Usually big edu-news doesn’t break during Christmas week. But, on Monday, DC Public Schools officials announced some troubling news concerning their acclaimed IMPACT teacher evaluation program. As the Washington Post’s savvy Nick Anderson reported, “Faulty calculations of the ‘value’ that D.C. teachers added to student achievement in the last school year resulted in erroneous performance evaluations for 44 teachers, including one who was fired because of a low rating.”
In response, Elizabeth Davis, president of the Washington Teachers’ Union, said, “IMPACT needs to be reevaluated. The idea of attaching test scores to a teacher’s evaluation -- that idea needs to be junked.” DCPS chief of human capital Jason Kamras said, “We take these kind of things extremely seriously. Any mistake is unacceptable to us.”
First, here’s a bit on what DCPS had to say. In an e-mail, Kamras wrote, “We (DCPS) made no mistakes. Our vendor, Mathematica, incorrectly calculated the value-added scores” due to “a programming error.” He said the teachers involved constitute “about 1% of the [DCPS] teacher force,” with 22 of the flawed ratings too high and 22 too low. Kamras said, “We are holding harmless anyone who should have had a lower rating [and] we are moving up anyone who should have had a higher rating. Those folks will get all the benefits they’re entitled to.” He said “only 1 person was erroneously fired” and “we’ve already offered” to reinstate them.
As we head into 2014, with lots of states and districts rolling out or amping up new teacher evaluation systems, there are at least four points worth keeping in mind.
One, this should be a huge caution flag for teacher evaluation enthusiasts. Remember, IMPACT is the gold standard for teacher evaluation and pay. DCPS has taken the time, money, talent, and discipline to really go after the data and technology issues. Even so, DC has stumbled. In other places, I’m struck at how often I chat with district or state leaders whose take on the number-crunching, vendors, and data systems seems to be, “Hey, the experts will take care of that technical stuff.” (I’ve come to think of this as the HealthCare.gov response.) As we’ve seen with health care reform, that can be a mistake. Folks better be bucking up.
Two, let’s maintain a sense of proportion. Remember that Tom Dee and Jim Wyckoff released an influential NBER study just this fall, which found the system seems to be working pretty much as intended. To the extent that we view reading and math gain scores as one element of teacher effectiveness, the system seems to be both keeping the right teachers and helping teachers improve.
Three, these systems are nascent and fragile. Proponents need to do everything they can to show that these will be fair, reliable, and workable. After all, unions are going to be well within their rights to bring legal challenges, especially when there are concerns that systems are inequitable or unreliable. Good intentions and complex algorithms won’t suffice to trump those concerns. That’ll be a matter of how these systems are designed and executed. And the inconvenient truth is that it’s not a question of whether the results are more right than wrong--it’s a question of whether the flaws are minor and defensible enough to pass muster in the courts of law and of public opinion.
Four, on this count, DCPS’s response should help. After all, some mistakes are going to be inevitable. An important consideration is how to respond to them. DCPS here has seemingly done a responsible job of ‘fessing up, coming clean, and moving to clean up its mess. Let’s hope that’s the norm for districts and states which misstep on teacher evaluation in the year ahead.