Teaching Profession

D.C. Teachers Improved After Overhaul of Evaluations, Pay

By Stephen Sawchuk — October 22, 2013 4 min read
  • Save to favorites
  • Print

The District of Columbia’s closely watched system for evaluating teachers and providing bonus pay appears to have motivated weak teachers to make improvements and spurred already-effective teachers to higher levels of performance, a new study concludes.

Teachers on the cusp of dismissal under the IMPACT evaluation system in the nation’s capital improved their performance by statistically significant margins, as did those on the cusp of winning a large financial bonus, according to the study, published as a working paper last week by the Cambridge, Mass.-based National Bureau of Economic Research.

The provocative study raises new questions about how recently revamped teacher-evaluation systems—and pay schedules linked to them—shape teacher behavior. It is among the first research to look at teacher evaluation empirically.

As an actual policy, not a pilot program, IMPACT was different in scope from other incentive programs. Teachers with lower initial ratings got access to instructional coaches to improve areas of their practice, one possible explanation for the gains.

“All of the actors involved are taking this system seriously, and it’s in those situations where you might expect greater responses to the incentives,” said James H. Wyckoff, a University of Virginia professor of education and policy and one of the study’s authors.

The results apply only to the populations close to the performance thresholds studied. They do not depict an “overall” effect of IMPACT.

Dividing Line

Established in the 2009-10 school year, the district’s IMPACT evaluation system relies on a complex mix of factors to score each teacher, including both multiple observations and measures of student achievement. Teachers deemed ineffective under the system can be dismissed, while those scoring at the “minimally effective” level, the second-lowest, get one year to improve. Teachers who earn the “highly effective” rating are eligible for bonuses of up to $25,000.

Since its rollout, IMPACT has led to the dismissal of several hundred teachers. The system, instituted by then-Chancellor Michelle A. Rhee, began just as the federal Race to the Top program brought teacher evaluation greater focus from policymakers.

Though IMPACT has won praise in policy circles, some teachers have been less happy with the system. In surveys and focus groups, they have complained about feeling monitored and expressed concerns that it’s harder for teachers in the most challenging schools to get top scores.

For the study, Mr. Wyckoff and Stanford University professor Thomas Dee analyzed records for district teachers and their students from 2009-10 through 2011-12. They examined those teachers falling just above or just below the dividing line separating “minimally effective” from “effective” teachers and compared how they fared on subsequent evaluations. Similarly, they analyzed the performance of those teachers just above and just below the decision point for awarding the financial bonuses, and compared how they did in the next year’s evaluation scoring.

The methodology exploits the similarities in characteristics between groups on either side of the cutoff, allowing their subsequent performance to be analyzed as a function of how they responded to their initial IMPACT scores. The design helps to rule out other factors that could have affected the results, though it is not considered as conclusive as a true randomized experiment.

The researchers found that in the 2011-12 year, those teachers at risk of losing their jobs improved their performance by about 13 evaluation points. Teachers on the cusp of winning a performance bonus also improved by about 11 IMPACT points. Such gains are about 50 percent and 41 percent, respectively, of the growth the average district teacher made in his or her first three years on the job.

Kaya Henderson, the chancellor of the District of Columbia school system, said the study validates IMPACT’s theory of action: to reward the best teachers, help struggling ones, and dismiss those unable to perform.

“The simple fact that this study tells us that the things we set out to do on our evaluation system are getting done makes me super proud and hopeful about where we’re going,” she said.

Adding to the Literature

The study also found that teachers deemed minimally effective in 2010-11 were less likely to stay in their jobs, with their retention rate falling by about 11 percentage points.

Rates of teacher turnover in the district have also earned the school system its fair share of criticism, although the paper shows that highly effective teachers had high overall retention rates. Also, the study notes, teachers newly hired in 2011-12 had average IMPACT scores that were more than 25 points higher than the teachers who left.

The study’s suggestion that a high-stakes evaluation system linked to pay decisions and professional development can be a motivating factor for teachers to improve contrasts with a run of recent studies on bonus-pay programs showing little or no effects overall on teacher behavior.

But the study did not examine in detail whether the teacher improvements translated to higher student performance, a criticism local union officials highlighted. “The goal of IMPACT was to improve student achievement, but in fact, [overall] student achievement has been flat,” Washington Teachers Union President Elizabeth Davis said.

Still others praised the study’s design but pushed back on the idea that teacher evaluations of this sort will lead to widespread improvements in teaching quality.

“Put simply, what this study says is that if we take a group of otherwise similar teachers and randomly label some as ‘OK’ and tell others they suck and their jobs are on the line, the latter group is more likely to seek employment elsewhere,” wrote Bruce D. Baker, an education professor at Rutgers University in New Brunswick, N.J., on his blog. “No big revelation there, and certainly no evidence that D.C. IMPACT ‘works.’ ”

Coverage of policy efforts to improve the teaching profession is supported by a grant from the Joyce Foundation, at www.joycefdn.org/Programs/Education. Education Week retains sole editorial control over the content of this coverage.
A version of this article appeared in the October 09, 2013 edition of Education Week as D.C. Teachers Improved After Overhaul of Evaluations, Pay

Events

English-Language Learners Webinar Helping English-Learners Through Improved Parent Outreach: Strategies That Work
Communicating with families is key to helping students thrive – and that’s become even more apparent during a pandemic that’s upended student well-being and forced constant logistical changes in schools. Educators should pay particular attention
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Mathematics Webinar
Addressing Unfinished Learning in Math: Providing Tutoring at Scale
Most states as well as the federal government have landed on tutoring as a key strategy to address unfinished learning from the pandemic. Take math, for example. Studies have found that students lost more ground
Content provided by Yup Math Tutoring
Classroom Technology Webinar Building Better Blended Learning in K-12 Schools
The pandemic and the increasing use of technology in K-12 education it prompted has added renewed energy to the blended learning movement as most students are now learning in school buildings (and will likely continue

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Teaching Profession Teachers Who Refuse to Comply With Vaccine Mandates Won't Face Consequences in Many Places
Some districts and states aren't even keeping track of how many teachers are vaccinated.
8 min read
Teachers protest against COVID-19 vaccination mandates in New York on Aug. 25, 2021. On Friday, Oct. 1, 2021, U.S. Supreme Court Justice Sonia Sotomayor denied an emergency appeal from a group of teachers to block New York City's COVID-19 vaccine mandate for public school teachers and other staff from going into effect.
Teachers protest against COVID-19 vaccination mandates in New York on Aug. 25.
Mary Altaffer/AP
Teaching Profession What New Teachers Need
Ideas from the real world on making teachers' first years less overwhelming and more fulfilling.
5 min read
Illustration of a classroom diorama sitting on a student desk.
Illustration by Laura Baker/Education Week (Images: iStock/Getty)
Teaching Profession Opinion This Year Almost Drove Me Out of Teaching. The Right Leader Made Me Stay
After seven years teaching and one class away from becoming an education specialist, I have seen the highs and lows of education leadership.
Samantha Richardson
4 min read
Illustration of woman sitting on a mountain top looking into the distant landscape.
iStock/Getty
Teaching Profession Maryland Teacher Wins $1 Million Global Prize
Keishia Thorpe received the prize for her work teaching immigrant and refugee students and helping them attend college.
2 min read
This photo provided by the Varkey Foundation shows Keishia Thorpe. The Maryland high school English teacher, who has worked to open up college education for her students, has won the $1 million Global Teacher Prize. The Varkey Foundation announced Wednesday, Nov. 10, 2021, that Thorpe, who teaches at International High School at Langley Park in Prince George’s County in Maryland, was selected from more than 8,000 nominations and applications from 121 countries around the world.
Keishia Thorpe, who has worked to open up college education for her students, has won the $1 million Global Teacher Prize.
Varkey Foundation via AP