Performance-Pay Model Shows No Achievement Edge

By Stephen Sawchuk — June 01, 2010 5 min read

Preliminary results from schools taking part in a Chicago program containing performance-based compensation for teachers show no evidence that the program has boosted student achievement on math and reading tests, compared with a group of similar, nonparticipating schools, an analysis released last week concludes.

The findings are notable not just for being at odds with other studies of the Teacher Advancement Program model, but also for the Chicago experiment’s unique background: During his tenure as chief executive officer of the district, Arne Duncan—now the U.S. secretary of education—oversaw the development of the initiative.

And through the economic-stimulus program’s Race to the Top and Teacher Incentive Fund competitions, the Obama administration has pushed states and districts to engage in changes to teacher-pay and -evaluation systems. Mr. Duncan has been a vocal supporter of the incentive-fund program, which provided some of the financing for the Chicago project.

The study by Mathematica Policy Research also found that the Chicago TAP, a local version of the national program, did not improve the rates of teacher retention in participating schools or in the district.

The findings from the Princeton, N.J.-based evaluation group cover the first two years of the program’s four-year implementation in select schools in the 409,000-student district.

At least one national independent study of TAP conducted in 2008 using a different research design found positive effects at the elementary level.

The new study offers few overt clues to explain its less-positive findings, and officials from both the district and from the U.S. Department of Education say it is premature to make a final judgment about the program’s impact.

Franklin R. Shuftan, a spokesman for the Chicago school system, noted that the district chose to focus on the professional-development aspect of the complex program, rather than on its bonus-pay feature, during the first two years of implementation.

“The report acknowledges that programs such as TAP take time to change attitudes and alter a school’s culture,” he said. “Measurables such as test scores and teacher retention might be better thought of as longer-term or final outcomes.”

Federal officials took a similar tack. “We know TAP and other reforms are hard work. We can’t expect immediate results,” said Peter Cunningham, a spokesman for the federal Education Department. “That’s why we’re committed to evaluating programs over the long term and identifying ones that deliver the results for children.”

Comparison Group

Seven of the 34 current federal incentive-fund grantees use a form of the TAP model in their programs. In addition to compensation bonuses, the program includes professional development keyed to a teacher-evaluation framework and a career ladder for teachers.

The Chicago program was designed jointly by the Chicago Teachers’ Union and the district, among other partners.

To determine what would have happened in the absence of Chicago’s TAP, analysts compared results from the 16 participating schools with a group of 200 schools with similar student demographics, teacher-retention rates, accountability status, and student-achievement levels.

They found no statistically significant differences in student scores or in teacher-retention rates among those schools.

The program’s impact also did not appear to increase after an additional year of experience. Several participating schools delayed implementation of TAP by a year so that, using a random-assignment methodology, the researchers could discern whether differences between schools with two years of experience in TAP and those with only one year could be attributed to the program.

They discovered that schools with only one year of exposure to TAP actually had slightly higher reading and math test scores, on average, than did the schools in their second year of implementation.

The Chicago findings are difficult to generalize to other TAP sites, as officials there did not adhere to all aspects of the TAP model. For one, the district spread the bonus funding to staff beyond teachers. (“TAP: More Than Performance Pay,” April 1, 2009.)

Because of problems with obtaining student-growth data linked to individual teachers, Chicago also paid bonuses based on schoolwide-, rather than classroom-achievement growth. The National Institute for Excellence in Teaching, which operates TAP, recommends that at least 30 percent of bonus pay be based on the results of classroom measures of student progress.

The study also notes that Chicago’s incentive payouts, which were lower than those in other programs nationwide, were not as heavily emphasized as other components. Payouts for teachers under the program averaged $1,100 for those in schools in their first year of implementation and $2,600 for their second year of TAP. That’s lower than the $2,000 and $4,000 per-teacher target set by the district.

Shortly after the release of the findings, the Santa Monica, Calif.-based institute commissioned an independent group, Interactive Inc,. to help determine why theChicago program didn’t produce results.

The findings could influence the larger field of performance pay, since they are likely to be scrutinized by districts hoping to win a cut of funds in the next round of federal incentive grants, said Steven Glazerman, a senior researcher at Mathematica.

“You have to wonder whether the result would have been different if the payouts had been larger or more meaningfully differentiated,” Mr. Glazerman said.

Guidelines for the next round of federal Teacher Incentive Fundprogramming do not specify the amount of incentive bonuses or how heavily to weigh growth in student test scores. But they do specify that average bonus payouts for educators should be “substantial,” perhaps 5 percent of the average teacher salary, and that top-performing educators should earn far beyond that amount, perhaps three times as much.

Mathematica plans to issue two more reports in coming years as it continues to evaluate the program.

A version of this article appeared in the June 09, 2010 edition of Education Week as Merit-Pay Model Pushed by Duncan Shows No Achievement Edge


Jobs The EdWeek Top School Jobs Virtual Career Fair
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Mathematics Webinar
Engaging Young Students to Accelerate Math Learning
Join learning scientists and inspiring district leaders, for a timely panel discussion addressing a school district’s approach to doubling and tripling Math gains during Covid. What started as a goal to address learning gaps in
Content provided by Age of Learning & Digital Promise, Harlingen CISD
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
How to Power Your Curriculum With Digital Books
Register for this can’t miss session looking at best practices for utilizing digital books to support their curriculum.
Content provided by OverDrive

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Accountability Biden Education Team Squashes States' Push to Nix All Tests but Approves Other Flexibility
The department has telegraphed its decision to deny states' requests to cancel federally mandated tests for weeks.
3 min read
A first-grader learns keyboarding skills at Bayview Elementary School in San Pablo, Calif on March 12, 2015. Schools around the country are teaching students as young as 6 years old, basic typing and other keyboarding skills. The Common Core education standards adopted by a majority of states call for students to be able to use technology to research, write and give oral presentations, but the imperative for educators arrived with the introduction of standardized tests that are taken on computers instead of with paper and pencils.
The U.S. Department of Education denied some states' requests to cancel standardized tests this year. Others are seeking flexibility from some testing requirements, rather than skipping the assessments altogether.
Eric Risberg/AP
Accountability Explainer Will There Be Standardized Tests This Year? 8 Questions Answered
Educators want to know: Will the exams happen? If so, what will they look like, and how will the results be used?
12 min read
Students testing.
Accountability Opinion What Should School Accountability Look Like in a Time of COVID-19?
Remote learning is not like in person, and after nine months of it, data are revealing how harmful COVID-19 has been to children's learning.
6 min read
Image shows a speech bubble divided into 4 overlapping, connecting parts.
DigitalVision Vectors/Getty and Laura Baker/Education Week
Accountability State Schools Chiefs Push Biden for Wiggle Room on Accountability During Pandemic
State schools chiefs say it's necessary to change how they use scores from mandated annual tests during the unprecedented disruption created by the coronavirus pandemic.
4 min read
Image of students taking a test.