Teaching Profession

‘Value Added’ Gauge of Teaching Probed

By Debra Viadero — July 01, 2009 7 min read
BRIC ARCHIVE
  • Save to favorites
  • Print

Includes updates and/or revisions.

A new study by a public and labor economist suggests that “value added” methods for determining the effectiveness of classroom teachers are built on some shaky assumptions and may be misleading.

The study, due to be published in February in the Quarterly Journal of Economics, is the first of a handful of papers now in the publishing pipeline that are widely seen to be contributing important evidence, both pro and con, regarding the use of value-added assessments, which gauge the effectiveness of schools and teachers by measuring the gains that their students make on standardized tests over the course of a school year.

A small number of states have used value-added measures since the early 1990s to track how well their schools are doing. But the method is coming in for new scrutiny amid growing interest from politicians in performance-pay plans for teachers.

Proponents say value-added measures provide a fair way of deciding which teachers deserve financial rewards by objectively measuring the learning gains students make from fall to spring, rather than students’ absolute achievement levels. By gauging that progress during the school year, proponents reason, teachers are not getting unjust blame for the learning deficits that students bring to their classes or undue rewards for being blessed with a class of high achievers.

The Quarterly Journal study was conducted by Jesse Rothstein, who left Princeton University recently to become an associate professor of public policy at the University of California, Berkeley. Mr. Rothstein, drawing on data for 99,000 North Carolina students who were in 5th grade during the 2000-01 school year, makes his case by using a “falsification” test. For example, he asked, what effect do 5th grade teachers have on their students’ test scores in 3rd and 4th grades?

Because it’s impossible for even the best teachers to have an impact on students’ previous learning, Mr. Rothstein reasons, there should be no impact.

Not Random?

But Mr. Rothstein’s analysis, using three different value-added models, found quite large effects—even though he was comparing teachers in the same years, grades, and schools.

To explain the findings, he suggested that students may well not have been randomly assigned to classrooms. Instead, they may have been sorted into classes based in some way on their prior achievement. A principal might, for example, assign students with behavior problems to teachers known to have a way with problem students or reward more senior teachers with high achievers.

“Anybody who’s had a kid in elementary school has tried to exert some influence over that kind of nonrandom assignment,” said Mr. Rothstein. Yet, he added, value-added calculations are based on the assumption that students’ classroom assignments are random, overlooking the day-to-day reality of what happens in schools.

Mr. Rothstein also looks at teachers’ long-run effects on individual students and finds that they tend to decay or fade out after the first year. He also finds wide variation in teachers’ effectiveness over time. For example, only about a third of the teachers who landed in the top quintile of the study sample based on their accumulated effects over two years fell in that same category based on their one-year effects. That suggests to Mr. Rothstein that some of those teachers were misclassified.

“There’s a tendency to assume these value-added scores may be noisy, but they are measures of teachers’ true effects,” Mr. Rothstein said. “We can’t make that leap as quickly as we’d like to.”

His study was posted online last month by Princeton’s Woodrow Wilson School of Public and International Affairs.

Cory R. Koedel, an assistant professor of economics at the University of Missouri in Columbia, said Mr. Rothstein’s paper is “huge in the literature, because he’s pointing out some things the literature had overlooked.”

In a paper that is being revised for publication in the journal Education Finance and Policy, Mr. Koedel and his research partner, Julian R. Betts of the University of California, San Diego, found that researchers can mitigate some of the effects of student-sorting bias by incorporating more years of data from teachers’ classes into value-added calculations.

To make their case, the researchers tried to replicate Mr. Rothstein’s techniques with testing data from four waves of 4th graders moving through San Diego public schools between the 1998-99 and 2001-02 school years. Although they found similar signs of sorting bias with one year of test scores, the size of that effect was reduced with more years of data.

For that reason, the researchers conclude, policymakers may want to be cautious about using achievement gains from a first-year teacher to make important decisions about salaries or retention. Many districts make teachertenure decisions in less than three or four years, experts said.

But in practice, sorting biases may not be as problematic as some experts think, assert the authors of a third article in the publishing pipeline.

An Experiment in L.A.

For their study, researchers Thomas J. Kane and Douglas O. Staiger use data from Los Angeles public schools to conduct what is widely considered to be the first experimental test of value-added modeling. The paper is being revised for possible publication in a not-yet-named economics journal, according to Mr. Staiger, who is an economics professor at Dartmouth College in Hanover, N.H.

The study was part of a larger evaluation of the National Board for Professional Teaching Standards, an Arlington, Va.-based group that confers special status on teachers who demonstrate that they can meet its criteria, in Los Angeles elementary schools. Researchers used data from the study to determine if value-added calculations could yield the same results on teachers’ effectiveness as an experiment in which students are randomly assigned to teachers.

To find out, the researchers recruited 78 principals of elementary schools in which one or more teachers had applied to be certified by the national board.

The principals identified experienced teachers in the same grades who were comparable to the nbpts applicants and then made up rosters of classes that they would be comfortable assigning to either the comparison teacher or the nbpts applicant. To further control for any possible student-assignment bias, district administrators later randomly switched

the rosters and notified principals that the switch had occurred.

In most cases, the researchers found, the value-added calculations matched the experimental results: They accurately identified which of the teachers in each school seemed to do a better job of producing bigger learning gains.

“What our paper suggests is that controlling for students’ prior-year setting is enough to eliminate the bias problem. Still, I think I want to see a lot more experiments like ours done in other settings,” said Mr. Staiger. “In our experiment, all you could learn about was two teachers in the same schools where the schools were also willing to randomize them.”

The 78 principals in the experiment, Mr. Staiger noted, were the only ones who agreed to the random-assignment procedure from among 200 principals who were invited to take part.

The study also found, as Mr. Rothstein did, that the teacher effects faded in their students’ performance from one year to the next, which may be the more important issue, according to Mr. Staiger.

“When calculating the potential value of shifting the teacher-effectiveness distribution, we and others have typically assumed that the effects of a strong teacher persist in the children they teach,” write Mr. Staiger and Mr. Kane, who is the faculty director of the Seattle-based Bill & Melinda Gates Foundation’s project on policy innovation.

“Critics will point to Jesse’s paper and say value-added doesn’t solve the [bias] problem,” said Douglas N. Harris, an economist at the University of Wisconsin-Madison. “Defenders will say, well, we’ve got this experiment. ....We’ve got Koedel and Betts saying the problem can be reduced. We still have the reliability problem, but it’s better than the alternatives.”

“Ultimately, it’s probably even the wrong debate,” Mr. Harris added. “We need to know how to use the measure in practice to improve school performance.”

Experts say a fourth experiment to get under way this fall may help shed light on how to do that.

Researchers at Mathematica Policy Research Inc. hope to randomly assign teachers with high value-added scores to classes in disadvantaged, hard-to-staff schools in seven districts. The target teachers are being offered $20,000 bonuses to agree to move to the disadvantaged schools and stay for two years.

At random, principals will fill existing vacancies by either recruiting from the “high value-added” pool or using their normal hiring procedures. But, according to Steven Glazerman, who is leading the study for the Princeton, N.J.-based research group, the final results are still a few years away.

A version of this article appeared in the July 15, 2009 edition of Education Week as ‘Value Added’ Gauge of Teaching Probed

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Start Strong With Solid SEL Implementation: Success Strategies for the New School Year
Join Satchel Pulse to learn why implementing a solid SEL program at the beginning of the year will deliver maximum impact to your students.
Content provided by Satchel Pulse
Teaching Live Online Discussion Seat at the Table: How Can We Help Students Feel Connected to School?
Get strategies for your struggles with student engagement. Bring questions for our expert panel. Help students recover the joy of learning.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Science Webinar
Real-World Problem Solving: How Invention Education Drives Student Learning
Hear from student inventors and K-12 teachers about how invention education enhances learning, opens minds, and preps students for the future.
Content provided by The Lemelson Foundation

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Teaching Profession How Teachers Are Spending Their Summer Vacation
Swimming, hiking, and an occasional academic project are on the agenda.
1 min read
Lifeguards watch over children and their families as they enjoy the shallow end of the Woodson Family Aquatic Center on the opening day of the 2022 pool season Saturday, May 28, 2022 in Odessa, Texas.
Lifeguards watch over children and their families at the Woodson Family Aquatic Center as pool season opens in Odessa, Texas.
Eli Hartman/Odessa American via AP
Teaching Profession Letter to the Editor Can Educators Agree to Disagree Respectfully?
We must acknowledge that there are strong, defensible differences in perspectives about divisive topics, writes an educator.
1 min read
Illustration of an open laptop receiving an email.
iStock/Getty
Teaching Profession Q&A The First 5 Years in the Classroom Are Tough. This Teacher Has Ideas to Lessen the Burden
A middle school teacher talks about why educators need to share stories about their jobs—and find schools that reflect their values.
7 min read
Patrick Harris
Patrick Harris
Teaching Profession Teachers in Texas Shooting Died Trying to Shield Students, Their Families Say
Eva Mireles and Irma Garcia, both veteran teachers, co-taught a 4th grade class at their Uvalde, Texas, elementary school.
3 min read
Fourth grade co-teachers Irma Garcia, left, and Eva Mireles.
Fourth grade co-teachers Irma Garcia, left, and Eva Mireles, were killed in the school shooting in Uvalde, Texas, alongside 19 children.
Courtesy of Uvalde CISD