School Choice & Charters Opinion

Achievement Gains in Voucher Research

By Martin Carnoy — February 28, 2001 6 min read
  • Save to favorites
  • Print

Earlier this month in these pages, William G. Howell, Paul E. Peterson, Patrick J. Wolf, and David E. Campbell defended their voucher research by responding to critics. (“In Defense of Our Voucher Research,” Commentary, Feb. 7, 2001.) As one who has found fault with their two-year, three-city study comparing pupils who received vouchers with those who did not, I am still not convinced. Here are just two of the reasons why.

Messrs. Howell, Peterson, Wolf, and Campbell claim that we should look at overall gains by ethnic group rather than grade-by-grade results.

When half the original sample in an experiment does not show up for the post-test, we really have little idea whether the estimated gains scores are biased or not.

This would be a reasonable demand if the pattern of gains were consistent across grades, and if gains were statistically significant in at least several grades in each city. In New York City, as reported by Mathematica Policy Research, the contractor on the study, only the one cohort of African- Americans who used vouchers to attend private school 5th grades in the study’s first year and 6th grades in the study’s second year showed significant test- score gains when compared with the same cohort who did not receive vouchers. Their entire gain was made in the study’s first year.

In Dayton, Ohio, according to Mr. Howell, who was kind enough to provide me unpublished data by grade, African-American voucher recipients finishing private school 3rd grades at the end of the study’s second year made very large two-year gains (in combined math and reading). Those finishing 4th grade performed slightly worse than pupils who did not get vouchers. Those finishing private school 5th grades made large gains, those finishing private school 6th grades, small losses, those finishing 7th grade large gains, and those finishing 8th and 9th grades, large losses compared to nonvoucher students. Only in Washington were achievement gains of voucher recipients attending private schools relatively consistent across grades.

By putting all African-American students together, the researchers get a positive, significant private school effect in each city; but in two of the cities (New York and Dayton), combining results obscures possible cohort effects that may have little to do with differences between public and private education.

A second and even more serious issue that clouds the research is nonresponse. In New York City, about 40 percent, and in Washington and Dayton, about one-half of voucher recipients and nonrecipients failed to show up for the follow-up first- and second-year tests. The researchers made an attempt to weight the scores of those who did show up to make them representative of the overall sample. They argue that they are able to “fill in” missing second- and third-round test scores implicity, based on the personal data gathered on voucher recipients and nonrecipients in the first, baseline round of testing.

But students who did not show up for the second and third round of testing may have done so based on how they performed during the school year, not just based on their personal characteristics and baseline test scores, and may have done so differently in the voucher-recipient and -nonrecipient samples. Clearly, many “unobservable” variables are important in predicting how a student performs on a particular test on a particular day, including how he or she expects to perform. When half the original sample in an experiment does not show up for the post- test, and we can only predict part of the variation of the outcome variable to fill in for the missing observations, we really have little idea whether the estimated gains scores are biased or not.

The nonresponse rate increases between the first and second follow-up tests, especially in Washington. There, almost 25 percent fewer students participated in the second follow-up than in the first. Gains are particularly volatile in Washington among middle school students. They suffered large losses in scores at the end of the first year and made enormous gains in the second. Was this due to a fraction of very low- scoring voucher students leaving the sample between the first and second years? Did some leave private school, or just not show up for the test? The researchers could easily answer this question with further analysis.

Such debates among researchers studying the same data are not only healthy; they are absolutely crucial to developing good public policy.

Indeed, they could go one better. They could release their data so that other researchers could subject the data to the scrutiny they deserve. This is exactly what one of the researchers, Paul Peterson, demanded of University of Wisconsin-Milwaukee professor John Witte in 1996, when it came to data from the Milwaukee voucher experiment. Once Mr. Peterson got the data, his analysis was different from Mr. Witte’s. Subsequently, Princeton University researcher Cecilia Rouse reanalyzed the data yet again and got another set of results. Such debates among researchers studying the same data are not only healthy; they are absolutely crucial to developing good public policy.

It is distinctly possible that a few hundred low-income African-American students, randomly drawn from families highly dissatisfied with Washington’s public schools and placed in one of the District of Columbia’s existing private schools with a group of higher-performing students, will make gains when compared with students who remain in public schools. It is also possible that these gains are partly due to characteristics of the private schools, not just the new set of peers surrounding the sample of voucher students. Right now, though, even this case—the most plausible based on the available results from the three cities—is still sufficiently riddled with potential estimation bias that it may or may not reflect the “true” effect of using a voucher in a private school.

Messrs. Howell, Peterson, Wolf, and Campbell cite Carolyn Hoxby’s description of randomized field trials as the “gold standard” of social science research, and want to bask in its glow. I agree with Ms. Hoxby’s characterization. But to the extent that the researchers’ results depend upon instrumental variables and upon weighting to correct for nonparticipation, they are no longer reporting results of a randomized field trial, but rather are reporting an empirical study with many of the dangers of assuming the relevance of characteristics that the “gold standard” attempts to avoid.

The Tennessee class-size study, to which they liken their own, suffers from similar problems, but to a lesser extent. In the Tennessee STAR study, all initial participants had identifiers, and could be tracked subsequent to their participation, even if they dropped out of the study, provided they remained within the state. As virtually all students who left the experimental schools and classrooms did remain in Tennessee, the instrumental variable of initial assignment could be utilized without the necessity of a more speculative weighting based on presumed similarities in family characteristics. Because Messrs. Howell, Peterson, Wolf, and Campbell had no similar way of tracking those who dropped out of their studies, their abandonment of the “gold standard” is necessarily more severe.

Martin Carnoy is a professor of education and economics at Stanford University. His original critique of the Howell, Peterson, Wolf, and Campbell research appeared in The American Prospect, Jan. 15, 2001. A more detailed version is available from the Economic Policy Institute in Washington.

Related Tags:

A version of this article appeared in the February 28, 2001 edition of Education Week as Achievement Gains in Voucher Research


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Classroom Technology Webinar
Seamless Integrations for Engagement in the Classroom
Learn how to seamlessly integrate new technologies into your classroom to support student engagement. 
Content provided by GoGuardian
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Recruitment & Retention Webinar
Be the Change: Strategies to Make Year-Round Hiring Happen
Learn how to leverage actionable insights to diversify your recruiting efforts and successfully deploy a year-round recruiting plan.
Content provided by Frontline
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Equity & Diversity Webinar
Critical Ways Leaders Can Build a Culture of Belonging and Achievement
Explore innovative practices for using technology to build an environment of belonging and achievement for all staff and students.
Content provided by DreamBox Learning

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School Choice & Charters Opinion The Biden Administration Is Right: Charters Need to Be More Accountable
The proposed changes to the federal Charter School Program are just common sense, write Jitu Brown and Randi Weingarten.
Jitu Brown & Randi Weingarten
3 min read
Illustration of students and teachers holding puzzle pieces.
<b>F. Sheehan/Education Week and iStock/Getty</b>
School Choice & Charters What's Behind the Fight Over the Biden Administration's Stance on Charter School Funding
Proposed new rules for federal charter school funding have drawn the ire of many in the charter school community.
8 min read
Publish Charter school parents stage a counter protest as thousands of public school teachers, administrators and supports march through the streets of Sacramento during a protest held at the California State Capitol urging state legislators to provide more funding for public schools in Sacramento, Calif., on May 22, 2019.
Publish Charter school parents stage a counter protest during a march in Sacramento, Calif., that advocated for more funding for public schools in 2019.
Jessica Christian/San Francisco Chronicle via AP
School Choice & Charters Opinion Families May Like Their School But Want More Options. That’s Where Course Choice Comes In
Educational choices have grown inside each school as a result of the pandemic. Families should take advantage of this.
3 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
School Choice & Charters Mich. Public School Advocates Launch Effort to Stop DeVos-Backed Proposal
The former secretary of education is backing an initiative that advocates say would create an unconstitutional voucher system.
Samuel J. Robinson, mlive.com
4 min read
Student with backpack.