Opinion Blog

Rick Hess Straight Up

Education policy maven Rick Hess of the American Enterprise Institute think tank offers straight talk on matters of policy, politics, research, and reform. Read more from this blog.

School Choice & Charters Opinion

Making Sense of the Lousy Test Results in Louisiana’s Voucher Program

By Rick Hess — April 29, 2019 3 min read
  • Save to favorites
  • Print

Last week, after four years spent evaluating the Louisiana Scholarship Program (LSP), the School Choice Demonstration Project issued a comprehensive summary of the findings—and the results were pretty grim.

The research team, led by University of Arkansas professor Patrick Wolf, concluded: “Overall, participating in the LSP had a statistically significant negative impact on student English Language Arts (ELA) and math scores across most years of the evaluation, including the fourth year, and across most samples of students studied.” Because Wolf has earned a reputation for both scholarly rigor and fair-mindedness when it comes to evaluating school choice, the negative findings cannot be lightly dismissed as the handiwork of anti-voucher ideologues.

What to make of these results, however, is less straightforward than the findings themselves. Wolf et al. explain that things are more complicated than the overall findings might suggest.

First, test scores are an imperfect gauge of outcomes, as Wolf et al. make clear. For instance, while LSP students posted lower test scores than did their peers, there was no commensurate effect on whether or not participating students wound up enrolling in college. It may be that students are benefiting in ways that don’t show up on the state tests.

Second, it also turned out that the test effects were only half as large when students were assessed using the iLEAP test (which is less aligned to the public school curriculum) than the LEAP (which is more aligned). This means that a substantial chunk of voucher underperformance on state tests might simply reflect that private school students spend a lot less time focusing on preparing for state tests than do students in public schools. What’s striking is that this question of how to make sense of movement in test scores is exactly the same one that arises—and is too often given short-shrift—when it comes to district schools.

Now, given that some choice advocates once insisted that test scores proved that school choice “works,” critics have a point in calling out those advocates who’ve flipped and now insist that it was never about test scores. But I feel comfortable setting that discussion aside, given that I’ve never used test scores to argue for school choice. Indeed, as I put it nearly a decade ago in “Does School Choice ‘Work’?”, “The mixed findings suggest that simply legislating ‘school choice’ programs, or enrolling a child in a charter school, will have no obvious short-term impact on achievement.” I feel like I’ve been pretty consistent, given that I observed in that essay:

For the past 20 years, the question of whether school choice 'works' has been understood to mean simply whether a school-choice program boosted reading and math test scores in a given year . . . Particularly problematic is how this way of thinking has caused school-choice proponents to ignore crucial questions of market design and implementation.

Third, the Louisiana program is distinctive in the substantial regulations it imposes upon participating private schools—including the requirement that they employ an open admissions policy. A crucial question is whether well-meaning regulations deter quality schools from accepting voucher students. For instance, Wolf et al. note:

To participate in the [LSP], private schools must meet certain criteria related to: enrollment, financial practices, student mobility, and the health, safety and welfare of students. Participating schools are prohibited from being selective in their enrollment of voucher students and must administer the state's accountability tests annually to voucher students in grades 3 through 8 and grade 10.

Anyone who has spent much time in or around schools knows that even seemingly innocuous criteria relating to “student health” or “safety” can entail onerous paper chases, complicated compliance, intrusive micromanagement, and disruption. This is why well-meaning rules may deter high-quality schools from participating—especially when those schools find it easy to fill up their seats without accepting voucher students. When that happens, the participating schools will tend to be those most desperate to add enrollment. If that’s what is leading to lousy performance, one conclusion is not that the scholarships were bad for kids—but that scholarships only help when kids can use them to attend good schools.

This all brings us back to something I wrote about last week, which is why obsessing on “What works?” can do more to obscure than illuminate why and how things work. My hat is off to Wolf and his colleagues for their invaluable work, but we would do well to recognize that their findings raise at least as many questions as they answer.

Related Tags:

The opinions expressed in Rick Hess Straight Up are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.

Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
6 Key Trends in Teaching and Learning
As we enter the third school year affected by the pandemic—and a return to the classroom for many—we come better prepared, but questions remain. How will the last year impact teaching and learning this school
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Attendance Awareness Month: The Research Behind Effective Interventions
More than a year has passed since American schools were abruptly closed to halt the spread of COVID-19. Many children have been out of regular school for most, or even all, of that time. Some
Content provided by AllHere
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Ensuring Continuity of Learning: How to Prepare for the Next Disruption
Across the country, K-12 schools and districts are, again, considering how to ensure effective continuity of learning in the face of emerging COVID variants, politicized debates, and more. Learn from Alexandria City Public Schools superintendent
Content provided by Class

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School Choice & Charters Virtual Charters in Hot Water Again. Accusations of Fraud Prompt $150M Lawsuit
Indiana officials seek to recoup more than $150 million they say was either wrongly obtained or misspent by a consortium of virtual schools.
Arika Herron, The Indianapolis Star
2 min read
Indiana's attorney general Todd Rokita speaks at a news conference on Sept. 16, 2020, in Indianapolis. Rokita filed a lawsuit against a group of online charter schools accused of defrauding the state out of millions of dollars Thursday, July 8, 2021.
Indiana's attorney general Todd Rokita speaks at a news conference on Sept. 16, 2020, in Indianapolis.
Darron Cummings/AP
School Choice & Charters How the Pandemic Helped Fuel the Private School Choice Movement
State lawmakers got a new talking point as they pushed to create and expand programs to send students to private schools.
8 min read
Collage showing two boys in classroom during pandemic wearing masks with cropped photo of feet and arrows going in different directions.
Collage by Gina Tomko/EducationWeek (Images: Getty)
School Choice & Charters Opinion Taking Stock After 30 Years of Charter Schools
Rick Hess speaks with Nina Rees, president and CEO of the National Alliance for Public Charter Schools, on charter schools turning 30.
8 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
School Choice & Charters In Fight Over Millions of Dollars for Charter Schools, a Marijuana Tax May Bring Peace
The Oklahoma State Board of Education voted unanimously to rescind a polarizing lawsuit settlement, pending certain stipulations.
Nuria Martinez-Keel, The Oklahoman
3 min read
Money bills cash funds close up Getty