As part of a fiscal 2017 budget deal, Congress plans to forbid the use of randomized, controlled trials in new evaluations of the only federal school voucher program.
The move comes a week after an experimental-design study found negative results for students who accepted vouchers compared to students who applied but were not chosen for the program, but it has been in the works for more than a year. The change in future methods may have ripple effects not only for those debating school vouchers, but also for states and districts trying to work out how to identify successful programs under the Every Student Succeeds Act.
“I think it’s fair to say it sends a signal about evidence that’s inconsistent with ESSA,” said Mark Dynarski, the lead author in the most recent evaluation of the voucher program. “ESSA is pretty clear about the role of evidence and experiments, and this does muddy the water.”
The 2017 omnibus spending bill to keep the government running through the end of September includes language requiring the U.S. Department of Education and the Institute of Education Sciences to evaluate the voucher program every year. But instead of using randomized, controlled trials, which have been used in prior evaluations of the program, it calls for the evaluations to use “an acceptable quasi-experimental research design ... that does not use a control study group consisting of students who applied for but did not receive opportunity scholarships.” Congress directs IES to compare students who get vouchers to a comparison group of “students with similar backgrounds in the District of Columbia public schools and the District of Columbia public charter schools.”
The budget deal does direct IES to continue the randomized trials that have already started, including the study that issued the report last week. Those studies will continue to compare scholarship applicants who did receive vouchers to those who didn’t, in terms of their short-term test scores and long-term college enrollment rates and outcomes.
“In some sense, it creates a window to continue following these kids. Congress is very keen on high school graduation and college. ... That’s a pretty important issue, when you already have indications that the voucher-using kids are more likely to graduate high school,” Dynarski said. Yet he noted that the current evaluation would have to continue follow-ups for years, since nearly a quarter of the students participating in the study began in kindergarten, and only 12 percent were in grade 8 or higher.
“It’s kind of going in different directions,” said Jon Baron, the vice president of evidence-based policy at the John and Laura Arnold Foundation. He had no comment on the legislation itself, but said districts and states have grown significantly in their capacity to conduct randomized trials in the last decade, particularly for programs like the voucher system that use lotteries to enroll students.
Change in Research Priority?
Under ESSA, randomized, controlled trials are considered the “gold standard” and the strongest tier of evidence to identify effective programs. This is the first time since the law was passed that Congress has specifically requested a less-rigorous evaluation method for a program.
The federal voucher program, which awards students based on a lottery of applicants, is uniquely suited to use a comparison group of students who applied for the scholarship but didn’t get it. Doing so removes the potential for selection bias, as students and families who put forth the effort to apply for a voucher may differ from those who don’t.
But evaluations of the voucher program have had their share of problems. Attrition from the program has not been equal between voucher holders and those in the control group, creating sampling differences. And in 2011, the program had enough spots for all its students, so it didn’t use a lottery at all. Critics of the evaluation have pointed out that follow-ups can be a hardship for students who did not get vouchers.
Baron noted that even using a quasi-experimental study, researchers will still have to test students and collect follow-up information from those in public and private schools. “We fund a lot of randomized, control trials, 35 to 40 in the field right now ... and more than 60 percent of those are school-based evaluations,” Baron said.
And Dynarski agreed. “Some of these issues might influence findings in one direction, and some might go in the other direction,” he said. “Considering the topic in question, and in the context of having used strong scientific methods for the previous study and the current one, it’s at least unclear why now is the time to swap out strong methods and swap in weaker ones.”