Education Opinion

What We Don’t Know About Regulation and Vouchers

By Douglas N. Harris — January 18, 2016 7 min read
  • Save to favorites
  • Print

The debate on Louisiana vouchers and the Over-Regulation Theory continues . . . Jason Bedrick responded to my argument that there is very little evidence that over-regulation drives down voucher effects, and what evidence we do have is actually inconsistent with that theory.

First, let’s be clear about the argument. The Over-Regulation Theory is that government regulation is counter-productive--in this case, because it scares away potentially effective private schools from participating in voucher programs. The flip side is that regulation can keep potentially ineffective schools out of the program and/or induce schools to change what they do to align with state and public objectives. This round of debate got started because of the negative effects of vouchers found in Louisiana.

In defense of the Over-Regulation Theory, Bedrick is seemingly critical of the fact that I don’t have evidence to support my alternative theories for the negative effects. But this only highlights my main point: that there are many theories other than Over-Regulation to explain the large negative effects in Louisiana and we don’t have a lot of evidence on any of them. The goal of my first post was to explain additional potential explanations for the negative effects and to argue that we shouldn’t jump to conclusions.

One of my alternative theories is that this has to do with the testing regime, which relies on the state standardized test, the same one that traditional public and charter school students take. That approach is sensible because it implicitly asserts that the public has a set of goals for students that it is willing to provide funding for, and it can hold different types of schools accountable in a common fashion for meeting those objectives.

But a further implication is that, at least in the short run, the private school curriculum is probably not aligned with state standards. It doesn’t take much evidence to show that if I teach you one thing and you take a test on something else, then you are going to do badly on the test even if you learned a great deal. Do I have hard evidence, no, but it sure is hard to argue with that logic. Bedrick doesn’t really respond this highly plausible alternative to his interpretation of the Louisiana results, though it has been a long topic of debate among reformers. (Rick Hess, for example, provides a summary of some earlier debates on this.)

Now, let’s take on the arguments Bedrick does make. First, he writes, “we know that the primary reason private schools opted out of the voucher program was their concerns over the regulatory burden, particularly those regulations that threatened their character and identity.” This is incorrect. As I wrote previously, the study he cites, by Patrick Wolf and colleagues, actually says that what private schools nationally most want changed is the voucher’s dollar value. In Louisiana, the authors reported that “the top concern was possible future regulations, followed by concerns about the amount of paperwork and reports. When asked about their concerns relating to student testing requirements, a number of school leaders expressed a strong preference for nationally normed tests” (italics added). These quotes give a very different impression that Bedrick states. The supposedly burdensome current regulations seem like less of a concern than funding levels and future additional regulations--and no voucher policy can ever insure against future changes in policy.

Also, note that even the private school leaders concerns echo my first point--that using state standardized tests rather than the norm-referenced ones matters a great deal. This just reinforces my theory that the results are due to the misalignment between the test and the curriculum. If the voucher scores improve significantly in the second year, this would be a sign that schools adjusted the curriculum--genuine school improvement takes time, but curriculum alignment can be done relatively quickly.

Second, Bedrick writes, “schools that were able to attract [non-voucher] students tended to reject the vouchers while voucher schools tended to be those where enrollment had been dropping. . . .The difference in enrollment trends suggests that the LSP’s regulatory burden had the opposite of its intended effect: discouraging higher-performing schools from participating.” Again, no. Obviously, schools with declining enrollments will be more willing to accept vouchers--but this is true regardless of the level of regulation. Schools with declining enrollment are trying to stay afloat so of course they are more likely than other schools to accept students at the lower tuition rates that come with vouchers.

I can hear the response coming ... “ah ha, private schools wouldn’t have to accept these students at lower tuition rates if not for the state regulation requiring them not to charge tuition on top of the voucher--that’s the Over-Regulation Theory!” That’s true, except only low-income students are eligible to participate and they wouldn’t be able to afford tuition. You could then decide to remove the family income requirement, except in that case the program would have no prospect of increasing access to quality schools among the most disadvantaged. As Neerav Kingsand wrote, “A system where every school can systematically discriminate based on wealth is not one that I want to be a part of.” Louisiana State Superintendent John White made similar comments. I didn’t immediately find discussion of these topics from those typically more critical of vouchers, though it’s evident that, if they had to choose between the two, they too would prefer regulated and targeted vouchers over the more broad-based, free market approach.

Third, Bedrick responds to my argument that the comparison schools might have been relatively more effective than those in the other voucher studies. He writes: “Harris’s ... theory is explicitly rejected by the NBER study. On the third page of the study, the authors write: ‘Negative voucher effects are not explained by the quality of public fallback options for LSP applicants: achievement levels at public schools attended by students lotteried out of the program are below the Louisiana average and comparable to scores in low- performing districts like New Orleans.’”

Again, no. The NBER statement focuses on test “levels” even though the NBER voucher analysis focuses, appropriately, on student growth. This is a fundamental difference since it is well established that many schools with relatively low test levels have higher-than-average growth, and vice versa. The question then is, what is the average growth of students in the schools that the students would have attended? Even if we knew the answer to that, which we don’t, recall that the goal here is to understand why the results are worse in Louisiana. So, the real question is, how does the achievement growth of these Louisiana public schools compare with other states where voucher studies have been carried out? This would be a great research project for someone to take on. At this point, we simply don’t know, but that just reinforces my point that we know very little about the role of regulation.

Finally, there is the theory that Bedrick doesn’t mention: that most of the potential participating private schools in Louisiana just aren’t very good at educating disadvantaged students. This would be understandable since it’s not something that private schools have generally been responsible for. But, again, it’s hard to know this since private schools generally do not report results. Ironically, if not for the “burdensome regulation” of testing we would never have seen these negative results.

So, let’s recap: My theory is that most of the negative result in Louisiana is due substantially, though probably not entirely, to test misalignment. This is consistent with what little evidence we do have--private school leaders say that, to the degree current regulation is a concern at all, the main worry is the use of the state test. There is no good evidence to the contrary. A second theory is that the comparison public and charter schools are better than in other states, and a third is that the potential private schools are ineffective with this student population. On these points, we just have no evidence.

The bottom line is that it’s always difficult to know why any program works or fails to work. It takes a lot of effort to figure that out and we simply haven’t done that work yet. With what we do know now about the Louisiana voucher program, however, the logic and evidence are more in line with the Testing Theory than the Over-Regulation Theory. If that holds up, it may also help us interpret prior studies of vouchers that have been based on norm-referenced tests. If the private schools are at a disadvantage in Louisiana for using the state test, then they may have been at an advantage in prior studies where norm-referenced tests were used.

Again, there will be more to come on this with the set of studies we will release next month on Louisiana vouchers from Pat Wolf of the University of Arkansas, Anna Egalite of NC State, and ERA-New Orleans’ own Jon Mills.

The opinions expressed in Urban Education: Lessons From New Orleans are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.