Education policy debates suffer from “research nihilism.” When research results are countered with findings from another study, policymakers, the press, and the attentive public tend to fall back on their ideology, interests, and biases instead of trying to sort out the competing claims of researchers. Education researchers, many fear, are just paid liars. Anybody can pay for a study that will conclude anything they want. As Ron Unz, the California businessman promoting an initiative to ban bilingual education, said about education research in The New York Times: “It’s all garbage.”
To some extent these suspicions of education research are well-founded. Many studies are crafted to find desired results, or are so poorly crafted that their results should not be believed. But to abandon reliance on research whenever that research is contested is to abandon all reason and accept instead the brute force of organized interests. Interest groups have learned that they can successfully check research contrary to their goals by producing their own studies, no matter how lousy, to sow confusion among policymakers, journalists, and the attentive public about what to believe.
How can we be rescued from this research nihilism? The consumers of education research need to learn that not all studies are created equal. Some studies are much better designed than others. Since it is hard for lay people to assess research quality, let me propose a simple rule of thumb: Education studies in which students are randomly assigned to different programs are vastly superior to any other research design. Random assignment, as in medical experiments, creates two groups that on average are exactly alike except for the “treatment” that is being tested. Any differences observed between the randomly assigned groups after a period of time can reasonably be attributed to the treatment, since the groups are otherwise identical. Education studies in which students are not randomly assigned to different programs are always plagued by the very real possibility that differences between the groups after a period of time are actually caused by differences before the experiment began. Background characteristics have such a strong effect on educational outcomes that statistical controls for these factors are often inadequate.
While random-assignment experiments are rare in education, the few that exist are extremely important and should carry great weight. Sometimes they do. For example, President Clinton’s proposal to reduce class sizes is based on a random-assignment study in Tennessee, in which students who were randomly assigned to smaller classes outperformed those randomly assigned to larger classes. The random assignment of students in Tennessee gives us high confidence that class size is an important factor in improving educational achievement. Similarly, the bipartisan backing of the Head Start program is supported by a long-term evaluation of students randomly assigned to a Head Start preschool in Michigan vs. students randomly rejected from the program.
The results of these random-assignment experiments have been readily accepted, in part, because no organized groups have been pushing poorly designed studies with contrary conclusions to muddy the waters. Reducing class size and funding Head Start are ideas that have broad-based support and are consistent with the interests of most organized groups in education. Random-assignment studies of school choice and bilingual education programs have not been similarly accepted. In a study of a school choice program in Milwaukee, in which students were randomly assigned, Paul Peterson and I have found significant gains in educational achievement for those students who were offered the option of going to a private school. And in a review of studies on the effectiveness of bilingual education, I was able to find five studies in which limited-English-proficient students were randomly assigned to being taught using at least some of their native language, or to being taught only in English. All five of those random-assignment studies show educational gains for students taught using at least some of their native language.
Why are the results from random-assignment studies on school choice and bilingual education not as readily accepted as the results from class size or Head Start studies? In part, it is because the random-assignment studies on the controversial subjects are countered with non-random-assignment studies of inferior quality that show different results. Suffering from research nihilism, policymakers and opinion leaders throw up their hands in disgust and ignore all research. This is a mistake. Those interested in improving education policy should consistently pay greater attention to studies based on random assignment. If we require random-assignment studies to approve medicines to be used on our children, shouldn’t we also give credence to random-assignment studies about how to educate our children?
Jay P. Greene is an assistant professor of government at the University of Texas at Austin and a research associate at Harvard University’s Program on Education Policy and Governance, in Cambridge, Mass.
