School & District Management Opinion

Rescuing Education Research

By Jay P. Greene — April 29, 1998 4 min read

Education policy debates suffer from “research nihilism.” When research results are countered with findings from another study, policymakers, the press, and the attentive public tend to fall back on their ideology, interests, and biases instead of trying to sort out the competing claims of researchers. Education researchers, many fear, are just paid liars. Anybody can pay for a study that will conclude anything they want. As Ron Unz, the California businessman promoting an initiative to ban bilingual education, said about education research in The New York Times: “It’s all garbage.”

To some extent these suspicions of education research are well-founded. Many studies are crafted to find desired results, or are so poorly crafted that their results should not be believed. But to abandon reliance on research whenever that research is contested is to abandon all reason and accept instead the brute force of organized interests. Interest groups have learned that they can successfully check research contrary to their goals by producing their own studies, no matter how lousy, to sow confusion among policymakers, journalists, and the attentive public about what to believe.

How can we be rescued from this research nihilism? The consumers of education research need to learn that not all studies are created equal. Some studies are much better designed than others. Since it is hard for lay people to assess research quality, let me propose a simple rule of thumb: Education studies in which students are randomly assigned to different programs are vastly superior to any other research design. Random assignment, as in medical experiments, creates two groups that on average are exactly alike except for the “treatment” that is being tested. Any differences observed between the randomly assigned groups after a period of time can reasonably be attributed to the treatment, since the groups are otherwise identical. Education studies in which students are not randomly assigned to different programs are always plagued by the very real possibility that differences between the groups after a period of time are actually caused by differences before the experiment began. Background characteristics have such a strong effect on educational outcomes that statistical controls for these factors are often inadequate.

While random-assignment experiments are rare in education, the few that exist are extremely important and should carry great weight. Sometimes they do. For example, President Clinton’s proposal to reduce class sizes is based on a random-assignment study in Tennessee, in which students who were randomly assigned to smaller classes outperformed those randomly assigned to larger classes. The random assignment of students in Tennessee gives us high confidence that class size is an important factor in improving educational achievement. Similarly, the bipartisan backing of the Head Start program is supported by a long-term evaluation of students randomly assigned to a Head Start preschool in Michigan vs. students randomly rejected from the program.

The results of these random-assignment experiments have been readily accepted, in part, because no organized groups have been pushing poorly designed studies with contrary conclusions to muddy the waters. Reducing class size and funding Head Start are ideas that have broad-based support and are consistent with the interests of most organized groups in education. Random-assignment studies of school choice and bilingual education programs have not been similarly accepted. In a study of a school choice program in Milwaukee, in which students were randomly assigned, Paul Peterson and I have found significant gains in educational achievement for those students who were offered the option of going to a private school. And in a review of studies on the effectiveness of bilingual education, I was able to find five studies in which limited-English-proficient students were randomly assigned to being taught using at least some of their native language, or to being taught only in English. All five of those random-assignment studies show educational gains for students taught using at least some of their native language.

Why are the results from random-assignment studies on school choice and bilingual education not as readily accepted as the results from class size or Head Start studies? In part, it is because the random-assignment studies on the controversial subjects are countered with non-random-assignment studies of inferior quality that show different results. Suffering from research nihilism, policymakers and opinion leaders throw up their hands in disgust and ignore all research. This is a mistake. Those interested in improving education policy should consistently pay greater attention to studies based on random assignment. If we require random-assignment studies to approve medicines to be used on our children, shouldn’t we also give credence to random-assignment studies about how to educate our children?

Jay P. Greene is an assistant professor of government at the University of Texas at Austin and a research associate at Harvard University’s Program on Education Policy and Governance, in Cambridge, Mass.

Related Tags:


Jobs The EdWeek Top School Jobs Virtual Career Fair
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
How to Power Your Curriculum With Digital Books
Register for this can’t miss session looking at best practices for utilizing digital books to support their curriculum.
Content provided by OverDrive
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Embracing Student Engagement: The Pathway to Post-Pandemic Learning
As schools emerge from remote learning, educators are understandably worried about content and skills that students would otherwise have learned under normal circumstances. This raises the very real possibility that children will face endless hours
Content provided by Newsela

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management We Pay Superintendents Big Bucks and Expect Them to Succeed. But We Hardly Know Them
National data is skimpy, making it hard to know what influences superintendents' decisions to move on, retire, or how long they stay. Why?
8 min read
Conceptual image of tracking with data.
School & District Management Data For the First Time in the Pandemic, a Majority of 4th Graders Learn in Person Full Time
The latest monthly federal data still show big racial and socioeconomic differences in who has access to full-time in-person instruction.
3 min read
Student with backpack.
School & District Management From Our Research Center To Offer Remote Learning in the Fall or Not? Schools Are Split
An EdWeek Research Center survey shows that nearly 4 of every 10 educators say their schools will not offer any remote instruction options.
4 min read
Image of a teacher working with a student through a screen session.
School & District Management Opinion What Does It Mean to Call a Program 'Evidence-Based' Anyway?
States and school districts need to help educators weigh the research on programs. Too often it stops at a single positive study.
Fiona Hollands, Yuan Chang & Venita Holmes
5 min read
A researcher points to charts and data