Data What the Research Says

What Does ‘Evidence-Based’ Mean? A Study Finds Wide Variation.

By Sarah D. Sparks — January 10, 2024 5 min read
photograph of a magnifying glass on an open book
  • Save to favorites
  • Print

What makes an education intervention evidence-based?

Over the last 20 years, that question has moved front and center as more federal and state agencies require programs to show evidence of effectiveness, and more education leaders look for proof that interventions used in other districts will help their own students.

But a new analysis in the Review of Educational Research finds wide variation on the kind of evidence that is required to show an education intervention is effective. In fact, large research clearinghouses, set up to review the evidence bases of programs for practitioners, reach the same conclusion on less than a third of the education programs they review.

What that means is that a teacher or principal trying to choose a reading curriculum or tutoring program for students may find it recommended by one clearinghouse and rejected by another.

Researchers from George Washington and Northwestern universities analyzed the evidence standards for 10 common research clearinghouses, including the federal What Works Clearinghouse and the National Dropout Prevention Center in New York, part of the nonprofit Successful Practices Network. The majority of these groups are supported by public agencies or nonprofit foundations.

The team, led by postdoctoral researchers Mansi Wadhwa and Jingwen Zheng of George Washington, compared evidence reviews for nearly 1,360 pre-K-20 education programs and interventions whose evidence base had been reviewed by at least one of the 10 clearinghouses.

The study found 83 percent of the education programs reviewed had only ever been rated by one clearinghouse. Of the programs with multiple ratings, fewer than 1 in 3 had consistent ratings across clearinghouses.

The clearinghouses were more likely to agree about what didn’t work; more than 80 percent of programs with at least two similar reviews were deemed ineffective by both. Less than 18 percent of programs had at least two “effective” ratings, and many had mixed reviews.

For example, five different clearinghouses reviewed the evidence for Peer-Assisted Learning Strategies, a peer-tutoring program focused on math and reading. One clearinghouse recommended the program as a whole and another found it promising.

Two others reviewed the program separately for each subject; one recommended the math program, while the other didn’t find the program promising in either math or reading.

And the final clearinghouse reviewed the PALs program effectiveness on a variety of different outcomes, finding evidence to recommend it for some purposes and not for others.

One reason for the disagreement is that standards differ from clearinghouse to clearinghouse, on what kinds of outcomes can be used to judge program effectiveness, how large a sample of students must be studied, and for how long.

They also differ on whether studies must use randomized controlled experiments, in which students are randomly assigned to a study or control condition, or other designs. Randomized studies are generally considered the most rigorous, but they are difficult and expensive to conduct in educational settings.

“Because [research clearinghouses] do not agree on such criteria for acceptable evidence, and because they are important enough to lead to different judgments about program effectiveness, ‘evidence-based’ seems to be an idea [with limited use] despite [clearinghouses] being funded precisely to identify which programs are most evidence based,” they conclude.

Tough to build consensus

The nonprofit Successful Practices Network, one of the clearinghouses in the study which reviews research on issues like dropout recovery and career and technical education, doesn’t try to align how it defines evidence quality with other groups, according to Bill Dagget, the network’s founder.

“If you’re trying to define ‘evidence-based,’ it’s very difficult to incorporate any of the skills that are harder to measure,” like critical thinking, collaboration, or social-emotional development, Dagget said.

“When you begin to look at these broader skills, you can’t evaluate those with a written test. Typically you have got to do some type of rubric,” Dagget said. “The problem with that is any time you use a rubric, I don’t care how carefully you train, the people using them are always somewhat subjective.”

In a prior study, Jean Stockard, an emerita professor at the University of Oregon, found that half of the What Works Clearinghouse’s intervention reports were based on a single study. Stockard, who was not part of the new study in the journal Review of Educational Research, found that out of more than 120 different studies of one broadly used literacy program, Reading Mastery, those that included evidence beyond randomized controlled studies had more consistent and precise reviews.

The effects of education interventions often fade over time, and the researchers said there’s little agreement on how long and how much follow-up should be done on evaluations. Clearinghouses most often required researchers to follow up a year after an intervention is used, but some allowed shorter follow-ups.

While national research groups have begun to advocate for more researchers to verify a program’s effectiveness, “education research isn’t in limitless supply,” said Julie Brosnan of the National Student Support Accelerator at Stanford university, which collects and conducts research related to tutoring programs.

“For instance, to use tutoring as an example, it is neither feasible nor cost-effective to have every tutoring program engage in a randomized controlled trial to test effectiveness given that there is such a strong evidence base,” Brosnan said. “Education leaders need to ensure the program characteristics align with those that have evidence behind them, while also monitoring implementation and collecting ongoing data.”

Stockard and Brosnan agreed that to build an evidence base for a given program, studies need to include more details about how and for whom the program was used, as well as more analyses of multiple studies to tease out individual aspects of an intervention that may work for different groups of students.

“If the evidence isn’t the right evidence, then the study isn’t of much value,” Dagget said. “So the essential question is, what’s our purpose? Is our purpose to prepare kids for the next grade and the next test and the next level of education? Or is it to prepare them for the world beyond school?”

A version of this article appeared in the January 31, 2024 edition of Education Week as What Does ‘Evidence-Based’ Mean? A Study Finds Wide Variation

Events

Jobs Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
English-Language Learners Webinar English Learners and the Science of Reading: What Works in the Classroom
ELs & emergent bilinguals deserve the best reading instruction! The Reading League & NCEL join forces on best practices. Learn more in our webinar with both organizations.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
Challenging the Stigma: Emotions and STEM
STEM isn't just equations and logic. Join this webinar and discover how emotions fuel innovation, creativity, & problem-solving in STEM!
Content provided by Project Lead The Way

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Data 'Hidden Homeless': A Key Measure of Homelessness Excludes Most Students
Federal agencies differ in how they measure homelessness—and many vulnerable students are left out.
3 min read
Photograph of a low angle view of children with backpacks climbing the school staircase.
E+/Getty
Data Spotlight Spotlight on Leveraging Data for Student Success
This Spotlight will help you learn how data can help schools target resources, explore how to improve instruction with data, and more.
Data What Superintendents Say They Need More of to Help Them Manage Districts
98% of those surveyed said better data would make them more comfortable making decisions.
2 min read
Image of a data dashboard.
Suppachok Nuthep/iStock/Getty
Data Opinion Standardized Tests Aren't the Only Meaningful Data on Student Achievement
"Street data" may be a better pathway to education equity. Here's what that means.
Shane Safir
5 min read
conceptual image of a young person walking a trendline
E+/Getty