School & District Management What the Research Says

What Does ‘Evidence-Based’ Mean? A Study Finds Wide Variation.

By Sarah D. Sparks — January 10, 2024 5 min read
photograph of a magnifying glass on an open book
  • Save to favorites
  • Print

What makes an education intervention evidence-based?

Over the last 20 years, that question has moved front and center as more federal and state agencies require programs to show evidence of effectiveness, and more education leaders look for proof that interventions used in other districts will help their own students.

But a new analysis in the Review of Educational Research finds wide variation on the kind of evidence that is required to show an education intervention is effective. In fact, large research clearinghouses, set up to review the evidence bases of programs for practitioners, reach the same conclusion on less than a third of the education programs they review.

What that means is that a teacher or principal trying to choose a reading curriculum or tutoring program for students may find it recommended by one clearinghouse and rejected by another.

Researchers from George Washington and Northwestern universities analyzed the evidence standards for 10 common research clearinghouses, including the federal What Works Clearinghouse and the National Dropout Prevention Center in New York, part of the nonprofit Successful Practices Network. The majority of these groups are supported by public agencies or nonprofit foundations.

The team, led by postdoctoral researchers Mansi Wadhwa and Jingwen Zheng of George Washington, compared evidence reviews for nearly 1,360 pre-K-20 education programs and interventions whose evidence base had been reviewed by at least one of the 10 clearinghouses.

The study found 83 percent of the education programs reviewed had only ever been rated by one clearinghouse. Of the programs with multiple ratings, fewer than 1 in 3 had consistent ratings across clearinghouses.

The clearinghouses were more likely to agree about what didn’t work; more than 80 percent of programs with at least two similar reviews were deemed ineffective by both. Less than 18 percent of programs had at least two “effective” ratings, and many had mixed reviews.

For example, five different clearinghouses reviewed the evidence for Peer-Assisted Learning Strategies, a peer-tutoring program focused on math and reading. One clearinghouse recommended the program as a whole and another found it promising.

Two others reviewed the program separately for each subject; one recommended the math program, while the other didn’t find the program promising in either math or reading.

And the final clearinghouse reviewed the PALs program effectiveness on a variety of different outcomes, finding evidence to recommend it for some purposes and not for others.

One reason for the disagreement is that standards differ from clearinghouse to clearinghouse, on what kinds of outcomes can be used to judge program effectiveness, how large a sample of students must be studied, and for how long.

They also differ on whether studies must use randomized controlled experiments, in which students are randomly assigned to a study or control condition, or other designs. Randomized studies are generally considered the most rigorous, but they are difficult and expensive to conduct in educational settings.

“Because [research clearinghouses] do not agree on such criteria for acceptable evidence, and because they are important enough to lead to different judgments about program effectiveness, ‘evidence-based’ seems to be an idea [with limited use] despite [clearinghouses] being funded precisely to identify which programs are most evidence based,” they conclude.

Tough to build consensus

The nonprofit Successful Practices Network, one of the clearinghouses in the study which reviews research on issues like dropout recovery and career and technical education, doesn’t try to align how it defines evidence quality with other groups, according to Bill Dagget, the network’s founder.

“If you’re trying to define ‘evidence-based,’ it’s very difficult to incorporate any of the skills that are harder to measure,” like critical thinking, collaboration, or social-emotional development, Dagget said.

“When you begin to look at these broader skills, you can’t evaluate those with a written test. Typically you have got to do some type of rubric,” Dagget said. “The problem with that is any time you use a rubric, I don’t care how carefully you train, the people using them are always somewhat subjective.”

In a prior study, Jean Stockard, an emerita professor at the University of Oregon, found that half of the What Works Clearinghouse’s intervention reports were based on a single study. Stockard, who was not part of the new study in the journal Review of Educational Research, found that out of more than 120 different studies of one broadly used literacy program, Reading Mastery, those that included evidence beyond randomized controlled studies had more consistent and precise reviews.

The effects of education interventions often fade over time, and the researchers said there’s little agreement on how long and how much follow-up should be done on evaluations. Clearinghouses most often required researchers to follow up a year after an intervention is used, but some allowed shorter follow-ups.

While national research groups have begun to advocate for more researchers to verify a program’s effectiveness, “education research isn’t in limitless supply,” said Julie Brosnan of the National Student Support Accelerator at Stanford university, which collects and conducts research related to tutoring programs.

“For instance, to use tutoring as an example, it is neither feasible nor cost-effective to have every tutoring program engage in a randomized controlled trial to test effectiveness given that there is such a strong evidence base,” Brosnan said. “Education leaders need to ensure the program characteristics align with those that have evidence behind them, while also monitoring implementation and collecting ongoing data.”

Stockard and Brosnan agreed that to build an evidence base for a given program, studies need to include more details about how and for whom the program was used, as well as more analyses of multiple studies to tease out individual aspects of an intervention that may work for different groups of students.

“If the evidence isn’t the right evidence, then the study isn’t of much value,” Dagget said. “So the essential question is, what’s our purpose? Is our purpose to prepare kids for the next grade and the next test and the next level of education? Or is it to prepare them for the world beyond school?”

A version of this article appeared in the January 31, 2024 edition of Education Week as What Does ‘Evidence-Based’ Mean? A Study Finds Wide Variation

Events

Reading & Literacy K-12 Essentials Forum Reading Instruction Across Content Disciplines
Join this free virtual event to hear from educators and experts implementing innovative strategies in reading across different subjects.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
Harnessing AI to Address Chronic Absenteeism in Schools
Learn how AI can help your district improve student attendance and boost academic outcomes.
Content provided by Panorama Education
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Science Webinar
Spark Minds, Reignite Students & Teachers: STEM’s Role in Supporting Presence and Engagement
Is your district struggling with chronic absenteeism? Discover how STEM can reignite students' and teachers' passion for learning.
Content provided by Project Lead The Way

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Opinion What's the No. 1 Way to Retain Principals?
When it comes to the demands of the job, principals share common concerns, according to a recent survey.
5 min read
Screenshot 2024 12 09 at 12.54.36 PM
Canva
School & District Management The Top 10 Things That Keep Principals Up at Night
Principals’ jobs are hard, but what are their most common concerns? We asked, principals answered.
5 min read
School & District Management Superintendents Wrapped: The Songs District Leaders Listened to This Year
Five brave superintendents shared their top songs and artists from the past year with Education Week.
1 min read
A bright blue and pink background with a hand holding a phone with the spotify logo. A pair of headphones frames the cellphone.
Collage by Gina Tomko/Education Week and Canva
School & District Management Opinion I Invited Students to Help Hire a New Assistant Principal. Here’s What Happened
What began as an opportunity for the students turned into a gift for our administrative team.
3 min read
Centering students in the school community.
Vanessa Solis/Education Week via Canva