What makes an education intervention evidence-based?
Over the last 20 years, that question has moved front and center as more federal and state agencies require programs to show evidence of effectiveness, and more education leaders look for proof that interventions used in other districts will help their own students.
But a new analysis in the Review of Educational Research finds wide variation on the kind of evidence that is required to show an education intervention is effective. In fact, large research clearinghouses, set up to review the evidence bases of programs for practitioners, reach the same conclusion on less than a third of the education programs they review.
What that means is that a teacher or principal trying to choose a reading curriculum or tutoring program for students may find it recommended by one clearinghouse and rejected by another.
Researchers from George Washington and Northwestern universities analyzed the evidence standards for 10 common research clearinghouses, including the federal What Works Clearinghouse and the National Dropout Prevention Center in New York, part of the nonprofit Successful Practices Network. The majority of these groups are supported by public agencies or nonprofit foundations.
The team, led by postdoctoral researchers Mansi Wadhwa and Jingwen Zheng of George Washington, compared evidence reviews for nearly 1,360 pre-K-20 education programs and interventions whose evidence base had been reviewed by at least one of the 10 clearinghouses.
The study found 83 percent of the education programs reviewed had only ever been rated by one clearinghouse. Of the programs with multiple ratings, fewer than 1 in 3 had consistent ratings across clearinghouses.
The clearinghouses were more likely to agree about what didn’t work; more than 80 percent of programs with at least two similar reviews were deemed ineffective by both. Less than 18 percent of programs had at least two “effective” ratings, and many had mixed reviews.
For example, five different clearinghouses reviewed the evidence for Peer-Assisted Learning Strategies, a peer-tutoring program focused on math and reading. One clearinghouse recommended the program as a whole and another found it promising.
Two others reviewed the program separately for each subject; one recommended the math program, while the other didn’t find the program promising in either math or reading.
And the final clearinghouse reviewed the PALs program effectiveness on a variety of different outcomes, finding evidence to recommend it for some purposes and not for others.
One reason for the disagreement is that standards differ from clearinghouse to clearinghouse, on what kinds of outcomes can be used to judge program effectiveness, how large a sample of students must be studied, and for how long.
They also differ on whether studies must use randomized controlled experiments, in which students are randomly assigned to a study or control condition, or other designs. Randomized studies are generally considered the most rigorous, but they are difficult and expensive to conduct in educational settings.
“Because [research clearinghouses] do not agree on such criteria for acceptable evidence, and because they are important enough to lead to different judgments about program effectiveness, ‘evidence-based’ seems to be an idea [with limited use] despite [clearinghouses] being funded precisely to identify which programs are most evidence based,” they conclude.
Tough to build consensus
The nonprofit Successful Practices Network, one of the clearinghouses in the study which reviews research on issues like dropout recovery and career and technical education, doesn’t try to align how it defines evidence quality with other groups, according to Bill Dagget, the network’s founder.
“If you’re trying to define ‘evidence-based,’ it’s very difficult to incorporate any of the skills that are harder to measure,” like critical thinking, collaboration, or social-emotional development, Dagget said.
“When you begin to look at these broader skills, you can’t evaluate those with a written test. Typically you have got to do some type of rubric,” Dagget said. “The problem with that is any time you use a rubric, I don’t care how carefully you train, the people using them are always somewhat subjective.”
In a prior study, Jean Stockard, an emerita professor at the University of Oregon, found that half of the What Works Clearinghouse’s intervention reports were based on a single study. Stockard, who was not part of the new study in the journal Review of Educational Research, found that out of more than 120 different studies of one broadly used literacy program, Reading Mastery, those that included evidence beyond randomized controlled studies had more consistent and precise reviews.
The effects of education interventions often fade over time, and the researchers said there’s little agreement on how long and how much follow-up should be done on evaluations. Clearinghouses most often required researchers to follow up a year after an intervention is used, but some allowed shorter follow-ups.
While national research groups have begun to advocate for more researchers to verify a program’s effectiveness, “education research isn’t in limitless supply,” said Julie Brosnan of the National Student Support Accelerator at Stanford university, which collects and conducts research related to tutoring programs.
“For instance, to use tutoring as an example, it is neither feasible nor cost-effective to have every tutoring program engage in a randomized controlled trial to test effectiveness given that there is such a strong evidence base,” Brosnan said. “Education leaders need to ensure the program characteristics align with those that have evidence behind them, while also monitoring implementation and collecting ongoing data.”
Stockard and Brosnan agreed that to build an evidence base for a given program, studies need to include more details about how and for whom the program was used, as well as more analyses of multiple studies to tease out individual aspects of an intervention that may work for different groups of students.
“If the evidence isn’t the right evidence, then the study isn’t of much value,” Dagget said. “So the essential question is, what’s our purpose? Is our purpose to prepare kids for the next grade and the next test and the next level of education? Or is it to prepare them for the world beyond school?”
