Curriculum What the Research Says

Picking ‘Evidence-Based’ Programs: 5 Mistakes for Educators to Avoid

By Sarah D. Sparks — January 19, 2024 6 min read
Conceptual image of magnifying glass and rating/grades.
  • Save to favorites
  • Print

With hundreds of education programs and interventions purporting to be “evidence based,” and dozens of websites and clearinghouses that compile thousands of studies and evaluations, it’s easy for practitioners to become overwhelmed by the sheer amount of information to sift through to figure out what will best help their students.

It’s understandable that teachers, as well as district and school leaders, get frustrated when studies and clearinghouses seem to come to different conclusions about a given program.

But hoping for a single seal of approval is “not realistic,” said Jonathan Jacobson, the the branch chief for the What Works Clearinghouse, National Library of Education, and ERIC at the National Center for Education Evaluation and Regional Assistance.

See Also

photograph of a magnifying glass on an open book
Valiantsin Suprunovich/iStock

The sometimes conflicting information is akin to the range of reviews consumers encounter for all kinds of products, Jacobson said.

“When I’m looking to buy a new car, and I read different magazines that have reviewed the car, the results or characterizations are not going to be identical, even though the vehicle might be the same,” Jacobson said. “That doesn’t mean that [evidence-based decision-making] isn’t scientific, but it’s not as simple as solving a math problem where everyone should get a single right answer.”

With that in mind, here are five pitfalls for educators to avoid when using research to choose evidence-based programs.

1. Equating research quality with program quality

Research clearinghouses have become a popular way for educators and leaders to review a lot of studies on a subject quickly, but Jacobson warned, “there’s a distinction between the quality of the research and the quality of the program or the practice or the intervention.”

“Just because studies are high-quality, that doesn’t mean they will show favorable results” for a product or program, he said.

A randomized controlled trial—considered the “gold standard” in research, might show that an intervention worked or didn’t, but it won’t necessarily give insight into why that happened.

Educators shouldn’t “assum[e] that the What Works Clearinghouse endorses a particular program or product simply because we report and confirm favorable findings,” Jacobson said referring to the U.S. Education Department’s education research review site. “We’re not endorsing that; we are simply characterizing the strength of the research and reporting whatever findings we were able to confirm with our standards or procedures. Educators need to make their own decisions.”

2. Taking ‘no effect’ for a conclusive answer

When a study finds that an intervention has “no effect,” it means that students who used the intervention performed equally well as similar students who did not. This comparison group might be using a different intervention, or just whatever they would normally do in school, often dubbed “business as usual.”

Finding no effect “doesn’t necessarily mean that is a bad finding,” said Erin Pollard, an education research analyst with IES’s knowledge utilization division. “If you have a very expensive program as your business-as-usual, and you’re [testing] a less expensive program and there’s no difference, that could be a good thing.”

3. Looking only at the summary (or rating)

It can be tempting to just skim the abstract of an evaluation study to determine whether the program was effective—or, similarly, to look at the overview given of an intervention on a clearinghouse—but looking only at a summary can give educators the wrong idea about a program, researchers said.

That’s because evaluations often measure several different outcomes in a variety of different areas, from specific academic subjects to social-emotional outcomes like motivation. The What Works Clearinghouse alone can include more than 70 different outcomes in its reports. It’s critical to dig into these findings.

“Sometimes users will take the high-level summary, like the evidence tier of the whole study and intervention, but not look at the individual outcome domains,” IES research scientist Betsy Wolf said. “It might be that the study has evidence of a positive finding [in the summary], but not in the outcome domain that people really care about for that intervention.”

For example, Pollard pointed to one What Works Clearinghouse review of a study on a pre-kindergarten reading intervention. The summary shows a “promising” rating for having at least one positive result in a rigorous study.

An intervention rating in the What Works Clearinghouse includes several different measures.

“People might look at it and say, ‘Well, we should do this reading intervention?’” Pollard said. “Well, you can, but actually the effects were on math.”

The study had assessed the children on early word identification, but also early number skills and two tasks measuring self-regulation, such as a “head, shoulders, knees and toes” game. Of all of those, students participating in the intervention only performed better than the control group of students in math.

Even within a given area, it’s important to look at the details of how an outcome is being measured. Wolf noted that researchers—and the WWC—have expanded the methods they use and ways they look at things like a program’s effects on teachers.

“In the past we had teacher practice, we had teacher retention and turnover—very observable things—but we didn’t have outcomes like how teachers were perceiving their jobs or their self-efficacy. So we have that now,” Wolf said.

4. Focusing too much on effect size

To ensure that a study can isolate the effects of, say, a reading program, researchers try to tightly control how it is put into practice. They may give several days of teacher training or provide their own tutors. They may create and use particular lesson plans and manipulatives. All of these implementation pieces can take time and money.

“One concern is that people might pick an intervention that has the largest effect size, and not think about the resources needed for implementing that,” said Liz Eisner, IES’s associate commissioner for knowledge utilization.

When choosing an intervention, Eisner advised teachers and leaders to balance its benefits with the effort of implementation: the materials, staff, and training, as well as how much of the intervention students need to see results. A cheap, 5-minute intervention that boosts background knowledge before a new lesson and a yearlong, intensive tutoring program can both benefit students, but they each require very different levels of investment from schools.

5. Forgetting whom the program serves

The most “evidence based” program in the world won’t be much help if it doesn’t help the students an educator needs to serve. Eisner said educators should remember to review the demographics, grades, and other aspects of students in a study’s treatment and control groups.

“For example,” she said, “if a school that is trying to improve math achievement serves mostly English-learner students, but the intervention of interest has not been studied in schools with many ELs, the staff may decide that they want to identify a different intervention that has some empirical evidence of positive outcomes with EL students.”

That goes for students in the comparison group, too, Jacobson said.

“It’s important to understand what was the intervention being compared against. For which population of students? For which outcomes?” he said. “It’s really important, I think, for decisionmakers to be thoughtful about what applies to them.”

See also

Evidence-Based Practice
July 9, 2020


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Attend to the Whole Child: Non-Academic Factors within MTSS
Learn strategies for proactively identifying and addressing non-academic barriers to student success within an MTSS framework.
Content provided by Renaissance
Classroom Technology K-12 Essentials Forum How to Teach Digital & Media Literacy in the Age of AI
Join this free event to dig into crucial questions about how to help students build a foundation of digital literacy.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Curriculum Opinion Media Literacy Is an Essential Skill. Schools Should Teach It That Way
From biased news coverage to generative AI, students (and adults) need help now more than ever to stay abreast of what’s real—or misleading.
Nate Noorlander
5 min read
Illustration of boy reading smartphone
Curriculum Interactive Play the EdWeek Spelling Bee
Educators use these words all the time. But can they spell them?
Image of a stage set up for a spelling bee.
Leonard Mc Lane/DigitalVision
Curriculum Outdoor Learning: The Ultimate Student Engagement Hack?
Outdoor learning offers a host of evidence-based benefits for students. One Virginia school serves as an example how.
7 min read
Students from Centreville Elementary School in Fairfax, Va., release brook trout they’ve grown from eggs in their classroom into Passage Creek at Elizabeth Furnace Recreational Area in the George Washington National Forest in Fort Valley, Va. on April 23.
Students from Centreville Elementary School in Fairfax, Va., release brook trout that they’ve grown from eggs in their classroom at a creek in Fort Valley, Va., on April 23.
Sam Mallon/Education Week
Curriculum Opinion Classical Education Is Taking Off. What’s the Appeal?
Classical schooling is an apprenticeship to the great minds and creators of the past, enabling students to develop their own thinking.
9 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty