Amid shrinking budgets, multiplying federal and state requirements for school improvement, and rapidly changing educational technology, it’s critical for education leaders to identify whether a given program will really improve learning for their students.
While leaders have more access than ever to education program evaluations and research clearinghouses, experts say it’s easy to overlook red flags in studies and evaluations of particular programs and interventions.
“It’s very challenging, and we’re asking educators whose job isn’t evidence to apply this to making their work more effective,” Nancy Madden, co-founder of the Success for All Foundation and an education professor at the Center for Research and Reform in Education (CRRE) at Johns Hopkins University, said during an Education Week webinar last week. “Educators are just now beginning to get down into the weeds enough to figure out how to make that evidence matter.”
Madden; Robert Peters, the senior vice president and chief academic officer for the nonprofit education research clearinghouses the Successful Practices Network and the National Dropout Prevention Center; and Amanda Neitzel, CRRE’s deputy director of evidence research, highlighted ways education leaders can find evidence-based interventions more effectively.
What does ‘evidence based’ mean?
The Every Student Succeeds Act formally defines three tiers of research evidence indicating how promising researchers have found a particular intervention to be. (Some states also allow a fourth tier for early studies exploring a new program with a strong rationale.) These tiers are used mostly to identify appropriate programs for school improvement under the law. To fit in any tier, the study must be well-conducted and completed by a researcher who is independent of the program, and it must show statistically significant findings.
- Tier 1 (strong) studies track the outcomes of a large group—350 or more students, teachers, or other participants. These studies randomly assign participants to the intervention or a control group in an effort to show cause and effect. This practice is considered the gold standard for causal research.
- Tier 2 (moderate) evidence comes from a study that also includes 350 or more participants tracked over time, but may use non-randomized, quasi-experimental designs in situations in which it is not ethical or possible to randomly assign participants.
- Tier 3 (promising) evidence may have a smaller sample group, use correlational versus causal designs, or be retrospective (with analysis performed after the intervention) rather than follow the sample over time.
Madden noted that terms like “research-based” and “evidence-based” are often used interchangeably to mean the same thing. However, different research groups and clearinghouses often disagree about exactly what is needed to prove that an intervention is effective.
The federal Institute of Education Sciences’ What Works Clearinghouse and CRRE have adopted the ESSA tiers when evaluating research. Neitzel said the highest-quality studies also follow a standardized intervention for at least 12 weeks, and include treatment and control groups that are similar and have had relatively few participants drop out.
Will an evidence-based intervention actually work in the classroom?
Many programs and interventions also sound great on paper or in the lab, but fizzle when they get to the classroom.
It’s important, experts said, to look for red flags in a study that suggest it could be hard to implement in a new class or school.
For example, no intervention will work if students don’t receive it, so an evaluation should state how much time students actually participated in a program that produced particular results.
“We see often, especially in cases of education technology, studies will compare students who used the program a lot to students who didn’t use it at all, and drop out all those kids in the middle who had low usage,” Neitzel said. “While that will tell you how this works for kids who use it five days a week, consistently for the whole year, it doesn’t necessarily tell you how it’s going to work for every one of your students.”
Even if a program shows strong evidence of effectiveness, it’s important for leaders and educators to consider how easily they could replicate the model with their own students.
“Choosing something with great evidence that doesn’t address the problem in your situation is not going to make the progress that you want to make,” Madden said.
She recommended that education leaders try to look for insight into the following questions when reading education research.
- What training do teachers or other intervention staff need to implement the program effectively?
- What aspects of the program may be harder to implement in your district? For example, rural districts may have less access to staff or resources used to implement a program in an urban school.
- How will the school monitor the effectiveness of the intervention with its own students?
Peters, who works with districts on improvement plans, said the highest-performing school systems also analyze the cost-effectiveness of their programs at the end of each school year to understand how to prioritize them.
“A lot of times we see large school districts across the country spend millions of dollars on programs that are not even connected to their strategic planning,” Peters said.
