Some good things never last.
It’s a well-established research finding that the effects of many academic interventions, from extra math help for a few weeks to some years-long preschool programs, can “fade out” over time. The additional supports can raise student achievement in the short run, but after a few years, the kids who received it aren’t any better off than the kids who didn’t.
This pattern raises thorny questions for school and district leaders, especially now as school systems have poured COVID relief funding into efforts to accelerate student learning.
What kinds of interventions are most likely to show persistent effects? Are there ways to make sure that the short-term gains in academics or student well-being will last?
Now, scientists have proposed some answers. In a panel this week hosted by the Association for Psychological Science, two researchers discussed the available evidence.
The presenters—Drew Bailey, an associate professor at the University of California, Irvine’s education school, and David Yeager, an associate professor of psychology at the University of Texas at Austin—also wrote a 2020 paper on this topic.
Here are a few takeaways for educators from the paper and the panel.
Most interventions show some fade out, but not all. Measurement is one factor
One example of fade out comes from a study of a high-quality preschool math curriculum in 2011. At the end of preschool, students who had experienced this curriculum had stronger math skills than students who hadn’t. But by the end of 4th grade, those differences had almost completely disappeared.
In this case, that gap closed because students who never had the early math curriculum caught up: In the intervening years between preschool and 4th grade, their math achievement grew at a faster rate than that of their peers’ who did have it.
This is one reason why fade out occurs in educational interventions, the researchers write—other students who didn’t get the intervention eventually reap the same benefits through regular teaching. Another is that learning doesn’t transfer: A lot of what students do in school requires the interaction of different skills. If an intervention only targets one discrete skill, it may not lead to overall increases in learning.
The effects of some interventions persist, though—usually, ones that alter students’ lives in big ways, like adding extra years of school.
And some interventions fade out at first, but then see other effects emerge over time.
A well-known example of this last category is the Perry Preschool Program study, which provided a group of low-income children in Michigan with high-quality preschool and then tracked their outcomes over time. A boost in IQ for Perry students compared to their peers faded out, but later, researchers found that Perry students had an increase in lifetime earnings compared to non-Perry students.
The emergence of long-term effects highlights another point: There are often benefits to educational interventions that can’t be captured by test scores.
“A critical perspective that we often overlook when trying to evaluate the efficacy of a program or initiative is [that] we don’t step back and try to think about what the primary goal of the program is, and what it is designed to do,” Matthew Kraft, an associate professor of education and economics at Brown University, said in an interview with Education Week.
Test scores are a readily available data point, and as such are often used to evaluate outcomes. But that’s not always the best approach, Kraft said. The effectiveness of a mentoring program, for example, might be better measured by looking at student behavior, persistence, or engagement with the juvenile justice system.
“There are a whole lot of ways that schools have a meaningful impact on students’ lives,” Kraft said.
Interventions with persistent effects may share some key characteristics
Given this variation, how can education leaders know which interventions are likely to fade out, and which might have staying power? Bailey, Yeager, and their colleagues suggest that interventions with persistent effects target skills that meet three criteria:
- These skills are malleable, meaning it’s possible for interventions to help students improve on them in the first place;
- They’re fundamental—they serve as building blocks for other skills;
- And the skills wouldn’t have developed if not for the intervention.
Early reading intervention is one example cited in the paper. Foundational reading skills like phonemic awareness and phonics are malleable—students can improve with explicit instruction. They’re also fundamental, because being able to decode the words on the page is a prerequisite to all reading.
For many students, the regular teaching they get on these skills will be enough to master them. But for students who struggle, the intervention gives them an opportunity to develop these skills that they might not get in a whole class setting.
Other interventions can be effective, too, even if they don’t share these characteristics—as long as they’re deployed at the right time.
Bailey and colleagues write that there are some “gateways” in the education system that decide students’ long-term trajectories—whether they graduate high school, for instance, or enroll in college. Those are smart places to time interventions, they say.
For example, one pair of studies demonstrated that taking two periods of math courses instead of one had a big effect on 9th grade students’ educational attainment, but none for 6th graders who underwent the same intervention.
One possible reason for this, Bailey and his co-authors write, is that 9th grade algebra is a gateway course—whether students pass it or not can expand or limit their future high school options.
Cost-benefit analyses can support district decisionmaking
Just because an intervention has a big effect doesn’t mean it’s the right one for every circumstance, the researchers write.
“We should always consider those effects relative to the costs,” said Kraft, who authored a 2020 paper on interpreting the policy relevance of different effect sizes. “Sometimes programs that have very modest or even small impacts may actually be very good investments because they are very inexpensive.”
Another consideration for school districts is how easy an intervention is to sustain and scale. Sometimes it’s just hard to find people who can keep an amazing program with big effects going long-term, Kraft said.
Still, he said, cost-benefit analyses shouldn’t drive decisionmaking alone. Sometimes expensive shifts, like hiring more high-quality teachers, are necessary.
“I think we also have to take a values based approach,” Kraft said. “What do we as a society think we should be investing in, and how do we invest in it?”
