How to Find Evidence-Based Fixes for Schools That Fall Behind

The Every Student Succeeds Act gives states and districts significant flexibility in how they turn around struggling schools, as long as the local approaches are backed by evidence. But without support, that flexibility runs the risk of putting smaller or more rural districts at a disadvantage.

“This is a sea change from the highly prescriptive approach to school improvement [under the No Child Left Behind Act] to what can seem like a bit of a Wild West structure under ESSA,” said Mike Magee, the chief executive officer of Chiefs for Change, which has created an ESSA working group of 15 experimentally minded state education leaders. “We have potentially unprecedented flexibility in how states address school improvement—but that’s just another factor in how high the stakes are.”

As states work out how to apply ESSA’s new standards of evidence, their quest highlights the need for more research on interventions at schools with a wider array of contexts. The pool of high-quality research on education programs remains relatively small, sporadic, and focused on shorter-term gains for students.

Tying Things Together

While the federal What Works Clearinghouse has reviewed more than 10,000 studies on various interventions, a forthcoming meta-analysis based on the clearinghouse’s reviews found only 29 different interventions showed significant effects—and the average effect was small, particularly when the interventions were in messy real-school contexts instead of highly controlled laboratory settings.

“If you look from 10,000 feet at education interventions, you can almost count on your hand the number of interventions that have truly scaled and established” themselves, said Jerome D’Agostino, a professor of educational studies at Ohio State University, who led the study presented at the 2016 American Educational Research Association in Washington this April.

“It’s not just the sheer volume of programs; there hasn’t been a wider effort to tie these [intervention evaluations] together in any way,” D’Agostino said. “It’s not like some grand designer said, ‘Do we have enough interventions in reading, in math, in different grade levels? It’s field-generated. ... People have been focusing on their parts of the elephant, and I’m not sure there would be a whole elephant if you brought them all together.”

ESSA lays out three levels of evidence that states can choose to apply to prove an intervention works:

• “Strong evidence” includes at least one well-designed and -implemented experimental study, meaning a randomized controlled trial.

• “Moderate evidence” includes at least one well-designed and -implemented quasi-experimental study. For example, a program evaluation could use a regression-discontinuity analysis, in which researchers might look at differences in outcomes for students who scored a point above and below the entrance cutoff score for a particular program or intervention.

• “Promising evidence” includes at least one well-designed and -implemented correlational study that controls for selection bias, the potential differences between the types of students who choose to participate in a particular program and those who don’t.

In separate guidance, the Education Department explained that districts and states should work to use the most rigorous evidence available, with intervention studies that not only meet high methodological quality, but also reflect similar students and school types as those where the intervention would be used.

“The logic model before was, pick a good intervention, implement it, and you’ll get results,” said Vivian Tseng, the vice president of the William T. Grant Foundation, which has been studying research use in education. “Now, whatever you implement, there’s this idea of ongoing evaluation to see where it worked, where did it not work, and for whom. I would say the ongoing cycle of learning is needed for programs at any of these [evidence] tiers.”

Proof Requirements

Those levels were developed in part from the proof required under the Obama administration’s Investing in Innovation competitive grants. It’s telling that only 43 projects met the grant’s moderate-evidence bar—and fewer than 10 have so far had the strong evidence required to win i3’s top-tier grant.

The few interventions that have established strong bases of evidence and use over time, such as Success for All and Reading Recovery, which D’Agostino evaluated for the federal Investing in Innovation program, created comprehensive infrastructure to implement programs in a wide variety of schools; trained and retrained staff that turned over; and sustained ongoing improvement and evaluation of the programs.

“When I meet with other [intervention programs], they are so far from even thinking and conceptualizing that need for infrastructure, I’ve come to the conclusion a lot of them will never get there,” he said.

It can take years of effort to build a strong evidence base for a program. One of those i3 grantees, the New Teacher Center, has been conducting multiple randomized controlled trials and quasi-experimental evaluations of its mentoring model since 2004, according to Ali Picucci, the center’s vice president of impact and improvement.

While the studies have produced promising results, “we know [randomized controlled trials] occur in controlled environments and are not ideal for addressing the social complexities that we find in classrooms,” Picucci said.

“Just as one-size-doesn’t fit all when it comes to clothes or educational initiatives, one study doesn’t fit all district and school contexts. It’s important for all of us to remember that interventions—even those backed by high-quality evidence—are beginnings and not ends,” said Ash Vasudeva, the vice president for strategic initiatives at the Carnegie Foundation for the Advancement of Teaching. “Simply selecting an evidence-based program does not ensure that similar results can be achieved in different settings and systems.”

Local Contexts Critical

In Cleveland, district officials are trying to integrate thinking about evidence into day-to-day decision making in schools.

The district has set up a website with a summary of every program available in the district, and is working to provide reviews on the effectiveness of each on key outcomes like reading and math achievement. The intervention “report cards” do include high-quality external studies if they are available, but also include the district’s own research on an intervention’s effects for schools that used it either a lot or just a little, and feedback from principals on how easy it was to use and how well it worked for them.

“We’re evaluating programs regardless of their external body of evidence, because context matters,” said Matthew A. Linick, executive director of research and evaluation for Cleveland public schools. “As a researcher, you know the more rigorous the study is, the less generalizable it becomes. While things that work in urban districts could be helpful, Cleveland’s urban context could be very different. Just because something works in Cincinnati doesn’t mean it will work in Cleveland.”

Both the U.S. Department of Education and nationwide groups like the Council of Chief State School Officers and Chiefs for Change have set up support networks for state officials to work together to identify evidence for what interventions will work in different school contexts. And the Institute of Education Sciences, the department’s research arm, is working to allow researchers and educators to search research for core elements of different programs and the effects of an intervention on specific populations.

“One of the things we’ve heard from people is they really want to see the context” of the school where an intervention was done, said Ruth C. Neild, the IES acting director. “Almost regardless of whether impacts are different for different [student] groups, ... a lot of times people need to see the intervention was done in their particular context in order to believe it.”