Opinion
School & District Management Opinion

What Does It Mean to Call a Program ‘Evidence-Based’ Anyway?

There’s a good reason not to stop after finding a single positive study
By Fiona Hollands, Yuan Chang & Venita Holmes — June 09, 2021 5 min read
A researcher points to charts and data
  • Save to favorites
  • Print

Millions of federal dollars are currently streaming into schools to help address learning shortfalls related to the pandemic. While it’s not clear what strings might be attached to these new funds, recent versions of the Elementary and Secondary Education Act have encouraged and, for some uses, required education decisionmakers to invest federal funds in evidence-based activities. Many school leaders themselves want evidence to inform their choices. But how does that work in practice?

The current federal approach may seem either needlessly restrictive or absolutely sound, but the fact is the bar for what counts as evidence isn’t awfully high: The Every Student Succeeds Act, the latest ESEA iteration, asks for only one study showing that a program improves student outcomes. And if there isn’t one, a sound rationale for the activity and a plan to evaluate it will often suffice.

To figure out how well schools and districts are able to meet this directive, our research team worked with a large, Southern school system to determine what proportion of its Title I funds are spent on evidence-based activities. Our findings—which relate to prepandemic federal funds—reveal weaknesses both in the law and in the evidence base. As a result, we have some advice to share with educators who are doing their best to adhere to ESSA, serve students well, and now wisely spend large amounts of COVID-19 relief funds in record time.

First, let’s address the question of what should count as adequate evidence when choosing where to invest education funds. Does it really make sense to declare a program “evidence-based” if one study shows positive outcomes for students while a bunch of others don’t?

In our analysis, we first determined what percentage of Title I funds at the district office and four sampled schools were spent on programs and practices backed by just one study: 70 percent in a three-year period. It’s likely that this number would have been substantially higher if we had descended below the top three tiers of evidence laid out in ESSA to that nebulous world of activities for which there is a research-based rationale and an ongoing evaluation plan. To evade this squishiness, we limited our definition of positive evidence to studies using one of three rigorous approaches for evaluating effectiveness: experimental, quasi-experimental, or correlational methods. Additionally, each study needed to show gains for students of approximately the same age as those being served at the district.

Next, we considered all the relevant studies we could find in four research repositories: What Works Clearinghouse, Evidence for ESSA, Education Endowment Foundation, and Education Resources Information Center (ERIC). We agreed on a summary rating for each program or practice to reflect the overall body of evidence. Using these ratings, we found that only 50 percent of the funds were spent on evidence-based activities. The punch line here is that, for 20 percent of the Title I investments, the conclusions drawn from a single study run counter to conclusions drawn from multiple studies of the same intervention.

v40 36 Hollands Chart C

The takeaway for those of you holding the purse strings is don’t stop at the first positive study you find.

Ironically, the more often a program is studied, the less likely it is always found to be effective. This means that ESSA can inadvertently tip the scales toward less studied—but not necessarily more promising—interventions. So a second lesson is when the evidence is mixed, look carefully at what kinds of students were served in each study, which outcomes were improved, and what skills and resources were needed to implement the program. If these don’t match your own context in at least some studies that found student improvements, be cautious about the likelihood that you can replicate positive results.

There’s also the problem that, for some applications of Title I funds suggested by ESSA, there just isn’t any evidence at all. This means that expecting 100 percent of Title I funds to be invested in evidence-based activities is an unreasonable goal. There are no rigorous studies, for instance, of many widely implemented and seemingly essential programs like student-data tracking and school nurses.

Most important, it’s worth questioning the apparent assumption that educators have the time and skills to search for and evaluate all this evidence. It took us a year to complete our study even though we’ve been trained extensively in how to do this kind of work. Who has that kind of time in a school or district office? It’s unrealistic and inefficient to expect educators and administrators in every school and district to thoroughly investigate each new practice they consider. It would be more helpful for state education departments and district central offices to provide clearer guidance to the decisionmakers they support on suitable investments of federal funds.

See also

Every Student Succeeds Act NCLB Rewrite Sets New Path on School Research
Sarah D. Sparks, January 5, 2016
7 min read

Some education agencies, for example, the New York state education department, are already doing this. Right now, suggestions on how to address learning loss are desperately needed—and these need to be more specific than “high-quality tutoring.” Detailed exemplars are called for to show what that looks like in practice to address different student needs.

Our research leads to one more major piece of advice. Even a well-curated list of evidence-based activities cannot assure effective implementation of promising practices. Schools may be more successful at scaling up effective programs they are already using or at tweaking implementation of their underperforming programs than at introducing brand-new strategies, even if the evidence for them is favorable. For example, one district was surprised to find that Reading Recovery was not producing positive results overall for its students despite ample research evidence to support its effectiveness elsewhere. Instead of ditching the program, district officials carefully reviewed whether schools with the greatest number of struggling readers were getting the appropriate share of trained teachers.

Our own research team’s evidence repository lists items currently in use in districts with which we work and aims to help decisionmakers know where to steer their efforts and their funding. It may behoove states and districts to exert their energy in supporting educators to better implement potentially effective programs than expecting them to become researchers overnight.

A version of this article appeared in the June 16, 2021 edition of Education Week as What Does It Mean To Call a Program ‘Evidence-Based’ Anyway?

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Assessment Webinar
Rethinking STEM Assessment: Strategies for Administrators
School and district leaders will explore strategies to enhance STEM assessment practices across their district, within schools and classrooms.
Content provided by Project Lead The Way
Federal Webinar Keeping Up with the Trump Administration's Latest K-12 Moves: Subscriber-Exclusive Quick Hit
EdWeek subscribers, join this 30-minute webinar to find out what the latest federal policy changes mean for K-12 education.
Artificial Intelligence Live Online Discussion A Seat at the Table: Math & Technology: Finding the Recipe for Student Success
How should we balance AI & math instruction? Join our discussion on preparing future-ready students.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Opinion How Can Principals Grow the Next Generation of Teachers? Listen to My Student
Here’s what made this high schooler want to become a public school teacher.
S. Kambar Khoshaba & Mina Etemadi
4 min read
Conceptual image of growing the next generation of teachers from students.
Vanessa Solis/Education Week via Canva
School & District Management How Los Angeles Schools Are Responding to the ICE Raids and Protests
The school year in Los Angeles is ending amid immigration raids, protests, and the activation of National Guard troops and Marines.
4 min read
Los Angeles police officers with batons and riot gear attempt to move back protesters in downtown Los Angeles on June 9, 2025.
Los Angeles police officers with batons and riot gear attempt to move back protesters in downtown Los Angeles on June 9, 2025. Los Angeles Unified School District Superintendent Alberto Carvalho said the district's police force planned to set up "perimeters of safety" around graduation ceremonies this week amid immigration raids in the city, protests, and the activation of National Guard troops and Marines.
Eric Thayer/AP
School & District Management Download How to Boost Teacher Morale: A Guide for District Leaders (DOWNLOADABLE)
Our discussion guide for district leaders has three takeaways about teachers' attitudes toward their job. Use it to jump start PD with your team.
1 min read
A leader meets with their team. Superintendents, principals, schools leaders, district maps.
Vanessa Solis/Education Week via Canva
School & District Management No More Fresh Fruits and Veggies: Schools Grapple With Loss of Federal Funding
The Local Food to Schools program, which was canceled by the Trump administration, helped schools get fresh, local produce.
7 min read
Dan Yarnick inspects produce at Yarnick's Farm in Indiana, Pa., on June 4, 2025. The farm is one of a number of local providers who partnered with Pittsburgh Public Schools to provide students with fresh fruits and veggies.
Dan Yarnick inspects produce at Yarnick's Farm in Indiana, Pa., on June 4, 2025. The farm is one of a number of local providers who partnered with Pittsburgh Public Schools to provide students with fresh fruits and vegetables. These types of partnerships are in jeopardy with the cancellation of the Local Food for Schools program.
Nate Smallwood for Education Week