Opinion
School & District Management Opinion

What Does It Mean to Call a Program ‘Evidence-Based’ Anyway?

There’s a good reason not to stop after finding a single positive study
By Fiona Hollands, Yuan Chang & Venita Holmes — June 09, 2021 5 min read
A researcher points to charts and data
  • Save to favorites
  • Print

Millions of federal dollars are currently streaming into schools to help address learning shortfalls related to the pandemic. While it’s not clear what strings might be attached to these new funds, recent versions of the Elementary and Secondary Education Act have encouraged and, for some uses, required education decisionmakers to invest federal funds in evidence-based activities. Many school leaders themselves want evidence to inform their choices. But how does that work in practice?

The current federal approach may seem either needlessly restrictive or absolutely sound, but the fact is the bar for what counts as evidence isn’t awfully high: The Every Student Succeeds Act, the latest ESEA iteration, asks for only one study showing that a program improves student outcomes. And if there isn’t one, a sound rationale for the activity and a plan to evaluate it will often suffice.

To figure out how well schools and districts are able to meet this directive, our research team worked with a large, Southern school system to determine what proportion of its Title I funds are spent on evidence-based activities. Our findings—which relate to prepandemic federal funds—reveal weaknesses both in the law and in the evidence base. As a result, we have some advice to share with educators who are doing their best to adhere to ESSA, serve students well, and now wisely spend large amounts of COVID-19 relief funds in record time.

First, let’s address the question of what should count as adequate evidence when choosing where to invest education funds. Does it really make sense to declare a program “evidence-based” if one study shows positive outcomes for students while a bunch of others don’t?

In our analysis, we first determined what percentage of Title I funds at the district office and four sampled schools were spent on programs and practices backed by just one study: 70 percent in a three-year period. It’s likely that this number would have been substantially higher if we had descended below the top three tiers of evidence laid out in ESSA to that nebulous world of activities for which there is a research-based rationale and an ongoing evaluation plan. To evade this squishiness, we limited our definition of positive evidence to studies using one of three rigorous approaches for evaluating effectiveness: experimental, quasi-experimental, or correlational methods. Additionally, each study needed to show gains for students of approximately the same age as those being served at the district.

Next, we considered all the relevant studies we could find in four research repositories: What Works Clearinghouse, Evidence for ESSA, Education Endowment Foundation, and Education Resources Information Center (ERIC). We agreed on a summary rating for each program or practice to reflect the overall body of evidence. Using these ratings, we found that only 50 percent of the funds were spent on evidence-based activities. The punch line here is that, for 20 percent of the Title I investments, the conclusions drawn from a single study run counter to conclusions drawn from multiple studies of the same intervention.

v40 36 Hollands Chart C

The takeaway for those of you holding the purse strings is don’t stop at the first positive study you find.

Ironically, the more often a program is studied, the less likely it is always found to be effective. This means that ESSA can inadvertently tip the scales toward less studied—but not necessarily more promising—interventions. So a second lesson is when the evidence is mixed, look carefully at what kinds of students were served in each study, which outcomes were improved, and what skills and resources were needed to implement the program. If these don’t match your own context in at least some studies that found student improvements, be cautious about the likelihood that you can replicate positive results.

There’s also the problem that, for some applications of Title I funds suggested by ESSA, there just isn’t any evidence at all. This means that expecting 100 percent of Title I funds to be invested in evidence-based activities is an unreasonable goal. There are no rigorous studies, for instance, of many widely implemented and seemingly essential programs like student-data tracking and school nurses.

Most important, it’s worth questioning the apparent assumption that educators have the time and skills to search for and evaluate all this evidence. It took us a year to complete our study even though we’ve been trained extensively in how to do this kind of work. Who has that kind of time in a school or district office? It’s unrealistic and inefficient to expect educators and administrators in every school and district to thoroughly investigate each new practice they consider. It would be more helpful for state education departments and district central offices to provide clearer guidance to the decisionmakers they support on suitable investments of federal funds.

See also

Every Student Succeeds Act NCLB Rewrite Sets New Path on School Research
Sarah D. Sparks, January 5, 2016
7 min read

Some education agencies, for example, the New York state education department, are already doing this. Right now, suggestions on how to address learning loss are desperately needed—and these need to be more specific than “high-quality tutoring.” Detailed exemplars are called for to show what that looks like in practice to address different student needs.

Our research leads to one more major piece of advice. Even a well-curated list of evidence-based activities cannot assure effective implementation of promising practices. Schools may be more successful at scaling up effective programs they are already using or at tweaking implementation of their underperforming programs than at introducing brand-new strategies, even if the evidence for them is favorable. For example, one district was surprised to find that Reading Recovery was not producing positive results overall for its students despite ample research evidence to support its effectiveness elsewhere. Instead of ditching the program, district officials carefully reviewed whether schools with the greatest number of struggling readers were getting the appropriate share of trained teachers.

Our own research team’s evidence repository lists items currently in use in districts with which we work and aims to help decisionmakers know where to steer their efforts and their funding. It may behoove states and districts to exert their energy in supporting educators to better implement potentially effective programs than expecting them to become researchers overnight.

A version of this article appeared in the June 16, 2021 edition of Education Week as What Does It Mean To Call a Program ‘Evidence-Based’ Anyway?

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Data Webinar
Education Insights with Actionable Data to Create More Personalized Engagement
The world has changed during this time of pandemic learning, and there is a new challenge faced in education regarding how we effectively utilize the data now available to educators and leaders. In this session
Content provided by Microsoft
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
Accelerate Learning with Project-Based Learning
Earlier this year, the George Lucas Educational Foundation released four new studies highlighting how project-based learning (PBL) helps accelerate student learning—across age groups, multiple disciplines, and different socio-economic statuses. With this year’s emphasis on unfinished
Content provided by SmartLab Learning
School & District Management Live Online Discussion Principal Overload: How to Manage Anxiety, Stress, and Tough Decisions
According to recent surveys, more than 40 percent of principals are considering leaving their jobs. With the pandemic, running a school building has become even more complicated, and principals' workloads continue to grow. If we

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management The Already Dire Substitute Shortage Could Get 'Worse Before It Gets Better'
School districts are trying all sorts of tactics, including increasing pay and relaxing requirements, to get more subs in classrooms.
10 min read
Image of an empty classroom.
urfinguss/iStock/Getty
School & District Management Opinion National School Boards Association Chooses to Be Part of the Problem
The NSBA chose to blur the distinction between permissible and suspect speech in suggesting the FBI should target unruly protesters.
5 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
School & District Management Facing Disruption and Firings, L.A. Extends COVID Vaccine Deadline for School Staff
The extension comes as the nation's second-largest school system has struggled to fill more than 2,000 teaching and other vacancies.
Howard Blume, Los Angeles Times
5 min read
In this March 2, 2021, file photo, a sign is displayed at a COVID-19 vaccination site for employees of the Los Angeles School District, LAUSD, in the parking lot of SOFI Stadium in Inglewood, Calif. Public schools have struggled for years with teacher shortages, particularly in math, science, special education and languages. But the pandemic has exacerbated the problem. The stresses of teaching in the COVID-era caused a spike in teacher retirements and resignations. On top of that, schools now have to hire all kinds of additional staff, like tutors and special aides to help kids make up for learning losses, and more teachers to run online school for those not ready to return.
In this March 2, 2021, file photo, a sign is displayed at a COVID-19 vaccination site for employees of the Los Angeles School District, LAUSD, in the parking lot of SOFI Stadium in Inglewood, Calif.
Marcio Jose Sanchez/AP Photo
School & District Management Opinion Graduation Must Depend on Learning, Not Time
We’re long overdue to redesign our education system around competency, argue six superintendents. Here’s what that could look like.
Morcease Beasley, Alberto Carvalho, William Hite, Jesus Jara, Monica Goldson & Jerry Almendarez
5 min read
A conceptual illustration of a mountain of paperwork before the goal is reached.
Vanessa Solis/Education Week and iStock