School & District Management

How Schools Can Identify ‘Evidence-Based’ Programs That Could Actually Work

Federal law urges educators to use evidence-based interventions for school improvement. What does that actually mean?
By Sarah D. Sparks — September 03, 2024 4 min read
Animated graphic showing the concepts of research, data, and analysis
  • Save to favorites
  • Print

Amid shrinking budgets, multiplying federal and state requirements for school improvement, and rapidly changing educational technology, it’s critical for education leaders to identify whether a given program will really improve learning for their students.

While leaders have more access than ever to education program evaluations and research clearinghouses, experts say it’s easy to overlook red flags in studies and evaluations of particular programs and interventions.

“It’s very challenging, and we’re asking educators whose job isn’t evidence to apply this to making their work more effective,” Nancy Madden, co-founder of the Success for All Foundation and an education professor at the Center for Research and Reform in Education (CRRE) at Johns Hopkins University, said during an Education Week webinar last week. “Educators are just now beginning to get down into the weeds enough to figure out how to make that evidence matter.”

Madden; Robert Peters, the senior vice president and chief academic officer for the nonprofit education research clearinghouses the Successful Practices Network and the National Dropout Prevention Center; and Amanda Neitzel, CRRE’s deputy director of evidence research, highlighted ways education leaders can find evidence-based interventions more effectively.

What does ‘evidence based’ mean?

The Every Student Succeeds Act formally defines three tiers of research evidence indicating how promising researchers have found a particular intervention to be. (Some states also allow a fourth tier for early studies exploring a new program with a strong rationale.) These tiers are used mostly to identify appropriate programs for school improvement under the law. To fit in any tier, the study must be well-conducted and completed by a researcher who is independent of the program, and it must show statistically significant findings.

  1. Tier 1 (strong) studies track the outcomes of a large group—350 or more students, teachers, or other participants. These studies randomly assign participants to the intervention or a control group in an effort to show cause and effect. This practice is considered the gold standard for causal research.
  2. Tier 2 (moderate) evidence comes from a study that also includes 350 or more participants tracked over time, but may use non-randomized, quasi-experimental designs in situations in which it is not ethical or possible to randomly assign participants.
  3. Tier 3 (promising) evidence may have a smaller sample group, use correlational versus causal designs, or be retrospective (with analysis performed after the intervention) rather than follow the sample over time.

Madden noted that terms like “research-based” and “evidence-based” are often used interchangeably to mean the same thing. However, different research groups and clearinghouses often disagree about exactly what is needed to prove that an intervention is effective.

The federal Institute of Education Sciences’ What Works Clearinghouse and CRRE have adopted the ESSA tiers when evaluating research. Neitzel said the highest-quality studies also follow a standardized intervention for at least 12 weeks, and include treatment and control groups that are similar and have had relatively few participants drop out.

Will an evidence-based intervention actually work in the classroom?

Many programs and interventions also sound great on paper or in the lab, but fizzle when they get to the classroom.

It’s important, experts said, to look for red flags in a study that suggest it could be hard to implement in a new class or school.

For example, no intervention will work if students don’t receive it, so an evaluation should state how much time students actually participated in a program that produced particular results.

“We see often, especially in cases of education technology, studies will compare students who used the program a lot to students who didn’t use it at all, and drop out all those kids in the middle who had low usage,” Neitzel said. “While that will tell you how this works for kids who use it five days a week, consistently for the whole year, it doesn’t necessarily tell you how it’s going to work for every one of your students.”

Even if a program shows strong evidence of effectiveness, it’s important for leaders and educators to consider how easily they could replicate the model with their own students.

“Choosing something with great evidence that doesn’t address the problem in your situation is not going to make the progress that you want to make,” Madden said.

She recommended that education leaders try to look for insight into the following questions when reading education research.

  • What training do teachers or other intervention staff need to implement the program effectively?
  • What aspects of the program may be harder to implement in your district? For example, rural districts may have less access to staff or resources used to implement a program in an urban school.
  • How will the school monitor the effectiveness of the intervention with its own students?

Peters, who works with districts on improvement plans, said the highest-performing school systems also analyze the cost-effectiveness of their programs at the end of each school year to understand how to prioritize them.

“A lot of times we see large school districts across the country spend millions of dollars on programs that are not even connected to their strategic planning,” Peters said.

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Artificial Intelligence Webinar
Teaching Students to Use Artificial Intelligence Ethically
Ready to embrace AI in your classroom? Join our master class to learn how to use AI as a tool for learning, not a replacement.
Content provided by Solution Tree
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
Empowering Students Using Computational Thinking Skills
Empower your students with computational thinking. Learn how to integrate these skills into your teaching and boost student engagement.
Content provided by Project Lead The Way
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
IT Infrastructure & Management Webinar
The Reality of Change: How Embracing and Planning for Change Can Shape Your Edtech Strategy
Promethean edtech experts delve into the reality of tech change and explore how embracing and planning for it can be your most powerful strategy for maximizing ROI.
Content provided by Promethean

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Opinion I Invited Students to Help Hire a New Assistant Principal. Here’s What Happened
What began as an opportunity for the students turned into a gift for our administrative team.
3 min read
Centering students in the school community.
Vanessa Solis/Education Week via Canva
School & District Management Opinion 5 Education Leadership Lessons From Chef Ina Garten
"Less is more," "quality is everything," and more tips inspired by the art of cooking to build trust, connection in your school community.
4 min read
Screen Shot 2024 11 29 at 1.11.40 PM
Canva
School & District Management Elon Musk Is Opening a School for Young Students. Here’s What We Know About It
The tech billionaire has claimed that the current Education Department is "basically paying people to hate America."
4 min read
Elon Musk listens as President-elect Donald Trump speaks during a meeting with the House GOP conference on Nov. 13 in Washington.
Elon Musk listens as President-elect Donald Trump speaks during a meeting with the House GOP conference, Wednesday, Nov. 13, 2024, in Washington.
Allison Robbert/AP
School & District Management A Principal Was Put on Leave for Her Election Message. What Leaders Need to Know
Principals have to tread a fine line to avoid getting too political in their role as public school leaders.
7 min read
Illustration of two people confined within red and blue circles.
iStock