Opinion
School & District Management Opinion

Why Evidence-Backed Programs Might Fall Short in Your School (And What To Do About It)

How close a program’s implementation matches its plan is important, though perhaps not as important as you think
By Heather C. Hill — May 25, 2021 5 min read
The process from idea to practice in the classroom.
  • Save to favorites
  • Print

Editor’s Note: This is part of a continuing series on the practical takeaways from research.

As funds from the American Rescue Plan start to arrive in schools and districts, many educators will find themselves tasked with quickly implementing new programs to meet the needs of students with pandemic-related unfinished learning. These programs include high-dosage tutoring, summer learning experiences, and school year curricula with linked professional development.

Many schools will choose such programs based on their evidence of success, perhaps using academic reviews or websites (like the late Robert Slavin’s evidenceforessa.org) that report specific programs’ effects on student achievement. Yet most school leaders also know that implementing a program does not automatically yield the promised bump in student achievement.

It’s a conundrum: Why do research syntheses find large effects on student learning, but schools less often realize the gains?

One reason is that the effect sizes promised by meta-analyses—studies that statistically combine the results of multiple evaluations—can be overly large: Small pilot programs are likely to make up the bulk of studies in meta-analyses, and they tend to yield larger effect sizes, which are then not replicated under real-world conditions. Another reason is that implementation—the degree to which a program or practice is carried out with fidelity—can sharply affect program outcomes.

With Anna Erickson, a researcher at the University of Michigan, I recently examined 72 studies of math, science, English/language arts, and social-emotional-learning programs, comparing the extent of program implementation to the likelihood of positive student outcomes. As expected, when schools did not adopt or maintain program elements, evaluations saw few positive impacts on students. This echoes findings from studies of individual programs, where positive effects often occur only in schools and classrooms with higher levels of fidelity.

However, in good news for schools, our analysis found that both moderate- and high-fidelity implementation tended to produce positive effects of similar size. Perfect need not be the enemy of the good when it comes to the implementation of new programs.

When principals and instructional leaders signal commitment to a program and follow up with set-asides of resources and time to learn about the program, teachers are more likely to follow their lead.

But low fidelity doesn’t cut it. So many district and school administrators will still want to work for better implementation. But how? Qualitative studies of implementation failure suggest addressing four root causes:

  • Will—literally, whether teachers decide to embrace new practices or materials or to shelve them. It’s worth noting that in some cases, teachers’ decisions to forgo new materials make sense in light of their students’ needs.
  • Skill—whether teachers have the knowledge and expertise to implement a given program. For instance, case studies suggest STEM teachers who lack strong content knowledge may provide disorganized and error-filled instruction when using science, technology, engineering, or math curriculum materials that focus on concepts and build in student inquiry.
  • Organizational capacity—whether an organization has in place tools, routines, and relationships that enable implementation. Capacity encompasses a range of factors, from the quality of connections between central-office staff and principals to the ability to overcome seemingly simple logistics challenges, like distributing curriculum materials to students.
  • Contexts and coherence—whether the program will work given its particulars and the local setting’s needs, strengths, and weaknesses. This includes whether the program aligns with existing instructional guidance as represented in school priorities, pacing guides, and tested content and also whether the program reflects students’ interests, strengths, and culture.

While most post-mortems of failed programs point to one of these four categories as the culprit, research on how to proactively address them is more primitive. However, several studies suggest ways school leaders can plan for a successful implementation.

School leadership is key to increasing teachers’ willingness to take up new programs and practices. When principals and instructional leaders signal commitment to a program and follow up with set-asides of resources and time to learn about the program, teachers are more likely to follow their lead.

Allowing teachers to help shape program adoption can also increase their commitment to the program and potentially help avoid a bad fit between the program and local context. Program “touch points” with teachers after initial program training—for instance, a brief workshop a few weeks after implementation begins—can also boost teacher will, program fidelity, and student performance.

Teacher skill in using a new program can be enhanced by concrete lesson materials: curricula, routines, or assessments. Professional development and coaching on practices specific to the new program also seem to build teacher skill and interest.

Enhancing organizational capacity to support implementation may be a heavier lift. Ideally, districts thinking about adopting a program or practice would assess their “readiness”: leader and teacher commitment to the program, earmarked resources for implementation, and the extent of conflict with existing programs or other instructional guidance. District leaders would either resolve these issues or find alternative programs. But little empirical evidence exists for how to do so.

Addressing context and coherence can also be tricky. Adapting programs to better fit local contexts is a promising practice but with a caveat: Teachers must implement with fidelity before adapting the program. James Kim of the Harvard Graduate School of Education and colleagues found such a result in a randomized study. They assigned a group of teachers first to a year-long, fidelity-oriented implementation of a reading program, then to a year of adaptation. During the adaptation year, teachers did things like better matching books to students’ reading level and adding a parent-outreach component. Student outcomes for this adaptation group outstripped those of both teachers who continued to implement the program with fidelity and those who adapted the program without first implementing with fidelity.

Finally, districts must take steps to protect the program following initial implementation, including continued professional development and mentoring in the program for new teachers. With average teacher turnover around 20 percent per year, getting new teachers up to speed on a particular program can both boost overall implementation and keep veteran teachers engaged.

Protecting the program also means preventing it from being “crowded out” by other initiatives. Too often, teachers report their work to learn a particular math or ELA program comes undone when their school or district adopts a conflicting classroom-observation rubric, pacing guide, or even a competing curriculum.

A program vetted by evidence is important for results, but so is implementation. With carefully crafted implementation plans, schools should see more returns on their American Rescue Plan investments.

Events

School Climate & Safety K-12 Essentials Forum Strengthen Students’ Connections to School
Join this free event to learn how schools are creating the space for students to form strong bonds with each other and trusted adults.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Reading & Literacy Webinar
Creating Confident Readers: Why Differentiated Instruction is Equitable Instruction
Join us as we break down how differentiated instruction can advance your school’s literacy and equity goals.
Content provided by Lexia Learning
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
IT Infrastructure & Management Webinar
Future-Proofing Your School's Tech Ecosystem: Strategies for Asset Tracking, Sustainability, and Budget Optimization
Gain actionable insights into effective asset management, budget optimization, and sustainable IT practices.
Content provided by Follett Learning

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Q&A How K-12 Leaders Can Better Manage Divisive Curriculum and Culture War Debates
The leader of an effort to equip K-12 leaders with conflict resolution skills urges relationship-building—and knowing when to disengage.
7 min read
Katy Anthes, Commissioner of Education in Colorado from 2016- 2023, participates in a breakout session during the Education Week Leadership Symposium on May 3, 2024.
Katy Anthes, who served as commissioner of education in Colorado from 2016-2023, participates in a breakout session during the Education Week Leadership Symposium on May 3, 2024. Anthes specializes in helping school district leaders successfully manage politically charged conflicts.
Chris Ferenzi for Education Week
School & District Management Virginia School Board Restores Confederate Names to 2 Schools
The vote reverses a decision made in 2020 as dozens of schools nationwide dropped Confederate figures from their names.
2 min read
A statue of confederate general Stonewall Jackson is removed on July 1, 2020, in Richmond, Va. Shenandoah County, Virginia's school board voted 5-1 early Friday, May 10, 2024, to rename Mountain View High School as Stonewall Jackson High School and Honey Run Elementary as Ashby Lee Elementary four years after the names had been removed.
A statue of confederate general Stonewall Jackson is removed on July 1, 2020, in Richmond, Va. Shenandoah County, Virginia's school board voted 5-1 early Friday, May 10, 2024, to rename Mountain View High School as Stonewall Jackson High School and Honey Run Elementary as Ashby Lee Elementary four years after the names had been removed.
Steve Helber/AP
School & District Management Quiz Quiz Yourself: How Much Do You Know About the School District Technology Leader?
The tech director at school districts is a key player when it comes to purchasing. Test your knowledge of this key buyer persona and see how your results stack up with your peers.
School & District Management Deepfakes Expose Public School Employees to New Threats
The only protection for school leaders is a healthy dose of skepticism.
7 min read
Signage is shown outside on the grounds of Pikesville High School, May 2, 2012, in Baltimore County, Md. The most recent criminal case involving artificial intelligence emerged in late April 2024, from the Maryland high school, where police say a principal was framed as racist by a fake recording of his voice.
Police say a principal was framed making racist remarks through a fake recording of his voice at Pikesville High School, a troubling new use of AI that could affect more educators. A sign announces the entrance to the Baltimore County, Md., school on May 2, 2012.
Lloyd Fox/The Baltimore Sun via AP