Understanding how and why students disengage or are starting to slide academically is especially important, given emerging findings from last spring’s national experiment with remote learning. In surveys, educators reported falling levels of engagement last spring the longer remote learning went on.
Unlike the other interventions in this series, an early-warning system doesn’t on its own help to re-engage students or fill in learning gaps. Like the lights on a dashboard telling you something’s wrong, an early-warning system uses indicators—missed days, falling grades, or a sudden rash of disciplinary actions—to identify which students need more help. And it provides a consistent framework district and school leaders can use to respond.
“It’s a nicely oiled machine, but it’s the seamless transitions from grade to grade and relationship building that really makes all of this work, so when you’re hit with a crisis, you can move and adjust,” said Jawana Akuffo, a middle school counselor in the White River school district in Washington state, about her school system’s robust early-warning system that spans both academic and social-emotional learning outcomes. “And by having that transition we can catch those kids who have fallen through the cracks.”
What indicators make up an early-warning system?
The purpose of an early-warning system is identification. The district selects a series of indicators—preferably ones that move in real time—that are statistically linked to dropping out, failing to complete major milestones in schooling, and other adverse effects. Generally, the indicators rely on data that districts should already be collecting.
District and school leaders are confronting difficult, high-stakes decisions as they plan for how to reopen schools amid a global pandemic. Through eight installments, Education Week journalists explore the big challenges education leaders must address, including running a socially distanced school, rethinking how to get students to and from school, and making up for learning losses. We present a broad spectrum of options endorsed by public health officials, explain strategies that some districts will adopt, and provide estimated costs.
Part 1: The Socially Distanced School Day
Part 2: Scheduling the School Year
Part 3: Tackling the Transportation Problem
Part 4: How to Make Remote Learning Work
Part 5: Teaching and Learning
Part 6: Overcoming Learning Loss
Full Report: How We Go Back to School
Then, districts create thresholds for each indicator that are consistent across all schools. For example, Johns Hopkins University, home to many of the researchers who have written on early-warning systems, recommends these thresholds for absenteeism at the high school level: Nine or more absences in a quarter indicate a student is “off track,” a student with five to eight such absences is “sliding,” and one with four or fewer is on track.
For high school, there’s widespread agreement about what should constitute the indicators: Grade point averages, attendance, course completion, and sometimes behavior data. Tests, it turns out, are not as predictive of dropping out at the high school level as the other combined indicators.
At the middle and elementary levels, attendance, marking period grades, and report card grades are often chosen. Districts have also added indicators they’ve validated on their own. Several districts in Montana, for example, found that mobility was predictive of future academic performance and incorporated that into their indicators, according to Sarah Frazelle, the director of early-warning systems for the Puget Sound Education Service District, in Washington state. Others have chosen 3rd-grade reading ability.
Above all, the system must be simple for educators to use. Indicator data should be compiled in an easy format for educators to review; some districts have hired third-party vendors to do that work, while others cobble together a user-friendly interface on their own.
“It should not be something that’s complicated; it should not be something that’s going to take a lot of extra time. Early-warning systems worked because they were very easy to understand and use so it’s not asking a whole lot in terms of people’s mental bandwidth for learning how to use this new system,” said Elaine Allensworth, the director of the Consortium on School Research at the University of Chicago, which has developed indicators for the Chicago Public Schools and produced many research reports.
What happens to students who are flagged under an early-warning system?
The key to success is what districts actually do when a student is identified. Sometimes intervention means a “light-touch” approach; students who trigger more flags in the system or don’t respond to initial efforts may need more-extensive help.
In the White River district, part of weekly professional learning community meetings is devoted to quickly reviewing the early-warning indicators and planning supports for any students they’ve flagged who need academic help. Support staff, counselors, and mental-health experts participate in weekly meetings too, just as teachers do, to review data on attendance, behavior, and social-emotional learning programming.
For example, at the secondary level, teachers on Monday review the academic indicators and academic targets for the week. If a student is having problems, a teacher goes into the student’s online planner to reserve “Hornet Time” or “Grizzly Time,” periods named after the schools’ mascots. During these 30-minute periods, which are worked into the master schedule on Wednesdays, Thursdays, and Fridays, students work with the appropriate teacher to master the specific missing skill or content. (They don’t miss out on regular grade-level teaching.)
Other interventions schools can consider, listed in order from lighter-touch to more intensive, include:
- Having a teacher send text messages to a student, if he or she is absent several days in a row or earns low grades on assignments.
- Ramping up parent engagement, for example by notifying them of homework assignments and quizzes.
- Prioritizing certain students for counseling.
- Conducting home visits.
- Assigning a staff person as a “case manager” to check in on a student several times a week.
- Providing mental health services.
- Providing extended learning time.
- Providing individual or small-group tutoring.
Finally, once the district has intervened with a student, it needs to follow up over time to see if the extra help is making a difference, or whether further assistance is necessary.
Do early-warning systems work for all students?
The research on the systems in high school is solid, concluding that they are better than test scores for identifying which students are at risk of dropping out. High school GPA seems to be an especially powerful indicator.
But the evidence is sparser on other populations. (One study of English-language learners found that early-warning indicators didn’t do a good job of identifying newcomer ELLs who would later go on to drop out.)
There’s generally less research evidence about how to structure early-warning systems at the elementary level, where grading is less emphasized, though many districts have adapted them for those grades as well.
The COVID-19 pandemic raises new questions and challenges for early-warning systems. Certain indicators like attendance take on new meanings, and the White River educators say they are contemplating how to keep the tenets in place as they transition to remote learning. One idea is reserving some in-class time at school for small group interventions; another is learning how to spot new clues that students may need more help.
“How do you find that right balance of online instruction with students interacting with the teacher and with one another from an engagement level, and how are we going to measure it? Maybe it means tracking who is showing up, and who, when we get on a Zoom meeting, always has their camera turned off. Or who isn’t eating well, or isn’t very well kept, or who is saying, ‘I can’t show up to algebra because I have to work,’” said Cody Motherhead, the principal at the district’s high school.
Putting It All Together
Districts will need to be creative about the attendance indicator.
Attendance is one of the most basic data points every district collects, and a core element of early-warning systems. But in a remote setting, the notion of attendance has taken on a myriad of new meanings.
Some districts are now looking at whether students are logging into the synchronous portion of instruction or filling in a shared spreadsheet on attendance. The problem with those measures is that they don’t say much about engagement. Researchers suggest a more useful gauge could be tracking whether they are completing assigned work.
“I think that what they will have to track is assignment completion. Anything else is going to be really tricky—all the data about whether students are logging in and how long are they on, a lot of times that’s not very good data,” Allensworth said.
Plus, assignment completion has the advantage of being workable across both remote and in-person contexts.
Involve a broad section of leadership in the development of indicators and deployment of interventions.
Amassing, disseminating, and interfacing with the data that powers the indicators can be a challenge, noted Frazelle. Information technology directors sometimes know where data live and can help connect disparate data sets; larger districts can bring their research teams to bear on the problem.
Above all, when you plan interventions, don’t put the responsibility solely on the counseling staff to respond; it’s already overburdened in most districts, she noted. Instead, think about who else could participate and receive training on interventions, including paraeducators, teacher-leaders, or teachers on special assignment.
Similarly, the systems only truly work through a combination of bottom-up and top-down buy-in. Superintendents need to support the data collection and reporting, but it’s the teacher teams and cross-sector communication that makes the interventions happen.
Don’t neglect social-emotional learning.
Isolation, lack of face-to-face contact, and other new challenges wrought by COVID-19 mean it’s critical to make an early-warning system responsive to SEL needs.
In White River, at the elementary level, the district uses a screener to examine behavior concerns, both externalized (theft, quarreling) and internalized (withdrawn, being bullied). Teachers fill it out three times a year and schools’ teams use the data to plan. And the district provides special SEL programming in the difficult transition years of 7th, 8th, and 9th grade.
The City Year tutoring program, which deploys AmeriCorps volunteers in schools in nearly 30 cities, uses early-warning data to prioritize the students it works with. Part of its intervention relies on involving students in goal setting and recognizing those students when they’ve met goals for attendance or behavior.
“There is some agency that I think is magical, when an AmeriCorps member can facilitate the student articulating for themselves what they need and what they believe can help them be successful to improve their academic performance,” said James Ellout, the managing director of impact for City Year Jacksonville in Jacksonville, Fla. “And they feel heard, and it’s really great to see.”
Jawana Akuffo, counselor, White River (Wash.) school district; Elaine Allensworth, director, University of Chicago Consortium on School Research; Amy D’Angelo, Regional superintendent and high school lead, Achievement First; Colleen Beaudoin, co-executive director, Springfield Empowerment Zone Partnership; Greg Benjamin, counselor, White River (Wash.) school district; Emily Burdick, Dean of Curriculum & Instruction, Emergency Academy, Springfield, Mass.; Carla Burgi, counselor, White River (Wash.) school district; Robbie Coleman, acting director, National Tutoring Programme; Stephanie Dann, mental health coordinator, White River (Wash.) school district; Heather Hill, professor of education, Harvard Graduate School of Education; James Ellout, managing director for impact, City Year Jacksonville; Sarah Frazelle, director of early warning indicator systems and multitiered systems of support, Puget Sound Education Service District; Emily Freitag, CEO, Instruction Partners; Michelle Kaffenberger, research fellow, Research on Improving Systems of Education (RISE) Programme; Matthew Kraft, associate professor of education and economics, Brown University; Ken Koedinger, professor of Computer Science, Carnegie-Mellon University; Steve Leifsen, executive director, equity and student support services, White River (Wash.) school district; Cody Mothershead, principal, White River (Wash.) school district; Misael Ramos, ELL math teacher and coach, Springfield (Mass.) public schools; Sonja Santelises, superintendent, Baltimore city schools; Beth Schueler, assistant professor of education and public policy, University of Virginia; Nathaniel Schwartz, leader, Brown University Annenberg Institute for School Reform; Anne Sinclair, Chief Learning Officer, Reading and Math, Inc.; Robert Slavin, professor, Johns Hopkins University and director of the Center for Research and Reform in Education; Christine SySantos Levy, special projects coordinator, Johns Hopkins School of Education’s Center for Research and Reform; Stephanie Wu, Chief Impact Officer, City Year.
Studies and Documents:
“Can States Take over and Turn Around School Districts? Evidence from Lawrence, Massachusetts.” 2016. By Beth E. Schueler, Joshua Goodman, and David J. Demming. National Bureau of Economic Research working paper 21895; “Making the Most of School Vacations; A Field Experiment of Small Group Instruction.” 2018. By Beth E. Schueler. Association for Education Finance and Policy; “Improving Academic Outcomes for Students: Scaling Up individualized Tutorials.” 2016. By Roseanna Ander, Jonathan Guryan, and Jens Ludwig. The Hamilton Project, Brookings Institution; “Effective Programs in Elementary Mathematics: A Meta-Analysis.” 2020. By Marta Pellegrini, Cynthia Lake, Amanda Neitzel, and Robert E. Slavin. Best Evidence Encyclopedia; “A Synthesis of Quantitative Research on Programs for Struggling Readers in Elementary Schools.” 2020. By Amanda J. Neitzel, Cynthia Lake, Marta Pellegrini, and Robert Slavin. Best Evidence Encyclopedia; “Failing to Plan? Estimating the Impact of Achieving Schooling Goals on Cohort Learning.” 2020. By Michelle Kaffenberger and Lant Pritchett. Research on Improving Systems of Education; “Human Capital Accumulation and Disasters: Evidence from the Pakistani Earthquake of 2005.” 2020. By Tahiri Andrabi, Benjamin Daniels, and Jishnu Das. Research on Improving Systems of Education; “Remedial Math Goes to High School: An Evaluation of the Tennessee SAILS Program.” 2018. By Thomas J. Kane, Angela Boatman, Whitney Kozakowski, Chris Bennett, Rachel Hitch, Dana Weisenfeld. Center for Education Policy Research, Harvard University; “Impact of School Closures on the Attainment Gap: Rapid Evidence Assessment.” 2020. Education Endowment Foundation; “Remote Learning: Rapid Evidence Assessment.” 2020. Education Endowment Foundation; “COVID-19 and Student Learning Loss in the United States: The Hurt Could Last a Lifetime.” 2020. By Emma Dorn, Bryan Hancock, Jimmy Sarakatsannis, and Ellen Viruleg. McKinsey; “The Impressive Effects of Tutoring on PreK-12 Learning: A Systematic Review and Meta-Analysis of the Experimental Evidence.” 2020. By Andre Nickow, Phillip Oreopoulos, and Vincent Quan; “The (Surprising) Efficacy of Academic and Behavioral Intervention with Disadvantaged Youth: Results from a Randomized Experiment in Chicago.” 2014. By Philip J. Cook, Kenneth Dodge, George Farkas, Roland G. Fryer, Jr., Jonathan Guryan, Jens Ludwig, Susan Mayer, Harold Pollack, and Laurence Steinberg. National Bureau of Economic Research working paper 19862; “The Aftermath of Accelerating Algebra: Evidence from a District Policy Initiative.” 2012. By Charles T. Clotfelter, Helen F. Ladd, and Jacob L. Vigdor. National Bureau of Economic Research working paper 18161; “Aiming High and Falling Short: California’s Eighth-Grade Algebra-for-All Effort.” 2015. By Thurston Domina, Andrew McEachin, Andrew Penner, and Emily Penner. Education Evaluation and Policy Analysis; “A Meta-Analysis of the Effectiveness of Intelligent Tutoring Systems on K-12 Mathematical Learning.” 2013. By Saiying Steenbergen-Hu and Harris Cooper. Journal of Education Psychology; “Intelligent Tutoring Systems and Learning Outcomes: A Meta-Analysis.” 2014. By Wenting Ma, Olusola O. Adesope, John C. Nesbit, and Qing Liu. Journal of Educational Psychology; “Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review.” 2016. By James A. Kulick and J.D. Fletcher. Review of Educational Research; “A Practitioner’s Guide to Implementing Early Warning Systems.” 2015. By Sarah Frazelle and Aisling Nagel. Education Northwest; “When the Saints Go Marching Out: Long-Term Outcomes for Student Evacuees from Hurricanes Katrina and Rita.” 2012. By Bruce Sacerdote. American Economic Journal: Applied Economics; “School Practices To Address Student Learning Loss.” 2020. By Elaine Allensworth and Nate Schwartz. EdResearch for Recovery, Annenberg Institute, Brown University; “The Big Problem with Little Interruptions to Classroom Learning.” 2020. By Matthew Kraft and Manuel Monti-Nussbaum. The Annenberg Institute at Brown University; “High-Dosage Tutoring and Reading Achievement: Evidence from New York City”. 2017. By Roland G. Fryer, Jr. and Megan Howard Noveck. National Bureau of Economic Research Working Paper 23792; “The Production of Human Capital in Developed Countries: Evidence from 196 Randomized Field Experiments.” 2016. By Roland G. Fryer, Jr., Harvard University, and National Bureau of Economic Research.