Special Report
Student Well-Being

Remote Learning Is Tough for Many Students. How ‘Early-Warning’ Data Can Help Schools Support Them

By Stephen Sawchuk — August 19, 2020 8 min read
BRIC ARCHIVE

Understanding how and why students disengage or are starting to slide academically is especially important, given emerging findings from last spring’s national experiment with remote learning. In surveys, educators reported falling levels of engagement last spring the longer remote learning went on. 

Unlike the other interventions in this series, an early-warning system doesn’t on its own help to re-engage students or fill in learning gaps. Like the lights on a dashboard telling you something’s wrong, an early-warning system uses indicators—missed days, falling grades, or a sudden rash of disciplinary actions—to identify which students need more help. And it provides a consistent framework district and school leaders can use to respond. 

“It’s a nicely oiled machine, but it’s the seamless transitions from grade to grade and relationship building that really makes all of this work, so when you’re hit with a crisis, you can move and adjust,” said Jawana Akuffo, a middle school counselor in the White River school district in Washington state, about her school system’s robust early-warning system that spans both academic and social-emotional learning outcomes. “And by having that transition we can catch those kids who have fallen through the cracks.” 

What indicators make up an early-warning system? 

The purpose of an early-warning system is identification. The district selects a series of indicators—preferably ones that move in real time—that are statistically linked to dropping out, failing to complete major milestones in schooling, and other adverse effects. Generally, the indicators rely on data that districts should already be collecting. 

BRIC ARCHIVE

District and school leaders are confronting difficult, high-stakes decisions as they plan for how to reopen schools amid a global pandemic. Through eight installments, Education Week journalists explore the big challenges education leaders must address, including running a socially distanced school, rethinking how to get students to and from school, and making up for learning losses. We present a broad spectrum of options endorsed by public health officials, explain strategies that some districts will adopt, and provide estimated costs.

Part 1: The Socially Distanced School Day
Part 2: Scheduling the School Year
Part 3: Tackling the Transportation Problem
Part 4: How to Make Remote Learning Work
Part 5: Teaching and Learning
Part 6: Overcoming Learning Loss
Full Report: How We Go Back to School

Then, districts create thresholds for each indicator that are consistent across all schools. For example, Johns Hopkins University, home to many of the researchers who have written on early-warning systems, recommends these thresholds for absenteeism at the high school level: Nine or more absences in a quarter indicate a student is “off track,” a student with five to eight such absences is “sliding,” and one with four or fewer is on track. 

For high school, there’s widespread agreement about what should constitute the indicators: Grade point averages, attendance, course completion, and sometimes behavior data. Tests, it turns out, are not as predictive of dropping out at the high school level as the other combined indicators.

At the middle and elementary levels, attendance, marking period grades, and report card grades are often chosen. Districts have also added indicators they’ve validated on their own. Several districts in Montana, for example, found that mobility was predictive of future academic performance and incorporated that into their indicators, according to Sarah Frazelle, the director of early-warning systems for the Puget Sound Education Service District, in Washington state. Others have chosen 3rd-grade reading ability.

Above all, the system must be simple for educators to use. Indicator data should be compiled in an easy format for educators to review; some districts have hired third-party vendors to do that work, while others cobble together a user-friendly interface on their own.

“It should not be something that’s complicated; it should not be something that’s going to take a lot of extra time. Early-warning systems worked because they were very easy to understand and use so it’s not asking a whole lot in terms of people’s mental bandwidth for learning how to use this new system,” said Elaine Allensworth, the director of the Consortium on School Research at the University of Chicago, which has developed indicators for the Chicago Public Schools and produced many research reports.

What happens to students who are flagged under an early-warning system? 

The key to success is what districts actually do when a student is identified. Sometimes intervention means a “light-touch” approach; students who trigger more flags in the system or don’t respond to initial efforts may need more-extensive help. 

In the White River district, part of weekly professional learning community meetings is devoted to quickly reviewing the early-warning indicators and planning supports for any students they’ve flagged who need academic help. Support staff, counselors, and mental-health experts participate in weekly meetings too, just as teachers do, to review data on attendance, behavior, and social-emotional learning programming.

For example, at the secondary level, teachers on Monday review the academic indicators and academic targets for the week. If a student is having problems, a teacher goes into the student’s online planner to reserve “Hornet Time” or “Grizzly Time,” periods named after the schools’ mascots. During these 30-minute periods, which are worked into the master schedule on Wednesdays, Thursdays, and Fridays, students work with the appropriate teacher to master the specific missing skill or content. (They don’t miss out on regular grade-level teaching.)

Other interventions schools can consider, listed in order from lighter-touch to more intensive, include: 

  • Having a teacher send text messages to a student, if he or she is absent several days in a row or earns low grades on assignments.
  • Ramping up parent engagement, for example by notifying them of homework assignments and quizzes.
  • Prioritizing certain students for counseling.
  • Conducting home visits. 
  • Assigning a staff person as a “case manager” to check in on a student several times a week.
  • Providing mental health services.
  • Providing extended learning time.
  • Providing individual or small-group tutoring.

Finally, once the district has intervened with a student, it needs to follow up over time to see if the extra help is making a difference, or whether further assistance is necessary. 

Do early-warning systems work for all students? 

The research on the systems in high school is solid, concluding that they are better than test scores for identifying which students are at risk of dropping out. High school GPA seems to be an especially powerful indicator. 

But the evidence is sparser on other populations. (One study of English-language learners found that early-warning indicators didn’t do a good job of identifying newcomer ELLs who would later go on to drop out.) 

There’s generally less research evidence about how to structure early-warning systems at the elementary level, where grading is less emphasized, though many districts have adapted them for those grades as well. 

The COVID-19 pandemic raises new questions and challenges for early-warning systems. Certain indicators like attendance take on new meanings, and the White River educators say they are contemplating how to keep the tenets in place as they transition to remote learning. One idea is reserving some in-class time at school for small group interventions; another is learning how to spot new clues that students may need more help.

“How do you find that right balance of online instruction with students interacting with the teacher and with one another from an engagement level, and how are we going to measure it? Maybe it means tracking who is showing up, and who, when we get on a Zoom meeting, always has their camera turned off. Or who isn’t eating well, or isn’t very well kept, or who is saying, ‘I can’t show up to algebra because I have to work,’” said Cody Motherhead, the principal at the district’s high school.

Putting It All Together

Districts will need to be creative about the attendance indicator. 

Attendance is one of the most basic data points every district collects, and a core element of early-warning systems. But in a remote setting, the notion of attendance has taken on a myriad of new meanings. 

Some districts are now looking at whether students are logging into the synchronous portion of instruction or filling in a shared spreadsheet on attendance. The problem with those measures is that they don’t say much about engagement. Researchers suggest a more useful gauge could be tracking whether they are completing assigned work.

“I think that what they will have to track is assignment completion. Anything else is going to be really tricky—all the data about whether students are logging in and how long are they on, a lot of times that’s not very good data,” Allensworth said. 

Plus, assignment completion has the advantage of being workable across both remote and in-person contexts.

Involve a broad section of leadership in the development of indicators and deployment of interventions.  

Amassing, disseminating, and interfacing with the data that powers the indicators can be a challenge, noted Frazelle. Information technology directors sometimes know where data live and can help connect disparate data sets; larger districts can bring their research teams to bear on the problem.

Above all, when you plan interventions, don’t put the responsibility solely on the counseling staff to respond; it’s already overburdened in most districts, she noted. Instead, think about who else could participate and receive training on interventions, including paraeducators, teacher-leaders, or teachers on special assignment.

Similarly, the systems only truly work through a combination of bottom-up and top-down buy-in. Superintendents need to support the data collection and reporting, but it’s the teacher teams and cross-sector communication that makes the interventions happen.

Don’t neglect social-emotional learning. 

Isolation, lack of face-to-face contact, and other new challenges wrought by COVID-19 mean it’s critical to make an early-warning system responsive to SEL needs.

In White River, at the elementary level, the district uses a screener to examine behavior concerns, both externalized (theft, quarreling) and internalized (withdrawn, being bullied). Teachers fill it out three times a year and schools’ teams use the data to plan. And the district provides special SEL programming in the difficult transition years of 7th, 8th, and 9th grade.

The City Year tutoring program, which deploys AmeriCorps volunteers in schools in nearly 30 cities, uses early-warning data to prioritize the students it works with. Part of its intervention relies on involving students in goal setting and recognizing those students when they’ve met goals for attendance or behavior.

“There is some agency that I think is magical, when an AmeriCorps member can facilitate the student articulating for themselves what they need and what they believe can help them be successful to improve their academic performance,” said James Ellout, the managing director of impact for City Year Jacksonville in Jacksonville, Fla. “And they feel heard, and it’s really great to see.” 

Sources: 
Jawana Akuffo, counselor, White River (Wash.) school district; Elaine Allensworth, director, University of Chicago Consortium on School Research; Amy D’Angelo, Regional superintendent and high school lead, Achievement First; Colleen Beaudoin, co-executive director, Springfield Empowerment Zone Partnership; Greg Benjamin, counselor, White River (Wash.) school district; Emily Burdick, Dean of Curriculum & Instruction, Emergency Academy, Springfield, Mass.; Carla Burgi, counselor, White River (Wash.) school district; Robbie Coleman, acting director, National Tutoring Programme; Stephanie Dann, mental health coordinator, White River (Wash.) school district; Heather Hill, professor of education, Harvard Graduate School of Education; James Ellout, managing director for impact, City Year Jacksonville; Sarah Frazelle, director of early warning indicator systems and multitiered systems of support, Puget Sound Education Service District; Emily Freitag, CEO, Instruction Partners; Michelle Kaffenberger, research fellow, Research on Improving Systems of Education (RISE) Programme; Matthew Kraft, associate professor of education and economics, Brown University; Ken Koedinger, professor of Computer Science, Carnegie-Mellon University; Steve Leifsen, executive director, equity and student support services, White River (Wash.) school district; Cody Mothershead, principal, White River (Wash.) school district; Misael Ramos, ELL math teacher and coach, Springfield (Mass.) public schools; Sonja Santelises, superintendent, Baltimore city schools; Beth Schueler, assistant professor of education and public policy, University of Virginia; Nathaniel Schwartz, leader, Brown University Annenberg Institute for School Reform; Anne Sinclair, Chief Learning Officer, Reading and Math, Inc.; Robert Slavin, professor, Johns Hopkins University and director of the Center for Research and Reform in Education; Christine SySantos Levy, special projects coordinator, Johns Hopkins School of Education’s Center for Research and Reform; Stephanie Wu, Chief Impact Officer, City Year.

Studies and Documents:
“Can States Take over and Turn Around School Districts? Evidence from Lawrence, Massachusetts.” 2016. By Beth E. Schueler, Joshua Goodman, and David J. Demming. National Bureau of Economic Research working paper 21895; “Making the Most of School Vacations; A Field Experiment of Small Group Instruction.” 2018. By Beth E. Schueler. Association for Education Finance and Policy; “Improving Academic Outcomes for Students: Scaling Up individualized Tutorials.” 2016. By Roseanna Ander, Jonathan Guryan, and Jens Ludwig. The Hamilton Project, Brookings Institution; “Effective Programs in Elementary Mathematics: A Meta-Analysis.” 2020. By Marta Pellegrini, Cynthia Lake, Amanda Neitzel, and Robert E. Slavin. Best Evidence Encyclopedia; “A Synthesis of Quantitative Research on Programs for Struggling Readers in Elementary Schools.” 2020. By Amanda J. Neitzel, Cynthia Lake, Marta Pellegrini, and Robert Slavin. Best Evidence Encyclopedia; “Failing to Plan? Estimating the Impact of Achieving Schooling Goals on Cohort Learning.” 2020. By Michelle Kaffenberger and Lant Pritchett. Research on Improving Systems of Education; “Human Capital Accumulation and Disasters: Evidence from the Pakistani Earthquake of 2005.” 2020. By Tahiri Andrabi, Benjamin Daniels, and Jishnu Das. Research on Improving Systems of Education; “Remedial Math Goes to High School: An Evaluation of the Tennessee SAILS Program.” 2018. By Thomas J. Kane, Angela Boatman, Whitney Kozakowski, Chris Bennett, Rachel Hitch, Dana Weisenfeld. Center for Education Policy Research, Harvard University; “Impact of School Closures on the Attainment Gap: Rapid Evidence Assessment.” 2020. Education Endowment Foundation; “Remote Learning: Rapid Evidence Assessment.” 2020. Education Endowment Foundation; “COVID-19 and Student Learning Loss in the United States: The Hurt Could Last a Lifetime.” 2020. By Emma Dorn, Bryan Hancock, Jimmy Sarakatsannis, and Ellen Viruleg. McKinsey; “The Impressive Effects of Tutoring on PreK-12 Learning: A Systematic Review and Meta-Analysis of the Experimental Evidence.” 2020. By Andre Nickow, Phillip Oreopoulos, and Vincent Quan; “The (Surprising) Efficacy of Academic and Behavioral Intervention with Disadvantaged Youth: Results from a Randomized Experiment in Chicago.” 2014. By Philip J. Cook, Kenneth Dodge, George Farkas, Roland G. Fryer, Jr., Jonathan Guryan, Jens Ludwig, Susan Mayer, Harold Pollack, and Laurence Steinberg. National Bureau of Economic Research working paper 19862; “The Aftermath of Accelerating Algebra: Evidence from a District Policy Initiative.” 2012. By Charles T. Clotfelter, Helen F. Ladd, and Jacob L. Vigdor. National Bureau of Economic Research working paper 18161; “Aiming High and Falling Short: California’s Eighth-Grade Algebra-for-All Effort.” 2015. By Thurston Domina, Andrew McEachin, Andrew Penner, and Emily Penner. Education Evaluation and Policy Analysis; “A Meta-Analysis of the Effectiveness of Intelligent Tutoring Systems on K-12 Mathematical Learning.” 2013. By Saiying Steenbergen-Hu and Harris Cooper. Journal of Education Psychology; “Intelligent Tutoring Systems and Learning Outcomes: A Meta-Analysis.” 2014. By Wenting Ma, Olusola O. Adesope, John C. Nesbit, and Qing Liu. Journal of Educational Psychology; “Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review.” 2016.  By James A. Kulick and J.D. Fletcher. Review of Educational Research; “A Practitioner’s Guide to Implementing Early Warning Systems.” 2015. By Sarah Frazelle and Aisling Nagel. Education Northwest; “When the Saints Go Marching Out: Long-Term Outcomes for Student Evacuees from Hurricanes Katrina and Rita.” 2012. By Bruce Sacerdote. American Economic Journal: Applied Economics; “School Practices To Address Student Learning Loss.” 2020. By Elaine Allensworth and Nate Schwartz. EdResearch for Recovery, Annenberg Institute, Brown University; “The Big Problem with Little Interruptions to Classroom Learning.” 2020. By Matthew Kraft and Manuel Monti-Nussbaum. The Annenberg Institute at Brown University; “High-Dosage Tutoring and Reading Achievement: Evidence from New York City”.  2017. By Roland G. Fryer, Jr. and Megan Howard Noveck. National Bureau of Economic Research Working Paper 23792; “The Production of Human Capital in Developed Countries: Evidence from 196 Randomized Field Experiments.” 2016. By Roland G. Fryer, Jr., Harvard University, and National Bureau of Economic Research.

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
A Safe Return to Schools is Possible with Testing
We are edging closer to a nationwide return to in-person learning in the fall. However, vaccinations alone will not get us through this. Young children not being able to vaccinate, the spread of new and
Content provided by BD
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
Meeting the Moment: Accelerating Equitable Recovery and Transformative Change
Educators are deciding how best to re-establish routines such as everyday attendance, rebuild the relationships for resilient school communities, and center teaching and learning to consciously prioritize protecting the health and overall well-being of students
Content provided by Campaign for Grade-Level Reading
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Reading & Literacy Webinar
Addressing Learning Loss: What Schools Need to Accelerate Reading Instruction in K-3
When K-3 students return to classrooms this fall, there will be huge gaps in foundational reading skills. Does your school or district need a plan to address learning loss and accelerate student growth? In this
Content provided by PDX Reading

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Student Well-Being Spotlight Spotlight on Student Health & Safety
In this Spotlight, assess what the data says and how educators can play a part in protecting their students, and more.
Student Well-Being Thousands of Kids Lost Parents to COVID-19. Schools Must Prepare to Help the Grieving
While some may view the back-to-school season as a return to “normal,” for those students who’ve lost someone, it will feel anything but.
9 min read
Vickie Quarles, a widow in Memphis, Tenn., lost her husband to COVID-19 in December 2020. She is now raising their five daughters alone. Her older daughter, Alyssa, 18, comforts her in their home.
Vickie Quarles, a widow in Memphis, Tenn., lost her husband to COVID-19 in December 2020. She is now raising their five daughters alone. Her oldest daughter, Alyssa, 18, comforts her in their home.
Karen Pulfer Focht for Education Week
Student Well-Being Nation's Pediatricians Call for All Students, Staff to Wear Masks in School
Countering recent guidelines from the Centers for Disease Control, the physicians say even vaccinated students should wear face coverings
5 min read
Students are reminded to wear a mask amidst other chalk drawings on the sidewalk as they arrive for the first day of school at Union High School in Tulsa, Okla., on Aug. 24, 2020.
A sidewalk-chalk drawing reminds students to wear a mask as they arrive for the first day of school last August at Union High School in Tulsa, Okla.
Mike Simons/Tulsa World via AP
Student Well-Being The Pandemic and Politics Made Life Especially Rough for LGBTQ Youth, Survey Finds
More than 80 percent of 13- to 24-year-olds who say they are lesbian, gay, bisexual, transgender, and questioning said 2020 was very hard.
2 min read
People wave pride flags and hold signs during a rally in support of LGBTQ students at Ridgeline High School, Wednesday, April 14, 2021, in Millville, Utah. Students and school district officials in Utah are outraged after a high school student ripped down a pride flag to the cheers of other students during diversity week. A rally was held the following day in response to show support for the LGBTQ community.
People rally in support of LGBTQ students at Ridgeline High School on April 14, 2021, in Millville, Utah. The day before a high school student ripped down a pride flag to the cheers of other students during diversity week.
Eli Lucero/The Herald Journal via AP