Over the last few years, researchers and policymakers have promoted tutoring as a key pandemic-recovery tool, citing the large body of evidence demonstrating that it can effectively move the needle on student academic outcomes.
But scaling up the kind of sustained, one-on-one or small-group programs that have been shown to improve student achievement has posed a challenge for many districts. This high-dosage model requires lots of tutors, dedicated time and money, and a strategy for reaching students most in need.
A new study of the Nashville public schools’ tutoring strategy evaluates one district’s attempt to thread this needle. The Tennessee district’s in-house program served more than 6,800 students over its first three years. But during that period, tutoring had a small to medium positive effect on students’ reading test scores and no effect on math test scores—less of a boost to student achievement than previous studies would suggest.
The research, published as an EdWorkingPaper, is one of the first pandemic-era studies to examine a district-designed tutoring program. The in-house effort differs from partnerships with external providers, which would supply tutors they hired and trained.
The findings suggest that tutoring at scale and operating in the real, often messy context of large districts may have a smaller impact than advocates might have hoped.
“Two things can be true,” said Matthew Kraft, an associate professor of education and economics at Brown University and one of the paper’s co-authors.
“The effects that are possible at scale are still large and meaningful. And the effects that we hoped for, and find in reviews of the literature—pre-COVID, in person, best implementation contexts—are also larger than we might be able to realize in most contexts at scale.”
That’s why these smaller effects shouldn’t be dismissed, he said. It’s a view that Sarah Chin, the chief strategy officer for the Nashville public schools, shares.
“Moderate [or] small effect sizes on large numbers of students is a very important strategy for large urban districts to be pursuing,” Chin said. “When we can reach a large number of students and have moderate effects for that population, that is a huge win for us.”
Choosing to build a tutoring infrastructure within the district, rather than working solely with an outside organization, means that Nashville will be able to continue operating the program even when federal ESSER funds expire, Chin said.
Still, there are logistical challenges to building a program from scratch, said Matthew Steinberg, the managing director of research and evaluation at Accelerate, a national initiative that funds effective tutoring programs. Steinberg was not involved with the study.
“What I don’t take away from this is tutoring doesn’t work. What I take away from this report is that there’s a lot of moving parts when trying to scale a very niche model to about 10 percent of the district,” Steinberg said, referencing the more than 6,800 students who received tutoring.
Research disentangles the effects of tutoring amid other recovery efforts
Nashville rapidly expanded its tutoring program over three years, starting in spring 2021 with three schools and eventually reaching 101 of the 128 traditional public schools in the system (the study did not include the district’s 31 public charter schools).
The district targeted students who scored between the 15th and 60th percentile on diagnostic assessments. They received 90 minutes of tutoring per week, though the duration of the program varied—students in the pilot received eight weeks of tutoring, those in the 2021-22 school year received 10 weeks, and those in 2022-23 received 12 weeks.
Students worked with tutors one-on-one or in small groups on district-created lessons, designed to be aligned to the core curriculum. Tutoring took place both during and before and after the school day. About three quarters of tutors were current teachers and other district staff; the rest were a mix of local undergraduate students, volunteers, and retired teachers.
One big takeaway from the study is that scaling tutoring at the district level in this way is feasible, said Kraft. “Most prior and ongoing efforts in the wake of COVID to scale up very rapidly have relied on third-party providers to be close partners, to recruit and train and oversee tutors,” he said.
Accelerating Scholars was one part of a broader academic-recovery strategy in Nashville, which also included implementing new core curricula in English/language arts and math. The district has reported year-over-year gains in test scores since 2022.
To disentangle the effects of tutoring from these broader improvements, the study compared the academic growth of students who received services with the growth of students who did not, at different points between spring 2021 and spring 2023. The researchers used statistical models to control for differences between students, so as to isolate the effect of the tutoring program alone.
In reading, tutoring had a small to medium effect on student test scores. There was no average effect on math test scores and no effects on end-of-course grades for either math or ELA.
Tutoring saw larger effects for higher-performing students
There are several potential reasons why this tutoring program didn’t improve student scores as much as those in previous studies, the researchers write.
One possibility has to do with other services that students who weren’t being tutored were receiving instead.
About half of students who received tutoring did so during a block of time in the school day dedicated to personalized learning. Students who weren’t getting tutoring during that time, students in the control group, were involved in other academic activities—going to intervention specialists, for example, or working on computer adaptive enrichment programs. This differs from many other studies, Kraft said, in which the control group isn’t receiving any academic enrichment.
It’s possible that these personalized learning activities boosted student achievement, too, and that tutoring didn’t provide that much of an edge over and above other personalized learning.
Another possibility is that the tutoring Nashville designed didn’t benefit the students who needed it most.
The district’s program had the largest effects for students who were in the middle to upper segment of the performance distribution, rather than at the bottom. This could be due to how the tutoring materials were written, the researchers hypothesize. The district created universal lessons aligned to grade-level content, rather than tailor instruction to individual students’ specific needs.
This heterogeneity in the impacts of tutoring highlights how important it is for districts to think intentionally about designing and implementing programs to maximize impact, Steinberg said. Could the program see stronger results, he asked, if it were targeted to students with the greatest need?
It’s a question that Chin and her team in Nashville are thinking about as well.
The research has helped the district refine its messaging for schools about which students to target, emphasizing the importance of reaching students who are below the 50th percentile, said Grace Bailey, the director of the Accelerating Scholars program.
Implementing districtwide programming of this kind, and changing student outcomes as a result, is a “slow, incremental process,” said Danielle Sanderson Edwards, an assistant professor of educational leadership and workforce development at Old Dominion University in Norfolk, Va., and a co-author of the paper.
“To expect these things to change quickly in the short term is likely unrealistic,” she emphasized.
