One of the most highly touted personalized-learning models in K-12 education had no significant impact on students’ scores on state math tests, according to a new, federally funded evaluation released this week.
But a second, less-rigorous study of the “Teach to One: Math” program, funded by the Bill & Melinda Gates Foundation and also released this week, suggested the program helps accelerate students’ academic growth in ways that most state tests don’t measure.
The mixed results highlight a rising source of friction in a K-12 sector increasingly experimenting with new approaches to customizing the learning experience for each student.
“There is a tension between grade-level accountability and meeting kids where they are,” said Joel Rose, the CEO of New Classrooms Innovation Partners, the nonprofit group behind Teach to One, in an interview.
“That in and of itself is a hugely valuable insight for the field.”
Focused on middle-school math, Teach to One is based on mapping out the skills and concepts necessary to learn algebra, regularly gauging which of those skills individual students have mastered, and then using algorithms to provide each student with a customized path through the remaining material. Thirty-nine schools across 11 states use the program. In 2017, Education Week reported that about a quarter of schools that started with Teach to One eventually stopped using the program.
Descriptive studies—including the new Gates-funded report released this week—have found signs that students’ math learning grew faster while they were using Teach to One. But more rigorous analyses have not been able to attribute such gains directly to the program.
“There is no causal evidence that Teach to One has either positive or negative effects on student outcomes,” said Douglas D. Ready, the director of the Consortium for Policy Research in Education at Teachers College, Columbia University, in an interview.
Investing in Innovation?
CPRE conducted its evaluation of Teach to One as part of a $3 million “Investing in Innovation” grant awarded by the U.S. Department of Education to New Classrooms in 2015.
At the time, the program was hailed as a promising new idea worthy of expansion and further study. Such luminaries as Bill Gates continue to tout the model as the “future of math.”
But like many of the initiatives funded via the Obama administration’s $1.4 billion i3 effort, Teach to One has seen choppy implementation in schools, and the program has yet to yield clear, consistently positive results.
CPRE’s evaluation focused on deployment of the program in the Elizabeth, N.J., public schools during the 2015-16, 2016-17, and 2017-18 school years.
The study compared changes in scores on standardized state math tests over that period at five Elizabeth K-8 schools using Teach to One: Math with the test-score changes at 16 academically and demographically similar schools in the same district that did not use the program.
The researchers were “unable to draw generalizable conclusions, positive or negative, regarding [Teach to One’s] impact on student mathematics performance,” according to the study.
One big reason: there was considerable variation in how the five Elizabeth schools chose to implement the program and the extent to which they chose to focus on providing students with grade-level content. Some schools changed their approach during the period being studied.
As a group, the CPRE researchers found, Elizabeth schools using Teach to One: Math scored slightly better than expected in the first year, slightly worse than expected the second year, and about as expected the third year.
But there was substantial variation in achievement among the five schools testing the model. Year-to-year performance at some of the individual schools also fluctuated considerably.
After comparing those changes in scores over time to the differences in test scores at the similar Elizabeth schools that did not use Teach to One, the researchers determined that the impact of the program was not statistically significant.
What should other K-12 administrators and policymakers who may be considering the model make of the results?
“There were no positive or negative effects in Elizabeth,” said Ready, who led the evaluation. “For me to recommend [Teach to One], I’d really love to see more causal studies in other places.”
Focusing on Growth
For the time being, though, such robust, rigorous studies of any approaches to personalized learning remain quite rare.
It’s also important to remember that the movement takes many forms, said John F. Pane, a senior scientist at the RAND Corporation, who has studied other models that rely less on technology and don’t focus on a single subject and grade span, as Teach to One: Math does.
“It’s a trap to get focused on this idea of, ‘Does personalized learning work?’” Pane said.
Instead, he suggested, researchers and practitioners should focus on gathering lots of evidence about a variety of specific personalized-learning practices, then triangulate the results to determine what is (and isn’t) effective.
The new i3 evaluation of Teach to One “is a data point in that exercise,” Pane said, with both a “fairly strong” methodology and some important limitations the K-12 field should consider.
Most notable, Pane said, is that end-of-year state tests are not well-suited to capture the kind of academic growth that Teach to One may actually be generating.
That’s why the other new study of the program released this week, conducted by the private research firm MarGrady Research, is also important to consider, he said.
For that report, researchers focused on the same three-year span as the i3 evaluation. But there were important differences: MarGrady included a broader sample of 14 Teach to One schools located in multiple districts. The firm used a less rigorous methodology with a weaker comparison group. And most notably, the MarGrady study focused on a different outcome measure: The Northwest Evaluation Association’s Measures of Academic Progress (MAP) test, which is designed to capture learning growth across grade levels, as opposed to grade-level proficiency.
That study found that students who were consistently enrolled in Teach to One: Math over the three-year period saw about 23 percent greater math test-score gains than students in a national reference group. Gains were even sharper for schools whose accountability systems prioritized student growth. And the study found “suggestive evidence” that when Teach to One delivered content that was closely matched to what students knew and were able to do, those students’ scores improved dramatically.
The results could not be conclusively attributed to Teach to One.
For Rose and New Classrooms, though, the MarGrady findings represent a challenge to the “conventional wisdom” that schools should focus on teaching students grade-level content as much as possible.
They also highlight a major flaw with the nation’s approach to K-12 assessment, Rose said.
Focusing narrowly on grade-level proficiency doesn’t appear to be a good to measure innovative new models aiming to meet students where they are and help them grow from there, he contended.
“Personalized learning isn’t pixie dust. You can’t sprinkle it on top of a school and watch all scores go up for all kids in a single year,” Rose said.
“When schools and districts play the long game, they see the benefits for kids.”
Photo: As many as 67 math students might be working in a single Teach to One classroom at Nathan Hale Middle School in Connecticut. Source: Mark Abramson for Education Week
This post has been updated with the current title for John F. Pane, senior scientist at the RAND Corporation.
See also: