An AI-powered tutoring assistant increased human tutors’ capacity to help students through math problems and improved students’ performance in math, according to a Stanford University study.
The digital tool, Tutor CoPilot, was created by Stanford researchers to guide tutors, especially novices, in their interactions with students.
The study is the first randomized controlled trial to examine a human-AI partnership in live tutoring, according to the researchers. The study examines whether the tool is effective for improving tutors’ skills and students’ math learning.
It comes as tutoring has become a key learning-recovery tool. Schools, however, have run into challenges in scaling and sustaining tutoring programs because they require a lot of human tutors, time, and money.
In an interview with Education Week, Susanna Loeb, an education professor at Stanford and one of the study’s authors, discussed the creation of the tool, the trial findings, and its implications for schools.
This interview has been edited for brevity and clarity.
How did Tutor CoPilot come about?
School districts have bought into tutoring and have worked to provide this kind of support for students. It’s not easy, because you have to fit it into the school schedule. It takes a lot of effort recruiting so many tutors.
One of the benefits of virtual tutoring is that you can draw tutors from a broader geographic area. Many of those tutors are novice tutors, and the good programs provide them training and materials, but they still have a learning curve.
Tutor CoPilot came out of this idea that maybe we can support tutors, not only with this upfront training and coaching every now and then, but with in-the-moment support.
How was Tutor CoPilot trained?
We partnered with FEV [a virtual tutoring company]. We did a first study using the data that they have to really understand the potential of generative AI for providing this information. We tried first with just the basic generative AI that’s available to all of us, things like ChatGPT. You can feed in the problem that the student is having: What is 10 minus 7? The student said 5. You can ask ChatGPT how an expert tutor would respond to the student’s mistake and it would just say 10 minus 7 is 3.
So we worked with a bunch of expert math teachers to understand the process they go through when they’re responding to students’ mathematical mistakes, trying to understand the reason for them, and thinking about different strategies. We tried a series of different prompts [for the chatbot]. One of them that worked quite well was, instead of asking how it would respond, we say, “respond to this problem with this strategy.” There were seven strategies that we had it respond to, and then the tutor had the option of choosing which strategy [to use].
What are the big takeaways from the results?
What we found was the students who were working with the [tutors] in the treatment group, who had the access to [Tutor CoPilot], did better. They were more likely to demonstrate mastery by 4 [percentage points]. Generally, about 62 percent of the time students demonstrate mastery, so this moved it up to 66 percent of the time that students were demonstrating mastery. That’s a strong effect.
What was more exciting was that if you looked at the heterogeneity, the effect is far stronger on the weaker [tutors]. [They experienced a 9 percentage point increase in students’ demonstrating mastery]. They started at a lower rate, so that’s a meaningful increase in the proportion of time students are demonstrating mastery. It was most effective for the novice teachers. So that’s where you really saw the positive effect.
Another finding that was exciting is the language difference. The tutors using Tutor CoPilot were more likely to do the good things, to prompt students to explain, to ask guiding questions, to focus on what students were doing really well, and much less likely to do the generic encouragement, but also significantly less likely to give away the solution or to answer the question directly.
What are the implications for schools and districts?
Tutors, educators, and many adults can really be helped by these small supports that happen in the moment. They can be much more helpful than big books and saying, “Oh, just learn this,” or a one-time class. The potential of this support that is light-touch but continuous is a very important thing to think about when we’re trying to help adults change behavior and to learn.
When people think about generative AI and its potential for improving education, they often go to generative AI tutors, where students sit in front of the computer, ask it a question, and it gives them an answer. That is one role for generative AI.
But there are lots of benefits of the relationships that are inherent in the kind of education that we do in tutoring and in classrooms. Those relationships are so important for engaging and motivating students, and generative AI can support those, as well—this is an example of that.
By providing generative AI to the educator instead of directly to the student, you may get some of the benefits of this immediate feedback while still maintaining the really important parts of the in-person, relationship-based educational approaches that we’ve taken traditionally.
