Connection and support from instructors is particularly important to encourage students from underrepresented groups to succeed in college. That’s a challenge for teachers of large lecture classes—the ones that many freshmen encounter.
Artificially intelligent chatbots could help them amplify that kind of instructional outreach, according to results from a pilot program at Georgia State University. In a study discussed at last weekend’s meeting of the Society for Research in Educational Effectiveness here, researchers found students who used an AI-powered teaching assistant called TA Pounce earned better grades and were more likely to complete the university’s two largest introductory lecture courses, in political science and economics.
Researchers tracked the class engagement and performance of more than 1,500 freshman students in Introduction to American Government classes and 915 students in Principles of Microeconomics, courses which historically have had high numbers of students receiving low or failing grades. The students were either randomly assigned to normal class supports or given access to the TA Pounce (named for the school’s panther mascot) and a self-quizzing tool.
A majority of assigned students in both subjects used the chatbot, while less than 30 percent used the quiz app.
In college—as in high school—"success is part academic engagement and part administrative navigation,” said Katharine Meyer, a co-author of the study and a government studies fellow at the Brookings Institution, who discussed the pilot at the conference. “It’s not simply learning the content and demonstrating mastery of the content, but also navigating the systems both in the university as a whole and in the micro-setting in the classroom.”
Across both subjects, students who used the chatbot were 5-6 percentage points more likely to earn a B or higher in their classes—a requirement to keep certain scholarships.
While the Georgia chatbot was made specifically for college students, it is one of a new wave of school-specific AI tools, which use large natural-language models like those of ChatGPT or GoogleBard to answer questions. Unlike the more general AI tools, however, these school-related AI tools base their answers on a closed pool of prescreened and frequently updated data, rather than scraping the full internet, to avoid the common problem of bots parroting incorrect but oft-repeated information.
Wider support net for vulnerable students
The Pounce chatbot seemed particularly beneficial for underrepresented students and those who historically were at higher risk of dropping the class.
For example, it appeared to aid women in the math-heavy microeconomics course, which is needed for several economics and business majors at the university. While men performed about as well in economics with or without the chatbot, 72 percent of the female students who used TA Pounce earned an A or B in the course compared to 60 percent of women in the control group who did not use the chatbot. While 9 percent of female students in the control group ultimately dropped the economics class, only 3 percent of those in the chatbot group did so.
Similarly, 55 percent of students who had a below-average GPA in high school were more likely to earn at least a B in political science if they used the chatbot, versus 48 percent of similarly low-performing peers who did not use the tool. While Black students were less likely than their white peers to use the chatbot, those who did also earned higher grades.
Researchers don’t know why students who used the chatbot performed better, but Meyer suggested it may have helped students feel more confident in managing their workload and navigating course resources. TA Pounce, like a human teacher’s assistant, sent students text messages a few times a week, reminding them of upcoming assignments, letting them know how they were doing in class, and encouraging them to use a variety of academic resources at the school. It also offered students suggestions on managing their time and answered questions about the course and the school.
In addition, human teacher assistants reviewed the bot transcripts weekly and followed up on some chats. For example, when one student told the chatbot that he would have to drop the class mid-semester because his tuition source fell through, the chatbot directed the student to financial aid options, but a live teacher’s assistant popped in to add that the student could continue to complete classwork while working out his financial aid to avoid losing credit.
“It’s really a combination. The bot does a lot of the administrative things and takes care of a lot of the low-hanging fruit,” such as building organizational and time-management skills, Meyer said, “but you still need people behind it to help out with tricky situations.”
