A new large-scale effort to map and evaluate AI-powered learning tools finds some areas where the technology has promise for propelling student achievement—but also warns that other products are “isolating,” “boring,” and could leave children’s learning “significantly damaged.”
The mixed-bag analysis, from education consulting nonprofit Instruction Partners, includes 16 deep dives into individual AI products. Each profile presents user feedback on the tool, reviews its instructional capabilities, and catalogs available research evidence on effectiveness.
The number of AI tools now marketed to schools is legion, so the project is just a small slice of what’s out there for schools to choose from. Nevertheless, it represents one of the first attempts to weigh the various types of services on offer and their implications for teaching and learning.
And it comes as educators and experts have called for more systematic assessments of the sprawling, and ever-growing, landscape of AI-enabled learning tools.
“Not all AI is created equal,” said Emily Freitag, the co-founder and CEO of Instruction Partners. “We need to have nuanced decisionmaking about where it can offer real value and where it creates real risk.”
Instruction Partners analyzed 20 tools, interviewing teacher users, students, district and building leaders, product developers, and observing the technology in action in 16 school systems.
In-depth product reviews have become common in the curriculum marketplace. Evaluations from the nonprofit EdReports and similar organizations are designed to analyze how likely it is that a curriculum will deliver a high-quality instructional experience. Does it align to academic standards? Is it easy for teachers to use?
In the AI realm, several organizations—including major nonprofits like Common Sense Media and Digital Promise—rate tools for safety and security risks, but not for instructional quality. There’s little comprehensive information about how well AI products support teaching and learning, even as they play an increasingly large role in students’ classroom experiences.
In part, that’s because the tools themselves are evolving so quickly, said Lewis Ferebee, the CEO of EdReports, and the former chancellor of the District of Columbia public schools. The organization recently published a report outlining trends in AI-enabled products, but hasn’t published reviews of individual tools.
“We don’t know yet what it would mean to consistently judge quality,” he said.
Some big risks are with general purpose models
The team at Instruction Partners examined three kinds of AI tools: 1) general-purpose chatbots like ChatGPT, Claude, and Gemini; 2) education-focused tools with many applications, like MagicSchool or Google Classroom; and 3) instructional tools with a discrete purpose, like Amira Learning, which only offers reading practice.
Each of the 16 “product profiles” explain how the tool in question works—what students do, what the AI does, what the teacher does, and what role school or district leaders can play.
The profiles include “wins” and “limitations” drawn from interviews with educator users, Instruction Partners’ own analyses, and a list of available studies that evaluate the tools’ effectiveness. Finally, each profile also includes a section “from the developer,” which includes statements from, or video interviews with, product designers.
The hope, said Freitag, is that the analysis “cuts through the marketing speak and helps [leaders] make sense of the product.”
Across the landscape, teachers said some tools seemed more of a hindrance than a help to learning.
“We definitely saw the biggest risks from the general-purpose chatbots,” said Freitag. “It’s also the way AI is showing up in search.
“We saw kids being asked questions from an AI tool, then they went to Google, asked Google the question, and then put the answer in the AI tool,” she continued, recalling classroom observations.
Teachers commonly reported students using chatbots to “get out of work that requires effortful thinking,” Instruction Partners’ analysis reads. “Learning anything requires effort, and products that intentionally or unintentionally reduce students’ effortful thinking will significantly compromise learning.”
Instructionally focused tools have more promise, but aren’t perfect
The review team was more bullish on tools designed for a specific instructional use, those with a “clear point of view about the parts of the lesson that teachers are best positioned to deliver and which parts AI can support,” the report reads.
In reading and English/language arts, this looked like tools that could provide actionable feedback on student writing faster than an individual teacher could on their own, or diagnose which phonics skills students had mastered and which they still struggled with.
In math, the review team praised tools that could identify students’ misconceptions, or collect and analyze many students’ explanations of their mathematical reasoning.
Still, purpose-built AI tools had their own shortcomings. For example, the Instruction Partners team cited concerns about tools that rewrote text at a lower reading level, which could potentially decrease rigor.
“We did not see anything that is ready to do the pedagogical job independently,” said Freitag. “We also heard from students, especially, that they value the relationships with their teachers, and they want to learn from their teachers.”
At the same time, leaders shouldn’t avoid AI entirely, she said. Educators should pick the approach that’s best suited to the job at hand.
“We do not think we should be precious about instructional methods when they are not producing the outcomes we want for kids,” she said.
Eventually, the field will need ‘raters and regulators’
Organizations like Instruction Partners and EdReports are wrestling with an ongoing tension: Because AI tools are still relatively new, there’s little consensus on how to evaluate their instructional quality. But companies are marketing these products now, and districts are adopting them—or have already done so. Usable evaluation systems are a present need.
On its website, Instruction Partners tries to stake a middle ground, underscoring that its product analyses aren’t ratings. They’re not evaluating the tools against a predetermined list of criteria.
Most of the potential “instructional wins” also come with caveats, meaning it’s up to district leaders to weigh the pros and cons.
For example, on tools’ ability to provide feedback on student writing, Instruction Partners outlines the upsides, noting that it can boost the amount of feedback students get overall—a shift that could improve their writing.
But the product profiles list some limitations to AI-generated feedback. It can be inconsistent, and students said some AI-generated comments felt impersonal.
(Relatedly, some research has shown that ChatGPT and Meta’s Llama change the kind and quality of writing feedback they give when they know a student’s demographic background. It’s unclear whether AI tools designed for instructional purposes operate similarly.)
“Eventually, the field needs raters and regulators,” Freitag said. “Right now, the field needs sense-makers.”
It’s a sentiment shared by the team at EdReports. The organization’s brief on AI in instructional materials, released earlier this month, raised a host of questions. Among them: Can AI-generated materials align coherently to an existing curriculum? How do adaptive features affect curriculum pacing?
EdReports’ work in AI “could evolve over time into some signaling,” said Ferebee, “but we’re not quite at that point yet.”
Instruction Partners’ project is also intentionally limited in scope—focusing solely on tools’ instructional capabilities, not other concerns such as data privacy.
Some observers say that choice leaves out important information.
“It misses the mark on accurately informing the people that these reviews are for, because it’s only giving one glimpse into it. It’s not giving the entire picture,” said Bridie Lee, one of the leaders of the group Schools Beyond Screens L.A., part of a national coalition that advocates for stricter guardrails around technology in the classroom.
The organization should incorporate other groups’ analyses of privacy and safety risks into its own reviews if it doesn’t plan on analyzing that on its own—especially if those analyses have found products unsafe for minors, Lee contends.
In response, Freitag said: “We totally agree—both as educators and as parents ourselves—that data privacy information is incredibly important, and organizations with expertise on these areas should absolutely be doing this kind of review.
“Where they do already exist, data privacy reviews (i.e., from ISTE, Common Sense Media) tend to be lighter on instructional information; this is the specific gap and need that we are targeting,” she continued.
This coming school year, the organization plans to analyze at least an additional 20 tools and partner with Stanford University’s AI in Education Lab to conduct additional reviews of evidence-of-impact studies.
