If a teacher uploads detailed lesson plans to her school’s AI-based curriculum tool, could her work or her students’ data end up training the next model?
Often, teachers have no way of knowing.
“We can get angrier and angrier, or we can act decisively,” said Randi Weingarten, the president of the nearly 1.9 million-member American Federation of Teachers in a statement. “Anything less than legally enforceable provisions is simply a wish list.”
On Wednesday, Microsoft became the first of the AI development giants to agree to adopt legally binding privacy and oversight standards for AI-related licenses in schools, via a 30-page memorandum of agreement between the company and the National Academy for AI Instruction in New York City. The academy was created last July through a five-year, $23 million partnership between AFT and Anthropic, Microsoft, and OpenAI.
At a briefing on the agreement, Weingarten noted that her union is still negotiating with Anthropic and OpenAI to adopt the standards for their education products. “We want it to be an industry standard,” she said, but added that the teachers’ union would “keep fighting” for federal and state governments to adopt legal standards for AI in education. While Anthropic has not officially signed onto the agreement, it said in July that it was working with AFT to “align our terms and privacy practices with gold standards they’re developing.”
Federal privacy laws protecting health and education data “never envisioned the advent of AI,” Weingarten said. “In the absence of those protections, tech companies need to take responsibility for the products they create and market to students, educators, and communities.”
The new Microsoft standards won’t be applied automatically. As of Nov. 1, school districts can ask to add the new language to new or existing contracts with Microsoft products.
But Brad Smith, Microsoft’s vice chairman and president, said it is important for the agreement to “have teeth,” including audit and legal enforcement provisions.
“I think that’s what is needed in this day and age to ensure that people can have confidence in the technology that their kids may use in school,” Smith said.
The landscape of data protections is murky
The standards apply an “intentionally broad” definition of student data—including directory information like names, grades, and behavioral records, but also writing prompts, memory files, and work outputs, as well as information like keystrokes or eye-tracking that could “reasonably be used” to identify or link to students.
Under the agreement, de-identified “telemetry” data about how students and teachers use the tools cannot be used to train AI models or create advertising, student profiles, or to infer student behavior, but can be used for debugging or security issues or to “improve products.”
De-identifying data doesn’t guarantee someone can never be identified, however, according to an emerging body of research.
Why? Imagine a teacher uploads information about reading interventions she uses with different groups in her class. One is specifically used for students with a type of dyslexia. Separately, one student’s uploaded essay about a family fight during a vacation to Maine shows hallmarks of the same disability.
Then, a separate teacher working with the same students in a group collaboration requests AI help in writing a supportive letter for a child whose parents just decided to divorce immediately on return from a family vacation to Maine. Taken together, an AI tool may be able to use disparate pieces of information like these to identify a student, their disability, and a sensitive family situation, several studies conducted in the last few years have shown.
While there has been no research using student data specifically, studies show current AI models often can easily re-identify people from wide array of anonymized data by cross-referencing background facts, writing styles, and other inferred information.
Anything less than legally enforceable [privacy] provisions is simply a wish list.
It’s hard to overestimate the sheer quantity of data potentially collected by Microsoft and other companies in schools. Microsoft’s “learning zone” program, being piloted in all Windows 11 devices this year, promise teachers “live classroom experiences with real-time visibility into student activity,” and its “copilot notebooks” are pitched as an “AI-powered workspace built around [students’] own materials.” In practice, such features mean large swathes of student classwork could be uploaded into the tool over the course of the year.
At times, even the firms themselves seem unsure of what data their tools might collect as part of normal use. For example, this summer, Anthropic similarly pledged to align with AFT’s guidelines when it launched its Claude for Teachers program, such as not using teachers’ inputs or outputs for training models and aligning student data protections with the Federal Education Rights and Privacy Act.
But in August it clarified that teachers uploading student work might still run afoul of FERPA without additional district approval, because it might include data that could be used to identify individual students. (Anthropic did not immediately respond to a request for comment.)
Michael Mulgrew, the president of the United Federation of Teachers, AFT’s largest affiliate in New York City, said student privacy was “a grave concern” that the new standards start to address, but he still supports the Big Apple district’s one-year ban on student use of AI tools in elementary, middle, and most high schools.
“We need a project in New York City where teachers who are trained already in the use of AI can start looking through these materials before we ever think about lifting the [student-facing AI] moratorium,” Mulgrew said. “That’s how you do this thing in a responsible way.”
Under the AFT’s agreement language with Microsoft:
- Aside from specific safety-related data, student, educator, or customer data could not be used to train or improve AI models.
- All data—including that processed by third-party vendors like Khanmigo or PowerSchool—would have to be encrypted, and access controlled, with security tested by independent groups and response plans in place for any data breaches.
- The company would have a permanent responsibility to alert customers and mitigate data breaches, even after a contract ends.
- The tools would minimize unneeded data collections, with “higher-risk” tools like biometrics, memory, profiling, and tracking requiring additional review.
- Students and customers would control the export, retention, and deletion of their data, which could not be sold for advertisements or product development.
- Students and families would be given plain-language guides to how the AI tools work and how they protect privacy.
- AI tools must provide fair, equitable access to features for students with special needs.
- AI “companions"—chatbots programmed to simulate human relationships and provide continuous, personalized social or emotional support for students—would be explicitly banned.
- Shools or districts holding a Microsoft license agreement,would receive notice about and control new features and data use, and would be allowed to export data to switch providers.
- Data breaches must be reported within 72 hours and fixed.
