Special Report
Equity & Diversity Q&A

Why Schools Need to Talk About Racial Bias in AI-Powered Technologies

By Benjamin Herold — April 12, 2022 6 min read
Colorful abstract circular, fragmented design with two Black students showing in pieces of the fragmented shapes.
  • Save to favorites
  • Print

Schools are embracing education technologies that use artificial intelligence for everything from teaching math to optimizing bus routes.

Their goals are to save money, personalize student learning, and free teachers from rote administrative tasks.

But how can educators know if the data and design processes those products rely on have been skewed by racial bias? And what happens if they’re afraid to ask?

“If you’re not breaking data down by subpopulations to really understand where harm might be caused, you’re perpetuating racist systems and structures that have been in place for centuries,” said Sierra Noakes, the Edtech Marketplace Project Director at the nonprofit Digital Promise.

The group has teamed up with fellow nonprofit Edtech Equity Project to offer a new product certification called Prioritizing Racial Equity in AI Design. The idea is to provide both schools and companies with a common language and process for evaluating whether the makers of education technology tools are taking steps to:

  • Identify and question their own biases and assumptions.
  • Ensure the data used to train their artificial intelligence systems aren’t tainted by bias.
  • Provide educators and families alike with more visibility into how their products actually work and the risks they might pose.

The initiative comes on the heels of three other certifications offered by Digital Promise. Twenty-seven ed-tech products have already been recognized for addressing “learner variability.” Sixty-one have earned certification for “research-based design,” and one product has earned a recently launched certification regarding the use of learning analytics.

How concerned are you about the potential for racial bias in the educational technology used in schools?

As part of the pilot for the new racial equity certification, an undisclosed number of ed-tech companies volunteered to send at least one product through the review process. Findings will be made public in May.

The results will come at a moment when school districts around the country face significant backlash for their efforts to teach about racism and promote racial justice. Still, Noakes and Edtech Equity Project co-founders Madison Jacobs and Nidhi Hebbar said in a wide-ranging interview with Education Week that now is the time for a broader conversation about bias in AI.

“Companies are taking advantage of schools’ misunderstandings of artificial intelligence,” Jacobs said. “It’s important for us to combat that.”

The conversation with Hebbar, Jacobs, and Noakes has been edited for length and clarity.

What are some examples of AI-powered ed-tech tools that could have blind spots or been designed using faulty or racist assumptions?

Hebbar: Personalized-learning software that adapts to the answers students give. Language-learning software that takes into account how students speak, but is often trained on data from native-English speakers born in America and treat other accents and dialects as wrong. Also, administrative tools that are used to identify who might be at risk for behavioral or academic reasons.

Jacobs: A lot of school districts are thinking about how they would utilize facial recognition technology. Those systems are clearly not built with the safety and security of Black folks in mind. It’s also all kind of connected. Students might get misidentified or mislabeled in one system, and that system then pushes the data to other systems.

Noakes: What we’ve learned in creating the certification so far is that there’s tremendous resistance from [companies] to collecting and disaggregating data by race. They’re sort of living in this place where ignorance is bliss. But unless a tool is intentionally looking at their impact data, they’re engaging in a harmful structure.

What harms can result for students of color? Are they hypothetical, or are we already seeing them in actual schools?

Noakes: AI tools often make decisions on behalf of teachers without teachers having any insight into what variables are leading to those decisions. And in some instances, there’s not an override function. So, for example, grouping students. If those decisions are dictated by an algorithm, educators are blind to what the decisionmaking process looks like. These tools are making decisions they disagree with, but they can’t go to school leadership to say this tool is not taking something into account.

Hebbar: It’s not like, “AI is bad, full stop.” But to be used in schools, it needs to be done in a way that allows educators who have a full understanding of a child to override [algorithmic decisions]. Giving users the ability to course-correct can mitigate a lot of the mistakes in technology, but we’re not seeing that with a lot of tools.

How aware of these problems are K-12 leaders?

Hebbar: It’s a mix. A lot of districts, especially the large ones with a lot of Black and brown students, are very aware of this. Others don’t even know which tools they’re purchasing that use AI. But what we generally see is that leaders often feel like, “I’m not an expert, I don’t even know what questions to ask.” The idea behind the certification is to make that process simpler.

Jacobs: Part of the problem is there is no across-the-board policy that requires companies to disclose what is actually operating inside their technology. And I think a lot of leaders in schools still have the perception that if you’re basing things on numbers, then systems must be operating in a neutral fashion. But these systems are built by people, and therefore people’s inherent biases, their unconscious biases, their explicit biases go into the systems. A lot of our conversations are about educating people on a general level that these are not neutral systems.

How do companies respond to that message?

Jacobs: There are definitely companies that are sort of hiding behind the protections that the black box of AI offers. They may have engineering teams that are skewed toward white males. Bringing that to light is hard to do. They may not want to expose their shortcomings. There’s also this culture in tech about being the best and doing the most, and that really is a horrible barrier to thinking about how you design through the lens of the collective liberation of the people utilizing your technology. But some companies understand that legislation is coming [to address racial bias in tech] and that the venture capital ecosystem is shifting, with more funders wanting to invest in companies that treat equity as a core principle in their product development.

What steps do you want to see companies take to address these problems?

Hebbar: Looking at the collection of data and the types of students whose data are used to train a company’s system is a big one. But there are places for bias to come in at almost every stage of the product development and design process. The certification is built off a toolkit to help companies understand how to mitigate that. Even in the ideation phase, you really want to check, “What are our assumptions and blind spots?” And then in the very last step, how are companies re-evaluating on a regular basis whether their products are effective?

There’s a backlash throughout the K-12 system to the way schools and districts are approaching racial equity. Many people just don’t agree with the idea that schools are systemically racist, and they’ll likely disagree with the idea that educators and developers should be trying to undo the existing system. Why should they support this certification?

Jacobs: When you have groups of folks that would fight against a practice of looking into whether students are being treated fairly, that’s just another data point and proof that racism and bias exists. And if you don’t think that bias exists, then what is the problem for us to look and see?

Noakes: I believe to my core that when we’re designing systems and solutions for folks at the margins who are often ignored in design and development, we’re creating tools that help all learners thrive.

education week logo subbrand logo RC RGB

Data analysis for this article was provided by the EdWeek Research Center. Learn more about the center’s work.

A version of this article appeared in the April 13, 2022 edition of Education Week as Why Schools Need To Talk About Racial Bias In AI-Powered Technologies


Special Education K-12 Essentials Forum Innovative Approaches to Special Education
Join this free virtual event to explore innovations in the evolving landscape of special education.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
STEM Fusion: Empowering K-12 Education through Interdisciplinary Integration
Join our webinar to learn how integrating STEM with other subjects can revolutionize K-12 education & prepare students for the future.
Content provided by Project Lead The Way
School & District Management Webinar How Pensions Work: Why It Matters for K-12 Education
Panelists explain the fundamentals of teacher pension finances — how they are paid for, what drives their costs, and their impact on K-12 education.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Equity & Diversity States That Require Period Products for Free in Schools
More and more states are either requiring K-12 schools to stock pads and tampons, or provide funding for schools to do so.
1 min read
A menstrual product dispenser inside a women's restroom in Purdue University Stewart Center on Feb. 6, 2020, in West Lafayette, Ind. More than half of the states have legislation on the books either requiring products be stocked in schools, or provide funding to purchase them.
A menstrual product dispenser inside a women's restroom in Purdue University Stewart Center on Feb. 6, 2020, in West Lafayette, Ind. Legislation in a number of states seeks to provide more access to pads and tampons for students in K-12 schools.
Nikos Frazier/Journal & Courier via AP
Equity & Diversity More Schools Stock Tampons and Pads, But Access Is Still a Problem
Period products are becoming more commonplace in schools. But there are gaps in funding—and in access, a barrier for lower-income students.
7 min read
Photograph of hygienic tampons and a sanitary pad on a blue background.
Equity & Diversity A School Board Reinstated Confederate School Names. Could It Happen Elsewhere?
Shenandoah County's school board voted in May to reinstate two Confederate names. Researchers wonder if others will, too.
7 min read
A statue of confederate general Stonewall Jackson is removed on July 1, 2020, in Richmond, Va. Shenandoah County, Virginia's school board voted 5-1 early Friday, May 10, 2024, to rename Mountain View High School as Stonewall Jackson High School and Honey Run Elementary as Ashby Lee Elementary four years after the names had been removed.
A statue of confederate general Stonewall Jackson is removed on July 1, 2020, in Richmond, Va. The Shenandoah County, Va. school board voted 5-1 on May 10, 2024, to restore the names of Confederate leaders and soldiers to two schools, four years after the names had been removed.
Steve Helber/AP
Equity & Diversity How 9 Leaders Think About Diversity, Equity, and Inclusion in Their Schools
District and school leaders share their take on DEI and what it means for all students to experience inclusion and belonging.
6 min read
An illustration of six speech bubbles that are different in size and of varying shades of blue.