Special Report
Equity & Diversity Q&A

Why Schools Need to Talk About Racial Bias in AI-Powered Technologies

By Benjamin Herold — April 12, 2022 6 min read
Colorful abstract circular, fragmented design with two Black students showing in pieces of the fragmented shapes.
  • Save to favorites
  • Print

Schools are embracing education technologies that use artificial intelligence for everything from teaching math to optimizing bus routes.

Their goals are to save money, personalize student learning, and free teachers from rote administrative tasks.

But how can educators know if the data and design processes those products rely on have been skewed by racial bias? And what happens if they’re afraid to ask?

“If you’re not breaking data down by subpopulations to really understand where harm might be caused, you’re perpetuating racist systems and structures that have been in place for centuries,” said Sierra Noakes, the Edtech Marketplace Project Director at the nonprofit Digital Promise.

The group has teamed up with fellow nonprofit Edtech Equity Project to offer a new product certification called Prioritizing Racial Equity in AI Design. The idea is to provide both schools and companies with a common language and process for evaluating whether the makers of education technology tools are taking steps to:

  • Identify and question their own biases and assumptions.
  • Ensure the data used to train their artificial intelligence systems aren’t tainted by bias.
  • Provide educators and families alike with more visibility into how their products actually work and the risks they might pose.

The initiative comes on the heels of three other certifications offered by Digital Promise. Twenty-seven ed-tech products have already been recognized for addressing “learner variability.” Sixty-one have earned certification for “research-based design,” and one product has earned a recently launched certification regarding the use of learning analytics.

How concerned are you about the potential for racial bias in the educational technology used in schools?

As part of the pilot for the new racial equity certification, an undisclosed number of ed-tech companies volunteered to send at least one product through the review process. Findings will be made public in May.

The results will come at a moment when school districts around the country face significant backlash for their efforts to teach about racism and promote racial justice. Still, Noakes and Edtech Equity Project co-founders Madison Jacobs and Nidhi Hebbar said in a wide-ranging interview with Education Week that now is the time for a broader conversation about bias in AI.

“Companies are taking advantage of schools’ misunderstandings of artificial intelligence,” Jacobs said. “It’s important for us to combat that.”

The conversation with Hebbar, Jacobs, and Noakes has been edited for length and clarity.

What are some examples of AI-powered ed-tech tools that could have blind spots or been designed using faulty or racist assumptions?

Hebbar: Personalized-learning software that adapts to the answers students give. Language-learning software that takes into account how students speak, but is often trained on data from native-English speakers born in America and treat other accents and dialects as wrong. Also, administrative tools that are used to identify who might be at risk for behavioral or academic reasons.

Jacobs: A lot of school districts are thinking about how they would utilize facial recognition technology. Those systems are clearly not built with the safety and security of Black folks in mind. It’s also all kind of connected. Students might get misidentified or mislabeled in one system, and that system then pushes the data to other systems.

Noakes: What we’ve learned in creating the certification so far is that there’s tremendous resistance from [companies] to collecting and disaggregating data by race. They’re sort of living in this place where ignorance is bliss. But unless a tool is intentionally looking at their impact data, they’re engaging in a harmful structure.

What harms can result for students of color? Are they hypothetical, or are we already seeing them in actual schools?

Noakes: AI tools often make decisions on behalf of teachers without teachers having any insight into what variables are leading to those decisions. And in some instances, there’s not an override function. So, for example, grouping students. If those decisions are dictated by an algorithm, educators are blind to what the decisionmaking process looks like. These tools are making decisions they disagree with, but they can’t go to school leadership to say this tool is not taking something into account.

Hebbar: It’s not like, “AI is bad, full stop.” But to be used in schools, it needs to be done in a way that allows educators who have a full understanding of a child to override [algorithmic decisions]. Giving users the ability to course-correct can mitigate a lot of the mistakes in technology, but we’re not seeing that with a lot of tools.

How aware of these problems are K-12 leaders?

Hebbar: It’s a mix. A lot of districts, especially the large ones with a lot of Black and brown students, are very aware of this. Others don’t even know which tools they’re purchasing that use AI. But what we generally see is that leaders often feel like, “I’m not an expert, I don’t even know what questions to ask.” The idea behind the certification is to make that process simpler.

Jacobs: Part of the problem is there is no across-the-board policy that requires companies to disclose what is actually operating inside their technology. And I think a lot of leaders in schools still have the perception that if you’re basing things on numbers, then systems must be operating in a neutral fashion. But these systems are built by people, and therefore people’s inherent biases, their unconscious biases, their explicit biases go into the systems. A lot of our conversations are about educating people on a general level that these are not neutral systems.

How do companies respond to that message?

Jacobs: There are definitely companies that are sort of hiding behind the protections that the black box of AI offers. They may have engineering teams that are skewed toward white males. Bringing that to light is hard to do. They may not want to expose their shortcomings. There’s also this culture in tech about being the best and doing the most, and that really is a horrible barrier to thinking about how you design through the lens of the collective liberation of the people utilizing your technology. But some companies understand that legislation is coming [to address racial bias in tech] and that the venture capital ecosystem is shifting, with more funders wanting to invest in companies that treat equity as a core principle in their product development.

What steps do you want to see companies take to address these problems?

Hebbar: Looking at the collection of data and the types of students whose data are used to train a company’s system is a big one. But there are places for bias to come in at almost every stage of the product development and design process. The certification is built off a toolkit to help companies understand how to mitigate that. Even in the ideation phase, you really want to check, “What are our assumptions and blind spots?” And then in the very last step, how are companies re-evaluating on a regular basis whether their products are effective?

There’s a backlash throughout the K-12 system to the way schools and districts are approaching racial equity. Many people just don’t agree with the idea that schools are systemically racist, and they’ll likely disagree with the idea that educators and developers should be trying to undo the existing system. Why should they support this certification?

Jacobs: When you have groups of folks that would fight against a practice of looking into whether students are being treated fairly, that’s just another data point and proof that racism and bias exists. And if you don’t think that bias exists, then what is the problem for us to look and see?

Noakes: I believe to my core that when we’re designing systems and solutions for folks at the margins who are often ignored in design and development, we’re creating tools that help all learners thrive.

education week logo subbrand logo RC RGB

Data analysis for this article was provided by the EdWeek Research Center. Learn more about the center’s work.

A version of this article appeared in the April 13, 2022 edition of Education Week as Why Schools Need To Talk About Racial Bias In AI-Powered Technologies


School Climate & Safety K-12 Essentials Forum Strengthen Students’ Connections to School
Join this free event to learn how schools are creating the space for students to form strong bonds with each other and trusted adults.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
IT Infrastructure & Management Webinar
Future-Proofing Your School's Tech Ecosystem: Strategies for Asset Tracking, Sustainability, and Budget Optimization
Gain actionable insights into effective asset management, budget optimization, and sustainable IT practices.
Content provided by Follett Learning
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Budget & Finance Webinar
Innovative Funding Models: A Deep Dive into Public-Private Partnerships
Discover how innovative funding models drive educational projects forward. Join us for insights into effective PPP implementation.
Content provided by Follett Learning

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Equity & Diversity Opinion 70 Years of Abandonment: The Failed Promise of 'Brown v. Board'
If the nation is going to refuse integration, Black people must demand we revisit the separate but equal doctrine, writes Bettina L. Love.
4 min read
A Black student is isolated from their classmates by an aisle in the classroom.
Xia Gordon for Education Week
Equity & Diversity Opinion 'Brown v. Board of Education' at 70: A Dream Dissolved
This anniversary should remind us that progress is not inevitable. We stand now at a critical juncture.
R. L’Heureux Lewis-McCoy
4 min read
A young Black woman's image dissolves in the smoke.
iStock/Getty Images
Equity & Diversity Opinion Equity? Equality? How Educators Can Tell the Difference
Educators offer advice and examples for giving students what they need, rather than simply treating everyone the same.
10 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
Equity & Diversity Judge Says State Can't Block Teachers From Discussing Critical Race Theory
The rule stops short of more broadly blocking Arkansas from enforcing its ban on certain topics.
2 min read
Students make their way into Little Rock Central High School on Aug. 24, 2020, for the first day of classes in the Little Rock School District. A federal judge ruled, Tuesday, May 7, 2024, that Arkansas cannot prevent two high school teachers from discussing critical race theory in the classroom, but stopped short of more broadly blocking the state from enforcing its ban on “indoctrination” in public schools. The prohibition is being challenged by two teachers and two students at Little Rock Central High School, site of the 1957 desegregation crisis.
Students make their way into Little Rock Central High School in Little Rock, Ark., on Aug. 24, 2020, for the first day of classes.
Tommy Metthe/Arkansas Democrat-Gazette via AP