Special Report
Equity & Diversity Q&A

Why Schools Need to Talk About Racial Bias in AI-Powered Technologies

By Benjamin Herold — April 12, 2022 6 min read
Colorful abstract circular, fragmented design with two Black students showing in pieces of the fragmented shapes.
  • Save to favorites
  • Print

Schools are embracing education technologies that use artificial intelligence for everything from teaching math to optimizing bus routes.

Their goals are to save money, personalize student learning, and free teachers from rote administrative tasks.

But how can educators know if the data and design processes those products rely on have been skewed by racial bias? And what happens if they’re afraid to ask?

“If you’re not breaking data down by subpopulations to really understand where harm might be caused, you’re perpetuating racist systems and structures that have been in place for centuries,” said Sierra Noakes, the Edtech Marketplace Project Director at the nonprofit Digital Promise.

The group has teamed up with fellow nonprofit Edtech Equity Project to offer a new product certification called Prioritizing Racial Equity in AI Design. The idea is to provide both schools and companies with a common language and process for evaluating whether the makers of education technology tools are taking steps to:

  • Identify and question their own biases and assumptions.
  • Ensure the data used to train their artificial intelligence systems aren’t tainted by bias.
  • Provide educators and families alike with more visibility into how their products actually work and the risks they might pose.

The initiative comes on the heels of three other certifications offered by Digital Promise. Twenty-seven ed-tech products have already been recognized for addressing “learner variability.” Sixty-one have earned certification for “research-based design,” and one product has earned a recently launched certification regarding the use of learning analytics.

How concerned are you about the potential for racial bias in the educational technology used in schools?

As part of the pilot for the new racial equity certification, an undisclosed number of ed-tech companies volunteered to send at least one product through the review process. Findings will be made public in May.

The results will come at a moment when school districts around the country face significant backlash for their efforts to teach about racism and promote racial justice. Still, Noakes and Edtech Equity Project co-founders Madison Jacobs and Nidhi Hebbar said in a wide-ranging interview with Education Week that now is the time for a broader conversation about bias in AI.

“Companies are taking advantage of schools’ misunderstandings of artificial intelligence,” Jacobs said. “It’s important for us to combat that.”

The conversation with Hebbar, Jacobs, and Noakes has been edited for length and clarity.

What are some examples of AI-powered ed-tech tools that could have blind spots or been designed using faulty or racist assumptions?

Hebbar: Personalized-learning software that adapts to the answers students give. Language-learning software that takes into account how students speak, but is often trained on data from native-English speakers born in America and treat other accents and dialects as wrong. Also, administrative tools that are used to identify who might be at risk for behavioral or academic reasons.

Jacobs: A lot of school districts are thinking about how they would utilize facial recognition technology. Those systems are clearly not built with the safety and security of Black folks in mind. It’s also all kind of connected. Students might get misidentified or mislabeled in one system, and that system then pushes the data to other systems.

Noakes: What we’ve learned in creating the certification so far is that there’s tremendous resistance from [companies] to collecting and disaggregating data by race. They’re sort of living in this place where ignorance is bliss. But unless a tool is intentionally looking at their impact data, they’re engaging in a harmful structure.

What harms can result for students of color? Are they hypothetical, or are we already seeing them in actual schools?

Noakes: AI tools often make decisions on behalf of teachers without teachers having any insight into what variables are leading to those decisions. And in some instances, there’s not an override function. So, for example, grouping students. If those decisions are dictated by an algorithm, educators are blind to what the decisionmaking process looks like. These tools are making decisions they disagree with, but they can’t go to school leadership to say this tool is not taking something into account.

Hebbar: It’s not like, “AI is bad, full stop.” But to be used in schools, it needs to be done in a way that allows educators who have a full understanding of a child to override [algorithmic decisions]. Giving users the ability to course-correct can mitigate a lot of the mistakes in technology, but we’re not seeing that with a lot of tools.

How aware of these problems are K-12 leaders?

Hebbar: It’s a mix. A lot of districts, especially the large ones with a lot of Black and brown students, are very aware of this. Others don’t even know which tools they’re purchasing that use AI. But what we generally see is that leaders often feel like, “I’m not an expert, I don’t even know what questions to ask.” The idea behind the certification is to make that process simpler.

Jacobs: Part of the problem is there is no across-the-board policy that requires companies to disclose what is actually operating inside their technology. And I think a lot of leaders in schools still have the perception that if you’re basing things on numbers, then systems must be operating in a neutral fashion. But these systems are built by people, and therefore people’s inherent biases, their unconscious biases, their explicit biases go into the systems. A lot of our conversations are about educating people on a general level that these are not neutral systems.

How do companies respond to that message?

Jacobs: There are definitely companies that are sort of hiding behind the protections that the black box of AI offers. They may have engineering teams that are skewed toward white males. Bringing that to light is hard to do. They may not want to expose their shortcomings. There’s also this culture in tech about being the best and doing the most, and that really is a horrible barrier to thinking about how you design through the lens of the collective liberation of the people utilizing your technology. But some companies understand that legislation is coming [to address racial bias in tech] and that the venture capital ecosystem is shifting, with more funders wanting to invest in companies that treat equity as a core principle in their product development.

What steps do you want to see companies take to address these problems?

Hebbar: Looking at the collection of data and the types of students whose data are used to train a company’s system is a big one. But there are places for bias to come in at almost every stage of the product development and design process. The certification is built off a toolkit to help companies understand how to mitigate that. Even in the ideation phase, you really want to check, “What are our assumptions and blind spots?” And then in the very last step, how are companies re-evaluating on a regular basis whether their products are effective?

There’s a backlash throughout the K-12 system to the way schools and districts are approaching racial equity. Many people just don’t agree with the idea that schools are systemically racist, and they’ll likely disagree with the idea that educators and developers should be trying to undo the existing system. Why should they support this certification?

Jacobs: When you have groups of folks that would fight against a practice of looking into whether students are being treated fairly, that’s just another data point and proof that racism and bias exists. And if you don’t think that bias exists, then what is the problem for us to look and see?

Noakes: I believe to my core that when we’re designing systems and solutions for folks at the margins who are often ignored in design and development, we’re creating tools that help all learners thrive.

education week logo subbrand logo RC RGB

Data analysis for this article was provided by the EdWeek Research Center. Learn more about the center’s work.

A version of this article appeared in the April 13, 2022 edition of Education Week as Why Schools Need To Talk About Racial Bias In AI-Powered Technologies


Special Education Webinar Reading, Dyslexia, and Equity: Best Practices for Addressing a Threefold Challenge
Learn about proven strategies for instruction and intervention that support students with dyslexia.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Families & the Community Webinar
How Whole-Child Student Data Can Strengthen Family Connections
Learn how district leaders can use these actionable strategies to increase family engagement in their student’s education and boost their academic achievement.
Content provided by Panorama Education
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
College & Workforce Readiness Webinar
The School to Workforce Gap: How Are Schools Setting Students Up For Life & Lifestyle Success?
Hear from education and business leaders on how schools are preparing students for their leap into the workforce.
Content provided by Find Your Grind

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Equity & Diversity Opinion The Real Value of Equity Directors for Districts
Though some in education dismiss equity directors as merely symbolic or even doomed to fail, the role is enormously consequential.
Decoteau J. Irby
6 min read
111622 opinion 14Irby equity director 1174940279
Equity & Diversity Opinion Stop Demonizing Black Boys. Let Them Play, Too
The play of Black boys is judged differently—more dangerous, more violent—than that of peers, writes teacher-educator Altheria Caldera.
Altheria Caldera
4 min read
Conceptual illustration of a black boy looking through a dream door at a glowing stairway.
Jorm Sangsom/iStock/Getty
Equity & Diversity Q&A How Is White Supremacy Embedded in School Systems Today? A Scholar Explains
John Diamond, a professor of sociology and education policy at Brown University, discusses how educators can make schools more equitable.
8 min read
Members of the 101st Airborne Division take up positions outside Central High School in Little Rock, Ark., on Sept. 26, 1957. A plan to only grant Little Rock partial control of its schools is drawing complaints that the district may further segregate 62 years after nine black students were escorted into an all-white high school, and a push to end the local teachers union's bargaining power is stirring fears of even more instability.
Central High School in Little Rock, Ark., on Sept. 26, 1957, the year nine black students, escorted by the National Guard, integrated the school.
Equity & Diversity Safe Space or Segregation? Affinity Groups for Teachers, Students of Color
See what affinity groups are and why they're coming under fire.
6 min read
Photograph of group of teachers meeting.
Getty/E+/SDI Productions