Special Report
Equity & Diversity Q&A

Why Schools Need to Talk About Racial Bias in AI-Powered Technologies

By Benjamin Herold — April 12, 2022 6 min read
Illustration of pop up windows and notifications of different programs and applications
  • Save to favorites
  • Print

Schools are embracing education technologies that use artificial intelligence for everything from teaching math to optimizing bus routes.

Their goals are to save money, personalize student learning, and free teachers from rote administrative tasks.

But how can educators know if the data and design processes those products rely on have been skewed by racial bias? And what happens if they’re afraid to ask?

“If you’re not breaking data down by subpopulations to really understand where harm might be caused, you’re perpetuating racist systems and structures that have been in place for centuries,” said Sierra Noakes, the Edtech Marketplace Project Director at the nonprofit Digital Promise.

The group has teamed up with fellow nonprofit Edtech Equity Project to offer a new product certification called Prioritizing Racial Equity in AI Design. The idea is to provide both schools and companies with a common language and process for evaluating whether the makers of education technology tools are taking steps to:

  • Identify and question their own biases and assumptions.
  • Ensure the data used to train their artificial intelligence systems aren’t tainted by bias.
  • Provide educators and families alike with more visibility into how their products actually work and the risks they might pose.

The initiative comes on the heels of three other certifications offered by Digital Promise. Twenty-seven ed-tech products have already been recognized for addressing “learner variability.” Sixty-one have earned certification for “research-based design,” and one product has earned a recently launched certification regarding the use of learning analytics.

How concerned are you about the potential for racial bias in the educational technology used in schools?

As part of the pilot for the new racial equity certification, an undisclosed number of ed-tech companies volunteered to send at least one product through the review process. Findings will be made public in May.

The results will come at a moment when school districts around the country face significant backlash for their efforts to teach about racism and promote racial justice. Still, Noakes and Edtech Equity Project co-founders Madison Jacobs and Nidhi Hebbar said in a wide-ranging interview with Education Week that now is the time for a broader conversation about bias in AI.

“Companies are taking advantage of schools’ misunderstandings of artificial intelligence,” Jacobs said. “It’s important for us to combat that.”

The conversation with Hebbar, Jacobs, and Noakes has been edited for length and clarity.

What are some examples of AI-powered ed-tech tools that could have blind spots or been designed using faulty or racist assumptions?

Hebbar: Personalized-learning software that adapts to the answers students give. Language-learning software that takes into account how students speak, but is often trained on data from native-English speakers born in America and treat other accents and dialects as wrong. Also, administrative tools that are used to identify who might be at risk for behavioral or academic reasons.

Jacobs: A lot of school districts are thinking about how they would utilize facial recognition technology. Those systems are clearly not built with the safety and security of Black folks in mind. It’s also all kind of connected. Students might get misidentified or mislabeled in one system, and that system then pushes the data to other systems.

Noakes: What we’ve learned in creating the certification so far is that there’s tremendous resistance from [companies] to collecting and disaggregating data by race. They’re sort of living in this place where ignorance is bliss. But unless a tool is intentionally looking at their impact data, they’re engaging in a harmful structure.

What harms can result for students of color? Are they hypothetical, or are we already seeing them in actual schools?

Noakes: AI tools often make decisions on behalf of teachers without teachers having any insight into what variables are leading to those decisions. And in some instances, there’s not an override function. So, for example, grouping students. If those decisions are dictated by an algorithm, educators are blind to what the decisionmaking process looks like. These tools are making decisions they disagree with, but they can’t go to school leadership to say this tool is not taking something into account.

Hebbar: It’s not like, “AI is bad, full stop.” But to be used in schools, it needs to be done in a way that allows educators who have a full understanding of a child to override [algorithmic decisions]. Giving users the ability to course-correct can mitigate a lot of the mistakes in technology, but we’re not seeing that with a lot of tools.

How aware of these problems are K-12 leaders?

Hebbar: It’s a mix. A lot of districts, especially the large ones with a lot of Black and brown students, are very aware of this. Others don’t even know which tools they’re purchasing that use AI. But what we generally see is that leaders often feel like, “I’m not an expert, I don’t even know what questions to ask.” The idea behind the certification is to make that process simpler.

Jacobs: Part of the problem is there is no across-the-board policy that requires companies to disclose what is actually operating inside their technology. And I think a lot of leaders in schools still have the perception that if you’re basing things on numbers, then systems must be operating in a neutral fashion. But these systems are built by people, and therefore people’s inherent biases, their unconscious biases, their explicit biases go into the systems. A lot of our conversations are about educating people on a general level that these are not neutral systems.

How do companies respond to that message?

Jacobs: There are definitely companies that are sort of hiding behind the protections that the black box of AI offers. They may have engineering teams that are skewed toward white males. Bringing that to light is hard to do. They may not want to expose their shortcomings. There’s also this culture in tech about being the best and doing the most, and that really is a horrible barrier to thinking about how you design through the lens of the collective liberation of the people utilizing your technology. But some companies understand that legislation is coming [to address racial bias in tech] and that the venture capital ecosystem is shifting, with more funders wanting to invest in companies that treat equity as a core principle in their product development.

What steps do you want to see companies take to address these problems?

Hebbar: Looking at the collection of data and the types of students whose data are used to train a company’s system is a big one. But there are places for bias to come in at almost every stage of the product development and design process. The certification is built off a toolkit to help companies understand how to mitigate that. Even in the ideation phase, you really want to check, “What are our assumptions and blind spots?” And then in the very last step, how are companies re-evaluating on a regular basis whether their products are effective?

There’s a backlash throughout the K-12 system to the way schools and districts are approaching racial equity. Many people just don’t agree with the idea that schools are systemically racist, and they’ll likely disagree with the idea that educators and developers should be trying to undo the existing system. Why should they support this certification?

Jacobs: When you have groups of folks that would fight against a practice of looking into whether students are being treated fairly, that’s just another data point and proof that racism and bias exists. And if you don’t think that bias exists, then what is the problem for us to look and see?

Noakes: I believe to my core that when we’re designing systems and solutions for folks at the margins who are often ignored in design and development, we’re creating tools that help all learners thrive.

education week logo subbrand logo RC RGB

Data analysis for this article was provided by the EdWeek Research Center. Learn more about the center’s work.

A version of this article appeared in the April 13, 2022 edition of Education Week as Why Schools Need To Talk About Racial Bias In AI-Powered Technologies

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Reading & Literacy Webinar
(Re)Focus on Dyslexia: Moving Beyond Diagnosis & Toward Transformation
Move beyond dyslexia diagnoses & focus on effective literacy instruction for ALL students. Join us to learn research-based strategies that benefit learners in PreK-8.
Content provided by EPS Learning
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
Cohesive Instruction, Connected Schools: Scale Excellence District-Wide with the Right Technology
Ensure all students receive high-quality instruction with a cohesive educational framework. Learn how to empower teachers and leverage technology.
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School Climate & Safety Webinar
How to Use Data to Combat Bullying and Enhance School Safety
Join our webinar to learn how data can help identify bullying, implement effective interventions, & foster student well-being.
Content provided by Panorama Education

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Equity & Diversity What Works to Help Students of Color Feel Like They Belong at School
New research focuses on how ethnic studies classes and local partnerships can help students of all races feel they belong in school.
5 min read
Group of diverse people (aerial view) in a circle holding hands. Cooperation and teamwork. Community of friends, students, or volunteers committed to social issues for peace and the environment.
iStock/Getty Images Plus
Equity & Diversity It's Banned Books Week. Have the Challenges to Books Slowed Down?
Attempts to ban books in public schools are still prevalent, according to two new reports.
5 min read
Image of a bookshelf.
Luoman/E+
Equity & Diversity Educators Tend to View Black Girls More Harshly. Here Are the Consequences
Schools discipline Black girls more frequently and severely than their white peers—even for similar incidents.
8 min read
Sign on door that reads "Principal's Office" from a school.
Liz Yap/education Week with E+
Equity & Diversity Students Fell Behind During the Pandemic. Who Stayed Behind?
Not enough students are receiving the support they need, and there's a disproportionate toll on the most vulnerable students.
7 min read
An elementary teacher delivers a lesson in Spanish in a dual-language immersion class.
An elementary teacher delivers a lesson in Spanish in a dual-language immersion class. A report found that vulnerable students bear the brunt of slow academic-recovery gains.
Allison Shelley for EDUimages