The New York City Council is considering a requirement that all city agencies publish the source code behind algorithms they use to target services to city residents, raising the specter of significant changes in how the country’s largest school district assigns students to high school, evaluates teachers, and buys instructional software.
In an age where public officials increasingly rely on big data to make decisions, proponents describe the measure as a first-of-its-kind attempt to bolster government transparency and accountability.
“While it is undeniable that these tools help city agencies operate more effectively and offer residents more targeted impactful services, algorithms are not without issue,” said the bill’s author, Councilmember James Vacca, during a hearing last month on the proposed legislation.
“In our city, it is not always clear when and why agencies deploy algorithms, and when they do, it is often unclear what assumptions [those algorithms] are based upon and what data they even consider.”
The bill has sparked strong, and mixed, reactions.
The office of Mayor Bill de Blasio says it supports the measure’s intent, but objects to its scope. Other observers point to the possibility of unintended consequences, including potential security risks and a possible chilling effect on businesses worried about protecting their proprietary computer code. And the creator of the 1.1-million student New York City school system’s most well-known automated decision-making system says its algorithms are already open—but that hasn’t prevented widespread confusion and complaints.
Whether and how such tensions get resolved could have national implications, said Claire Fontaine, a researcher at Data & Society, a New York-based think tank.
“The goal of open algorithms for the provision of municipal services is a laudable and necessary one,” Fontaine said.
“However, there will be issues around technical capacity, feasibility, buy-in, and enforcement.”
Pros and Cons of Algorithmic Decision-Making
Broadly speaking, an algorithm is a set of rules to solve a problem.
Software and digital tools often use algorithms to analyze massive amounts of data to identify patterns, deliver recommendations, and make predictions about the future.
When used by public agencies, proponents say algorithm-driven software can help improve efficiency, reduce human error, and help generate desired policy outcomes.
Critics, however, say that such automated systems for making decisions or recommendations can exacerbate existing inequalities, particularly when they rely on historical data that may be tainted by bias.
Int. 1696, as Vacca’s bill is formally known, does not seek to prevent the use of algorithms in public-agency decision-making. Instead, it aims to make those algorithms more available for public scrutiny and testing.
To date, much of the public discussion around the bill’s possible impact has focused on the criminal-justice system, where officials are using or testing algorithmic systems to help decide where to deploy officers, how to sentence criminals, and whether to offer bail to the accused.
But as it currently stands, 1696 would also apply to any New York City agency that uses algorithms or automated data-processing systems “for the purposes of targeting services to persons, imposing penalties upon persons, or policing.” That includes the New York City Department of Education.
Indeed, one of the examples that Vacca has cited repeatedly when discussing the bill is New York’s complicated high-school assignment system, one of several across the country that relies on software to match students with schools. (For a detailed description of how the system works, see this 2013 Education Week story.)
“I strongly believe the public has a right to know when decisions are made using algorithms, and they have a right to know how these decisions are made,” Vacca said during the October hearing. “When the Department of Education uses an algorithm to assign children to different high schools, and a child is assigned to their sixth choice, they and their family have a right to know how that algorithm determined that their child would get their sixth choice.”
In written response to questions from Education Week, Vacca said he hoped his legislation would also bring transparency and improvements to the district’s “inaccurate or erratic teacher evaluations,” which he said “can occasionally spit out pretty different scores for the same teachers from year to year, or low scores for good teachers.”
And outside observers point out that nearly all schools, including those in New York, use a wide range of instructional and administrative software programs that rely on algorithms.
“It would be fascinating to get a peek under the hood of proprietary educational software that purports to be adaptive and personalized,” said Fontaine of Data & Society. “In addition, many charter-management organizations are deeply invested in data-driven decision-making and invest significant resources in the collection, aggregation, and presentation of data to demonstrate their effectiveness.”
In a statement, the 1.1-million student New York City school system said that is reviewing the bill’s potential impact.
Neil Dorosin had a more substantive take, highlighting some of the on-the-ground challenges associated with making automated decision-making accessible to the public.
Now the executive director of the nonprofit Institute for Innovation in Public School Choice, or IIPSC, Dorosin previously worked as the director of high-school-admissions operations for the New York City schools, where he helped implement the algorithm-driven high-school-assignment process that is in place today.
In an interview, Dorosin said parents and the general public are certainly entitled to know how students are assigned to schools, including the specific algorithms at work.
The problem, he said, is that the algorithm in question (which has also been applied to a wide range of other real-world situations, including matching kidneys to donors) is already open to public inspection. It’s called the Gale-Shapley algorithm. You can find it on Wikipedia. One of its creators, Lloyd S. Shapley, shared the 2012 Nobel Prize in Economics with Alvin Roth, who chairs IIPSC’s scientific advisory board.
In the context of New York City schools, the algorithm works by analyzing information from students and parents themselves (a rank-order listing of schools they want to attend) and from the Department of Education (each school’s admissions rules and preferences.) Dorosin pointed out that the latter set of information is also public, in the district’s annual high school directory.
So why are lawmakers pushing to make open an algorithm that is already publicly available?
“It’s complicated material,” Dorosin said. “A lot of people still don’t get it, and that’s a problem.”
Other concerns with Int. 1696 center around possible unintended consequences.
In its current form, the bill would “generate considerable risk, providing a roadmap for bad actors to attack crucial city systems,” said Don Sunderland, the deputy commissioner for enterprise and solution architecture in the city’s information-technology department, during the October hearing. Such security concerns are most pronounced for older city software systems, for which it may be difficult to even identify the relevant source code, Sunderland said.
Industry groups also raised objections.
“Mandating proprietary information, which many companies have built their businesses on, be shared on public websites could cause a chilling effect on local companies willing to do business with the city,” testified Taline Sanassarian, the policy director for Tech NYC, a non-profit trade group with about 500 industry members.
In response to questions from Education Week, Vacca said his office is “thrilled” with the feedback, and that will inform significant revisions to the bill.
“To my knowledge, we are the first legislative body of any size in the U.S. to take this issue on,” Vacca said. “This legislation is chiefly aimed at increasing democratic accountability and understanding, and we are now reviewing further mechanisms to ensure that.”
A version of this news article first appeared in the Digital Education blog.