Philadelphia District Suspends School Rating System, Seeks Fix

By Benjamin Herold — November 01, 2012 5 min read

The Philadelphia School District revealed Wednesday that its system for rating schools is faulty.

The District has suspended use of its “School Performance Index,” or SPI.

District leaders are now seeking outside help to fix the complicated formula that converts more than a dozen factors into a single score given to every public school in the city, including charters.

For the last two years, SPI scores have been used to help guide a wide range of major decisions, including which schools should be closed down or converted into charters. It has also been used to evaluate charters’ bids for renewal or expansion.

Leaders of several of the city’s charter schools have long taken issue with the index.

“We are, at this point, confident that there were some mistakes made,” said new District Deputy Superintendent Paul Kihn. “We honestly don’t know how extensive the problem is.”

Kihn said the issues with SPI stem from human error in how the accountability measure was calculated—not from faulty data resulting from cheating on state standardized tests.

“They are two different matters,” Kihn said.

A state-commissioned analysis of results from 2009-11 found evidence of widespread cheating at dozens of schools across the state, including 53 traditional public schools and three charters in Philadelphia. For the spring 2012 testing, after tight new test security measures were put in place, scores at many of those schools dropped dramatically—in some cases by 50 percentage points or more.

“I am not prepared to comment about the cheating matter in relation to this,” Kihn said.

A Far-Reaching Tool

The District developed its School Performance Index in 2009, when Arlene Ackerman was superintendent. The formula boils down either 13 indicators (for elementary and middle schools) or 17 indicators (for high schools) into a single score of 1-10, with 1 being the best.

For all schools, indicators relating to student scores on the Pennsylvania System of School Assessment (PSSA) exams are weighted most heavily, and measures of parent, student and teacher satisfaction also taken into consideration. High schools’ success in preparing students for college and careers also factors in.

In an interview Wednesday, Kihn acknowledged the central role that SPI has played in the District’s Renaissance Schools initiative, facilities master planning process, and decisions regarding charter schools. But Kihn stressed that the rankings were never “used as the sole determinant for any decision that we have made.”

The measure will be disregarded altogether as District officials target several dozen more schools for possible closure by next fall, he said.

“Going forward, we’ve made the decision that we will not use SPI at all in the facilities master planning process.”

Transition a Factor

The District first became aware of a potential problem with its performance index in May, Kihn said. In response, officials conducted an internal three-month investigation that concluded in late August. Action was not taken earlier because of the District’s recent change in leadership, he said.

Kihn was not specific about the nature of the errors and how they may have impacted the reliability of the school’s rating. But it is clear from the District’s detailed online explanation of SPI that the calculations rely on complex statistical modeling.

In addition to looking at the overall percentage of students in each school that achieved proficiency on standardized tests, it used student-level data to calculate how well the school helped individual students make significant progress from year to year—the so-called “growth model.”

SPI also relies on the individual student test-score data to determine whether schools were able to narrow the racial and ethnic achievement gap between White and Asian and Black and Latino students.

“Because [incoming Superintendent William Hite] and I were just transitioning in, a decision was made to hold on taking any action until we could be briefed and make the decision about how to proceed,” Kihn said.

The District’s Office of accountability remains responsible for developing and maintaining the School Performance Index.

In August, the School Reform Commission approved the termination of Daniel Piotrowski, then the District’s executive director of accountability and assessment, effective July 14.

Neither District officials nor Piotrowski would comment on whether his termination was connected to the problems with SPI.

Earlier this month, NewsWorks and the Philadelphia Public School Notebook reported that Piotrowski, who also served as the head of the District’s testing security program, had called in March for a full investigation of the testing procedures at Wagner Middle School, where he had reported more than a dozen testing violations. Piotrowksi was removed from Wagner, and other officials overturned his judgment that an investigation was warranted.

Charter Concerns

Some charter operators have criticized SPI since its inception, claiming the District’s numbers were not trustworthy and criticizing the formula.

“I’m glad that they recognized [the problem],” said David Hardy, the CEO of Boys’ Latin charter in West Philadelphia.

But Hardy, who has long advocated for an accountability measure that puts more emphasis on students’ postsecondary outcomes and less on their test scores, said his issues with the performance index go beyond a “math problem” resulting from faulty calculations.

“What they need to do is change the types of things they’re measuring and the weight of their measurements and take a broader view of success,” Hardy said. “In an environment of school choice, parents have to have [good] information so they know what they’re buying.”

Kihn acknowledged such concerns and said he personally notified leaders in the charter community of the problems with SPI on Wednesday morning.

Kihn also said he recently received a proposal for how to improve the performance index from a working group of the Great Schools Compact committee, which has convened District officials and charter leaders to examine the issue for months.

“We are certainly taking those very seriously as a set of thoughtful recommendations,” Kihn said.

Most immediately, though, the District is looking for an outside vendor who can recalculate SPI scores from 2009-10 and 2010-11, and calculate 2011-12 scores for the first time.

Kihn said that he hopes to receive bids by mid-November and that the work will take no more than a few months. All results and analysis of SPI in the future will be publicly posted to encourage transparency, he stressed.

Making recommendations about how to improve the measure will be optional for whichever vendor is selected, Kihn said.

Additional reporting by Notebook contributing editor Dale Mezzacappa.

This story was reported as part of a partnership in education coverage between WHYY/NewsWorks and the Notebook.

Republished with permission from The Philadelphia Public School Notebook. Copyright © 2012 The Philadelphia Public School Notebook.


Jobs The EdWeek Top School Jobs Virtual Career Fair
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
How to Power Your Curriculum With Digital Books
Register for this can’t miss session looking at best practices for utilizing digital books to support their curriculum.
Content provided by OverDrive
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Embracing Student Engagement: The Pathway to Post-Pandemic Learning
As schools emerge from remote learning, educators are understandably worried about content and skills that students would otherwise have learned under normal circumstances. This raises the very real possibility that children will face endless hours
Content provided by Newsela

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Accountability Biden Education Team Squashes States' Push to Nix All Tests but Approves Other Flexibility
The department has telegraphed its decision to deny states' requests to cancel federally mandated tests for weeks.
3 min read
A first-grader learns keyboarding skills at Bayview Elementary School in San Pablo, Calif on March 12, 2015. Schools around the country are teaching students as young as 6 years old, basic typing and other keyboarding skills. The Common Core education standards adopted by a majority of states call for students to be able to use technology to research, write and give oral presentations, but the imperative for educators arrived with the introduction of standardized tests that are taken on computers instead of with paper and pencils.
The U.S. Department of Education denied some states' requests to cancel standardized tests this year. Others are seeking flexibility from some testing requirements, rather than skipping the assessments altogether.
Eric Risberg/AP
Accountability Explainer Will There Be Standardized Tests This Year? 8 Questions Answered
Educators want to know: Will the exams happen? If so, what will they look like, and how will the results be used?
12 min read
Students testing.
Accountability Opinion What Should School Accountability Look Like in a Time of COVID-19?
Remote learning is not like in person, and after nine months of it, data are revealing how harmful COVID-19 has been to children's learning.
6 min read
Image shows a speech bubble divided into 4 overlapping, connecting parts.
DigitalVision Vectors/Getty and Laura Baker/Education Week
Accountability State Schools Chiefs Push Biden for Wiggle Room on Accountability During Pandemic
State schools chiefs say it's necessary to change how they use scores from mandated annual tests during the unprecedented disruption created by the coronavirus pandemic.
4 min read
Image of students taking a test.