The Philadelphia School District revealed Wednesday that its system for rating schools is faulty.
The District has suspended use of its “School Performance Index,” or SPI.
District leaders are now seeking outside help to fix the complicated formula that converts more than a dozen factors into a single score given to every public school in the city, including charters.
For the last two years, SPI scores have been used to help guide a wide range of major decisions, including which schools should be closed down or converted into charters. It has also been used to evaluate charters’ bids for renewal or expansion.
Leaders of several of the city’s charter schools have long taken issue with the index.
“We are, at this point, confident that there were some mistakes made,” said new District Deputy Superintendent Paul Kihn. “We honestly don’t know how extensive the problem is.”
Kihn said the issues with SPI stem from human error in how the accountability measure was calculated—not from faulty data resulting from cheating on state standardized tests.
“They are two different matters,” Kihn said.
A state-commissioned analysis of results from 2009-11 found evidence of widespread cheating at dozens of schools across the state, including 53 traditional public schools and three charters in Philadelphia. For the spring 2012 testing, after tight new test security measures were put in place, scores at many of those schools dropped dramatically—in some cases by 50 percentage points or more.
“I am not prepared to comment about the cheating matter in relation to this,” Kihn said.
A Far-Reaching Tool
The District developed its School Performance Index in 2009, when Arlene Ackerman was superintendent. The formula boils down either 13 indicators (for elementary and middle schools) or 17 indicators (for high schools) into a single score of 1-10, with 1 being the best.
For all schools, indicators relating to student scores on the Pennsylvania System of School Assessment (PSSA) exams are weighted most heavily, and measures of parent, student and teacher satisfaction also taken into consideration. High schools’ success in preparing students for college and careers also factors in.
In an interview Wednesday, Kihn acknowledged the central role that SPI has played in the District’s Renaissance Schools initiative, facilities master planning process, and decisions regarding charter schools. But Kihn stressed that the rankings were never “used as the sole determinant for any decision that we have made.”
The measure will be disregarded altogether as District officials target several dozen more schools for possible closure by next fall, he said.
“Going forward, we’ve made the decision that we will not use SPI at all in the facilities master planning process.”
Transition a Factor
The District first became aware of a potential problem with its performance index in May, Kihn said. In response, officials conducted an internal three-month investigation that concluded in late August. Action was not taken earlier because of the District’s recent change in leadership, he said.
Kihn was not specific about the nature of the errors and how they may have impacted the reliability of the school’s rating. But it is clear from the District’s detailed online explanation of SPI that the calculations rely on complex statistical modeling.
In addition to looking at the overall percentage of students in each school that achieved proficiency on standardized tests, it used student-level data to calculate how well the school helped individual students make significant progress from year to year—the so-called “growth model.”
SPI also relies on the individual student test-score data to determine whether schools were able to narrow the racial and ethnic achievement gap between White and Asian and Black and Latino students.
“Because [incoming Superintendent William Hite] and I were just transitioning in, a decision was made to hold on taking any action until we could be briefed and make the decision about how to proceed,” Kihn said.
The District’s Office of accountability remains responsible for developing and maintaining the School Performance Index.
In August, the School Reform Commission approved the termination of Daniel Piotrowski, then the District’s executive director of accountability and assessment, effective July 14.
Neither District officials nor Piotrowski would comment on whether his termination was connected to the problems with SPI.
Earlier this month, NewsWorks and the Philadelphia Public School Notebook reported that Piotrowski, who also served as the head of the District’s testing security program, had called in March for a full investigation of the testing procedures at Wagner Middle School, where he had reported more than a dozen testing violations. Piotrowksi was removed from Wagner, and other officials overturned his judgment that an investigation was warranted.
Charter Concerns
Some charter operators have criticized SPI since its inception, claiming the District’s numbers were not trustworthy and criticizing the formula.
“I’m glad that they recognized [the problem],” said David Hardy, the CEO of Boys’ Latin charter in West Philadelphia.
But Hardy, who has long advocated for an accountability measure that puts more emphasis on students’ postsecondary outcomes and less on their test scores, said his issues with the performance index go beyond a “math problem” resulting from faulty calculations.
“What they need to do is change the types of things they’re measuring and the weight of their measurements and take a broader view of success,” Hardy said. “In an environment of school choice, parents have to have [good] information so they know what they’re buying.”
Kihn acknowledged such concerns and said he personally notified leaders in the charter community of the problems with SPI on Wednesday morning.
Kihn also said he recently received a proposal for how to improve the performance index from a working group of the Great Schools Compact committee, which has convened District officials and charter leaders to examine the issue for months.
“We are certainly taking those very seriously as a set of thoughtful recommendations,” Kihn said.
Most immediately, though, the District is looking for an outside vendor who can recalculate SPI scores from 2009-10 and 2010-11, and calculate 2011-12 scores for the first time.
Kihn said that he hopes to receive bids by mid-November and that the work will take no more than a few months. All results and analysis of SPI in the future will be publicly posted to encourage transparency, he stressed.
Making recommendations about how to improve the measure will be optional for whichever vendor is selected, Kihn said.
