Philadelphia District Suspends School Rating System, Seeks Fix

By Benjamin Herold — November 01, 2012 5 min read
  • Save to favorites
  • Print

The Philadelphia School District revealed Wednesday that its system for rating schools is faulty.

The District has suspended use of its “School Performance Index,” or SPI.

District leaders are now seeking outside help to fix the complicated formula that converts more than a dozen factors into a single score given to every public school in the city, including charters.

For the last two years, SPI scores have been used to help guide a wide range of major decisions, including which schools should be closed down or converted into charters. It has also been used to evaluate charters’ bids for renewal or expansion.

Leaders of several of the city’s charter schools have long taken issue with the index.

“We are, at this point, confident that there were some mistakes made,” said new District Deputy Superintendent Paul Kihn. “We honestly don’t know how extensive the problem is.”

Kihn said the issues with SPI stem from human error in how the accountability measure was calculated—not from faulty data resulting from cheating on state standardized tests.

“They are two different matters,” Kihn said.

A state-commissioned analysis of results from 2009-11 found evidence of widespread cheating at dozens of schools across the state, including 53 traditional public schools and three charters in Philadelphia. For the spring 2012 testing, after tight new test security measures were put in place, scores at many of those schools dropped dramatically—in some cases by 50 percentage points or more.

“I am not prepared to comment about the cheating matter in relation to this,” Kihn said.

A Far-Reaching Tool

The District developed its School Performance Index in 2009, when Arlene Ackerman was superintendent. The formula boils down either 13 indicators (for elementary and middle schools) or 17 indicators (for high schools) into a single score of 1-10, with 1 being the best.

For all schools, indicators relating to student scores on the Pennsylvania System of School Assessment (PSSA) exams are weighted most heavily, and measures of parent, student and teacher satisfaction also taken into consideration. High schools’ success in preparing students for college and careers also factors in.

In an interview Wednesday, Kihn acknowledged the central role that SPI has played in the District’s Renaissance Schools initiative, facilities master planning process, and decisions regarding charter schools. But Kihn stressed that the rankings were never “used as the sole determinant for any decision that we have made.”

The measure will be disregarded altogether as District officials target several dozen more schools for possible closure by next fall, he said.

“Going forward, we’ve made the decision that we will not use SPI at all in the facilities master planning process.”

Transition a Factor

The District first became aware of a potential problem with its performance index in May, Kihn said. In response, officials conducted an internal three-month investigation that concluded in late August. Action was not taken earlier because of the District’s recent change in leadership, he said.

Kihn was not specific about the nature of the errors and how they may have impacted the reliability of the school’s rating. But it is clear from the District’s detailed online explanation of SPI that the calculations rely on complex statistical modeling.

In addition to looking at the overall percentage of students in each school that achieved proficiency on standardized tests, it used student-level data to calculate how well the school helped individual students make significant progress from year to year—the so-called “growth model.”

SPI also relies on the individual student test-score data to determine whether schools were able to narrow the racial and ethnic achievement gap between White and Asian and Black and Latino students.

“Because [incoming Superintendent William Hite] and I were just transitioning in, a decision was made to hold on taking any action until we could be briefed and make the decision about how to proceed,” Kihn said.

The District’s Office of accountability remains responsible for developing and maintaining the School Performance Index.

In August, the School Reform Commission approved the termination of Daniel Piotrowski, then the District’s executive director of accountability and assessment, effective July 14.

Neither District officials nor Piotrowski would comment on whether his termination was connected to the problems with SPI.

Earlier this month, NewsWorks and the Philadelphia Public School Notebook reported that Piotrowski, who also served as the head of the District’s testing security program, had called in March for a full investigation of the testing procedures at Wagner Middle School, where he had reported more than a dozen testing violations. Piotrowksi was removed from Wagner, and other officials overturned his judgment that an investigation was warranted.

Charter Concerns

Some charter operators have criticized SPI since its inception, claiming the District’s numbers were not trustworthy and criticizing the formula.

“I’m glad that they recognized [the problem],” said David Hardy, the CEO of Boys’ Latin charter in West Philadelphia.

But Hardy, who has long advocated for an accountability measure that puts more emphasis on students’ postsecondary outcomes and less on their test scores, said his issues with the performance index go beyond a “math problem” resulting from faulty calculations.

“What they need to do is change the types of things they’re measuring and the weight of their measurements and take a broader view of success,” Hardy said. “In an environment of school choice, parents have to have [good] information so they know what they’re buying.”

Kihn acknowledged such concerns and said he personally notified leaders in the charter community of the problems with SPI on Wednesday morning.

Kihn also said he recently received a proposal for how to improve the performance index from a working group of the Great Schools Compact committee, which has convened District officials and charter leaders to examine the issue for months.

“We are certainly taking those very seriously as a set of thoughtful recommendations,” Kihn said.

Most immediately, though, the District is looking for an outside vendor who can recalculate SPI scores from 2009-10 and 2010-11, and calculate 2011-12 scores for the first time.

Kihn said that he hopes to receive bids by mid-November and that the work will take no more than a few months. All results and analysis of SPI in the future will be publicly posted to encourage transparency, he stressed.

Making recommendations about how to improve the measure will be optional for whichever vendor is selected, Kihn said.

Additional reporting by Notebook contributing editor Dale Mezzacappa.

This story was reported as part of a partnership in education coverage between WHYY/NewsWorks and the Notebook.

Republished with permission from The Philadelphia Public School Notebook. Copyright © 2012 The Philadelphia Public School Notebook.


Student Well-Being Webinar After-School Learning Top Priority: Academics or Fun?
Join our expert panel to discuss how after-school programs and schools can work together to help students recover from pandemic-related learning loss.
Budget & Finance Webinar Leverage New Funding Sources with Data-Informed Practices
Address the whole child using data-informed practices, gain valuable insights, and learn strategies that can benefit your district.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Classroom Technology Webinar
ChatGPT & Education: 8 Ways AI Improves Student Outcomes
Revolutionize student success! Don't miss our expert-led webinar demonstrating practical ways AI tools will elevate learning experiences.
Content provided by Inzata

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Accountability Education Secretary: Standardized Tests Should No Longer Be a 'Hammer'
But states won't ease accountability requirements until federal law tells them to do so, policy experts say.
5 min read
Close up of a student holding pencil and writing the answer on a bubble sheet assessment test with blurred students at their desks in the background
Accountability Timeline: How Federal School Accountability Has Waxed and Waned
From its origins in the 1990s to the most-recent tack, see how the federal approach to accountability has shifted.
4 min read
President George W. Bush, left, participates in the swearing-in ceremony for the Secretary of Education Margaret Spellings, center, at the U.S. Dept. of Education on Jan. 31, 2005 in Washington. On the far right holding a bible is her husband Robert Spellings.
President George W. Bush, left, participates in the swearing-in ceremony for the Secretary of Education Margaret Spellings, center, at the U.S. Dept. of Education on Jan. 31, 2005 in Washington. On the far right holding a bible is her husband Robert Spellings.
AP Photo/Pablo Martinez Monsivais
Accountability School Accountability Is Restarting After a Two-Year Pause. Here's What That Means
For a moment, the COVID-19 pandemic succeeded in doing what periodic protests about school accountability couldn't: Halting it.
10 min read
Illustration of a gauge.
Accountability Opinion Let's Take a Holistic Approach to Judging Schools
Parents wouldn't judge their kids based on a single factor. So, says Ron Berger of EL Education, why must schools use a lone test score?
8 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."