An internal review of suspicious 2009 standardized test score results at 28 city schools has yielded no firm evidence of cheating, but 13 of the schools had results that warrant more intensive investigation, Philadelphia school district officials announced on Monday.
Officials declined to identify the schools and said the district is awaiting more data, including any irregularities from subsequent testing years, and requesting further assistance from the Pennsylvania Department of Education before undertaking a more complete probe.
“We are asking to partner with the state to get more comprehensive data for 2009, 2010, and 2011 so we can understand where we need to investigate, and we also ask the Pennsylvania Department of Education to assist us in that investigation,” said Fran Newburg, the district’s deputy chief of accountability and educational technology.
In July, the state department of education directed 40 districts across the state to investigate a total of 89 schools that were flagged multiple times for statistical irregularities in a recently surfaced July 2009 report. The report used statistical analysis to look for suspicious erasure patterns, improbable jumps in schools’ scores, and unlikely jumps in student performance levels, among other things.
In Philadelphia, 28 district schools and 10 charters were deemed worthy of further inquiry.
Typically, district investigations into possible cheating involve interviews with students and staff. In this case, however, district officials relied solely on an analysis of the data provided to them by state department of education, which they criticized as incomplete and not sufficiently comprehensive to warrant full-fledged inquiries.
“What we have is very vague allegations against schools,” said Daniel Piotrowski, a staffer in the district’s accountability office. “As of now, we don’t have the type of data to really start an investigation knowing that there is going to be more information coming in a month or so from PDE.”
Similar forensic analyses of 2010 and 2011 results on the Pennsylvania System of School Assessment exams are expected sometime in September.
Pennsylvania Department of Education spokesperson Tim Eller declined to comment on any specific district reports, but emphasized that they are only a “first step” in the process of determining if there was cheating on the PSSA exams.
“The department will conduct its own review and analysis of the forensic data, as well as the reports provided by the districts, to determine if further action is necessary,” said Eller.
At fifteen of the schools the district was asked to investigate, officials said they were able to “explain away” the suspicious results simply by reviewing the data provided by the state department of education.
Seven of those schools were flagged only for things like unexpected changes in the participation rates of particular subgroups.
“There is no real reason to believe that these schools modified student information improperly,” said Piotrowski.
Another eight were flagged for some combination of factors that district officials determined were neither abnormal nor suspicious. Some of those schools had unusually high numbers of wrong-to-right erasures or improbable jumps in the percentage of students scoring proficient, but the district determined that additional investigations were unnecessary.
Thirteen schools, however, were flagged for a combination of suspicious erasure patterns and unlikely changes in student performance levels.
At Wagner Middle School in West Oak Lane, for example, dozens of 7th graders had their response sheets flagged for highly suspicious numbers of incorrect answers that were erased and changed to the correct answer. Between 2008 and 2009, the percentage of Wagner 7th graders scoring proficient in both reading and math jumped more than 23 percentage points.
Results such as those warrant a further look, said district officials, although it is not confirmed that Wagner is among the 13 schools.
Officials also concluded that they should “step up” their monitoring protocols at some schools during future administrations of the exam. One possibility is using a “tiered model” that could result in some schools not being allowed to handle their own testing materials.
Though they expressed numerous concerns with the 2009 forensic analysis, district officials said they were not attempting to discredit the report and stressed that they take any allegations of cheating seriously.
If improper conduct is uncovered, “people can lose jobs and will lose jobs,” said Newburg. “But we want to have as much information as we can before we go marching down that road.”
The charter schools flagged did their own investigations. The Charter High School of Architecture and Design, which had six flags, said its review “did not discover any cheating on the part of our students and we checked this very carefully.”
Instead, the irregularities were caused “some carelessness on our part,” according to CEO Peter Kountz. He said there was a “mix up” with the testing company regarding the proper demographic coding for students, and because the school missed its own labeling mistakes it didn’t make allowable corrections in time.
