2009 Report Identified Dozens of Pa. Schools for Possible Cheating

By Benjamin Herold — July 11, 2011 6 min read

Dozens of schools across the city and state were flagged in a study of 2009 state standardized test scores that sought to use statistical analysis to ferret out possible examples of cheating on the PSSA exam.

The analysis, prepared for the Pennsylvania Department of Education in July 2009, highlights roughly 60 schools with suspicious results due to multiple statistical irregularities, including 22 Philadelphia district schools and seven Philadelphia charters.

Among the Philadelphia district schools referenced in the report is Roosevelt Middle School, which has been at the center of a controversy this year involving alleged cheating on the PSSA. In 2009, the analysis reveals, results of both the reading and math PSSA exams taken by Roosevelt’s 7th and 8th graders showed a highly unlikely number of wrong answers that were erased and changed to the correct answer. The results also showed highly improbable increases over the previous year in the percentage of students who scored proficient or advanced.

For example, the odds that the wrong-to-right erasure patterns that showed up on Roosevelt’s 7th grade reading response sheets occurred purely by chance were slightly less than 1 in 100 trillion.

The “data forensics technical report,” prepared by the Data Recognition Corporation, was made available to the Notebook by the state. At the Notebook‘s request, Andrew Porter, dean of the Graduate School of Education at the University of Pennsylvania and an expert on testing, reviewed the report and some of its data.

Porter stressed that that statistical analysis alone, without witnesses or confessions, cannot definitively prove that there was cheating. But he added that the report “describes a reasonable approach to identifying schools where there may have been cheating.”

Nevertheless, it appears that the state never followed up with any further investigations. The forensic data analysis was discontinued in 2010, although state Department of Education spokesperson Timothy Eller said it would be reinstated this year.

“We’re going to reinstitute those reports this year and they are actually going to mean something,” said Eller. He criticized the 2009 report as “convoluted.”

Earlier this year, teachers from Roosevelt and FitzSimons High told the Inquirer that they had witnessed numerous instances of cheating at their schools in 2010 and 2011. The Roosevelt teachers also spoke to the Notebook. FitzSimons was not one of the schools flagged in the state report for suspicious results in 2009.

Eller acknowledged that the state Department of Education has received two complaints from Philadelphia about possible cheating and said the department had ordered the district to investigate. Earlier this week, district officials said they found no wrongdoing at Roosevelt or FitzSimons. On Friday, district spokesperson Shana Kemp told the Inquirer that the district is investigating 15 schools and would wait until all are complete before submitting the reports to the state.

Eller sought to minimize the relevance of the 2009 report, pointing out that it was done under the previous administration of Gov. Ed Rendell and saying that “it is difficult to glean anything from that information.”

Eller stopped short, however, of saying it was wrong, emphasizing rather that it had not been presented in comprehensible form.

“It’s not information the average person would understand,” he said.

The report and accompanying analysis cost the state $113,000.

A DRC spokespersonsaid that the company could not answer any questions about the significance or context of report’s findings without state Department of Education’s permission, which was not granted.

Prominent in the DRC report is an analysis of erasures on the state tests. The study says that schools flagged for irregular erasure patterns are “of particular interest” because “these results may strongly suggest that a testing irregularity occurred in the school.”

At some Philadelphia schools, erasure patterns were flagged across numerous tested grades and subjects. For example, Olney Elementary, a K-8 school, was flagged for its erasure patterns in both reading and math in every tested grade at the school.

But the study looked at other possible evidence of test cheating as well.

A Notebook review showed roughly 60 schools across Pennsylvania received three or more “flags” in a single grade for improbable jumps in students’ performance levels and unlikely increases in schools’ scale scores on reading and math tests, as well as unusual erasure patterns. Nearly half of those schools singled out for having particularly suspicious results were in Philadelphia: 22 district schools and six seven Philadelphia charter schools had “flags” on three or more measures in a single grade.

“Multiple flags mean that the school data were highly unusual on more than one indicator,” said Penn’s Porter. “There are many ways to cheat, so it’s not wise to look at the data in only one way.”

Six of the 22 district schools had more than three “flags” for more than one grade.

Wagner Middle School in West Oak Lane, for example, was flagged in the report six times for its 7th grade results and three times for its 8th grade results. Overall, the likelihood of the erasure patterns on the school’s 8th grade response sheets in reading and math occurred purely by chance was more than 1 in 100 trillion.

From 2007-08 to 2008-09, the percentage of Wagner 8th graders scoring proficient on the PSSA exam jumped almost 14 percentage points, from 56.5 percent to 69.3 percent.

Statewide, 22 high schools had three or more flags. Three of those were Philadelphia district high schools, including Northeast High, which was flagged on every analyzed dimension of its 11th grade math results. Three local charter high schools were also flagged, including Imhotep Institute Charter High School, which was flagged seven of a possible eight times, including for a wrong-to-right erasure rate on its 11th grade math results that had a one in 10,000,000,000,000,000,000,000,000,000,000,000,000,000 chance of occurring purely by chance.

Questions about test score cheating in Philadelphia come on the heels of the district’s recent announcement that Philadelphia public schools improved their test scores for the ninth straight year in 2010-11. Over the past several months there have been several cheating scandals across the country.

Earlier this year, USA Today published a major investigation analyzing several years of test scores in six states and the District of Columbia that found statistical evidence of potential cheating. The series prompted a probe in Washington, D.C., where unusually high erasure rates were found in more than half of the schools.

In Atlanta, the state ordered an investigation into cheating after years of reporting by the Atlanta Journal-Constitution on similar unusual statistical patterns and other testing irregularities. The results, released this week, confirmed cheating in 44 of the 56 schools state investigators examined. Based on more than 2000 interviews, it named 178 educators as engaging in unethical behavior, said 80 of them had confessed, and concluded that the administration of former superintendent Beverly Hall had ignored or covered up evidence. Hall issued a statement Friday apologizing for any “shortcomings.”

A focus in both of those scandals was on suspicious erasure patterns detected by testing companies on student response sheets.

Eller, of the state Department of Education, said that new Pennsylvania Secretary of Education Ron Tomalis, appointed by Republican Gov. Tom Corbett earlier this year, has directed that forensic analysis and security audits be reinstated for the current year.

Eller gave no timetable for the completion of these reports. The PSSAs were administered in March.

To some, the need for more scrutiny of test score results grows in tandem with the intense pressure for schools to do well on standardized tests.

“As a general rule, the higher the stakes for high test performance, the more incentive to cheat, [and] the more cheating will be done,” Porter said.

Republished with permission from The Philadelphia Public School Notebook. Copyright © 2011 The Philadelphia Public School Notebook.


Jobs The EdWeek Top School Jobs Virtual Career Fair
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
How to Power Your Curriculum With Digital Books
Register for this can’t miss session looking at best practices for utilizing digital books to support their curriculum.
Content provided by OverDrive
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Embracing Student Engagement: The Pathway to Post-Pandemic Learning
As schools emerge from remote learning, educators are understandably worried about content and skills that students would otherwise have learned under normal circumstances. This raises the very real possibility that children will face endless hours
Content provided by Newsela

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Accountability Opinion Absenteeism Is the Wrong Student Engagement Metric to Use Right Now
In a post-pandemic era for school accountability, let’s focus on measuring what matters.
Sara Johnson, Annette Anderson & Ruth R. Faden
4 min read
Figure being erased.
Accountability Biden Education Team Squashes States' Push to Nix All Tests but Approves Other Flexibility
The department has telegraphed its decision to deny states' requests to cancel federally mandated tests for weeks.
3 min read
A first-grader learns keyboarding skills at Bayview Elementary School in San Pablo, Calif on March 12, 2015. Schools around the country are teaching students as young as 6 years old, basic typing and other keyboarding skills. The Common Core education standards adopted by a majority of states call for students to be able to use technology to research, write and give oral presentations, but the imperative for educators arrived with the introduction of standardized tests that are taken on computers instead of with paper and pencils.
The U.S. Department of Education denied some states' requests to cancel standardized tests this year. Others are seeking flexibility from some testing requirements, rather than skipping the assessments altogether.
Eric Risberg/AP
Accountability Explainer Will There Be Standardized Tests This Year? 8 Questions Answered
Educators want to know: Will the exams happen? If so, what will they look like, and how will the results be used?
12 min read
Students testing.
Accountability Opinion What Should School Accountability Look Like in a Time of COVID-19?
Remote learning is not like in person, and after nine months of it, data are revealing how harmful COVID-19 has been to children's learning.
6 min read
Image shows a speech bubble divided into 4 overlapping, connecting parts.
DigitalVision Vectors/Getty and Laura Baker/Education Week