School & District Management

Questions Raised on New York Test System’s Reliability

September 29, 2009 5 min read
05NYTests

The recent news that 97 percent of New York City public schools got an A or B under the district’s grading system might be seen as reason for celebration, but critics suggest the grades hold little value and highlight the need to revisit the state assessment system.

The results, they say, reveal far more about flaws in the city’s so-called “progress reports”—and the state testing regime that largely drives them—than they do about the quality of education in the 1.1 million-student district.

Eighty-four percent of the city’s 1,058 public elementary and middle schools received an A on the city’s report cards this year, compared with 38 percent in 2008, while 13 percent received a B, city officials announced this month.

“It tells us virtually nothing about the actual performance of schools,” Aaron M. Pallas, a professor of sociology and education at Teachers College, Columbia University, said of the city’s grades.

Diane Ravitch, an education historian at New York University, was even sharper: She declared the school grades “bogus” in a Sept. 9 opinion piece for the Daily News of New York, saying the city’s report card system “makes a mockery of accountability.”

But Andrew J. Jacob, a spokesman for the New York City Department of Education, defended the ratings, even as he said the district’s demands on schools would continue to rise next year.

“A lot of our schools simply zoomed past what they had been able to achieve in the past,” he said, helping their students “make a great amount of progress.”

He cautioned, however: “The fact that a school earned an A does not mean that it’s where it needs to be.” And Mr. Jacob said the report cards provide plenty of information beyond a simple overall letter grade.

Complex Methodology

The new grades come as New York City Mayor Michael R. Bloomberg, an Independent who oversees the city school system, is making education a key issue in his re-election bid. His Democratic challenger, William C. Thompson Jr., has been attacking him on the topic.

The city employs a complex methodology to devise its overall letter grades, with the primary driver being results from statewide assessments in reading and mathematics, which have also encountered considerable skepticism lately.

The city’s grades are based on three categories: student progress on state tests from one year to the next, which accounts for 60 percent; student performance for the most recent school year, which accounts for 25 percent; and school environment, which makes up 15 percent.

Mr. Pallas of Teachers College argues that one key flaw with the city’s rating system is that it depends heavily on a what he deems a “wholly unreliable” measure of student growth on test scores from year to year that fails to account adequately for statistical error.

Further complicating matters is controversy about the reliability of the New York state testing system in gauging student achievement.

Since the implementation of a revamped curriculum and testing system in 2006, the state has posted some apparently impressive gains in the proportion of students meeting state standards, called Level 3, or exceeding state standards, Level 4.

In June, for example, the state department of education announced that 86 percent of students in grades 3-8 statewide had a passing score on the math exam, up from 66 percent in 2006. For New York City, the passing rate in math for those grades was 82 percent, up from 57 percent three years earlier.

Some other New York cities saw even bigger gains. In Buffalo, the proportion of students who passed the math test rose to 63 percent, from 29 percent, over that time period; in Rochester, it grew to 63 percent, from 33 percent.

The state agency cautioned in a press release that increases in the average scale scores for students in particular grades were “sometimes relatively smaller.” One explanation: More students who were just below the passing rate may have made slight gains to cross the threshold.

Skepticism on Gains

The reliability of the state’s assessment system has been debated for a variety of reasons, from concerns about whether the state may be manipulating the results to whether the exams are especially vulnerable to teaching to the test. Observers say a lot of the questions are very similar from year to year.

“What people really want to know in New York is, can we trust these big increases?” said DanielM. Koretz, an education professor at Harvard University. “And my answer is, I don’t know.”

Some analysts say recent data from the National Assessment of Educational Progress, dubbed “the nation’s report card,” raise questions about the New York tests. Student achievement for 4th and 8th graders in New York state in both reading and math on NAEP has stayed relatively flat. Results from the 2009 reading and math tests are expected out in coming months.

Ms. Ravitch has been especially outspoken about the state tests. Writing this month on the Bridging Differences blog she co-writes on Education Week’s Web site, she accused the state education department of rigging the testing system to produce inflated results, saying the “double-digit gains are phony.” The proportion of points a student must earn on the tests to move to a higher level “has steadily fallen in many grades,” she said.

“When states play games with cut scores and conversions from raw scores to scale scores, testing becomes a mighty scam,” she wrote.

Changes Ahead

Jonathan Burman, a spokesman for the state education department, did not explicitly address that and other criticisms when asked to respond for this story, but he said the state is working on changes to the testing program.

The New York state board of regents has indicated that the state “will raise standards next year, with a higher cut score required to demonstrate proficiency on the state exams,” he said in an e-mail.

State officials have previously said the percentage of points required to meet state standards has declined on some of the tests because the questions have become harder. The adjustments were made to ensure comparability from year to year, according to those officials.

“That makes perfect sense if the test has become sufficiently harder,” said Mr. Koretz, from Harvard. “But we don’t know if that’s true.”

Related Tags:

A version of this article appeared in the September 30, 2009 edition of Education Week as Questions Raised on New York Test System’s Reliability

Events

Student Well-Being Webinar Boosting Teacher and Student Motivation During the Pandemic: What It Takes
Join Alyson Klein and her expert guests for practical tips and discussion on how to keep students and teachers motivated as the pandemic drags on.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
A Holistic Approach to Social-Emotional Learning
Register to learn about the components and benefits of holistically implemented SEL.
Content provided by Committee for Children
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
How Principals Can Support Student Well-Being During COVID
Join this webinar for tips on how to support and prioritize student health and well-being during COVID.
Content provided by Unruly Studios

EdWeek Top School Jobs

Arizona School Data Analyst - (AZVA)
Arizona, United States
K12 Inc.
Software Engineer
Portland, OR, US
Northwest Evaluation Association
Proposal Writer
Portland, OR, US
Northwest Evaluation Association
CCLC Program Site Director
Thornton, CO, US
Adams 12 Five Star Schools

Read Next

School & District Management New York City's Equity-Minded Schools Chief Resigns
Richard A. Carranza, the chancellor of the New York City schools, announced Feb. 26 he will step down from the job this month.
4 min read
Richard Carranza, Chancellor of the New York City Department of Education, arrives to Public School 188 The Island School as students arrive for in-person classes, on, Sept. 29, 2020, in the Manhattan borough of New York.
Richard A. Carranza announced he will depart the top New York City schools job in March.
John Minchillo/AP
School & District Management Opinion New Resource Tracks School System Reopening
The Return to Learn Tracker identifies the current instructional model of all regular public school districts with three or more schools.
5 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
School & District Management San Francisco School Board Pauses Renaming 44 Schools, Promises to Consult Historians
The renaming of 44 schools in the San Francisco Unified School District is apparently being put on hold after intense blowback.
Greg Keraghosian
1 min read
A pedestrian walks below a sign for Dianne Feinstein Elementary School in San Francisco, on Dec. 17, 2020. The San Francisco Unified School District put the renaming of 44 schools, including Dianne Feinstein Elementary School, on hold after local and national blowback.
A pedestrian walks below a sign for Dianne Feinstein Elementary School in San Francisco, on Dec. 17, 2020. The San Francisco Unified School District put the renaming of 44 schools, including Dianne Feinstein Elementary School, on hold after local and national blowback.<br/><br/>
Jeff Chiu/AP
School & District Management Superintendent Who Led During COVID-19 School Shutdowns Gets Top Honors
Michelle Reid of Washington state's Northshore district, one of the very first to close schools last March, was named National Superintendent of the Year.
3 min read
Michelle Reid, superintendent of the Northshore district in Washington
Michelle Reid, the superintendent of the Northshore district in Washington, was named National Superintendent of the Year.
courtesy of AASA, the School Superintendents Association