States and districts are increasingly using test scores to reward or punish schools, but a study out of Chicago cautions that norm-referenced, standardized tests are not appropriate for such purposes.
The study by the Consortium on Chicago School Research, a federation of local groups, focused on Chicago’s use of the Iowa Tests of Basic Skills to identify low-performing schools. It concluded that using the testing system for those purposes raises significant problems for several reasons:
- The ITBS is actually a system of tests, with different versions given in different grades and in different years. When researchers gave the same students different forms of the exam, their scores--reported in terms of grade equivalents--varied. That, they say, suggests that the testing system should not be used to measure students’ progress over time.
- The content assessed by the ITBS has also changed over the past 10 years, so the standards against which schools and students are being measured are a “moving target.”
- Chicago identifies schools as low-achieving if fewer than 15 percent of their students are reading at or above grade level on the ITBS. But the study argues that schools could meet that target simply by improving the scores for a small group of students who cluster just below the national norm. Meanwhile, schools that significantly improved learning for a large group of very low-achieving students could still fail to meet the cutoff.
“If there’s an overall theme of this study,” said Anthony S. Bryk, the senior director of the consortium and the lead author of the report, “it’s that it’s very hard to do accountability when you’re using these kinds of tests because they simply weren’t designed for those purposes.”
John Oswald, the president of Riverside Publishing, which produces the ITBS, said last week that he had not seen the study. But he said different versions of the test are statistically equivalent and could be used to track gains in achievement. While the tests’ content may change over time, he said, “the overall curriculum areas that we’re providing scores on are consistent.”
Substantial Gains
The good news for Chicago is that after using statistical techniques to make test content more comparable from year to year, math and reading scores have improved substantially over the past decade in many elementary schools, the study found.
The consortium recommends that, for accountability purposes, Chicago replace the ITBS with a test that would be directly linked to the city’s academic standards.
The researchers also suggest holding schools accountable for the learning gains that their students make over time, rather than for meeting a cutoff score. By focusing on such a “value added” approach, the researchers suggest, Chicago could avoid penalizing schools whose students have improved substantially but started at a low point.
Mr. Bryk emphasized that standardized assessments of basic skills are an important measure. “These tests serve a useful purpose of telling us how our kids are doing relative to a national sample,” he said. “They’re problematic when we want to use the same data to make judgments about the increasing productivity of schools or teachers or principals.”
Standards-Based Tests
Paul G. Vallas, the chief executive officer of the 420,000-student district, said last week that “we agree with the consortium 100 percent that we need to have a standards-based assessment system, and that’s the direction in which we’re moving.”
But he said it would take at least three years to develop such a system. Though the reliance on the ITBS for accountability will diminish over time, he added, the tests remain the primary instrument for now. “The Iowa will remain as our nationally norm-referenced test.”
