As a champion for urban education, the Council of the Great City Schools has been working overtime in recent months to get the word out: Public schools in the nation’s cities are on the upswing, and rising test scores are part of the proof.
The latest effort to drive home the point came last week with the council’s release of “Signs of Progress: Preliminary Evidence of Urban School Comeback,” a nine-page compendium of encouraging statistics from 41 city districts.
“We have never seen progress across the board occurring in so many districts at once,” Michael D. Casserly, the council’s executive director, said last week in releasing the report in Washington. “The test scores are pretty hard and solid data.”
Yet while council officials fervently believe that urban schools are turning the corner, proving their case to an often skeptical press and public is no easy feat. Although many individual cities show signs of improvement, demonstrating that it all adds up to a rebound in urban achievement is more problematic.
Owing to a dearth of testing data that are comparable across districts in different states, establishing that students in America’s cities are in fact performing better is a tricky proposition.
“It’s hard to get data to generalize across the urban districts,” said Margaret E. Goertz, the co-director of the Consortium for Policy Research in Education, a federally financed research group based in Philadelphia. “Nobody has looked at it systematically.”
Nonetheless, some outside observers agree with the council that the signs of urban progress are promising. The council’s report follows news from such cities as Atlanta, Boston, Chicago, Houston, New York, Philadelphia, and San Francisco that test scores are up, in some cases for several years running.
“What I see happening is a real determination--not in all but in many districts--to turn performance around, and a readiness to take strong measures to get it,” said Marc S. Tucker, the president of the National Center on Education and the Economy, a Washington-based group focusing on standards-based education.
“The real problem,” Tucker added, “is we have no way to gather the uniform data across districts that would allow us to be more confident about what we are seeing.”
Pressure Is On
Although experts caution against relying too heavily on test scores, there is no denying their importance to educators, politicians, and the public. If anything, the focus on tests is intensifying as more states and districts adopt new academic standards and devise systems aimed at holding schools, educators, and students accountable for meeting them.
In urban schools, the pressure is even greater given the alarming disparity in performance between city students and their nonurban peers.
But with all the attention devoted to big-city test scores, educators, researchers, and the public are often left wondering what they really mean.
Finding data that measure student performance over time in cities within a single state is often confounding enough. Such challenges are only compounded when trying to compare test results in urban districts nationwide.
“If a parent in Atlanta wants to compare their kid with a kid in Miami-Dade, there’s no way,” said Edward D. Roeber, the director of student-assessment programs for the Council of Chief State School Officers. “Within a state you can find quite a bit of comparable data. But there is not much comparable across states.”
The National Assessment of Educational Progress, a federal program that regularly tests samples of 4th, 8th, and 12th graders in key subjects, is generally considered the most reliable national barometer of student achievement.
With rare exceptions, however, the results are available only at the state and national levels, not for individual districts. Although some large cities could receive district-level NAEP results, in most cases they have chosen not to do so because of the cost.
That there is a wide performance gap between urban and nonurban students on NAEP was recently documented by a first-ever analysis conducted for Quality Counts ‘98: The Urban Challenge, a report on city schools published in January by Education Week. But that project analyzed scores only from single administrations of the tests, and thus offered no insight into national trends in urban performance.
In the absence of district-by-district NAEP scores, commercially available standardized tests offer another means of trying to put individual cities’ scores in a national context. But the way in which those tests are used severely limits their value as a tool for comparing cities across state lines or evaluating long-term trends.
Apples to Oranges
For starters, districts administer a smorgasbord of such assessments. The Iowa Tests of Basic Skills, the Stanford Achievement Test, the Metropolitan Achievement Test, the Comprehensive Tests of Basic Skills, and the California Achievement Test are among the better known, but by no means the only such measures.
And even if two districts administer the same basic tests, they may not give them in the same grade, in the same subjects, or at the same time of the school year.
What’s more, they may be using versions of the tests that are of varying vintage. One district may be administering a version with norms established a decade ago, while another may be giving one with benchmarks set in the past few years. The older the norms, the greater the likelihood that the typical student’s score will be deemed above average.
Another complicating factor is the propensity of districts and states to change their tests on a regular basis.
Most educators and experts say it makes sense to update testing programs to reflect evolving standards and to guard against teachers’ and students’ becoming too familiar with the exams. But the upshot is that urban districts--especially those that switch test publishers--often have only a year or two of comparable testing data. Both Boston and Philadelphia, for example, reported gains last year on the Ninth Edition of the Stanford Achievement Test. But both cities had started administering it only the previous year.
An additional obstacle to comparing districts across states are differences in the types of students who get tested. Policies vary from place to place on exempting certain kinds of special education students, for example, and whether they are included in the aggregate scores reported to the public. The same holds true for children who are not yet fluent in English. Whether and how many of such students are included may affect how a district’s scores stack up to national norms.
“There’s no one correct way to handle these exclusion questions,” said Daniel Koretz, a senior social scientist at the RAND Corp. in Santa Monica, Calif.
Reporting Methods Vary
Still another problem is variations in how districts report their scores.
In San Francisco, for example, which has seen five years of improvements on the CTBS, officials report districtwide results based on the scores of students who have been in the system long enough to have taken the tests at least once before.
The idea is that if schools are going to be held accountable for results from one year to the next, fairness dictates excluding the scores of test-takers who haven’t had a year’s worth of instruction in the district.
So last year the district counted scores of some 31,800 students out of the 40,300 who were tested systemwide in grades 1-11. Based on that method, the district reported a systemwide average score last year of 50.8 in reading and of 55.3 in math, compared with national norms of 50 on a 99-point scale. If the full 40,300 scores had been counted, the figures would have dipped slightly, to 49.3 in reading and 53.9 in math.
Some testing experts praise San Francisco’s reporting method as a useful and sophisticated way of assessing its programs’ effectiveness. Nonetheless, experts say, it raises potential problems when comparing scores there with those of the original national sample of students used to set the norms.
“For national norms to be accurate, participating schools that are reporting their scores in terms of norms have to try to mimic the inclusion policies of the norming sample,” Mr. Koretz explained. “You have to be reasonably close to the samples that the norms are based on.”
San Francisco Superintendent Waldemar Rojas, who is also the chairman of the Council of the Great City Schools, a membership organization of nearly 50 large urban districts, said his district is careful to explain its reporting system to the public. He said the method should be a model for other districts seeking to hold schools accountable for performance. At the same time, he lamented the lack of data comparing cities across states.
“We’ve had public education for over 100 years,” he said. “It would be nice to tell the public what the performance levels are.”
Cities’ Gains Cited
To help show that those performance levels are improving in urban schools, the Council of the Great City Schools asked its member districts to provide whatever evidence they had of progress in student achievement.
Of the 41 districts that responded, 18 cited higher scores on “national, state, and/or local student-achievement tests,” over at least a one-year period, the council reported last week.
Beyond test scores, 22 of the districts cited falling dropout rates or rising student attendance. Other indicators of progress included higher scores on college-entrance and Advanced Placement exams, as well as greater numbers of students taking both kinds of tests.
Mr. Casserly said his group would welcome assessments that could provide performance data across districts over time. One sign of that openness, he said, was the willingness of many of his member districts to participate in the voluntary national reading and math tests that the Clinton administration has proposed.
In the meantime, Mr. Casserly said, the absence of such a standardized achievement measure by no means disproves the existence of a upward trend in city schools. “I hate to think that somehow people would not believe there was progress in urban schools until they had definitive comparable data,” he said. “When you’ve got this many districts improving, that I think is pretty convincing.”
