States Turn to Student Performance As New Measure of School Quality
Spurred by political and business leaders, states and the federal government are moving rapidly toward what one leading educator calls a "paradigm shift" in the way schools are judged.
Unlike in the past, when officials evaluated schools on whether they had complied with regulations governing "inputs"--such as pupil-teacher ratios, library books, and course offerings--governments are increasingly measuring school quality by how well students are performing.
President Bush and the nation's governors signaled their support for the new view at the Charlottesville, Va., summit in September. In their joint statement, the executives agreed to "establish clear measures of performance and then issue annual report cards on the progress of students, schools, the states, and the federal government."
But even before the summit, accountability was high on the agenda in state capitals throughout the nation. In fact, notes Emerson J. Elliott, acting commissioner of the National Center for Education Statistics, "a healthy majority of states can cite recent policy initiatives that, in one way or another, try to hold schools accountable for the quality of instructional processes and outcomes."
This month alone, for example:
New Jersey officials will send to parents the state's first "report card," detailing how each of the state's schools performed on state high-school proficiency tests, elementary basic-skills tests, the Scholastic Aptitude Test, and the Advanced Placement tests, as well as in student attendance and graduation rates.
The North Carolina Board of Education is expected to issue regulations outlining the criteria districts can use to apply for state funds to implement differential-pay plans for teachers and regulatory flexibility. Under legislation approved this year, districts must meet at least 75 percent of the performance goals they set, using state-determined criteria, to be eligible for the awards.
The Maryland Board of Education is expected to consider a draft plan for implementing the recommendations of its Governor's Commission on School Performance. That panel had proposed establishing an accreditation system that would evaluate schools at least in part on the basis of student performance.
Such activity, suggests Chester E. Finn Jr., professor of education and public policy at Vanderbilt University, reflects a "paradigm shift" in education.
In the near future, he predicts, "the enterprise of education will be defined entirely by actual learning accomplished and accounted for. Indeed, no 'education' will have taken place unless there is evidence learning occurred."
A Premature Shift?
But, because schools continue to rely on what they consider inadequate measures of student performance, critics warn, this shift may be premature. Until more accurate tools are in place, charge these critics--who include Gov. Evan Bayh of Indiana and Albert Shanker, president of the American Federation of Teachers--the accountability efforts could harm education by leading schools to employ the wrong policy prescriptions.
Moreover, contends Lorraine McDonnell, senior political scientist at the rand Corporation, the drive toward accountability threatens to collide head-on with the equally strong move to restructure schools by granting authority at the building level.
"A big issue," she says, "is how to devolve power down to schools and keep accountability to the larger system."
But perhaps the most pressing question, suggests Susan H. Fuhrman, director of the Center for Policy Research in Education, is whether the states' efforts will lead to genuine improvements in student learning. And on that question, she contends, the jury is still out.
States can point to "a lot of winners in all categories of schools,'' she says. "That's improvement. But whether that will lead to the kind of workforce we want remains to be seen."
Return on Investment
The issue of accountability has risen rapidly to become one of the top priorities in state capitals. Some 30 legislatures scheduled the subject for discussion in this year's sessions, according to Chris Pipho, director of the information clearinghouse at the Education Commission of the States.
Such activity has also captured the attention of researchers.
Last month, for example, confereNCES on the issue were held by the U.S. Education Department's office of educational research and improvement and the federally sponsored Center for Research on Evaluation, Standards, and Student Testing at the University of California at Los Angeles's graduate school of education.
But while the interest in accountability is intense, the subject is not altogether new, according to Mr. Pipho.
In the early 1970's, he recalls, some 31 states enacted legislation dealing with accountability, including establishing learning goals, tests to monitor progress toward such goals, "program-planning budgeting systems," and uniform accounting systems.
Those early efforts were generally aimed at applying business techniques to education to improve management efficiency, Mr. Pipho points out. By contrast, he says, more recent proposals are aimed at gauging the results of instruction.
"I'm hearing more legislators saying, 'We've escalated money for reform, how do we know we've gotten anything for it?"' he says.
Such calls reflect the growing involvement of business leaders in education, notes Terry Peterson, executive director of South Carolina's joint business-education subcommittee.
"They have become key actors in school reform in many states," he notes. "As part of that, particularly if there is new money, they want to know what they are getting for their investment."
Likewise, political leaders are also interested in seeing the results of large increases in education spending, suggests Marla Ucelli, education adviser to Gov. Thomas H. Kean of New Jersey. "The citizens of New Jersey have a right to know what we are getting," she says.
In response to such demands, more than half the states have put in place systems that link state actions--rewards or sanctions--with school performance, according to a survey conducted by the Council of Chief State School Officers. Some 26 states, the survey found, withhold funds or intervene in low-performing schools, compared with 19 in 1983, and 29 states, compared with 2 in 1983, reward high-performing schools.
Another 22 states compile and publish lists of how schools perform without attaching consequeNCES to the results.
"A lot of states have moved on many issues, but there aren't many that have figured out how to put the pieces together in ways they are comfortable with," says Michael Cohen, associate director of education programs for the National Governors' Association. "It's easier to reach an agreement that [accountability for performance] is the direction they ought to be going than pulling it off."
Tool for Improvement?
The systems that have been in place so far have generally succeeded in focusing educators' attention on raising the level of student performance, suggests Mr. Peterson of South Carolina, which has one of the most extensive state accountability systems.
In that state, districts that fail to meet state-established criteria on a range of indicators--including student-test scores, student and teacher attendance, dropout rates, and the proportion of teachers teaching out of their field--are listed as "seriously impaired" and must develop plans for improvement. Those that exceed state requirements and show improvement over time are eligible for public recognition, cash awards, and regulatory flexibility.
"People know about it, they think it's important, and they think it's a tool for school improvement," Mr. Peterson says of the state's accountability program.
He notes that a survey of teachers and parents found that most respondents said being judged "impaired" was embarrassing to a district, but that improvements outweighed the negative stigma attached to the label.
In other states, however, accountability systems may not lead to improvement because school officials often ignore the broad array of data on student performance states provide, says Joan L. Herman, associate director of the center for the study of evaluation at the UCLA.
"The use of the information in schools is virtually absent," Ms. Herman says. "We like to talk about the use of data, but people in schools don't pay much attention to it. School people don't have time to worry about the quality of schools. That's a tragedy."
"We know the information that ought to be included on an indicator system," Ms. Herman adds. "That's not going to solve the problem of how to get the information marshalled for improvement."
Virginia Rosen, director of strategic planning for the Dade County (Fla.) Public Schools, agrees that schools seldom use the detailed performance information her district provides. But she adds that the data have proven attractive to local real-estate agents, who use the information to sell houses near high-performing schools.
'Very Powerful Lever'
Other state officials and researchers contend that accountability systems have motivated schools, but in the wrong direction. One piece of data--student test scores--is a "very powerful lever" for changing instruction, notes Ms. McDonnell of the rand Corporation. Because such scores are often published in newspapers--with schools ranked according to performance--school officials often tailor their curriculum and teaching methods to raise scores, she says.
But she cautions that many of those changes have not necessarily been for the better, since traditional standardized tests measure a narrow range of student abilities.
"I have certainly seen schools, many at the low end of the achievement distribution, that have really changed their behavior" in response to accountability pressures, she says. "But many have changed in ways we feel are inappropriate."
Moreover, notes Mr. Elliott of the NCES, citing a well-publicized study that showed 48 states reported scores "above national averages," the reported results are often "deliberately misleading."
"Officials at the state level have known for years that schools sometimes either selectively suppress the lowest individual scores or, even worse, allow answers to fall into students' hands," he says.
Rather than rely on such flawed measures to judge school performance, schools should scrap standardized tests "as they are now," Mr. Shanker of the aft has urged.
Similarly, Governor Bayh of Indiana last month publicly questioned whether his state should continue to use scores on the Indiana Statewide Testing for Educational Progress program to determine whetherschools are eligible for financial incentives and accreditation. He noted that a state panel had urged the state to improve the test to emphasize higher-order skills. (See Education Week, Nov. 1, 1989.)
A Reliable Gauge?
But some local officials counter that, despite their flaws, standardized tests can help school improvement. Floraline I. Stevens, director of evaluation and assessment for the Los Angeles Unified School District, says tests provide a reliable gauge of "whether students are receiving the instruction they need to receive."
"Some teachers had not taught the subject matter, even though they told you they taught it," she points out.
In addition, notes Ruben Carriedo, assistant superintendent for planning, research, and evaluation for the San Diego Unified School District, many minority residents in his district are reluctant to abandon traditional tests.
"Blacks and and Hispanics have put pressure on the district to turn around achievement," he says. "They are suspicious about talk about de-emphasizing test scores. They feel a new approach will eliminate indicators that allow them to make judgments about quality."
To improve the reliability of test data, Mr. Peterson of South Carolina suggests that states use other indicators in addition to test scores to gauge school performance.
"Advocates of more accountability should be willing to say we should not put all our eggs in one basket," he says. "Using multiple indicators helps compensate for inaccuracies and problems with measures."
In addition, Mr. Peterson says, states should focus on measuring and rewarding improvement.
"Our feeling is, you start where you start," he says. "It's not fair to beat you over the head becauseyou're a low-performing school."
Alternatively, notes Jeannie Oakes, associate professor of education at UCLA, officials can adjust performance data to take into account differeNCES in student population.
But, she cautions, states should use care in making such adjustments. "Does that mean you expect less of schools" that have large numbers of traditionally low-performing students, she asks. "You can institutionalize a lower set of expectations by adjusting scores."
Better Data or Brickbats?
Rather than make adjustments to take into account the limitations of standardized tests, states should develop new assessment tools that measure a broader range of student abilities, argues Ramsay W. Selden, director of the state education-assessment center at the Council of Chief State School Officers.
Without such improvements, suggests Larry Gess, director of strategic planning for the Georgia Department of Education, schools that are sanctioned for poor performance could claim in court that they were punished unfairly.
"On those kind of judgments," he says, "you're more likely to get suits."
Some state officials argue that the forthcoming expansion of the National Assessment of Educational Progress will provide states with a more accurate gauge of student performance. Next year, naep will administer a pilot state-level test of 8th-grade mathematics that will permit the first state-by-state comparisons of data on student achievement in a key subject area.
Roy E. Truby, executive staff director of naep's governing board and a former chief state school officer in West Virginia and Idaho, says the expanded naep would help state officials focus on where to make improvements.
"If the results say we score low in science," he says, "I would look at other states that are similar, and say, 'What textbooks are they using? What is their teacher-certification policy?"'
But Judith A. Billings, superintendent of public instruction in Washington State, which is not participating in next year's state-level naep, denies that the test would provide useful data. The test scores, she says, will more likely be used as "brickbats" to scold teachers for poor performance.
"We know far more about how students are doing from our [state] test than from the national assessment," she says. "If all we want is another test score to make comparisons among states, I don't want to hear the word accountability again."
"Information for information's sake is not good enough," she adds. "It has to be used productively."
Mr. Selden of the ccsso suggests that the growing movement toward performance-based assessment could improve the quality of data about student achievement.
In addition to assessing a broader range of student abilities than multiple-choice tests, adds Richard J. Shavelson, dean of the graduate school of education at the University of California at Santa Barbara, such assessments also encourage teachers to use the kind of instruction researchers say will enhance student learning.
"If schools spend three or four weeks a year teaching to a [performance-based] test," he says, "at least they'll be teaching things they ought to be teaching in ways they ought to be teaching it."
But Mr. Elliott of the NCES cautions that such tests are years away from being implemented widely in the schools. Although several states, including California, Connecticut, Michigan, New York, and Vermont, have experimented with performance testing or plan to put such tests in place in the next few years, he notes, "'under development' does not equal 'proved reliable and objective."'
In addition to questioning whether accountability systems are leading to improved performance, critics also warn that the states' efforts could clash with other school-reform strategies.
"The technical difficulties in designing [accountability] systems are significant," Ms. McDonnell of rand says. "But these challenges pale in comparison with the philosophical and political issues accountability raises in public education: Who should be held accountable, to whom, for what? That's always been a difficult question, but school restructuring further complicates the resolution."
Those advocating greater autonomy at the school site, she points out, contend that school-level personnel should be held accountable for their school's performance. But, she says, state "indicator measures influence what schools do. They can act as a constraint on school-based management."
For example, she notes, a school may decide that it wants to de-emphasize science in order to focus on the arts. But if the state accountability system includes a measure of student performance in science, the school may be unable to give up a science course if it does not want to look bad on the state measure.
Roy Forbes, director of the center for school accountability at the University of North Carolina at Greensboro, notes that his state will allow local schools some flexibility by not requiring schools to perform well on every state measure to be eligible for state awards.
"If we're going to make an error, we ought to make an error on the side of giving people more flexibility," he says. "At the present time, the list of things the state requires is very lengthy. Only those districts with a lot of energy can go beyond the list and do things they want to do."
Mr. Selden says states increasingly seem willing to make their accountability systems more responsive to restructuring goals.
"States are setting their instructional sights on goals that represent much deeper, higher-order learning," he says. "They don't want their accountability systems to level downward the expectations of what students learn."