North Carolina has devoted insufficient time, money, and staff to its testing program over the past few years—shortcomings that, if not properly addressed, could continue to jeopardize its overall reliability and credibility, a panel of testing experts told the state school board last week.
Read the report, “Independent Audit Panel Review of North Carolina Testing and Accountability Issues,”, from North Carolina public schools. (Requires Adobe’s Acrobat Reader.)
While the panel’s audit praised the state’s accountability program for raising student achievement, it warned that without clearly documented testing policies, the state may have difficulty defending itself against lawsuits that could result as the stakes for individual students rise.
“In spite of the fact that North Carolina has a good program, [the state] is trying to do more at this point than it is willing to devote the time and staff to,” said Mark D. Musick, the president of the Southern Regional Education Board in Atlanta, which oversaw the independent audit. “The testing program has grown, the responsibilities of the testing staff have grown, and the stakes have grown, but the money and the staffing haven’t kept pace.”
The report urges policymakers to yield their demand for expediency whenever it threatens to compromise the complex technical operations of statewide assessment systems.
Louis M. Fabrizio, the director of accountability services for the North Carolina Department of Public Instruction, said, “I was happy that the report pointed out that developing valid and reliable tests for a statewide testing program is a much more complicated and involved and time-consuming process than most folks understand.”
Problems in the state’s testing program arose last spring when local and state education officials discovered that the scores used to determine whether students demonstrated proficiency in mathematics were set too low. As a result, far more elementary and middle school students passed the exam than expected, and fewer 5th graders were held back as the state began to phase in tough new promotion requirements. (“Testing Glitch Prompts N. Carolina to Order System Audit,”, May 30, 2001.)
Changes Made Hastily
The tests were designed to reflect a more rigorous math curriculum that had been phased in over the previous two years. But experts believe the sweeping adjustments were made too quickly, in response to tight deadlines set by the state board. Similar criticisms have been leveled at testing programs in other states as well.
And the task stretched North Carolina’s testing staff beyond reasonable limits, the panel said. The process of field-testing new questions for the exams, for example, was rushed, and not enough time was built into the grading process to check for problems with the cutoff scores used to determine proficiency levels.
State legislators on both sides of the aisle have shown unprecedented support for the state’s 5-year-old accountability program. As a result, teachers and staff members involved in improving schools have shared as much $140 million each year in bonus money. But the $12 million testing program has lacked the necessary monetary and technical support, according to the audit panel.
The state education department has 10 employees to oversee all aspects of crafting and administering the testing program. The program includes annual tests in math and reading in grades 3 through 8, a writing assessment in 4th, 7th, and 10th grades, a computer-skills exam for 8th graders, and a high school exit test.
The testing staff also oversees a high school competency test and alternative assessments for students with special needs, and helps create and field-test new test items.
The state board had asked the legislature to expand the payroll for the division for seven new positions in its 2002-2003 biennial budget. But under the constraints of a nearly $1 billion budget shortfall, the division instead lost one position.
State education officials agreed with some of the panel’s recommendations. State board Chairman Philip J. Kirk Jr. acknowledged that he and his colleagues on the board, particularly those experienced in business management, may have pressed for speed over quality.
“It seems to at least the business members of the state board that it takes too long to get tests validated and in use,” he said. “However, this study has convinced us that it is better to go a little slower and get it right than to proceed and take some shortcuts.”
Critics Not Appeased
The report, which was commissioned by the state board, did little to quell critics of the accountability plan. The questions undertaken by the audit panel may be beside the point, according to John Hood, the president of the John Locke Foundation, a conservative think tank in Raleigh. Mr. Hood has argued that the current testing program offers a narrow view of achievement among the state’s 1.3 million students.
“Our annual testing data are useful only to the extent that they compare us to ourselves,” said Mr. Hood. He has advocated the use of a nationally normed test, which would measure North Carolina students against those throughout the country. “We don’t get a good sense if the yardstick we’re using is the right length.”
