States Need Updates for Managing Data, Analysis Concludes

States will need to make major upgrades to their data-collection systems in order to take full advantage of the mounting wealth of student test results, according to a new analysis from a group tracking state accountability plans.

“The Value of Longitudinal Student Data” is available from the National Center for Educational Accountability.

Just 21 states, for example, are tracking individual student performance with a so-called student identifier that records achievement across a pupil’s K-12 career—a key element in interpreting test scores and using them to improve school policies, according to the National Center for Educational Accountability.

Without such information, state officials aren’t able to figure out what they can do to raise achievement for struggling students, or for others who have similar demographic characteristics, said Chrys Dougherty, the research director for the Austin, Texas, nonprofit group.

“You just can’t interpret [a student’s test scores] unless you know how they were when they came into the school,” Mr. Dougherty said in an interview last week. “It’s absolutely essential if you’re going to do any kind of analysis at the middle and high school level.”

In fact, unless a state has a way to track achievement, student by student, it will struggle to comply with the accountability mandates in the federal No Child Left Behind Act.

“It’s implicitly called for in No Child Left Behind,” said Margaret E. Raymond, the director of education policy and evaluation at the Hoover Institution, an independent think tank at Stanford University. She joined the NCEA advisory board after the study was completed this fall. “It’s the next innovation in student-information systems.”

States are working hard to upgrade data collection, according to Mr. Dougherty and Ms. Raymond. But overall, the two analysts said, states lack student identifiers and many basic ingredients that Mr. Dougherty’s group suggests they should have to get the full value from their test scores.

IDs and More

After surveying state officials and researchers, the National Center for Educational Accountability identified the nine “essential elements” of state data systems.

First on the list is a unique code that identifies each student so that the state can follow enrollment, test scores, and other factors as the student matriculates. The identifiers also help track students as they migrate from school to school within districts or across a state.

Other elements on the list include collecting information on individual student coursetaking, recording scores on SAT, ACT, and Advanced Placement tests, and following student success in higher education. The final piece is to ensure the state has a thorough audit of all the data it collects and receives from districts.

So far, only Texas and Florida have all the pieces of what a NCEA brochure calls the “Nine Essential Elements of Statewide Data-Collection Systems.” Georgia and Louisiana have eight of the nine elements.

But 22 states have three or fewer of the pieces, according to the NCEA’s state-by-state analysis.

States are making progress, however. Since 1999, the NCEA found, 13 states have added student identifiers to their data systems, and others are working to add them.

“The progress is very strong in some states, and very tentative or limited in other states,” Mr. Dougherty said.

States may soon receive federal subsidies to upgrade their databases. The U.S. Senate’s fiscal 2004 appropriations bill would institute an $80 million competitive-grant program to help states with efforts to create data systems needed to comply with the No Child Left Behind Act.

House and Senate negotiators are now working out a compromise on their differing appropriations bills.

States have to be careful in choosing what information they want, one researcher said.

“In the past, people have created real beasts,” said Joan L. Herman, a co-director of the National Center for Research on Evaluation, Standards, and Student Testing, based at the University of California, Los Angeles. “You run the risk of using time and energy collecting data that no one ever uses.”

Keeping Track

The need for student identifiers—and other individual student data—reflects the movement in the school reform debate from tracking schools’ or districts’ progress toward learning goals, to following individual students’ achievement of those goals.

Just for the Kids Inc.—the National Center for Educational Accountability’s parent organization—has used school- level information collected under the Texas testing program to publish detailed analyses of Texas schools’ performance on state tests. The Education Commission of the States and the University of Texas at Austin are the Austin- based Just for the Kids’ partners in the NCEA.

In addition to using data in new ways, states will soon be collecting a huge amount of fresh statistics when complying with the No Child Left Behind Act’s requirement that states assess the reading abilities and math skills of every student from the 3rd through the 8th grade, and once in high school.

All that information will do little good, though, if it isn’t used to make a difference, according to Matt Gandal, the executive vice president of Achieve, a Washington-based group led by governors and corporate executives that promotes increased student achievement.

“This stuff has to come alive and be used in schools and statehouses to make important decisions,” Mr. Gandal said.

For example, Ohio policymakers are starting to reap the benefits of creating a unique code identifying every student. Before, the state could analyze test scores only at the district level, according to Kimberly A. Murnieks, the deputy chief of staff for the Ohio Department of Education.

Now, in addition to providing statistics that help schools make informed decisions about individuals, the student information “sets us up nicely” to comply with the accountability and reporting requirements of the No Child Left Behind Act, Ms. Murnieks said.

Auditing Role

In addition to collecting and reporting data, states should conduct thorough audits of that information, the NCEA suggests.

Such audits should include statistical reviews of district data to flag anything that looks suspicious, as well as random inspections of the data a district collects, the center says.

The recent firestorm in Houston over misreported dropout figures demonstrates the importance of auditing data to ensure the information reflects what’s actually happening in schools, Mr. Dougherty said. (“Houston Case Offers Lesson on Dropouts,” Sept. 24, 2003.)

Ohio’s new individual student-level data give the state another source of information to double-check when it audits the statistics from districts on dropouts and other issues, Ms. Murnieks added.