Harvard University researcher Daniel M. Koretz has some good news and some bad news for policymakers looking ahead to the reauthorization of the federal No Child Left Behind Act: The nation’s K-12 students are now attending Lake Wobegon schools.
Because of rampantly inflated standardized test scores, Mr. Koretz contended at a panel discussion here last week, all children seem to be above average, as in the fictional town made famous by the radio personality Garrison Keillor—or at least better academically than they actually are.
“We know that we are creating, in many cases, with our test-based accountability system, illusions of progress,” Mr. Koretz told a roomful of policymakers and educators at a Sept. 22 forum hosted by the American Enterprise Institute, a Washington think tank.
Mr. Koretz, a professor of education at Harvard’s graduate school of education who has written a new book, Measuring Up: What Educational Testing Really Tells Us, said that under the No Child Left Behind law, widespread teaching to the test, strategic reallocation of teaching talent, and other means of gaming the high-stakes testing system have conspired to produce scores on state standardized tests that are substantially better than students’ mastery of the material.
“If you tell people that performance on that tested sample is what matters, that’s what they worry about, so you can get inappropriate responses in the classroom and inflated test scores,” he said.
Mr. Koretz pointed to research in the 1990s on the state standardized test then used in Kentucky, which was designed to measure similar aspects of proficiency as the National Assessment of Educational Progress, the federally sponsored testing program often called “the nation’s report card.”
Scores on both tests should have moved more or less in lock step, he said. But instead, 4th grade reading scores rose sharply on the Kentucky Instructional Results Information System test, which resembled an NCLB-test prototype, from 1992 to 1994, while sliding slightly on NAEP over the same period.
“If you’re a parent in Kentucky, what you care about is whether your kids can read, not how well they can do on the state test,” Mr. Koretz said. “And the national assessment told us that, in fact, the gains in the state test were bogus.”
Shortage of Studies
In his book, Mr. Koretz says a similar situation occurred during the “Texas miracle” of the 1990s: Minority students posted large gains on the high-stakes Texas Assessment of Academic Skills, but much smaller ones on NAEP.
Mr. Koretz said the relative dearth to date of comparative studies on large-scale state assessments isn’t for lack of trying. He said he and other scholars have often been rebuffed after approaching officials about the possibility of studying their assessment systems.
“There have not been a lot of studies of this,” Mr. Koretz said, “for the simple reason that it’s politically rather hard to do, to come to a state chief and say, ‘I’d like the chance to see whether your test scores are inflated.’?”
Bella Rosenberg, an education consultant and a longtime special assistant to Albert Shanker, the late president of the American Federation of Teachers, echoed many of Mr. Koretz’s critiques.
“This is officially sanctioned malpractice—if we did this in health, in medical practice, we’d all be dead by now,” she said of the testing system under the NCLB law. “The fact is, we’re doing a great job of obscuring instead of illuminating achievement gaps.”
But fellow panelist Roberto Rodriguez, a senior education adviser to U.S. Sen. Edward M. Kennedy, D-Mass., the chairman of the Senate Health, Education, Labor, and Pensions Committee who helped shepherd the NCLB law through Congress in 2001, cautioned against “throwing the baby out with the bath water.”
While a more effective accountability system “needs to be a goal, I would argue, of the upcoming reauthorization of the [Elementary and Secondary Education Act],” he said, that “summative assessment is a critically important area of accountability.” The No Child Left Behind law is the current version of the ESEA, first enacted in 1965.
On the tug of war between advocates of the current system and detractors who prefer that tests better reflect the course content that students are supposed to be mastering, Mr. Rodriguez said, “I think we can have our cake and eat it too.”
Incentives for Instruction
But Mr. Koretz said some parts of the nearly 7-year-old NCLB law need to be fundamentally reworked.
“I think we ought to mandate, as part of No Child Left Behind, independent evaluations of state accountability programs,” he said. “There is nobody involved in this system who has an incentive to look for good instruction anymore—all the incentives are lined up in one direction: Increase scores on the summative tests at any costs.
“We need to create a system in which somebody … has incentives to make sure we’re not just gaming the system.”
Williamson M. Evers, the U.S. Department of Education’s assistant secretary for planning, evaluation, and policy development, who attended the panel discussion, suggested that any changes to the ESEA law should preserve its essence.
“Some are proposing changes in accountability that remove consequences; this would make it accountability in name only,” Mr. Evers said in an e-mail. “Results should be comparable statewide, or parents, taxpayers, and teachers themselves can’t tell how students are doing.”
Bruce Fuller, a professor of education and public policy at the University of California, Berkeley, praised the points Mr. Koretz makes about testing under NCLB in his book, and suggested steps to address the law’s flaws.
“Washington should first ensure that states use a common benchmark that defines proficient student achievement in reading, writing, and math,” Mr. Fuller, who is the director of the Policy Analysis for California Education research center based at Berkeley and Stanford University, said in an e-mail. “Second, neo-No Child policies should ensure that each state sets a valid way of tracking student growth over time, and provide states the resources to build necessary data systems.”
