A second round of research on the latest federal reading-test results suggests that Kentucky’s 1998 scores represented a statistically significant increase over 1994 after all, the U.S. Department of Education announced last week.
By comparing Kentucky 4th graders excluded from the 1998 national assessment with similar students who had taken the state’s own reading test, a department-subsidized researcher concluded that Kentucky’s gains were as large as originally reported, according to a report from the National Center for Education Statistics.
The impact of excluding students with disabilities from last year’s National Assessment of Educational Progress reading test was “modest,” writes Lauress L. Wise, the president of the Human Resources Research Organization, an Alexandria, Va., nonprofit group that conducts education and training research.
Even in the worst-case scenario, the rise in Kentucky’s scores from 1994 to 1998 was “clearly significant,” Mr. Wise writes in the research commissioned by the NCES, a branch of the Education Department.
When the NCES released state-by-state NAEP scores earlier this year, it said Kentucky’s 4th grade scores rose a statistically significant amount. After critics pointed out that the state had excluded higher proportions of disabled students in 1998 than in 1994, the NCES hired researchers to probe whether higher exclusion rates in Kentucky and elsewhere had tainted the states’ assessment results.
An initial review by the Educational Testing Service, which runs the assessment under contract with the Education Department, speculated that scores in Kentucky and Maryland may have benefited from higher exclusion rates. (“Board Won’t Revise State NAEP Scores,” May 19, 1999.)
In response to a request from Kentucky officials, the NCES hired Mr. Wise, who conducts research for the state, to mine data from the former Kentucky testing program to see if he could estimate what would have been the NAEP scores of the 4th graders who were held out of the 1998 exam.
The NCES chose to review Kentucky’s scores because Mr. Wise had access to individual student scores, said Peggy G. Carr, NCES’ associate commissioner in charge of assessment.
While the research did not produce “an exact one-for-one match,” it comes as close as possible to assigning NAEP scores to students who didn’t take the national reading tests, Mr. Wise said.
Apples and Oranges?
But the study paints too rosy a picture of disabled students’ scores, according to the critic who first questioned Kentucky’s gains.
“The basic, fundamental premise is absolutely out to lunch,” charged Richard G. Innes, an airline pilot and a persistent critic of the state’s education policies. “What [Mr. Wise] did is purely and simply compare apples to oranges.”
Mr. Innes estimates that three-fourths of the students with disabilities who were excluded from the national test received such accommodations as having questions read to them when they took the state test. Such extra help, which is not allowed on NAEP, skewed the state results upward, he argues.
“It’s a great way for states to jimmy their NAEP scores,” said Mr. Innes, who lives in Villa Hills, Ky., a suburb of Cincinnati.
Mr. Wise said he didn’t question the results of Kentucky’s test, but tried only to understand how they might have translated to NAEP’s scales.
Gary W. Phillips, the acting commissioner of the NCES, said. the federal agency would not underwrite any other research on the impact of excluding disabled students on last year’s NAEP results.
Instead, the NCES will examine what policy changes it might make in reporting scores, according to Ms. Carr, who oversees the NAEP program.
