Kentucky should postpone rewarding or penalizing schools next fall based on recent state test scores, a new study concludes, because changes to last year’s assessment make conclusions about student-learning trends unreliable.
Instead, the study says, the state should interrupt its third two-year assessment cycle, which is set to conclude this spring, and restart it with 1997 as a baseline year. That action could delay verdicts about school performance until 2000.
As it stands now, schools are to learn next fall if their scores improved enough from administrations of the exam in 1995 and 1996 to those in 1997 and this year. If so, their teachers could earn millions of dollars in bonuses statewide; if not, schools could face sanctions, such as outside intervention.
If lawmakers heed the report’s advice, teachers would have at least a two-year wait to get the money promised them for generating improved performance on an assessment system that has been watched by educators nationwide.
The report’s authors write that they would be “uncomfortable giving schools rewards and sanctions” unless the change was made, but some educators warn that such a shift could sap the program’s credibility and momentum.
While it points to needed improvements in the state assessment, the report, a draft of which Education Week obtained last week, could have concluded that the Kentucky Instructional Results Information System, or KIRIS, be junked altogether. A final version was due out last week.
State education officials view the report as “useful and generally supportive,” Jim Parks, a department spokesman, said.
Inflated Scores?
The Legislative Research Commission, the research arm of the legislature, commissioned the study last year. It was written by a team led by James S. Catterall, an education professor at the University of California, Los Angeles.
The report looks at whether KIRIS is a valid, reliable, and appropriate measure of student achievement and school success. The assessment generates scores for individual students, but imposes accountability on schools.
Some state legislators have been highly critical of KIRIS, created in 1991 as part of a court-ordered overhaul of Kentucky’s education system. The report arrives as the legislature is beginning its biennial session, in which decisions about KIRIS--hailed as cutting-edge and criticized as unreliable--will figure prominently.
One state senator is proposing that KIRIS be replaced by a national standardized test. Observers said last week that the report could be used both by those backing radical changes to the assessment system and those seeking more modest alterations--and all in time to affect the significance of this spring’s testing.
To support its call for a suspension of the current testing cycle, the report explains that shifting the grade levels that took parts of KIRIS last year could throw off the accuracy of year-to-year comparisons. For the first time last year, 5th graders took what had been 4th grade subject tests and 7th graders took tests that had previously been given only to 8th graders--in part to reduce the burden on a given grade level’s students and teachers.
The change in grade levels taking the test “almost certainly” led to inflated scores stemming from students doing better on the tests, the report says.
Robert Linn, an education professor at the University of Colorado at Boulder, noted the concern about inflated scores--especially when compared with Kentucky’s less impressive performance over time on the federally sponsored National Assessment of Educational Progress--is “also true for other states.”
The report recommends that beginning with the testing that took place in the 1996-97 school year, KIRIS maintain consistent test components, testing across grade levels, and test-administration rules.
‘Breaking a Promise’
The Catterall report also calls “unjustifiable” Kentucky’s practice of designating even a high-performing school “in crisis” if it fails, over two years, to improve sufficiently against its own previous scores. It advises state policymakers to review that categorization. But the study applauds the controversial inclusion in 1996-97 of multiple-choice items in KIRIS and urges that they be used to help hold schools accountable for improvements. Those kinds of questions are not now counted.
The report also observes that “KIRIS policymakers seem very resistant to the suggestion to re-establish baseline years” and interrupt the current cycle of accountability reports.
Mr. Parks of the education department acknowledged that department officials would be reluctant to recommend suspending the accountability reports. “We’ve got 40,000 teachers in the state who worked hard last year to improve learning who might get rewards,” he said. “If we suspend [the reports], they’re going to view it as a breach in faith, as the state breaking a promise.”
Mr. Linn and Roger Pankratz, the executive director of the Kentucky Institute for Education Research, an independent, nonprofit organization based in Frankfort, agreed.
Virtually all of the UCLA researchers’ recommendations made sense to him, Mr. Pankratz said, but he warned against losing momentum with abrupt policy shifts. “Sure, make these changes we need to make,” he said, “but use a process of transition so that the efforts people have made don’t get discarded.”
