Roger Pankratz, the director of the Kentucky Institute for Education Research, says the longevity of the reforms has allowed time for study.
--Lonnie Harp
In 1989, the Kentucky Supreme Court handed the state’s lawmakers and educators a once-in-a-lifetime opportunity. Start over, the justices said. Build a better school system.
The next year, legislators cobbled together a plan from what was then considered the best of education’s cutting-edge ideas. They called for student-performance measures that included projects, portfolios, and tests with open-ended questions. They set new academic standards and goals and added rewards and sanctions to hold schools to the new targets.
Schools were also asked to create multiage classes in the primary grades, give teachers more on-the-job training, and set up preschool programs for poor 4-year-olds. In turn, they received more authority and more-equitable funding.
At that time, a few states were tinkering with some of those strategies. But no other state was trying all of them at once.
All of that activity has transformed a poor, mostly rural state with lagging schools into a living laboratory for state-of-the-art educational innovations. Researchers from around the country--as well as from within the state--headed for Kentucky to examine everything from preschool programs to teacher training initiatives.
In addition to its grand scale, Kentucky’s reform effort offers researchers the added bonus of longevity. Despite some fine-tuning, the major pieces have largely been in place for seven years. “Most statewide efforts don’t last more than three years,” says Roger Pankratz, the executive director of the Kentucky Institute for Education Research, based in Frankfort.
As a result, school administrators and policymakers have amassed an impressive stack of studies on the state’s progress. The research has provided insights on a wide range of issues--school finance, accountability, pay incentives for teachers, and strategies for improving failing schools, to name a few.
As more states have sought to create comprehensive and integrated approaches to education policy, many have looked to Kentucky for guidance.
But, seven years and more than 500 studies into the program, researchers still can’t say for sure whether Kentucky’s reforms have made a real difference in the classroom.
“We don’t have great information on how kids’ learning is different,” said Jane Clark Lindle, an associate professor of education at the University of Kentucky and a co-director of the University of Kentucky/University of Louisville Joint Center for the Study of Educational Policy. “And that’s the critical issue.”
Part of the problem is that the very thing that has made Kentucky attractive to researchers has also made it difficult to study effectively: A lot is going on at once. In the same schools where educators are flexing newfound autonomy, students grapple with unfamiliar tests and the use of portfolios. And, in a state where all the schools are changing, it’s hard to find control groups for comparison.
“The disadvantage from a research point of view is trying to tease out which factors were responsible for the changes that occurred,” observed Connie Bridge, a professor of education at North Central College in Naperville, Ill. “Was it the assessment, or were teachers getting more staff development?”
Early Findings
Researchers drew a few conclusions about the reforms early on. For example, studies show that the new funding system has successfully halved the gap between Kentucky’s highest-spending districts and its lowest-spending.
Surveys also show that most schools have elected governing councils of educators and parents to decide everything from which textbooks to buy to who will be the principal--decisions once left to superintendents and school boards.
The state has fared less well in its effort to establish mixed-age classrooms in grades K-3, a recommendation that is strongly encouraged by regulations governing the reforms but is no longer mandated. Surveys show that only 10 percent of teachers group children from three or more age levels in their classrooms; most group children from just two grades. And a smattering of studies shows that the numbers of teachers assigned to mixed-age classrooms decreased further once the mandate was removed.
As for student achievement, the picture is mixed. Scores on the new assessment have for the most part risen since 1992-93, but most of the movement has come at the lower end of the scoring scale. The state has had a tougher time nudging students to reach “proficient” and “distinguished” levels of performance. And elementary students tend to outperform their high school and middle school counterparts.
| Over the years, some investigators have questioned whether the gains in the early years were real. | 
|
The picture changed somewhat this school year. The latest round of test results shows gains at all levels of schooling in all subjects. And, in the two areas with the biggest gains--reading and mathematics--fewer students scored at the “novice” level and more reached the proficient and distinguished levels.
Still, over the years, some investigators and educators have questioned whether the gains in the early years of the program were real.
Assessment Scrutinized
As the linchpin of the reform effort, the state testing program--known as the Kentucky Instructional Results Information System, or KIRIS--was designed to be far-reaching. The idea was to develop tests worth teaching to. Thus, rather than focus on basic skills as many standardized tests do, Kentucky’s assessments emphasize writing, hands-on tasks, and open-ended test questions.
Students also compile portfolios of their best work in math and writing, though currently only the writing portfolios are scored.
To ensure that the tests would drive changes in classroom practice, lawmakers built in a system that rewards schools that improve and penalizes those where scores either remain steady or decline. A “distinguished educator” is assigned to evaluate and assist each school in the latter category, and the entire staff is placed on probation.
The high stakes attached to KIRIS have made it one of the most heavily scrutinized pieces of Kentucky’s reforms. “People are interested in what happens in Kentucky’s accountability system even if they never set foot in Kentucky,” said Dan Koretz, a Washington-based researcher with the RAND Corp. who has studied the testing system.
Over the years, several researchers have criticized technical aspects of the program, such as the validity of the scores on writing portfolios. And they have questioned why yearly gains on the state tests were not reflected in some other tests, such as the ACT college-entrance exam.
But a more recent study by Anthony Nitko, a University of Pittsburgh researcher, concludes that the open-ended questions on the state tests are, for the most part, measuring what they’re supposed to: how well schools cover the 37 academic expectations set by the state. Many commercially available standardized tests do not.
A team of University of Kentucky researchers led by education professor Joseph Petrosko also studied 19 elementary schools and found that schools with high KIRIS scores tended to be those that had also put in place the reforms called for by the 1990 law.
|
| The law requires districts to provide teachers with at least four days of in-service training. |
Such positive findings, which have come out only in the past year, still may not be enough to sway teachers, who report in survey after survey that they do not trust the test results. Many believe students are scoring higher because they are becoming more familiar with the tests.
Emphasis on Writing
There is less doubt, on the other hand, that the tests have changed the mix of activities that take place in the classroom.
Ms. Bridge, of North Central College, studied instructional practices at four Kentucky elementary schools in 1982. She returned to the same schools in 1995, five years into the reforms, to see whether teaching had changed.
She found that in the grades she studied--1st, 3rd, 4th, and 5th--teachers were doing 1.2 to two times as much writing with their students as they did before the reforms. Students were writing expository essays, newspaper articles, stories, paragraphs, and essay responses of the type called for in the state testing system.
“If you think about doubling the amount of time that teachers spend [on] writing, I think that’s pretty significant,” Ms. Bridge said recently. A handful of other studies echo her findings.
But teachers also complain that all the writing, rewriting, and conferring over writing involved in the new approaches eats up too much classroom time. “Teachers in Kentucky felt they had to give up important material,” said Mr. Koretz, who surveyed 500 Kentucky teachers and principals during the 1994-95 school year.
What they have sacrificed, teachers typically report in surveys, is instruction in basic computational and language skills.
To test whether students were indeed losing out on those skills, the state hired the Human Resources Research Organization, a Radcliff, Ky.-based group, to compare examples of student writing from 1993 and 1996. The researchers analyzed the samples for spelling, capitalization, punctuation, and subject-verb agreement.
“The long story is that 4th grade girls’ performance essentially remained flat, and the boys improved,” said Gene Hoffman, the manager of the group’s center for learning, evaluation, and assessment research.
Success With Preschool
The 1990 law called upon districts to set up preschool programs for at-risk 4-year-olds and for children ages 3 and 4 with disabilities.
A year later, researchers began tracking groups of children who entered the program and comparing them with other youngsters who were eligible for preschool but whose parents decided against enrolling them. The sample now includes about 3,000 children, the oldest of whom are in 5th grade, according to Mary Louise Hemmerter, an associate education professor at the University of Kentucky who is working on that project.
“What we are finding is that these children make progress,” she said. “In kindergarten, teachers rate them as prepared as children from higher-income families and more prepared than children who could’ve gone through the preschool programs but did not.”
Over time, the study also found, the former preschool pupils kept pace with their better-off peers. And those students’ grades did not drop off in 3rd grade--the point at which studies of other preschool programs suggest that achievement gains made in the early years start to fade.
Training Focus Studied
In the area of professional development for teachers, another bulwark of the reform program, the research evidence is more mixed. The law requires districts to submit their plans for professional development to the state and to provide teachers with at least four days of in-service training. Schools also received extra money to pay for the added training.
But Thomas Corcoran, a co-director of the Consortium for Policy Research in Education, a national organization based at the University of Pennsylvania, said schools have not made the wisest use of those opportunities. He and two colleagues studied 20 schools noted for doing “interesting things” in professional development. They reviewed those schools’ plans and talked to heads of their professional-development committees.
“The focus in most places was on the short term,” Mr. Corcoran said. “It was, ‘What can we do with the money we have and the four days we have to jack up our KIRIS scores?’” As a result, activities tended to focus on pedagogical techniques, such as cooperative learning, and new assessment strategies. Little attention was paid to deepening teachers’ understanding of the subjects they teach.
On the other hand, he added, “the good news is that there’s more professional-development activity and more of it is schoolwide, and there’s really an effort to get some practical payoff out of it.”
“Even though the definitive word on KERA’s success or failure is yet to come, no one in Kentucky has advocated throwing out the reforms.” Roger Pankratz, |
|
The reform package also called upon schools to establish extended programs to offer help to struggling students before and after school and during the summer. Schools with a high proportion of poor students were also required to set up resource centers where families could receive help obtaining eyeglasses or medical referrals for their children. And studies on those programs suggest they are up and running and doing their jobs.
Broader Research Needed
While such research on individual provisions of the reform law is useful, Ms. Lindle of the University of Kentucky says, overall the studies on its impact have been mostly small, fragmented, and not nearly as numerous as researchers and state officials might have hoped. “Here we’ve got a massive reform system and no real window into it,” she said.
Research gained a higher profile after Wilmer S. Cody became the state’s second appointed superintendent in 1995. He named William White last year to be the first research manager for the education department since it was re-created in 1990. The department is also devising a peer-review process to raise the quality of the studies it pays for.
Mr. White says the time has come to shift the focus away from studies that look just at the reform program’s “inputs.”
“We are at the point that most of the pieces are in place, and we need to devote more time to evaluating the effectiveness of programs,” he said. “For example, we know a lot about what kinds of technology we have in place and where, but we don’t know a lot about how technology is actually being used in the classroom by our teachers.” The department is in the process of commissioning studies to answer that question.
In a similar vein, the Kentucky Institute for Education Research, which Mr. Pankratz heads, is re-examining its mission. The organization, with support from the Annie E. Casey Foundation, has bankrolled much of the research on the reform program to date. The institute’s funding is scheduled to run out before the close of the century, prompting its leaders to review what role it will play in future research.
Even though the definitive word on KERA’s success or failure is yet to come, Mr. Pankratz says, no one in Kentucky has advocated throwing out the reforms.
“I think we have been effective in trying to convince legislators and others to improve what we have rather than starting over,” he said. “Whether we’re going to be fortunate enough to learn from our mistakes and successes--that’s not been answered.”
