| Success in whole-school reform, researchers say, depends on how well schools implement the designs. |
If whole-school reforms practiced truth-in-advertising, even the best would carry a warning like this: “Works if implemented. Implementation variable.”
As states and districts embrace the concept of schoolwide change, the degree to which a school carries out the ideas and practices of a particular reform model in the way its designers envisioned has emerged as the weak link.
Research suggests that if they’re well-implemented, some of these designs can produce substantial gains in student achievement. The better the implementation, the bigger the payoff.
But study after study has found that implementation is often problematic and inconsistent, even at school sites that have been identified as exemplars.
“Across every single one of those studies, implementation matters a lot,” says Samuel C. Stringfield, a principal research scientist at Johns Hopkins University in Baltimore, who has carried out numerous studies of whole-school reforms. “Implementation matters regardless of the program.”
But researchers are a long way from a full understanding of the conditions that lead to successful implementation. The next research frontier, Mr. Stringfield suggests, is to figure out which designs work best under which circumstances.
The answers are crucial. Many states and districts are investing heavily in the whole-school concept, and the federal government has made available $150 million in grants for schools to adopt reform designs under the Comprehensive School Reform Demonstration Program.
The question of how to translate education programs from paper into practice is hardly new, however.
“This is a long-standing and intractable problem,” says Ron Anson, a senior analyst at the U.S. Department of Education’s office of educational research and improvement. “People have been trying to figure this out for 20 or 30 years.”
‘Layers of Variability’
As early as the 1970s, researchers discovered that schools didn’t just adopt education designs, they adapted them--sometimes beyond recognition.
“There are many layers of variability that exist when you’re trying to implement a reform program,” Mr. Anson says. Each school has its own set of circumstances: teachers and teaching competence; school climate and culture; district-level values and strategies; and state standards and requirements. “So you kind of march through each set of variables,” Mr. Anson explains, “and at each intersection, you have to find the approach, the strategy, the materials that will work best.”
Rather than thinking about schools simply adopting an existing design, researchers say it might be more productive to think about schools “co-constructing” models that will fit their environments.
That’s especially true as the designs grow more ambitious and the problems surrounding implementation become more complex. Approaches such as the Comer School Development Program or Accelerated Schools do not just alter the reading program in a building; they address everything from parent-teacher relationships to the structure of the school day.
And with such complex prescriptions for improvement come myriad ways for individual schools to alter the dosage.
When Mr. Stringfield conducted the Special Strategies study of federal Title I programs, he recalls, “it quickly became obvious that there were a couple of Success for All schools that were just doing wonderfully. And it became equally obvious that there were some not doing so well.”
The same was true for other popular designs.
Last year, researchers at the RAND Corp., a nonprofit research institute based in Santa Monica, Calif., released a study of 40 schools that had adopted one of eight whole-school designs. The designs were sponsored by New American Schools, a nonprofit company created in 1991 to finance comprehensive reforms.
After two years, the RAND study found only about half the schools were clearly implementing the core elements of a design across the school. Nearly 45 percent were below that level. And four schools were still stuck in the planning stages.
The study found that schools differed greatly in their ability to carry out reforms, design teams varied widely in their capacity to help schools, and districts offered varying levels of support.
Many of the design teams that created the reform models have clear and promising ideas for how they believe a school should look, says Susan J. Bodilly, a senior social scientist at RAND who conducted the study. But, she adds, “I don’t think they have the level of expertise in implementation that would enable schools to do this on a systematically successful basis.”
‘Craft Knowledge’
Researchers and practitioners are beginning to amass a body of knowledge about what it takes to make whole-school reforms work and the barriers that get in the way.
Many of the designers are learning “what the strategies are for making implementation easier, which we haven’t looked at or codified enough,” says Thomas B. Corcoran, the co-director of the Consortium for Policy Research in Education at the University of Pennsylvania in Philadelphia.
Implementation tends to be better, for example, when schools have chosen a design of their own free will, when there’s strong leadership at the school site, and when the designs themselves are clear and specific.
Common barriers to implementation include turnover at the school and district levels, dissension among faculty members, shifts in state and district priorities and funding, and pressure to prepare students for high-stakes tests.
“A lot of times, the teachers didn’t see the designs as well-aligned with the state assessments,” Mr. Corcoran says of one study that looked at whole-school designs in three districts. “So the teachers were intentionally deviating from the designs to do what they thought they had to do to prepare for the tests.”
The problem is that designers are trying to balance two competing demands, says Rebecca Herman, who recently oversaw a rating of whole-school reforms for the Washington-based American Institutes for Research. On the one hand, they want teachers to feel ownership of a design and to be able to shape what it looks like in their school. On the other hand, they want to maintain enough integrity for the design to remain intact.
“There’s no program that you can just take and stick in a school and make it happen the way it is on paper,” Ms. Herman argues. “And sometimes, when you adapt it to the situation, you lose sight of what’s critical to make the program work.”
Researchers at the Learning Research and Development Center at the University of Pittsburgh note that ambitious reforms often suffer from either too much or too little specification. Too much--in the form of codified materials, routines, and teaching strategies--can result in outward compliance with a design while missing the underlying intent. Too little specification can confuse and annoy teachers.
Selecting a Design
Researchers emphasize the importance of the selection process. Schools that carefully identify their needs and freely commit to a design have more success. “If a design is forced upon a school, you have a high probability that it will not go forward,” Ms. Bodilly says.
The RAND study of New American Schools found that the initial match between designs and schools was often rushed. “There are more times than we would like to think when schools either have been told that they have to do this, or at least feel they have been told,” says Thomas K. Glennan Jr., a senior adviser for education policy at RAND.
New Jersey, for example, recently mandated that elementary schools in 28 urban districts adopt one of five whole-school designs, following a state court order.
And Miami-Dade County, Fla., and New York City have, in the past, strong-armed some of their lowest-performing schools to carry out certain designs.
But while choice is important, “without really strong guidance as schools go about making decisions, they don’t make very good ones,” cautions Robert E. Blum, the director of school improvement programs for the Northwest Regional Educational Laboratory, a federally funded center in Portland, Ore. Schools may select a model because it’s popular or convenient, he says, rather than look at the strength of the research or whether the model addresses their needs.
The Perfect Fit
Unfortunately, researchers don’t know as much as they would like about which designs work best under which circumstances.
“One of the things that I think we can evolve as a science over time is which reforms work where, when, and why,” Mr. Stringfield says.
Already, some trends are emerging. Designs that provide teachers with a set of guiding principles but few specifics, for example, may require more capacity on the part of schools than designs that are more prescriptive.
“The latter tend to have a smoother implementation, at least in the first few years,” Ms. Bodilly observes. “They require less teacher time and less teacher expertise.”
“I’m not saying these more philosophically based approaches are wrong,” she adds, “but by their very nature, they’re depending much more on teachers as professionals. And in an urban district that’s very stressed out, it may be beyond their means and their energy level to really go after this.”
Readiness Is Crucial
Some researchers even suggest that the question of whether schools are “ready” for whole-school reform has been overlooked.
The Special Strategies study noted that “strategies cannot be put in place when school administrators and/or faculty are reluctant to change, have no or little expectations that anything will happen, or are poorly managed either at the school or classroom level.” There are conditions, Mr. Stringfield warns, “under which every reform wouldn’t work.”
H. Jerome Freiberg, a professor of education at the University of Houston, agrees. Readiness is critical, he argues. Yet often, “no time is spent on that issue.”
If an organization isn’t ready for change, it won’t, adds Mr. Freiberg, who is the founder of the Consistency Management/Cooperative Discipline Program, based in Houston. His program tries to measure the climate in a school before training begins: first, to identify the school’s needs from differing perspectives and, second, to ensure that everyone is on board.
Ms. Bodilly of RAND suggests that readiness can be built, by phasing in designs or gradually moving schools from more to less prescriptive approaches. “But no one wants to talk about that,” she adds. “They want a design to claim a school and that’s the end of it.”
In some deeply troubled schools, Ms. Bodilly suggests, it may be better to start simple, with less comprehensive programs focused on building the teaching capacity to ensure basic skills. Then, she adds, “when they’ve built the capacity among them, it may be time to go on to something more ambitious.”
Mr. Anson of the Education Department’s OERI agrees. “The real problem,” he says, “is how you get these programs to work in schools that don’t have great principals, teachers, or superintendents and can’t get them.”
‘A Problem of Drift’
As developers gain experience in helping schools with implementation, they’re also confronting the problem of how to sustain it.
“As we get larger numbers of 6-year-old schools, we’re finding some are still absolutely dynamite, and some have gotten into fairly big trouble,” says Robert E. Slavin, the founder of the Success for All program. “There is a problem of drift.”
Mr. Freiberg concurs. “It really is hard work to stay and provide logistical support over multiple years,” he says. “And I don’t think people have done a good job of thinking that through both from the school district’s perspective and from the developer’s perspective.”
Many of the design teams are trying to measure implementation and provide schools with benchmarks on how well they’re doing.
Success for All, for example, uses an implementation checklist that is revised regularly. And there’s a whole process for how to use it and provide schools with feedback.
Mr. Slavin has also begun compiling computerized information about implementation in hundreds of schools, so that schools can benchmark themselves against one another.
Similarly, Expeditionary Learning Outward Bound, a comprehensive school design that engages students in long-term, multidisciplinary projects known as “expeditions,” conducts implementation checks at least once a year. Each spring, participating schools also conduct self-reviews focused on one or two core practices.
“We have a lot of concrete pieces that you can see,” says Greg Farrell, the nonprofit company’s president. “They’re either there or they’re not.”
Examples include the average number of “expeditions” per student per year, the number of teachers who stay with the same students for more than one year, and the number of teachers who collect students’ work in portfolios.
Implementation “is a long, slow accumulation of details and establishment of habits,” Mr. Farrell emphasizes. “It’s like doing the dishes or mowing the lawn. And I think there’s a natural human tendency to rest and relax.”
In Memphis, Tenn., researchers found that schools using whole-school designs posted better achievement gains than schools that did not.
But, says Steve Ross, a professor of educational psychology and research at the University of Memphis, “what everybody forgets is that every one of those Memphis schools got a full, 30-page report at the end of their first year with information about how they were doing, and I’d like to think that had a lot to do with it.”
Mr. Ross is now working with the Appalachia Educational Laboratory in Charleston, W.Va., to put together an “implementation evaluation package” that schools could buy for about $3,000.
It would include two external visits during the year by outside experts, help in setting up specific implementation benchmarks, and a written report based on the data.
Mr. Ross hopes the package will provide schools with a practical, cost-efficient way to evaluate their efforts.
‘Shifting Sand’
Even knowing when a design is fully implemented is not a clear-cut issue. That’s particularly true for less prescriptive programs that rely on schools to work out many of the specifics themselves.
The Oakland, Calif.-based Coalition of Essential Schools, for example, prides itself on “standards without standardization.” Each school is expected to interpret the coalition’s principles and redesign its curriculum, pedagogy, and assessments to fit its own context. But coalition leaders acknowledge that such an intentionally messy strategy is hard for researchers to stomach.
“The word ‘implementing’ is one I dislike because it suggests there’s a solution, it’s fashioned outside, it needs tailoring, but it can be more or less plugged in,” says Joseph P. McDonald, a professor of teaching and learning at New York University.
He believes some of the changes that need to take place in schools are difficult if not impossible to measure, but still very important. “If you’re talking about shifting people’s beliefs, then you’re talking about a level of impact that I equate with [religious] conversions or, on a more mundane scale, with things like overcoming obesity or alcohol dependence,” Mr. McDonald says.
Further muddying the waters, Ms. Bodilly points out, is that many of the designs have evolved over time, so that implementation becomes a moving target. “The very things that they say are being implemented at certain periods of time are changing,” she says, “so we’re sort of on shifting sand here.”
For example, many program designers have backed away from plans to devise their own standards and assessments, and they’ve placed more emphasis on basic skills than they had initially planned.
“You can basically see a maturation of the designs,” Ms. Bodilly says, “but you also see this interaction with the system--a district and its policies--that’s made them retreat, in a sense, from certain stances they originally took.”
‘Experimental Phase’
Meanwhile, the demand for better information on whole-school reform is growing.
Groups such as the American Institutes for Research and the Northwest Regional Educational Laboratory have begun to publish guidebooks to help schools with selection and implementation.
Mr. Stringfield is coordinating an updated study of the Special Strategies schools, some of which have been implementing whole-school reforms for more than a decade. The OERI also is supporting a $10 million research project that will seek to better define what comprehensive reform means, evaluate the effectiveness of different approaches, and try to develop a framework for people to use when they look at such programs.
In addition, the Education Department plans to spend about $4.35 million over three years for a detailed evaluation of the Comprehensive School Reform Demonstration Program.
The study will also explore the effectiveness of specific reform models. But Mary R. Rollefson, a deputy division director in the department’s office of elementary and secondary education, notes that the study’s small sample size may make it difficult to evaluate all of the “brand name” models.
The department also hopes to synthesize the research about whole-school reform across numerous, ongoing studies.
Experts hope such research will begin to fill in the gaps on what is considered one of the most promising innovations in education in years--but an innovation that remains unproven on a large scale.
“It’s kind of like we’re in an experimental phase with whole-school reform,” Mr. Freiberg of the University of Houston says. “We don’t always have all the answers.”
Mr. Stringfield adds a caution. “These reforms are now in a market environment, and developers are marketing their products,” he says. “And, in that environment, the buyer obviously has to be somewhat cautious. What is a mistake is to think you can buy one of these and plug it in.”
