RAND served as an “independent third-party evaluator” throughout the history of New American Schools’ efforts to support the development and dissemination of Comprehensive School Reform models or “designs.” RAND’s evaluation activities were not confined to whether a design “worked” and should continue to be funded - although that was one objective of the work. The nonprofit think tank took a disinterested view of how NAS managed the effort. RAND briefings were also a cornerstone of the several meetings NAS staff and Design Team managers held each year. While the discussion almost always involved whether RAND had its facts right, most of the discussion concerned why the facts were as they were (with caveats), what had happened since RAND collected those facts, and what more needed to be done by teams and NAS to make things better or keep them heading in a positive direction.
As NAS and the Teams shifted from design development to what I called the challenge of “quality at scale” in my first presentation to the NAS board as Director of Design Team Development in 1997, RAND’s evaluation engaged the school districts spawning a new marketplace for school improvement. Up to this point, Designs were certainly not in some end state categorized as “developed”; like all professional services, they were in continuous development. But they were more or less understood as a coherent set of interventions to be implemented in some sequence against a more or less defined criteria. The real challenge was that to date they were in place in a relatively small number of sites, free of charge, with the focused assistance of the designs developers. Now the designs would be implemented in a large number of schools, all over the nation, for a price, with newer staff as well as experienced developers.
The meetings of NAS and Teams were now opened up to the district managers with direct responsibility for implementing the MOUs described yesterday. Once again, RAND’s work not only helped NAS determine which jurisdictions and which teams would continue in the dissemination effort, it gave NAS, districts and teams an unusual opportunity to improve the quality of implementation “in process.” Yes, there was a certain amount of debate over RAND’s facts when NAS, the teams and now district leaders met,; and there was some finger pointing, and examples of “the blame game.” Still, the players recognized their joint responsibility for the effort’s outcomes, their shared interest in its success, and the relationship between their individual work the broader context of district reform.
Whether and where the NAS dissemination efforts led to success is the subject of some debate. My view is that they worked where the school district wanted them to work, and actively supported their implementation. But regardless of this debate, I am quite sure that the model of “practical evaluation” developed in that strategy was essential to whatever level of success was attained, and something any school district attempting systemic reform avoids at its peril.
Next: Practical Evaluation Today.
