Despite all the recent rhetoric about the importance of using research evidence to guide education policy and practice, the sad reality is that research has had less constructive influence during the past 10 to 15 years than it did over the previous decade or two. The many calls for “evidence based” decisionmaking notwithstanding, most of our major policy and reform initiatives have either been launched without any prior research on their efficacy, and without provisions for assessing their intended and unintended effects, or have ignored or misused whatever relevant research evidence was available when they were launched.
Consider the record of policy initiatives in the first category—those that were launched without a meaningful research base. They include the following:
• High-stakes, test-based accountability systems. These have clearly been the most consequential policy changes of our time. What started in Kentucky and Texas, spread to other states, and morphed into the No Child Left Behind Act at the federal level was never based on supportive research. Moreover, there were no funds in any of the enabling legislation to monitor unintended consequences. The evidence gathered since then (through other means) shows no clear pattern of effects on student achievement—even in tested subjects—and reveals a host of damaging side effects.
The prospects for NCLB’s reauthorization may be cloudy now for various reasons, including failure to demonstrate effectiveness. But it is far from certain that state accountability systems will be altered in fundamental ways, or that whatever replaces No Child Left Behind will itself be grounded in good research.
• Charter schools. Over the past decade, billions of federal and state tax dollars have been invested in creating charter schools, without an initial evidence base and without much accountability thereafter. A large number of studies now show that, overall, charters are no better and possibly worse than “regular” public schools at promoting student achievement. Thus far, these negative or neutral findings have had little effect on charter legislation or funding.
• Voucher programs. Like charters, voucher programs began without an initial evidence base, though a decade earlier. Large amounts of private, politically conservative funding have been invested in vouchers, with the goal of providing parents with greater choice among schools, and thereby creating a more competitive and responsive marketplace. Government money also has been invested: An estimated 60,000 students now participate in publicly funded voucher programs. According to a comprehensive review of the research evidence, achievement gains have been negligible.
• Supplemental educational services. Under the No Child Left Behind law, $1 billion in federal funds annually has been allocated to private tutoring programs, with no initial evidence base to support their value and an actual ban in the enabling legislation on evaluating their effectiveness. Studies funded by other means show these programs to be ineffective.
Examples in the second category—initiatives for which relevant research evidence was available but was ignored, misinterpreted, or misused in policy decisions—include:
• Reading First. From 2002 until this year, $1 billion in federal funding per year has been allocated to improving reading instruction in grades K-3 in high-need schools. The requirements for receiving grant money under this purported “research based” initiative were largely derived from a framework developed by the National Reading Panel, which excluded a great deal of worthwhile reading research by looking only at evidence from randomized trials. Because of this, the panel painted itself into a corner from which it was forced to give inordinate weight to studies that focused primarily on students with learning disabilities. Moreover, the federal officials who were appointed to oversee the Reading First program were, from the start, biased toward (or even financially linked to) certain reading programs and assessments, and away from at least one of the most carefully and comprehensively vetted reading programs.
Here, at least, the initiative was assessed, belatedly, with the use of federal funding. Initial findings, released last May, showed no effect on students’ reading-comprehension capacities—even though students in Reading First schools received on average 10 more minutes per day of reading instruction than students in comparison schools.
• Retention-in-grade initiatives. These policies have spread widely despite a large and long-standing body of evidence showing them to be consistently counterproductive.
• Class-size reduction. Beginning in California in the mid-1990s, such reform efforts have spread to 40 states (and to federal legislation), at a cost of several billion dollars per year. There was prior evidence of effectiveness from a well-conducted study—Tennessee’s Project STAR, which showed significant benefits when class size was reduced by 40 percent to 50 percent. Later studies also showed positive results. But substantial reductions in class size are extremely costly to implement, and often cause dislocations of several kinds, including temporary, if not permanent, shortages of qualified teachers. Simply put, the cost-effectiveness of class-size reduction pales in comparison with the value of less-expensive, less-disruptive reform approaches.
• State reading adoptions. A number of states, California and Texas most notably, view literacy instruction as so important that they create their own detailed specifications for its form and content. They then require publishers to create programs that conform to those specifications, and make districts choose from among the programs—usually basal reading programs—approved by their selection committees. Granted, some state specifications (such as providing instruction in comprehension strategies) can be traced to a body of research, but many others cannot (ensuring a high percentage of “decidable” words in the early grades, for example).
Most important, there is no evidentiary basis for concluding that the combination of elements in these programs will lead to more learning or higher achievement. A case in point: The What Works Clearinghouse recently found that there is insufficient evidence to conclude that either of California’s approved basal programs (Open Court and Houghton Mifflin) is effective. It should be no surprise, then, that those who have observed the states’ specification-setting and selection processes report that ideology rather than research evidence often prevails in how decisions are made.
In addition to the catalogue of sins above, there are many ways that solid research findings could have been used but weren’t. For example, research has amply demonstrated the importance of providing summer learning opportunities to low-income students, of building a sense of belonging or “connectedness” in school, and of providing social and emotional learning programs in school settings. Most states, and certainly the federal government, have been slow to make any of these a high priority.
Where has research played a useful role in recent times? One potentially positive development is represented by the several clearinghouses that have been created to bring evaluation evidence to bear on program-adoption decisions. At the federal level, these include the What Works Clearinghouse, sponsored by the Department of Education; the Model Programs Guide, sponsored by the Justice Department’s office of juvenile justice and delinquency prevention; and the National Registry of Evidence-Based Programs and Practices, sponsored by the Substance Abuse and Mental Health Services Administration. In the private sector, an especially user-friendly resource, Safe and Sound: An Educational Leader’s Guide to Evidence-Based Social and Emotional Learning Programs, has been developed and widely disseminated by the nonprofit Collaborative for Academic, Social, and Emotional Learning.
The challenges and complexities of implementing these clearinghouses have proved to be considerable, including problems of determining how to assess the quality, strength, and general application of research findings, and of translating conclusions into language that prospective program adopters can readily understand. In two yet-to-be-published studies, researchers assessed the influence of these clearinghouses on the adoption rates of “evidence based” drug-prevention programs. The studies’ findings show that the clearinghouses did increase adoption rates, albeit only modestly. The need for the clearinghouses is undeniable, however. It has become nearly impossible for nonresearchers to make informed judgments of their own, as research methods have become ever more sophisticated, and program developers’ claims of “research based” and “research proven” more and more deceptive.
To sum up, research evidence is being ignored or abused much more frequently than it is being used appropriately. Ideology, politics, and vested financial interests often trump substance when it comes to how “rigorous” research is defined and evidence is either privileged or dismissed; how research findings are compiled, interpreted, and disseminated; and how policy initiatives and grants programs are conceived and implemented.
When they are used at all, research findings serve more often as a cloak of respectability for policy decisions made for other reasons, rather than as a genuine guide to action.
