Opinion
Federal Opinion

Missing in Action

By Eric Schaps — November 04, 2008 7 min read
  • Save to favorites
  • Print

Despite all the recent rhetoric about the importance of using research evidence to guide education policy and practice, the sad reality is that research has had less constructive influence during the past 10 to 15 years than it did over the previous decade or two. The many calls for “evidence based” decisionmaking notwithstanding, most of our major policy and reform initiatives have either been launched without any prior research on their efficacy, and without provisions for assessing their intended and unintended effects, or have ignored or misused whatever relevant research evidence was available when they were launched.

Consider the record of policy initiatives in the first category—those that were launched without a meaningful research base. They include the following:

High-stakes, test-based accountability systems. These have clearly been the most consequential policy changes of our time. What started in Kentucky and Texas, spread to other states, and morphed into the No Child Left Behind Act at the federal level was never based on supportive research. Moreover, there were no funds in any of the enabling legislation to monitor unintended consequences. The evidence gathered since then (through other means) shows no clear pattern of effects on student achievement—even in tested subjects—and reveals a host of damaging side effects.

The prospects for NCLB’s reauthorization may be cloudy now for various reasons, including failure to demonstrate effectiveness. But it is far from certain that state accountability systems will be altered in fundamental ways, or that whatever replaces No Child Left Behind will itself be grounded in good research.

Charter schools. Over the past decade, billions of federal and state tax dollars have been invested in creating charter schools, without an initial evidence base and without much accountability thereafter. A large number of studies now show that, overall, charters are no better and possibly worse than “regular” public schools at promoting student achievement. Thus far, these negative or neutral findings have had little effect on charter legislation or funding.

Voucher programs. Like charters, voucher programs began without an initial evidence base, though a decade earlier. Large amounts of private, politically conservative funding have been invested in vouchers, with the goal of providing parents with greater choice among schools, and thereby creating a more competitive and responsive marketplace. Government money also has been invested: An estimated 60,000 students now participate in publicly funded voucher programs. According to a comprehensive review of the research evidence, achievement gains have been negligible.

Supplemental educational services. Under the No Child Left Behind law, $1 billion in federal funds annually has been allocated to private tutoring programs, with no initial evidence base to support their value and an actual ban in the enabling legislation on evaluating their effectiveness. Studies funded by other means show these programs to be ineffective.

Examples in the second category—initiatives for which relevant research evidence was available but was ignored, misinterpreted, or misused in policy decisions—include:

Reading First. From 2002 until this year, $1 billion in federal funding per year has been allocated to improving reading instruction in grades K-3 in high-need schools. The requirements for receiving grant money under this purported “research based” initiative were largely derived from a framework developed by the National Reading Panel, which excluded a great deal of worthwhile reading research by looking only at evidence from randomized trials. Because of this, the panel painted itself into a corner from which it was forced to give inordinate weight to studies that focused primarily on students with learning disabilities. Moreover, the federal officials who were appointed to oversee the Reading First program were, from the start, biased toward (or even financially linked to) certain reading programs and assessments, and away from at least one of the most carefully and comprehensively vetted reading programs.

Here, at least, the initiative was assessed, belatedly, with the use of federal funding. Initial findings, released last May, showed no effect on students’ reading-comprehension capacities—even though students in Reading First schools received on average 10 more minutes per day of reading instruction than students in comparison schools.

Retention-in-grade initiatives. These policies have spread widely despite a large and long-standing body of evidence showing them to be consistently counterproductive.

Class-size reduction. Beginning in California in the mid-1990s, such reform efforts have spread to 40 states (and to federal legislation), at a cost of several billion dollars per year. There was prior evidence of effectiveness from a well-conducted study—Tennessee’s Project STAR, which showed significant benefits when class size was reduced by 40 percent to 50 percent. Later studies also showed positive results. But substantial reductions in class size are extremely costly to implement, and often cause dislocations of several kinds, including temporary, if not permanent, shortages of qualified teachers. Simply put, the cost-effectiveness of class-size reduction pales in comparison with the value of less-expensive, less-disruptive reform approaches.

State reading adoptions. A number of states, California and Texas most notably, view literacy instruction as so important that they create their own detailed specifications for its form and content. They then require publishers to create programs that conform to those specifications, and make districts choose from among the programs—usually basal reading programs—approved by their selection committees. Granted, some state specifications (such as providing instruction in comprehension strategies) can be traced to a body of research, but many others cannot (ensuring a high percentage of “decidable” words in the early grades, for example).

Most important, there is no evidentiary basis for concluding that the combination of elements in these programs will lead to more learning or higher achievement. A case in point: The What Works Clearinghouse recently found that there is insufficient evidence to conclude that either of California’s approved basal programs (Open Court and Houghton Mifflin) is effective. It should be no surprise, then, that those who have observed the states’ specification-setting and selection processes report that ideology rather than research evidence often prevails in how decisions are made.

In addition to the catalogue of sins above, there are many ways that solid research findings could have been used but weren’t. For example, research has amply demonstrated the importance of providing summer learning opportunities to low-income students, of building a sense of belonging or “connectedness” in school, and of providing social and emotional learning programs in school settings. Most states, and certainly the federal government, have been slow to make any of these a high priority.

Where has research played a useful role in recent times? One potentially positive development is represented by the several clearinghouses that have been created to bring evaluation evidence to bear on program-adoption decisions. At the federal level, these include the What Works Clearinghouse, sponsored by the Department of Education; the Model Programs Guide, sponsored by the Justice Department’s office of juvenile justice and delinquency prevention; and the National Registry of Evidence-Based Programs and Practices, sponsored by the Substance Abuse and Mental Health Services Administration. In the private sector, an especially user-friendly resource, Safe and Sound: An Educational Leader’s Guide to Evidence-Based Social and Emotional Learning Programs, has been developed and widely disseminated by the nonprofit Collaborative for Academic, Social, and Emotional Learning.

The challenges and complexities of implementing these clearinghouses have proved to be considerable, including problems of determining how to assess the quality, strength, and general application of research findings, and of translating conclusions into language that prospective program adopters can readily understand. In two yet-to-be-published studies, researchers assessed the influence of these clearinghouses on the adoption rates of “evidence based” drug-prevention programs. The studies’ findings show that the clearinghouses did increase adoption rates, albeit only modestly. The need for the clearinghouses is undeniable, however. It has become nearly impossible for nonresearchers to make informed judgments of their own, as research methods have become ever more sophisticated, and program developers’ claims of “research based” and “research proven” more and more deceptive.

To sum up, research evidence is being ignored or abused much more frequently than it is being used appropriately. Ideology, politics, and vested financial interests often trump substance when it comes to how “rigorous” research is defined and evidence is either privileged or dismissed; how research findings are compiled, interpreted, and disseminated; and how policy initiatives and grants programs are conceived and implemented.

When they are used at all, research findings serve more often as a cloak of respectability for policy decisions made for other reasons, rather than as a genuine guide to action.

Related Tags:

A version of this article appeared in the November 05, 2008 edition of Education Week as Missing in Action:

Events

Jobs Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
Ed-Tech Policy Webinar Artificial Intelligence in Practice: Building a Roadmap for AI Use in Schools
AI in education: game-changer or classroom chaos? Join our webinar & learn how to navigate this evolving tech responsibly.
Education Webinar Developing and Executing Impactful Research Campaigns to Fuel Your Ed Marketing Strategy 
Develop impactful research campaigns to fuel your marketing. Join the EdWeek Research Center for a webinar with actionable take-aways for companies who sell to K-12 districts.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal What Would Happen to K-12 in a 2nd Trump Term? A Detailed Policy Agenda Offers Clues
A conservative policy agenda could offer the clearest view yet of K-12 education in a second Trump term.
8 min read
Republican presidential candidate and former President Donald Trump speaks at a campaign rally, March 9, 2024, in Rome Ga.
Former President Donald Trump speaks at a campaign rally, March 9, 2024, in Rome, Ga. Allies of the former president have assembled a detailed policy agenda for every corner of the federal government with the idea that it would be ready for a conservative president to use at the start of a new term next year.
Mike Stewart/AP
Federal Opinion Reading Scores Have Plunged Since the Pandemic. What This Senator Wants to Do About That
How can the nation support literacy efforts? The ranking Republican on the Senate education committee wants to hear from educators.
6 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
Federal Biden Calls for Teacher Pay Raises, Expanded Pre-K in State of the Union
President Joe Biden highlighted a number of his education priorities in a high-stakes speech as he seeks a second term.
5 min read
President Joe Biden delivers the State of the Union address to a joint session of Congress at the Capitol on March 7, 2024, in Washington.
President Joe Biden delivers the State of the Union address to a joint session of Congress at the Capitol on March 7, 2024, in Washington.
Shawn Thew/Pool via AP
Federal Low-Performing Schools Are Left to Languish by Districts and States, Watchdog Finds
Fewer than half of district plans for improving struggling schools meet bare minimum requirements.
11 min read
A group of silhouettes looks across a grid with a public school on the other side.
Vanessa Solis/Education Week via Canva