Opinion
Federal Commentary

Missing in Action

By Eric Schaps — November 04, 2008 7 min read

Despite all the recent rhetoric about the importance of using research evidence to guide education policy and practice, the sad reality is that research has had less constructive influence during the past 10 to 15 years than it did over the previous decade or two. The many calls for “evidence based” decisionmaking notwithstanding, most of our major policy and reform initiatives have either been launched without any prior research on their efficacy, and without provisions for assessing their intended and unintended effects, or have ignored or misused whatever relevant research evidence was available when they were launched.

Consider the record of policy initiatives in the first category—those that were launched without a meaningful research base. They include the following:

High-stakes, test-based accountability systems. These have clearly been the most consequential policy changes of our time. What started in Kentucky and Texas, spread to other states, and morphed into the No Child Left Behind Act at the federal level was never based on supportive research. Moreover, there were no funds in any of the enabling legislation to monitor unintended consequences. The evidence gathered since then (through other means) shows no clear pattern of effects on student achievement—even in tested subjects—and reveals a host of damaging side effects.

The prospects for NCLB’s reauthorization may be cloudy now for various reasons, including failure to demonstrate effectiveness. But it is far from certain that state accountability systems will be altered in fundamental ways, or that whatever replaces No Child Left Behind will itself be grounded in good research.

Charter schools. Over the past decade, billions of federal and state tax dollars have been invested in creating charter schools, without an initial evidence base and without much accountability thereafter. A large number of studies now show that, overall, charters are no better and possibly worse than “regular” public schools at promoting student achievement. Thus far, these negative or neutral findings have had little effect on charter legislation or funding.

Voucher programs. Like charters, voucher programs began without an initial evidence base, though a decade earlier. Large amounts of private, politically conservative funding have been invested in vouchers, with the goal of providing parents with greater choice among schools, and thereby creating a more competitive and responsive marketplace. Government money also has been invested: An estimated 60,000 students now participate in publicly funded voucher programs. According to a comprehensive review of the research evidence, achievement gains have been negligible.

Supplemental educational services. Under the No Child Left Behind law, $1 billion in federal funds annually has been allocated to private tutoring programs, with no initial evidence base to support their value and an actual ban in the enabling legislation on evaluating their effectiveness. Studies funded by other means show these programs to be ineffective.

Examples in the second category—initiatives for which relevant research evidence was available but was ignored, misinterpreted, or misused in policy decisions—include:

Reading First. From 2002 until this year, $1 billion in federal funding per year has been allocated to improving reading instruction in grades K-3 in high-need schools. The requirements for receiving grant money under this purported “research based” initiative were largely derived from a framework developed by the National Reading Panel, which excluded a great deal of worthwhile reading research by looking only at evidence from randomized trials. Because of this, the panel painted itself into a corner from which it was forced to give inordinate weight to studies that focused primarily on students with learning disabilities. Moreover, the federal officials who were appointed to oversee the Reading First program were, from the start, biased toward (or even financially linked to) certain reading programs and assessments, and away from at least one of the most carefully and comprehensively vetted reading programs.

Here, at least, the initiative was assessed, belatedly, with the use of federal funding. Initial findings, released last May, showed no effect on students’ reading-comprehension capacities—even though students in Reading First schools received on average 10 more minutes per day of reading instruction than students in comparison schools.

Retention-in-grade initiatives. These policies have spread widely despite a large and long-standing body of evidence showing them to be consistently counterproductive.

Class-size reduction. Beginning in California in the mid-1990s, such reform efforts have spread to 40 states (and to federal legislation), at a cost of several billion dollars per year. There was prior evidence of effectiveness from a well-conducted study—Tennessee’s Project STAR, which showed significant benefits when class size was reduced by 40 percent to 50 percent. Later studies also showed positive results. But substantial reductions in class size are extremely costly to implement, and often cause dislocations of several kinds, including temporary, if not permanent, shortages of qualified teachers. Simply put, the cost-effectiveness of class-size reduction pales in comparison with the value of less-expensive, less-disruptive reform approaches.

State reading adoptions. A number of states, California and Texas most notably, view literacy instruction as so important that they create their own detailed specifications for its form and content. They then require publishers to create programs that conform to those specifications, and make districts choose from among the programs—usually basal reading programs—approved by their selection committees. Granted, some state specifications (such as providing instruction in comprehension strategies) can be traced to a body of research, but many others cannot (ensuring a high percentage of “decidable” words in the early grades, for example).

Most important, there is no evidentiary basis for concluding that the combination of elements in these programs will lead to more learning or higher achievement. A case in point: The What Works Clearinghouse recently found that there is insufficient evidence to conclude that either of California’s approved basal programs (Open Court and Houghton Mifflin) is effective. It should be no surprise, then, that those who have observed the states’ specification-setting and selection processes report that ideology rather than research evidence often prevails in how decisions are made.

In addition to the catalogue of sins above, there are many ways that solid research findings could have been used but weren’t. For example, research has amply demonstrated the importance of providing summer learning opportunities to low-income students, of building a sense of belonging or “connectedness” in school, and of providing social and emotional learning programs in school settings. Most states, and certainly the federal government, have been slow to make any of these a high priority.

Where has research played a useful role in recent times? One potentially positive development is represented by the several clearinghouses that have been created to bring evaluation evidence to bear on program-adoption decisions. At the federal level, these include the What Works Clearinghouse, sponsored by the Department of Education; the Model Programs Guide, sponsored by the Justice Department’s office of juvenile justice and delinquency prevention; and the National Registry of Evidence-Based Programs and Practices, sponsored by the Substance Abuse and Mental Health Services Administration. In the private sector, an especially user-friendly resource, Safe and Sound: An Educational Leader’s Guide to Evidence-Based Social and Emotional Learning Programs, has been developed and widely disseminated by the nonprofit Collaborative for Academic, Social, and Emotional Learning.

The challenges and complexities of implementing these clearinghouses have proved to be considerable, including problems of determining how to assess the quality, strength, and general application of research findings, and of translating conclusions into language that prospective program adopters can readily understand. In two yet-to-be-published studies, researchers assessed the influence of these clearinghouses on the adoption rates of “evidence based” drug-prevention programs. The studies’ findings show that the clearinghouses did increase adoption rates, albeit only modestly. The need for the clearinghouses is undeniable, however. It has become nearly impossible for nonresearchers to make informed judgments of their own, as research methods have become ever more sophisticated, and program developers’ claims of “research based” and “research proven” more and more deceptive.

To sum up, research evidence is being ignored or abused much more frequently than it is being used appropriately. Ideology, politics, and vested financial interests often trump substance when it comes to how “rigorous” research is defined and evidence is either privileged or dismissed; how research findings are compiled, interpreted, and disseminated; and how policy initiatives and grants programs are conceived and implemented.

When they are used at all, research findings serve more often as a cloak of respectability for policy decisions made for other reasons, rather than as a genuine guide to action.

Related Tags:

A version of this article appeared in the November 05, 2008 edition of Education Week as Missing in Action:

Events

Student Well-Being Webinar Boosting Teacher and Student Motivation During the Pandemic: What It Takes
Join Alyson Klein and her expert guests for practical tips and discussion on how to keep students and teachers motivated as the pandemic drags on.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
A Holistic Approach to Social-Emotional Learning
Register to learn about the components and benefits of holistically implemented SEL.
Content provided by Committee for Children
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
How Principals Can Support Student Well-Being During COVID
Join this webinar for tips on how to support and prioritize student health and well-being during COVID.
Content provided by Unruly Studios

EdWeek Top School Jobs

CCLC Program Site Director
Thornton, CO, US
Adams 12 Five Star Schools
Customer Support Specialist, Tier 1
Portland, OR, US
Northwest Evaluation Association
Customer Support Specialist, Tier 1
Portland, OR, US
Northwest Evaluation Association
Customer Support Specialist, Tier 1
Portland, OR, US
Northwest Evaluation Association

Read Next

Federal Biden Legal Team Steps Back From Trump Stance on Transgender Female Sports Participation
The Education Department's office for civil rights pulls a letter that said Connecticut's transgender-inclusive policy violates Title IX.
4 min read
Bloomfield High School transgender athlete Terry Miller, second from left, wins the final of the 55-meter dash over transgender athlete Andraya Yearwood, far left, and other runners in the Connecticut girls Class S indoor track meet at Hillhouse High School in New Haven, Conn on Feb. 7, 2019. Transgender athletes are getting an ally in the White House next week as they seek to participate as their identified gender in high school and college sports. Attorneys on both sides say they expect President-elect Joe Biden’s Department of Education will switch sides in legal battles that could go a long way in determining whether transgender athletes are treated by the sex on their birth certificates or by how they identify.
Bloomfield High School transgender athlete Terry Miller, second from left, wins over transgender athlete Andraya Yearwood, far left, and other runners in an event in New Haven, Conn. The two transgender athletes are at the center of a legal fight in Connecticut over the participation of transgender female athletes in girls' or women's sports.
Pat Eaton-Robb/AP
Federal Congress Again Tries to Pass Eagles Act, Focused on School Shootings After Parkland
A group of bipartisan Congressional lawmakers is once again trying to get a law passed aimed at preventing school violence.
Devoun Cetoute & Carli Teproff
2 min read
Suzanne Devine Clark, an art teacher at Deerfield Beach Elementary School, places painted stones at a memorial outside Marjory Stoneman Douglas High School on Feb. 14, 2019 during the first anniversary of the school shooting in Parkland, Fla.
Suzanne Devine Clark, an art teacher at Deerfield Beach Elementary School, places painted stones at a memorial outside Marjory Stoneman Douglas High School on Feb. 14, 2019 during the first anniversary of the school shooting in Parkland, Fla.
Wilfredo Lee/AP
Federal Some Districts Extend Paid Leave Policies as They Hope for Passage of Biden Relief Plan
With federal provisions having expired, some school employees have had to dip into their own banks of leave for COVID-19 purposes.
5 min read
Linda Davila-Macal, a seventh grade reading teacher at BL Garza Middle School in Edinburg, Texas, works from her virtual classroom at her home on Aug. 31, 2020.
A teacher leads a virtual classroom from her home.
Delcia Lopez/The Monitor via AP
Federal President Biden Is Walking a 'Careful Tightrope' When It Comes to School Reopenings
CDC guidance and confusion over his rhetoric turn up the pressure, and could overshadow progress in schools and nuanced public opinion.
9 min read
President Joe Biden answers questions during a televised town hall event at Pabst Theater in Milwaukee on Feb. 16, 2021.
President Joe Biden answers questions during a televised town hall event in Milwaukee earlier this month.
Evan Vucci/AP