Opinion
Federal Opinion

Missing in Action

By Eric Schaps — November 04, 2008 7 min read
  • Save to favorites
  • Print

Despite all the recent rhetoric about the importance of using research evidence to guide education policy and practice, the sad reality is that research has had less constructive influence during the past 10 to 15 years than it did over the previous decade or two. The many calls for “evidence based” decisionmaking notwithstanding, most of our major policy and reform initiatives have either been launched without any prior research on their efficacy, and without provisions for assessing their intended and unintended effects, or have ignored or misused whatever relevant research evidence was available when they were launched.

Consider the record of policy initiatives in the first category—those that were launched without a meaningful research base. They include the following:

High-stakes, test-based accountability systems. These have clearly been the most consequential policy changes of our time. What started in Kentucky and Texas, spread to other states, and morphed into the No Child Left Behind Act at the federal level was never based on supportive research. Moreover, there were no funds in any of the enabling legislation to monitor unintended consequences. The evidence gathered since then (through other means) shows no clear pattern of effects on student achievement—even in tested subjects—and reveals a host of damaging side effects.

The prospects for NCLB’s reauthorization may be cloudy now for various reasons, including failure to demonstrate effectiveness. But it is far from certain that state accountability systems will be altered in fundamental ways, or that whatever replaces No Child Left Behind will itself be grounded in good research.

Charter schools. Over the past decade, billions of federal and state tax dollars have been invested in creating charter schools, without an initial evidence base and without much accountability thereafter. A large number of studies now show that, overall, charters are no better and possibly worse than “regular” public schools at promoting student achievement. Thus far, these negative or neutral findings have had little effect on charter legislation or funding.

Voucher programs. Like charters, voucher programs began without an initial evidence base, though a decade earlier. Large amounts of private, politically conservative funding have been invested in vouchers, with the goal of providing parents with greater choice among schools, and thereby creating a more competitive and responsive marketplace. Government money also has been invested: An estimated 60,000 students now participate in publicly funded voucher programs. According to a comprehensive review of the research evidence, achievement gains have been negligible.

Supplemental educational services. Under the No Child Left Behind law, $1 billion in federal funds annually has been allocated to private tutoring programs, with no initial evidence base to support their value and an actual ban in the enabling legislation on evaluating their effectiveness. Studies funded by other means show these programs to be ineffective.

Examples in the second category—initiatives for which relevant research evidence was available but was ignored, misinterpreted, or misused in policy decisions—include:

Reading First. From 2002 until this year, $1 billion in federal funding per year has been allocated to improving reading instruction in grades K-3 in high-need schools. The requirements for receiving grant money under this purported “research based” initiative were largely derived from a framework developed by the National Reading Panel, which excluded a great deal of worthwhile reading research by looking only at evidence from randomized trials. Because of this, the panel painted itself into a corner from which it was forced to give inordinate weight to studies that focused primarily on students with learning disabilities. Moreover, the federal officials who were appointed to oversee the Reading First program were, from the start, biased toward (or even financially linked to) certain reading programs and assessments, and away from at least one of the most carefully and comprehensively vetted reading programs.

Here, at least, the initiative was assessed, belatedly, with the use of federal funding. Initial findings, released last May, showed no effect on students’ reading-comprehension capacities—even though students in Reading First schools received on average 10 more minutes per day of reading instruction than students in comparison schools.

Retention-in-grade initiatives. These policies have spread widely despite a large and long-standing body of evidence showing them to be consistently counterproductive.

Class-size reduction. Beginning in California in the mid-1990s, such reform efforts have spread to 40 states (and to federal legislation), at a cost of several billion dollars per year. There was prior evidence of effectiveness from a well-conducted study—Tennessee’s Project STAR, which showed significant benefits when class size was reduced by 40 percent to 50 percent. Later studies also showed positive results. But substantial reductions in class size are extremely costly to implement, and often cause dislocations of several kinds, including temporary, if not permanent, shortages of qualified teachers. Simply put, the cost-effectiveness of class-size reduction pales in comparison with the value of less-expensive, less-disruptive reform approaches.

State reading adoptions. A number of states, California and Texas most notably, view literacy instruction as so important that they create their own detailed specifications for its form and content. They then require publishers to create programs that conform to those specifications, and make districts choose from among the programs—usually basal reading programs—approved by their selection committees. Granted, some state specifications (such as providing instruction in comprehension strategies) can be traced to a body of research, but many others cannot (ensuring a high percentage of “decidable” words in the early grades, for example).

Most important, there is no evidentiary basis for concluding that the combination of elements in these programs will lead to more learning or higher achievement. A case in point: The What Works Clearinghouse recently found that there is insufficient evidence to conclude that either of California’s approved basal programs (Open Court and Houghton Mifflin) is effective. It should be no surprise, then, that those who have observed the states’ specification-setting and selection processes report that ideology rather than research evidence often prevails in how decisions are made.

In addition to the catalogue of sins above, there are many ways that solid research findings could have been used but weren’t. For example, research has amply demonstrated the importance of providing summer learning opportunities to low-income students, of building a sense of belonging or “connectedness” in school, and of providing social and emotional learning programs in school settings. Most states, and certainly the federal government, have been slow to make any of these a high priority.

Where has research played a useful role in recent times? One potentially positive development is represented by the several clearinghouses that have been created to bring evaluation evidence to bear on program-adoption decisions. At the federal level, these include the What Works Clearinghouse, sponsored by the Department of Education; the Model Programs Guide, sponsored by the Justice Department’s office of juvenile justice and delinquency prevention; and the National Registry of Evidence-Based Programs and Practices, sponsored by the Substance Abuse and Mental Health Services Administration. In the private sector, an especially user-friendly resource, Safe and Sound: An Educational Leader’s Guide to Evidence-Based Social and Emotional Learning Programs, has been developed and widely disseminated by the nonprofit Collaborative for Academic, Social, and Emotional Learning.

The challenges and complexities of implementing these clearinghouses have proved to be considerable, including problems of determining how to assess the quality, strength, and general application of research findings, and of translating conclusions into language that prospective program adopters can readily understand. In two yet-to-be-published studies, researchers assessed the influence of these clearinghouses on the adoption rates of “evidence based” drug-prevention programs. The studies’ findings show that the clearinghouses did increase adoption rates, albeit only modestly. The need for the clearinghouses is undeniable, however. It has become nearly impossible for nonresearchers to make informed judgments of their own, as research methods have become ever more sophisticated, and program developers’ claims of “research based” and “research proven” more and more deceptive.

To sum up, research evidence is being ignored or abused much more frequently than it is being used appropriately. Ideology, politics, and vested financial interests often trump substance when it comes to how “rigorous” research is defined and evidence is either privileged or dismissed; how research findings are compiled, interpreted, and disseminated; and how policy initiatives and grants programs are conceived and implemented.

When they are used at all, research findings serve more often as a cloak of respectability for policy decisions made for other reasons, rather than as a genuine guide to action.

Related Tags:

A version of this article appeared in the November 05, 2008 edition of Education Week as Missing in Action:

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Reading & Literacy Webinar
The Future of the Science of Reading
Join us for a discussion on the future of the Science of Reading and how to support every student’s path to literacy.
Content provided by HMH
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
College & Workforce Readiness Webinar
From Classrooms to Careers: How Schools and Districts Can Prepare Students for a Changing Workforce
Real careers start in school. Learn how Alton High built student-centered, job-aligned pathways.
Content provided by TNTP
Student Well-Being Live Online Discussion A Seat at the Table: The Power of Emotion Regulation to Drive K-12 Academic Performance and Wellbeing
Wish you could handle emotions better? Learn practical strategies with researcher Marc Brackett and host Peter DeWitt.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal No 'Gender Ideology': Ed. Dept.'s New Focus for Mental Health Grants It Yanked
The Trump administration abruptly canceled $1 billion in mental health grants in April that it said reflected Biden-era priorities.
5 min read
Amelia, 16, sits for a portrait in a park near her home in Illinois on Friday, March 24, 2023. “We are so strong and we go through so, so much," says the teenage girl who loves to sing and wants to be a surgeon. Amelia has also faced bullying, toxic friendships, and menacing threats from a boy at school who said she “deserved to be raped."
The U.S. Department of Education has revealed new priorities for two mental health grants after it abruptly canceled awards the Biden administration made.
Erin Hooley/AP
Federal Trump Admin. Starts Moving CTE to Labor Dept. After Supreme Court Order
The Education Department put arrangements to move some of its programs on hold while court battles over downsizing played out.
4 min read
Students make measurements to wood to add to a tiny home project during their shop class at Carrick High School in Pittsburgh, Pa., on Dec. 13, 2022.
Students make measurements to wood to add to a tiny home project during their shop class at Carrick High School in Pittsburgh, Pa., on Dec. 13, 2022. The Trump administration is shifting management of career and technical education programs to the U.S. Department of Labor now that the Supreme Court have given the go-ahead to proceed with downsizing of the U.S. Department of Education.
Nate Smallwood for Education Week
Federal Hope Shattered for Laid-Off Ed. Dept. Staff After Supreme Court Order
The Supreme Court on Monday allowed the Trump administration to proceed with 1,400 Education Department layoffs.
6 min read
Supporters hold signs and cheer Education Department employees as they leave after retrieving their personal belongings from the Education Department building in Washington on March 24, 2025.
Supporters hold signs and cheer Education Department employees as they leave after retrieving their personal belongings from the Education Department building in Washington on March 24, 2025. The Supreme Court on July 14, 2025, allowed the Trump administration to proceed with department layoffs that a lower-court judge had put on hold.
Jose Luis Magana/AP
Federal Trump Admin. Says Undocumented Students Can't Attend Head Start, Early College
The administration issued notices saying undocumented immigrants don't qualify for Head Start and some Education Department programs.
7 min read
Children play during aftercare for the Head Start program at Easterseals South Florida, an organization that gets about a third of its funding from the federal government, on Jan. 29, 2025, in Miami.
Children play during aftercare for the Head Start program at Easterseals South Florida, an organization that gets about a third of its funding from the federal government, on Jan. 29, 2025, in Miami. The Trump administration said Thursday that undocumented children are ineligible for Head Start and a number of other federally funded programs that the administration is classifying as similar to welfare benefits.
Rebecca Blackwell/AP