Federal

Studies of Popular Reading Texts Don’t Meet Reviewers’ Rigor Test

By Kathleen Kennedy Manzo — August 13, 2008 4 min read
  • Save to favorites
  • Print

Two well-known commercial reading programs, which have been adopted by some of the nation’s largest school districts and have met the strict requirements for research-based programs under the federal Reading First initiative, failed to earn ratings from the What Works Clearinghouse because they do not have any studies that satisfy the agency’s rigorous evidence standards.

Reports on Open Court Reading and Reading Mastery, both highly structured texts published by the Columbus, Ohio-based SRA McGraw-Hill, were released Tuesday by the clearinghouse, a program of the Institute of Education Sciences, the research arm of the U.S. Department of Education, to vet effectiveness studies on educational programs and practices.

“At this time, the WWC is unable to draw any conclusions based on research about the effectiveness or ineffectiveness” of either program, the reports say.

The publisher of the programs holds a large share of the elementary school textbook market for reading, largely on the popularity of Open Court, which is used throughout the 708,000-student Los Angeles Unified School District and is one of just two programs approved for use in K-2 reading instruction in California.

Although Open Court has been criticized in the past for not having the kind of independent, scientific evidence that federal officials called for in launching Reading First in 2002, Reading Mastery is perceived as having significant proof of its effectiveness.

Officials with the National Institute for Direct Instruction, based in Eugene, Ore., have complained to the IES, saying the review failed to consider all the available research on Reading Mastery going back decades, and mistakenly rejected more recent studies that fit the criteria.

“They have an arbitrary time limit of 1985 when most of the seminal research on Direct Instruction was conducted prior to that, including Project Follow Through, which is the largest study in the history of the United States,” said Kurt E. Engelmann, the president of the institute. Some of the “studies” that were included, he said, are nothing more than two-page promotional pieces, with charts, circulated by the publisher. “The review is a clear misrepresentation. If the full research base were included, it would receive a higher rating, even with WWC emphasis on randomized trials, which are terribly difficult to do.”

SRA/McGraw-Hill officials argue that both programs are based on solid research and have led to significant improvements in student proficiency in districts large and small.

“SRA/McGraw-Hill’s Open Court Reading and Reading Mastery programs have been developed from nearly 50 years of intense study and field-testing. Where these scientifically researched programs have been implemented, students’ reading test scores increased, and teachers and principals have seen gains in student achievement overall,” company spokesman Tom Stanton wrote in an e-mail.

Why Now?

In the past, so few studies have met the criteria outlined by the clearinghouse, and reviews have found few or “potentially positive” effects of programs, that the agency has been dubbed the “Nothing Works” clearinghouse by critics.

In a review of beginning reading last year, for example, the clearinghouse concluded that few comprehensive or supplemental programs have proof that they work. Many observers also questioned the value of a review coming five years after the federal push for research-based instruction and the demand that schools taking part in the $1 billion-a-year Reading First use programs that are effective. (“Federal Reading Review Overlooks Popular Texts,” Aug. 29, 2007.)

“What is most interesting is the question of why they took until August 2008, to do this. Open Court was among the top three basals adopted under Reading First,” said Robert E. Slavin, the founder of Success for All, which earned a somewhat positive rating from the clearinghouse last year. His program is not widely used in Reading First schools, and Mr. Slavin’s complaints that those schools were discouraged from adopting Success for All led to congressional reviews of Reading First’s implementation in 2005.

The clearinghouse, created in 2002, got off to a slow start. It released its first reviews in 2004 and then another batch two years later, but it did not include evaluations of reading programs.

“If there is no evidence on Reading Mastery or Open Court now, there was no evidence in 2003, 2004, 2005, 2006, or 2007, when Reading First schools were being virtually required to use these programs or the other three basals that the WWC has not yet reviewed,” he said.

The Education Department had issued several statements to state officials that it did not prescribe any particular commercial programs for use in Reading First schools. Subsequent investigations indicated that federal consultants for the program appeared to have steered states and districts to specific reading programs.

The clearinghouse identified 61 studies of Reading Mastery between 1985 and 2007, 15 of which used the required quasi-experimental design. Those studies, however, were deemed inadequate either because the comparison group did not meet the review requirements or other factors made it impossible to link changes in student achievement to use of the program.

The Open Court review looked at 30 studies, but 15 did not have the type of quasi-experimental design—either using randomized trials or a comparison group—the clearinghouse requires. In two of the seven that met the criteria, there was insufficient evidence that the control group was comparable to the treatment group, and the other five did not demonstrate that changes in achievement were attributable to the program.

A version of this article appeared in the August 27, 2008 edition of Education Week as Studies of Popular Reading Texts Don’t Meet Reviewers’ Rigor Test

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Start Strong With Solid SEL Implementation: Success Strategies for the New School Year
Join Satchel Pulse to learn why implementing a solid SEL program at the beginning of the year will deliver maximum impact to your students.
Content provided by Satchel Pulse
Teaching Live Online Discussion Seat at the Table: How Can We Help Students Feel Connected to School?
Get strategies for your struggles with student engagement. Bring questions for our expert panel. Help students recover the joy of learning.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Science Webinar
Real-World Problem Solving: How Invention Education Drives Student Learning
Hear from student inventors and K-12 teachers about how invention education enhances learning, opens minds, and preps students for the future.
Content provided by The Lemelson Foundation

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal The Senate Gun Bill: What It Would Mean for School Safety, Mental Health Efforts
Details of a bipartisan Senate agreement on guns outline additional funding to support student mental health programs.
6 min read
Protesters take to the streets of downtown Detroit June 11 to call for new gun laws. One holds up a sign that says "policy and change."
Protesters call for new gun laws in Detroit's March for Our Lives event earlier this month.
KT Kanazawich for Education Week
Federal What Educators Need to Know About Senators' Bipartisan Deal on Guns, School Safety
In addition to gun restrictions, a tentative compromise would also fund mental health and school safety programs—but it faces hurdles.
4 min read
Protesters hold up a sign that shows the outline of a rifle struck through with a yellow line at a demonstration in support of stronger gun laws.
Protesters gather for the March For Our Lives rally in Detroit, among the demonstrations against gun violence held on the heels of recent mass shootings in Buffalo, N.Y., and Uvalde, Texas.
KT Kanazawich for Education Week
Federal Senate Negotiators Announce a Deal on Guns, Breaking Logjam
The agreement offers modest gun curbs and bolstered efforts to improve school safety and mental health programs.
5 min read
Sen. Chris Murphy, D-Conn., speaks during a rally near Capitol Hill in Washington, Friday, June 10, 2022, urging Congress to pass gun legislation. (AP Photo/Susan Walsh)
Federal Education Secretary: 'Let's Transform Our Appreciation of Teachers to Action'
Miguel Cardona shared strategies to help recruit, develop, and retain effective teachers.
5 min read
Education Secretary Miguel Cardona speaks during the 2022 National and State Teachers of the Year event in the East Room of the White House in Washington, Wednesday, April 27, 2022.
Education Secretary Miguel Cardona speaks during the 2022 National and State Teachers of the Year event in the White House on April 27.
Susan Walsh/AP