Federal

Studies of Popular Reading Texts Don’t Meet Reviewers’ Rigor Test

By Kathleen Kennedy Manzo — August 13, 2008 4 min read
  • Save to favorites
  • Print

Two well-known commercial reading programs, which have been adopted by some of the nation’s largest school districts and have met the strict requirements for research-based programs under the federal Reading First initiative, failed to earn ratings from the What Works Clearinghouse because they do not have any studies that satisfy the agency’s rigorous evidence standards.

Reports on Open Court Reading and Reading Mastery, both highly structured texts published by the Columbus, Ohio-based SRA McGraw-Hill, were released Tuesday by the clearinghouse, a program of the Institute of Education Sciences, the research arm of the U.S. Department of Education, to vet effectiveness studies on educational programs and practices.

“At this time, the WWC is unable to draw any conclusions based on research about the effectiveness or ineffectiveness” of either program, the reports say.

The publisher of the programs holds a large share of the elementary school textbook market for reading, largely on the popularity of Open Court, which is used throughout the 708,000-student Los Angeles Unified School District and is one of just two programs approved for use in K-2 reading instruction in California.

Although Open Court has been criticized in the past for not having the kind of independent, scientific evidence that federal officials called for in launching Reading First in 2002, Reading Mastery is perceived as having significant proof of its effectiveness.

Officials with the National Institute for Direct Instruction, based in Eugene, Ore., have complained to the IES, saying the review failed to consider all the available research on Reading Mastery going back decades, and mistakenly rejected more recent studies that fit the criteria.

“They have an arbitrary time limit of 1985 when most of the seminal research on Direct Instruction was conducted prior to that, including Project Follow Through, which is the largest study in the history of the United States,” said Kurt E. Engelmann, the president of the institute. Some of the “studies” that were included, he said, are nothing more than two-page promotional pieces, with charts, circulated by the publisher. “The review is a clear misrepresentation. If the full research base were included, it would receive a higher rating, even with WWC emphasis on randomized trials, which are terribly difficult to do.”

SRA/McGraw-Hill officials argue that both programs are based on solid research and have led to significant improvements in student proficiency in districts large and small.

“SRA/McGraw-Hill’s Open Court Reading and Reading Mastery programs have been developed from nearly 50 years of intense study and field-testing. Where these scientifically researched programs have been implemented, students’ reading test scores increased, and teachers and principals have seen gains in student achievement overall,” company spokesman Tom Stanton wrote in an e-mail.

Why Now?

In the past, so few studies have met the criteria outlined by the clearinghouse, and reviews have found few or “potentially positive” effects of programs, that the agency has been dubbed the “Nothing Works” clearinghouse by critics.

In a review of beginning reading last year, for example, the clearinghouse concluded that few comprehensive or supplemental programs have proof that they work. Many observers also questioned the value of a review coming five years after the federal push for research-based instruction and the demand that schools taking part in the $1 billion-a-year Reading First use programs that are effective. (“Federal Reading Review Overlooks Popular Texts,” Aug. 29, 2007.)

“What is most interesting is the question of why they took until August 2008, to do this. Open Court was among the top three basals adopted under Reading First,” said Robert E. Slavin, the founder of Success for All, which earned a somewhat positive rating from the clearinghouse last year. His program is not widely used in Reading First schools, and Mr. Slavin’s complaints that those schools were discouraged from adopting Success for All led to congressional reviews of Reading First’s implementation in 2005.

The clearinghouse, created in 2002, got off to a slow start. It released its first reviews in 2004 and then another batch two years later, but it did not include evaluations of reading programs.

“If there is no evidence on Reading Mastery or Open Court now, there was no evidence in 2003, 2004, 2005, 2006, or 2007, when Reading First schools were being virtually required to use these programs or the other three basals that the WWC has not yet reviewed,” he said.

The Education Department had issued several statements to state officials that it did not prescribe any particular commercial programs for use in Reading First schools. Subsequent investigations indicated that federal consultants for the program appeared to have steered states and districts to specific reading programs.

The clearinghouse identified 61 studies of Reading Mastery between 1985 and 2007, 15 of which used the required quasi-experimental design. Those studies, however, were deemed inadequate either because the comparison group did not meet the review requirements or other factors made it impossible to link changes in student achievement to use of the program.

The Open Court review looked at 30 studies, but 15 did not have the type of quasi-experimental design—either using randomized trials or a comparison group—the clearinghouse requires. In two of the seven that met the criteria, there was insufficient evidence that the control group was comparable to the treatment group, and the other five did not demonstrate that changes in achievement were attributable to the program.

A version of this article appeared in the August 27, 2008 edition of Education Week as Studies of Popular Reading Texts Don’t Meet Reviewers’ Rigor Test

Events

Special Education Webinar Reading, Dyslexia, and Equity: Best Practices for Addressing a Threefold Challenge
Learn about proven strategies for instruction and intervention that support students with dyslexia.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
The Key to Better Learning: Indoor Air Quality
Learn about the importance of improved indoor air quality in schools, and how to pick the right solutions for educators, students, and staff.
Content provided by Delos
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
Leading Systemic Redesign: Strategies from the Field
Learn how your school community can work together to redesign the school system, reengineer instruction, & co-author personalized learning.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal What the Federal 'Don't Say Gay' Bill Actually Says
The bill would restrict federal funds for lessons on LGBTQ identities. The outcome of this week's election could revive its prospects.
4 min read
Demonstrators gather on the steps of the Florida Historic Capitol Museum in front of the Florida State Capitol on March 7, 2022, in Tallahassee, Fla. Florida House Republicans advanced a bill, dubbed by opponents as the "Don't Say Gay" bill, to forbid discussions of sexual orientation and gender identity in schools, rejecting criticism from Democrats who said the proposal demonizes LGBTQ people.
Demonstrators gather on the steps of the Florida Historic Capitol Museum in Tallahassee on March 7, 2022. Florida's "Don't Say Gay" law was a model for a federal bill introduced last month.
Wilfredo Lee/AP
Federal Fed's Education Research Board Is Back. Here's Why That Matters
Defunct for years, the National Board for Education Sciences has new members and new priorities.
2 min read
Image of a conference table.
vasabii/iStock/Getty
Federal Opinion NAEP Needs to Be Kept at Arm’s Length From Politics
It’s in all our interests to ensure NAEP releases are buffered from political considerations and walled off from political appointees.
4 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
Federal Feds Emphasize Legal Protections for Pregnant or Recently Pregnant Students, Employees
The U.S. Department of Education has released a new resource summary related to pregnancy discrimination in schools.
2 min read
Young girl checking her pregnancy test, sitting on beige couch at home.
iStock/Getty Images Plus