Federal

Studies of Popular Reading Texts Don’t Meet Reviewers’ Rigor Test

By Kathleen Kennedy Manzo — August 13, 2008 4 min read
  • Save to favorites
  • Print

Two well-known commercial reading programs, which have been adopted by some of the nation’s largest school districts and have met the strict requirements for research-based programs under the federal Reading First initiative, failed to earn ratings from the What Works Clearinghouse because they do not have any studies that satisfy the agency’s rigorous evidence standards.

Reports on Open Court Reading and Reading Mastery, both highly structured texts published by the Columbus, Ohio-based SRA McGraw-Hill, were released Tuesday by the clearinghouse, a program of the Institute of Education Sciences, the research arm of the U.S. Department of Education, to vet effectiveness studies on educational programs and practices.

“At this time, the WWC is unable to draw any conclusions based on research about the effectiveness or ineffectiveness” of either program, the reports say.

The publisher of the programs holds a large share of the elementary school textbook market for reading, largely on the popularity of Open Court, which is used throughout the 708,000-student Los Angeles Unified School District and is one of just two programs approved for use in K-2 reading instruction in California.

Although Open Court has been criticized in the past for not having the kind of independent, scientific evidence that federal officials called for in launching Reading First in 2002, Reading Mastery is perceived as having significant proof of its effectiveness.

Officials with the National Institute for Direct Instruction, based in Eugene, Ore., have complained to the IES, saying the review failed to consider all the available research on Reading Mastery going back decades, and mistakenly rejected more recent studies that fit the criteria.

“They have an arbitrary time limit of 1985 when most of the seminal research on Direct Instruction was conducted prior to that, including Project Follow Through, which is the largest study in the history of the United States,” said Kurt E. Engelmann, the president of the institute. Some of the “studies” that were included, he said, are nothing more than two-page promotional pieces, with charts, circulated by the publisher. “The review is a clear misrepresentation. If the full research base were included, it would receive a higher rating, even with WWC emphasis on randomized trials, which are terribly difficult to do.”

SRA/McGraw-Hill officials argue that both programs are based on solid research and have led to significant improvements in student proficiency in districts large and small.

“SRA/McGraw-Hill’s Open Court Reading and Reading Mastery programs have been developed from nearly 50 years of intense study and field-testing. Where these scientifically researched programs have been implemented, students’ reading test scores increased, and teachers and principals have seen gains in student achievement overall,” company spokesman Tom Stanton wrote in an e-mail.

Why Now?

In the past, so few studies have met the criteria outlined by the clearinghouse, and reviews have found few or “potentially positive” effects of programs, that the agency has been dubbed the “Nothing Works” clearinghouse by critics.

In a review of beginning reading last year, for example, the clearinghouse concluded that few comprehensive or supplemental programs have proof that they work. Many observers also questioned the value of a review coming five years after the federal push for research-based instruction and the demand that schools taking part in the $1 billion-a-year Reading First use programs that are effective. (“Federal Reading Review Overlooks Popular Texts,” Aug. 29, 2007.)

“What is most interesting is the question of why they took until August 2008, to do this. Open Court was among the top three basals adopted under Reading First,” said Robert E. Slavin, the founder of Success for All, which earned a somewhat positive rating from the clearinghouse last year. His program is not widely used in Reading First schools, and Mr. Slavin’s complaints that those schools were discouraged from adopting Success for All led to congressional reviews of Reading First’s implementation in 2005.

The clearinghouse, created in 2002, got off to a slow start. It released its first reviews in 2004 and then another batch two years later, but it did not include evaluations of reading programs.

“If there is no evidence on Reading Mastery or Open Court now, there was no evidence in 2003, 2004, 2005, 2006, or 2007, when Reading First schools were being virtually required to use these programs or the other three basals that the WWC has not yet reviewed,” he said.

The Education Department had issued several statements to state officials that it did not prescribe any particular commercial programs for use in Reading First schools. Subsequent investigations indicated that federal consultants for the program appeared to have steered states and districts to specific reading programs.

The clearinghouse identified 61 studies of Reading Mastery between 1985 and 2007, 15 of which used the required quasi-experimental design. Those studies, however, were deemed inadequate either because the comparison group did not meet the review requirements or other factors made it impossible to link changes in student achievement to use of the program.

The Open Court review looked at 30 studies, but 15 did not have the type of quasi-experimental design—either using randomized trials or a comparison group—the clearinghouse requires. In two of the seven that met the criteria, there was insufficient evidence that the control group was comparable to the treatment group, and the other five did not demonstrate that changes in achievement were attributable to the program.

A version of this article appeared in the August 27, 2008 edition of Education Week as Studies of Popular Reading Texts Don’t Meet Reviewers’ Rigor Test

Events

Jobs Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
Ed-Tech Policy Webinar Artificial Intelligence in Practice: Building a Roadmap for AI Use in Schools
AI in education: game-changer or classroom chaos? Join our webinar & learn how to navigate this evolving tech responsibly.
Education Webinar Developing and Executing Impactful Research Campaigns to Fuel Your Ed Marketing Strategy 
Develop impactful research campaigns to fuel your marketing. Join the EdWeek Research Center for a webinar with actionable take-aways for companies who sell to K-12 districts.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Biden Calls for Teacher Pay Raises, Expanded Pre-K in State of the Union
President Joe Biden highlighted a number of his education priorities in a high-stakes speech as he seeks a second term.
5 min read
President Joe Biden delivers the State of the Union address to a joint session of Congress at the Capitol on March 7, 2024, in Washington.
President Joe Biden delivers the State of the Union address to a joint session of Congress at the Capitol on March 7, 2024, in Washington.
Shawn Thew/Pool via AP
Federal Low-Performing Schools Are Left to Languish by Districts and States, Watchdog Finds
Fewer than half of district plans for improving struggling schools meet bare minimum requirements.
11 min read
A group of silhouettes looks across a grid with a public school on the other side.
Vanessa Solis/Education Week via Canva
Federal Biden Admin. Says New K-12 Agenda Tackles Absenteeism, Tutoring, Extended Learning
The White House unveiled a set of K-12 priorities at the start of an election year.
4 min read
U.S. Education Secretary Miguel Cardona participates in a roundtable discussion with students from Dartmouth College on Jan. 10, 2024, on the school's campus, in Hanover, N.H.
U.S. Education Secretary Miguel Cardona participates in a roundtable discussion with students from Dartmouth College on Jan. 10, 2024, on the school's campus, in Hanover, N.H.
Steven Senne/AP
Federal Lawmakers Want to Reauthorize a Major Education Research Law. What Stands in the Way?
Lawmakers have tried and failed to reauthorize the Education Sciences Reform Act over the past nearly two decades.
7 min read
Sen. Bill Cassidy, R-La., left, joins Senate Health, Education, Labor and Pensions Committee Chairman Bernie Sanders, I-Vt., left, as Starbucks founder Howard Schultz answers questions about the company's actions during an ongoing employee unionizing campaign, at the Capitol in Washington, on March 29, 2023.
Sen. Bill Cassidy, R-La., left, joins Senate Health, Education, Labor and Pensions Committee Chairman Bernie Sanders, I-Vt., left, at the Capitol in Washington, on March 29, 2023. The two lawmakers sponsored a bill to reauthorize the Education Sciences Reform Act.
J. Scott Applewhite/AP