Federal

Studies of Popular Reading Texts Don’t Meet Reviewers’ Rigor Test

By Kathleen Kennedy Manzo — August 13, 2008 4 min read

Two well-known commercial reading programs, which have been adopted by some of the nation’s largest school districts and have met the strict requirements for research-based programs under the federal Reading First initiative, failed to earn ratings from the What Works Clearinghouse because they do not have any studies that satisfy the agency’s rigorous evidence standards.

Reports on Open Court Reading and Reading Mastery, both highly structured texts published by the Columbus, Ohio-based SRA McGraw-Hill, were released Tuesday by the clearinghouse, a program of the Institute of Education Sciences, the research arm of the U.S. Department of Education, to vet effectiveness studies on educational programs and practices.

“At this time, the WWC is unable to draw any conclusions based on research about the effectiveness or ineffectiveness” of either program, the reports say.

The publisher of the programs holds a large share of the elementary school textbook market for reading, largely on the popularity of Open Court, which is used throughout the 708,000-student Los Angeles Unified School District and is one of just two programs approved for use in K-2 reading instruction in California.

Although Open Court has been criticized in the past for not having the kind of independent, scientific evidence that federal officials called for in launching Reading First in 2002, Reading Mastery is perceived as having significant proof of its effectiveness.

Officials with the National Institute for Direct Instruction, based in Eugene, Ore., have complained to the IES, saying the review failed to consider all the available research on Reading Mastery going back decades, and mistakenly rejected more recent studies that fit the criteria.

“They have an arbitrary time limit of 1985 when most of the seminal research on Direct Instruction was conducted prior to that, including Project Follow Through, which is the largest study in the history of the United States,” said Kurt E. Engelmann, the president of the institute. Some of the “studies” that were included, he said, are nothing more than two-page promotional pieces, with charts, circulated by the publisher. “The review is a clear misrepresentation. If the full research base were included, it would receive a higher rating, even with WWC emphasis on randomized trials, which are terribly difficult to do.”

SRA/McGraw-Hill officials argue that both programs are based on solid research and have led to significant improvements in student proficiency in districts large and small.

“SRA/McGraw-Hill’s Open Court Reading and Reading Mastery programs have been developed from nearly 50 years of intense study and field-testing. Where these scientifically researched programs have been implemented, students’ reading test scores increased, and teachers and principals have seen gains in student achievement overall,” company spokesman Tom Stanton wrote in an e-mail.

Why Now?

In the past, so few studies have met the criteria outlined by the clearinghouse, and reviews have found few or “potentially positive” effects of programs, that the agency has been dubbed the “Nothing Works” clearinghouse by critics.

In a review of beginning reading last year, for example, the clearinghouse concluded that few comprehensive or supplemental programs have proof that they work. Many observers also questioned the value of a review coming five years after the federal push for research-based instruction and the demand that schools taking part in the $1 billion-a-year Reading First use programs that are effective. (“Federal Reading Review Overlooks Popular Texts,” Aug. 29, 2007.)

“What is most interesting is the question of why they took until August 2008, to do this. Open Court was among the top three basals adopted under Reading First,” said Robert E. Slavin, the founder of Success for All, which earned a somewhat positive rating from the clearinghouse last year. His program is not widely used in Reading First schools, and Mr. Slavin’s complaints that those schools were discouraged from adopting Success for All led to congressional reviews of Reading First’s implementation in 2005.

The clearinghouse, created in 2002, got off to a slow start. It released its first reviews in 2004 and then another batch two years later, but it did not include evaluations of reading programs.

“If there is no evidence on Reading Mastery or Open Court now, there was no evidence in 2003, 2004, 2005, 2006, or 2007, when Reading First schools were being virtually required to use these programs or the other three basals that the WWC has not yet reviewed,” he said.

The Education Department had issued several statements to state officials that it did not prescribe any particular commercial programs for use in Reading First schools. Subsequent investigations indicated that federal consultants for the program appeared to have steered states and districts to specific reading programs.

The clearinghouse identified 61 studies of Reading Mastery between 1985 and 2007, 15 of which used the required quasi-experimental design. Those studies, however, were deemed inadequate either because the comparison group did not meet the review requirements or other factors made it impossible to link changes in student achievement to use of the program.

The Open Court review looked at 30 studies, but 15 did not have the type of quasi-experimental design—either using randomized trials or a comparison group—the clearinghouse requires. In two of the seven that met the criteria, there was insufficient evidence that the control group was comparable to the treatment group, and the other five did not demonstrate that changes in achievement were attributable to the program.

A version of this article appeared in the August 27, 2008 edition of Education Week as Studies of Popular Reading Texts Don’t Meet Reviewers’ Rigor Test

Events

School & District Management Webinar Examining the Evidence: Catching Kids Up at a Distance
As districts, schools, and families navigate a new normal following the abrupt end of in-person schooling this spring, students’ learning opportunities vary enormously across the nation. Access to devices and broadband internet and a secure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
Branding Matters. Learn From the Pros Why and How
Branding your district matters. This webinar will provide you with practical tips and strategies to elevate your brand from three veteran professionals, each of whom has been directly responsible for building their own district’s brand.
Content provided by EdWeek Top School Jobs
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
How to Make Learning More Interactive From Anywhere
Nearly two-thirds of U.S. school districts are using hybrid learning right now with varying degrees of success. Students and teachers are getting restless and frustrated with online learning, making curriculum engagement difficult and disjointed. While
Content provided by Samsung

EdWeek Top School Jobs

Principal-Elementary School
San Antonio, TX, US
Southwest Independent School District
Principal-Elementary School
San Antonio, TX, US
Southwest Independent School District
Principal-Elementary School
San Antonio, TX, US
Southwest Independent School District
Special Education Teacher
Chicago, Illinois
JCFS Chicago

Read Next

Federal Biden Launches New Strategy to Combat COVID-19, Reopen Schools
The president plans a more centralized strategy that includes broader vaccine efforts, more data on the pandemic, and new school guidance.
5 min read
Public School 95 in the Gravesend section of Brooklyn is one of many schools in New York ordered to close due to a flare-up of coronavirus cases in the area on Oct. 5, 2020.
Public School 95 in the Gravesend section of Brooklyn is one of many schools in New York ordered to close due to a flare-up of coronavirus cases in the area on Oct. 5, 2020.
Kathy Willens/AP
Federal Biden Revokes Trump's 'Patriotic Education' Order, Will Shield DACA
Joe Biden took a flurry of executive actions on his first day as president that included a new government-wide emphasis on racial equity.
3 min read
 Joe Biden departs a news conference after introducing his nominees and appointees to economic policy posts at The Queen theater, Tuesday, Dec. 1, 2020, in Wilmington, Del.
Joe Biden departs a news conference after introducing his nominees and appointees to economic policy posts Dec. 1 in Wilmington, Del.
Andrew Harnik/AP
Federal Opinion Blessings and Best Wishes, President Biden
Rick Hess takes a moment to offer President-elect Biden his best wishes and to reflect on Inauguration Day.
1 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
Federal Biden Picks San Diego Superintendent for Deputy Education Secretary
San Diego Superintendent Cindy Marten was a classroom teacher for 17 years before she became a school and district administrator.
2 min read
Image of the White House seal
Bet Noire/Getty