Federal

Studies of Popular Reading Texts Don’t Meet Reviewers’ Rigor Test

By Kathleen Kennedy Manzo — August 13, 2008 4 min read

Two well-known commercial reading programs, which have been adopted by some of the nation’s largest school districts and have met the strict requirements for research-based programs under the federal Reading First initiative, failed to earn ratings from the What Works Clearinghouse because they do not have any studies that satisfy the agency’s rigorous evidence standards.

Reports on Open Court Reading and Reading Mastery, both highly structured texts published by the Columbus, Ohio-based SRA McGraw-Hill, were released Tuesday by the clearinghouse, a program of the Institute of Education Sciences, the research arm of the U.S. Department of Education, to vet effectiveness studies on educational programs and practices.

“At this time, the WWC is unable to draw any conclusions based on research about the effectiveness or ineffectiveness” of either program, the reports say.

The publisher of the programs holds a large share of the elementary school textbook market for reading, largely on the popularity of Open Court, which is used throughout the 708,000-student Los Angeles Unified School District and is one of just two programs approved for use in K-2 reading instruction in California.

Although Open Court has been criticized in the past for not having the kind of independent, scientific evidence that federal officials called for in launching Reading First in 2002, Reading Mastery is perceived as having significant proof of its effectiveness.

Officials with the National Institute for Direct Instruction, based in Eugene, Ore., have complained to the IES, saying the review failed to consider all the available research on Reading Mastery going back decades, and mistakenly rejected more recent studies that fit the criteria.

“They have an arbitrary time limit of 1985 when most of the seminal research on Direct Instruction was conducted prior to that, including Project Follow Through, which is the largest study in the history of the United States,” said Kurt E. Engelmann, the president of the institute. Some of the “studies” that were included, he said, are nothing more than two-page promotional pieces, with charts, circulated by the publisher. “The review is a clear misrepresentation. If the full research base were included, it would receive a higher rating, even with WWC emphasis on randomized trials, which are terribly difficult to do.”

SRA/McGraw-Hill officials argue that both programs are based on solid research and have led to significant improvements in student proficiency in districts large and small.

“SRA/McGraw-Hill’s Open Court Reading and Reading Mastery programs have been developed from nearly 50 years of intense study and field-testing. Where these scientifically researched programs have been implemented, students’ reading test scores increased, and teachers and principals have seen gains in student achievement overall,” company spokesman Tom Stanton wrote in an e-mail.

Why Now?

In the past, so few studies have met the criteria outlined by the clearinghouse, and reviews have found few or “potentially positive” effects of programs, that the agency has been dubbed the “Nothing Works” clearinghouse by critics.

In a review of beginning reading last year, for example, the clearinghouse concluded that few comprehensive or supplemental programs have proof that they work. Many observers also questioned the value of a review coming five years after the federal push for research-based instruction and the demand that schools taking part in the $1 billion-a-year Reading First use programs that are effective. (“Federal Reading Review Overlooks Popular Texts,” Aug. 29, 2007.)

“What is most interesting is the question of why they took until August 2008, to do this. Open Court was among the top three basals adopted under Reading First,” said Robert E. Slavin, the founder of Success for All, which earned a somewhat positive rating from the clearinghouse last year. His program is not widely used in Reading First schools, and Mr. Slavin’s complaints that those schools were discouraged from adopting Success for All led to congressional reviews of Reading First’s implementation in 2005.

The clearinghouse, created in 2002, got off to a slow start. It released its first reviews in 2004 and then another batch two years later, but it did not include evaluations of reading programs.

“If there is no evidence on Reading Mastery or Open Court now, there was no evidence in 2003, 2004, 2005, 2006, or 2007, when Reading First schools were being virtually required to use these programs or the other three basals that the WWC has not yet reviewed,” he said.

The Education Department had issued several statements to state officials that it did not prescribe any particular commercial programs for use in Reading First schools. Subsequent investigations indicated that federal consultants for the program appeared to have steered states and districts to specific reading programs.

The clearinghouse identified 61 studies of Reading Mastery between 1985 and 2007, 15 of which used the required quasi-experimental design. Those studies, however, were deemed inadequate either because the comparison group did not meet the review requirements or other factors made it impossible to link changes in student achievement to use of the program.

The Open Court review looked at 30 studies, but 15 did not have the type of quasi-experimental design—either using randomized trials or a comparison group—the clearinghouse requires. In two of the seven that met the criteria, there was insufficient evidence that the control group was comparable to the treatment group, and the other five did not demonstrate that changes in achievement were attributable to the program.

A version of this article appeared in the August 27, 2008 edition of Education Week as Studies of Popular Reading Texts Don’t Meet Reviewers’ Rigor Test

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Equity, Care and Connection: New SEL Tools and Practices to Support Students and Adults
As school districts plan to welcome students back into buildings for the upcoming school year, this is the perfect time to take a hard look at both our practices and our systems to build a
Content provided by Panorama Education
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Classroom Technology Webinar
Here to Stay – Pandemic Lessons for EdTech in Future Development
What technology is needed in a post pandemic district? Learn how changes in education will impact development of new technologies.
Content provided by AWS
School & District Management Live Online Discussion A Seat at the Table: Strategies & Tips for Complex Decision-Making
Schools are working through the most disruptive period in the history of modern education, facing a pandemic, economic problems, social justice issues, and rapid technological change all at once. But even after the pandemic ends,

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Biden Pick for Education Civil Rights Office Has History With Racial Equity, LGBTQ Issues
Biden selected Catherine Lhamon to lead the Education Department's civil rights work, a role she also held in the Obama administration.
2 min read
Flags decorate a space outside the office of the Education Secretary at the Education Department in Washington on Aug. 9, 2017.
Flags decorate a space outside the office of the Education Secretary at the Education Department in Washington on Aug. 9, 2017.
Jacquelyn Martin/AP
Federal Lawmakers Press CDC About Teachers' Union Influence on School Reopening Guidance
Republican senators asked CDC Director Rochelle Walensky about reports a teachers' union had input on guidance for schools on COVID-19.
3 min read
Dr. Rochelle Walensky speaks during an event in Wilmington, Del., to announce President-elect Joe Biden's health care team on Dec. 8, 2020.
Dr. Rochelle Walensky, CDC director, speaks during an event in Wilmington, Del., to announce then-President-elect Joe Biden's health care team on Dec. 8, 2020.
Susan Walsh/AP
Federal Biden Taps Ex-Obama Aide Roberto Rodriguez for Key Education Department Job
Rodriguez served as a top education staffer to President Barack Obama and currently leads a teacher-advocacy organization.
3 min read
BRIC ARCHIVE
Getty
Federal Biden Pitches Plan to Expand Universal Pre-K, Free School Meal Programs, Teacher Training
The president's $1.8 trillion American Families Plan faces strong headwinds as Congress considers other costly administration proposals.
8 min read
President Joe Biden addresses Congress from the House chamber. Behind him are Vice President Kamala Harris and Speaker of the House Nancy Pelosi.
President Joe Biden addresses a joint session of Congress Wednesday night, as Vice President Kamala Harris, left, and House Speaker Nancy Pelosi of Calif., applaud.<br/>
Chip Somodevilla/AP