When a federal review of beginning-reading programs was commissioned four years ago, experts and educators hoped it would help school leaders sift through the vast marketplace of instructional materials and find those most effective for improving achievement. But the long-awaited study by the What Works Clearinghouse, released this month, may not fully deliver on that promise.
In fact, the analysis found that few comprehensive or supplemental programs have empirical proof that they work. And none of the most popular commercial programs on the market—including McGraw-Hill’s Open Court, Scott Foresman Reading, and Houghton Mifflin Reading, which have earned hundreds of millions of dollars in sales to districts—had sufficiently rigorous studies to be included in the review by the clearinghouse.
“They tended not to have studies with randomized-control trials or with experimental designs that met the clearinghouse’s evidence standards,” said Jill Constantine, the principal director of the review.
Most of the programs deemed to have “positive effects” or “potentially positive effects” in the review were supplemental or intervention programs, not core reading series. In the review of nearly 900 studies, just 51 met the standard for evidence, meaning just one or two for each program.
More than 120 other programs, however, had no studies that met the requirements.
The complexity of the study, and the scarcity of suitable research on the topic, point to the difficulty of determining what works in the classroom, said James W. Kohlmoos, the president of the Knowledge Alliance, a Washington-based nonprofit organization that works with education groups to translate research findings to instruction.
“In the very early stages, we thought it was something that could be of value, but it’s only of marginal value right now,” Mr. Kohlmoos said of the federal review. “We all wish there was more in there and the results were more definitive,” he said, “but it just shows that this is a far more complex and difficult effort than we all originally thought.”
Smaller Programs Reviewed
The clearinghouse, which the U.S. Department of Education’s Institute of Education Sciences set up in 2002 to vet research on “what works” in education, has given few of its coveted positive ratings.
Source: Institute of Education Sciences
Just one of the reading programs examined was found to have positive effects or potentially positive effects across all four of the domains in the review: alphabetics, fluency, comprehension, and general reading achievement. That program, Reading Recovery, involving intensive, one-on-one tutoring, has drawn criticism over the past few years from prominent researchers and federal officials who claim it is costly and lacks evidence that it helps struggling readers.
Success for All, a whole-school-reform program developed at Johns Hopkins University in Baltimore, got the favorable rating on alphabetics and general reading achievement, but mixed results on comprehension. Voyager Universal Literacy System, published by the Dallas-based Voyager Learning, was found to have potentially positive effects on alphabetics but potentially negative effects on comprehension. Accelerated Reader, distributed by Renaissance Learning Inc., was found to have a potentially positive impact on comprehension and general reading achievement.
The Success for All review is not as glowing as previous federal and private studies, which have put it in an exclusive group of programs that meet the highest standard of evidence that they are effective at teaching children to read. (“Long-Awaited Study Shows ‘Success for All’ Gains,” May 11, 2005.)
Robert E. Slavin, the founder of Success for All, said the clearinghouse review discounted such studies, which were based on large numbers of students exposed to the whole-school-reform program, and instead favored research on targeted interventions for small groups of students.
Those products—such as Start Making a Reader Today, Kaplan SpellRead, and Peer-Assisted Learning Strategies, or PALS—also got positive reviews.
But Mr. Slavin said the review is missing important information on the large commercial programs.
“The What Works Clearinghouse was designed to ... tell educators in a fair, nonpolitical, scientifically justifiable way what is the strength of evidence of various programs,” he said. “How could it not have done so on the largest programs? To go for five years without getting that job done boggles the imagination.”
In fact, the clearinghouse report lists 36 products, including some widely used series, that are in the process of being reviewed, but notes that each has just one study that meets the criteria “with reservations.”
First Appraisal
Tom Stanton, a spokesman for the New York City-based McGraw-Hill, wrote in an e-mail that the company’s programs have been developed over decades of “intense study, scientific research, and field testing.”
“The key question,” he added, “is, ‘do our programs drive student achievement?’ Our answer is that they do, and that we have results to prove it.”
The reading review, which has been under way for more than three years, is the first in an ongoing appraisal of reading programs, according to Phoebe H. Cottingham, the commissioner of the National Center for Education Evaluation and Regional Assistance, which oversees the clearinghouse.
