Leading Commercial Series Don’t Satisfy ‘Gold Standard’

By Kathleen Kennedy Manzo — October 01, 2004 6 min read
  • Save to favorites
  • Print

A number of commercial reading programs have satisfied the requirement under the federal reading law for embodying a strong research base, yet there appears to be limited outside evidence that any of them produces a conclusive and consistent effect on overall reading achievement.

The publishers of each of the top-selling reading programs make convincing cases for their effectiveness, offering research summaries, case studies, and commissioned empirical investigations to support their claims.

Scott Foresman Reading, published by the New York City-based Pearson Education, comes with a bound, 1,000-page synopsis of journal articles and scientific studies on which it is based. The effectiveness of McGraw-Hill’s Open Court is outlined in a commissioned study of California schools that used the program and saw some improvements in test results.

And the Boston-based Houghton Mifflin, which has cornered the profitable California market and proved popular in districts across the country, has a link on its Web site explaining the program’s alignment with the findings of the National Reading Panel and the work of prominent scholars who have influenced policymakers.

Despite such impressive documentation, it appears that none of the top commercial series—including Open Court, Houghton Mifflin Reading, Harcourt, Scott Foresman Reading, and Macmillan/McGraw-Hill of New York City, all of which have been accepted for use under the stringent federal Reading First rules—meets what is called the gold standard for research.

They don’t have randomized studies pitting their products against other methods or materials; the studies they have commissioned have not been published in scholarly journals; and the companies have not documented improvements in student achievement across the range of schools and students. The programs have thrived, however, on their reputations among educators as having met the specified—and perceived—research standards in the Reading First legislation, which is part of the No Child Left Behind Act.

Those perceptions were bolstered by federal officials after some of the materials were held up as examples at reading academies sponsored by the U.S. Department of Education to give state officials an overview of the Reading First requirements.

Researchers and publishers criticized the officials’ action as a subtle endorsement of specific curricula, which is prohibited by the No Child Left Behind Act, as well as previous federal legislation.

“Assertions made by government officials in advising Reading First applicants, and claims made by publishers in advertising their products, that certain comprehensive programs are ‘research based’ (with the implication that others and teacher-constructed programs are not) are not supported by anything in the National Reading Panel report,” panel member Joanne Yatvin wrote in a Commentary piece for Education Week last year.

Confusion Over Research

Other programs, meanwhile, such as Success for All, which has a number of studies supporting its effectiveness, have been shunted aside. That whole-school-reform program—developed at Johns Hopkins University—has seen a general decrease in the number of schools using it. Its founder, Robert Slavin, complains that the success of a product depends more on marketing than on science.

“There is bright potential in really using evidence and research to make a difference in reading instruction,” said Mr. Slavin, a professor of education at Johns Hopkins. “But there’s a huge difference between marketing research and genuine research.”

Exacerbating the issue, some experts say, is the widespread confusion over what constitutes credible research and how to interpret it for practical purposes.

“No Child Left Behind made them aware of the research, but whether they are becoming critical consumers of the research, I wouldn’t go that far,” said Susan B. Neuman, a researcher at the University of Michigan in Ann Arbor who helped roll out the federal law as the U.S. Department of Education’s assistant secretary for elementary and secondary education early in the Bush administration.

Federal officials themselves may have added to the confusion.

G. Reid Lyon, the chief of the reading-research branch of the National Institute of Child Health and Human Development, for example, has suggested that commercial programs should have published studies demonstrating their effectiveness with the kinds of students for which districts intend to use them.

When asked about New York City’s choice of reading textbook last year, he said it lacked published experimental or quasi-experimental studies on its effectiveness in a variety of classrooms. Mr. Lyon hinted in a New York Times article that the text would not pass the rigorous review under Reading First, potentially threatening the city’s $111 million share. In the hope of appeasing federal reviewers for the grant program, city officials eventually agreed on another reading program for schools applying for Reading First money—one that was accepted by state reviewers even though it, too, lacked that very kind of scientific proof.

But Ms. Neuman said at the time that she did not believe that Reading First required that higher standard of evidence.

Publishers’ Proof

Publishers are working hard to make sure their products accurately reflect research principles, and to gather convincing evidence that they work, said Marci Baughmann, the director of academic research for the Pearson Education school group.

Like many other publishers, Pearson, the parent company of Scott Foresman, has hired a small army of researchers to consult throughout every stage of development and production of reading texts. The publisher has been gathering data from clients to map changes in student achievement, and has commissioned independent studies or reviews from prominent researchers.

Studies on the Scott Foresman program, for instance, have been submitted to the What Works Clearinghouse of the federal Education Department’s Institute of Education Studies, Ms. Baughmann said, in the hope of getting a positive peer review to back up the product’s claims.

Many publishers have also asked the Florida Center for Reading Research to evaluate the research justifications behind their products. The center, directed by Joseph Torgesen at Florida State University, one of a select group of researchers called on to help launch Reading First, has produced eight reports on core reading programs and more than three dozen evaluations of interventions for struggling readers.

Mr. Torgesen, who has won research funding from the NICHD and runs one of three technical-assistance centers for Reading First as part of a $37 million Education Department grant, has conducted numerous efficacy studies on reading interventions. The center’s reports summarize the principles behind the programs, evaluate the content, and identify the elements of the programs that do and do not align with research conclusions. The reports do not make claims about programs’ effectiveness.

Beyond those appraisals, however, publishers are unlikely to see their studies in peer-reviewed journals—the stamp of approval to many in the field. The publishing process is often lengthy and could distract researchers from their other projects, according to Ms. Baughmann.

The new focus on research evidence “places a certain amount of responsibility on the publishers to hire good academic researchers willing to work to the highest standards,” she said. “But we don’t anticipate being able to get [their work] published in a journal.”

Related Tags:


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
IT Infrastructure Webinar
A New Era In Connected Learning: Security, Accessibility and Affordability for a Future-Ready Classroom
Learn about Windows 11 SE and Surface Laptop SE. Enable students to unlock learning and develop new skills.
Content provided by Microsoft Surface
Classroom Technology K-12 Essentials Forum Making Technology Work Better in Schools
Join experts for a look at the steps schools are taking (or should take) to improve the use of technology in schools.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Budget & Finance Webinar
The ABCs of ESSER: How to Make the Most of Relief Funds Before They Expire
Join a diverse group of K-12 experts to learn how to leverage federal funds before they expire and improve student learning environments.
Content provided by Johnson Controls

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Curriculum Want Students to Become Better Problem Solvers? Then Teach Them to Fly Planes and Drones
A free, relatively new curriculum that teaches students about the concepts of flying planes and drones is growing quickly nationwide.
10 min read
Sophomore Byron Barksdale, left, and classmates from Magruder High School’s aviation program examine the inside of a charter plane at the Montgomery County Airpark in Gaithersburg, Md., on April 6, 2022.
Sophomore Byron Barksdale, left, and classmates from Magruder High School’s aviation program examine the inside of a charter plane at the Montgomery County Airpark in Gaithersburg, Md., on April 6, 2022.
Jaclyn Borowski/Education Week
Curriculum Florida's Math Textbook Brouhaha: Your Questions Answered
Florida rejected several math textbooks for 'indoctrination.' But what do we really know about them—and what happens next for districts?
9 min read
Conceptual Illustration of a chalkboard with math equations, where many are blacked out and unreadable
Vanessa Solis/Education Week and E+/Getty
Curriculum How Prevalent Are Book Bans This Year? New Data Show Impact
Two million students in 86 school districts have had access to books restricted due to book bans, according to a report by PEN America.
4 min read
Image of library shelves.
VTT Studio/iStock/Getty
Curriculum Opinion Ignoring Racism in Schools Actually Increases Prejudice
Good intentions aren't enough to address race and racism in schools; good practice is.
Ross Wiener & Francesca López
5 min read
Conceptual Illustration of a hand pointing the way forward over a field of arrows pointing in all directions.