Leading Commercial Series Don’t Satisfy ‘Gold Standard’

By Kathleen Kennedy Manzo — October 01, 2004 6 min read
  • Save to favorites
  • Print

A number of commercial reading programs have satisfied the requirement under the federal reading law for embodying a strong research base, yet there appears to be limited outside evidence that any of them produces a conclusive and consistent effect on overall reading achievement.

The publishers of each of the top-selling reading programs make convincing cases for their effectiveness, offering research summaries, case studies, and commissioned empirical investigations to support their claims.

Scott Foresman Reading, published by the New York City-based Pearson Education, comes with a bound, 1,000-page synopsis of journal articles and scientific studies on which it is based. The effectiveness of McGraw-Hill’s Open Court is outlined in a commissioned study of California schools that used the program and saw some improvements in test results.

And the Boston-based Houghton Mifflin, which has cornered the profitable California market and proved popular in districts across the country, has a link on its Web site explaining the program’s alignment with the findings of the National Reading Panel and the work of prominent scholars who have influenced policymakers.

Despite such impressive documentation, it appears that none of the top commercial series—including Open Court, Houghton Mifflin Reading, Harcourt, Scott Foresman Reading, and Macmillan/McGraw-Hill of New York City, all of which have been accepted for use under the stringent federal Reading First rules—meets what is called the gold standard for research.

They don’t have randomized studies pitting their products against other methods or materials; the studies they have commissioned have not been published in scholarly journals; and the companies have not documented improvements in student achievement across the range of schools and students. The programs have thrived, however, on their reputations among educators as having met the specified—and perceived—research standards in the Reading First legislation, which is part of the No Child Left Behind Act.

Those perceptions were bolstered by federal officials after some of the materials were held up as examples at reading academies sponsored by the U.S. Department of Education to give state officials an overview of the Reading First requirements.

Researchers and publishers criticized the officials’ action as a subtle endorsement of specific curricula, which is prohibited by the No Child Left Behind Act, as well as previous federal legislation.

“Assertions made by government officials in advising Reading First applicants, and claims made by publishers in advertising their products, that certain comprehensive programs are ‘research based’ (with the implication that others and teacher-constructed programs are not) are not supported by anything in the National Reading Panel report,” panel member Joanne Yatvin wrote in a Commentary piece for Education Week last year.

Confusion Over Research

Other programs, meanwhile, such as Success for All, which has a number of studies supporting its effectiveness, have been shunted aside. That whole-school-reform program—developed at Johns Hopkins University—has seen a general decrease in the number of schools using it. Its founder, Robert Slavin, complains that the success of a product depends more on marketing than on science.

“There is bright potential in really using evidence and research to make a difference in reading instruction,” said Mr. Slavin, a professor of education at Johns Hopkins. “But there’s a huge difference between marketing research and genuine research.”

Exacerbating the issue, some experts say, is the widespread confusion over what constitutes credible research and how to interpret it for practical purposes.

“No Child Left Behind made them aware of the research, but whether they are becoming critical consumers of the research, I wouldn’t go that far,” said Susan B. Neuman, a researcher at the University of Michigan in Ann Arbor who helped roll out the federal law as the U.S. Department of Education’s assistant secretary for elementary and secondary education early in the Bush administration.

Federal officials themselves may have added to the confusion.

G. Reid Lyon, the chief of the reading-research branch of the National Institute of Child Health and Human Development, for example, has suggested that commercial programs should have published studies demonstrating their effectiveness with the kinds of students for which districts intend to use them.

When asked about New York City’s choice of reading textbook last year, he said it lacked published experimental or quasi-experimental studies on its effectiveness in a variety of classrooms. Mr. Lyon hinted in a New York Times article that the text would not pass the rigorous review under Reading First, potentially threatening the city’s $111 million share. In the hope of appeasing federal reviewers for the grant program, city officials eventually agreed on another reading program for schools applying for Reading First money—one that was accepted by state reviewers even though it, too, lacked that very kind of scientific proof.

But Ms. Neuman said at the time that she did not believe that Reading First required that higher standard of evidence.

Publishers’ Proof

Publishers are working hard to make sure their products accurately reflect research principles, and to gather convincing evidence that they work, said Marci Baughmann, the director of academic research for the Pearson Education school group.

Like many other publishers, Pearson, the parent company of Scott Foresman, has hired a small army of researchers to consult throughout every stage of development and production of reading texts. The publisher has been gathering data from clients to map changes in student achievement, and has commissioned independent studies or reviews from prominent researchers.

Studies on the Scott Foresman program, for instance, have been submitted to the What Works Clearinghouse of the federal Education Department’s Institute of Education Studies, Ms. Baughmann said, in the hope of getting a positive peer review to back up the product’s claims.

Many publishers have also asked the Florida Center for Reading Research to evaluate the research justifications behind their products. The center, directed by Joseph Torgesen at Florida State University, one of a select group of researchers called on to help launch Reading First, has produced eight reports on core reading programs and more than three dozen evaluations of interventions for struggling readers.

Mr. Torgesen, who has won research funding from the NICHD and runs one of three technical-assistance centers for Reading First as part of a $37 million Education Department grant, has conducted numerous efficacy studies on reading interventions. The center’s reports summarize the principles behind the programs, evaluate the content, and identify the elements of the programs that do and do not align with research conclusions. The reports do not make claims about programs’ effectiveness.

Beyond those appraisals, however, publishers are unlikely to see their studies in peer-reviewed journals—the stamp of approval to many in the field. The publishing process is often lengthy and could distract researchers from their other projects, according to Ms. Baughmann.

The new focus on research evidence “places a certain amount of responsibility on the publishers to hire good academic researchers willing to work to the highest standards,” she said. “But we don’t anticipate being able to get [their work] published in a journal.”

Related Tags:


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Reading & Literacy Webinar
Your Questions on the Science of Reading, Answered
Dive into the Science of Reading with K-12 leaders. Discover strategies, policy insights, and more in our webinar.
Content provided by Otus
Mathematics Live Online Discussion A Seat at the Table: Breaking the Cycle: How Districts are Turning around Dismal Math Scores
Math myth: Students just aren't good at it? Join us & learn how districts are boosting math scores.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Achievement Webinar
How To Tackle The Biggest Hurdles To Effective Tutoring
Learn how districts overcome the three biggest challenges to implementing high-impact tutoring with fidelity: time, talent, and funding.
Content provided by Saga Education

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Curriculum Video VIDEO: What AP African American Studies Looks Like in Practice
The AP African American studies course has sparked national debate since the pilot kicked off in 2022. A look inside the classroom.
Ahenewa El-Amin leads a conversation with students during her AP African American Studies class at Henry Clay High School in Lexington, Ky., on March 19, 2024.
Ahenewa El-Amin leads a conversation with students during her AP African American Studies class at Henry Clay High School in Lexington, Ky., on March 19, 2024.
Jaclyn Borowski/Education Week
Curriculum Anti-Critical-Race-Theory Laws Are Slowing Down. Here Are 3 Things to Know
After a wave of bills limiting class discussions on race and gender, an Education Week analysis shows the policies have slowed.
5 min read
A man holds up a sign during a protest against Critical Race Theory outside a Washoe County School District board meeting on May 25, 2021, in Reno, Nev.
A man holds up a sign during a protest against critical race theory outside a Washoe County School District board meeting on May 25, 2021, in Reno, Nev. This year, the numbers of bills being proposed to restrict what schools can teach and discuss about race and racism have slowed down from prior years.
Andy Barron/Reno Gazette-Journal via AP
Curriculum History Group Finds Little Evidence of K-12 'Indoctrination'
Most social science educators say they keep politics out of the classroom, but need help identifying good curriculum resources
6 min read
Photo of U.S. flag in classroom.
iStock / Getty Images Plus
Curriculum How an International Baccalaureate Education Cuts Through the ‘Noise’ on Banned Topics
IB programs offer students college credit in high school and advanced learning environments.
9 min read
James Minor teaches his IB Language and Literature class at Riverview High School in Sarasota, Fla., on Jan. 23, 2024.
James Minor teaches his IB Language and Literature class at Riverview High School in Sarasota, Fla., on Jan. 23, 2024.
Zack Wittman for Education Week