Leading Commercial Series Don’t Satisfy ‘Gold Standard’

By Kathleen Kennedy Manzo — October 01, 2004 6 min read
  • Save to favorites
  • Print

A number of commercial reading programs have satisfied the requirement under the federal reading law for embodying a strong research base, yet there appears to be limited outside evidence that any of them produces a conclusive and consistent effect on overall reading achievement.

The publishers of each of the top-selling reading programs make convincing cases for their effectiveness, offering research summaries, case studies, and commissioned empirical investigations to support their claims.

Scott Foresman Reading, published by the New York City-based Pearson Education, comes with a bound, 1,000-page synopsis of journal articles and scientific studies on which it is based. The effectiveness of McGraw-Hill’s Open Court is outlined in a commissioned study of California schools that used the program and saw some improvements in test results.

And the Boston-based Houghton Mifflin, which has cornered the profitable California market and proved popular in districts across the country, has a link on its Web site explaining the program’s alignment with the findings of the National Reading Panel and the work of prominent scholars who have influenced policymakers.

Despite such impressive documentation, it appears that none of the top commercial series—including Open Court, Houghton Mifflin Reading, Harcourt, Scott Foresman Reading, and Macmillan/McGraw-Hill of New York City, all of which have been accepted for use under the stringent federal Reading First rules—meets what is called the gold standard for research.

They don’t have randomized studies pitting their products against other methods or materials; the studies they have commissioned have not been published in scholarly journals; and the companies have not documented improvements in student achievement across the range of schools and students. The programs have thrived, however, on their reputations among educators as having met the specified—and perceived—research standards in the Reading First legislation, which is part of the No Child Left Behind Act.

Those perceptions were bolstered by federal officials after some of the materials were held up as examples at reading academies sponsored by the U.S. Department of Education to give state officials an overview of the Reading First requirements.

Researchers and publishers criticized the officials’ action as a subtle endorsement of specific curricula, which is prohibited by the No Child Left Behind Act, as well as previous federal legislation.

“Assertions made by government officials in advising Reading First applicants, and claims made by publishers in advertising their products, that certain comprehensive programs are ‘research based’ (with the implication that others and teacher-constructed programs are not) are not supported by anything in the National Reading Panel report,” panel member Joanne Yatvin wrote in a Commentary piece for Education Week last year.

Confusion Over Research

Other programs, meanwhile, such as Success for All, which has a number of studies supporting its effectiveness, have been shunted aside. That whole-school-reform program—developed at Johns Hopkins University—has seen a general decrease in the number of schools using it. Its founder, Robert Slavin, complains that the success of a product depends more on marketing than on science.

“There is bright potential in really using evidence and research to make a difference in reading instruction,” said Mr. Slavin, a professor of education at Johns Hopkins. “But there’s a huge difference between marketing research and genuine research.”

Exacerbating the issue, some experts say, is the widespread confusion over what constitutes credible research and how to interpret it for practical purposes.

“No Child Left Behind made them aware of the research, but whether they are becoming critical consumers of the research, I wouldn’t go that far,” said Susan B. Neuman, a researcher at the University of Michigan in Ann Arbor who helped roll out the federal law as the U.S. Department of Education’s assistant secretary for elementary and secondary education early in the Bush administration.

Federal officials themselves may have added to the confusion.

G. Reid Lyon, the chief of the reading-research branch of the National Institute of Child Health and Human Development, for example, has suggested that commercial programs should have published studies demonstrating their effectiveness with the kinds of students for which districts intend to use them.

When asked about New York City’s choice of reading textbook last year, he said it lacked published experimental or quasi-experimental studies on its effectiveness in a variety of classrooms. Mr. Lyon hinted in a New York Times article that the text would not pass the rigorous review under Reading First, potentially threatening the city’s $111 million share. In the hope of appeasing federal reviewers for the grant program, city officials eventually agreed on another reading program for schools applying for Reading First money—one that was accepted by state reviewers even though it, too, lacked that very kind of scientific proof.

But Ms. Neuman said at the time that she did not believe that Reading First required that higher standard of evidence.

Publishers’ Proof

Publishers are working hard to make sure their products accurately reflect research principles, and to gather convincing evidence that they work, said Marci Baughmann, the director of academic research for the Pearson Education school group.

Like many other publishers, Pearson, the parent company of Scott Foresman, has hired a small army of researchers to consult throughout every stage of development and production of reading texts. The publisher has been gathering data from clients to map changes in student achievement, and has commissioned independent studies or reviews from prominent researchers.

Studies on the Scott Foresman program, for instance, have been submitted to the What Works Clearinghouse of the federal Education Department’s Institute of Education Studies, Ms. Baughmann said, in the hope of getting a positive peer review to back up the product’s claims.

Many publishers have also asked the Florida Center for Reading Research to evaluate the research justifications behind their products. The center, directed by Joseph Torgesen at Florida State University, one of a select group of researchers called on to help launch Reading First, has produced eight reports on core reading programs and more than three dozen evaluations of interventions for struggling readers.

Mr. Torgesen, who has won research funding from the NICHD and runs one of three technical-assistance centers for Reading First as part of a $37 million Education Department grant, has conducted numerous efficacy studies on reading interventions. The center’s reports summarize the principles behind the programs, evaluate the content, and identify the elements of the programs that do and do not align with research conclusions. The reports do not make claims about programs’ effectiveness.

Beyond those appraisals, however, publishers are unlikely to see their studies in peer-reviewed journals—the stamp of approval to many in the field. The publishing process is often lengthy and could distract researchers from their other projects, according to Ms. Baughmann.

The new focus on research evidence “places a certain amount of responsibility on the publishers to hire good academic researchers willing to work to the highest standards,” she said. “But we don’t anticipate being able to get [their work] published in a journal.”

Related Tags:


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Data Webinar
Education Insights with Actionable Data to Create More Personalized Engagement
The world has changed during this time of pandemic learning, and there is a new challenge faced in education regarding how we effectively utilize the data now available to educators and leaders. In this session
Content provided by Microsoft
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Accelerate Learning with Project-Based Learning
Earlier this year, the George Lucas Educational Foundation released four new studies highlighting how project-based learning (PBL) helps accelerate student learning—across age groups, multiple disciplines, and different socio-economic statuses. With this year’s emphasis on unfinished
Content provided by SmartLab Learning
School & District Management Live Online Discussion Principal Overload: How to Manage Anxiety, Stress, and Tough Decisions
According to recent surveys, more than 40 percent of principals are considering leaving their jobs. With the pandemic, running a school building has become even more complicated, and principals' workloads continue to grow. If we

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Curriculum From Our Research Center Privacy, Porn, and Parents in the Room: Sex Education's Pandemic Challenges
After more than a year of instructional shifts and social isolation, students need sex education that is media-savvy and relationship-wise.
7 min read
Conceptual image of students feeling isolated, but also trying to connect.
Mary Haasdyk for Education Week
Curriculum Calls to Ban Books by Black Authors Are Increasing Amid Critical Race Theory Debates
Books about race and the experiences of Black Americans are being challenged by parents who claim they make white children feel uncomfortable.
8 min read
Fans of Angie Thomas, a Jackson, Miss., resident whose book, "The Hate U Give," has been on a national young adult best-seller list for over 80 weeks, show off their copies at a reception and book signing for the author, in Jackson on Oct. 10, 2018. Thomas' novel has crossed over to a wider audience than simply young adults. The reception honored her writing as well as the coming release of the big screen adaption of the first novel.
The young adult best-seller "The Hate U Give" was one of the top 10 most challenged books of 2020.
Rogelio V. Solis/AP
Curriculum District That Banned Diverse Books Reverses Its Decision After Pushback
A Pennsylvania district voted unanimously to reinstate a four-page list of resources from some of today's most acclaimed creators of color.
Tina Locurto, The York Dispatch, Pa.
3 min read
Image of books on a library shelf.
Curriculum He Taught About White Privilege and Got Fired. Now He's Fighting to Get His Job Back
Matthew Hawn is an early casualty in this year's fight over how teachers can discuss with students America's struggle with racism.
13 min read
Social studies teacher Matthew Hawn is accused of insubordination and repeated unprofessional conduct for sharing Kyla Jenèe Lacey's, 'White Privilege', poem with his Contemporary Issues class. Hawn sits on his couch inside his home on August 17, 2021.
Matthew Hawn is accused of insubordination and repeated unprofessional conduct for lessons and materials he used to teach about racism and white privilege in his Contemporary Issues class at Sullivan Central High School in Blountville, Tenn.<br/>
Caitlin Penna for Education Week