Curriculum

‘What Works’ Reviewers Find No Learning Edge for Leading Math Texts

By Debra Viadero — January 23, 2007 | Corrected: February 22, 2019 6 min read
  • Save to favorites
  • Print

Corrected: This story on bid specifications for a new five-year What Works Clearinghouse contract should have said that the current contract is worth $23.4 million. The story also should have made clear that while the clearinghouse gives its highest ratings to randomized control studies, it also considers other kinds of studies that compare treatment and control groups.

Includes updates and/or revisions.

As the federal What Works Clearinghouse rolls out long-awaited ratings on the effectiveness of math programs for the elementary grades, one trend is becoming clear: Most major commercial textbooks can’t yet muster the proof that they are any better than their competitors at improving student achievement.

Of four reviews published by the online clearinghouse since September, only one elementary school math program has received even a qualified nod from evaluators for its research record.

Yet while publishers and textbook evaluators are concerned about the message those lukewarm effectiveness ratings may send, they also say the ratings may have more to do with the clearinghouse’s strict reporting system than with the programs themselves.

The What Works site says a handful of rigorously conducted experiments show that Everyday Mathematics,published by Wright Group/McGraw-Hill of DeSoto, Texas, has “potentially positive effects” on achievement compared with more traditional math programs.

The other programs—Houghton Mifflin Mathematics, Saxon Elementary School Math,and Scott Foresman-Addison Wesley Elementary Mathematics—were found in What Works reviews to have “no discernible effects” on learning.

Together, the four programs represent about half the U.S. market for elementary school math textbooks, according to a 2005 survey by Robert M. Resnick, the founder and president of Education Market Research, of Rockaway Park, N.Y.

The What Works project favors randomized-control trials—experiments in which a program or practice is randomly assigned to either a treatment or a control group—and ignores or discounts most other kinds of studies. (“‘One Stop’ Research Shop Seen as Slow to Yield Views That Educators Can Use,” Sept. 27, 2006.)

That’s a problem for publishers, because the comparison groups in their cases tend to consist of schools or classrooms using other popular textbooks.

Math Marketing

SOURCE: Education Market Research

BRIC ARCHIVE

“It’s very difficult to find significant differences in a one-year study when you’re looking at two programs teaching essentially the same content,” said Marcy L. Baughman, the director of academic research for Pearson Education. The Boston-based company publishes the Scott Foresman-Addison Wesley Elementary series, which in December became the latest math-textbook program to be reviewed by U.S. Department of Education-sponsored clearinghouse.

Rebecca S. Herman, the clearinghouse’s project director, said such criticisms miss the point. The clearinghouse, which is funded by the department’s Institute of Education Sciences and housed at the American Institutes for Research, a nonprofit organization based in Washington, is designed to identify the most-effective educational programs and strategies, she noted.

“The point is: Does one work better? Is one really a star?,” said Ms. Herman, the project director for the clearinghouse. “In math, there’s never going to be a situation where students are not getting a math textbook or some sort of math curriculum.”

Misleading Impressions

Publishers and some scholars complain, though, that spare judgments such as “no discernible effect” or “potentially positive effects” give the wrong impression.

“If a layperson reads the ratings, he or she is going to think a kid will not improve with this program,” said Mariam Azin, the president of Planning Research and Evaluation Services Associates, an independent group in Jackson Hole, Wyo., that has evaluated several commercial textbook programs. “Well, that’s not true.”

An experiment could turn up no effects if both the textbook under study and the program with which it is being compared are equally good—or equally bad—at improving student learning.

Ms. Herman said clearinghouse officials are aware of such concerns, which publishers have been voicing since the ratings system was in the planning stages. In response, she said, federal reviewers have moved some information about the composition of the comparison groups from a technical appendix to the body of the main report.

Still, the clearinghouse should go further, argued Bill Wilkinson, the vice president for research for Harcourt Achieve, the Austin, Texas-based company that publishes the Saxonmath program. He said the What Works site should also put learning-growth data from studies that are not randomized into its main reports to give educators and policymakers more information.

“When you’ve got so many math programs showing ‘no discernible effects,’ it really makes it hard for the education community to make judgments,” Mr. Wilkinson said, “and, really, the clearinghouse is there for the education community to make judgments.”

Practical Impact

What’s not yet clear, though, is whether the ratings will carry any weight with the people charged with selecting instructional materials for schools. Texas and California—two of the three largest states that adopt textbooks at the state level—are scheduled to approve new elementary mathematics series this year.

Adding It Up

SOURCE: Education Market Research

Leading Programs’ Market Shares Grades 3-5

The curriculum directors in those states said that the official state criteria for their upcoming adoptions require publishers to ensure that their programs are research-based. Yet they do not require publishers to submit proof that their products work, although that could change in Texas, where lawmakers this spring plan to rewrite the state’s textbook-adoption policies.

Some educators at the district level, for their part, have been demanding that kind of evidence for a few years, in part because of requirements under the No Child Left Behind Act that programs be research based, according to publishers. Mr. Wilkinson of Harcourt said he was even asked recently to provide research showing that students learn more than they otherwise would when they use the flashcards his company produces.

“If there’s a place where they take an objective view of the research, that’s something I would certainly be interested in,” said Lianne B. Jackson, the textbook and standards program coordinator for Nevada’s 65,000-student Washoe County school district, which expects to adopt new elementary math texts over the next few years.

But the bottom line for most districts and states is ensuring that the instructional programs they choose adhere to local and state standards on what to teach and when to teach it.

“If you weren’t significantly outperforming your competitor,” Ms. Baughman said, “I don’t know that it would keep a state or district from adopting you.”

The What Works Clearinghouse is part of a movement at the federal level to spur demand for rigorous research attesting to the effectiveness of educational programs.

The No Child Left Behind Act, for instance, requires that schools receiving funds under the law rely on “scientifically based evidence” in choosing a wide variety of educational programs, products, and practices.

Textbook publishers, in turn, have responded by commissioning randomized trials of their products. While the growth in company-sponsored studies has raised some eyebrows, experts by and large see the mounting piles of evidence as a positive development. (“Houghton Mifflin’s Sale to Software Maker Reflects Trend,” Dec. 6, 2006.)

“Before, it was definitely more loosey-goosey,” said Ms. Azin of Planning Research and Evaluation Services Associates. But she believes the clearinghouse should expand the range of research that qualifies as evidence—a change that clearinghouse operators say they are not likely to favor.

“It’s also dangerous to go to a more formulaic approach, which might be extreme,” Ms. Azin said.

Coverage of education research is supported in part by a grant from the Spencer Foundation.
A version of this article appeared in the January 24, 2007 edition of Education Week as ‘What Works’ Reviewers Find No Learning Edge For Leading Math Texts

Events

Classroom Technology Webinar How Pandemic Tech Is (and Is Not) Transforming K-12 Schools
The COVID-19 pandemic—and the resulting rise in virtual learning and big investments in digital learning tools— helped educators propel their technology skills to the next level. Teachers have become more adept at using learning management
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Building Teacher Capacity for Social-Emotional Learning
Set goals that support adult well-being and social-emotional learning: register today!


Content provided by Panorama
Jobs October 2021 Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Curriculum Holocaust Books Must Be Countered With ‘Opposing’ Views, Texas School Administrator Says
Teachers were told they're required to offer alternative information for debated and controversial topics. The district later apologized.
Brian Niemietz, New York Daily News
2 min read
A book about David Boder's recordings of concentration camp survivors sits next to one of his wire recorders on display at the University of Akron's Center for History of Psychology on Thursday, Feb. 2, 2017 in Akron, Ohio.
A book about David Boder's recordings of concentration camp survivors sits next to one of his wire recorders on display at the University of Akron's Center for History of Psychology on Thursday, Feb. 2, 2017 in Akron, Ohio.
Phil Masturzo/Akron Beacon Journal via AP
Curriculum From Our Research Center Privacy, Porn, and Parents in the Room: Sex Education's Pandemic Challenges
After more than a year of instructional shifts and social isolation, students need sex education that is media-savvy and relationship-wise.
7 min read
Conceptual image of students feeling isolated, but also trying to connect.
Mary Haasdyk for Education Week
Curriculum Calls to Ban Books by Black Authors Are Increasing Amid Critical Race Theory Debates
Books about race and the experiences of Black Americans are being challenged by parents who claim they make white children feel uncomfortable.
8 min read
Fans of Angie Thomas, a Jackson, Miss., resident whose book, "The Hate U Give," has been on a national young adult best-seller list for over 80 weeks, show off their copies at a reception and book signing for the author, in Jackson on Oct. 10, 2018. Thomas' novel has crossed over to a wider audience than simply young adults. The reception honored her writing as well as the coming release of the big screen adaption of the first novel.
The young adult best-seller "The Hate U Give" was one of the top 10 most challenged books of 2020.
Rogelio V. Solis/AP
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Curriculum Sponsor
Executive Q&A: Building Back Better: The Future of Digital Curriculum
Edmentum’s Chief Strategy Officer Amanda Kocon and Chief Product Officer Todd Mahler discuss the future of digital curriculum.
Content provided by Edmentum Inc.