School & District Management

Math Programs Seen to Lack a Research Base

By Debra Viadero — November 23, 2004 4 min read
  • Save to favorites
  • Print

Most of the off-the-shelf mathematics programs used in middle schools across the country have little or no rigorous evidence attesting to their effectiveness, concludes a federal research review released last week.

The U.S. Department of Education analysis was based on a review of studies undergirding 44 math programs used in grades 6-9, including some of the nation’s most popular textbooks for those grade levels.

The report, “Curriculum-Based Interventions for Increasing K-12 Math Achievement-Middle School,” is scheduled to be available online from the What Works Clearinghouse. ()

The researchers found only five that had a research record strong enough to meet their standards. Of those, just two—a pair of computer-based algebra programs called I CAN Learn Mathematics and Cognitive Tutor—had studies showing that students actually learned more with their programs compared with other programs.

“I see this really as a wake-up call on the state of research in mathematics,” said Tom Loveless, a researcher at the Brookings Institution, a Washington think tank. Mr. Loveless was invited to comment on the findings during a press conference here last week organized by the Education Department.

The department was eager to trumpet the findings because the report was the first broad topic review to come out of its new What Works Clearinghouse. Part of the Bush administration’s plan to transform education into an evidence-based field, the clearinghouse vets research on educational programs, policies, and products, and publishes the results on a Web site where practitioners and policymakers can easily find them. The site, which cost $18.5 million to set up and run, has been operating since July. Up till now, though, it has only produced reviews of individual studies. (“‘What Works’ Research Site Unveiled,” July 14, 2004.)

Federal officials said another reason they found so few strong studies for middle school math programs is that they purposely set the standards so high. As in medical research, the studies that get the highest marks from the clearinghouse are those that use a research method known as randomized control trials. Those are experiments in which large numbers of subjects are randomly assigned to either a test group or a comparison group. Quasi-experimental studies—those that use demographically matched comparison groups to test out an intervention—also qualify as evidence but they are rated lower.

“We hope this will have the effect of bootstrapping efforts in the field so that high-quality studies get done in the future,” said Grover J. “Russ” Whitehurst, the director of the Institute of Education Sciences, the Education Department branch that oversees the clearinghouse.

‘Jury Is Still Out’

However, some experts wondered whether those standards were too much of a leap for educational research to be useful to the field—at least for now.

Research Void

A new report by the federal Institute of Education Sciences concludes that no solid research is available that examines the effectiveness of these math programs.

• Addison-Wesley Mathematics basal program

• AGS Publishing

• A+ny where Learning System

• Academic Systems’ Interactive Math Curriculum

• Math Applications and Connections

• Heath Mathematics Connections

• Heath Math Passport

• Hold Middle School Math

• Key Math Teach and Practice

• Larson Developmental Math Series
• Lightspan Achieve Now

• Math Advantage

• Math Learning Center

• Mathematics Plus

• MathScape: Seeing and Thinking Mathematically

• Macmillan/McGraw Hill

• Middle Grades Math

• Middle School Math

• Middle School Mathematics Through Applications Program

• The New Century Integrated Instructional System

• Real Math basal mathematics program

• Scott Foresman Math Diagnostic & Intervention System

• SimCalc: Cognitive Foundations for a Multiplicative Structures Curriculum

• Singapore Mathematics

SOURCE: Institute of Education Sciences

“The realities of schools are so complex that doing the kind of studies we thought we needed may simply not be practical,” said Cathy Seeley, the president of the National Council of Teachers of Mathematics, a professional group based in Reston, Va. “You can’t control all the variables in schools like you can in a medical setting.”

The researchers turned up no credible evaluations at all, for example, for math programs published under such popular textbook imprints as Addison-Wesley, Heath, Holt, Glencoe, and Scott Foresman. Singapore Mathematics, a program favored by some prominent mathematicians, also landed on the “no studies” list as did the Middle School Mathematics Through Applications Program, which had been labeled “promising” in 1999, the last time federal education officials attempted to review mathematics curricula.

Phoebe H. Cottingham, the institute’s commissioner of education evaluation and regional assistance, said the fact that those programs have little evidence does not necessarily mean they don’t work.

“That just means the jury is still out,” she cautioned. “This is not an endorsement. It is a rating system of the evidence for an intervention.”

It’s also difficult to build a research track record, publishers said, when programs undergo continual revisions, making them moving targets for studies.

For instance, a spokeswoman for Pearson School Group, the Upper Saddle River, N.J.-based publisher of the Middle Grades Math series published under the Scott Foresman/Addison-Wesley imprint, said her company has already replaced that program with a new one.

800 Reports Reviewed

In all, the study looked at 800 reports, screening out 77 in the first cut. Only 11 studies, though, made the final grade because they met clearinghouse standards for well-done randomized or quasi-experimental studies.

The positive results for I CAN Learn Mathematics, for example, came in one small, randomized study conducted in a northwestern Georgia middle school and three comparison studies in urban districts in Florida and Louisiana. The program is currently used in about 500 schools.

The other computer-based algebra program that produced positive results, Cognitive Tutor, is used in 1,500 schools nationally. The three remaining programs with enough strong studies to get on the clearinghouse’s short list are: the Connected Mathematics Project, a program devised at Michigan State University in East Lansing; The Expert Mathematician, a computer-based program designed by James J. Baker, a Minneapolis educator; and Saxon Math, a popular program now owned by Harcourt Achieve of Austin, Texas, a division of Harcourt.

To be more useful to educators, federal officials might consider casting a broader net in future study reviews, said Glen H. Harvey, the chief executive officer of WestEd, a San Francisco-based research organization.

“There are only two studies here that show positive impact as I can tell. So what guidance does that provide a school?” she said. “They need to maintain their standards of rigor while giving policymakers a way to look at best bets.”

Related Tags:

A version of this article appeared in the November 24, 2004 edition of Education Week as Math Programs Seen to Lack a Research Base

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Reading & Literacy Webinar
(Re)Focus on Dyslexia: Moving Beyond Diagnosis & Toward Transformation
Move beyond dyslexia diagnoses & focus on effective literacy instruction for ALL students. Join us to learn research-based strategies that benefit learners in PreK-8.
Content provided by EPS Learning
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School Climate & Safety Webinar
How to Use Data to Combat Bullying and Enhance School Safety
Join our webinar to learn how data can help identify bullying, implement effective interventions, & foster student well-being.
Content provided by Panorama Education
Classroom Technology Live Online Discussion A Seat at the Table: Is AI Out to Take Your Job or Help You Do It Better?
With all of the uncertainty K-12 educators have around what AI means might mean for the future, how can the field best prepare young people for an AI-powered future?

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Quiz Quiz Yourself: How Much Do You Know About The District Academic Officer Persona?
The district academic officer is a key player when it comes to purchasing. Test your knowledge of this key buyer persona and see how your results stack up with your peers.
School & District Management Download Navigating Tense Conversations at Work: A Guide for Educators (Downloadable)
A downloadable guide to help educators navigate polarizing conflicts.
3 min read
Polar opposite hands hold u a triangular flag. Teamwork, resolution, truce.
Vanessa Solis/Education Week + DigitalVision Vectors
School & District Management Opinion Education Leaders, You Can't Do Your Job in Isolation
An unusual way to begin a leadership team retreat leads to a deeper understanding of why teachers and leaders need to work together.
5 min read
Screen Shot 2024 10 01 at 7.05.34 AM
Shutterstock
School & District Management Educators Rush to Get Food and Shelter to Their Students After Hurricane Helene
Districts slammed by an unprecedented natural disaster have become shelter zones for their communities.
7 min read
A passerby checks the water depth of a flooded road, Saturday, Sept. 28, 2024, in Morganton, N.C. Torrential rain from Hurricane Helene left many area streets flooded. In addition, traffic lights are inoperable due to no power, with downed power lines and trees.
A passerby checks the water depth of a flooded road, Sept. 28, 2024, in Morganton, N.C. Torrential rain from Hurricane Helene left area streets flooded, and strong winds downed power lines and trees. Schools have become hubs to support their communities as recovery begins.
Kathy Kmonicek/AP