Federal Study Finds No Edge for Students Using Technology-Based Reading and Math Products

By Andrew Trotter — April 04, 2007 4 min read
  • Save to favorites
  • Print

A major federal study of reading and mathematics software has found no difference in academic achievement between students who used the technology in their classrooms and youngsters who used other methods.

The $10 million study of 15 educational software products is the most extensive federal study yet to follow methods that the U.S. Department of Education considers scientifically rigorous.

The report on the first year of a two-year study is expected to be presented today to Congress, which commissioned the study in the No Child Left Behind Act. The report is slated to be posted on the Education Department’s Web site tomorrow, according to sources.

The lead researcher for the study was Mark Dynarski of Mathematica Policy Research Inc., an independent, for-profit research organization based in Princeton, N.J. Also taking part was the Menlo Park, Calif.-based SRI International Inc.

“Because the study implemented products in real schools and with teachers who had not used the products, the findings provide a sense of product effectiveness under real-world conditions of use,” the report says.

The study followed an experimental design that was drawn up with the help of leading education researchers and vetted by the Education Department’s Institute for Education Sciences.

Software products were selected in four categories: 1st grade early reading, 4th grade reading comprehension, 6th grade pre-algebra, and 9th grade algebra. While the companies involved will receive results for their own products, the public will see only aggregated findings for the four categories of programs.

The products were chosen from more than 160 products submitted in 2003 by their developers. One selection criteria was that the product had shown previous evidence of effectiveness.

The developers or publishers of the software are well-known in K-12 education: PLATO Learning Inc., Carnegie Learning Inc., Houghton Mifflin Co., Scholastic Inc., iLearn, Leapfrog Schoolhouse, Autoskill International Inc., Pearson PLC, and Headsprout Inc.

Test Scores Compared

The study compares classes overseen by teachers who used the technology-based products with those of other teachers who used different methods. Those other approaches also included the use of technology in some cases, though the selection team tried to avoid schools that used technologies similar to the ones being studied.

Student achievement in math or reading was measured by standardized-test scores, with complete data collected for 9,424 students.

The study team recruited the school districts, favoring districts that had low student achievement and large proportions of students in poverty. Researchers sought districts and schools that did not already use products like those being studied.

The school districts identified schools, based in part on their having adequate technology infrastructure and being involved in other initiatives.

The finding of no gains from the software is sure to complicate the efforts of advocates of technology in education, who are lobbying the Bush administration and members of Congress to continue providing millions of dollars annually in support for classroom technology.

The release of the study will take place about a year after its original target of spring 2006. It appears at a time when the Education Department officials are speaking more about education technology than they have in recent years.

Several officials, including Education Secretary Margaret Spellings, have commented recently that the public has not seen much of a return on the federal government’s investments of millions of dollars in grants to states and school districts for educational technology.

The findings may be disturbing to the companies that provided their software for the trial. As rumors spread this week about the study’s findings, companies revived old complaints about how the study has been conducted—particularly the government’s decision not to disclose individual performance results for the 15 computerized curriculum packages being studied.

Company officials point out that the blended findings also blur the results for any specific software product, which may have fared better than others in its category.

Random Assignment

Teachers who volunteered for the trial in the selected schools were randomly assigned to use the products or not. The teachers were expected to be equivalent in their teaching skills in both groups, the report says. All told, 439 teachers took part in 132 schools.

Teachers who used the software products implemented them as part of their reading or math instruction. Teachers in the control group were expected to teach reading or math as they would have normally, possibly using some form of technology.

How well the technology is implemented is a critical question in any evaluation, so the study placed trained classroom observers to assess the quality of implementation of the products.

Each classroom was visited three times during the school year, with observers following a common format for their observations. The teachers were also interviewed about implementation issues and filled out questionnaires.

Technical glitches and similar problems cropped up—unavoidable in educational technology—but were mostly of the minor variety and were easily corrected or worked around. Nearly all the teachers said they would use the products again.

Related Tags:


Student Well-Being K-12 Essentials Forum Boosting Student and Staff Mental Health: What Schools Can Do
Join this free virtual event based on recent reporting on student and staff mental health challenges and how schools have responded.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
Practical Methods for Integrating Computer Science into Core Curriculum
Dive into insights on integrating computer science into core curricula with expert tips and practical strategies to empower students at every grade level.
Content provided by

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Science Opinion Want to Get Students Excited About Physics? Try Using a Glass of Water
The physics field needs more new voices. Celebrating the everyday awe around you is a good place to start, writes a Yale physicist.
Charles D. Brown II
4 min read
A glass with clean clear water and sharp shadows stands on a white wood table with subtle superimposed molecules.
Istock/Getty Images + Education Week
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Science Sponsor
The Future of STEM Education is Social Impact
In today’s world where so many different elements of the news and of our culture are competing for students’ attention, one of the hardest things to do in grades 6 to 12 science education is to show young people the impact of what they are learning and why it matters. For a teacher, it’s a true gift to make space in your classroom – and in your curriculum – for letting students see how what they are learning can impact society in a truly positive way.
Content provided by Samsung
Group photo of: Holly Warren, Kirstin Milks, Karen Jepson-Innes, Kiefer Kettenis, Emma Shaw
Bloomington High School South's Samsung Solve for Tomorrow student team showcased their Temp Mural project at the Wonderlab Museum ceremony, where they unveiled a mural art piece created with highly reflective, climate-positive barium sulfate paint, aimed at mitigating the effects of global warming. Left to right:<br/>• Holly Warren, interim director of economic and sustainable development at the City of Bloomington<br/>• Kirstin Milks, science teacher at Bloomington High School South<br/>• Karen Jepson-Innes, Executive Director, WonderLab Museum of Science and Technology<br/>• Kiefer Kettenis, Temp Mural team member<br/>• Emma Shaw, Temp Mural team member
Garrett Ann Walters
Science U.S. Teachers Lag Behind Global Peers in Teaching About Sustainability. Here's Why
Many say they want materials and supports to help them weave topics like clean energy across subjects.
4 min read
Teacher talking to students about recycling in the classroom at school
Science Q&A How High School Students Are Making STEM Education Accessible for Younger Kids
Team STEAM is a program where high school students help elementary students develop STEM skills.
3 min read
Students from MC2 STEM High School in Cleveland critique their classmates’ projects for an event that blends STEM and art on March 16, 2017.
Students critique their classmates’ projects for an event that blends STEM and art in Cleveland on March 16, 2017.
Allison Shelley for All4Ed