A federal study of curriculum materials used in two “enhanced” after-school programs has found that a math program produced significant gains in student achievement while a reading program did not.
Students who took part in the enhanced math program for one year showed significantly more improvement than their peers in a regular after-school program. A second year in the program, however, produced gains no greater than those for students enrolled in a regular after-school program. The study examines students in grades 2 through 5.
The math program, Mathletics, was developed especially for the study by Harcourt School Publishers, based in Orlando, Fla., and was used in 15 after-school centers. The reading program, called Adventure Island, was also developed for the study, by the Baltimore-based Success for All Foundation, and was taught in 12 after-school centers. The centers were located in rural, urban, and suburban areas in 10 states.
“The impact translates into an extra month’s worth of math—the ‘enhanced’ group grew more, by about a month’s worth of learning,” said Fred Doolittle, a co-author of the study, along with Alison R. Black.
The study was commissioned by the U.S. Department of Education’s Institute of Education Sciences and conducted by the New York City-based MDRC, a nonprofit organization known for its large-scale evaluations of social programs.
Not only did the study show that the enhanced reading program had no effect in the first year of the after-school program, but it found that students taking part in it performed significantly worse in reading during the second year, compared with students in a regular after-school program.
Implementation Problems
Mr. Doolittle said the study wasn’t designed to provide an explanation for the mixed results across the two subjects. “It wasn’t set up to be a horse race between reading and math,” he said.
But, he added, the researchers found some implementation issues with the reading program. Teachers had difficulty keeping up with the fast pace of the reading program, a problem that didn’t exist with the math program.
In addition, said Ms. Black, the goal of the study was to work with students who were performing below grade level. It turned out, she said, that the students in the reading sample were further below grade level than those in the math sample, which could have contributed to the different results in the two subject areas.
Mr. Doolittle said the study’s overall findings—that an after-school math program had an effect while an after-school reading program did not—are consistent with those of other research. “The proportion of positive findings in math is higher,” he noted.
Robert Slavin, the director of the Center for Research and Reform in Education at Johns Hopkins University in Baltimore and the chairman of the Success for All Foundation, said the study was a good one and he accepts its findings. As a result, he said the foundation would not disseminate the Adventure Island materials.
But Mr. Slavin said he believes the study says more about after-school programs than it does about the enhanced reading program. “What it reinforces is the importance of focusing on what teachers do during the regular school day, rather than expecting that a relatively brief after-school experience is going to make a big difference.”
At the same time, he said, “there may be reason to have after-school programs other than to improve reading outcomes, and those are fine.”
Mr. Slavin said one likely reason the enhanced math program had an effect while the enhanced reading program didn’t is that math tends to be easier to measure.
“Giving kids more practice or explanation of a well-defined problem-solving skill, for example, is quite a different thing than teaching kids reading, which is more difficult to measure and just a bigger task that happens over many years.”
The first year of the study included a sample of about 4,000 students in the treatment and the control groups; the second year included about half that number.
