Thoughtful practitioners often become confused when they look to the research evidence to back up interventions aimed at preventing crime, reducing poverty, or improving schools. One study says a program works. Another maintains it doesn’t.
That’s why the Campbell Collaboration was born four years ago. Made up of scientists, statisticians, and policymakers from around the world, the group’s aim is to shepherd systematic reviews of studies and distill nuggets of reliable information practitioners could use.
Last week, the group got an early glimpse at the nuggets its efforts are producing. More than 250 practitioners and researchers from 16-plus countries came to the nation’s capital to hear “reviews of reviews” of studies on teenage-pregnancy prevention, justice and anti-crime programs, peer-assisted learning, block scheduling, after-school programs, and other interventions.
Most of the reviews have not yet earned the group’s official stamp of approval. But, by showcasing the studies in its pipeline and others, the group is hoping to convince skeptics that its approach is both feasible and potentially useful.
“There are challenges, but I think the growing message from the field is that these are doable,” said Susan Goerlich Zief, a doctoral student at the University of Pennsylvania in Philadelphia and the co-author of a study discussed here.
Ms. Zief’s pilot study of after-school programs, technically called a “test-bed” review, focused only on randomized controlled trials—in other words, studies in which subjects are randomly assigned to either experimental or comparison groups. Of the eight such studies found by Ms. Zief and her research partner, Sherri Lauver, five met their review criteria.
Once they synthesized the studies’ effects, the researchers concluded that after-school programs may be a wash when it comes to improving students’ grades or reducing the number of hours they spend unsupervised.
“Do we want to say these programs don’t work?” asked Ms. Lauver.
|
Not yet. Like many of the researchers who presented preliminary work here, the Pennsylvania team said many factors could explain the null findings. Such factors include: too few rigorous studies, varied levels of program implementation, differences in the way studies measured the same outcomes, and differences among the programs themselves.
“We’re learning a lot,’' said Robert F. Boruch, Campbell’s co-chairman. “It’s going to take time to get this stuff pumped out.”
Gauging Effects
Other study reviews presented at the meeting also suggest that:
- Block scheduling has “no harmful effect” on high school students’ academic achievement in the short run. That conclusion comes from a review of seven studies by Chance W. Lewis and Marc A. Winokur, both researchers from Colorado State University in Fort Collins. They added that schools might have reasons other than bolstering achievement for implementing those programs.
- Peer-assisted learning, a process involving students in collaborative, group learning, can produce learning gains among elementary pupils—particularly those who are poor or members of minority groups. In a study first published last year, researchers Marika D. Ginsburg-Block of the University of Delaware in Newark and Cynthia A. Rohrbeck of George Washington University here base that conclusion on an analysis of 81 studies.
- Teenage-pregnancy-prevention programs seem to have little effect on reducing pregnancy rates, delaying sexual initiation, or promoting contraceptive use. For her pilot review, Lauren Scher of the University of Pennsylvania analyzed 41 randomized studies. But she, too, said policymakers shouldn’t interpret her findings to mean such programs won’t work.
‘Informed Choices’
Organizers were not discouraged that so many of the findings seemed to show no effect. They noted that the same happened with early reviews from the Cochrane Collaboration, the medical- research group on which the Campbell Collaboration is modeled.
The going may also be slow, conference-goers said, because the rigorous “meta-analyses” the group is hoping to produce can be challenging, time-consuming, and hard to get funded.
That so many of the reviews here focused on education, however, reflects the high priority Washington policymakers and some foundations have put on generating “scientifically based research” in the field. The Campbell group, in fact, is also a partner in developing the U.S. Department of Education’s What Works Clearinghouse, a parallel effort to vet research for the field.
|
Whether the group’s work will translate into changes in social policy remains to be seen. Successive Campbell reviews of “Scared Straight’’ programs, which use convicts to describe harsh truths about prison life to potential juvenile offenders, suggest those programs may encourage a life of crime, rather than deter it. Even so, Illinois lawmakers voted last year to continue the program in their state.
“I don’t know that anyone would realistically expect any of this would lead policymakers to shut down a program and start a whole new set of programs,” said Dennis W. Cheek, a spokesman for the collaboration. “But we would hope that they at least would make informed choices.”
