I have a story in today’s online edition of Education Week that describes a spate of disappointing findings coming out of the large-scale, randomized studies that the Institute of Education Sciences has been underwriting in recent years.
Experts contend that randomized studies—in other words. experiments in which participants are randomly assigned to either treatment or control groups—are the “gold standard” for determining what works in education and in many other fields. So there was much hope that this new generation of studies would point to some strong programs that practitioners would feel confident about using in their own schools.
But, of the eight such study reports posting results this year, six offer a mix of findings that can be characterized as showing mostly no effects for the programs tested.
But “no effects” are in the eye of the beholder, I guess. As one reader pointed out to me, four of the eight studies contain at least one positive finding. For example, one study tested 10 different commercial software programs, finding one that consistently worked better than what teachers were already doing in their classrooms. Well, that’s certainly positive for that particular program but not so much so for the other nine.
Still, this reader notes, a “hit rate” of 50 percent is high compared to some other fields that have embraced randomized controlled trials, or RCTs. In the pharmaceutical industry, it’s estimated that only 10 percent to 12 percent of studies of new drugs produce positive effects in clinical trials.
What do you think? Is the high rate of “no effects” coming out of these federal education studies cause for concern or for celebration?