The National Education Policy Center at the University of Colorado Boulder this week unveiled its Bunkum Awards for Shoddy Research, “recognizing the lowlights in education research” conducted by think tanks in 2014. Its leaders also reflected on common problems that keep cropping up in research in the nine years the Bunkum has been awarded.
As a tongue-in-cheek warning, the center also noted in its announcement of the “awards,” that:
Judging by the bunk we've been reviewing—we'll admit to feeling a little uneasy about the possibility that we may actually be enticing think tanks to produce awful reports. So, just to be clear to any think tank folks who may be reading this: Receipt of these awards isn't something to be proud of. Please stop competing to out-bunk one another."
Learning from Mistakes
After nine years of pointing out the problems with popular education reports, Kevin Welner, the NEPC’s director, and William J. Mathis, the center’s managing director, told me some clear problems come up again and again:
- Mistaking correlation for causation.
- Not accounting for potential reasons for results, such as selection bias.
- Taking too large of leaps from a finding in one area to a conclusion that isn’t neccessarily supported by the results. “I’m remembering a report years ago about the dropout problem in Florida,” Mr. Welner said. “The first part of the report was relatively solid, providing good evidence about the problem. It then leaped over to, ‘Therefore, we need vouchers.’”
- Using a small or barely-statistically significant result to recommend a broad policy change.
- Starting a study from an ideological point rather than an objective hypothesis. “In its purest form, there are many ‘evidence free’ reports that rely on gushy narratives but no real evidence,” Mr. Mathis said.
“We don’t consider the Think Tank Review Project reviews to be the final word, nor is our goal to prevent think tanks’ participation in the public dialogue over school reform,” Mr. Welner said. “That dialogue is, in fact, what we hope to improve and encourage. In our view, the best ideas come about through rigorous but substantive critique and debate, and ideas presented in think tank reports should be part of the process.”