Today’s reformers rightly demand to know how well teachers are teaching and students are learning. The trouble is that statistics are the only evidence they will accept. Numbers by themselves, however, are not sacrosanct. They are subject to manipulation by special interests and to misinterpretation by unsophisticated readers.
I first learned how this is done by a piece that was published in Harper’s in October 1964. Otto Friedrich wrote “There Are 00 Trees in Russia.” He showed how even “the most careful checking of facts by a platoon of researchers does not necessarily add up to the whole truth.” He explained how what he calls “the zip” allows the writer to focus on the drama of the story, while indiscriminately using facts. The title of his piece was an example of the way a “wholly improbable but wholly unchallengeable statistic” can be foisted on readers.
Most recently, I was reminded of how numbers can be abused after reading Proofiness (Viking, 2010) by Charles Seife. The author lays out a compelling case for the way numbers are distorted in order to create the impression of truth. Seife calls this process proofiness, which is “the art of using bogus mathematical arguments to prove something that you know in your heart is true - even when it’s not.”
Seife cites three main strategies that are employed: Potemkin numbers (meaningless statistics), disestimation (distorted numbers), and fruit- packing (selective data). Although these techniques are not limited to education by any means, they are particularly suited to the reform movement, which insists on numbers.
Despite the publicity that Seife is getting, he is not the first to write a book about the subject. In 1954, Darrell Huff wrote How to Lie with Statistics (W.W. Norton). One of his chapters was titled “How to Statisticulate.” Like Seife, he shows how easy it is to intimidate anyone by using numbers, graphs and charts. And like Seife, Huff uses examples from politics, economics and health, as well as from education, to make his points.
Yet for the most relevant book about the connection between statistics and education, I still think that Reading Educational Research (Heinemann, 2006) by the late Gerald Bracey is the best of the three. He shows how such phenomena as Simpson’s Paradox continue to cause erroneous inferences to be made about teaching and learning. For readers unfamiliar with data interpretation, Bracey lists 32 principles that provide the basis for a better understanding of educational issues in the news, or at least to create a healthy skepticism.
It’s also helpful to keep in mind that statistics were originally called political arithmetic. The term changed because those collecting data were known as statists, since they worked for the state. So despite all the data being published about education today, we would all be well advised to remember Friedrich’s caveat that was written in 1964: "...neither facts nor news is necessarily the truth.”