Assessment Opinion

Data, ‘Crises’, and the Way We Interpret Them

By Deborah Meier — November 01, 2007 4 min read

Dear Diane,

You’re right! There is data and then there are various ways to interpret it. The NAEP data is data. But given that the average reader—such as me—doesn’t know a lot about how the scores were derived, making sense of it requires interpretation. For example: (1) what kind of questions are on the tests? (2) how great are the differences in right/wrong answers that separate the benchmarks? and above all (3) how valid are the categories proficient, advanced? Some of this requires statistical sophistication, some just more information than is readily available. Most of the time most of us are satisfied if we feel confidence in those that made these decisions to start with. We leave it there.

There’s no question that NAEP has helped us to recognize the absurdity of all the other scores the states are using. That’s your main point, and I agree with it.

I feel the same way about the international comparisons. Gerald Bracey almost yearly does a report demonstrating how many ways you can interpret the same data, particularly re. math. I just read a new critique from Business Week by Vivek Wadhwa in which he makes the claim that the U.S. looks very good in science and literacy, and pretty good in math! He goes through it pretty thoroughly and has the credentials to do so. But I grabbed onto it, I know, because I was looking for data to support my claim that the “crisis” around education is being drummed up around the wrong issues. That’s what I meant about the human tendency to rely on experts one trusts, or who say what one wants to hear.

There may be a crisis, but how we define it will determine our approach to overcoming it. Controlling “the story” is what money helps us do—lots of think tanks, media exposure makes a story seem more reasonable. I think the crisis facing us is one about trust, mutual respect, human relationships on one hand, and around understanding the complexity of the democratic idea, and the role of uncertainty on the other. It requires being exposed to conflicting views and data and trying to sort it out. Both crises, for example, require a better understanding of statistics vs. algebra (choices need to be made). A lot more understanding about the nature of evidence in various fields, and greater respect for our right to make sense of the world is not the likely outcome of current reforms. Too many schools have traditionally robbed us of our “common sense” and more of the same may further weaken democracy. There are always trade-offs.

I spent 48 hours in Switzerland. Seeing schools and even “data” from different cultural norms is always startling. They are proud of their schools—but of course admit that immigrant (foreign?) children are doing badly. So they are trying to do something about it. Surprises? The kindergarten I visited—in one of their best multi-cultural schools—was colorful and lovely and pristine! No clay, paint, sand, water and minimal blocks—although also not “academic”. Why? Because at the end of each day the rooms must be left pristine so the cleaning staff can get through the whole building in three hours. Odd way to decide priorities? In general, I found the school delightful and thoughtfully organized. I met two boys from Seattle in the immigrant integration class! Much of the education language is familiar—including our reform language. At the moment they pride themselves on not having bought into the testing mania. There are, they say, no national tests. BUT… at the age of around 14 kids do take exams that determine whether they go to university or secondary technical schools! A grand SAT? Their struggle over what it means to be Swiss reminded me of how differently we define “being American"—and yet the similarities are striking as well. Many of our current debates about outsiders echo those they are having and result in similar schooling issues. Everybody I spoke to—I also attended two university education courses—pushed my thinking. It was an exciting visit. And Zurich is beautiful. (The ceremony at which I received an award was also thrilling for me.)

Diane, in the end, do you think it would be possible to have the kind of national curriculum that would only be “suggestive”, without NAEP data becoming an even fiercer and tighter test-monitored curriculum? Why not many suggested ones? The litmus test to me is whether we’re willing to do any tests on a sampled periodic basis. Note the “comment” from the anonymous Federal bureaucrat who would like the results sent home to every parent—"Uncle Sam’s report on your child”.

Because I believe that the crisis we’re in cannot be measured by multiple-choice or short-answer tests I opt for measures that reduce the risk of focusing on tests and one-size-fits-all solutions. One picks one’s risks. Where you and I disagree tends to involve our greater or lesser fears of centralization of power?


The opinions expressed in Bridging Differences are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.