Opinion
Assessment Opinion

Campbell’s Law and testing

By Deborah Meier — May 07, 2007 4 min read

Dear Diane,

I’ve also been pondering how we could resolve some of our disagreements about national testing—without necessarily resolving our differences on curriculum! Thanks for trying to word it succinctly (less than 500 words!). My response is twice as long, alas. But I wanted to approach it in a way that helped me think through why your solution (a national curriculum with a low-stakes test) doesn’t work for me.

We’re both distressed—to put it mildly—about the misleading misinformation we’re fed about one or another school system’s successes and failures. The following data, which I ran across recently, is an example of such mystification.

**Only 33 percent of Swedish 4th graders would meet NAEP’s “proficiency or better” standards. (NAEP is the low-stakes National Assessment of Educational Proficiency test designed many years ago to collect data for the federal government, and increasingly being proposed for more high-stakes purposes.)

**Only half of Singapore’s 8th graders would be labeled “proficient or better” if measured by NAEP.

These are two puzzlers since Swedish 4th graders and Singapore 8th graders are # 1 internationally. Here’s another:

**While American 4th grade kids rank 3rd among 26 nations in science, only 29% measured “proficient or better” on NAEP.

These puzzling statistics look tame compared to the even wilder disparities we get when we compare kids in the nation’s 50 states in math or literacy scores. What counts as advanced in some states barely makes basic in another. Even if these kinds of tests were semi-accurate at measuring reading or science achievement (which I have more doubts about than you do) they clearly aren’t measuring the same thing. I can see why uniformity might have appeal.

Lee Shulman, in an essay written just a week ago for Carnegie Corp., echoes our concern. He evokes Campbell’s Law: “The more any quantitative social indicator is used for social decisionmaking, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” No mention of “rewards and punishments” you’ll note.

Something else worried Campbell. And Shulman.

You suggest that we could invent a measurement tool that wouldn’t be subject to corruption or ideology. As long as it required agreement about what all kids are expected to learn, then I think the answer is “not likely” on either score. Maybe it could were it restricted to “basic” reading, writing and arithmetic standards. But if we’re trying to better understand how our practices impact on the exercise of judgment—I do not think a national test based on a standardized national curriculum will do.

In my idle driving hours I imagine NY State announcing that, given the health and obesity crisis, a panel of experts recommend that we simplify phys ed, requiring high school graduates (unless physically handicapped) to learn to run a mile in 4 minutes (Advanced), 3 (Proficient), and so on. No high stakes attached. If every kid is tested I assume parents, schools, physical ed teachers, and above all kids would want to know how they did, even as they grumbled. Some would privately declare it impossible, but would wait for some politically more naive coach to mention it (like the little boy in The Emperor’s Clothes). Others would view news stories about coaches who met their targets with skepticism. (One was a citywide sports magnet.) Schools might ask for more practice time by dropping football and basketball, which weren’t being tested. Others would propose a similar test in all sports. Some would find reasons to exclude more kids based on new definitions of what constituted a physical handicap, some kids would try breaking their own legs, and some would just laugh. Some schools or coaches would get new stopwatches designed to meet the new standards. Those who noted that it seemed to have no effect on obesity—in fact it led to more snacking—would be ignored. Missing? Any lively and robust discussion at each school site on what role the school could play in defining and producing healthier children.

Furthermore, I’m not sure if many of the current testing maniacs would see merit in it unless the results were made public and comparisons available—in ways that willy-nilly create their own sanctions. I suspect that those that favor assessment primarily as a way of controlling—not for the purposes of gaining truthful information upon which to better discuss schooling practices—would create rank orders, offer rewards privately, etc. Some track coaches in the scenario above might applaud it because it provided a larger pool of potential runners to compete in the Olympics. Some might like it because only when we’re in a crisis mode do we allow ourselves to do the “unthinkable"—like dismantling a long tradition of public education on the grounds that it is beyond saving.

Sampled Federal studies—ala the old NAEP, investments in research that matched local concerns, a new 8-year type study of the long-term impact of various different schools—these we could agree on, Diane.

It’s amusing, sort of, that Harvard’s Eleanor Duckworth and I were invited to China to discuss promising educational reforms by officials who think that the Chinese exam-based tradition is stifling the kind of creativity and ingenuity that they believe has made America technologically and scientifically so outstanding. They are looking for ways to produce well-educated youth whose ambitions are focused on more than getting the right answers on exams.* They might also discover that creativity and ingenuity if spread too widely have repercussions for democracy as well. Happily you and I agree about that.

Deborah

*See Daryl Smith and Gwen Garrison (Teachers College Record) “The Impending Loss of Talent: An Exploratory Study Challenging Assumptions About Testing and Merit”.

The opinions expressed in Bridging Differences are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.