As a boy in Ireland, I attended Mass every Sunday during which priests with their backs to the congregation mumbled in incomprehensible Latin. None of us knew what it meant, but we were assured these mysteries were good for the soul.
In education today, measurement experts occupy the exalted status of Irish priests. With their figurative back to the schools, these prelates genuflect at the altar of Item Response Theory and mumble confidently amongst themselves in a language known as psychometrics. No one in the school congregation understands a word of it, but we are assured these mysteries are based on science.
It’s a troubling situation. As I argued earlier on this blog, assessment results are but the tip of an iceberg. All the assumptions that produce the numbers the public sees are buried beneath the waterline. Discussion between the measurement experts who develop the assessments and front-line educators who must act on them is almost non-existent. It’s a broken conversation.
Professor Pizmony-Levy’s scholarly review of the history of the governance of international large-scale assessments introduces a new element that makes the situation even more problematic. Researchers once dominated the governance mechanisms of ILSAs. That’s not true today. Nearly three-quarters of the members of the policy governance committee of the International Association for the Evaluation on Education Achievement (IEA), for example, are from government or the bureaucracy. It is a safe bet that many of these people don’t possess even a rudimentary understanding of the “voodoo that we do so well,” as Jakob Wandall of Denmark described the arcana of psychometrics at a 2012 conference sponsored by Teachers College’s Assessment and Evaluation Research Initiative.
Pizmony-Levy’s history reminds me of Torsten Husén’s 1982 testimony before the National Commission on Excellence in Education, which I served as deputy director. Husén, one of the towering figures in the early history of IEA, made a plea to the commission: Don’t interpret international large-scale assessments as a horse race between nations. ILSAs weren’t developed by bookies to handicap horses but by researchers to help clarify the goals each nation pursued for itself through its schools.
Unfortunately, the politically appointed members of the excellence commission could not resist the catnip of ranking nations by mean results. In its startling 1983 report, A Nation at Risk, the commissioners thundered: “On 19 [international] academic tests, American students were never first or second and, in comparison with other industrialized nations, were last seven times.”
What would Husén make of where we are today? It is not just that the horse-race mentality of mean scores on ILSAs dominate the conversation, domestically and internationally. It is also that bureaucrats in both individual nations and international associations have demonstrated remarkable skill and cunning in providing more of the horse-race catnip politicians crave. And, truth be told, in the quiet of their university and think-tank offices many measurement experts acknowledge that, although troubled by some directions in assessment today, they have to follow the money.
So, where does this leave us? Several things might be said. First, no responsible educator denies the need for assessment. Second, accountability needs to run in both directions. It is clear that teachers and administrators are today at the mercy of how public officials interpret ILSA results. This puts a special onus on measurement leaders to live up to the best practices of the profession. They shouldn’t be in the business of providing catnip. Third, it’s time for another look at the governance structures of these assessments.
Finally, we should openly acknowledge that what’s “below the waterline” is simply incomprehensible to policymakers and the lay audience. It’s time to clarify communications among the three sectors (measurement experts, politicians, and the educators). After all, even in the churches of Ireland, priests long ago abandoned Latin and turned their faces to the congregation.
James Harvey
National Superintendents Roundtable