If there’s one thing that is sure to turn off most editors, it’s essays about standardized test scores. Whether it’s because they assume that all tests are created equal or that the subject intrinsically lacks sex appeal, attempts to clarify matters usually fall on deaf ears. One exception is “Test Scores Are No Sure Guide to What Students Know” (The Wall Street Journal, Jul. 12).
The takeaway line is that standardized tests “reveal less about what children know than about the way the test makers decide to measure that knowledge.” This encompasses two key elements that are poorly understood by taxpayers. The first refers to the design of standardized tests, and the second refers to the validity of inferences drawn about test results.
It’s not hard to design a test that has a high likelihood of producing just about any outcome desired. Let’s not forget that a test cannot possibly measure all that is taught. Therefore, designers rely on sampling what they believe are the most important concepts in any given subject. But suppose the mission is to sort out students (e.g., the SAT). If the test were heavily loaded up with items assessing only the most important material that was well taught, scores would likely be clumped together, making comparisons unsatisfactory. In that case, the testing company would not remain in business very long because it had not delivered on its promise.
To avoid that possibility, designers deliberately include some items that have a low probability of being taught. This is not at all fair, but it is extremely useful. As a result, the public will draw invalid inferences about the effectiveness of instruction in the schools they support with their taxes. The same invalid inferences can be drawn by the arbitrary way that cut scores are determined. The public demands to know how many students are “proficient” in any given subject. By manipulating the scale changes, designers can help mold judgments about how many students fall into that category.
As unpopular as it is, assessment is an indispensable part of the teaching-learning process. Without feedback, it’s extremely difficult to know what’s working. I realize that experienced teachers can tell whether their students are getting what is taught by strategies other than traditional tests. But the accountability movement demands hard data. And nothing seems to satisfy reformers like standardized tests. That’s unfortunate because these instruments hardly tell the whole story.