If you’ve been reading the New York papers this week, you’ve already heard of Andrew Ho, an educational psychologist who teaches at the University of Iowa’s School of Education. Ho studies high-stakes score trends, and has done some excellent work comparing NAEP and state score trends.
If you want to hear more about why measuring achievement trends with proficiency scores is problematic, you should watch his talk, Trend-Flipping, Gap-Bowing, and Growth Stretching: The Pliability of Popular High-Stakes Statistics. Here’s a description:
The most important large-scale policy questions in education - Are students learning? Are gaps between advantaged and disadvantaged students decreasing? - are answered in part by test score trends. These trends can be reported in different ways. One popular approach is to look at the change in the percent of proficient students, where “proficient” is defined as scoring above a chosen cut score. In an clear, lively presentation, Professor Andrew Ho describes how misleading these trend statistics can be - they can be larger, smaller, and even undergo sign-reversal under a different choice of cut score. He explains the basis for this “pliability,” and describes alternative approaches to reporting and comparing score trends that avoid the troublesome properties of the proficiency-based reporting that has become widespread under No Child Left Behind.
And here’s a little clip from the NYT article about the New York state tests:
[Andrew Ho] said that while there was no question that students had improved substantially on New York’s exams, such gains were not mirrored in the national tests.
“They are on the order of what you might see in a 25-year trend on the national assessment,” Mr. Ho said. “Even the most pro-testing regime would have to admit there is a small component of inflation at the very least here.”