Opinion
Federal Opinion

Accountability Tests’ Instructional Insensitivity: The Time Bomb Ticketh

By W. James Popham — November 13, 2007 6 min read
  • Save to favorites
  • Print

Would you ever want your temperature to be taken with a thermometer that was unaffected by heat? Of course not; that would be dumb. Or would you ever want to weigh yourself with bathroom scales that weren’t influenced by the weight of the person using them? Of course not again; that would be equally dumb. But today’s educators are allowing their instructional success to be judged by students’ scores on accountability tests that are essentially incapable of distinguishing between effective and ineffective instruction. Talk about dumb.

What’s worse is that we are now racing toward the 2014 deadline of the federal No Child Left Behind Act, the point at which all students are supposed to have attained test-based “proficiency.” But the 2002-2014 schedules that most states devised when establishing their goals for annual required numbers of proficient students will soon demand some staggering increases in how many students must earn proficient scores on state NCLB tests each year. These balloon-payment improvement schedules were, in most instances, adopted as a way of deferring the pain stemming from having too many state schools and districts flop in reaching their goals for adequate yearly progress, or AYP.

Such cunningly crafted, soft-to-start improvement schedules will lead in a very few years to altogether unrealistic requirements for improved test scores. Without such improvements, huge numbers of U.S. schools and districts will be seen as AYP failures. If the American public is skeptical now about the quality of public schools, how do you think citizens will react when, in the next several years, test-based AYP failure becomes the rule rather than the exception? Can you hear the ticking of this nontrivial time bomb?

How could American educators let themselves get into a situation in which the tests being used to evaluate their instruction are unable to distinguish between effective and ineffective teaching? The answer, though simple, is nonetheless disquieting. Most American educators simply don’t know that their state’s NCLB tests are instructionally insensitive. Educators, and the public in general, assume that because such tests are “achievement tests,” they accurately measure how much students have learned in schools. That’s just not true.

Two types of accountability tests are currently being used to satisfy the No Child Left Behind law’s assessment requirements. About half of the nation’s NCLB tests consist of traditional, off-the-shelf, standardized achievement tests, usually supplemented by a sprinkling of new items, so that the slightly expanded tests will supposedly be better aligned with a particular state’s content standards. Other NCLB tests are made-from-scratch, customized standards-based accountability tests, built specifically for a given state. Let’s see, briefly, why both these types of tests are instructionally insensitive.

Traditional standardized achievement tests, such as the Stanford Achievement Test-10th Edition, are intended to provide comparative information about test-takers. So the performance of a student who scores at, for instance, the 96th percentile can be contrasted with that of students who score at lower percentiles. To accomplish this comparative-measurement mission, these tests must produce a substantial degree of “score spread,” so there are ample numbers of high scores, middle scores, and low scores. Most items on such tests are of middle-difficulty levels because such items, statistically, maximize score spread.

Over the years, however, many of these middle-difficulty items turn out to be closely linked to students’ socioeconomic status. More-affluent kids tend to answer these socioeconomically linked items correctly, while less-affluent kids tend to miss them. This occurs because socioeconomic status, or SES, is a nicely distributed variable, and one that doesn’t change rapidly; SES-linked items help generate the score spread required by traditional standardized achievement tests. When such tests are used as accountability assessments, however, they tend to measure the socioeconomic composition of a school’s student body, rather than the effectiveness with which those students have been taught. The more SES-linked items there are on a traditional standardized achievement test, the more instructionally insensitive that test is bound to be.

How could American educators let themselves get into a situation in which the tests being used to evaluate their instruction are unable to distinguish between effective and ineffective teaching?

The other type of NCLB accountability test used in the United States is usually described as a “standards-based test,” because such tests are deliberately built to assess students’ mastery of a given state’s content standards, that is, its curricular aims. In all but a few states, though, the number of content standards to be assessed is so large that there is no way to accurately assess—via an annual accountability test—students’ mastery of this immense array of skills and knowledge. Instead, each year’s accountability test must sample from the profusion of the state’s curricular aims. Such a sampling-based approach to annual assessment means that teachers end up guessing about which curricular aims will be assessed each year. And, given the huge numbers of potentially assessable curricular targets, most teachers guess wrong.

After a few years of incorrect guessing, many teachers simply give up on trying to mesh their teaching with what’s to be assessed on each year’s accountability tests. And when this happens, it turns out that the major determinant of how well a school’s students perform on accountability tests is the very same factor that governed students’ performances on traditional standardized achievement tests: socioeconomic status. Thus, even on customized standards-based tests, a school’s scores are influenced less by what students are taught than by what the students brought to that school. Most standards-based accountability tests are every bit as instructionally insensitive as traditional standardized achievement tests.

The instructional insensitivity of accountability tests does not represent an insuperable problem, however. Remember when, several decades ago, we began to recognize that there was considerable test bias in our high-stakes educational assessments? Once this difficulty had been identified, it was attacked with both empirical and judgmental bias-detection procedures. As a consequence, today’s educational tests are markedly less biased than were their predecessors. Once the test-bias problem had been identified, we set out to fix it—and in less than a decade, we did.

That’s precisely what we need to do now. Using a mildly technical definition, a test’s instructional sensitivity represents the degree to which students’ performances on that test accurately reflect the quality of instruction specifically provided to promote students’ mastery of what is being assessed. We need to discover how to build accountability tests that will be instructionally sensitive and, therefore, can provide valid inferences about effective and ineffective instruction. It may take several years to get the required procedures in place, but we need to get started right now.

In the short term, though, we must make citizens, and especially educational policymakers, understand that almost all of today’s accountability tests yield an invalid picture of how well students are being taught. Accountability systems based on the use of such instructionally insensitive tests are flat-out senseless. We need accountability tests capable of distinguishing between students who have been properly taught and those who have not. Until such tests are at hand, we might as well relabel our accountability systems as what they are—elaborate and costly socioeconomic-status identifiers.

A version of this article appeared in the November 14, 2007 edition of Education Week as Accountability Tests’ Instructional Insensitivity: The Time Bomb Ticketh

Events

Classroom Technology Webinar How Pandemic Tech Is (and Is Not) Transforming K-12 Schools
The COVID-19 pandemic—and the resulting rise in virtual learning and big investments in digital learning tools— helped educators propel their technology skills to the next level. Teachers have become more adept at using learning management
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Building Teacher Capacity for Social-Emotional Learning
Set goals that support adult well-being and social-emotional learning: register today!


Content provided by Panorama
Jobs October 2021 Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal 'A Snitch Line on Parents.' GOP Reps Grill AG Over Response to Threats on School Officials
Attorney General Merrick Garland said his effort is meant to address violent threats against school boards, not to stifle parents' dissent.
5 min read
LEFT: Attorney General Merrick Garland speaks during a House Judiciary Committee oversight hearing of the Department of Justice on Capitol Hill in Washington, Thursday, Oct. 21, 2021. RIGHT: Rep. Jim Jordan, R-Ohio, questions Attorney General Merrick Garland.
Attorney General Merrick Garland, left, speaks during a House Judiciary Committee oversight hearing of the U.S. Department of Justice on Capitol Hill on Thursday, questioned by Rep. Jim Jordan, R-Ohio, right, among others.
Greg Nash via AP, Andrew Harnik/AP
Federal School Boards, 'Domestic Terrorism,' and Free Speech: Inside the Debate
From critical race theory to COVID policy, the heat on schools has raised issues involving free speech and the safety of public officials.
13 min read
Brenda Stephens, a school board member with Orange County Public Schools in Hillsborough, N.C. has purchased a weapon and taken a concealed carry class over concerns for her personal safety.
Brenda Stephens, a school board member in Hillsborough, N.C., says board members face threats and bullying, an atmosphere far different from what she's encountered in years of board service.
Kate Medley for Education Week
Federal Senate Confirms Catherine Lhamon to Civil Rights Post; Kamala Harris Casts Decisive Vote
Joe Biden's controversial pick to lead the Education Department's office for civil rights held that job in the Obama administration.
2 min read
Catherine Lhamon, nominee to be assistant secretary for civil rights at the Department of Education, testifies during a Senate Health, Education Labor and Pensions Committee confirmation hearing in Dirksen Building on Tuesday, July 13, 2021.
Catherine Lhamon, then-nominee to be assistant secretary for civil rights at the U.S. Department of Education, testifies during a Senate Health, Education Labor and Pensions Committee confirmation hearing in July.
Tom Williams/CQ Roll Call via AP Images
Federal White House Outlines COVID-19 Vaccination Plans for Kids 5-11
The Biden administration will rely on schools, pharmacies, and pediatricians to help deliver the COVID-19 shots to younger children.
3 min read
Ticket number 937 sits on a COVID-19 vaccination at the drive-thru vaccination site in the Coweta County Fairgrounds on Jan. 14, 2021, in Newnan, Ga.
A ticket number sits on a COVID-19 vaccination at the drive-thru vaccination site in the Coweta County Fairgrounds in Newnan, Ga.
Curtis Compton/Atlanta Journal-Constitution via AP