Opinion
States Opinion

The Proficiency Delusion

By Mark Schneider — March 01, 2010 6 min read
BRIC ARCHIVE
  • Save to favorites
  • Print

The No Child Left Behind Act of 2001 set as a national goal that all children would be proficient in reading and math by 2014. But perhaps reflecting bruising battles over President Bill Clinton’s proposed “voluntary national test” in the late 1990s, the law left both the design of the tests and the setting of “cut scores” to the states. The dubious outcome of this choice is well documented: States set their cut scores so low that large numbers of students were judged proficient even though they lacked basic skills.

When the National Center for Education Statistics mapped these state proficiency standards onto the National Assessment of Educational Progress scale, it found that in 2007, seven states had set their 4th grade math proficiency standards below NAEP’s basic level. Only Massachusetts had set its proficiency standard equal to NAEP’s. Using different data, the Thomas B. Fordham Institute found similar results, calling its findings “The Proficiency Illusion.”

NAEP’s recent Trial Urban District Assessment, or TUDA, covering math in 18 cities presents another window into this illusion. Given the concentration of low-performing students in large urban districts and the need to radically transform their education, the stakes in accurately measuring student skills and progress in these school systems are profound. The following overview of findings from the TUDA study focuses on the 4th grade math results, but those for the 8th grade are virtually identical.

If we looked only at state test results, we could conclude that most students in these large urban districts are doing reasonably well. In six of the districts, more than three-quarters of pupils are proficient, and in four others more than two-thirds are.

But the TUDA study describes a different world. Across these 18 districts, only 24 percent of students meet NAEP’s proficiency standard, far fewer than the 67 percent average across state assessments.

Outrage is often a powerful prod to reform, and state tests may be minimizing that potent force by presenting an unrealistically rosy picture of student proficiency."

Another way of thinking about this disconnect is in terms of the gap between percent proficient on state assessments and on NAEP. The gaps range from nearly 70 points in Baltimore and Detroit, to about 30 points in San Diego, Jefferson County, Ky., and the District of Columbia. Boston is the only school system in the study in which more students scored proficient on NAEP than on the state test.

This suggests that even the Fordham Institute’s provocative title understates reality. When it comes to these urban districts, the “proficiency illusion” is more a “proficiency delusion.”

One of America’s biggest challenges is to make progress on the number of students who are proficient in math (and, of course, in reading). Indeed, the No Child Left Behind law’s focus on “adequate yearly progress,” or AYP, was perhaps its most challenging and consequential dimension. Do NAEP and state measures paint a similar picture in terms of progress?

To help determine that, we can calculate changes in the percentage of students proficient on both NAEP and the state assessments for the 11 districts that took part in TUDA in 2007 as well as 2009. In nine of those districts, at least the direction of the change was the same on both assessments (Atlanta and Austin, Texas, are the exceptions). In terms of the magnitude of change, seven districts recorded more growth on state assessments than on NAEP, and in Cleveland, the decline in the percent proficient was smaller on the state test than on NAEP.

As a rough indication of how state assessments and NAEP differ in measuring progress, the average gain in percent proficient on state assessments across these 11 districts was 10 percent over that two-year period. The average gain on NAEP was only 4 percent. If we rank the districts based on the size of their gains on each assessment, the correlation is only 0.20 and not statistically significant.

In short, we’re using different yardsticks to measure progress—and finding only limited agreement between them.

Clearly, setting a national goal of having all students proficient by 2014, and yet leaving states to create their own tests and set their own cut scores, has produced a mess. Of course, state assessments and NAEP do not have identical purposes, but can these differences account for the wide variation in estimates of percent proficient? NAEP’s proficiency standards are often viewed as “aspirational,” setting a high bar, yet international comparisons show our students lagging behind major competitors. So what may be “aspirational” for our nation is about basic for many others.

We’re using different yardsticks to measure progress—and finding only limited agreement between them."

Increased scrutiny from the federal government and requirements to make test results more widely available than ever before have led states to this defining-down of proficiency. But it is a move that will thwart efforts to improve their schools.

Outrage is often a powerful prod to reform, and state tests may be minimizing that potent force by presenting an unrealistically rosy picture of student proficiency. When parents in, say, Detroit are told that more than 70 percent of their 4th graders are proficient in math, they will feel quite different about their schools than when told that just 3 percent have cleared the NAEP proficiency bar. Indeed, announcement of the TUDA results set off a barrage of national and local press reports calling Detroit the worst school district in the nation and demanding action. Setting a low bar for state-determined proficiency may weaken the push for needed changes.

These results also highlight the stakes involved in the Common Core State Standards Initiative being led by the National Governors Association and the Council of Chief State School Officers. No matter how hard the standards-setting part of this project has been, it will be easier than what remains to be done: turning these common standards into actual assessments, and then defining proficiency cut scores. The NGA and the CCSSO are still more than a little vague about how this will happen. And the U.S. Department of Education, which is putting $350 million into developing common assessments, has also left the path forward undefined.

Clearly, the whole common-standards effort is fraught with challenges. And one of the most fundamental, as my analysis shows, will be determining who sets the cut scores that define proficiency. With tensions between setting a common cut score and deferring to the states rising, political pressure to allow states to do it will be intense. But the disparities between state and TUDA results show just how consequential the stakes are in this decision.

One possible solution would be what is termed “partial pre-emption,” in which the federal government sets a uniform standard that states are free to exceed. This has been widely used in federal environmental laws and in setting safety standards through the Occupational Safety and Health Act, and is built into the Individuals with Disabilities Education Act. Not surprisingly, states are often unhappy with these pre-emptions, and push-back is common. Nonetheless, under the No Child Left Behind Act, we have already tried a system in which states were free to do as they pleased in setting proficiency standards. It didn’t work. Setting a national cut score, while allowing more aggressive states such as Massachusetts the freedom to exceed it, would be a big step forward.

This would require legislative action. The long-overdue reauthorization of the Elementary and Secondary Education Act, of which NCLB is the latest version, is the most appropriate place to revisit the current chaotic definition of proficiency. We can mend its flaws without weakening the national commitment to improving the education of all our students.

A version of this article appeared in the March 03, 2010 edition of Education Week as The Proficiency Delusion

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
A Whole Child Approach to Supporting Positive Student Behavior 
To improve student behavior, it’s important to look at the root causes. Social-emotional learning may play a preventative role.

A whole child approach can proactively support positive student behaviors.

Join this webinar to learn how.
Content provided by Panorama
Recruitment & Retention Live Online Discussion A Seat at the Table: Why Retaining Education Leaders of Color Is Key for Student Success
Today, in the United States roughly 53 percent of our public school students are young people of color, while approximately 80 percent of the educators who lead their classrooms, schools, and districts are white. Racial
Jobs January 2022 Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

States N.H. Teacher ‘Loyalty’ Law Could Expand to Include Race Debate
A Cold War-era law in New Hampshire targeting "teachers' loyalty" would be updated with today's hot button issues.
2 min read
Collage of an American Flag.
Collage: Laura Baker/Education Week (Images: iStock/Getty)
States Pivoting to Remote Learning: Why It Is Harder in Some States Than Others
In calling the shots on the switch back to remote instruction, states have very different rules, an Education Week analysis finds.
8 min read
Macy Schulman, left, and Mason Yeoh, both students at Fairfield Warde High School, carry pro-remote learning signs during a rally of parents and students fighting to have an online option for school this year, Monday, Aug. 16, 2021, in Fairfield, Conn.
Macy Schulman, left, and Mason Yeoh, both students at Fairfield Warde High School in Connecticut, carry pro-remote learning signs during a rally in August of parents and students fighting to have an online option for school this academic year.
Ned Gerard/Hearst Connecticut Media via AP
States Ind. Teachers Push Back Against Bill That Would Let Parents Vet School Curricula
Sparking opposition from dozens of teachers, the legislation seeks to require all school curricula to be vetted by parent review committees.
4 min read
Rep. Vernon Smith, left, D-Gary, looks at his notes during the first day of the legislative session at the the Statehouse, Tuesday, Jan. 4, 2022, in Indianapolis.
Rep. Vernon Smith, left, D-Gary, looks at his notes during the first day of the legislative session at the the Statehouse, Tuesday, Jan. 4, 2022, in Indianapolis.
Darron Cummings/AP
States Ariz. Families Can Now Get Private School Vouchers If Their Schools Go Remote
Gov. Doug Ducey says he is taking "preemptive action" to keep students in classrooms despite rising hospitalizations as the Omicron variant spreads.
4 min read
Arizona Republican Gov. Doug Ducey speaks at a ceremony on Dec. 7, 2021, in Phoenix. Gov. Ducey on Tuesday, Jan. 4, 2022, took what he called "preemptive action" to keep school public schools open and give students access to in-person instruction despite rising numbers of COVID-19 cases in Arizona and nationwide as the more contagious omicron virus variant spreads.
Arizona Republican Gov. Doug Ducey speaks at a ceremony on Dec. 7, 2021, in Phoenix.
Ross D. Franklin/AP