Opinion
States Opinion

The Proficiency Delusion

By Mark Schneider — March 01, 2010 6 min read
BRIC ARCHIVE
  • Save to favorites
  • Print

The No Child Left Behind Act of 2001 set as a national goal that all children would be proficient in reading and math by 2014. But perhaps reflecting bruising battles over President Bill Clinton’s proposed “voluntary national test” in the late 1990s, the law left both the design of the tests and the setting of “cut scores” to the states. The dubious outcome of this choice is well documented: States set their cut scores so low that large numbers of students were judged proficient even though they lacked basic skills.

When the National Center for Education Statistics mapped these state proficiency standards onto the National Assessment of Educational Progress scale, it found that in 2007, seven states had set their 4th grade math proficiency standards below NAEP’s basic level. Only Massachusetts had set its proficiency standard equal to NAEP’s. Using different data, the Thomas B. Fordham Institute found similar results, calling its findings “The Proficiency Illusion.”

NAEP’s recent Trial Urban District Assessment, or TUDA, covering math in 18 cities presents another window into this illusion. Given the concentration of low-performing students in large urban districts and the need to radically transform their education, the stakes in accurately measuring student skills and progress in these school systems are profound. The following overview of findings from the TUDA study focuses on the 4th grade math results, but those for the 8th grade are virtually identical.

If we looked only at state test results, we could conclude that most students in these large urban districts are doing reasonably well. In six of the districts, more than three-quarters of pupils are proficient, and in four others more than two-thirds are.

But the TUDA study describes a different world. Across these 18 districts, only 24 percent of students meet NAEP’s proficiency standard, far fewer than the 67 percent average across state assessments.

Outrage is often a powerful prod to reform, and state tests may be minimizing that potent force by presenting an unrealistically rosy picture of student proficiency."

Another way of thinking about this disconnect is in terms of the gap between percent proficient on state assessments and on NAEP. The gaps range from nearly 70 points in Baltimore and Detroit, to about 30 points in San Diego, Jefferson County, Ky., and the District of Columbia. Boston is the only school system in the study in which more students scored proficient on NAEP than on the state test.

This suggests that even the Fordham Institute’s provocative title understates reality. When it comes to these urban districts, the “proficiency illusion” is more a “proficiency delusion.”

One of America’s biggest challenges is to make progress on the number of students who are proficient in math (and, of course, in reading). Indeed, the No Child Left Behind law’s focus on “adequate yearly progress,” or AYP, was perhaps its most challenging and consequential dimension. Do NAEP and state measures paint a similar picture in terms of progress?

To help determine that, we can calculate changes in the percentage of students proficient on both NAEP and the state assessments for the 11 districts that took part in TUDA in 2007 as well as 2009. In nine of those districts, at least the direction of the change was the same on both assessments (Atlanta and Austin, Texas, are the exceptions). In terms of the magnitude of change, seven districts recorded more growth on state assessments than on NAEP, and in Cleveland, the decline in the percent proficient was smaller on the state test than on NAEP.

As a rough indication of how state assessments and NAEP differ in measuring progress, the average gain in percent proficient on state assessments across these 11 districts was 10 percent over that two-year period. The average gain on NAEP was only 4 percent. If we rank the districts based on the size of their gains on each assessment, the correlation is only 0.20 and not statistically significant.

In short, we’re using different yardsticks to measure progress—and finding only limited agreement between them.

Clearly, setting a national goal of having all students proficient by 2014, and yet leaving states to create their own tests and set their own cut scores, has produced a mess. Of course, state assessments and NAEP do not have identical purposes, but can these differences account for the wide variation in estimates of percent proficient? NAEP’s proficiency standards are often viewed as “aspirational,” setting a high bar, yet international comparisons show our students lagging behind major competitors. So what may be “aspirational” for our nation is about basic for many others.

We’re using different yardsticks to measure progress—and finding only limited agreement between them."

Increased scrutiny from the federal government and requirements to make test results more widely available than ever before have led states to this defining-down of proficiency. But it is a move that will thwart efforts to improve their schools.

Outrage is often a powerful prod to reform, and state tests may be minimizing that potent force by presenting an unrealistically rosy picture of student proficiency. When parents in, say, Detroit are told that more than 70 percent of their 4th graders are proficient in math, they will feel quite different about their schools than when told that just 3 percent have cleared the NAEP proficiency bar. Indeed, announcement of the TUDA results set off a barrage of national and local press reports calling Detroit the worst school district in the nation and demanding action. Setting a low bar for state-determined proficiency may weaken the push for needed changes.

These results also highlight the stakes involved in the Common Core State Standards Initiative being led by the National Governors Association and the Council of Chief State School Officers. No matter how hard the standards-setting part of this project has been, it will be easier than what remains to be done: turning these common standards into actual assessments, and then defining proficiency cut scores. The NGA and the CCSSO are still more than a little vague about how this will happen. And the U.S. Department of Education, which is putting $350 million into developing common assessments, has also left the path forward undefined.

Clearly, the whole common-standards effort is fraught with challenges. And one of the most fundamental, as my analysis shows, will be determining who sets the cut scores that define proficiency. With tensions between setting a common cut score and deferring to the states rising, political pressure to allow states to do it will be intense. But the disparities between state and TUDA results show just how consequential the stakes are in this decision.

One possible solution would be what is termed “partial pre-emption,” in which the federal government sets a uniform standard that states are free to exceed. This has been widely used in federal environmental laws and in setting safety standards through the Occupational Safety and Health Act, and is built into the Individuals with Disabilities Education Act. Not surprisingly, states are often unhappy with these pre-emptions, and push-back is common. Nonetheless, under the No Child Left Behind Act, we have already tried a system in which states were free to do as they pleased in setting proficiency standards. It didn’t work. Setting a national cut score, while allowing more aggressive states such as Massachusetts the freedom to exceed it, would be a big step forward.

This would require legislative action. The long-overdue reauthorization of the Elementary and Secondary Education Act, of which NCLB is the latest version, is the most appropriate place to revisit the current chaotic definition of proficiency. We can mend its flaws without weakening the national commitment to improving the education of all our students.

A version of this article appeared in the March 03, 2010 edition of Education Week as The Proficiency Delusion


Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
6 Key Trends in Teaching and Learning
As we enter the third school year affected by the pandemic—and a return to the classroom for many—we come better prepared, but questions remain. How will the last year impact teaching and learning this school
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
Ensuring Continuity of Learning: How to Prepare for the Next Disruption
Across the country, K-12 schools and districts are, again, considering how to ensure effective continuity of learning in the face of emerging COVID variants, politicized debates, and more. Learn from Alexandria City Public Schools superintendent
Content provided by Class
Teaching Profession Live Online Discussion What Have We Learned From Teachers During the Pandemic?
University of California, Santa Cruz, researcher Lora Bartlett and her colleagues spent months studying how the pandemic affected classroom teachers. We will discuss the takeaways from her research not only for teachers, but also for

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

States To Quarantine or Not? Florida Is Letting Parents Decide
The choice to quarantine exposed students will be up to parents, not school districts, under a rule signed by Florida’s new surgeon general.
Scott Travis, South Florida Sun-Sentinel
6 min read
Florida Gov. Ron DeSantis, shown at a news conference last month, appointed a new surgeon general for the state who issued an emergency rule that the decision to quarantine students will be up to parents, not school districts.
Florida Gov. Ron DeSantis, shown at a news conference last month, appointed a new surgeon general for the state who issued an emergency rule that the decision to quarantine students will be up to parents, not school districts.
Bob Self/The Florida Times-Union via AP
States During Site Visit From Cardona, Illinois Governor Defends Vaccine, Testing Policies
“The testing regimen is there in order to make sure that they’re not entering the institution where they work and spreading COVID-19.”
Karen Ann Cullotta, Chicago Tribune
3 min read
The Student Council lead the creation of “sensory hallways” at Western Branch Middle School in Chesapeake, Va.
U.S. Secretary of Education Miguel Cardona looks on as Gov. J.B. Pritzker speaks with reporters after touring Access Hawthorne Family Health Center, which is offering COVID-19 vaccines at 3040 S. Cicero Ave. in Cicero, as part of the Department of Education's "Return to School Road Trip" events in the Chicago area, Tuesday afternoon, Sept. 21, 2021.
Ashlee Rezin/Chicago Sun-Times via AP
States Kentucky Ends All Statewide Mask Mandates After Governor's Vetoes Overridden
The Republican-led legislation strips the Democratic governor's ability to issue statewide mask mandates in schools or anywhere else.
Jack Brammer and Alex Acquisto, Lexington Herald-Leader
4 min read
In this Wednesday, Sept. 23, 2020, file photo, Kentucky Governor Andy Beshear addresses the media in Frankfort, Ky. Kentucky's governor said Sunday, Oct. 11, that he will quarantine after a member of his security detail who drove with his family the day before later tested positive for COVID-19. Democratic Gov. Andy Beshear said he and his family feel fine, show no coronavirus symptoms and have tested negative for the virus.
In this Sept. 23, 2020, file photo, Kentucky Governor Andy Beshear addresses the media in Frankfort, Ky.
Timothy D. Easley/AP
States Bill to Restrict How Race and Racism Is Taught in Schools Headed to Texas Governor
If the "critical race theory" bill sounds familiar, that's because lawmakers passed a similar one during the regular legislative session.
Eleanor Dearman, Fort Worth Star-Telegram
4 min read
Texas Gov. Greg Abbott speaks at a news conference in Austin, Texas, on June 8, 2021.
Texas Gov. Greg Abbott speaks at a news conference in Austin, Texas, on June 8, 2021.
Eric Gay/AP