Opinion
Assessment Commentary

NAEP’s Odd Definition of Proficiency

By James Harvey — October 25, 2011 6 min read

Released in August, the U.S. Department of Education study mapping state proficiency standards onto the National Assessment of Educational Progress scales produced a remarkable statement from Joanne Weiss, the chief of staff to U.S. Secretary of Education Arne Duncan. According to an article in the Aug. 24 issue of Education Week, Weiss said the practice of permitting each state to set its own proficiency standards amounts to “lying to parents, lying to children, lying to teachers and principals about the work they’re doing.” Her intemperate outburst crosses the line, not only by the standards of what passes for civil discourse in Washington these days, but also for what it says about the assessment itself.

Indeed, a plausible case can be made that when it comes to telling fibs about proficiency, NAEP has a nose that annually grows longer, for its definition of proficiency is seriously flawed.

As an assessment, let us be clear, NAEP is highly regarded. It is thought of as “the nation’s report card.” Yet a controversy has surrounded NAEP’s achievement levels of basic, proficient, and advanced since they were developed in the 1990s. Congress still insists that every NAEP report include this disclaimer: “[The] National Center for Education Statistics has determined that NAEP achievement levels should continue to be used on a trial basis and should be interpreted with caution.”

BRIC ARCHIVE

Far from interpreting the NAEP achievement levels “with caution,” Ms. Weiss threw caution to the winds in questioning those state educational leaders who followed the law and common sense in defining proficiency around performance at grade level.

Since definition is crucial in any discussion of standards, let’s define the terms of the discussion. The No Child Left Behind Act, passed by Congress in 2001 as the latest reauthorization of the Elementary and Secondary Education Act of 1965, permitted states to develop their own assessments and set their own proficiency standards to measure student achievement. Most states, for their purposes, quite sensibly defined proficiency as performance at grade level.

What about NAEP? Oddly, NAEP’s proficient standard has little to do with grade-level performance or even proficiency as most people understand the term. NAEP officials like to think of the assessment standard as “aspirational.” In 2001, long before the current contretemps around state assessments, two experts associated with the National Assessment Governing Board—Mary Lynne Bourque, staff member to the governing board, and Susan Loomis, a member of the board—made it clear that “the proficient achievement level does not refer to ‘at grade’ performance. Nor is performance at the proficient level synonymous with ‘proficiency’ in the subject. That is, students who may be considered proficient in a subject, given the common usage of the term, might not satisfy the requirements for performance at the NAEP achievement level.”

It is hardly surprising, then, that most state assessments aimed at establishing proficiency as “at grade” produce results different from a NAEP standard in which proficiency does not refer to “at grade” performance or even describe students that most would think of as proficient. Far from supporting the NAEP proficient level as an appropriate benchmark for state assessments, many analysts endorse the NAEP basic level as the more appropriate standard because NAEP’s current standard sets an unreasonably high bar.

What is striking in reviewing the history of NAEP is how easily its governing board has shrugged off criticisms about the board’s standards-setting processes.

In 1993, the National Academy of Education argued that NAEP’s achievement-setting processes were “fundamentally flawed” and “indefensible.” That same year, the General Accounting Office concluded that “the standard-setting approach was procedurally flawed, and that the interpretations of the resulting NAEP scores were of doubtful validity.” The National Assessment Governing Board, or NAGB, which oversees NAEP, was so incensed by an unfavorable report it received from Western Michigan University in 1991 that it looked into firing the contractor before hiring other experts to take issue with the university researchers’ conclusions that counseled against releasing NAEP scores without warning about NAEP’s “conceptual and technical shortcomings.”

Most state assessments aimed at establishing proficiency as 'at grade' produce results different from a NAEP standard."

In addition, NAGB absorbed savage criticism from the National Academy of Sciences, which concluded in 1999 that “NAEP’s current achievement-level-setting procedures remain fundamentally flawed. The judgment tasks are difficult and confusing; raters’ judgments of different item types are internally inconsistent; appropriate validity evidence for the cut scores is lacking; and the process has produced unreasonable results. ... The results are not believable.”

For the most part, such pointed criticism has rolled off the governing board like so much water off a duck’s back.

As recently as 2009, the U.S. Department of Education received a report on NAEP from the University of Nebraska’s Buros Institute. This latest document expressed worries about NAEP’s “validity framework” and asked for a “transparent, organized validity framework, beginning with a clear definition of the intended and unintended uses of the NAEP assessment scores. We recommend that NAGB continue to explore achievement-level methodologies.” In short, for the last 20 years, it has been hard to find any expert not on the Education Department’s payroll who will accept the NAEP benchmarks uncritically.

Those benchmarks might be more convincing if most students outside the United States could meet them. That’s a hard case to make, judging by a 2007 analysis from Gary Phillips, a former acting commissioner of the National Center for Education Statistics. Phillips set out to map NAEP benchmarks onto international assessments in science and mathematics and found that only Taipei (or Taiwan) and Singapore have a significantly higher percentage of proficient students in 8th grade science than the United States does. In math, the average performance of 8th grade students in six jurisdictions could be classified as proficient: Singapore, South Korea, Taipei, Hong Kong, Japan, and Flemish Belgium. Judging by Phillips’ results, it seems that when average results, by jurisdiction, place typical students at the NAEP proficient level, the jurisdictions involved are typically wealthy—many with “tiger mothers” or histories of excluding low-income students or those with disabilities.

None of this is to say that the method of determining the NAEP achievement levels is entirely indefensible. Like other large-scale assessments—the International Mathematics and Science Study, the Progress on International Reading Literacy Survey, and the Program on International Student Assessment—NAEP is an extremely complex endeavor, depending on procedures in which experts make judgments about what students should know and construct assessment items to distinguish between student responses. Panels then make judgments about specific items, and trained scorers, in turn, bring judgment to bear on constructed-response items, which typically make up about 40 percent of the assessment.

That said, it is hard to avoid some obvious conclusions. First, NAEP’s achievement levels, far from being engraved on stone tablets, are administered, as Congress has insisted, on a “trial basis.” Second, NAEP achievement levels are based on judgment and educated guesses, not science. Third, the proficiency benchmark seems reachable by most students in only a handful of wealthy or Asian jurisdictions.

Finally, enough questions exist about these achievement levels that Congress should commission an independent exploration to make sense of the many diverse definitions of proficiency found in state, NAEP, and international assessments. A national assessment that suggests proficiency is beyond the reach of students throughout the Western world promises to confuse our educational challenges, not clarify them.

A version of this article appeared in the October 26, 2011 edition of Education Week as NAEP’s Odd Definition of Proficiency

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Measuring & Supporting Student Well-Being: A Researcher and District Leader Roundtable
Students’ social-emotional well-being matters. The positive and negative emotions students feel are essential characteristics of their psychology, indicators of their well-being, and mediators of their success in school and life. Supportive relationships with peers, school
Content provided by Panorama Education
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
Making Digital Literacy a Priority: An Administrator’s Perspective
Join us as we delve into the efforts of our panelists and their initiatives to make digital skills a “must have” for their district. We’ll discuss with district leadership how they have kept digital literacy
Content provided by Learning.com
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
How Schools Can Implement Safe In-Person Learning
In order for in-person schooling to resume, it will be necessary to instill a sense of confidence that it is safe to return. BD is hosting a virtual panel discussing the benefits of asymptomatic screening
Content provided by BD

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment Opinion Alternatives to Standardized Tests During a Pandemic Year
Three educators suggest alternatives to federally mandated standardized testing during this year undercut by COVID-19.
7 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
iStock/Getty
Assessment Opinion AP Exams Can't Be Business as Usual This Year
The College Board seems unconcerned with the collateral damage of its pandemic approach, writes an assistant superintendent of curriculum and instruction.
Pete Bavis
5 min read
Illustration of large boat in turbulent waters with other smaller boats falling into the abyss.
iStock/Getty Images Plus
Assessment Federal Lawmakers Urge Miguel Cardona to Let States Cancel Tests, Highlighting Discord
A letter from Democratic members to the new education secretary calls for an end to the "flawed" system of annual standardized exams.
3 min read
Jamaal Bowman speaks to reporters after voting at a polling station inside Yonkers Middle/High School in Yonkers, N.Y. on June 23, 2020.
Jamaal Bowman speaks to reporters after voting at a polling station inside Yonkers Middle/High School in Yonkers, N.Y. on June 23, 2020.
John Minchillo/AP
Assessment How Two Years of Pandemic Disruption Could Shake Up the Debate Over Standardized Testing
Moves to opt out of state tests and change how they're given threaten to reignite fights over high-stakes assessments.
9 min read
Image of a student at a desk.
patat/iStock/Getty