NAEP Board Fires Researchers Critical of Standards Process

By Robert Rothman — September 04, 1991 4 min read
  • Save to favorites
  • Print

The governing beard of the National Assessment of Educational Progress has fired a team of researchers that prepared a critical evaluation of the beard’s process for setting achievement levels for use in reporting results of NAEP’s 1990 mathematics assessment.

In a letter to researchers and policymakers who had received a draft copy of the final report, officials of the National Assessment Governing Board called the draft “so thoroughly flawed as to be unsalvageable.”

In addition to numerous “egregious errors and misstatements of fact,” the officials stated, there is a “purposeful lack of objectivity permeating the entire document.”

The officials also charged that the reviewers had committed a “political act” by submitting the draft to policymakers, such as Congressional aides and Gov. Roy Romer of Colorado, rather than just to technical experts who could make informed comments.

“It was clear from the beginning they didn’t believe in achievement levels, they didn’t want levels set,” Roy E. Truby, executive director of the N.A.G.B., said in an interview. “It was even more clear they don’t like a N.A.G.B.-type board.”

He said the beard was terminating the evaluators’ services because it was unwilling to invest additional funds in editing the final report.

Richard M. Jaeger, a professor of education at the University of North Carolina at Greensboro and a member of the evaluation team, responded that the board’s charges were “absurd,” and he took offense at the charge that the team was biased and had failed to follow the standards of education-program evaluation.

The chairman of the review panel, Daniel L. Stufflebeam, director of the evaluation center in the college of education at Western Michigan University, wrote the evaluation standards, Mr. Jaeger noted.

The charges are “code words for saying they didn’t like our conclusions,” Mr. Jaeger said. “This is a case of not liking the message and acting to kill the messenger.”

Other researchers familiar with the incident voiced outrage at the beard’s action, and warned it could threaten the credibility of NAEP.

But Governor Romer, who just stepped down as chairman of the National Education Goals Panel, said the controversy would not deter the goals panel from including the disputed data in its forthcoming report.

“My judgment is, we will continue to use the material, and label it the best judgment we have,” he said. ‘“We are aware of the judgmental nature of this, and we will use these figures with appropriate conservatism.” John F. Jennings, an aide to the House Education and Labor Committee, said the committee has asked the General Accounting Office to look into the matter.

“We want them to make a judgment about whether the Stufflebeam team was correct on their points, or whether N.A.G.B. was correct in their rebuttal,” he said.

At issue in the dispute is the NAEP governing board’s decision last May to change the way it reports NAEP results. Unlike in the past, when the congressionally mandated project simply reported how students performed on the assessment, the board agreed to compare student performance with agreed-upon standards for “basic,” “proficient,” and “advanced” levels of achievement.

As part of its effort to set standards, a technically complex procedure, the beard had agreed last year to hire a team of experts to evaluate the process. In addition to Mr. Stufflebeam and Mr. Jaeger, the team included Michael Scriven, a professor of education at Western Michigan University. Although the team’s formal $11,000 contract expired last December, the board asked the researchers to continue their work and to produce other reports. The board has paid the researchers an additional $7,000, Mr. Truby said.

In its draft final report, the evaluation team concluded that, despite numerous positive aspects, the process the board used to set achievement levels “must be viewed as insufficiently tested and validated, politically dominated, and of questionable credibility.”

“If used at all, these achievement levels should be applied with caution,” the report states. “Their psychometric properties are suspect to the degree that they should not be used to infer the quality of 1990 mathematics achievement of grades 4, 8, and 12 students in the United States, and certainly should not be used to form the baseline of a time series representing U.S. students’ mathematics achievement.”

The team also urged the Congress to reconstitute the N.A.G.B. to include a greater number of technical exports, and to invest in a research-and-development project to investigate standards-setting issues.

In a 22-page rejoinder, the board took strong issue with the researchers’ assertions, and concluded that ! their report was “a document riddled with inaccuracies, articulated in unsubstantiated generalizations, and devoid of what most evaluators look for in such a report--data.”

Achievement Levels Defended

“The board believes,” the rejoinder states, “that the achievement levels it has adopted for reporting results of the 1990 NAEP mathematics assessment are reasonable, defensible, and useful; that, despite the rhetoric and bias of the evaluation consultants, the facts argue for release and evaluation of the achievement levels; and that releasing the achievement levels will help to inform the ongoing discussion related to national education standards.”

Mr. Truby denied that the board was attempting to squelch the evaluators’ report, and noted that he was making available the draft report, along with the board’s rejoinder, to all interested parties.

Mr. Jaeger said that, despite the firing, the panel has submitted its final report to the N.A.G.B., and added that he hoped others would read it and come to their own conclusions.

“I’m hopeful people who are less passionate about the argument will evaluate the evidence on its merits,” Mr. Jaeger said.

A version of this article appeared in the September 04, 1991 edition of Education Week as NAEP Board Fires Researchers Critical of Standards Process


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
Strategies for Incorporating SEL into Curriculum
Empower students to thrive. Learn how to integrate powerful social-emotional learning (SEL) strategies into the classroom.
Content provided by Be GLAD
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Leadership in Education: Building Collaborative Teams and Driving Innovation
Learn strategies to build strong teams, foster innovation, & drive student success.
Content provided by Follett Learning
School & District Management K-12 Essentials Forum Principals, Lead Stronger in the New School Year
Join this free virtual event for a deep dive on the skills and motivation you need to put your best foot forward in the new year.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Briefly Stated: June 12, 2024
Here's a look at some recent Education Week articles you may have missed.
9 min read
Education Briefly Stated: May 29, 2024
Here's a look at some recent Education Week articles you may have missed.
9 min read
Education Briefly Stated: May 8, 2024
Here's a look at some recent Education Week articles you may have missed.
8 min read
Education Briefly Stated: April 17, 2024
Here's a look at some recent Education Week articles you may have missed.
8 min read