Also reporting the results in the traditional fashion would cost an additional $500,000, the resolution notes.
Asserting that it has solved many of the problems that drew sharp criticism the last time out, the National Assessment Governing Board this month began considering a new set of standards for student performance in mathematics.
At a meeting here, officials from American College Testing, the Iowa City-based firm that administers the widely used college-admissions test, outlined the procedure it used to set standards on the 1992 National Assessment of Educational Progress.
The A.C.T. officials also submitted a report containing preliminary written descriptions of what students at each level of achievement should know and be able to do, as well as sample test items illustrating such knowledge.
If the levels are approved, the N.A.G.B. will use the standards to report the number of students who performed at the “basic,’' “proficient,’' and “advanced’’ levels on the assessment. That approval is expected in August, after a public hearing and some modifications.
Roy E. Truby, the executive director of the governing board, said the panel is “very confident about the work A.C.T. has done.’'
“It won’t answer all the objections,’' he said. “Some people object to doing it in the first place.’'
“But I think they’ve done a good job,’' he added. “They learned from the 1990 experience, and built in many improvements. I’m confident they’ll bring in levels that are reliable.’'
Thomas Saterfiel, the vice president of the A.C.T., offered a more cautious assessment, however. He noted that the firm still has yet to analyze all of the data from the 1992 assessment in order to translate the results of the standards-setting process into achievement levels that can be reported to the public.
“We feel very good about the process,’' Mr. Saterfiel said. “The question is always, no one knows what it will say until we get all the information in. That won’t be until a couple, three months from now.’'
Sharp Criticism
The effort to set standards for NAEP represents a substantial change in the way the results of the 23-year-old assessment program are reported.
Unlike in the past, when NAEP simply reported how students performed on the assessment, the new method is aimed at comparing their performance to agreed-on standards. It would show, board officials said, whether student performance was “good enough.’'
In the first use of the standards, in reporting the results of the 1990 math assessment, the N.A.G.B. found that more than a third of students at the 4th, 8th, and 12th grades failed to attain the “basic’’ level of achievement, and that fewer than 20 percent demonstrated at least “proficient’’ achievement. (See Education Week, Oct. 2, 1991.)
But those results were thrown into doubt by a number of experts, who sharply criticized the standards-setting process.
Most recently, for example, the General Accounting Office, in a preliminary report on its investigation into the process, concluded that there were “problems of procedure, of reliability, of validity, and of reporting.’' The G.A.O. recommended against further use of the standards without substantial changes. (See Education Week, March 18, 1992.)
Mr. Truby said the board has already taken steps to improve the process, most notably by hiring the A.C.T. to oversee it and providing $1.3 million for the effort. Under its contract with the governing board, the testing firm is also expected to set standards for the 1992 reading and writing assessments.
“A.C.T. has been setting standards for a long time,’' Mr. Truby said. “They’re as competent as anyone.’'
Definitions of Levels
In its report here this month, the testing firm expressed confidence in the project.
To conduct the effort, the firm convened a group of 68 people--half of whom were classroom teachers--who met for four days in March in St. Louis. The group first agreed on definitions of what students at the “basic,’' “proficient,’' and “advanced’’ levels should know and be able to do, and then analyzed each item on the 1992 assessment to determine whether those at each level should be able to answer them correctly.
The group determined that:
- At the 4th grade, those at the “basic’’ level should show some evidence of conceptual and procedural understanding of mathematics; those at the “proficient’’ level should consistently demonstrate the integration of such understanding in solving problems; and those at the “advanced’’ level should fully integrate knowledge and understanding in problem-solving.
- At the 8th grade, those at the “basic’’ level should begin to describe objects, to compare and contrast, to find patterns, to reason from graphs, and to understand spatial reasoning; those at the “proficient’’ level should apply mathematical concepts consistently to more complex problems; those at the “advanced’’ level should be able to generalize and synthesize concepts and principles.
- At the 12th grade, “basic’’ students should understand fundamental algebraic operations with real numbers; “proficient’’ students should master such operations, and understand complex numbers; and “advanced’’ students should show mastery of trigonometric, exponential, logarithmic, and composite functions; zeros and inverses of functions; polynomials of the 3rd degree and higher; rational functions; and graphs of all of these.
The A.C.T. held a public hearing on the proposal in Washington earlier this month; it has scheduled another hearing in Seattle this week.
Some Limitations
Mr. Saterfiel said preliminary findings from the raters’ evaluation of the process indicate that they were satisfied with the results of their work. All but 1 of the 68 raters, he noted, said they would sign a statement recommending the use of the achievement levels on the 1992 math assessment.
But he noted that the process also had its limitations. For one thing, Mr. Saterfiel said, because of budget constraints, the firm was unable to employ enough raters.
“If the goal is to have the absolute smallest amount of error, you need 108 people per grade level,’' the A.C.T. vice president said. “We had 22 to 24.’'
“I don’t know if it’s worth spending four times more,’' he added.
Mr. Saterfiel also noted that the N.A.G.B. has not yet conducted a study to determine if the achievement levels are valid. Such a study would determine, for example, whether students at the “proficient’’ level are well prepared for the next level of schooling, as the board’s policy states.
Mr. Truby said the board remains committed to the use of the achievement levels. At its meeting this month, the panel unanimously adopted a resolution affirming that the new method will be “the initial and primary means’’ of reporting the results of all of the 1992 NAEP assessments.
Also reporting the results in the traditional fashion would cost an additional $500,000, the resolution notes.
“The board is not persuaded that such costs are justified at this time,’' it states.
