Achievement Levels
The governing board that oversees the National Assessment of Educational Progress has awarded a $1.5 million contract to ACT Inc. of Iowa City to help set new achievement levels for the NAEP math test in grade 12.
The achievement levels describe what students should know and be able to do on NAEP tests and establish the scores needed to meet three standards: “basic,” “proficient,” and “advanced.”
If the National Assessment Governing Board approves ACT’s recommendations, the new standards will be used for the first time in 2004, when the NAEP 12th grade math test introduces substantial new content, including more advanced algebra and geometry. Math tests in grades 4 and 8 will change only slightly, and the present scale scores and achievement levels will be maintained.
Although the percent of students reaching each achievement level is the primary means of reporting NAEP results, the levels remain controversial. In 1998, a panel of experts convened by the National Research Council called the current method for setting the standards— known as the modified-Angoff method—"fundamentally flawed” and said the governing board should replace it.
The performance standards the board sets are often too rigorous and don’t reflect student achievement on other tests, the panel said.
ACT Inc. will explore using a different standards-setting procedure. Under the modified-Angoff method, panels of experts determine on an item-by-item basis how likely it is that a student at the basic, proficient, or advanced level would get the item correct. ACT experts will field-test a new procedure, referred to as “map mark,” which is a modification of the “bookmark” technique.
Under the bookmark method, panelists get a loose-leaf booklet of the test items, in order of difficulty from easiest to hardest. They study the items and what they’re measuring, and then put a bookmark in the notebook to indicate where they think the cutoff points should be for each level.
Jim Carlson, the governing board’s assistant director for psychometrics, said the contract calls for a field test of the new method, upon which the board will decide the procedure to actually use.
The new 18-month contract also requires consultation with teachers, subject-matter specialists, parents, and other members of the public.
