Chiefs Weighing Use of NAEP for Pupil Comparisons
New York--The nation's chief state school officers, who a year ago set themselves the task of developing a way to compare student achievement on a state-by-state basis, will consider a proposal at their meeting next month to use test questions from the National Assessment of Educational Progress for that purpose.
Initial proposals also call for giving states a more influential voice in the operations of the national assessment itself. Possibilities range from influencing the areas to be tested to awarding of the contract for its administration.
Preliminary recommendations presented here last week call for the Council of Chief State School Officers to test students at the same grade levels and ages that the naep does, but to develop its own pool of test items based on the naep questions rather than use the examination itself.
The initial draft of the plan--called for by the council last year--will be presented to its committee on coordinating educational information and research next week.
If the recommendations are then approved in November, the first state-by-state report on student achievement--in the areas of English, mathematics, reading, science, and social studies--probably will not occur until 1988, said Stephen S. Kaagan, commissioner of education in Vermont and chairman of the research committee. Testing in other areas, such as the arts and higher-order thinking skills, would not occur until after that.
The next two years will be spent gathering and reporting on educational indicators that are already available, he said, such as states' per-pupil expenditures.
Mr. Kaagan presented the proposed recommendations in New York City, at the annual conference of the council's study commission.
The commission, which includes representatives from each of the 50 states, last year drafted the policy statement on educational indicators that was adopted by the chiefs.
This year, it is developing a series of proposals related to international education. (See related article on page 12.)
According to Mr. Kaagan, the proposed plan will recommend that the chiefs report on educational indicators in three areas: contextual factors that schools cannot change, such as demographics; educational policies and practices that schools can change, such as graduation requirements; and educational outcomes, such as student achievement.
The preliminary list of indicators was chosen on the basis of three criteria: whether the indicators are important and useful to policymakers and the public; whether it is technically possible to gather information about them; and whether they are feasible in terms of the cost and data-gathering difficulties involved.
To help make state-by-state comparisons that are "fair," Mr. Kaagan said, the commission recommended the development of a state-classification system based on broader information about each state.
"This is the apples and oranges issue," he said of the problem of preventing people from inappropriately comparing educational outcomes between states like Texas and Vermont or Connecticut and Mississippi.
The recommendations also ask the council to develop a classification system for comparing school districts. The ccsso, however, would not gather or report that information, but would ask individual states to do so.
The proposed plan also calls on the chiefs to develop a scaling system that would compare states' educational progress in a more descriptive way than simple state8rankings, Mr. Kaagan said.
The recommendations also ask the council to establish national standards on selected educational policies, practices, and outcomes, "with the intent that all states would meet them," Mr. Kaagan said. And they call for a second list of "goals" toward which states would aim but would not be expected to meet in the near future.
The council will report both national and state data, according to the draft plan. The form these might take would vary, depending on the audiences and the issues addressed.
Mr. Kaagan said the draft plan also includes a list of principles for changing the naep program itself.
"The theme behind the principles is that if the states are going to be involved heavily in the use of naep items, then we would need greater influence, greater control over what happens within the national-assessment system," he said.
In particular, he noted, states might have greater influence over the subject areas tested, the governance of the national assessment, the awarding of the naep contract, sampling procedures, and reporting. In addition, he said, state officials should be involved more closely in the work of the national assessment.
Mr. Kaagan cautioned that even if the chiefs adopt the proposed recommendations, their implementation would be slow. "The theme," he said, "is walk before we run. If we need to go a little bit slow, we will, rather than being precipitous as some others have."
Many of the indicators that states now use lack "standardization and definition," said John Pisapia, assistant state superintendent of the bureau of learning systems in the West Virginia Department of Education and chairman of the steering committee for the council's new center on assessment and evaluation.
"We didn't want to follow in that path," he said, "reporting information that was not technically sound or credible at the end. There were very few indicators that could be reported in a credible, valid way immediately."
He said the recommendations will be "phased in" over time. And he reassured state officials that "nothing is going to be dumped on you immediately that you can't live with."
Mr. Kaagan said that at least in the area of student achievement, the plan recommends that each state pay for testing its own students, with a possible infusion of federal funds in "appropriate" areas such as test development.
He added that states that already have statewide testing programs administered in the spring would only have to administer the proposed sample of naep test items to some of their students along with their regular test.
"We're attempting to do this in a fashion that binds onto existing testing programs as much as possible," he said.
Although he gave examples of possible indicators that may be included in the draft plan--such as attendance rates and the allocation of state funds--Mr. Kaagan cautioned that the list may change considerably before it is presented to the chiefs in November.