The board that oversees the National Assessment of Educational Progress wants to reduce the number of questions asked of students and teachers that accompany the federally sponsored exams.
A draft of “Background Information Framework for the National Assessment of Educational Progress” is available from the National Assessment Governing Board. (Requires Microsoft Word.)
But at a meeting here this month, researchers and others urged caution in revising what many view as a key feature of the “nation’s report card.”
In addition to monitoring student achievement, NAEP gathers information on students’ experiences in and out of school through hundreds of background questions posed to students, teachers, and administrators at participating sites.
The National Assessment Governing Board, which sets policy for NAEP, held the May 1 meeting to gather comments on a proposed framework for rethinking such questions. The board hopes to focus them more squarely on information needed to report test results or that has a proven relation to student achievement.
While generally supportive of the board’s efforts, respondents recommended prudence in revamping a closely watched part of NAEP’s data-gathering efforts.
The in-school and out-of-school information collected by NAEP is “a crucial tool” for fostering intelligent discourse about education policy and practice, according to Sandra Feldman, the president of the American Federation of Teachers. “Test scores surely tell us something,” she said in written testimony. “But without context, there is little hope of analysis.”
Need for Improvement
Jamel Abedi, a professor of education at the University of California, Los Angeles, said that any changes would “surely have significant impact on the analyses and reporting of NAEP,” including the ability to compare results across time.
Agreement is widespread, however, that the questionnaires, which have been part of NAEP since its inception 34 years ago, need to be improved. The number of background questions has grown considerably over the years, and the governing board, known as NAGB, is concerned both about their quality and the burden on respondents, particularly given the increased frequency Congress has mandated for the assessment.
Some parents and policymakers also contend that a number of those questions are an invasion of privacy and focus too much on students’ beliefs and attitudes.
The “No Child Left Behind” Act of 2001 requires states to take part in NAEP tests in reading and mathematics in grades 4 and 8 every other year. The law, a reauthorization of the Elementary and Secondary Education Act, gives the governing board new authority to approve the background questions. Previously, the board simply advised the Department of Education.
NAGB has proposed refocusing the questions on the assessment’s primary mission: providing a fair and accurate measure of student achievement. Information gathered on students’ race, ethnicity, and socioeconomic status, for instance, helps officials report performance for different student populations.
The board also has proposed maximizing the use of data from other sources in order to limit the amount of background data that NAEP must collect. And it has proposed cycling in some background questions periodically, preferably for limited samples of students, rather than asking the same questions every time.
Many of those who testified agreed with the board’s general principles.
Sally Kilgore, a former director of the office of research in the Education Department’s office of educational research and improvement, advised the board to invest in a “few metrics” of high quality. In particular, she applauded NAGB’s proposal to formulate a better measure of students’ socioeconomic status than their eligibility for subsidized lunches, perhaps by devising an index based on multiple indicators.
“This could be a substantial contribution to this country,” Ms. Kilgore argued.
But Paul W. Holland, a measurement expert with the Princeton, N.J.-based Educational Testing Service, contended that “a huge effort will be put into getting nowhere with that.”
He recommended that the board focus, instead, on collecting better measures of educational practices. “NAEP can give a better description of what is happening in schools to particular groups of students than any other vehicle,” he said, arguing that those portraits are its greatest contribution to education policy.
Others expressed worries that the board would go too far in its quest to reduce the amount of contextual information NAEP collects, either jettisoning some information entirely or collecting it on too infrequent a basis.
“Rather than having fewer noncognitive items, NAEP should have more,” said Harold Wenglinsky, an associate professor at the City University of New York’s Baruch College of Public Affairs. He noted that secondary analyses of NAEP have had a substantial influence on educational research.
Patte Barth, a senior associate with the Washington-based Education Trust, said some contextual questions are so important for students’ opportunity to learn that they should be monitored every two years.
She cited information on teacher credentials; student coursetaking; subject-specific resources, such as access to science labs and libraries; and teachers’ instructional practices. Because such information is central to understanding the achievement gaps between students of different backgrounds, she added, “any background question worth asking is worth reporting by group.”
While some people at the NAGB session questioned the quality of teachers’ self-reports, Ms. Barth and others said ways were available to improve measures of educational practices. “We would not like to see these questions eliminated,” she said, though acknowledging that the number could be streamlined.
Out-of-School Factors
In making decisions about which information to collect, Ms. Barth counseled the board to focus on factors “that are within the school’s control, since these can more easily be changed by public policy.”
But others pressed NAGB to retain information about out-of-school factors that may have a strong relationship to student achievement.
“Given that the research has consistently shown that out-of-school factors explain the lion’s share of variance in achievement, and given, too, that many of these factors, like in-school ones, are neither fixed nor immutable, they should be an important priority for improved background-data collection,” said Ms. Feldman of the AFT.
In one of their rare instances of agreement, both the AFT and the Washington-based Heritage Foundation, a conservative think tank, suggested that the board reconsider producing a parent questionnaire.
Patrick F. Fagan, a fellow at the foundation, also urged the board to collect more information on family structure and on students’ participation in religious communities because of what he says is the critical import of such factors to children’s sense of belonging.
“Perhaps you’re aware of the law, which prohibits NAEP from getting into intrusive questions of family beliefs and attitudes?” John H. Stevens, the NAGB member who is spearheading development of the framework, countered. “How could we possibly ask this in a way that could be viewed by ordinary citizens, as well as members of Congress, as nonintrusive?”
The No Child Left Behind Act prohibits NAEP from asking about “personal or family beliefs and attitudes.” It also requires all questions to be “secular, neutral, and nonideological.”
Mr. Stevens, who is the executive director of the Texas Business and Education Coalition, asked why NAEP, in particular, would be the appropriate or best vehicle for collecting such information.
In an answer that reflects the growing pressures on NAEP, Mr. Fagan responded: “Because it is at the center of the education policy discussion.”
After the meeting, Mr. Stevens pointed out that with limited resources, the governing board has to weigh tradeoffs. “Do we expand the background questions,” he asked, “or maybe grow the samples” to provide more accurate reporting for subgroups of students?
“We need to really be sure that we get the essential part of our mission as right as we can.”
“Expansion is fine,” agreed Thomas H. Fisher, another board member and the director of assessment services for the Florida education department, “as long as Congress and the president enact the necessary changes and provide the necessary funding.”
