NRC Panel: Rethink, Revamp Testing

By Lynn Olson — April 11, 2001 7 min read

The tests that are playing an increasingly central role in education need to be changed substantially to reflect new knowledge about how people think and learn, a report released last week asserts.

For More Information

The report is available from the National Academy Press for $45, plus $4.50 shipping and handling, by calling (202) 334-3313 or (800) 624-6242.

The report, by a committee of the National Research Council, argues that the combination of new research about cognition and advances in technology and measurement provides an “opportune time” to rethink the theoretical underpinnings of assessments.

“One of the central dilemmas in testing policy in the United States has been the collision between tests used for accountability purposes and the same tests’ being used to inform instruction,” said Michael J. Feuer, the executive director of the NRC’s Center for Education. “What this report gives us is a road map that would go a long way to connecting tests that can be useful for accountability but that can also provide information for teachers and students.”

Many existing tests, the report notes, focus on discrete bits of knowledge and skill rather than on the most complex aspects of student achievement. They do not chart students’ progress over time, nor do they provide adequate insights into students’ thinking strategies, the nature of their misunderstandings in particular subjects, or the types of help that could best improve learning.

The council’s 17-member Committee on the Foundations of Assessment proposes a “major program of research” that would synthesize current thinking about cognition and measurement in order to design new tests that could yield fairer and more accurate information about students.

It also recommends that states and districts sample a broader range of student competencies and understandings by using a variety of testing techniques or multiple measures of student performance.

“One of the things that we’re pushing for is more coordinated systems of assessments,” which might include such alternative measures as portfolios and tasks that students complete during the course of ongoing classroom instruction, said Naomi Chudowsky, the study’s director.

As part of that effort, the report advocates shifting more attention and resources toward the classroom assessments given by teachers and away from high-stakes, large-scale exams.

“The current educational assessment environment in the United States assigns much greater value and credibility to external, large- scale assessments of individuals and programs than to classroom assessment designed to assist learning,” the report says. “A vision for the future is that assessments at all levels—from classroom to state—will work together in a system that is comprehensive, coherent, and continuous.”

Although creating such coherent assessment systems would require more time and money, Mr. Feuer said, doing so is not unrealistic. He noted that the U.S. Senate has proposed $400 million in fiscal 2002 to help states create new tests. “Even on Capitol Hill, they know if they’re going to get this next round of testing right, it’s going to require a sizable investment,” he said.

Bits and Pieces

The National Research Council’s report, “Knowing What Students Know: The Science and Design of Educational Assessment,” argues that in the past four decades, studies of the human mind have provided considerable insight into how children develop conceptual understanding, how people acquire expertise in particular subjects, and which thinking processes are associated with competent performance.

At the same time, advances in measurement technology mean it’s now possible to characterize student achievement in terms of multiple aspects of proficiency rather than a single test score, to track students’ growth over time, and to analyze which factors contribute to student learning.

Despite such advances, the report says, most tests remain mired in the past, reflecting earlier theories of learning characterized as the step-by- step accumulation of facts, procedures, and definitions.

“What we have is an approach which literally takes bits and pieces of information and treats them as largely substitutes for one another, and doesn’t do justice to the nature of knowledge and understanding,” said James W. Pellegrino, a co-chairman of the NRC committee and a professor of cognitive studies at Vanderbilt University.

Although current tests do a reasonable job of measuring students’ knowledge of basic facts and procedures, the report says, they provide very limited information that educators can use to identify why students do not perform well or how teachers can best modify instruction.

A New Generation

The committee, whose work was supported by the National Science Foundation, suggests crafting a new generation of assessments founded on advances in cognitive research, technology, and measurement.

“If we would spend more time on careful, thoughtful crafting of the assessments and getting them to the point where they could answer the kinds of questions we want answered,” said Mr. Pellegrino, “then ultimately, these instruments would be far more valuable, and the investments would have a far greater return.”

The committee recommends basing assessments on an underlying theory of how student understanding develops in a content area. That theory, in turn, should be based on empirical studies and models of how people learn and exhibit expertise in a particular subject, such as geometry.

Such assessments should focus less on isolated facts and skills and more on how students organize factual and procedural knowledge in a subject, the report argues. Tests also should probe how students monitor or reflect on their own thinking—a process known as “metacognition” that is one of the hallmarks of expertise. And tests should examine the specific strategies that children use to solve problems.

To make assessments fairer, the committee also urges that students’ opportunities to learn the material being tested be taken into account both in designing tests and in interpreting children’s responses.

Too often, the report says, teachers’ classroom-based assessments mirror the same testing formats and scoring practices found in large-scale tests. The report advocates that instruction about assessments and about how students learn become a required part of teacher-preparation and professional-development programs. Such education should be linked to actual experiences in classrooms, both in assessing students and interpreting their development of competence, the report says. And it suggests that states require such teacher education through their licensing and accreditation standards.

In addition, the report calls for new ways of reporting test results that focus less on ranking students and schools and more on tracking students’ development of competence within a subject. Such reporting would include specific benchmarks and examples of where students stand on a continuum of learning.

Harnessing Technology

The panelists argue that alternatives to on-demand, standardized testing of every child already exist. If individual scores are needed, their report says, much richer information about children’s learning could be extracted from classroom work produced during the course of instruction, through such means as portfolios and tasks that students complete while learning the curriculum, but that are common across schools or districts.

If the primary purpose of the tests is program evaluation or accountability, the committee says, it may not be necessary to test every student.

The report is particularly optimistic that new advances in technology could help remove some of the constraints that have limited testing practices in the past. Sophisticated computer-based programs, for example, can track the sequence of actions students take to solve problems, analyze patterns of correct and incorrect reasoning, and provide rapid and informative feedback to students and teachers, while also serving accountability purposes.

One of the best examples, Mr. Pellegrino said, is “intelligent tutoring” systems, such as those used in algebra. Students answer questions on a computer, which matches their responses to a cognitive map of how student thinking in algebra develops over time.

When a student asks for help, the computerized “tutor” can estimate where the student is in his or her development and provide hints that are tailored to the student’s particular approach to the problem.

Bringing together scientists, educators, task designers, and psychometricians to work on such efforts, the report argues, could result in a “significant leap forward in the field of assessment.”

A version of this article appeared in the April 11, 2001 edition of Education Week as NRC Panel: Rethink, Revamp Testing


Classroom Technology Webinar Making Big Technology Decisions: Advice for District Leaders, Principals, and Teachers
Educators at all levels make decisions that can have a huge impact on students. That’s especially true when it comes to the use of technology, which was activated like never before to help students learn
Professional Development Webinar Expand Digital Learning by Expanding Teacher Training
This discussion will examine how things have changed and offer guidance on smart, cost-effective ways to expand digital learning efforts and train teachers to maximize the use of new technologies for learning.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
The Social-Emotional Impact of the Covid-19 Pandemic on American Schoolchildren
Hear new findings from an analysis of our 300 million student survey responses along with district leaders on new trends in student SEL.
Content provided by Panorama

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment Standardized Tests Could Be in Jeopardy in Wake of Biden Decisions, Experts Say
Has the Biden administration shored up statewide tests this year only to risk undermining long-term public backing for them?
6 min read
Illustration of students in virus environment facing wave of test sheets.
Collage by Vanessa Solis/Education Week (Images: iStock/DigitalVision Vectors/Getty)
Assessment How Can Teachers Better Understand Students? A New Breed of Assessment Will Try to Help
Researchers will work to create formative assessments that can give teachers a window into students’ emerging identities and strengths
4 min read
In this Nov. 19, 2020, file photo, sixth-grade students listen to instruction in class at Cape Cod Lighthouse Charter School in East Harwich, Mass.
Researchers hope to create new assessments to help teachers gain deeper insights into the identities and strengths of their students, like these 6th graders at Cape Cod Lighthouse Charter School in East Harwich, Mass.
Elise Amendola/AP
Assessment Opinion It's Time We Begin Using Assessments to Look Forward, Instead of Back
Schools do not get much value from high-stakes tests. Many are now allowing schools to use better assessments to guide student learning.
Seth Feldman
5 min read
shutterstock 19525837
Assessment Opinion Grading Has Always Been an Imperfect Exercise. COVID-19 Made It Worse
It’s hard reducing the complexity of each student’s social, emotional, and academic learning to a letter grade. Maybe we’re doing it wrong.
Lory Walker Peroff
4 min read
A student's grades are unknown
Robert Neubecker for Education Week