Special Report

Quality of Questions on Common Tests at Issue

By Stephen Sawchuk — January 21, 2010 9 min read
  • Save to favorites
  • Print

Most experts in the testing community have presumed that the $350 million promised by the U.S. Department of Education to support common assessments would promote those that made greater use of open-ended items capable of measuring higher-order critical-thinking skills.

But as measurement experts consider the multitude of possibilities for an assessment system based more heavily on such questions, they also are beginning to reflect on practical obstacles to putting such a system into place.

The issues now on the table include the added expense of those items, as well as sensitive questions about who should be charged with the task of scoring them and whether they will prove reliable enough for high-stakes decisions. Also being confronted are matters of governance—the quandary of which entities would actually “own” any new assessments created in common by states and whether working in state consortia would generate savings.

“The reality is that it does cost more to base a system on open-ended items, no question about it,” said Scott Marion, the vice president of the Dover, N.H.-based Center for Assessment, a test-consulting group, who is advising several states. “If the model we’re thinking about has got to be on-demand and high-stakes and used in systems with scores that are returned quickly, then it’s going to cost a lot.”

Higher Costs?

State dependence on multiple-choice testing under the federal No Child Left Behind Act has led to a backlash by those who say the tests, while cheap and technically reliable, come at a cost: not measuring complex cognitive skills.

Uphill Climb

A push for open-ended assessments comes after states reduced their use in the wake of the No Child Left Behind Act.


Source: Government Accountability Office

Using a slice of money from the $4.35 billion Race to the Top Fund, created last year under the American Recovery and Reinvestment Act, U.S. Secretary of Education Arne Duncan has called for state consortia to craft richer item types aligned to common standards that would include constructed-response questions, extended tasks, and performance-based items in which students would apply knowledge in new ways. (“Stimulus Seeks Enriched Tests,” Aug. 12, 2009.)

His department is expected to open up a competition in March for the assessment aid from the stimulus law. Work is under way, meanwhile, on a national project to produce a “common core” of academic standards for adoption by states.

No Redos

Assessment experts caution that open-ended test items carry with them a number of practical challenges. For one, items that measure higher-order skills are generally more expensive to devise, depending on how extensive the items are and how much of the total test such items make up.

For instance, with their detailed prompts and scenarios, questions that require students to engage in extensive writing or to defend their answer choices often are “memorable,” meaning the items can’t be reused for many years and must be replaced in the meantime.

Wes Bruce, the chief assessment officer for the Indiana education department, recalled one prompt that required 5th graders to write about what would happen if a kangaroo bounded into the classroom.

“All across the state, kids were talking about the prompt,” he said. “From an assessment perspective, that’s not good. Teachers will use it [subsequently] as an example for classroom work.”

The scoring process for open-ended items is also far more complicated than sticking a bunch of test papers into a computer scanner. It relies on “hand scorers” who are trained according to a scoring guide for each question and a set of “anchor papers” that give examples of performance on the item at each level. Each open-ended item typically goes through multiple reviews to ensure consistent scoring.

Depending on the complexity of the item and how long it takes to score, the costs can increase dramatically. A short constructed-response item with four possible scores might take one minute to score, said Stuart R. Kahl, the president of Measured Progress, a nonprofit test contractor based in Dover, N.H. But an extended performance-based or portfolio item might take up to an hour, he said.. With test scorers paid in the range of $12 to $15 an hour, such costs would add up.

For a midsize state with about 500,000 students within the tested grades and subjects, the scoring of tests based even partly on constructed-response items would make up more than a fifth of the total annual contract cost, Mr. Kahl estimated.

Scoring Scenarios

For some, the idea of expanding human-scored items raises issues of reliability: Performance-based items are typically less mathematically reliable than those based entirely on multiple choice.

Todd Farley, a 15-year veteran of the test-scoring business who detailed his experiences in a recent book, Making the Grades, is among the skeptics. In the book, Mr. Farley alleges that the scoring guidelines for open-ended items were frequently counterintuitive, and that as a “table leader”—an individual supervising other scorers’ work—he occasionally changed other reviewers’ scores.

Though test publishers interviewed for this story dismissed Mr. Farley’s account, independent sources do point to areas of concern. At least two reports issued by the Education Department’s office of inspector general last year, for instance, found lapses in Florida’s and Tennessee’s oversight of test contractors charged with scoring open-ended items.

In part to ameliorate the errors and costs associated with human scoring, test publishers are investing heavily in automated-response systems that use artificial-intelligence software to “read” student answers.

Such programs are already in use to minimize the number of human scorers needed for major tests such as the Graduate Record Examination. A handful of states, including Indiana, have piloted the technology for their own tests, and some experts, like Joan Herman, a director of the National Center for Research on Evaluation, Standards, and Student Testing, or CRESST, a group of assessment researchers headquartered at the University of California, Los Angeles, believe that the technology will be widespread within five years.

“When that happens, it will open up entirely new windows for doing more complex, open-ended items on large-scale assessments,” Ms. Herman said.

But such systems are not perfect, and experts including Mr. Bruce, in Indiana, noted that one of their limitations is that they typically don’t generate interpretative information about where a student needs to improve.

And in an era in which most teachers are distanced from the assessment process, unions and other stakeholders argue that teachers should have a greater role in the development and scoring of assessments, which can serve an important function for gathering information on student performance.

Teacher scoring of assessments has generally been eschewed under the NCLB law, the current version of the Elementary and Secondary Education Act, in part because of fears that the law’s accountability pressure would cause educators to inflate their students’ scores. But other countries have tackled the problem by using a blind-scoring process, in which teachers meet in central locations to score exams with names removed.

Hiring substitute teachers and paying the scoring teachers for their time would be costly as well. But there are benefits in the form of professional development, said Marcia Wilbur, the executive director of curriculum and content development for the College Board’s Advanced Placement program, which uses such a system to grade open-ended essays on AP exams.

Teachers, Ms. Wilbur said, spend a week looking at sample responses at each level of the scoring guides before they begin to score, and the guides often help them better understand students’ learning progressions.

“Teachers will bare their souls about what they do in their classrooms while discussing the samples,” she said. “It’s not that they go back to their classrooms and teach to the test, but they have a better understanding of the skills and the features of the skills that students struggle with.”

Another model, drawn from international practice in places such as Australia, England, Hong Kong, and New Zealand, would rely on teachers’ scoring assessments that are embedded within classroom curricular activities. Kentucky used such a system in the days before the 8-year-old NCLB law.

Not only could such assessments provide better feedback on where instruction needs to be improved, but they also could be included in state gauges of achievement if there were a central auditing process to ensure appropriate administration and reliable scoring, CRESST’s Ms. Herman said.

Consortia Challenges

Aside from costs, the problem with systems that rely heavily on teacher scoring, Mr. Kahl argued, is that results cannot be scored as quickly or efficiently as those done by machine. That means it could prove tough to turn results around under the quick timeline envisioned by the NCLB law and current state accountability systems.

Both Ms. Herman and Mr. Kahl said it might be possible to build systems that coupled curriculum-embedded assessments, scored throughout the year, with information from more-typical “on demand” standardized tests. But such a model hasn’t been tested in the United States.

“While most people agree that some amount of analyzing student work is good professional development, every teacher probably doesn’t have to do 100 papers to get the full value of it,” Ms. Herman added.

Also uncertain is which entities would actually own test forms or items developed in common, and which would bear the responsibility for updating them. Under the current system, such decisions are now made through contracts with individual states.

A common test, Mr. Marion of the Center for Assessment said, argues for some kind of open-source setup, but the details are “dicey,” he concedes. “The reality is that if this is paid for out of stimulus funds, the country should own them,” he said, “but at some point, someone has to house these items.”

Mr. Kahl, meanwhile, warns that one of the major selling points about state consortia—cheaper tests—might yield only limited savings for some states. Consortia would probably ease costs more for small states, where test development is a high percentage of overall assessments, but might not help larger states, where most testing costs involve factors such as printing, scoring, and reporting results.

Such questions of governance, finance, and sustainability have drawn the concern of policy experts, too.

Doubts and Hopes

Chester E. Finn Jr., the president of the Thomas B. Fordham Institute, warned in a recent article for the Washington think tank’s newsletter that the federal competition could lock in features of an assessment system that would be difficult to change in the future.

He also worries that the Obama administration’s ambitious goals for the assessment funding—which include generating information about both school and student performance as well as data about teacher effectiveness—could prove to be irreconcilable.

“If all the glitterati ... remains in the grant competition, anyone that wants to win the competition is going to have to pretend they can do all those things,” Mr. Finn said in an interview. “But since we know that they can’t all be done by the same assessment, in the same period of time at a finite price, something will get left in the dust.”

But for all those challenges, state testing experts hope to see breakthroughs with the federal funding.

“They are expecting that there be some innovation in the assessment area,” Mr. Bruce of Indiana said. “As a state that has been committed to using what at one point were innovative item types, and is still looking at ways to innovate in the scoring, it’s exciting.”

Coverage of the American Recovery and Reinvestment Act is supported in part by grants from the William and Flora Hewlett Foundation, at www.hewlett.org, and the Charles Stewart Mott Foundation, at www.mott.org.
A version of this article appeared in the January 27, 2010 edition of Education Week as Open-Ended Test Items Pose Challenges


School Climate & Safety K-12 Essentials Forum Strengthen Students’ Connections to School
Join this free event to learn how schools are creating the space for students to form strong bonds with each other and trusted adults.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Reading & Literacy Webinar
Creating Confident Readers: Why Differentiated Instruction is Equitable Instruction
Join us as we break down how differentiated instruction can advance your school’s literacy and equity goals.
Content provided by Lexia Learning
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
IT Infrastructure & Management Webinar
Future-Proofing Your School's Tech Ecosystem: Strategies for Asset Tracking, Sustainability, and Budget Optimization
Gain actionable insights into effective asset management, budget optimization, and sustainable IT practices.
Content provided by Follett Learning

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Standards Explainer What’s the Purpose of Standards in Education? An Explainer
What are standards? Why are they important? What's the Common Core? Do standards improve student achievement? Our explainer has the answers.
11 min read
Photo of students taking test.
F. Sheehan for EdWeek / Getty
Standards Florida's New African American History Standards: What's Behind the Backlash
The state's new standards drew national criticism and leave teachers with questions.
9 min read
Florida Governor and Republican presidential candidate Ron DeSantis speaks during a press conference at the Celebrate Freedom Foundation Hangar in West Columbia, S.C. July 18, 2023. For DeSantis, Tuesday was supposed to mark a major moment to help reset his stagnant Republican presidential campaign. But yet again, the moment was overshadowed by Donald Trump. The former president was the overwhelming focus for much of the day as DeSantis spoke out at a press conference and sat for a highly anticipated interview designed to reassure anxious donors and primary voters that he's still well-positioned to defeat Trump.
Florida Governor and Republican presidential candidate Ron DeSantis speaks during a press conference in West Columbia, S.C., on July 18, 2023. Florida officials approved new African American history standards that drew national backlash, and which DeSantis defended.
Sean Rayford/AP
Standards Here’s What’s in Florida’s New African American History Standards
Standards were expanded in the younger grades, but critics question the framing of many of the new standards.
1 min read
Vice President Kamala Harris speaks at the historic Ritz Theatre in downtown Jacksonville, Fla., on July 21, 2023. Harris spoke out against the new standards adopted by the Florida State Board of Education in the teaching of Black history.
Vice President Kamala Harris speaks at the historic Ritz Theatre in downtown Jacksonville, Fla., on July 21, 2023. Harris spoke out against the new standards adopted by the Florida state board of education in the teaching of Black history.
Fran Ruchalski/The Florida Times-Union via AP
Standards Opinion How One State Found Common Ground to Produce New History Standards
A veteran board member discusses how the state school board pushed past partisanship to offer a richer, more inclusive history for students.
10 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty