The federal government and dozens of states are slowly paving the way for widespread use of high-stakes online exams that adjust the difficulty of their questions based on the skill levels of individual test-takers.
But dueling congressional proposals to overhaul the Elementary and Secondary Education Act, currently known as No Child Left Behind, highlight lingering disagreements over how computer-adaptive assessment should function and have reignited long-standing concerns held by advocates for students with disabilities.
At the heart of the debate is the extent to which computer-adaptive exams should ask questions that are above or below a test-taker’s grade level. Skeptics worry that too much leeway, along with a relaxation of the federal requirement that state tests cover the full range of grade-level content, could result in dumbed-down assessments—and eventually dumbed-down instruction—for struggling students.
“Prior to NCLB, states were allowed to give fifth grade students a third grade test and call them proficient,” said Laura Kaloi, who co-chairs the education task force for the Consortium for Citizens with Disabilities, a Washington-based advocacy group that has 1,115 member organizations. “We don’t want computer-adaptive testing to go down a similar path.”
Full Potential
In the U.S. House of Representatives, a proposal to overhaul the ESEA would explicitly allow off-grade-level test items. Rep. Tom Petri, R-Wis., who sponsored an earlier bill that was folded into that broader legislation, said that approach would allow states “to use these new assessments to their full potential” by precisely pinpointing what students know and are able to do.
A competing Senate proposal to rewrite the law includes a requirement to assess students on grade level, but Ms. Kaloi said it leaves open a “loophole” that “bad actors” could exploit to deliver tests that are heavily off grade level.
Broadly speaking, the benefits of computer-adaptive testing are widely agreed upon: shorter, more secure exams; more precise information on what students know; and faster turnaround of test results for use by educators, administrators, and policymakers.
But the stakes surrounding the contentious details could be high.
Beginning in 2014-15, more than 20 states intend to administer high-stakes computer-adaptive tests to millions of children. Those exams, which are being created by the Smarter Balanced Assessment Consortium and will be tied to the Common Core State Standards, are expected to include a limited number of off-grade-level questions.
The U.S. Department of Education is funding the Smarter Balanced effort to the tune of $175 million, but the group’s exams will still need federal approval before they can be used.
Ms. Kaloi said disability-rights advocates are still in “wait-and-see mode” regarding Smarter Balanced’s approach.
In its present form, the ESEA neither prohibits nor favors any specific type of test—computer-adaptive or otherwise. But it does require students to be tested on the full depth and breadth of on-grade-level content, which has made it difficult to win federal approval to use some adaptive exams for accountability purposes.
New Breed of Tests
Learn how adaptive testing works in this 2012 video and read the related story, Adaptive Testing Evolves to Assess Common-Core Skills.
In the next version of the act, lawmakers from both parties and houses of Congress want the new breed of assessments officially acknowledged to ensure they can be used by states.
“I think I’ve sort of won these battles on recognizing the need to do computer-adaptive testing,” said Sen. Al Franken, D-Minn., who has pushed proposals on the issue since 2011.
Mr. Franken said the fixed-form NCLB-era assessments do not provide accurate measures of what struggling and advanced students know, creating a “race to the middle.”
Computer-adaptive assessments, on the other hand, rely on complex algorithms to feed students questions targeted to their individual skill levels based on their prior responses. The more questions a student gets right, the harder the subsequent questions will be.
A current Senate proposal sponsored by Sen. Tom Harkin of Iowa, a Democrat, and supported by Mr. Franken, to reauthorize the ESEA calls for all exams used for federal accountability purposes to assess students in two ways: whether they are performing at grade level or not, and the specific grade level at which they are performing.
Mr. Franken supports the inclusion of test questions that are above and below grade level, though proponents of the Senate legislation stressed that the bill requires all exams used for federal accountability purposes to measure the full range of grade-level standards so that lower-performing students are not held to lower expectations.
Ms. Kaloi said the Consortium for Citizens with Disabilities supports the overall Senate proposal sponsored by Mr. Harkin to revise the ESEA, but is disappointed with the bill’s language around assessment.
“We remain concerned about the loophole that could allow out-of-level testing,” she said, referring to the bill’s specific allowance to test for students’ grade-level performance, which Ms. Kaloi said could be open to interpretation and abuse.
Disability-rights groups do not support the House Republican proposal to overhaul the law, which Rep. Petri’s office said would also require states to measure students’ actual grade-level performance, as well as whether they are on grade level or not.
Ms. Kaloi pointed to exams created by the American Institutes for Research, which has been contracted by six states to provide computer-adaptive assessments, as an example of how the tests should protect struggling students.
On the Washington-based institutes’ 5th grade exams, for example, reading passages might vary from student to student according to the complexity of the language and concepts they contain. But all test-takers would be asked to engage in grade-appropriate higher-order thinking skills, such as making inferences about what characters in the passage might do next.
“It’s important that computer-adaptive testing be standards-based, meaning there is a blueprint that ensures every kid sees test questions that reflect the full breadth and depth of on-grade-level content,” said Jon Cohen, AIR’s executive vice president and director of assessment.
Delaware, Hawaii, and Oregon have all won federal approval for those types of adaptive exams and have been successfully using them for years, said Mr. Cohen. His organization’s approach, he said, helps ensure that struggling students are not limited to being assessed—or taught—only basic skills, such as recall of facts or rote applications of formulas.
Mr. Cohen decried a “big push” from lobbyists on behalf of “companies that sell adaptive tests that are not tied to a strong testing blueprint” to allow off-grade-level testing and tests that do not cover the full range of grade-level content.
“I think that would be a step backwards,” he said.
The biggest test of the viability of high-stakes computer-adaptive assessment will come in 2014-15, when many of the schools covered by the Smarter Balanced Assessment Consortium could struggle with the technical hurdles of administering hundreds of exams online at once.
Challenges Ahead
Figuring out an effective, equitable way to implement the exams’ adaptive functions will also be a challenge, said Joe Willhoft, the consortium’s executive director.
“We want to be very careful about our use of out-of-grade level items for students who are struggling,” he said. “If we do use [them], we would restrain ourselves to only one or two times” per test.
Such limits constitute a trade-off, Mr. Willhoft acknowledged.
“That approach doesn’t take as much advantage of the effectiveness of computer-adaptive testing as possible,” he said. “But it does adhere to the value that all students are assessed according to content standards.”
Smarter Balanced is still developing its bank of test items. A field test of the fully adaptive exams, in which roughly 20 percent of the students represented by the consortium will participate, will take place next spring.
Ms. Kaloi said that disability-rights groups won’t be able to effectively evaluate Smarter Balanced’s adaptive exams until closer to that time.
Mr. Willhoft said he did not anticipate any problems for states seeking Education Department approval to use Smarter Balanced exams to meet their accountability requirements.
“By virtue of its grant award, the department has clearly approved the overall design,” he said.
A spokesman for the department offered a different take.
“The Smarter Balanced Assessment Consortium exams, just like any other state test, will have to go through a new peer review if they will be used for accountability purposes,” said Daren Briscoe. “Being funded for the project does not obviate the need for the review.”
