Vendors at Odds Over Nevada Testing Problems

When breakdowns disrupted Nevada’s online testing last month, state officials were left searching for answers—and they blamed a vendor charged with administering the test, as well as the consortium of states that designed the common-core-aligned exams.

But the testing provider accused of botching the assessments, Measured Progress, has said the responsibility lies at least partly with another vendor, the American Institutes for Research, which it claims was late in delivering critical software needed to make sure the exams could be administered smoothly.

The dust-up is the latest of the battles that have erupted in a number of states, including California and New Mexico, over common-core testing contracts.

But Nevada’s woes underscore another reality, at least for states belonging to the Smarter Balanced Assessment Consortium: In some cases, they’re counting on different testing vendors—who may be rivals—with different responsibilities to help them carry out ambitious new online exams under tight deadlines.

“I’m extremely frustrated at the disruption this has caused in our state. Nevada was ready. Our system was there,” said Steve Canavero, the state’s deputy superintendent for student achievement.

The AIR, a Washington-based nonprofit, received approximately $6 million of its $14.3 million contract from Smarter Balanced to develop an “open source” platform for delivering the test in Nevada and other consortium states that wanted to use that technology. But Measured Progress and Nevada officials say the technology was not ready, and Smarter Balanced also says that portions of the code were not delivered to states until earlier this year.

The AIR, however, has defended its performance.

In 2012, Smarter Balanced contracted with the AIR to build a test-delivery system to register students, present the assessment items to them, and deliver the results from the tests to reporting systems. The contract was awarded in collaboration with DRC to modify the AIR’s existing test-delivery system to meet Smarter Balanced specifications.

DRC, based in Maple Grove, Minn., declined to comment for this article.

The consortium’s announcement about the contract award had highlighted the proven platform the AIR would use as the foundation for the open-source software.

“AIR’s system is used by several states to deliver statewide online assessments,” the press release said. “It is compatible with the range of technology in schools, including computers, laptops, and tablets—requiring only a secure Internet browser for administration. The system will be optimized for release in its open-source version in fall 2014.”

The open version would be designed to allow any company or developer to build upon, or enhance, the technology because its code would be publicly available. That was necessary, so different vendors could access the code and use it for test administration.

Smarter Balanced states made independent decisions about which vendor they would contract with to actually administer the assessments. Eleven states chose the AIR. Montana, Nevada, and North Dakota chose Measured Progress, a Dover, N.H.-based assessment company.

Smarter Balanced is one of two major state consortia that devised tests tied to the Common Core State Standards. Most states have begun administering such tests this year.

Signs of Trouble

The initial problems in Nevada happened in mid-March, when the state had to delay the start of Smarter Balanced tests by two weeks because the system did not perform predictably, said Mr. Canavero.

“The code wasn’t functioning like it needed to function,” he said. “The system wasn’t able to be stress-tested at a level where we could feel confident that our students could start a testing session and have a productive testing experience.”

Even after the two-week delay in March, test administration continued to be plagued by the inability of students to log in or to complete their exams. Multiple patches provided by the AIR that the organization said would solve the issue failed to do so, Mr. Canavero said. And when students in Nevada’s Clark County schools—the fifth-largest district in the country, with an enrollment of about 318,000—tried to start the tests, the system also balked, he said.

On April 15, the state had to halt the exams altogether, and after problems continued, it resumed testing only on a limited basis.

Nevada officials sent a letter notifying Measured Progress and Smarter Balanced that the state deemed them to be in “breach of contract.”

Just 7,000 students statewide were able to take the Smarter Balanced exams on April 24, Mr. Canavero said.

In response to the breach-of-contract letter, Measured Progress released its own statement. Without naming names, the company pointed to “multiple and delayed deliveries of the source code” that it said was necessary to ensure that the Nevada assessment could be sufficiently tested so it would perform as expected when large numbers of test-takers were on it.

Of 19 major components required from the AIR for the open-source test delivery platform, five were released in January or later, and one was delivered in November, which was after the Sept. 30, 2014 release of most of the code, according to Smarter Balanced. The consortium indicated in an email that the AIR made some of the source code or precursors to it available months before the final delivery dates.

‘Minimal Delays’

In response to questions about the AIR’s performance, Jon Cohen, the president of its assessment division, said his organization had been charged with delivering an enormously complex system across multiple states, under tight deadlines, and has performed well.

“There have been minimal delays,” Mr. Cohen said in a phone interview. He added that most software products experience delays in the development and release process.

“Our scheduled delivery of the system was in September,” he said. “We got most of it delivered in time and [provided] updates after that.”

Martin Borg, the president and CEO of Measured Progress, said in a statement that his company performed “exhaustive quality-assurance testing every time we received updated code from the Smarter Balanced vendor.” However, he said, the delays in receiving the code from the vendor—he did not identify the AIR by name—limited the ability of Measured Progress to predict how the assessment-delivery system would perform during peak traffic times.

Of the roughly 840,000 Smarter Balanced tests slated to be taken by about 211,000 students in Nevada, between 8 percent and 23 percent of the exams had been completed by last week, with students finishing a greater share of their English/language arts exams than their math tests. Those figures are far off pace for all students in grades 3-8 to finish their Smarter Balanced exams, according to Mr. Canavero.

The state testing window officially closes June 12, but many Nevada districts end their school years in May.

Problems Beyond Nevada

Major controversies about testing have emerged in other states, too.

In Montana, Measured Progress increased the computer-server capacity after issues arose with delivery of Smarter Balanced tests there. On April 15, Superintendent of Public Instruction Denise Juneau announced that the common-core-aligned tests would be voluntary because of a delay in opening the testing window.

In California, Pearson, an education company based in the United Kingdom, wants state officials to re-bid the testing contract that was awarded to the Lawrenceville, N.J.-based Educational Testing Service. Pearson contends the contract, which could be worth a quarter of a billion dollars, was “invalid” and “illegal,” on the grounds that Pearson was the lowest bidder, public records were destroyed in the procurement process, and one of its proposed strategies wound up in the winning ETS proposal.

In New Mexico, meanwhile, the AIR is suing the state to stop a contract award to Pearson for work associated with the Partnership for Assessment of Readiness for College and Careers, or PARCC, the other main consortium developing common-core tests. The AIR is arguing that the proposal for the work unfairly favored Pearson because it was bundled with work the company had already done for PARCC. The AIR estimates the maximum value of the contract to be $1 billion.