Rhode Island PARCC Scores Lower on Computer-Based Exams

Rhode Island students who took the 2014-15 PARCC exams by computer tended to score lower than those who took the exams by paper, raising further questions about the validity and usefulness of results from the tests taken last school year by more than 5 million students in the multi-state Partnership for the Assessment of Readiness for College and Careers.

The differences were sharpest on the English/language arts exam, where 42.5 percent of Rhode Island students who took the test on paper scored proficient, compared to 34 percent of those who took the test by computer. A spokesman for the state department of education said the variability in scores appears to be due in large measure to “student and system readiness for technology.”

Findings vary among other states that took part in PARCC testing last year:

• Illinois has also found higher proficiency rates among paper-based test takers, as Education Week first reported earlier this month.
  
• Colorado has uncovered similar trends in specific grade-subject combinations.
  
• Maryland officials have completed an analysis comparing students’ PARCC results by mode of administration, but are not releasing the results until later this month.
  
• Officials in the District of Columbia say they did not have enough paper-based test-takers to conduct a full analysis that controlled for student characteristics.
  
• Mississippi officials say they are currently taking a closer look at their state’s results.
  
• Officials in Arkansas, the District of Columbia, Ohio, Louisiana, New Jersey, and New Mexico said they have not conducted analyses searching for a possible “mode effect,” generally because the vast majority of students in their states took the exams via a single format, or because the state has since dropped the PARCC exams.
  

The results from Rhode Island, where almost 22 percent of students took the PARCC exam on paper, provide further fuel to concerns raised last week, when Education Week reported that PARCC scores tended to be lower on computer-based exams. Officials from the consortium acknowledged that the differences were likely attributable in part to students’ familiarity with the computer-based test delivery platform, rather than their academic knowledge and skill.

The score discrepancies vary across states, districts, grades, and subjects, with the advantage for paper-based test-takers appearing to be most pronounced in English/language arts and in upper-grade math.

Such differences in scores pose a serious short-term concern, especially given the high stakes that some states and districts intend to attach to the PARCC results, according to assessment experts consulted by Education Week. But long-term, a possible “mode effect” on the exams should be seen as a technical challenge that should dissipate over time, said Morgan Polikoff, an assistant professor of education at the University of Southern California.

“There’s no question in my mind that moving to computerized testing makes a lot of sense, but the process is not necessarily going to be smooth,” Polikoff said in an interview.

“That suggests the need for caution in how the results are used in the first few years, but it doesn’t suggest that we need to abandon the idea of computer-based testing altogether,” he said.

Comparing Paper-Pencil and Computer Test Scores

For more than two decades, researchers have found signs that some students tend to do slightly worse on computer-based versions of an exam, for reasons that have more to do with their familiarity with technology than with their academic knowledge and skills.

One challenge in conducting such analyses is that the students who take each format of an exam may have different demographic and academic backgrounds.

In December, officials in the 111,000-student Baltimore County school district sought to account for that possibility of differences in student populations in an analysis of the district’s 2014-15 PARCC results, using statistical techniques to isolate the impact of testing format. They found that after controlling for student and school characteristics, students were between 3 percent and 9 percent more likely to score proficient on the paper-and-pencil version of the math exam, depending on their grade level. Students were 11 percent to 14 percent more likely to score proficient on the paper version of the ELA exam.

It’s also important for states and districts to look at possible mode effects among different subgroups of students, said Polikoff of USC.

“The concern is if there are differential effects for different groups,” Polikoff said. “If, for example, taking tests on computers causes score declines among low-income but not high-income students, that would be a problem.”

A 2015 working paper from the National Center for Educational Statistics found such differences among 4^th graders taking a computer-based pilot version of the NAEP writing exam.

As a result, it’s important that states such as New Jersey, which are already administering exams almost entirely online, to consider the impact of the mode of test administration, Polikoff said.

“If it’s harder for some kids to do well on computer-based tests, that could affect within-state comparisons, even if everyone is taking the test on computer,” he said. “There still could be effects that are systematically biasing results against certain students, teachers, or schools.”

Most states with an overwhelming majority of online test takers have declined to conduct any such analysis, however.

Below is a summary of the information that states that took part in PARCC testing during the 2014-15 school year have to date provided to Education Week:

Arkansas

Of the roughly 245,000 students in the state who took the 2014-15 PARCC exams, about 93 percent took the tests online, according to a spokeswoman for the state department of education.

State officials have not conducted any analyses to compare results by mode of administration, nor do they have any plans to conduct such an analysis in the future.

Arkansas has since dropped PARCC and will administer the ACT-Aspire exams this April. The state expects administration of the new exams to be 100 percent online, except for students who need accommodations, beginning this year.

Colorado

The state department of education found that 12.2 percent of Colorado 3^rd

graders took the PARCC English/language arts exam on paper, and these students tended to score higher than their classmates who took the exams by computer. There were some signs of a similar trend on upper-grades math exams, although drawing clean comparisons was tricky because of varying test participation rates.

In the lower grades, where roughly 12 percent of Colorado students took PARCC math exams on paper, students scored similarly on the PARCC math exam, regardless of testing format.

In Colorado, the paper-based version of the PARCC ELA exam was only available as an accommodation to students in grades 4-11; roughly one-tenth of 1 percent of students in those grades took the paper-based ELA exam, the vast majority of whom were students with disabilities and English language learners.

About 474,000 students took the 2014-15 PARCC exams in Colorado. It was the state’s third year of online testing.

Results from last year’s PARCC exams will not be used for state accountability or teacher evaluation purposes this year, thanks to a one-year pause approved by the state general assembly.

PARCC math and English/language arts exams will be given in the state this April.

District of Columbia

In the D.C. schools and the district’s public charter schools, 92 percent of the more than 31,000 students who took the PARCC exams used a computer-based delivery format, according to a spokeswoman with the Office of the State Superintendent of Education.

That high percentage of online test-takers meant that officials could not construct a representative sample of paper-based test-takers that would allow for a comparison of scores by testing format that controlled for students’ demographic and academic backgrounds. District officials could not provide a descriptive comparison of proficiency rates among paper- and computer-based test-takers.

“We are confident in the results and our focus continues to be on using those results to help students, families and educators understand the critical concepts students need to master to be successful at the next grade level and be prepared for college and careers,” according to a statement from the OSSE. “More than 95 percent of students will test online this spring and we expect 100 percent to test online by 2017, other than those students who need a paper-based test as an individualized education program (IEP) accommodation.”

D.C.'s PARCC tests will be administered between March 28 and May 20 of this year.

Illinois

A comparison of online and paper-and-pencil scores done by the state board’s data-analysis division was also posted on the board’s website and reported on last week by Education Week. Here was our original breakdown of the Illinois scores:

State education officials declined interview requests. The state will again administer PARCC exams this spring.

Louisiana

All 316,000 or so Louisiana students who took the 2014-15 PARCC exams did so on paper. As a result, the state has not conducted any analysis to compare scores by mode of administration, according to a spokesman with the state department of education.

Louisiana has since moved to bar schools from using PARCC test items for the majority of its 2015-16 exams. The upcoming Louisiana Educational Assessment Program exams, to be administered April 25-29, will include both PARCC items and a majority of locally developed test items.

Maryland

Of the roughly 877,000 Maryland students who took the 2014-15 PARCC exams, 81 percent took the tests online, according to an analysis provided by the state department of education.

State officials have completed an analysis of PARCC results by mode of test administration, but results will not be shared publicly until a February 23 meeting of the state board of education, according to a spokesman for the state education department.

Maryland districts will be giving this year’s PARCC exams during a 30-day window of their choosing, between April 4 and June 10.

The state has delayed the use of PARCC results for high-stakes purposes such as student graduation and teacher evaluation until the 2016-17 school year.

Mississippi

Of the roughly 250,000 students Mississippi students who took the 2014-15 PARCC exams online, roughly 82 percent did so online, according to an analysis provided by the state department of education.

Officials are currently investigating possible differences in scores by mode of administration, as well as some concerns over the establishment of cut scores, according to a spokeswoman.

“The MDE is continuing to evaluate the PARCC results prior to their inclusion into the state’s accountability model,” according to a statement. “Our initial quality control, along with anecdotal information from the field, has increased the degree of analysis being conducted, to include, but not limited to, examining mode effects and score volatility at the ‘cut scores’ for Levels 3 and 4.”

Mississippi has since dropped PARCC and will administer Mississippi Assessment Program exams in April and May.

New Jersey

Of the more than 550,000 New Jersey students who took the 2014-15 PARCC exams, 99.4 percent did so by computer, according to a spokesman with the state department of education.

Concerns regarding a possible mode effect “simply isn’t an issue in New Jersey, because virtually all of New Jersey’s students took the PARCC assessments on computer.”

Regular administration of PARCC exams in New Jersey will take place between April 11 - May 20.

New Mexico

Of the nearly 377,000 New Mexico students who took the 2014-15 PARCC exams, nearly 93 percent did so online, according to a spokesman for the state department of education. The state has not conducted any analyses to compare results by mode of test administration.

New Mexico will administer the PARCC exams again this school year.

“We expect to have nearly all students testing online later this year,” according to the spokesman. “As we have said from the beginning, this was a new test for students in our state last year and we expect that as students and teachers become more familiar with the expectations and format of the test, our students will rise to the challenge.

Ohio

A spokeswoman for the Ohio Department of Education said that 65 percent of the state’s students who took the 2014-15 PARCC exams did so online. The department was unable to provide an exact number of test-takers. No analysis has been done to compare results by mode of administration, and no such analysis has been planned.

Ohio has since dropped the PARCC exams. Tests developed by the American Institutes for Research will be given this school year.

Rhode Island

Of the roughly 151,000 students in the state who took PARCC exams in 2014-15, 21.6 percent took the test on paper, while 78.4 percent took the exams online.

Across all grades, 42.5 percent of Rhode Island students who took the PARCC English/language arts exam scored proficient, compared to 34 percent of those who took the exam online.

Across all grades, 26.8 percent of Rhode Island students who took PARCC math exams on paper scored proficient, compared to 24.4 percent of those who took the exam online.

The scope of the differences varies by district, school, subject and grade.

In a statement, state Commissioner of Elementary and Secondary Education Ken Wagner said the “variability appears to be related primarily to student and system readiness for technology.”

“We have no plans to revise the intended use of our PARCC results during this period of transition,” Wagner said. “Our goal is to complete the transition to an all-online test administration by spring 2017, other than for students with individual accommodations.”

See also:

• PARCC Scores Lower for Students Who Took Exams on Computer
• Comparing Paper-Pencil and Computer Test Scores: 7 Key Research Studies