Pitfalls and Progress in Moving from Paper- to Computer-Based Testing

By Sarah D. Sparks — April 28, 2018 4 min read
  • Save to favorites
  • Print

Asking students and schools to switch to a new mode of assessment does affect students’ performance, but problems might be more short-lived than some educators fear, finds a new study by the American Institutes of Research.

In the first year of online testing for the Partnership for Assessment of Readiness for College and Careers, or PARCC, researchers found that students who took the test online across grades 3-8 showed the equivalent of five months less academic progress in math and as much as 11 months less academic progress in English/language arts compared to the performance of students who took the test on paper.

“Switching to the online assessment led to much lower measured achievement,” said Benjamin Backes, senior researcher at AIR’s National Center for Analysis of Longitudinal Data in Education Research. Students of all demographic groups showed similar dips in achievement when they took computer-based tests, though the researchers found students who were already struggling the most in reading had bigger declines from changing the mode of the test.

Once schools and students had a year of online testing under their belts, though, the test-mode effects shrunk by half in English/language arts and by a third in math, found co-authors Backes and James Cowan of AIR.

The Bay State allowed districts to volunteer to move to online tests in 2015 or 2016, and about half of them—mostly high-performing districts—did so. In both years, the state developed pools of students taking paper-based and computer-based versions of the same tests, and used them to compare how similar students at the same schools performed on different versions.

Backes noted that some questions were presented differently on the paper version of the test:

... and the online version of the same question:

“It’s pure speculation, but ... in paper format, it does look somewhat different,” Backes said. “For a reading passage, it’s just all in front of you on paper, but you need to use a scroll bar to go through it [online], so maybe it’s harder to refer to certain parts of the passage?”

Massachusetts’ students are perennial top-performers on large-scale tests, from the consortium-based PARCC to the National Assessment of Educational Progress, to international tests like the Program for International Student Assessment or the Trends in Mathematics and Science Study. And the state’s Deputy Education Commissioner Jeff Wulfson said the state viewed the move from paper-and-pencil tests to computer-based assessment—which tests at all levels have undertaken in the last five years—as a “natural progression.”

“We were aware of some of the downsides. I don’t think any of it really surprised us,” said Wulfson. The state chose to move to online testing as “a companion to the fact that we are spending a lot of time and effort putting computers into the classroom for instruction,” Wulfson said. “Kids these days are spending more time writing on computers than pen and paper.”

Massachusetts has since moved away from the PARCC, and it has corrected for the mode effect statistically when releasing its test results for its own online state test, dubbed MCAS 2.0. The researchers are continuing to track how students perform when moving from one online test to a new one, to see if the handicap for online testing continues to shrink.

“It’s just a reminder that we as a state should not be making any high-stakes decisions on the basis of one test where the mode effect could be within the margin of error,” Wulfson said. He noted that 98 percent of Bay State students will be learning and being tested on a computer by next year. “I think five years from now it will almost be a non-issue.”

There are three possibilities for the state’s next steps, in Backes’ view. If the decline is just caused by students adjusting to a new test, the mode effect will disappear over time. If a few items are inherently more difficult to do online, the testing developer can adjust them.

But the researchers plan to conduct a predictive study as part of the next round of online assessment, to look at a third possibility: that the online tests are assessing slightly different and more complex skills than paper and pencil—and these differences actually make the tests a better predictor of students’ long-term academic achievement in the subject.

“Just because there are mode effects, that’s not necessarily a bad thing if they are telling us something about student performance or preparation that were not being captured before,” Backes said.

Bob Lee, the chief analyst for MCAS, noted that the state’s 2010 college- and career-readiness frameworks require students to be able to read and write online.

“We are seeing in the data our students getting used to writing online and our teachers are getting used to giving tests online,” Lee said, adding, “we’re learning more about the close relationship between keyboarding and literacy in early grades.”

Illustrations Source: PARCC, Benjamin Backes

A version of this news article first appeared in the Inside School Research blog.

Commenting has been disabled on effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
What’s Next for Teaching and Learning? Key Trends for the New School Year
The past 18 months changed the face of education forever, leaving teachers, students, and families to adapt to unprecedented challenges in teaching and learning. As we enter the third school year affected by the pandemic—and
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
How Data and Digital Curriculum Can Drive Personalized Instruction
As we return from an abnormal year, it’s an educator’s top priority to make sure the lessons learned under adversity positively impact students during the new school year. Digital curriculum has emerged from the pandemic
Content provided by Kiddom
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Equity & Diversity Webinar
Leadership for Racial Equity in Schools and Beyond
While the COVID-19 pandemic continues to reveal systemic racial disparities in educational opportunity, there are revelations to which we can and must respond. Through conscientious efforts, using an intentional focus on race, school leaders can
Content provided by Corwin

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment Spotlight Spotlight on Assessment in 2021
In this Spotlight, review newest assessment scores, see how districts will catch up with their supports for disabled students, plus more.
Assessment 'Nation's Report Card' Has a New Reading Framework, After a Drawn-Out Battle Over Equity
The new framework for the National Assessment of Educational Progress will guide development of the 2026 reading test.
10 min read
results 925693186 02
Assessment Opinion Q&A Collections: Assessment
Scores of educators share commentaries on the use of assessments in schools.
5 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
Assessment Standardized Tests Could Be in Jeopardy in Wake of Biden Decisions, Experts Say
Has the Biden administration shored up statewide tests this year only to risk undermining long-term public backing for them?
6 min read
Image of a test sheet.