Assessment

Impact of Paper-and-Pencil, Online Testing Is Compared

By Lynn Olson — August 30, 2005 3 min read
  • Save to favorites
  • Print

How students perform on computer-delivered tests depends, in part, on how familiar they are with the technology, concludes a set of studies conducted by the Princeton, N.J.-based Educational Testing Service.

The studies looked at how students performed when given mathematics and writing items from the National Assessment of Educational Progress by paper and pencil vs. computer. The results of the studies were released this month by the National Center for Education Statistics, which oversees the federal testing program.

In the math study, nationally representative samples of 4th and 8th graders in 2001 took a computer-based math test and a test of computer facility, among other measures. In addition, at the 8th grade level, a randomly selected control group of students took a paper-based exam containing the same math items as the computer-based test.

“Online Assessment in Mathematics and Writing: Reports From the NAEP Technology-Based Assessment Project, Research and Development Series” is available from National Center for Education Statistics.

Average scores for 8th graders taking the computerized test were about 4 points lower than for those taking the paper version, a statistically significant difference. On average, 5 percent more students responded to individual items correctly on paper than on a computer.

At both grade levels, students’ facility with the computer—based on hands-on measures of input speed and accuracy—predicted their performance on the online exam.

The writing study compared the performance of a nationally representative sample of 8th graders who took a computer-based writing test in 2002 with that of a second, nationally representative sample of 8th graders taking the same test on paper as part of the regular NAEP administration that year.

Results showed that average scores on the computer-based writing test generally were not significantly different from average scores on the paper-based exam. But, as with the math test, individual students with better hands-on computer skills tended to achieve higher online scores, after controlling for their level of paper writing skills.

Arnold A. Goldstein, the director of reporting and dissemination for the assessment division of the NCES, said that the findings suggest a possible problem in administering the national assessment online, but that further research is needed. “I think we would need to have a larger field test in a more traditional NAEP testing setting in order to determine that,” he said.

Mr. Goldstein added that, while this was a one-time study, the NCES—an arm of the U.S. Department of Education—may do further work in the future to explore the administration of the assessment online.

Scoring by Computer

The studies also examined the feasibility and costs of generating and scoring NAEP math and writing items by computer.

While the machine’s grades on simple math items were generally interchangeable with those of human scorers, that was less true for items requiring extended text responses. On those items, the computer tended to treat correct responses that were misspelled as incorrect, a technical shortcoming that could be addressed by including common misspellings in the automated scoring key or including a spell-check before an answer is submitted, according to the study’s authors.

On the writing test, however, automated scores did not agree closely enough with the scores awarded by human readers to consider the two types of scores interchangeable.

“That is something that needs to be considered in any further development work,” said Mr. Goldstein. “For example, whether a trend can be extended from a paper-and-pencil to an online administration of the assessment.”

Related Tags:

Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
What’s Next for Teaching and Learning? Key Trends for the New School Year
The past 18 months changed the face of education forever, leaving teachers, students, and families to adapt to unprecedented challenges in teaching and learning. As we enter the third school year affected by the pandemic—and
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Curriculum Webinar
How Data and Digital Curriculum Can Drive Personalized Instruction
As we return from an abnormal year, it’s an educator’s top priority to make sure the lessons learned under adversity positively impact students during the new school year. Digital curriculum has emerged from the pandemic
Content provided by Kiddom
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Equity & Diversity Webinar
Leadership for Racial Equity in Schools and Beyond
While the COVID-19 pandemic continues to reveal systemic racial disparities in educational opportunity, there are revelations to which we can and must respond. Through conscientious efforts, using an intentional focus on race, school leaders can
Content provided by Corwin

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Assessment Whitepaper
Proven Techniques for Assessing Students with Technology
Dr. Doug Fisher’s proven assessment techniques help your students become active learners and increase their chances for higher learning g...
Content provided by Achieve3000
Assessment Long a Testing Bastion, Florida Plans to End 'Outdated' Year-End Exams
Florida Gov. Ron DeSantis said the state will shift to "progress monitoring" starting in the 2022-23 school year.
5 min read
Florida Governor Ron DeSantis speaks at the opening of a monoclonal antibody site in Pembroke Pines, Fla., on Aug. 18, 2021.
Florida Gov. Ron DeSantis said he believes a new testing regimen is needed to replace the Florida Standards Assessment, which has been given since 2015.
Marta Lavandier/AP
Assessment Spotlight Spotlight on Assessment in 2021
In this Spotlight, review newest assessment scores, see how districts will catch up with their supports for disabled students, plus more.
Assessment 'Nation's Report Card' Has a New Reading Framework, After a Drawn-Out Battle Over Equity
The new framework for the National Assessment of Educational Progress will guide development of the 2026 reading test.
10 min read
results 925693186 02
iStock/Getty