Add Maryland to the list of states to find that students tended to score better when taking the 2014-15 PARCC exams on paper instead of by computer.
Paper results in the state were higher than online results for every tested grade in English/language arts and on higher-level math exams, according to an analysis prepared by the state department of education and presented to the state board of education Tuesday.
Differences between the student populations who took the exams via different modes of administration accounted for a portion of the score discrepancies, according to the analysis. But a bigger factor was students’ readiness for the technology platform, including developing extended written responses on a computer and responding to multimedia content.
“A lot of this has to do with getting our instruction on board [with the new tests and Common Core State Standards they are meant to assess] and making students more aware of what’s required for an adequate response,” said Doug Strader, the assessment director for the Maryland education department, in an interview.
Those findings echo a pattern of score discrepancies by mode of administration found across the multi-state Partnership for the Assessment of Readiness for College and Careers, one of two consortia to develop and administer last year new tests aligned with the common core. The apparent mode-of-administration effect was first reported by Education Week earlier this month.
Colorado, Illinois, and Rhode Island have found similar patterns within their states. The Baltimore County schools, one of the largest districts in Maryland, also found such a pattern, then used statistical techniques to isolate the impact of testing format. The district found that after controlling for student and school characteristics, students were between 3 percent and 9 percent more likely to score proficient on the paper-and-pencil version of the math exam, depending on their grade level. Students were 11 percent to 14 percent more likely to score proficient on the paper version of the E/LA exam.
According to the new analysis from the Maryland education department, 80 percent of the more than 876,000 Maryland students who took the PARCC exams last spring did so online.
In math, the advantage for paper-based test-takers was most pronounced on upper-grade and higher-level tests, such as 8th grade math and Algebra I and II. About 55 percent of students who took the Algebra I exam on paper scored proficient or advanced, for example, compared to fewer than 30 percent of those who took the exam online.
The advantage for paper-based test-takers was found in every grade (3-8 and 10) on the English/language arts exam. Among 4th graders, for example, just over 50 percent of students who took the E/LA test on paper scored proficient or advanced, compared to fewer than 40 percent of those who took the exam online.
Paper-best test-takers tended to be higher-performing on previous state exams, a difference that accounted for about 40 percent of the discrepancy between paper and online results on this year’s PARCC. African-American students, English-language learners, students from low-income families, and students with disabilities did not appear to be significantly disproportionately affected, according to Strader.
And there was “no evidence of technical flaws in the development, administration, scoring, or reporting of the assessment,” according to the state analysis.
That leaves student readiness for technology as the biggest factor explaining the score discrepancies. In other words, scores in some cases reflected not what students know and are able to do, but their familiarity and capabilities with the computer-based platform on which the test was administered.
Test items that required more involved responses—for example, developing a longer written response to be entered via computer—appeared to be the biggest problem in the online version of the exams, according to the state report.
“Through informal discussions with test-takers of both modes, there was a common theme of not knowing what was expected to score well on the extended response items [via computer],” according to the report. “Students shared that they would default to the size of the response box to better determine how much was enough.”
Students also described having little experience “developing extended responses from start to finish with the use of technology,” according to the report.
Several states have dropped the PARCC exams since last school year, but the overall push for more online testing appears to be moving forward unabated.
Maryland schools will again administer the PARCC exams, during a 30-day window between April 4 and June 10. The state expects the test to be fully online by next school year, with the exception of students who require an accommodation.
Like many states, Maryland has delayed using the results of PARCC exams for high stakes, such as student graduation or teacher evaluation, during the transition.
Photo: A student at Marshall Simonds Middle School in Burlington, Mass., reviews a question on a PARCC practice test before 2014 field-testing of the computer-based assessments.--Gretchen Ertl for Education Week-File
See also: