Guest blog post by Carmen Constantinescu
High school students in the class of 2017 who take the ACT college admission exam will see their writing scores reported on a new scale starting September 2016.
The new reporting will show scores ranging from 2-12, a similar scale to the one originally used for the writing test but eliminated in 2015 in favor of a wider 1-36 scale. The 1-36 scale is used for all other ACT tests. The testing company said the return to the previous scale was needed to remove confusion among test users.
“Students were comparing their writing scores to their other multiple-choice scores and many of them thought that their scores were lower than they should have been based on that comparison,” stated Ed Colby, the senior director of media and public relations for ACT.
However, Adam Ingersoll, one of the founders and principals of a California-based test preparation company, Compass Education, disagrees. “The scale of 1-36 had an extremely large standard error of measurement which means that the test is essentially not reliable and the former 2-12 scale essentially masks that,” he said. In an analysis published on the company’s website by Art Sawyer, also a founder and principal of Compass Education, the same score is shown to correspond to different percentile ranks when placed on the previous and current scale. (A percentile rank is a measure of relative position which indicates the percentage of score equal or below a given score.)
Here is an example: The writing test, which uses a single response to an essay prompt to measure the domains of ideas and analysis, development and support, organization, and language use and conventions, used to receive two scores of up to 6 points by two independent readers which were, then, summed up for a final score. In its new version, however, although the score range stays the same for the domains, scoring by the two raters will be calculated as an average and then rounded up. For instance, two sets of hypothetical domain scores on an essay by raters—one of 4, 3, 4, 3 and another of 4, 4, 4, 3—would earn a raw score of 29 corresponding to a scaled score of 21 (on the 1-36 scale) and a percentile rank of 84. But, these same scores, when calculated through averaging and rounding on the new 2-12 scale, would yield a final score of 7.25 which, when rounded, would correspond to a 59th percentile rank.
ACT defends its decision and explains that, “our research suggests again that scores on the writing test perform virtually the same as they did in the previous writing test. Any differences that exist now also existed before,” said Colby. Instead of looking at scaled scores, ACT also recommends that test users—students, parents, or colleges—use percentile ranks when interpreting them.
“Comparing ACT subject scores without referencing the percentile ranks (i.e., norms) can lead to misinterpretation and use,” concludes an ACT report on the subject. Colby also refuted any concerns with regard to the reliability of the writing test: “The writing test has strong reliability but it is a different type of test than the multiple-choice tests; it’s based on an essay as opposed to a number of multiple-choice questions, it’s based on ratings by individual graders as opposed to just counting the numbers of incorrect answers, so there is going to be more variability there and that’s not surprising,” argued Colby.
Ingersoll is not convinced, though. “They’ve pitched this to us today as eliminating confusion but I’d say they haven’t eliminated confusion by removing its root cause. They have simply hidden it from view,” he concluded.
A version of this news article first appeared in the High School & Beyond blog.