Wisconsin and Wyoming have been notified by the U.S. Department of Education that they cannot win approval to use the ACT to measure high school achievement until they submit “substantial” amounts of evidence supporting its use.
The two states learned of the education department’s decision in letters in December and January.
Those letters came as part of the department’s “peer review” process, which requires states to undergo periodic, detailed evaluation of their assessment systems. Peer review has been in place under the previous two versions of the federal education law, the No Child Left Behind Act and the Improving America’s Schools Act of 1994, and continues under the newest version, the Every Student Succeeds Act.
ESSA allows states to substitute “nationally recognized high school tests” such as the ACT or SAT instead of their own state tests to measure secondary school achievement. But states must win the approval of the federal department’s peer reviewers, just as they must do for the assessments they use in elementary and middle school.
In the most recent round of the peer-review process, Wisconsin and Wyoming both ran into problems getting full approval for their use of the ACT. Thirty-eight states submitted some or all of their assessment systems for review last year, but the decision letters for Wyoming and Wisconsin are the only ones to emerge so far with feedback on the ACT.
Sixteen states require all juniors to take the ACT. A few of those—including Wyoming and Wisconsin—use the ACT as a federal accountability test at the high school level.
Both states received letters from the U.S. Department of Education saying it could not approve their use of the ACT until the states supplied “substantial additional information” and specific kinds of evidence backing up its use. In the meantime, the letters said, the federal department would place conditions on part of the states’ federal Title I funding allocations, and would hold quarterly progress calls with them to track their progress.
ACT spokesman Ed Colby said the company is “eager to assist” Wyoming and Wisconsin in supplying the needed information to the U.S. Department of Education. He noted that many other states’ testing systems also were rated less than “substantially meets” requirements—the standard of approval in the peer-review process. Typically, states work with the department over a period of months to provide the evidence requested until the department is satisfied that their assessment systems are valid for their intended use.
“The current peer review letters don’t state that the ACT is not compliant,” Colby wrote in an email. “They simply state that there are some aspects and additional data that US ED wants to see. We intend to support those states in responding to requests for more information.”
The Education Department’s letter to Wyoming requested documentation that “independent” alignment studies were conducted showing that the ACT fully reflects the state’s academic content standards. It also wanted documentation on the accommodations provided to students with special needs who take the ACT (which has been a sore subject for both the ACT and College Board). Additionally, U.S. education officials asked Wyoming to provide evidence that the ACT—as well as its own state tests, the Proficiency Assessments for Wyoming Students, or PAWS—was “administered with fidelity to test administration procedures.”
The U.S. Department of Education’s letter to Wisconsin asked the state for evidence that the ACT “measures the full range of the state’s academic content standards. Like the letter to Wyoming, this letter also asked for independent evidence of alignment between the ACT and the state’s standards. The letter also asks Wisconsin to provide evidence to support its use of ACT results to report writing and reading scores for accountability purposes.
Wisconsin’s letter also raised the question of whether some students might have been disadvantaged on the ACT. Federal officials asked it to provide “differential item functioning” analyses that show whether certain kinds of questions, such as essays or performance tasks, “function differently for relevant student groups.”
The state must also supply evidence showing that aspects of the exam “do not provide inappropriate barriers for measuring the achievement of all students.” It must clarify “what specific accessibility tools are available to all students, including students with disabilities,” and demonstrate that it has a process to determine that the accommodations it provides allow fair access for English-learners and students with special needs.
The letter to Wisconsin raised another issue: whether there is too much variation in the way different scorers score the writing tests, a phenomenon the assessment world calls “inter-rater reliability.” Wisconsin must submit evidence about the way scorers were trained, and how “range-finding"—establishing what performance should look like in specific score bands—was performed.