Accountability

Accountability Systems ‘Mediocre,’ Study Finds

By Lynn Olson — February 18, 2004 4 min read
  • Save to favorites
  • Print

Not a single state in America has its whole system of standards, tests, and accountability policies “right,” argues a report from two Washington-based groups.

“Grading the Systems: The Guide to State Standards, Tests and Accountability Policies,” is available from the Thomas B. Fordham Foundation. For more information about study methodology, e-mail AccountabilityWorks at: contact@accountabilityworks.org. (Report requiresAdobe’s Acrobat Reader.)

“Grading the Systems: The Guide to State Standards, Tests, and Accountability Policies” judges states on the quality of their reading and math standards, the content of their tests, the match between their standards and tests, test rigor, their technical trustworthiness, and state accountability policies.

The project, underwritten by the Smith Richardson Foundation, evaluated 30 state systems in each category on a scale from 1 to 5, with 5 denoting “outstanding” performance.

Although some states—such as Massachusetts, Pennsylvania, and Virginia—got high marks in three out of six categories, the multistate average is only “mediocre,” according to the report published by the Thomas B. Fordham Foundation and AccountabilityWorks, a consulting group.

But what’s most notable about the review is what’s missing.

See Also...

Read the accompanying story, “Ratings by State.”

States were included in the study based on whether copies of the state-administered tests in reading and mathematics (or an equivalent form of such tests) could be obtained for analysis. A few states, such as Massachusetts, post previously administered tests on their Web sites. Colorado, Illinois, Michigan, New York, and Pennsylvania agreed to make secure test forms available for review. In other cases, the reviewers used an alternate form of an off-the-shelf test that was available from the publisher but not identical to the one given in the state.

But in the vast majority of cases, the authors complained, states would not release basic information that should be available.

“While we did feel that it was a significant accomplishment that we could look at 30 state testing systems, it was unfortunate that 20 states did not feel the need to share their tests under secure conditions,” said Theodor Rebarber, AccountabilityWorks’ president.

Given that state accountability systems drive so much of what happens at the district, school, and classroom level, he said, “there’s a need for some external, independent review of what’s under the hood, so to speak.”

Bill Reinhard, a spokesman for the Maryland education department, said officials there could not recall the request. But, “we rarely let any kind of researchers look at our tests, even under secure conditions, because we don’t own a lot of the test items.

“There’s a lot of hoops to go through,” he added, noting that the process is difficult and time-consuming, “and if we started to do that, there’s some concern that we’d have to start doing it for a lot of different organizations.”

No Guarantee of Quality

To review state systems, the project assembled a team of individuals with expertise in the relevant academic-content areas and grades. With their advice, AccountabilityWorks established a set of “reference standards” that cover what the organization considers to be essential math and reading skills at each grade level. The reviewers then judged state systems against those standards.

The project also reviewed the “trustworthiness,” or reliability, of state tests based on criteria devised with help from Susan E. Phillips, a psychometrician and lawyer in private practice.

The project examined both criterion-referenced tests crafted by states to specifically match their standards and commercially produced norm-referenced tests that are used as part of state accountability systems. The latter measure how students stack up against a nationally representative sample of their peers.

But the study didn’t find that one type of test was always better.

In math, for example, the reviewers discovered that the content of norm-referenced tests was significantly better than that of criterion-referenced tests in the elementary and middle grades, and significantly worse at the high school level. That’s because, with a few exceptions, the norm-referenced tests for high schools incorporated only limited amounts of high school math, such as algebra or geometry.

The authors also found that while states’ custom- designed tests were better aligned with state content standards than were off- the-shelf exams, “the difference is not nearly as large as one might expect.”

But Mr. Rebarber said the weakest dimension of state systems was the rigor of state tests, or where the states set the cutoff scores needed to perform at the proficient level. Even states with challenging standards and tests often had cutoff scores that more closely resembled “minimum competency expectations,” he said. Massachusetts alone earned a strong overall rating on test rigor.

The researchers concluded that 18 states would have solid accountability systems if the federal No Child Left Behind Act was “fully and properly implemented.”

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Start Strong With Solid SEL Implementation: Success Strategies for the New School Year
Join Satchel Pulse to learn why implementing a solid SEL program at the beginning of the year will deliver maximum impact to your students.
Content provided by Satchel Pulse
Teaching Live Online Discussion Seat at the Table: How Can We Help Students Feel Connected to School?
Get strategies for your struggles with student engagement. Bring questions for our expert panel. Help students recover the joy of learning.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Science Webinar
Real-World Problem Solving: How Invention Education Drives Student Learning
Hear from student inventors and K-12 teachers about how invention education enhances learning, opens minds, and preps students for the future.
Content provided by The Lemelson Foundation

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Accountability School Accountability Is Restarting After a Two-Year Pause. Here's What That Means
For a moment, the COVID-19 pandemic succeeded in doing what periodic protests about school accountability couldn't: Halting it.
10 min read
Illustration of a gauge.
4zevar/iStock/Getty
Accountability Opinion Let's Take a Holistic Approach to Judging Schools
Parents wouldn't judge their kids based on a single factor. So, says Ron Berger of EL Education, why must schools use a lone test score?
8 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
iStock/Getty
Accountability Opinion Are K-12 State Tests Like a Visit to the Pediatrician?
Even if the doctor’s trip isn’t pleasant, at least parents get something out of it they believe is worthwhile.
3 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
Accountability Opinion What Does the Future Hold for School Accountability?
Testing and accountability advocates have an opportunity to think anew about how to make the case for testing.
4 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty