Assessment

Florida Scoring Glitch Sparks Broad Debate

By Scott J. Cech — June 12, 2007 5 min read
  • Save to favorites
  • Print

What started as a subtle flaw in the Florida Comprehensive Assessment Test of 3rd grade reading has widened into a full-scale debate with national implications: Is too much riding on one fallible assessment?

The mistakes in the 2006 exam were discovered last month, as state education officials were looking at this year’s test results and puzzling over a sudden performance drop. In contrast to 2006, when the proportion of 3rd graders reading at grade level jumped from 67 percent to 75 percent, and previous years, when that percentage rose only 1 to 3 points annually, the scores for 2007 fell back by 6 percentage points to 69 percent.

The nosedive on the test itself was significant enough, but in Florida, any ripple in FCAT performance can send waves of consequences crashing through the state’s education system. High school graduation is tied to the test. So are the annual letter grades, A through F, with which schools get labeled. Teachers’ merit pay is also contingent upon the scores, as is student promotion from 3rd to 4th grade.

In no other state, according to assessment experts, does so much hinge on a single exam, though they say others are following Florida’s lead.

“It’s getting more and more common—the pressure’s ratcheting up nationwide,” said Daniel M. Koretz, a professor of education at Harvard University and an associate director of the National Center for Research on Evaluation, Standards, and Student Testing, based at the University of California, Los Angeles. “The problem is that if something goes wrong, it makes it that much worse.”

What went wrong in 2006 was invisible to the untrained eye, and even to the trained ones at San Antonio-based Harcourt Assessment—the company that created the FCAT. It also slipped by the Florida Department of Education, which has accepted responsibility for the error and is regrading roughly 204,000 exams.

According to education department spokesman Tom Butler, some 2006 “anchor questions”—ones whose difficulty must remain consistent from year to year, though the questions themselves are different—“were probably easier for the kids than they should have been.”

Compounding the problem, those questions were moved from the back of the exam, where they were in 2005, to the front, artificially increasing the chance that students would do better on them. “A student is going to be fresher in the beginning,” Mr. Butler said.

Inquiry’s Scope Widens

Florida Commissioner of Education Jeanine Blomberg announced May 23 that 3rd graders who ought not have been promoted, based on their 2006 scores, would be given a pass. School grades will also stand, she said.

But it’s not yet known how the federal government will calculate schools’ adequate yearly progress under the No Child Left Behind Act, or whether other accountability measures indexed to the FCAT will be grandfathered in.

The FCAT Advisory Committee—a group of critics, parents, and others convened by Ms. Blomberg to pick experts to audit last year’s exam—has yet to recommend whether teachers might lose merit pay based on the 2006 results. But the panel has already widened its scope, and is now asking questions about the FCAT as a whole.

It’s far from alone. The Florida Coalition for Assessment Reform, a nonprofit organization opposed to the FCAT, has written an open letter to Gov. Charlie Crist, a Republican, asking for a probe of all recent FCAT scores. The state’s House Democratic Caucus also wants the governor to investigate whether the promotion of 3rd graders broke any state statutes requiring that promotion be based on student proficiency.

State Sen. Don Gaetz, the chairman of his chamber’s pre-K-12 committee, also has launched an inquiry. But he cautioned those whom he called “critics of accountability” not to throw the baby out with the bath water.

“If we found that less than 10 percent of X-rays in a hospital were being misread, we would not throw out all the X-ray machines in the building,” the Republican lawmaker said. “We would find out what went wrong and resolve that problem.”

Contract Renewal Ahead

Some observers say part of the problem is the testing companies themselves.

“Testing is a human endeavor, and humans make mistakes,” said Robert Schaeffer, the public education director of the National Center for Fair & Open Testing, or FairTest, a watchdog group based in Cambridge, Mass., that is opposed to misuse of standardized tests. But, he added, “humans make more mistakes when they’re stretched thin, which is the case with the testing industry … since No Child Left Behind.”

The 5-year-old federal NCLB law, which requires annual testing in all states in grades 3-8 and once in high school, has created “a real shortage of top-level testing professionals,” Mr. Schaeffer said. “But also at the lower end of the employment chain, there is a problem getting sufficient folks to process and score tests.”

He pointed to a recent rash of problems associated with state standardized tests, including a series of technical glitches that interrupted Virginia’s online Standards of Learning testing last month. Thousands of students will have to retake those exams.

Jay A. Diskey

Jay A. Diskey, the executive director of the Association of American Publishers’ School Division, a testing-industry group based in Washington, dismissed questions about test companies’ capacity.

“I see an industry that is performing well,” he said. “That’s not to say there aren’t occasional issues. I’m not seeing the strain that some pundits may be seeing.”

Harcourt Assessment’s contract with the Florida education department is up for renewal this year, and despite the problems with last year’s FCAT, Mr. Butler, the department spokesman, said the intent is to renew the contract.

David Miller

“[Testing companies] work hard, and they know what they’re doing,” said David Miller, a professor of educational psychology at the University of Florida who writes often about assessment. “With the amount of testing going on in this country, the testing companies are straining. It doesn’t take much for problems to appear.”

Related Tags:

A version of this article appeared in the June 13, 2007 edition of Education Week as Florida Scoring Glitch Sparks Broad Debate

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Data Webinar
Education Insights with Actionable Data to Create More Personalized Engagement
The world has changed during this time of pandemic learning, and there is a new challenge faced in education regarding how we effectively utilize the data now available to educators and leaders. In this session
Content provided by Microsoft
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
Accelerate Learning with Project-Based Learning
Earlier this year, the George Lucas Educational Foundation released four new studies highlighting how project-based learning (PBL) helps accelerate student learning—across age groups, multiple disciplines, and different socio-economic statuses. With this year’s emphasis on unfinished
Content provided by SmartLab Learning
School & District Management Live Online Discussion Principal Overload: How to Manage Anxiety, Stress, and Tough Decisions
According to recent surveys, more than 40 percent of principals are considering leaving their jobs. With the pandemic, running a school building has become even more complicated, and principals' workloads continue to grow. If we

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment Long a Testing Bastion, Florida Plans to End 'Outdated' Year-End Exams
Florida Gov. Ron DeSantis said the state will shift to "progress monitoring" starting in the 2022-23 school year.
5 min read
Florida Governor Ron DeSantis speaks at the opening of a monoclonal antibody site in Pembroke Pines, Fla., on Aug. 18, 2021.
Florida Gov. Ron DeSantis said he believes a new testing regimen is needed to replace the Florida Standards Assessment, which has been given since 2015.
Marta Lavandier/AP
Assessment Spotlight Spotlight on Assessment in 2021
In this Spotlight, review newest assessment scores, see how districts will catch up with their supports for disabled students, plus more.
Assessment 'Nation's Report Card' Has a New Reading Framework, After a Drawn-Out Battle Over Equity
The new framework for the National Assessment of Educational Progress will guide development of the 2026 reading test.
10 min read
results 925693186 02
iStock/Getty
Assessment Opinion Q&A Collections: Assessment
Scores of educators share commentaries on the use of assessments in schools.
5 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
iStock/Getty