Study Relates Cautionary Tale Of Misusing Data

By Lynn Olson — May 21, 2003 4 min read
  • Save to favorites
  • Print

Under federal law, states have to break out test results by race, ethnicity, and other student subgroups. But two Texas researchers caution that such data could be misused if educators aren’t better prepared to interpret the information.

Just such a situation, they say in a new paper, occurred at a school where the researchers were working. For five years, a research team from the University of Texas at Austin partnered with an unnamed, high-poverty urban high school to improve mathematics instruction.

At the end of that time, write researchers Jere Confrey and Katie Makar, the school had posted a 25 percent increase in the passing rate on the math portion of the state exit test, the Texas Assessment of Academic Skills.

But when the 2000 TAAS results came out, the passing rate in math for African-American students in the school dropped below the 50 percent that was deemed acceptable by the state. (The proportion that passed was 48.4 percent.) As a result, the school was labeled “low performing.”

The “No Child Left Behind Act” of 2001, which was modeled in part on the Texas accountability system, requires all public schools to disaggregate test-score data.

According to the authors of the Texas study, teachers were admonished to focus their instruction on raising the achievement of black students. The school suggested adjusting its improvement plan so that all African-American 9th and 10th graders would be assigned peer tutors. It also called for meeting with teachers from a neighboring high school that had gotten off the low-performing list by requiring all its black students to attend lunchtime tutoring.

At a meeting with district administrators to discuss the test results, the authors write, virtually no talk about the 25.5 percent of Hispanic students or 14.3 percent of white students who also failed the test took place.

The researchers expressed concern to school administrators that the proposals would unfairly target African-American students for specific interventions despite how they performed. Shortly thereafter, they say, their partnership with the school was abruptly terminated.

Natural Variation

In the paper, to be published later this year by Harvard University’s graduate school of education, the researchers note that in planning its improvement strategies, the school failed to look at the performance of African-American students over time or at the distribution of test scores.

Had they done so, conclude Ms. Confrey and Ms. Makar, whose paper was presented at a conference at Harvard this spring, they would have seen that the passing rate for black students had been on an upward trajectory for the past five years and generally mirrored improvements at the state and district levels.

The one-year drop in the passing rate for black students, based on a test group of only 31 African-American youngsters, still fell within one “standard deviation” of their projected passing rate, close enough so that the drop could have been caused by measurement or sampling errors, the paper says.

If one student had answered just one or two more questions correctly, the authors estimated, the school would not have earned the low-performing label.

Moreover, when Ms. Confrey and Ms. Makar compared the distribution of African-American students’ scores with those for nonblack 10th graders, they found a substantial overlap, with individual black students scoring across the spectrum.

“While we certainly recognize that a passing rate of only half the students is cause for concern and significant efforts at intervention,” they caution in the paper, “we believe that the policy should be based on better-informed ideas of variation.”

‘Bubble Kids’

Noting that many schools try to improve their performance by focusing on “bubble kids"—or those whose scores fall just below passing—the researchers undertook two analyses.

First, they looked at whether the state was closing the achievement gaps between different racial and ethnic groups in 10th grade math over time by looking at the percent of students in each group passing the exam and, then, at the average math score for each subgroup. When the latter method was used, any closing of such gaps became much less apparent.

Next, the authors examined the math-test scores of 7th graders at a Texas middle school compared with the same students’ scores the previous year. Surprisingly, of students who failed but were close to passing in grade 7, more than one-third had been high performers in 6th grade.

“This variation in the performance of particular students raised significant questions about what is being measured for these students each year,” write Ms. Confrey and Ms. Makar.

When the pair plotted the performance of 300 students in grades 8-10, chosen randomly across Texas, again they found that quite a few students performed differently on TAAS across the grades. While many students lost ground over the two-year period, a significant number performed substantially better.

“One of the concerns with just having a single measure is we sometimes see schools then acting on that measure to try and bring their scores up, rather than thinking about what the needs of the students are,” said Ms. Makar, a doctoral candidate in mathematics education.

Based on their analyses, the University of Texas researchers advise schools to consider statistics besides the percent passing in any subgroup before designing instructional strategies. Those include sample size; test performance over time; mean and median scores; and the distribution of scores by performance quartiles, because students in the lowest quartile often get ignored if their scores fall well below the passing standard.

“It’s important not to simply react to ‘fix’ the problem and get out of the situation,” cautioned Ms. Confrey, a professor of mathematics education. “One has to interpret the data in light of the activities at the school.”

Related Tags:


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Achievement Webinar
How To Tackle The Biggest Hurdles To Effective Tutoring
Learn how districts overcome the three biggest challenges to implementing high-impact tutoring with fidelity: time, talent, and funding.
Content provided by Saga Education
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Reframing Behavior: Neuroscience-Based Practices for Positive Support
Reframing Behavior helps teachers see the “why” of behavior through a neuroscience lens and provides practices that fit into a school day.
Content provided by Crisis Prevention Institute
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Mathematics Webinar
Math for All: Strategies for Inclusive Instruction and Student Success
Looking for ways to make math matter for all your students? Gain strategies that help them make the connection as well as the grade.
Content provided by NMSI

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment What the Research Says What Teachers Should Know About Integrating Formative Assessment With Instruction
Teachers need to understand how tests fit into their larger instructional practice, experts say.
3 min read
Students with raised hands.
E+ / Getty
Assessment AI May Be Coming for Standardized Testing
An international test may offer clues on how AI can help create better assessments.
4 min read
online test checklist 1610418898 brightspot
Assessment The 5 Burning Questions for Districts on Grading Reforms
As districts rethink grading policies, they consider the purpose of grades and how to make them more reliable measures of learning.
5 min read
Grading reform lead art
Illustration by Laura Baker/Education Week with E+ and iStock/Getty
Assessment As They Revamp Grading, Districts Try to Improve Consistency, Prevent Inflation
Districts have embraced bold changes to make grading systems more consistent, but some say they've inflated grades and sent mixed signals.
10 min read
Close crop of a teacher's hands grading a stack of papers with a red marker.