Study Relates Cautionary Tale Of Misusing Data

By Lynn Olson — May 21, 2003 4 min read
  • Save to favorites
  • Print

Under federal law, states have to break out test results by race, ethnicity, and other student subgroups. But two Texas researchers caution that such data could be misused if educators aren’t better prepared to interpret the information.

Just such a situation, they say in a new paper, occurred at a school where the researchers were working. For five years, a research team from the University of Texas at Austin partnered with an unnamed, high-poverty urban high school to improve mathematics instruction.

At the end of that time, write researchers Jere Confrey and Katie Makar, the school had posted a 25 percent increase in the passing rate on the math portion of the state exit test, the Texas Assessment of Academic Skills.

But when the 2000 TAAS results came out, the passing rate in math for African-American students in the school dropped below the 50 percent that was deemed acceptable by the state. (The proportion that passed was 48.4 percent.) As a result, the school was labeled “low performing.”

The “No Child Left Behind Act” of 2001, which was modeled in part on the Texas accountability system, requires all public schools to disaggregate test-score data.

According to the authors of the Texas study, teachers were admonished to focus their instruction on raising the achievement of black students. The school suggested adjusting its improvement plan so that all African-American 9th and 10th graders would be assigned peer tutors. It also called for meeting with teachers from a neighboring high school that had gotten off the low-performing list by requiring all its black students to attend lunchtime tutoring.

At a meeting with district administrators to discuss the test results, the authors write, virtually no talk about the 25.5 percent of Hispanic students or 14.3 percent of white students who also failed the test took place.

The researchers expressed concern to school administrators that the proposals would unfairly target African-American students for specific interventions despite how they performed. Shortly thereafter, they say, their partnership with the school was abruptly terminated.

Natural Variation

In the paper, to be published later this year by Harvard University’s graduate school of education, the researchers note that in planning its improvement strategies, the school failed to look at the performance of African-American students over time or at the distribution of test scores.

Had they done so, conclude Ms. Confrey and Ms. Makar, whose paper was presented at a conference at Harvard this spring, they would have seen that the passing rate for black students had been on an upward trajectory for the past five years and generally mirrored improvements at the state and district levels.

The one-year drop in the passing rate for black students, based on a test group of only 31 African-American youngsters, still fell within one “standard deviation” of their projected passing rate, close enough so that the drop could have been caused by measurement or sampling errors, the paper says.

If one student had answered just one or two more questions correctly, the authors estimated, the school would not have earned the low-performing label.

Moreover, when Ms. Confrey and Ms. Makar compared the distribution of African-American students’ scores with those for nonblack 10th graders, they found a substantial overlap, with individual black students scoring across the spectrum.

“While we certainly recognize that a passing rate of only half the students is cause for concern and significant efforts at intervention,” they caution in the paper, “we believe that the policy should be based on better-informed ideas of variation.”

‘Bubble Kids’

Noting that many schools try to improve their performance by focusing on “bubble kids"—or those whose scores fall just below passing—the researchers undertook two analyses.

First, they looked at whether the state was closing the achievement gaps between different racial and ethnic groups in 10th grade math over time by looking at the percent of students in each group passing the exam and, then, at the average math score for each subgroup. When the latter method was used, any closing of such gaps became much less apparent.

Next, the authors examined the math-test scores of 7th graders at a Texas middle school compared with the same students’ scores the previous year. Surprisingly, of students who failed but were close to passing in grade 7, more than one-third had been high performers in 6th grade.

“This variation in the performance of particular students raised significant questions about what is being measured for these students each year,” write Ms. Confrey and Ms. Makar.

When the pair plotted the performance of 300 students in grades 8-10, chosen randomly across Texas, again they found that quite a few students performed differently on TAAS across the grades. While many students lost ground over the two-year period, a significant number performed substantially better.

“One of the concerns with just having a single measure is we sometimes see schools then acting on that measure to try and bring their scores up, rather than thinking about what the needs of the students are,” said Ms. Makar, a doctoral candidate in mathematics education.

Based on their analyses, the University of Texas researchers advise schools to consider statistics besides the percent passing in any subgroup before designing instructional strategies. Those include sample size; test performance over time; mean and median scores; and the distribution of scores by performance quartiles, because students in the lowest quartile often get ignored if their scores fall well below the passing standard.

“It’s important not to simply react to ‘fix’ the problem and get out of the situation,” cautioned Ms. Confrey, a professor of mathematics education. “One has to interpret the data in light of the activities at the school.”

Related Tags:


Jobs Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
Teacher Perspectives: What is the Future of Virtual Education?
Hear from practicing educators on how virtual and hybrid options offer more flexibility and best practices for administrative support.
Content provided by Class
Reading & Literacy Webinar How Background Knowledge Fits Into the ‘Science of Reading’ 
Join our webinar to learn research-backed strategies for enhancing reading comprehension and building cultural responsiveness in the classroom.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment A Huge Publisher and a Big Testing Company Are Teaming Up. What This Means for Educators
Four key questions to consider about how the pairing of Houghton Mifflin Harcourt and NWEA might affect educators.
3 min read
Students testing.
Assessment Spotlight Spotlight on Assessment
This Spotlight will help you examine updated testing guidance from the U.S. Dept. of Ed, analyze college-placement test scores, and more.

Assessment Opinion We Are More Than Our Grades: A Student's Perspective
Students have come to believe that their GPA and test scores are the ultimate reflection of their self-worth, writes a college senior.
Bailey Striepling
3 min read
Conceptual illustration of Students emerging from a field of giant discarded letter grades.
Vanessa Solis/Education Week and iStock/Getty Images
Assessment Letter to the Editor Grading for Growth Through Competency-Based Education
Competency-based education can better prepare today's children for tomorrow's challenges, writes this letter to the editor.
1 min read
Illustration of an open laptop receiving an email.