Study Puts Results of International Tests on Common Metric

By Sean Cavanagh — June 16, 2009 5 min read

International tests known by odd acronyms like PISA and TIMSS have become fixed in the American educational and political vernacular. Newspaper editorial writers and elected officials at all levels hash over U.S. students’ scores on those nation-by-nation exams with a zeal and frequency they once reserved for the release of state and local test results.

Now an American researcher has attempted to make those comparisons more meaningful to the public, by comparing the performance of students in U.S. states and cities against that of their foreign peers using a well-understood metric: letter grades.

A new study, “The Second Derivative: International Benchmarks in Mathematics for U.S. States and School Districts,” shows students in almost all U.S. states and cities performing at a C+ or C level. That is below the B grade that the study’s author contends should be the international target for American students, and which is achieved by the highest-performing nations.

In addition, the study reveals that the gap between a B and a C grade is a “quantum leap,” according to the author of the report, Gary W. Phillips, a vice president and a chief scientist at the American Institutes for Research, in Washington. The gap that separates the United States from top-performing jurisdictions like Hong Kong is about the size of the gulf between the highest-achieving U.S. state in the study, Massachusetts, and the lowest, Mississippi.

“We’re below the international benchmark, and we’re way below it,” Mr. Phillips said in an interview. The goal of the study, he said, is to compare U.S. and international performance in a clear fashion, which “resonates with readers and does it in a way that’s scientifically justifiable.”

Letter Grades Assigned

The study allows grade comparisons across cities, states, and nations. Florida and Washington state, for instance, earn C+ grades in 4th grade math, which the study shows as comparable to the performance of England, Germany, and Denmark, among others.

In two previous studies, Mr. Phillips compared the performance of students in U.S. states and cities, respectively, on an international scale. He did so by statistically linking the results of a prominent global exam, the Trends in International Mathematics and Science Study, or TIMSS, with the results of a U.S. test, the National Assessment of Educational Progress, or NAEP.

City Grades: A Global Perspective

A study of international achievement grades nations, U.S. states, and select cities on a common scale, using American jurisdictions’ NAEP scores to estimate their performance on an international test, TIMSS. The OECD figure represents the average for developed nations.


SOURCE: American Institutes for Research

In his latest effort, Mr. Phillips uses the cut scores on the TIMSS test to fashion a new statistical comparison with NAEP, and then equates scores with letter grades ranging from A to “BD,” or below D. Linking the two tests is possible, Mr. Phillips says, because TIMSS and NAEP are administered to equivalent, representative samples of students, in the same grades.

He selects a B grade as the U.S. target because his methodology shows that students who reached the “proficient” level on NAEP are statistically comparable to those reaching the “high” international level on the TIMSS.

American students’ performance, as judged by this metric, is mixed.

In 4th grade math, roughly half of U.S. states perform above the C+ mean for developed nations, according to the report. Just five states—Massachusetts, Minnesota, New Jersey, New Hampshire, and Kansas—reach the international target grade of B, and one state, Vermont, scores a B-minus in 4th grade.

In 8th grade, the U.S. states’ results were even less impressive, with only Massachusetts scoring a B-minus. Those scores fit a pattern: On the whole, American students’ math performance was weaker at 8th grade than at 4th grade, a disparity that has emerged in other international studies. (“U.S. Students’ Scores Drop By 8th Grade,” Dec. 13, 2000.)

Of 11 large U.S. cities studied, the majority scored a C grade at the 4th and 8th grade level, and none reached a B grade. Just two—Charlotte, N.C., and Austin, Texas—notched scores above the estimated TIMSS score, in both 4th and 8th grade. The District of Columbia schools had the lowest scores for both 4th and 8th grade.

No jurisdiction in the study—not even top-performing nations like Japan—earned an A grade. That result caused one expert on testing, Sharif Shakrani of Michigan State University, to question whether Mr. Phillips’ methods had set too high a bar, which might in turn make the U.S. results look worse than they actually are. Mr. Shakrani, who directs his university’s Education Policy Center, has helped develop math content for NAEP.

“Nobody got an ‘A,’ so is this [goal] a pie in the sky?” Mr. Shakrani said in an interview. “A very, very, good student should be able to reach that level,” he argued, as should a top-tier nation.

U.S. States as Models?

Mr. Shakrani attributed the drop-off in U.S. performance between 4th and 8th grade to American middle schools’ tendency to track students into middle grades math classes—algebra, pre-algebra, and very basic math—of strikingly different quality. Higher-performing nations have more uniform expectations for what math content is taught and tested, he said.

Comparing States on an International Scale

Most U.S. states meet or exceed the average of developed nations in 4th grade math.


SOURCE: American Institutes for Research
* U.S. Department of Defense schools

Hal Salzman, a professor of public policy at Rutgers University, said in an e-mail that the study reinforces an “unproved assumption” that international test scores are connected to a country’s ability to compete economically. Mr. Salzman has argued that U.S. policymakers misinterpret international exam results and exaggerate worries about American performance.

“Using these scores and international comparisons too often substitutes for substantive discussion of what is needed for real education improvement,” Mr. Salzman added.

More variation in test scores occurs within countries, among their own student populations, than from one nation to the next, Mr. Salzman said. American policymakers would be better off focusing on closing gaps between high- and low-perfoming states and districts, than on nation-by-nation comparisons, he argued.

In an e-mail, Mr. Phillips said that, on his grading scale, it is possible, albeit very difficult, for jurisdictions to obtain an A. He chose a B grade as the international target because it was “high, but achievable.”

Mr. Phillips responded to Mr. Salzman’s point by saying that it is important for U.S. officials to be aware of the high standards set by foreign countries.

But he agreed with the Rutgers professor that American policymakers should probe the successes of states like Massachusetts. Those states’ high grades, he wrote in the study, show “it is possible in the United States for students to learn mathematics at a level that is competitive with the best in the world.”

Related Tags:

A version of this article appeared in the June 17, 2009 edition of Education Week as Study Puts Results Of International Tests On Common Metric


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
Interactive Learning Best Practices: Creative Ways Interactive Displays Engage Students
Students and teachers alike struggle in our newly hybrid world where learning takes place partly on-site and partly online. Focus, engagement, and motivation have become big concerns in this transition. In this webinar, we will
Content provided by Samsung
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Classroom Technology Webinar
Educator-Driven EdTech Design: Help Shape the Future of Classroom Technology
Join us for a collaborative workshop where you will get a live demo of GoGuardian Teacher, including seamless new integrations with Google Classroom, and participate in an interactive design exercise building a feature based on
Content provided by GoGuardian
School & District Management Live Online Discussion A Seat at the Table With Education Week: What Did We Learn About Schooling Models This Year?
After a year of living with the pandemic, what schooling models might we turn to as we look ahead to improve the student learning experience? Could year-round schooling be one of them? What about online

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

International Global Test Finds Digital Divide Reflected in Math, Science Scores
New data from the 2019 Trends in International Math and Science Study show teachers and students lack digital access and support.
3 min read
Image of data.
International Pre-COVID Learning Inequities Were Already Large Around the World
A new international benchmarking highlights gaps in training for digital learning and other supports that could deepen the challenge for low-income schools during the pandemic.
4 min read
International Part of Global Trend, 1 in 3 U.S. High Schoolers Felt Disconnected From School Before Pandemic
UNESCO's annual report on global education progress finds countries need to make more effort to include marginalized students, particularly in the United States.
4 min read
International How Schools in Other Countries Have Reopened
Ideas from Australia, Denmark, and Taiwan can help American district and school leaders as they shape their reopening plans.
11 min read
Students at the Taipei American School in Taipei, Taiwan, perform The Little Mermaid in full costume and masks.
Students at the Taipei American School in Taipei, Taiwan, perform The Little Mermaid in full costume and masks.
Photo courtesy of Dustin Rhoades/Taipei American School