International

Study Puts Results of International Tests on Common Metric

By Sean Cavanagh — June 16, 2009 5 min read
  • Save to favorites
  • Print

International tests known by odd acronyms like PISA and TIMSS have become fixed in the American educational and political vernacular. Newspaper editorial writers and elected officials at all levels hash over U.S. students’ scores on those nation-by-nation exams with a zeal and frequency they once reserved for the release of state and local test results.

Now an American researcher has attempted to make those comparisons more meaningful to the public, by comparing the performance of students in U.S. states and cities against that of their foreign peers using a well-understood metric: letter grades.

A new study, “The Second Derivative: International Benchmarks in Mathematics for U.S. States and School Districts,” shows students in almost all U.S. states and cities performing at a C+ or C level. That is below the B grade that the study’s author contends should be the international target for American students, and which is achieved by the highest-performing nations.

In addition, the study reveals that the gap between a B and a C grade is a “quantum leap,” according to the author of the report, Gary W. Phillips, a vice president and a chief scientist at the American Institutes for Research, in Washington. The gap that separates the United States from top-performing jurisdictions like Hong Kong is about the size of the gulf between the highest-achieving U.S. state in the study, Massachusetts, and the lowest, Mississippi.

“We’re below the international benchmark, and we’re way below it,” Mr. Phillips said in an interview. The goal of the study, he said, is to compare U.S. and international performance in a clear fashion, which “resonates with readers and does it in a way that’s scientifically justifiable.”

Letter Grades Assigned

The study allows grade comparisons across cities, states, and nations. Florida and Washington state, for instance, earn C+ grades in 4th grade math, which the study shows as comparable to the performance of England, Germany, and Denmark, among others.

In two previous studies, Mr. Phillips compared the performance of students in U.S. states and cities, respectively, on an international scale. He did so by statistically linking the results of a prominent global exam, the Trends in International Mathematics and Science Study, or TIMSS, with the results of a U.S. test, the National Assessment of Educational Progress, or NAEP.

City Grades: A Global Perspective

A study of international achievement grades nations, U.S. states, and select cities on a common scale, using American jurisdictions’ NAEP scores to estimate their performance on an international test, TIMSS. The OECD figure represents the average for developed nations.

BRIC ARCHIVE

SOURCE: American Institutes for Research

In his latest effort, Mr. Phillips uses the cut scores on the TIMSS test to fashion a new statistical comparison with NAEP, and then equates scores with letter grades ranging from A to “BD,” or below D. Linking the two tests is possible, Mr. Phillips says, because TIMSS and NAEP are administered to equivalent, representative samples of students, in the same grades.

He selects a B grade as the U.S. target because his methodology shows that students who reached the “proficient” level on NAEP are statistically comparable to those reaching the “high” international level on the TIMSS.

American students’ performance, as judged by this metric, is mixed.

In 4th grade math, roughly half of U.S. states perform above the C+ mean for developed nations, according to the report. Just five states—Massachusetts, Minnesota, New Jersey, New Hampshire, and Kansas—reach the international target grade of B, and one state, Vermont, scores a B-minus in 4th grade.

In 8th grade, the U.S. states’ results were even less impressive, with only Massachusetts scoring a B-minus. Those scores fit a pattern: On the whole, American students’ math performance was weaker at 8th grade than at 4th grade, a disparity that has emerged in other international studies. (“U.S. Students’ Scores Drop By 8th Grade,” Dec. 13, 2000.)

Of 11 large U.S. cities studied, the majority scored a C grade at the 4th and 8th grade level, and none reached a B grade. Just two—Charlotte, N.C., and Austin, Texas—notched scores above the estimated TIMSS score, in both 4th and 8th grade. The District of Columbia schools had the lowest scores for both 4th and 8th grade.

No jurisdiction in the study—not even top-performing nations like Japan—earned an A grade. That result caused one expert on testing, Sharif Shakrani of Michigan State University, to question whether Mr. Phillips’ methods had set too high a bar, which might in turn make the U.S. results look worse than they actually are. Mr. Shakrani, who directs his university’s Education Policy Center, has helped develop math content for NAEP.

“Nobody got an ‘A,’ so is this [goal] a pie in the sky?” Mr. Shakrani said in an interview. “A very, very, good student should be able to reach that level,” he argued, as should a top-tier nation.

U.S. States as Models?

Mr. Shakrani attributed the drop-off in U.S. performance between 4th and 8th grade to American middle schools’ tendency to track students into middle grades math classes—algebra, pre-algebra, and very basic math—of strikingly different quality. Higher-performing nations have more uniform expectations for what math content is taught and tested, he said.

Comparing States on an International Scale

Most U.S. states meet or exceed the average of developed nations in 4th grade math.

BRIC ARCHIVE

SOURCE: American Institutes for Research
* U.S. Department of Defense schools

Hal Salzman, a professor of public policy at Rutgers University, said in an e-mail that the study reinforces an “unproved assumption” that international test scores are connected to a country’s ability to compete economically. Mr. Salzman has argued that U.S. policymakers misinterpret international exam results and exaggerate worries about American performance.

“Using these scores and international comparisons too often substitutes for substantive discussion of what is needed for real education improvement,” Mr. Salzman added.

More variation in test scores occurs within countries, among their own student populations, than from one nation to the next, Mr. Salzman said. American policymakers would be better off focusing on closing gaps between high- and low-perfoming states and districts, than on nation-by-nation comparisons, he argued.

In an e-mail, Mr. Phillips said that, on his grading scale, it is possible, albeit very difficult, for jurisdictions to obtain an A. He chose a B grade as the international target because it was “high, but achievable.”

Mr. Phillips responded to Mr. Salzman’s point by saying that it is important for U.S. officials to be aware of the high standards set by foreign countries.

But he agreed with the Rutgers professor that American policymakers should probe the successes of states like Massachusetts. Those states’ high grades, he wrote in the study, show “it is possible in the United States for students to learn mathematics at a level that is competitive with the best in the world.”

Related Tags:

A version of this article appeared in the June 17, 2009 edition of Education Week as Study Puts Results Of International Tests On Common Metric

Events

Ed-Tech Policy Webinar Artificial Intelligence in Practice: Building a Roadmap for AI Use in Schools
AI in education: game-changer or classroom chaos? Join our webinar & learn how to navigate this evolving tech responsibly.
Education Webinar Developing and Executing Impactful Research Campaigns to Fuel Your Ed Marketing Strategy 
Develop impactful research campaigns to fuel your marketing. Join the EdWeek Research Center for a webinar with actionable take-aways for companies who sell to K-12 districts.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Privacy & Security Webinar
Navigating Cybersecurity: Securing District Documents and Data
Learn how K-12 districts are addressing the challenges of maintaining a secure tech environment, managing documents and data, automating critical processes, and doing it all with limited resources.
Content provided by Softdocs

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

International What the Research Says It's Not Just U.S. Students. Civics Scores Have Dropped Around the World
Eighth graders are less engaged and knowledgeable about government than they were before the pandemic, a global study finds.
5 min read
vote ballot initiatives money 1371378601 01
LAUDISENO/iStock/Getty and EdWeek
International England Pushes for Cellphone Bans in Schools. Could the U.S. Be Next?
England is the latest country seeking to keep cellphones out of class.
3 min read
Tight crop photo of a student looking at their cellphone during class. The background is blurred, but shows students wearing uniforms.
E+
International Photos PHOTOS: Take a Round-the-World Tour of the Return to School
Here's what back to school looks like in classrooms around the globe.
1 min read
A teacher gives a lesson on the first day of school at a cadet lyceum in Kyiv, Ukraine on Sept. 4, 2023.
Young cadets sing the national anthem during a ceremony on the first day of school at a cadet lyceum in Kyiv, Ukraine, on Sept. 4, 2023.
Efrem Lukatsky/AP
International Opinion School Reform Is Tough All Over, Not Just in the U.S.
Even though some reforms produce evidence of student success, that often isn't enough to overcome political hurdles.
6 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty