Opinion
Equity & Diversity Commentary

International Tests Reveal Surprises at Home and Abroad

By Martin Carnoy & Richard Rothstein — January 18, 2013 6 min read

Policymakers and pundits raise alarms whenever international test results are announced. In December, upon release of new scores from the Trends in International Mathematics and Science Study, or TIMSS, U.S. Secretary of Education Arne Duncan called them “unacceptable,” saying they “underscore the urgency of accelerating achievement ... and the need to close large and persistent achievement gaps.”

It was no different a little over two years ago, when the Program for International Student Assessment, or PISA, released its latest scores. Secretary Duncan said they showed Americans “napping at the wheel. ... As disturbing as these national trends are for America, enormous achievement gaps among black and Hispanic students portend even more trouble for the United States in the years ahead.”

Such conclusions are oversimplified, frequently exaggerated, and misleading. They ignore the complexity of testing and may lead policymakers to pursue inappropriate and even harmful reforms.

BRIC ARCHIVE

Both TIMSS and PISA publish not only average national scores, but also a rich database from which analysts can disaggregate scores by students’ socioeconomic characteristics. Examining these can lead to more nuanced conclusions than those suggested from average national scores alone.

Yet, for some reason, although TIMSS published average national results this past December, it only released its underlying database last week. This puzzling procedure ensures that commentators draw quick but ill-informed interpretations. Analysis of the database takes time, and headlines from the initial release are codified before scholars can complete more careful study.

Since the last PISA release in 2010 (of a test given in 2010), we have been digging deeper into its database and examining older databases for TIMSS and for our domestic National Assessment of Educational Progress. We concentrated on adolescent scores—8th graders on TIMSS and NAEP, 15-year-olds on PISA—in the United States; three top-scoring countries (Canada, Finland, and South Korea); three similar postindustrial countries (France, Germany, and the United Kingdom); seven American states; and three Canadian provinces for which trends are available.

The TIMSS executive director deemed the report of our findings helpful (without endorsing our analysis in detail), but officials of PISA and of the U.S. Department of Education were harshly critical. We have posted their criticism, and our response to it, online.

Some of our conclusions are obvious; some are counterintuitive or startling. Here are a few:

A larger proportion of students in the United States is disadvantaged than in any comparison country. The number increased rapidly over the last decade, while in comparison countries it did not. Nonetheless, reading and mathematics achievement of lower-social-class U.S. students improved substantially, while achievement of similarly disadvantaged students declined in countries to which the United States is frequently unfavorably compared.

Thus, while the reading achievement on PISA of the lowest-social-class students in the United States grew by more than 0.2 standard deviations from 2000 to 2009, it fell by an even larger amount in Finland. In math, U.S. students from the lowest social class also gained substantially, while scores of comparable Finnish students declined. This is surprising because the proportion of disadvantaged students in Finland also fell, and we might expect this to make the task of devoting resources to them easier.

Certainly, even for the lowest social class of students, Finland’s scores remain higher than ours, but examination of trends (performance changes over time), as well as levels (performance at a single point in time), challenges the easy assumption that simply imitating Finnish education is a recipe for U.S. success.

Unsurprisingly, every country has an achievement gap between its most- and least-disadvantaged students. But unexpected is that this gap is smaller in the United States than in similar postindustrial countries, and often only slightly larger than gaps in top-scoring nations.

In some comparisons, the U.S. achievement gap is relatively small, while the scores of both disadvantaged and advantaged students are relatively low. But in other comparisons, the relatively small gap is attributable to our disadvantaged students’ scores being relatively high, while the scores of our advantaged students are relatively low, especially in math.

This brings into question our widespread policy consensus that American educational problems are concentrated among disadvantaged students.

Creating truly representative national samples is difficult for test-makers, and errors can have big consequences. For example, in 2009, PISA apparently oversampled low-income U.S. students who attended schools with very high proportions of similarly disadvantaged students, artificially lowering the average U.S. score. While 40 percent of the PISA sample in 2009 was drawn from schools in which half or more of the students were eligible for the subsidized-lunch program, only 32 percent of students nationwide attended such schools, according to National Center for Education Statistics data.

A top PISA official disputes our claim. While acknowledging that 40 percent of the PISA sample for the United States was, indeed, composed of students who attended schools where more than half qualified for subsidized lunches, he suggests that free-lunch participation must have jumped because of the recession.

This explanation is theoretically possible, but not plausible, considering that the NCES database shows that for the same year (2009) a considerably smaller proportion of all secondary school students (not a representative sample) attended schools with such high poverty concentrations.

So we stand by this conclusion: If we make two reasonable adjustments to the reported U.S. average, our international ranking substantially improves. The first adjustment reweights the social-class composition of U.S. test-takers to the average composition of similar postindustrial countries. The other reweights the distribution of lunch-eligible students by the actual concentration of such students in schools. Once we further eliminate ranking differences based on tiny and meaningless statistical differences, these adjustments raise the U.S. international ranking on the 2009 PISA from the publicly reported 14th to sixth in reading, and from the publicly reported 25th to 13th in mathematics. While there is still room for improvement, these are more respectable showings.

We should be cautious about drawing conclusions about international comparisons from any single test. From 2000 to 2006, U.S. math scores on PISA fell substantially, causing great alarm among U.S. policymakers and pundits. But few noticed that during roughly the same period, U.S. math scores on TIMSS were rising, as they did on NAEP. We cannot attribute this to alleged superior alignment of TIMSS to U.S. curricula because in the next period, the pattern reversed: From 2006 to 2009, PISA and NAEP scores both rose, while TIMSS scores were flatter.

Just as population sampling is complex, so is the choice of topics to test. New TIMSS results show that the United States’ and Finland’s scores are now about the same overall in mathematics. But underlying data show that the United States does better than Finland in algebra but worse in number properties (e.g., using fractions). Algebra and numbers now each constitute 30 percent of test items (proportions have changed over time, adding another challenge to interpretation). But in the United States, we have made a big policy push to expose all students to algebra in 8th grade. If algebra had a bigger timss weight, our average score would be higher.

When it comes to domestic tests, U.S. analysts are relatively sophisticated. Federal law requires that average scores be disaggregated by race, ethnicity, and family income. But in our rush to condemn U.S. international performance, we ignore this common-sense approach. It is time for analysts to apply the same sophistication to international scores that is now second nature to us domestically.

A version of this article appeared in the January 23, 2013 edition of Education Week as International Tests Reveal Surprises at Home and Abroad

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
Interactive Learning Best Practices: Creative Ways Interactive Displays Engage Students
Students and teachers alike struggle in our newly hybrid world where learning takes place partly on-site and partly online. Focus, engagement, and motivation have become big concerns in this transition. In this webinar, we will
Content provided by Samsung
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Classroom Technology Webinar
Educator-Driven EdTech Design: Help Shape the Future of Classroom Technology
Join us for a collaborative workshop where you will get a live demo of GoGuardian Teacher, including seamless new integrations with Google Classroom, and participate in an interactive design exercise building a feature based on
Content provided by GoGuardian
School & District Management Live Online Discussion A Seat at the Table With Education Week: What Did We Learn About Schooling Models This Year?
After a year of living with the pandemic, what schooling models might we turn to as we look ahead to improve the student learning experience? Could year-round schooling be one of them? What about online

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Equity & Diversity Opinion Trans Youth Are Under Attack. Educators Must Step Up
What can schools do in the face of the extreme hostility trans and gender-nonconforming young people now face across the country?
Harper B. Keenan & Z Nicolazzo
4 min read
A butterfly lands on balanced stones in front of tranquil waters and a sunset
Pict Rider/iStock/Getty images<br/>
Equity & Diversity Opinion When Does Educational Equity Become Educationally Unethical?
Equity stumbles into a truly gruesome place when educators are directed to shortchange students based on how they look or where they live.
3 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
Equity & Diversity Opinion Why More Teachers Need to See the Beauty and Brilliance in Black Girls
Black girls are often accused of being loud or having an attitude. We need teachers to change that harmful perspective, because it matters.
Bola Delano-Oriaran
5 min read
Black Girls Misunderstood
Shutterstock
Equity & Diversity Anti-Asian Violence: What Schools Should Start Doing About It
Asian-Americans are often erased from the curriculum and even from schools' social justice work. Five educators discuss how to change that.
The crowd at Hing Hay Park responds to speakers calls to "fight hate"and against the attacks, physical and verbal on Asian Americans during a rally to speak out against anti-Asian hate and violence on March 13, 2021 in Seattle.
A crowd at Hing Hay Park in Seattle protests physical and verbal attacks against Asian Americans during a rally earlier this month.
Alan Berner/The Seattle Times via AP