International Opinion

What PISA Can’t Teach Us

By Martin Carnoy, Emma Garcia & Tatiana Khavenson — February 09, 2016 6 min read
  • Save to favorites
  • Print

The results of the Program for International Student Assessment, or PISA, the test that is used to do an international ranking of students’ academic performance, will be trumpeted later this year, and the responses will undoubtedly repeat those of previous years: U.S. officials will wring their hands and lament that American student achievement is stagnant, that it is lagging woefully behind our economic competitors’, and that we therefore need to import features of schooling from higher-scoring countries.

This edu-masochism—a distinctly American way of focusing on our educational shortcomings—can be traced at least back to the late 1950s and the Soviet Union’s launch of Sputnik. Perhaps the nation’s early trailblazing successes in establishing mass schooling and developing a uniquely excellent system of higher education have left U.S. educators particularly vulnerable to charges that other nations are surpassing us.

These fears unfortunately have led education policymakers astray. They focus on what other nations are achieving, and they fall prey to those urging us to copy Finland’s, Singapore’s, or South Korea’s schooling practices.

But why compare national student performance in the United States with average scores in other countries, when U.S. students attend schools in 51 separate education systems responsible to states and the District of Columbia, not the federal government? The U.S. education system is a construct that does not exist operationally. What’s more, a well-respected test, the National Assessment of Educational Progress (NAEP) provides a state-by-state picture of our schools that is much more relevant than either PISA or other major international tests, such as the Trends in International Mathematics and Science Study (TIMSS).


President Barack Obama’s signing of the Every Student Succeeds Act in December underscores this reality: Although the federal government retains the ability to measure and compare how different states are doing with NAEP, the new law makes clear that the authority to decide school policy resides with the states.

Our recent study of two decades of PISA, TIMSS, and NAEP scores offers new data that highlight why education policymakers should emphasize national test results.

To begin with, schools in the United States are not doing as poorly as international-test scores suggest. It makes sense to look to high performers in Europe and Asia for new education strategies if we lack our own success stories, but that is not the case. After we adjusted for socioeconomic differences in the samples of students taking the PISA and TIMSS tests, we found that performance on international math and reading tests in states such as Massachusetts and North Carolina is as high as, or higher than, in the highest-scoring countries in Europe. We also showed that gains in TIMSS mathematics scores over the past 12 to 16 years in several states are much higher than gains in other countries. Furthermore, students in these same states and others have made very large, steady gains on NAEP, especially in mathematics, over the last two decades, despite the dip in scores in 2015. That’s in marked contrast with the average U.S. results on PISA, which did not rise between 2000 and 2012.

Policymakers also must consider whether international tests are influenced by factors that are as much social and cultural as they are educational. There is no causal evidence that students in some Asian countries, for example, score higher on international tests mainly because of better schooling. Their achievement is more likely the result of large investments made by families on outside-of-school tutoring and cram courses to hone test-taking skills. Such differences between other countries and U.S. states in these social and educational contexts make it difficult to infer relevant educational policy from correlations with student test scores.

Worrying about how schools in other nations are better than ours doesn’t get us much closer to answers."

By contrast, the conditions and context of education in the high-scoring states are more similar to those in other, lower-performing states than they are to other countries’. Teacher labor markets are not drastically different from state to state. Furthermore, the state systems are regulated under the same federal rules. If students with similar family academic resources attending schools with similar socioeconomic and ethnic composition in some states make much larger gains than in other states, those larger gains are more likely to be related to specific state policies that could be applied elsewhere in the United States.

In our study, we showed that the basis for such fruitful comparisons exists. We found that since 1992, the average annual increase in demographically adjusted NAEP 8th grade math scores of students in the top-gaining 10 states was 1.6 points per year, double that of students in the bottom-gaining 10 states. Thus, for a 21-year period between 1992 and 2013, 8th graders in the higher-gaining states increased their math performance 16 points more than students in the lower-gaining states. This represents about one-half a standard deviation of the individual student variation in NAEP 8th grade math scores—a huge difference in performance gain by typical educational improvement standards.

States that made large reading gains were not necessarily the same states that made large math gains. For example, Maryland and Florida made relatively larger gains in reading than in mathematics. And Texas and Massachusetts made large gains in math, but not reading.

Another direction for further policy research is to pair off states with different patterns of gains in 8th grade math. We showed in our study, for instance, that 8th grade students in Massachusetts made much larger gains in math after 2003 than their counterparts in neighboring Connecticut; that students in Texas greatly outpaced demographically similar California from the early 1990s until 2013; and that students in New Jersey made larger gains than students in New York after 2003. These and other state comparisons could provide important insights into the kinds of policies that enabled students in some states to make much larger demographically adjusted gains in math scores than students next door.

A case in point is Massachusetts and Connecticut, where students had the same NAEP math score in 2003. By 2013, Massachusetts students had increased by 17 points over similar students attending similar schools in Connecticut. We need to learn why students in Massachusetts took off in math after 2003 while students in Connecticut did not.

To be sure, international comparisons can be instructive. It is useful to know that teachers in high-scoring Finland are prepared much more thoroughly than teachers in most U.S. states, and that high teacher salaries in Singapore and Taiwan have eliminated shortages in math instruction.

But too much time in the United States is spent fear-mongering and declaring that our economy is about to tank because of how U.S. schools purportedly stack up against schools in other nations. It’s almost as if we want to punish ourselves.

Despite these cries, our economy has grown, and the nation remains the leader in innovation. While improvement is needed, worrying about how schools in other nations are better than ours doesn’t get us much closer to answers.

We can put an end to our edu-masochism: If researchers spend more effort on assessing our own states’ successes and failures in improving student performance and less on trying to draw lessons from countries with very different social and educational contexts, they are sure to spark a much more productive national educational policy debate than we have had in the past decade.

Follow the Education Week Commentary section on Facebook and Twitter.
A version of this article appeared in the February 10, 2016 edition of Education Week as America’s ‘Edu-Masochism’


Classroom Technology Webinar How Pandemic Tech Is (and Is Not) Transforming K-12 Schools
The COVID-19 pandemic—and the resulting rise in virtual learning and big investments in digital learning tools— helped educators propel their technology skills to the next level. Teachers have become more adept at using learning management
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Building Teacher Capacity for Social-Emotional Learning
Set goals that support adult well-being and social-emotional learning: register today!

Content provided by Panorama
Jobs October 2021 Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

International Opinion Why Other Countries Keep Outperforming Us in Education (and How to Catch Up)
Money from the American Rescue Plan could be our last chance to build the school system we need, writes Marc Tucker.
Marc Tucker
5 min read
A student climbs stacks of books to reach the top
Tatyana Pivovarova/iStock/Getty Images Plus
International Global Test Finds Digital Divide Reflected in Math, Science Scores
New data from the 2019 Trends in International Math and Science Study show teachers and students lack digital access and support.
3 min read
Image of data.
International Pre-COVID Learning Inequities Were Already Large Around the World
A new international benchmarking highlights gaps in training for digital learning and other supports that could deepen the challenge for low-income schools during the pandemic.
4 min read
International Part of Global Trend, 1 in 3 U.S. High Schoolers Felt Disconnected From School Before Pandemic
UNESCO's annual report on global education progress finds countries need to make more effort to include marginalized students, particularly in the United States.
4 min read