International Opinion

What PISA Can’t Teach Us

By Martin Carnoy, Emma Garcia & Tatiana Khavenson — February 09, 2016 6 min read

The results of the Program for International Student Assessment, or PISA, the test that is used to do an international ranking of students’ academic performance, will be trumpeted later this year, and the responses will undoubtedly repeat those of previous years: U.S. officials will wring their hands and lament that American student achievement is stagnant, that it is lagging woefully behind our economic competitors’, and that we therefore need to import features of schooling from higher-scoring countries.

This edu-masochism—a distinctly American way of focusing on our educational shortcomings—can be traced at least back to the late 1950s and the Soviet Union’s launch of Sputnik. Perhaps the nation’s early trailblazing successes in establishing mass schooling and developing a uniquely excellent system of higher education have left U.S. educators particularly vulnerable to charges that other nations are surpassing us.

These fears unfortunately have led education policymakers astray. They focus on what other nations are achieving, and they fall prey to those urging us to copy Finland’s, Singapore’s, or South Korea’s schooling practices.

But why compare national student performance in the United States with average scores in other countries, when U.S. students attend schools in 51 separate education systems responsible to states and the District of Columbia, not the federal government? The U.S. education system is a construct that does not exist operationally. What’s more, a well-respected test, the National Assessment of Educational Progress (NAEP) provides a state-by-state picture of our schools that is much more relevant than either PISA or other major international tests, such as the Trends in International Mathematics and Science Study (TIMSS).


President Barack Obama’s signing of the Every Student Succeeds Act in December underscores this reality: Although the federal government retains the ability to measure and compare how different states are doing with NAEP, the new law makes clear that the authority to decide school policy resides with the states.

Our recent study of two decades of PISA, TIMSS, and NAEP scores offers new data that highlight why education policymakers should emphasize national test results.

To begin with, schools in the United States are not doing as poorly as international-test scores suggest. It makes sense to look to high performers in Europe and Asia for new education strategies if we lack our own success stories, but that is not the case. After we adjusted for socioeconomic differences in the samples of students taking the PISA and TIMSS tests, we found that performance on international math and reading tests in states such as Massachusetts and North Carolina is as high as, or higher than, in the highest-scoring countries in Europe. We also showed that gains in TIMSS mathematics scores over the past 12 to 16 years in several states are much higher than gains in other countries. Furthermore, students in these same states and others have made very large, steady gains on NAEP, especially in mathematics, over the last two decades, despite the dip in scores in 2015. That’s in marked contrast with the average U.S. results on PISA, which did not rise between 2000 and 2012.

Policymakers also must consider whether international tests are influenced by factors that are as much social and cultural as they are educational. There is no causal evidence that students in some Asian countries, for example, score higher on international tests mainly because of better schooling. Their achievement is more likely the result of large investments made by families on outside-of-school tutoring and cram courses to hone test-taking skills. Such differences between other countries and U.S. states in these social and educational contexts make it difficult to infer relevant educational policy from correlations with student test scores.

Worrying about how schools in other nations are better than ours doesn’t get us much closer to answers."

By contrast, the conditions and context of education in the high-scoring states are more similar to those in other, lower-performing states than they are to other countries’. Teacher labor markets are not drastically different from state to state. Furthermore, the state systems are regulated under the same federal rules. If students with similar family academic resources attending schools with similar socioeconomic and ethnic composition in some states make much larger gains than in other states, those larger gains are more likely to be related to specific state policies that could be applied elsewhere in the United States.

In our study, we showed that the basis for such fruitful comparisons exists. We found that since 1992, the average annual increase in demographically adjusted NAEP 8th grade math scores of students in the top-gaining 10 states was 1.6 points per year, double that of students in the bottom-gaining 10 states. Thus, for a 21-year period between 1992 and 2013, 8th graders in the higher-gaining states increased their math performance 16 points more than students in the lower-gaining states. This represents about one-half a standard deviation of the individual student variation in NAEP 8th grade math scores—a huge difference in performance gain by typical educational improvement standards.

States that made large reading gains were not necessarily the same states that made large math gains. For example, Maryland and Florida made relatively larger gains in reading than in mathematics. And Texas and Massachusetts made large gains in math, but not reading.

Another direction for further policy research is to pair off states with different patterns of gains in 8th grade math. We showed in our study, for instance, that 8th grade students in Massachusetts made much larger gains in math after 2003 than their counterparts in neighboring Connecticut; that students in Texas greatly outpaced demographically similar California from the early 1990s until 2013; and that students in New Jersey made larger gains than students in New York after 2003. These and other state comparisons could provide important insights into the kinds of policies that enabled students in some states to make much larger demographically adjusted gains in math scores than students next door.

A case in point is Massachusetts and Connecticut, where students had the same NAEP math score in 2003. By 2013, Massachusetts students had increased by 17 points over similar students attending similar schools in Connecticut. We need to learn why students in Massachusetts took off in math after 2003 while students in Connecticut did not.

To be sure, international comparisons can be instructive. It is useful to know that teachers in high-scoring Finland are prepared much more thoroughly than teachers in most U.S. states, and that high teacher salaries in Singapore and Taiwan have eliminated shortages in math instruction.

But too much time in the United States is spent fear-mongering and declaring that our economy is about to tank because of how U.S. schools purportedly stack up against schools in other nations. It’s almost as if we want to punish ourselves.

Despite these cries, our economy has grown, and the nation remains the leader in innovation. While improvement is needed, worrying about how schools in other nations are better than ours doesn’t get us much closer to answers.

We can put an end to our edu-masochism: If researchers spend more effort on assessing our own states’ successes and failures in improving student performance and less on trying to draw lessons from countries with very different social and educational contexts, they are sure to spark a much more productive national educational policy debate than we have had in the past decade.

Follow the Education Week Commentary section on Facebook and Twitter.
A version of this article appeared in the February 10, 2016 edition of Education Week as America’s ‘Edu-Masochism’


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Equity, Care and Connection: New SEL Tools and Practices to Support Students and Adults
As school districts plan to welcome students back into buildings for the upcoming school year, this is the perfect time to take a hard look at both our practices and our systems to build a
Content provided by Panorama Education
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Classroom Technology Webinar
Here to Stay – Pandemic Lessons for EdTech in Future Development
What technology is needed in a post pandemic district? Learn how changes in education will impact development of new technologies.
Content provided by AWS
School & District Management Live Online Discussion A Seat at the Table: Strategies & Tips for Complex Decision-Making
Schools are working through the most disruptive period in the history of modern education, facing a pandemic, economic problems, social justice issues, and rapid technological change all at once. But even after the pandemic ends,

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

International Global Test Finds Digital Divide Reflected in Math, Science Scores
New data from the 2019 Trends in International Math and Science Study show teachers and students lack digital access and support.
3 min read
Image of data.
International Pre-COVID Learning Inequities Were Already Large Around the World
A new international benchmarking highlights gaps in training for digital learning and other supports that could deepen the challenge for low-income schools during the pandemic.
4 min read
International Part of Global Trend, 1 in 3 U.S. High Schoolers Felt Disconnected From School Before Pandemic
UNESCO's annual report on global education progress finds countries need to make more effort to include marginalized students, particularly in the United States.
4 min read
International How Schools in Other Countries Have Reopened
Ideas from Australia, Denmark, and Taiwan can help American district and school leaders as they shape their reopening plans.
11 min read
Students at the Taipei American School in Taipei, Taiwan, perform The Little Mermaid in full costume and masks.
Students at the Taipei American School in Taipei, Taiwan, perform The Little Mermaid in full costume and masks.
Photo courtesy of Dustin Rhoades/Taipei American School