School & District Management

Three Better Ways to Use Global Tests

By Sarah D. Sparks — April 06, 2018 6 min read
  • Save to favorites
  • Print

International benchmarking tests always make headlines, but reading between the lines would help education researchers and policymakers better use the tests to improve schools.

That’s the upshot of a new report on international education assessments by the National Academy of Education and an accompanying article in the journal Science. A working group of testing experts in the Academy analyzed methods, use, and media coverage of large-scale tests including the Program for International Student Assessment (PISA), the Trends in International Mathematics and Science Study (TIMSS), and the Progress in International Reading Literacy Study (PIRLS) during the past 50 years.

“In general, equating of tests across countries is a good idea,” said Judith Singer, Harvard University education professor and chair of the report committee. “The challenge is, countries aren’t sports teams ... If you are reducing a country to a ranking, it is concealing so much.”

The group found that over time, international tests have come to be used not just to describe and different education systems and track their changes over time, but also to compare countries as though they had de facto international standards, evaluate the effectiveness of policies or even curricula, and examine the relationships between schools’ contexts and student achievement.

“We should use international assessments for the purposes they serve best, rather than squeezing into them things they will never do very well,” said Andreas Schleicher, the director of the OECD’s directorate for education and skills, who was not part of the report group. “International assessments like PISA are extremely valuable tools for generating hypotheses. Who would have thought we’d be able to learn something from Finland or Shanghai? They open our eyes about what we think. ... but it will never get to the causal answers. I do not see international assessments as substitutes for causal instruments like [randomized controlled trials], but as complements.”

The report suggested several ways to make international test data more useful for practitioners and policymakers. Here are a few:

1. Stop pulling rank.

In reporting the results of global tests, the report noted countries are listed in rank order, but many countries’ average scores are statistically the same as others listed above and below them on the list.

This can lead policymakers to adopt individual policies from the leading countries without determining whether they caused that country’s achievement, Singer said. “A few years ago it was homework: Finland was on top, Finland doesn’t assign homework, so maybe we shouldn’t assign homework. Then a few years later, Finland isn’t on top anymore, so maybe eliminating homework isn’t the answer,” she said. “It’s a fool’s errand to try to identify the one secret sauce and try to roll it out. We do have a lot to learn, but we are not thinking broadly enough.”

Focus on rank can also give the wrong impression of a country’s performance. For example, Japan’s rank on the PISA in science rose from fourth in the world in 2012 to second in 2015, and local newspapers trumpeted its improvement. But the mean test scores on science literacy during that time actually declined, from 547 to 538.

The report called for tests to include more context on comparison groups, such as matching countries to others with similar resources and contexts and comparing education systems like Singapore’s to those of cities or states rather than countries. For example, a recent report based on PISA data compared how quickly Canada and the United States improved achievement for new immigrant students.

William Schmidt, director of the Education Policy Center at Michigan State University, was not part of the study committee, but he agreed that tests should provide more information about countries’ policy contexts. However, he cautioned against focusing too much on individual state or city scores.

“If you believe these studies are really about comparing where the United States is, you really need the United States” as the point of comparison, Schmidt said. “I’m not very comforted by Massachusetts doing well. If it was a horse race, maybe you’d want to put up our best against their best, but ... the bigger issues are our national policies around our system. We are a country.”

2. Dig into contexts.

All of the international assessments generate mountains of data that most policymakers and researchers never see, from background surveys given to students along with the tests. PISA alone collects more than 1,000 elements describing school policies, instructional practices, and student behaviors, all of which can affect student learning.

For example, Schleicher noted that the OECD has published the codes for all of its data elements and put both test results and background data online, but said it can still be complicated for researchers to use these data.

Schmidt warned, however, that the report left out a critical problem with international assessments: curriculum. The test administrators—the Organization for Economic Cooperation and Development for PISA, and the International Association for the Evaluation of Educational Achievement for TIMSS and PIRLS—determine the subject matter to be covered on each test, but it is difficult to tell how much of that content individual countries teach in the years before their students are tested.

“That’s the heart and core of it,” Schmidt said. “Comparisons really don’t make sense if you don’t know what the students cover in terms of say, mathematics. ... That’s a really critical issue to avoid misinterpreting the horse race ... It provides that context in which you can look at the top-achieving countries and ask what’s different.”

The report recommended that test groups augment their data on student and school backgrounds with outside data—for example, using data from the U.S. Census Bureau to supplement background data for the United States—to allow more consistent measures of indicators like a student’s socioeconomic level.

3. Use emerging technology.

Schleicher said PISA is already starting to respond to some of the criticisms of international tests leveled in the report. As of the 2018 administration, the PISA has moved to computer-adaptive testing, which he suggested would give a better indication of what each student knows and can do, regardless of what country they live in.

Schmidt agreed. “That’s a new form of testing that would probably make the whole testing process more sensitive,” he said, “because you don’t waste time asking the kids questions they will never have a chance of answering, and spending more time asking questions around the knowledge they do know.”

The report suggested that as more international tests move to computer-based formats, their supervising organizations will also have the opportunity to rethink how they design test questions and structure the tests to provide more useful information to educators.

The National Academy of Education will discuss the findings of the report in more detail at a symposium in Washington later today.

Photo Source: Getty


Want more research news? Get the latest studies and join the conversation.

Related Tags:

A version of this news article first appeared in the Inside School Research blog.

Commenting has been disabled on effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
6 Key Trends in Teaching and Learning
As we enter the third school year affected by the pandemic—and a return to the classroom for many—we come better prepared, but questions remain. How will the last year impact teaching and learning this school
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Equity & Diversity Webinar
Evaluating Equity to Drive District-Wide Action this School Year
Educational leaders are charged with ensuring all students receive equitable access to a high-quality education. Yet equity is more than an action. It is a lens through which we continuously review instructional practices and student
Content provided by BetterLesson
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Attendance Awareness Month: The Research Behind Effective Interventions
More than a year has passed since American schools were abruptly closed to halt the spread of COVID-19. Many children have been out of regular school for most, or even all, of that time. Some
Content provided by AllHere

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Letter to the Editor School Mask Mandates: Pandemic, ‘Panicdemic,’ or Personal?
"A pandemic is based on facts. A 'panicdemic' is based on fears. Today, we have both," writes a professor.
1 min read
School & District Management How 'Vaccine Discrimination' Laws Make It Harder for Schools to Limit COVID Spread
In Montana and Ohio, the unvaccinated are a protected class, making it tough to track and contain outbreaks, school leaders say.
4 min read
Principal and District Superintendent Bonnie Lower takes the temperature of a student at Willow Creek School as the school reopened, Thursday, May 7, 2020, in Willow Creek, Mont.
Bonnie Lower, a principal and district superintendent in Willow Creek, Mont., checks the temperature of a student as Willow Creek School reopened for in-person instruction in the spring.
Ryan Berry/Bozeman Daily Chronicle via AP
School & District Management Opinion 'Futures Thinking' Can Help Schools Plan for the Next Pandemic
Rethinking the use of time and place for teachers and students, taking risks, and having a sound family-engagement plan also would help.
17 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
School & District Management Opinion The Consequence of Public-Health Officials Racing to Shutter Schools
Public-health officials' lack of concern for the risks of closing schools may shed light on Americans' reticence to embrace their directives.
5 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty