Federal

PISA Called Inappropriate for U.S. Benchmarking

By Sean Cavanagh — February 26, 2009 3 min read
  • Save to favorites
  • Print

Includes updates and/or revisions.

As state leaders and education advocates weigh evaluating U.S. students using international benchmarks, a new report argues that one prominent test, PISA, is flawed and may not be appropriate for judging American schools on global standards.

The author, Tom Loveless, a senior fellow at the Washington-based Brookings Institution, also contends that questions asked on the Program for International Student Assessment surveys of students’ beliefs and attitudes about science reflect an ideological bias, which undermines the test’s credibility.

He cites an example from one PISA questionnaire, which seeks to gauge “a sense of students’ responsibility for sustainable development,” and asks test-takers if they agree with certain statements, such as “having laws that protect the habitats of endangered species.”

A response requires a “political judgment,” Mr. Loveless writes. Also, the questions are vague, making it difficult for the scientifically literate to know how to answer, he argues.

“It is difficult to see how declaring support or opposition to a policy without knowing the details” is related to responsible citizenship, Mr. Loveless adds.

Andreas Schleicher, the head of education indicators for the Organization for Economic Cooperation and Development, the Paris-based group that oversees the test, called the report “disingenuous” and misleading on some points.

‘A First Reading’

He noted that the student questionnaire is not in any way connected to the main, publicly reported PISA scores for science and math, which are most commonly cited in the news media and by policymakers. It is clear, he said, that the test scores and the questionnaire give policymakers two different sets of information. Results from the questionnaire are put in separate indices in PISA reports, he noted.

“These questions explore significant science-related contemporary issues,” Mr. Schleicher said in an e-mail, and give policymakers “a first reading” of students’ attitudes about science, even if the phrases are not perfect.

Mr. Loveless also casts doubt on whether PISA’s practice of measuring skills that students pick up both in and out of school makes it useful for state policymakers who want to improve their K-12 systems. Another international test, the Trends in International Mathematics and Science Study, or TIMSS, and the U.S.-based National Assessment of Educational Progress, or NAEP, focus primarily on in-school skills.

In addition, he said the OECD takes policy positions that it should not be doing if it collects and interprets score data, because it creates potential for conflict.

Mr. Schleicher said PISA emphasizes students’ ability to apply knowledge in an out-of-school context, but that doesn’t mean students necessarily learned those skills outside the classroom.

One central PISA goal is to assess students’ “capacities to extrapolate from what they know and transfer and apply their knowledge and skills to novel settings,” Mr. Schleicher said, which, he added, is a prized skill in science.

Last September, the National Governors Association, Achieve, and the Council of Chief State School Officers announced plans to create an advisory group to produce a “road map” to benchmark U.S. school performance with that of top-performing nations.

Mr. Loveless writes that the NGA would “like states to use PISA” in that process. But Dane Linn, the director of the NGA’s education division, disputed that, saying the organizations are not committed to any particular approach, but considering a range of rigorous international exams.

“It behooves us to not exclude PISA in examining how other countries measure performance,” Mr. Linn said. Different elements of PISA, TIMSS, and other international tests are likely to appeal to state policymakers. Debates about which kind of test material, emphasizing in-school “content,” as opposed to the “application of knowledge,” miss the point, Mr. Linn added. “It’s both.”

A version of this article appeared in the March 04, 2009 edition of Education Week as PISA Called Inappropriate for U.S. Benchmarking

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Professional Development Webinar
Inside PLCs: Proven Strategies from K-12 Leaders
Join an expert panel to explore strategies for building collaborative PLCs, overcoming common challenges, and using data effectively.
Content provided by Otus
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Science Webinar
Making Science Stick: The Engaging Power of Hands-On Learning
How can you make science class the highlight of your students’ day while
achieving learning outcomes? Find out in this session.
Content provided by LEGO Education
Teaching Profession Key Insights to Elevate and Inspire Today’s Teachers
Join this free half day virtual event to energize your teaching and cultivate a positive learning experience for students.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Vaccine Skeptic Robert F. Kennedy Jr. Is Sworn in as Trump’s Health Chief
Kennedy, who has called for public health agencies to focus on chronic diseases, was sworn in after a close Senate vote.
3 min read
Supreme Court Associate Justice Neil Gorsuch swears in Robert F. Kennedy Jr. as Health and Human Services Secretary as Kennedy's wife Cheryl Hines holds the Bible in the Oval Office at the White House, Thursday, Feb. 13, 2025, in Washington.
Supreme Court Associate Justice Neil Gorsuch swears in Robert F. Kennedy Jr. as Health and Human Services Secretary as Kennedy's wife Cheryl Hines holds the Bible in the Oval Office at the White House, Feb. 13, 2025, in Washington.
Alex Brandon/AP
Federal 5 Key Takeaways From Linda McMahon's Confirmation Hearing
President Trump's education secretary nominee pledged to maintain school funding even as she works to end the U.S. Department of Education.
6 min read
Linda McMahon, President Donald Trump’s nominee to be Secretary of Education, arrives for her Senate Health, Education, Labor, and Pensions Committee confirmation hearing, at the U.S. Capitol, in Washington, on Feb. 13, 2025.
Linda McMahon, President Donald Trump’s nominee to be Secretary of Education, arrives for her confirmation hearing before the Senate Health, Education, Labor, and Pensions Committee at the U.S. Capitol in Washington on Feb. 13, 2025.
Graeme Sloan for Education Week
Federal Linda McMahon Backs Trump’s Push to Dismantle Ed. Dept.: 'It's Not Working'
McMahon faced lawmakers' questions amid changes to the Education Department that are already underway.
5 min read
Linda McMahon, President Donald Trump’s nominee to be Secretary of Education, testifies during her Senate Health, Education, Labor, and Pensions Committee confirmation hearing, at the U.S. Capitol, in Washington, on Feb. 13, 2025.
Linda McMahon, President Donald Trump’s nominee to be Secretary of Education, testifies during her confirmation hearing before the Senate Health, Education, Labor, and Pensions Committee at the U.S. Capitol in Washington on Feb. 13, 2025.
Graeme Sloan for Education Week
Federal McMahon Declines to Say If Black History Classes Are Allowed Under Trump Order
The comment came during an exchange with Sen. Chris Murphy, D-Conn., about Trump's executive order on "radical indoctrination."
3 min read
Senator Chris Murphy (D-CT) and Senator Susan Collins (R-ME) greet Linda McMahon, President Donald Trump’s nominee to be Secretary of Education, before her Senate Health, Education, Labor, and Pensions Committee confirmation hearing, at the U.S. Capitol, in Washington, on Feb. 13, 2025.
Sen. Chris Murphy, D-Conn., and Sen. Susan Collins, R-Maine, greet Linda McMahon, President Donald Trump’s nominee to be secretary of education, before her confirmation hearing before the Senate Health, Education, Labor, and Pensions Committee at the U.S. Capitol in Washington on Feb. 13, 2025.
Graeme Sloan for Education Week