Most Nations Seen Falling Short of NAEP ‘Proficiency’

By Debra Viadero — May 01, 2007 6 min read

Outside of a handful of Asian nations, the typical 8th grader in many foreign countries would not meet “proficient” levels on U.S. tests of mathematics and science, a new reanalysis of international achievement shows.

Then again, the study also shows, neither do most American students.

The study, which was posted April 24 on the Web site of the American Institutes for Research, a Washington-based research organization, offers fodder for national debates over whether the United States has set its achievement targets too high. The new analysis, the second in a year to draw nearly the same conclusions from reinterpreted international data, comes from the AIR’s chief scientist, Gary W. Phillips.

“Expressing International Educational Achievement in Terms of U.S. Performance Standards: Linking NAEP Achievement Levels to TIMSS” is posted by the American Institutes for Research.

Mr. Phillips’ idea was to statistically “link” scores from two well-known testing programs: the Trends in International Mathematics and Science Study, or TIMSS, which is given every few years to students in more than three dozen countries, and the National Assessment of Educational Progress, a program mandated by the U.S. Congress that is known as “the nation’s report card.”

“I wanted to re-express the TIMSS results in terms of a set of standards that U.S. policymakers would be familiar with,” said Mr. Phillips, who was acting commissioner of the National Center for Education Statistics, the data-gathering arm of the U.S. Education Department, from 1999 to 2002. “It’s a lot like looking at world-poverty levels. It’s impossible to get your head around it until you get a common metric.”

Defining Proficient

Global Comparisons

A study designed to predict how other countries would fare on U.S. mathematics and science tests found that the typical 8th grader in most places, including the United States, would score at the “basic” level or below on the National Assessment of Educational Progress math test.


Note: “Basic” denotes a minimum score of 469, “proficient” a minimum of 556, and “advanced,” 637.
SOURCE: American Institutes for Research

Although Mr. Phillips saw his study as primarily a methodological test, the results come at a time when the NAEP achievement levels, which have been debated on and off for years, are attracting renewed attention in national discussions over what it means for U.S. students to be academically “proficient” or “world class.”

The federal No Child Left Behind Act calls for all students to be proficient by the 2013-14 school year. But studies show huge gaps between NAEP’s definition of proficient and the definitions states use in their own assessment programs. (“Gaps in Proficiency Levels on State Tests And NAEP Found to Grow,” April 18, 2007.)

“The notion of all students being proficient is an oxymoron,” said Richard Rothstein, a research associate for the Economic Policy Institute, a Washington-based think tank. “You can’t have a standard that challenges students at all ends of the distribution system.”

Using slightly different methodology, Mr. Rothstein put out a paper last fall that found results similar to Mr. Phillips’: Countries that score highest on the TIMSS tests don’t ensure that all students meet the NAEP definition of proficient.

“My conclusion is that we need to take a good look at where all the cut points are for defining basic skills or proficiency,” said Tom Loveless, the director of the Brown Center on Education Policy at the Brookings Institution, a Washington-based research organization, referring specifically to the NAEP tests.

However, Chester E. Finn Jr., the president of the Thomas B. Fordham Foundation, another Washington group that studies school improvement issues, said the results confirmed the current levels for the NAEP tests. “If you have normal students from normal public schools in normal countries and more than half of them are reaching proficient levels, then we have set exactly the right target that everybody should aim for,” said Mr. Finn, who once sat on the governing board that sets policy for the federal assessment program.

Mr. Phillips, who undertook the study on his own initiative, said he agreed. “A lot of people have said the NAEP achievement levels are too high,” he noted. “I think it’s pretty clear that’s not the case.”

‘A World Apart’

NAEP uses three levels to gauge student achievement: “basic,” which generally denotes partial mastery of the academic material; “proficient,” for solid mastery of challenging academic content; and “advanced,” to describe superior achievement. Students who reach none of those levels are described as performing “below basic.”

By Mr. Phillips’ estimate, of the 8th graders from 46 nations who took TIMSS mathematics tests in 2003, only students from Hong Kong, Japan, Singapore, South Korea, and Taiwan would reach the proficient level, on average, if they were to take the 8th grade NAEP in that subject.

For example, in Singapore, the nation that topped the international tests, the analysis shows that 73 percent of 8th graders would score at or above the proficient level on NAEP, compared with 26 percent in the United States. The percentage of 8th graders reaching the NAEP advanced level in that Asian country was estimated to be 35 percent—seven times greater than the U.S. rate.

On the other hand, in four nations—Botswana, Ghana, Saudi Arabia, and South Africa—almost no students scored at the advanced level.

The science results suggest a similar pattern. On that test, the analysis shows, 8th graders in only two of the TIMSS countries—Singapore and Taiwan—would, on average, achieve at the proficient level. In Singapore, 55 percent of students were estimated to score at that level, compared with 31 percent in the United States.

On both the science and math tests, the average U.S. 8th grader, like those in the largest group of TIMSS countries, tends to score at levels considered basic on the U.S. NAEP. On the math test, 22 of 46 countries were projected to achieve that status; in science, the number was 20 of 38. The percentage of students who score at that level in the United States is 66.

However, almost as many nations, most of them in Africa and the Middle East, would achieve at below-basic levels on the 8th grade math and science NAEP, according to the analysis, which suggests they would not meet the U.S. definition for grade-level competency. On the math tests, 19 nations fell in that category.

“When you compare the results for those countries to a place like Hong Kong, you can see that’s a world apart,” said Mr. Phillips.

Interpretations Mixed

Mr. Loveless and other experts warned about reading too much into the findings, because the two tests cover different content. But Mr. Phillips said the coverage areas, though not exact, are similar enough to make his comparisons worthwhile.

To arrive at his results, Mr. Phillips drew on earlier studies that analyzed results for two national samples of U.S. students—one of which took TIMSS in 1999 and another that took NAEP in 2000—to find statistical relationships to predict how well students who took one test would do on the other one. The purpose of that earlier study was to determine how students in particular U.S. states would have performed on the international tests. Mr. Phillips used essentially the same process and extended it so that he could estimate how students taking the international tests in 2003 would fare on the U.S. exams.

He acknowledged, though, that the method may have some drawbacks, one of which is that the statistics on which the calculations are drawn are based on U.S. students because they took both NAEP and TIMSS. In the other countries included in the study, students took only the international tests.

Dennis W. Cheek, the vice president of education for the Ewing Marion Kauffman Foundation, based in Kansas City, Mo., said such caveats suggest the study should be viewed as more of a methodological exercise than a policymaking guide: “When you pile on assumption on top of assumption, it’s kind of a shaky platform on which to link all these things.”

Related Tags:

A version of this article appeared in the May 02, 2007 edition of Education Week as Most Nations Seen Falling Short of NAEP ‘Proficiency’


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Measuring & Supporting Student Well-Being: A Researcher and District Leader Roundtable
Students’ social-emotional well-being matters. The positive and negative emotions students feel are essential characteristics of their psychology, indicators of their well-being, and mediators of their success in school and life. Supportive relationships with peers, school
Content provided by Panorama Education
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Making Digital Literacy a Priority: An Administrator’s Perspective
Join us as we delve into the efforts of our panelists and their initiatives to make digital skills a “must have” for their district. We’ll discuss with district leadership how they have kept digital literacy
Content provided by
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
How Schools Can Implement Safe In-Person Learning
In order for in-person schooling to resume, it will be necessary to instill a sense of confidence that it is safe to return. BD is hosting a virtual panel discussing the benefits of asymptomatic screening
Content provided by BD

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment Opinion Alternatives to Standardized Tests During a Pandemic Year
Three educators suggest alternatives to federally mandated standardized testing during this year undercut by COVID-19.
7 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
Assessment Opinion AP Exams Can't Be Business as Usual This Year
The College Board seems unconcerned with the collateral damage of its pandemic approach, writes an assistant superintendent of curriculum and instruction.
Pete Bavis
5 min read
Illustration of large boat in turbulent waters with other smaller boats falling into the abyss.
iStock/Getty Images Plus
Assessment Federal Lawmakers Urge Miguel Cardona to Let States Cancel Tests, Highlighting Discord
A letter from Democratic members to the new education secretary calls for an end to the "flawed" system of annual standardized exams.
3 min read
Jamaal Bowman speaks to reporters after voting at a polling station inside Yonkers Middle/High School in Yonkers, N.Y. on June 23, 2020.
Jamaal Bowman speaks to reporters after voting at a polling station inside Yonkers Middle/High School in Yonkers, N.Y. on June 23, 2020.
John Minchillo/AP
Assessment How Two Years of Pandemic Disruption Could Shake Up the Debate Over Standardized Testing
Moves to opt out of state tests and change how they're given threaten to reignite fights over high-stakes assessments.
9 min read
Image of a student at a desk.