Most Nations Seen Falling Short of NAEP ‘Proficiency’

By Debra Viadero — May 01, 2007 6 min read
  • Save to favorites
  • Print

Outside of a handful of Asian nations, the typical 8th grader in many foreign countries would not meet “proficient” levels on U.S. tests of mathematics and science, a new reanalysis of international achievement shows.

Then again, the study also shows, neither do most American students.

The study, which was posted April 24 on the Web site of the American Institutes for Research, a Washington-based research organization, offers fodder for national debates over whether the United States has set its achievement targets too high. The new analysis, the second in a year to draw nearly the same conclusions from reinterpreted international data, comes from the AIR’s chief scientist, Gary W. Phillips.

“Expressing International Educational Achievement in Terms of U.S. Performance Standards: Linking NAEP Achievement Levels to TIMSS” is posted by the American Institutes for Research.

Mr. Phillips’ idea was to statistically “link” scores from two well-known testing programs: the Trends in International Mathematics and Science Study, or TIMSS, which is given every few years to students in more than three dozen countries, and the National Assessment of Educational Progress, a program mandated by the U.S. Congress that is known as “the nation’s report card.”

“I wanted to re-express the TIMSS results in terms of a set of standards that U.S. policymakers would be familiar with,” said Mr. Phillips, who was acting commissioner of the National Center for Education Statistics, the data-gathering arm of the U.S. Education Department, from 1999 to 2002. “It’s a lot like looking at world-poverty levels. It’s impossible to get your head around it until you get a common metric.”

Defining Proficient

Global Comparisons

A study designed to predict how other countries would fare on U.S. mathematics and science tests found that the typical 8th grader in most places, including the United States, would score at the “basic” level or below on the National Assessment of Educational Progress math test.


Note: “Basic” denotes a minimum score of 469, “proficient” a minimum of 556, and “advanced,” 637.
SOURCE: American Institutes for Research

Although Mr. Phillips saw his study as primarily a methodological test, the results come at a time when the NAEP achievement levels, which have been debated on and off for years, are attracting renewed attention in national discussions over what it means for U.S. students to be academically “proficient” or “world class.”

The federal No Child Left Behind Act calls for all students to be proficient by the 2013-14 school year. But studies show huge gaps between NAEP’s definition of proficient and the definitions states use in their own assessment programs. (“Gaps in Proficiency Levels on State Tests And NAEP Found to Grow,” April 18, 2007.)

“The notion of all students being proficient is an oxymoron,” said Richard Rothstein, a research associate for the Economic Policy Institute, a Washington-based think tank. “You can’t have a standard that challenges students at all ends of the distribution system.”

Using slightly different methodology, Mr. Rothstein put out a paper last fall that found results similar to Mr. Phillips’: Countries that score highest on the TIMSS tests don’t ensure that all students meet the NAEP definition of proficient.

“My conclusion is that we need to take a good look at where all the cut points are for defining basic skills or proficiency,” said Tom Loveless, the director of the Brown Center on Education Policy at the Brookings Institution, a Washington-based research organization, referring specifically to the NAEP tests.

However, Chester E. Finn Jr., the president of the Thomas B. Fordham Foundation, another Washington group that studies school improvement issues, said the results confirmed the current levels for the NAEP tests. “If you have normal students from normal public schools in normal countries and more than half of them are reaching proficient levels, then we have set exactly the right target that everybody should aim for,” said Mr. Finn, who once sat on the governing board that sets policy for the federal assessment program.

Mr. Phillips, who undertook the study on his own initiative, said he agreed. “A lot of people have said the NAEP achievement levels are too high,” he noted. “I think it’s pretty clear that’s not the case.”

‘A World Apart’

NAEP uses three levels to gauge student achievement: “basic,” which generally denotes partial mastery of the academic material; “proficient,” for solid mastery of challenging academic content; and “advanced,” to describe superior achievement. Students who reach none of those levels are described as performing “below basic.”

By Mr. Phillips’ estimate, of the 8th graders from 46 nations who took TIMSS mathematics tests in 2003, only students from Hong Kong, Japan, Singapore, South Korea, and Taiwan would reach the proficient level, on average, if they were to take the 8th grade NAEP in that subject.

For example, in Singapore, the nation that topped the international tests, the analysis shows that 73 percent of 8th graders would score at or above the proficient level on NAEP, compared with 26 percent in the United States. The percentage of 8th graders reaching the NAEP advanced level in that Asian country was estimated to be 35 percent—seven times greater than the U.S. rate.

On the other hand, in four nations—Botswana, Ghana, Saudi Arabia, and South Africa—almost no students scored at the advanced level.

The science results suggest a similar pattern. On that test, the analysis shows, 8th graders in only two of the TIMSS countries—Singapore and Taiwan—would, on average, achieve at the proficient level. In Singapore, 55 percent of students were estimated to score at that level, compared with 31 percent in the United States.

On both the science and math tests, the average U.S. 8th grader, like those in the largest group of TIMSS countries, tends to score at levels considered basic on the U.S. NAEP. On the math test, 22 of 46 countries were projected to achieve that status; in science, the number was 20 of 38. The percentage of students who score at that level in the United States is 66.

However, almost as many nations, most of them in Africa and the Middle East, would achieve at below-basic levels on the 8th grade math and science NAEP, according to the analysis, which suggests they would not meet the U.S. definition for grade-level competency. On the math tests, 19 nations fell in that category.

“When you compare the results for those countries to a place like Hong Kong, you can see that’s a world apart,” said Mr. Phillips.

Interpretations Mixed

Mr. Loveless and other experts warned about reading too much into the findings, because the two tests cover different content. But Mr. Phillips said the coverage areas, though not exact, are similar enough to make his comparisons worthwhile.

To arrive at his results, Mr. Phillips drew on earlier studies that analyzed results for two national samples of U.S. students—one of which took TIMSS in 1999 and another that took NAEP in 2000—to find statistical relationships to predict how well students who took one test would do on the other one. The purpose of that earlier study was to determine how students in particular U.S. states would have performed on the international tests. Mr. Phillips used essentially the same process and extended it so that he could estimate how students taking the international tests in 2003 would fare on the U.S. exams.

He acknowledged, though, that the method may have some drawbacks, one of which is that the statistics on which the calculations are drawn are based on U.S. students because they took both NAEP and TIMSS. In the other countries included in the study, students took only the international tests.

Dennis W. Cheek, the vice president of education for the Ewing Marion Kauffman Foundation, based in Kansas City, Mo., said such caveats suggest the study should be viewed as more of a methodological exercise than a policymaking guide: “When you pile on assumption on top of assumption, it’s kind of a shaky platform on which to link all these things.”

Related Tags:

A version of this article appeared in the May 02, 2007 edition of Education Week as Most Nations Seen Falling Short of NAEP ‘Proficiency’


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Attend to the Whole Child: Non-Academic Factors within MTSS
Learn strategies for proactively identifying and addressing non-academic barriers to student success within an MTSS framework.
Content provided by Renaissance
Classroom Technology K-12 Essentials Forum How to Teach Digital & Media Literacy in the Age of AI
Join this free event to dig into crucial questions about how to help students build a foundation of digital literacy.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment Should Teachers Be Tough Graders? Here's What They Have to Say
Teachers on social media give their opinions on whether stricter grading helps their students learn more.
2 min read
Close cropped photo of a teacher's grade on an essay graded 'F' in red with the words "See Me"
Assessment The State of Teaching Where Teachers Say the Pressure to Change Grades Comes From
Teachers are more likely to be pressured by parents than school leaders.
4 min read
Conceptul image in blues of a teacher handing out graded papers.
Liz Yap/Education Week and E+
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Assessment Sponsor
Testing Season: Who Are We Really Testing For? Transforming Assessments from Obstacles to Opportunities
As another testing season approaches, a familiar question weighs heavily on our minds: who are these tests serving?
Content provided by Achievement Network
Assessment What the Research Says AI and Other Tech Can Power Better Testing. Can Teachers Use the New Tools?
Assessment experts call for better educator supports for technology use.
3 min read
Illustration of papers and magnifying glass
iStock / Getty Images Plus