Assessment

Most Nations Seen Falling Short of NAEP ‘Proficiency’

By Debra Viadero — May 01, 2007 6 min read
  • Save to favorites
  • Print

Outside of a handful of Asian nations, the typical 8th grader in many foreign countries would not meet “proficient” levels on U.S. tests of mathematics and science, a new reanalysis of international achievement shows.

Then again, the study also shows, neither do most American students.

The study, which was posted April 24 on the Web site of the American Institutes for Research, a Washington-based research organization, offers fodder for national debates over whether the United States has set its achievement targets too high. The new analysis, the second in a year to draw nearly the same conclusions from reinterpreted international data, comes from the AIR’s chief scientist, Gary W. Phillips.

“Expressing International Educational Achievement in Terms of U.S. Performance Standards: Linking NAEP Achievement Levels to TIMSS” is posted by the American Institutes for Research.

Mr. Phillips’ idea was to statistically “link” scores from two well-known testing programs: the Trends in International Mathematics and Science Study, or TIMSS, which is given every few years to students in more than three dozen countries, and the National Assessment of Educational Progress, a program mandated by the U.S. Congress that is known as “the nation’s report card.”

“I wanted to re-express the TIMSS results in terms of a set of standards that U.S. policymakers would be familiar with,” said Mr. Phillips, who was acting commissioner of the National Center for Education Statistics, the data-gathering arm of the U.S. Education Department, from 1999 to 2002. “It’s a lot like looking at world-poverty levels. It’s impossible to get your head around it until you get a common metric.”

Defining Proficient

Global Comparisons

A study designed to predict how other countries would fare on U.S. mathematics and science tests found that the typical 8th grader in most places, including the United States, would score at the “basic” level or below on the National Assessment of Educational Progress math test.

BRIC ARCHIVE

Note: “Basic” denotes a minimum score of 469, “proficient” a minimum of 556, and “advanced,” 637.
SOURCE: American Institutes for Research

Although Mr. Phillips saw his study as primarily a methodological test, the results come at a time when the NAEP achievement levels, which have been debated on and off for years, are attracting renewed attention in national discussions over what it means for U.S. students to be academically “proficient” or “world class.”

The federal No Child Left Behind Act calls for all students to be proficient by the 2013-14 school year. But studies show huge gaps between NAEP’s definition of proficient and the definitions states use in their own assessment programs. (“Gaps in Proficiency Levels on State Tests And NAEP Found to Grow,” April 18, 2007.)

“The notion of all students being proficient is an oxymoron,” said Richard Rothstein, a research associate for the Economic Policy Institute, a Washington-based think tank. “You can’t have a standard that challenges students at all ends of the distribution system.”

Using slightly different methodology, Mr. Rothstein put out a paper last fall that found results similar to Mr. Phillips’: Countries that score highest on the TIMSS tests don’t ensure that all students meet the NAEP definition of proficient.

“My conclusion is that we need to take a good look at where all the cut points are for defining basic skills or proficiency,” said Tom Loveless, the director of the Brown Center on Education Policy at the Brookings Institution, a Washington-based research organization, referring specifically to the NAEP tests.

However, Chester E. Finn Jr., the president of the Thomas B. Fordham Foundation, another Washington group that studies school improvement issues, said the results confirmed the current levels for the NAEP tests. “If you have normal students from normal public schools in normal countries and more than half of them are reaching proficient levels, then we have set exactly the right target that everybody should aim for,” said Mr. Finn, who once sat on the governing board that sets policy for the federal assessment program.

Mr. Phillips, who undertook the study on his own initiative, said he agreed. “A lot of people have said the NAEP achievement levels are too high,” he noted. “I think it’s pretty clear that’s not the case.”

‘A World Apart’

NAEP uses three levels to gauge student achievement: “basic,” which generally denotes partial mastery of the academic material; “proficient,” for solid mastery of challenging academic content; and “advanced,” to describe superior achievement. Students who reach none of those levels are described as performing “below basic.”

By Mr. Phillips’ estimate, of the 8th graders from 46 nations who took TIMSS mathematics tests in 2003, only students from Hong Kong, Japan, Singapore, South Korea, and Taiwan would reach the proficient level, on average, if they were to take the 8th grade NAEP in that subject.

For example, in Singapore, the nation that topped the international tests, the analysis shows that 73 percent of 8th graders would score at or above the proficient level on NAEP, compared with 26 percent in the United States. The percentage of 8th graders reaching the NAEP advanced level in that Asian country was estimated to be 35 percent—seven times greater than the U.S. rate.

On the other hand, in four nations—Botswana, Ghana, Saudi Arabia, and South Africa—almost no students scored at the advanced level.

The science results suggest a similar pattern. On that test, the analysis shows, 8th graders in only two of the TIMSS countries—Singapore and Taiwan—would, on average, achieve at the proficient level. In Singapore, 55 percent of students were estimated to score at that level, compared with 31 percent in the United States.

On both the science and math tests, the average U.S. 8th grader, like those in the largest group of TIMSS countries, tends to score at levels considered basic on the U.S. NAEP. On the math test, 22 of 46 countries were projected to achieve that status; in science, the number was 20 of 38. The percentage of students who score at that level in the United States is 66.

However, almost as many nations, most of them in Africa and the Middle East, would achieve at below-basic levels on the 8th grade math and science NAEP, according to the analysis, which suggests they would not meet the U.S. definition for grade-level competency. On the math tests, 19 nations fell in that category.

“When you compare the results for those countries to a place like Hong Kong, you can see that’s a world apart,” said Mr. Phillips.

Interpretations Mixed

Mr. Loveless and other experts warned about reading too much into the findings, because the two tests cover different content. But Mr. Phillips said the coverage areas, though not exact, are similar enough to make his comparisons worthwhile.

To arrive at his results, Mr. Phillips drew on earlier studies that analyzed results for two national samples of U.S. students—one of which took TIMSS in 1999 and another that took NAEP in 2000—to find statistical relationships to predict how well students who took one test would do on the other one. The purpose of that earlier study was to determine how students in particular U.S. states would have performed on the international tests. Mr. Phillips used essentially the same process and extended it so that he could estimate how students taking the international tests in 2003 would fare on the U.S. exams.

He acknowledged, though, that the method may have some drawbacks, one of which is that the statistics on which the calculations are drawn are based on U.S. students because they took both NAEP and TIMSS. In the other countries included in the study, students took only the international tests.

Dennis W. Cheek, the vice president of education for the Ewing Marion Kauffman Foundation, based in Kansas City, Mo., said such caveats suggest the study should be viewed as more of a methodological exercise than a policymaking guide: “When you pile on assumption on top of assumption, it’s kind of a shaky platform on which to link all these things.”

Related Tags:

A version of this article appeared in the May 02, 2007 edition of Education Week as Most Nations Seen Falling Short of NAEP ‘Proficiency’

Events

Classroom Technology Webinar How Pandemic Tech Is (and Is Not) Transforming K-12 Schools
The COVID-19 pandemic—and the resulting rise in virtual learning and big investments in digital learning tools— helped educators propel their technology skills to the next level. Teachers have become more adept at using learning management
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Building Teacher Capacity for Social-Emotional Learning
Set goals that support adult well-being and social-emotional learning: register today!


Content provided by Panorama
Jobs October 2021 Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment State Test Results Are In. Are They Useless?
While states, districts, and schools pore over data from spring 2021 tests, experts urge caution over how to interpret and use the results.
9 min read
FILE - In this Jan. 17, 2016 file photo, a sign is seen at the entrance to a hall for a college test preparation class in Bethesda, Md. The $380 million test coaching industry is facing competition from free or low-cost alternatives in what their founders hope will make the process of applying to college more equitable. Such innovations are also raising questions about the relevance and the fairness of relying on standardized tests in admissions process.
A sign is posted at the entrance to a hall for a test-preparation class. Assessment experts say educators should use data from spring 2021 tests with caution.
Alex Brandon/AP
Assessment Data Young Adolescents' Scores Trended to Historic Lows on National Tests. And That's Before COVID Hit
The past decade saw unprecedented declines in the National Assessment of Educational Progress's longitudinal study.
3 min read
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Assessment Whitepaper
Proven Techniques for Assessing Students with Technology
Dr. Doug Fisher’s proven assessment techniques help your students become active learners and increase their chances for higher learning g...
Content provided by Achieve3000
Assessment Long a Testing Bastion, Florida Plans to End 'Outdated' Year-End Exams
Florida Gov. Ron DeSantis said the state will shift to "progress monitoring" starting in the 2022-23 school year.
5 min read
Florida Governor Ron DeSantis speaks at the opening of a monoclonal antibody site in Pembroke Pines, Fla., on Aug. 18, 2021.
Florida Gov. Ron DeSantis said he believes a new testing regimen is needed to replace the Florida Standards Assessment, which has been given since 2015.
Marta Lavandier/AP