The international study released here last week offers “confirming evidence” that previous cross-national studies, which have been criticized by some as invalid, did not misrepresent the low levels of American students’ achievement, federal officials maintain.
“When every study comes out with about the same conclusion,” said Diane S. Ravitch, the assistant secretary of education for educational research and improvement, “and the methodological flaws in earlier studies were corrected, you have to say that the others are not moot.”
Education Department officials last week released a new report concluding that, despite their flaws, international studies over the past three decades consistently show that students from the United States lag behind their counterparts in other industrialized countries in mathematics and science achievement.
And the new 20-nation study, conducted by the Educational Testing Service, bolstered that conclusion by adhering to rigorous technical standards for data collection and analysis, according to Emerson J. Elliott, the acting commissioner of the National Center for Education Statistics.
“I have no hesitancy whatsoever saying this is the most consistent, competent international study to date,” Mr. Elliott said.
But Iris C. Rotberg, a senior social scientist at the RAND Corporation, who has been perhaps the most vocal critic of the international comparisons, said the improvements in the new study have failed to eliminate the fundamental problem that plagues all cross-national studies.
The fact remains, Ms. Rotberg said, that differences in test scores among nations do not reflect differences in the quality of education systems.
“The E.T.S. has done what can be done with the research,” she said. “Research can’t be done to produce meaningful findings.
Comparisons ‘Misleading?’
Since the 1960’s, researchers from around the world have conducted six cross-national studies of math and science achievement, and the United States has participated in each one. The International Association for the Evaluation of Educational Achievement, or the i.E.A., conducted four of the studies; the E.T.s. conducted the other two.
Largely because of the attention focused on such studies, President Bush and the nation’s governors, in developing the national education goals, pledged that, by the year 2000, U.S. students will be first in the world in mathematics and science achievement.
The National Education Goals Panel, which has been charged with monitoring progress toward the goals, decided last year to include the I.E.A. and E.T.S. test results as measures of progress.
But in a series of articles over the past two years, Ms. Rotberg has sharply criticized the findings of such studies.
For one thing, she noted, the universe of students who participated in the studies were not consistent across countries, and a country’s relative standing could change dramatically if the tested population changed.
As an example, Ms. Rotberg pointed out that Hong Kong’s 8th graders performed in the middle of the international ranking on an I.E.A. math assessment, while their 12th graders were first.
The difference, she said, is not that Hong Kong’s high schools are dramatically better than their middle schools; rather, it reflects the fact that only 3 percent of Hong Kong’s 12th graders are enrolled in math.
As further evidence of the problems with the international studies, Ms. Rotberg pointed to other indicators of student performance, such as the scientific research productivity of graduate students. And on that score, she said, the U.S. leads the world.
“It is misleading to measure problems or accomplishments simply by scores on multiple-choice tests,” Ms. Rotberg said.
‘A Weak Position’
Norman M. Bradburn, the chairman of the beard on international comparative studies in education, an arm of the National Academy of Sciences created in 1988 to oversee U.S. participation in such studies, acknowledged the problems with the previous studies.
Many of them, he said, arose from the fact that the studies were poorly funded, which hampered researchers’ abilities to conduct proper cross-national analyses.
Nevertheless, he said, the problem did not invalidate the consistent conclusion that American students lag far behind their counterparts from abroad in math and science.
“Even if you take care of [the flaws], the results still show us in a weak position relative to other countries,” said Mr. Bradburn, the director of the National Opinion Research Center at the University of Chicago.
Ms. Rotberg’s emphasis on the “high end” of student performance, Mr. Bradburn added, “misses the point.”
“Most people don’t think the system as a whole has problems,” he said. “International comparisons are saying that what we think of as middle-level schools really aren’t very good by international standards.
Moreover, said William Schmidt, a professor of applied statistics at Michigan State University, the international studies have also provided a wealth of information on the instructional factors associated with high levels of achievement.
“The least important aspect of the international studies is the ‘horse race,’” Mr. Schmidt said.
Quality Controls
In an effort to rid the studies of their technical flaws, the E.T.S., working with the National Academy’s board, attempted to correct many of the problems Ms. Rotberg had identified, according to Archie E. Lapointe, the director of the center for the assessment of educational progress at the E.T.s.
For example, he said, a subcontractor, Westat Inc., carefully drew samples of participating schools and students for each country. The firm also conducted a “quality-control check” at 20 percent of the testing sites to ensure that the test was administered uniformly.
In addition, Mr. Lapointe noted, in reporting the results, the report’s authors pointed out which countries tested a non-representative sample of their population, and separated out those whose samples represented fewer than 90 percent of their population at the testing age.
“The E.T.S. is very confident with the quality of the data,” he said. “Iris has done us a favor.”
But even with the improvements, said Willis D. Hawley, the director of the center for education and human-development policy at Vanderbilt University’s institute for public policy studies, the study fails to address the most serious problems in American schools, such as the poor quality of teacher training.
“There is no question we need to raise standards in the U.S. to compete in the world economy,” he said. “That’s not the issue.”
“The issue,” he added, “is whether international comparisons tell us what to do.”
But Ms. Ravitch, the assistant secretary of education, said researchers should seek ways to improve student performance in math and science, rather than argue over the comparisons.
“The test is a thermometer,” she said. “It doesn’t tell you what the cure is; it tells you what the problem is.”
“But some people’s reaction is to break the thermometer,” Ms. Ravitch said.
