The latest international study of math and science achievement is “misleading and seriously flawed,” according to a critique published this month in Science.
|
The article by Iris C. Rotberg, a research professor of educational leadership at George Washington University in Washington, in the journal’s May 15 issue picks apart results from the Third International Mathematics and Science Study. The findings drew wide attention earlier this year after they showed that American high school seniors ranked near the bottom among the 21 nations that participated. (“U.S. Seniors Near Bottom in World Test,” March 4, 1998.)
But Ms. Rotberg, a longtime critic of comparative international studies, said the results from TIMSS are “misinterpretable” because the countries involved differed in too many ways. Some adhered more closely to testing guidelines than others, she said. Others tested greater proportions of students within the end-of-school age group. Some countries had higher proportions of poor students, and others routinely track their best math and science students into specialized schools.
“Because of all the methodological problems and uncontrolled variables, the tests simply are meaningless for comparing the quality of education in any of the participating countries,” Ms. Rotberg said last week.
But Michael O. Martin, the deputy international-study director for TIMSS, and other testing experts said the results--though not perfect--still offer a sound basis for making comparisons.
“The U.S. performance isn’t going to get any better by just complaining about comparisons,” added Mr. Martin, who is also an education professor at Boston College. “U.S. students just don’t take as much advanced mathematics and science.”
Comparable Samples?
The international study, which tested students in 4th grade, 8th grade, and the final year of secondary school, is widely described as the largest, most comprehensive study of student achievement ever undertaken. It tested performance in both general and advanced math and science, analyzed the curricula in participating countries, and even videotaped classes in a few countries.
But Ms. Rotberg is among a small group of researchers and experts who have been vocal in their criticism of the study--and the 12th grade results in particular. The complaints have even prompted the National Center for Education Statistics to post a response on its World Wide Web site.
“I don’t conclude from the study that the U.S. doesn’t have educational problems,” Ms. Rotberg said. “What I’m saying is our problems or strengths or those of any other country are not identified one way or another from the study.”
One of her primary concerns is that only five of the 21 nations that took part in the study’s general-achievement test met its guidelines for sampling and procedures. For the advanced tests, only six of 16 countries qualified.
Russia, for example, did not test vocational school students or non-Russian-speakers. Students in four regions in Italy did not participate, and Cyprus tested only its best science and math students.
Ms. Rotberg says the varying participation rates can skew the results. A country with a low participation rate, for example, might show inflated scores because low-achieving students were excluded.
But Mr. Martin said the project’s guidelines were strict--more so than those in previous international assessments.
And, aside from obvious exceptions such as Russia and Cyprus, in most countries “there wasn’t any clear evidence that certain kinds of students were not being included,” said Keith Rust, a private consultant who acted as an international referee on some of the testing procedures used by timss. “It throws another caution,” he said, “but I don’t think it means you can say the results of the study are in question.”
Age Gaps Questioned
Ms. Rotberg also notes that students in some countries were older than American high school seniors. The average age of U.S. test-takers was 18.1 years--just a few months below the international average but years younger than in Iceland, for example, where the average age of participating students was 21.
Countries where students moved down in rank between the 8th grade test and the test at the end of high school, such as the United States, had the smallest average age gap between those grade levels, Ms. Rotberg writes. The nations that gained in rank between those grades, on the other hand, had bigger age gaps.
But TIMSS researchers say the age issue misses the point.
“What we chose to do was to compare students at the end of secondary school,” Mr. Martin said. Regardless of how old students are when they leave school, he added, “it’s still legitimate to ask, at the end of the day, how much math and science have they learned?”
“I guess the most important message here is that no research should be looked on as being of such a state of grace that it should be accepted without any thought or questions,” said Paul LeMahieu, a researcher from the University of Delaware, Newark, who was not connected with the TIMSS project. “But that doesn’t mean the opposite is true and that one should dismiss what has been a very valuable effort.”
