The recent avalanche of reports and media stories that bemoan the average performance of American students in science on the 41-country Third International Mathematics and Science Study has set off an equally large avalanche of policy suggestions about what should be done about our apparent lack of international competitiveness. As has happened each time one of these international studies rolls out its initial results, average scores (“mean scores” in the jargon of statisticians) for each country on achievement tests are reported with little, if any, additional analysis of the hoards of other information contained in the multimillion-dollar study. Because countries and their relative average performances on almost anything of value make for interesting comparisons, these initial rankings produce a great deal of untested speculation. Often this is harmless. But as we have seen in the past, all too often the policymaking process gets carried away with very premature discussions of what’s wrong and how we should fix it, before more detailed assessments of the problem and its underlying causes can be completed.
We are on the verge of another such binge of runaway speculation about a “slump” in average American science performance from 8th grade on. And just about anything from the hormone-riddled American adolescent to the organization of the average middle school is now being scrutinized and is at risk of being hit with waves of reform-minded rhetoric.
It is true that our average 4th grade performance on the TIMSS science test places U.S. students well above the so-called “international mean” (the average of all countries), while the average for our 8th graders was just above the international mean, and our 12th graders fell well below that mean. Even so, without further detailed analyses, we can’t so easily conclude that as a nation our elementary schools and young students are doing something very right in science, while our middle and secondary schools, or their adolescent students, are doing mediocre to downright awful things in science.
Apparently, a wide array of analysts from across the political spectrum think that these initial score reports make the need for middle school reform a foregone conclusion. In a recent policy paper written for the U.S. Department of Education, Edward Silver proposes sweeping math reforms for U.S. middle schools based on the TIMSS-scores slump. Middle school researchers Joan Lipsitz, Anthony Jackson, and Leah Meyer Austin, in a Phi Delta Kappan article, use America’s low scores as the rationale for suggesting that “poor academic achievement is especially prevalent in middle-grades schools serving high concentrations of low-income students.” And former U.S. Secretary of Education William J. Bennett, speaking at a congressional hearing about these international findings, suggested that a major problem with American education in the middle grades was a permissive culture of adolescence.
It is not that we think that the average scores are in and of themselves wrong; they are not. Recent attacks on the Third International Mathematics and Science Study as a database, such as those that have appeared in Educational Researcher or Phi Delta Kappan, reflect an ideological reaction against international comparisons, rather than a serious examination of the TIMSS methods. Rather, it is premature to focus on just one interpretation of these simple statistical indicators from the study, and it is extremely dangerous to begin formulating policy from what amount to no more than preliminary descriptive statistics. There are many other things we should know before we start to assume that there is a real slump in American performance that would have us focus on middle grades and the conditions of American adolescence.
For example, how common are “slumps” or increases among other nations in TIMSS from the elementary to secondary school scores? Is this just an American phenomenon? Given the centrality of puberty in human development, don’t all nations have adolescents with the accompanying “sturm und drang”? Is the arrangement of placing young adolescent students in middle schools really the culprit?
Just a few additional analyses of the TIMSS data present a strikingly different picture of a middle school slump. These analyses are not nearly as detailed and statistically sophisticated as we would prefer, but they add a cautionary note to the way the policy discussions are currently headed.
While our 4th graders on average do better than the international mean, there is some evidence that suggests we are not building a significant amount of science knowledge at these lower grades. Some of the TIMSS countries, including the United States, administered the science test to two grades, so we can get some rough idea of the average gain from one year of schooling to the next in these countries. The United States’ 3rd to 4th grade gains are not very large compared with other countries'--suggesting that science performance may be “de-accelerating” in the upper-elementary grades. The international picture, however, is complex: Some of the highest-performing countries, such as Japan, actually had lower gains than the United States during the elementary years. The United States continues to post significantly smaller-than-average 7th to 8th grade gains compared with other nations’, but to further complicate matters, some high-scoring countries like Singapore show modest gains in elementary grades and very large gains in middle grades.
All of this shows the danger in speculating while staring at a set of ranked averages, no matter how much fun it might be. Clearly, we need to know much more about the patterns underlying these scores and the ways in which curriculum and instruction in various nations influence national trends.
A similar danger is evident in the assumption that somehow the American middle school, the occasional “organizational whipping boy” in education reform, is to blame for the assumed slump. Even if we were to accept that there is a substantial slump in science training in the middle grades, preliminary analyses suggest that it probably will not be the fault of the middle schools as compared to the old junior high school or extended elementary school. When we look at how much students in each type of school gained from 7th to 8th grade, we find substantively significant gains were made in middle schools. Middle school students, on average, gained about 34 points on the science test, compared with about 22 points for students in old-style elementary schools and barely 18 points for students in junior high schools. This simple test clearly shows it is too early to call for massive reformation of math or science instruction in just U.S. middle schools.
It appears, too, that different countries have different trajectories over grade levels relative to the international mean. This suggests that adolescence, as a difficult stage of life, may not have much to do with cross-national patterns of science achievement. Certainly, more immediate “causes,” such as the long-term organization of science curricula and the professional development of a nation’s science teachers, need examination.
Finally, recent calls for policy reform based on speculation over mean scores tend to commingle the results of science and math scores. Yet, the trajectories in math and science scores appear to follow different patterns: Even at the elementary level, the United States is doing just average in math. The more dramatic changes in rank on science scores may not be closely tied to what is happening in math.
Let us not make important policy decisions about our educational system out of the rankings of average scores across countries. Using preliminary “horse ranking” reports as a basis for educational policy represents the most illiterate use of these data. Clearly, the forming debate over a slump in U.S. scores (and the subsequent calls for school reform) is misleading. The discussions of what we need to do to reform math and science education for adolescents based on TIMSS represent a worst-case scenario of using preliminary data releases to inform public policy in the absence of a systematic and thorough inquiry.
TIMSS contains a wealth of information about teachers, students, and the curriculum. Most of it has received only the most cursory examination. It would perhaps be wise to have a moratorium on “policy” recommendations flowing out of TIMSS until more-detailed analyses can be performed and incorporated into the discussion.
