Over the past 30 years, periodic statements of dismay and disappointment over the performance of research funded by the U.S. Department of Education have been a reliable staple in the field. Some of the complaints about the department’s office of educational research and improvement seem to have considerable merit. These include criticisms that center on the following:
- Responding to short-range congressional mandates, which diverts scarce funds;
- Trying to respond to an administration’s priorities, which often may be short-lived;
- Maintaining too cozy a relationship with major federal research-and-development centers and regional education laboratories, a situation that eats up half the available funding;
- Trying to use the remaining money for small, short-term studies of dubious impact; and
- Not using a peer-review system, such as the panels employed by the National Institutes of Health.
The real problem of educational research lies in the inflated expectations of consumers as to what can or should be accomplished through educational innovation.
Attempts to restructure the OERI’s organization and programs could marginally improve the agency’s operation, but I would propose that the real problem of educational research lies elsewhere. It lies in the inflated expectations of consumers as to what can or should be accomplished through educational innovation and research.
Such expectations appear to be drawn from consumers’ experience with research in the areas of health or technology. The development of drugs such as Prednisone or Valium, for example, has given doctors a powerful tool for changing their patients’ lives and outlooks, regardless of what the social context of their medical condition may be. Similarly, the invention of a new family of computer chips that can increase the speed of operation by 100-fold is a dramatic technological advance with hundreds of applications.
But educational research is different. It deals not with the physical or biological world but with the social and behavioral world, a research environment where contextual variations matter greatly, but are elusive to gauge. The educational “treatment"—whether applied behavioral analysis, cooperative learning, or teaching social skills—must be studied in the context of powerful societal and biological factors that limit the impact of even the best of such innovations. Poverty, dysfunctional families, and hostile peer groups can and do cut into the effectiveness of education innovations.
Moreover, the payoff is of necessity limited. Meta-analyses of educational or social-behavioral interventions reveal that consistent gains can be obtained of from 0.5 to 0.75 standard deviations in whatever educational, social, and behavioral goals. These gains are certainly meaningful advances from the current status of the students affected, but they can be counted upon to be a great disappointment to those who had hoped for a breakthrough, a revolutionary improvement in education. It is the public’s search for such a breakthrough—when everything we know about human behavior suggests that only modest changes are possible through intervention, and then only with monumental and persistent effort—that is the problem. (This is clear when we consider, for example, psychotherapy, improving social skills for children with autism, or raising IQ scores.)
The matter is compounded by potential investigators who, aware of consumer interest and expectations, promise outcomes that cannot possibly be reached in order to gain research funds for their work. When the inevitable evaluation of their efforts is published, and the gains are perhaps one-quarter or one-half of what they had promised, another cycle of disappointment and despair ensues, even though gains have in fact been made.
The American people aren't looking for modest gains; they want a "cure." Since no such elixir exists, we once again will be disappointed by the lack of magic in educational innovation.
One of the current hot educational issues, for example, is how we might close the “achievement gap” between minority and majority students that has been evident for a number of years. A large number of potential solutions have been proffered by ambitious educators and researchers who are content to ignore the fact that the basis for this gap lies only slightly in the educational process and teacher preparation, and more deeply in the sociocultural background of various subgroups in our society.
Although there would seem to be good evidence that the achievement gap involves cultural practices and habits that are extremely hard to change in the short run (one generation), these researchers focus mainly on the variables they can influence, namely the classroom and the educational process.
Certainly, improving the schools’ approach to these achievement-gap issues should result in some modest gains, and it does. But the American people aren’t looking for modest gains; they want a “cure.” The achievement gap should vanish with the application of some potent learning drug. Since no such elixir exists, we once again will be disappointed by the lack of magic in educational innovation.
Another example of how thoroughly embedded are our expectations is the fact that we can consider a standard solution to our problems of “inadequately designed educational research.” We are told that we should conduct intervention research using only randomized samples of experimental and control groups in our studies. Yet, this, again, ignores the essential truth that easily 80 percent of the factors influencing the outcome variables for both the experimental and control groups lie outside the “treatment” or “experimental program.” This is in stark contrast to studies of the effects of drug treatments in medicine, where the sheer potency of the treatment often can overwhelm any contextual factors in the study.
Of course, there is nothing wrong with using randomized selection of subjects, when that is possible and relevant, but doing so will hardly change the fact that the educational treatment—whatever it is—just does not have the power to dramatically change the behavior of the subjects. Not, at least, in a way that will satisfy the American consumeror the politician looking for solutions to problems, rather than improvements in coping with those problems.
Let us reform the OERI. But let us also remember that we must make such improvement a continuing goal, rather than an instant cure.
Educators are not alone in having to settle for incremental gains. The physical therapist helping a child with cerebral palsy walk, the speech therapist helping a teenager control his stuttering, the rehabilitation personnel helping those with drug addictions learn to cope—all of these professionals have to settle for small gains. This is not because these specialists are incompetent, but rather because the behavior patterns of human beings are exceedingly difficult to change quickly or dramatically.
So, yes, let us reform the OERI, which like all organizations can and should improve its functioning. But let us also remember that we must make such improvement a continuing goal, rather than an instant cure. Only then we can avoid these periodic lapses into a generalized depression over the performance of our educational establishment.
James J. Gallagher is the Kenan professor of education at the University of North Carolina at Chapel Hill and a senior investigator at the university’s Frank Porter Graham Child Development Institute.
