Education Letter to the Editor

Cost and Effect

May 24, 2005 3 min read
  • Save to favorites
  • Print

To the Editor:

Experiments are powerful analytical tools. They can provide cause-and-effect conclusions more certainly than other approaches. With respect to educational evaluations, however, they are also very costly tools.

Costly tools should be used when there is something that needs to be found out, rather than to add a little bit of confirmation about something already understood.

Your reporting of the second-year results from a true experiment on the Success for All program (“Long-Awaited Study Shows ‘Success for All’ Gains,” May 11, 2005.) raised for me the question of whether we needed such an experiment to decide if that program produces small, positive effects on beginning reading. There is a lot of existing evaluation data on Success for All, and a reasonable summary is that the approach often produces small, positive effects on beginning language arts, although not always.

While it is logically possible that all of the positive, nonexperimental evaluations of Success for All were somehow biased in the program’s favor, that seems unlikely. It would require an efficiency of corruption beyond even the most resourceful educational entrepreneur. As often as the correlation between Success for All and small, positive effects on language arts has been observed, a causal relationship is a pretty good bet. What third variable could possibly account for the repeated linkages between Success for All and beginning-reading achievement that has been observed? Since I can’t think of one, I have my doubts about whether a costly true experiment just to make certain there is a causal relationship made economic sense.

My doubts were heightened as I reflected on Herbert J. Walberg’s remark in your article discounting the true experimental result because the developers of Success for All were involved in the study. As a deeply experienced experimenter, as well as an experienced reader and reviewer of the experimental literature in education, I recognize that a very high proportion of experiments are done by folks who either have developed or have a great affinity for the intervention they are studying. Following Mr. Walberg’s logic, much of the experimental education literature that exists should be dismissed.

Is the solution to do another very expensive experimental evaluation of Success for All, this time conducted by some dispassionate third party? Believe me, if such a study were done, there would be serious doubts raised about it, too. In fact, I have never read a single experimental study about which some kind of doubt (often serious) could not be raised.

The only way to get to a firm conclusion about causality is for there to be many experiments by many parties in many settings—with each experiment having differing weaknesses in the eyes of various critics. Given that we already know the program’s use is associated with small, positive effects on beginning language arts, I could never justify the resources that would be required to prove conclusively the causal relationship between the program and the effects.

Rather than funding experiments on interventions that have been studied a lot, the federal government should spend the limited research dollars available on the development of new interventions, ones that have the potential, at least, to produce something better than small, positive effects. And if we get such studies, we’re going to have to have more faith than critics like Mr. Walberg, for first experiments often will be done by program developers, and one (or a very few) experiments may be all the experimentation we can afford.

It is time for the education research leadership to help the country develop an understanding that experiments are precious and rare resources that should be employed judiciously. The scientific leadership also needs to make clear to the country that although the results of individual experiments always should be used cautiously, they should be used even after a single experiment. A second experiment may never come—and the perfect experiment definitely will never come.

Michael Pressley

Professor of Education

Michigan State University

East Lansing, Mich.


Jobs Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
Challenging the Stigma: Emotions and STEM
STEM isn't just equations and logic. Join this webinar and discover how emotions fuel innovation, creativity, & problem-solving in STEM!
Content provided by Project Lead The Way
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Professional Development Webinar
Leveraging Student Voice for Teacher Retention & Development
Join our webinar on using student feedback to improve teacher performance, retention & student achievement.
Content provided by Panorama Education

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Briefly Stated: February 7, 2024
Here's a look at some recent Education Week articles you may have missed.
8 min read
Education Briefly Stated: January 31, 2024
Here's a look at some recent Education Week articles you may have missed.
9 min read
Education Briefly Stated: January 17, 2024
Here's a look at some recent Education Week articles you may have missed.
9 min read
Education In Their Own Words The Stories That Stuck With Us, 2023 Edition
Our newsroom selected five stories as among the highlights of our work. Here's why.
4 min read
102523 IMSE Reading BS
Adria Malcolm for Education Week