Reading & Literacy Letter to the Editor

‘No Effects’ Reading Study Was Poorly Designed

June 14, 2010 1 min read

To the Editor:

Once again, Education Week has published a summary of a study that, with one small exception, suggests “no effects” for programs designed to improve student engagement and depth of reading comprehension (“Study of Reading Programs Finds Little Proof of Gains in Student Comprehension,” May 12, 2010). This follows in a long line of “no effects” studies coming from the federal Institute of Education Sciences (“‘No Effects’ Studies Raising Eyebrows,” April 1, 2009).

These reports are frustrating and misleading. A test of the null hypothesis in statistics only allows one to make probabilistic conclusions when there is a positive effect. Unfortunately, the data from these null-result experiments are often reported with the implication that the programs are ineffective, when it is more likely that the design and measurements were poorly controlled and failed to provide a test of the reading programs. There are many ways to obtain no results in large-scale “quasi-scientific” experiments.

Suppose we want to examine the effectiveness of the drug Prozac. Half the subjects are assigned to a treatment condition in which they take Prozac daily, and half the subjects to a placebo condition. If we find no significant difference between the experimental and the control group, can we conclude that Prozac is not effective in treating depression? What if we also know that only 30 percent of the subjects in the treatment group actually took Prozac, and we find out that several subjects in the control condition took Prozac?

Such a study clearly would not allow us to make conclusions about the effectiveness of the drug. Yet this example of “research” is almost a direct parallel to the latest large-scale studies of reading-program effectiveness.

Your newspaper must stop reporting poorly designed, null-result studies with inflammatory headlines such as “Supplementary Reading Programs Found Ineffective” (May 13, 2009). These headlines should instead read “Millions of Federal Dollars Wasted on Poorly Designed Study of Reading.”

Carol M. Santa
John L. Santa

Kalispell, Mont.

Carol M. Santa is the founder and co-owner of Project CRISS, one of the reading programs included in the Institute of Education Sciences study, and a past president of the International Reading Association.

Related Tags:

A version of this article appeared in the June 16, 2010 edition of Education Week as ‘No Effects’ Reading Study Was Poorly Designed


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Classroom Technology Webinar
Educator-Driven EdTech Design: Help Shape the Future of Classroom Technology
Join us for a collaborative workshop where you will get a live demo of GoGuardian Teacher, including seamless new integrations with Google Classroom, and participate in an interactive design exercise building a feature based on
Content provided by GoGuardian
School & District Management Live Online Discussion A Seat at the Table With Education Week: What Did We Learn About Schooling Models This Year?
After a year of living with the pandemic, what schooling models might we turn to as we look ahead to improve the student learning experience? Could year-round schooling be one of them? What about online
School & District Management Webinar What's Ahead for Hybrid Learning: Putting Best Practices in Motion
It’s safe to say hybrid learning—a mix of in-person and remote instruction that evolved quickly during the pandemic—is probably here to stay in K-12 education to some extent. That is the case even though increasing

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Reading & Literacy Whitepaper
Dr. Louisa Moats on Why Literacy PD Is Essential
In the white paper, Literacy PD: 10 Reasons Why It’s Essential, renowned literacy expert and author of LETRS® (Language Essentials for Te...
Content provided by Voyager Sopris Learning
Reading & Literacy Most States Fail to Measure Teachers' Knowledge of the 'Science of Reading,' Report Says
The majority of states don’t evaluate whether prospective teachers know how to teach reading effectively, a new analysis finds.
6 min read
Image shows two children ages 5 to 7 years old and a teacher, an African-American woman, holding a digital tablet up, showing it to the girl sitting next to her. They are all wearing masks, back to school during the COVID-19 pandemic, trying to prevent the spread of coronavirus.
Reading & Literacy Opinion The Coming Literacy Crisis: There’s No Going Back to School as We Knew It
Many schools failed to properly teach reading long before the pandemic, write Comer Yates, Renée Boynton-Jarrett, and Maryanne Wolf.
Comer Yates, Renée Boynton-Jarrett & Maryanne Wolf
4 min read
Illustration shows boy of color holding a cage with floating star dust escaping from the cage into the open night sky.
iStock/Getty Images Plus
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Reading & Literacy Sponsor
How have students’ reading habits changed due to the COVID-19 pandemic?
Without a doubt, the average classroom looks more different now than ever before. With schools and districts across the nation engaging in a mix of remote, hybrid, and in-person learning, getting books into the hands of students can be difficult.
Content provided by Renaissance Learning