Opinion
Reading & Literacy Letter to the Editor

‘No Effects’ Reading Study Was Poorly Designed

June 14, 2010 1 min read
  • Save to favorites
  • Print

To the Editor:

Once again, Education Week has published a summary of a study that, with one small exception, suggests “no effects” for programs designed to improve student engagement and depth of reading comprehension (“Study of Reading Programs Finds Little Proof of Gains in Student Comprehension,” May 12, 2010). This follows in a long line of “no effects” studies coming from the federal Institute of Education Sciences (“‘No Effects’ Studies Raising Eyebrows,” April 1, 2009).

These reports are frustrating and misleading. A test of the null hypothesis in statistics only allows one to make probabilistic conclusions when there is a positive effect. Unfortunately, the data from these null-result experiments are often reported with the implication that the programs are ineffective, when it is more likely that the design and measurements were poorly controlled and failed to provide a test of the reading programs. There are many ways to obtain no results in large-scale “quasi-scientific” experiments.

Suppose we want to examine the effectiveness of the drug Prozac. Half the subjects are assigned to a treatment condition in which they take Prozac daily, and half the subjects to a placebo condition. If we find no significant difference between the experimental and the control group, can we conclude that Prozac is not effective in treating depression? What if we also know that only 30 percent of the subjects in the treatment group actually took Prozac, and we find out that several subjects in the control condition took Prozac?

Such a study clearly would not allow us to make conclusions about the effectiveness of the drug. Yet this example of “research” is almost a direct parallel to the latest large-scale studies of reading-program effectiveness.

Your newspaper must stop reporting poorly designed, null-result studies with inflammatory headlines such as “Supplementary Reading Programs Found Ineffective” (May 13, 2009). These headlines should instead read “Millions of Federal Dollars Wasted on Poorly Designed Study of Reading.”

Carol M. Santa
John L. Santa

Kalispell, Mont.

Carol M. Santa is the founder and co-owner of Project CRISS, one of the reading programs included in the Institute of Education Sciences study, and a past president of the International Reading Association.

Related Tags:

A version of this article appeared in the June 16, 2010 edition of Education Week as ‘No Effects’ Reading Study Was Poorly Designed

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Data Webinar
Drive Instruction With Mastery-Based Assessment
Deliver the right data at the right time—in the right format—and empower better decisions.
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Profession Webinar
How Does Educator Well-Being Impact Social-Emotional Awareness in Schools?
Explore how adult well-being is key to promoting healthy social-emotional behaviors for students. Get strategies to reduce teacher stress.
Content provided by International Baccalaureate
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
IT Infrastructure Webinar
A New Era In Connected Learning: Security, Accessibility and Affordability for a Future-Ready Classroom
Learn about Windows 11 SE and Surface Laptop SE. Enable students to unlock learning and develop new skills.
Content provided by Microsoft Surface

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Reading & Literacy Students Write Their Way to Hope, Courage: Read Their Poems
Five poems from students in Los Angeles and Miami, written to make sense of difficult times.
2 min read
Conceptual image of poetry.
Laura Baker/Education Week (Images: Digital VisionVectors, E+, Pateresca/iStock)
Reading & Literacy ‘It Can Save Lives’: Students Testify to the Power of Poetry
For National Poetry Month, see how teachers and students are exploring the art form.
5 min read
In a Wednesday, April 19, 2017 photo, Eric Charles, left, smiles after performing his poem, "Goodbye to High School Football," for classmates at Sharpstown High School in Houston. Charles compared the rush of performing to the emotions he felt during a football game. Charles had played football since young age, and he planned to play at an elite level in college. However, after injuring his left knee a second time, he found he enjoyed poetry and writing. "That's the glory in me getting hurt," he said.
Eric Charles, left, smiles after performing his poem, "Goodbye to High School Football" for classmates at Sharpstown High School in Houston in 2017. For some students and their teachers, studying and writing poetry has been transformative amid the losses of the pandemic and the wrenching national dialogue about racial justice.
Jon Shapley/Houston Chronicle via AP
Reading & Literacy What the 'Science of Reading' Should Look Like for English-Learners. It's Not Settled
ELLs need additional supports for, and bring different strengths to, early reading instruction, experts say.
10 min read
Sarah Mireles kneels down to work on reading skills with students at Maplewood Elementary in Greeley, Colo., in January of 2018.
Sarah Mireles kneels down to work with Abdigani Hussein, 10, left, and Muhammod Amanullah, 10, during class on at Maplewood Elementary in Greeley, Colo., on Jan. 30, 2018. (Joshua Polson/The Daily Tribune via AP)
Joshua Polson/The Daily Tribune via AP
Reading & Literacy The 'Science of Reading' and English-Language Learners: What the Research Says
As more states embrace certain evidence-based methods, advocates question: Will they work for English learners?
9 min read
Odalys Tebalan works on an assignment at Fairview Elementary in Carthage, Mo., on Nov. 26, 2018. Millions of children are suddenly learning at home as a result of school closures prompted by the global coronavirus pandemic. Many parents are trying to guide their children through assignments, but many face the challenge of English comprehension.
Odalys Tebalan works on an assignment at Fairview Elementary in Carthage, Mo., on Nov. 26, 2018. Millions of children are suddenly learning at home as a result of school closures prompted by the global coronavirus pandemic. Many parents are trying to guide their children through assignments, but many face the challenge of English comprehension.
Roger Nomer/The Joplin Globe via AP