Opinion
Reading & Literacy Letter to the Editor

‘No Effects’ Reading Study Was Poorly Designed

June 14, 2010 1 min read
  • Save to favorites
  • Print

To the Editor:

Once again, Education Week has published a summary of a study that, with one small exception, suggests “no effects” for programs designed to improve student engagement and depth of reading comprehension (“Study of Reading Programs Finds Little Proof of Gains in Student Comprehension,” May 12, 2010). This follows in a long line of “no effects” studies coming from the federal Institute of Education Sciences (“‘No Effects’ Studies Raising Eyebrows,” April 1, 2009).

These reports are frustrating and misleading. A test of the null hypothesis in statistics only allows one to make probabilistic conclusions when there is a positive effect. Unfortunately, the data from these null-result experiments are often reported with the implication that the programs are ineffective, when it is more likely that the design and measurements were poorly controlled and failed to provide a test of the reading programs. There are many ways to obtain no results in large-scale “quasi-scientific” experiments.

Suppose we want to examine the effectiveness of the drug Prozac. Half the subjects are assigned to a treatment condition in which they take Prozac daily, and half the subjects to a placebo condition. If we find no significant difference between the experimental and the control group, can we conclude that Prozac is not effective in treating depression? What if we also know that only 30 percent of the subjects in the treatment group actually took Prozac, and we find out that several subjects in the control condition took Prozac?

Such a study clearly would not allow us to make conclusions about the effectiveness of the drug. Yet this example of “research” is almost a direct parallel to the latest large-scale studies of reading-program effectiveness.

Your newspaper must stop reporting poorly designed, null-result studies with inflammatory headlines such as “Supplementary Reading Programs Found Ineffective” (May 13, 2009). These headlines should instead read “Millions of Federal Dollars Wasted on Poorly Designed Study of Reading.”

Carol M. Santa
John L. Santa

Kalispell, Mont.

Carol M. Santa is the founder and co-owner of Project CRISS, one of the reading programs included in the Institute of Education Sciences study, and a past president of the International Reading Association.

Related Tags:

A version of this article appeared in the June 16, 2010 edition of Education Week as ‘No Effects’ Reading Study Was Poorly Designed

Events

Teaching Profession K-12 Essentials Forum New Insights Into the Teaching Profession
Join this free virtual event to get exclusive insights from Education Week's State of Teaching project.
Jobs Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and K-12 education jubs at the EdWeek Top School Jobs virtual career fair.
Mathematics K-12 Essentials Forum Helping Students Succeed in Math

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Reading & Literacy Fewer Parents Are Reading Aloud to Their Kids. Why That Matters
Pleasure reading is the best way for emerging readers to maintain their skills in the summer. Getting parents on board can be challenging.
5 min read
Image of a parent and child reading together at home.
iStock/Getty
Reading & Literacy Opinion 6 Active Reading Strategies to Engage Students (DOWNLOADABLE)
How can teachers scaffold literacy instruction for developing readers? Start here.
Jennifer Throndsen
1 min read
Hand lifting a dumbbell made of colorful books against a blue background. Concept of active reading, knowledge, and strength. 3d rendering
Antonio Solano/iStock
Reading & Literacy How Spelling Bees Can Improve Students' Reading Skills
Faizan Zaki, age 13, spelled the word “eclaircissement” and collapsed on the floor in victory as confetti showered him at the 100th Scripps National Spelling Bee last week.
3 min read
Faizan Zaki, 13, of Dallas, reacts after he sees him as the last speller as he competes in the finals the 2025 Scripps National Spelling Bee at the Gaylord National Resort & Convention Center Thursday, May 29, 2025, in Oxon Hill, Md.
Faizan Zaki, 13, of Dallas, reacts after he realizes he's the last speller as he competes in the finals of the 2025 Scripps National Spelling Bee at the Gaylord National Resort & Convention Center on May 29, 2025, in Oxon Hill, Md. Schools can improve reading and writing proficiency through spelling bees.
Jose Luis Magana/AP
Reading & Literacy Trump's Ed. Dept. Wants to Fund 'Evidence-Based' Reading. What Will That Mean for Schools?
The agency has placed literacy squarely on its agenda amid an ongoing national "science of reading" movement.
9 min read
Richard Evans, a teacher at Hyde Park Elementary School, helps a student sound out a word during a reading circle in class on Thursday, Oct. 20, 2022, in Niagara Falls, N.Y.
Richard Evans, a teacher at Hyde Park Elementary School, helps a student sound out a word during a reading circle in class on Thursday, Oct. 20, 2022, in Niagara Falls, N.Y. Federal support for reading has waxed and waned over the years, and now the Trump Administration says it will prioritize "evidence-based" reading supports in its grant programs.
Joshua Bessex/AP