Opinion
Reading & Literacy Letter to the Editor

‘No Effects’ Reading Study Was Poorly Designed

June 14, 2010 1 min read
  • Save to favorites
  • Print

To the Editor:

Once again, Education Week has published a summary of a study that, with one small exception, suggests “no effects” for programs designed to improve student engagement and depth of reading comprehension (“Study of Reading Programs Finds Little Proof of Gains in Student Comprehension,” May 12, 2010). This follows in a long line of “no effects” studies coming from the federal Institute of Education Sciences (“‘No Effects’ Studies Raising Eyebrows,” April 1, 2009).

These reports are frustrating and misleading. A test of the null hypothesis in statistics only allows one to make probabilistic conclusions when there is a positive effect. Unfortunately, the data from these null-result experiments are often reported with the implication that the programs are ineffective, when it is more likely that the design and measurements were poorly controlled and failed to provide a test of the reading programs. There are many ways to obtain no results in large-scale “quasi-scientific” experiments.

Suppose we want to examine the effectiveness of the drug Prozac. Half the subjects are assigned to a treatment condition in which they take Prozac daily, and half the subjects to a placebo condition. If we find no significant difference between the experimental and the control group, can we conclude that Prozac is not effective in treating depression? What if we also know that only 30 percent of the subjects in the treatment group actually took Prozac, and we find out that several subjects in the control condition took Prozac?

Such a study clearly would not allow us to make conclusions about the effectiveness of the drug. Yet this example of “research” is almost a direct parallel to the latest large-scale studies of reading-program effectiveness.

Your newspaper must stop reporting poorly designed, null-result studies with inflammatory headlines such as “Supplementary Reading Programs Found Ineffective” (May 13, 2009). These headlines should instead read “Millions of Federal Dollars Wasted on Poorly Designed Study of Reading.”

Carol M. Santa
John L. Santa

Kalispell, Mont.

Carol M. Santa is the founder and co-owner of Project CRISS, one of the reading programs included in the Institute of Education Sciences study, and a past president of the International Reading Association.

Related Tags:

A version of this article appeared in the June 16, 2010 edition of Education Week as ‘No Effects’ Reading Study Was Poorly Designed

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
Leadership in Education: Building Collaborative Teams and Driving Innovation
Learn strategies to build strong teams, foster innovation, & drive student success.
Content provided by Follett Learning
School & District Management K-12 Essentials Forum Principals, Lead Stronger in the New School Year
Join this free virtual event for a deep dive on the skills and motivation you need to put your best foot forward in the new year.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Privacy & Security Webinar
Navigating Modern Data Protection & Privacy in Education
Explore the modern landscape of data loss prevention in education and learn actionable strategies to protect sensitive data.
Content provided by  Symantec & Carahsoft

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Reading & Literacy Q&A Can Taylor Swift Get Students to Love Poetry?
A college professor will train middle and high school teachers on how to use Swift's lyrics in their curriculum.
8 min read
Singer Taylor Swift performs on stage during her Eras Tour at the Murrayfield Stadium in Edinburgh on June 7, 2024.
Taylor Swift performs on stage during her Eras Tour at the Murrayfield Stadium in Edinburgh on June 7, 2024.
Jane Barlow/PA via AP
Reading & Literacy Photos Drama and Delight: The Faces of the National Spelling Bee
The 2024 Scripps National Spelling Bee came down to a high-stakes spell-off. Here's a look at the faces behind the event.
1 min read
Shrey Parikh, 12, of Rancho Cucamonga, Calif., reacts to a fellow competitor's word during the finals of the Scripps National Spelling Bee, in Oxon Hill, Md., on May 30, 2024.
Shrey Parikh, 12, of Rancho Cucamonga, Calif., reacts to a fellow competitor's word during the finals of the Scripps National Spelling Bee, in Oxon Hill, Md., on May 30, 2024.
Nathan Howard/AP
Reading & Literacy Q&A A New Plan to Raise the Lowest Literacy Rates in the Nation
Daily summer reading instruction for thousands of students is part of a bigger plan to improve literacy in New Mexico.
5 min read
Arsenio Romero, secretary of New Mexico’s Public Education Department, addresses the audience at the Albuquerque Earth Day Festival on April 21, 2024.
Arsenio Romero, the New Mexico secretary of education, speaks at the Albuquerque Earth Day Festival on April 21, 2024. Romero is leading a statewide effort to improve literacy.
Courtesy of New Mexico Public Education Department