Opinion
Education Opinion

Bad Science I: Bad Measures

By Robert Slavin — July 19, 2012 2 min read

“My multiple choice test on bike riding was very reliable.
How come none of my kids can ride a bike?”

As an advocate for evidence-based reform in education, I’m always celebrating the glorious possibilities of having educational policies and practices be based on the findings of “rigorous” research. Who could disagree? For this idea to have a little bite in it, however, it is important to understand what I mean by “rigorous.”

In general, a rigorous study evaluating an educational program is one that compares, say, some number of teachers or schools in an experimental program using program X, to others in a control group of very similar characteristics using program Y, which may just be traditional education. Clear enough so far.

One problem arises when we ask, “On what measures should programs X and Y be compared?” Often, this debate revolves around measures felt to be insensitive to real learning gains, as when a study of a science program uses a multiple choice science test. Such studies tend to understate likely program effects.

An even bigger problem occurs when experimenters make up their own measures that are closely linked to the experimental program (X) but not the control program (Y). For example, imagine that a researcher develops a vocabulary-building treatment for English learning and then creates a test around the words emphasized in the program (these words may never have even been introduced to control group). Or, imagine that a researcher develops a science program that spends twice as much time as usual on properties of light, and then develops a test with a heavy emphasis on the very concepts about light added in the extra time. Or a researcher introduces a topic earlier than usual (such as topics of mathematics in preschool) and then uses a measure of the content taught, to which the control group was never exposed. In each of these cases, the experimental group has a huge advantage over the control group, just because they received a lot more teaching on the topic being assessed.

There is a simple solution to this problem. Hold constant the content of instruction while varying the methods, or use widely accepted measures not developed by the experimenter. Studies using measures that are fair to the experimental group and control group tend to report much smaller impacts, but these impacts are a lot more believable than those from studies using measures slanted toward the experimental treatments.

Illustration: Slavin, R.E. (2007). Educational research in the age of accountability. Boston: Allyn & Bacon. Reprinted with permission of the author.

Next Week: Bad Science II: Brief, Small, and Artificial Studies
Find Bob Slavin on Facebook!

The opinions expressed in Sputnik are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Recruitment & Retention Webinar
Recruiting and Retaining a More Diverse Teaching Workforce
We discuss the importance of workforce diversity and learn strategies to recruit and retain teachers from diverse backgrounds.
Content provided by EdWeek Top School Jobs
Student Well-Being Webinar Boosting Teacher and Student Motivation During the Pandemic: What It Takes
Join Alyson Klein and her expert guests for practical tips and discussion on how to keep students and teachers motivated as the pandemic drags on.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
A Holistic Approach to Social-Emotional Learning
Register to learn about the components and benefits of holistically implemented SEL.
Content provided by Committee for Children

EdWeek Top School Jobs

Customer Support Specialist, Tier 1
Portland, OR, US
Northwest Evaluation Association
Customer Support Specialist, Tier 1
Portland, OR, US
Northwest Evaluation Association
Customer Support Specialist, Tier 1
Portland, OR, US
Northwest Evaluation Association
Principal
Meredith, New Hampshire
Inter-Lakes School District

Read Next

Education Briefly Stated Briefly Stated: February 3, 2021
Here's a look at some recent Education Week articles you may have missed.
8 min read
Education Briefly Stated Briefly Stated: January 20, 2021
Here's a look at some recent Education Week articles you may have missed.
9 min read
Education Briefly Stated Briefly Stated: January 13, 2021
Here's a look at some recent Education Week articles you may have missed.
8 min read
Education Obituary In Memory of Michele Molnar, EdWeek Market Brief Writer and Editor
EdWeek Market Brief Associate Editor Michele Molnar, who was instrumental in launching the publication, succumbed to cancer.
5 min read