Opinion
Education Opinion

Educational Research: To Address Rather Than to Answer

By Justin Reich — September 04, 2013 6 min read
  • Save to favorites
  • Print

As per usual, the trolls at the New York Times have been following the conversations here at EdTechResearcher, and now they’re piggybacking on our work with an article on education research, randomized experiments, and the What Works Clearinghouse.

To review, last week, I got a little frumpy at two headlines summarizing results from a Department of Education/RAND study of a blended learning learning platform: Cognitive Tutor: Alegbra. One headline said “Cognitive Tutor: Algebra Doubles Student Learning,” and I wrote a post that said, “sort of, but not always, and high schools may find lower performance in year 1 before gains in year 2, and we’re not totally sure for middle school.”

Then, something lovely happened: a whole bunch of people started a civil discussion about the research. Christina Quattrocchi who wrote one of the headlines that I quibbled with wrote an article further investigating the findings and interpretation of the study. John Pane, the study’s lead author and I exchanged some email correspondence where he agreed with some of my interpretations and suggested some alternative perspectives on others. A colleague of Betsy Corcoran, the founder of EdSurge, invited me to join them on a panel discussion (which I couldn’t do, but I hope to in the future). Steve Ritter, founder of Carnegie Learning, weighed in with his perspectives in a comment here.

The conversations pushed my thinking. Steve Ritter noted in his comment that I characterized the study as four smaller studies—two year-long studies in high schools and two year-long studies in middle school—and he argued that they are best understood as two studies of the same high schools over two years and the same middle schools over two years. Steve’s point is quite valid; the fact that the study follows the same schools for two years is very important (I was highlighting something different, the fact that we have four sets of results to examine, which is also important). Steve and I both discussed the fact that there wasn’t quantitative, empirical evidence that proved that the improvement in high school outcomes from year 1 to year 2 was due to teacher improvement or practice changes. But to that point, John Pane notes in our email correspondence that the sample size of teachers was much smaller than the sample size of students, and therefore the study didn’t necessarily have the statistical power to detect those changes. He noted that other studies have identified the phenomena of an “innovator’s dip,” and there aren’t really very many compelling alternative explanations for why scores would be low in year 1 and higher in year 2. Even though we don’t have great quantitative evidence from this study, it’s not crazy to think that the gains between year 1 and year 2 were because teachers got better. I think John has a good point, here, if not a decisive one. (RAND has policies limiting publishing John’s remarks until the full study is published in a journal, which is reasonable. Otherwise, I’d publish our correspondence). If I re-wrote that blog post today, or made it longer, I think these would be worthy perspectives to dig more into.

Let me highlight one important thing about these conversations: we all agree on the same body of facts. The study, as far as I can tell, was very carefully done. Steve, John, and I are all turning to the same body of findings, the same statistical analysis, and using those findings as the shared text for a conversation. We then seem to have some disagreements (along with many areas of near-universal concurrence) about how to interpret those findings, how to communicate those findings, and how to help educational decision makers leverage those findings in their own contexts. The research absolutely helps advance our understanding of blended learning in mathematics. It informs conversations about what we should do. It doesn’t provide a straightforward answer for everyone.

When I was writing my first grant application as a doctoral student, my advisor, the economist of education Richard Murnane, changed some wording in a way that I now consider quite illuminating. I wrote that we were going to “answer these three research questions about excellence and equity.” Dick changed it to read that we would “address these three research questions.” In most cases, research addresses questions rather than answers them.

It’s that nuance that is missing from the recent New York Times piece on educational research. The article suggests that experimental research can remove the guess-work from education. But just like doctors still have to make informed judgments (“guesses”) rather than simply having computers automatically recommend treatments based on symptoms or vital signs, teachers and educators will still need to make informed judgments (“guesses”) using a wide variety of educational research. Research narrows the bounds of our debates, and it narrows the range of informed judgments. It addresses, rather than answers.

To some extent, public understanding of the affordances of research is set back by branding from the Institute of Education Sciences. The very name “What Works Clearinghouse” suggests that studies can definitely answer the question “What Works?” But even well-conducted randomized control trials that have statistically significant results can only tell us what works on average across a variety of typical settings. Most educators don’t need to answer “What Works?” They need to answer “What works, for whom, when, and under what conditions?” As the discussion on Cognitive Tutor:Algebra shows, the results from even our best studies often don’t lead to definitive answers for decision makers, they lead to better information to support more informed decisions.

So, should this send us into a tizzy of skeptical malaise? No. I’m not saying that we’ll never learn anything from research, or that everything is subjective, or that everyone should willy-nilly make decisions based on their gut or what worked for their kid. (Nor, by the way, did I claim in my original post that Cognitive Tutor: Algebra had no value or didn’t work; that claim is as wrong as the claim that it always doubles learning.) Good educational decisions should be built on a foundation of research. Each study is like a brick, which can be combined with other studies for a sturdy foundation. On top of that foundation, educators will make specific decisions for their specific community of students, aligned with the strengths, weaknesses, and cultures of their systems.

In fact, part of the problem is that we don’t have enough bricks. Many fields seeking to make improvements devote 5 to 15% of their resources to research and development; in education that figure is much lower; in 2003 it was closer to 0.01%

So, do more research. Do lots of high quality experimental research when asking questions about comparative efficacy. Have good conversations about how to interpret findings. Train educators to read educational research and join those conversations. Ensure that important studies are not only published in technical language, but that summaries are published in accessible language for people without sophisticated methodological training. Support educators in making thoughtful, informed judgments about their specific circumstances based on the foundations of research and collaborative inquiry.

That’s the plan. Get to work everybody.

For regular updates, follow me on Twitter at @bjfr and for my publications, C.V., and online portfolio, visit EdTechResearcher.

The opinions expressed in EdTech Researcher are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.

Events

Classroom Technology Webinar How Pandemic Tech Is (and Is Not) Transforming K-12 Schools
The COVID-19 pandemic—and the resulting rise in virtual learning and big investments in digital learning tools— helped educators propel their technology skills to the next level. Teachers have become more adept at using learning management
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Building Teacher Capacity for Social-Emotional Learning
Set goals that support adult well-being and social-emotional learning: register today!


Content provided by Panorama
Jobs October 2021 Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Tiny Wrists in Cuffs: How Police Use Force Against Children
An investigation finds children as young as 6 and a disproportionate amount of Black children have been handled forcibly by police officers.
15 min read
Jhaimarion, 10, reacts as he listens to his mother, Krystal Archie talking with an Associated Press reporter in Chicago on Sept. 23, 2021. Archie’s three children were present when police, on two occasions, just 11 weeks apart, kicked open her front door and tore through their home searching for drug suspects. She’d never heard of the people they were hunting. Her oldest child, Savannah was 14 at the time; her youngest, Jhaimarion, was seven. They were ordered to get down on the floor.
Jhaimarion, 10, reacts as he listens to his mother, Krystal Archie talking with an Associated Press reporter in Chicago on Sept. 23, 2021. Archie’s three children were present when police, on two occasions, just 11 weeks apart, kicked open her front door and tore through their home searching for drug suspects. She’d never heard of the people they were hunting. Her oldest child, Savannah was 14 at the time; her youngest, Jhaimarion, was seven. They were ordered to get down on the floor.
Nam Y. Huh/AP
Education Gunman in 2018 Parkland School Massacre Pleads Guilty
A jury will decide whether Nikolas Cruz will be executed for one of the nation’s deadliest school shootings.
3 min read
Annika Dworet and her husband, Mitch Dworet, wipe away tears as their son's name is read aloud during Marjory Stoneman Douglas High School shooter Nikolas Cruz's guilty plea on all 17 counts of premeditated murder and 17 counts of attempted murder in the 2018 shootings, at the Broward County Courthouse in Fort Lauderdale, Fla. on Wednesday, Oct. 20, 2021. The Dworet's son, Nicholas Dworet, 17, was killed in the massacre.
Annika Dworet and her husband, Mitch Dworet, wipe away tears as their son's name is read aloud during Marjory Stoneman Douglas High School shooter Nikolas Cruz's guilty plea on all 17 counts of premeditated murder and 17 counts of attempted murder in the 2018 shootings, at the Broward County Courthouse in Fort Lauderdale, Fla. on Wednesday, Oct. 20, 2021. The Dworet's son, Nicholas Dworet, 17, was killed in the massacre.
Amy Beth Bennett/South Florida Sun Sentinel via AP
Education Briefly Stated: October 20, 2021
Here's a look at some recent Education Week articles you may have missed.
8 min read
Education Gunman in Parkland School Massacre to Plead Guilty
The gunman who killed 14 students and three staff members at a Florida high school will plead guilty to their murders, his attorneys said.
4 min read
Parkland school shooter Nikolas Cruz is sworn in before pleading guilty, Friday, Oct. 15, 2021, at the Broward County Courthouse in Fort Lauderdale, Fla., on all four criminal counts stemming from his attack on a Broward County jail guard in November 2018, Cruz's lawyers said Friday that he plans to plead guilty to the 2018 massacre at a Parkland high school.
Parkland school shooter Nikolas Cruz is sworn in before pleading guilty, Friday, Oct. 15, 2021, at the Broward County Courthouse in Fort Lauderdale, Fla., on all four criminal counts stemming from his attack on a Broward County jail guard in November 2018, Cruz's lawyers said Friday that he plans to plead guilty to the 2018 massacre at a Parkland high school.
Amy Beth Bennett/South Florida Sun Sentinel via AP