Education Opinion

Finding Rigor in Diverse Online Research Methods

By Justin Reich — May 30, 2012 3 min read
  • Save to favorites
  • Print

I had a great time last week on RadioBoston chatting with Matthew Chingos about his study comparing an online statistics course with face to face course. Matt and his colleagues wanted to know whether the online version of a course had the same effect on student achievement (as measured by passing rates, grades, and standardized test scores) as a fairly traditional intro stats class. To make this comparison, they used a research method called a randomized control trial, where participants (students in this case) volunteer to be randomly assigned to either the regular (control) class or the online (intervention class). Most people are familiar with the idea of randomized control trials from medical research, where patients are given an experimental drug and a placebo.

The major advantage of this method is that, when executed correctly, only one thing differs between the control group and the intervention. In this study, the only difference between the two groups of students was which course they took.

We could imagine more problematic research designs without random assignment. For instance, let’s say that we had all the students at one college take regular intro stats and all the students at another college take online intro stats. We can compare outcomes between the two groups, but the problem is that the two groups differ in lots of ways: they go to different colleges, have different demographic background, have different school cultures, and are different in all kinds of other unobservable ways. When we look at difference in scores, it is hard to tell what is due to the online intervention and what is due to all those other differences.

It’s very hard work to set up randomized experiments, and Matt and his colleagues have my commendation not just for running the experiment, but offering quite a bit of experiment design details and advice in their paper.

All that said, randomized control trials do not have a monopoly on “rigor.” My one strong disagreement with Matt is his characterization, in the radio broadcast, of randomized trials as the only “rigorous” methods in education technology research. That’s the kind of characterization that policy researchers make that just drive the rest of us crazy, and it obscures more than it clarifies.

As I have said above, for ascertaining the efficacy of an intervention (like a new online course) at scale, randomized trials are the way to go. But we need to learn much more than the efficacy of interventions. For instance, at the end of this study, all we know is that the hybrid stats courses achieve the same outcomes as the traditional ones. This study, and other similar randomized trials, are nearly useless for telling us why hybrid courses work (indeed, Matt mentioned during the interview that he hadn’t even examined the entire online course).

We need rigorous anthropological research to learn more about the study and learning habits of students in these online courses. We need rigorous design-based research to take those insights and develop ever more effective learning environments through iterative trial and design. We need rigorous survey research to track how universities are deploying these new online teaching strategies. We need rigorous learning analytics research to identify patterns in course taking behaviors that can inform design.

It’s silly to call randomized control trials the “gold standard” of educational research, when such such studies are only one piece of the puzzle necessary to develop the potential for technology to enhance learning. Moreover, such languages marginalizes the important work that is done, especially by qualitative researchers, that is vital to the field of education technology.

I’m happy to consider random trials the gold standard of comparative efficacy research. But comparative efficacy research is not the sum total of work that is needed; we need diverse scholars examining diverse subjects through diverse methods, and as a field we need to make such that all of it is rigorous.

For regular updates, follow me on Twitter at @bjfr and for my papers, presentations and so forth, visit EdTechResearcher.

The opinions expressed in EdTech Researcher are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.

Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
6 Key Trends in Teaching and Learning
As we enter the third school year affected by the pandemic—and a return to the classroom for many—we come better prepared, but questions remain. How will the last year impact teaching and learning this school
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Ensuring Continuity of Learning: How to Prepare for the Next Disruption
Across the country, K-12 schools and districts are, again, considering how to ensure effective continuity of learning in the face of emerging COVID variants, politicized debates, and more. Learn from Alexandria City Public Schools superintendent
Content provided by Class
Teaching Profession Live Online Discussion What Have We Learned From Teachers During the Pandemic?
University of California, Santa Cruz, researcher Lora Bartlett and her colleagues spent months studying how the pandemic affected classroom teachers. We will discuss the takeaways from her research not only for teachers, but also for

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Schools Get the Brunt of Latest COVID Wave in South Carolina
In the past few weeks, South Carolina has set records for COVID-19 hospitalizations and new cases have approached peak levels of last winter.
4 min read
Two Camden Elementary School students in masks listen as South Carolina Gov. Henry McMaster talks about steps the school is taking to fight COVID-19, Wednesday, Sept. 15, 2021, in Camden, S.C. McMaster has adamantly and repeatedly come out against requiring masks in schools even as the average number of daily COVID-19 cases in the state has risen since early June. (AP Photo/Jeffrey Collins)
Education More States Are Requiring Schools to Teach Native American History and Culture
Advocates say their efforts have gained some momentum with the nation’s reckoning over racial injustice since the killing of George Floyd.
3 min read
A dancer participates in an intertribal dance at Schemitzun on the Mashantucket Pequot Reservation in Mashantucket, Conn., Saturday, Aug. 28, 2021. Connecticut and a handful of other states have recently decided to mandate students be taught about Native American culture and history. (AP Photo/Jessica Hill)
Education Judge's Temporary Order Allows Iowa Schools to Mandate Masks
A federal judge ordered the state to immediately halt enforcement of a law that prevents school boards from ordering masks to be worn.
4 min read
Iowa Gov. Kim Reynolds speaks to reporters following a news conference, Thursday, Aug. 19, 2021, in West Des Moines, Iowa. Reynolds lashed out at President Joe Biden Thursday after he ordered his education secretary to explore possible legal action against states that have blocked school mask mandates and other public health measures meant to protect students against COVID-19. Reynolds, a Republican, has signed a bill into law that prohibits school officials from requiring masks, raising concerns as delta variant virus cases climb across the state and schools resume classes soon. (AP Photo/Charlie Neibergall)
Education Hurricane Ida Deals New Blow to Louisiana Schools Struggling to Reopen
The opening of the school year offered teachers a chance to fully assess the pandemic's effects, only to have students forced out again.
8 min read
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021. Louisiana students, who were back in class after a year and a half of COVID-19 disruptions kept many of them at home, are now missing school again after Hurricane Ida. A quarter-million public school students statewide have no school to report to, though top educators are promising a return is, at most, weeks away, not months.
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021.
John Locher/AP