Education Opinion

Four Types of MOOC Research

By Justin Reich — December 18, 2013 5 min read
  • Save to favorites
  • Print

From the recent MOOC Research Initiative Conference (#mri13) to the data-driven education workshop at NIPS, more research based on courses run by edX, Coursera and Udacity will be coming down the ‘Pike. (We say “coming down the ‘Pike” in Massachusetts, because we have this big turnpike that transects the state. Do other people say “coming down the ‘Pike”? Do you say “coming down the ‘Pike wicked fast?”) Here are four kinds of research to look out for:

  1. Fishing Expeditions in the Exhaust
  2. Experiments in the Periphery
  3. Anthropology in the Field
  4. Design Research in the Core

[[UPDATE (2/2/14): Here’s a graphic that I’ve been using to illustrate these ideas.]]

Fishing Expeditions in the Exhaust

Computer scientists sometimes refer to the tracking logs that come out of software as the “exhaust.”

Much of the early research that we are seeing on MOOCs (my own forthcoming included) is simply the reporting of observations from these clickstream data. Institutions create a bunch of courses, educators run those courses, researchers go in afterwards to see if anything is interesting. Since the educators and researchers are not often working closely in tandem, many courses are not built with carefully designed research questions and methods for answering those questions prepared in advance. So researchers are going fishing.

As an early stage research approach, this seems perfectly fine. I think we may get some useful data from a policy perspective--for instance, examining how closely the rhetoric of “democratizing education” lines up with the demographic characteristics of actual MOOC participants, or getting a sense of how much interest there is in MOOCs in different populations of higher educations non-conusmers.

I also think that we’re going to have some interesting things pop up in the data that inspire more carefully designed research experiments about teaching and learning. If we see interesting correlations between course structure or teaching approach and outcomes, we can start probing more intentionally in those spaces.

The great risk of this approach is that we’re looking at what we have rather than what we care most about. It’s the drunkard looking for his keys under the lampost because that where the light is. We risk developing fixations on clickstream behavior, persistence, attrition, certification--which are all imperfect proxies for learning, which is what we actually care about.

So, this is good stuff for the early stages, but we’ll need to build beyond it.

Experiments in the Periphery

There are experiments happening throughout xMOOCs, but again because of a disconnect between researchers and course teams, many of these experiments are happening at the periphery rather than within the instructional core. This disconnect is totally understandable; it’s a massive effort to build any kind of course, but as a result many experiments are being designed in ways that could be implemented without any attention from the faculty or course teams. Look for priming statements in surveys and courseware, follow up emails to randomly selected teams, and other kinds of experiments that don’t actually get into the core of the course.

I suspect that we may find some nifty nudges, primes, and user experience enhancements here, and lots of small effect sizes can add up. But we shouldn’t expect big breakthroughs from innovations outside of the core.

Observations from the Field

So we have trillions of cells of data about what people are clicking, and not so much data about what’s changing in people’s heads. We have a zillion time stamped logs, and not a clear sense of what’s happening in between the timestamps.

I have heard very little about qualitative research in the MOOC field (I know of one interview study happening in one HarvardX course), and I think there is a wide open space for people to do some really informative work here. Where are people taking courses? Who are they talking with about them? Are there typical patterns of studying or engaging? Are there importnat learning activities not being captured by the clickstream?

I’m a big believer in methodological pluralism, and as much quantitative data as gets produced by these online platforms, making sense of that data is going to require lots of insights from human beings: through interviews, field observations, screencasts, and content analyses of forums and other peer-production environments. danah boyd’s and Kate Crawford’s Six Provocations for Big Data is helpful for thinking through this, and Mimi Ito et. al. ethnography of teens and interest based learning, Hanging Out, Messing Around and Geeking Out is a great model.

There are human learners taking these courses, and their voices matter; it’s not just user accounts.

Design Research in the Core

The research that I’m most excited about, and the research that is hardest to pull together, will be design research, where faculty, course development teams, and researchers work closely together to design courses that have important and interesting questions baked into the architecture of the course, and where the courses are carefully instrumented to be able to capture data that will answer these important questions. The courses will be designed with multiple opportunities for assessment, so that we don’t have to wait an entire course run to start iterating designs and refine hypotheses.

Research in these environments is a forethought not an afterthought.Course faculty will have clearly defined learning goals and a theory of action for how their instructional methods will help students meet these goals. The key research efforts will take place in the heart of that theory of action. Rather than experiments that tinker with parts of the course that instructors aren’t paying attention to, they’ll be experiments that manipulate the parts that faculty care most about.

So as you see the research unfolding from these efforts, see if this taxonomy helps you think about different approaches that researchers are taking. I think all of these modes of research have the potential to make important contributions. That said, I suspect the most impactful research will be anthropological work in the midst of courses and design work that begins long before courses launch, rather than approaches that comb through tracking logs after all has been said and done.

For regular updates, follow me on Twitter at @bjfr and for my publications, C.V., and online portfolio, visit EdTechResearcher.

The opinions expressed in EdTech Researcher are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Data Webinar
Education Insights with Actionable Data to Create More Personalized Engagement
The world has changed during this time of pandemic learning, and there is a new challenge faced in education regarding how we effectively utilize the data now available to educators and leaders. In this session
Content provided by Microsoft
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Accelerate Learning with Project-Based Learning
Earlier this year, the George Lucas Educational Foundation released four new studies highlighting how project-based learning (PBL) helps accelerate student learning—across age groups, multiple disciplines, and different socio-economic statuses. With this year’s emphasis on unfinished
Content provided by SmartLab Learning
School & District Management Live Online Discussion Principal Overload: How to Manage Anxiety, Stress, and Tough Decisions
According to recent surveys, more than 40 percent of principals are considering leaving their jobs. With the pandemic, running a school building has become even more complicated, and principals' workloads continue to grow. If we

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education California Makes Ethnic Studies a High School Requirement
California is among the first in the nation to require students to take a course in ethnic studies to get a diploma starting in 2029-30.
4 min read
FILE - In this Jan. 22, 2020, file photo, Democratic Assembly members, from left, James Ramos, Chris Holden Jose Medina, and Rudy Salas, Jr., right, huddle during an Assembly session in Sacramento, Calif. Medina's bill to make ethnic studies a high school requirement was signed into law by California Gov. Gavin Newsom on Friday, Oct. 8, 2021. (AP Photo/Rich Pedroncelli, File)
Education California Requires Free Menstrual Products in Public Schools
The move comes as women’s rights advocates push nationwide for affordable access to pads, tampons, and other items.
1 min read
Tammy Compton restocks tampons at Compton's Market, in Sacramento, Calif., on June 22, 2016. California public schools and colleges must stock their restrooms with free menstrual products under a new law signed by Gov. Gavin Newsom, Friday, Oct. 8, 2021.
Tammy Compton restocks tampons at Compton's Market, in Sacramento, Calif., on June 22, 2016. California public schools and colleges must stock their restrooms with free menstrual products under a new law signed by Gov. Gavin Newsom, Friday, Oct. 8, 2021.
Rich Pedroncelli/AP
Education Florida to Dock School District Salaries for Requiring Masks
Florida is set to dock salaries and withhold funding from local school districts that defied Gov. Ron DeSantis' ban on mask mandates.
2 min read
Florida Gov. Ron DeSantis speaks, Tuesday, Sept. 14, 2021, at the Doral Academy Preparatory School in Doral, Fla.
Florida Gov. Ron DeSantis speaks, Tuesday, Sept. 14, 2021, at the Doral Academy Preparatory School in Doral, Fla.
Wilfredo Lee/AP
Education More Than 120,000 U.S. Kids Had Caregivers Die During Pandemic
The toll has been far greater among Black and Hispanic Americans, a new study suggests.
3 min read
FILE - In this Thursday, Sept. 2, 2021 file photo, a funeral director arranges flowers on a casket before a service in Tampa, Fla. According to a study published Thursday, Oct. 7, 2021, by the medical journal Pediatrics, the number of U.S. children orphaned during the COVID-19 pandemic may be larger than previously estimated, and the toll has been far greater among Black and Hispanic Americans. (AP Photo/Chris O'Meara, File)