I’ve recently concluded two years as a research fellow at HarvardX. To bring things to a close, last week I held a workshop with course developers looking at the question: What have we learned from the last two years of MOOC research that could help improve the design of courses?
Over the next few days, I’ll release a series of short post on seven general themes from MOOC research that could inform the design of large-scale learning environments in the years ahead.
- MOOC students are diverse, but trend towards auto-didacts (July 2)
- MOOC students value flexibility, but benefit when they engage frequently (Today!)
- The best predictor of persistence and completion is intention, though every activity predicts every other activity
- MOOC students (tell us they) leave because they get busy with other things, but we may be able to help them stay on track
- Students learn more from doing than watching
- Lots of student learning activities are happening beyond our observation: including note-taking, socializing, and using other references
- Improving student learning outcomes will require measuring learning, experimenting with different approaches, and baking research into courses from the beginning
2. MOOC students value flexibility, but benefit when they engage frequently
Students Value Flexibility
One of the first researchers with access to the HarvardX tracking log data was Tommy Mullaney, a fabulous undergraduate who essential did a double major in Social Studies and Computer Science. My kind of guy. in his senior thesis research, Tommy had noted that students appeared to follow very diverse pathways through courses. In this figure, Tommy shows that even among six students who earn a certificate in the first run of HeroesX, they do so in very different, asynchronous, temporal patterns.
The week of the course is on the x-axis and each chapter of the course is on the y-axis. The student in light green starts a little late but then basically keeps up with the course. The student in navy blue starts very late and crams it all in at the end.
It looks like even among students with similar aims, they can get to those aims in diverse ways. A natural experiment that occured in the HeroesX class, about classical conceptions of heroes in Ancient Greece, let us test this observation futher. In the HeroesX natural experiment, two versions of HeroesX were launched one right after the conclusion of the other. The only major change between the two courses was that the first course released content every two weeks, and the second course release all of the content at course launch. The next trio of figures shows the differences observed between these two courses. On the x-axis of each panel is the week in the course, and on the y-axis are the chapters of the course. Bubble size represents the number of students who spent the plurality of their time in a given chapter in a given week.
In the first course, some students stayed on track with the recommended syllabus, but by very early in the course, more than half of students engaging with content in each week are not “on track.” In the second course, virtually no students are ever on track (excepting the few hundred students from the College who were required to stay on track). As far as we could discern, there were no negative consequences in student performance from releasing all content at once--both courses had similar rates of persistence, participation, and completion. We argue, however, that the second HeroesX course showed a “revealed preference” for flexibility: when students were offered more flexibility in the timing of completing a course, they took advantage.
There was no harm to releasing all content at once, and students, through their actions, seem to prefer it.
Students Benefit from Spaced Study Sessions and Regular Course Engagement
While students show a preference for flexibility, they also appear to benefit from consistent interaction with courses. One finding from our study of HeroesX is that one of the strongest predictors of course success, stronger than staying on track, is returning to the course each week. The following taxonomy of fitted logistic regression models shows that the proportion of weeks that a student is active in a course is among the strongest predictors of course completion.
This finding was reinforced in a second study examining how students allocate their time in courses. Yoshuke Miyamoto, a student in the School of Engineering and Applied Sciences, examined activity data from courses and found that both total time on task and the number of distinct study sessions that a user conducted predicted course completion. However, Yoshuke and co-authors found that controlling for total time, students with more study sessions outperformed students who concentrated their studies into fewer sessions. This is generally consistent with the idea from cognitive science of “spaced practice,” for a given amount of time, students usually benefits from breaking that time up into multiple sessions. Even when looking within individual students who took two courses, students perform better in courses when they space their study more widely.
The figures below are a little tricky to follow, though explained in the paper. The top left shows that both total time and session number correlate with earning a certificate in a course. The top right shows that at every level of total time, students with more widely spaced sessions outperform students with more massed practice. The bottom right shows that when you look at students who took two courses, they were more likely to earn a certificate in the course where they spaced their study time over more sessions.
From these studies, we can suggest two takeaways. These aren’t hard and fast rules, but guidelines from early research. Students appreciate flexibility, but they benefit from spaced, regular engagement in courses. In the aggregate, there don’t seem to be any harms from releasing all of the content from a course right in the beginning. Some course instructors space the release of their content to “keep students moving through the material as a group.” But as Tommy’s research shows, even this doesn’t necessarily work. Lots of students fall behind, and move along at their own pace (more strict deadlines might remediate this).
MOOC course designers might consider making content available at the beginning of each course, but they should also consider developing routines and incentives--bonus content, office hours, extra material, email messages--that encourage students to return regularly to classes. Over both of these studies, we see evidence that regardless of exactly how students make their way through courses, returning regularly to the course is a strong predictor of completion.
For regular updates, follow me on Twitter at @bjfr and for my publications, C.V., and online portfolio, visit EdTechResearcher.