MOOCs Revisited

Despite this month’s title, I have refrained from writing about MOOCs, Massive Open Online Courses, in this column before now. The initial burst of interest always seemed overdone to me. Now that the enthusiasm has waned, we are beginning to see the emergence of meaningful information about when and how they can be useful.

As I argued in my co-authored piece in the AMS Notices, Musings on MOOCs [1], they do seem to hold promise as a source of supplementary material that enables flipped classes, supplementary instruction, alternate approaches, or opportunities for exploring topics that extend beyond the course syllabus. Two questions immediately emerge: How hard is it to take advantage of these materials? Do students actually benefit?

This past summer, Rebecca Griffiths and her team at Ithaka S+R, an academic consulting and research service, released Interactive Online Learning on Campus [2], its study of the use of hybrid MOOCs within the University System of Maryland. Hybrid MOOCs are face-to-face classes for which instructors draw on online courses, in this case developed by Coursera or the Open Learning Initiative, to supplement their own instruction. Griffiths et al. conducted seven side-by-side studies, direct comparisons of the same courses taught with and without these online supplemental materials, and ten case study investigations of courses that were only taught with supplemental materials derived from MOOCs. The side-by-side comparisons are of greatest interest to me because of the usefulness of direct comparisons and because these courses included STEM subjects: three sections of introductory biology and one each of pre-calculus, statistics, and computer science, plus a course in communications.

In answer to the first question—How hard is it to incorporate material from these online courses?—the answer is hard, but probably will become easier when repeated. Griffiths et al. found that self-reported instructor time spent selecting the materials and preparing how they would incorporate them into their hybrid course had a median of 68 and a mean of 144 hours, roughly two to four weeks. The variation was tremendous, from only one full-time week to an entire summer. Most of this is, almost certainly, a one-time investment. For some hybrid courses, face time was reduced by as much as 50%. For others, there was no reduction in face time. Once the start-up time is invested, there appears to be potential for some time—and therefore cost—savings, although it would be modest at best.

The biggest question is whether this improved student outcomes. For the most part in the side-by-side comparisons, there was little difference in pass rates or student performance on a common post test. One biology section had a substantially and significantly better pass rate for the hybrid course, but the other two hybrid biology sections had slightly lower (though not statistically significantly lower) pass rates than the sections with which they were paired. With two exceptions, results on the post tests were indistinguishable between hybrid and non-hybrid courses. Those exceptions were the biology section with the high pass rate and the pre-calculus class. In both of these cases, the hybrid classes posted substantially higher post test results that were significant at p < 0.001.

Griffiths et al. also looked at pass rates and post test results by key subgroups involving race, gender, socio-economic status, and SAT scores. Averaging across all of the side-by-side comparisons for each of the subgroups, pass rates and post test results improved with the hybrid courses, although none of the pass rate differences were significant at p < 0.01. However, several of the post test comparisons were. Although all students saw gains from the hybrid approach, the greatest gains were to White and Asian students, females, and those with parental income between $50,000 and $100,000, at least one parent with a BA, and combined SAT scores above 1000.

There were other factors that came into play. Students preferred the traditional course format and felt that they learned more from it, although they did prefer to do their homework assignments, quizzes, and exams online. Technical glitches did arise in the hybrid courses and may have been a factor in student dislike of online instruction.

One of the most intriguing differences was in how much time students spent on the course outside of classtime. Here the effects were in opposite directions for: under-represented minorities (URM) versus non-URM, low income versus high income, first generation college student versus not first generation, SAT scores below 1000 versus SAT score above 1000. In all cases, the first group saw a decrease in time spent outside of class with the hybrid course, the second group an increase. It may be that online materials allowed students in these traditionally under-represented subgroups to make more efficient use of their time, thus needing to spend less time. But that is a hypothesis that would require study. On its face, this distinction is troubling.

[1] Bonfert-Taylor, P., Bressoud, D.M., and Diamond, H. 2014. Musings on MOOCs. Notices of the AMS. Vol 61, pp. 69–71. www.ams.org/notices/201401/rnoti-p69.pdf

[2] Griffiths, R., Chingos, M., Mulhern, C., and Spies, R. 2014. Interactive Online Learning on Campus: Testing MOOCs and Other Platforms in Hybrid Formats in the University System of Maryland. New York, NY: Ithaka S+R. www.sr.ithaka.org/research-publications/interactive-online-learning-on-campus