The potential of MOOCs — massive open online courses — to provide educational access to all has generated significant interest and no shortage of hype. New York Times columnist Thomas Friedman described them as an educational “revolution,” while the Washington Post suggested they could provide “elite education for the masses.” Some businesses see the potential for bridging the “skills gap” between workers and employers, while universities view online offerings as a way to maximize the value of their branded pedagogical content and potentially shore up revenue.

There has been much theorizing, but real empirical evidence is starting to come in. Studies on early experiences with MIT’s and Harvard’s EdX platforms have now been released. In a 2014 working paper, “HarvardX and MITx: The First Year of Open Online Courses, Fall 2012-Summer 2013,” researchers note that the “registrants are not ‘students’ in a conventional sense, and they and their behavior differ from traditional students in K-12 and post-secondary institutions.” The authors of that research paper, a summary of a cluster of related studies, conclude both that “there will be no grand unifying theory of MOOCs” and that “open online courses are neither useless nor the salvation of higher-education.” A 2013 study by Pennsylvania State University examined the behavior of one million enrollees in 16 Coursera offerings, and found that completion rates averaged just 4%. Rates were higher for courses with fewer assignments, but even then only reached about 6%.

A May 2014 report from Teachers College, Columbia University, examines the experiences of faculty, staff and administrators at 82 institutions to assess “expectations and realities” nationwide.

Up to now, however, the success of MOOCs in fulfilling the hopes — and earning back the money — invested in them has been decidedly mixed. Other questions include the viability of pay models: The University of California developed UC Online and spent $4.3 million to promote it, but the early results have been distinctly underwhelming. In July, San Jose State “paused” Udacity after six months of experimentation, and concern has risen that MOOCs may ultimately prove to be passing fad.

The scale and open nature of MOOCs definitely create challenges — the courses examined in the Penn State study had enrolled anywhere from 13,000 to 110,000 students. Other problems can include discomfort with user interfaces and procedures, questions about pedagogical rigor, and technical hitches. A Coursera class, “Fundamentals of Online Education: Planning and Application,” was cancelled in February 2013 after a series of design flaws, issues with multi-author editing and assorted glitches. Attempts to rethink how MOOCs work are ongoing, even if efforts to increase completion rates have so far stumbled.

And even for students who do finish courses, the actual market value of a certificate of completion remains to be determined. For many, improving job skills may play a role: A 2012 survey indicated that 41% of those studying online were working professionals, while 31% were undergraduates and graduates. Nearly 40% of respondents reported enrolling because of casual subject interest. Some educators see MOOCs as the next step in the development of technology-enhanced learning practices, including blended learning and online learning portals such as Khan Academy, YouTubeU and TedEd.

For a critical examination of these trends, and their intersection with business interests, see this lecture by Justin Reich, fellow at Harvard’s Berkman Center for Internet & Society, titled “Personalized Learning, Backpacks Full of Cash, Rockstar Teachers, and MOOC Madness.”

In “The Troubled Future of Colleges and Universities,” Harvard’s Gary King and Maya Sen of the University of Rochester explore the state of higher education in the U.S., including “no touch” options such as online courses. Digital culture theorist and NYU professor Clay Shirky has written that it’s worth bearing in mind the huge shifts that rapidly upended the music, books and newspaper industries. “In the academy, we lecture other people every day about learning from history,” Shirky noted. “Now its our turn, and the risk is that we’ll be the last to know that the world has changed, because we can’t imagine — really cannot imagine — that story we tell ourselves about ourselves could start to fail.”

Below are recent government publications and scholarly articles that explore the potential and effectiveness of MOOCs as well as their impact on traditional education methods:

—————————

“The Life Cycle of a Million MOOC Users”
Perna, Laura; Ruby, Alan; Boruch, Robert; Wang, Nicole; Scull, Janie; Evans, Chad; Ahmad, Seher. University of Pennsylvania, December 2013.

Summary: “The Penn GSE study analyzed the movement of a million users through sixteen Coursera courses offered by the University of Pennsylvania from June 2012 to June 2013. The project aimed to identify key transition points for users — such as when users enter and leave courses — as well as when and how users participate in the courses. The study also considered how engagement and persistence vary based on various course characteristics. The courses studied ranged widely in topic, target audience, length of study, instructional time, use of quizzes and assignment of homework, and other dimensions. Preliminary findings: Course completion rates are very low, averaging 4% across all courses and ranging from 2% to 14% depending on the course and measurement of completion. Across the 16 courses, completion rates are somewhat higher, on average, for courses with lower workloads for students and fewer homework assignments (about 6% versus 2.5%). Variations in completion rates based on other course characteristics (e.g., course length, availability of live chat) were not statistically significant. Across all courses, about half of those who registered viewed at least one lecture within their selected course. The share of registrants viewing at least one lecture ranged from [27% to 68%].”

 

“Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies”
2010 Technical Report. U.S. Department of Education, Washington, D.C.

Excerpt: “A systematic search of the research literature from 1996 through July 2008 identified more than a thousand empirical studies of online learning…. The meta-analysis found that, on average, students in online learning conditions performed modestly better than those receiving face-to-face instruction. The difference between student outcomes for online and face-to-face classes — measured as the difference between treatment and control means, divided by the pooled standard deviation — was larger in those studies contrasting conditions that blended elements of online and face-to-face instruction with conditions taught entirely face-to-face. Analysts noted that these blended conditions often included additional learning time and instructional elements not received by students in control conditions. This finding suggests that the positive effects associated with blended learning should not be attributed to the media, per se. An unexpected finding was the small number of rigorous published studies contrasting online and face-to-face learning conditions for K-12 students. In light of this small corpus, caution is required in generalizing to the K-12 population because the results are derived for the most part from studies in other settings (e.g., medical training, higher education).”

 

“State U Online”
New American Foundation, April 23, 2013.

Findings: “At a time when educational credentials are more important to individual and collective prosperity than ever before, students need online courses and degree programs that are effective, affordable, and grounded in public values. This report…finds that state institutions have tremendous untapped potential to grow enrollment, increase revenues, contribute to economic development, and fulfill their historical missions—if they adopt a series of policies that a few innovative states and public higher-educationsystems have already pioneered.”

 

“Interpolated Memory Tests Reduce Mind Wandering and Improve Learning of Online Lectures”
Proceedings of the National Academy of Sciences, April 1, 2013.

Findings: “Recent research in cognitive psychology suggests that, when implemented appropriately, tests can be used to significantly enhance learning. The present results demonstrate one such function of testing and highlight the specific cognitive mechanism by which testing can facilitate learning. In particular, testing can be used to help students sustain attention to lecture content in a manner that discourages task-irrelevant (mind wandering) and encourages task-relevant (note taking) activities, and hence improves learning.”

 

“Effectiveness of Fully Online Courses for College Students: Response to a Department of Education Meta-Analysis”
Community College Research Center, Teachers College, Columbia University. June 2011.

Excerpt: “Proponents of postsecondary online education were buoyed by a 2009 meta-analysis sponsored by the U.S. Department of Education suggesting that student learning outcomes in online courses are superior to those in traditional face-to-face courses. This paper demonstrates that this finding does not hold for the studies included in the meta-analysis that pertain to fully online, semester-length college courses; among these studies, there is no trend in favor of the online course mode. Furthermore, the authors argue that because these studies examine courses that were taken by relatively well-prepared university students, their results may not generalize to traditionally underserved populations. The authors conclude that while advocates argue that online learning is a promising means to increase access and improve student progression through college, the Department of Education report does not provide evidence that fully online college courses produce superior learning outcomes, particularly among low-income and academically underprepared students.”

 

“A Pedagogy of Abundance or a Pedagogy to Support Human Beings? Participant Support on Massive Open Online Courses”
International Review of Research in Open and Distance Learning, November 2011, Vol. 12, No. 7, 74-93.

Abstract: “This paper examines how emergent technologies could influence the design of learning environments. It will pay particular attention o the roles of educators and learners in creating networked learning experiences on massive open online courses (MOOCs). The research shows that it is possible to move from a pedagogy of abundance to a pedagogy that supports human beings in their learning through the active creation of resources and learning places by both the learners and course facilitators. This pedagogy is based on the building of connections, collaborations, and the exchange of resources between people, the building of a community of learners, and the harnessing of information flows on networks. This resonates with the notion of emergent learning as learning in which actors and systems co-evolve within a MOOC and where the level of presence of actors on the MOOC influences learning outcomes.”

 

“How the Embrace of MOOCs Could Hurt Middle America”
Chronicle of Higher Education, October 2012, Vol. 59, No. 6, B22-B23.

Abstract: “The article discusses massive open online courses (MOOCs) and argues that they will ultimately harm students, particularly lower-income ones, by turning face-to-face education into a luxury rather than a mainstream method of delivering learning. The attractiveness of MOOCs and other types of online education due to their cost-effectiveness is examined, and the value of in-person relationships between teachers and students is commented on. MOOCs offered by former Stanford University professor Sebastian Thrun are also examined.”

 

“Why Some Distance Education Programs Fail While Others Succeed in a Global Environment”
The Internet and Higher Education, June 2010, Vol. 13, No. 3, 141-147. doi: 10.1016/j.iheduc.2009.07.001.

Abstract: “Many universities increase their recruiting efforts to reach a larger and more diverse audience. Some universities also extend their reach with cross-border initiatives and seek international students in order to promote enrollment growth and global learning. The economic potential of distance education and academic globalization has attracted numerous higher education providers, many of which operate on a for-profit basis. The result is an increase in competition for students, which leads to added pressure on universities to control costs and rising tuition. Those online programs unable to successfully adapt to this competitive environment are at risk of failing. This article draws from the research literature and US Securities and Exchange Commission (SEC) filings to examine seven important factors that help determine the success or failure of online programs. These factors are planning, marketing and recruitment, financial management, quality assurance, student retention, faculty development, and online course design and pedagogy.”

 

“Can MOOCs and Existing E-Learning Efficiency Paradigms Help Reduce College Costs?”
George Mason University School of Public Policy working paper, June 2012.

Abstract: “E-Learning enrollments in post-secondary education are growing significantly but the annual cost of tuition continues to increase more than inflation. Does this mean that E-Learning cannot reduce costs? After describing some salient details of the tuition cost problem, this article examines four paradigms that have the potential to reduce tuition significantly over the long term, perhaps by half. First, it reviews the potentially game-changing effects of free Massively Open On-line Courses (MOOCs) and the new MIT/Harvard Ed X project. Both are very new and eventually target as many as a billion on-line students. Second, it describes the underutilized but highly scalable National Center for Academic Transformation’s NCAT) course redesign paradigm, capable of reducing aggregate administrative and teaching costs by billions annually, and examines problems associated with its wider deployment. Third, Western Governors University’s competency-based approach is presented as another way of drastically reducing labor costs while increasing availability of college courses to low income students. Finally, the export/import approach is introduced, in which MOOCs for credit become available (at a fee) beyond the offering institutions, thereby setting up a disruptive, but potentially highly economical reduction in faculty and administrative labor expense.”

 

“The Retention Fallacy”
National Center for Academic Transformation, July 2010.

Findings: “Sometimes completion rates [for redesigned online courses] will go down for good reasons. We have frequently experienced the phenomenon of improved student learning outcomes supported by clear assessment data coupled with decreased completion rates. This phenomenon is typically due to prior grade inflation… There is a relationship among section size, pass rate improvements and possible cost reduction. The larger the section size, the higher the pass rate improvement must be in order to reduce costs. The smaller the section size, the lower the pass rate improvement must be in order to reduce costs. However, reductions in smaller section sizes result in lower amounts of the cost reduction…. even if the number of students enrolled in the course is large and you take the necessary steps to reduce the number of sections offered, the impact on cost reduction may be quite small.”

 

“Understanding the Implications of Online Learning for Educational Productivity”
Department of Education, Office of Educational Technology, January 2012.

Excerpt: “While rigorously researched models are lacking, the review of the available literature suggested nine applications of online learning that are seen as possible pathways to improved productivity: 1) Broadening access in ways that dramatically reduce the cost of providing access to quality educational resources and experiences, particularly for students in remote locations or other situations where challenges such as low student enrollments make the traditional school model impractical; 2) Engaging students in active learning with instructional materials and access to a wealth of resources that can facilitate the adoption of research-based principles and best practices from the learning sciences, an application that might improve student outcomes without substantially increasing costs; 3) Individualizing and differentiating instruction based on student performance on diagnostic assessments and preferred pace of learning, thereby improving the efficiency with which students move through a learning progression; 4) Personalizing learning by building on student interests, which can result in increased student motivation, time on task and ultimately better learning outcomes; 5) Making better use of teacher and student time by automating routine tasks and enabling teacher time to focus on high-value activities; 6) Increasing the rate of student learning by increasing motivation and helping students grasp concepts and demonstrate competency more efficiently; 7) Reducing school-based facilities costs by leveraging home and community spaces in addition to traditional school buildings; 8) Reducing salary costs by transferring some educational activities to computers, by increasing teacher-student ratios or by otherwise redesigning processes that allow for more effective use of teacher time; and 9) Realizing opportunities for economies of scale through reuse of materials and their large-scale distribution.”

 

“Emergent Learning and Learning Ecologies in Web 2.0”
International Review of Research in Open and Distance Learning, March 2011, Vol. 12, No. 3, 40-59. doi: 10515/sy5hh6ck2.

Findings: “In this paper, we have described the unprecedented affordances that Web 2.0 offers for interaction and communication and for emergent learning, as well as some of the substantial challenges in realising this potential in education… Such a framework would be based first on the technical or infrastructural conditions for emergent learning. ICT is fast morphing into the social software of Web 2.0 and the augmented reality of cloud-based Web 3.0. ICT has changed beyond recognition, providing global open access at extremely low cost, for not only consuming, producing, and distributing texts and artefacts but for interaction, communication, and networking. Secondly, emergence requires new institutional and social memes and structures. Some innovative legal frameworks which are already in place provide clear protocols and resources for collaboration and sharing, notably Open Source licences for collaborative software and Creative Commons licences for collaborative and shared content. Many free (mostly advertising-driven) platforms are also in place, from Google to a range of social software and cloud-based “apps” — downloadable applications. And thirdly, there is a need for a shift from a monolithic learning environment in which everything must be controlled and predictable to a more pluralistic learning ecology in which both prescriptive and emergent application domains and modes of learning have their place, and in which it is possible to celebrate the unpredictable. This requires quite a different mindset, in which there is a role for safe/fail as well as fail-safe management, a role for resilience as well as robustness, and a balance between pro-spective and retro-spective sense-making of teaching and learning (see section on Managing Emergence).”

 

“What 40 Years of Research Says About the Impact of Technology on Learning: A Second-Order Meta-Analysis and Validation Study”
Review of Educational Research, March 2011, Vol. 81, No. 1, 4-28. doi:10.3102/0034654310393361.

Excerpt: “The average student in a classroom where technology is used will perform 12 percentile points higher than the average student in the traditional setting that does not use technology to enhance the learning process. It is important to note that these average effects must be interpreted cautiously because of the wide variability that surrounds them. We interpret this to mean that other factors, not identified in previous meta-analyses or in this summary, may account for this variability. We support Clark’s (1983, 1994) view that technology serves at the pleasure of instructional design, pedagogical approaches, and teacher practices and generally agree with the view of Ross, Morrison, and Lowther (2010) that “educational technology is not a homogeneous ‘intervention’ but a broad variety of modalities, tools, and strategies for learning. Its effectiveness, therefore, depends on how well it helps teachers and students achieve the desired instructional goals” (p. 19). Thus, it is arguable that it is aspects of the goals of instruction, pedagogy, teacher effectiveness, subject matter, age level, fidelity of technology implementation, and possibly other factors that may represent more powerful influences on effect sizes than the nature of the technology intervention. It is incumbent on future researchers and primary meta-analyses to help sort out these nuances, so that computers will be used as effectively as possible to support the aims of instruction… there is the suggestion that one of technology’s main strengths may lie in supporting students’ efforts to achieve rather than acting as a tool for delivering content.”

 

“Class Differences: Online Education in the United States, 2010”
Babson Survey Research Group, Babson College, November 2010.

Excerpt: “The first study in this series found that a majority of chief academic officers rated the learning outcomes for online education “as good as or better” than those for face-to-face instruction, but a sizable minority considered online to be inferior. Each subsequent year’s report displayed similar results. Do academic leaders still hold the same opinion, given the rapid growth in the numbers of online students?… The 2010 results show some small improvements in the perception of the relative quality of online instruction as compared to face-to-face. In the first report of this series in 2003, 57% of academic leaders rated the learning outcomes in online education as the same or superior to those in face-to-face. That number is now 66%, a small but noteworthy increase. Over three-quarters of academic leaders at public institutions report that online is as good as or better than face-to-face instruction (compared to only 55.4% of private nonprofits and 67.0% of for-profits.)”

 

“Conceptions of and Approaches to Learning Through Online Peer Assessment”
Learning and Instruction, February 2010,Vol. 20, No. 1, 72-83. doi: 10.1016/j.learninstruc.2009.01.003.

Abstract: “The present study investigated junior college students’ conceptions of and approaches to learning via online peer assessment (PA) using a phenomenographic approach. Participants were 163 college students. Students were asked to accomplish a given learning task via an online PA system. Of the participants, 62 were interviewed after the activity. The interviews revealed hierarchically related and qualitatively different categories of conceptions and approaches to learning via online PA. The main and achieved levels of conceptions of and approaches to learning were determined. The results showed that, within each level, conceptions emphasizing a fragmented and cohesive learning tended to be associated with approaches focusing on surface and deep learning, respectively. In addition, students with cohesive learning conceptions and deep learning approaches were likely to make greater progress in the early stages of online PA activity. The present study finally found that approaches to learning via online PA were less related to the learning outcomes than conceptions of learning.”

 

“Current Status of Research on Online Learning in Postsecondary Education”
Ithaka S&R research group, May 2012.

Excerpt: “Students who took all or part of their class online performed modestly better, on average, than did those taking the course through traditional, face-to-face instruction. Although both purely online instruction and hybrid forms of instruction had positive effects, the hybrid mode had a larger advantage — which is hardly surprising since hybrid forms of instruction involve the commitment of more resources. Time-on-task was found to be a significant variable, and effect sizes were larger when a new instructional approach was tried, as opposed to simply varying the medium of delivery. Significantly, the meta-analysis found that online learning can be enhanced by giving learners control of their interactions with delivery mechanisms and prompting learner reflection.”

 

“Motivation in Online Learning: Testing a Model of Self-Determination Theory”
Computers in Human Behavior, July 2010, Vol. 26, No. 4, 741-752. doi: 10.1016/j.chb.2010.01.011.

Findings: “This study found a mediating effect of need satisfaction between contextual support and motivation/self-determination. In other words, supports of autonomy and competency positively affected online students’ perceived autonomy, relatedness, and competency, the satisfaction of the three basic needs. Students’ need satisfaction, in turn, positively affected online students’ self-determination… haphazard and aimless supports without addressing students’ needs are likely to lead to adverse — even worse than “no effects” — outcomes. It is through the enhancement of students’ perceptions of autonomy, relatedness, and competency that makes contextual support effective and meaningful to online students.”

 

”Student Interactions in Online Discussion Forum: Empirical Research from ‘Media Richness Theory’ Perspective”
Journal of Interactive Online Learning, Spring 2010, Vol. 9, No. 1.

Findings: “The present study is significant in its attempts to draw on different theories to investigate the antecedents and outcomes of using ODFs along with classroom lectures. The findings indicate that facilitating discourse, reflective thinking, assessment and connectedness contribute to interactions in ODFs. From the practical perspective, the present study suggests that using multiple mediums of instruction enriches the communication context and leads to enhanced learning. Some limitations of this study may warrant further consideration in future research. First, the regression results must be interpreted with caution. It is advisable to use more powerful statistical approaches like structural equation modeling or experimental design to enable the testing of relationships among the constructs. Second, instructional tools other than ODF could be investigated and compared with classroom lectures for their perceived effectiveness in meeting the learning objectives. Further, qualitative methods, as used in previous research are needed to analyze the content of the postings and make inferences about the level of reflective thinking, and sense of community among the forum members.”

 

“Anxiety and Attitude of Graduate Students in On-Campus vs. Online Statistics Courses”
Journal of Statistics Education, 2010, Vol. 18, No. 1.

Abstract: “This study compared levels of statistics anxiety and attitude toward statistics for graduate students in on-campus and online statistics courses. The Survey of Attitudes Toward Statistics and three subscales of the Statistics Anxiety Rating Scale were administered at the beginning and end of graduate level educational statistics courses. Significant effects were observed for two anxiety scales (Interpretation and Test and Class Anxiety) and two attitude scales (Affect and Difficulty). Observed decreases in anxiety and increases in attitudes by online students offer encouragement to faculty that materials and techniques can be used to reduce anxiety and hopefully enhance learning within online statistics courses.”

 

“To Blog or Not to Blog: Student Perceptions of Blog Effectiveness for Learning in a College-Level Course”
The Internet and Higher Education, December 2010, Vol. 13, No. 4, 206-213. doi: 10.1016/j.iheduc.2010.04.001.

Abstract: “Blogs have the potential to increase reflection, sense of community and collaboration in undergraduate classrooms. Studies of their effectiveness are still limited. The purpose of this study was to investigate whether the use of blogs in a large lecture class would enhance students’ perceived learning. Students in an undergraduate nutrition course were required to engage in blog conversations over the course of the semester to promote reflective learning. Sixty-seven undergraduates responded to a survey with dimensions on perceived learning and sense of community. Sense of community and computer expertise were identified as significant predictors of perceived learning, when controlled for age, gender, and previous blogging experience. While a majority of the students reported that blogging enhanced their learning and led them to think about course concepts outside the classroom, fewer perceived value in peer comments. Implications for integrating blogging into undergraduate classrooms are discussed.”

 

“Procrastination, Participation, and Performance in Online Learning Environments”
Computers & Education, January 2011, Vol. 56, No. 1, 243-252. doi: 10.1016/j.compedu.2010.07.025.

Abstract: “The present study focuses on a specific learner characteristic in the management of time — procrastination — and its role in an online learning environment. More specifically, it was expected that procrastination would influence the successfulness of online learning and that this could be explained by the level of participation of learners in discussion forums. A study was conducted to test this hypothesis among a sample of learners taking a 10-week course on environmental and land use issues. As predicted, a negative relationship was found between procrastination and performance, and this relationship was mediated by the level of the learners’ participation in discussion forums. In other words, it appears that if high procrastinators are less successful online learners than low procrastinators, it is partly due to their lack of participation in discussion forums during the learning process. Additionally, some behavioral differences between high and low procrastinators were found in the times they decided to (re)start working at a distance, felt motivated to work on their course, and felt like dropping out of the course. To conclude, some practical implications for tutoring online activities and for stimulating participation in online learning environments have been proposed.”

 

“Impact of Podcasting on Student Motivation in the Online Learning Environment”
Computers & Education, September 2010, Vol. 55, No. 2, 714-722. doi: 10.1016/j.compedu.2010.03.004.

Abstract: “Researchers investigated the impact of podcasting on student motivation in the online environment during fall 2008 and spring 2009. Data were collected from students enrolled in fourteen online courses at a research university in the United States. One hundred and ninety-one students completed a modified version of the Instructional Materials Motivation Survey; it has four subscales: attention, relevance, confidence, and satisfaction. Strong positive relationships between all subscales were detected. Results indicate students were moderately motivated by the use of podcasts in their online courses. Statistically significant differences in student motivation based on gender, class standing, and prior online learning experience were found. Benefits of using podcasts and recommendations for improvement of the multimedia files were offered by users.”

 

Tags: cognition, inequality, technology