Why Do Institutions Offer MOOCs?
Fiona M. Hollands
Teachers College, Columbia University Devayani Tirthali
Brown University
Abstract
By reviewing the literature and interviewing 83 individuals knowledgeable about massive open online courses (MOOCs), we investigate the goals of institutions of higher education that are currently developing and delivering such courses. We identify six major goals for MOOC initiatives: extending reach and access, building and maintaining brand, improving economics by reducing costs or increasing revenues, improving educational outcomes, innovation in teaching and learning, and conducting research on teaching and learning. Comparing these goals with the data being collected about MOOCs, their participants, and educational outcomes, as well as the resource requirements and cost drivers of the development and delivery process, we assess whether these goals are being met, or are likely to be in the future. While quantification of success in achieving these goals is for the most part lacking, we conclude that institutions are experiencing at least partial success in achieving each of these goals except for improving economics. We identify obstacles to fuller achievement of the goals and some potential solutions.
Introduction
Over the past few years, observers of higher education have speculated about dramatic changes that must occur to accommodate more learners at lower costs and to facilitate a shift away from accumulation of knowledge to acquisition of a variety of cognitive and noncognitive skills (e.g., “Connecting the Dots,” 2013). All scenarios feature a major role for technology and online learning. Massive open online courses (MOOCs) are the most recent candidates being pushed forward to fulfill these ambitious goals. To date, there has been little evidence collected that would allow an assessment of whether MOOCs do indeed provide a cost-effective mechanism for producing desirable educational outcomes at scale. It is not even clear that these are the goals of the institutions offering MOOCs. This paper is a report on the actual goals of institutions creating MOOCs or integrating them into their programs and reviews the current evidence regarding whether and how these goals are being achieved.
Online education at the college level has been expanding rapidly over the last decade with students participating in single courses or even earning entire degrees without setting foot in a brick-and-mortar institution. According to Allen and Seaman (2013, 2014), the online enrollment growth rate in the
United States has ranged between 6.1% and 36.5% per year since 2002. Means, Bakia, and Murphy (2014) outline and document four major trends in how universities are using online learning: “self-paced, adaptive instruction and competency-based learning; blended learning; learning analytics; and MOOCs” (p. 46). The arrival of MOOCs, which allow hundreds of thousands of students to participate simultaneously in a course and are free and open to any interested participant, constitutes a phenomenon that extends preexisting initiatives to provide free, educational resources online, such as MIT OpenCourseWare, Stanford Engineering Everywhere, and Khan Academy.
The goals, structure, and pedagogical philosophy of connectivist MOOCs (cMOOCs), first offered by Siemens, Downes, and other Canadian instructors (see Downes, 2008; Cormier & Siemens, 2010) and “xMOOCs”, first offered by Ng, Thrun, Norvig, and Widom at Stanford University (see Markoff, 2011; Waldrop, 2013) are radically different, with the only commonality being that they are scalable and technology based. Whereas in cMOOCs learning occurs through participant interactions with a network of individuals and participants who are expected to create, share, and build upon each other’s artifacts (e.g., videos, blog posts), xMOOCs are primarily designed to deliver education at scale and involve more structured and sequenced direct transmission of knowledge. There has been little agreement as to what actually constitutes a MOOC and what educational or other objectives they can and should address. However, as observed by Lewin (2013), one universal impact of MOOCs is clear: “The intense publicity about MOOCs has nudged almost every university toward developing an Internet strategy” (para. 11). Allen and Seaman (2014) report that, in 2013, 5% of 2,831 U.S. institutions responding to an annual survey about online learning were offering a MOOC, 9% were planning to do so, and 53% were undecided as to whether to engage in this innovation. Larger institutions were more likely to offer a MOOC than smaller ones, as were doctoral/research institutions compared with institutions offering less advanced degrees. The value and purposes of engaging with MOOCs have been less clear, and many undecided institutions are struggling with whether and how to join the race to provide education using this relatively untested method of pedagogical delivery. At the same time, development and delivery of MOOCs requires a significant commitment of personnel time and institutional resources. Hollands and Tirthali (2014) document that MOOC production and delivery almost invariably involve multiple actors contributing several hundred hours of their time, costing their institutions between $39,000 and $325,000 per MOOC. Clearly, this demand on effort and resources needs to be justified in terms of expected benefits.
It is curious that MOOCs have taken hold without much evidence as to whether they are effective in improving participants’ skills and knowledge, or in addressing other objectives, and without evidence of their economic value. As Means et al. (2014) observe, “Both irrational exuberance and deep-seated fear concerning online learning are running high” (p. 42). This study seeks to address the lack of clarity surrounding institutional motivations for developing and delivering MOOCs and to investigate whether MOOCs are worth the considerable resources and effort being expended on their creation. We first review what is known about institutional goals for pursuing MOOCs. Subsequently, we describe our methods for interviewing a variety of stakeholders in the MOOCscape. We summarize our findings in terms of six goal areas, in each case assessing whether the goals are being met, what the obstacles to their realization are, and how they could be better fulfilled in the future.
Literature Review
Peer-reviewed publications on MOOCs first began appearing in 2008, (see review by Liyanagunawardena, Adams, & Williams, 2013), and some early indicators of student performance in MOOCs have emerged over the past two years (e.g., Breslow et al., 2013; Firmin, Schiorring, Whitmer, Willett, & Sujitparapitaya, 2013; Champaign et al., 2014), but there is a dearth of rigorous studies investigating the effectiveness of MOOCs in addressing educational objectives. Researchers are still
formulating ideas about how to assess the effectiveness of different types of MOOCs. Some researchers (e.g., Grover, Franz, Schneider, & Pea, 2013; DeBoer, Ho, Stump, & Breslow, 2014) point out that current discussions about the effectiveness of MOOCs are based on assumptions and outcome variables relevant to previous learning environments, and that there is a need to reframe the discussions and reconceptualize the variables. In addition, MOOCs are being pursued at many institutions for reasons other than the improvement of teaching and learning, creating the need for alternative metrics to assess the impact of MOOCs on these other objectives.
Hollands and Tirthali (2014) report that colleges and universities have adopted several different stances toward engaging with MOOCs. Some are actively developing MOOCs and may be termed “producers,” some are using MOOCs developed by other institutions in their programs and could be termed “consumers,” and a few are doing both. Others are adopting a wait-and-see approach, and some have considered MOOCs and have either decided against any form of official engagement or have not met with interest from faculty members to pursue them. While many people assume that MOOC production is limited to “elite” universities, other institutions, including a few community colleges, have concluded that they know how to serve their particular students best and have created MOOCs tailored to the needs of their own student populations. Consumers of MOOCs are integrating MOOCs created by other institutions into their course offerings in flipped classrooms (e.g., San José State University, see Cheal, 2012) or simply as supplemental resources for their students. Instructors at the University System of Maryland are experimenting with a variety of approaches to embedding MOOCs created by others into their classes (Griffiths, 2013; Ithaka S+R, 2013; Ithaka S+R, 2014). A small number of institutions have declared willingness to consider MOOCs for credit (e.g., Georgia State University, University of Maryland University College). A few universities, such as Vanderbilt University and the University of California, Berkeley, could be categorized as both producers and consumers, with some instructors creating MOOCs and others using MOOCs created by their own or other institutions in their classes.
Allen and Seaman (2014) report that, in 2013, 5% of institutions responding to their survey of online education had already offered a MOOC. Among the 140 or so MOOC-offering institutions, 27% indicated that their primary objective was to increase the institution’s visibility, 20% primarily sought to increase student recruitment, 18% wanted to innovate pedagogy, and 17% wished to provide flexible learning opportunities. Other primary objectives, mentioned by fewer than 10 institutions each, included reaching new students, supplementing on-campus experiences, exploring cost reductions, learning about scaling, and creating revenues. Surprisingly, increasing the quality of teaching and learning did not appear to be a priority. As the survey asked only for the primary objective, it is possible that increasing the quality of teaching and learning may have been a secondary objective for some. Two thirds of the institutions felt it was too early to judge whether their objectives were being met, while one third felt some or all objectives were being met. It is not, however, clear whether these perceptions were based on substantive evidence.
The Alliance for Higher Education and Democracy (AHEAD) at the University of Pennsylvania recently conducted a poll of administrators, faculty members, and other personnel at institutions of higher education (IHEs) in the United States and found that, among the approximately 44 respondents (reported as “around 29%” [p.4] of 153 respondents) at institutions offering a MOOC, 57% strongly agreed that MOOCs may be a potentially effective mechanism for “raising institutional profile” (AHEAD, 2014). Fifty percent of the respondents strongly agreed that MOOCs could help improve access around the globe, 40% that MOOCs can help improve access in the United States, 34% that MOOCs can improve pedagogy, and only 19% that they can reduce costs. Confidence in the potential for MOOCs to reduce costs was higher among respondents from institutions that had not yet offered MOOCs. For all other goal options offered by AHEAD, MOOC offerers appeared more confident than MOOC abstainers about the potential for MOOCs to achieve the goals.
This limited information from the Allen and Seaman and AHEAD surveys on the goals of institutions offering MOOCs suggests a variety of motivations that extend well beyond the dissemination of knowledge at scale. Our study was designed to investigate institutional goals and objectives for engaging with MOOCs through in-depth interviews, and to collect evidence of whether and how these goals are being achieved.
Method
Using a methodology similar to that employed by Bacow, Bowen, Guthrie, Lack, and Long (2012) to investigate barriers to online learning in higher education, we conducted a qualitative study (see Merriam, 2009) comprising interviews of 83 individuals across a range of institutions. With the aim of eliciting perspectives on MOOCs from several angles, we sought representation from both public and private IHEs, both two-year and four-year institutions, researchers, online learning platform providers, other for-profit education companies, and several additional stakeholders in the online learning space. Table 1 shows the distribution of interviewees across institutional types.
Table 1 Affiliations of Interviewees
Type of institution
Number of institutions
represented*
Number of
interviewees
Public university
16
20
Private university
14
26
Community college
9
10
Research organization
7
8
Platform provider
5
6
Other for-profit education company
4
5
Other institution**
7
8
Total
62
83
*At the majority of institutions one person was interviewed, but at a few institutions several individuals were interviewed to include interviewees from multiple areas (for example, administrators, faculty members, and researchers).
**Other institutions consisted of the following: one museum (two interviewees); one K–12 school district; one educational technology advocacy group; one higher education association; one venture capital firm; one private foundation; and one independent consultant.
We contacted by e-mail individuals who appeared to be knowledgeable about MOOCs or online learning based on their position in deciding whether and how to participate in the MOOCspace, their experience teaching or planning MOOCs, or their writing and research in this area. Interviewees were identified from the existing academic and journalistic literature on MOOCs, by reviewing the names of conference presenters and panelists for sessions on MOOCs or online learning in higher education, by researching the MOOC activities of institutions on the Internet, and by consulting with known experts in the field of educational technology. Additionally, many of our interviewees suggested other people for us to interview either at their own institutions or elsewhere. Given the targeted nature of our recruitment process, our sample is not representative of U.S. educational institutions as a whole. However, in addition to stratifying our sample across public and private four-year institutions and community colleges, we aimed to include institutions and interviewees that were supportive of MOOCs and those that were not.
We contacted a total of 100 individuals on a rolling basis at 66 different institutions, 39 of which were colleges or universities. Of the 100 contacted, 83 people at 62 institutions were successfully interviewed within the time frame of our study. At least one person from every college or university we contacted agreed to participate. At most institutions we interviewed one person, but at a few institutions we interviewed two or more. Most interviewees were based in the United States, two were in China, two in the United Kingdom, and several were in Canada. Thirty of our interviewees were administrators at IHEs; 22 were faculty members, of which 7 also had administrative duties; 16 were executives at other institutions; 13 were researchers; one was an educational technologist; and one was a program officer at a foundation.
Interviews were conducted between June 2013 and February 2014, and follow up by e-mail with interviewees to obtain updates and to verify information continued until May 2014. Slightly under half of the interviews were conducted face-to-face, and the others were conducted by telephone or Skype. Interviews lasted an average of 75 minutes and followed a semistructured interview protocol (Merriam, 2009). Interview questions addressed what the interviewee’s institution was currently doing with respect to MOOCs; institutional goals for pursuing MOOCs; the interviewee’s role in the process; how the interviewee and his or her institution define a MOOC; the characteristics of MOOCs, including structure, purpose, enrollment level, fees, and credentials offered; educational objectives of specific MOOCs being offered; educational outcomes being measured; data being collected before, after, and during MOOC delivery; how MOOC-based research is being used to improve pedagogy on-campus; personnel and other resource requirements; cost drivers; how costs of MOOC production compare with face-to-face courses; how the interviewee sees the MOOC phenomenon developing over the next five years, and how this development might help the institution meet its ongoing goals.
Most interviewees granted permission for the interview to be recorded, and the resulting digital audio files were transcribed. All interview notes and transcriptions were coded in NVivo software using deductive codes initially derived from the interview protocol. These initial codes were supplemented by inductive codes that were iteratively refined as more granular themes were identified (LeCompte & Schensul, 1999). Interviewee perspectives were supplemented with a review of the MOOC-related literature and perspectives offered by presenters at various academically oriented MOOC-related events. A comprehensive report of interview findings was circulated to all interviewees in April 2014 to allow for corrections and clarifications. This paper focuses on the data relevant to the goals of institutions engaging with MOOCs.
Results
Of the 62 institutions that participated in this study, 29 had already offered a MOOC or had used one or more MOOCs in their programs, or were in the midst of doing so. Our interviewees identified a variety of institutional goals, which fell into one of six categories:
•
extending the reach of the institution and access to education
•
building and maintaining brand
•
improving economics by lowering costs or increasing revenues
•
improving educational outcomes for MOOC participants and on-campus students
•
innovation in teaching and learning
•
conducting research on teaching and learning
Table 2 indicates the number of each type of educational institution in our sample at which one or more interviewees mentioned each goal, as well as the percentage of all 83 interviewees who mentioned each goal.
Table 2 Institutional Goals for Developing and Delivering or Using MOOCs
Institutional goal
Number of institutions offering/using MOOCs stating
this as a goal interviewees who % of all raised this as a
goal (n = 83) % of total
(n = 29)*
Private universities
(n = 10)
Public universities
(n = 15)
Community colleges
(n = 3)
Extending reach and access 65 7 8 3 42
Building and maintaining
brand 41 3 8 0 25
Improving economics 38 2 8 1 29
Improving educational
outcomes 38 4 5 2 20
Innovation in teaching and
learning 38 3 7 1 19
Research on teaching and
learning 28 4 3 1 18
* Includes one museum in addition to the universities and colleges
Goal 1: Extending Reach and Access to Education
Extending the reach of the institution to a wider audience and improving access to education was the most commonly identified goal for offering a MOOC, mentioned by 65% of the institutions in our study that were offering or using MOOCs and 42% of our interviewees overall. These interviewees included administrators and faculty members from seven private universities, eight public universities, three community colleges, and a museum. Some institutions claimed to be pursuing an altruistic goal of reaching the masses globally with high-quality educational experiences, while others presented more defined goals of reaching a specific population or solving a particular challenge related to access. Several instructors described being motivated to offer MOOCs because of their passion for their subject and desire to make it accessible to the general public. Ways in which MOOCs are expected to increase access to education include the following:
•
“broadcasting” to global audiences
•
alleviating infrastructure constraints domestically and in rapidly developing
countries where the existing physical campus infrastructure and level of faculty
expertise cannot accommodate the growing demand for postsecondary education
•
easing the pressure on oversubscribed programs or “bottleneck” courses
•
providing flexibility in time and place of study for nontraditional students
•
providing a no-risk, low-cost option for at-risk students in developmental
education, setting them on an accelerated path to credit-bearing courses and more
timely completion of a degree
•
increasing access to instructors skilled in specialized domains and niche subjects;
•
flexibility for students to create their own programs using courses from various
institutions
•
continuing education or professional development for alumni and other working
adults
Evidence of MOOCs Extending Reach and Access Institutions offering MOOCs via the major MOOC platforms have attracted a substantial number of participants: edX reported around two million unique users as of March 2014, and Coursera reported over seven million. In January 2014, Coursera reported
over 22 million enrollments, with many participants enrolling in multiple courses. These enrollees were spread across 571 courses and were based in 190 different countries (see https://www.coursera.org/about/community). While these numbers are impressive, only a small fraction of those who registered actually participated in the courses, and far fewer completed them. Ho et al. (2014), using data from 17 HarvardX and MITx MOOCs offered between fall 2012 and summer 2013, found that 841,687 people registered for the MOOCs, but 35% of them never engaged with the content, and 56% engaged with less than half of it. Around 43,000, or 5% of participants, completed their courses. A typical registrant was “male with a bachelor’s degree who is 26 or older” (p. 2). This finding is in line with Christensen et al.’s (2013) claim that 79% of MOOC participants have a bachelor's degree, based on a survey of participants in 24 MOOCs offered on the Coursera platform by the University of Pennsylvania. With respect to global reach, 72% of the registrants in the HarvardX and MITx MOOCs were from outside the United States, and 2.7% were from countries on the United Nations’ list of Least Developed Countries (Ho et al., 2014). Among the 34,779 participants in the University of Pennsylvania sample, 31% were from non-U.S., OECD countries; 15% were from BRICS (Brazil, Russia, India, China, and South Africa); and 20% were from other developing countries.
These data suggest that MOOCs are providing educational opportunities to millions of individuals across the world. However, most are already well educated, and only a small fraction of these participants fully engages with the courses. Among our interviewees, we found mixed views as to whether MOOCs have succeeded in improving access to their institutions’ offerings and reaching wider audiences. Some had no doubt that MOOCs had already expanded access to their courses while others acknowledged that the initial expectations that MOOCs would “democratize” education had not been realized and that MOOCs may even be doing more to increase gaps in access to education than to diminish them. Quantifying the impact of MOOCs offered by a particular institution on improving access to its regular courses is difficult unless institutions accurately document the level and diversity of participation in their programs both before and after offering MOOCs. For the most part, institutions were not yet making a concerted effort to document these potential changes.
Barriers to Using MOOCs to Extend Reach and Access, and PossibleSolutions Several interviewees pointed out that simply providing free resources does not necessarily make them accessible to everyone. The end user may still experience infrastructure constraints and cultural barriers. In some locations Internet bandwidth is inadequate for MOOC participation, which involves downloading materials and watching videos. Buying additional bandwidth is very expensive in some countries, such as China, and may be prohibitive for many would-be MOOC participants. Adaptations have been made in certain countries to overcome bandwidth limitations. For example, according to one interviewee, in Pakistan, where the infrastructure outside of cities cannot support MOOCs, the government is partnering with FutureLearn and Coursera to transfer MOOC content to CDs. ZayaLabs, a technology startup, provides tablets and a router loaded with MOOC content in a backpack to schools in India, Indonesia, and Mongolia. Taking the content offline allows participants to engage with it in the absence of any wireless infrastructure. As platform providers such as NextThought and Coursera move toward mobile access to their content, some of these infrastructure constraints may be alleviated.
Less easily addressed were some pessimistic views about the capacity of online courses, massive ones in particular, to reach any but the most dedicated students. One interviewee observed that access to educational resources is only one part of the equation to guarantee success for the majority of college students, with the need to motivate students, track their progress, and guide them along a trajectory toward completion being equally important. Another observed that education is more complex than simply providing access to content. In addition to logistical and pedagogical barriers to improving access to education through MOOCs, a number of regulatory obstacles were mentioned, especially for community colleges and K–12 schools. Such obstacles will only be resolved if accreditation agencies and state education departments review and revise regulations to accommodate online offerings that reach far beyond local confines.
Goal 2: Building and Maintaining Brand
For IHEs, building and maintaining brand serves to attract and retain students, faculty members, and partnership opportunities with other institutions, funders, and alumni networks. Interviewees from 41% of the institutions that were offering or using MOOCs stated that branding, positioning, or attracting students was a strategic goal for the initiative. These interviewees represented seven public universities, three private universities, and a museum. None of the community colleges mentioned branding as a goal for engaging in MOOCs. Branding involves different strategies for different types of institutions, depending on their current positioning. For flagship state universities in the United States that have already established local appeal, MOOCs allow for more national and international recognition. For elite institutions, being at the forefront of a highly publicized innovation serves as a signaling mechanism to protect their global ranking among the top universities. Community colleges were less concerned about branding than improving access to education for an underprepared student population.
Institutions that were employing MOOCs as a vehicle to expand their brand made strategic decisions to showcase programs, specialties, and research capabilities in which they believed they were leaders or wanted to be among the first to stake a claim to world-class expertise. Some institutions perceived building brand as a direct pathway to higher recruitment and enrollment in tuition-earning courses and programs. MOOCs offer participants a “frictionless”—that is, a low-risk, no-cost— introduction to a program area and the offering institution can simultaneously identify candidates who might perform well as degree-earning students. In addition to recruiting students, some institutions emphasized the need to showcase themselves as the leader in academic content areas and in pedagogical innovation so as to attract and retain the best faculty members. MOOCs also offer an opportunity to maintain connections to alumni, who are an important source of funding. Finally, MOOCs provide opportunities to forge mutually beneficial partnerships with other educational institutions, and with corporate partners and funding agencies, potentially leading to increased grant revenues.
While many institutions have received significant media attention as a result of their MOOC activities, isolating and measuring the impact of any new initiative on brand is a difficult exercise. Most institutions are only beginning to think about how to capture and quantify branding-related benefits—for example, by comparing historical data on applications and admissions with post-MOOC statistics. One interviewee from an elite institution observed that selective institutions must balance the apparently contradictory goals of increasing access to their offerings with building and maintaining brand. Another complication raised was the question of where the brand actually lies. Do participants opt for a course because it is on a particular platform or because it is offered by a particular university? It was noted that as MOOC platform providers such as edX and Coursera open their platforms to a wider range of institutions, some of the initial cachet of belonging to these consortia is being lost.
Goal 3: Improving Economics: Reducing Costs or Increasing Revenues Thirty-eight percent of the institutions in our sample that were offering or using MOOCs, and 29% of our interviewees overall, claimed that a goal for their MOOC initiatives was to lower costs or increase revenues, or both. Among the 29 institutions currently offering or using MOOCs, this goal was mentioned by representatives of eight of the 15 public, four-year universities, two of the ten private, four-year universities, and one of the three community colleges, suggesting that it was more important to public universities than private ones. Many interviewees expressed concern at the lack of sustainability for MOOC initiatives given the heavy burden on faculty time and other institutional resources. There was widespread acknowledgement that the current expenditures on MOOC development could not continue indefinitely without financial justification.
Potential Cost Savings from MOOCs To date, MOOC production and delivery have resulted in a significant drain on time and money for institutions, but interviewees offered several possibilities for eventual cost savings:
• reusing MOOC materials multiple times
• sharing MOOC materials across instructors and campuses • developing common courses to offer across institutions • replacing on-campus courses with MOOCs
• faculty time savings
• reducing the need for facilities • recruitment efficiencies
• less costly student support services provided by nonfaculty members
•
increasing student throughputNot surprisingly given the infancy of MOOC initiatives, we were able to identify very few examples of actual costs savings realized. To the contrary, we were able to gather significant evidence of the high costs of development and delivery of MOOCs. Realizing cost and time savings may be more difficult than anticipated. Some interviewees, based on their experiences with regular online courses, questioned the likelihood of being able to reuse MOOC materials without significant adjustments each time. Others observed that sharing MOOCs across campuses or developing common courses would require the resolution of logistical issues such as how tuition and credits are transferred, and a shift in culture away from creation of content at each site. Few interviewees believed that MOOCs could fully substitute for on-campus instruction, expecting that they would need to be accompanied by face-to-face interactions between a live instructor and the students.
One example we encountered where less costly student support services are being provided is the MOOC-based Online Master of Science in Computer Science (OMS CS) program that the Georgia Institute of Technology (Georgia Tech) is developing with Udacity. Each course is staffed by a Georgia Tech instructor and a Head Teaching Assistant (TA) serving a projected 127 students in Year 1, rising to 456 students in Year 3 (although the actual average course size for the five courses offered in spring 2014 was 115 students). The Head TA, while selected by Georgia Tech, is employed by Udacity. In addition, Udacity-based Course Managers approved by Georgia Tech each provide technical support to around 390 students. Georgia Tech graduate students, serving 40 students each, serve as TAs to help with grading and responding to academic questions. Georgia Tech and Udacity are working on ways to manage the workload as enrollments increase, including more use of automated grading. Courses are initially taught by a faculty member, but reruns may be taught by a nontenured instructor or the Head TA. While this model may rely less than regular programs on expensive faculty time, it is not clear yet whether it will be sufficient to adequately support students’ needs.
MOOCs as a Source of Revenue A small number of interviewees speculated that some IHEs are pursuing MOOCs as a potential source of revenue, but only two representatives from state universities explicitly stated revenue generation as a goal. Potential sources of revenue included the following:
• offering credit for MOOC completion and charging tuition • creating new for-fee courses and programs
• drawing MOOC participants into existing, full-tuition degree programs • increasing class sizes
• licensing fees for use of MOOC materials or data by other institutions
• fees for additional services offered to MOOC participants—for example, online tutoring or face-to-face instruction with a local instructor
• increased grant revenues
• matchmaking or training for employers
There are a few instances in which MOOCs or MOOC-like courses have been offered for credit with tuition being charged. San José State University piloted online courses that were offered free and without credit to the public, but a smaller number of formally enrolled students paid $150 each to earn three to five credits per course. Georgia Tech’s OMS CS program charges $134 per credit. In each instance where MOOC-like courses are offered for credit, the offering institution has provided some degree of supplementary face-to-face instruction or online support for course participants. However, based on our review of the financial and enrollment projections for Georgia Tech’s OMS CS program, we do not expect that the revenues will exceed the high fixed costs of MOOC development over the initial three-year period. We did not have access to financial projections for San José State University’s MOOC initiative that would allow a similar review. At the University of Oklahoma, 20 credit-bearing, online courses have been developed with on-campus students as the primary audience in a flipped classroom model. These courses carry the same credits as the university’s traditional courses and charge the same tuition. The courses are also open as online only, tuition- and credit-free MOOCs to any member of the public.
Evidence to support the likelihood of revenues materializing from other sources was more limited. We heard several anecdotal reports of increased interest in an institution’s regular courses subsequent to a MOOC offering, but we found only one documented instance in which MOOC participants were being drawn into an existing, full-tuition program. University of Texas at Arlington’s School of Nursing offered a MOOC through MOOC2Degree, and an undisclosed number of participants were in the process of applying to the university’s program. Several interviewees claimed that using MOOC materials to flip the classroom and automating certain aspects of on-campus courses had allowed more students to enroll in a class, but it was unclear whether there was a net increase in number of students at the university or whether these students were simply being drawn away from other existing courses and programs. Some fees have been earned by offering MOOC participants verified certificates of completion, but the percentage uptake for this service has, to date, only been on the order of 1% of initial enrollees. While we heard several reports of licensing arrangements being signed to allow one institution to use another’s MOOC content, we encountered no instances where fees had actually changed hands. Revenue increases from grants or from matchmaking or training services for employers currently appeared to be purely aspirational.
Goal 4: Improving Educational Outcomes
Thirty-eight percent of the institutions participating in our study that were offering or using MOOCs and 20% of our interviewees overall expected MOOCs to lead to an improvement in educational outcomes, some believing improvement would occur directly within the MOOC format and some others believing improvement would happen indirectly through the transfer of new strategies and techniques to on-campus teaching. This goal was mentioned by representatives of five public universities, four private universities, and two community colleges. Ways in which MOOCs were expected to lead to improvements in educational outcomes included the following:
• providing instant feedback to course participants • gamification and badging to increase motivation
• outreach to MOOC participants to encourage persistence • adaptive learning, personalization, or mastery-based learning • flipping the classroom using MOOCs to provide the online content • motivating instructors to rethink pedagogy
• redesigning regular courses to incorporate MOOC strategies, such as “chunking” lectures and interspersing questions, and increasing opportunities for peer-to-peer learning
• using MOOCs in K–12 to prepare students for college • fine-tuning instructional materials
According to a number of interviewees, the most significant impact of MOOCs has been on the motivation they have created for instructors to rethink how they teach. Several interviewees reported that MOOCs have prompted even typically intransigent faculty members to reconsider their teaching styles and have facilitated faculty professional development that might otherwise be resisted. However, several interviewees remarked on the lack of evidence that better learning experiences are actually being created for students, observing that change in practice is not inherently positive. Indeed, while interviewees provided many examples of how MOOCs have been used to change instruction, for the most part, actual impact on educational outcomes has not been documented in any rigorous fashion. Consequently, in most cases, it is unclear whether the goal of improving educational outcomes has been achieved. Cima (2014) and Ghadiri, Qayoumi, Junn, Hsu, and Sujitparapitaya (2013) provide two exceptions in which instructional changes and their effects on student performance have been carefully recorded. While not gold-standard, random-assignment experiments, they provide promising evidence of improvement in student performance as a result of integrating MOOCs into flipped on-campus courses, or adopting some of the MOOC strategies, such as frequent assessment and automatic feedback. On the other hand, Ithaka S+R (2014) conducted several side-by-side comparison tests of courses at the University System of Maryland and found no significant differences in posttest gains between students participating in hybrid courses that included MOOC materials and those participating in traditional versions of the courses.
While many interviewees pointed to the potential for MOOCs to contribute significantly to the development of personalized and adaptive learning, it is clear that current turnaround time for analyzing MOOC platform data is too long to allow for even midcourse corrections, let alone to provide just-in-time customization for individual participants. Without exception, interviewees acknowledged that this potential is far from being realized. A great deal of coordination and collaboration among content experts, instructors, researchers, instructional designers, and programmers will be necessary to result in substantive progress towards personalized and adaptive learning.
Goal 5: Innovation in Teaching and Learning
Thirty-eight percent of the institutions in our study that were engaging with MOOCs and 19% of our interviewees overall presented MOOCs as vehicles for experimenting with and innovating pedagogy and models of higher education. Innovation as an end goal was mentioned by representatives of seven public universities, three private universities, and one community college. We note, however, that the other five goals we report involve innovating as a means to a particular end.
A number of interviewees portrayed experimentation with MOOCs and online learning as preparation for an uncertain future. Institutions do not want to be left behind in what they perceive could be a game-changing phenomenon for higher education. MOOCs were also presented as a disruptive innovation within the higher education business model that could help universities become more competitive simply by forcing them to reconsider the status quoand to ask fundamental questions about commonly accepted practices. One interviewee asserted that MOOCs have pushed college personnel to open up to the research about how people learn. A senior administrator observed that, especially for large state universities, change cannot be implemented swiftly. If these institutions do not at least experiment with innovative ideas as they arise, they will always lag behind smaller, more nimble competitors. A researcher at one of the pioneering MOOC producers described a trajectory of innovation in MOOCs starting with the initial infatuation with scale, proceeding to the search for a peer-mediated solution to the challenge of grading thousands of assignments, to the current recognition that participants may learn best when connected with other participants and finding ways to facilitate this through online chat rooms and small group discussions.
Not all of our interviewees were as convinced that MOOCs would introduce more productive innovation than existing experiments with online and blended learning. Interviewees with experience in online education and open learning resources over the last two decades pointed out that many MOOC developers are reinventing and relearning the missteps and successes of online learning and failing to take advantage of scale, the most characteristic aspect of MOOCs that differentiates them from other online education.
The large number of courses that have been developed or substantively redesigned since the appearance of MOOCs can be considered as evidence of innovation—for example, MOOC.list currently catalogs around 1,700 courses. It is abundantly clear that MOOCs have prompted many institutions and faculty members to engage in new educational activities. The strategies employed online, such as frequent assessments and short lectures interspersed with questions, are subsequently being used on campus. It is less clear what has been gained by these new initiatives because the value of innovation is hard to measure unless it can be tied to a more tangible objective. While a few institutions are developing metrics to assess the impact of MOOC-related innovations on various objectives, most institutions are not yet making any rigorous attempt to assess whether MOOCs are more or less effective than other strategies to achieve their goals.
Goal 6: Research on Teaching and Learning
Research on teaching and learning was stated as a goal for MOOC initiatives by 28% of the 29 institutions offering or using MOOCs and by 18% of our 83 interviewees. This was a stated goal for one community college, three public research universities, and four private research universities. There is considerable overlap between formal research and ad hoc efforts to improve teaching and learning. While there is often a fine line of distinction between Goal 4 and Goal 6, in this section we generally address work that is being conducted by individuals who consider themselves researchers first and foremost and aim to publish their work, while efforts described under Goal 4 were less about creating generalizable knowledge and more about improving educational outcomes in a specific situation.
We heard from our interviewees about several types of research being conducted that use a MOOC as the vehicle for delivering an intervention or as a source of data on participant behavior and performance. The researchers themselves were in a range of disciplines, which influenced the kinds of questions being asked. Computer scientists and engineers have tended to focus either on descriptive studies, summarizing demographics or establishing how much each educational resource (video, e-text, discussion forum, etc.) was used, or on design research to create badging systems, discussion opportunities, peer grading, and auto-grading applications. Sociologists and education (or learning science) researchers have more often focused on experimental research—for example conducting A/B tests by e-mailing different messages or materials to different participants selected at random. Areas in which interviewees described research explorations and advances included the following:
• the role of social media networks in teaching and learning
• testing pedagogical strategies—for example, Firmin et al. (2013) and Champaign et al. (2014)
• student engagement and motivation—for example, Coetzee, Fox, Hearst, and Hartmann (2014); Kizilcec, Piech, and Schneider (2013); and Kizilcec, Schneider, Cohen, and McFarland (2014)
• machine learning and modeling research—for example, Piech et al. (2013); Halawa, Greene, and Mitchell (2014); and Nguyen, Piech, Huang, and Guibas (2014)
• natural language processing
• human–computer interaction—for example, Kulkarni et al. (2013) and Cambre, Kulkarni, Bernstein, and Klemmer (2014)
• personalized and adaptive learning—for example, Buffardi and Edwards (2014) • comparing hybrid with traditional courses—for example, Ithaka S+R (2013, 2014)
Griffiths (2013); and Bruff, Fisher, McEwen, and Smith (2013)
• developing data standards and a common platform for data mining—for example, Veeramachaneni, Dernoncourt, Taylor, Pardos, and O'Reilly (2013) and Dernoncourt et al. (2013)
A great deal of effort has been expended on trying to improve participant engagement and completion of MOOCs and less effort on determining whether participants actually gain skills or knowledge from the courses. The majority of instructors and researchers appear content with equating the completion of a MOOC with having learned something. One exception is the work of RELATE at MIT (see Colvin et al., in press) which rigorously documents both absolute and relative learning in a physics MOOC using pre- and posttesting and Item Response Theory. A few interviewees noted that this question of efficacy is rarely asked with respect to regular college courses. Others questioned whether findings from MOOCs can be generalized to other populations and contexts. Many noted that progress in using MOOCs for educational research is being impeded by difficulty using the platform data and lack of clarity regarding regulations applicable to the participants and their data. Another obstacle is that in a MOOC researchers do not make the decisions about what gets taught and how, and faculty members may or may not be predisposed toward the researcher’s goals.
Conclusions and Recommendations
It appears that the goal of MOOC producers to extend reach to a wider audience and improve access to education has been met to some extent in terms of geographical spread but less so in terms of reaching individuals with fewer educational opportunities. Participants are mostly limited to highly motivated learners who have access to high-bandwidth Internet connections, and the majority have already earned at least a bachelor’s degree. These may be new audiences for many existing college and university courses and programs, but it appears that MOOCs are mostly educating the educated and are therefore increasing the divide between those who have access to education and those who do not. It may be more accurate to state that they are extending access to lifelong education rather than making education accessible to a broad group of people. On the other hand, this educated audience is also more likely to be able to pay for certificates and other services, allowing the production and delivery of MOOCs to be financially sustainable. While many institutions express the desire to improve access to education—and this is indubitably a laudable goal—there is little real incentive or financial justification for institutions to do so, except for those participants who can pay for it.
Going forward, if institutions genuinely wish to broaden access to less educated audiences, they must identify multiple channels of communication to reach potential recruits. For example, they may need to use social media networks and advertise through high schools, employment agencies, or community organizations in the United States and abroad. Additionally, courses must be designed to serve learners who are less self-directed by incorporating motivational features and optional instructional scaffolding to address differences in participant preparation levels. However, as IHEs can ill afford to offer free and unsubsidized educational opportunities to the world, the costs of developing MOOCs will need to be covered by some fraction of participants who have adequate resources.
If success in achieving the goal of building and maintaining brand was measured purely by media publicity surrounding MOOCs, many institutions have indeed become more visible as a result of their MOOC activities. This publicity has been mostly positive, but not always. Although several institutions suggested anecdotally that this goal has been partially accomplished, quantifying success could be achieved by comparing pre- and post-MOOC metrics on student recruitment and enrollment statistics,
success in faculty recruitment and retention, breadth and generosity of donors, and quantity and size of grants. In addition, the costs of MOOC production and delivery should be compared to other strategies known to be effective at building and maintaining brand in order to ascertain which alternatives are most cost-effective.
Typical of any early-stage, technology-based innovation, achievement of the third goal, to improve the economics of higher education, is still elusive. We found scant evidence that MOOCs have increased revenues substantially and much evidence that they have increased costs significantly. Gradually materializing revenues could help shift the balance over the next few years, but unless MOOC producers and platform providers continue to expand ways to confer economic value on MOOC completion—for example, in the form of employer-recognized credentials—the market and associated revenue streams will remain limited. One way or another, MOOCs will need to prove they can solve a problem or provide a service more efficiently than existing alternatives.
MOOCs could potentially reduce the costs of higher education if they were used to eliminate the reproduction of similar courses across many campuses. If the licensing issues surrounding such sharing of materials can be resolved, and if MOOCs can be rerun several times without significant adjustments at each offering, the high costs of initial MOOC development could be amortized over all the instances of use. However, as costs of delivering education are mostly personnel related, overall costs of higher education can only fall if personnel numbers or salaries are reduced. Reduction of costs could be realized if academic institutions were willing to shift more of the responsibility for instructional support away from tenured faculty members and onto nontenured instructors, teaching assistants, or outsourced personnel. Tenured faculty positions could not be easily eliminated in the short term, but they could be reduced over time, contributing to the current long-term trend towards “adjunctization” of higher education. This trend does not, however, bode well for the capacity of academic institutions to further various academic disciplines through research, nor is it likely to be welcomed by faculty members and their unions. For now, MOOCs are being promoted as a substitute for repetitive activities, such as delivering the same lecture every year or grading uniform assignments, while faculty members spend time in the classroom with students on problem solving, discussion, and debate. This approach may prove to be more educationally effective than current practice, but it will also be more expensive and therefore even more financially unsustainable.
From the perspective of learners, MOOCs will only favorably impact the economics of obtaining an education if they remain free or charge minimal fees but can be used to substitute for costlier on-campus or online courses to obtain college credits or other credentials that are accepted by employers. While this scenario may be beneficial to the students and to the taxpayer, it may not be in the interests of the academic institutions themselves, as it will imply a loss of revenue per student, which may or may not be possible to make up by increasing enrollments. State and federal education policymakers could adjust regulations to create pathways for MOOCs to be accepted for credit in high schools or to satisfy government-mandated continuing education for professionals, but in higher education it is less clear how accreditation agencies could be persuaded to move in this direction. Establishment of an accrediting organization for individual MOOCs and other nontraditional educational experiences would allow learners to accumulate a portfolio of credentials that serve as a viable supplement or alternative to a college degree. Furthermore, such accreditation of individual courses or other nontraditional educational experiences should confer the ability to use public funds toward the costs of these credentials. Fain (2013) reports on two potential efforts in this direction and highlights some of the surrounding regulatory issues. Ironically, MOOCs have introduced possibilities that, should they materialize, could significantly compete with the very institutions that are racing to develop them.
To assess the sustainability of MOOCs, researchers and evaluators will need to document how much the costs of delivery fall for repeat offerings of MOOCs, how enrollment numbers change, and what the uptake rate is for any revenue-generating services. To determine whether MOOCs are a cost-effective means to deliver education or achieve other objectives, it will be necessary to compare the costs of MOOCs to the costs of alternative interventions, as well as the effectiveness of each alternative in addressing a common outcome of interest, such as increasing participants’ knowledge or skills. It would also be useful to gauge the reaction of employers to such nontraditional credentials and to inquire as to what modifications employers might require to allow these credentials to be considered in employment or promotion decisions. The more value employers are willing to place on these credentials, the more fees MOOC offerers can charge to participants in order to offset institutional costs of production and delivery.
Many interviewees would readily assert that the fourth goal of improving educational outcomes has been achieved through MOOCs, at least for a limited group of learners. These positive reports arise in situations where MOOCs have been integrated with on-campus courses or where on-campus courses have been redesigned to incorporate MOOC-like components. Documentation of a small number of these experiments has shown early signs of improving student performance as measured by assessments and course completion rates. Typical strategies include flipping the classroom, implementing frequent assessments, and spending class time on problem-solving activities, often in small groups, rather than on lecture.
Little effort has been made to ascertain whether participants in stand-alone MOOCs gain useful skills and knowledge that can be applied in productive, real-world contexts. To determine whether the goal of improving educational outcomes can indeed be achieved through MOOCs alone will require rigorous evaluation employing pre- and postassessments of knowledge and skills, and comparisons with the outcomes of face-to-face or other online courses. Educators must progress beyond the notion that course completion per se equates with learning and must consider how to allow learners to demonstrate valuable competencies, both cognitive and noncognitive, that may be acquired through MOOC participation. Longitudinal studies tracking post-MOOC outcomes—such as sequences of courses taken, professional certifications obtained, or job opportunities received—would help assess the longer term value of participating in MOOCs. More effort is also required to assess whether MOOCs as stand-alone experiences can be designed effectively to increase knowledge and cultivate skills for any but the most motivated learners. Future research could be designed to broaden the types of learners represented in studies of MOOC activity and impact in order to avoid the presentation of results that are not applicable to the majority of learners.
The goal of innovation in teaching and learning has certainly been met at institutions where online learning did not previously exist, as evidenced by the sheer number of new courses produced and course redesign efforts undertaken. Metrics on the frequency of these efforts could be compared with pre-MOOC activity to provide an assessment of how well the goal of innovation has been achieved, but it is not clear that innovation per se is a sufficient aspiration. In many cases, institutions are innovating with a further goal in mind, such as improving educational outcomes or building brand. Assessing the impact of MOOC initiatives on these distal goals would be a more worthwhile endeavor in judging their contribution to higher education. Some institutions are unclear as to why they are embarking on MOOC initiatives and until they can agree internally on suitable and realistic goals, they will struggle to justify the expense and effort. For institutions that have been offering online and hybrid courses for many years, MOOCs represent more of an incremental step along a preexisting trajectory than a major innovation. Some even consider MOOCs to be regressive because they fail to incorporate best practices in distance learning and repeat many of the mistakes of earlier attempts to educate at scale. Going forward, researchers and practitioners who are familiar with evidence-based best practices in online learning could usefully recommend pedagogical strategies that can be effectively migrated to MOOCs. Finally, the goal of conducting research on teaching and learning has encountered many obstacles, but a few well-funded
institutions with strong computer programming capacity have not been deterred from efforts to use the “big data” from MOOC delivery platforms to document online learner behavior and to experiment with variations in pedagogical strategies. Regrettably, MOOCs are more often serving as a playground populated by captive participants within which researchers tinker with the artifacts than as an object of research to determine whether MOOCs, compared with existing models of education, are an effective and cost-effective means of educating a broad range of learners. MOOC researchers could benefit from soliciting the involvement of experienced online learning practitioners or researchers as they plan studies to experiment with MOOCs. A potentially useful application of the big data from MOOCs is to provide learner analytics. These can be used to iteratively improve courses or to catalyze the advent of high-quality adaptive learning and personalized educational experiences that meet the needs of a variety of learners. It is apparent, however, that the useful application of data-mining techniques to data from MOOC platforms could be facilitated by standardization of data formats across the various platforms. It is also clear that the high demand on resources to develop sophisticated adaptive-learning mechanisms will require the establishment of working partnerships between educators, instructional designers, and programmers. Federal approval of cross-institutional Institutional Review Board agreements could facilitate projects in which expertise and resources can be pooled. Additionally, clarifications regarding the applicability of FERPA to MOOCs would alleviate current confusion over how to handle participant data.
Given the considerable investment that MOOCs represent, we strongly recommend that prior to embarking on MOOC-related initiatives, institutions carefully consider their goals and whether MOOCs present a realistic and financially justifiable means to achieve them compared with existing alternatives. Administrators at academic institutions should work with representative faculty members and, where relevant, their unions to establish a strategy for MOOC engagement and a process for production or adoption of MOOCs. Data collection protocols should be devised up front to measure relevant indicators both prior to and following MOOC engagement in order to permit an objective assessment of whether stated goals are being met. Costs of such initiatives can be estimated by tracking personnel time commitments with respect to MOOC-related efforts and accounting for displacement of other productive activities, such as teaching or grant writing. These costs can be compared with the evidence of goal achievement to judge whether MOOCs are a worthwhile investment of limited educational resources.
References
Allen, E., & Seaman, J. (2013). Changing course: Ten years of tracking online education in the United States. Babson Survey Research Group Report. Retrieved from
http://sloanconsortium.org/publications/survey/changing_course_2012
Allen, E., & Seaman, J. (2014). Grade change: Tracking online education in the United States. Babson Survey Research Group Report. Retrieved from
http://sloanconsortium.org/publications/survey/grade-change-2013
Alliance for Higher Education and Democracy (AHEAD) at the University of Pennsylvania. (2014, April). What’s AHEAD key trends in education Poll #1: Massive open online courses (MOOCs). Retrieved from http://www.gse.upenn.edu/pdf/ahead/whats_ahead/01_moocs.pdf
Bacow, L. S., Bowen, W. G., Guthrie, K. M., Lack, K. A., & Long, M. P. (2012). Barriers to adoption of online learning systems in U.S. higher education. Ithaka S+R. Retrieved from http://www.sr.ithaka.org/research-publications/barriers-adoption-online-learning-systems-us-higher-education
Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., & Seaton, D. T. (2013). Studying learning in the worldwide classroom: Research into edX’s first MOOC. Research & Practice in
Assessment 8, 13–25. Retrieved from http://www.rpajournal.com/studying-learning-in-the-worldwide-classroom-research-into-edxs-first-mooc/
Bruff, D. O., Fisher, D. H., McEwen, K. E., & Smith, B. E. (2013). Wrapping a MOOC: Student perceptions of an experiment in blended learning. Journal of Online Teaching and Learning, 9(2). Retrieved from https://my.vanderbilt.edu/douglasfisher/files/2013/06/JOLTPaperFinal6-9-2013.pdf
Buffardi, K., & Edwards, S. H. (2014). Introducing CodeWorkout: An adaptive and social learning environment. Proceedings of the 45th ACM Technical Symposium on Computer Science Education, 724. Abstract retrieved from http://dl.acm.org/citation.cfm?doid=2538862.2544317 Cambre, J., Kulkarni, C., Bernstein, M. S., & Klemmer, S. R. (2014). Talkabout: Small-group discussions
in massive global classes. Retrieved from
https://hci.stanford.edu/publications/2014/PeerStudio/LAS2014-CambreTalkabout.pdf Champaign, J., Fredericks, C., Colvin, K., Seaton, D., Liu, A., & Pritchard, D. (2014, March).
Correlating skill and improvement in 2 MOOCs with a student’s time on task. Paper presented at Learning@Scale Conference, Atlanta, GA. Retrieved from
http://dx.doi.org/10.1145/2556325.2566250
Cheal, C. (2012, August 14). Creating MOOCs for college credit [Research bulletin]. Louisville, CO: EDUCAUSE Center for Applied Research. Retrieved from http://www.educause.edu/ecar
Christensen, G., Steinmetz, A., Alcorn, B., Bennett A., Woods, D., & Emanuel, E. J. (2013). The MOOC phenomenon: Who takes massive open online courses and why? [Working paper]. University of Pennsylvania. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2350964 Cima, M. J. (2014). A mastery-based learning and assessment model applied to 3.091r (Introduction to
solid-state chemistry) [Internal MIT report].
Coetzee, D., Fox, A., Hearst, M. A., & Hartmann, B. (2014, February). Should your MOOC forum use a reputation system? Proceedings of the 17th ACM Conference on Computer-Supported
Collaborative Work, Baltimore, MD. Retrieved from http://dl.acm.org/citation.cfm?doid=2531602.2531657
Colvin, K. F., Champaign, J., Liu, A., Fredericks, C., Zhou, Q., & Pritchard, D. E. (in press). Learning in an introductory physics MOOC: All cohorts learn equally, including an on-campus class. The International Review of Research in Open and Distance Learning.
Cormier, D., & Siemens, G. (2010). Through the open door: Open courses as research, learning, and engagement. EDUCAUSE Review, 45(4), 30–39. Retrieved from
http://www.educause.edu/ero/article/through-open-door-open-courses-research-learning-and-engagement
DeBoer, J., Ho, A., Stump, G., & Breslow, L. (2014). Changing “course”: Reconceptualizing educational variables for massive open online courses. Educational Researcher, 43(2), 74–84. doi: 10.3102/0013189X14523038.
Dede, C. (Ed.). (2013). Connecting the dots: New technology-based models for postsecondary learning.
EDUCAUSE Review, September/October 2013.
Dernoncourt, F., Taylor, C., O’Reilly, U., Veeramachaneni, K., Wu, S., Do, C., & Halawa, S. (2013, December). MoocViz: A large scale, open access, collaborative, data analytics platform for MOOCs. Paper presented at NIPS Workshop on Data-Driven Education, Lake Tahoe, Nevada. Retrieved from
http://groups.csail.mit.edu/EVO-DesignOpt/groupWebSite/uploads/Site/MoocViz.pdf
Downes, S. (2008). Places to go: Connectivism & connective knowledge. Innovate, 5(1). Retrieved from http://www.innovateonline.info/index.php?view=article&id=668
Fain, P. (2013, January 1). Paying for proof. Inside Higher Ed. Retrieved from http://www.insidehighered.com/news/2013/01/09/courseras-fee-based-course-option
Firmin, R., Schiorring, E., Whitmer, J., Willett, T., & Sujitparapitaya, S. (2013). Preliminary summary SJSU+ Augmented Online Learning Environment pilot project. Retrieved from http://www.sjsu.edu/chemistry/People/Faculty/Collins_Research_Page/AOLE Report -September 10 2013 final.pdf
Ghadiri, K., Qayoumi, M. H., Junn, E., Hsu, P., & Sujitparapitaya, S. (2013). The transformative potential of blended learning using MIT edX’s 6.002x online MOOC content combined with student team-based learning in class. JUCE (Japanese Universities Association for Computer Education) Journal, 3.
Griffiths, R. (2013). MOOCs in the classroom? Ithaka S+R. Retrieved from http://www.sr.ithaka.org/sites/default/files/files/S-R_BriefingPaper_Moocs_20131028.pdf
Grover, S., Franz, P., Schneider, E., & Pea, R. (2013). The MOOC as distributed intelligence: Dimensions of a framework & evaluation of MOOCs. Paper presented at the 10th International Conference on Computer Supported Collaborative Learning, Madison, U.S.A. Retrieved from
http://lytics.stanford.edu/wordpress/wp-content/uploads/2013/04/Framework-for-Design-Evaluation-of-MOOCs-Grover-Franz-Schneider-Pea_final.pdf
Halawa, S., Greene, D., & Mitchell, J. (2014). Dropout prediction in MOOCs using learner activity features. Proceedings of the European MOOC Stakeholder Summit (EMOOCS 2014), Lausanne, Switzerland. Retrieved from http://www.stanford.edu/~halawa/cgi-bin/files/emoocs2014.pdf Ho, A. D., Reich, J., Nesterko, S. O., Seaton, D. T., Mullaney, T., Waldo, J., & Chuang, I. (2014).
HarvardX and MITx: The first year of open online courses [HarvardX and MITx Working Paper No. 1]. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2381263
Hollands, F. M., & Tirthali, D. (2014). MOOCs: Expectations and reality. Full report. Center for Benefit-Cost Studies of Education, Teachers College, Columbia University, NY. Retrieved from http://cbcse.org/wordpress/wp-content/uploads/2014/05/MOOCs_Expectations_and_Reality.pdf Ithaka S+R. (2013). Interim report: A collaborative effort to test MOOCs and other online learning
platforms on campuses of the University System of Maryland. Retrieved from http://www.sr.ithaka.org/sites/default/files/reports/S-R_Moocs_InterimReport_20131024.pdf Ithaka S+R. (2014). Interactive online learning on campus: Testing MOOCs and other platforms in
hybrid formats in the University System of Maryland. Retrieved from
http://www.sr.ithaka.org/research-publications/interactive-online-learning-on-campus
Kizilcec, R., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. Proceedings of the Third International Conference on Learning Analytics and Knowledge, pp.170–179. Retrieved from http://www.stanford.edu/~cpiech/bio/papers/deconstructingDisengagement.pdf
Kizilcec, R. F., Schneider, E., Cohen, G. L., & McFarland, D. A. (2014, March). Encouraging forum participation in online courses with collectivist, individualist, and neutral motivational framings.
eLearning Papers, 37, 13–22. ISSN: 1887–1542. Retrieved from http://rene.kizilcec.com/wp-content/uploads/2013/02/kizilcec2014encouraging2014elearning.pdf
Kulkarni, C., Koh, P. W., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013). Peer and self-assessment in massive online classes. ACM Transactions on Computer-Human Interactions, 9(4) Article 39, 31 pages. Retrieved from
http://www.cogsci.ucsd.edu/~mboyle/COGS1/readings/Klemmer-COGS1-Peer%20and%20self%20assesment%20in%20massive%20online%20classes.pdf
LeCompte, M. D., & Schensul, J. J. (1999). Analyzing and interpreting qualitative data. Walnut Creek, CA: Altamira Press.
Lewin, T. (2013, December 10). After setbacks, online courses are rethought. New York Times. Retrieved from
http://www.nytimes.com/2013/12/11/us/after-setbacks-online-courses-are-rethought.html?_r=0
Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: A systematic study of the published literature 2008–2012. The International Review of Research in Open and Distance Learning, 14(3), 202–227. Retrieved from
http://www.irrodl.org/index.php/irrodl/article/view/1455
Markoff, J. (2011, August 15). Virtual and artificial, but 58,000 want course. The New York Times. Retrieved from http://www.nytimes.com/2011/08/16/science/16stanford.html
Means, B., Bakia, M., & Murphy, R. (2014). Learning online: What research tells us about whether, when and how. New York, NY: Routledge.
Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. San Francisco, CA: Jossey-Bass.
Nguyen, A., Piech, C., Huang, J., & Guibas, L. (2014). Codewebs: Scalable homework search for massive open online programming courses. Proceedings of the 23rd International World Wide Web Conference, Seoul, Korea. Retrieved from
http://www.stanford.edu/~jhuang11/research/pubs/www14/nphg-www14.pdf
Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Koller, D. (2013). Tuned models of peer assessment in MOOCs. Proceedings of The 6th International Conference on Educational Data Mining, Memphis, TN. Retrieved from
http://www.stanford.edu/~jhuang11/research/pubs/edm13/edm13.pdf
Veeramachaneni, K., Dernoncourt, F., Taylor, C., Pardos, Z., & O'Reilly, U. (2013). MOOCdb:
Developing data standards for MOOC data science. Proceedings of the 1st Workshop on Massive Open Online Courses at the 16th Annual Conference on Artificial Intelligence in Education, Memphis, TN. Retrieved from
http://edf.stanford.edu/sites/default/files/Verramachaneni%20et%20al.%202013.pdf
Waldrop, M. M. (2013). Online learning: Campus 2.0. Nature, 495, 160–163. Retrieved from http://www.nature.com/news/online-learning-campus-2-0-1.12590
