University of Southern California Dissertations and Theses

Assessment, accountability & accreditation: a study of MOOC provider perceptions

(USC Thesis Other)

Assessment, accountability & accreditation: a study of MOOC provider perceptions

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content
ASSESSMENT, ACCOUNTABILITY & ACCREDITATION:
A STUDY OF MOOC PROVIDER PERCEPTIONS

by

Richard May

A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION

August 2015

Copyright 2015 Richard May

ii
ACKNOWLEDGEMENTS
There are many people who I would like to thank for helping me complete my
dissertation. First, I am thankful to the faculty, staff and fellow students I worked with at the
University of Southern California’s Rossier School of Education. I am especially grateful to my
dissertation committee – Drs. Robert Keim, P.J. Woolston, and Patricia Tobey - for their
guidance throughout this process. Dr. Keim provided clarity as I shifted through topics and study
designs; Dr. Woolston passed on his incredible knowledge of accreditation; and Dr. Tobey
provided non-stop positive reassurance.
Secondly, I owe gratitude to my family who supported me throughout my doctoral
pursuit. I am thankful to my brothers - David, Michael, and Christopher – for their continual
encouragement; my parents for instilling the importance of education in me; and my parents-in-
laws for their constant support. I’d also like to thank my son Carter for staying up and hanging
out with me on late night writing sessions. Finally, I am most thankful to my wife Stephanie. Her
patience and optimistic outlook everyday helped me complete this dissertation.

iii
TABLE OF CONTENTS

ACKNOWLEDGEMENTS ii

LIST OF TABLES v

ABSTRACT vi

CHAPTER ONE: OVERVIEW OF THE STUDY 1
History of MOOCs 2
Argument for the Study 7
Statement of the Problem 7
Purpose of the Study 8
Importance of the Study 8

CHAPTER TWO: LITERATURE REVIEW 10
MOOCs 10
Understanding and Defining MOOCs 12
Business Models 14
Assessment Tools 15
Criticisms of MOOCs 19
Accreditation 22
History of Accreditation 22
International Accreditation Practices 30
Alternatives to Accreditation 33
Effects of Accreditation on Assessment 36
MOOCs and Accreditation 48

CHAPTER THREE: METHODOLOGY 54
Review of Research Questions 54
Method of Study 54
Sample and Population 56
Data Collection and Instrumentation 57
Data Collection and Analysis 57
Reliability and Validity 58
Limitations and Delimitations 58

CHAPTER FOUR: RESULTS 60
Participants 60
Non-Participation 61
Results 61
Research Question #1 (Assessment) 63
Research Question #2 (Accountability) 66
Research Question #3 (Accreditation) 69
Other Themes 73
Document Analysis 75

iv
Summary 78

CHAPTER FIVE: DISCUSSION OF FINDINGS 79
Summary of Findings 79
Implications for Practice 80
Determining Quality 80
Expectations of Teaching and Learning 81
Current Accreditation Practices 82
Tools Needed for Accreditation 83
Alternatives to Accreditation 84
Future Research 84
Conclusions 85

REFERENCES 87

v
LIST OF TABLES
Table 1: MOOC Provider Profiles 61
Table 2: Overview of Themes and Findings 62

vi
ABSTRACT
This qualitative study explored how Massive Open Online Course (MOOC) providers
perceive the concepts of assessment, accountability, and accreditation. Seven MOOC providers
were interviewed via Skype, telephone and email correspondence. Their responses were
transcribed, coded and analyzed to identify common themes. Respondents perceived MOOCs as
part of a larger shift in education towards online learning, which requires assessment to become
more student-centered. Regarding accountability, students are the primary stakeholders. By
focusing on direct measures of student success, MOOCs will potentially decouple traditional
measures of higher education prestige and rankings. Regarding accreditation, MOOCs could be
institutionally accredited when they are delivered as part of a degree program, and directly
accredited when used for professional education. Additional research is needed to determine how
accrediting bodies can better incentivize MOOCs to participate in the accreditation process. The
outcomes from this study can inform educational leaders, policy makers, and accrediting bodies
on how MOOC providers themselves wish to be assessed.

1
CHAPTER ONE: OVERVIEW OF THE STUDY
American institutions of higher education have been increasingly scrutinized over the
past decade. Tuition costs have outpaced inflation for nearly twenty years (Baum & Ma, 2013).
Gaps in educational outcomes persist for ethnic and racial minority groups (Bensimon, 2005).
Federal and state financial support has gradually eroded, blurring the notion of public higher
education (Ikenberry, 2006). One consequence of these criticisms is increased scrutiny of the
accreditation process for higher education institutions.
Accreditation is a process of external quality review created and used by higher education
to scrutinize colleges, universities and programs for quality assurance and quality improvement
(Eaton, 2012b). Critics argue the current accreditation system impedes potential innovations
needed to address rising costs and access issues of higher education (L. M. Burke & Butler,
2012; Robert C. Dickeson, 2006). While criticism of the accreditation process is not necessarily
a new phenomenon (Powell, 2013), recent calls for accreditation reform are more pronounced.
These issues, coupled with a swift increase of for-profit colleges, requires a rethinking of
traditional measures of quality and assessment (Van Der Werf & Sabatier, 2009).
In his 2013 State of the Union address, President Obama asked Congress to “change the
Higher Education Act, so that affordability and value are included in determining which colleges
receive certain types of federal aid” (Obama, 2013c). The address was followed by The
President’s Plan for a Strong Middle Class and a Strong America, which suggested achieving
the above change to the Higher Education Act “either by incorporating measures of value and
affordability into the existing accreditation system; or by establishing a new, alternative system
of accreditation that would provide pathways for higher education models and colleges to receive
federal student aid based on performance and results.” (Obama, 2013b). Furthermore, in August

2
2013, President Obama called for a performance-based rating system that would connect
institutional performance with financial aid distributions (Obama, 2013a). While the
accreditation process may need some altering to incorporate new innovations, new innovative
solutions must first be proposed.
Currently, the most disruptive innovation in higher education that attempts to address
both rising costs and lack of access in higher education is the Massive Open Online Course
(MOOC) (Ch & Popuri, 2013; Christensen, Horn, Caldera, & Soares, 2011; Economist, 2013b;
Friedman, 2013; Horn & Christensen, 2013; Karsenti, 2013). While often compared to traditional
distance education, MOOCs differ in a few regards. First, they can reach a vast population of
students in one class. Secondly, their new business models are drastically different than current
financial models in higher education. The following section will review their brief history in
higher education.
History of MOOCs

The emergence of MOOCs can be viewed as an evolutionary growth out of two trends in
higher education. First, distance education and online learning initiatives expanded in higher
education over the last decade. In parallel, open education initiatives began to form (Kazakoff-
Lane, 2014). From these two trends emerged the MOOC. The first wave of (cMOOCs) focused
on a learner centered knowledge model and the principle of connectivism. The second wave
(xMOOCs) focused on scalable teacher centered instruction (Yuan & Powell, 2013). The
following sections will discuss the brief history of MOOCS, focusing on the precursors (distance
education and open initiatives), common MOOC types, and current MOOC demographics.

3
Precursors to MOOCs
Distance Education is a formal educational process in which the majority of the
instruction in a course occurs when students and instructors are not in the same place. While the
exact beginning of distance education is debated, correspondence courses beginning the mid
1800s are considered the first formal distance education course. Correspondence education refers
requires sending of materials back and forth between the instructor and student. The interaction
between the instructor and the student is limited, and is primarily initiated by the student. As
such, the courses are typically self-paced (Marques, 2014).
In the 1900’s, new delivery mediums for educational information emerged with the
creation of radio and then television. These inventions allowed for one teacher or instructor to
reach a broader, scalable audience of learners. In 1969, The Open University was established in
the United Kingdom with the mission of providing widening participation in higher education
through distance education. In 1970, Canada created a similar institution by establishing
Athabasca University (Byrne, 1989). These two universities promoted the ideals of open
educational access.
However it was the creation of computers and the Internet that rapidly expanded the use
of distance education in higher education. Online education is a form of distance education
delivered via the Internet. As of 2013, over 21 million students were enrolled in an online course
(Wasik, 2014). The ideals of open education fused with new Internet technologies lead to the
creation of Open Educational Resources (OER).
Hylén (2005) defines OERs as “digitized materials offered freely and openly for
educators, students and self-learners to use and re use for teaching, learning and research” (p. 1).
These resources are not necessarily organized in a structured format. Rather, they provide

4
content that can be used as desired and accessed via the Internet. In contrast, OpenCourseWare
(OCR) is a type of OER that organizes online material in a course format. Therefore all OCWs
are OERs, but not all OERs are OCWs (Terrell & Caudill, 2012).
In 2001, MIT’s OpenCourseWare (OCW) established a platform for institutions to create
course content freely available online. This course content is published under an open license
and can be reused by anyone. As of March 2014, the MIT OCW website averages 1 million
visits each month, mostly from self-learners (43%) and students (42%). In 2006, The Khan
Academy, a website set up by a former hedge fund analyst, began posting short videos of
lectures dedicated to various academic and practical subjects.
Created in 2008, the OpenCourseWare Consortium is a community of over 250 higher
education institutions. With the primary aim of advancing free, open, high-quality educational
materials, over 13,000 courses in 20 languages have been published on the consortium’s website.
With increased technical capacity to connect and monitor students, the online environment
became ripe for connecting students to content, their peers and instructors at a new massive
scale. A few professors began experimenting with new modes of class delivery called the
MOOC.
Emergence of MOOCs
The first MOOC, entitled “Connectivisim and Connectivist Knowledge”, was offered by
the University of Manitoba in 2008. While over 2,000 students enrolled, of perhaps more
importance were the principles of connectivism (autonomy, openness and interactivity) which
guided the course delivery (Kazakoff-Lane, 2014). Rather than being taught in a linear fashion,
students engaged the course content from various theme areas. While exploring these topics, the
students relied on peers for feedback, and created new, openly available educational content

5
based on the wisdom of the crowd. This style of course delivery would later be labeled a
cMOOC. The first waves of MOOCs were generally constructed under the principles of
cMOOCs, and mostly distributed by Canadian universities. (Ebben & Murphy, 2014).
In 2011, a Stanford professor offered a MOOC on artificial intelligence using a course
designed for around the principles of scalability. This class used technology to facilitate a
“technology enriched teacher centered [aka knowledge-transmission model of ] instruction” that
came to be known as an xMOOC, rather than the learner-centered knowledge construction
model (cMOOC) (Yuan & Powell, 2013). By 2012, that Stanford professor left academia to start
Udacity, a MOOC provider. Two other Stanford professors would also give up their tenure to
create Coursera, a private company that currently stands as the leading MOOC provider. Near
the same time, MIT and Harvard joined to create edX, a non-profit MOOC provider (Watters,
2013).
Since 2012, numerous more companies have formed, often with the backing of
considerable venture capital funding. Notable groups included Canvas Network, Udemy, and
FutureLearn. In the news media, MOOCs are arguably the most debated topic in higher
education. The New York Times hailed 2012 as the “Year of the MOOC” (Pappano, 2012).
However, public opinion is still divided on their utility (Abeywardena, 2014). Academic forums
of education and technology experts have generated intense debate regarding the role of MOOCs
in higher education (Day, 2014). These debates are creating serious tension among leadership of
higher education institutions (Stripling, 2012).
University presidents and chancellors are publically staking their positions on the value
of MOOCs in the future of higher education. Many are incorporating them into existing course
offerings, albeit for no credit. However, a small group of universities are taking strong stands

6
against their use in higher education (Boren, 2012; Nickias, 2012). Provosts from 15 large public
universities released a joint position paper that emphasizes the need to infuse academic values
and pedagogical principles in emerging educational technologies such as MOOCs (CIC, 2013).
A leading concern includes whether they genuinely increase access to underrepresented students
(Block, 2014).
MOOC Demographics

Obtaining current metrics on MOOC provider offerings is difficult due to their rapid
expansion. In 2013, MOOCs enrolled over 4.5 million students (Wasik, 2014). Shah (2013)
provides a comprehensive breakdown of market share and subject distribution by course. As of
December 2013, Coursera remains the largest MOOC provider, offering nearly half of all
MOOCs. The next largest providers are edX and Cancas Networks – each at 8%. Despite
Coursera’s large course distribution, their market share is slowly being reduced as new providers
emerge all over the world.
Shah (2013) also reviews the proportion of MOOC offerings by subject. The first courses
mostly concentrated on computer science and engineering subjects. But from 2011-2013, subject
offerings began to include more breadth. Humanities courses comprised of 20% of all offerings,
followed by computer science and programming (16%), business and management (15%),
science (11%), and health/medicine (11%). Nearly three quarters are offered in English. But with
the rise of international MOOC providers, non-english based offerings are quickly growing.
Leading MOOC providers in other languages include Miridia X (Spanish), XuetangX (Mandarin
Chinese), Iversity (German) and France Universite Numerique (French). In addition, ease of
creating new MOOCs will likely increase in the upcoming years. For example, MOOC.org is a
partnership between Google and edX to create a platform for future course designers.

7
Argument for the Study
The rise of multiple open educational initiatives (OERs, OCW, etc) has happened in a
rapid, unorganized manner. As such, a unified policy agenda from the accrediting agencies and
the U.S. government towards open initiatives remains lacking. In contrast, European nations
have a more developed policy framework for open educational initiatives (e-InfraNet, 2013). For
example, The European Network for Coordination of Policies and Programmes in e-
Infrastructure has made significant progress on creating a unified policy framework that also
maps onto a European Union educational agenda (Corrall & Pinfield, 2014). With so many
colleges and universities experimenting with MOOCs, U.S. accreditors will be increasingly
forced to assess them in future reviews. For this reason, it is important to develop a better
understanding of how to properly assess quality in MOOCs for higher education through
empirical research.
Statement of the Problem

The current problem is a lack of consensus on how to assess MOOCs for integration into
higher education. While principles of assessment remain the same in an online environment, the
manner in which assessment is implemented does change (Rovai, 2000). At the same time,
accreditors are struggling to encourage new innovative practices in higher education (Spangehl
& Lindborg, 2012). Input from the all stakeholders involved is needed to move this discourse
forward.
Perceptions of MOOCs have been provided by instructors (Kolowich, 2013b), students
(Bruff, Fisher, McEwen, & Smith, 2013), and even potential employers (Radford et al., 2014).
However, no formal survey has been given to MOOC provider’s regarding their perceptions of
their place in higher education. Currently only blog posts, individual press releases and public

8
appearances by MOOC provider CEOs exist. This study addresses this gap in research by
conducting a survey of MOOC providers regarding their perceptions of MOOCs’ place in higher
education.
Purpose of the Study

The purpose of this study was to understand how MOOC providers view the concepts of
assessment, accountability, and accreditation. In order to develop this understanding the study
focused on the following research questions:
• What are MOOC provider perceptions regarding how MOOCs should be assessed?
• What are MOOC provider perceptions regarding how MOOCs should be held
accountable, and to what stakeholders?
• What are MOOC provider perceptions regarding how MOOCs should be integrated into
the current U.S. accreditation system of higher education?
Importance of the Study
The Council for Higher Education Accreditation (CHEA) states MOOCs are the most
“conspicuous, challenging and controversial” innovation facing higher education (Eaton, 2012a).
The traditional higher education model could change if a MOOC based program of study leads to
a degree from an accredited institution. Therefore, addressing the quality of the learning
experience in these environments is of the utmost importance (Mazoue, 2013). However, before
accreditation agencies can properly assess MOOCs, research is needed on what constitutes
appropriate metrics of quality.
Five pressing questions were asked by CHEA regarding MOOCs and accreditation: a)
Through what lens do we examine MOOCs for quality? b) Do MOOCs require altering
expectations of teaching and learning? c) Does current accreditation review address the key

9
features of MOOCs? d) If accreditation will address MOOCs, what tools do accreditors need? e)
If accreditation is not an appropriate venue to address quality for MOOCs, what is an alternative
version of quality review? (Eaton, 2012a). These five questions provide an outline for accreditors
to start reviewing what are appropriate mechanisms for quality assurance and improvement.
In addition, The National Institute for Learning Outcomes Assessment (NILOA) argues
that student learning outcomes assessment is the key to MOOCs future. This mirrors a larger
trend among regional accrediting commissions of increased focus across higher education on
student learning outcomes (Provezis, 2010). However before MOOCs can realize their potential,
various assessment issued need to be addressed. These include a) standards regarding peer
grading and feedback b) plagiarism detection tools c) proctoring of final exams d) validity of
robo-grading and e) establishment of sustainable business models (Boston & Helm, 2012).
Overseas, The Higher Education Academy in the United Kingdom acknowledges MOOCs as a
potentially significant shift towards increased digital education. Future policy efforts should
investigate the role of accreditation for MOOC, and possible implications for teaching (Bayne &
Ross, 2013).
There is strong evidence that online education will continue to be one of the fastest
growing markets in higher education. Ensuring a quality supply of online courses is needed to
meet the increasing student demand (Moloney & Oakley, 2010). This study explored the proper
assessment metrics and paths for accreditation for MOOCs as perceived by MOOC providers.
The outcomes from this study will inform educational leaders, policy makers, and accrediting
bodies on how MOOCs themselves wish to be assessed.

10
CHAPTER TWO: LITERATURE REVIEW

The following review will survey the body of literature that has been written on MOOCs
and accreditation. The first section will review the existing literature on MOOCs. Topics include
the evolution of MOOC types, MOOC business models, MOOC assessment techniques for
students, and criticisms of MOOCs. The second section will review accreditation in higher
education. Topics include accreditation’s long history, international accreditation practices,
alternatives to accreditation, and accreditations’ effect on student assessment. The third section
reviews literature and events surrounding MOOCs’ potential inclusion into current accreditation
processes.
MOOCs

Existing Literature Reviews of MOOCs

Despite the short history of MOOCs, researchers are beginning to conduct systematic
literature reviews to understand their impact and place in education. This section reviews
existing MOOC literature reviews, and common themes that emerge from these studies. The
most common themes will guide the direction of this section. To highlight the quick emergence
of MOOCs, the existing literature reviews will be discussed in the chronological order they were
published – a time period of ten months.
Existing studies. In March 2013 The Centre for Educational Technology and
Interoperability and Standards (CETIS), a non-profit in the United Kingdom, reviewed MOOCs
and their implications for higher education (Yuan & Powell, 2013). While written from a U.K.
perspective, this report is largely informed by MOOCs from the U.S. and reviews literature from
non-academic sources (reports, blogs, and press releases). The report first argues that MOOCs
contain key characteristics of a disruptive innovation (an innovation that helps create a new

11
market and eventually disrupts an existing market). The report points out potential implications
for two distinct groups. First, higher education institutions will need to explore new business
models and innovative teaching practices. Secondly, policy makers should embrace MOOCs and
open learning initiatives to ensure affordability and access for future students.
In July 2013, Liyanagunawardena, Adams and Williams (2013) published the first
scholarly review of MOOC literature. After reviewing forty-five peer-reviewed papers from
2008-2012, each paper was placed into one of eight categories: introductory, concept, case
studies, educational theory, technology, participant focused, and other. Liyanagunawardena et al.
(2013) point out three major gaps in the literature. First there is no consensus on how to define
different types of MOOCs. Second, proper assessment of learners is still vague. The outcomes of
this research will directly impact how MOOCs are paired with accreditation. Finally, most
existing research focuses on the learner’s perspective. There is little published research focusing
on the MOOC provider’s perspective.
In September 2013, The United Kingdom’s Department for Business, Innovation and
Skills (BIS) released a literature review of MOOCs and other online distance learning (Haggard,
2013). After reviewing 100 authors, the literature is divided into three categories: a) MOOC
perspectives from educational institutions and learners, b) Formal surveys, and c) Articles from
the press and other journalistic sources. The BIS report recommends future research to
understand how disruptive MOOCs will become in higher education. In addition, policy
development around them should focus on issues of accreditation and pedagogical innovation.
In January 2014, Ebben and Murphy (2014) reviewed MOOC literature from nine leading
academic databases. Using a chronological framework, they categorize their history in two
distinct phases. Phase One, from 2009-2012, encompasses the development of cMOOCs as a

12
technology platform, and the concept of Connectivism as a learning theory. Phase Two, from
2012-2013, focuses on the rise of xMOOCs and use of learning analytics for assessment. In
addition, a critical discourse regarding MOOCs is beginning to emerge among academics.
Common themes. Four themes emerged across these existing literature reviews. First,
the definition and classification of MOOCs is still debated as they continually evolve. Second,
potential business models remain unproven and mostly hypothetical at the present time. Third,
the assessment tools for MOOCs are still evolving. Fourth, there are strong criticisms of MOOCs
from various stakeholders. These four themes frame the remainder of this literature review on
MOOCs.
Understanding and Defining MOOCs

The term MOOC is popularly used in discussions of online learning, but often misused
and misunderstood. This section will focus on the complexity that arises when using this term.
Specifically, it will discuss the differences between MOOC types, newly emerging MOOCs and
their various classifications, and arguments for future classification frameworks.
Classifying MOOCs. Despite a short history, MOOCs are becoming increasingly
difficult to classify. There has been a rapid rise the new acronyms to describe them. Examples
include the BOOC (big open online course), DOCC (distributed open collaborative course),
MOOA (massive open online administrations), MOOD (massive open online discussion), SMOC
(synchronous massive online course), and SPOC (small private online courses) (Watters, 2013).
The need and utility of these new acronyms are increasingly debated. As a consequence,
researchers are beginning to create organizational frameworks for the naming of MOOCs.
Rosselle, Caron, and Heutte (2014) provide a succinct history of attempts to define and
classify MOOCs. They review four separate attempts to classify MOOCs. First was the

13
distinction between cMOOCs (focused on connectivism) and xMOOCs (focused on transfer and
scalability). Second, the term “tMOOC” was introduced, which encompasses MOOCs that focus
on tasks the learners have to perform. Third, eight types of MOOCs were defined based on
pedagogy. Finally, researchers asked what dimensions are opened or closed from the learner’s
point of view. While cMOOCs allow all dimensions to be open, an xMOOC is mostly closed.
Therefore a new MOOC type was proposed that resides in the middle of the continuum of
openness – called the iMOOC.
Unfortunately these classifications are still vague and create confusion (Rosselle et al.,
2014). So in response, more frameworks are being created to simplify or streamline how
MOOCs are classified. Conole (2013) categorizes them based on key characteristics, and
specifically uses the 7Cs of Learning Design Framework (Conceptualize, Capture,
Communicate, Collaborate, Consider, Combine, and Consolidate). Schneider (2013) argues,
“each MOOC needs to be characterized in terms of its subject matter, audience, and use” (p. 5).
She hopes to develop a shared language for both researchers and MOOC designers. Her
taxonomy balances a scalable approach to classifying with an appreciation for capturing the
pedagogical approaches and content of each course. Rosselle et al. (2014) also attempt to create
an initial typology, framed by the “Hy-Sup” (p. 5) model of describing blended learning systems
used in higher education. Whether these proposals are simplifications to the existing
classifications, or just more additions, is unclear.
One obvious theme from these proposals is a lack of consensus. And the potential for
future MOOCs to adapt in real time to their students need creates even more complexity
(Sonwalkar, 2013). What are the leading factors in determining a classification system? Whose
point of view matters the most (teacher, learner, course designer)? How do you create

14
classification boundaries that capture future, not-yet invented MOOC designs? How does
assessment influence these naming systems? These questions must be further discussed before
further classification systems are developed.
Business Models

As MOOC types remain dynamic and rapidly evolving, so do their potential business
models. The early MOOCs were originally created with the intent of offering a course for free to
learners, and upon completion a learner could earn a certificate or credential of some kind saying
they passed the course by paying a small fee. The Oxford dictionary defines MOOCs as “a
course of study made available over the Internet without charge to a very large number of
people”, but this definition is already slightly outdated. Just as MOOC types and intent have
evolved, so have their potential business models (Economist, 2013a).
Leading MOOC providers are still experimenting with how to create a sustainable
revenue stream. One company (Coursera) provides certificates to those who complete courses for
a small fee, and licenses course materials to universities. Another group (Udacity) is working
directly with companies to train current and future employees. And another (edX) is selling its
technology to universities, in turn allowing universities to create their own MOOCs. The
potential to use MOOC platforms to deliver advertising space, while criticized by many
academics, is very appealing to venture capitalists (Economist, 2013a). Despite these options, a
proven path towards financial sustainability remains missing.
One constant is the concern institutions of higher education have regarding MOOCs
impact on traditional revenue streams. Gallup (2013) surveyed over a thousand college and
university chief academic officers. Forty-seven percent of the respondents agreed or strongly
agreed that MOOCs could threaten the business model of their respective institutions. Some

15
universities with existing online education programs are quick to point out they already have a
sustainable business model, and see no reason to dabble in the MOOC space (Nickias, 2012).
However, MOOCs are really part of a larger trend of partnerships between non-profit and for-
profit sectors to deliver higher education services online.
Bundled service providers (BSPs) are for-profit companies that help institutions create
and expand online programs, and in turn, get a portion of tuition from enrolled students. In 2013
alone, over 200 non-profit institutions of higher education partnered with a for-profit service
provider. This number is expected to double by 2015 (Leblanc, 2013). Large companies such as
Pearson, John Wiley & Sons, and Blackboard have entered this lucrative market. New companies
like Learning House, Academic Partnerships, and 2U have emerged as major players too. These
BSPs need accredited universities to access students and their tuition streams; universities and
colleges want BSPs to handle the complexity of creating online delivery platforms and expand
marketing towards potential new students.
Current business models of higher education are already being disrupted by these new
online offerings. But MOOCs providers are just of the larger trend of colleges and universities
partnering with outside companies to deliver educational content. And while the business models
are widely discussed, MOOCs largest impact may not be a financial change in higher education.
In the long run, their biggest impact may be the potential flipping of previous pedagogical
approaches in higher education (Lawton & Katsomitros, 2012).
Assessment Tools
As MOOCs attempt to change business models of higher education, they also may
change pedagogical approaches for delivering instruction. Online education has given rise to
blended learning, where online resources complement formal in-class education. Taken a step

16
further, students using online materials can enter a flipped classroom. Instead of listening to a
lecture in class, and going home to apply the new material in homework assignments, the student
can now listen or watch the lecture at home, and come into class to apply this new knowledge
(Economist, 2013b). Now MOOCs have the potential to continue this trend of pushing pedagogy
towards a flipped classroom model (Bruff et al., 2013).
In addition, MOOCs can use their own students as an experimental lab for new
instructional practices. By combing through analyze large data sets of student interaction,
researchers can observe, change and improve various pedagogical approaches given the specific
needs of the student population (Koller, Ng, Do, & Chen, 2013). Unfortunately, models for
assessing their own student learning, which will be essential to accreditation, are not fully
developed (Haggard, 2013).
New assessment tools. Jordan (2013) argues that future assessment of MOOCs will in
general require increased electronic assessment (e-assessment). As the formal boundaries
between teaching, assessment and learning are blurred, there’s an increased need for a variety of
automated assessment tools. E-assessment helps fill this demand, with use of techniques such as
peer-assessment and assessed e-portfolios, blogs, wikis and online forums. In addition, there will
be an increased reliance on the use of learning analytics to shift through increasingly large
datasets.
Currently MOOCs often rely on machine or peer grading tools to evaluate student’s
work. A survey of instructors by The Chronicle of Higher Education indicated 74% use machine
or automated grading, and 34% used peer grading. While 67% of instructors found machine
grading to be very reliable, only 25% believed peer grading to be very reliable (Kolowich,
2013b). In an attempt to determine validity of peer assessments, one study reviewed student

17
paper’s grades assigned by teaching assistants versus self-evaluated student grades. Minimal
differences existed between the two sets of grades (Wilkowski, Russell, & Deutsch, 2014). One
downside to automated grading is the lack of evidence on how to properly assess writing.
Humanities and creative writing courses are unable to use machine grading techniques with a
high degree of accuracy (Balfour, 2013).
With this in mind, research is beginning on how to personalize automated responses to
students to improve learning. One approach to personalizing assessment is the creation of a
“feedback generator”. This automated software toolkit aims to provide feedback at a scalable
level for any number of students that’s comparable to feedback from a traditional course. The
primary concept guiding the creation of this new toolkit to support assessment and feedback is
using intended learning outcomes to guide the feedback generators design (Whitelock, Gilbert, &
Wills, 2013).
Authentication and proctoring. With so many students enrolled in a MOOC, issues of
authentication and proctoring are increasingly important. For MOOCs to become formally
integrated into traditional higher education, steps must be taken to ensure identities of students
are verified during and upon completion of each course. Proctoring is a method for students to
verify their successful completion. Sandeen (2013) points out that because of cost, proctoring
occurs for summative assessments in one of two ways. One method requires the student to
complete an exam at a physical testing center (library, university, etc). The second method
involves a webcam monitoring a student throughout the entire exam. Once a MOOC is
successfully completed, student’s can receive some sort of credential to show others. The most
popular credential awarded is a digital badge.

18
Badging. Digital badges are increasingly used to certify achievement in a specific topic
or skill. While digital badges are visually nothing more than a simple image file (such a PNG
file), they represent attainment of knowledge or skills in a competency area. These virtual
certifications are developed with flexibility in mind to allow organizations the ability to
individualize badge requirements according to their own needs. Perhaps because of this open
framework, badges are rapidly affecting human resource policies of businesses in how they
recruit and hire new employees (Schrage, 2012).
The largest badge provider for MOOCs is The Open Badge Infrastucture (OBI), a project
sponsored by the Mozilla Corporation. Launched in September 2011, the primary goal of OBI is
to create a free, open standards system to recognize and verify learning. Using OBI, any
organization can create, issue and verify digital badges using their own standards. This allows
individuals to gather multiple badges from various sources or educational providers, and easily
compile them into a virtual repository, referred to as a “backpack”. Badges can be displayed in
different online locations, such as personal websites and social networks. (Laso, Peco, & Luján-
Mora, 2013).
But the badging community is not yet unified, as other platforms exist to earn badges.
Groups including MITx, Khan Academy, University of California, Davis and many other land-
grant universities have spent years working on their own badging systems (Carey, 2012). While
numerous organizations are dabbling in how to create their own badging requirements, the
concept of “open” remains consistent. And by allowing any organization to inquire, learn about,
and participate in the badging process, traditional universities and colleges are increasingly
affected. While badges do not yet represent the equivalent of bachelors and masters degrees,
badges have the potential to augment credentialing requirements of future employers. In

19
addition, this open-source alternative may erode traditional universities’ monopoly on degree
conference in higher education.
However, there remain many unresolved issues of badging and its potential effect on
students learning. Given their novelty, they are still not widely accepted as legitimate credentials
for skills and education. Though employers have expressed interest in badging, there is little
research to show that this interest has actually changed recruitment and hiring policies. And
perhaps most relevant to this study is the criticism regarding the true value of a badge. While
reputation and accreditation drives the value of university and college degrees, a badge can be
conferred from an unknown or unaccredited source (EDUCAUSE, 2012). This is just one of
many recurrent criticisms regarding MOOCs.
Criticisms of MOOCs

While MOOCs have garnered considerable interest for their potential in delivering
education to a wide audience at low cost, they have also received considerable criticism. The
critiques can be grouped into the following areas: a) Low completion rates of MOOC students, b)
No measurable increase in access of quality education for underserved students, c) MOOCs are
only useful for specific employer job training requests.
Completion rates. Perhaps the most common criticism of MOOCs is their low
completion and retention rates. Jordan (2014) pooled existing data in the public domain to
provide a comprehensive view of factors affecting enrollment and completion. She claims the
average MOOC enrolls 43,000 students with a completion rate of 6.5%. Completion rates are
negatively correlated with course length, but remain consistent across university rank and total
class size.

20
The University of Pennsylvania conducted a study on a million users through sixteen
Coursera courses from 2012-2013. The most shocking finding low course completion rates,
averaging of 4% across all courses (Stein, 2013). Adamopoulos (2013) argues that self-paced
courses have a negative effect on completion, compared to courses that follow a specific
timetable. He notes the most important factor in online course retention is the quality of the
professor(s). Herman (2014) notes that despite differences in methods and study design for the
studies above, low completion rates for MOOCs are a consistent finding.
However, some argue that completion rates need to be viewed in a proper context. The
founders of Coursera believes retention should first be viewed in the context of learner intent
(Koller et al., 2013). They argue students enter MOOCs for different purposes, and given these
different motivations, low retention is often a reasonable outcome. But this viewpoint is not held
by all MOOC providers, some of which see low completion rate as a disturbing trend. Sebastian
Thrun, founder of Udacity, called MOOCs a “lousy product” after a failed partnership between
Udacity and San Jose State University (Chafkin, 2013).
Limited access. The Campaign for the Future of Higher Education (CFHE) argues
MOOCs do not address two realities in education - the digital divide and the online achievement
gap (CFHE, 2013). First, the digital divide refers to those who have reliable access to the Internet
versus those who do not. High quality education cannot be reliably delivered to citizens that lack
broadband Internet access. In addition to the digital divide, research consistently shows these
same students also experience an online achievement gap (Figlio, Rush, & Yin, 2010; Jaggars &
Xu, 2013; Kaupp, 2012).
In fact, the majority of students benefitting from MOOCs already have post-secondary
credentials. Emanuel (2013) reviewed data from a July 2013 survey of participants in 32

21
MOOCs. Out of nearly 35,000 respondents, 83% already had a two or four year post-secondary
degree, and 44% reported education beyond a bachelor’s degree. Of participants in other
countries including Brazil, Russia, China, and India, almost 80% come from the most well
educated 6% of the population.
Other nations are beginning to assess how MOOCs have been implemented in their own
countries, and hope to create best practices to improve educational access (O’Connor, 2014;
Rizzardini, Gütl, Chang, & Morales, 2014; Torres et al., 2013). Perhaps time is needed to
understand how MOOCs and online resources can reach those learners who need it most. But as
currently delivered, they disproportionally assist high-achieving students.
Another form of employer training. Some critics argue MOOCs impact won’t even
reach the average higher education student. Rather, they have little utility past gaining specific
competencies and completing employer-specific courses. This does not mean that traditional
education institutions should ignore MOOCs or other open educational resources. Rather,
MOOCs will never be a direct competitor to institutions of higher education for degree-seeking
students (de Langen & van den Bosch, 2013)
New studies are increasingly critical of the real impact MOOCs are having in higher
education. Baggaley (2014) argues the previous widespread acceptance of them has been more a
publicity myth than reality. A team of researchers from the United Kingdom reviewed their
current place in America and Europe, and pointed out numerous concerns including: a) Lack of
evidence regarding impact, b) Absence of pedagogy, c) Lack of support for learners, and d)
Unrealistic expectations of new learners (Boyatt, Joy, Rocks, & Sinclair, 2013). The Wall Street
Journal gave an early report card to MOOCs, identifying multiple areas that still need to be
improved upon (Fowler, 2013).

22
At the same time studies are increasingly critical of MOOCs in higher education, other
reports show future employers may shape the future of MOOCs given their interest in scalable
training solutions. Duke University and RTI International conducted an online survey of 398
employers in the state of North Carolina regarding their knowledge and experiences with
MOOCs (Radford et al., 2014). Most of the employers were highly receptive to the potential role
MOOCs may play in future professional development training. The respondents noted they could
play a positive role in recruiting and hiring decisions.

Accreditation
Note: The “Accreditation” section of Chapter 2 was jointly authored. Nathan J. Barlow
developed the section entitled “History of Accreditation”. Kristopher Tesoro drafted the section
entitled “Alternatives to Accreditation”. Ban Dimapindan and Win Shih collaboratively
constructed the section entitled “Effects of Accreditation on Assessment”. The researcher, before
inclusion in this study, edited all jointly authored sections.
In contrast to MOOCs, the current accreditation system in higher education has a long
history. This section will review accreditation in higher education. Topics include accreditation’s
long history, international accreditation practices, alternatives to accreditation, and
accreditations’ effect on student assessment.
History of Accreditation
Accreditation has a long parentage among the universities and colleges of the United
States dating back to the self-initiated external review of Harvard in 1642. This external review,
done only six years after Harvard’s founding, was intended to ascertain rigor in its courses by
peers from universities in Great Britain and Europe (Brittingham, 2009; Davenport, 2000). This

23
type of self-study is not only the first example in America of peer-review, but it also highlights
the need for self and peer regulation in the U.S. educational system due to the lack of federal
governmental regulation (Brittingham, 2009).
While the federal government does not directly accredit educational institutions, the first
example of an accrediting body was through a state government. In 1784 the New York Board of
Regents was established as the first regionally organized accrediting organization. The Board
was set up like a corporate office with the educational institutions being franchisees. The Board
created mandated standards that had to be met by each college or university if that institution was
to receive state financial aid (Blauch, 1959).
Not only did Harvard pioneer accreditation in the U.S. with its early external review of its
own courses, but the president of Harvard University initiated a national movement in 1892
when he organized and chaired the Committee of Ten, which was an alliance formed among
educators (mostly college and university presidents) to seek standardization regarding
educational philosophies and practices in the U.S. through a system of peer approval (Shaw,
1993).
Around this same time there began to be different associations and foundations that
undertook an accreditation review of educational institutions in the U.S. based on their own
standards. Associations such as the American Association of University Women, the Carnegie
Foundation, and the Association of American Universities would evaluate various institutions
and generate lists of approved or accredited schools. These associations were responding to the
desire of their constituents to have accurate information regarding the validity and efficacy of the
different colleges and universities (Orlans, 1974; Shaw, 1993).

24
Regional accreditation, 1885 to 1920. When these associations declined to broaden or
continue their accrediting practices, individual institutions began to unite together to form
regional accrediting bodies to assess secondary schools’ adequacy in preparing students for
college (Brittingham, 2009). Colleges were then measured by the quality of students they
admitted based on standards at the secondary school level that were measured by the accrediting
agency. The regional accrediting agencies began to focus also on creating a list of colleges that
were good destinations for in-coming freshmen. If an institution was a member of the regional
accreditation agency, it was considered an accredited college; or more precisely the institutions
that belonged to an accrediting agency were considered colleges while those that did not belong
were not (Blauch, 1959; Ewell, 2008; Orlans, 1974; Shaw, 1993).
Regional accrediting bodies were formed in the following years: New England
Association of Schools and Colleges (NEASC) in 1885, the Middle States Association of
Colleges and Secondary Schools (MSCSS and Middle States Commission on Higher Education
[MSCHE]) in 1887, the North Central Association of Colleges and Schools (NCA) and the
Southern Association of Colleges and Schools (SACS) in 1895, the Northwest Commission on
Colleges and Universities (NWCCU) in 1917, and finally the Western Association of Schools
and Colleges (WASC) in 1924 (Brittingham, 2009).
Regional accrediting associations began to create instruments for the purpose of
establishing unity and standardization in regards to entrance requirements and college standards
(Blauch, 1959). For example, in 1901 MSCHE and MSCSS created the College Entrance
Examination Board to standardize college entrance requirements. The NCA also published its
first set of standards for its higher education members in 1909 (Brittingham, 2009).

25
Although there were functioning regional accreditation bodies in most of the states, in
1910 the Department of Education created its own national list of recognized (accredited)
colleges. Because of the public’s pressure to keep the federal government from controlling
higher education directly, President Taft blocked the publishing of the list of colleges and the
Department of Education discontinued the active pursuit of accrediting schools. Instead, it
reestablished itself as a resource for the regional accrediting bodies in regards to data collection
and comparison (Blauch, 1959; Ewell, 2008; Orlans, 1974).
Regional accreditation, 1920-1950. With the regional accrediting bodies in place, the
ideas of what an accredited college was became more diverse (e.g. vocational colleges,
community colleges). Out of the greater differences among schools in regards to school types
and institutional purposes, there arose a need to apply more qualitative measures and a focus on
high outcomes rather than minimum outcomes (Brittingham, 2009). School visits by regional
accreditors became necessary once a school demonstrated struggles, since qualitative standards
became the norm. The regional organizations began to measure success (and therefore grant
accredited status) on whether an institution met its own standards outlined in its own mission,
rather than a predetermined set of criteria. The accreditation process later became a requirement
for all member institutions. Self and peer-reviews became a standard part of the accreditation
process, and were undertaken by volunteers from the member institutions (Ewell, 2008).
Accrediting bodies began to be challenged as to their legitimacy in classifying colleges as
accredited or not. The Langer Case in 1938 is a landmark case that established the standing of
accrediting bodies in the United States. Governor William Langer of North Dakota lost in his
legal challenge of the NCA’s denial of accreditation to North Dakota Agricultural College. This
ruling carried over to other legal cases that upheld the decision that accreditation was legitimate

26
as well as a voluntary process (Fuller & Lugg, 2012; Orlans, 1974).
In addition to the regional accrediting bodies, there arose other associations that were
meant to regulate the accrediting agencies themselves. The Joint Commission on Accrediting
was formed in 1938 to validate legitimate accrediting agencies and discredit questionable or
redundant ones. After some changes to the mission and the membership of the Joint Commission
on Accreditation, the name was changed to the National Commission on Accrediting (Blauch,
1959).
History of accreditation 1950 to present. The period 1950 to 1985 has been coined the
golden age of higher education and was marked by increasing federal regulations. During this
period, key developments in the accreditation process included standardization of self-study, site
visits by colleagues from peer institutions, and institutions visited on a regular cycle (Woolston,
2012). With the passage of the Veterans' Readjustment Assistance Act of 1952, the U.S.
Commissioner of Education was required to publish a list of recognized accreditation
associations (Bloland, 2001). This act provided for education benefits to veterans of the Korean
War directly rather than to the educational institution being attended, increasing the importance
of accreditation as a mechanism for recognition of legitimacy (Woolston, 2012). A more pivotal
event occurred in 1958 with the National Defense Education Act's (NDEA) allocation of funding
for NDEA fellowships and college loans (Weissburg, 2008). The NDEA limited participating
institutions to those that were accredited (Gaston, 2013).
In 1963, the U.S. Congress passed the Higher Education Facilities Act. This act required
that higher education institutions receiving federal funds through enrolled students be accredited.
Arguably the most striking expansion in accreditation's mission coincided with the passage of the
Higher Education Act in 1964 (Gaston, 2013). Title IV in this legislation expressed the intent of

27
Congress to use federal funding to broaden access to higher education. According to Gaston
(2013), having committed to this much larger role in encouraging college attendance, the federal
government found it necessary to affirm that institutions benefitting from such funds were
worthy of it. That same year, the National Committee of Regional Accrediting Agencies
(NCRAA) became the Federation of Regional Accrediting Commissions of Higher Education
(FRACHE).
In 1965, the Higher Education Act was first signed into law. That law strengthened the
resources available to higher education institutions and provided financial assistance to students
enrolled at those institutions. The law was especially important to accreditation because it forced
the U.S. Department of Education to determine and list a much larger number of institutions
eligible for federal programs (Trivett, 1976). In 1967, Parsons College lost its accreditation due
to administrative weaknesses and a $14 million debt. The college appealed, but the courts
denied it on the basis that the regional accrediting associations were voluntary bodies (Woolston,
2012).
The need to deal with a much larger number of potentially eligible institutions led the
office of Education's U.S Commissioner of Education to create in 1968 the Bureau of Higher
Education the Accreditation and Institutional Eligibility Staff (AIES). The purpose of the AIES
was to administer the federal recognition and review process involving the accrediting agencies
(Dickey & Miller, 1972). In 1975, the National Committee on Accrediting (NCA) and FRACHE
merged to form a new organization called the Council on Postsecondary Accreditation (COPA).
The newly created national accreditation association encompassed an astonishing array of types
of postsecondary education to include community colleges, liberal arts colleges, proprietary

28
schools, graduate research programs, bible colleges, trade and technical schools, and home-study
programs (Chambers, 1983).
Since 1985, accountability has become the issue of paramount importance in the field of
education. According to Woolston (2012), key developments in the accreditation process during
this period include higher education experiencing rising costs resulting in high student loan
default rates as well as accreditation enduring increasing criticism for a number of apparent
shortcomings, most ostensibly a lack of demonstrable student learning outcomes. Similarly,
accreditation is increasingly and formally defended by various champions of the practice. For
example, congressional hostility reached a crisis stage in 1992 when Congress, in the midst of
debates on the reauthorization of the Higher Education Act, threatened to bring to a close the role
of the accrediting agencies as gatekeepers for financial aid. During the early 1990s the federal
government grew increasingly intrusive in matters directly affecting the accrediting agencies
(Bloland, 2001). As a direct consequence, Subpart 1 of Part H of the Higher Education Act
amendments involved an increase role for the states in determining the eligibility of instructions
to participate in the student financial aid programs of the aforementioned Title IV. For every
state, this meant the creation of a State Postsecondary Review Entity (SPRE) that would review
institutions that the USDE secretary had identified as having triggered such review criteria as
high default rates on student loans (Bloland, 2001). The SPREs were short lived and in 1994
were abandoned largely because of a lack of adequate funding. The 1992 reauthorization also
created the National Advisory Committee on Institutional Quality and Integrity (NACIQI) to
replace the AIES.
For several years, the regional accrediting agencies had entertained the idea of pulling out
of COPA and forming their own national association. Based on dissatisfaction with the

29
organization, regional accrediting agencies proposed a resolution to terminate COPA by the end
of 1993. Following a successful vote on the resolution, COPA was effectively terminated. A
special committee, generated by the COPA plan of dissolution of April 1993, created the
Commission on Recognition of Postsecondary Accreditation (CORPA) to continue the work of
recognizing accrediting agencies. However, CORPA was formed primarily as an interim
organization to continue national recognition of accreditation. In 1995, national leaders in
accreditation formed the National Policy Board (NPB) to shape the creation and legitimation of a
national organization overseeing accreditation. The national leaders in accreditation were
adamant that the new organization should reflect higher education's needs rather than those of
postsecondary education. Following numerous intensive meetings, a new organization named the
Council for Higher Education Accreditation (CHEA) was formed in 1996 as the official
successor to CORPA (Bloland, 2001). In 1996, the Spellings Commission (2006) argued that
accreditation has significant shortcomings and is both ineffective and a barrier to innovation.
Since the release of the Spellings Commission, the next significant statement on the
subject of accreditation came during a State of the Union Address by President Barack Obama
(Obama, 2013c). In conjunction with the president's address, the White House released a nine-
page document titled "The President's Plan for a Strong Middle Class and a Strong America".
The document stated that the President was going to call on Congress to consider value,
affordability, and student outcomes in making determinations about which colleges and
universities receive access to federal student aid, either by incorporating measures of value and
affordability into the existing accreditation system; or by establishing a new, alternative system
of accreditation that would provide pathways for higher education models and colleges to receive
federal student aid based on performance and results (Obama, 2013b).

30
International Accreditation Practices

Distance and online education can easily span across national borders. It’s therefore
important to understand the complexity in aligning international accreditation practices. The
United States has no doubt developed its own unique accreditation process (Brittingham, 2009).
The most obvious difference between the U.S. and other countries is in the way education is
governed. In the U.S. education is governed at the state level whereas other nations are often
governed by a ministry of education (Ewell, 2008; Middaugh, 2012). Dill (2007) outlined three
traditional models of accreditation. These include “the European model of central control of
quality assurance by state educational ministries, the US model of decentralized quality
assurance combining limited state control with market competition, and the British model in
which the state essentially ceded responsibility for quality assurance to self-accrediting
universities” (p. 3). These models have been used in some form by other nations in South
America, Africa, and Asia. Historically, direct government regulation (the European model) of
higher education has been the most prevalent form of institutional oversight outside of the United
States (Dickeson, 2009).
Unfortunately the low level of autonomy historically granted to post-secondary
institutions has limited their ability to effectively compete against institutions in the United
States and other countries (Dewatripont, van Pottelsberghe, Sapir, & Veugelers, 2010; Jacobs &
Van der Ploeg, 2006; Sursock & Smidt, 2010). Overall, European institutions struggle with poor
governance, and are offer insufficient incentives for academics to focus on research
(Dewatripont et al., 2010). Many European countries have a “very centralized” system of higher
education, such as France, Germany, Italy, and Spain (Van der Ploeg & Veugelers, 2008). In
addition, the level of governmental intervention inhibits European universities from innovating

31
and reacting quickly to changing demands (Van der Ploeg & Veugelers, 2008).
Institutions in Europe with low levels of autonomy have historically had little to no
control in areas including hiring faculty, managing budgets, and setting wages (Aghion,
Dewatripont, Hoxby, Mas-Colell, & Sapir, 2008). Thus, it is difficult for universities with low
autonomy to attract and retain the faculty needed to compete for top spots in global ranking
indices (Aghion et al., 2008; Dewatripont et al., 2010; Jacobs & Van der Ploeg, 2006).
However, some European nations have conducted serious reform to their higher
education systems, including Denmark, Netherlands, Sweden, and the United Kingdom. Not
surprisingly, universities with high autonomy in these countries have higher levels of research
performance compared to European countries with low levels of institutional autonomy
(Dewatripont et al., 2010). This sentiment is echoed by Aghion et al. (2008), who argues
research performance (which impacts academic prestige and rankings) is negatively impacted by
less institutional autonomy.
While research on accreditation’s direct impact on student learning outcomes is sparse,
Jacobs and Van der Ploeg (2006) argue the European system of greater regulation has some
benefits. They concluded that institutions in continental Europe had better access for students
with lower socioeconomic status, better outcomes in terms of student completion, and even
lower spending per student.
Internationalization of accreditation. Due to globalization, there is an increased focus on how
to assure quality of standards in higher education across nations. Assessment frameworks are
being initiated and modified to meet these increased demands for accountability (Salmi, 2003).
Recent studies have tried to compare these assessment trends across multiple countries.

32
Bernhard (2012) conducted a comparative analysis of such reforms in six countries
(Austria, Germany, Finland, the United Kingdom, the United States, and Canada). Stensaker and
Harvey (2006) identified a growing trend that nations are relying on forms of accreditation
distinctly different from the U.S. accreditation processes. Specifically, they identified the
academic audit as an increasingly used alternative in countries such as Australia and Hong Kong.
Yung-chi Hou (2013) examined challenges the Asia-Pacific region faces in implementing quality
standards that cross national boundaries.
Another outcome of globalization is the internationalization of the quality-assurance
process itself. Rather than each nation setting it’s own assessment frameworks, international
accords are attempting to bridge academic quality issues between nations. Student mobility
across national borders has driven the need for cooperative international accreditation networks
(Van Damme, 2000). There are many loosely or unconnected initiatives that have formed over
the last decade.
The United Nations Educational, Scientific and Cultural Organization (UNESCO) began
the discussion on guidelines for international best practices in higher education (Reinalda &
Kulesza-Mietkowski, 2005). The International Network for Quality Assurance Agencies in
Higher Education (INQAAHE) is a network of quality assurance agencies aimed to help ensure
cross-border quality assurance measures. Public-policy led initiatives in Europe include the
establishment of the European Standards and Guidelines for quality assurance in higher
education, which falls into the framework of the Bologna Process. (Cremonini, Epping,
Westerheijden, & Vogelsang, 2012). The CHEA International Quality Group (CIQG) provides a
forum to discuss quality assurance issues in an international context.

33
In conclusion, the U.S. system of accreditation has served as a model for higher
education assessment worldwide. Nonetheless, there is considerable difference in how other
nations govern quality assurance. While internationalization of the higher education accreditation
process will continue to increase, the precise frameworks used to achieve cross-national quality
standards remains undetermined. For the immediate future, nations will continue to use their own
frameworks for accreditation. International accreditation processes may eventually supersede
these existing frameworks, but not anytime soon.
Alternatives to Accreditation

As the role of accreditation has been thrust into the public spotlight within the United
States, it is important to review the alternatives to the current system that have been proposed in
previous years. Generally speaking, the alternatives to accreditation that have been proposed by
scholars or administrators in the past revolved around the common theme of increased
government involvement (either at the state or federal government level). To illustrate this notion,
Orlans (1974) described the development (at the national level) of a Committee for Identifying
Useful Postsecondary Schools that would ultimately allow accrediting agencies to focus on a
wider range of schools. This committee was part of Orlans’s plan to increase the amount of
competition amongst accrediting agencies in order to further the advancement of education.
Trivett (1976) stated that there was a triangular relationship between accrediting agencies, state
governments, and federal governments:
In its ideal form, the states establish minimum legal and fiscal standards, compliance with
which signifies that the institutions can enable a student to accomplish his objectives
because the institution has the means to accomplish what it claims it will do. Federal
regulations are primarily administrative in nature. Accrediting agencies provide depth to

34
the evaluation process in a manner not present in either the state or federal government’s
evaluation of an institution by certifying academic standards. (pg. 7)
Trivett’s statement speaks to the ever-present relationship between accreditation agencies, state
governments, and federal governments.
Harcleroad (1976, 1980) identified six different methods for accreditation in his writings;
three vouched for an expansion of responsibility for state agencies, one called for an expansion of
federal government responsibility, and the remaining two asked for a modification of the present
system (by increasing staff members or auditors) or keeping the present system in place.
Harcleroad (1976, 1980) wrote that:
A combination of the second (present system with modifications) and third options
(increased state agency responsibility without regional and national associations) seems
the most likely plan for the near future. This possibility will become even more viable if
both regional and national associations continue refinements in their process and increase
the objectivity of an admittedly subjective activity. (pg. 46)
These methods clearly demonstrate a preference for increased state government involvement
within the accreditation process. Harcleroad (1976) also spoke about the use of educational
auditing and accountability as an internal review to increase both external accountability and
internal quality. This concept is modeled after the auditing system developed by the Securities
and Exchange Commission (SEC) that was used to accredited financial organizations
(Harcleroad, 1976).
Another example of internal and external audits was demonstrated by the proposals in the
essay produced by three scholars (Graham, 1995). The essay (also known as the Mellon report)
was the result of a grant funding the study of accountability of higher education institutions to

35
their three major constituencies - students, government, and the public (Bloland, 2001). This
essay emphasized the notion that accountability had both an internal and external aspect and the
authors suggested that institutions conduct internal reviews (primarily within their teaching and
research units) every 5-10 years. Once this internal review was completed, an external review
would then be conducted in the form of an audit on the procedures of the internal review.
Specifically, this external audit would be conducted by regional accrediting agencies while
institutional accrediting agencies were encouraged to pay close attention to the internal processes
in order to determine if the institution has the ability to learn and address its weaknesses (Bloland,
2001). These concepts surrounding auditing were later explored by other authors and, most
recently, have been linked to discussions regarding the future of higher education accreditation
(Bernhard, 2012; Burke, 2005; Dill, Massy, Williams, & Cook, 1996; Ikenberry, 2009).
In examining alternatives to accreditation, it is important to note the alternative programs
that have been established by regional accreditors as enhancements to current accreditation
processes. For example, the Higher Learning Commission (an independent commission within
the North Central Association of Colleges and Schools) established (in 2000) an alternative
assessment for institutions that have already been accredited: the Academic Quality Improvement
Program (AQIP). According to Spangehl (2012), this process instilled the notion of continuous
quality improvement through the processes that would ultimately provide evidence for
accreditation. An example of AQIP offering continuous improvement for higher educational
institutions would be its encouragement of institutions to implement the use of various categories
(i.e. the Helping Students Learn, category allows for institutions to continuously monitor their
ongoing program and curricular design) to stimulate organizational improvement (Spangehl,
2012).

36
Another example of an alternative program is the use of the Quality Enhancement Plan
(QEP) by the Southern Association of Colleges and Schools (Jackson, Davis, & Jackson,
2010). The QEP was adopted in 2001 and defined as an additional accreditation requirement
that would help guide institutions to produce measurable improvement in the areas of student
learning. A few common themes of student learning that have been utilized by institutions
(through the use of QEP) include student engagement, critical thinking, and promoting
international tolerance (Jackson et al., 2010).
This section has offered a glimpse into the alternatives to accreditation that have been
proposed and implemented in the past. It is important to note that while accreditation has been
criticized by many, the general thoughts of many is that accreditation is a critical piece of
academia and vital to accomplishing the goal of institutional quality assurance and accountability
(Bloland, 2001). There is no consensus yet on whether MOOCs should be formally incorporated
into current accreditation practices or some other alternative.
Effects of Accreditation on Assessment
This section of the literature review will examine the effects of accreditation, focusing
primarily on the assessment of student learning outcomes. Specifically, outcome assessment
serves two main purposes — quality improvement and external accountability (Bresciani, 2006;
PT Ewell, 2009). Over the years, institutions of higher education have made considerable strides
with regard to learning assessment practices and implementation. Yet despite such progress, key
challenges still remain.
Trend toward learning assessment. The shift within higher education accreditation
toward greater accountability and student learning assessment began in the mid-1980s (Beno,
2004; Peter Ewell, 2001; Wergin, 2005, 2012). During that time, higher education was portrayed

37
in the media as costly and inefficient (Bloland, 2001). The impetus behind the public’s concern
stemmed from two reasons: first was the perception that students were underperforming
academically, and second was the demand of the business sector (Peter Ewell, 2001). Employers
and business leaders expressed their need for college graduates who could demonstrate high
levels of literacy, problem solving ability, and collaborative skills in order to support the
emerging knowledge economy of the 21
st
Century. In response to these concerns, institutions of
higher education started emphasizing student learning outcomes as the main process of
evaluating effectiveness (Beno, 2004).
Framework for learning assessment. Accreditation is widely considered to be a
significant driving force behind advances in both student learning and outcomes assessment.
According to Rhodes (2012), in recent years, accreditation has contributed to the proliferation of
assessment practices, lexicon, and even products such as e-portfolios, which are used to show
evidence of student learning.
Kuh and Ikenberry (2009) surveyed provosts or chief academic officers at all regionally
accredited institutions granting undergraduate degrees, and found that student assessment was
driven more by accreditation than by external pressures such as government or employers.
Another major finding was that most institutions planned to continue their assessment of student
learning outcomes despite budgetary constraints. They also found that gaining faculty support
and involvement remained a major challenge - an issue that will be examined in more depth later
in this section.
Additionally, college and university faculty and student affairs practitioners have stressed
how students must now acquire proficiency in a wide scope of learning outcomes to adequately
address the unique and complex challenges of today’s ever-changing, economically competitive,

38
and increasingly globalizing society. In 2007, the Association of American Colleges and
Universities published a report focusing on the aims and outcomes of a 21
st
Century collegiate
education, with data gathered through surveys, focus groups, and discussions with postsecondary
faculty. Emerging from the report were four “essential learning outcomes” which include: (1)
knowledge of human cultures and the physical and natural world, through study in science and
mathematics, social sciences, humanities, history, languages, and the arts; (2) intellectual and
practical skills, including inquiry and analysis, critical and creative thinking, written and oral
communication, quantitative skills, information literacy, and teamwork and problem-solving
abilities; (3) personal and social responsibility, including civic knowledge and engagement,
multicultural competence, ethics, and foundations and skills for lifelong learning; and (4)
integrative learning, including synthesis and advanced understanding across general and
specialized studies. With the adoption of such frameworks or similar tools at institutions,
accreditors can be well-positioned to connect teaching and learning and, as a result, better engage
faculty to improve student learning outcomes (Rhodes, 2012).
Benefits of accreditation on learning. Accreditation and student performance
assessment have been the focus of various empirical studies, with several pointing to benefits of
the accreditation process. Ruppert (1994) conducted case studies in 10 states – Colorado, Florida,
Illinois, Kentucky, New York, South Carolina, Tennessee, Texas, Virginia, and Wisconsin – to
evaluate different accountability programs based on student performance indicators. The report
concluded “quality indicators appear most useful if integrated in a planning process designed to
coordinate institutional efforts to attain state priorities” (p. 155).
Furthermore, research has also demonstrated how accreditation is helping shape
outcomes inside college classrooms. Specifically, Cabrera, Colbeck, and Terenzini (2001)

39
investigated classroom practices and their relationship with the learning gains in professional
competencies among undergraduate engineering students. The study involved 1,250 students
from seven universities. It found that the expectations of accrediting agencies may be
encouraging more widespread use of effective instructional practices by faculty.
A study by Volkwein, Lattuca, Harper, and Domingo (2007) measured changes in student
outcomes in engineering programs, following the implementation of new accreditation standards
by the Accreditation Board for Engineering and Technology (ABET). Based on the data
collected from a national sample of engineering programs, the authors noted that the new
accreditation standards were indeed a catalyst for change, finding evidence that linked the
accreditation changes to improvements in undergraduate education. Students experienced
significant gains in the application of knowledge of mathematics, science, and engineering;
usage of modern engineering tools; use of experimental skills to analyze and interpret data;
designing solutions to engineering problems; teamwork and group work; effective
communication; understanding of professional and ethical obligations; understanding of the
societal and global context of engineering solutions; and recognition of the need for life-long
learning. The authors also found accreditation also prompted faculty to engage in professional
development-related activity. Thus, the study showed the effectiveness of accreditation as a
mechanism for quality assurance (Volkwein et al., 2007).
Organizational effects of accreditation. Beyond student learning outcomes,
accreditation also has considerable effects on an organizational level. Procopio (2010) noted that
the process of acquiring accreditation influences perceptions of organizational culture.
According to the study, administrators are more satisfied than staff – and especially more so than
faculty – when rating organizational climate, information flow, involvement in decisions, and

40
utility of meetings. He also notes “these findings suggest institutional role is an important
variable to consider in any effort to affect organizational culture through accreditation buy-in”
(p. 10). Similarly, a study by Wiedman (1992) describes how the two-year process of
reaffirming accreditation at a public university drives the change of institutional culture.
Meanwhile, Brittingham (2009) explains that accreditation offers organizational-level
benefits for colleges and universities. The commonly acknowledged benefits include students’
access to federal financial aid funding, legitimacy in the public, consideration for foundation
grants and employer tuition credits, positive reflection among peers, and government
accountability. However, Brittingham (2009) points out that there are “not often recognized”
benefits as well (p. 18). For example, accreditation is cost-effective, particularly when
contrasting the number of personnel to carry out quality assurance procedures here in the U.S.
versus internationally, where it’s far more regulated. Second, “participation in accreditation is
good professional development” because those who lead a self-study come to learn about their
institution with more breadth and depth (p. 19). Third, self-regulation by institutions – if done
properly – is a better system than government regulation. And fourth, “regional accreditation
gathers a highly diverse set of institutions under a single tent, providing conditions that support
student mobility for purposes of transfer and seeking a higher degree” (p. 19).
Future assessment recommendations. Many higher education institutions have
developed plans and strategies to measure student learning outcomes, and such assessments are
already in use to improve institutional quality (Beno, 2004). For future actions, the Council for
Higher Education Accreditation, in its 2012 Final Report, recommends to further enhance
commitment to public accountability:

41
Working with the academic and accreditation communities, explore the adoption and
implementation of a small set of voluntary institutional performance indicators based on
mission that can be used to signal acceptable academic effectiveness and to inform
students and the public of the value and effectiveness of accreditation and higher
education. Such indicators would be determined by individual colleges and universities,
not government. (p. 7)
In addition, Brittingham (2012) outlines three developments that have the capacity to
influence accreditation and increase its ability to improve educational effectiveness. First,
accreditation is growing more focused on data and evidence, which strengthens its value as a
means of quality assurance and quality improvement. Second, “technology and open-access
education are changing our understanding of higher education” (p. 65). These innovations – such
as massive open online courses – hold enormous potential to open up higher education sources.
As a result, this trend will heighten the focus on student learning outcomes. Third, “with an
increased focus on accountability – quality assurance – accreditation is challenged to keep, and
indeed strengthen, its focus on institutional and programmatic improvement” (p. 68). This
becomes particularly important amid the current period of rapid change.
Challenges to student learning outcomes. Assessment is critical to the future of higher
education. As noted earlier, outcome assessment serves two main purposes – quality
improvement and external accountability (Bresciani, 2006; Ewell, 2009). The practice of
assessing learning outcomes is now widely adopted by colleges and universities since its
introduction in the mid-1980s. Assessment is also a requirement of the accreditation process.
However, outcomes assessment in higher education is still a work in progress and there is still a
fair amount of challenges (Kuh & Ewell, 2010).

42
Organization learning challenges. First, there is the organizational culture and learning
issue. Assessment, as clearly stated by the American Association for Higher Education (1992),
“is not an end in itself but a vehicle for educational improvement.” The process of assessment is
not a means unto its own end. Instead, it provides an opportunity for continuous organizational
learning and improving (Maki, 2010). Too often, institutions assemble and report sets of
mountainous data just to comply with federal or state accountability policy or accreditation
agency’s requirements. However, after the report is submitted, the evaluation team left, and the
accreditation confirmed, there are little incentives to act on the findings for further improvement.
The root causes of deficiencies identified are rarely followed up and real solutions are never
sought (Ewell, 2005; Wolff, 2005).
Another concern pointed out by Ewell (2005) is that accreditation agencies tend to
emphasize the process of, rather than the outcomes, once the assessment infrastructure is
established. The accreditors are satisfied with a formal statements and goals of learning
outcomes, but do not query further about how, the appropriateness, and to what degree these
learning goals are applied in the teaching and learning process. As a result, the process tends to
be a single loop learning where changes reside at a surface level, instead of a double-loop
learning, where changes are incorporated in the practices, belief, and norms (Bensimon, 2005).
Lack of faculty buy-in. Lack of faculty’s buy-in and participation is another hurdle in the
adoption of assessment practice (Kuh & Ewell, 2010). In a 2009 survey by the National Institute
for Learning Outcomes Assessment, two-third of all 2,809 surveyed schools noted that more
faculty involvement in learning assessment would be helpful (Kuh & Ikenberry, 2009).
According to Ewell (2005; 1993, 2002), there are several reasons that faculty want to be directly
involved in the assessment process. First, faculty views teaching and curriculum development

43
their domain. Assessing their teaching performance and student learning outcomes by external
groups can be viewed as an intrusion of their professional authority and academic freedom.
Second, the extra efforts and time required for engaging outcome assessment and the
unconvincing added-value perceived by faculty can be another deterrent. Furthermore, the
compliance-oriented assessment requirements are imposed by external bodies and most faculty
members participate in the process indirectly. They tend to show a lukewarm attitude and leave
the assessment work to administrative staff. In addition, faculty might have a different view on
the definitions and measures of “quality” than that of institution or accreditors (Perrault,
Gregory, & Carey, 2002) (p. 273). Finally, the assessment process incurs tremendous amount of
work and resources. To cut costs, the majority of the work is done by administration at the
institution. Faculty consequently perceives that assessment as an exercise performed by
administration for external audiences, instead of embracing the process.
Lack of institutional investment. Shortage of resources and institutional support is
another challenge in the implementation of assessment practice. As commented by Beno (2004),
“[d]eciding on the most effective strategies for teaching and for assessing learning will require
experimentation, careful research, analyses, and time” (p. 67). With continuously dwindling
federal and state funding in the last two decades, higher education, particularly at the public
institutions, is stripped of resources to support such an endeavor. A case in point is the recession
in early 1990s. Budget cuts forced many states to abandon the state assessment mandates
originated in mid-1980s and switched to process-based performance indicators as a way to gain
efficiency in large public institutions (Ewell, 2005). The 2009 National Institute for Learning
Outcomes Assessment survey shows that majority of the surveyed institutions undercapitalized
resources, tools, and expertise for assessment work. Twenty percent of respondents indicated

44
they had no assessment staff and 65% had two or less (Kuh & Ewell, 2010; Kuh & Ikenberry,
2009). The resource issue is further described by Beno (2004):
A challenge for community colleges is to develop the capacity to discuss what the results
of learning assessment mean, to identify ways of improving student learning, and to make
institutional commitments to that improvement by planning, allocating needed resources,
and implementing strategies for improvement. (p. 67)
Difficulty of integration into local practice. Integrating the value and institutionalizing
the practice of assessment into daily operations can be another tall order in many institutions. In
addition to redirecting resources, leadership’s involvement and commitment, faculty’s
participation, and adequate assessment personnel contribute to the success of cultivating a
sustainable assessment culture and framework on campus (Banta, 1993; Kuh & Ewell, 2010;
Maki, 2010). Furthermore, assessment activities, imposed by external authorities, tend to be
implemented as an addition to, rather than an integral part of, an institutional practice (Ewell,
2002). Assessment, like accreditation, is viewed as a special process with its own funding and
committee, instead of being part of regular business operations. Finally, the work of assessment,
program reviews, self-study, and external accreditation at institutional and academic program
levels tends to be handled by various offices on campus and coordinating the work can be
another challenge (Perrault et al., 2002).
Colleges also tend to adopt the institutional isomorphic approach by modeling itself after
those peers who are more legitimate or successful in dealing with similar situation and the
practice widely used to gain acceptance (DiMaggio & Powell, 1983). As reported by Ewell
(1993), institutions are prone to “second-guess” and adopt the type of assessment practice
acceptable by external agencies as a safe approach instead of adopting or customizing the one

45
appropriate to the local needs and situation (Ewell, 1993). Institutional isomorphism offers a
safer and more predictable route for institutions to deal with uncertainty and competition, to
confirm to government mandates or accreditation requirements, or to abide by professional
practices (Bloland, 2001). However, the strategy of following the crowd might hinder in-depth
inquiry of a unique local situation, as well as the opportunity for innovation and creativity.
Furthermore, decision makers may be unintentionally trapped in a culture of doing what
everyone is doing without carefully examining unique local situation, the logic, the
appropriateness, and the limitations behind the common practice (Miles, 2012).
Lack of assessment standards and clear terminology presents another challenge in
assessment and accreditation practice (Ewell, 2001). With no consensus on vocabulary, methods,
and instrument, assessment practice and outcomes can have limited value. As reported by Ewell
(2005), the absence of outcome metrics makes it difficult for state authorities to aggregate
performance across multiple institutions and to communicate the outcomes to the public. The
exercise of benchmarking is also difficult. Bresciani (2006) stressed the importance of
developing a conceptual definition, framework, and common language at institutional level.
Outcome equity. Outcome assessment that focuses on students’ academic performance
while overlooks the equity and disparity of diverse student population, as well as the student
engagement and campus climate issues is another area of concern. In discussing local financing
of community colleges, Dowd and Grant (2006) stressed the importance of including “outcome
equity” in additional to performance-based budget allocation. Outcome equity pays special
attention to the equal outcomes of educational attainment among populations of different social,
economic, and racial groups (Dowd, 2003).

46
Tension between improvement and accountability. The tension between the twin goals
of outcomes assessment, quality improvement and external accountability, can be another factor
affecting outcome assessment practice. According to Ewell (2008; 2009), assessment practice
has evolved over the years into two contrasting paradigms. The first paradigm, assessment for
improvement, emphasizes on constant evaluating and enhancing the process or outcomes, while
the other paradigm, assessment for accountability, demands conformity to a set of established
standards mandated by the state or accrediting agencies. The strategies, the instrumentation, the
methods of gathering evidences, the reference points, and the way results are utilized of these
two paradigms tend to be at the opposite end of the spectrum (2009; Ewell, 2008). For example,
in the improvement paradigm assessment is mainly used internally to address deficiencies and
enhance teaching and learning. It requires periodic evaluation and formative assessment to track
progress over time. On the other hand, the accountability paradigm assessment is designed to
demonstrate institutional effectiveness and performance to external constituencies and to comply
with pre-defined standards or expectations. The process tends to be performed on set schedules
as a summative assessment. The nature of these two constraints can create tension and conflict
within an institution. Consequently, an institution’s assessment program is unlikely to achieve
both objectives. Ewell (2009) further pointed out that “when institutions are presented with an
intervention that is claimed to embody both accountability and improvement, accountability
wins” (p. 8).
Transparency Challenges. Finally, for outcome assessment to be meaningful and
accountable the process and information need to be shared and open to the public (Ewell, 2005).
Accreditation has long been criticized as mysterious or secretive with little information to share
with stakeholders (Ewell, 2010). In a 2006 survey, the Council of Higher Education

47
Accreditation reported that only 18% of the 66 accreditors surveyed provide information about
the results of individual reviews publicly; less than 17% of accreditors provide a summary on
student academic achievement or program performance; and just over 33% of accreditors offer a
descriptive summary about the characteristics of accredited institutions or programs (Council of
Higher Education Accreditation, 2006). In the 2014 Inside Higher Education survey, only 9% of
the 846 college presidents indicate that it is very easy to find student outcomes data on the
institution’s website, and only half of the respondents agree that it is appropriate for federal
government to collect and publish data on outcomes of college graduates (Jaschik & Ledgerman,
2014). With the public disclosure requirements of the No Child Left Behind Act, there is an
impetus for higher education and accreditation agencies to be more open to public and policy
makers. It is expected that further openness will contribute to more effective and accountable
business practices as well as the improvement of educational quality.
Summary of accreditation and assessment. It has been three decades since the birth of
the assessments movement in U.S. higher education and a reasonable amount of progress has
been made (Ewell, 2005). Systematic assessment of student learning outcomes is now a
common practice at most institutions. The 2009 National Institute for Learning Outcomes
Assessment shows that more than 75% of surveyed institutions have adopted common learning
outcomes for all undergraduate students and most institutions conduct assessments at both the
instructional and program level (Kuh & Ikenberry, 2009).
As the public concern about the performance and quality of American colleges and
universities continues to grow, it is more imperative than ever to embed assessment in the
everyday work of teaching and using assessment outcomes to further improve practice, to inform
decision makers, to communicate effectively with the public, and to be accountable for preparing

48
the national learners in the knowledge economy. The ability of MOOCs providers to embrace
these traits is still not yet fully understood.

MOOCs & Accreditation

This final section of the literature review will briefly explore how MOOCs are currently
positioned for accreditation in higher education. After the initial wave of MOOCs, debates
quickly ensued of how to understand and assess quality (Adair, 2013). Afterall, the development
of mass educational resources online does not necessarily constitute a decline in quality (Lomas,
2002). To reinforce this point, there are many examples of distance online education programs
becoming successfully accredited in America. In 1996, Jones International University became
the first fully online university accredited by a regional accrediting association, and several
others have followed suit.
In 2014, The University of the People (a non-profit, tuition free online university)
received formal accreditation from the Distance Education and Training Council. This came just
in time for the first graduating class 7 students across the globe. Currently over 700 students are
enrolled from 142 countries, including 30% from Africa. Classes are purposefully low-tech with
text-based open source materials. This allows access to students without broadband or video
capabilities. Classes are structured in a 10-week format for 20 to 30 students at a time (Lewin,
2014). However MOOCs are unique in due to their large scope of students per class, which
increase the need for proven assessment techniques.
The MOOC Research Initiative is attempting to address this research gap by evaluating
their effectiveness. Created by the Bill and Melinda Gates Foundation, researchers are funded to
explore how MOOCs impact teachers, learners, and education in general (Grossman, 2013).

49
Overseas, European entities are beginning to define quality in MOOCs. The University of
London has outlined quality assurance processes for assessing their MOOC programs (Price,
2013). The MOOC Quality Project was recently created by the European Foundation for Quality
in E-Learning to also stimulate discourse on the issue of quality (EFQUEL, 2014). And the E-
xcellence methodology was established for assessing quality in e-learning, and could potentially
support future assessment by accreditation agencies (Rosewell, Kear, & Williams, 2012).
Endorsements for integrating MOOCs into formal academic credit are beginning to
emerge. The American Council on Education (ACE) is an influential association of higher
education institutions that advises university and college presidents across the nation. One
service they provide is recommending whether individual courses are suitable for academic
credit. Then ACE informs their member universities whether or not to accept credits from these
individual courses. In one case, ACE reviewed five MOOC to make a judgment on their
potential recognition for credit. Upon completion of the project, ACE recommended all five
courses for credit (Kolowich, 2013a).
Despite these events, institutions of higher education typically remain unsure about
integrating MOOCs into formal academic credit. How can these courses create quality learning
environments, and what mechanisms can be used properly to assess student outcomes? A few
assessment methods are discussed below, including use of proper design, and then the potential
of using prior learning or competency-based assessments.
Assessment for MOOC Design

Assessment can help inform MOOCs before they are even created and released.
Researchers are just beginning to understand how designers can affect quality of online learning
environments (Sharif, Moeini, & Gisbert, 2013). Given the various reasons students register for

50
MOOCS, course designers must create courses that meet this wide range of goals (Wilkowski,
Deutsch, & Russell, 2014). One of the most mature frameworks for the design and evaluation of
MOOCs is grounded in the theory of distributed intelligence (Grover, 2013).
Distributed intelligence refers to the notion that the resources shaping and enabling
learning can positively be influenced by those people involved in designing and taking the
MOOC. Using this framework, a MOOC contains four distinct dimensions across which
intelligence is distributed. First the Interactive Learning Environment (ILE) is shaped by the
course creators’ intent alongside the technology used for implementation. Second, the learner
background and intention captures the student’s intent for enrolling in a MOOC. Third, the
technology infrastructure drives potential use of web-based social media and delivery of content.
Finally, evidence-based improvement becomes increasingly important to determine how courses
can be improved upon with future iterations. Data mining and analytics are increasingly
important for producing the evidence-base which emerges upon completion of each MOOC
(Grover, 2013).
Prior Learning Assessment
Moss (2013) explores the role of using prior learning assessment as a potential
mechanism to recognize formal academic credit from MOOCs. Prior learning assessment
recognizes learning gained from experiences outside of postsecondary education. It is
increasingly relevant for assessing student’s knowledge as the distinction between formal and
informal classrooms settings are blurred from the rise of Open educational resources. In addition,
Moss (2013) argues this approach has already been explicitly supported by some MOOC
providers alongside educational experts. One downside however is a lack of cohesion in how
prior learning assessment mechanisms are currently implemented. In addition, prior learning

51
assessment is not easily scalable across different courses and disciplines. Nonetheless, it has
received positive endorsements from educational groups including the American Council on
Adult and Experiential Learning (Tate, 2013).
Competency-Based Assessment
Western Governors University is one of the most highly regarded online universities with
regional accreditation due to its innovative practices. Diverting from traditional course delivery,
Western Governors championed the concept of competency-based education (White &
Glickman, 2007). Competency-based education allows students to progress as they demonstrate
proficiency of an academic subject, regardless of time in the course or location of the student. At
Western Governors University, students pay a fixed fee per semester, and can then take as many
courses as they want. Instead of worrying about class attendance or time in class, students can
complete assessments whenever they are ready to prove mastery of course content.
Many institutions are experimenting with competency-based education programs,
including over twenty recently financed by the Lumina Foundation. This should come as no
surprise, given the Lumina foundation’s extensive investigation of how to properly measure
“credit-hours” (Adelman, Ewell, Gaston, & Schneider, 2011). Aside from philosophical debates
regarding the quality of competency-based education versus traditional mediums, there remain
practical hurdles. Moving from traditional online courses to MOOCs requires a larger
infrastructure for data management. Traditional techniques for conducting assessments are not
always applicable, and more research is needed to understand how to properly implement
competency-based assessments in MOOCs (Irakliotis & Johnstone, 2014)

52
Accreditor’s Role in Assessing MOOCs
Given the lack of evidence as to what is proper assessment for a MOOC, some prominent
groups argue that accreditation bodies should remain flexible in their assessment. The
President’s Council of Advisors on Science and Technology (PCAST) gave three specific
recommendations to the President. First, they recommend that market forces should drive online
teaching and learning innovations. Secondly, accrediting bodies should remain flexible in
response to these new educational innovations. And finally, the federal government should
support research and sharing of results on effective teaching and learning (Gates, Mundie, &
Jackson, 2013). The Accrediting Council for Independent Colleges and Schools (2013) makes
further recommendations for how accreditation supports these new innovations. These include
assessment based on peer review, balancing accountability with quality improvement,
participation from a broad stakeholder pool to preserve credibility.
Others have argued for a new alternative accreditation process to incorporate MOOCs
into higher education. Bergeron and Klinsky (2013) advocate the creation of a new private sector
accrediting body, the “Modern States Accrediting Agency”. In this new accrediting model,
specific courses could be accredited, not just the entire degree-granting institution. Students who
complete tuition-free online courses and pass a Modern States assessment could earn accredited
course hours from Modern States itself. Students could take a mix of accredited MOOC courses
and traditional courses at an accredited institution of higher education to earn a degree. This
model could pave the way for potential degree attainment by a student who never physically
steps into an accredited institution of higher education. Bergeron and Klinsky (2013) emphasize
that this model would require collaboration and support from current accrediting bodies to
become a reality.

53
A minority view suggests that the accreditation process is already capable of
incorporating new innovations in higher education, and there is no need for drastic change to the
current accreditation process. Kezar (2014) argues that accreditation supports evidence-based
innovation, and points out alternative accrediting systems could do more harm than good. The
increased presence of the federal government in the accreditation process will not accelerate
adoption of innovative practices.
Ironically, by the time MOOCs are (if ever) formally accepted into traditional higher
education, they will have evolved again (Legon, 2013). And with so many types (cMOOC,
xMOOC, iMOOC, hMOOC, etc), each MOOC provider may have drastically different views on
the importance of accreditation. Unfortunately none of the currently existing research attempts to
understand how MOOC providers perceive accreditation. Therefore further research is needed
before accreditors or other external entities can properly assess quality.

54
CHAPTER THREE: METHODOLOGY
Currently, there is a lack of consensus on how to assess MOOCs for integration into
higher education. At the same time, accreditors are struggling to encourage new innovative
practices in higher education (Spangehl & Lindborg, 2012). At present time, no formal survey
exists of MOOC provider perceptions on these issues. This qualitative study addressed this
research gap by conducting a survey of MOOC providers regarding their perceptions of MOOCs
place in higher education. This chapter provides an overview of the methodology used for this
study, including sample, research design, data collection techniques, data analysis and potential
limitations.
Review of Research Questions

This study focused on the following research questions:
• What are MOOC provider perceptions regarding how MOOCs should be assessed?
• What are MOOC provider perceptions regarding how MOOCs should be held
accountable, and to what stakeholders?
• What are MOOC provider perceptions regarding how MOOCs should be integrated into
the current U.S. accreditation system of higher education?
Method of Study
This study interviewed MOOC providers based on a qualitative research approach.
Qualitative research has three principal characteristics: it is a) naturalistic b) descriptive, and c)
focused on meaning and explanation (McEwan & McEwan, 2003).
Using a social constructionism paradigm, this study attempted to uncover the ways in
which MOOC providers participate in the construction of their perceived social reality. Social
constructionism allows for viewing the way social phenomena are developed, institutionalized

55
into formal processes and eventually made into traditions (Creswell, 2009). This process is
dynamic and ongoing. By using a social constructionism approach as outlined by Creswell
(2009), this study makes three assumptions.
First, meanings are constructed by human beings as they interact with their surrounding
world. Therefore, this study used open-ended questions so that respondents can express their
views.
Secondly, humans engage their world and make sense of it based on their historical and
ecologically placed perspective. In other words, culture shapes perspective. Therefore, this study
sought to understand the context in which MOOC providers exist relative to other educational
and business organizations. In addition, the interviewer’s own experience and background with
MOOCs can profoundly affect the interpretation of results.
Finally, the generation of meaning is always a social construct, and arises out of
interaction with a human community. As such, this study attempted to generate meaning from
the data collected in the field during the interview process. In order to triangulate data, mission
statements from websites of MOOC providers were reviewed. Denzin (1984) refers to this
process as data-source triangulation - when the researcher looks for the data to remain the same
in different contexts. These artifacts helped create a better understanding of the culture in which
MOOC providers exist.
Throughout the study, the researcher systematically reflected on his role during the
interview process. Acknowledgement of biases, values, and interests are essential to creating an
honest and open inquiry process (Creswell, 2009). This is especially important due to the use of
emergent sampling techniques. The researcher’s acknowledgment of personal biases became

56
paramount given the increased ability of the researcher to pick respondents as the study
unfolded.
Sample and Population

The population for this study was MOOC providers. This study reached out to the largest
MOOC providers, and specifically asked to speak with a representative responsible for
assessment and/or accountability. After approval from an Institutional Review Board at USC, ten
MOOC providers were emailed an invite to participate in this study. These MOOC providers
were Coursera, Canvas Networks, edX, Udacity, Open2Study, CourseSites, FutureLearn,
iversity, NovoEd, and Udemy. Together, these companies offer the majority of all MOOCs
available (Shah, 2013).
After the initial contact was made via email to all companies listed above, the researcher
used an emergent sampling design to guide the study. Emergent sampling (sometimes referred to
as opportunistic sampling) allows the researcher to address when both barriers and opportunities
in the sampling process occur. It takes place after the study begins in order to take advantage of
developing events (Onwuegbuzie & Leech, 2007).
Emergent sampling design is especially relevant to this study given the fluid dynamics of
MOOC providers. New companies are appearing every year, some backed by new rounds of
venture capital, others formed by university or non-profit consortiums. With such a short history,
new MOOC providers emerged during the execution of this study. A flexible study design fits
the context of this evolving space.
Additional MOOC providers that were contacted over the course of this study included
EdCast, OpenLearning, XuetangX, digiSchool, and France Université Numérique,

57
Data Collection and Instrumentation
Given the exploratory nature of this inquiry semi-structured interviews were conducted, guided
by the three major research questions. The researcher simply asked MOOC providers their
perceptions of:
• How should MOOCs be assessed?
• How should MOOCs be held accountable, and to what stakeholders?
• How should MOOCs be integrated into the current U.S. accreditation system of higher
education?
The researcher used a mix of video chat, telephone, and email interviews depending on the
availability and location of respondents. The interviews were scheduled for 30 to 45 minutes.
The three research questions were explicitly asked in the same order, and the respondent was
allowed time to answer each respective question as he/she sees fit. The researcher used prompts
and follow up questions based upon the responses. Upon completion of the interview, the
researcher asked for any documents, material or information the respondent would like to share
that illustrates or expands upon their views regarding assessment, accountability and
accreditation. These documents could include text documents as well as audio or visual material.
Data Collection and Analysis
All interviews were recorded using an iPhone, and the researcher took separate notes
during the interviews. One study has shown iPhones are reliable tools for recording qualitative
research in education, especially when conducting interviews. (Beddall-Hill, Jabbar, & Al
Shehri, 2011). All audio files were downloaded onto a computer. Member checking was
employed to allow each MOOC provider to review his/her responses as collected by the
researcher, and verify the accuracy of the data collection.

58
The transcriptions, along with any other documents gathered during the interview process
were uploaded to a private computer for record keeping. The notes and transcriptions were
analyzed and coded using Creswell’s six-step model (Creswell, 2009). Data was cleaned of
typos, and then re-read to understand the larger picture. The notes were then categorized into
major themes that arise during the coding process. These themes framed the results of the study.
Reliability and Validity

The interview questions were not piloted before this study was conducted. Nonetheless,
strategies were used to ensure verify both reliability and validity of data throughout the study
design. These include using sufficient sampling, developing a dynamic relationship with
sampling, transparent data collection and analysis techniques. A dynamic relationship with
sampling encourages flexibility in design to capture the true phenomena you are researching.
Transparent data collection allows outsiders to review and better understand the execution of the
study design. And finally, transparent analysis techniques reduce ambiguity over results and
methods for reaching conclusions. In addition, qualitative research should not be linear, but
rather an iterative process that allows for corrections. This process also ensures congruence
between the study purpose, methods, data collection and analysis (Morse, Barrett, Mayan, Olson,
& Spiers, 2002).
Limitations and Delimitations
A major limitation on qualitative research is the role the researcher can play in the study
design. Personal biases and conclusions may have inadvertently affected the responses given
MOOC providers. As such, reflection and acknowledgment of the researcher’s biases are crucial
to minimizing these effects. The researcher kept a dairy of reflexivity along with field notes from
the interview.

59
At the beginning of the study, the researcher had not taken a MOOC. Once the study
began, the researcher made the conscious decision to avoid taking one. The researcher believed a
lack of experience in this space would decrease biased personal perceptions of MOOC quality.

60
CHAPTER FOUR: RESULTS
The purpose of this study was to understand how MOOC providers view the concepts of
assessment, accountability, and accreditation. In order to develop this understanding the study
focused on the following research questions:
• What are MOOC provider perceptions regarding how MOOCs should be assessed?
• What are MOOC provider perceptions regarding how MOOCs should be held
accountable, and to what stakeholders?
• What are MOOC provider perceptions regarding how MOOCs should be integrated into
the current U.S. accreditation system of higher education?
This chapter will introduce the seven MOOC providers that participated in this study, and then
discuss their perceptions regarding assessment, accountability, and accreditation. Common
themes will be grouped relative to one of the specific research questions listed above. Common
themes that fall outside of the three research questions will then be reviewed separately.
Participants
Seven MOOC providers agreed to be interviewed for this study. These participants are
based in The United States, Australia, and Germany. Five interviews were conducted over
Skype, and one verbal interview was conducted via telephone. These interviews ranged from
fifteen to forty-five minutes long. One respondent sent text answers via email. Table 1 below
summarized the profile of the seven participating MOOC providers.

61
Table 1
MOOC Provider Profiles
NAME TYPE LOCATION
CanvasNetwork

For-Profit United States
EdCast

For-Profit United States
Iversity

Non-Profit Germany
Open2Study

Non-Profit Australia
OpenLearning

For-Profit Australia
Udacity

For-Profit United States
Udemy

For-Profit United States

Non-Participation
Eight MOOC providers contacted did not agree to participate in this study. The reasons
for non-participation fell into one of two categories. The first category of non-participants was
due to a non-response. Five MOOC providers never responded to my interview requests. A lack
of response may be partially due to a language barrier, as some of the MOOC providers are
located overseas where English is not the first language. The remaining three MOOC providers
fell into a second category of “over-commitment”. First, the respondent stated it was a start-up,
and had limited ability to answer such a query. After continued follow-up, the respondent would
very politely decline, citing over-commitment in so many other research projects. Responses
were often encouraging, noting interest in the study, but a general lack of resources available to
participate.
Results
Once the interviews were converted from audio recordings to word documents, analysis
of the transcribed text began. The respondents were given pseudonyms (Participant A-G).

62
Transcripts were read and re-read to identify similar concepts and arguments. The sections below
presents the major themes expressed by the participants. Each theme is matched with a
corresponding research question. Emergent themes that fall outside of the original research
questions are then presented. Table 2 below provides an overview of the findings from this
qualitative inquiry.

Table 2
Overview of Themes and Findings
RESEARCH QUESTION THEME FINDINGS
How do you think MOOCs
should be assessed?

• Importance of Context

• Student-Centered
Demonstration

• First need to address how to
assess education in general, and
then more specifically online.
MOOCs are a type of online
education, not a separate entity

• Important to assess students
through demonstration of
competency

How do you think MOOCs
should be held accountable,
and to what stakeholders?

• Student choice

• Perceptions versus
reality

• The primary stakeholders are
the students
• Professional accountability
matters in teaching

• The title of non-profit is
overvalued in the accountability
debate
• MOOCs have potential to
introduce new key performance
indicators

How do you think MOOCs
should be integrated into the
current U.S. accreditation
system of higher education?
And if not, are there
alternatives?
• Type of Accreditation

• Regulation of
Education
• MOOCs used to gain a
traditional degree should be
accredited through the
institution providing the course;
professional education courses
could be programmatically
accredited

• Accreditors should allow market
to guide MOOC evolution

63
N/A • Understanding of
MOOC providers

• Educational Access
• Increasing diversity of non-
traditional higher education
providers
• Need more distinction between
content providers, platforms

• MOOCs are an opportunity to
democratize education without
geographical barriers

Research Question #1 - Assessment
The first of question asked to all participants was “How should MOOCs be assessed”.
Two major themes emerged. First was the importance of providing context when addressing
educational assessment. The second theme was the need to incorporate student-centered
demonstrations as part of assessment.
Importance of context. Nearly all respondents collectively tried to frame this question in
the appropriate context. The key question is not how MOOCs should be assessed, but first “how
people should be assessed in education and then how they should be assessed online”
(Participant G, personal communication, January 11, 2015). Respondents stressed that MOOCs
are a type of online education, not a separate entity.
Next, course design depends on the goals and desired outcomes a student expects to gain
from that MOOC. The creator of a MOOC should ask, “Is it stand-alone thing or is it something
that's embedded in a larger activity that an institution or organization is doing” (Participant A,
personal communication, September 24, 2014). When the MOOC is part of a larger activity, such
as a required course for a bachelor’s degree, it should be assessed “similar to regular university
courses” (Participant C, personal communication, October 9, 2014). Assessment should be
“based on what is the goal of the course” (Participant F, personal communication, December 9,
2014).

64
Many of the respondents give tools and resources to create the MOOC, but do not want to
interfere with the manner in which teachers evaluate their own courses. Rather than stipulating
how a MOOC should be assessed, respondents did not want to explicitly state how to judge
quality of a MOOC. Participant A stated, “We'd rather be the experimental platform than have
the particular position that here's how MOOCs should be” (personal communication, September
24, 2014). The course creator should have the flexibility and authority to assess their MOOCs by
their own criteria.
While a MOOC is a new medium through which educational information is delivered, the
same principles that apply to traditional higher education courses apply. The need to match
course assessment with desired course outcomes remains intact. The difficulty however resides
in scaling assessment, and the appropriate mix of methods used to measure students’ knowledge
gained.
Student-centered demonstration. After a MOOC’s position in the educational space is
correctly understood, MOOCs themselves could be assessed based on a more flexible student-
centric model. A majority of the respondents stated they care “very little about course
completion” (Participant B, personal communication, September 25, 2014). Instead, they argue,
“The real marker of success in the course, or at least how we think of it, is has this course helped
you achieve your learning goals” (Participant B, personal communication, September 25, 2014).
Participant G stated, “We don't use the word ‘assess’. We use the word how people should
‘demonstrate’ the learning” (personal communication, January 11, 2015). He advocated for a
“much more flexible student-centric model that enables the student to ultimately enjoy their
experience, practically learn and demonstrate their knowledge” (Participant G, personal
communication, January 11, 2015).

65
To achieve student-centered demonstration, a majority of respondents advocated for more
competency-based assessments. Participant C noted, “competency based learning is the official
methodology here” (Participant C, personal communication, October 9, 2014). This approach is
already gaining traction in higher education. Participant B explained, “There are groups doing
really interesting things, like [The Council for Adult and Experiential Learning], that do
assessment of prior learning in order to get college credit. You put together a portfolio of your
experiences and that can include online courses to demonstrate competency” (personal
communication, September 25, 2014). In addition, Participant D argued, “The success of a
course could be measured by the success of its students, whether they were promoted, got hired,
or reached some other goal they set for themselves…However, as you may have guessed, this is
not an easy or scientifically accurate thing to measure” (Participant D, personal communication,
October 22, 2014). One hurdle is finding common ground on how to assess students during the
class.
Peer evaluation has a valuable place in student assessment, but only a part. Some
MOOCs are asynchronous, which makes peer assessment logistically impractical. As one
respondent stated, “Given that courses are cohort based, there's not a singular start time, so
everyone is not working on the same assignment at the same time - peer assessment piece
becomes secondary for us” (Participant B, personal communication, September 25, 2014).
Participant A advocated that peer assessment “needs to be a part of the grade but it can't be a
high-stakes element in the whole experience” (personal communication, September 24, 2014).
On the other hand, the use of artificially intelligent computer programs to grade essays is
still debatable. One respondent stated, “I have a hard time seeing this being accepted by higher

66
education…For creative assignments, we will always need to have some form of human
evaluation” (Participant C, personal communication, October 9, 2014).
One novel approach to evaluation of students is use of “cloud teaching”, which uses an
online platform to match people’s expertise area and level (bachelors, masters, doctorate, etc.) to
what they are permitted to evaluate. This concept is already done at a much smaller scale in
higher education. Participant C queried, “who actually does the evaluation at universities? It is
rarely the professor. It is all the teaching assistants who do this” (personal communication,
October 9, 2014).
While the exact mix of methods used to evaluate students is still in flux, the respondents
agreed that students should be evaluated based on their own learning goals. The importance of
the student’s in guiding assessment carries weight not just in how a MOOC is assessed, but also
in deciding who holds MOOCs accountable.
Research Question #2 - Accountability
The second question asked to all participants was “How should MOOCs be held
accountable, and to what stakeholders”. The two common themes that emerged are the first the
importance of student choice in education; and secondly the need to challenge current
perceptions of traditional higher education institutions.
Student choice. All seven respondents argued that students are the most important
stakeholders in education. Participants A and D both stated accountability starts “first and
foremost” with the students (personal communication, September 24, 2014; personal
communication, October 22, 2014). Participant B stated “the value of a MOOC is…students are
the customers. The student reports back that they had good experience with the course and it
helps them in their lives. I think that's the best way to assess the quality” (personal

67
communication, September 25, 2014). Participant G agreed stating, “The first [measure of
quality] is the student satisfaction levels” (personal communication, January 11, 2015).
Participant F echoed similar sentiments saying “The number one person ... everybody is
accountable to is the student…the student has to get the value” (personal communication,
December 9, 2014).
The student is the primary stakeholder because there are no exit costs to dropping out of a
MOOC. This allows the student greater control over their own learning experiences. This differs
from the traditional model of enrolling for a semester’s worth of courses in a university. As one
respondent pointed out, “In a face to face institution, you're locked in…it's entrapment. There's
this big exit cost. On a MOOC, there's no exit cost. You can leave with a click” (Participant E,
personal communication, October 26, 2014). Respondents viewed student mobility as increasing
student’s ability to choose his/her own educational path. Reinforcing the notion that completion
rates are not intrinsically important, Participant G argued, “There is no point to have [higher
completion rates] if we don't have the student satisfaction” (Participant G, personal
communication, January 11, 2015).
After the emphasizing the importance of students in holding MOOCs accountable, the
second most discussed stakeholder was the teacher holding him or herself to a high level of
professional accountability. As one respondent noted, “We have a duty to educate well, and to
help people change.” (Participant E, personal communication, October 26, 2014). Remaining
stakeholders included capital investors and the universities partners. As the next section will
discuss, the role of investors was not seen as a new development in higher education.

68
Perceptions versus reality. The respondents identified two perceptions of higher
education that need some re-examination. First, the traditional perceptions between for-profit and
non-profit educational providers are not always accurate in reality. Participant F argued:
Every company has an investor whether you are a for-profit or a non-profit, okay? Even
if you are a non-profit, you still have an investor, they're just not called investors they're
called donors. But, donors are seeking some kind of ROI, otherwise donors would be
completely dumb. Right? But they are not…So, I think the whole for profit/non-profit has
no meaning. You see all these non-profit higher institutions that are making a ton of
money. (personal communication, December 9, 2014)
This is not rejecting the notion that MOOC providers aim to make a profit. Rather, it argues that
generating large revenue streams is also a motivation of most top-tier research universities.
Participant F added, “MOOCs are a great innovation because until now, if you ask institutions,
even the top 20 institutions in the country and say, ‘who are you accountable to?’ They'll say
they're accountable to the board of trustees because otherwise they'll get fired” (Participant F,
personal communication, December 9, 2014). The value MOOCs add to higher education is the
ability to put students as the primary stakeholder.
This leads to the second faulty perception – the belief that traditional institutions of
higher education are always held to rigorous quality standards in teaching. As Participant C
notes, “they say the best teachers are from the universities, which can be highly questionable.
Universities are based on research, and not on teaching” (personal communication, October 9,
2014). It is hard to quantify teaching quality. However, MOOCs provide an avenue to advance
new metrics related to quality of teaching and teaching performance. Participant E explained:

69
If you've got great research people notice because they can count publications, and they
can count grant money, and all these things can be measured. What I love about MOOC's
is they make teaching measurable... If you're a professor that does really bad teaching but
has really good research, in the past no one would ever know…Now, increasingly, if you
do a bad job teaching, then everyone can see it. Your videos are out there, and everyone
laughs at them. Your courses are there and all the students say, "This is horrible"…all
these things that previously were secret are now open, just like research is open...I think
eventually the [key performance indicators] for the vice chancellors and presidents will
start to reflect this now that it's measurable in public. (personal communication, October
26, 2014)
Research Question #3 - Accreditation
The third and final question asked to each participant was “How should MOOCs be
integrated into the current U.S. accreditation system of higher education?” Respondents had very
different approaches to if their own offerings are accredited or ever plan to be accredited. One
respondent already offers accredited courses, one plans on entering accreditation soon, one was
not interested in integrating into the current accreditation system, and another feels positively
towards the need for accreditation, but has no plans to offer accredited courses. Nevertheless,
common themes emerged from respondents regarding how MOOCs could be accredited in the
current system of higher education. Two major themes emerged. First, the type of accreditation
depends on the goal of the MOOC. Second, the accreditation bodies should avoid undue
regulations when implementing new standards in higher education.
Types of accreditation. Despite their differences in how they currently approach
accreditation, respondent opinions on accreditation were similar. When serving as an

70
undergraduate course, or part of a degree program, MOOCs should be accredited through the
higher education institution in which they are created. Participant A stated, “Accreditors should
understand that the MOOC is like a textbook, it's like a lab, it's like a internship. It's just another
opportunity to gather experiences that fits into this larger puzzle. It should be a part of the bigger
accrediting process.” (personal communication, September 24, 2014). Participant E also noted,
“For MOOCs that are offered by institutions, then I believe they should be accountable in the
same way that all of their institutions’ educational offerings are accountable” (Participant E,
personal communication, October 26, 2014).
As another respondent noted, MOOCs “will not replace the accreditation from the
traditional system” (Participant G, personal communication, January 11, 2015). Rather, MOOCs
used by traditional universities for degree granting programs should be integrated into the current
existing accreditation process. Participant B pointed out:
This is not new territory and there are lots of folks who have figured out how to do this
accreditation and validation. Whatever you think about [for-profit online universities] and
their tactics and whatever, they've figured it out how to do online delivery of courses for
credit. (personal communication, September 25, 2014)
However, when serving as a professional or continuing education program, many respondents
agreed that MOOCs could be directly accredited. One respondent drew a parallel with existing
accreditation practices in professional education:
Medical schools, they’ll do grand rounds and people will go and they’ll spend an hour
and they'll do this lecture with the doctor and they get an hour continuing professional
education credit. That's accredited by the American Medical Association or whatever it is
in Medicine. The same thing is in Law. The same thing is in Education. This is where (I

71
think) MOOCs could be really interesting. They [MOOCs] could be direct accredited
activities for the continuing professional education hours. (Participant A, personal
communication, September 24, 2014)
While the above statement does not accurately convey the true nature of accreditation in
professional education, it exemplifies a common theme among respondents. Participant D also
emphasized the importance of addressing accreditation in professional education, since most
enrolled students are already in the workforce. She stated:
Regarding accreditation, we actually find a high number of our students are college
graduates who are looking to specialize in a certain field. I think there's an important role
for MOOCs to fill here, as baccalaureate education serves to introduce students to a field.
After graduation, students learn an awful lot on the job, and MOOCs can provide an
alternative means for students to gain knowledge in a specific subject area. (personal
communication, October 22, 2014)
There is currently much debate regarding how to test these students for credit. However, this
notion of how to authenticate and proctor misses the greater point. Participant E made clear there
is no easy technological solution to authentication. People can always find a way to cheat the
system. However if incentives and objectives are lined up correctly, the motivation to cheat will
be removed. Participant E explained their process used to reduce cheating:
I have an invigilated face-to-face exam at the end, just like a normal university exam, and
the rest of their assessment is them mounting a case, putting together a portfolio based on
what they've done over the whole course, referencing everything that happened in the
MOOC as evidence, and then their pattern of behavior over the whole MOOC is assessed,
and that's much harder to fake. You can get a guy in India to write a software assignment

72
for you, to write a proper one. You can't get him to be online, talking and being you for
13 weeks, and having reflective diary entries, and doing incremental works. That's just
too expensive and awkward to do. You'd have to hire an impersonator to do that.
(Participant E, personal communication, October 26, 2014)
Hopefully within a few years “online proctoring will be accepted as official proctoring for
universities” (Participant C, personal communication, October 9, 2014).
Regulation of education. Another theme that emerged in the area of accreditation
focused on the role of accrediting or government bodies in regulating education. Respondents
claimed accreditors must not dictate how MOOCs evolve or are created through federal quality
control standards. Participant E passionately argued:
The notion of "should" sort of implies rules and things, and I think the whole wonderful
flourishing [of MOOCs] has happened because there were no rules…I would like the
whole online education space to be as big, and broad, and rich, and deep, and constantly
evolving as possible, and not constrained. (personal communication, October 26, 2014)
This fast evolution of online education is not limited to higher education. The majority of
respondents believe their largest role remains outside higher education. As Participant E
explained,
I think at the moment, inside of [higher education] is where most of the existing content
is. Your content providers are there. For getting content up quickly onto the MOOC's, I
reckon it'll come out of [higher education] to start with…I see long run that MOOC's
extend beyond higher education. (personal communication, October 26, 2014)
As the workforce changes with new technology, MOOCs are an avenue for specific skill
training. Participant B stated, “When you think about the value of a degree as a signal, it is really

73
important. But when you think about what you actually need to learn, that's another important
conversation. How do you continue to keep ahead of the skills as [new] technology goes and
changes the necessarily skills [needed for] the labor market” (personal communication,
September 25, 2014).
Another area of expansion is into teacher training in K-12. Participant A stated, “I think
K-12 teachers are hungry for really good content. They don’t have the resources to make really
good content, especially content aligned to Common Core now and that if we could find a way
for MOOCs to really help them in that space, that can be really powerful” (Participant A,
personal communication, September 24, 2014).
Other Themes
As the respondents answered the three research questions listed above, two other themes
emerged which fall outside the realm of assessment, accountability and accreditation. The first is
the need for a more nuanced understanding of the term “MOOC provider”. The second theme
was the respondents’ hope to increase educational access new technologies and online
educational resources.
Understanding of MOOC providers. Respondents felt the term “MOOC provider is
often misunderstood, and other times overused. This term describes just a portion of what
companies are aiming to achieve. Participant F explained:
We can support MOOC-type courses, which are open to anyone in the world (and free
and scaled and all of that). But we do support other models that are not open. Which is
private courses, which are not free courses, and we also support many other full credit
courses and many other certificate courses. Non-free, non-credit courses, all of them.
(personal communication, December 9, 2014)

74
Participant G made a similar comment about MOOCs being just a part of this expansion into
online education, stating, “Not everything that is happening in education and in higher education
and in online learning is all about MOOCs. We have a very balanced view on MOOCs”
(Participant G, personal communication, January 11, 2015).
The “MOOC” is the acronym that has captured the public’s attention – but that term
alone does not encapsulate all the new approaches happening in the online educational space.
Some respondents don’t like to focus on using this terminology, instead thinking of it as a
“quality online learning initiative to change things in our business and to change things for better
for students” (Participant G, personal communication, January 11, 2015).
In addition, there is a need to further distinguish between online platforms and content
providers. They have extremely different roles in the online education space, but are often
lumped together. Participant E wondered out loud, “if the general public even cares about
platform providers or if they really mainly care about the content that's delivered using the
platform” (personal communication, October 26, 2014). Ironically, the platforms are the groups
most likely to dramatically change education delivery. Participant E continued his train of
thought and stated: “I think as far as the general public's concerned, all platforms look more or
less the same even though I know that's not the case. Different platforms have widely different
philosophical underpinnings behind them, and educational objectives” (Participant E, personal
communication, October 26, 2014).
Educational access. Another theme that emerged from respondents was the potential for
MOOCs to increase educational access for students across the world. The technology used in
online education in general can “really help democratize education…in 2014 we shouldn't be
limiting and restricting students and say ‘you need to fly out to Los Angeles and you have to pass

75
a face-to-face test’” (Participant F, personal communication, December 9, 2014). Participant B
expanded on this same issue:
There's tremendous potential for online learning to help folks who may not have access to
traditional education or ongoing education to get ahead or to learn…We see two-thirds of
our students are coming from outside the U.S. where relevant skills education is much
less accessible. I think that's where really a lot of the potential of MOOCs and online
learning in general to have an impact and open up access is pretty important. (personal
communication, September 25, 2014)
Participant B also noted however that as educational access increased, degrees from prestigious
higher education institutions aren’t going to quickly become diminished. She stated that “the top
50, top 100, top 150 [universities] aren't going anywhere. I don't think you're going to replace a
Stanford degree with a bunch of MOOCs” (personal communication, September 25, 2014).
Document Analysis
For this study, a comparative document analysis was conducted to cross-check the
respondent’s perceptions of assessment, accountability, and accreditation with their own
organization’s respective mission statements. These mission statements were gathered from the
seven participating MOOC provider websites, and analyzed to determine if the respondent’s
perceptions are consistent with their own institutions’ mission statement. The researcher first
looked for terms related to assessment, accountability, and accreditation – and then examined
each mission statement to understand the context in which these keywords were used. In cases
where the emergent themes of the respondents matched mission statement text, triangulation has
been accomplished.

76
The themes that emerged from the document analysis included concepts related to
open/increased access, collaborative environments and personalized education. All seven of the
respondent websites included at least two of the aforementioned themes. The majority of these
themes were used in the context of student learning and student experiences. However,
sometimes they were also used in reference to other stakeholders such as teachers, traditional
educational institutions and employers.
The keywords “open” or “anyone” or “anywhere” were used by all seven MOOC
providers, all in the same context of increasing learning opportunities for students willing to
learn. For example: Canvas Network (2015) provides “open, online courses”; Open Universities
Australia (2015) is “open to anyone, anywhere”; iversity (2015) hopes to “share a broad range
courses with students from around the globe” and let them “study anywhere”; and Udemy (2015)
aims to “help anyone learn”. These examples align with the emergent theme that respondents
believed MOOCs can drastically increase educational access in the future.
The theme of personalized education also emerged throughout the most of the MOOC
provider websites. Udemy (2015) creates services so “students can learn at their own pace, on
their own time, and on any device”. Canvas Network (2015) lets students “chart their own course
for personal growth”. In addition, iversity (2015) provides “provide students and independent
learners with a customized higher education experience”. This emphasis on the importance of
allowing personalized education services support the respondents common theme of putting the
student at the center of assessment. All learners taking MOOCs should be assessed via a flexible
and student-centric model. In addition, the student’s direct satisfaction and ability to obtain
his/her own learning goals should anchor whatever assessment framework is used.

77
Finally, the concept of creating and supporting collaborative environments appeared
throughout all seven respondent websites. One important note is the context of this theme is not
just related to students, but also teachers, employers, content creators, and traditional educational
institutions. Australian based OpenLearning (2015) aims to create “an online learning platform
that goes beyond content delivery to focus on community, connectedness, and student
engagement”. They hope to combine the learning environment and student hangout areas into
“the same” space. In addition, EdCast (2015) will not only “enable millions of students to
collaborate with each other and learn”, they will aim to offer a “personal learning network to
enhance human ability to collaborate and learn across educational materials, instructors, students
and employers. In a similar tone, Canvas Network (2015) hopes that “institutions worldwide can
connect” while using their online platforms. The concept of creating collaborative environments
aligns with the respondents’ viewpoint that students are of central importance, and must be
supported through formal and informal methods. By creating collaboration spaces across
teachers/institutions/employers, students will also retain greater choice over their own
educational delivery, while traditional lines between educational institutions become blurred.
One additional outcome of the document analysis was alignment of respondent views on
accreditation specifically regarding their own MOOCs. For example, iversity (2015) provides
students with the “opportunity to earn recognized credentials, no matter where they are”.
Another group (Open2Learn) states that through Open Universities Australia (2015), “you can
study units and work towards qualifications online…and if you wish, gain a degree from one of
those providers.” Essentially, you can complete online “the same qualification as an on-campus
student”. However, these respondents still noted the need for supporting traditional higher

78
education rather than become a substitute. As iversity (2015) stated, “our aim is not to replace
universities, but to empower academics”.
Summary
Participant narratives in this study uncover several themes relevant to how MOOCs
should be assessed, held accountable, and potentially accredited. The findings indicate that
MOOC providers view MOOCs as part of a larger online-based movement in education. The true
value of MOOCs resides in their ability to give students more voice and control over their own
educational path. Increasing student choice can potentially create new quality based teaching
metrics. Traditional institutions of higher education will not be able to rely on just research
prestige when marketing their teaching quality. Accreditors in turn must prepare for MOOCs in
higher education, but not over regulate their use. When used in a traditional degree-granting
program, MOOCs ought to be institutionally assessed and accredited like all the other existing
courses. These findings have possible implications on future policy and practice for higher
education, which will be further discussed in the next chapter.

79
CHAPTER FIVE: DISCUSSION OF FINDINGS

This study attempted to develop a better understanding of how to properly assess quality
in MOOCs for higher education through empirical research. Currently there is a lack of
consensus on how to assess them for integration into higher education. While numerous studies
have been conducted that incorporate views from instructors (Kolowich, 2013b), students (Bruff,
Fisher, McEwen, & Smith, 2013), and even potential employers (Radford et al., 2014), the
opinions of MOOC providers have not been aggregated in one place. This study aimed to fill this
gap in research by conducting a survey of MOOC providers regarding their perceptions of
MOOCs place in higher education.
More specifically, the purpose of this study was to understand how MOOC providers
perceive the concepts of assessment, accountability, and accreditation. In order to develop this
understanding, semi-structured interviews were conducted with seven MOOC providers. These
participants were each asked their perceptions of:
• How should MOOCs be assessed?
• How should MOOCs be held accountable, and to what stakeholders?
• How should MOOCs be integrated into the current U.S. accreditation system of higher
education?
Summary of Findings
After conducting interviews with seven MOOC providers, several themes emerged.
Respondents argued MOOCs should first be understood in the correct educational context, and
then assessed based on a student-centered approach. After all, students are by far their most
important stakeholder. Despite concerns that MOOC providers are primarily driven to generate
profits, the respondents claimed they are actually tethered to the students’ whims. The student’s

80
voice is so powerful because there are no exit costs to leaving a MOOC, which empowers the
student in creating their own educational trajectory. Regarding accreditation, MOOCs should be
institutionally accredited when they are delivered as part of a degree program, and directly
accredited when used for professional education.
Implications for Practice
As previously noted in this study, CHEA listed five pressing questions regarding MOOCs
and accreditation: a) Through what lens do we examine MOOCs for quality? b) Do MOOCs
require altering expectations of teaching and learning? c) Does current accreditation review
address the key features of MOOCs? d) If accreditation will address MOOCs, what tools do
accreditors need? e) If accreditation is not an appropriate venue to address quality for MOOCs,
what is an alternative version of quality review? (Eaton, 2012a). The section below will address
each of CHEA’s questions from the MOOC provider perspective, based on the results gathered
from this qualitative inquiry.
Determining Quality

The first question from CHEA asks, “Through what lens do we examine MOOCs for
quality?” Currently traditional higher education uses a baseline of quality that includes curricula,
faculty and student support. Most respondents argue while these topics are relevant, the primary
lens through which MOOCs should be judged are via the student’s own experiences. Was the
student satisfied with the delivery of the course? Did the course help a student achieve his/her
intended goals or outcomes?
Students enroll in MOOCs for a variety of reasons, and these must be taken into account
when judging the performance of a MOOC. This approach does not devalue the need for good
curricula, faculty and student support. Rather, it emphasizes the students’ voice as the primary

81
quality indicator for higher education. The majority of respondents want direct input from
students to guide assessment and course improvement, rather than indirect quality measures that
can be misleading (i.e. retention rates).
By focusing on direct measures of student success, MOOCs will potentially decouple
traditional measures of university prestige and rankings. High quality research is too often
assumed to correlate with quality teaching. However MOOCs will put a premium on each course
achieving not just a baseline of quality, but creating a “race-to-the-top” mentality to see who can
create the best MOOC for a given content area. The market forces that drive research prestige
could be harnessed to increase expectations of teaching quality in higher education.
Expectations of Teaching and Learning

The second question from CHEA asks, “Do MOOCs require altering expectations of
teaching and learning?” In general respondents would argue the answer is no. The expectations
of teaching and learning should be the same across all forms of education, online and traditional.
All teachers across these mediums have a professional accountability that holds them to
standards regardless of their own background or the type of course they teach. The value
MOOCs add is the transparency provided to outsiders regarding quality of teaching.
Perhaps a more nuanced question that addresses CHEA’s concern is “What new indirect
metrics should CHEA incorporate to measure student success?” A traditional measure of
counting tenure-track faculty versus adjunct faculty does not apply to the MOOC model. And
while incorporating outside curricula is not common in traditional higher education, it is one of
the perceived benefits of utilizing open educational resources. Finally, new forms of mentorship
through peer and online community engagement may augment traditional student support.

82
There is no consensus from respondents on how to properly weigh these indirect
measures of student success. Currently there is still debate on how to even directly measure
student success. Peer-to-peer grading has a role, but should not be a large portion of the student’s
evaluation. Automated assessment is not yet universally accepted, and may only be relevant to
certain subjects that allow for discrete solutions. Creative and entrepreneurial courses will
always require some form of human assessment.
Hopefully MOOCs will create new measurable outcomes of teaching quality. These
outcomes will perhaps be more directly related to a student’s personal interactions with the
course, and the teacher’s individual course design. While the indirect and direct methods used to
measure these outcomes are not yet agreed upon, quality of teaching and learning should (at a
minimum) become more transparent when using MOOCs.
Current Accreditation Practices
The third question from CHEA asks, “Does current accreditation review address the key
features of MOOCs?” There are vast differences in respondents’ approaches to accreditation of
their own courses. However, their opinions on where MOOCs could fit into national
accreditation practices are fairly consistent. When offered as part of a larger degree-granting
curriculum through a university, MOOCs should be assessed and accredited through that
university’s normal processes. In this case, accreditation review does address the key features of
MOOCs.
When used for professional education credits, the MOOC itself could be directly
accredited by its corresponding trade organization. This is more complicated, and may fall
outside the scope of current accreditation practices. Accreditors have already judged quality of

83
online courses already from numerous online education institutions – but MOOCs are much
larger in scale. The issues of authentication and proctoring must be addressed.
Surprisingly, respondents believed the best way to authenticate and proctor students
won’t be from a technological advance. Rather, ensuring course incentives and potential
outcomes are properly aligned is the number one deterrent to online cheating and plagiarism. A
well-designed MOOC can achieve these goals through multiple layers of engagement, various
meetings, different types of assignments, and requiring a submission of portfolios to showcase
student work. Proper alignment of MOOC incentives and outcomes is something accreditors can
begin reviewing immediately.
Tools Needed for Accreditation
The fourth question from CHEA asks, “If accreditation will address MOOCs, what tools
do accreditors need?” This question has already been partially addressed as CHEA is creating a
new quality platform for non-traditional institutions of higher education. This new platform will
be designed to handle the complexity of MOOCs and other new online course offerings by
following similar methods of accreditation (external peer review, assessment as a lever for
quality improvement, etc.). One major difference - the speed of the review process will become
dramatically faster, and allow for input from geographically separated participants.
But at its fundamental level, the accreditation process should not change just for MOOCs.
While new technology can help accreditor’s understand and quickly review a MOOC’s quality,
the basic principles of accreditation remain. Perhaps the bigger issue for accreditors is that
coupling federal financial aid to accreditation holds less importance to MOOCs. While MOOC
providers are still working out their own business models, they are not as reliant on financial aid
as traditional institutions of higher education.

84
As more attention is given to how MOOCs can be accredited (or even if they should be
accredited), CHEA and other accrediting bodies should remain flexible in their approach to
evaluation. The online education market is evolving at a rapid pace, and over-regulation could
stifle potential innovation. Most MOOCs are currently used in higher education and professional
development, but in the future could delve into new spaces like K-12 learning and teacher
training.
Alternatives to Accreditation
The fifth question from CHEA asks, “If accreditation is not an appropriate venue to
address quality for MOOCs, what is an alternative version of quality review?” Most respondents
agreed that some MOOCs can be accredited in the current system of accreditation. However,
some do not need accreditation. Many are designed for students who want a particular skill-set
that may be applicable to a very specific job or task. In those cases, accreditation is probably not
the best venue to assure quality. Rather, the employer or market will decide if the student’s skill-
set has improved based on that student’s own employment and career outcomes.
The alternative to accreditation then becomes allowing the market to decide quality in
MOOCs. While this approach is vague, it allows for MOOC providers to shift their offerings
based on the needs of the student. It also ensures competition between new MOOC providers, as
they attempt to create a higher quality learning experience for potential students.
Future Research

The five questions listed above provide an outline for accreditors to start reviewing what
are appropriate mechanisms for quality assurance and improvement in MOOCs. However, this
study is not able to fully answer each question. Further research is needed to understand how
MOOCs should be assessed from other viewpoints regarding accreditation. Perhaps even more

85
fundamental is the question if they need to even be assessed outside of market-driven forces. In
addition, further clarity is needed regarding the appropriate use of new assessment technologies.
One limitation of this study is the lack involvement from the largest MOOC provider(s).
These groups have the largest market-share of the MOOC space, which consequently means their
opinions hold greater impact on the education community. One caveat is that as time passes, and
new providers are established, the market-share of these currently larger companies decline.
Nonetheless, including additional MOOC providers would generate an even deeper
understanding of their perceptions regarding assessment, accountability and accreditation.
Conclusion
Currently MOOCs are one of the most discussed topics in higher education due to their
lack of enrollment costs and open access. At the same time, traditional institutions of higher
education have come under scrutiny for a lack of cost-effectiveness and restricted access.
Nonetheless, the potential for MOOCs to fully integrate into higher education relies first on
developing a common understanding of their capabilities. Many MOOC providers believe they
can supplement and support course delivery in traditional institutions of higher education. While
MOOCs are not the proper venue for all degree-granting courses, they can be used by students to
enrich and expand certain skills and knowledge. Perhaps most importantly, MOOCs can also
encourage a healthy debate on how to assess quality in teaching.
While the exact methods needed to assess teaching and student learning in MOOCs are
debated, there is knowledge gained by allowing non-traditional educational providers explore the
possibilities. Therefore, MOOC providers are weary of over-regulation from accrediting bodies
that could stifle future innovation. A lack of consensus on how to incorporate MOOCs into
higher education does not necessitate a rapid answer. Instead, a thoughtful discussion is needed

86
that includes representation from all groups involved (teachers, students, government bodies,
etc.). Accrediting bodies must proactively work with all non-traditional institutions of higher
education to continually advance the dialogue. While this study provides insight on MOOC
provider perceptions, this is just one step towards incorporating MOOCs into higher education’s
accreditation process.

87
REFERENCES

Abeywardena, I. S. (2014, January). Public opinion on OER and MOOC: A sentiment analysis of
twitter data. Paper presented at the International Conference on Open and Flexible
Education (ICOFE 2014), Hong Kong SAR, China.
Accrediting Council for Independent Colleges and Schools. (2013). How effective accreditation
supports innovation in postsecondary education. Washington, DC: Accrediting Council
for Independent Colleges and Schools.
Adair, D. (2013, December). Quality matters for MOOCs: Results & implications of the first QM
MOOC reviews [Webinar]. Educause Learning Initiative. Retrieved from
https://educause.acms.com/p1x4dugu17j/?launcher=false&fcsContent=true&pbMode=no
rmal
Adamopoulos, P. (2013, December). What Makes a Great MOOC? An Interdisciplinary Analysis
of Student Retention in Online Courses. Paper presented at the Proceedings of the 34th
International Conference on Information Systems, ICIS.
Adelman, C., Ewell, P., Gaston, P. L., & Schneider, C. G. (2011). The degree qualifications
profile. Washington, DC: Lumina Foundation.
Aghion, P., Dewatripont, M., Hoxby, C., Mas-Colell, A., & Sapir, A. (2008). Higher aspirations:
An agenda for reforming European universities. Brussels: Bruegel.
Baggaley, J. (2014). MOOC postscript. Distance Education, 35(1), 126-132.
Balfour, S. P. (2013). Assessing writing in MOOCs: automated essay scoring and calibrated peer
review. Research & Practice in Assessment, 8(1), 40-48.

88
Banta, T. W. (1993). Summary and conclusion: Are we making a difference? In T. W. Banta
(Ed.), Making a difference: Outcomes of a decade of assessment in higher education (pp.
357-376). San Francisco, CA: Jossey-Bass.
Baum, S., & Ma, J. (2013). Trends in college pricing. New York, NY: The College Board.
Bayne, S., & Ross, J. (2013). The pedagogy of the massive open online course: The UK view.
York, England: The Higher Education Academy.
Beddall-Hill, N. L., Jabbar, A., & Al Shehri, S. (2011). Social mobile devices as tools for
qualitative research in education: iPhones and iPads in ethnography, interviewing, and
design-based research. Journal of the Research Center for Educational Technology, 7(1),
67-89.
Beno, B. A. (2004). The role of student learning outcomes in accreditation quality review. New
directions for community colleges, 2004(126), 65-72.
Bensimon, E. M. (2005). Closing the achievement gap in higher education: An organizational
learning perspective. New Directions for Higher Education, 2005(131), 99-111.
Bergeron, D., & Klinsky, S. (2013, October 28). Debt-free degrees. Inside Higher Ed. Retrieved
from http://www.insidehighered.com/views/2013/10/28/essay-need-new-innovation-
focused-accreditor
Bernhard, A. (2012). Quality Assurance in an international higher education area: A summary of
a case-study approach and comparative analysis. Tertiary Education and Management,
18(2), 153-169.
Blauch, L. E. (1959). Accreditation in higher education. Washington, DC: United States
Government Printing Office.

89
Block, G. (2014, Mar 28). Viewpoints: Don’t let online education widen the digital divide. The
Sacramento Bee. Retrieved from
http://www.sacbee.com/2014/03/28/6274064/viewpoints-dont-let-online-education.html
Bloland, H. G. (2001). Creating the council for higher education accreditation (CHEA).
Phoenix, AZ: Oryx Press.
Boren, D. (2012, Aug 14). Letter: President David Boren on instructional technology: University
of Oklahoma. Retrieved from
http://immagic.com/eLibrary/ARCHIVES/GENERAL/OU_OKUS/O120814G.pdf
Boston, W., & Helm, J. S. (2012). Why student learning outcomes assessment is key to the
future of MOOC [Blog post]. Retrieved from http://illinois.edu/blog/view/915/84723
Boyatt, R., Joy, M., Rocks, C., & Sinclair, J. (2013, September). What (use) is a MOOC? Paper
presented at the The 2nd International Workshop on Learning Technology for Education
in Cloud.
Bresciani, M. J. (2006). Outcomes-based academic and co-curricular program review: A
compilation of institutional good practices. Sterling, VA: Stylus Publishing, LLC.
Brittingham, B. (2009). Accreditation in the United States: How did we get to where we are?
New Directions for Higher Education, 2009(145), 7-27.
Brittingham, B. (2012). Higher Education, Accreditation, and Change, Change, Change: What’s
Teacher Education To Do? In M. LaCelle-Peterson & D. Rigden (Eds.), Inquiry,
evidence, and excellence: The promise and practice of quality assurance (pp. 59-75).
Washington, DC: Teacher Education Accreditation Council.

90
Bruff, D. O., Fisher, D. H., McEwen, K. E., & Smith, B. E. (2013). Wrapping a MOOC: Student
perceptions of an experiment in blended learning. Journal of Online Learning &
Teaching, 9(2).
Burke, J. C. (2005). Achieving Accountability in Higher Education: Balancing Public, Academic,
and Market Demands. San Francisco, CA: Jossey-Bass.
Burke, L. M., & Butler, S. M. (2012). Accreditation: Removing the Barrier to Higher Education
Reform (Vol. 2728). Washington, DC: The Heritage Foundation.
Byrne, T. C. (1989). Athabasca University: The evolution of distance education. Calgary,
Canada: University of Calgary Press.
Cabrera, A. F., Colbeck, C. L., & Terenzini, P. T. (2001). Developing performance indicators for
assessing classroom teaching practices and student learning. Research in Higher
Education, 42(3), 327-352.
Campaign for the Future of Higher Education. (2013). The promises of higher education: Access.
http://futureofhighered.org/wp-content/uploads/2013/10/Promises-of-Online-Higher-Ed-
Access2.pdf
Canvas Network. (2015). About Us. Retrieved from https://www.canvas.net/pages/about-us
Carey, K. (2012). A future full of badges. Chronicle of Higher Education, 58, 32.
Ch, S. K., & Popuri, S. (2013, December). Impact of online education: A study on online
learning platforms and edX. Paper presented at the Innovation and Technology in
Education (MITE), 2013 IEEE International Conference in MOOC.
Chafkin, M. (2013). Udacity's Sebastian Thrun, Godfater of free online education, changes
course. Fast Company. http://www.fastcompany.com/3021473/udacity-sebastian-thrun-
uphill-climb

91
Chambers, C. M. (1983). Council on Postsecondary Education. In K. E. Young, C. M. Chambers
& H. Kells (Eds.), Understanding Accreditation. San Francisco, CA: Jossey-Bass.
Christensen, C. M., Horn, M. B., Caldera, L., & Soares, L. (2011). Disrupting College: How
Disruptive Innovation Can Deliver Quality and Affordability to Postsecondary Education.
Innosight Institute.
CIC. (2013). CIC online learning collaboration: A vision and framework. Champaign, IL:
Committee on Institutional Cooperation.
Conole, G. (2013). MOOCs as disruptive technologies: strategies for enhancing the learner
experience and quality of MOOCs. [Preprint]. Retrieved from
https://docs.google.com/document/d/1B6QAx6OiwK3VW16idU7mnHDuZljyy6r7gLXh
TzUa5co/edit?pli=1
Corrall, S., & Pinfield, S. (2014, March). Coherence of" Open" Initiatives in Higher Education
and Research: Framing a Policy Agenda. Paper presented at the iConference 2014
Proceedings.
Cremonini, L., Epping, E., Westerheijden, D. F., & Vogelsang, K. (2012). Impact of quality
assurance on cross-border higher education. Enschede, Netherlands: Center for Higher
Education Policy Studies.
Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods
approaches. Thousand Oaks, CA: Sage Publications.
Davenport, C. A. (2000). Recognition chronology. Retrieved from: http://www.aspa-
usa.org/documents/Davenport.pdf

92
Day, D. (2014). "Mooc World": Experts clash over differing visions of education technology
[Press release]. Retrieved from Princeton University website:
http://www.princeton.edu/main/news/archive/S39/71/57E81/index.xml?section=topstorie
s
de Langen, F., & van den Bosch, H. (2013). Massive open online courses: Disruptive innovations
or disturbing inventions? Open Learning: The Journal of Open, Distance and e-Learning,
28(3), 216-226.
Denzin, N. (1984). The research act. Englewood Cliffs, NJ: Prentice Hall.
Dewatripont, M., van Pottelsberghe, B., Sapir, A., & Veugelers, R. (2010). Boosting innovation
in Europe. Brussels: Bruegel policy contribution.
Dickeson, R. C. (2006). The Need for Accreditation Reform. Washington, DC: Secretary of
Education’s Commission on the Future of Higher Education.
Dickeson, R. C. (2009). Recalibrating the accreditation-federal relationship. Washington, DC:
Council for Higher Education Accreditation.
Dickey, F. G., & Miller, J. W. (1972). Federal involvement in nongovernmental accreditation.
Educational Record, 53(2), 139.
Dill, D. (2007). Quality assurance in higher education: Practices and issues. In B. McGaw, E.
Baker & P. P. Peterson (Eds.), The 3rd International Encyclopedia of Education.
Dill, D. D., Massy, W. F., Williams, P. R., & Cook, C. M. (1996). Accreditation & academic
quality assurance: can we get there from here? Change: The Magazine of Higher
Learning, 28(5), 17-24.

93
DiMaggio, P. J., & Powell, W. W. (1983). The iron cage revisited: Institutional isomorphism and
collective rationality in organizational fields. American Sociological Review 48(2), 147-
160.
Dowd, A. C. (2003). From access to outcome equity: Revitalizing the democratic mission of the
community college. The ANNALS of the American Academy of Political and Social
Science, 586(1), 92-119.
Dowd, A. C., & Grant, J. L. (2006). Equity and efficiency of community college appropriations:
The role of local financing. The Review of Higher Education, 29(2), 167-194.
e-InfraNet. (2013). e-InfraNet: ‘Open’ as the default modus operandi for research and higher
education. Dublin, Ireland: European Network for co-ordination of policies and
programmes on e-infrastructures (e-InfraNet Project).
Eaton, J. S. (2012a). MOOCs and accreditation: Focus on the quality of "direct-to-students"
education. Inside Accreditation, 9(1). Retrieved from:
http://www.chea.org/ia/IA_2012.10.31.html
Eaton, J. S. (2012b). An Overview of US Accreditation--Revised. Council for Higher Education
Accreditation. Washington, DC.
Ebben, M., & Murphy, J. S. (2014). Unpacking MOOC scholarly discourse: A review of nascent
MOOC scholarship. Learning, Media and Technology (ahead-of-print), 1-18.
Economist. (2013a, July 20). The attack of the MOOCs. The Economist. Retrieved from
http://www.economist.com/news/business/21582001-army-new-online-courses-scaring-
wits-out-traditional-universities-can-they
Economist. (2013b). Teachers, students and machines: The democratisation of education. The
Economist. Retrieved from

94
http://www.economistinsights.com/sites/default/files/downloads/Article 3 -
Education_Humans & machines.pdf
EdCast. (2015). About Us. Retrieved from http://www.edcast.com/corp/about_us
EDUCAUSE. (2012). 7 things you should know about badges. Retrieved from:
http://net.educause.edu/ir/library/pdf/eli7085.pdf
Emanuel, E. J. (2013). Online education: MOOCs taken by educated few. Nature, 503(7476),
342.
European Foundation for Quality in e-Learning. (2014). MOOC Quality Project - perspectives on
quality of MOOC-based education.
Ewell, P. (2001). Accreditation and student learning outcomes: A proposed point of departure
from Council for Higher Education Accreditation
http://www.chea.org/award/StudentLearningOutcomes2001.pdf
Ewell, P. (2005). Can assessment serve accountability? It depends on the question. In J. C. Burke
(Ed.), Achieving accountability in higher education: Balancing public, academic, and
market demands (pp. 78-105). San Francisco: Jossey-Bass.
Ewell, P. (2009). Assessment, accountability, and improvement: Revisiting the tension (NILOA
Occasional Paper No. 1). Chicago, IL: University of Illinois and Indiana University,
National Institute of Learning Outcomes Assessment.
Ewell, P. (2010). Twenty years of quality assurance in higher education: what’s happened and
what’s different? Quality in Higher education, 16(2), 173-175.
Ewell, P. T. (1993). The role of states and accreditors in shaping assessment practice. In T. W.
Banta (Ed.), Making a difference: Outcomes of a decade of assessment in higher
education (pp. 339-356). San Francisco, CA: Jossey-Bass.

95
Ewell, P. T. (2002). An emerging scholarship: A brief history of assessment. In T. W. Banta
(Ed.), Building a scholarship of assessment (pp. 3-25). San Francisco, CA: John Wiley &
Sons.
Ewell, P. T. (2008). Assessment and accountability in America today: Background and context.
New Directions for Institutional Research, 2008(S1), 7-17.
Figlio, D., Rush, M., & Yin, L. (2010). Is it live or is it internet? Experimental estimates of the
effects of online instruction on student learning. Cambridge, MA: National Bureau of
Economic Research.
Fowler, G. (2013, October 8). An early report card on massive open online courses. The Wall
Street Journal. Retrieved from
http://online.wsj.com/news/articles/SB10001424052702303759604579093400834738972
Friedman, T. (2013, January 26). Revolution hits the universities. The New York Times.
Retrieved from http://www.nytimes.com/2013/01/27/opinion/sunday/friedman-
revolution-hits-the-universities.html
Fuller, M. B., & Lugg, E. T. (2012). Legal precedents for higher education accreditation. Journal
of Higher Education Management, 27(1).
Gallup. (2013). The 2013 Inside Higher Ed survey of College and University Chief Academic
Officers. In S. Jaschik & D. Lederman (Eds.). Washington, DC: Inside Higher Ed.
Gaston, P. L. (2013). Higher Education accreditation: how it's changing, why it must. Sterling,
VA: Stylus Publishing, LLC.
Gates, J., Mundie, C., & Jackson, S. A. (2013). PCAST considers massive open online courses
(MOOCs) and related technologies in higher education [Press release]. Retrieved from

96
http://www.whitehouse.gov/blog/2013/12/18/pcast-considers-massive-open-online-
courses-moocs-and-related-technologies-higher-ed
Graham, P. A. (1995). Accountability of colleges and universities: An essay. New York, NY:
Columbia University.
Grossman, S. (2013). New research effort aims to examine effectiveness of MOOCs. The
Chronicle of Higher Education. Retrieved from
http://chronicle.com/blogs/wiredcampus/new-research-effort-aims-to-examine-
effectiveness-of-moocs/44217
Grover, S., Franz, P., Schneider, E. and Pea, R. (2013, June). The MOOC as distributed
intelligence: Dimensions of a framework for the design and evaluation of MOOCs. Paper
presented at the 10th International Conference on Computer Supported Collaborative
Learning, Madison, WI.
Haggard, S. (2013). The maturing of the MOOC: Literature review of massive open online
courses and other forms of online distance learning. (Vol. 130). London, England: UK
Department for Business, Innovation and Skills.
Harcleroad, F. F. (1976). Educational auditing and accountability. Washington, DC: Council on
Postsecondary Accreditation.
Harcleroad, F. F. (1980). Accreditation--history, process, and problems (Vol. 6). Washington,
DC: American Association for Higher Education
Herman, R. L. (2014). Letter from the Editor-in-Chief: MOOCs-How are they doing? The
Journal of Effective Teaching, 1.
Horn, M. B., & Christensen, C. (2013, February 20). Beyond the buzz, where are MOOCs really
going? WIRED.

97
Hou, A. Y. C., Morse, R., Chiang, C.-L., & Chen, H.-J. (2013). Challenges to quality of English
medium instruction degree programs in Taiwanese universities and the role of local
accreditors: a perspective of non-English-speaking Asian country. Asia Pacific Education
Review, 14(3), 359-370.
Hylén, J. (2005). Open educational resources: Opportunities and challenges. OECD - The Centre
for Educational Research and Innovation (CERI): Retrieved from
http://www.oecd.org/dataoecd/1/49/35733548.doc
Ikenberry, S. (2006). American higher education: The new balancing act. In R. Clark & M.
d'Ambrosio (Eds.), The new balancing act in the business of higher education.
Northhampton, MA: Edward Elgar.
Ikenberry, S. (2009. January). Where do we take accreditation? Paper presented at the CHEA
Annual Meeting, Washington, DC.
Irakliotis, L., & Johnstone, S. (2014). Competency-based education programs versus traditional
data management. EDUCAUSE Review Online. Retrieved from:
http://www.educause.edu/ero/article/competency-based-education-programs-versus-
traditional-data-management
iversity. (2015). About Us. Retrieved from https://iversity.org/en/pages/about
Jackson, R. S., Davis, J. H., & Jackson, F. R. (2010). Redesigning regional accreditation: The
impact on institutional planning. Planning for Higher Education, 38(4), 9-19.
Jacobs, B., & Van der Ploeg, F. (2006). Guide to reform of higher education: a European
perspective. Economic Policy, 21(47), 535-592.

98
Jaggars, S. S., & Xu, D. (2013). Adaptability to online learning: Differences across types of
students and academic subject areas. New York, NY: Community College Research
Center. Teachers College Columbia University.
Jaschik, S., & Ledgerman, D. (2014). The 2014 Inside Higher Ed survey of college & university
presidents. Washington, DC: Inside Higher Ed.
Jordan, K. (2014). Initial trends in enrolment and completion of massive open online courses.
The International Review of Research in Open and Distance Learning, 15(1), 133-160.
Jordan, S. (2013). E-assessment: Past, present and future. New Directions, 9(1), 87-106.
Karsenti, T. (2013). What the research says. International Journal of Technologies in Higher
Education, 10(2), 23-37.
Kaupp, R. (2012). Online penalty: The impact of online instruction on the Latino-White
achievement gap. Journal of Applied Research in the Community College, 12(2), 1-9.
Kazakoff-Lane, C. (2014). Environmental scan and assessment of OERs, MOOCs and libraries.
Chicago, IL: Association of College and Research Libraries.
Kezar, A. (2014, April 11). Innovator or protector of status quo? Inside Higher Ed. Retrieved
from http://www.insidehighered.com/views/2014/04/11/accreditation-pushes-colleges-
innovate-not-stagnate-essay - sthash.wgRTVeo2.YNU2tYNQ.dpbs
Koller, D., Ng, A., Do, C., & Chen, Z. (2013). Retention and intention in massive open online
courses: In depth. EDUCAUSE Review Online.
Kolowich, S. (2013a, February 7). American Council on Education recommends 5 MOOCs for
credit. The Chronicle of Higher Education. Retrieved from
http://chronicle.com/article/American-Council-on-Education/137155/

99
Kolowich, S. (2013b). The professors who make the MOOCs. The Chronicle of Higher
Education. Retrieved from http://chronicle.com/article/The-Professors-Behind-the-
MOOC/137905/ - id=overview
Kuh, G. D., & Ewell, P. T. (2010). The state of learning outcomes assessment in the United
States. Higher Education Management and Policy, 22(1), 1-20.
Kuh, G. D., & Ikenberry, S. (2009). More than you think, less than we need: Learning outcomes
assessment in American higher education. Chicago, IL: National Institute for Learning
Outcomes Assessment.
Laso, J. A., Peco, P. P., & Luján-Mora, S. (2013). Using open badges as certification on a
MOOC. ICERI 2013 Proceedings, 1809-1818.
Lawton, W., & Katsomitros, A. (2012). MOOCs and disruptive innovation: The challenge to HE
business models. Retrieved from: http://www.obhe.ac.uk/documents/view_details?id=929
Leblanc, P. (2013). The new for-profits. Inside Higher Ed. Retrieved from
http://www.insidehighered.com/views/2013/05/31/nonprofit-colleges-should-be-wary-
new-breed-profit-players-essay - sthash.XB8zrTFO.dpbs
Legon, R. (2013). MOOCs and the quality question. Inside Higher Ed. Retrieved from
http://www.insidehighered.com/views/2013/04/25/moocs-do-not-represent-best-online-
learning-essay - sthash.ZQRddjrr.dpbs
Lewin, T. (2014, February 13). Free online university receives accreditation, in time for
graduating class of 7. The New York Times. Retrieved from
http://www.nytimes.com/2014/02/14/education/free-online-university-receives-
accreditation-in-time-for-graduating-class-of-7.html

100
Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: A systematic
study of the published literature 2008-2012. International Review of Research in Open &
Distance Learning, 14(3).
Lomas, L. (2002). Does the development of mass education necessarily mean the end of quality?
Quality in Higher education, 8(1), 71-79.
Maki, P. L. (2010). Assessing for learning: Building a sustainable commitment across the
institution (2nd ed.). Sterling, VA: Stylus Publishing, LLC.
Marques, J. (2014). A short history of MOOCs and distance learning. Retrieved from
http://moocnewsandreviews.com/a-short-history-of-moocs-and-distance-learning/
Mazoue, J. G. (2013). The MOOC model: Challenging traditional education. EDUCAUSE
Review Online. Retrieved from: http://www.educause.edu/ero/article/mooc-model-
challenging-traditional-education
McEwan, E. K., & McEwan, P. J. (2003). Making sense of research: What's good, what's not,
and how to tell the difference. Thousand Oaks, CA: Corwin Press.
Middaugh, M. F. (2012). Introduction to themed PHE issue on accreditation in higher education.
Planning for Higher Education, 40(3), 6-7.
Miles, J. A. (2012). Management and Organization Theory: A Jossey-Bass Reader (Vol. 9).
Hoboken, NJ: John Wiley & Sons.
Moloney, J. F., & Oakley, I. (2010). Scaling online education: Increasing access to higher
education. Journal of Asynchronous Learning Networks, 14(1).
Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Verification strategies for
establishing reliability and validity in qualitative research. International journal of
qualitative methods, 1(2), 13-22.

101
Moss, L. (2013). The tolerable contradictions of massive open online courses (MOOCs) and the
potential role of prior learning assessment (PLA). PLA Inside Out: An International
Journal on Theory, Research and Practice in Prior Learning Assessment, 2(1).
Nickias, C. L. M. (2012). Online education - hype and reality [Press release]. Retrieved from
http://www.immagic.com/eLibrary/ARCHIVES/GENERAL/USC_CAUS/U120827N.pdf
O’Connor, K. (2014). MOOCs, institutional policy and change dynamics in higher education.
Higher Education, 1-13.
Obama, B. (2013a). Fact Sheet on the President’s Plan to Make College More Affordable: A
Better Bargain for the Middle Class. Washington, DC: The White House Retrieved from
http://www.whitehouse.gov/the-press-office/2013/08/22/fact-sheet-president-s-plan-
make-college-more-affordable-better-bargain-.
Obama, B. (2013b). The President’s Plan for a Strong Middle Class and a Strong America.
Washington, DC: The White House. Retrieved from
http://www.whitehouse.gov/sites/default/files/uploads/sotu_2013_blueprint_embargo.pdf.
Obama, B. (2013c). State of the Union Address. Washington, DC: The White House. Retrieved
from http://www.whitehouse.gov/the-press-office/2013/02/12/president-barack-obamas-
state-union-address.
Onwuegbuzie, A. J., & Leech, N. L. (2007). A call for qualitative power analyses. Quality &
Quantity, 41(1), 105-121.
Open Universities Australia. (2015). Who we are. Retrieved from
http://www.open.edu.au/about-us/who-we-are/.
OpenLearning. (2015). What is OpenLearning? Retrieved from
https://www.openlearning.com/About.

102
Orlans, H. O. (1974). Private accreditation and public eligibility (Vol. 1 & 2). Washington, DC:
United States Department of Education.
Pappano, L. (2012, November 2). The year of the MOOC. The New York Times. Retrieved from
http://www.nytimes.com/2012/11/04/education/edlife/massive-open-online-courses-are-
multiplying-at-a-rapid-pace.html
Perrault, A. H., Gregory, V., & Carey, J. (2002). The Integration of assessment of student
learning outcomes and teaching effectiveness. JELIS, 43(4), 270-282.
Powell, C. (2013). Accreditation, assessment, and compliance: Addressing the cyclical
challenges of public confidence in american education. . Journal of Assessment and
Institutional Effectiveness, 3(1), 54-74.
Price, E. (2013). Quality assurance for massive open online courses. London, England:
University of London.
Procopio, C. H. (2010). Differing administrator, faculty, and staff perceptions of organizational
culture as related to external accreditation. Academic Leadership Journal, 8(2), 100-109.
Provezis, S. (2010). Regional accreditation and student learning outcomes: Mapping the
territory. Champaign, IL: National Institute for Learning Outcomes Assessment,
University of Illinois at Urbana-Champaign.
Radford, A. W., Robles, J., Cataylo, S., Horn, L., Thorton, J., & Whitfield, K. (2014). The
employer potential of MOOCs: A survey of human resource professionals' thinking on
MOOCs. Durham, NC & Research Triangle Park, NC: Duke University and RTI
International.
Reinalda, B., & Kulesza-Mietkowski, E. (2005). The Bologna process: Harmonizing Europe's
higher education. Farmington Hills, MI: Barbara Budrich

103
Rhodes, T. L. (2012). Show me the learning: Value, accreditation, and the quality of the degree.
Planning for Higher Education, 40(3), 36-42.
Rizzardini, R. H., Gütl, C., Chang, V., & Morales, M. (2014). MOOC in Latin America:
Implementation and lessons learned. Paper presented at the The 2nd International
Workshop on Learning Technology for Education in Cloud, Netherlands.
Rosewell, J., Kear, K., & Williams, K. (2012). Next steps for excellence in the quality of e-
learning. Paper presented at the ALT-C 2012: A Confrontation with Reality, Manchester,
UK.
Rosselle, M., Caron, P.-A., & Heutte, J. (2014, February). A typology and dimensions of a
description framework for MOOCs. Proceedings of the European MOOC Stakeholder
Summit 2014, 130-139.
Rovai, A. P. (2000). Online and traditional assessments: What is the difference? The Internet and
Higher Education, 3(3), 141-151.
Ruppert, S. S. (1994). Charting Higher Education Accountability: A Sourcebook on State-Level
Performance Indicators.
Salmi, J. (2003). Constructing knowledge societies: new challenges for tertiary education.
Higher Education in Europe, 28(1), 65-69.
Sandeen, C. (2013). Assessment’s place in the new MOOC world. Research & Practice in
Assessment, 8(1), 5-12.
Schneider, E. (2013). Welcome to the moocspace: A proposed theory and taxonomy for massive
open online courses. Paper presented at the AIED 2013, Memphis, TN.
Schrage, M. (2012). Four Innovation Trends to Watch in 2013. Retrieved from
http://blogs.hbr.org/2012/12/four-innovation-trends-to-watc/.

104
Shah, D. (2013, December 22). MOOCs in 2013: Breaking down the numbers. EdSurge.
Retrieved from https://http://www.edsurge.com/n/2013-12-22-moocs-in-2013-breaking-
down-the-numbers
Sharif, A., Moeini, H., & Gisbert, M. (2013). Quality of online learning: Adding MOOC into the
mix? Paper presented at the World Conference on Educational Multimedia, Hypermedia
and Telecommunications.
Shaw, R. (1993). A backward glance: To a time before there was accreditation. NCA Quarterly,
68(2), 323-335.
Sonwalkar, N. (2013). The first adaptive MOOC: A case study on pedagogy framework and
scalable cloud architecture - part I. Paper presented at the MOOCs Forum, New York,
NY.
Spangehl, S. D. (2012). AQIP and accreditation: Improving quality and performance. Planning
for Higher Education, 40(3), 29-35.
Spangehl, S. D., & Lindborg, H. J. (2012). Does accreditation inhibit innovation? In A. Hoffman
& S. D. Spangehl (Eds.), Innovations in Higher Education: Igniting the Spark for
Success. New York, NY: Rowman & Littlefield Publishers.
Spellings, M. (2006). A test of leadership: Charting the future of US higher education.
Washington, DC: US Department of Education.
Stein, K. (2013). PENN GSE Study shows MOOCs have relatively few active users, with only a
few persising to course end [Press release]. Retrieved from
http://www.gse.upenn.edu/pressroom/press-releases/2013/12/penn-gse-study-shows-
moocs-have-relatively-few-active-users-only-few-persisti

105
Stensaker, B., & Harvey, L. (2006). Old wine in new bottles? A comparison of public and private
accreditation schemes in higher education. Higher Education Policy, 19(1), 65-85.
Stripling, J. (2012, July 2). UVa's painfully public lesson in leadership. The Chronicle of Higher
Education. Retrieved from http://chronicle.com/article/UVas-Painfully-Public-
Lesson/132701/
Sursock, A., & Smidt, H. (2010). Trends 2010: A decade of change in European higher
education. Brussels, Belgium: European University Association.
Keeping college within reach: Improving higher education through innovation, U.S. House of
Representatives (2013).
Terrell, R. L., & Caudill, J. G. (2012). OpenCourseWare: Open sharing of course content and
design. Journal of computing sciences in colleges, 27(3), 38-42.
Torres, A. L. M., Pacheco, N. B., Pacheco, T., Novo, C., Galego, J., & Dias, J. M. (2013,
October). MOOC: The first experience in a Portuguese institute of higher education.
Paper presented at the World Conference on E-Learning in Corporate, Government,
Healthcare, and Higher Education.
Trivett, D. A. (1976). Accreditation and institutional eligibility. Washington, DC: American
Association for Higher Education.
Udacity. (2015). About Us. Retrieved from https://www.udacity.com/us.
Udemy. (2015). About Us. Retrieved from https://about.udemy.com/.
Van Damme, D. (2000). Internationalization and quality assurance: towards worldwide
accreditation? European Journal for Education Law and Policy, 4(1), 1-20.
Van der Ploeg, F., & Veugelers, R. (2008). Towards evidence-based reform of European
universities. CESifo Economic Studies, 54(2), 99-120.

106
Van Der Werf, M., & Sabatier, G. (2009). The college of 2020: Students. Washington, DC:
Chronicle Research Services.
Volkwein, J. F., Lattuca, L. R., Harper, B. J., & Domingo, R. J. (2007). Measuring the impact of
professional accreditation on student experiences and learning outcomes. Research in
Higher Education, 48(2), 251-282.
Wasik, J. (2014, March 17). What color is your online adult course? The New York Times.
Retrieved from http://www.nytimes.com/2014/03/18/education/what-color-is-your-
online-adult-course.html
Watters, A. (2013). Top ed-tech trends of 2013: MOOCs and anti-MOOCs. Retrieved from
http://hackeducation.com/2013/11/29/top-ed-tech-trends-2013-moocs/
Weissburg, P. (2008). Shifting alliances in the accreditation of higher education: On the long
term consequences of the delegation of government authority to self-regulatory
organizations. (Doctoral Dissertation), George Mason University, Fairfax, VA.
Wergin, J. F. (2005). Waking up to the importance of accreditation. Change, 37(3), 35-41.
Wergin, J. F. (2012). Five essential tensions in accreditation. In M. LaCelle-Peterson & D.
Rigden (Eds.), Inquiry, evidence, and excellence: The promise and practice of quality
assurance (pp. 27-38). Washington, DC: Teacher Education Accreditation Council.
White, S. C., & Glickman, T. S. (2007). Innovation in higher education: Implications for the
future. New Directions for Higher Education, 2007(137), 97-105.
Whitelock, D., Gilbert, L., & Wills, G. (2013). Feedback generators: providing feedback in
MOOCs. Paper presented at the Computer Assisted Assessment Conference,
Southampton, UK.

107
Wiedman, D. (1992). Effects on academic culture of shifts from oral to written traditions: The
case of university accreditation. Human Organization, 51(4), 398-407.
Wilkowski, J., Deutsch, A., & Russell, D. M. (2014). Student skill and goal achievement in the
mapping with Google MOOC.
Wilkowski, J., Russell, D. M., & Deutsch, A. (2014). Self-evaluation in advanced power
searching and mapping with Google MOOCs. L@ S 2014, 04-05.
Wolff, R. A. (2005). Accountability and accreditation: Can reforms match increasing demands?
In J. C. Burke (Ed.), Achieving accountability in higher education: Balancing public,
academic, and market demands. (pp. 37-67). San Francisco, CA: Jossey-Bass.
Woolston, P. (2012). The Costs of Institutional Accreditation: a Study of Direct and Indirect
Costs. (Doctoral Dissertation), University of Southern California, Los Angeles, CA.
Yuan, L., & Powell, S. (2013). MOOCs and open education: Implications for higher education.
Cetis White Paper.

Abstract (if available)

Abstract This qualitative study explored how Massive Open Online Course (MOOC) providers perceive the concepts of assessment, accountability, and accreditation. Seven MOOC providers were interviewed via Skype, telephone and email correspondence. Their responses were transcribed, coded and analyzed to identify common themes. Respondents perceived MOOCs as part of a larger shift in education towards online learning, which requires assessment to become more student-centered. Regarding accountability, students are the primary stakeholders. By focusing on direct measures of student success, MOOCs will potentially decouple traditional measures of higher education prestige and rankings. Regarding accreditation, MOOCs could be institutionally accredited when they are delivered as part of a degree program, and directly accredited when used for professional education. Additional research is needed to determine how accrediting bodies can better incentivize MOOCs to participate in the accreditation process. The outcomes from this study can inform educational leaders, policy makers, and accrediting bodies on how MOOC providers themselves wish to be assessed.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Perspectives on accreditation and leadership: a case study of an urban city college in jeopardy of losing accreditation

PDF

Learning outcomes assessment at American Library Association accredited master's programs in library and information studies

PDF

Priorities and practices: a mixed methods study of journalism accreditation

PDF

An examination of the direct/indirect measures used in the assessment practices of AACSB-accredited schools

PDF

The effects of accreditation on the passing rates of the California bar exam

PDF

An evaluation of nursing program administrator perspectives on national nursing education accreditation

PDF

A descriptive analysis focusing on similarities and differences among the U.S. service academies

PDF

The costs of institutional accreditation: a study of direct and indirect costs

PDF

The benefits and costs of accreditation of undergraduate medical education programs leading to the MD degree in the United States and its territories

PDF

The goals of specialized accreditation: A study of perceived benefits and costs of membership with the National Association of Schools of Music

PDF

States of motivation: examining perceptions of accreditation through the framework of self-determination

PDF

A cost benefit analysis of professional accreditation by ABET for baccalaureate engineering degree programs

PDF

Assessment and accreditation of undergraduate study abroad programs

PDF

Input-adjusted transfer scores as an accountability model for California community colleges

PDF

A study of the pedagogical strategies used in support of students with learning disabilities and attitudes held by engineering faculty

PDF

Accreditation and accountability processes in California high schools: a case study

PDF

Examining the implications of return on investment using the gap analysis framework on the executive master of business administration program at a four year research university

PDF

An exploratory, quantitative study of accreditation actions taken by the Western Association of Schools and Colleges' Accrediting Commission for Community and Junior Colleges Since 2002

PDF

Learning 21st century skills In a multicultural setting

PDF

PowerPoint design based on cognitive load theory and cognitive theory of multimedia learning for introduction to statistics

Asset Metadata

Creator May, Richard (author)

Core Title Assessment, accountability & accreditation: a study of MOOC provider perceptions

School Rossier School of Education

Degree Doctor of Education

Degree Program Education (Leadership)

Publication Date 06/17/2015

Defense Date 03/09/2015

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag accountability,accreditation,assessment,MOOC,OAI-PMH Harvest

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Keim, Robert G. (committee chair), Tobey, Patricia Elaine (committee member), Woolston, Paul J. (committee member)

Creator Email RichardMay08@gmail.com,rlmay@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-571941

Unique identifier UC11301730

Identifier etd-MayRichard-3474.pdf (filename),usctheses-c3-571941 (legacy record id)

Legacy Identifier etd-MayRichard-3474.pdf

Dmrecord 571941

Document Type Dissertation

Format application/pdf (imt)

Rights May, Richard

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA