University of Southern California Dissertations and Theses

Adaptive learning in higher education: an evaluation study

(USC Thesis Other)

Adaptive learning in higher education: an evaluation study

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content

Adaptive Learning in Higher Education: An Evaluation Study
by
Greg Akai

Rossier School of Education
University of Southern California
A dissertation submitted to the faculty
in partial fulfillment of the requirements for the degree of
Doctor of Education

December 2020

© Copyright by Greg Akai 2020
All Rights Reserved

The Committee for Greg Akai certifies the approval of this Dissertation

Kimberly Hirabayashi
Karen Vignare
Kenneth Yates, Committee Chair

Rossier School of Education
University of Southern California
2020

iv
Abstract
This study consisted of a needs’ analysis to examine knowledge, motivation, and
organizational influences on the implementation of adaptive learning courseware. Clark and
Estes’s (2008) gap analysis framework was used to analyze and evaluate courseware
implementation. To address challenges, some colleges and universities adopted adaptive
learning, integrating adaptive courseware, to support and augment learning and instruction. The
purpose of this study was to conduct an evaluation of adaptive learning implementation at eight
higher education institutions. This analysis focused on stakeholders’ areas of knowledge and
skill, motivation, and organizational resources for adaptive learning programs. While a complete
study and evaluation would address all stakeholders, both in higher education institutions and
associated entities, for practical purposes, the stakeholders of focus were the project leads
responsible for the adaptive learning implementation programs at their respective universities.

v
Acknowledgements
The past three and a half years have been incredible journey for me. I am grown
tremendously in many ways, both personally and professionally. I have been lucky enough to
have an amazing family and group of friends, colleagues and professors that have supported and
kept me motivated throughout this process. To my wife, I would not have been able to do all this
without you and your support. You are my soul mate and I love you so much! To my daughter,
daddy can now spend more time with you. I love you! To my mother, you have supported me
through everything since the beginning. Your support made this doctoral journey possible and I
am very grateful for your love and how you have helped me throughout my life. To Cory, you
have been the best cohort mate I could have asked for and your support has allowed me to make
it to finally make this happen. To Shannon, you have supported me from the very first day of my
doctoral program, always with encouraging words, allowing me to adjust my schedule for school
and just being an unbelievably compassionate and caring person. I am most fortunate to have a
boss like you! To my colleagues, Michael, Annette and Darin, your continuing support has been
incredible. To my committee Dr. Kenneth Yates, Dr. Kimberly Hirabayashi and Dr. Karen
Vignare, thank you for your guidance, patience and support throughout the program.

vi
Table of Contents
Abstract ...........................................................................................................................................iv
Acknowledgements ..........................................................................................................................v
List of Tables ..................................................................................................................................xi
List of Figures ................................................................................................................................xii
Chapter One: Introduction of the Problem of Practice ....................................................................1
Organizational Context and Mission ...................................................................................2
Association of Public and Land-Grant Universities ......................................................2
Personalized Learning Consortium ................................................................................3
Organization Performance Goal...........................................................................................5
Related Literature.................................................................................................................7
Moving Toward Student-Centered Learning .................................................................8
An Understanding of Adaptive Learning.......................................................................9
The Impact of Adaptive Learning ................................................................................10
Importance of the Evaluation .............................................................................................11
Stakeholders and Stakeholders’ Performance Goals .........................................................12
Stakeholder for the Study and Stakeholder Performance Goal .........................................12
Project Leads’ Critical Behaviors ......................................................................................13
Purpose of the Project and Questions ................................................................................14
Methodological Framework ...............................................................................................14
Definitions..........................................................................................................................15
Organization of the Study ..................................................................................................16
Chapter Two: Review of the Literature .........................................................................................17

vii
Adaptive Learning..............................................................................................................17
Adaptive Courseware .........................................................................................................21
Adaptive Courseware Implementation...............................................................................23
Adaptive Courseware Implementation - Faculty Perspective......................................26
Adaptive Courseware Implementation - Student Perspective .....................................27
Adaptive Courseware Implementation - Future Expectations .....................................28
Conceptual Framework ......................................................................................................29
Stakeholder Knowledge, Motivation and Organizational Influences ................................30
Knowledge and Skills ..................................................................................................30
Motivation Influence: PLs Need to be Motivated to Implement Adaptive Courseware
......................................................................................................................................35
Organization.................................................................................................................38
Chapter Three: Methodology .........................................................................................................42
Conceptual and Methodological Framework .....................................................................42
Assessment of Performance Influences .............................................................................43
Knowledge Assessment ...............................................................................................44
Motivation Assessment ................................................................................................46
Organization/Culture/Context Assessment ..................................................................47
Participating Stakeholders and Sample Selection ..............................................................49
Sampling ......................................................................................................................49
Recruitment ..................................................................................................................49
Interview Protocols ............................................................................................................49
Document Analysis ............................................................................................................50

viii
Data Collection ..................................................................................................................50
Interviews.....................................................................................................................50
Document Analysis ......................................................................................................50
Data Analysis .....................................................................................................................51
Interviews.....................................................................................................................51
Documents ...................................................................................................................51
Trustworthiness of Data .....................................................................................................52
Role of Investigator............................................................................................................52
Limitations and Delimitations............................................................................................52
Chapter Four: Results and Findings...............................................................................................54
Participating Stakeholders..................................................................................................56
Interviews...........................................................................................................................57
Document Analyses ...........................................................................................................58
Findings..............................................................................................................................58
Knowledge Influence 1: Question 1.............................................................................58
Knowledge Influence 1: Question 2.............................................................................62
Knowledge Influence 1: Question 3.............................................................................66
Knowledge Influence 2: Question 1.............................................................................69
Knowledge Influence 2: Question 2.............................................................................73
Knowledge Influence 2: Question 3.............................................................................76
Knowledge Influence 2: Question 4.............................................................................79
Knowledge Influence 3: Question 1.............................................................................84
Knowledge Influence 3: Question 5.............................................................................88

ix
Knowledge Influence 4 ................................................................................................91
Motivation Influence....................................................................................................95
Organizational Influence 1.........................................................................................102
Organizational Influence 2.........................................................................................106
Summary ..........................................................................................................................111
Chapter Five: Discussion .............................................................................................................114
Recommendations to Address Knowledge, Motivation, and Organization Influences ...114
Knowledge Recommendations ..................................................................................116
Motivation Recommendations ...................................................................................123
Organization Recommendations ................................................................................129
Summary of Knowledge, Motivation, and Organization Recommendations ..................136
Integrated Implementation and Evaluation Plan ..............................................................138
Organizational Purpose, Need, and Expectations ......................................................138
Implementation and Evaluation Framework..............................................................139
Level 4: Results and Leading Indicators....................................................................140
Level 3: Behavior.......................................................................................................142
Level 2: Learning .......................................................................................................148
Level 1: Reaction Evaluation Tools...........................................................................153
Data Analysis and Reporting .....................................................................................155
Summary of the Implementation and Evaluation ............................................................156
Limitations and Delimitations..........................................................................................157
Recommendations for Future Research ...........................................................................159
Conclusion .......................................................................................................................160

x
References ....................................................................................................................................162
Appendix A: Immediate Evaluation Tool ....................................................................................167
Appendix B: Delayed Evaluation Tool ........................................................................................169
Appendix C: Informed Consent/Information Sheet .....................................................................171

xi
List of Tables
Table 1 Summary of Assumed Knowledge Influences on Stakeholder’s Ability to Achieve
the Performance Goal.....................................................................................................................34
Table 2 Summary of Assumed Motivation Influences on Stakeholder’s Ability to Achieve
the Performance Goal.....................................................................................................................37
Table 3 Summary of Assumed Organization Influences on Stakeholder’s Ability to Achieve
the Performance Goal.....................................................................................................................40
Table 4 Summary of Assumed Knowledge Influences on Project Leads’ Ability to Achieve
the Performance Goals ...................................................................................................................45
Table 5 Summary of Motivation Influences and Method of Assessment......................................46
Table 6 Summary of Organization Influences and Method of Assessment...................................48
Table 7 Summary of Knowledge Influences and Recommendations ..........................................116
Table 8 Summary of Motivation Influences and Recommendations...........................................124
Table 9 Summary of Organization Influences and Recommendations .......................................129
Table 10 Outcomes, Metrics, and Methods for External and Internal Outcomes........................141
Table 11 Critical Behaviors, Metrics, Methods, and Timing for Evaluation ..............................143
Table 12 Required Drivers to Support Critical Behaviors ...........................................................145
Table 13 Evaluation of the Components of Learning for the Program........................................152
Table 14 Components to Measure Reactions to the Program ......................................................154

xii
List of Figures
Figure 1 The Eight Universities Participating in the Adaptive Learning Courseware program......4
Figure 2 Adaptive Learning Position Within Technology in Higher Education ...........................18
Figure 3 Sequence of Steps in the Gap Analysis Process ..............................................................43
Figure 4 Assets as Critical Influences..........................................................................................115

1

Chapter One: Introduction of the Problem of Practice
Entrusted with a uniquely important and influential role, higher education continues to
face significant opportunities and critical challenges in an ongoing effort to serve an increasingly
growing, evolving, and transforming student population. With postsecondary student enrollment
projected to decrease until 2026, graduation rates are expected to fall accordingly (National
Center for Education Statistics, 2019). Acting and functioning as a conduit to opportunity,
knowledge creation, and social mobility, higher education serves an affirmed purpose of
blending knowledge, utility, and function by providing a broad-based education, stimulating
personal and intellectual growth, and instilling career-ready skills (U.S. Department of
Education, 2017). This balance between individual growth and the development of work-related
skills ideally precipitates success and fulfillment for graduates.
Today, challenges facing higher education, for both on-campus and online student
populations, are many. Large class sizes, expanded general education requirements, remedial and
developmental requisites, falling course completion rates, section and instructor variability, and
disparate student engagement are among the issues that confront both two- and four-year
institutions. While newer innovations such as learning management systems, real-time
collaborative cloud-based applications, online synchronous conferencing and inclusive
technology design aid long-serving traditional methods of instruction, progressive and dynamic
learning technologies promote and facilitate a new model of learning. The proliferation and
advancement of higher quality learning technology in recent years has corresponded with a
pronounced enthusiasm regarding technology use (Fain, 2013; Waters, 2014; Zimmer, 2014).
With the student at the center of instruction, personalized learning benefits from advances in
2

artificial intelligence, performance analytics, and big data to, in real-time, individualize and
adjust instruction for students and provide learner performance data to guide the instructor.
A current and continuing point of discussion to address the myriad of institutional
challenges in higher education is the growing influence and impact of technology. For the
nation’s 4,300 colleges and universities, school, program, and course technology-related
implementations, adoptions, and decisions in the near-term will likely result in far-reaching
implications for students, faculty, and administrators over the long term.
One emerging and evolving potential solution to address challenges in higher education is
adaptive learning (Educause, 2017). McGraw Hill defines adaptive learning as a field that uses
artificial intelligence to actively tailor content to each individual’s needs—draws upon
knowledge domains as diverse as machine learning, cognitive science, predictive analytics, and
educational theory—to make this learner-centered vision of education a reality. Without
universal guidelines, shared frameworks, or endorsed standards, implementations of adaptive
learning and adaptive courseware are in an inaugural and discipline-wide period of discovery and
inquiry. Without a comprehensive implementation plan and process, integration results of
adaptive courseware has shown to be inconsistent.
Organizational Context and Mission
Association of Public and Land-Grant Universities
The Association of Public and Land-grant Universities (APLU) is a research, policy, and
advocacy organization of public research universities, land-grant institutions, state university
systems, and higher education organizations based in Washington, DC. The association’s
members are 241 institutions, consisting of 212 campuses and 24 university systems.
3

APLU’s mission is to expand and improve student success to deliver the innovative
workforce of tomorrow; advance and promote research and discovery to improve society, foster
economic growth, and address global challenges; and build healthy, prosperous, equitable, and
vibrant communities locally and globally.
Personalized Learning Consortium
Founded in 2013, the Personalized Learning Consortium (PLC), under the purview of the
APLU, serves as a multi-collaboration of public universities to exchange information and best
practices about personalized learning technologies that will improve student outcomes and
success. The PLC seeks to leverage the benefits of scale by understanding new learning
technologies and process changes needed at the institutional level. The PLC offers its members
resources and opportunities to improve understanding, implementation, and scaling of higher
education learning technologies to personalize and improve the education experience. PLC
membership positions universities to better capture the significant economies of scale inherent in
technology and improving learning for students while containing costs.
The PLC emerged from an APLU project funded by the Bill & Melinda Gates
Foundation, and it currently operates under the aegis of APLU as a separate due-funded activity.
The consortium carries out important functions for members through a national office in
Washington DC, including market analyses and tracking, sharing a review of vendor
assessments, organizing product licensing, piloting new adaptive learning technologies, and
sharing of learning data.
The APLU PLC, in conjunction with the Bill and Melinda Gates Foundation (BMGF),
has launched and funded a research-based, multiple-school project to identify successful
implementations of adaptive learning in higher education. The APLU’s adoption of adaptive
4

courseware grant program is an effort to launch, communicate, and share discrete
implementations of adaptive learning at universities across the country. As such, this evaluation
examines the APLU’s 3-year effort to implement adaptive learning at eight leading universities
as shown in Figure 1.
Figure 1
The Eight Universities Participating in the Adaptive Learning Courseware Program

5

Organization Performance Goal
In the summer of 2016, the APLU awarded grants to eight universities to adopt,
implement and scale adaptive learning courseware in undergraduate and general education
The Accelerating Adoption of Adaptive Courseware at Public Universities (AAACPU) grant
program set a goal of achieving adaptive courseware use in a minimum of 15% to 20% general
education course enrollments at the eight participating institutions within the 3-year program
term (Spring 2017 to Fall 2019), equating to over 85,000 enrollments. The adaptive courseware
adoption plans at grantee institutions should provide a disproportionate impact for disadvantaged
learners and follow these guidelines:
Course-Level
● Concentrate on general education (100- and 200-level) courses for credit
● Be delivered primarily in blended learning environments
● Inclusion of multiple high-enrollment courses and/or high D/F/W courses
● Disproportional benefit for disadvantaged learners
● Engage multiple disciplines/departments including physical sciences, life
sciences, social sciences, and the humanities
Institution-Level
● Demonstrate commitment to faculty development and sustained engagement with
a serious review of teaching and learning in the general education curriculum and,
as part of the effort, scaled deployment of high-quality adaptive courseware from
approved supplies/vendors
● Connect to other institution-wide improvement initiatives with chancellor-level
and provost-level leadership and engagement
6

● Targeted adaptive courseware use in a minimum of 15% to 20% general
education course enrollments within the 3-year program term (Spring 2017 - Fall
2019)
The 3-year program’s goal is to improve undergraduate education and promote student
success through the implementation of adaptive learning courseware in high-enrollment, high-
risk courses. Additionally, the program aims to “speed post-secondary educators toward effective
use of high-quality adaptive courseware, and ensure the public universities personalize learning
for students in ways that promote completion while containing costs.” The eight schools
participating in the adaptive learning project are Arizona State University (ASU), Colorado State
University (CSU), Georgia State (GSU), Northern Arizona University (NAU), Oregon State
University (OSU), Portland State University (PSU), the University of Louisville (UL), and the
University of Mississippi (UM). The eight institutions comprise a cohort that shares
implementation best practices across institutions through an online community, monthly calls,
and annual convenings. Institutions are encouraged to share their experiences beyond the cohort
to improve the efficiency and effectiveness of adaptive courseware implementations.
To ensure that program grantees adopt high-quality instructional products, participating
institutions choose the adaptive technology providers and preferred courseware titles to be
adopted from a list of suppliers meeting the highest quality standards as determined by the Bill
and Melinda Gates Foundation. This list is updated to reflect new products entering the market
and changes in current offerings every six months. The most promising findings and best
practices from the grant program will be scaled through APLU’s national network of over 200
public institutions and shared publicly to better meet the educational needs of students across the
postsecondary landscape.
7

Now in its third year, the eight schools have published their mid-year 2019 reports which
detail grant activities and progress to this point. The results, based on 2018 data, show that,
across the eight universities, adaptive courseware is being used in 146 courses enrolling 55,600
students. These numbers compare favorably with first-year data which showed adaptive
courseware implemented in 66 courses enrolling 26,756 students. The number of classes and
students using adaptive courseware more than doubled from the 2017 to 2018 school years.
Some of the courseware used included products from McGraw Hill, Smart Sparrow, Realizeit,
CogBooks, Lumen Learning, and Knewton. The eight universities cite their courseware
enrollments through reports published by the APLU.
The results above generally represent the proximal goals of implementing the program
within the desired time period and guidelines established by APLU and the Gates Foundation.
The grant does not target specific achievement or graduation goals but, rather, enumerates
specific implementation usage across particular groups of undergraduate classes. The grant looks
for evidence of implementation of adaptive software, followed by progress towards adaptive
software usage at scale.
Related Literature
Technology can be a powerful tool for transforming and facilitating learning and
teaching. For post-secondary education, the attention and focus on technology integration is
pervasive. The National Center for Education Statistics’ (2019) Education Technology Equity
Initiative focuses on the effective and prudent integration of learning, skills, and teaching with
technology.
With reference to advancement and progress, technology’s ingression and inclusion into
many areas of higher education are undaunted and ubiquitous. The capabilities, applications
8

practices, and functions of educational technologies continue to emerge, evolve, and moderate.
As an example, developments in artificial intelligence, enabled by the economies and efficiencies
of big data and cloud computing and fueled by predictive and evaluative analytics, positively
affect and impact student academic outcomes (Educause, 2017). Additionally, newer next-
generation learning technologies in higher education expand opportunity by promoting and
facilitating equity, access, and inclusion (Educause, 2019).
At a February 2019 Educause conference, a director of adaptive learning at ASU,
asserted that “this is the golden age of educational technology, as the past 10 years have allowed
courseware providers and digital curriculum to operationally mature, match higher education’s
instructional requirements and improve the overall user experience” (D. Johnson, personal
communication, February 7, 2019). Tesene (2018) writes that the proliferation and advancement
of high-quality learning technology in recent years has corresponded with a pronounced
enthusiasm regarding the potential of adaptive learning courseware. Some believe that adaptive
learning technologies will lead the way for a pedagogical renaissance. In a recent Gallup and
Inside Higher Ed survey, two-thirds of university and college presidents believed that adaptive
learning would make a “positive impact on higher education” (Zimmer, 2014, p. 62).
Moving Toward Student-Centered Learning
Falling under the educational umbrella of one-on-one or individualized learning, the
synergistic instructional relationship between teacher and learner, as exercised and applied
through adaptive learning, moves to supplant and displace long-standing teaching practices such
as whole-class instruction, direct instruction, teach to the middle of the class and sage on the
stage. Bloom (1984), in seminal research, found that the average student tutored one-to-one
using mastery learning techniques performed two standard deviations better than students who
9

learned via conventional instructional methods, that is, “the average tutored student was above
98% of the students in the control class” (Bloom, 1984, p. 4). Additionally, the variation of the
students’ achievement changed: “about 90% of the tutored students ... attained the level of
summative achievement reached by only the highest 20%” of the control class (Bloom, 1984,
p. 4).
As a new way of instruction and as a new way to learn, adaptive learning follows a
general and familiar path of use and acceptance. Rogers (1962) explained, how, over time, an
idea or product gains momentum and diffuses through a group of people or through a population.
The diffusion of innovation theory (Rogers, 1962) refers to the process that occurs as people
adopt a new idea, practice, or product. The “Innovators” and “Early Adopters” are the pioneers
and risk-takers and are followed by “Early Majority,” “Late Majority” and “Laggards.” Over
time, the innovative idea, practice, or product becomes diffused through a population until a
saturation point is achieved (Rogers, 1962, Kaminski, 2011). A subsequent enhancement to
Rogers’ theory is a group of non-adopters who do not follow the adoption process.
Although in its relative infancy by technology adoption standards, Johnson asserts that
adaptive learning’s capabilities to deliver “the right content, to the right student at the right time”
will improve with scale. By continuing to process large amounts of data from student
interactions course after course, adaptive learning systems, as with artificial intelligence’s deep
learning, continues to learn and become smarter over time.
An Understanding of Adaptive Learning
As part of a new category of educational technologies, adaptive learning is considered to
be at the forefront of an interactive and multi-modal form of learning and instruction for higher
education. A dynamic working relationship among schools (including faculty, administration,
10

and staff), content providers (such as Pearson, McGraw Hill, CogBooks, and Cengage) and data
service providers (such as cloud services from Google, Microsoft, IBM, and Amazon) \serves to
frame, found and fuel adaptive learning technologies.
For instructional context, adaptive learning, as integration throughout higher education,
incorporates and closely relates to the fields of artificial intelligence, big data, predictive
analytics, learning analytics, and personalized learning. Adaptive learning, together with other
new learning technologies, expect to have a significant impact on the ways in which colleges and
universities approach their core mission of teaching, learning, and inquiry (O’Sullivan, 2018). As
a rising expectation for today’s higher education students, accessibility to platforms, learning
materials, content, and resources provides opportunities for learning anywhere and at any time
(Tesene, 2018).
Several key themes indicate continued growth in adaptive learning (Adams Becker et al.,
2017). One identified theme is that adaptive courseware developers better understand the needs
of the sector overall, which may be a case of the producer better learning the consumer market.
Secondly, technology’s strength through differing instructional deliveries, including hybrid,
blended, and online learning environments, has shown the importance and viability of adaptive
learning platforms (Adams Becker et al., 2017).
The Impact of Adaptive Learning
In higher education, adaptive learning enables the delivery of personalized learning at
scale, one course, or one class at a time. The software’s ability to monitor student progress and
use data to modify instruction at any time supports and differentiates the process of learning. The
effects of non-linear curriculum mapping, embedded assessments, student feedback, and
instructor adjustments/interventions combine to respect prior knowledge and respond to learning.
11

A goal for adaptive learning, according to Horizon (2018), is to move students through a
learning path, empowering active learning, targeting at-risk student populations, and assessing
factors affecting completion and student success. Additionally, advocates for adaptive learning
believe that “it can be a solution for the “iron triangle” of educational challenges: cost, access,
and quality.
The advancement of adaptive learning parallels and integrates with the progressing
technical capabilities and innovations of Next-Gen Learning Management Systems, Next-Gen
mobile learning, and Natural Language Processing (NLP). As these systems move to become a
virtual teaching and learning center, mobile learning has flipped to a student-centered focus
where LX, the learner experience, is the lens from which content providers, instructors, and
instruction designers combine to create a robust and inclusive mobile learning experience. NLP
brings the ability to understand human forms of communication. NLP helps computers
understand, interpret, and manipulate human language. Also, NLP enables researchers to go deep
into text and speech to mine language for correlations, patterns, and other findings—teasing out
meaning that might be detectable only through powerful computation as well as expediting
analyses that would take humans relatively significant effort to complete on their own (ELI,
2018).
Importance of the Evaluation
Meeting the needs of an increasingly diverse and distinct college student population in a
changing and competitive higher education institution landscape requires a continued focus on
opportunities and solutions that might improve student outcomes, positively affecting academic
achievement. Furthermore, as learning technologies continue to present new opportunities in
higher education, implementations of adaptive learning often represent unique, single-campus
12

examples, and cases. However, best practices and shared knowledge across institutions have not
been readily available. From the 3-year project grant, the most promising findings and best
practices will be scaled through APLU’s national network of over 200 public institutions and
shared publicly to support and advance the postsecondary community and higher education
landscape.
Stakeholders and Stakeholders’ Performance Goals
Stakeholders in this emerging and evolving field of adaptive learning fall into four
groups: students, faculty and instructors, Project Leads, and university site administrators. These
four stakeholder groups are represented at each of the eight project grant universities. As part of
the project grant, each university participant was required to submit an annual report
summarizing adaptive courseware implementation progress and program status.
Stakeholder for the Study and Stakeholder Performance Goal
For the AAACPU grant program, each participating university selected an individual or
small team to serve as Project Lead. The Project Lead guides the implementation from start to
finish. All other stakeholders will take direction from and coordinate with the Project Lead . The
Project Lead will be responsible for project management activities such as leading meetings,
planning, documenting, and communicating with other stakeholders in every phase.
The adaptive courseware implementation plan was designed in six phases:
1. Plan
a. Phase 1 - Establish Support
b. Phase 2 - Discover and Decide
2. Build
a. Phase 3 - Design
13

b. Phase 4 - Develop
3. Use
a. Phase 5 - Pilot and Iterate
b. Phase 6 - Scale
The goal of Project Leads is to implement adaptive learning courseware for 15% of the
first-year student courses within three years. To effectively and successfully lead an adaptive
courseware implementation program, the Project Lead must be able to perform the following
actions and functions on a daily basis as set forth in the APLU’s Guidebook (2018) discussed in
Chapter Two.
1. Create and maintain a collaborative and cooperative working relationship with faculty in
the pedagogical design, technology selection, courseware implementation, and evaluation
of the adaptive courses,
2. Provide and support ongoing courseware product training and guidance during the
implementation of the adaptive courses, and
3. Provide and support faculty in a program to assess student outcomes and evaluate the
implementation of the adaptive courses.
Project Leads’ Critical Behaviors
To achieve their goal to best support adaptive courseware implementation, Project Leads
must perform the following critical behaviors on a weekly basis: create and foster a collaborative
and cooperative working relationship with faculty, provide and support ongoing product training
and implementation guidance for faculty, and establish open communication and encourage
recurring feedback from faculty. The aforementioned critical behaviors were extracted from
numerous industry reports and an APLU guidebook published in 2018.
14

Purpose of the Project and Questions
To address specific and identified challenges, some colleges and universities have
adopted adaptive learning, integrating adaptive courseware to support and augment learning and
instruction. The purpose of this study is to conduct an evaluation of adaptive learning
implementations at eight higher education institutions across the country participating in the
APLU program. The analysis will focus on stakeholders, areas of knowledge and skill,
motivation and organizational resources. While a complete study and evaluation would address
all stakeholders, both in higher education institutions and associated entities, for practical
purposes, the stakeholders of focus will be the Project Leads responsible for the eight adaptive
learning implementation programs. To survey, appraise and evaluate current implementations of
adaptive learning in the eight universities, this study will consider specific school
implementation programs at the eight universities selected for the AAACPU grant.
This analysis focuses on the goals of the implementation project and the extent to which
they are being met. The analysis will also focus on the assets and needs of the AAACPU Project
Leads in the areas of knowledge and skill, motivation, and organizational factors necessary to
achieve their goals. Each of the participating schools has identified a Project Lead who will
coordinate, organize, and lead the grant process. The question that will guide this evaluation
study is:
What Knowledge, Motivation, and Organization influences affect the implementation of
adaptive learning courseware?
Methodological Framework
Qualitative methods will be used for data gathering and analysis to evaluate the APLU’s
AAACPU grant program in the areas of knowledge, motivation, and organizational resources as
15

they relate to program implementation. Project Leads from each of the eight participating
universities will be interviewed and annual program progress reports will be analyzed. These
various instruments will serve to triangulate the data.
Definitions
Technology and technology-related instruction in higher education, as used in this
dissertation, include specific and relative terms:
Adaptive courseware collects student data through assessment, analyzes that data, and
uses it to offer personalized learning paths to each student or reports and recommendations to
instructors to help personalize the learning experience. Through adaptive courseware, instructors
are able to collect, access and utilize the data they need to deliver quality learning experiences to
individual students at scale (APLU, 2018).
Adaptive learning is one technique for providing personalized learning, which aims to
provide efficient, effective, and customized learning paths to engage each student. Adaptive
learning systems use a data-driven approach to adjust the path and pace of learning, enabling the
delivery of personalized learning at scale. Adaptive systems can support changes in the role of
faculty, enable innovative teaching practices, and incorporate a variety of content formats to
support students according to their learning needs (Educause, 2017).
Learning Analytics is the use of data, analysis, and predictive modeling to improve
teaching and learning. Learning analytics examines data from various sources, looking for
patterns and correlations that can provide insight to learners, instructors, and those who support
them about how to improve learning. In this way, learning analytics can help students, faculty,
and institutions achieve their respective goals (Educause, 2017).
16

Personalized learning refers to instruction in which the pace of learning and the
instructional approach are optimized for the needs of each learner. Learning objectives,
instructional approaches, and instructional content (and its sequencing) all may vary based on
learner needs. In addition, learning activities are meaningful and relevant to learners, driven by
their interests, and often self-initiated (U.S. Department of Education, 2017).
Organization of the Study
Five chapters are used for this study. This chapter provides the reader with key concepts
and terminology commonly found in a discussion about adaptive learning in a higher education
setting. The organization’s mission, goals, and stakeholders as well as the review of the
evaluation framework was provided. Chapter Two provides a review of the current literature
surrounding the scope of the study. Topics of adaptive learning, adaptive courseware, artificial
intelligence, personalized learning, curriculum and content providers, differentiated instruction,
and learning analytics as all relate to higher education will be addressed. Chapter Three details
the assumed causes for this study as well as methodology when it comes to the choice of
participants, data collection, and analysis. In Chapter Four, the data and results are assessed and
analyzed. Chapter Five provides solutions, based on data and literature, for closing the perceived
gaps as well as recommendations for an implementation and evaluation plan for the solutions.

17

Chapter Two: Review of the Literature
With the potential to transform learning experiences and improve student outcomes at
scale, advances in educational technologies provide new opportunities across postsecondary
education institutions to adapt instruction to meet the needs of all learners. As the capability,
capacity, and potential of technologies in education continue to emerge and evolve, approaches
to learning, instruction, and practice are poised for pedagogical change and shifts in
methodology. In this section, the roles of technology in higher education, teaching with
technology, an overview of personalized learning, an introduction to adaptive learning, and a
specific focus on adaptive courseware will be presented through empirical research, industry
reports, and journal articles. Additionally, an explanation of knowledge, motivation, and
organizational influences will be explained from the aforementioned instructional perspectives.
Lastly, the chapter will conclude with a presentation of the conceptual framework.
Adaptive Learning
Falling under the umbrella of personalized learning, adaptive learning incorporates real-
time monitoring, collection, and analysis of learner interactions and performance levels which
continuously adjust and adapt the dynamics of instruction. Adaptive learning, as a simple
definition, responds to a student’s interactions in real-time by dynamically providing the student
with individual support (Johnson, 2016). A more robust definition of adaptive learning is real-
time monitoring, collection, and analysis of learner interactions and performance levels that
continuously adjust and adapt the dynamics of instruction. An additional definition is “Adaptive
Learning technologies dynamically adjust to the level or type of course content based on an
individual’s abilities or skill attainment, in ways that accelerate a learner’s performance with
both automated and instructor interventions” (Educause, 2018).
18

As a subset of personalized learning in higher education, adaptive learning employs
technologies to provide efficient, effective, and individual student learning interactions and
paths. “Adaptive learning technologies dynamically adjust to the level or type of course content
based on an individual’s abilities of skill attainment in ways that accelerate a learner’s
performance with both automated and instructor interventions and feedback” (Educause, 2018).
Figure 2 shows the position of adaptive learning within higher education’s technology
implementation.
Figure 2
Adaptive Learning Position Within Technology in Higher Education

19

Adaptive learning is an instructional methodology that aims to provide an efficient,
effective, and individualized learning path for each student. Adaptive learning systems use
student-generated interactions, performance, and data to dynamically adjust, adapt, and modify
instruction (Johnson, 2016).
Literature on adaptive learning often begins with a broader discussion of instructional
technology or technology integration. As such, Barajas-Murphy’s (2018) foreword to a volume
of an academic journal devoted entirely to adaptive learning starts broadly with a discussion on
learning and technology. The journal, Current Issues in Emerging eLearning, shares articles that
cover specific adaptive learning tools and technologies and their efficacy in addressing the aims
and learning outcomes in the instances they were used and studied and common challenges in
adaptive learning: faculty adoption. The Barajas-Murphy narrative discusses current
implementation examples of adaptive learning courseware including the Realizeit platform at the
University of Central Florida and the state of courseware adoption at the University of
Mississippi, Colorado Technical University, and Georgia State University.
As with new instructional technology introductions, challenges are confronted with the
innovative initiative of adaptive learning. In an industry report, Alexander et al. (2017) provide a
general reference by citing six challenges faced in implementing learning technologies. With an
identified goal of improving digital literacy and academic equity, Alexander et al. (2017) define
an achievement gap as “The achievement gap also referred to as the college completion gap,
reflects a disparity in the enrollment and academic performance between student groups, defined
by socioeconomic status, race, ethnicity, or gender. While emerging technological developments
such as digital courseware and open educational resources have made it easier to engage with
learning resources, significant issues of access and equity persist among students from low-
20

income, minority, single-parent families, and other disadvantaged groups. The one-size-fits-all
approach of traditional higher education paradigms, coupled with overwhelming tuition costs, is
in stark contrast with an increasingly diverse global student population; more flexible degree
plans are needed.”
With a focus on integrating new learning technologies, Alexander et al. (2017) discuss
rethinking the roles of educators: “Educators are increasingly expected to employ a variety of
technology-based tools, such as digital learning resources and courseware, and engage in online
discussions and collaborative authoring. Further, they are tasked with leveraging active learning
methodologies like project- and problem-based learning. This shift to student-centered learning
requires them to act as guides and facilitators.”
With a focus on adaptive learning in higher education, Miranda (2017) shares how
adaptive learning has been implemented across various US university and college programs and
their efficacy in competency-based learning and skills uptake. Miranda (2017) shows how
adaptive learning philosophies can transform a course through flexible policies, course structure,
class size, and faculty roles. Miranda (2017) shows how adaptive learning can affect attrition
rates in college courses.
Adding to research written in 2017, Alexander et al. (2019) annual industry report
discusses the state of educational technology in higher education and focuses on survey-
identified areas of importance and challenge. Under an umbrella of personalized learning,
Alexander et al. (2019) discuss the connection between student-generated data and an adjustment
in instruction by sharing “encompassed by the personalized learning movement and closely
linked to learning analytics, adaptive learning refers to technologies that monitor student
progress and use data to modify instruction at any time. From a student perspective, these
21

technologies adapt to what they need by providing real-time feedback and learning paths to help
them advance—no matter the level at which they begin.”
With broad context and an industry-wide perspective, Alexander et al. (2019) assert that
“in discussing adaptive learning, some institutions are waiting and watching while early adopters
pilot, implement, and share what they have learned.”
As adaptive learning modifies and adjusts the presentation of content based on student
activity and performance, adaptive courseware is technology that collects and assesses student
data, evaluates the data, and uses it to curate personalized learning paths for each student.
Adaptive Courseware
Under the personalized learning conceptual umbrella, adaptive learning – focusing on
adaptive courseware as a digital instructional tool, provides a dynamic, personalized learning
experience for each student.
Adaptive courseware, as an application, practice, and implementation of adaptive
learning, represents adaptive learning in a higher education class or course. Content and
curriculum providers offer three basic types of courseware (Educause, 2017):
1. Closed systems which offer off-the-shelf courseware for ready course/class
implementation
2. Hybrid systems which offer limited modification by the provider and/or faculty/school
3. Open systems that allow users (instructor/school) control of all configuration and content
decisions.
An ongoing and continuing challenge in higher education is often focused on first-year
courses, general education courses, or high-enrollment courses. Additionally, courses with poor
22

academic outcomes such as high D/F/W classes, high transfer rates, or high attrition rates may
also be areas in need of support.
Adaptive courseware allows for instruction to be personalized and individualized for a
student. Critical to the instruction and learning process, the adaptive courseware must determine
what individual students already know and what they need to learn, then modifying instruction to
meet those diverse and differentiated needs efficiently and effectively (Vignare et al., 2019).
To be practical and effective, adaptive courseware should include dynamic and engaging
embedded assessments to scaffold and elicit and demonstrate learner content mastery in course-
and program-specific contexts. Equally essential and critical is the role that instructors and
pedagogical experts play in the design and curation of instructional and assessment strategies. As
a relatively new instructional resource, adaptive courseware has the opportunity to positively
improve academic outcomes for thousands of postsecondary students by providing a
personalized learning experience that is equally immersive and interactive as well as
communicative with faculty and instructional staff (Vignare et al., 2019).
It is important to note that technology integration in itself, whether adaptive learning or
adaptive courseware, supports learning, but the medium is not the key to learning. As stated by
Clark, Yates, Early, and Moulton (2010), media is the vehicle that delivers instruction but is not
the reason for learning.
Proximal, short-term project grant goals include adaptive courseware implementation
across the eight schools with fidelity and consistency while distal, extended-term goals would be
measurable, course, or program analytics demonstrating higher student achievement and student
outcomes.
23

At the center of adaptive learning technology implementation is adaptive courseware
which may be a closed system, containing ready to use off-the-shelf course content, an open
system, allowing users (school and staff) to control content, instructional delivery, and course
configuration, or a hybrid system which allows for limited configuration. Adaptive courseware is
defined as “a digital instruction tool that provides a personalized learning experience for each
student. It includes instructional content and assessment that is scoped and sequenced to support
an entire course (APLU, 2018). These adaptive systems, using embedded deep learning artificial
intelligence, learn from student interactions and learner performance and then adjust content,
assessment, and sequence of instruction, providing a differentiated learning experience for each
student (Gebhardt, 2018).
As might be expected in a nascent and emergent field, and represented by the innovators
and early adopters from Rodgers’ (1962) diffusion of innovations theory, applications and
practices (implementation) of adaptive learning are disparate, distinctive and often siloed.
Common challenges and areas in need of attention for two-and four-year institutions are
low course completions, high D/F/W grades, or high transfer rates, all issues that might be
assisted by adaptive learning technologies (Barajas-Murphy, 2018). Additionally, adaptive
courseware, in place of physical books, can reduce the financial costs of college attendance and
reduce potential instructional variability between course sections, instructors, campuses, and
semesters (Johnson & Zone, 2018).
Adaptive Courseware Implementation
Adaptive courseware, as a technological product, necessitates a plan and process for
effective implementation. In higher education, courseware implementation plans vary according
to a myriad of factors including leadership influence, financial resources, academic or enrollment
24

objectives, faculty attitude, staffing, product knowledge, and prevailing school learning culture
and climate.
Sharing courseware implementation practice in higher education, research by Dziuban et
al. (2018) shows that through adaptive learning, the four dimensions – knowledge acquisition,
engagement activities, communications, and growth remain constant across courses, disciplines,
and universities. With more specificity, Dziuban et al. (2018) state that “an increasingly diverse
student population is enrolling in postsecondary schools. To address their unique needs, adaptive
learning strategies and interventions can help bridge the gaps between different skills and
competencies in a way that is personalized to students.”
With a similar sentiment, O’Sullivan (2018) discusses, in a case study, various ways by
which adaptive courseware can be used to address diverse educational outcomes for diverse
university student populations. Continuing, Alexander et al. (2019) state that adaptive
technologies and a focus on measuring learning are driving institutional decision-making while
personalizing student learning experiences; leaders must now consider how to evaluate the
acquisition of vocational skills, competencies, creativity, and critical thinking.
Moving to topic-specific courseware, adaptive learning courseware has the potential to
positively affect student outcomes by making personalized learning scalable. Gebhardt (2018)
reports that personalized learning has improved student performance through statistical
techniques used in microeconomics. Further, Gebhardt (2018) states that personalized learning is
important for various non-traditional student populations.
The financial considerations of learning technologies, including adaptive learning
courseware, are a factor in implementation and scale. In an effort to find ways to decrease costs
while delivering high-quality education, adaptive learning has the potential to address the “iron
25

triangle” of quality, cost, and access. Gebhardt (2018) concludes that across higher education,
there has been a call to find “ways to decrease costs while delivering high-quality education to an
expanded and more diverse student body.” This recognition of the importance of teaching while
acknowledging that teachers are expensive, demands that higher education embrace new models
for learning.
Illuminating benefits of adaptive learning courseware show that the courseware requires
students to master the same learning objectives, but the order and timing of content are
determined by the adaptive software engine that assesses the student’s performance on a number
of factors and then guides the student through the course content. Adaptive learning diverts from
a one-size-fits-all model to an individual perspective of meeting the needs of each learner.
Using the adaptive learning program Realizit, the University of Central Florida piloted
faculty-created courses to give students a personalized pathway through instructional content and
coursework. UCF began a pilot investigation of the use of adaptive learning in 2014. After
exploring vendors and with faculty input, Realizeit was selected as the university’s enterprise
adaptive learning platform. Case studies at UCF show that adaptive courseware in nursing allows
students to analyze clinical problems based on real-life scenarios. Because of perceived
increased engagement, case-based learning has been used extensively in nursing education. A
growing body of evidence provides support for the educational benefits of case-based pedagogy
including improved learner outcomes such as critical thinking, understanding of difficult
concepts, and clinical skills.
Hinkle and Moskal (2018), continuing a discussion of courseware, share that Realizit is
an adaptive learning platform that uses Bayesian estimation techniques within a faculty-created
course to give each student a personalized pathway through the instructional content. Realizeit is
26

a content-agnostic adaptive learning platform which allows faculty to create the learning content
and assessment or ingest content from sources such as open educational resources. As the student
progresses through course content, a comprehensive stream of data is generated that guides the
algorithmic adaptivity and personalization. Realizeit’s curriculum map involves a series of
nodes, depicting granular course content, that is connected by edges depicting the pathways of
prerequisites that students must traverse to achieve mastery.
Adaptive Courseware Implementation - Faculty Perspective
Focusing on faculty engagement, a study at Colorado Technical University (CTU)
chronicled a scaled implementation of adaptive learning while outlining the successes and
challenges encountered through the rollout of this personalized approach to student learning.
Johnson and Zone (2018) state that during the last decade, there has been an increased interest in
the engagement of faculty with online learning and the use of digital tools, including, but not
exclusive to, adaptive learning. Digitals tools and courseware are being utilized in online courses
as well as blended and flipped classrooms, and faculty are being asked to employ new
pedagogies incorporating digital tools into their classroom experience.
Through a lens of beginning-of-term, diagnostic assessment, Johnson and Zone (2018)
assert that faculty are faced with the challenge of determining the level of knowledge of each
student in the class and then, how best to support each student. When a cohort of students with a
wide range of base knowledge about a topic begins a course, faculty seek technologies, such as
adaptive courseware, to personalize the experience while increasing student success. Adding to a
faculty perspective, Johnson and Zone (2018) share that CTU has two core beliefs from this
work: 1) Faculty are better equipped to define the training protocols needed to effectively use
27

technology in the classroom, and, 2) Technology adoption by students in a course is greatly
influenced by the faculty experience of the technology.
The realities of introducing a new initiative in higher education accentuate a sometimes
less-than-welcoming environment. Johnson and Zone (2018) share that research indicates that
faculty are reluctant to engage in online teaching due to concerns with change, technology,
student outcomes, and workload. Specific barriers to adoption of platforms that promised to
personalize the student experience include additional time required for faculty, efficacy of digital
courseware in improving student outcomes, and reduced control over course content and student
experience.
Adaptive Courseware Implementation - Student Perspective
The use of adaptive learning courseware to support students and their learning brings a
multi-dimensional approach to andragogy and instruction. Adaptive learning can facilitate
improvements in student retention, student satisfaction, and the achievement of student
outcomes.
With a focus on student enrollment, O’Sullivan (2018) states that adaptive courseware is
a key personalized learning strategy designed to benefit traditionally underserved, minority, and
first-generation students in higher education. At Georgia State University, a 3-year adaptive
courseware implementation project aims to improve undergraduate education and promote
student success through the implementation of adaptive learning courseware in high-enrollment,
high-risk courses.
Chen et al. (2017) state that, at the University of Central Florida, adaptive learning
technologies are emerging as an effective way to promote access and quality at a large scale in
education. Enabled by machine learning, these technologies can provide each student with a
28

personalized learning experience that helps accelerate a learner’s performance while adapting to
his or her needs and skillsets in real-time. Focusing on the actual student experience, Chen et al.
(2017) share that in this type of adaptive courseware system, content is presented to the student
in small chunks and is followed by several assessment questions. If students excel in a particular
content area, the system will move them toward advanced content and questions. If students are
not doing well in a particular area, the system will recommend that they review content areas
where their knowledge is weaker and will decrease the difficulty of the questions. The amount of
content and the types of assessment questions depend on how much information and how many
questions are created and entered into the system by the instructor or course builder.
Adaptive Courseware Implementation - Future Expectations
As the number of adaptive courseware implementations increases and as implementation
programs scale, a broad focus identifies key strengths and benefits to adaptive technology
adoption. Vignare et al. (2019) articulate that while many public colleges and universities have
embraced the mission of increasing access to higher education by recruiting more diverse student
populations, these institutions continue to see significant persistence and completion gaps for
their low-income students, students of color, and first-generation students. In response to this
problem, many 2- and 4-year institutions have set goals around equity, success, and completion.
Continuing, Vignare et al. (2019) share that among the technologies gaining traction
today are adaptive learning solutions, which help deliver a personalized learning experience to
each student. One of the most promising forms of adaptive technology is adaptive courseware,
and Institutes of Higher Education are implementing adaptive courseware to support a range of
institutional goals, including increased course completion, increased student engagement, greater
course flexibility, and access, greater opportunity for student self-remediation, reduced
29

variability between sections/instructors, reduced equity gaps and lower costs for instructional
materials.
In conclusion, the introduction of new technology at any level of education is often not a
simple process. In higher education, there is no single way to implement adaptive courseware
and no singular timeline for success. Expectedly, implementation goals vary from school to
school. Examples of effective implementation programs continue to be reported at the
community college and university level. With a growing student-centered focus, adaptive
courseware has the transformational potential to create a more personalized learning
environment in college courses, allowing students to progress through an individually curated
instructional path of content based on their demonstrated skills and knowledge.
Conceptual Framework
The gap analysis framework and approach, established by Clark and Estes (2008),
highlights, identifies, and resolves performance gaps in the areas of knowledge, motivation, and
organizational causes. Within higher education, the knowledge gap focuses on instructors having
the understanding and skills to employ and integrate adaptive courseware throughout their
courses. The motivation gap examines value factors related to the instructors’ self-efficacy,
mindset, and attitude to facilitate and guide the implementation of adaptive courseware.
Organizational causes address resources, policies, and procedures as well as cultural models that
can impede performance goals. The data collected from the gap analysis process allows informed
decision-making required to close the performance gap.
In the study, the Clark and Estes (2008) framework will be applied and aligned to
highlight promising practices of adaptive learning from colleges and universities across the
country.
30

Stakeholder Knowledge, Motivation and Organizational Influences
From a review of the literature, influences and determinants for effective courseware
program implementations were identified. In the literature, case studies were shared from various
community college and university adaptive courseware implementations across the country.
Knowledge and Skills
While circumstances and objectives for courseware implementation programs differ from
institution to institution, the following key procedural knowledge and skill influences repeated
throughout the various programs and were therefore recognized to be critical and essential.
Knowledge Influence: Project Leads need to know how to select appropriate adaptive
courseware.
Knowledge Influence: Project Leads need to know how to implement adaptive
courseware at their school.
Knowledge Influence: Project Leads need to know how to collaborate with others in
order to implement adaptive courseware.
Knowledge Influence: Project Leads need to know how to utilize adaptive courseware to
achieve student outcomes.
Anderson and Krathwohl (2001) describe procedural knowledge as the steps in which
something is accomplished or completed. Methods, procedures, techniques and criteria for
applying, assessing and evaluating skills are part of this knowledge type.
More than two dozen companies offer adaptive courseware products ranging from open
systems where faculty provides the course content, to hybrid systems where content is provided
by both vendor and faculty and closed systems which are topic-specific and ready to be delivered
to students. Schools often employ all three types of courseware. Courseware selection criteria
31

include Project Lead recommendations, product cost, budget constraints, faculty preferences,
vendor-school relationships and matching academic disciplines or courses with product
availability.
At Georgia State University, the Center for Excellence in Teaching and Learning (CETL)
is charged with research and identification of appropriate courseware for campus use. According
to Tesene (2018), GSU’s CETL offers an overview of the systematic exploration and selection of
adaptive learning courseware. The CETL hopes that the extensive evaluative process can offer
insights to individuals and institutions that are interested in navigating and experimenting with
adaptive learning courseware. In outlining the steps taken to evaluate and select adaptive
courseware, the CETL expects to provide a model that is both replicable and flexible.
Courseware use is expected to grow as the synergy between technology and learning
continues to evolve and expand. Technology use among higher education students continues to
grow, for on-campus students as well as in online programs. Technology use among higher
education students continues to grow, for on-campus students as well as in online programs.
According to a report by Intentional Futures (2017), “nationally, the number of students engaged
in digital learning is growing rapidly. One driver of this growth is rising demand for distance
learning, which often relies on digital learning environments. Distance learning programs saw
enrollment increases of approximately 4% between 2015 and 2016, with nearly 30% of higher
education students taking at least one digital distance learning course. Much of this growth in
distance learning is occurring at the undergraduate level. The number of students who take
distance learning courses exclusively is growing as well.”
A recent poll of college and university presidents shows that a majority of the survey’s
respondents (66%) see potential in adaptive learning to make a “positive impact on higher
32

education” (Lederman, 2013). As examples, the Bill & Melinda Gates Foundation initiated in
2013 the Adaptive Learning Market Acceleration Program (ALMAP) to advance evidence-based
understanding of how adaptive learning technologies could improve opportunities for low-
income adults to learn and to complete postsecondary credentials (SRI Education, 2016). The
APLU founded the Personalized Learning Consortium, also in 2013, to facilitate public
universities to exchange information about personalized learning technologies, such as adaptive
learning courseware, that will improve student success (APLU, 2016).
Herckis (2018) shares that a two-year study of the barriers and affordances to the
successful implementation of evidence-based instructional tools and strategies at scale makes
clear that instructional autonomy, a reliance on peer networks, risk-averse instructional
development, and unidentified pedagogical misalignments intersect such that educator buy-in
often comes at the cost of digital literacy.
The increasing availability and utility of learning technologies continues to support and
assist students in higher education. Not only must educators recognize appropriate technological
tools for instructional use, they must compare and differentiate tools to provide the most
effective learning environment for students.
In higher education, a shift from classroom-based instruction continues towards online
and digital representation of learning. Kumar et al., (2017) share that the “higher education
landscape is undergoing significant change as a result of technological innovations. Students
have an option to learn through self-paced or real-time online courses offered by the MOOCs
and Coursera platform. Education already has moved from the mode of chalkboards to chat
boards. Most learning management systems have already switched over to cloud-based systems.
Worldwide, most faculty members have adopted eLearning platforms and have incorporated chat
33

boards, social media, wikis, blogs, and other digital tools for their class delivery, student
engagement, and class participation. MOOCs, Coursera degrees, and similar programs will
become more common as part of degree completion. Virtual technologies and social media sites,
such as Facebook, Twitter, and others, could become the primary forum for “instantaneous idea
sharing, tutoring, learning and training.” E-learning will incorporate more interactive video or
game-based platforms for learners to participate in and experience than listening to traditional
lectures.”
Newer learning technologies continue to focus instruction on the individual student,
allowing for a differentiated learner experience. Implementation evaluation and reflection should
be formatively and summatively assessed and appraised. Table 1 shows the stakeholder’s
knowledge influences and the related literature.

34

Table 1

Summary of Assumed Knowledge Influences on Stakeholder’s Ability to Achieve the
Performance Goal

Critical Behaviors
1. Create and maintain a collaborative and cooperative working relationship on a daily
basis with faculty in the pedagogical design, technology selection, courseware
implementation, and evaluation of the adaptive courses.

2. Provide and support ongoing courseware product training and guidance during the
implementation of the adaptive courses on a daily basis.

3. Provide and support faculty in a program to assess student outcomes and evaluate the
implementation of the adaptive courses.
Assumed Knowledge Influences Research Literature
Author, Year; Author, Year.
Knowledge Influence: Project Leads need to know how to select appropriate adaptive
courseware.
Knowledge Influence: Project Leads need to know how to implement adaptive courseware at
their school.
Knowledge Influence: Project Leads need to know how to collaborate with others in order to
implement adaptive courseware.
Knowledge Influence: Project Leads need to know how to utilize adaptive courseware to
achieve student outcomes.
Procedural
Project Leads need to know how to select
appropriate adaptive courseware.

(Sharma et al., 2017); (Tesene, 2018);
(Herckis, 2018); (SRI Education, 2016)
Project Leads need to know how to
collaborate with others in order to implement
adaptive courseware.

(Sharma et al., 2017); (Tesene, 2018);
(Herckis, 2018); (SRI Education, 2016)

35

Procedural
Project Leads need to know how to utilize
adaptive courseware to achieve student
outcomes.
(Sharma et al., 2017); (Tesene, 2018);
(Herckis, 2018); (SRI Education, 2016)

Motivation Influence: PLs Need to be Motivated to Implement Adaptive Courseware
From review of the literature, motivation influences and determinants for effective
courseware program implementations were identified. The literature revealed case studies that
shared numerous community college and university adaptive courseware implementations across
the country. While circumstances and objectives for courseware implementation programs differ
from institution to institution, the following key motivation influence emerged from the various
programs and was therefore recognized to be critical and essential.
General Theory
Motivation is an internal process that directs and maintains behavior (Clark & Estes,
2008). According to Clark and Estes, motivation, as a psychological system, can be observed
with three indices: choice, persistence and mental effort. Choice refers to deciding one action
over another – to take action or not. Persistence is the commitment to continue a task or action,
even over an extended period of time. Mental effort, as shared by Clark and Estes (2008), is the
work required to attain new knowledge and achieve the learning goals. For adaptive learning and
adaptive courseware, all three motivational factors influence the performance and effectiveness
of the overall implementation program.

36

Stakeholder/Topic-Specific Factors
Clark and Estes (2008) identified variables that affect performance that include value,
self-efficacy, mood and expectancy. According to Pintrich (2003), value refers to how important
a person identifies a task to be. Rueda (2011) adds that value answers the question, “Why should
I do this task?” Clark and Estes state that value is one of the three powerful ways people express
their perceptions about what they believe will make them effective. For the implementation of
adaptive learning, Project Leads understand the value and benefit of the courseware for faculty
and students.
According to Pintrich (2003), self-efficacy is the most important factor in a person’s
commitment and motivation to work tasks and in the quality and quantity of mental effort to
invest. Clark and Estes (2008) expand on this idea and explain that people with lower self-
efficacy will not choose to engage in, persist at, and work hard at a task or activity. For adaptive
learning, Project Leads that are confident in their ability to lead may benefit from a collaborative
and cooperative working relationship with faculty.
Nadler (2013) connects the influence of mood with motivation. Nadler shares that
positive mood is positively related to learning. Clark and Estes (2008) add that positive mood
support a person’s work commitment while negative mood impedes it. As a result, Project Leads
who create a positive work environment may benefit from a collaborative and cooperative
working relationship with faculty.
Expectancy, as stated by Maddow, Sherer, and Rodgers (1982), is a belief about the
likelihood of the behavior leading to a specific outcome. Similarly, Bandura (1977) shares that
an outcome expectancy is defined as a person’s estimate that a given behavior will lead to certain
outcomes. Project Leads expect that developing and maintaining a collaborative and cooperative
37

working relationship with faculty will support and benefit adaptive learning program
implementation.
To attain the goal of effective and productive technology integration, educators must
enact a willingness to take risks and expand their practice. Mayer (2011) states that learners work
more diligently to learn and master the material if they attribute their effort to success. This
would suggest that, when educators attribute and connect their success to hard work and
technological savvy and attainment, they are more likely to develop a positive attitude and
perception towards technology use and technology integration. Table 2 shows the stakeholder’s
motivation influences and the related literature.
Table 2

Summary of Assumed Motivation Influences on Stakeholder’s Ability to Achieve the Performance
Goal

Critical Behaviors

1. Create and maintain a collaborative and cooperative working relationship on a daily
basis with faculty in the pedagogical design, technology selection, courseware
implementation, and evaluation of the adaptive courses.

2. Provide and support ongoing courseware product training and guidance during the
implementation of the adaptive courses on a daily basis.

3. Provide and support faculty in a program to assess student outcomes and evaluate the
implementation of the adaptive courses.

Assumed Motivation Influences Research Literature
Author, Year; Author, Year.
Motivation Influence: Project Leads need to be motivated to implement adaptive courseware.
Motivation is an internal process that directs
and maintains behavior. Motivation, as a
psychological system, can be observed with
three indices: choice, persistence and mental
effort.
(Clark and Estes, 2008)
38

Motivation value refers to how important a
person identifies a task to be.
(Pintrich, 2003)

Assumed Motivation Influences Research Literature
Author, Year; Author, Year.
Self-efficacy is the most important factor in a
person’s commitment and motivation to work
tasks and in the quality and quantity of
mental effort to invest.
(Pintrich, 2003)
A positive mood is positively related to
learning. Positive mood supports a person’s
work commitment while negative mood
impedes it.
(Nadler, 2013); (Clark and Estes, 2008)
Motivation Influence: Project Leads need to be motivated to implement adaptive courseware.
Project Leads expect that developing and
maintaining a collaborative and cooperative
working relationship with faculty will
support and benefit adaptive learning
program implementation.

(Maddow, Sherer, and Rodgers, 1982);
(Bandura, 1977)
When educators attribute and connect their
success to hard work and technological savvy
and attainment, they are more likely to
develop a positive attitude and perception
towards technology use and technology
integration.
(Mayer, 2011)

Organization
From review of the literature, organization influences and determinants for effective
courseware program implementations were identified. The literature revealed case studies that
shared numerous community college and university adaptive courseware implementations across
the country. While circumstances and objectives for courseware implementation programs differ
from institution to institution, the following key organization influences repeated throughout the
various programs and were therefore recognized to be critical and essential.
39

Organization Influence: Organizations need to convey the value for the implementation
of adaptive courseware.
Organization Influence: Organizations need to provide resources for the implementation
of adaptive courseware.
Organization, within a knowledge, motivation, and organization (KMO) framework,
encompasses resources, policies, procedures and processes as well as cultural factors. Clark and
Estes (2008) state that individuals within an organization may possess the knowledge, skills, and
motivation required to accomplish an established goal; however, inadequate resources,
bureaucracies, and structures may prevent a goal from being achieved. Organizational barriers
can also create problems with knowledge, skills, and motivation within an organization (Rueda,
2011). The culture of the organization can also determine how well individuals and teams
perform to accomplish a goal (Clark & Estes, 2008). Gallimore and Goldenberg (2001) express
that cultural implementation relative to learning, can be authentic if viewed through a lens of
cultural models and cultural settings. Cultural models, according to Gallimore and Goldenberg,
are shared mental schemas or normative understandings of how the world works, or ought to
work. The concept includes behavioral activity or as well as cognitive and affective components.
Cultural settings occur when two or more people come together, over time, to accomplish
something. Gallimore and Goldenberg (1991) describe how organizational development
improves through intellectual stimulation and opportunities to develop new skills and
knowledge. Anticipated and expected with the introduction of innovative instructional
methodologies and new learning resources, the successful implementation of adaptive learning
must include the authentic, procedural and coordinated participation of stakeholders. Table 3
shows the stakeholder’s organizational influences and the related literature.
40

Table 3

Summary of Assumed Organization Influences on Stakeholder’s Ability to Achieve the
Performance Goal

Critical Behaviors
1. Create and maintain a collaborative and cooperative working relationship on a daily
basis with faculty in the pedagogical design, technology selection, courseware
implementation, and evaluation of the adaptive courses.

2. Provide and support ongoing courseware product training and guidance during the
implementation of the adaptive courses on a daily basis.
Critical Behaviors
3. Provide and support faculty in a program to assess student outcomes and evaluate the
implementation of the adaptive courses.
Assumed Organization Influences Research Literature
Author, Year; Author, Year.
Organization Influence: Organizations need to convey the value for the implementation of
adaptive courseware.
Organization Influence: Organizations need to provide resources for the implementation of
adaptive courseware.
Individuals within an organization may
possess the knowledge, skills, and motivation
required to accomplish an established goal;
however, inadequate resources,
bureaucracies, and structures may prevent a
goal from being achieved.
(Clark and Estes, 2008)
Organizational barriers can also create
problems with knowledge, skills, and
motivation within an organization
(Rueda, 2011)

The culture of the organization can also
determine how well individuals and teams
perform to accomplish a goal
(Clark and Estes, 2008)
Organizational development improves
through intellectual stimulation and
opportunities to develop new skills and
knowledge.
(Gallimore and Goldenberg, 2001)

41

Assumed Organization Influences Research Literature
Author, Year; Author, Year.
Implementation relative to learning, can be
authentic if viewed through a lens of cultural
models and cultural settings.

(Gallimore and Goldenberg, 2001)

Successful implementation of adaptive courseware requires strong program leadership as
well as attentive supervision, collaboration and guidance throughout the school year.
Concurrently, organization factors such as product and teaching efficacy are vital in sustaining
the overarching goal of synergizing technology integration with instruction. The influences
identified in this chapter will be used as a foundation for data collection in Chapter Three.
42

Chapter Three: Methodology
To address specific and identified challenges, some colleges and universities have
adopted adaptive learning, integrating adaptive courseware to support and augment learning and
instruction. The purpose of this study was to conduct an evaluation of adaptive learning
implementations at eight higher education institutions. This analysis focused on stakeholders,
areas of knowledge and skill, motivation and organizational resources for adaptive learning
programs. While a complete study and evaluation would address all stakeholders, both in higher
education institutions and associated entities, for practical purposes, the stakeholders of focus
were the Project Leads referred to herein as Project Leads and were responsible for the adaptive
learning implementation programs. The question that guided this study was:
1. What Knowledge, Motivation and Organization influences affect the implementation of
adaptive learning courseware?
Conceptual and Methodological Framework
The gap analysis approach used in this evaluation, established by Clark and Estes (2008),
identifies, examines and resolves performance gaps in the areas of knowledge, motivation and
organization causes that prevent organizations from achieving their goals. The gap analysis
process is represented below in Figure 3. First, measurable and attainable performance goals are
determined. Next, current achievement is assessed at different organizational levels. The gap
between goals and present performance is then established and base influences are identified and
classified as either knowledge, motivational or organizational.

43

Figure 3
Sequence of Steps in the Gap Analysis Process

For this study, the subsequent steps of Clark and Estes’ (2008) framework were adapted
to evaluate implementations of adaptive learning in eight higher education institutions. As
adaptive learning is an emergent and evolving field, report findings by early adopting
universities varied greatly in scope, scale and objective. Adaptive learning program outcomes
were discussed as well as significant challenges and impacts, relative to the influences of
knowledge, motivation and organizational factors.
Assessment of Performance Influences
To assess and appraise the adoption and scope of adaptive learning in higher education,
the Clark and Estes (2008) gap analysis process was used. A focus of this evaluation was to
provide data to identify lessons learned and best practices as shared by the eight institutions’
Project Leads. Knowledge, motivation and organization influences were established based on the
44

following Project Leads’ critical behaviors and extended through data gleaned from field
research and institutional reports. The tables that follow detail items for interviews and document
analysis that align to critical behavior influences outlined in Chapter Two.
Knowledge Assessment
According to Clark and Estes (2008), knowledge influences attempt to answer how,
when, why, where and who questions related to an individual’s ability to achieve a goal.
Knowledge focuses on communication, procedure and experience issues. Support is inclusive
and personalized to the stakeholder. Product proficiency, implementation dynamics and
guidance, assessment practices and data analytics support knowledge acquisition and actionable
feedback. Ongoing training, accountability objectives and supervision implant conceptual and
strategic knowledge stakeholders may apply to unexpected situations or novel circumstances
(Clark and Estes, 2008).
Anderson and Krathwohl (2001) describe procedural knowledge as the steps in which
something is accomplished or completed. Methods, procedures, techniques and criteria for
applying, assessing and evaluating skills are part of this knowledge type. In more depth,
procedural knowledge expresses area-specific skills, techniques and methods of “how to do
something” and when to use appropriate procedures (Anderson & Krathwohl, 2001). Table 4
details Chapter Two’s knowledge influences and methods of assessment.

45

Table 4

Summary of Assumed Knowledge Influences on Project Leads’ Ability to Achieve the
Performance Goals

Assumed Knowledge
Influences
Interview Questions Document Analysis
Knowledge Influence:
Project Leads need to
know how to select
appropriate adaptive
courseware.
1. Of the three general types of
adaptive courseware, what have
you observed as having the best
outcomes?
2. Are there subjects that
particularly match well with
adaptive courseware? Are there
subjects that do not work well
with adaptive courseware?

3. What are reasons why
universities want to introduce or
use adaptive learning
courseware?
Review artifacts or behaviors
for evidence of knowledge of
products, practices,
applications, assessments,
facts, information, operations
and terminology.
Knowledge Influence:
Project Leads need to
know how to implement
adaptive courseware at
their school.
4. Describe the current state of
adaptive courseware
implementation at your school?

5. From your observations, how
is the adaptive courseware
being utilized?

6. What are they keys to a
successful adaptive courseware
implementation program?

7. How is continuous
improvement for courseware
implementation applied or
exercised?
Review artifacts or behaviors
for evidence of knowledge of
products, practices,
applications, assessments,
facts, information, operations
and terminology.

46

Assumed Knowledge
Influences
Interview Questions Document Analysis
Knowledge Influence:
Project Leads need to
know how to collaborate
with others in order to
implement adaptive
courseware.
8. Please talk about the
significance of a coordinated
team for courseware
implementation.

9. What has been the faculty
experience to date with the
implementation of adaptive
learning courseware?
Review artifacts or behaviors
for evidence of knowledge of
products, practices,
applications, assessments,
facts, information, operations
and terminology.
Knowledge Influence:
Project Leads need to
know how to utilize
adaptive courseware to
achieve student outcomes.
10. What indicators/outcomes
does your institution use to
measure student success?
Review artifacts or behaviors
for evidence of knowledge of
products, practices,
applications, assessments,
facts, information, operations
and terminology.

Motivation Assessment
Table 5 presents value, self-efficacy, mood and expectancy as the motivational factors
which impact adaptive learning implementation programs in higher education. as revealed by the
literature Chapter Two described motivational influences and will be confirmed through
interviews and document analysis.
Table 5
Summary of Motivation Influences and Method of Assessment
Assumed Motivation
Influences
Interview Questions Document Analysis
Motivation Influence:
Project Leads need to be
motivated to implement
adaptive courseware.
11. How can Project Leads
be motivated to implement
adaptive courseware.
Review documents to assess
artifacts of levels of
engagement (choice,
persistence, effort) in the
task.

47

According to Clark and Estes (2008), motivation is described as a psychological process
comprising three indices: choice, persistence, and mental effort. Choice is defined as actively
pursuing a goal while persistence is described as ongoing focus on a task. When individuals
become distracted and lose focus, they are identified as having a persistence problem. Similarly,
mental effort is the ability to expend effort to seek new knowledge and persist to achieve a goal.
Confidence is a key contributing factor like lack of confidence as well as overconfidence, which
result in persistence and choice challenges (Clark & Estes, 2008). According to Wigfield and
Eccles (2000), the value an individual places on a task influences performance, effort, and
persistence. Similarly, self-efficacy is defined as the assessment of one’s performance ability to
achieve a goal. A strong sense of self-efficacy will result in higher competence and a potentially
positive outcome. Individuals who value the task will work more diligently to learn and master
the material, especially if they attribute the effort to their own success (Bandura, 2006; Mayer,
2011).
An assumed motivational influence, presented as an open-ended question, will be
analyzed to identify the psychological constructs that may be present with Project Leads. Table 5
details Chapter Two’s motivation influences and their method of assessment.
Organization/Culture/Context Assessment
Organizational influences were categorized as cultural models, and resources. Clark and
Estes (2008) state barriers or misalignment in the aforementioned factors results in a delay of
work and lack of goal performance. Organizational influences outlined in Chapter Two were
assessed through interviews and document analysis.
Organizational culture is complex and, according to Scott, Mannion, Davies, and
Marshall (2003), change encompasses a thorough understanding of both conscious and
48

unconscious behavioral factors. Gallimore and Goldenberg (2001) explain cultural models as
integral components of an organization’s structure. Cultural models are defined as a shared
mental schemas that are often invisible to stakeholders. They define the way things should be
and are often automated and go unnoticed, creating a shared understanding. Cultural models
develop from shared experiences and collective information. Additionally, cultural models are
restricted to the environment in which they are located. Cultural models generally go unnoticed
by the members of the environment. According to Clark and Estes (2008), resources are tangible
supplies and equipment are necessary to achieve organizational goals. Clark and Estes (2008)
also suggest that an organization’s tools and material supplies need to be available and designed
in a way that supports an organization’s goals. Table 6 details Chapter Two’s organization
influences and their method of assessment.
Table 6
Summary of Organization Influences and Method of Assessment
Assumed Organization Influences Interview Questions Document Analysis
Cultural Model (beliefs, shared
beliefs, and values)

Organization Influence:
Organizations need to convey the
value for the implementation of
adaptive courseware.
How can organizations
convey the value for the
implementation of
adaptive courseware?
Review documents to
assess perceived value
represented through
implementation
practices, applications
and outcomes.
Resources
(what time, energy, etc. do they
have?)

Organization Influence:
Organizations need to provide
resources for the implementation of
adaptive courseware.
How can organizations
provide resources for the
implementation of
adaptive courseware?
Review documents to
assess utilized, areas of
need and requested levels
of resources.
49

Participating Stakeholders and Sample Selection
The stakeholder group of focus for this study is the Project Leads who direct the
implementation of adaptive courseware programs at universities. For the APLU grant, eight
universities are participating in a 3-year adaptive learning and adaptive courseware
implementation program.
Sampling
The criteria used in the sampling for this study was typical purposeful sampling.
According to Merriam and Tisdell (2016), typical purposeful sampling is a subset of population
representatives in the research setting of interest. The selection criteria used in the sampling
identified the stakeholder leaders of adaptive learning implementation programs at higher
education institutions.
Recruitment
For the purpose of this study, the researcher attended the APLU Adaptive Courseware
Convening on September 11 and 12 in Denver, Colorado. Convening attendees represented the
eight adaptive learning grant-participating universities as well as executive director and assistant
director for the APLU’s PLC. Interview participants were the representatives from the grant-
participating universities and from the APLU PLC leadership team.
Interview Protocols
The initial data was collected through semi-structured interviews with Project Leads
representing grant-participating universities. Semi-structured interview protocols were used to
ensure the ability for flexibility in questioning and for topic exploration (Merriam & Tisdell,
2016). The interview questions that were asked are listed in Table 4 for knowledge, Table 5 for
motivation and Table 6 for organization factors. Interview items consisted mainly of open-ended
50

questions, sometimes followed by a probing question, to assess a respondent’s assumed
knowledge, motivation and organization needs. The interview itself was scheduled to last
approximately 30 minutes. Interview participants were emailed interview questions prior to the
interviews, which were conducted via an online meeting platform.
Document Analysis
The eight university grantees publish annual progress reports which were reviewed and
analyzed.
Data Collection
Following University of Southern California Institutional Review Board (IRB) approval,
interview participants were contacted via email and interviewed through a video conferencing
session. Documents reviewed were accessed through program-participating university websites
and through industry reports.
Interviews
Interview requests were solicited in person during an APLU convening held in September
2019. Basic contact information and a general interview expected time frame was discussed with
each prospective participant. With prospective participants residing across the country near their
respective schools, it was agreed that a video conference interview would be convenient and
suitable. Each interview was approximately 30 minutes long. All interviews were recorded and
transcribed.
Document Analysis
Implementation reports on adaptive learning programs are published from various
universities. Individual school accounts of adaptive learning range from 2015 and continue
51

through 2019. For the APLU 3-year grant, each university has been required to publish an annual
progress report, sharing adaptive learning implementation progress, best practices and outcomes.
Data Analysis
Interviews
The first of two research sources contributed to this study - five interviews with leaders in
higher education adaptive learning programs and document analysis from a group of universities
in year-three of their respective adaptive learning implementation programs. The series of
interviews was conducted through web video conferences in April 2020 to collect qualitative
data, focusing on their ongoing, multiple-year courseware implementation programs. The
interviews were recorded (video and audio), transcribed, coded, and responses were synthesized,
reconciled and summarized.
The recorded interviews were transcribed, reviewed and evaluated. Interview data was
assimilated through Clark and Estes’ (2008) Knowledge, Motivation and Organization
framework. Frequencies and common themes of knowledge were noted. The variables and
constructs associated with motivation were discerned and interpreted. Finally, the
representations, exemplifications and constructs for organization were identified and articulated.
Documents
As a second research source, qualitative data was collected through document analysis
from the eight universities who began adaptive learning courseware implementation programs in
2017. Courseware implementation was reviewed for a three-year longitudinal research window
from 2017 to 2019. Published through the APLU, the eight universities submitted annual reports
chronicling implementation results – with detailed program status, implementation impacts,
success factors, challenges met and best practices identified.
52

Documents including published reports, university website information, industry
publications and empirical research were analyzed. Questions such as how and when the artifact
was produced, for what purpose, as well as how it was used and by whom, were included in the
document analysis process for each artifact collected (Merriam & Tisdell, 2016).
Trustworthiness of Data
To maintain the credibility and trustworthiness of this study, data will be triangulated. A
variety of information collected from a range of participants and sources were evaluated
(Maxwell, 2013). Interviews with Project Leads from adaptive learning implementation
programs provided firsthand (primary source) information from each school and were
assimilated with program-specific published documentation.
Role of Investigator
The investigator in this study provided purpose and context for participating respondents,
collected data and reviewed documentation. The investigator is independent of all the
stakeholders and of all adaptive courseware implementation programs. The investigator works
for a university that is not related, connected or associated with instructional programs at any
other postsecondary institution.
Limitations and Delimitations
The limitations of this study were the sample size and the emerging and progressive
nature of this higher education learning methodology. The grant participants and reviewed
reports were a small sample of the programs employing adaptive learning on a national scale.
Additionally, Implementation programs at the community college level were not evaluated or
reviewed. Since the long-term impact and outcomes of adaptive learning have yet to be
universally established, the data from this study was relative/related to individual schools and
53

selected stakeholders. Since many factors are unique for each university and specific to adaptive
learning implementations, evaluated programs cannot be generalized, assumed to substitute or
expected to replicate.

54

Chapter Four: Results and Findings
Assumed knowledge, motivation, and organizational influences delineated in Chapter
Three were assessed to learn the ways in which institutions of higher education introduced
adaptive learning through adaptive courseware. University objectives, financial considerations,
instructional goals and learning expectations influenced the scope and scale of adaptive programs
at the respective campuses. Two stakeholder groups - a personalized learning director for a
national association and Project Leads at individual universities - were purposefully selected to
gain both a broad perspective and school-specific viewpoint of adaptive courseware
implementation, application and practice. A second source of qualitative data was a longitudinal
series of reports from the group of schools implementing adaptive learning courseware.
The gap analysis approach (Clark & Estes, 2008: Rueda 2011) will assist in the analysis
of the performance, challenges and barriers in knowledge and skills, motivation and
organizational culture. Based on a review of the research, an evaluative knowledge, motivation
and organization framework was employed to analyze and assess the amassed data. From the
collected data, influences, behaviors and determinants were identified and an outline was
developed. These courseware implementation considerations helped to guide and provide scope
and context for the evaluation process.
To most effectively review and assimilate the research findings, an evaluative outline and
profile has been developed. This principled and structured methodology followed Clark and
Estes’ (2008) Knowledge, Motivation and Organization framework. Processed through the KMO
framework, accountable influences were identified and synthesized. The requisite influences
were symbolized through express critical behaviors expected of Project Leads to achieve
program and implementation goals. Interview questions put appropriate focus and query onto the
55

critical behaviors. The coordinated and interconnected evaluative outline and framework is
shared as follows:
• Knowledge
o Project Leads need to know how to select appropriate adaptive courseware.
▪ Of the three general types of adaptive courseware, what have you
observed as having the best outcomes?
▪ Are there subjects that particularly match well with adaptive courseware?
Are there subjects that do not work well with adaptive courseware?
▪ What are reasons why universities want to introduce or use adaptive
learning courseware?
o Project Leads need to know how to implement adaptive courseware at their
school.
▪ Describe the current state of adaptive courseware implementation at your
school?
▪ From your observations, how is the adaptive courseware being utilized?
▪ What are they keys to a successful adaptive courseware implementation
program?
▪ How is continuous improvement for courseware implementation applied
or exercised?
o Project Leads need to know how to collaborate with others in order to implement
adaptive courseware.
▪ Please talk about the significance of a coordinated team for courseware
implementation.
56

▪ What has been the faculty experience to date with the implementation of
adaptive learning courseware?
o Project Leads need to know how to utilize adaptive courseware to achieve student
outcomes.
▪ What indicators/outcomes does your institution use to measure student
success?
• Motivation
o Project Leads need to be motivated to implement adaptive courseware.
▪ Identify any contributors from anywhere in the interview.
• Organizational
▪ Organizations need to convey the value for the implementation of adaptive
courseware.
▪ Organizations need to provide resources for the implementation of
adaptive courseware.
Participating Stakeholders
Two research sources contributed to this study - five interviews with leaders in higher
education adaptive learning programs and document analysis from a group of universities in
year-three of their respective adaptive learning implementation programs. The series of
interviews was conducted through web video conferences in April 2020 to collect qualitative
data. Interviews varied from 45 to 60 minutes each and chronicled implementation programs
from five states, ranging from the East Coast to the West Coast of the U.S. The interviews were
recorded (video and audio), transcribed, coded, and responses were synthesized, reconciled and
summarized.
57

Qualitative data was collected through document analysis from the eight universities who
began adaptive learning courseware implementation programs in 2017. Courseware
implementation was reviewed for a three-year longitudinal research window from 2017 to 2019.
Published through the APLU, the eight universities submitted annual reports chronicling
implementation results – with detailed program status, implementation impacts, success factors,
challenges met and best practices identified.
Additionally, as a reference source, a 2018 report entitled A Guide For Implementing
Adaptive Courseware published by a national education consortium was reviewed and appraised.
Interviews
Requests for interviews were sent to ten leaders of adaptive learning courseware
programs, including to the Program Leads the eight schools participating in the Accelerating
Adoption of Adaptive Courseware at Public Universities grant program (AAAC). Four Program
Leads agreed to be interviewed as well a Director for the AAAC grant program, for a total of
five participants.
For this research dissertation, five online video conference interviews were conducted
within a seven-day period in the middle of April, 2020. The interviews were recorded and
transcribed, with results synthesized. As evidenced by authored research papers (each of five
interview participants’ publications were included in Chapter Two’s Literature Review),
university leadership positions and affiliation with the national Personalized Learning
Consortium (part of Association of Public and Land-Grant Universities), all participants have
been involved with adaptive learning courseware implementations in higher education for three
years or more with several for more than six years.

58

Document Analyses
Documents analyzed were annual reports (for the years 2017, 2018 and 2019) from eight
universities (designated University #1 to #8) who participated in the APLU (Association of
Public Land-Grant Universities) Adaptive Courseware Scaling grant. Geographically, the eight
schools represented large metropolitan areas from coast to coast. Undergraduate enrollment from
each of the eight schools ranged from 22,000 to 72,000 students. The grant-required, formative
annual reports from each school chronicled the efforts, progress and challenges encountered with
courseware implementation. A grant-concluding 2020 report will be available later this year.
Findings
Knowledge was assessed through interviews and document analysis. The findings are
categorized by Clark and Estes’ (2008) Knowledge, Motivation and Organization framework,
further focused by requisite and expected influences. For each interview question, key participant
excerpts were followed by a summary of participants’ responses. A synopsis of analyzed
documents was followed by “Key Points from the Data.” The interview question was then
concluded with a collective summary of findings.
Knowledge Influence 1: Question 1
PLs need to know how to select appropriate adaptive courseware.
● Of the three general types of adaptive courseware, what have you observed as having the
best outcomes? With focus on their schools’ courseware implementation programs,
interview participants were asked to describe and discuss specifics of courseware
integration.
○ Participant 1: “The most used is off the shelf where faculty are adding in (some)
modification and (there is) limited customization for faculty.”
59

○ Participant 2: “We're using all of them (off the shelf, hybrid and custom-built).”
○ Participant 3: “I see there are two basic types (of adaptive courseware) ‘construct’
and ‘configure’.”
○ Participant 4: “The issue is working backwards (to see what is the best fit). That's
how you make your determination of, you know, off the shelf or hybrid or build
your own.”
○ Participant 5: “The more successful adaptive courses were created by the faculty
members themselves.”
Interviews
It was evident from the interviews that there was not one universal or prevailing
employment of adaptive software among the universities. The adoption of the three main types
of adaptive software – “off-the-shelf,” “hybrid” or “custom-built” were mentioned across the
universities, with several using two or all three types within their respective schools. However,
the agreement from participants was that the choice of adaptive courseware type was ultimately
in the hands of a faculty or department lead or asserted and advocated by a faculty champion.
The utilization of adaptive courseware, through the three distinct models, was strongly
represented throughout the interviews.
Considerations and justifications for particular adaptive courseware use was influenced
by multiple factors including price, product support, course subject, time constraints, UI (user
interface) as well as campus familiarity with curriculum providers and guidance from school
instructional designers. Two factors related to the selection of adaptive courseware were
mentioned by several participants. One was the ability of faculty to modify an “off-the-shelf”
courseware product, appealing to instructors wanting to incorporate some of their subject-matter
60

content to a course. Also mentioned was the comparative time involved between the relative ease
of adopting an “off-the-shelf” adaptive courseware versus the time-demanding nature of building
an adaptive courseware from scratch.
Examples of distinct university-preference and faculty-inclination were apparent.
Participant 2 stated that at his university, “We're using all of them (off the shelf, hybrid and
custom-built).” Participant 3 shared his university’s approach, “I see there are two basic types (of
adaptive courseware) ‘construct’ and ‘configure’” which generalized to “custom-built” and “off-
the-shelf.” Participant 5 directly identified the most effective type of courseware for her campus,
“the more successful adaptive courses were created by the faculty members themselves.”
Document Analysis
For all adaptive courseware models, the recurrent and cooperative working relationships
with vendors stated in many university annual reports reveals this influence to be an asset. With
regard to curriculum and content developers, University One worked with four vendors across
courses in biology, micro and macroeconomics, history, mathematics, physics and psychology,”
using all types of courseware. University Two employed different models of courseware from
three vendors for 15 courses, covering accounting, biology, botany/zoology, micro and
macroeconomics, engineering, modern languages and physics. University Three contracted with
three vendors for five courses in economics, global issues, American government and
introduction to psychology. University Four began with courses in biology, business, chemistry
and psychology and in year three, added geology, nutrition, philosophy and psychology.
University Six utilized five vendors for courses covering the disciplines of mathematics and
statistics, chemistry, biology, business and physics. University Eight found success outside of
61

STEM courses with positive outcomes in two English Composition courses - Writing 101 and
102.
Key Points from the Data
The adoption of the three main types of adaptive software – “off-the-shelf,” “hybrid” or
“custom-built” were mentioned across the universities, with several using two or all three types
within their respective schools. Interviews revealed that the section and impetus for courseware
product implementation was dependent on a multitude of program-specific factors.
• Choice of adaptive courseware type was determined by the department head or faculty
leader.
• Particular adaptive courseware use was influenced by multiple factors including price,
product support, course subject, time constraints, UI (user interface) as well as campus
familiarity with curriculum providers and guidance from school instructional designers.
• Ability to modify an “off-the-shelf” courseware product, appealed to instructors wanting
to incorporate some of their subject-matter content to a course.
Summary
While all three types of adaptive courseware (off the shelf, hybrid and build your own)
were used by different universities, the recommended type of courseware seemed to be a
preference based on faculty discretion, course consideration or instructional need. Faculty-
created custom applications were also mentioned. Identified in the interviews and documents, the
fact that universities and faculty purposely employ all types of adaptive courseware based on
particular circumstances or context demonstrates that they know the various types of adaptive
courseware and appropriate respective employments.

62

Knowledge Influence 1: Question 2
● Are there subjects that particularly match well with adaptive courseware? Are there
subjects that do not work well with adaptive courseware?
o Participant 1: “ALEKS (Developmental Math, Algebra and Pre-Calculus courses)
(by) McGraw Hill is the one with the most high-quality platform and the quality is
(shown) by the number of topics in the course. (The large amount of content) can
be parsed and (encourages) faculty customization and has a very high level of
satisfaction.”
o Participant 2: “We have spent two years developing physics courses. We have
10,000 to 15,000 questions with feedback (for the physics classes) into the system
now (faculty collaborated together).”
o Participant 3: “There's some mature products in the market that have had much
more success. ALEKS (Developmental Math, Algebra and Pre-Calculus) has been
commercialized for 20 years so the level of maturity is very high.
o Participant 4: “The market for adaptive both expanded and constricted. The
constriction came with start-ups and smaller companies such as Junction, Smart
Sparrow and Knewton were bought out by bigger companies, and some
companies merged such as Cengage (Social Sciences, Humanities, STEM,
Business, Economics and Career Prep) and McGraw Hill (mostly STEM).
o Participant 5: “Realizeit (STEM disciplines) (with instructor provided content)
has stood out with its pretty sophisticated adaptivity features.” “In terms of
(being) user-friendly for learners, other systems definitely lead.”

63

Interviews
It was evident from the interviews that all five participants could share positive examples
of adaptive courseware implementations. “Success stories” from specific adaptive learning
employments commended courseware from several curriculum vendors including McGraw Hill,
Realizeit, Cengage, Smart Sparrow, CogBooks, Junction, Knewton, Hawkes Learning, Top Hat
and Macmillan’s Sapling indicated this influence to be an instructional asset. Three of five
participants explicitly praised ALEKS by McGraw Hill for its higher education adaptive courses
in Developmental Math, College Algebra and Precalculus. Participant 1 specifically referenced
the ALEKS courseware, “ALEKS (by) McGraw Hill is the one with the most high-quality
platform and the quality is (shown) by the number of topics in the course. (The large amount of
content) can be parsed and (encourages) faculty customization and has a very high level of
satisfaction.” Among the reasons mentioned supporting ALEKS courses across the universities
were high faculty evaluations, increased student course completions and higher course final
grades, compared to previous semester courses not using the ALEKS software.
With adaptive learning and adaptive courseware use still pressing forward towards
implementation at scale, with initial focus on the lower division course offerings, several
participants optimistically noted the expected expansion by adaptive courseware vendors into
smaller enrollment classes such as physics, anthropology, history, philosophy, psychology, upper
division STEM and health. Two participants mentioned the experienced and collegial benefit of
pedagogically pairing active learning with adaptive courseware. As Participant 3 conclusively
asserted, the synergetic educational benefit of uniting active and adaptive learning resulted in
compound and extended instructional and learning gains. Participant 2 shared procedural details
of courseware implementation by saying “We have spent two years developing physics courses.
64

We have 10,000 to 15,000 questions with feedback (for the physics classes) into the system now
(faculty collaborated together).” Participant 1 provided insight into the curation of curriculum
stating “ALEKS (by) McGraw Hill is the one with the most high-quality platform and the quality
is (shown) by the number of topics in the course. (The large amount of content) can be parsed
and (encourages) faculty customization and has a very high level of satisfaction.”
Document Analysis
Each university’s reports cited effective examples of courseware implementation across
multiple disciplines, demonstrating this influence to be an asset. University One has deployed
adaptive courseware within individual courses as well as throughout a sequenced course path in
specific majors. To elaborate the school’s innovative direction, a quote from Participant 3’s
interview provides further context and reference. “We have had two different strategies for
adaptive courseware. First strategy was horizontal strategy, doing individual education, college
algebra, and then we went through biology, chemistry, physics, economics, psychology,
philosophy, all individual courses. Okay. Two years ago, we shifted our strategy was a vertical
strategy, we went to an integrated program an entire academic program in biology, Bachelor of
Science in Bio spine, right Bio spine. So, our current strategy is to continue expanding in both
directions. We're going to continue to end until and, and add more vertical.”
As an example, the adaptive courseware in a first year STEM course will “remember”
and “guide” the student through successive and higher levels of the science major. This
horizontal (multi-discipline) and vertical (major-specific) implementation of adaptive courseware
has provided technology-rich, individualized instruction and learning for students and faculty at
scale.
65

Utilizing the instructional development procedure of “design thinking,” University Five
embarked on a college algebra redesign. Working “backwards” from course-level outcomes to
weekly objectives to individual assignments, the implementation team worked closely with the
courseware vendor to produce a completely new course that has been highly successful.
Key Points From the Data
• Positive learning outcomes in Developmental Math, College Algebra and Precalculus
courseware courses.
• Successful course results and improved student outcomes with adaptive courseware in
lower division and STEM courses.
• The expansion by adaptive courseware vendors into smaller enrollment classes such as
physics, anthropology, history, philosophy, psychology, upper division STEM and health.
• Disciplines that don’t work well include those that are hands-on or experiential such as
fine arts.
Summary
As might be expected with the introduction of a new instructional initiative, the
integration of technology with a requisite shift in pedagogy resulted in differing executions and
employments of adaptive learning. Implementation programs shared through interviews and
reports were distinct to an individual university. Great variances were noted among courseware
vendor products, course applications, instructional deployment, goal orientation and faculty
practice. The multiple participant recounts and document reportings of effective implementations
of adaptive courseware highlight particular academic disciplines and course offerings.

66

Knowledge Influence 1: Question 3
● What are reasons why universities want to introduce or use adaptive learning
courseware?
o Participant 1: “Lower-level courses are definitely the biggest area.” (Also) high
enrollments are probably just as likely as your high D/F/W now.” “I see
absolutely no differences of (between) on-campus or online (courses).”
o Participant 2: “I would say that our biggest driving force was our graduation rate.
And although we knew that it was not likely that we'd ever be able to connect
courseware directly to graduate, we were hoping that any contribution that we
made to that would help.”
o Participant 3: “We have one mission statement and four learning objectives for
this (adaptive learning) project. Our mission statement is to enable students’
success. The objectives are to "increase their knowledge of the application of that
lesson.” “We want to increase their success rate in the class to more than 90% of
the grade of C or better.” “We want to reduce the withdrawal rate from class to
under 5%.” “And we want to enable the faculty members to identify struggling
students within the first two weeks (of) class.”
o Participant 4: “To support student success, particularly among first-generation and
low-income students, through the implementation of personalized learning
strategies.”
o Participant 5: “I think (a top administrator) believed that adaptive learning could
really be in use for large enrollment, gateway and undergraduate courses,
67

especially for courses with high and D/F/W. I think that's the reason that we
started with adaptive learning.”
Interviews
It was evident from the interviews that anticipated or prospective learning outcomes from
the use of adaptive courseware were guided by specific institutional directives or departmental
objectives. Universities had individual and unique perspectives and expectations towards student
outcomes for courseware. Aggregate academic indicators such as graduation rates and course-
specific expectations such as final grade distributions were noted by participants. Two
participants did comment that graduation rates - the presumptive academic goal for learners and
the institution, is a cumulative, nuanced and composite declaration of learning at the higher
education level, though not necessarily a representation of adaptive courseware efficacy. In a
summative statement of adaptive courseware across higher education, Participant 1 cited
“Adaptive is just a little part of AI (artificial intelligence) and is definitely going to help make us
much more efficient when it comes to learning.”
Through the interviews, numerous measures and indicators of student outcomes were
identified. Mentioned as factors influencing learning results included analytics pulling from
multiple sources of data – information from multiple sources across campus for example, the
growing influence and the emerging potential of artificial intelligence to make learning more
efficient, raising student-received course grades in identified classes and the ability to
personalize learning through adaptive courseware for struggling or under-represented student
groups.
Looking towards the future of improving student outcomes, Participant 1 commented
“Adaptive is just a little part of AI (artificial intelligence) and is definitely going to help make us
68

much more efficient when it comes to learning.” Participant 2 succinctly shared “I would say that
our biggest driving force was our graduation rate.” Participant 3 directly and summarily added
“We have one mission statement and four learning objectives for this (adaptive learning) project.
Our mission statement is to enable students’ success. The objectives are to "increase their
knowledge of the application of that lesson.” “We want to increase their success rate in the class
to more than 90% of the grade of C or better.” “We want to reduce the withdrawal rate from
class to under 5%.” “And we want to enable the faculty members to identify struggling students
within the first two weeks of class.”
Document Analysis
University reports revealed disparate and distinct presumptions and expectations for
courseware implementation disclosing this influence to be a need. University Seven shared
recent positive student outcomes supported by implemented adaptive courseware. “Over the last
three years, College Mathematics has seen a 20% drop in the rate of Ds Fs and Withdrawals
(DFW rates). The University has found success through the use of an adaptive platform and the
Emporium Model. Classes are held in a computer lab with students working at their own pace.
Color-coded flags signal the instructor for help or display progress. Learning Assistants (Grad
Students and Undergraduate Student Workers) walk around the room answering questions”.
University Two reports stated administration’s target of specific goals of improved retention,
completion and the elimination of success gaps. Reports from University Three highlighted
courseware’s contribution towards improved pedagogy and overall student success.
Key Points From the Data
• Improved graduation rates and final grade distributions.
• Improved course completions, lower withdrawal rates.
69

• Adaptive courseware implementation was mentioned to address enrollment challenges
and as well as to support academic equity, student inclusion and course access.
• Adaptive courseware’s AI makes learning more efficient.
• The courseware’s ability to personalize learning for struggling or under-represented
student groups.
Summary
Student outcomes and specifically, student success, was at the center of adaptive
courseware implementation program goals. Universities championed the use of data and learning
analytics to support instruction and to promote and progress academic equity. While student
success was cited as the communal, broad-based and sweeping objective, representations and
affirmation of such objectives were more variant and less accordant. Viewed indicators,
measures and demographics varied from program to program. Corroboration of heterogeneous
instructional expectations from interviews and document analysis show that goals and objectives
were specific for a school, department or course.
Knowledge Influence 2: Question 1
PLs need to know how to implement adaptive courseware at their school.
● Describe the current state of adaptive courseware implementation at your school?
o Participant 1: “Pre-COVID, about 1/3 of the faculty teaching undergraduate level
are using technology tools (adaptive).” “With this current state of teaching
remotely, adoption of adaptive courseware will be much higher.”
o Participant 2: “We have had great support from administration (for adaptive
courseware), a tremendous effort in physics and excellent data from early college
math classes.”
70

o Participant 3: “We have two strategies for adaptive courseware – horizontal
(individual courses including algebra, biology, chemistry, physics, economics,
psychology and philosophy) and vertical implementation (a full course sequenced
biology major and an in-progress business major) that are working well.”
o Participant 4: “Approximately one-fifth of faculty use digital learning tools for
teaching. Often these are used as homework supplements or as a quizzing
platform as well as providing a digital copy of the textbook.”
o Participant 5: “Our school has piloted (adaptive courseware) for three, four years
but now it (adaptive courseware) is just normal operation for us.”
Interviews
It was evident from the interviews conducted in April 2020 that all five participants were
keenly aware that higher education up to that point, on a national level, was in the midst of
systemic and fundamental change. Unsolicited, participants commented that teaching and
learning in post-secondary education was in an unprecedented, reflexive and reactive state of
transition due to the COVID-19 pandemic. Teaching and learning, as they knew it, constrained
by factors well outside of education, would shape and shift through an online learning construct
moving forward. Forced to scale without notice, definition and recognition of online learning
exemplars, capabilities to equalize and enhance learning access, inclusion and equity at a
distance and consensus of instructional best practices were in a process of being determined.
Continuing, it was also evident that all five participants were resolutely optimistic and
bullish on the outlook for and value of adaptive learning courseware in a newly focused, online
higher education environment illustrating this influence to be an asset. Participant 1 shared that
“Pre-COVID, about one third of the faculty teaching undergraduate level are using technology
71

tools (adaptive)” and expected that percentage to increase with the move into distance learning at
scale.
Evaluating adaptive courseware implementation through student data, Participant 1
shared that faculty observed excellent learner data from physics and early college math classes,
adding to faculty’s perception that adaptive courseware could make a difference. As an “early
adopter” to adaptive courseware, Participant 3’s university used four years of improved student
learning outcomes (increased course completions, higher student grades, lower attrition and
increased course access and inclusion for students) to expand adaptive courseware
implementation “vertically” into a four-year biology major with a four-year business major
currently in the works. Participant 2 summarily stated “there is a lot of belief that it (adaptive
courseware) can make a difference.” Participant 4 disclosed that a key to implementation success
was “fostering a culture of innovation among faculty.”
Document Analysis
Adaptive courseware instructors utilizing learning analytics data together with positive
input from students represented this influence to be an asset. Representations of metacognition -
knowledge about cognition and the regulation about cognition, appear in many university
reports. For new technology and methodology at scale, evaluation and reflection appear in both
formative and summative accounts. Student input towards adaptive learning, shared through
focus groups and surveys, were reported in six of eight university reports.
University Three reported that “exit surveys from 2,800 student showed that the majority
of students believed adaptive and other digital tools helped them with their studies.”
In Spring 2019, University Seven reported a 20% decrease over the past three years in the
rate of Ds, Fs and Withdrawals for college mathematics. University Seven attributed success due
72

to the use of an adaptive platform and the employment of an emporium model (computer lab)
with students working at their own pace, assisted by graduate and undergraduate students
walking around to answer questions.
Key Points From the Data
• Adaptive learning courseware was increasing scale year over year.
• In the midst of a pandemic, with a renewed focus on distance education, expectations are
that the number of programs using adaptive courseware would significantly increase.
• One university, known as the national higher education innovator and leader in adaptive
courseware, used four years of improved student learning outcomes (increased course
completions, higher student grades, lower attrition and increased course access and
inclusion for students) to expand adaptive courseware implementation “vertically” into
the upper division courses for a four-year biology major.
Summary
The procedural and metacognitive influences of program implementation and reflection
were expressly and openly shared across interviews and university annual reports. As the
interviews were conducted in April 2020, in the midst of a Covid-19 impacted education system,
a greater emphasis and significance on learning technologies was mentioned as an instructional
asset and needed learning resource. All interviews voiced that higher education moving forward,
at a distance, would benefit from digital and online capabilities and capacities to engage and
promote teaching and learning both synchronously and asynchronously. Several interview
participants noted that adaptive learning courseware, in particular, was well positioned to
welcome and receive a suddenly burgeoning, mostly involuntary and considerably apprehensive
student population. While much is unsettled about moving forward with new or revised
73

instructional models in higher education, the known capabilities and experienced benefits of
adaptive courseware bode well for distance learning on a large and formidable scale.
Knowledge Influence 2: Question 2
● From your observations, how is the adaptive courseware being utilized?
o Participant 1: “Number one and two reasons are homework systems and student
practice.” “I think (these uses) could grow very fast in this post-COVID world.”
o Participant 2: “You can get improvement out of the courseware regardless of how
you use it.” “They make good homework systems.” With “adaptive courseware,
we're able to measure how much the student is engaging, we're able to measure
where the student is reading and where the student is focusing.”
o Participant 3: “The optimal design (for adaptive courseware) is (for) lecture
replacement.” “You want to input all of your instructional resources into the
adaptive system, (then) videotape your lecture.”
o Participant 4: “In the beginning, courseware was being used as a digital textbook
and a homework platform. That's what most faculty we're using it for.”
o Participant 5: “The majority (of faculty) use it (adaptive courseware) as a
textbook replacement.” “They (faculty) create their own content, record their own
videos and then the students do not have to buy a textbook.”
Interviews
It was evident from the interviews that all five participants knew, at the class-level and
course-level, adaptive courseware was being used in multiple and distinct instructional ways.
The applications and employments of the adaptive courseware included use as a “homework
system (Participant 4), for “student practice” (Participant 1), to offer “affordable instructional
74

materials” (Participant 4), as “lecture replacement” (Participant 3) and as “textbook replacement”
(Participant 1). Participant 4 shared that courseware vendors were astutely marketing their
products as inclusive instructional solutions to faculty as “here’s a package that has the textbook,
digital textbook, quizzes based on the textbook, homework based on the textbook and it’s auto
graded. It also gives you exams and PowerPoint slides.”
As all interviews were conducted in April, 2020, involuntarily framed through a mid -
COVID-19 backdrop, several participants noted the likely and progressive instructional benefit
from educational technologies that enhance and engage a burgeoning online student population.
With focus on combining the use of adaptive courseware with the increasing pedagogical
integration of “active learning” in higher education, Participant 3 commented that “the optimal
design (for adaptive courseware) is (for) lecture replacement. You want to input all of your
instructional resources into the adaptive system, (then) videotape your lecture.” Participant 5
shared that adaptive courseware not only acted as a textbook replacement but provided a digital
platform for faculty to embed their own curriculum content, add videos and link to Open
Educational Resources.
Document Analysis
All annual university reports expressively cited multiple examples of utilizing the
capabilities and capacities of adaptive courseware, deeming this influence to be an asset. With an
eye towards promoting scope and scale, University Eight’s annual adaptive courseware report
recounted adaptive courseware implementation across a broad range of first- and second-year
courses. “While many consider adaptive courseware to be the domain of STEM disciplines,
University Eight found success in English Composition - Writing 101 and 102 as faculty worked
with Lumen Learning on a course using Open Educational Resource (OER) material. This
75

approach produced courseware that is relevant, engaging and connected to the needs and
experience of faculty and students.” To advance and sustain courseware scale, University One
employed Course Coordinators to train new faculty, support courseware integration and liaise
with vendors. The Course Coordinators served to guard against negative effects of faculty and
staff turnover. Similarly, University Five created a new position - Director of Foundational Math
to provide oversight for first-year math courses making use of adaptive courseware and active
learning. University Seven utilized an Emporium Model with students in an open lab
environment, staffed by learning assistants trained in the use of adaptive courseware.
Key Points From the Data
• Schools used adaptive courseware as a homework system, student practice, lecture
replacement and textbook replacement.
• Courseware vendors sold their products to schools as an inclusive instructional solution
that included the textbook, digital textbook, quizzes based on the textbook, homework
based on the textbook, PowerPoint slides, tests and auto grading.
• Adaptive courseware capabilities were employed and applied by faculty based on
individual discretion, course consideration or instructional need.
• Courseware capabilities were used to regulate course and instructor variability.
Summary
The employment and application of adaptive courseware seems to be a preference based
on faculty discretion, course consideration or instructional need. The role and application of
adaptive courseware ranges from instructional support for faculty to student support as a
curriculum resource to program support to regulate course and instructor variability. In a dual
sense, faculty are adopting adaptive software based on curriculum topic and instructional
76

purpose. In addition, one participant adds a third factor and consideration – the integration of an
LMS (Learning Management System), which may support or challenge adaptive courseware use.
The various functional employments of adaptive courseware cited through the interviews and
reports asserts clear and applicable instructional utility and purpose.
Knowledge Influence 2: Question 3
● What are they keys to a successful adaptive courseware implementation program?
o Participant 1: “You have to have a core group of staff that are capable of
understanding the power of the tool. And they have to be introducing this to
faculty on a regular basis. So, there's a lot of planning that has to go into this, then
there's some discovery around (and attempting to obtain) buy in from the faculty.”
“From there, you really want to think about a pilot and that pilot should focus (not
only on) the (courseware) tool but also the redesign that needs to occur.”
o Participant 2: “If I were to design the utopian best way of implementing, I would
start with complete course redesign. I think traditionally, universities nationwide,
faculty, professors all across the country have focused on content, or, you know,
programmatically making changes to courses (content focused), when we should
have been focusing on the learner to begin with, you should have been involved
with the backwards design approach (and) faculty don't do that. We should have
been thinking about when my student leaves my class, I want them to be able to
do this” (and know this).”
o Participant 3: “Where you get the most learning gain is when you combine it
(adaptive courseware) with an active learning model. So, we've called that the
77

‘Adaptive-Active approach’. But if you give them active learning, sometimes you
can double or even triple the learning.”
o Participant 4: (The keys are the ability to do) “course revision and course
customization (to align) to the student population and evidence-based continuous
improvement efforts.” “I think (keys are) course revisions so that you can align
what's happening in the course where what's happening in the classroom.”
o Participant 5: “We had started a course redesign initiative in the past three years
ago. When we talk about course redesign, I think it takes probably at least two or
three deliveries (semesters) before the instructor should see some significant
improvement.”
Interviews
It was evident from the interviews that all five participants clearly identified keys to a
successful adaptive courseware implementation. Successful programs addressed both the new
adaptive learning courseware technology to be learned but also assistance for the instructional
methodology of using the new courseware. Participants spoke of the dual importance of
supporting both the technology and pedagogy to affect a successful implementation program.
Response perspectives varied from sharing past program experiences, reporting current program
influences, to imparting learned best practices.
With multiple courseware implementation across many schools and subjects at his large
university, Participant 2 posited that, ideally, he would start with courseware redesign, meaning
that, with substantial faculty support, the “custom-built” courseware model would be the most
effective method and framework. Participants idealized, “If I were to design the utopian best way
of implementing, I would start with complete course redesign.” His idea would begin with a
78

“backwards design approach,” identifying instructional outcomes first and then instructionally
and programmatically configuring the software in working to meet the needs of the learners. He
conceptualized his pedagogical model by meta-cognitively asking “what do we need the student
to know and be able to do after they complete this class.”
With the longest tenure and experience (six years and counting) in adaptive courseware
implementation, Participant 3 adeptly summarized the essential factors required for an effective
program deployment. His keys to successful courseware implementation began with leadership
and faculty commitment, followed by the requisite support through monetary resources, program
management, instructional design and the robust relationship with a technology partner
(courseware vendor). Looking from a broad pedagogical perspective, Participant 1 shared “So I
think it's really important during the roll out to make sure you're building expertise around using
the data, leveraging the data. And then what does that mean in terms of a different kind of
teaching or a different set of teaching practices.” Participant 3 disclosed the collective and
collaborative keys and participants to successful implementation: leadership, faculty
commitment, monetary resources, program management, instructional design and technology
partner (courseware vendor).
Document Analysis
Keys to implementation programs were cited in all university documents over the three-
year reporting period. University Three’s courseware implementation program “consists of three
phases - pilot, evaluate and scale - to reach full implementation in micro and macroeconomics,
global issues, introduction to American government and introduction to general psychology.”
University Five had plans in place to “monitor course level student success metrics, retention and
graduation from year to year.” University Eight saw sustainable success in courses where a
79

consistent course coordinator is in place, and instructors are included in the process of
continuous change through feedback sessions, training, and course revisions”.
Key Points From the Data
• Keys to successful courseware implementation began with leadership and faculty
commitment, followed by the requisite support through monetary resources, program
management, Instructional design and the robust relationship with a technology partner
(courseware vendor).
• Learning is amplified when combining active learning with adaptive learning courseware.
• Start with courseware redesign, meaning that, with substantial faculty support, the
“custom-built” courseware model would be the most effective method and framework.
Summary
Through several years of adaptive courseware implementation experience, universities
had identified specific facets and dynamics for successful adaptive learning courseware program
implementation. The cooperative and allied efforts of staff, faculty and vendors for successful
integration was mentioned throughout interviews and reports. With the evolving technology and
nascent pedagogy embodied by the relative recent arrival of adaptive courseware at two- and
four-year institutions, this metaphoric chapter is continuing to be written. Mid-pandemic, the
expanding application of adaptive courseware and its attendant instructional methodology will
likely increase as distance learning becomes more of a conventional learning solution for higher
education students.
Knowledge Influence 2: Question 4
● How is continuous improvement for courseware implementation applied or exercised?
80

o Participant 1: “The continuous improvement is definitely where you continuously
I'd say redesign (during a semester) which is easier to call active learning, So the
continuous improvement is not only tweaking how you use the software (adaptive
courseware), but it's definitely trying to find enough things in your toolbox so that
when you do the rest of the course, you'll have enough choices to personalize it.”
o Participant 2: “As someone that leads workshops to help faculty development and
improve teaching and learning. I keep the motto that we're always improving.
There's always something that we can do growth mindset.”
o Participant 3: “We use a step function for improvement each semester we debrief,
and we make adjustments for the next semester. So, in terms of stability, we try
not to make too many changes during a semester, because (doing so) could create
a lot of confusion and chaos in a hurry.”
o Participant 4: “Our Spanish program, I think this is probably the best model that
I've seen is that it is one master course. But then the instructors who are not
adjuncts, and they're not graduate students, they are full time instructors. It's very
collaborative. It's very strict in terms of we're all going to be working on the same
chapter at the same time, we're all giving the same exams, but let's talk together
about what's working and what's not. And (they) design and improve (their)
classes that way.”
o Participant 5: (Regarding a continuous improvement model of adaptive
courseware implementation between school support staff and faculty during the
semester) “We finish the process of course design and development before the
81

course delivery. So once the course has started, we don't recommend (for faculty
to) make any changes.”
Interviews
It was evident from the interviews that all five participants planned for and experienced
the benefits of a collaborative implementation program effort. Four of five participants
mentioned the presence of a culture-based, pervasive continuous improvement model
environment within departments or with faculty documenting this influence to be an asset. Each
participant quickly recalled instances of building culture or examples of in-place culture
supportive of the adaptive courseware implementation.
Support provided to faculty of adaptive courseware was deliberate, purposed and multi-
dimensional, according to all five participants. An across-the-board adaptive courseware
implementation support team included academic management, Project Leads, faculty leads,
course leads, school tech support, school instructional designers, vendor technical support,
vendor instructional designers, vendor curriculum developers and analytics staff. Participants
spoke of pre-semester adaptive courseware training sessions, in-semester technical focused
asynchronous tutorials sent to faculty. Vendors hosted virtual “open mic” Q & A sessions for
faculty during the school year. Several participants mentioned financial incentives to faculty for
adaptive courseware adoption and continued use.
Participant 2 shared the mood of his culture-building leadership sessions with faculty.
“As someone that leads workshops to help faculty development and improve teaching and
learning. I keep the motto that we're always improving. There's always something that we ‘can
do’ growth mindset.” Behind Participant 2’s buoyant positivity for adaptive learning is affirming
support from leadership. “I definitely think the university administration is behind it sees the
82

improvement and the gains that we have (from the adaptive courseware implementation
programs), and its potential” shared Participant 2. Participant 5 noted that adaptive courseware
using faculty felt supported, even mid-semester. “And that's really one of the benefits or the
benefits to adaptive courseware. You know, during the middle of a course, being able to change
their teaching their instruction, because they found some information from the adaptive
courseware, maybe in week four or week eight, or whenever it is” helped build a culture, a
collaboration of support.
Participant 3 detailed an inclusive support program and culture-building approach for his
university math department. “We meet before every semester for a one-day workshop on
analyzing the pedagogy and the technology and making any adjustments. So, it's an all-faculty
meeting. Sometimes there are 50 faculty members in this meeting. And it really gives them a
sense of solidarity, because they get a chance to share ideas and insights that they wouldn't
normally during their isolated teaching experience, right because they're in their individual
classrooms.”
Document Analysis
The sweeping acknowledgement of the essential presence of a conducive culture for
implementation substantiates the importance of relationships and cooperation. University One
employed Course Coordinators to assist implementation at scale. Course Coordinators were
responsible for training new faculty and liaising with curriculum vendors while also serving as
guards against the challenges of faculty and staff turnover.
Administration at University Two shared that school “culture is moving to a more digital
and analytical approach to understanding and promoting student engagement in and out of class.”
University Two publicized that “a thoughtful course redesign of a course could be the first step
83

towards success in implementation of adaptive courseware.” Using the example of their
Fundamentals of Accounting course, University Two reworked the course to meet the needs of
students - correcting the mix of financial and managerial accounting classes previously offered.
University Three offered an experienced and nuanced reflection on effective program
integration and moving towards program scale. “There is no single way to implement adaptive
courseware and no single timeline for success. Each implementation requires a different
approach that brings faculty, administrators, instructional designers, technologists and
courseware providers together to create a course that has the greatest potential to positively
impact student success.”
Key Points From the Data
• Program leadership scheduled pre-semester adaptive courseware training sessions and in-
semester technical focused asynchronous tutorials for faculty.
• An across-the-board adaptive courseware implementation support team included
academic management, Project Leads, faculty leads, course leads, school tech support,
school instructional designers, vendor technical support, vendor instructional designers,
vendor curriculum developers and analytics staff.
• Vendors hosted virtual “open mic” Q & A sessions for faculty during the school year to
assist continuing implementation and to work towards program efficacy.
Summary
A continuous improvement model, sometimes explicitly stated, was mentioned by several
interview participants, was based on the collaborative efforts of various stakeholders and
stakeholder groups. The collaborative and cooperative undertakings were directed to adapting,
adjusting and improving the learning model for adaptive courseware.
84

Reviewing the interview transcripts, participants shared multiple examples supporting
the presence of conducive and contributory cultural settings within their programs. Repeatedly,
the implementation of adaptive courseware, revealed in interviews and reports, was described as
collaborative, cooperative and collegial.
Knowledge Influence 3: Question 1
PLs need to know how to collaborate with others in order to implement adaptive
courseware.
● Please talk about the significance of a coordinated team for courseware implementation.
o Participant 1: “We have incredible faculty, innovators, but we also have a lot of
faculty who just need, you know, a really good course model to help them
(become a more effective, efficient instructor).” “We want faculty to feel like
they're much more effective at helping their students learn.”
o Participant 2: “We know that there are better implementations than others. And
we're trying to share that feedback with the faculty that are currently using it.”
“And as we find people that are willing to try adaptive courseware for the first
time, we're talking about those best practices up front.”
o Participant 3: “So our first most important point is that faculty lead this initiative.
It's not a technical initiative. It's an academic initiative. And I talk a lot about that.
It's a team sport.”
o Participant 4: “The collaboration between the university personnel and the team at
Lumen initially took about nine months, with several in-person sessions over the
summer months on our campus. What was great about this approach to
85

developing a course is that it balanced the needs of our school’s faculty and
students with the expertise and experience Lumen has in digital learning.”
o Participant 5: “We work with any professors who have gone through training
(adaptive courseware) and who have not gone through training, we support them,
we support them 100% we don't just teach them fish, I think a couple of them, we
teach them fish, and then they can fish themselves with little things.”
Interviews
It was evident from all of the interviews that all participants worked collaboratively with
faculty to implement adaptive courseware within the scaling grant’s guidelines and policies. In-
place school and department processes and protocols for the introduction of new instructional
materials were also cooperatively followed. Each participant spoke of a well-planned,
coordinated and deliberate process that, after initial launch, facilitated and evaluated the
implementation and ongoing employment of adaptive courseware, a collective effort indicating
this influence to be an asset.
Specific examples of implementation methods and procedures were cited by participants.
Suggesting a practical method and process to help faculty with implementation, Participant 1
voiced “a lot of faculty who just need, you know, a really good course model to follow to help
them (become a more effective, efficient instructor). It is essentially we want faculty to feel like
they're much more effective at helping their students learn.”
Participant 3 advances two examples, one emphasizing an “organizational cultural
setting” and one highlighting “organizational structure and strategy” that support an effective
adaptive courseware implementation. First, the implementation “is an academic initiative, not a
technical initiative. It is a team sport.” Second, “a lesson learned is that you need four levels of
86

leadership to come together. You need faculty leadership, you need the department chair, you
need the dean and you need a provost.”
Participant 4 emphasized the pivotal development of a relationship between the school
and courseware vendors. To supplement a freshman writing course, the university worked with
the vendor and over nine months, developed courseware that balanced the needs of their school’s
faculty and students with the expertise and experience of the software vendor. The collaborative
process included several in-person meetings and work over the summer but resulted in strong
faculty support and high student satisfaction.
Document Analysis
Representing a teaching and learning environment open and supportive of innovation and
adoption, the presence of a positive and welcoming cultural setting resonated within several
university reports. The reports’ many examples of faculty-centered adaptive courseware
implementation activities confirm this influence to be an asset.
Institutional, departmental and program support of faculty for the new adaptive
courseware implementation program was demonstrated in a variety of ways. University One
“supported 20 faculty members who attended a conference on active learning and will now help
spearhead the dissemination of the active and adaptive approach.” University One identified two
essential elements of success for any new implementation - faculty buy-in and overall
momentum. To aid faculty development efforts, a new centralized teaching and learning unit, the
Provost Teaching Academy, was utilized by faculty to launch new courses and set new goals for
courseware implementation.
University Five created two YouTube videos to raise awareness about the successful
college algebra redesign, using exemplars to advocate for increasing adaptive courseware
87

implementation. Additionally, University Five had “ongoing efforts to update their advising
community about course redesign and technology implementation to support students and to
provide student experience feedback.”
University Six hosted annual adaptive courseware faculty events where current
participants shared their work and results. This event was a continuation of an early-adopters
community at their university. University Seven hosted an end of semester Course Redesign
Institute where faculty learned to integrate active and adaptive learning components into courses.
Follow up programming was implemented to meet faculty need.
Key Points From the Data
• Collaborative culture-building sessions between Project Leads and faculty courseware
users was cited as effective.
• A collaborative implementation program included not only program leadership and
faculty but vendor support personnel, instructional designers, IT staff, instructional staff
support and school leadership.
• A culture-based, collaborative improvement model and plan contributed significantly to
implementation efficacy.
• Examples of a positive in-place, existing culture supported the adaptive courseware
implementation.
Summary
All annual reports cited multiple activities, exercises and engagements to support a
collaborative courseware implementation program by the respective universities. Universities
mentioned pre-semester or post-semester adaptive courseware professional development
opportunities – some hosted by the schools or departments, while others were led by the
88

curriculum software vendors. These distinct and purposed activities and requisite involvement of
multiple stakeholders for program support are evidence of a coordinated team effort.
Knowledge Influence 3: Question 5
● What has been the faculty experience to date with the implementation of adaptive
learning courseware?
o Participant 1: “The ones (faculty) that are using well, they're really excited about
the fact that they have this data. They're really excited that they believe while
students may have done homework, it wasn't done well or effectively. So they
think these tools are structuring their homework practice in a way that really helps
the student understand how to do things over and over again until, like the cliché,
they learn how to learn.”
o Participant 2: (Oftentimes) “I think that faculty response always starts out
negative. And I hate to say that, but I think that it always begins negative. It's
almost like you have to prove to them that the work that they've put in had any
return.”
o Participant 3: “There's a cycle that I describe as three times through. It requires
three times through an adaptive course in order to master the teaching process.
The first time, you're really trying to learn how the students are using the adaptive
system, and then how you can do active learning. So that is usually very rough.
First time through the second cycle, which is usually the next semester, it's a
much smoother technology cycle and much more active learning in the classroom.
By the third semester, the professors have developed a high level of expertise in
both the adaptive system and the active learning. And that's where you start
89

seeing the dramatic gains in their outcomes. So, as you're thinking about
implementation, you have to allow time, two years, for example, to develop that
capability in your faculty, and many organizations are not that patient.”
o Participant 4: “I definitely think that it (using adaptive courseware) helps them
(instructors) manage large classes. There's a lot of auto grading that goes on.
There's a lot of students can use the platform to answer questions. And so there's
maybe a reduced amount of questions in class or a reduced amount of visits to
office hours.
o Participant 5: “We have really, really good faculty members who will explore the
adaptivity features to the extreme, integrated every kind of variable questions and
formulas and graphing apps and everything inside to Realizeit, and then follow
the analytics and trying to provide students with suggestions.”
Interviews
It was evident from the interviews that positive and effective experiences with adaptive
courseware implementations were shared between faculty and all five participants. The
relationship-benefit from administration to faculty and faculty to administration were
underscored throughout the five interviews indicating courseware to be a positive instructional
influence.
With several years of adaptive courseware implementation experience as a guide, two
participants disclosed discovering a revealing and significant best practice that combined and
synergized adaptive courseware use with “active learning.” The first attempt (first semester) with
adaptive courseware, as with most new things, was a learning process for faculty to see how
students were using the adaptive system and, contributorily, how could faculty extend and
90

compound courseware learning with active learning. Participant 3 shared that semester one was
usually “pretty rough.” The second cycle, which was usually the next semester, was typically a
much smoother technology cycle and by the third semester, professors had developed a high
level of expertise in both the adaptive system and active learning. Participant 3 summarized that
for effective courseware implementation, two years is required to develop the capability with the
faculty. Participant 2 succinctly stated “you got to go three semesters out before you can ever
start talking about what's happened in reviewing progress” (of an adaptive courseware course).
Document Analysis
The value of an inclusive adaptive courseware implementation program was evidenced
throughout all school reports revealed this influence to be an asset. At University Two, the
APLU’s Accelerating Adoption of Adaptive Courseware grant became part of an institution-
wide commitment to scale the use of adaptive courseware. University Two had in place an
institution-wide commitment to creating a culture that leveraged learning analytics. The
alignment of multiple internal and external data was critical to moving the implementation
program forward.
In the first year of program implementation, University Seven launched an Adaptive
Learning Faculty Learning Community (AL-FLC) of faculty champions to spearhead awareness
and engagement building on campus. University Seven promoted the use of faculty learning
communities on Exploration of Flipped Teaching and Learning to provide faculty with time to
learn about adaptive courseware and how it could provide appropriate levels of challenge to
support flipped approach to curriculum design. University Eight saw “sustainable success with a
consistent course coordinator in place and when instructors were included in the process of
continuous change through feedback sessions, training and course revisions.”
91

Key Points From the Data
• Faculty shared that a significant best practice combined and synergized adaptive
courseware use with “active learning.”
• Faculty shared that for courseware implementation, semester one was usually often
difficult. The second cycle, which was usually the next semester, was typically a much
smoother technology cycle and by the third semester, professors had developed a high
level of expertise using the adaptive system.
• Program Leads noted that a fair review of courseware implementation is after about three
semesters of courseware integration.
Summary
A commonly repeated remark from interview participants and from annual reports is the
direct correlation and influence of faculty support to an adaptive courseware implementation
program. This disclosed influence, according to Project Leads, centered on the faculty’s
perceived “value” of the adaptive courseware itself. The “value” from the faculty perspective
could be increased “learning support” for students, assistance in “class management” for faculty
or the synergistic instructional gains from combining active learning with adaptive learning
courseware. The value or benefit of supportive and cooperative working relationships is not
unique to higher education or an instructional staff. The voice and authority of faculty was
influential and decisive.
Knowledge Influence 4
PLs need to know how to utilize adaptive courseware to achieve student outcomes.
● What indicators/outcomes does your institution use to measure student success?
92

o Participant 1: (Institutions are implementing adaptive courseware with) “what I
would call quasi experimental design, a case study type of approach.” (What)
“institutions (are) really looking for, generally speaking, (are) increases in student
success.”
o Participant 2: “I think that (course) completion rate was a good metric for us. We
know on a state level, the recommendations that we are getting from the state is to
help people get through mathematics. And so that's why looking at some of these
courses, and especially within a math program is how are we getting successful
completion of that course.”
o Participant 3: “The bigger programmatic goal was driven by another number,
which is the freshmen retention rate, high retention here. When we started this, we
were only about 70% of our freshman. Now, that was 20 years ago. Today we're
at 86%. So, our president has created the goal of 90% retention, and that's why
we're so motivated.”
o Participant 4: “From what I have seen, institutions are looking at student final
grades and student progression to measure implementation success. Some
institutions are also conducting faculty and student surveys to measure
satisfaction.”
o Participant 5: “We had started a course redesign initiative in the last three years
ago (to) go with the cost redesign initiative. The adaptive running courses have
better learning outcomes and also improved the DFW rates - especially compared
with the non-adaptive courses.”

93

Interviews
It was evident from the interviews that all five participants, as informed by their
classroom and online instructors, were able to articulate the confluence of influences, dynamics,
considerations and expectations that impact and shape quantitative and qualitative measures of
learning outcomes. All five participants shared that, the bottom line or foundation goal for the
school and departments, was the focus on improving student success, clearly revealing this
influence to be an asset. Faculty, as the instructional extension of the school or department,
worked towards developing their courses, optimizing course delivery and using adaptive
courseware to dually support and improve their teaching and to support and improve the student
learning experience.
The metrics, measures and representations of positive learning outcomes through
adaptive courseware varied from school to school and program to program. Three of five
participants mentioned student grades as a school component in determining student success,
with all three specifically mentioning that adaptive courseware implementation has shown to
improve data on D, F, W (withdrawal) rates. Participants mentioned other indicators of student
success that were addressed by adaptive courseware including course completion rates, first year
retention, retention from fall to spring semester in the first year and second year, student
progression analytics, academic progress for transfer students, academic progress for non-
traditional and under-served student populations, graduation rates as a whole and across
demographics and faculty and student satisfaction surveys.
Participant 3 shared that the school president had set a university wide goal of
specifically improving freshman retention rates. “When we started this, we were only about 70%
of our freshman. Now, that was 20 years ago. Today we're at 86%. So, our president has created
94

the goal of 90% retention, and that's why we're so motivated.” This school-wide initiative is
fueled by many campus factors including adaptive courseware implementation programs. As
mentioned by Participant 2, the faculty attitude towards student grades is not always cooperative
or congenial. “Some faculty do not like the fact that we are measuring D/F/W rates on things but
it was already a metric that's used by not just our institution, but (a) nationwide metric.”
Document Analysis
All university reports cited the importance of strong working relationships between
administration and faculty indicating this influence to be an asset. University Six’s report shared
that their school’s “Board of Trustees had expressed a strong interest in active and adaptive
work. The strategic plan cites improving student success, and raising retention and graduation
rates and OAI (Office of Academic Innovation) believes active and adaptive are key to raising
metrics.” University One’s annual report highlighted “Creating a Culture of Faculty Success”
which featured a discipline-specific faculty learning community to support curriculum creation
and faculty development in math, psychology and biology. Additionally, University One
supported 20 faculty members to attend an active learning conference which helped to spearhead
the dissemination of the active and adaptive learning approach.
To support the implementation of adaptive learning courseware, University Two’s faculty
received professional development opportunities to learn how to incorporate digital learning into
pedagogy with faculty receiving individualized instructional design support. University Two’s
faculty met three times per semester to celebrate successes, share challenges and troubleshoot
implementation situations.

95

Key Points From the Data
• A large-scale adaptive courseware implementation program stated objectives were to
"increase their knowledge of the application of that lesson,” “increase their success rate
in the class to more than 90% of the grade of C or better,” “reduce the withdrawal rate
from class to under 5%” and “enable the faculty members to identify struggling students
within the first two weeks of class.”
• Aggregate academic indicators such as graduation rates and course-specific expectations
such as final grade distributions were noted by participants.
• Artificial intelligence in adaptive courseware will make learning more efficient.
Summary
Applications, integrations and practices of adaptive courseware varied among
universities. Institutional objectives led some implementation programs while specific indicators
of academic standing directed other strategies. Contrastively, some universities used adaptive
courseware to confront D-F-W rates while other schools used the software for other enrollment
or instructional challenges. The motivation to adopt adaptive courseware, varying from school to
school, was cited as being both institutionally-led and academically-directed. The overarching
view was, however, that adaptive courseware could address student outcomes or instructional
challenges. Accordingly, as a multi-dimensional resource of employment and application, the
implementation of adaptive courseware has addressed specific academic and enrollment
benchmarks and objectives.
Motivation Influence
● PLs need to be motivated to implement adaptive courseware. (Identify any contributors
from anywhere in the interview.)
96

o Participant 1: “In this (COVID-19) world where we have to prepare for multiple
modalities, the future for courseware is actually very bright. I mean, in terms of
an LMS but I definitely think faculty are going to see the need (for adaptive
courseware). But now without that connection to your student (being able to look
into the eyes of students in a physical classroom), your need to rely on other sets
of tools, right. And the adaptive (courseware) really fits this bill, on how we're
going to actually improve, particularly in lower-level classes around the learning
and how we help students do more practice in an effective way.” “Right, like so.
rolling out the tool (adaptive courseware) is probably simple. But if you don't use
it well, you know, so there's two things. I mean, simple as a little bit unfair
because people need to understand how to use the tool, what things I'm going to
customize what assessments I'm going to do. But that part is time consuming. And
it often means because it's so time consuming. You're not using the data well,
right. So, I think it's really important during the roll out to make sure you're
building expertise around using the data, leveraging the data. So, it (courseware
data) gives faculty a lot more information about how to intervene instructionally
sooner.”
o Participant 2: “I describe the ideal scenario around adaptive courseware. My
response then was, I wish we had one provider that we could say, we're going to
adopt this provider and we're going to build everything out in this one platform.
Our instructional designers would be trained in this platform and know how to
develop it, right? We'd have more and more people come to us and start to use
this. And I do think that is a great ideal situation.” “And we have to validate those
97

feelings of faculty members, we have to say that, yes, I understand that you're
concerned about your student feedback. And you're concerned about the time that
you're putting in there. But you know, on the flip side of that, I've also come back
with him, here's semesters worth of student feedback, that's very positive. So, this
should help you feel better about that. Right. And I think once they find a product
that they're familiar with, okay, they feel better and it starts to go, the more a
vendor is willing to Step out and, and try to help create things that the faculty
member likes, I think a better response you're going to get from faculty. I think
that as these products develop and get easier for faculty, I think that you will see
them start to respond differently and make use of the data that's coming to them.”
o Participant 3: “Yes, COVID-19 has split the entire educational industry wide
open. There are 1.5 billion students out of school right now. So, in this moment in
time, I'm hoping this is an inflection point that will allow us to implement this
transformational technology (adaptive courseware) at scale.”
o Participant 4: “Let's say, I'm like the Elon Musk of adaptive courseware. I would
be looking at virtual reality and how I, especially (as we are facing) COVID-19,
how can I bring all these students together in one virtual experience, so they're
interacting with each other but it's adaptive because they're doing things. They’re
doing things in the courseware.” “I definitely think that it (using adaptive
courseware) helps them (instructors) manage large classes. There's a lot of auto
grading that goes on. There's a lot of students can use the platform to answer
questions. And so, there's maybe a reduced amount of questions in class or a
reduced amount of visits to office hours.”
98

o Participant 5: (An ongoing dilemma is with) “the chairs and departments input
into it and getting everybody buying (adaptive courseware) navigating the conflict
between program implementation and the instructors and academics and their
freedom to teach whatever they would like to do. So, I think those are our
challenges that we have in terms of (adaptive courseware) implementation.” “And
that's really one of the benefits or the benefits to adaptive courseware. You know,
being able to like, during the middle, of course, being able to change their, their
teaching their instruction, because they found some information from the adaptive
courseware, maybe in week four or week eight, or whatever it is. So, I'm glad
they're doing that, because that's one of the selling points, correct?”
Interviews
It was evident from the interviews that all five participants’ current realizations and
forward expectations were processed through a backdrop of the current COVID-19 pandemic.
Through their responses, all five participants expressed high expectations and a fortuitous timely
opportunity for the future of adaptive courseware in higher education, inadvertently more
contributory after the abrupt and unanticipated move to distance learning. All five participants
were involved in year three of a three-year adaptive courseware scaling grant, were able to
discern and ascertain the positive value of adaptive courseware on an increasing scale,
throughout departments and across disciplines.
In support of adaptive courseware utility and efficacy moving forward, Participant 1
highlighted the courseware’s “modality flexibility,” supporting faculty to promote academic
equity, facilitate instructional access and foster learner inclusion for students at a distance.
Through three years of providing adaptive courseware support for multiple instructors over
99

multiple courses, Participant 4 motivational mindset foreshadowed greater educational
possibilities for adaptive artificial intelligence-fueled technologies. Participant 4 spoke of the
conceivable collaboration between adaptive courseware and VR (virtual reality) headsets to
create a “Star Trek-like ‘holodeck’ – or virtual world, where “together,” students in a virtual
classroom, led by their real teacher in an avatar, virtual form, would experience individualized
and differentiated learning and lessons within discrete environments, locations and mediums.
Participant 4 added, “I think about the future of a virtual classroom (with students participating
from home) and it could be very different in five years.”
Participant 3 shared that, while there may have been ongoing overtones of “a cultural
resistance to online education” pre-COIVD-19, “that contention has largely dissipated with the
new realities of higher education learning.” Continuing, participant 3 voiced “So in this moment
in time, I'm hoping this is an inflection point that will allow us to implement this
transformational technology (adaptive courseware) at scale.” Through a broad lens, Participant 1
shared perspective for adaptive courseware in general. “What we're looking for in a post COVID
world, which could last two to five years, we are looking for modality flexibility. (Adaptive
courseware) “is a much more ubiquitous tool we have to focus on multiple modalities.”
As a cautionary balance to motivation, Participant 5 shared the virtues of patience and
having a longer-window perspective. “When we talk about course redesign (implementing
adaptive courseware), I think it takes probably at least at least two or three deliveries (semesters)
before the instructor should see some significant improvement because it (takes some time) to
get used to the new environment and everything.”

100

Document Analysis
Through three years of adaptive learning courseware program implementation, all
university reports referenced positive expectations moving forward. With focus and intent on
increasing scale, largely in first and second year, high enrollment and high attrition courses, the
expressed value and benefit of adaptive courseware affirm this influence to be an asset.
University Five, employing a broad perspective, discovered that building collaborative
teams of faculty with diverse perspectives resulted in implementation approaches that supported
students of all backgrounds. Harnessing their significant influence, University Five spotlighted
instructors that had successfully implemented adaptive courseware as the best champions for
scaling across departments and disciplines.
University Six’s leadership, together with the Office of Academic Innovation (OAI), met
with faculty on a recurring basis to promote and support gateway course redesign using adaptive
courseware and to affirm alignment in times of leadership change and adjustments to strategic
priorities.
University Seven wrote “Adaptive learning supports the school’s strategic plan.
Administrators and faculty are committed to the 21st Century Initiative and student success.”
Key Points From the Data
o Adaptive courseware implementation, as expected with the integration of a new
instructional methodology, necessitates a bigger-picture and sustained deployment
of about three semesters to see positive program results.
o Faculty were most inspired about adaptive courseware by hearing and learning
from other courseware-using faculty.
101

o Increased program interest was expected due to the pandemic-directing
instructional focus to online instruction and asynchronous learning.
o Interest in and motivation towards adaptive courseware is expected to increase
with the anticipated advances in virtual learning environments.
Summary
Through the lens of adaptive learning and adaptive courseware, all interview participants
and all annual reports expressed a positive, general outlook for this new educational teaching and
learning initiative. Motivation, supported and represented through improved student outcomes,
addressed enrollment objectives and augmented student learning experiences, collectively
progress program implementation towards scale.
Once again, with interviews conducted mid-Covid-19 in April 2020, and learning
technologies near the forefront of educational conversations and discourse for nearly a decade,
the future of education technologies fueled, in part, with the advancing and evolving capabilities
and applications of artificial intelligence, consensus was ubiquitous describing the utility and
standing of adaptive learning and adaptive courseware.
To advance and progress courseware implementation, participants elicited a wide range
of responses, from course-based perspectives to global collaborative initiatives. Interview
transcripts revealed multiple assertions of optimism, confidence and high efficacy for adaptive
courseware. Through the framework of Rogers’ Diffusion of Innovation Theory, program
leadership represents the “innovators” and “early adopters” for introducing and building adaptive
courseware towards scale for a course, through a department or across academic disciplines.

102

Organizational Influence 1
● 12. Organizations need to convey the value for the implementation of adaptive
courseware.
o Participant 1: “So there's definitely high levels of faculty satisfaction for those
who use it (adaptive courseware) well. “What we're looking for in a post COVID
world, which could last two to five years, we are looking for modality flexibility.
(Adaptive courseware) “is a much more ubiquitous tool we have to focus on
multiple modalities.”
o Participant 2: “I definitely think the university administration is behind it sees the
improvement and the gains that we have (from the adaptive courseware
implementation programs), and its potential. And so they want me to continue to
develop that potential and it’s great.”
o Participant 3: “So when you're looking for reasons why we would invest all of
this time and energy in this technology and this pedagogy. It's because we have
very clear goals. And that's an important part of the conversation. When I
mentioned four levels of leadership, all four levels have to be aware of those goals
and supportive of those goals. Make sense? Our president has been very focused
for a very long time on these goals. It's not a technical initiative. It's an academic
initiative.”
o Participant 4: “A really interesting study would to go back to that data (for a) set
of the students who were in those classes that used adaptive courseware, where
from the beginning, consistently used it for a four- or five-year period to see if
they did any better.”
103

o Participant 5: “We have a close relationship with the leadership (of the
courseware vendor) because I think leadership, they're very concerned about that
(school – vendor relationship). They want to know about what's going on with
adaptive learning, they believe in it, they want to hear results, good and bad. So, I
think that's why they try to report everything to them as well. We have a closer
relationship like this (vendor than with) any other vendors. So, we talk with our
vendor every other week. We email them continuously. And they also talk with
their leadership to once a month just talk about how they can support us and they
have a different research group where they talk with our research group and they
do significant research. Certain publications together as well.”
Interviews
With several years of adaptive courseware implementation experience as a guide, two
participants disclosed discovering a revealing and significant best practice that combined and
synergized adaptive courseware use with “active learning.” The first attempt (first semester) with
adaptive courseware, as with most new things, was a learning process for faculty to see how
students were using the adaptive system and, contributorily, how could faculty extend and
compound courseware learning with active learning. Participant 3 shared that semester one was
usually “pretty rough.” The second cycle, which was usually the next semester, was typically a
much smoother technology cycle and by the third semester, professors had developed a high
level of expertise in both the adaptive system and active learning. Participant 3 summarized that
for effective courseware implementation, two years is required to develop the capability with the
faculty. Participant 2 succinctly stated “you got to go three semesters out before you can ever
start talking about what's happened in reviewing progress” (of an adaptive courseware course).
104

Document Analysis
The value of an inclusive adaptive courseware implementation program was evidenced
throughout all school reports revealed this influence to be an asset. The APLU Scaling Grant
provided financial, instructional and technical resources for a three-year period beginning in
2017. Individual universities augmented courseware implementation programs with discrete
resources and support measures. University One received “continuous significant overall and
financial support from the Provost’s Office, along with an institutional board that focused on a
commitment to student success in introductory courses and contributed to a steady upward
retention rate.”
At University Two, the APLU’s grant became part of an institution-wide commitment to
scale the use of adaptive courseware. University Two had in place an institution-wide
commitment to creating a culture that leveraged learning analytics. The alignment of multiple
internal and external data was critical to moving the implementation program forward.
In the first year of program implementation, University Seven launched an Adaptive
Learning Faculty Learning Community (AL-FLC) of faculty champions to spearhead awareness
and engagement building on campus. University Seven promoted the use of faculty learning
communities on Exploration of Flipped Teaching and Learning to provide faculty with time to
learn about adaptive courseware and how it could provide appropriate levels of challenge to
support flipped approach to curriculum design.
University Eight saw “sustainable success with a consistent course coordinator in place
and when instructors were included in the process of continuous change through feedback
sessions, training and course revisions.”
105

Highlighting specific courses, University Seven shared that “faculty champions in
anthropology, biology, business communications and physics shared excitement about adaptive
courseware through presentations and networking with colleagues.” At University Four, faculty
were motivated to find ways to help students better prepare for class and adaptive learning
addressed this pedagogical need. Faculty were provided support from both e-Learning Center
Instructional Designers and adaptive courseware vendors. At University Five, growing faculty
interest in adaptive courseware was spurred by success of a college algebra redesigned course.
Additionally, adaptive courseware appealed to faculty and departments for its potential to aid in
transforming high attrition courses.
Key Points From the Data
o Value of a cooperative and supportive working relationship between program
leadership and faculty was emphasized.
o Faculty support and buy-in were influential to program implementation success.
o High levels of faculty satisfaction were noted for those who knew how to use the
courseware effectively.
o Perceived value of adaptive courseware required a sustained deployment of about
three semesters to see positive program results.
Summary
A commonly repeated remark from interview participants and from annual reports is the
direct correlation and influence of faculty support to an adaptive courseware implementation
program. This disclosed influence, according to Project Leads, centered on the faculty’s
perceived “value” of the adaptive courseware itself. The “value” from the faculty perspective
could be increased “learning support” for students, assistance in “class management” for faculty
106

or the synergistic instructional gains from combining active learning with adaptive learning
courseware. The value or benefit of supportive and cooperative working relationships is not
unique to higher education or an instructional staff. Having great influence throughout the
program planning and implementation, the voice and authority of faculty was clearly pivotal and
pervasive.
Organizational Influence 2
● 13. Organizations need to provide resources for the implementation of adaptive
courseware.
o Participant 1: “How you use the software (adaptive courseware) is definitely
trying to find enough things in your toolbox so that when you do the rest of the
course, you'll have enough choices to personalize it. (One of the) “challenges
becomes how do we teach people (instructors) to pay attention and to integrate
this (adaptive courseware), in course design? How you're going to use that data to
then personalize your instruction, particularly to those who are at risk of not being
successful in the course.”
o Participant 2: “Using adaptive courseware is one of our biggest opportunities for
Affordable Learning Solutions. Okay. And many universities are using the Open
Educational Resources language now and we tried to start looking at as an
institution is Affordable Learning Resources. And I think that this is our big
opportunity because we have a platform with algorithms designed to (help) the
student and apply problem solving to what's being read. Using electronic means,
such as adaptive courseware, we're able to measure how much the student is
107

engaging, we're able to measure where the student is reading and where the
student is focusing.”
o Participant 3: “For example, with the math faculty, we meet at every before every
semester for one day workshop on analyzing the pedagogy and the technology
and making any adjustments. So, it's an all-day faculty meeting. Sometimes there
are 50 faculty members in this meeting. And it really gives them a sense of
solidarity, because they get a chance to share ideas and insights that they wouldn't
normally during their isolated teaching experience.”
o Participant 4: “Our First Year Writing Program team, which consisted of two lead
instructors and an instructional designer who collaborated with researchers,
SMEs, and course designers at Lumen Learning to develop OER modules in
Lumen’s Waymaker software as a supplement to a freshmen writing course. The
collaboration between the university personnel and the team at Lumen initially
took about nine months, with several in-person sessions over the summer months
on our campus. What was great about this approach to developing a course is that
it balanced the needs of our school’s faculty and students with the expertise and
experience Lumen has in digital learning.”
o Participant 5: “Our Vice Provost believed that adaptive learning could really be in
use for large enrollment gateway and undergraduate courses, especially for
courses with high DFW. I think that's the reason that we started with adaptive
learning.” “For us majority of our adaptive learning courses are fully online
classes. We have more online than face to face is because of our funding model.
108

Okay, because we have funded those fully online courses for their course with
design students. They don't have to pay when they use adaptive learning course.”
Interviews
It was evident from all of the interviews that a multitude and confluence of requisite
resources contributed to the development and efficacy of a courseware program implementation.
Planning, best practices, training leadership, administrative support, program buy-in,
collaborative culture, cultivating relationships, technical expertise, innovation and
implementation experience and pedagogical knowledge were some of the resources referenced
throughout discussions.
All participants worked collaboratively with faculty to implement adaptive courseware
within the scaling grant’s guidelines and policies. In-place school and department processes and
protocols for the introduction of new instructional materials were also cooperatively followed .
Each participant spoke of a well-planned, coordinated and deliberate process that, after initial
launch, facilitated and evaluated the implementation and ongoing employment of adaptive
courseware, a collective effort indicating this influence to be an asset.
Specific examples of implementation methods and procedures were cited by participants.
Suggesting a practical method and process to help faculty with implementation, Participant 1
voiced “a lot of faculty who just need, you know, a really good course model to follow to help
them (become a more effective, efficient instructor). It is essentially we want faculty to feel like
they're much more effective at helping their students learn.”
Participant 3 advances two examples of resources, one highlighting the nature of the
implementation resource and another that underscores organizational leadership resources. First,
the implementation “is an academic initiative, not a technical initiative. It is a team sport.”
109

Second, “a lesson learned is that you need four levels of leadership to come together. You need
faculty leadership, you need the department chair, you need the dean and you need a provost.”
Participant 4 emphasized the pivotal development of a relationship between the school
and courseware vendors. To supplement a freshman writing course, the university worked with
the vendor and over nine months, developed courseware that balanced the needs of their school’s
faculty and students with the expertise and experience of the software vendor. The collaborative
process included several in-person meetings and work over the summer but resulted in strong
faculty support and high student satisfaction.
Document Analysis
Resources made available to assist adaptive courseware implementation included
institutional, administrative, financial, technical, pedagogical and human capital support. With
multi-year plans in place for courseware adoption and to advance to scale, program resources
and faculty supports indicated this influence to be an asset.
The adaptive courseware program at University Two partnered with the Learning
Assistant Program and the Catalyst Learning Community to capitalize on the synergy of these
programs. Additionally, at University Two, program implementation received support from the
Institute for Learning and Teaching (TILT) which provided opportunities for faculty to learn
how to incorporate digital learning into pedagogy. Faculty embarking on course redesign
received individualized instructional design support.
University Six’s adaptive courseware program has been integrated with a strong push to
leverage open education resources.” As a growing trend, the availability and advancing quality
of open education resources offers a content source to integrate curriculum into courseware or as
instructional support or extension for an existing courseware program.
110

Key Points From the Data
• Resources made available to assist adaptive courseware implementation included
institutional, administrative, financial, technical, pedagogical and human capital support.
• On-campus programs such as Learning Centers, Teaching Institutes and instructional
design centers vitally support implementation programs.
• Content and curriculum resources such as OER (Open Educational Resources) commonly
support and expand courseware integration.
• A “toolbox of resources” for program implementation was inclusive and multi-
dimensional.
Summary
Synthesizing assertions from interview participants and university reports found differing
approaches to establish, promote and continue adaptive courseware programs. First, several
noted the steep learning curve experienced to bring adaptive courseware to effective
implemented use. Second, the importance was underscored of a collaborative and cooperative
effort that included the copious involvement of the curriculum vendors, Project Leads,
department leads and campus-based instructional designers. The “toolbox of resources” for
effective program implementation was strongly shared as varied and comprehensive.
Assessment of program implementation resources of adaptive learning courseware
elicited differing perspectives. One large university with multiple campuses, each with on-
campus and fully online enrollments, cited “urban geography” as an inherent challenge to
implementing and supporting technology initiatives. Assuredly, circumstances and conditions for
the adoption of adaptive courseware was relative to each university, as each institution is unique
to its community and demography.
111

The planning for, the attention to and the coordination of an implementation team and
resources was shared by all participants. While each execution and facilitation looked different,
the “it takes a village” principle was clearly communicated through numerous interview
comments and report findings.
Summary
This evaluation study focused on the outcomes of adaptive learning courseware
implementation programs in higher education institutions from 2017 to 2020. Each
implementation varied by academic scope, enrollment objective and deployment scale, among
other variables. These implementation programs are ongoing and continue to develop as these
concluding dissertation chapters are being written.
Research and document data was synthesized through the assumed Knowledge,
Motivation and Organization framework. Specifically, interview questions and responses were
reconciled through transcripted excerpts and program reports were gleaned for application and
practice implementation findings and results. Participant feedback and annual university
implementation documents, while decided in detail, voiced and affirmed consistent, recurrent
and concurrent insights, developments and considerations across courseware program
implementations.
Knowledge influences, through study and analysis, addressed four critical areas of
program implementation. Initial focus was on project leadership’s knowledge of selecting
appropriate adaptive courseware, matching the identified needs and objectives of a program.
Second, attention moved to the project leadership’s knowledge of adaptive courseware
implementation for their respective school. Third, inquiry shifted to the significance of project
leadership knowledge of how to collaborate with others in order to implement adaptive
112

courseware. Lastly for knowledge influences, the importance of project leadership to effectively
utilize adaptive courseware to achieve pursued student outcomes was examined.
With respect to motivation as an influence, project leadership needs to be inspired to
implement adaptive courseware and, as a leader, to be able to infuse inspiration with other
program stakeholders. A new learning initiative, such as adaptive learning courseware
implementation, requires the cooperation and enterprise of an engaged program team.
From an organization perspective, institution and department leadership needs to convey
the value for the implementation of adaptive courseware. The benefits to learning and improved
enrollment and student outcomes needs to be uniformly recognized and acknowledged by
program participants. Additionally, resources necessary to assist program implementation
included institutional, administrative, financial, technical, pedagogical and human capital
support.
In summary, research data, assessed through a Knowledge, Motivation and Organization
framework, identified pronounced strengths and decided attributes common through courseware
implementation programs. Recognized strengths included effective courseware implementations
in lower division, high enrollment and in particular, STEM courses. Improved graduation rates,
course completions, final grade distributions and lower withdrawal rates were attributed to
courseware implementations. With an increased focus on distance education, online learning
technologies, such as adaptive courseware, were expected to increase scale across schools,
departments and disciplines. Best practices gleaned from past experience included agreement
that effective implementations took three semesters to be realized, a collaborative and
cooperative stakeholder team was required for program success and support from high level
113

school administration was a key program resource. Chapter Five will present recommendations
based on findings assessed in Chapter Four.

114

Chapter Five: Discussion
The purpose of this project is to evaluate adaptive learning courseware implementation
programs in higher education. While there are multiple stakeholders involved with any
technology implementation program, this study primarily focuses on two stakeholder groups:
Project Leads directing the implementation programs and courseware-using faculty members.
Analysis is performed through a framework of knowledge, motivation, and organization
influences, perspectives, and resources.
The research question that guided this evaluation study was the following:
What Knowledge, Motivation, and Organization influences affect the implementation of
adaptive learning courseware?
Recommendations to Address Knowledge, Motivation, and Organization Influences
The assumed knowledge, motivation, and organization influences in the following tables
have been established through program participant interviews and document analysis. The
purpose of this section is to identify and develop potential strategies and practices to support and
advance program implementation efficacy. Each category begins with a table that summarizes
the knowledge, motivation, or organizational assumed influence, the evidence-based principles
supporting any related recommendations, and a brief statement on the context-specific
recommended solutions.
To develop context and position recommendations to the knowledge, motivation, and
organization influences, a brief and general discussion follows. In an effort to adopt and apply,
preserve and perpetuate the assets, resources, and capital of an implementation program, the
following conceptual diagram visualizes the relationship of continuity, consistency, and a
commitment to continuous improvement, as these three determinants interrelate to support a
115

program implementation initiative. Continuity is defined as “uninterrupted duration or
continuation especially without essential change”. Consistency is defined as “steadfast adherence
to the same principles, course, form, etc.” Commitment to Continuous Improvement is defined as
“an ongoing effort to improve products, services, or processes.”
Figure 4
Assets as Critical Influences

116

Knowledge Recommendations
Knowledge influences were assessed through data collection and examination, which
included interviews and document analysis. For all knowledge influences, evidence-based
principles have been identified to inform context-based recommendations to improve and
maintain performance in those areas. Table 7 outlines the assumed knowledge influence,
principle, and citation, and context-specific recommendation.
Table 7
Summary of Knowledge Influences and Recommendations
Assumed Knowledge
(Factual, Conceptual,
Procedural &
Metacognitive) Influences
Principle and Citation Context-Specific
Recommendation
Project Leads need to know how to select appropriate adaptive courseware.
1. Of the three general types
of adaptive courseware, what
have you observed as having
the best outcomes?

2. Are there subjects that
particularly match well with
adaptive courseware? Are
there subjects that do not
work well with adaptive
courseware?

3. What are the reasons why
universities want to introduce
or use adaptive learning
courseware?
How individuals organize
knowledge influences how
they learn and apply what they
know (Schraw & McCrudden,
2006).

Learning is enhanced as
individuals identify and
understand important points
(Schraw & McCrudden, 2006)

Help individuals connect new
knowledge to prior knowledge
and to construct meaning
(Schraw & McCrudden,
2006).
Provide Project Leads (PLs)
with information and job aids
through an adaptive
courseware implementation
digital manual and guidebook
that would connect, organize,
reinforce, and remind current
courseware-using faculty, and
direct new courseware-using
faculty.

This manual would include
knowledge about adaptive
courseware types, courseware
to course utility, the
components of evaluation, the
capabilities and benefits of
courseware products, best
practices learned, best
practices from other schools’
adaptive programs, and
pedagogical implementation.
117

Project Leads need to know how to select appropriate adaptive courseware.
Information learned
meaningfully and connected
with prior knowledge is stored
more quickly and remembered
more accurately because it is
elaborated with prior learning
(Schraw and McCrudden,
2006).

Facilitating transfer promotes
learning (Mayer, 2011).

Project Leads need to know how to implement adaptive courseware at their school.
4. Describe the current state
of adaptive courseware
implementation at your
school?

5. From your observations,
how is the adaptive
courseware being utilized?

6. What are the keys to a
successful adaptive
courseware implementation
program?

7. How is continuous
improvement for courseware
implementation applied or
exercised?
How individuals organize
knowledge influences how
they learn and apply what
they know (Schraw &
McCrudden, 2006).

Help individuals connect new
knowledge to prior knowledge
and to construct meaning
(Schraw & McCrudden,
2006).

Provide collaborative
opportunities for PLs and
faculty to evaluate program
implementation.

Provide PLs with
opportunities to collect
longitudinal, year-over-year
data, analyze the data, and
evaluate program and student
outcomes through the data.

Provide opportunities for PLs
to share program data
analytics with faculty,
administration, and program
support staff.

Provide PLs with a job aid, in
the form of a concept map
within the manual and
guidebook, that organizes and
shows the relationships and
connections between PLs and
faculty, faculty and
implementation efficacy and
program effectiveness, and
student achievement.

118

Project Leads need to know how to implement adaptive courseware at their school.
Provide a courseware
implementation plan that first,
prioritizes the maintenance of
the program assets, resources,
and capital and second,
commit to the principles and
practices of continuity,
consistency, and commitment
to continuous improvement to
boost program viability and
performance.
Project Leads need to know how to collaborate with others in order to implement adaptive
courseware.
8. Please talk about the
significance of a coordinated
team for courseware
implementation.

9. What has been the faculty
experience to date with the
implementation of adaptive
learning courseware?
To develop mastery, training
must show teachers how to
utilize the strategies to
achieve the goal (Clark
and Estes, 2008).

To develop mastery, teachers
must be given the opportunity
to repetitively practice the
strategies they have been
taught. The more they
practice, the more they will
know (Meyer, 2011).

Mastery is developed when
individuals acquire component
skills, practice utilizing them,
and know when to apply what
they have learned (Schraw &
McCrudden, 2006).

Provide training for faculty on
how to integrate the
courseware into their practice,
not just how to use the
courseware.

Provide training for new and
continuing faculty on
navigating and applying the
manual to integrate the
courseware into their practice.

Provide training to select
courseware that will best
support and facilitate positive
learning outcomes.

Provide PLs with
opportunities to collect
longitudinal, year-over-year
data, analyze the data and
evaluate the program and
student outcomes through the
data.

Provide collaborative
opportunities for PLs and
faculty to evaluate program
implementation.
119

Project Leads need to know how to utilize adaptive courseware to achieve student outcomes.
10. What
indicators/outcomes does
your institution use to
measure student success?
Help individuals connect new
knowledge to prior knowledge
and to construct meaning
(Schraw & McCrudden,
2006).

Learning is enhanced as
individuals identify and
understand important points
(Schraw & McCrudden, 2006)
Provide PLs with
opportunities to collect
longitudinal, year-over-year
data, analyze the data and
evaluate the program and
student outcomes through the
data.

Project Leads need to know how to utilize adaptive courseware to achieve student outcomes.
The use of metacognitive
strategies facilitate learning
(Baker, 2006).

Learning and motivation are
enhanced when learners set
goals, monitor their
performance, and evaluate
their progress towards
achieving their goals
(Ambrose et al, 2012; Mayer,
2011).

Provide collaborative
opportunities for PLs and
faculty to evaluate program
implementation.

Factual Knowledge Solutions
The overall recommendation is to provide documented and accessible reference and
guidance for the continued effective implementation of adaptive learning courseware. The
findings and results presented factual knowledge influences to be assets, resources, and capital.
To maintain the factual knowledge assets, resources, and capital, Information Processing Theory
can be used to make informed recommendations. Schraw and McCrudden (2006) state that
information learned meaningfully and connected with prior knowledge is stored more quickly
and remembered more accurately because it is elaborated with prior learning. Moreover, Schraw
120

and McCrudden (2006) assert that learning is enhanced as individuals identify and understand
important points. This would suggest that Project Leads, courseware-using faculty, and new
faculty would benefit from a knowledge source that would connect application and
implementation. Thus, it is recommended that an adaptive courseware implementation manual
and guidebook be developed that would maintain and protect the factual knowledge assets,
resources, and capital while minimizing implementation variability and promoting program
viability and performance.
Establishing a base of implementation knowledge can benefit program participants in
multiple ways. According to Schraw and McCrudden (2006), in order to develop mastery,
individuals must acquire component skills, practice integrating them, and know when to apply
what they have learned. The proposed manual and guidebook would help individuals organize
knowledge influences and apply what they know (Schraw & McCrudden, 2006). In a study that
developed a guidebook as a resource to organize pertinent information for technology
coordinators, Frazier (2003) concluded that a “technology leader's guidebook is a useful resource
for implementing and supporting educational technology.” Additionally, Frazier asserted that a
guidebook can be a resource to successfully impact technology implementation and use.
Therefore, the proposed and recommended manual and guidebook would efficiently organize
critical courseware implementation information, act as a reference source for current faculty, and
function as an instructional guide for new faculty.
Conceptual Knowledge Solutions
The recommendation for the conceptual knowledge influences is to provide PLs with a
job aid, in the form of a concept map as part of the manual and guidebook, that organizes and
depicts the relationships and connections between program stakeholders, implementation
121

components, and learned courseware outcomes. The findings and results presented conceptual
knowledge influences to be assets, resources, and capital. To maintain the conceptual knowledge
assets, resources, and capital, Information Processing Theory can be used to make informed
recommendations. Schraw and McCrudden (2008) position that individuals organize knowledge
influences by how they learn and how to apply what they know. Furthermore, Anderson and
Krathwohl (2001) define conceptual knowledge as a way experts think about a discipline. This
would suggest that implementation stakeholders would benefit from a conceptual working
framework that values, understands, and perpetuates these key influences. Therefore, a concept
map or visual representation would effectively connect program participants, implementation
efficacy, course components, and student outcomes.
Research shows that instructional supports can assist in the process of learning. Schraw
and McCrudden (2006) submit that individuals connect new knowledge to prior knowledge to
construct meaning. Mayer (2011) shares that the incorporation of visual learning, as in the
proposed concept map, will increase one’s memory capacity. Schraw et al. (2009) state that the
development of schemata assists learners in the organization of knowledge in a specific area of
study. Overall, visual representation, through a concept map job aid, prominently distinguishes
and characterizes the array of associations, alliances, and influences between program
stakeholders and program components,
Procedural Knowledge Solutions
The recommendation is to provide training for faculty on how to integrate the courseware
into their practice, not just how to use the courseware. The findings and results presented the
procedural knowledge influences to be assets, resources, and capital. To maintain the procedural
knowledge assets, resources, and capital, Information Processing Theory can be used to make
122

informed recommendations. Clark and Estes (2008) assert that to develop mastery, training must
show teachers how to utilize the strategies to achieve the goal. Moreover, Meyer (2011) affirms
that to develop mastery, teachers must be given the opportunity to repetitively practice the
strategies they have been taught. The more they practice, the more they will know. This would
suggest that the factual knowledge solution of developing an implementation manual and
guidebook would be a source of in-service training for faculty and their practice. Therefore, it is
recommended that to support adaptive courseware implementation efficacy, a manual and
guidebook be created and curated to operationally formalize program routines and procedures.
Training, which includes demonstration, practice, and feedback, as a procedural
knowledge support, would promote learning, guidance, and practice as effective ways to
integrate pedagogy and technology. Mayer (2011) shares that, with a focus on technology
integration, important learning occurs when learners engage in three processes: selecting,
organizing, and integrating. The proposed manual and guidebook, as a training job aid, can be a
useful resource for implementing and supporting educational technology, and that a guidebook
can be a resource for those serving as technology coordinators (Frazier, 2003). Schraw and
McCrudden, (2006) add that mastery is developed when individuals acquire component skills,
practice utilizing them, and know when to apply what they have learned. Accordingly, the
attainment of knowledge and skills, practice, and application become requisite influences to
successful technology implementation.
Metacognitive Knowledge Solutions
The recommendation is to provide PLs with opportunities to collect longitudinal, year-
over-year data, analyze the data, and evaluate program and student outcomes through the data.
The findings and results presented the metacognitive knowledge influences to be assets.
123

resources and capital. To maintain the metacognitive assets, resources, and capital, Information
Processing Theory can be used to make informed recommendations. Baker (2006) found that the
use of metacognitive strategies facilitated learning. Moreover, learning and motivation are
enchanted when learners set goals, monitor performance, and evaluate progress towards the goals
(Ambrose et al., 2012; Meyer, 2011). This would suggest that both participant and program
efficacy would benefit from metacognitive inquiry and critique. Therefore, it is recommended
that PLs and faculty collaboratively evaluate and appraise adaptive courseware implementation
from a broader program perspective as well as from a narrower section and course viewpoint.
Metacognitive thinking and self-regulation are cognitive processes that learners use to
monitor, control, and regulate thinking and learning (Pintrich, 2002). Calegari et al. (2015) state
that the use of structured reflective discussions to facilitate the improvement of faculty members’
knowledge and skills. Furthermore, Dowd et al. (2012) encourage organizations that are
establishing or improving the use of outcome-based decision making to engage in discussions
about the process of analysis. Dowd et al. (2012) explain that the discussion of outcomes is
important to determine if conclusions based on outcomes are justifiable. In sum, the regulation of
thinking and learning as well as the reflective intent, share the metacognitive facets of
understanding, realization, and insight to better understand and therefore more easily improve
program implementation.
Motivation Recommendations
According to the conceptual framework utilized for this study by Clark and Estes (2008),
motivation influences include value, self-efficacy, mood, and expectancy. Motivation influences
were assessed through data collection and examination, which included interviews and document
124

analysis. Table 8 outlines the assumed motivation influence, principle and citation, and the
context-specific recommendation.
Table 8
Summary of Motivation Influences and Recommendations
Assumed Motivation
Influence
Principle and Citation Context-Specific
Recommendation
11. PLs need to be motivated
to implement adaptive
courseware.

Rationales that include a
discussion of the importance
and utility value of the work
or learning can help develop
positive values (Eddles, 2006;
Pintrich, 2003).

Learning and motivation are
enhanced if the learner values
the task (Eccles, 2006).

Self-efficacy, or the belief in
their capacity, is a top
factor in a person’s
commitment to the
organization’s goal (Clark &
Estes, 2008; Pajares, 2006).

Modeling to-be-learned
strategies or behaviors
improves self-efficacy,
learning, and performance
(Denler, Wolters, &
Benzon, 2009).

Learning and motivation are
enhanced when learners have
positive expectancies for
success (Pajares, 2006).

Positive emotional
environments support
motivation (Clark & Estes,
2008).
Provide positive reminders
for faculty of why their work
supports course efficacy,
program integrity, and
implementation goals.

Provide opportunities for PLs
to share implementation
successes with colleagues,
faculty, and school leaders.

Provide recognition for
faculty champion leaders of
courseware implementation.

Provide opportunities for
faculty champions to help
fellow colleagues with
courseware implementation.

Provide collaborative
opportunities for PLs and
faculty to monitor and
evaluate program assets,
resources, and capital, and
performance.

Provide stakeholders with
clear adaptive courseware
program implementation
expectations.

125

Assumed Motivation
Influence
Principle and Citation Context-Specific
Recommendation
A positive emotional
environment is
identified as one of the
critical conditions for
increasing motivation
(Bruning & Horn, 2010).

Learning and motivation are
enhanced when learners have
positive expectancies for
success (Pajares, 2006).

Learning and motivation are
enhanced when learners set
goals, monitor their
performance, and evaluate
their progress towards
achieving their goals
(Ambrose et al, 2012; Mayer,
2011).

Higher expectations for
success and perceptions of
confidence can positively
influence learning and
motivation (Eccles, 2006)

Value Motivation Solutions
The recommendation is to provide collaborative opportunities for PLs and faculty to
monitor and evaluate program assets, resources, and capital, and performance. The findings and
results presented the value motivation influences to be assets, resources, and capital. To maintain
the motivation assets, resources, and capital, Expectancy Value Theory can be employed to make
informed recommendations. Eccles (2006) suggests that learning and motivation are enhanced if
the learner values the task. Moreover, a discussion of the importance and utility value of the
work or learning can help develop positive values (Eddles, 2006; Pintrich, 2003). This would
126

suggest that program implementation support and buy-in from faculty is essential, influential,
and highly valued. Therefore, to enhance participant value as it relates to technology integration,
it is recommended that PLs and faculty work cooperatively to identify and recognize areas of
program success including the affirmation of positive student outcomes.
Research supports group collaboration, a constructivist learning approach, and openness
to risk-taking as a means to support faculty value for technology integration (Baylor & Ritchie,
2002; Ertmer, 2005). Additionally, positive values coupled with open discussions can assist in
the utility value of learning (Eccles, 2006; Pintrich, 2002). Finally, Eccles (2006) stated that
perceived value includes four constructs: intrinsic value, attainment value, utility value, and the
cost value. In summary, these four perspectives on value influence program efficacy as seen
from implementation results and program outcomes.
Self-Efficacy Motivation Solutions
The recommendation is to provide opportunities for PLs to share implementation
successes with colleagues, faculty, and school leaders. The findings and results presented the
self-efficacy motivation influences to be assets, resources, and capital. To maintain the
motivation assets, resources, and capital, Self-Efficacy Theory can be employed to make
informed recommendations. Bandura (2006) describes self-efficacy as a perceived capability in
achieving a task or outcome. Moreover, as a subset of Social Cognitive Theory, Self-Efficacy
Theory conditions making it clear that individuals are capable of learning what is being taught or
are capable of performing a task (Pajares, 2006). This would suggest that that faculty would
benefit from increased and applicative confidence and positivity in using adaptive courseware.
Therefore, it is recommended that program leaders provide opportunities for faculty champions
127

to share their implementation successes and advise fellow colleagues, building faculty
confidence, and raising program sentiment.
To be effective, self-efficacy, or the belief in their capacity, is a top factor in a person’s
commitment to the organization’s goal (Clark & Estes, 2008; Pajares, 2006). Bandura (2000)
suggested that for a team to function effectively, is it critical for the individuals to have high
efficacy. Modeling (such as examples shared by colleagues) to-be-learned strategies or behaviors
improves self-efficacy, learning, and performance (Denler, Wolters, & Benzon, 2009).
Moreover, Pajares (2006) states that learning and motivation are enhanced when learners have
positive expectancies for success. As a result, colleagues exchanging and engaging in the real-
world application of learning technologies as well as increasing confidence through the
connection of shared program goals and objectives, highlight valuable ways of promoting self-
efficacy.
Mood Motivation Solutions
The recommendation is to provide positive reminders for faculty of why their work
supports course efficacy, program integrity, and implementation goals. The findings and results
presented the mood motivation influences to be assets, resources, and capital. To maintain the
motivation assets, resources, and capital, Self-Efficacy Theory can be employed to make
informed recommendations. Clark & Estes (2008) state that positive emotional environments
support motivation. Moreover, learning and motivation are enhanced when learners have positive
expectancies for success (Pajaraes, 2006). This would suggest that creating and maintaining a
positive attitude towards courseware implementation would benefit program outcomes.
Therefore, it is recommended that activating and reminding faculty of program and courseware
utility and learning advantages will increase positivity for technology integration.
128

Positive faculty mood influences long-lasting and inspired instructional technology use.
Bruning and Horn (2010) assert a positive emotional environment is identified as one of the
critical conditions for increasing motivation. According to Groff & Mouza (2008), teachers’
knowledge and skill level play a significant role in technology integration. Reynolds (2001)
discussed the importance of one’s own emotions in a workplace environment with levels of
positivity fueling higher expectations for success and favorably impacting learning and
motivation. The recommendation is to acknowledge and champion faculty courseware
implementation leaders and facilitate peer support and communication between faculty
courseware users.
Expectancy Motivation Solutions
The recommendation is to provide stakeholders with clear adaptive courseware program
implementation expectations. The findings and results presented the expectancy motivation
influences to be assets, resources, and capital. To maintain the motivation assets, resources, and
capital, Expectancy Value Theory can be employed to make informed recommendations.
Pajaraes (2006) states that learning and motivation are enhanced when learners have positive
expectancies for success. Moreover, as expectancy-value believes that a given behavior will or
will not lead to a given outcome, a courseware implementation program can benefit from shared
successful examples from early adopting faculty and completed course sections. This would
suggest that faculty would benefit from focusing on instructional improvement and positive
outcomes of courseware implementations. Therefore, it is recommended that faculty review
successful examples of courseware adoption during the initial phases of implementation,
examine supplementary best practices shared during the academic year, and evaluate additional
program exemplars during the post-course summative assessment.
129

Learning and motivation are supported when values, enthusiasm, and interest in a task or
initiative are framed and founded. Anderman and Anderman (2010) emphasized that learning
and motivation are enhanced when individuals attribute success or failures to effort rather than
ability. Eccles (2006) stated that higher expectations for success and perceptions of confidence
can positively influence learning and motivation. Eccles (2000) continued that expectancy and
value are affected by perceived difficulty and individual goals. The recommendation is to
activate personal interest by emphasizing the program’s bigger-picture benefits that highlight
expectations for success and the perceived usefulness of the educational technology.
Organization Recommendations
According to the conceptual framework utilized for this study by Clark and Estes (2008),
organization influences include cultural model, cultural settings, policies, processes and
procedures, and resources. Organization influences were assessed through data collection and
examination, which included interviews and document analysis. Table 9 outlines the assumed
organization influence, the principle and citation, and the context-specific recommendation.
Table 9
Summary of Organization Influences and Recommendations
Assumed Organization
Influence
Principle and Citation Context-Specific
Recommendation
12. Organizations need to
convey the value for the
implementation of adaptive
courseware.
Effective organizations ensure
that organizational messages,
rewards, policies, and
procedures that govern the
work of the organization are
aligned with or are supportive
of organizational goals and
values (Clark & Estes, 2008).
Provide positive reminders
for faculty of why their work
supports course efficacy,
program integrity, and
implementation goals.

130

Assumed Organization
Influence
Principle and Citation Context-Specific
Recommendation
Performances improve when
organizational goals, policies,
and procedures are aligned
with the organizational
culture (Clark & Estes, 2008).

Provide a courseware
implementation plan that first,
prioritizes the maintenance of
program assets, resources,
and capital and second,
commit to the principles and
practices of continuity,
consistency, and commitment
to continuous improvement to
boost program viability and
performance.

Provide collaborative
opportunities for PLs and
faculty to monitor and
evaluate program assets,
resources, capital, and
performance. Faculty buy-in
is key to the success of an
adaptive courseware
implementation program.

Provide training for faculty on
how to integrate the adaptive
courseware into their practice,
not just how to use the
courseware.
13. Organizations need to
provide resources for the
implementation of adaptive
courseware.

Effective change efforts
ensure that everyone has the
resources (equipment,
personnel, time, etc.) needed
to do their job, and that if
there are resource shortages,
then resources are aligned
with organizational priorities
(Clark & Estes, 2008).
Provide faculty with an
implementation manual and
guidebook that would
connect, organize, reinforce,
and remind current
courseware-using faculty, and
direct new courseware-using
faculty.

Provide faculty with stipends
to adopt and integrate
adaptive courseware in their
practice.

131

Assumed Organization
Influence
Principle and Citation Context-Specific
Recommendation
Provide stewardship to
support faculty on integrating
policies, processes, and
procedures from the manual
and guidebook into their
practice.

Cultural Model Organization Solutions
The recommendation is to provide a courseware implementation plan that first, prioritizes
the maintenance of the program assets, resources, and capital, and second, commits to the
principles and practices of continuity, consistency, and commitment to continuous improvement
to boost program viability and performance. The findings and results presented the cultural
model organization influences to be assets, resources, and capital. To maintain the organization’s
assets, resources, and capital, organizational change principles can be applied to make informed
recommendations. Clark and Estes (2008) suggest that effective organizations ensure that
organizational messages, rewards, policies, and procedures that govern the work of the
organization are aligned with or are supportive of organizational goals and values. Moreover, a
cultural model is bolstered when effective change efforts use evidence-based solutions, such as
best practices or implementation exemplars, to reinforce and extend an organization’s culture
(Clark & Estes, 2008). This would suggest that courseware-using faculty, guided by the precepts
of continuity, consistency, and commitment to continuous improvement, would benefit from
learning about successful courseware implementations from fellow faculty, related sections, or
other programs. Therefore, it is recommended that efforts to affect change in an organization
model, while maintaining and protecting critical assets, resources, and capital, should focus first
132

on identifying key organizational elements warranting change, and second, on the need to
articulate how evidence-based change can be adapted for the organization.
Research supports the significance of regularly monitoring organizational cultural
influences and making adjustments as necessary. According to Sugai and Horner (2002),
collective working structures and behaviors that encourage the adoption of research-based
practices have become a central focus in impacting positive participant behavioral outcomes.
Positive faculty interactions must occur on an individual as well as, on a larger group basis
(Sugai & Horner, 2002). Clark and Estes (2008) state that performances improve when
organizational goals, policies, and procedures are aligned with the organizational culture and
note that effective change efforts be communicated regularly and frequently to all key
stakeholders. In summary, research shows that cultural model influences must be monitored and
assessed and practices adapted as necessary to maintain a healthy organizational culture.
Cultural Setting Organization Solutions
The recommendation is to provide collaborative opportunities for PLs and faculty to
monitor and evaluate program assets, resources, and capital, and performance. Faculty buy-in is
key to the success of an adaptive courseware implementation program. The findings and results
presented the cultural settings organization influences to be assets, resources, and capital. To
maintain the organization’s assets, resources, and capital, organizational change principles can be
applied to make informed recommendations. Clark and Estes (2008) suggest that performances
improve when organizational goals, policies, and procedures are aligned with the organizational
culture. Moreover, Banks and Mayes (2001) assert that instructors are more effective in work
environments characterized by mutual dependence where sharing is the norm. This would
suggest that learning is enhanced when implementation efficacy, as part of a collaborative
133

cultural setting, improves when not only factual and procedural knowledge influences are
integrated, but when motivational influences such as value, expectancy, and mood are
emphasized and assimilated. Therefore, to further embrace an inclusive and supportive
organizational setting, it is recommended that implementation programs provide training for
faculty on how to best integrate the adaptive courseware into their practice, not just how to use
the courseware.
Research supports the importance of regularly monitoring organizational cultural
influences and making adjustments as necessary. As stated by Rueda (2011), cultural settings are
the visible characteristics of the daily workings in an organization, while the cultural models are
often invisible and a shared mental representation of the organization’s structures, values,
practices, and policies. Clark and Estes (2008) state that effective change efforts use evidence-
based solutions and adapt them, where necessary, to the organization's culture. Additionally,
effective organizations ensure that organizational messages, rewards, policies, and procedures
that oversee the organization also align with or are supportive of organizational values and goals
(Clark & Estes, 2008). According to Kraft and Papay (2014), teachers working in more
supportive professional environments improve their effectiveness more over time than teachers
working in less supportive contexts. In summary, research shows that cultural setting influences
must be monitored and assessed and practices adapted as necessary to maintain a healthy
organizational culture.
Policies, Processes, & Procedures Organization Solutions
The recommendation is to provide stewardship to support faculty on integrating policies,
processes, and procedures from an adaptive courseware implementation manual and guidebook
into their instructional practice. The findings and results presented the policies, processes, and
134

procedures organization influences to be assets, resources, and capital. To maintain the
organization’s assets, resources, and capital, organizational change principles can be applied to
make informed recommendations. Clark and Estes (2008) state that effective organizations
ensure that organizational messages, rewards, policies, and procedures that govern the work of
the organization are aligned with or are supportive of organizational goals and values. Moreover,
performances improve when organizational goals, policies, and procedures are aligned with the
organizational culture (Clark & Estes, 2008). This would suggest that program performance and
implementation efficacy would strongly benefit from a job aid as represented by a proposed
comprehensive source of program reference. The recommendation, with a focus on maintaining
and protecting the courseware implementation program assets, resources, and capital, is to create
an adaptive courseware implementation manual and guidebook that would organize, reinforce,
and remind current courseware-using faculty, and train new courseware-using faculty through
the application of acknowledged best practices.
As a digital reference source, for the purposes of knowledge management, knowledge
guidance, and knowledge transfer, the implementation manual and guidebook formalizes
processes and procedures as organizational influences. Frazier (2003) asserts that a technology
leader's guidebook is a useful resource for implementing and supporting educational technology.
Frazier adds that a guidebook can be a resource for those serving as technology coordinators and
that knowledgeable technology coordinators can successfully impact technology implementation
and use. Fratura and Capper (2007) assert that policies should maintain the goals indicated by an
organization. Thoonen et al. (2012) affirm that the successful implementation of transformative
practices within an organization is based on strong central leadership, teacher buy-in, resources,
time, and support. In sum, the aggregate and collated base of knowledge that the implementation
135

manual and guidebook represents, leads the organizational practice, collates organizational
assets, resources, and capital, and highlights organizational direction.
Resources Organization Solutions
The recommendation is to provide faculty with an implementation manual and guidebook
that would connect, organize, reinforce, and remind current courseware-using faculty, and direct
new courseware-using faculty. The findings and results presented the resources organization
influences to be assets, resources, and capital. To maintain the organization’s assets, resources,
and capital, organizational change principles can be applied to make informed recommendations.
Clark and Estes (2008) condition that effective change efforts ensure that everyone has the
resources (equipment, personnel, time, etc.) needed to do their job. Moreover, a robust adaptive
courseware implementation program should be founded and sustained by organizational, factual,
procedural, and value resources to ensure the continuing operational viability of the technology
initiative. This would suggest that program achievement and implementation efficacy would
strongly benefit from established and validated resources, developed and curated by program
leads and faculty contributors. Therefore, it is recommended that the proposed implementation
manual and guidebook, the compilation of knowledge, motivation, and organization program
resources, may become the most important instructional asset, specifically for courseware-using
faculty, and in general, for program efficacy, management and performance.
Research supports guidance, training, and information as important resources to facilitate
technology integration through a new initiative. Clark and Estes (2008) position that effective
change efforts use evidenced-based solutions and adapt them, where necessary. Resources must
be organized efficiently to encourage high-quality instruction (Sugai & Horner, 2002). Frazier
(2003) asserts that a technology leader's guidebook is a useful resource for implementing and
136

supporting educational technology. Frazier (2003) adds that a guidebook can be a resource for
those serving as technology coordinators and that knowledgeable technology coordinators can
successfully impact technology implementation and use. A manual and guidebook convey
successful implementations as solutions and shares established practices and applications to limit
program variability and promote instructional consistency.
Summary of Knowledge, Motivation, and Organization Recommendations
The following recommendations were guided by evidence-based principles and address
influences for each knowledge category. The recommendation for factual knowledge is for an
adaptive courseware implementation manual and guidebook to be developed that would maintain
and protect the factual knowledge assets, resources, and capital while minimizing
implementation variability and promoting program viability and performance. The
recommendation for conceptual knowledge is to develop a concept map or visual representation
that would effectively connect program participants, implementation efficacy, course
components, and student outcomes. To support adaptive courseware implementation efficacy,
the recommendation for procedural knowledge is for a manual and guidebook to be created and
curated that would operationally formalize program routines and procedures. The
recommendation for metacognitive knowledge is for Project Leads and faculty to collaboratively
evaluate and appraise adaptive courseware implementation from a broader program perspective
as well as from a narrower section and course viewpoint.
The following recommendations were guided by evidence-based principles and address
the assets, resources, and capital for each motivation category. To enhance participant value as it
relates to technology integration, the recommendation for motivation value is for Project Leads
and faculty to work cooperatively to identify and recognize areas of program success including
137

the affirmation of positive student outcomes. The recommendation for motivation self-efficacy is
for program leaders to provide opportunities for faculty champions to share their implementation
successes and advise fellow colleagues, therefore building faculty confidence and raising
program sentiment. The recommendation for motivation mood is to activate and remind faculty
of program and courseware utility and learning advantages, thereby increasing positivity for
technology integration. The recommendation for motivation expectancy is for faculty to review
successful examples of courseware adoption during the initial phases of implementation,
examine supplementary best practices shared during the academic year and evaluate additional
program exemplars during the post-course summative assessment.
The following recommendations were guided by evidence-based principles and address
the assets, resources, and capital for each organization category. The recommendation for the
organization’s cultural model is that efforts to affect change in an organization model, while
maintaining and protecting critical assets, resources, and capital, should focus first on identifying
key organizational elements warranting change, and second, on the need to articulate how
evidence-based change can be adapted for the organization. To further embrace an inclusive and
supportive operational setting, the recommendation for the organization’s cultural setting is for
implementation programs to provide training for faculty on how to best integrate the adaptive
courseware into their practice, not just how to use the courseware. With a focus on maintaining
and protecting the courseware implementation program assets, resources, and capital, the
recommendation for organization policies, processes and procedures are to create an adaptive
courseware implementation manual and guidebook that would organize, reinforce, and remind
current courseware-using faculty, and train new courseware-using faculty through the application
of acknowledged best practices. The recommendation for organization resources is a proposed
138

implementation manual and guidebook, representing a compilation of knowledge, motivation,
and organization program resources, employed specifically for courseware-using faculty, and in
general, for program efficacy, management and performance.
Integrated Implementation and Evaluation Plan
Organizational Purpose, Need, and Expectations
Common to large, urban universities is a mission statement that aspires to promote a
culture of inquiry, extend knowledge, and provide a quality and progressive educational
experience. In a continuing effort to advance learning, many institutions in higher education
welcome new methods of instruction facilitated through innovative educational technologies,
with a goal of supporting academic achievement and improving student outcomes. Adaptive
learning, facilitated through adaptive learning courseware, is a promising learning technology
that seeks to meet the needs of an increasingly diverse and disparate student population.
As an evaluative study, current and ongoing adaptive learning courseware
implementations were researched and appraised. Through first-hand accounts and published
reports, implementation programs were viewed from planning stages, course accounts, and
learning outcomes. From higher education schools across the country, program implementations
have shown marked similarities as well as distinct differences, often accentuating experienced
best practices and effective integration strategies.
While multiple stakeholder groups play a significant role in a successful adaptive
learning courseware implementation, two stakeholder groups have been purposefully highlighted
for their conspicuous and pivotal roles in program efficacy and capacity: project implementation
leads and courseware-using faculty members. The Project Leads guide courseware
implementation and integration while faculty users implement adaptive courseware into their
139

courses and into their instruction. Proposed recommendations focus on maintaining and
disseminating identified knowledge, motivation, and organization assets, resources, and capital.
Implementation and Evaluation Framework
The implementation and evaluation plan is framed by the New World Kirkpatrick
Model (Kirkpatrick & Kirkpatrick, 2016), which is based on the Kirkpatrick Four-Level Model
of Evaluation. The Kirkpatrick Model is designed to guide and monitor the impact training has
on an organization. One of the foundational principles of the Kirkpatrick Model is that “the end
is the beginning” (Kirkpatrick & Kirkpatrick, 2016), similar in theory to current educational
practices of “design thinking” or “understanding by design” that begin with the end, which refers
to the fact that any effective training needs to be built around the end-goal and desired results of
the organization and then worked backward towards participants’ behavior, learning and
reactions. Thus, the reimagined New World Kirkpatrick Model relies on the Four Levels,
beginning with Level 4 and ending with Level 1.
The Four Levels of the Kirkpatrick Model are as follows: Level 4 - Results measures the
extent to which intended outcomes can be attributed to learning from a training program. Level 3
- Behavior measures how critical behaviors from training modules support participants in
applying their learning to their job. Level 2 - Learning teaches the knowledge and skills required
for participants to transfer skills from training to on-the-job application. Level 1 - Reaction
measures how participants associate with the training. In other words, training facilitators must
consider if the learning experience is engaging and enjoyable while meeting participants’ needs.
(Kirkpatrick & Kirkpatrick, 2016).
Differences between the Kirkpatrick Model and the New World Kirkpatrick Model begin
with the inclusion of processes that promote or hinder the application of learned knowledge or
140

skills, which is found in Level 3. Additionally, the New Kirkpatrick Model adds elements of
learners' confidence, commitment, and engagement to Level 2 and Level 1. Collectively, the
binary Kirkpatrick Models bridge immediate solutions with anticipated larger outcomes to
support program efficacy and effectiveness.
Level 4: Results and Leading Indicators
In the New World Kirkpatrick Model (2016), Level 4 - Results are the degree to which
targeted outcomes occur as a result of the training and the support and accountability package.
Connecting training to performance to results is essential for an organization and knowing the
true Level 4 can be recognized by leading indicators. Kirkpatrick and Kirkpatrick (2016) define
leading indicators as short-term observations and measurements that provide a metric to
determine whether or not critical behaviors will positively influence the organization's desired
results. Common leading indicator examples include efficiency, quality, employee (faculty)
satisfaction, customer (faculty and student) ratings, and customer (faculty and student) attrition
(Kirkpatrick & Kirkpatrick, 2016). Internal indicators are defined as an individual, team,
departmental, and organizational outcomes. External indicators are defined as customer, client,
market, and industry response. The proposed leading external and internal indicators, metrics,
and methods are shown in Table 10.

141

Table 10
Outcomes, Metrics, and Methods for External and Internal Outcomes
Outcome Metric(s) Method(s)
External Outcomes
Increased publicity and
exposure for a school’s
adaptive learning
programs through various
forms of marketing.
Number of references through
different media (i.e. mentions
from local newspapers,
inquiries from external
students, other schools and
interested parties, recognition
from higher education
organizations and associations,
invitations to share professional
expertise with other industry
and institutions, etc.)
School and program leadership
will actively seek and develop
marketing opportunities (in
conjunction with school-site
media departments) to
communicate adaptive learning
programs advances and
successes to elevate school and
program recognition.
The increased perception
that the university is an
organization of
continuous learning and
innovation.
Number of references
highlighting a school’s
progressive and dynamic
learning culture. (i.e.
conference speaking invitations,
communication from other
universities, references in
periodicals, interest in research
partnerships, etc)
School and program leadership
will actively seek and develop
marketing opportunities (in
conjunction with school-site
media departments) and seek
external funding sources to
build external recognition of
the school’s innovative and
progressive learning programs
and opportunities.

Increased communication
and knowledge exchange
with other universities
implementing adaptive
learning courseware.
Number of new and continuing
relationships and knowledge
sharing alliances with other
university programs.
Program leadership will
actively pursue opportunities to
share implementation
experiences and best practices
(i.e. speaking at conferences,
authoring of papers and reports,
initiating research projects,
etc.) with interested higher
education parties.

142

Outcome Metric(s) Method(s)
Internal Outcomes
The increased scale of
adaptive courseware use
throughout general
education courses.
The number of adaptive
courseware implementations
among general education
courses.
School and program leadership
will actively engage
departments with general
education classes to
recommend courseware
adoption.
Increased course
completions.
The number of students
completing classes employing
adaptive courseware.
Collaborative efforts between
program leadership, new and
continuing faculty users,
curriculum developers, and
instructional support staff to
improve courseware use
efficacy.
Increased student
performance outcomes.
Number of students earning
higher grades in classes
employing adaptive
courseware.
Collaborative efforts between
program leadership, new and
continuing faculty users,
curriculum developers, and
instructional support staff to
improve student use and
engagement with courseware.

Level 3: Behavior
Level 3 - Behavior of the New World Kirkpatrick Model refers to “the degree to which
participants apply what they learned during training when they are back on the job” (Kirkpatrick
& Kirkpatrick, 2016, p. 13). Level 3 is a comprehensive, continuous performance monitoring and
improvement system.
Critical Behaviors
Level 3 is considered to be the most important level because training alone may not yield
the expected or successful results. Level 3 appropriately and procedurally bridges the gap
between Level 4 results and Level 2 learning. Critical behaviors are "the few, key behaviors that
143

the primary group will have to consistently perform on the job to bring about targeted outcomes"
(Kirkpatrick & Kirkpatrick, 2016, p. 14).
Kirkpatrick and Kirkpatrick (2016) state that critical behaviors need to be specific,
observable, and measurable and work behaviors become critical if they are performed reliably. In
this study, the behavior of the key stakeholder groups of Project Leads and faculty will be
viewed through an objective lens of preservation and perpetuation of adaptive courseware
implementation program assets, resources, and capital. The critical behaviors, specific metrics,
methods, and timing for evaluating outcome behaviors are shown in Table 11.
Table 11
Critical Behaviors, Metrics, Methods, and Timing for Evaluation
Critical Behavior Metric(s) Method(s) Timing
1 - With extensive
input and influence
from experienced
adaptive learning
courseware-using
faculty and adding key
best practices gleaned
from other programs,
Project Lead (PL)
develops an
implementation
manual and guidebook.

Project Lead (PL)
develops an
implementation
manual and guidebook.
Completion of manual
and guidebook to be
one semester prior to
the first
implementation of the
courseware.
Number of completed
sections of the
implementation manual
and guidebook.
Number of courseware-
using faculty accessing
the manual and
guidebook.

Number of new faculty
accessing the manual
and guidebook as they
implement courseware
into their course.

PL launches a
school-based
adaptive learning
courseware
website which
contains the
implementation
manual and
guidebook, among
other resources.
Launch of school-
based adaptive
learning courseware
website at least one
semester prior to
starting of
courseware program
implementation.

144

Critical Behavior Metric(s) Method(s) Timing
2 - Each school creates
a Professional
Learning Community
(PLC) composed of
PL, courseware faculty
users, curriculum
developers, and
instructional staff that
meets bi-monthly.
The number of
faculty, developers,
and staff participating
in the Professional
Learning Community
forums.
PL monitors activity,
engages members,
and interjects
operational guidance
and knowledge in the
PLC forum.
PLC is established
simultaneously with
the launch of the
adaptive learning
courseware website.
PLC forum is integral
to the courseware
website.
3 - Provide monthly
ongoing training for
new and continuing
faculty on how to
effectively integrate
and utilize courseware
into their practice.
The number of
trainings and check-
ins conducted with
faculty.
PL provides in-
semester and post-
semester check-ins
with courseware
using faculty.
Synchronous training
pre-semester.
Asynchronous
training tutorials
available on demand.
Monthly check-ins
between PL and
courseware-using
faculty.

Required Drivers
According to Kirkpatrick and Kirkpatrick, required drivers are "processes and systems
that reinforce, monitor, encourage and reward the performance of critical Behaviors on the job"
(p. 14). “A required driver package - methods and systems of monitoring, reinforcing,
encouraging and rewarding performance of critical behaviors on the job needs to be created and
implemented” (Kirkpatrick & Kirkpatrick, 2016, p. 59). Common examples of required drivers
are job aids, work review, coaching, performance-related pay, and recognition. Required drivers
are key to accomplishing the desired on-job application of the training. The success of
reinforcing the skills and providing knowledge acquisition through training supports an
organization and its operations. Additionally, execution and ongoing monitoring of required
drivers are essential to a program’s success. Table 12 shows the required drivers to support
critical behaviors.
145

Table 12
Required Drivers to Support Critical Behaviors
Method(s) Timing
Critical
Behaviors
Supported
(1, 2, 3, etc.)
Reinforcing
Provide introductory training for faculty new
to implement adaptive courseware.
Prior to the beginning of
a semester.
1, 3
Provide, in the form of a job aid, an
implementation manual, and guidebook.
Prior to the beginning of
a semester.
1, 3
Provide a formal training introduction and
walk thru of the implementation manual and
guidebook for new and continuing faculty.
Prior to the beginning of
a semester.
1, 3
Provide ongoing training for and check-ins
with faculty to effectively integrate and utilize
courseware into their practice.
Monthly 1, 3
Provide implementation guidance through
active participation in the PLC forum.
Bi-weekly 2, 3
Encouraging
Champion examples of faculty implementation
successes to date.
Monthly 1, 2, 3
Highlight improved student outcomes from
adaptive courseware courses.
Monthly 1, 2, 3
Share implementation courseware successes
recounted by other schools and programs.
Monthly 1, 2, 3
Rewarding
Recognize and champion faculty members who
are effectively implementing adaptive
courseware in school and department
communications.
Monthly 1, 2

146

Rewarding
Recognize and champion faculty members who
are effectively implementing adaptive
courseware to external stakeholders and
educational groups.
End of semester 1, 2
Recognize and champion improved student
outcomes from adaptive courseware courses to
school leaders and administration.
End of semester 1, 2
Monitoring
Project Leads monitor, track, and document,
over the semester, for all implementations,
adaptive courseware course outcomes.
End of semester 1, 3
Project Leads monitor, track, and document,
over the semester, faculty member, adaptive
courseware course outcomes.
End of semester 1, 3
Project Leads monitor, track, and document,
over the semester, by department, adaptive
courseware course outcomes.
End of semester 1, 3
Project Leads monitor, track, and document,
over the semester, by course, adaptive
courseware course outcomes.
End of semester 1, 3

Organizational Support
In order to support recommendations of critical behaviors from Table 12, it is
recommended that specific organizational changes must occur. First, a program-encompassing
job aid, as recommended by the development of an adaptive learning courseware implementation
manual and guidebook, will reinforce program procedures, processes, and practices for all
participants. This comprehensive reference source will be available to not only the Project Lead
and courseware-using faculty but also to department leads of other school programs and faculty
considering the adoption of adaptive courseware, potentially broadening scale across a campus.
147

Additionally, this reference source will allow school leadership to review courseware program
implementation and reinforce support staff who may work directly with students.
It is also recommended that critical behaviors also be encouraged through the required
drivers. First, implementation program participants will engage in a professional learning
community, specific to the adaptive course program, and represent one of many resources found
on the adaptive learning courseware website. Second, encouragement will be expressed through
the disclosures of improved student outcomes, both from specific courses and across academic
programs. Finally, the website will highlight program implementations from other schools,
deepening the collective factual and procedural knowledge base.
Rewarding is another driver that supports critical behaviors. The website should
showcase implementation successes, with both recognition for faculty courseware users as well
as sharing best practices for effective instructional use. The implicit clout of leading faculty users
should support program viability and acknowledge early adopters and department influencers.
Following the lead of course implementers should drive scale across course sections and
academic departments.
Monitoring is the final required driver that supports critical behaviors. It is recommended
that Project Leads oversee multiple operational perspectives and numerous measures of
formative and summative program performance. With a more specific focus, program
performance should be monitored and appraised based on wide-ranging data points which
include faculty achievement, instructional efficacy, learner progress, individual, section or
department academic outcomes, student engagement, and institutional goal attainment. Finally,
program evaluation should be analyzed longitudinally over time (one semester to another and
148

one academic year to another) and as well as against learning outcomes in non-courseware using
courses, sections, and academic disciplines.
Level 2: Learning
According to Kirkpatrick and Kirkpatrick (2016), Level 2 - Learning is defined as the
knowledge and skills, attitude, confidence, and commitment participants acquired from training
for application in their daily jobs. Confidence and commitment were added to The New World
Kirkpatrick Model (2016) to bridge Level 3 - Behavior with Level 2 - Learning. Evaluating
knowledge centers on which method will be used to determine if participants know and
understand the content. Evaluating skills requires participants to demonstrate or prove
something. Evaluating attitude focuses on determining if participants see the value in what they
are being asked to do on the job (Kirkpatrick & Kirkpatrick, 2016). Evaluating confidence and
commitment is based on an effective training program that provides the opportunity for practice
and discourse.
Learning Goals
Based on the recommendations identified in this chapter, learning goals have been
developed. Upon fulfillment and completion of the recommended solutions, program participants
will be able to:
Project Leads
● Develop a comprehensive, resource-rich adaptive learning implementation
manual and guidebook. (Factual, Procedural)
● Develop a courseware training program for new and continuing faculty.
(Procedural, Cultural Setting, Resources)
149

● Develop a system and schedule to monitor, check-in with, and assist courseware-
using faculty. (Policies, Processes, and Procedures)
● Create a formative and summative evaluation procedure and process to oversee
courseware implementation performance. (Policies, Processes, and Procedures)
Faculty
● Develop and premise their implementation practice through a growth-minded
instructional directive that encompasses continuity, consistency, and commitment
to continuous improvement. (Conceptual, Value, Mood)
● Understand, monitor, and reflect upon their courseware implementation progress
and results for their courses. (Metacognitive)
● Compare and contrast their implementation progress and results to other
courseware-using sections. (Metacognitive, Self-efficacy, Expectancy)
● Compare and contrast their implementation progress and results to other
courseware-using courses in their department. (Metacognitive, Self-efficacy,
Expectancy)
● Compare and contrast their implementation progress and results to other
courseware-using courses in other academic disciplines at their school.
(Metacognitive, Self-efficacy, Expectancy)
● Contribute learned best practices and effective instructional strategies to the
implementation manual and guidebook, supporting and facilitating program
performance. (Factual, Procedural, Cultural Setting)

150

Program
It is recommended that the Program goals will be achieved through training, proactive
guidance, shared knowledge, and reflective practice. This proposed, multi-faceted program will
increase knowledge and support motivation for faculty and aggregate, coordinate, and
consolidate organization influences for Project Leads.
Two types of training can support adaptive learning courseware implementation - product
training and implementation training. Product training begins pre-semester, with expected heavy
demand in the first weeks of a new semester. Initial training for courseware use will be delivered
from product-curriculum developers, led by staff members from the companies who sell the
courseware programs. This initial training, in-person or on Zoom, will directly support Project
Leads, faculty users, and instructional staff. This initial training will be recorded and available
asynchronously on the adaptive learning courseware website for on-demand access. Further, pre-
made video “how-to” tutorials from the product-curriculum developer company will be available
on the school-based, program-created website. In-semester, additional training from the
curriculum developers will be scheduled according to need, initiated by faculty members or
Project Lead.
Implementation training could be led and facilitated by the Project Lead. Factors that will
influence and prompt implementation training will be the needs of faculty users, program
efficacy and “program health,” enrollment numbers, student engagement, instructional staff
requests, and learning outcomes. In-semester training and support may require the assistance of a
staff instructional designer or help from the school IT department. A proactive, hands-on, and
forward-looking approach by a Project Lead will lead and guide to better program outcomes.
151

A singular, accessible, and self-building knowledge source characterizes the development
and curation of an adaptive learning courseware implementation manual and guide. To benefit
Project Leads, new and existing faculty users, and instructional staff, this information and
resource portal protects, shares, and perpetuates knowledge, motivation, and organization
program assets, resources, and capital. Each manual and guide will be specific for each
implementation program as schools differ by institutional objectives, instructional goals,
department criteria, and enrollment benchmarks. With a goal to support implementation and
operational efficacy, the manual and guidebook will operationally formalize and instructional
document program routines, practices, and procedures.
Timely and forthright reflective practices could frame professional development and
learning for all program stakeholders. A cumulative base of experience and knowledge, gleaned
through courseware implementations across subjects, enrollments, and contexts, will guide the
Project Lead to better guide faculty users and refine supervision of instructional staff. Activated
motivation influences of value, self-efficacy, mood, and expectancy combine to support, sustain,
and advance a technology-facilitated implementation program. An established organizational
cultural setting that influences and inspires participant introspection will build program capacity,
capability, and attainment.
Evaluation of the Components of Learning
Kirkpatrick and Kirkpatrick (2016) describe learning as the degree to which participants
obtain the intended knowledge, skills, attitude, confidence, and commitment based on their
participation in training. Therefore, in order to assess learning, intentional methods, tools, and
techniques should be developed and employed. Table 13lists the methods and activities that will
152

be utilized to evaluate the declarative knowledge, procedural skills, attitude, confidence, and
commitment of the adaptive learning implementation program participants.
Table 13
Evaluation of the Components of Learning for the Program
Method(s) or Activity(ies) Timing
Declarative Knowledge “I know it.”
Project Leads lead planning meetings pre-
semester, check-ins mid-semester and
evaluations post-semester with faculty
courseware users, in part, to ascertain product
and implementation knowledge.
During and after the training sessions
Monitor if courseware implementations
accurately follow program routines, practices,
and procedures as referenced in the manual
and guidebook.

After the training sessions
Check-ins with instructional support staff. During and after the training sessions
Procedural Skills “I can do it right now.”
Checklist of implementation procedures and
steps.
During the training sessions
Checklist of how to utilize courseware
capabilities and functions.
During the training sessions
Shared courseware program implementations
from other sections, departments, and schools.
During the training sessions
Attitude “I believe this is worthwhile.”
Open-ended summative survey for faculty
after the semester to assess training and
guidance.
After the training sessions
Open-ended summative survey for
instructional support staff after the semester to
assess training and guidance.
After the training sessions

153

Method(s) or Activity(ies) Timing
Confidence “I think I can do it on the job.”
During training sessions, share reports of
effective implementation programs.
During the training sessions
Survey program participants to assess self-
efficacy as it relates to training and job
performance.
After the training sessions
Commitment “I will do it on the job.”
Plan of implementation developed
collaboratively between Project Lead and
faculty user during training.
During the training sessions
Plan of implementation developed
collaboratively between Project Lead and
instructional support staff during training.
During the training sessions

Level 1: Reaction Evaluation Tools
According to Kirkpatrick and Kirkpatrick (2016), Level 1 - Reaction is the degree to
which participants find the training favorable, engaging, and relevant to their jobs. The New
World Kirkpatrick Model defines Level 1 - Reaction as the measurement of favorability,
relevancy, and engagement as it relates to training (Kirkpatrick & Kirkpatrick (2016). Level 1 is
usually measured immediately after a training session to obtain the highest rate of response. As
the following table shows, engagement refers to the degree to which training participants are
involved in the learning experience. Relevance is the degree to which training participants have
the opportunity to employ what they learned on the job. Customer satisfaction may be assessed
formatively or summative and has a positive connection to learning. Concisely, “the key to Level
1 is to quickly and efficiently get the information you need to confirm that the quality of the
program and instructor is acceptable” (p. 39). Table 14 outlines the methods that will be used to
determine how adaptive courseware implementation participants react to the training sessions.
154

Table 14
Components to Measure Reactions to the Program
Method(s) or Tool(s) Timing
Engagement
Asking meaningful questions about the courseware product During the training
Asking meaningful questions about the courseware
implementation process and procedures
During the training
Completion of the implementation checklist During the training
Relevance
Pulse checks throughout the training sessions During the training
Training evaluation at the end of each session During the training
Customer Satisfaction
Pulse checks throughout the training sessions During the training
Training evaluation at the end of each session During the training

According to Kirkpatrick and Kirkpatrick (2016), there are two types of evaluation tools:
immediate and delayed. Immediate evaluation tools measure participant reaction directly after
the conclusion of training and assess areas that include favorability, relevancy, and timing.
Delayed evaluation tools are equally critical as they provide time and perspective for participants
to apply what they have learned in training sessions on the job. Each type of evaluation serves a
distinct purpose and overall evaluation combines both assessment methodologies.
Immediately Following the Program Implementation
The goal for evaluation directly after the program’s implementation training is to evaluate
Level 1 (customer satisfaction, relevance, and engagement) and Level 2 (knowledge and
motivation) outcomes (Kirkpatrick & Kirkpatrick, 2016). Immediately following the
implementation training, participants could complete an evaluation. The evaluation will indicate
155

the relevance of the components and content of the training sessions to their job, the satisfaction
of the training program, and indicate the commitment, attitude, and confidence in applying what
was learned. The evaluation tool will consist of a Likert scale and open-ended questions and can
be found in Appendix 1.
Delayed for a Period After the Program Implementation
The delayed evaluation tool, as seen in Appendix B would be distributed to training
participants approximately 45-60 days (approximately mid-semester) after the training session
concludes. The delayed evaluation tool would consist of a Likert scale and open-ended
questions. Likert scale questions will be used to assess critical behaviors for both Level 1 -
Reaction, indicating participant engagement, relevance, and customer satisfaction and for Level
2 - Learning, denoting declarative knowledge, procedural knowledge, attitude, confidence, and
commitment. Open-ended questions will be used to assess critical behaviors for Level 3 -
Behaviors as well as participants’ application of training to achieve the program implementation
goal for Level 4 - Results.
Data Analysis and Reporting
In order to support and achieve program and organizational goals, it is essential to
evaluate and report both formative and summative data gathered from critical behaviors. These
critical behaviors include a) Project Lead (PL) develops an implementation manual and
guidebook. Completion of manual and guidebook to be one semester prior to the first
implementation of courseware; b) Each school creates a Professional Learning Community
(PLC) composed of PL, courseware faculty users, curriculum developers, and instructional staff
that meets bi-monthly; c) Provide monthly training for new and continuing faculty on how to
effectively integrate and utilize courseware into their practice.
156

Data from both immediate and delayed evaluation tools will be aggregated and used to
evaluate current program performance as well as guide program efficacy moving forward . In
collaboration with the curriculum development companies (courseware vendors), courseware
implementation data will be aggregated and displayed in a “digital dashboard.” A cursory
version of the dashboard will be displayed on the program website while a “down to the nuts and
bolts” version available to program leadership. This representation and utilization of data
analytics will, among other appraisements and variable dissections, draw inferences from “cohort
by cohort training implementation outcomes,” “this year’s pre-semester training outcomes to last
year’s training outcomes,” “year over year outcomes,” “fall semester to spring semester
outcomes,” “current fall semester to last fall semester,” “specific courseware product to student
outcomes” and “course to student outcomes” as representations of program performance.
Summary of the Implementation and Evaluation
The New World Kirkpatrick Model (Kirkpatrick & Kirkpatrick, 2016) provides the
framework for implementation and evaluation of this study. The Four Levels of training and
evaluation are used to determine that program participants have the knowledge, motivation, and
organizational support to effectively and successfully implement adaptive learning courseware.
Systems are also embedded in the framework to monitor and adjust training and support as
necessary. Additionally, this model offers a framework for determining measurable outcomes
that range from the broad, high-level review to specific, individual indicator review to
comprehensively and efficiently assess training and learning results and reactions.
The New World Kirkpatrick Model begins with Level 4 - Results. For this level,
outcomes are measured through the facilitation of training and attendant supports, while leading
indicators, both short term goals, and observations, determine if training sessions achieve overall
157

objectives. Level 3 - Behavior assesses to what extent participants apply training knowledge to
their daily jobs. Level 2 - Learning incorporates the knowledge and skills, attitude, confidence,
and commitment participants obtain from training towards application and employment in their
daily jobs. Lastly, Level 1- Reaction measures favorability, relevancy, and engagement as it
relates to training while utilizing both immediate and delayed evaluation tools to determine how
participants perceived training as well as applied knowledge learned in their workplace.
The value of utilizing the New World Kirkpatrick Model for this specific project is that it
will provide adaptive learning courseware program leadership with strategic information
regarding the potential effectiveness of the proposed program of training as well as identifying a
plan and pathway to move forward with implementation. Throughout the four levels, formative
inquiry should ask and determine if expectations are being addressed and met. If actions and
results fall short of expectations, required drivers can be re-examined, reintroduced, and acted
upon as necessary. Finally, towards a goal of optimized adaptive courseware implementation,
evaluating the quality and efficacy of the program will help guide future training by adjusting,
augmenting, and supplanting the format and composition of learning and knowledge acquisition.
Limitations and Delimitations
Limitations of this study include the sample size, scope and selection of documents
analyzed, the relative early adoption rate of adaptive learning in higher education, the relative
precursory and emergent developmental state of adaptive learning courseware, and the author’s
positionality. Adaptive courseware use is far from ubiquitous but moving towards scale in some
sectors of education. Today, programs of implementation are increasing at both the community
college and university levels, with courseware use particularly focused on general education,
lower division, multiple-section, and high enrollment courses. As courseware use increases
158

across two and four-year institutions, moving beyond first or second-year courses and
transitioning from a textbook supplement to lecture or textbook replacement, the applications
and employments of courseware programs will concurrently expand, providing new contexts and
novel connections for opportunities in research.
Interview participants were active stakeholders in their school’s adaptive learning
courseware implementation programs. As such, their individual perspectives of mid-program
implementation should be viewed through a formative lens, with summative determination
forthcoming. Over time, accrued knowledge and implementation experience build insight,
efficacy, and expertise, introducing more progressive and involved program implementations.
Asa result, an in-progress viewpoint today may become only part of a larger discussion of
diverse implementations at scale.
The limited sample size of courseware implementations studied, however, does not
preclude the potential impact of this genre of learning technologies. Advances in artificial
intelligence and the emergence of learning science expect to elevate courseware towards a true
personalized and differentiated learning experience for students in higher education. These
advancements benefit teachers as well, allowing instructional efforts to focus on pedagogy and
less on traditional knowledge transfer. This same evaluation study, conducted a few years from
now, would likely illuminate highly evolved adaptive courseware implementations, fueled by
AI-rich augmented learning, resulting in improved student outcomes across a broader range of
topics and classes. Lastly, the author of this study works in the field of education technology in
higher education and teaches an undergraduate course on online teaching for pre-service K-12
student-teachers.

159

Recommendations for Future Research
Adaptive learning courseware, off to a respectable start, will increasingly and
systematically shift and transform the format and delivery of general education classes, to be
followed shortly by expanding assimilation within upper-level courses. Within a few years, the
impact of adaptive courseware with embedded chatbots acting as an available-anytime
instructional coach will decidedly reform undergraduate instruction. Students will have a
physical teacher and a virtual teacher in the same on-campus or online class, both providing
content delivery and instructional guidance, both being subject matter experts and assessment
proctors, and both able to reply to spoken questions and provide pertinent feedback. Once this
level of “dual instruction” is reached, the clear benefactors will be learning, instruction, and the
student.
Qualitative and quantitative research on student outcomes of the application of this new
generation of learning would be enlightening. Questions to study would be: Are there changes in
student engagement in a course? Are more students completing their classes? Have students’
formative and summative grades changed? Has enrollment in chatbot-embedded adaptive
courseware courses changed? Are the bigger-picture-effects changes in graduation rates?
The embed of curriculum-infused, AI chatbots in adaptive courseware will usher in a new
dimension of teaching and learning. Reaffirming that knowledge attainment and knowledge
transfer is a main objective of education, the concurrent dual modes of instruction will
predictably benefit students and teachers alike.
From a research perspective, a study in itself could be evaluating the “LX” or learner
experience of students taking this dual-mode class. Another perspective could center on a
discussion of students enrolled in fully online programs - what would be measures of course
160

efficacy of an asynchronous class led by only a chatbot? With AI continuing to amass and
aggregate learning and enrollment data towards “optimizing” a course, would, at some point, a
first-year, introduction to psychology course be nearly “perfect” in its delivery, presentation,
interface, and instruction? The questions are many.
Conclusion
This evaluation study essentially “sets the table” for an expected push towards AI-led
learning and instruction for undergraduate education. The role of the instructor progresses to
“curator,” “facilitator,” “guide” or similar to an orchestra conductor - administering all the pieces
and making sure things run smoothly.
The realizations and realities from a pandemic have strongly underscored the limited
scope of available instructional resources and inherent limitations of our educational models.
Technology, while not an inclusive answer to a myriad of challenges, does act as an instrument
and catalyst for good pedagogy and versant instructional practice.
Adaptive learning courseware, circa 2020, will pale in comparison to a 2024 model, for
example. Currently, IBM, Pearson, and others are in beta integrating chatbots into digital
textbooks. Much like Google Duplex’s close-to-humanlike conversational chatbot capabilities
think of a new generation of Siri, pre-loaded with the course textbook and lecture notes, the
course’s two community college prerequisite course textbooks and notes, the course syllabus,
formative and summative assessment examples, an integrated study group, office hours and
teacher assistants’ calendar and an AI-accumulated bank of ready-to-go, course-specific
curriculum responses, and feedback FAQs.
The duality of instruction is intriguing. Two teachers are better than one, right? The
classroom or Zoom teacher leads and guides the course. The virtual AI teacher helps to
161

asynchronously deliver the curriculum and is able to answer questions, direct learning, supply
explanations, provide feedback, and differentiate learning to best suit and engage each student.
While never to replace human course instructors, the abilities of AI virtual teachers supplements
and accelerates instruction for many levels and disciplines of learning.

162

References
Alexander, B.,Adams Becker, S., Cummins, M., Davis, A., Freeman, A., Hall Glesinger, C., &
Ananthanarayanan, V. (2017). NMC Horizon Report: 2017 higher education edition. The
New Media Consortium.
Alexander, B., Ashford-Rowe, K., Barajas-Murphy, N., Dobbin, G., Knott, J., McCormack,
M.,… & Weber, N. (2019). Educause Horizon Report: 2019 Higher Education Edition.
Educause
Alli, N., Rajan, R., & Ratliff, G. (2016, March 7). How personalized learning unlocks student
success [blog post]. https://er.educause.edu/articles/2016/3/how-personalized-learning-
unlocks-student-success
Anderson, L., & Krathwohl, D. (2001). A taxonomy for learning, teaching, and assessing
(1st ed.). Longman.
Bailey, A., Vaduganathan, N., Henry, T., Laverdiere, R., & Pugliese, L. (2018). Making digital
learning work: Success strategies from six leading universities and community colleges.
The Boston Consulting Group.
Barajas-Murphy, N. (2018). Foreword: Leveraging adaptive courseware and adaptive learning.
Current Issues in Emerging eLearning, 5(1), 1–6.
Bill & Melinda Gates Foundation. (2016, September 14). The changing face of U.S. higher
education. The Washington Post. http://www.washingtonpost.com/sf/ brand-
connect/gates/the-changing-face-of-us-higher-education/
Bill & Melinda Gates Foundation. (n.d.). Postsecondary success.
https://www.gatesfoundation.org/What-We-Do/US-Program/Postsecondary-Success
163

Brown, M., Dehoney, J., & Millichap, N. (2015). The next generation digital learning
environment. EDUCASE.
California Department of Developmental Services. (2008). Fact book..
http://www.dds.ca.gov/FactsStats/docs/factbook_11th.pdf
Chen, B., Bastedo, K., Kirkley, D., Stull, C., & Tojo, J. (2017). Designing personalized adaptive
learning courses at the University of Central Florida.
https://library.educause.edu/resources/2017/8/designing-personalized-adaptive-learning-
courses-at-the-university-of-central-florida
Cimera, R. E., & Cowan, R. J. (2009). The costs of services and employment outcomes achieved
by adults with autism in the US. Autism, 13(3), 285–302.
https://doi.org/10.1177/1362361309103791
Clark, R. E., & Estes, F. (2008). Turning research into results: A guide to selecting the right
performance solutions. CEP Press.
Dockterman, D. (2018). Insights from 200+ years of personalized learning. npj Science of
Learning 3(1), 15. https://doi.org/10.1038/s41539-018-0033-x
Dziuban, C., Howlin, C., Moskal, P., Johnson, C., Parker, L., & Campbell, M. (2018). Adaptive
learning: A stabilizing influence across disciplines and universities. Online Learning,
22(3), 7–39. https://doi.org/10.24059/olj.v22i3.1465
Engle, J. (2016). Answering the call: Institutions and states lead the way toward better measures
of postsecondary performance. Bill & Melinda Gates Foundation.
Gebhardt, K. (2018). Adaptive learning courseware as a tool to build foundational content
mastery: evidence from principles of microeconomics.” Current Issues in Emerging
Elearning, 5(1), 7–19.
164

Hinkle, J. F., & Moskal, P. (2018). A preliminary examination of adaptive case studies in nursing
pathophysiology. Current Issues in Emerging eLearning, 5(1).
Intentional Futures. (2017). High-tech high-touch: Serving student needs at scale. Online
Learning Consortium.
Johanes, P., & Lagerstrom, L. (2017, June). Adaptive learning: The premise, promise, and
pitfalls. [Paper presentation]. ASEE 124th Annual Conference and Exposition,
Columbus, OH, United States.
Johnson, C. (2016). Adaptive learning platforms: Creating a path for success.
https://er.educause.edu/articles/2016/3/adaptive-learning-platforms-creating-a-path-for-
success
Johnson, C., & Zone, E. (2018). Achieving a scaled implementation of adaptive learning through
faculty engagement: A case study. Current Issues in Emerging eLearning, 5(1), 7.
Kaplan, A. M., & Haenlein, M. (2016). Higher education and the digital revolution: About
MOOCs, SPOCs, social media, and the cookie monster. Business Horizons, 59(4), 441–
450. https://doi.org/10.1016/j.bushor.2016.03.008
King, J., & South, J. (2017). Reimagining the role of technology in higher education: A
supplement to the national education technology plan. U.S. Department of Education.
McCowan, T. (2016). Three dimensions of equity of access to higher education. Compare: A
Journal of Comparative Education, 46(4), 645–665.
https://doi.org/10.1080/03057925.2015.1043237
Nodine, T. R. (2016). How did we get here? A brief history of competency-based higher
education in the United States. The Journal of Competency-Based Education, 1(1), 5–11.
https://doi.org/10.1002/cbe2.1004
165

O’Sullivan, P. (2018). APLU adaptive courseware grant, a case study: Implementation at the
University of Mississippi. Current Issues in Emerging eLearning, 5(1), 45–61.
Sharma, S. K., Palvia, S. C. J., & Kumar, K. (2017). Changing the landscape of higher education:
From standardized learning to customized learning. Journal of Information Technology
Case and Application Research, 19(2), 75–80.
https://doi.org/10.1080/15228053.2017.1345214
SRI Education. (2016). Lessons learned from early implementations of adaptive courseware.
https://library.educause.edu/resources/2016/6/lessons-learned-from-early-
implementations-of-adaptive-courseware
Tate, M. (2018, July 30). Jill Watson’s terrific twos. Georgia Tech News.
https://www.news.gatech.edu/features/jill-watsons-terrific-twos
Tesene, M. M. (2018). Adaptable selectivity: A case study in evaluating and selecting adaptive
learning courseware at Georgia State University. Current Issues in Emerging eLearning,
5(1), Article 6.
Vignare, K., Kelsey, R., & Guney, S. V. (2019, February 21). Improving student outcomes with
adaptive courseware: The Every Learner Everywhere Network Initiative.
https://er.educause.edu/blogs/2019/2/improving-student-outcomes-with-adaptive-
courseware-the-every-learner-everywhere-network-initiative
Vignare, K., Lammers Cole, E., Greenwood, J., Buchan, T., Tesene, M., DeGruyter, J.,… &
Kruse, S. (2018). A guide for implementing adaptive courseware: From planning through
scaling. Association of Public and Land-grant Universities.
166

Young, J. R. (2018, April 13). Can a family of bots reshape college teaching? EdSurge.
https://www.edsurge.com/news/2018-04-13-can-a-family-of-bots-reshape-college-
teaching

167

Appendix A: Immediate Evaluation Tool
Level 1: Reaction and Level 2: Learning

1. The training curriculum held my attention. (Engagement)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

2. I was responsible for my own learning by being involved in the training (Engagement)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

3. The training information is applicable to my daily job. (Relevance)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

4. I enjoyed the training course content. (Customer Satisfaction)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

5. I learned the necessary knowledge and skills from the training content to use in my job.
(Declarative Knowledge)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

6. I practiced the necessary strategies during the training I can use in my job. (Procedural
Knowledge)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree
168

7. I believe the training was worthwhile to my job responsibilities. (Attitude)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

8. I feel confident to apply the knowledge and skills I learned from the training to my daily
job. (Confidence)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

9. I intend to apply the knowledge and skills learned from the training to my job.
(Commitment)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

10. Was there any product or implementation knowledge or skills missing from the training?
Please be specific.

______________________________________________________________________
______________________________________________________________________

169

Appendix B: Delayed Evaluation Tool
45 - 60 Days after training
Level 1: Reaction, Level 2: Learning, Level 3: Behavior, Level 4: Results

1. I have had the opportunity to use the course content in my job. (Relevance)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

2. Upon reflection, enrolling in this professional development training was a valuable use of
my time. (Customer Satisfaction)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

3. I have applied the knowledge and skills I learned in the training for my job. (Declarative
Knowledge)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

4. I have applied the strategies I learned in the training for my job. (Procedural Knowledge)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

5. The training was a worthwhile experience. (Attitude)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

6. I know where to find additional support if I should need it now that I have returned to my
job. (Confidence)
A. Strongly agree
B. Agree
170

C. Disagree
D. Strongly disagree

7. I have applied training concepts to my job. (Commitment)
A. Strongly agree
B. Agree
C. Disagree
D. Strongly disagree

8. How have you applied training principles to your daily work? (Behavior)
________________________________________________________________________
________________________________________________________________________
9. What outcomes do you expect from your efforts? (Behavior)
________________________________________________________________________
________________________________________________________________________
10. What impact has the professional development training had on the organization?
(Results)
________________________________________________________________________
________________________________________________________________________

171

Appendix C: Informed Consent/Information Sheet
University of Southern California
Rossier School of Education
3470 Trousdale Pkwy, Los Angeles CA, 90089

[Name of the Study]
You are invited to participate in a research study. Research studies include only people who
voluntarily choose to take part. This document explains information about this study. You should
ask questions about anything that is unclear to you.

PURPOSE OF THE STUDY
This study aims to

Write your

PARTICIPANT INVOLVEMENT
If you agree to take part in this study, you will be asked to

CONFIDENTIALITY
There will be no identifiable information obtained in connection with this study. Your name,
address or other identifiable information will not be collected.

Required language:
The members of the research team, the funding agency and the University of Southern
California’s Human Subjects Protection Program (HSPP) may access the data. The HSPP reviews
and monitors research studies to protect the rights and welfare of research subjects.

When the results of the research are published or discussed in conferences, no identifiable
information will be used. (Remove this statement if the data are anonymous)

INVESTIGATOR CONTACT INFORMATION
The Principal Investigator is [NAME, EMAIL, PHONE]
The Faculty Advisors are [NAME, EMAIL, PHONE] and [NAME, EMAIL, PHONE].

IRB CONTACT INFORMATION
University Park Institutional Review Board (UPIRB), 3720 South Flower Street #301, Los
Angeles, CA 90089-0702, (213) 821-5272 or upirb@usc.edu

Abstract (if available)

Abstract This study consisted of a needs’ analysis to examine knowledge, motivation, and organizational influences on the implementation of adaptive learning courseware. Clark and Estes’s (2008) gap analysis framework was used to analyze and evaluate courseware implementation. To address challenges, some colleges and universities adopted adaptive learning, integrating adaptive courseware, to support and augment learning and instruction. The purpose of this study was to conduct an evaluation of adaptive learning implementation at eight higher education institutions. This analysis focused on stakeholders’ areas of knowledge and skill, motivation, and organizational resources for adaptive learning programs. While a complete study and evaluation would address all stakeholders, both in higher education institutions and associated entities, for practical purposes, the stakeholders of focus were the project leads responsible for the adaptive learning implementation programs at their respective universities.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Civic learning program policy compliance by a state department of higher education: an evaluation study

PDF

Adaptability characteristics: an evaluation study of a regional mortgage lender

PDF

Prior learning assessment portfolios: an evaluation study

PDF

First-generation college students and persistence to a degree: an evaluation study

PDF

Organizational agility and agile development methods: an evaluation study

PDF

Increasing parent involvement in social-emotional learning workshops in high school using the gap analysis approach

PDF

College and career readiness through high school STEM programs: an evaluation study

PDF

Strengthening community: how to effectively implement a conflict resolution and peer mediation program on a secondary school campus: an evaluation study

PDF

Managers’ learning transfer from the leadership challenge training to work setting: an evaluation study

PDF

One to one tablet integration in the mathematics classroom: an evaluation study of an international school in China

PDF

Establishing a systematic evaluation of positive behavioral interventions and supports to improve implementation and accountability approaches using a gap analysis framework

PDF

Customer satisfaction with information technology service quality in higher education: an evaluation study

PDF

Relationship between employee disengagement and employee performance among facilities employees in higher education: an evaluation study

PDF

A qualitative study of educators’ attitudes towards blended learning institutional adoption

PDF

Mandatory reporting of sexual violence by faculty and staff at Hometown University: an evaluation study

PDF

Improving foundational reading skills growth in middle school: a promising practices study

PDF

Implementing proactive safety strategies in place of reactive safety strategies at a manufacturing organization: an evaluation study

PDF

A qualitative examination of the methods church leaders use to increase young adult attendance in Christian churches: an evaluation study

PDF

Increasing organizational capacity at a small college to deploy revenue diversification strategies: an evaluation study

PDF

Improving the evaluation method of a military unit: a gap analysis

Asset Metadata

Creator Akai, Greg (author)

Core Title Adaptive learning in higher education: an evaluation study

School Rossier School of Education

Degree Doctor of Education

Degree Program Education (Leadership)

Publication Date 12/22/2020

Defense Date 12/16/2020

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag adaptive courseware,adaptive courseware implementation, courseware,adaptive learning,adaptive learning courseware,OAI-PMH Harvest

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Yates, Kenneth (committee chair), Hirabayashi, Kimberly (committee member), Vignare, Karen (committee member)

Creator Email ghakai@hotmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c89-415628

Unique identifier UC11666560

Identifier etd-AkaiGreg-9228.pdf (filename),usctheses-c89-415628 (legacy record id)

Legacy Identifier etd-AkaiGreg-9228.pdf

Dmrecord 415628

Document Type Dissertation

Rights Akai, Greg

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA