Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Creating effective online educational content: an evaluation study of the influences affecting course developers’ abilities to create online content with engaging learning strategies
(USC Thesis Other)
Creating effective online educational content: an evaluation study of the influences affecting course developers’ abilities to create online content with engaging learning strategies
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
Running head: CREATING EFFECTIVE EDUCATIONAL CONTENT 1
CREATING EFFECTIVE ONLINE EDUCATIONAL CONTENT: AN EVALUATION
STUDY OF THE INFLUENCES AFFECTING COURSE DEVELOPERS’ ABILITIES TO
CREATE ONLINE CONTENT WITH ENGAGING LEARNING STRATEGIES
by
Erika Maldonado
A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the Requirements for the Degree
DOCTOR OF EDUCATION
August 2018
Copyright 2018 Erika Maldonado
CREATING EFFECTIVE EDUCATIONAL CONTENT 2
Dedication
This work is dedicated to my family, who has supported me throughout my life: my
parents (Cecilia and Oscar Maldonado), Mamita (Alicia Garrido), Abuelita (Carmen Tejada), and
my sisters (Natasha and Alexis Maldonado).
Mom and Dad: without you I would not have the drive to accomplish everything I do in
life. You have taught me the importance of grit. I may not always be the best or the smartest, but
I will always work tirelessly to accomplish what I want to achieve. Your constant wisdom and
dedication to my growth encourages me to strive to achieve my dreams. When faced with
obstacles and decisions in life in which I feel as though no one will believe in me, I always
believe in myself. And you are responsible for that. When I have had feelings of self-doubt and
the terrible voices in my head that sometimes tell me, “no you can’t do that” or “what makes you
think you might be worthy of…” there is always one strong voice that boldly states “yes, you
can” and “yes, you are worthy.” I have that voice because of you. In addition to your constant
support, I love and admire your zest for life and adventure – constant reminders that the life we
live is short and everyday should be seized!
Natasha aka MakeMeMaldonado, you are my hustler, dreamer, doer, Instagram-Queen,
and beauty-writer extraordinaire. Your drive to achieve your dreams pushes me to keep pressing
against my own limits and reminds me to never get comfortable and settle for where I am in life.
As my little sister, I know you usually look up to me, but as you have grown up I am in awe of
you and your confidence. Besides that, your makeup skills have kept me from looking like a
disheveled mess every day of my life. Thank you for teaching me how to contour, highlight, and
conceal so I can look mostly put-together. I would look like a blotchy and tired mess without
you.
CREATING EFFECTIVE EDUCATIONAL CONTENT 3
Alexis, you are the best nurse that I know. You are dedicated, smart, committed, and a
force to be reckoned with. I appreciate you bringing a custom-made IV vitamin drip to get me up
and going just before I lost all hope of productivity. Thank you for supporting me.
And finally, I would be remiss if I did not acknowledge the tremendous strength my
grandmothers taught me. Abuelita, you passed away too soon. When I think of you, I still get
teary-eyed, but I am proud of how strong you were throughout your years of suffering with
cancer. I never saw how much you suffered because you always had a smile on your face and
focused on how proud you were of me. I still think of you, your delicious Salvadorian quesadilla,
perfect Cuban flans, and sparkling clean home. To my Mamita, you are the nuttiest grandma I
know. At 93, everyone is in awe of your mobility, humor, and tenacity. I have so many
childhood memories of you pushing me to be better and taking me to and from piano, swimming,
and school. My adulthood memories are full of European vacations with you and my favorite:
our time at Oktoberfest in Munich, Germany. I am so grateful for the memories I have been able
to make with you. Mamita and Abuelita, I hope that I can make you as proud of me as I am of
being your granddaughter. You are both the definition of the word fierce. Mamita y Abuelita,
ustedes son la razón por la cual nuestra familia ha podido disfrutar de la libertad y las
oportunidades de los Estados Unidos. Las palabras no pueden describir la fuerza que siento
dentro de mí debido a lo que han inspirado.
CREATING EFFECTIVE EDUCATIONAL CONTENT 4
Acknowledgements
I would like to thank my Tuesday Trojans: Roseanne Cameron, Stephanie Trela-George,
Arvind Ramakrishnand, Christopher Riddick, Annie Rinaldi, Melissa Singh, and Andy Surber.
You all motivated me to get to this point, with your words of encouragement and constant
support. I am forever grateful for our group text messages, spontaneous get-togethers around
California and for the memories we have built together. I am truly looking forward to seeing
what the Tuesday Trojans achieve in the future! #FightOn
Tyler Botana, my partner, soul mate and the main reason that I am able to remain
somewhat composed and calm in my professional and academic relationships. You listen to and
tolerate, “Stressed and Anxious Erika” so that I get to be my best as, “Calm and Composed
Erika” at work and at school. When I have felt close to pulling all of the hairs out of my head,
you have been there to comfort me. I am also grateful that you have essentially dubbed yourself
our event planner so that we always have a fun show to watch or a cool activity to do after
productive workdays.
I am also deeply grateful for my committee: Dr. Anthony Maddox, Dr. Kimberly
Hirabayashi and Dr. Ilya. Dr. Maddox, your wisdom and innovative thinking have helped me to
expand my thinking and remain calm throughout the writing process. Dr. Hirabayashi, I am so
grateful for your meticulous notes and attention to detail. I don’t know where I’d be without your
specific and measurable action steps to successfully finish my dissertation. Your direction is
truly appreciated. Dr. Goldin, I am equally as grateful for your attention to detail and candid
feedback. Your support in this endeavor has helped me not only to finish but to also be motivated
to continue to learn more about the field of educational technologies and how I can apply them to
our organization.
CREATING EFFECTIVE EDUCATIONAL CONTENT 5
TABLE OF CONTENTS
Table of Contents 5
List of Tables 10
List of Figures 11
Abstract 13
Chapter 1: Statement of the Problem 14
Organizational Context and Mission 15
Organizational Goal 16
Related Literature 17
Importance of the Evaluation 18
Description of Stakeholder Groups 19
Stakeholder Groups’ Performance Goal 20
Stakeholder Group for the Study 20
Purpose of the Project and Questions 21
Conceptual Methodological Framework 21
Definitions 22
Engagement 22
Learning analytics 22
Learning management systems 22
Organization of the Project 23
Chapter 2: Review of the Literacture 24
Student Engagement in Online Degree Programs 24
Student Retention 26
Challenges to Student Engagement and Retention 26
Learning Theories and Frameworks 27
Community of inquiry 27
Learning communities 29
CREATING EFFECTIVE EDUCATIONAL CONTENT 6
Critical value factors 29
The Seven Principles of Online Education 30
Learning Analytics and Data 32
Clickstream data 33
Summary 34
Clark and Estes’ (2008) Organizational Problem Solving Framework 34
Stakeholder Knowledge, Motivation, and Organizational Influences 35
Knowledge and Skills 35
General Theory 35
Knowledge influences 36
Conceptual knowledge 37
Procedural knowledge 37
Metacognitive knowledge 38
Motivation 40
Attribution theory 41
Self-efficacy 41
Organizational Influences 42
Cultural models 43
Cultural settings 44
Conclusion 47
Chapter 3: Methodology 49
Participating Stakeholders 49
Survey Sampling and Criterion and Rationale 49
Criterion 1 49
Criterion 2 49
Survey Sampling (Recruitment) Strategy and Rationale 49
Interview Sampling Criterion and Rationale 50
Criterion 1 50
Criterion 2 50
CREATING EFFECTIVE EDUCATIONAL CONTENT 7
Interview Sampling (Recruitment) Strategy and Rationale 50
Conceptual Framework: The Interaction of Stakeholders’ Knowledge and Motivation and
the Organizational Context 50
Data Collection and Instrumentation 51
Surveys 52
Interviews 53
Data Analysis 54
Credibility and Trustworthiness 55
Validity and Reliability 56
Ethics 57
Limitations and Delimitations 58
Chapter 4: Results and Findings 60
Participating Stakeholders 60
Surveys 61
Interviews 62
Results and Findings Regarding Knowledge 62
Conceptual Knowledge 63
Active learning 63
Procedural Knowledge 65
Metacognitive Knowledge 70
Reflecting on content as a student 71
Declarative Knowledge 71
Synthesis 72
Results and Findings Regarding Motivation 73
Attribution 73
Student engagement 73
Self-efficacy 77
CREATING EFFECTIVE EDUCATIONAL CONTENT 8
Confidence 77
Synthesis 78
Results and Findings Regarding Organization 79
Faculty and University Influences 79
Relationship Management 79
University culture influences 80
Up to Date Information and Training 81
Need for supportive resources 81
Need for student data 84
Synchronous sessions 86
Accountability 86
Course developers are not accountable for creating content with engaging learning
strategies 86
Corporate Culture 86
Timing 86
Collaboration and leadership 88
Synthesis 88
Chapter 5: Recommendations 90
Recommendations for Practice to Address the Knowledge, Motivation, and
Organizational Influences 90
Knowledge Recommendations 91
Declarative knowledge solutions, or descriptions of needs or assets 92
Conceptual knowledge solutions, or descriptions of needs or assets 93
Procedural knowledge solutions, or descriptions of needs or assets 94
Metacognitive knowledge solutions, or descriptions of needs or assets 95
Motivation Recommendations 95
Value 97
Attribution 98
Self-efficacy 98
CREATING EFFECTIVE EDUCATIONAL CONTENT 9
Organization Recommendations 99
Accountability 100
Leadership 101
Corporate culture 102
Integrated Implementation and Evaluation Plan 103
Implementation and Evaluation Framework 103
Organizational Purpose, Need, and Expectations 103
Level 4: Results and Leading Indicators 104
Level 3: Behavior 106
Critical behaviors 106
Required drivers 107
Organizational support 110
Level 2: Learning 111
Learning goals 111
Program 111
Components of learning 113
Level 1: Reaction 115
Evaluation Tools 117
Immediately following the program implementation 117
Delayed for a period after the implementation 117
Data Analysis and Reporting 118
Summary 120
References 122
Appendix A: Recruitment Email 134
Appendix B: Survey Protocol 135
Appendix C: Interview Protocol 137
Appendix D: Informed Consent/Information Sheet 138
CREATING EFFECTIVE EDUCATIONAL CONTENT 10
List of Tables
Table 1. Organizational Mission, Global Goal, and Stakeholder Performance Goals 20
Table 2. Assumed Knowledge Influences 39
Table 3. Assumed Motivation Influences 42
Table 4. Assumed Organizational Influences 46
Table 5. Gender Identity 61
Table 6. Critical Behaviors, Metrics, Methods, and Timing for New Reviewers 61
Table 7. Summary of Knowledge Influences and Recommendations 91
Table 8. Components of learning 96
Table 9. Summary Of Organization Influences And Recommendations. 99
Table 10. Outcomes, Metrics, and Methods for External and Internal Outcomes 104
Table 11. Critical Behaviors, Metrics, Methods, and Timing for New Reviewers 106
Table 12. Required Drivers to Support New Reviewers’ Critical Behaviors 107
Table 13. Components of learning 113
Table 14. Components to Measure Reactions to the Program 115
CREATING EFFECTIVE EDUCATIONAL CONTENT 11
List of Figures
Figure 1. Community of inquiry framework. 28
Figure 2. Conceptual framework. 51
Figure 3. Developing asynch activities is important to engaging an online learner. 63
Figure 4. Do you try to encourage the use of online discussions in the Learning Management
Systems? 64
Figure 5. I understand how online assessments work to engage students. 66
Figure 6. I feel confident in my ability to identify content that is engaging or not engaging.
69
Figure 7. How do you define student engagement 78
Figure 8. Student engagement is affected by how I plan content with professors. 80
Figure 9. Student engagement is affected by how we build content in the learning management
systems. 81
Figure 10. What, if any of the items below, drives the means you use to plan and produce content
with professors? - Selected Choice 82
Figure 11. I feel confident in my ability to identify content that is engaging or not engaging.
83
Figure 12. I want to continue to learn how I can create content that is engaging to online learners
84
Figure 13. Influences on course developers 86
Figure 14. The organization has provided me enough training to understand how online tools
engage students. 88
Figure 15. I feel that my organization provides me with enough resources to plan engaging
CREATING EFFECTIVE EDUCATIONAL CONTENT 12
content. 89
Figure 16. How do you think, if you are interested, you can be provided with more information
on how to build and plan content that engages online students? - selected choice 90
Figure 17. Would you like more information on how students engage with the content you create
with professors? 91
Figure 18. How do you think, if at all, you can gain more insight into how students engage with
the content you’ve produced? - selected choice 92
Figure 19. Timeline, Pedagogy and Professor Feedback 94
CREATING EFFECTIVE EDUCATIONAL CONTENT 13
Abstract
As the role of online learning in graduate education continues to expand, course developers must
have the resources needed to create engaging instructional content. Course content that leverages
effective learning strategies is more likely to successfully engage online students so they
complete all online coursework. Student engagement not only plays an important role in
educating students, but is also a factor in student retention numbers. Organizations that hire
course developers must train them so they possess the skills needed to design learning for future
generations. In this study, course developers from a large educational technology firm
participated in surveys and one-on-one interviews regarding their experience with developing
online courses. The Clark and Estes (2008) framework was used to categorize the findings by
knowledge, motivation, and organizational influences that affect course developers’ abilities to
produce engaging learning content. The study’s findings reveal that course developers
instinctually have some sense of how to engage online learners, but they lack the confidence to
use that understanding in guiding the faculty with whom they work. The study also found that
course developers need training resources, student data, and organizational support to create
effective online courses.
CREATING EFFECTIVE EDUCATIONAL CONTENT 14
CHAPTER 1: STATEMENT OF THE PROBLEM
Online education dates back to the 1980s, before the creation of the World Wide Web in
1992. Since its inception, the phenomenon of learning online has experienced a paradigm shift in
perspectives that people have toward earning a degree in a virtual environment (Harasim, 2000).
Given this paradigm shift, new businesses in the field of virtual learning have boomed. Some
have been focused on providing Massive Open Online Courses (MOOCs) while a steady flow of
companies have grown in the Software-as-a-Service (SaaS) market.
MOOCs and SaaS based online programs continue to grow because universities have
become more driven to satisfy student demand, expand their student body and capitalize on the
growing industry. Allen, Seaman, Poulin, and Straut, 2016 report that, in the United States, more
than 63% of higher education institutions feel that online education is vital to their long-term
growth strategy. The researchers also found that the number of students enrolled in at least one
online course is at an all-time high of 7.1 million (Allen et al., 2016). This means that the
compound annual growth rate of students in online courses is at a steady growth rate of 16.1
percent since the report started in 2002.
The growth in online learning is not only felt in the United States, but it is also
worldwide. For example, in Australia, (Steven & Moore, 2014) found that between the year 2008
and 2012, student enrollment in the Swinburne University of Technology’s undergraduate
marketing program grew to become five times larger after expanding to online courses. While
the amount of growth that the field of online learning is experiencing is impactful, it does not
come about without negative ramifications (Capra, 2011). Universities need to learn a new way
of building content in online Learning Management Systems (LMS), and course developers are
tasked with the challenge of figuring out the best ways to teach online.
CREATING EFFECTIVE EDUCATIONAL CONTENT 15
One of the most profound issues with building and developing effective content for
online programs is that there is still much investigation that needs to be done in order to be
conclusive about what types of online learning tools work best to achieve learning outcomes.
More importantly, however, is that often times, because of the growth of the online learning
industry, course developers don’t receive the proper training needed to create engagin online
learning content. This affects both the learning experience for students and potentially student
engagement and retention.
Organizational Context and Mission
The name of the organization is a pseudonym. Any identifiable citations and references
are omitted. Identifying, non-critical details regarding the study site are altered for anonymity
(e.g., location, dates).
Ed Tech Inc. is an educational technology company that was founded by a leader in the
education space because they felt that online education could not only be good but also could be
great or even better than brick and mortar classes. Fueled by a passion for education and desire to
innovate, the organization launched its first program, which was a Master’s in Education. Since
then, the company has scaled from just one program at one school to multiple programs across
several universities. Its mission is to partner with quality colleges and universities to provide the
world’s best online education to today’s students.
In addition to being an educational technology company, Ed Tech Inc. provides its
partners with all services needed from marketing, production, and student/faculty support to
create scalable and immersive online programs. The degrees are scalable because they reach
students at a global, rather than regional or national, level. Each program aims to be immersive
by building courses with proprietary online tools that are meant to foster asynchronous
CREATING EFFECTIVE EDUCATIONAL CONTENT 16
interaction amongst students.
According to the organization’s 2015 Impact Report, the number of students enrolled in
Ed Tech Inc.’s programs is about 12,000 across 79 countries. The mean class size is 10.48, and
there are over 20,000 students in field placements, such as social work students in on-site
internships. The average age of students is 33, and in 2014, 1,422 students graduated from
programs enabled by Ed Tech Inc.
Organizational Goal
Ed Tech’s goal is to improve 1 million lives through learning by 2030. The organization
strives to do this by building the best online learning platform and business that leads to the
successful development of graduates for its university partners. While the business aspect of the
organization focuses on retaining students and building them to be successful graduates, the
mission is to create an online learning experience that is equal to, if not better than, their on
ground counterparts. In brick and mortar schools, lecture halls are often crowded with hundreds
of students, and the least engaged or the sheer volume of colleagues in their courses shadows
more reserved. In online courses, sections promise students to only share the virtual lecture hall
with an average of ten to fifteen students per section.
The goal of the Course Development team at Ed Tech Inc. is to make future decisions on
produced educational content and courses based on data that can be allocated from the existing
learning platform and to train the course developers to build online content with effective
learning strategies. The learning platform is the Learning Management System (LMS) that
facilitates all educational activities for online students enrolled in Ed Tech Inc.’s programs.
Assessments, asynchronous lectures, forums, social groups created by peers and synchronous
classes are all facilitated via the LMS.
CREATING EFFECTIVE EDUCATIONAL CONTENT 17
Related Literature
The following section provides an overview of the literature that contributes to an
understanding of student engagement and retention.
Some research shows that retention rates may be attributed to outside factors within
students’ lives, not just the quality of the online content. Deng, Hunter, and Hyllegard (2008)
found that over 70 % of students at the Borough of Manhattan Community College (BMCC)
withdrew from online programs because of personal problems and commitments that were taking
up most of their time. In addition, Greenland and Moore (2014) interviewed students who
withdrew from the Australian school’s online marketing program and found that the top two
reasons were work-related factors and personal reasons (e.g., health, family and personal
relationships).
In addition to personal issues, online students also tend to struggle with a newfound sense
of autonomy in their online courses because they must manage their time differently in a virtual
environment. Kaur and Sidhu (2010) reported that students in a local Malaysian online university
had a difficult time making decisions on strategies to address their own learning objectives.
Additionally, they had poor planning skills, which are necessary to create and achieve goals.
Grade point average and attrition rates are also correlated in online programs. In a study of the
University of Tennessee’s Masters in Social Work program, Cummings, Chaffin, and Clifford
(2015) found that online students’ GPAs were lower than their on-ground counterparts.
However, while personal factors in students’ lives are a large part of student attrition, there are
still a number of elements that course developers face as challenges to consider when creating
engaging content, such as understanding how interaction with asynchronous content can affect
student engagement and the importance of understanding how to create effective educational
CREATING EFFECTIVE EDUCATIONAL CONTENT 18
videos.
Numerous studies suggest that there are several factors to consider when building
successful online programs that engage and retain online students. Constructivist and social
constructivist learning approaches, for example, have been examined to show a relationship
between their implementation and students’ perception of their learning (Arbaugh & Benbunan-
Fich 2006). Arbaugh and Benbunan (2006) suggest that students report higher scores for
perceived learning in courses that implement a collaborative learning model.
Lin, Aiken, Seaton, Douglas, Greco, Thos and Schatz (2016) examined over 70 online
videos that were created for a one-semester introductory mechanics course for a large accredited
university. The study found that students were most engaged with content that pertained to
laboratory activities in comparison to those that were recorded as lectures. Additionally, Lin et
al. (2016) findings showed that 80 % of students were consistently engaged with laboratory
related content and that engagement with lecture-based content dropped to less than 40 percent
by the term’s end.
Importance of the Evaluation
At an empirical level, the evaluation is important because it can contribute to the limited
knowledge base that exists with regard to how course developers’ knowledge, motivation, and
organizational influences affect how online courses are built. One of the most unexplored areas
in studying online learning is related to the roles of teaching approach (Arbaugh & Fich, 2006).
Arbaugh and Fich (2006) state that most empirical research ignores the role of teaching
approaches or only test one theory in online learning. As learning evolves, professors of online
courses and course developers must be equipped with the knowledge and skills to adapt to
teaching online (Barbour & Harrison, 2016).
CREATING EFFECTIVE EDUCATIONAL CONTENT 19
At an organizational level, Ed Tech Inc. is accountable to the students that enroll in the
online programs it develops. With tuition rates that run from about $70,000 to nearly $100,000
per degree, the experiences it provides must be exceptional. That is, if the expectations are not
met by the organization to the students, the company risks losing them as consumers.
Additionally, Ed Tech Inc. is accountable to faculty because without their satisfaction, and buy-
in, the online programs will not succeed and grow.
Ed Tech Inc. is also accountable to universities to create content in its programs that help
them to fulfill accreditation guidelines, produces graduates that can pass the necessary
assessments to succeed in their intended careers and achieve success in their field of work. If
enrolled students and graduates of the online programs do not meet the goals that the university
partners place, Ed Tech Inc. risks losing its partners and the possibility of forming new
partnerships with the schools. The relationship between the universities and Ed Tech Inc. in
regards to assessments and student success are reflective of professional accountability.
Description of Stakeholder Groups
At Ed Tech Inc., the stakeholders include students, the Course Development team, and
University lead professors. Students affect the achievement of the organization’s performance
goal because their enrollment drives program growth across university partners. The Course
Development team contributes to the organizational goal because it is responsible for planning,
producing and executing hundreds of courses per year. The lead professors are ultimately
responsible for verifying that their courses, both synchronously and asynchronously, achieve
their learning objectives. This dissertation will focus on the knowledge, motivation and
organizational elements that affect how the Course Development produces courses.
The Course Development team is comprised of around twenty Course Developers, but
CREATING EFFECTIVE EDUCATIONAL CONTENT 20
continues to grown every year as the company scales. The population was chosen for this
dissertation because course developers work closely with professors from university partners to
plan and produce courses from start to finish for online students.
Stakeholder Groups’ Performance Goal
Table 1
Organizational Mission, Global Goal, and Stakeholder Performance Goals
Organizational Mission
Ed Tech Inc.’s mission is to partner with quality colleges and universities to provide the
world’s best online education to today’s students.
Organizational Performance Goal
Ed Tech’s goal is to improve 1 million lives through learning by 2030. It will do this by
creating course content that aligns with the organization's aspirations around being more
efficient and effective in everything that it delivers: from more engaging learning experiences
to more streamlined and adaptable internal processes.
Student Goal Course Development Team Goal University Professors’ Goal
By January 2019,
students will engage
in course content that
is built with engaging
learning strategies.
By January 1, 2019, all Course
Developers at Ed Tech Inc. will
have the knowledge, motivation,
and organizational resources
required to create online
coursework with engaging
learning strategies.
University Lead Professors
Plan and produce content with
EdTech Inc. that mirrors, or is
better than, what is used with on
ground programs.
Stakeholder Group for the Study
Although a complete analysis would involve all stakeholder groups, for practical
purposes this paper will focus on the Course Development group at Ed Tech Inc. The reason this
group was chosen is that of the control they have on the development and production of the
asynchronous and synchronous content for online students (e.g., lectures, assessments,
CREATING EFFECTIVE EDUCATIONAL CONTENT 21
roundtable discussions, interviews and roleplays). Course developers work closely with
university faculty by meeting with them every week to plan and create media content that is
consumed by the online students. They also work with faculty to develop lesson plans for their
online, face-to-face classes.
The Course Development team’s performance goal is that all dourse developers at Ed
Tech Inc. will have the knowledge, motivation, and organizational resources required to create
online coursework with engaging learning strategies
Purpose of the Project and Questions
The purpose of this project is to evaluate the degree to which organization is meetings its
goal of being able to create content with engaging learning strategies. The analysis will focus on
knowledge, motivation and organizational influences related to achieving the organizational
goals. While a complete performance evaluation would focus on all stakeholders, for practical
purposes the stakeholder to be focused on in this analysis is the Course Development team. The
questions that guide this study are the following:
1. What are the knowledge, motivation and organizational (KMO) influences that impact how
course developers produce online courses with student engagement strategies in order to
achieve the goal of creating more engaging learning experiences for students?
2. How could organizational practices in the area of knowledge, motivation and organizational
resources be changed to facilitate the use of engaging learning strategies in Course
Development?
Conceptual Methodological Framework
Clark and Estes’ (2008) modified gap analysis, a systematic, analytical method that helps
to clarify organizational goals and identify the knowledge, motivation and organizational
CREATING EFFECTIVE EDUCATIONAL CONTENT 22
influences, will be adapted to evaluation model and implemented as the conceptual framework.
Knowledge, motivation and organizational influences were outlined based on a general literature
review. Solutions based on the research will be suggested as a means of improving future
development of tools for the organization.
Definitions
The following glossary of definitions ensures a common understanding of key concepts
in online learning and online Course Development.
Engagement. According to the Glossary of Education Reform (2016), student
engagement refers to the amount of attention, curiosity, interest, optimism, and passion that
students show when they are learning or being taught. This, in turn, affects how they are or are
not motivated to continue learning and progress in their education. The Glossary of Education
Reform (2016) states that the general assumption when talking about student engagement is that
when students are more engaged, they are more "inquisitive, interested, or inspired." In contrast,
if students are disengaged they are generally "bored, dispassionate, disaffected." Therefore,
student engagement is a common goal amongst educators.
Learning Analytics. Learning analytics is defined as “the measurement, collection,
analysis, and reporting of data about learners and their contexts, for purposes of understanding
and optimizing learning and the environments in which it occurs” (Long & Siemens, 2011). It is
the collection, analysis, and application of data accumulated to assess the behavior of educational
communities (Larusson, 2014).
Learning Management Systems. The term, learning management system, refers to any
type of software that is available online to educators as a means of hosting their educational
content. There are several types of platforms available to schools online, and each of them has
CREATING EFFECTIVE EDUCATIONAL CONTENT 23
their own different characteristics of settings that can be customized by their users. Some are
available for free online and others must be paid for in order to be activated. Philips (2005)
defines an LMS as a system that has tools that allow educators to track students, student
engagement, communications, and assessments.
Organization of the Project
Five chapters are used to organize this study. This chapter provided the reader with the
key concepts and terminology commonly found in a discussion about online student learning and
engagement. It also provided the organizational mission, stakeholders and an introduction to the
methodological framework. Chapter Two continues with a review of the current literature
surrounding the scope of the study. Topics on the knowledge, motivation and organizational
influences on course developers will be addressed as well as themes that affect online student
engagement. Chapter Three details the methodology that was used to choose the participants,
data collection, and analysis. It will also address ethical factors we well as reliability, credibility,
trustworthiness, and validity. In Chapter Four, the data and results are described and analyzed.
Chapter Five provides a discussion of the findings as well as a recommendation for practice,
based on data and literature. It also includes recommendations for an implementation and
evaluation plan.
CREATING EFFECTIVE EDUCATIONAL CONTENT 24
CHAPTER 2: REVIEW OF THE LITERATURE
This chapter outlines the effect of student engagement on retention rates in online
programs as well as how online course developers’ perceptions and beliefs affect their course
content construction for online degrees. The first section provides an historical overview of
factors influencing student engagement in online programs and why engagement is important.
The second segment describes how student retention drives revenue and university success since
the organization is accountable to its outside stakeholders and students to ensure that student
enrollment numbers do not falter. The next part focuses on learning theory and pedagogical
strategies used to enhance learning. Finally, the chapter ends with an analysis of student
engagement through the lens of engagement literature using the gap analysis dimensions of
knowledge, motivation, and organization.
Student Engagement in Online Degree Programs
When technology is used strategically and combined with pedagogical practices and
learning theory, student learning and engagement increases (Kuh & Vesper, 2001). Kuh and
Vesper (2001) state that when online courses use strategies specifically designed for online
learning, they are more likely to engage students in responding to online discussions and lecture
viewing. A plethora of factors contribute to the effectiveness of an online program, include:
student engagement, student perceptions, learning outcomes, perceptions of the designers and
course developers, and the quality of course instructors (Kuh & Vesper, 2001).
The National Survey of Student Engagement (NSSE) states that student engagement is
represented by the measurement of two elements: (1) the amount of time and effort students put
into their school work and activities and (2) how the institution utilizes resources and designs
curriculum to motivate students to participate in exercises linked to student linking as
CREATING EFFECTIVE EDUCATIONAL CONTENT 25
demonstrated by research studies (Kuh & Vesper, 2001).
The term engagement has become increasingly popular, likely because of the amount of
new knowledge generated about the role certain "intellectual, emotional, behavioral, physical,
and social factors play in the learning process and social development” (Kuh & Vesper, 2001).
The topic of engagement often arises when educators talk about strategies and teaching styles
that address the many factors that affect student learning. It is significant that the term
engagement holds different definitions depending on context. For example, a researcher might
view students' attendance and grades as a measure of engagement, while another might evaluate
engagement using the perceived feelings and emotional state of students (e.g., curiosity,
motivation, and interest). However, as mentioned above, the Glossary of Education (2016)
defines five types of engagement: emotional, behavioral, cultural, social, and physical.
Emotional engagement refers to how instructors promote a positive environment for
students, so they can engage with the learning materials, undistracted by negative behaviors
(Student Engagement, 2016). A professor who asks how students are feeling to assess their need
for services to improve their mental health is an example of focusing on emotional engagement.
Behavioral engagement refers to how teachers use routines to drive learning, like consistently
using breakout rooms or having students facilitate set activities each week. Cultural engagement
concerns how instructors promote inclusiveness by incorporating course content that represents a
variety of cultures or even offering translation services to support international and ESL students,
so they will persist in the class or program (Student Engagement, 2016). Physical engagement is
the use of kinesthetic learning to activate student thinking (Student Engagement, 2016).
Examples include asking students to stand up and write an answer on a board and, finally, social
engagement refers to how students engage with each other as a community (Kuh & Vesper,
CREATING EFFECTIVE EDUCATIONAL CONTENT 26
2001). Fostering social engagement can mean having students develop a capstone project that
includes community service, pairing or grouping students into groups for classwork, or being
cognizant of a community of inquiry in a program. Student engagement is a critical element in
online education because it can directly affect student retention.
Student Retention
Student retention drives revenue, making it a key concern in online education.
Additionally, attrition rates affect the perceived success of colleges and universities (Haydarov,
Moxley, & Anderson, 2013). However, the variety of challenges that online students face in
virtual learning management systems makes the process of driving retention difficult and will be
discussed below. Student retention has an effect not only on a program’s success, but also on the
university funding that program. Student tuition drives revenue, which in turn fosters growth for
more initiatives.
Challenges to Student Engagement and Retention. The variety of challenges that
online students face in virtual learning management systems makes retention difficult. The
decision that a student makes to withdraw from an online graduate program is a complicated
phenomenon with multiple variables (Rice, Rojjanasrirat, & Trachsel). Students’ rationale for
withdrawing include: a lack of interaction, feelings of isolation, and excessively challenging
course content.
For online degree programs to be successful, universities must understand the factors
affecting how students engage with virtual coursework and programs since retention is more
challenging in this realm (Moskal, Dziuban, Upchurch, Hartman & Truman, 2006). To launch
and sustain online degrees, program directors must understand the issues that typically arise. Part
of the complexities and issues that arise could be ameliorated by understanding existing learning
CREATING EFFECTIVE EDUCATIONAL CONTENT 27
theories and frameworks when guiding discussions about creating effective learning materials.
Learning Theories and Frameworks
Education institutions and organizations encourage online course developers and
instructors to understand the relationship between learning theories and student engagement to
improve attrition rates. Knowledge of theoretical frameworks and expertise in creating engaging
content is vital to developing courses that result in effective online educational programs. Several
of these learning theories and frameworks are discussed below.
Community of Inquiry. To address the complexities of online learning, Garrison,
Anderson, and Archer (2001) developed the Community of Inquiry (CoI), a framework that
outlines the components of online learners’ educational experience. This model is comprised of
three central elements: cognitive presence, social presence, and teaching presence. The
researchers, Garrison, Anderson and Archer (2000), first introduced the model as a means of
demonstrating the optimal use of online technology to facilitate meaningful learning for students.
According to the CoI, learning occurs as a result of the intersection of the three core elements.
Figure 1 illustrates the relationship between the three main presences.
CREATING EFFECTIVE EDUCATIONAL CONTENT 28
Figure 1. Community of Inquiry Framework.
Cognitive presence is the most fundamental element in this model and is aligned with
Dewey’s theory of Constructivism (1910). Constructivism involves four phases of reflective
thought, which are all part of cognitive presence: triggering an event, exploration, integration,
and solution (Garrison, Anderson, & Archer, 2010). That is, critical thinking and meaning-
making through a student’s interaction and ongoing communication within a learning community
are staples of cognitive presence.
Social presence, as defined by Garrison, Anderson, and Archer (2000), refers to how
members (students) of a community interact with their peers. That is, it is the manner in which
the members interact as “real people.”
Teaching presence refers to both the design of the educational experience and the manner
in which it is facilitated (Garrison et al., 2000). Garrison et al. (2000) define the educational
experience as how educational content is organized, designed, developed, assessed and selected.
For example, in a course that has both asynchronous and synchronous components, the
CREATING EFFECTIVE EDUCATIONAL CONTENT 29
“experience” refers to how the asynchronous components are formatted. They might be
comprised of lectures, roleplays, interviews, and quizzes. In contrast, Garrison et al. (2000)
define facilitation as the way in which the instructor and students interact to make meaning of the
learning.
Learning communities. Young and Bruce (2011) also explore how instruction, when
designed to actively involve learners in meaningful tasks, elevates student engagement. Young
and Bruce conducted a survey of 1,410 students across five colleges to measure how online
classroom community and student engagement correlate. A twenty-three question survey
measured classroom community with instructors (8), classroom community with classmates (8)
and engagement in learning (7). The researchers define student engagement as the interest and
motivation students had in their own learning (Young & Bruce, 2011). Young and Bruce (2011)
also found that there is a difference in student engagement across disciplines. For example,
students in colleges of education felt stronger feelings of community with their instructors and
classmates, while students enrolled in schools of business, arts, and science reported weaker
feelings of community with instructors and classmates.
In a study of three universities, Robinson and Hullinger (2008) found that online students
were modestly engaged in selective NSSE dimensions and had a pattern of engagement different
from students taking classes on campus. The researchers contend that general research in online
learning focuses on three key elements: student outcomes focused on test scores and grades,
student attitudes about learning, and overall student satisfaction about learning. Thus, curriculum
design should focus on increasing student-faculty interaction on readings and class-related
materials.
Critical value factors (CVFs). Levy (2008) pinpoints five critical value factors (CVFs)
CREATING EFFECTIVE EDUCATIONAL CONTENT 30
that affect how online students view their educational material. Those factors are (a)
collaborative, social, and passive learning activities (CSLA); (b) formal communication activities
(FCA); (c) formal learning activities (FLA); (d) logistic activities (LGA); and (e) printing
activities (PA). Levy (2008) also noted that there is a difference between how females and males
interact with their peers online.
The Seven Principles of Online Education
Chickering and Gamson (1987) formulated seven principles of good education practice
that create a foundation for active learning. These principles are as relevant today as they were
when they were first developed. Originally published in the AAHE Bulletin (Chickering &
Gamson, 1987), this popular framework provides guidance for evaluating teaching in traditional,
face-to-face courses. Additionally, the seven principles are based on 50 years of higher education
research (Chickering & Reisser, 1993). The seven principles are:
1. Good practice encourages contact between students and faculty.
2. Good practice develops reciprocity and cooperation among students.
3. Good practice uses active learning techniques.
4. Good practice gives prompt feedback.
5. Good practice emphasizes time on task.
6. Good practice communicates high expectations.
7. Good practice respects diverse talents and ways of learning.
According to Phillips (2005), these seven principles provide educators learner-centered
teaching and learning guidelines, not only for a face-to-face classroom, but also in an online
environment. Graham, Cagiltay, Craner, Lim, and Duffy (2001) a team of five evaluators from
Indiana University's Center for Research on Learning and Technology (CRLT), used these
CREATING EFFECTIVE EDUCATIONAL CONTENT 31
principles to examine four online courses from "a large Midwestern university." Graham et al.
(2001) state that faculty members who also taught face-to-face classes taught these courses.
Their evaluation analyzed the online, asynchronous course content, student and instructor forum
postings, and interviews with the faculty. The researchers identified examples of the seven
principles to create a list of "lessons learned" in online teaching. The synthesized list follows.
Principle 1: Good practice encourages student-faculty contact. The lesson for online
instruction is that instructors should provide students with clear guidelines for interaction
(Chickering & Gamson, 1987). Syllabi, contact information, class meeting times, assignment
deadlines, rubrics, and expectations should be clear. Also, students should know how to get
support when they need it. Should they run into technical problems, they need to know that the
instructor is not the first point of contact. For example, instructors should provide the toll-free
number or helpdesk email address to students.
Principle 2: Good practice encourages cooperation among students. The lesson for
online instruction is that well-designed discussion assignments facilitate meaningful
conversation among students (Chickering & Gamson, 1987).
Principle 3: Good practice encourages active learning. The lesson for online instruction
is that students should complete and present course projects. Even in the online world, presenting
course projects is getting easier (Chickering & Gamson, 1987). Most, if not all, web conferencing
software provides a means of sharing screens, conducting discussions, and having breakout
rooms. Also, software such as VoiceThread makes it possible for students to add video and voice
overs to their group projects.
Principle 4: Good practice gives prompt feedback. Online instructors need to provide
two types of feedback--information and acknowledgment (Chickering & Gamson, 1987).
CREATING EFFECTIVE EDUCATIONAL CONTENT 32
Graham et al. (2001) define informative feedback as giving a student more information on their
topic or assignment grade, while acknowledgment refers to a student receiving attention for their
efforts right away. The latter is more effective when done one-on-one, but since professors are
often managing several tasks simultaneously, simple wall posts, emails or messages in an LMS
are also beneficial.
Principle 5: Good practice emphasizes time on task. Online courses need deadlines
(Chickering & Gamson, 1987). Students should not be allowed to be too flexible in their
assignments to avoid student procrastination. While it is true that students have full-time jobs
and responsibilities, and some leeway is acceptable, it is important to be vigilant about the
general guidelines of the course in this regard.
Principle 6: Good practice communicates high expectations. Challenging tasks,
sample cases, and praise for quality work communicate high expectations in an online course
(Chickering & Gamson, 1987). Graham et al. (2001) suggest the use of case-based learning and
providing models for assignments. They also recommend the use of public praising for high-
quality work.
Principle 7: Good practice respects diverse talents and ways of learning. Allow
students to choose project topics to incorporate diverse views in online courses (Chickering &
Gamson, 1987). A part of understanding diverse ways of learning is being able to access data
that gives insight into how students interact with content with learning analytics and data.
Learning Analytics and Data
Video analytics provide valuable insight for the individuals planning and executing
educational content. The data can be used to create engaging content for students and make
informed business decisions about how to allocate money for building and purchasing certain
CREATING EFFECTIVE EDUCATIONAL CONTENT 33
types of content. Educational media content includes tutorials, lectures, roleplays, and interactive
elements that requires students to complete an activity before being able to continue. Each
content type has a different monetary figure attached to it based on the man hours and expertise
necessary to execute creation. However, not all types of content are suitable for the pedagogical
needs of a course. Being able to decipher what is valuable to students and why creates better
course developers and course strategists.
Clickstream data. Aguiar, Nagrecha, and Chawla (2014) analyzed the effects of
clickstream data on predicting engagement data for video content. The researchers most notably
indicate that for certain categories, the percentage of users watching videos in their entirety can
reach as low as 20% (Aguiar et al., 2014). This type of insight into educational media
consumption is beneficial not only to the learners, but also to company personnel who create the
content. With the constant growth and speed of the internet, there is little room to deliver inferior
content (Aguiar et al., 2014). Such growth has increased the demand for content creators to
develop engaging content for consumers, including online students.
Aguiar et al. (2014) successfully predicted online user engagement with video content by
taking several factors into account. The factors include a 10-factor cross-validation on available
user data (Aguiar et al., 2014). This type of knowledge helps content creators produce content
that is meaingfully personalized for their audience.
Video dropouts and interaction in online lecture content. Kim, Guo, Seaton, Mitros,
Gajos, and Miller (2014) analyzed dropout rates and peaks in views as well as student activity in
862 videos for MOOCs on a large, free, online learning platform. The researchers’ findings
conclude that higher dropout rates are to be expected in longer videos, and peaks in viewership
can generally be expected in tutorial style videos, as opposed to lectures.
CREATING EFFECTIVE EDUCATIONAL CONTENT 34
Summary
There are several factors that course developers should consider when creating engaging
content: learning analytics, data, good educational practices, existing educational frameworks
and student values are some key examples. As the industry of educational technologies continues
to grow so does the amount of research done on students. Focusing on learning analytics, for
example, is a popular trend among researchers as they use this data to decipher how students
interact with content. Plus, the ever-increasing amount of content makes learning analytics that
much more relevant and fiscally useful. Course developers, however, should not be overlooked
by researchers and nor should their teaching practices, due to their impact on student learning
experiences.
Clark and Estes’ (2008) Organizational Problem-Solving Framework
The Clark and Estes (2008) modified gap analysis, a systematic, analytical method that
helps clarify organizational goals and identify the gap between the actual performance level and
the preferred performance level within an organization, will be implemented in order to evaluate
Ed Tech Inc. According to Rueda (2011), the gap analysis process “provides a way to clarify
both short-term and long-term organizational goals, assess them, and describe gaps from the
actual levels of performance or achievement to the desired levels” (p. 73). The gap analysis
process begins with defining measurable goals. Determining the organization’s current
performance is the next step in the process. Both the measurable goals and current performance
were presented in Chapter 1. Hypothesizing possible performance-related needs, causes, or assets
is the next step, followed by assessment via data gathering. After needs have been established
and prioritized, the final two steps occur: developing solutions and evaluating outcomes (Clark &
Estes, 2008).
CREATING EFFECTIVE EDUCATIONAL CONTENT 35
Individuals often assume needs, assets, or causes for problems within organizations without
appropriate examination. This jumping to conclusions often leads to the application of
inappropriate solutions or inaction as a result of perceived complexity in solving the problem. As
a result, individuals may omit or misdiagnose causes for performance gaps. As such, a thorough
investigation into the causes of performance gaps should include (a) scanning (informal)
interviews with stakeholders; (b) learning, motivation, and organization/culture theory; and (c) a
review of related literature. What follows is a discussion of the knowledge, motivation, and
organizational assumed influences for the stakeholder performance goal.
Stakeholder Knowledge, Motivation, and Organizational Influences
Knowledge and Skills
General theory. Anderson and Krathwohl (2001) discuss the four major types of
knowledge: factual, conceptual, procedural, and metacognitive. In order to perform successfully,
employees must have the knowledge and skills necessary to complete their job. According to
Clark and Estes (2008), leaders must understand the four types of knowledge and skills to give
their teams the tools needed to complete tasks effectively: information, job aids, training, and
education. Information is knowledge provided to an individual about their job. Similarly, a job
aid is a piece of higher level information that assists employees in solving problems on their
own. Training refers to the “how-to” aspect of a job, while education is the conceptual
understanding a person possesses to help them maneuver unique and unanticipated challenges
(Clark & Estes, 2008). In order for Ed Tech Inc. to be successful in training its course
developers, all knowledge types must be addressed.
This section focuses on the knowledge-related influences relevant to Ed Tech Inc.’s
stakeholder goal of producing content rich with engaging learning strategies. The stakeholders
CREATING EFFECTIVE EDUCATIONAL CONTENT 36
are the course developers, members of the Content Department at Ed Tech Inc. The Content
Group is comprised of subsections: Iteration, Course Strategy, Implementation, Video
Production, and Course Development. The Iteration Team is responsible for updating current
online courses so the information delivered remains relevant for students. The Implementation
Team is responsible for managing digital assets, the Video Production Team takes the lead on
filming content on set. The Course Strategy Team works with professors at a high level to plan
their course content, and the Course Development Team strategizes at a more granular level
about how the content will be produced. This section of the literature review will focus on course
developers.
The course development team works with professors to design and produce course
content using the company’s proprietary online learning tools. Course developers also manage
courses with specific timelines to ensure they launch on schedule for students. Thus, a discussion
of the literature related to the course developer role focuses on instructional design and project
management in technology companies.
Knowledge influences. The knowledge types discussed in this section are conceptual,
metacognitive, and procedural, classifications derived from Bloom’s Taxonomy of knowledge
categories (Krathwohl, 2002). According to Kummaran, Hassabis, and Maguire (2009),
conceptual knowledge is a result of the brain organizing different aspects of its environment in a
manner that allows it to pull past experiences into current situations. Krathwohl (2002) defines
conceptual knowledge as the brain’s ability to categorize information in a manner that allows it
to be applied to generalized settings. Metacognition is an individual’s knowledge and control of
their own memory processes (Baker, 2006). According to Krathwohl (2002) metacognition is the
newest addition to the knowledge categories and involves an individual’s awareness of their own
CREATING EFFECTIVE EDUCATIONAL CONTENT 37
cognitive processes. Procedural knowledge refers to an individual’s understanding of how to
perform a task (Colman, 2016). Each knowledge influence discussed in this section will be
categorized into one of the knowledge types defined above. Each influence must be classified in
order to assess individuals accurately.
Conceptual knowledge. Course developers should understand the tools in the Learning
Management System (LMS) as well as effective instructional strategies that foster student
engagement. Additionally, all course developers should consider problem-based learning as a
means of engaging students. Sendag and Odabasi (2009) found that critical thinking skills and
content knowledge were improved in an undergraduate learning environment when instructional
designers implemented problem-based learning exercises. For example, Kim and Shaw (2009)
discovered that problem-solving activities coupled with collaborative exercises were not only
beneficial in content acquisition, but also gave students a chance to interact and form
relationships. Borokhovski, Bernard, Tamim, Schmid, and Sokolovskaya’s (2016) findings also
support the efficacy of online instructional design that provides student collaboration
opportunities and facilitates the development of a learning community.
Procedural knowledge. Course developers should also know how to build exercises that
motivate student engagement. If a course developer is methodological in structuring group
argumentation activities, the learning can be more effective than when using traditional methods.
In his research 2011, Leong found that having students introduce themselves at the beginning of
the semester and post to discussion boards created a positive social learning environment.
Further, he found that online instructors’ timely responses to discussion posts helped students
feel acknowledged. In support of social online learning environments, Wheeler (2006)
discovered that students appreciate opportunities to collaborate with peers.
CREATING EFFECTIVE EDUCATIONAL CONTENT 38
Knowing how to leverage available assessment tools in designing online courses will also
be critical to the Course Development Team’s success. Assessments are not only important in
gathering evidence of learning outcomes, but also in evaluating student technology expertise.
Students cannot be successful in online courses if they do not know how to use the tools required
to complete an assessment (Milligan & Buckenmeyer, 2008). Even more important is
implementing assessments to gauge students’ comprehension and progress in meeting learning
goals. Shute (2008) determined that formative feedback should be explicit, helpful, and timely
and that formative assessment supports student success.
Course developers should also be adept at incorporating discussion boards and
Computer-Mediated Communication (CMC) aids into online courses. Online forums provide
quiet students an opportunity to contribute and prepare all students for synchronous discussions.
When students engage in focused communication with one another, they improve their discourse
abilities (Leonard, 2011).
Genc and Tinmaz (2016) asked study participants to rate fifteen online learning items,
including video conferencing, videos, exams, and the chat feature in synchronous class meetings.
They found that instructional designers value forum tools more than any other item when
creating online courses. Understanding what tools course developers value most provides a
window into how their preferences affect the ultimate course product.
Metacognitive knowledge. Being self-aware and reflecting on strengths and weaknesses
when developing course content is critical to a course developer’s effectiveness. According to
Baker (2006), metacognition improves learning because it requires individuals to engage their
knowledge about their cognitive processes. This awareness also boosts on-task confidence and
decreases self-handicapping (Jiang & Kleitman, 2015).
CREATING EFFECTIVE EDUCATIONAL CONTENT 39
McAvoy and Butler (2009) examined the detrimental effects of a software team’s
mindless approach to projects. The study found that the Abilene Paradox handicapped their
collaboration. In the Abilene Paradox (Harvey 1988), the complexities of agreeing and
disagreeing during in-group decision-making make it difficult for individuals to speak up in a
critical decision-making period. Therefore, a team might make a group decision based on only
one opinion because the rest of the group does not contribute ideas or feel confident in their
opinions.
Being reflective and self-aware during critical decision-making conversations is crucial in
fostering the development of course developers for online learning. Yusop and Correia (2014)
found that self-reflection, discussions among colleagues, and the completion of educational
coursework were agents of change in the courses instructional designers brought to fruition. So
the course developers at Ed Tech Inc. should make metacognitive reflection an important part of
their practice as they work to produce hundreds of courses every year.
Table 2 shows the three knowledge influences discussed above labeled with their
corresponding category: conceptual, procedural, and metacognitive. As previously explained, the
metacognitive knowledge type plays a key role in the continuous development of effective online
learning design and a strong team because each member must be self-aware and able to articulate
their thoughts when making group decisions. Procedural knowledge is related to the how-to of
processes, while conceptual knowledge involves taking prior knowledge and applying it to
current relevant situations.
Table 2
Assumed Knowledge Influences
Organizational Mission
CREATING EFFECTIVE EDUCATIONAL CONTENT 40
Ed Tech Inc.’s mission is to partner with quality colleges and universities to provide the
world’s best online education to today’s students.
Organizational Global Goal
Ed Tech’s goal is to improve 1 million lives through learning by 2030. It will do this by
creating course content that aligns with the organization's aspiration of being more efficient
and effective in everything that it delivers: from more engaging learning experiences to more
streamlined and adaptable internal processes.
Stakeholder Goal
By January 1, 2019, all Course Developers at Ed Tech Inc. will have the knowledge,
motivation, and organizational resources required to create online coursework with engaging
learning strategies.
Knowledge Influence
Knowledge Type
(i.e., declarative
(factual or
conceptual),
procedural, or
metacognitive)
Knowledge Influence
Assessment
Course developers understand the
LMS tools and effective
instructional strategies to foster
student engagement.
Conceptual Interviews and surveys
Course developers understand
how to build exercises that
motivate students to engage with
the content.
Procedural Interviews and surveys
Course developers are more self-
aware and reflective about their
strengths and weaknesses when
producing course content.
Metacognitive Interviews and surveys
Motivation
The following section focuses on motivation-related influences relevant to achieving Ed
Tech Inc.’s stakeholder goal. Equally important as knowledge-based influences, motivation
keeps individuals on the move to achieve stakeholder goals (Clark & Estes, 2002). According to
Clark and Estes, research has focused on three main motivational aspects: choice, persistence,
and mental effort. Choice refers to whether or not a person makes a decision to be active in their
pursuit of a goal. Persistence is a person’s ability to focus on one task and achieve it. Mental
effort refers to the number of cognitive processes one chooses to dedicate to a specific job.
CREATING EFFECTIVE EDUCATIONAL CONTENT 41
Attribution Theory. Understanding why and how certain outcomes occur after an
individual invests time on a certain task affects how that same person will behave in the future.
The general premise of attribution theory is that people want to understand their environment to
grasp why certain events happen and attribute meaning to them based on external or internal
factors (Anderman, 2009). For example, if an employee receives a low score on their quarterly
review, they will attribute the cause of the failure to themselves or an external factor.
Subsequently, the attribution that they select will affect how they engage in future activities
(Anderman, 2009). Korn, Rosenblau, Buritica, Rodriguez, and Heekeren (2016) found that
people will change their behavior based on the feedback they receive, even if their old behavior
was more desirable. Thus, feedback should be carefully considered.
Course developers should assume that student engagement is due in part to the design of
the content rather than student’s desire to learn.
Self-efficacy. According to Albert Bandura’s (1986) self-efficacy theory, the belief that a
person has about their own abilities plays a substantial role in their performance on any given
task. People with high self-efficacy are more likely to complete a task with confidence and an
expectation of high-performance (Rueda, 2011). In contrast, an individual with low self-efficacy
will doubt their abilities and predict that they will fail (Rueda). Collective efficacy, how a group
perceives their capability and that of their peers, can also affect how a group of individuals
performs (Pajares, 2006).
Table 3 matches the assumed motivation influences discussed above with the proposed
assessments. The first motivational influence listed in the table, attribution, is important to
consider because course developers should assume that student engagement is generally a result
of how well content is or is not designed. The second motivational influence, self-efficacy, is
CREATING EFFECTIVE EDUCATIONAL CONTENT 42
vital to the team because it affects how individuals perceive their own abilities and that of their
team.
Table 3
Assumed Motivation Influences
Organizational Mission
Ed Tech Inc.’s mission is to partner with quality colleges and universities to provide the
world’s best online education to today’s students.
Organizational Global Goal
Ed Tech’s goal is to improve 1 million lives through learning by 2030. It will do this by
creating course content that aligns with the organization's aspiration of being more efficient
and effective in everything that it delivers: from more engaging learning experiences to more
streamlined and adaptable internal processes.
Stakeholder Goal
By January 1, 2019, all Course Developers at Ed Tech Inc. will have the knowledge,
motivation, and organizational resources required to create online coursework with engaging
learning strategies.
Assumed Motivation Influences Motivational Influence Assessment
Attribution: Course developers should assume that
low student engagement is due in part to the design
of the content rather than student’s desire to learn.
Written survey item: “Student
achievement is affected by how I build
content in the learning management
systems (strongly agree-strongly
disagree)
Interview item: What are some of the
causes of low student interaction?
Self-efficacy: Course developers should feel
confident in their abilities to create educational
content that will engage students.
Written survey item: “I feel confident in
my ability to identify content that is
engaging or not engaging” (strongly
agree-strongly disagree)
Interview item: How confident do you
feel about your ability to identify
engaging content and non-engaging
course materials?
Organizational Influences
In addition to knowledge and motivation gaps, there are organizational barriers that
contribute to performance deficiencies. Clark and Estes (2008) state that successful performance
CREATING EFFECTIVE EDUCATIONAL CONTENT 43
goals must stem from the business goals of the organization. Effective leadership drives an
organization’s mission by demonstrating commitment, setting clear goals, and supporting all
those who are part of the institution. A team has a strong chance of being successful if its leaders
are capable, committed, and track attainable performance measures (Holtzman and Anderberg,
2011). Attention, therefore, should be given to the notion that people in an organization will
perform better when they have specific outcomes in mind. Often times, according to Clark and
Estes (2008), organizations’ leaders set specific goals for teams without being clear about how
those goals affect the business’s overall business plan.
Ravanfar (2015) states that there are particular behaviors leaders need to exhibit to earn
and keep the respect of their teams, so they can successfully lead initiatives. That is, motivating
employees and communicating in an effective manner is key to driving change. In addition,
understanding the multifaceted influences that affect how transitions occur is essential to leading
successful change efforts. According to Gilley, Gilley, and McMillan (2009) transitional change
is guided by small, incremental changes to a company’s process.
Cultural models. Only a limited body of research examines the values, beliefs, and
course developers’ attitudes concerning organizational culture connected to implementing ideal
online learning strategies. However, researchers found that time, budget, and audience heavily
influence course developers, more so than educational theory, when planning courses
(Thompson-Sellers & Calandra, 2012). Also, as previously referenced, while course developers
essentially act as an instructional designer, they also act as project managers. Individuals who
hold this position are responsible for overseeing the online course timeline and ensuring its
delivery into the appropriate program’s LMS for the start of the semester.
According to Honebein & Honebein (2015), course developers believe that online course
CREATING EFFECTIVE EDUCATIONAL CONTENT 44
design should be guided by stakeholders’ (e.g. faculty) beliefs. However, course development
decisions should remain unaffected by the influences of faculty members’ beliefs and instead be
guided by their knowledge of effective learning strategies that are also supported by the
organization.
Cultural settings. Corporate culture creates a company’s personality, rules, and
regulations for how employees interact with one another and perform their work. Across much of
the research on corporate culture, the term is defined as “the basic assumptions and beliefs that
are shared by organizational members” (Schein, 1985, p. 9). Therefore, organizations that
provide Software-as-a-Service (SaaS) technology should create a culture of inquiry for their own
methods. Sørensen (2002) explains that “most members work from a shared knowledge base and
common beliefs, which enhances organizational reliability (p. 10).” That is, all organizations
contain an employee base, which creates an opportunity to create a shared set of values within
the workplace. Numerous factors affected by workplace culture, either positive or negative,
makes cultural settings a hot-button issue. Though values may not be documented within an
organization, they still exist.
As previously mentioned, culture is a company asset (or liability in some instances).
Because culture affects an organization’s reputation and financial success, leaders must
understand these dynamics to create a strong environment. Research by Flatt and Kowalczyk
(2008) indicates that culture not only affects financial performance, but also a company’s public
perception, which, in turn, affects financial success. This research pulls data from Kotter and
Heskett’s (1992) culture dataset, reputation rankings from Fortune’s Most Admired Companies
(MAC), and financial data from Dow Jones Interactive in 1986 and 1987. The latter datasets
were then matched to the culture datasets from 1992. The correlations found in the study include:
CREATING EFFECTIVE EDUCATIONAL CONTENT 45
(a) a strong relationship exists between culture and reputation, and (b) culture is related to market
to book cost of the valuation of a company (Flatt and Kowalczyk, 2008).
Course development goals. The goals of any online program should be explicit so that
course developers understand the desired outcomes of the courses they are developing. Kumar
and Swanson (2014) determined the importance of creating courses in an Ed.D. program that are
intentionally aligned with the program’s outcome goals. In addition to having clear goals,
universities and organizations should focus on course developers’ feeling supported throughout
the development period.
Research and university influences. Barker, Hovey, and Gruning (2015) conducted a
study of sixty-six computer science faculty members, who were teaching courses in thirty-six
postsecondary institutions in the United States. The researchers' findings demonstrate that many
professors learn about new teaching strategies by participating in government or corporate-
funded initiatives or when they are motivated to solve a problem, and thus, search for new
teaching methods on their own (Barker et al., 2015). For example, if faculty members find that
students are bored in their classrooms or if they are not quite grasping the content, professors are
more motivated to seek engaging teaching strategies. In terms of government and corporate
initiatives, conferences are one popular method used to inform instructors about innovative
methods. More importantly, Barker et al. indicate state that, in some cases, faculty buy-in is best
routinized when granting agencies for pilot projects are institutionalized (2015).
Research is the main barrier to implementing reform in online education. Indeed, faculty
members who learn new strategies for their courses do so at "their own risk” (Barker et al.,
2015). That is, any additional strategies or information that invest time in acquiring is typically
done on their own accord. Interestingly, research faculty members are not likely to use findings
CREATING EFFECTIVE EDUCATIONAL CONTENT 46
from educational research as a basis for changing their teaching techniques (Barker et. al., 2015).
Rather, they are most influenced by the expectations of their university, its policies, the
perceived cost and benefits (both for themselves and for their students), and role models (Barker
et. al., 2015). Role models refer to any person in a professor's network who is a respected teacher
with proven results (e.g., stellar student evaluations), someone who is a respected researcher in
their field, or even their own former instructors (Barker et al., 2015). In terms of cost-benefit
analyses, Barker et al. (2015) stated that faculty either explicitly or implicitly weigh both costs
and benefits of using new methods in their courses before implementing them. Some costs might
be related to the amount of time spent on creating knowledge check-in quizzes for students
versus spending time on research, which brings money into the school.
Accountability. Course developers need to feel accountable for creating content that
utilizes engagement strategies in online learning. According to Hentschke and Wohlstetter
(2004), the most simple definition of the term “accountability” is a “contractual relationship
between two parties:” a director and a provider. The provider brings the “good or service,” while
the director has the power to either punish or reward (Hentschke and Wohlstetter, 2004 p. 11).
The application of accountability structures to the course development process should be
considered a means to ensuring intentional strategies are used during development.
Table 4
Assumed Organizational Influences
Organizational Mission
Ed Tech Inc.’s mission is to partner with quality colleges and universities to provide the world’s
best online education to today’s students.
Organizational Global Goal
Ed Tech’s goal is to improve 1 million lives through learning by 2030. It will do this by creating
course content that aligns with the organization's aspirations around being more efficient and
effective in everything that it delivers: from more engaging learning experiences to more
CREATING EFFECTIVE EDUCATIONAL CONTENT 47
streamlined and adaptable internal processes.
Stakeholder Goal (If Applicable)
By January 1, 2019, all Course Developers at Ed Tech Inc. will have the knowledge, motivation
and organizational resources required to create online coursework with engaging learning
strategies.
Assumed Organizational Influences
Organization Influence Assessment
Cultural Model Influence 1: There is a general
belief among course developers that online
course design should be guided by
stakeholders’ (e.g. faculty) beliefs.
Course developers are confident in leading
conversations about online course design.
Cultural Model Influence 2: Time, budget, and
audience heavily influence course developers,
more than educational theory when planning
courses.
Course developers support focusing on
instructional design methods and course
outcomes throughout the project.
Cultural Setting Influence 1: Course
developers are affected by organizational
culture, department goals, and faculty
influences.
Professional development provides training
opportunities and supports the implementation
of online learning strategies in course content..
Cultural Setting Influence 2: Course
developers are not accountable for creating
courses with effective online learning
strategies.
Course developers have accountability
structures in place to ensure their work aligns
with course outcomes.
Conclusion
According to the literature, student engagement and its effect on retention rates is not
easy to discern. Several components contribute to a student’s experience in an online
environment, combining to deliver a quality and effective experience. Chapter 2 attempted to
outline the effects of student engagement on retention rates in online programs and how
stakeholder knowledge, motivation, and organizational factors influence the university partners’
achievement of student engagement and high retention rate goals. According to Clark and Estes
(2008), four different types of knowledge and skills are important for leaders to understand to
provide their teams the tools needed to complete tasks effectively: conceptual, procedural,
metacognitive, and conceptual knowledge. Addressing these skills within an organization’s
course development group are vital first steps to creating positive organizational change as the
CREATING EFFECTIVE EDUCATIONAL CONTENT 48
organization continues to grow. Additionally, the organization must be mindful of motivation
when training course developers. This chapter focused on attribution and self-efficacy: course
developers should attribute student engagement to their content development methods, and they
should feel confident in applying their knowledge when working with faculty. When considering
organizational influences, organizations should be mindful of the culture created to allow course
developers to work collaboratively, of the influences of timelines and audience on course
developers, and of the faculty influence on course developers when creating content. Finally,
accountability is an important structure to consider when thinking about evaluating course
developers’ work.
CREATING EFFECTIVE EDUCATIONAL CONTENT 49
CHAPTER 3: METHODOLOGY
This study was designed to explore the perceptions of and practices used in the
incorporation of effective engagement strategies for online course development. The research
concerning elements of effective online learning strategies were presented in Chapter 2.
Assumed knowledge, motivation and organizational influences that impact course developers’
ability to create online courses with engaging learning strategies were also highlighted. The
assumed KMO influences include
● declarative, procedural, and metacognitive knowledge;
● attribution, self-efficacy, and goal orientation theories of motivation; and
● organizational goals, support, culture, and research and university influences.
Participating Stakeholders
This convergent, parallel, mixed methods study used course developers as the stakeholder
because they work closely with professors to create and produce online content. The research for
this dissertation will use purposive sampling.
Survey Sampling and Criterion and Rationale
Criterion 1. Course developer was employed with Ed Tech Inc. as a permanent
employee, as reported through organizational documentation.
Criterion 2. Employed by Ed Tech Inc. for at least sixty days, as reported
through organizational documentation.
Survey Sampling (Recruitment) Strategy and Rationale
Purposive sampling was used to indicate the circumstances affecting how course
developers build and plan online courses in a degree program. The strategy was purposive
CREATING EFFECTIVE EDUCATIONAL CONTENT 50
population because each course developer has an impact on how courses are produced more
generally. Their influence can affect protocol and processes implemented across the entire team.
Interview Sampling Criterion and Rationale
Criterion 1. Course developer that was employed with Ed Tech Inc.as a
permanent employee, as reported through organizational documentation.
Criterion 2. Employed by Ed Tech Inc. for at least sixty days, as reported
through organizational documentation.
Interview Sampling (Recruitment) Strategy and Rationale
A purposeful sampling approach was used in this instance. Course developers were asked
to participate in non-incentivized, one-on-one, semi-structured interviews that included questions
about knowledge, motivation, and organizational elements. The interviews lasted about one hour
in duration and were primarily conducted via web conference because the team is dispersed
across the country; interviewing team members face-to-face would have been costly and
challenging. The web conferencing software was also used to record audio and visual interview
components.
Conceptual Framework: The Interaction of Stakeholders’ Knowledge and Motivation and
the Organizational Context
According to Maxwell (2013), the purpose of a conceptual framework is to outline the
theory of one’s research, even if it is temporary and incomplete. While the potential influences
have already been listed, they do not exist in isolation from one another. The diagram below
illustrates how course developers create asynchronous materials as well as the knowledge,
motivational, and organizational influences that affect how these materials are created.
CREATING EFFECTIVE EDUCATIONAL CONTENT 51
Figure 2. Conceptual Framework.
The figure above represents the stakeholder group of this study: course developers. The Course
Development Team is influenced by factors within Ed Tech Inc. and the university partners’
professors with whom they work. Their development decisions should, however, be primarily
affected by effective engagement strategies for online students.
Data Collection and Instrumentation
A convergent, parallel, mixed methods approach was used to understand the feelings,
values, and perceptions related to how the course developers produce online courses. This is the
preferred method for this study since it merges both quantitative and qualitative data and aids in
CREATING EFFECTIVE EDUCATIONAL CONTENT 52
developing a deeper understanding of the knowledge, motivation, and organizational influences
on the Content Department.
The triangulation of methods and sources created a more thorough study of the setting
due to the inclusion of both interviews with and surveys by the Course Development Team.
According to Maxwell (2013), triangulation of data means cross checking results using several
types of data. By employing the strengths and weaknesses of multiple methods, a researcher has
a better chance of using data that lead to the same conclusion (Maxwell, 2013). To make sense of
the qualitative and quantitative data, relationships and common themes were identified between
course developer survey responses and one-on-one interview answers. This analysis provided a
deeper understanding of the course developers’ knowledge and motivation influences.
Using qualitative and quantitative measures for this study created challenges in terms of
reliability and validity. Thus, both concepts are discussed in the data collection methods outlined
below. Maxwell (2013) found that researchers’ inattention to validity is a common characteristic
of ineffective research proposals. Validity must be considered from the first step in research
design. The concept of validity is controversial for qualitative researchers because many question
its suitability for non- quantitative research (Maxwell, 2013). This controversy has generated the
use of terms like “trustworthiness” in the field. However, validity of quantitative data is less
disputed because researchers have control groups, randomized sampling, and other techniques to
ensure data accuracy (Maxwell, 2013).
Surveys
An online survey about knowledge, motivation, and organizational influences was
distributed to 14 potential participants. Twelve of the 14 course developers responded. The
survey included multiple choice and short-answer questions. According to Fink (2013), multiple-
choice questions are often the best method for surveys because they are the most reliable and
CREATING EFFECTIVE EDUCATIONAL CONTENT 53
efficient question type because a researcher can easily score and gather the needed data. This
survey, which used a continuous scale (strongly disagree to strongly agree), was sent to course
developers via a program called Qualtrics.
According to Creswell (2008), the three types of validity are content validity, predictive
or concurrent validity, and construct validity. Content validity relates to whether or not the
collected data measures what is intended. Predictive or concurrent validity relates to whether or
not scores foresee a criterion measure. And finally, construct validity refers to whether or not the
data measures hypothetical concepts (Creswell, 2008). To ensure reliability, a pilot test was
conducted before the study began to determine whether the research design was suitable to
proceed.
Interviews
After completing the survey, the 12-member Course Development Team was asked to
participate in non-incentivized, one-on-one, semi-structured interviews. According to Merriam
and Tisdell (2016), semi-structured interviews are beneficial to a research study because they
allow for flexibility. Additionally, the researcher can respond to the interviewee and create a
natural flow between each question (Merriam & Tisdell, 2016). Patton (2002) refers to this type
of interview as a standardized open-ended interview and notes that, in addition to the benefits
that have been outlined, these interviews also ensure each participant has a similar experience
with the interviewer.
The interviews for this study were approximately one hour in duration and primarily
conducted using web conferencing software. The web conferencing software was also used to
record audio and visual components of the interviews.
During the recorded interview, notes were also taken by the researcher. According to
CREATING EFFECTIVE EDUCATIONAL CONTENT 54
Bogdan and Bilken (2007), transcribing is important because without an adequate understanding
of how to code a large number of recorded interviews, the researcher may end up with a large
amount of recorded data that is impossible to code later.
Data Analysis
An analysis of the data presented in the results and findings section was used to inform
the research questions and make recommendations in Chapter 5. Per recommendations made by
Singleton and Straits (2005), the data was mapped back to the literature to interpret surveys and
interviews.
The first phase of approaching the survey data was analysis. Since the survey was
distributed via Qualtrics, all quantitative data was gathered in the program. Qualtrics computed
descriptive statistics, but I preferred to export the data and use Excel instead. I created a code
book to gather, store, and analyze codes found in the surveys and interviews.
In addition to using a codebook for interviews, short analytic memos were created after
each discussion to capture post-interview reflections, ideas, and thoughts as the work progressed.
Additionally, I analyzed the data in tandem with the collection of additional information from
interviews. Doing so helped to establish themes based on the conceptual framework and to keep
the content organized.
As data collection and analysis progressed, categories and themes were identified and
compared to categories created during earlier interviews. In other words, the initial analysis was
inductive, while the latter part of the process was more deductive as the point of saturation
occurred because no new themes emerged. An analysis of the data presented in the results and
findings section was used to inform the research questions and make recommendations in
Chapter 5. Ensuring data quality is critical to the findings of a study (Merriam & Tisdell, 2016).
CREATING EFFECTIVE EDUCATIONAL CONTENT 55
Without credibility, trustworthiness, validity, and reliability the data will be questionable, which
decreases the strength of the study.
Credibility and Trustworthiness
While quantitative research is typically recognized by its use numbers for data,
qualitative data still make use of them as well. According to Miles, Hubberman, and Saldana
(2014), it is vital for a researcher to know when counting is used and when it is acceptable to
identify frequencies self-consciously. Miles et al. (2014) stress the importance of using numbers
because qualitative researchers can easily make generalizations and assumptions based on how
frequently they encounter certain topics or words in their data.
Careful attention was placed on identifying secondary sources of evidence that contribute
to the creation of identified patterns. As Miles et al. point out (2014), data should be approached
with a certain amount of skepticism--both by the researcher and colleagues.
An interview guide was also used when conducting interviews with participants.
According to Patton (2002), interview guides help focus interviews so that the time with the
subjects is used efficiently. Additionally, Patton (2002) states that outlining questions in detail
ensures that every participant is asked the same questions or given the same stimuli in the same
order and with equal probing. While the interview questions were all open-ended, standardizing
them gave participants the freedom to answer openly but with strong guidance from their
interviewer.
The questions that were asked in the interview process were tested with a pilot group
before the research was conducted. Experimenting with the questions and reassessing those that
might confuse participants ensured that subjects had the clearest content possible. If members of
CREATING EFFECTIVE EDUCATIONAL CONTENT 56
the experiment group had trouble answering the questions or struggled with the meaning of
certain questions, changes were made before the questions were given to the study participants.
With qualitative research, the researcher is the key instrument in data gathering and
analysis. Therefore, all collected data was subject to interpretation not aligned with the subject’s
intent during the observations and interviews (Merriam, 2016). Researchers can use respondent
validation, peer examination, reflexivity, and explicit statement of their own biases to minimize
misinterpretations (Merriam and Tisdell, 2016).
As previously mentioned, data triangulation was implemented. Respondent validation or
member checks were also used to ensure validity. Merriam and Tisdell (2016) state that the
process of conducting member checks involves sharing preliminary findings and the researcher’s
analysis with the participants to verify that what was deduced is consistent with what the
participants’ intended meaning. Therefore, all interview participants received a copy of their
transcripts to verify the accuracy of the interview. They were also asked to participate in follow-
up interviews to discuss the findings as well as the transcripts that were provided.
Finally, the researcher used a reflexive journal to consider how existing biases about
online learning strategies might affect data analysis. This strategy was meant to reduce
subjectivity and to was especially important since the researcher is employed at the organization
where the research was conducted.
Validity and Reliability
According to Fink (2013), the four essential components of creating surveys that produce
accurate information are sampling and design, data processing, pilot testing, and response rate.
The survey questions for this study were formatted as forced-choice questions. Fink (2013)
encourages careful thought about the questions asked of respondents. Therefore, the questions
CREATING EFFECTIVE EDUCATIONAL CONTENT 57
were directly tied to the conceptual framework used for this study. In terms of sampling and
design, the research utilized a cross-sectional method across all members of the Course
Development Team. During data analysis, the researcher strived to find relationships between
knowledge, motivation, and organizational factors and how online courses are created.
Additionally, to address the concerns that Fink (2013) highlights regarding pilot testing and
response rates, a focus group was conducted to ensure that all survey questions were suitable for
the research questions and would collect accurate results. Pilot test respondents were asked to
inform the researcher if they had a difficult time completing the survey. This step was crucial to
the data collection process because questions that could adversely affect the data’s accuracy were
removed immediately (Fink, 2013). Fink (2013) also explains that making a survey user-friendly
is essential to gathering reliable and valid data. Therefore, all quantitative responses were
recorded via Qualtrics, which is easy to use and accessible via mobile and desktop applications.
As discussed in the previous section, noting patterns in data is not an appropriate
mechanism to determine whether or not the data is valid. Instead, a researcher should approach
the results of a study with skepticism (Miles et al., 2014). Miles et al. (2014) also maintain that
assuming study findings are valid simply because they are plausible is dangerous. ndividuals
are inclined to make inferences based on how reasonable or believable they are thus, the
researcher approached the data with a degree of doubt and sought evidence from the qualitative
measurements to draw the same conclusions.
Ethics
All research is concerned with providing data that is reliable, trustworthy, and ethical
(Merriam & Tisdell, 2013). In order to conduct ethical research, privacy matters were considered
from the start of the study. Understanding the study’s ethical implications was crucial since
CREATING EFFECTIVE EDUCATIONAL CONTENT 58
much of the data was gathered from interviews. Informed consent forms were provided to all
study participants. According to Merriam & Tisdell (2016), informed consent must be
considered before beginning a study as it empowers research participants and clarifies that
participation is voluntary, they may choose to end participation at any time, and some aspects of
the study could affect their well-being (Glesne, 2011).
The study was submitted to the University of Southern California Institutional Review
Board (IRB) to ensure safety for participants. All IRB rules and regulations were adhered. Due to
the participants’ employment status, strict confidentiality was maintained and since the
researcher is a manager in the Course Development Department, no direct-reports were invited to
participate in the study. Additionally, no added incentives were offered, so participants would
not feel coerced. However, at its conclusion, thank you cards and small non-monetary gifts of
appreciation were sent.
Limitations and Delimitations
The time frame was a notable limitation of the study, especially since the interviews and
surveys were conducted when the organization was undergoing considerable transition. For
example, at the conclusion of the study, additional course developers were hired, more university
partners signed on, and a significant change was underway.
Conducting the study using Zoom may have made some course developers uneasy about
the privacy of their responses. While their privacy was guaranteed by the researcher, it is
possible they were not fully confident with this assurance.
Some survey questions could have been more elaborate or more focused on the KMO
influences. During data analysis, it was noted that there were not enough questions about
organizational influences. While interview questions satisfied the requirement for data on this
CREATING EFFECTIVE EDUCATIONAL CONTENT 59
topic, more multiple-choice survey questions about the organization and resources would have
balanced the survey given the number of questions about knowledge and motivation influences it
contained.
Biases were also a potential problem because the researcher was an employee and senior
manager at the organization.
Finally, this study does not include the survey results, interviews, or learning analytics
that involve online students. Including such data would require more time and resources but
could also provide valuable insight into why a student would or would not engage with certain
content.
CREATING EFFECTIVE EDUCATIONAL CONTENT 60
CHAPTER 4: RESULTS AND FINDINGS
This study evaluates the perceived knowledge, motivation, and organization (KMO)
influences among Ed Tech Inc.’s course developers and how these influences impact their ability
to produce course content that uses engaging learning strategies for online learning. The
questions guiding this study were:
1. What are the knowledge, motivation, and organizational (KMO) influences that impact how
course developers structure asynchronous content using student engagement strategies to
create more engaging learning experiences for students?
2. How should organizational practices in the areas of knowledge, motivation, and
organizational resources shift to facilitate the use of engaging learning strategies in course
development?
In Chapter 2, assumed knowledge, motivation, and organizational influences impacting
course developers’ abilities to plan courses with efficacious learning strategies were verified
against theories and related literature. Chapter 3 described the research methodology and
validation process used to validate assumed needs. This chapter presents key findings concerning
Ed Tech Inc. course developers’ knowledge, motivation, and organizational resources.
Participating Stakeholders
Purposive sampling was used to select course developers for this study. This chapter
describes the participants and findings from the surveys and interviews, including a 25-item
survey comprised of 20 fixed-response questions and five open-ended questions disseminated
between August 7, 2017, and November 17, 2017. The sample response rate was 85 %; 12 out of
14 invited participants responded. Table 2 represents the demographics of the course developers
who participated in the study: three participants were male and nine were female.
CREATING EFFECTIVE EDUCATIONAL CONTENT 61
Table 5
Gender Identity
Percentage n
Male 25% 3
Female 75% 9
Total 12
Surveys
In the first phase of data collection, self-administered questionnaire data was collected
from Ed Tech Inc. course developers through an online survey. The survey assessed the
significance of knowledge, motivational, and organizational influences that impact their ability
to plan online courses with engaging learning strategies. Participants were selected based on the
following sampling criterion: (a) employed with Ed Tech Inc. for at least 60 days, and (b) is a
course developer for Ed Tech Inc. who is currently planning with, or has planned with, Ed Tech
Inc.’s proprietary course planning software. Table 6 outlines the survey data collection timeline.
Table 6
Survey Recruitment Communications
Survey
Day
Description Sent From
0 Survey opened in Qualtrics N/A
0 Investigator sent an email to the 14 potential
participants outlining information included in
survey information sheet.
USC email server
0 Investigator sent a link to the 14 potential
participants.
Qualtrics distribution email
CREATING EFFECTIVE EDUCATIONAL CONTENT 62
19 Investigator sent a reminder email to 14 potential
participants.
Qualtrics distribution email
48 Investigator sent a reminder email to 14 potential
participants.
Qualtrics distribution email
91 Investigator closed survey Qualtrics distribution email
Interviews
In addition to survey data, one-on-one in-depth interviews were conducted. A total of 12
participants were interviewed for up to an hour. All interviews were conducted and recorded
using Zoom software. The interviews were strategically held between October and December
2017, the end of the fourth quarter and the end of a strategic departmental reorganization.
Results from the survey data and themes from the one-on-one interviews are presented in
this chapter. All participants that completed the survey were invited to participate in the
interviews. The results and findings refer back to the literature. This chapter is organized into
three parts: knowledge, motivation, and organization.
Results and Findings Regarding Knowledge
According to Clark and Estes (2008), conceptual, procedural, and meta-cognitive
knowledge is necessary to improve an individual’s performance. This study launched with the
hypothesis that course developers need knowledge to understand online learning tools and
engaging learning strategies (conceptual), how to build exercises that are made to motivate
students and engage them with the learning content (procedural), and how to be self-aware and
reflective about their strengths and weaknesses when producing course content (metacognitive).
Kuh and Vesper (2001) explain that when online course developers use strategies designed for
online learners, they are more likely to engage students in responding to online discussions and
lecture viewing. During data collection and analysis, an additional theme emerged: a need for a
CREATING EFFECTIVE EDUCATIONAL CONTENT 63
shared definition of what student engagement means (declarative).
Conceptual knowledge
Active learning. The data showed revealed most course developers instinctively have
knowledge about creating effective courses. In addition, they value the concept of active learning and
aim to include interactive components in their course planning. Figure 3 shows that 50% of the
participants strongly agreed that developing asynchronous activities is important in engaging online
learners; the other 50% agreed.
Figure 3. Developing asynch activities is important to engaging an online learner.
Phillips (2005) states that active learning strategies are crucial in online education, including creating
multiple methods for students to interact and creating activities that prompt the students to apply
knowledge. The results in Figure 4 also support the notion that course developers instinctively try to
use the tools in ways that encourage online students to interact with their asynchronous coursework.
CREATING EFFECTIVE EDUCATIONAL CONTENT 64
Figure 4. Do you try to encourage the use of online discussions in the Learning Management
Systems?
In the survey, 41% of respondents strongly agreed that they encourage the use of online
discussions in course work, while 33% agreed. Only 16% remained neutral and 10% disagreed.
While the survey provides general knowledge about how course developers design active
asynchronous activities, such as online discussions, written responses in the survey offered more
in-depth information about the types of activities that course developers include as well as their
rationale for that inclusion. For example, one course developer stated, “I think the questions in
asynch make students stop and think before they answer. They might need to go back and re-
watch a lecture if they don't understand. Without the questions, student might be inclined to skip
lectures.” Another course developer wrote, “I think that requiring students to answer questions
invites them to be an active participant in the learning experience and helps provide some
facsimile of the classroom.” These responses show that the developers intentionally include
opportunities for online learners to engage with online learning modules rather than being a
passive audience. Another course developer stated that requiring students to answer questions in
CREATING EFFECTIVE EDUCATIONAL CONTENT 65
between watching videos online, “encourages (the) student to stay tuned-in.” And, another said
that, “(questions) encourage (students) to reflect on asynch learning and think critically before
submitting a response.” Yet another developer said that “(activities) force the students to think
about the question and generate a response based on a specific topic.” And, finally, another
course developer stated that they thought adding interactive questions to asynchronous content is
useful, “because it's more involved and allows them to actively think and respond to the content
instead of just passively viewing it.”
In one-on-one interviews, responses offered even more insight about the types of active
learning activities that course developers use. For example, one course developer shared “(My)
feeling is by including…asynchronous activities within the course plan, that it does encourage
student engagement, that they're not just watching lectures and getting through it, they're actually
engaging with the content. They're discussing things with their classmates.” Encouraging
discussions among classmates is one of the tenets of active learning, and the interview responses
show developers actively implementing this strategy.
Another example of how course developers use active learning strategies was revealed by
this respondent who valued specific questions relevant to the course work: “Having (students)
answer questions can be considered engagement, but if it's a basic question or a question that
they needed to answer when they haven't taken the course, it's not specifically engaging for that
course.” A final course developer stated: “You want to have something that engages them and
forces them to interact (with the asynchronous content), rather than having them just stare at
(video) content for 90 minutes."
Procedural Knowledge
Course developers should know how to build online courses that motivate student content
engagement. However, based on the data, most course developers have a different perception of the
CREATING EFFECTIVE EDUCATIONAL CONTENT 66
procedures that should be used to build online courses. One course developer stated that the course
development process is “very nebulous. (It is) working with the faculty in some way to develop a
structure or a course, probably.” Another stated that “(course development is) just more of guiding,
because I don't have a lot of experience. So it's guiding them through our tools.” Assessment, a
critical aspect of any learning cycle, was addressed during data collection with the course developers.
Assessments. Course developers reported that they understood how online assessments work
to engage online students.
Figure 5. I understand how online assessments work to engage students.
Figure 5 illustrates the results of the survey in which 66% of respondents agreed that they
understand how online assessments can engage students while 25% remained neutral and only
one strongly agreed. However, it is important to note that the survey only collected a self-report
of the knowledge type and not their actual knowledge, which is why interview data is so
important.
In the one-on-one interviews, where assessment was discussed in more depth, course
developers did not generally express extensive knowledge about how to plan assessments with
CREATING EFFECTIVE EDUCATIONAL CONTENT 67
faculty members. They relied on professors to lead the assessment-creation process. For
example, one course developer said, “some of the faculty I've worked with, they have said they
thought that (strategizing ahead of time with a member of the strategy team, not the course
development team, at Ed Tech Inc.) was helpful because they could think through like the course
objectives and the goals and assessments.” However, since the course developer was not in the
strategy discussions with the professors, they were only able to deliver on what the faculty and
the strategy team felt were the best practices and were unable to offer additional knowledge.
Another course developer stated, “when (they) start talking about assessments, going back again
to the (strategy report is helpful).” But the course developer was unable to provide additional
information about how the assessments should be built. In the same vein of not knowing how to
build course assessments, a developer stated:
There's so many times I think we get in to it, we have our process, (and everybody) has
their process…in some ways it would be a little more creative and freeing if (professors)
were like: ‘Okay, well, we want to have these projects and assignments and activities,
and then we'll fill it in with the asynch.’ We'll fill in everything else around it.
This comment illustrates how this developer’s knowledge is process-focused rather than
solution-focused.
Video lectures. Course developers confirmed that they know how to plan video content
that engages students. Figure 6 shows that 66% strongly agreed that they understand how to use
video lectures to engage students, 25% agreed and one remained neutral. However, the survey
responses only reflect perceived knowledge, not actual knowledge.
CREATING EFFECTIVE EDUCATIONAL CONTENT 68
Figure 6. I understand how video lectures work to engage students.
In one-on-one interviews, course developers said they tend to follow some best practices
for planning recorded content, but there were several discrepancies in responses. For example,
one course developer stated that they often get questions from faculty about why the guidance
from the Ed Tech Inc. team is that recorded content should “be 8 to 10 minutes,” and course
developers have “no idea” how to answer that question. The department’s best practice stipulates
keeping videos at around that length but does not provide a rationale for the length. In fact,
research actually states that 6 minutes or under is an ideal video length for educational content
(Guo, Kim & Rubin, 2014). In a similar comment, another course developer stated:
(I say) lectures should be between 8 to 12 minutes, and I always say (in planning calls)
that, but I don't really know, like I know that that's something hopefully that's backed
supported by data, but I'm just kind of like this is what we try to keep it as within this
timeframe, so that it's easy to consume by students and not too long and you know,
keeping the students engaged and retaining information. But I don't have like, because
this, this research report says this or because we found in our other programs that this is
CREATING EFFECTIVE EDUCATIONAL CONTENT 69
before this, after 12 minutes students fall off. So just kind of like guesses, kind of like just
faking it till you make it. I feel like a little bit is what we've been doing.
A third course developer, when asked what type of video content is effective, said, “it's a
difficult question to answer, and I typically am BSing my way through it.” Based on the
experiences the course developers shared, a considerable knowledge gap exists in their
understanding of how to create effective online learning videos.
Live session planning. Live sessions are another piece of the course development puzzle
that seems nebulous to course developers. During one-on-one interviews, the live sessions
generally came up as an important part of creating a holistic online experience for students, but
most course developers expressed a knowledge gap in understanding how the sessions work to
engage students and how to build effective live session lesson plans with professors.
Course developers mentioned that when they outline their current live session lesson
plans, they typically try to match them to what is covered in the asynchronous video content. .
For example, one developer said:
Essentially, the asynchronous materials [are] a foundation for the live session that (they
make sure) it's not disjointed, and the synchronous session isn't simply another lecture. I
don't want them to plan just willy nilly because there's so much to plan that it's easy to get
lost.
This example is typical of how course developers discussed and characterized live sessions--as
part of the holistic experience. Yet they did not have additional strategies to increase the
engagement level of live sessions. Another developer said of live sessions, “that is probably one
area I still don't feel as confident as I would like with it. I know at least right now how I'm
handling it, as we are going through planning, I kind of weave it in. I talk about connecting the
CREATING EFFECTIVE EDUCATIONAL CONTENT 70
async to the sync. And if there is that connection, I say, "Okay, let's put a note right here in the
live session area to remind you that that…exercise or that whatever exercise is something that
you want to bring up again in live session.” This also illustrates how, while course developers
acknowledge that live session plans should match the learning objectives that the asynchronous
content covers, they do not have any specific strategies for achieving that goal. Another course
developer said, “I'm a little bit at fault of focusing a little bit too much on asynchronous at first
and then later on saying, you know, we also have these live session planning sessions. You
should also be filling out.” Another course developer referenced live session planning as,
“helping (faculty)” and said that, “it's just more of guiding, because I don't have a lot of
experience. So it's guiding them through our tools.”
Templates and course flow. Templates and course flow were not addressed in the
survey questions and arose as an emergent theme during one-on-one interviews. Most course
developers create and use a template to guide how they develop courses with faculty. Templates
support the development process because they structure what should be included in each week of
the course to achieve specific learning objectives. Templates also create a consistent course
flow, so students can easily acclimate to to how a course is organized and know what to expect
each week. One course developer explained, “[during the] final meeting, I like to go through the
course overall. Talk through it and make sure that flow is there, it feels good to the professor.”
And another said, “I feel like we kind of develop a flow, we have an idea of what a typical week
looks like, we can use that information to plan the rest of the weeks.”
Metacognitive Knowledge
Course developers should be self-aware and reflective of their strengths and weaknesses
in the course creation process. Thus, possessing metacognitive skills across the Course
CREATING EFFECTIVE EDUCATIONAL CONTENT 71
Development Department would mean all course developers have the ability to monitor and
regulate their comprehension and development process.
Reflecting on content as a student. Some course developers explained that they think
about course content through a student lens, either by putting themselves in the user’s shoes or
by gathering anecdotal information about student course experiences from colleagues. One
course developer stated that they, “always try to watch professor's content through a student's
eyes and (think) ‘like, am I finding this engaging, is this something I'm interested in and feel like
I'm learning from?”
Another stated:
I try to think through how they've set up (content) in their course. So if it's like there, they
have a lecture, and they're wondering if they should do like a file upload I’ll talk through
it and say, ‘OK. So students are going to go through this lecture. They're going to hear
you talk about this. Then they'll see this car and then they can upload the document and
then they'll see, you know, x, y, z and continue on.’ So just kind of giving them a student
perspective and then, um, yeah, that's pretty much what I do some, uh, interesting times
for the content team.
The interview data shows that course developers actively think about how their how students will
process content and how they can make the connection between what they perceive students’
experience will be in their own practice.
Declarative Knowledge
Finally, while survey data demonstrates that most course developers feel confident in
leveraging online engagement strategies, interview results reveal slightly differing regarding what
student engagement is. Figure 7 is a visual representation of the qualitative data that collected on
course developers’ definitions of student engagement. As the graphic illustrates, 90% of the
CREATING EFFECTIVE EDUCATIONAL CONTENT 72
respondents view completion of asynchronous activities as a student engagement activity. And 27%
of respondents said they view students’ video drop-off rates as an engagement indicator However,
they acknowledged that because the learning management system does not gather data about whether
or not students are actually watching the videos, this is a metric they would consider using when, or
if, it becomes available. An equal number of course developers, 27%, said they view participation in
live sessions as a form of engagement and 18% described grade data as a vital engagement metric.
Figure 7. How do you define student engagement?
Synthesis
Course developers reported having conceptual knowledge about what makes for engaging
online course content. However, their procedural knowledge for how to apply these concepts was
lacking, especially as it relates to building effective assessments and developing live session
plans alongside asynchronous content.
In terms of metacognition, course developers generally thought through the course
development process from a student perspective or through the lens of their coworkers that are
CREATING EFFECTIVE EDUCATIONAL CONTENT 73
students in online programs. Thinking this way helps to direct their planning process. Finally,
the data reveals that course developers need a consistent, shared definition student engagement.
Results and Findings Regarding Motivation
It was initially stated that course developers need to understand student engagement as
resulting from the course development methods they employ, and that they should feel self-
efficacious in their abilities to plan engaging content.
Attribution
Course developers should assume that any effects on student engagement are due, at least
in part, to how courses are developed.
Student engagement. Course developers should attribute successful student engagement
to how courses are planned with professors.
CREATING EFFECTIVE EDUCATIONAL CONTENT 74
Figure 8. Student engagement is affected by how I plan content with professors.
As Figure 8 indicates, 66% of participants strongly agreed that student engagement is affected
by how they develop content with professors, and the remaining 33% of developers agreed. The
data shows that they value their planning process with faculty as a means of developing engaging
content for students. But course developers should also be cognizant of how courses are built
after the planning process. The survey results in Figure 9 indicate that 41% of course developers
strongly agreed that they attribute student engagement to how the final course product is built for
students and 50% agreed. One person did not respond to the question, so their answer was not
factored in.
CREATING EFFECTIVE EDUCATIONAL CONTENT 75
Figure 9. Student engagement is affected by how we build content in the learning management
systems.
Figure 10’s data shows that most course developers are motivated by creating engaging learning
experiences for students. In the selected-choice survey item, 75% of course developers identified
that creating learning experiences for students was a driving factor in how they plan content.
Interestingly, 75% of course developers also identified that creating adequate online learning
experiences for students is important. In this survey item, organizational influences were also
identified; These will be discussed in the organization section that follows.
CREATING EFFECTIVE EDUCATIONAL CONTENT 76
Figure 10. What, if any of the items below, drives the means you use to plan and produce
content with professors? - selected choice
During one-on-one interviews, course developers expanded on their views about student
engagement’s relationship to course planning and development. One stated, “yes, I do believe
that (student engagement is affected by how I develop courses) actually.” Another stated, “if
you're looking at more of a micro level of structure, I think the way that we strive to make each
discreet segment digestible, I think, is gonna have a big impact on how they learn and doesn't
feel like they're just turning on a lecture and sitting back for an hour and a half and watching
someone talk and maybe taking notes.” A third developer elaborated saying, “we have to be
intentional, and the types of questions, open response or so forth, that we're posing to students,
and making sure that either student responses are helping to assist students in coming to a
specific conclusion about, maybe a topical lecture that the instructors provided, or is gonna help
to add some colorful discussion starting the synchronous session.”
CREATING EFFECTIVE EDUCATIONAL CONTENT 77
Self-efficacy
Self-efficacy is the belief a person has that they can accomplish a given task. This
element can determine if a course developer begins a task and follows it through to completion.
Confidence. Course developers need to feel confident in their abilities to create engaging
learning materials for online students. As Figure 11 shows, 58 % of course developers agreed
that they are confident in identifying content that is engaging or not engaging to students; 16%
strongly agreed; another 16% remained neutral, while one respondent disagreed.
Figure 11. I feel confident in my ability to identify content that is engaging or not engaging.
During one-on-one interviews, the responses did no align with the survey data regarding
confidence. For example, one course developer said, “I have a course right now where the
professor is like so hung up on figuring out what should be asynch and what should be synch,
and I'm giving him suggestions, but it doesn't mean my suggestions are correct.” This quotation
shows that while the course developer was able to manage a conversation about building a
CREATING EFFECTIVE EDUCATIONAL CONTENT 78
course with a professor, they were not confident about their planning methods. Another stated, in
response about their feelings of confidence in the planning process that “I feel like at this point,
more training would definitely increase (my) confidence.” And, a third course developer reported
that their confidence was boosted when the team coordinated mock planning meetings and said
that, “even if that was a totally manufactured environment, just having done it that first time
makes me feel like I'm not trying something out for the first time ever in a more 'high stakes'
scenario.” Course developers’ self-reported lack of self-confidence also brought to light their
motivation to learn more. As Figure 12 indicates, 83% of course developers reported wanting to
learn more about how to create engaging content for online learners.
Figure 12. I want to continue to learn how I can create content that is engaging to online learners
Synthesis
Course developers reported that they view their course development methods as being
crucial to building engagement with online students. Course developers also reported that they
are not always confident in how to best approach course planning meetings with faculty when it
comes to offering guidance about what the best content planning and building methods are. The
CREATING EFFECTIVE EDUCATIONAL CONTENT 79
relationship between course developers and university faculty is clearly delicate because both the
professors and course developers have specialized knowledge they bring to the course
development process. But course developers need to feel confident in their own abilities so that
they do not rely so heavily on professors’ influences.
Results and Findings Regarding Organization
Previously, three organizational influences were identified: corporate culture,
accountability, and leadership. All three were validated and are discussed below.
Faculty and University Influences
Relationship management. In one-on-one interviews, course developers discussed how
they are commonly affected by managing their relationships with faculty members during the
planning process. One course developer said, “one thing I do find is that instructors often just
want someone to listen to them, and who can empathize with the situation that they're in, who
can be really an advocate and a team player for them.” Figure 13 is a graphic representation of
how important maintaining a high level of faculty satisfaction is to developers. In fact, almost all
course developers reported valuing faculty experiences and satisfaction the most in their
development process.
CREATING EFFECTIVE EDUCATIONAL CONTENT 80
Figure 13. Influences on course developers
In addition to individual relationships course developers manage with faculty, internal
technical quality assurance processes and project timeliness are key factors in the course
development practice, as shown in Figure 13. What is not represented in the figure is the effect
that universities have on the planning process, which was a theme in the one-on-one interviews.
University culture influences. In one conversation, a course developer stated, “every
program has different cultures and approaches and styles and views. How they view online
education. How they view their professors and everything under the sun. I think that stems from
both just the culture of the university and the subject matter of the degree program.” Another
course developer said:
Counselors are going to have more creative videos or round tables where you discuss a
case or something like that. But in a business class you would still discuss a case, but it's
more fact driven I would feel, whereas many counseling programs that I've worked with
will be like, oh, there are just many approaches you can take. We're going to examine just
CREATING EFFECTIVE EDUCATIONAL CONTENT 81
this one, but you can have a multitude of ways. But a business school, for example,
would be like, this is the answer for that question. This is how you would do it. So it's a
very broad range of possibilities for certain programs and then just a very narrow
definition for others.
Continuing with the theme of university and program subject matter culture influences, another
course developer recognized that program directors also determine the types of content the
school values. This developer said:
You know (there is a program director that) likes to think outside the box, and she really
pushes things to her professors…they really have a lot of role-plays and different things
that are useful…They're a little bit more work, but I always know it's a little bit different.
Then you have some that are, you know...might not be as exciting.
So course developers not only navigate organizational influences from the company , but must
also manage the content influences from the educational institutions.
Up to Date Information and Training
Need for supportive resources. As indicated in Figure 14, 58% of respondents
disagreed when asked whether the organization provides sufficient resources to train them how
to engage online students, while 33% chose to remain neutral on the topic, and one strongly
disagreed and another agreed.
CREATING EFFECTIVE EDUCATIONAL CONTENT 82
Figure 14. The organization has provided me enough training to understand how online tools
engage students.
Similarly, as Figure 15 reveals, when asked about the sufficiency of resources provided,
course developers also reported a gap. While 50% chose to remain neutral, 33% disagreed, one
strongly disagreed, and one agreed.
Figure 15. I feel that my organization provides me with enough resources to plan engaging
CREATING EFFECTIVE EDUCATIONAL CONTENT 83
content.
When asked about what they learned in previous training on the job, respondents made it
clear that the training did not ensure they could that achieve their goal of creating more engaging
learning content. One said, “I'd say a lot of the sort of planning training sessions that we've had,
to sort of gear [course developers] up for taking over the planning process, helped increase my
confidence a bit. But I will say, just speaking for me personally, I feel like we need a lot more of
that at this point…there's a gap there, for sure. I feel like at this point, more training would
definitely increase my confidence.”
Figure 16. How do you think, if you are interested, you can be provided with more information
on how to build and plan content that engages online students? - selected choice
CREATING EFFECTIVE EDUCATIONAL CONTENT 84
In an effort to determine the types of training and resources course developers most
value, they were asked to identify them, as depicted in Figure 16. Unsurprisingly, all course
developers value in-person trainings and workshops. Yet unexpectedly, all course developers felt
attending conferences in the field of educational media and technology would be useful for them.
Need for student data. All course developers value student data collected from the learning
management systems. In a one-on-one interview, one developer stated, “most recently I had a
professor ask about a forum, how often do students post...those are the types of questions I tend
to get.” And another expressed that they, “don't have any information like that to be able to
provide to [the professors they work with]. It's tough.” Further, in the distributed survey, answers
about helpful and needed data included: “back-end LMS analytics that demonstrate student
engagement” and “data from student interactions with our system.” All course developers felt the
need to learn more about student data in order to be knowledgeable about creating online learning
content. One respondent stated that professors ask about student data during planning, “all the
freaking time.” This developer went on to explain that they were asked that “a lot more often than at
times [they felt] comfortable answering.”
CREATING EFFECTIVE EDUCATIONAL CONTENT 85
The survey data, as shown in Figure 17, also aligned with the interviews in that 91% of the
course developers strongly agreed with wanting more information about how students engage with
the content that the developers produce. Only one person remained neutral.
Figure 17. Would you like more information on how students engage with the content you create
with professors?
CREATING EFFECTIVE EDUCATIONAL CONTENT 86
Figure 18. How do you think, if at all, you can gain more insight into how students engage with
the content you’ve produced? - selected choice
Figure 18 shows that course developers value student insight the most, followed by
surveys, then training. The course developers that selected “other” stated a desire for backend
analytics, data from student interactions, and more information on online learning pedagogy.
Synchronous sessions. As discussed in the knowledge section, a common theme arose
regarding planning live sessions with faculty. Most course developers reported not having the
necessary training to be able to plan live sessions effectively. One course developer stated, “I’m
still getting comfortable with live session planning. Theoretically, I understand the purpose
behind it, but I don’t speak it as well as I can speak asynch jargon.” Another said they need
more training in “live session planning for sure and things like workshops and best practices and
a chance to shadow as well (would be) useful.”
CREATING EFFECTIVE EDUCATIONAL CONTENT 87
Accountability
Course developers are not accountable for creating content with engaging learning
strategies. One course developer explained, “internally (they) have a quality assurance process
obviously that we use to vet courses before they go live to students or before students access.” But
that process does not include any vetting to ensure that engaging learning strategies are used. Instead,
the quality assurance process focuses solely on technical errors. One developer even referenced the
QA process as a box that has to be checked and said the factors they focus on at the end of course
creation are “has it gone through pre-QA, QA, and come back looking clean? For me, those are
boxes that I want to check.”
Corporate Culture
Timing. Some course developers reported time as a major concern when working on
courses with faculty. One course developer stated, “It’s always a race against the clock. I feel
like even the courses that we start early with the intention of getting ahead and spending a lot of
time on them, it still seems like we are rushing to get ready for the study.” Another developer
related: “I think sometimes if you have a busy professor…we can get delayed.” When asked
about balancing the demands of timelines and the workload, another person said:
It's a lot to juggle and do it effectively. I am managing it, but I don't feel as though it's
sustainable, just given the level of detail, the level of thought and preparation that you
have to put into even reviewing the course planner multiple times during different stages
of planning. It's being done, but it is not sustainable at the current level.
In support of the qualitative data that shows that course developers are affected by their strict
timelines. Figure 19 shows that 66% of the respondents are most influenced by the timelines that
are inflicted on them for course development.
CREATING EFFECTIVE EDUCATIONAL CONTENT 88
Figure 19. Timeline, Pedagogy and Professor Feedback
Collaboration and Leadership. Course developers reported that they value
collaboration with outside groups and direction from leadership. But they also said they do not
feel as though the other groups in the department are collaborative or that leadership asked them
to execute on specific goals. One course developer stated, “it's important that the leadership team
is transparent with conversations that are happening with program and school administrators
because sometimes updates that they give me can provide almost immediate guidance or
assistance in communications that I'm having with faculty or other school leaders.”
Synthesis
Course developers identified that they need more training resources to fulfill the demands
of their job. A high number of course developers also emphasized that they value student data
from the learning management system. Most stated that a lack of understanding of the students’
learning experience contributed to their lack of confidence in planning courses. Course
CREATING EFFECTIVE EDUCATIONAL CONTENT 89
developers are also affected by the time constraints within which they have to work. This
significant factor has a negative impact on how much time developers can spend discussing
learning strategies with the faculty members with whom they are planning. Finally, course
developers shared that the culture of the their department affects how they work.
CREATING EFFECTIVE EDUCATIONAL CONTENT 90
CHAPTER 5: RECOMMENDATIONS
In Chapter 4, detailed findings obtained from data collected via qualitative and quantitative
methods were presented. Quantitative data from the surveys, along with qualitative data from the
surveys and interviews provided abductive information regarding the two research questions:
1. What are the knowledge, motivation and organizational (KMO) influences that impact
how course developers produce online courses with student engagement strategies in order to
achieve the goal of creating more engaging learning experiences for students?
2. How could organizational practices in the area of knowledge, motivation and
organizational resources be charged to facilitate the use of engaging learning strategies in Course
Development?
The recommendations in Chapter 5 are presented in knowledge, motivation, and
organizational categories based on the results and findings in Chapter 4. The knowledge
recommendations are based on influences that provide current information related to student
engagement, knowledge of templates and assessments for building online courses, and guidance
for how to structure live sessions. The motivational influences are engaging in conversations
with course developers about how their content production affects the student experience and
developing their confidence in their abilities in course development. The organizational
influences have strict timelines for creating content, a lack of supportive departmental culture
and a lack of training and resources.
Recommendations for Practice to Address the Knowledge, Motivation, and Organizational
Influences
The following knowledge, motivation and organization recommendation sections are
supported by the literature review and data analysis. In each table, the assumed influence
identified in the literature review and data analysis is stated. All validated assumptions are
CREATING EFFECTIVE EDUCATIONAL CONTENT 91
determined to be a high priority, and non-validated assumptions are not a priority. Principles
supported by literature are determined, and recommendations are made based on Ed Tech Inc.’s
need.
Knowledge Recommendations
Clark and Estes (2008) recommend a group’s knowledge be assessed with regard to
performance goals and gaps because, in general, people are unlikely to be willing to attest to lack
of knowledge or skills. Therefore, Table 7 reflects the general knowledge that course developers
must have in order to achieve the stakeholders’ goal of creating online courses with engaging
learning strategies.
Table 7
Summary of Knowledge Influences and Recommendations
Assumed Knowledge
Influence: Cause, Need,
or Asset*
Priority
Yes, No
(Y, N)
Principle and
Citation
Context-Specific
Recommendation
Declarative: All new
course developers should
be assessed in order for
Ed Tech Inc. leadership
to understand their
knowledge and skill
levels, especially as it
relates to a shared
definition of student
engagement.
Y According to
Krathwohl (2002),
factual knowledge
includes foundational
elements that
employees need to
understand to
function in the
workplace at a basic
level.
Training, Job Aids, and
Information
Training guides empower
employees by sharing the
organization’s best practices and
definition of what student
engagement means. Information is
the type of knowledge that is
given to an individual to tell them
something regarding their job. A
job aid helps individuals to solve
complicated matters
independently. (Clark & Estes,
2008)
CREATING EFFECTIVE EDUCATIONAL CONTENT 92
Conceptual: Course
developers need to
understand the tools in
the learning management
systame and effective
engagement strategies
that can foster student
engagement.
Y Basic knowledge is
needed to function
effectively, and
problem solve
(Rueda, 2011).
Job Aids and Information
Provide current knowledge in the
field of educational media to
increase the confidence of course
developers in the use of effective
learning strategies for online
learners.
Training
Provide training to course
developers on engagement
strategies for online learners that
are specific to video production
and assessment methods.
Procedural: Course
developers need to
understand how to build
exercises that are built to
motivate students to
engage with the online
learning content.
Y Basic knowledge
needed to function
effectively and
problem solve
(Rueda, 2011).
Job Aids and Information
Provide current information to
increase the knowledge of course
developers and faculty on
effective learning strategies for
online learners.
Metacognitive: Course
developers should be
more sef-aware and
reflective of their
strengths and
weaknesses when
producing course
content.
Y According to Mayer
(2011), self-regulated
learners have
metacognitive
awareness and
control.
Provide training that utilizes
simulations of course development
planning to allow course
developers to intentionally plan
courses with online learning
strategies.
Declarative knowledge solutions, or description of needs or assets. During data
collection, it was noted that course developers had different perceptions about what student
engagement is. As noted in Table 7, all new Course Developers’ knowledge should be assessed
prior to beginning training. For example, one of the concepts that are important to assess is
whether or not course developers have a shared definition of what student engagement is for
online learning as well as how online tools can help to create engaging e-learning content.
CREATING EFFECTIVE EDUCATIONAL CONTENT 93
Therefore, assessing all course developers in the topic will give the training team an idea of
where to begin. If new employees have prior experience in the field of educational media and
technology, then information and job aids can help to support them as they produce courses with
Ed Tech Inc., while employees who have not worked in the field of educational media will need
more guidance and training (Clark & Estes, 2008).
Job aids and information provide course developers with the support that they need to do
their job while training guides them through specific exercises and feedback (Clark & Estes,
2008). Job aids could consist of specific definitions that are important in creating engaging
learning content in a shared wiki. Anderson et al. (2001) state that in order to be effective
problem solvers, course developers need to know the basic definitions that are associated with
their jobs. And, training helps to put the concepts into practice before applying them on the job
(Clark & Estes, 2008).
Conceptual knowledge solutions, or description of needs or assets. Course developers
should have a basic understanding of the online tools that are used in online learning. Basic
knowledge is needed to function effectively, and problem solve (Rueda, 2011). Thus, the use of
job aids and information can be used to provide current knowledge about how to use online
learning tools and strategies in a manner that will be more engaging to students.. Training
sessions based on the up-to-date concepts that are ever-evolving in the field can also guide
course developers to be most effective at their job.
Mayer (2011) states that the use of guidance, modeling, coaching, and other scaffolding
during a performance is more effective than one long training session. Therefore, the use of
frequent training as it relates to course development would be helpful to instill concepts as they
relate to educational media. Also in support of Mayer’s (2011) recommendation for frequent
CREATING EFFECTIVE EDUCATIONAL CONTENT 94
training, Grossman and Salas (2011) state that employees that are trained and assessed in
frequent intervals have a higher likelihood of adapting to the culture of their organization and
being successful at their job (Grossman & Salas, 2011).
Procedural knowledge solutions, or description of needs or assets. Members of the
course development department should also understand how to build exercises that are built
to motivate students to engage with the content. There are several frameworks that can be
considered when building online modules with goals to complete certain learning objectives
(Clark, Sampson, Weinberger, & Erkens, 2007). If a course developer is methodological in how
they structure group argumentative activities, the learning that is fostered may even become
more effective than when using traditional methods.
In 2011, Leong suggested that designers of online classes should create a social
environment by having students introduce themselves to one another at the beginning of the
semester and post on discussion walls. Further course developers should encourage the faculty
members with whom they are working with to reply to posts in online forums in a timely manner
so that students do not feel as though their input in their courses is ignored. In support of the
notion of online collaboration, Wheeler (2006) states that students appreciated opportunities that
were given to collaborate with their peers. Finally, e-mentoring can alleviate the feelings of
alienation that are experienced by online students and foster a social environment. In order to
help facilitate engagement with online content, e-mentors should also be considered and
understood by the organization as a means of providing more interaction (Norziania, 2012).
CREATING EFFECTIVE EDUCATIONAL CONTENT 95
Metacognitive knowledge solutions, or description of needs or assets. Training that
utilizes simulations of course development planning could allow course developers to
intentionally plan courses with online learning strategies in a low-stakes environment. The data
showed that course developers think through their content planning process through the lens of a
student and even reflect on their own experiences as students, if applicable, to guide their
conversations with faculty.
Motivation Recommendations
The motivational influences in Table 8 represent a list of assumed motivational
influences that affect the achievement of the stakeholders’ goal.
Motivation, as described by Clark, Howard, and Early (2006), is the drive that and energy
that, when present in an individual, helps learning to come to fruition. Therefore, when otherwise
capable learners fail to achieve learning goals, hypothesizing a lack of motivation is not
unreasonable (Clark, Howard & Early, 2006). Different individuals have different value systems.
Motivating a team is often difficult because people are driven by different goals, values, beliefs,
and expectations (Clark 2005). But, according to Clark (2005) when a group of people joins
together, their different personalities can positively influence one another if each person
contributes to the group.
Examples of some intrinsic motivators and inhibitors in the literature on faculty
motivation in online learning are technology self-efficacy, perceptions regarding the
effectiveness of online education, faculty’s experiences with technology, desires for professional
growth and the shared desire with their university for increased student access (Batts, Pagliari,
Mallett, & McFadden, 2010; Gibson et al., 2008; Regan et al., 2012; Schulte, 2010; Styron et al.,
2009; Waheed & Farooq, 2010; Ward et al., 2010; Wickersham & McElhany, 2010). Examples
of extrinsic motivators are an increase in pay for designing an online course for the university
CREATING EFFECTIVE EDUCATIONAL CONTENT 96
and, as T. Green et al., 2009; Osika et al., 2009; Wasilik & Bolliger (2009) state, an increase in
flexibility for teaching, since they can do so from home or anywhere but the university. While
research on the topic of motivation for faculty focus on both intrinsic and extrinsic motivators,
institutions should focus on the former.
Table 8
Summary of Motivation Influences and Recommendations
Assumed Motivation
Influence: Cause, Need,
or Asset*
Priority
Yes, No
(Y, N)
Principle and
Citation
Context-Specific
Recommendation
Self-Efficacy. Course
Developers need to
believe they can use
effective engagement
strategies for online
learners.
Y Self-efficacy is
increased as
individuals succeed
in a task (Bandura
1997). High self-
efficacy can
positively influence
motivation (Pajares,
2006).
Management should make it clear
that course developers are capable
of learning how to use effective
engagement strategies. Training
should be provided to all, but
especially stressed on for those
who feel as though they are not
capable of producing or teaching
an online course effectively.
Attribution. Course
developers should
attribute student
engagement to their
content development
processes.
Y Performance
improvement must
start with the setting
of clear goals
(Bandura 1997;
Clark & Estes 2006).
Provide measurable goals that are
frequently checked in on by the
production team.
Attribution. Course developers should consider adding a certain amount of activities that
promote social interaction amongst students. This is because a lack of social interaction
contributes to a higher dropout rate in online courses, in comparison to traditional brick and
mortar classes. In 2011, Leong suggested that instructors of online classes should create a social
environment by having students introduce themselves to one another at the beginning of the
CREATING EFFECTIVE EDUCATIONAL CONTENT 97
semester and post in discussion walls. Course Developers could encourage such discussions by
replying to posts in online forums in a timely manner so that students don’t feel as though their
input in their courses is ignored.
In order to achieve a successful relationship amongst the organization’s goal and the
values that course developers and faculty hold in creating online learning content a collaborative
environment between the faculty members and course developers are essential to creating
effective online learning content. There is a delicate relationship between both the person
designing the course in its online format and the faculty member, a subject matter expert, who
will ultimately teach the course. The relationship is delicate because both the professors and the
online learning experts have specialized knowledge that they bring to the course development
process, but neither should overburden the other (Brigance, 2011). Professors do not want to be
managed by course developers, but they typically do welcome being led (Brigance, 2011).
Self-efficacy. According to Clark and Estes (2006), motivation stems from the belief that
a person has about themselves and their capabilities to achieve a given task. Self-efficacy, as
defined by Pajares (2006) is defined as the beliefs that an individual holds about their capabilities
to be able to learn or achieve tasks at certain levels. In short, self-efficacy is the perception that
an individual holds with regard to capabilities (Pajares 2006).
Course developers should feel confident in their ability to create courses that harness the
engagement of online learners. Hoffman, in a dissertation written in 2013, cited several
researchers that found that faculty members could be motivated to participate in online education
by positive self-efficacy regarding their ability to teach in an online environment (Tabata &
Johnsrud, 2008; VanHorn, Pearson, & Child, 2008). In order to foster an environment where
CREATING EFFECTIVE EDUCATIONAL CONTENT 98
course developers feel self-efficacious, the leadership team should express confidence in the
team’s ability to produce educational content.
Organization Recommendations
An organization’s effectiveness increases when the employees are motivated to work
well. And, motivation is tied to the organization’s ability to articulate their goals and direction.
Knowles (1980) states that people are more likely to be good participants when they understand
the goals that need to be achieved. In order to achieve goals, benchmarks should be set in place
as a means of measurement. Thus, Bogue and Hall (2003) and Marsch (2012) encourage the use
of data to drive decisions and actions that lead up to a goal. Such goals work as benchmarks and
clearly show the gaps between what the organization wants to achieve versus how it is currently
performing. Table 9 provides recommendations, based on theoretical principles, to close the
organizational gaps in the areas of cultural models and settings within EdTech Inc.
Table 9
Summary of Organization Influences and Recommendations
Assumed Organization
Influence: Cause, Need, or
Asset*
Priority
Yes, No
(Y, N)
Principle and
Citation
Context-Specific
Recommendation
Accountability. Course
developers are not accountable
for creating courses with
effective online learning
strategies.
Y Different types of
benchmarking
contribute data to
improve
organizational
performance
(Bogue & Hall, 2003;
Marsch,
2012).
Data-driven benchmarking
should be used in the
organization and
department to demonstrate
how performance compares
to desired goals.
CREATING EFFECTIVE EDUCATIONAL CONTENT 99
Corporate culture. Course
developers are affected by
organizational culture,
department goals, and faculty
influences.
Y Strong corporate
cultures improve
firm performance by
facilitating internal
behavioral
consistency.
(Sorensen, 2002)
Meetings and
communication within the
department should focus on
messaging the value of
sound pedagogy in course
production.
Time, budget and audience
heavily influence course
developers, more than
educational theory when
planning courses.
Y The organization
must be involved in
communicating the
appropriate
expectations so that
course developers
are not sidetracked
and influenced by
the organization’s
partners (Clark &
Estes, 2008)
Meetings and
communication within the
department should focus on
instilling confidence in the
course developers’ so that
they are less influenced by
outside factors.
There is a general belief among
course developers that online
course design should be guided
by stakeholders’ (e.g. faculty)
beliefs.
Y Communication
triggers change
(Lewis, 2011).
Communication to the
course development team
that instills in them the
belief that they should be
expected to guide course
development processes.
Course developers should also
have access to up-to-date
information related to online
learning (e.g., videos that are <6
minutes are more engaging;
videos produced with personal
feel are more engaging than
high-production content; talking
head lectures are more engaging
than screen captures).
.
Student learning
outcomes are related
with the resource
needs of staff being
met (Walters,
Marzano, &
McNulty, 2003).
Job Aids and Information
Provide current knowledge
in the field of educational
media to increase the
confidence of course
developers in the use of
effective learning strategies
for online learners.
Training
Provide training to course
developers on engagement
strategies for online
learners that are specific to
video production methods.
CREATING EFFECTIVE EDUCATIONAL CONTENT 100
Accountability. Holding course developers accountable to utilizing recommended
engagement strategies that are proposed by the strategy team and existing research could creative
competitive culture if not implemented strategically. Indeed, there are both advantages and
disadvantages to a culture of competition in society. On one hand, according to Bogue and Hall
(2003), a lack of competition creates an environment where the absence of goals can set a trend
of mediocrity in an organization. On the other, over-zealous personalities may jump on top of
each other to fulfill their own selfish needs to be the best (Bogue & Hall, 2003). Regardless,
accountability measures are crucial in ensuring the best quality product in an organization
(Bogue & Hall, 2003). In the case of the course development department, setting up an
accountability structure would need to be created carefully such that a hostile environment of
competition is not created and instead fosters a community that values the use of effective online
learning strategies. For example, leaders of the department should consider creating a quality
assurance process in which courses are evaluated to establish whether or not intentional
strategies were used to create the online course content. Having an objective rubric to decipher
what is an optimal course versus what would be a suboptimal course would help the course
developers to not feel threatened by an accountability process.
Corporate culture. A strong organizational culture affects the behavior of its members
and can prevent an organization from making changes for revolutionizing for a changing
industry (Schein, 2004). The benefits of a strong culture in a corporation are enhanced
coordination and control, improved goal alignment and increased employee effort (Sorenson,
2002). Sorenson (2002) states that one of the key outcomes of strong corporate culture is the
increase in behavioral consistencies across employees in an organization. Schein (2004) states
CREATING EFFECTIVE EDUCATIONAL CONTENT 101
that organizational culture is created through sharing experiences, shared learning and a feeling
of stability in the group. Corporate culture cannot be learned or imposed onto individuals
(Schein, 2004).
Elmore (2002) states that in order to ameliorate student learning and improve overall
performance, the knowledge of educators must first be invested in. While members of the
Content and Technology team are not educators per se, they work closely with university faculty
to plan, strategize and execute the production of online courses. Therefore, their knowledge and
understanding of online learning strategies, the proprietary tools offered by the organization and
assessments are vital to the development of pedagogically sound course materials. Marsh et al.
(2002) caution that while understanding student achievement scores is helpful, workshops and
tutorials for educators should focus more on how to use evaluation scores to structure
educational content.
Leadership. Effective leadership is the ability to drive an organization’s mission by
demonstrating one’s commitment, setting clear goals and supporting all those who are part of the
institution. A team has a strong chance of being successful if its leaders are capable, committed
and track attainable performance measures (Holtzman, & Anderberg, 2011). What leaders pay
attention to and measure affects their subordinates’ view of the organization and mission
(Schein, 2004). If a leader harnesses this knowledge and focuses their actions on their goals for
their department or organization, then they can become powerful in messaging their intentions.
On the other hand, if they are weak at being consistent in their messaging they will be unable to
make their goals relevant to their subordinates truly.
When creating online courses, leaders of universities and online learning firms should
make sure that their designers understand online learners’ characteristics and the types of tools
CREATING EFFECTIVE EDUCATIONAL CONTENT 102
that would be best in addressing their needs. According to Oliver, Kellogg, Townsend, & Brady
(2010) knowing the types of tools and how they relate to learning outcomes improves the
ultimate educational product. Kulski (2002) stated that institutions should be strategic and clear
about their goals.
Time. Additionally, in order to utilize the data taken from the quality assurance process,
leaders in the course development department should verify that individuals’ have an adequate
amount of staff time to dedicate to analyzing assessments and finish courses. According to
Marsh et al. (2002) staff capacity and support help to enable data use in educational practices.
Up-to-date information. Course developers should also have access to up-to-date
information related to online learning. Basic knowledge is needed to function effectively, and
problem solve (Rueda, 2011). Thus, the use of job aids and information can be used to provide
current knowledge in the field of educational media to increase the confidence of course
developers in the use of effective learning strategies for online learners. Training sessions based
on the up-to-date concepts that are ever-evolving in the field can also guide course developers to
be most effective at their job.
Integrated Implementation and Evaluation Plan
Implementation and Evaluation Framework
The model that informed this implementation and evaluation plan is the New World
Kirkpatrick Model (Kirkpatrick & Kirkpatrick, 2016) which is based on the original Kirkpatrick
Four-Level Model of Evaluation (Kirkpatrick & Kirkpatrick, 2006). Kirkpatrick (2016) stresses
the importance of an effective evaluation strategy and the use of triangulated data to show
results. Using qualitative and quantitative data as a means of triangulation shows the most robust
CREATING EFFECTIVE EDUCATIONAL CONTENT 103
outcome of training and thus show the effectiveness of training as it relates to an organization’s
goals.
Organizational Purpose, Need, and Expectations
Having laid the foundation for scalable architecture (for example, course planning
software to help make course planning more efficient) in 2016, the Technology Development
team at EdTech Inc. in 2017 focused on continuing to developing the company’s suite of tools,
as well as create further integrations between Content and the business as a whole. The goals
accomplished in 2016 greatly increase the volume and complexity of the course data that will be
collected and stored in the organization’s infrastructure. 2017 focused on acquiring that data and
creating workflows to allow the content team to make use of this data across the breadth of the
business.
The stakeholder’s, the course development team, performance goal is to structure
asynchronous content with engagement strategies in order to be able to achieve the goal of
creating course content that aligns with the organization's aspirations around being more efficient
and effective in everything that it delivers. From more engaging learning experiences to more
streamlined and adaptable internal processes. In order to achieve this goal, the course developers
in the content team must be trained and educated in research-based teaching strategies for online
learning in order to enhance the student experience.
Level 4: Results and Leading Indicators
Kirkpatrick (2016) states that it is important to initially consider each of the New World
Model’s levels in reverse. Doing so helps to keep the focus on what is most important
(Kirkpatrick 2016). Thus, Table 10 shows the proposed Level 4: Results and Leading Indicators
in the form of outcomes, metrics, and methods for both external and internal outcomes for Ed
CREATING EFFECTIVE EDUCATIONAL CONTENT 104
Tech Inc. If the internal outcomes are met as expected as a result of the training and
organizational support for new course developer’s performance on the job, then the external
outcomes should also come to fruition.
Table 10
Outcomes, Metrics, and Methods for External and Internal Outcomes
Outcome Metric(s) Method(s)
External Outcomes
1. Increase the number of courses that
are created with engaging online
learning strategies.
The number of courses
that course developers
deem as being planned
with pedagogy rather
than timelines at the
forefront.
Survey data from course
developers.
2. Improve Course Developer
knowledge on engaging learning
strategies for online students.
Number of courses
launched with learning
outcomes as a priority.
Conduct knowledge check-
ins with course developers
to gauge their knowledge on
learning strategies.
3. Improved relationships with
university faculty.
The length of time that
professors take to plan
and produce courses with
university professors.
Compare the length of time
that professors take to
produce courses to previous
methods.
4. Increased notability/marketability
in the field of educational
technologies due to the use of
pedagogically sound techniques used
to create online content.
The presence of the
organization in
educational technology
literature and/or media
content.
Compare the number of
times the organization was
cited as using specific
engagement strategies to
increase learner outcomes.
CREATING EFFECTIVE EDUCATIONAL CONTENT 105
5. Increased learner outcomes. The scores that students
provide specifically on
their online coursework
as it relates to how
engaged they are and
how they value it.
Compare scores from
previous years to what
students score after
engaging learning strategies
are implemented.
Internal Outcomes
6. Increased Course Developer
confidence and satisfaction in the way
they produce courses with faculty.
Results on key questions
as they relate to
satisfaction and
confidence in how they
produce courses.
Compare quarterly results to
survey and set aside regular
times for 1:1 development
check-ins between course
developers, senior course
developers, and directors.
Feedback from senior
course developers and
directors
Set aside regular times for
1:1 conversations
(development check-ins)
between new course
developers and top course
developers.
7. Increased use of online learning
strategies by course developers in
their courses.
The number of strategies
that are identified by
senior course developers.
QA process for each course
that checks explicitly a
course for its use of
pedagogy.
8. Increased confidence levels in the
organization and department.
Feedback from course
developers.
Survey data from course
developers.
Level 3: Behavior
Critical behaviors. The stakeholders of focus are the course developers that work with
university faculty to create online courses. The first critical behavior is that new developers must
identify opportunities to include effective engagement strategies for online students. The second
CREATING EFFECTIVE EDUCATIONAL CONTENT 106
critical behavior is that they must guide professors in the planning process to use the strategies in
their courses. The third critical behavior is that they must review the courses for the use of
effective engagement strategies, identify gaps and complete the courses on or before the
deadline. The specific metrics, methods, and timing for each of these outcome behaviors appear
in Table 6, below.
Table 6
Critical Behaviors, Metrics, Methods, and Timing for New Reviewers
Critical Behavior Metric(s) Method(s) Timing
1. Identify opportunities
to include effective
engagement strategies
for online students
The number of
courses that course
developers identify
as including
effective
engagement
strategies.
The senior course
developers should
track the creation of
courses by
developers.
Halfway through the
course
planning/production
process.
2. Course developers
should guide professors
in the planning and
production process to
use the strategies in their
courses.
The number of
course developers
who report using
engagement
strategies in their
online courses.
Surveys sent to
course developers
after a course has
been completed.
Ongoing – every
quarter.
CREATING EFFECTIVE EDUCATIONAL CONTENT 107
3. Must review the
courses for the use of
effective engagement
strategies, identify gaps
and complete the courses
on or before the
deadline.
The number of
online engagement
strategies used in a
course.
Senior Course
Developers will
monitor the system
for creating courses
with effective
engagement
strategies.
Ongoing – every
quarter.
Required drivers. Course developers require the support of senior leadership and
directors to reinforce what they learn in training and encourage them to apply what they have
learned in ongoing course planning and production meetings with university faculty. Kirkpatrick
(2016) states that the use of drivers is necessary to reinforce, monitor, encourage and reward the
performance of critical behaviors on the job. Rewards should be created for the achievement of
goals that relate to creating pedagogically sound courses. Thus, Table 7 shows the recommended
drivers to support critical behaviors of course developers.
Table 7
Required Drivers to Support New Reviewers’ Critical Behaviors
Method(s) Timing Critical
Behaviors
Supported
1, 2, 3 Etc.
Reinforcing
Job Aid including a glossary of terms that are used in educational
technology.
Ongoing 1, 2, 3
CREATING EFFECTIVE EDUCATIONAL CONTENT 108
Job Aid including a checklist for the use of effective engagement
strategies for online learning.
Ongoing 1, 2, 3
Team meetings with new and experienced course developers to
establish goals for using effective engagement strategies and
progress in their work.
Quarterly 1, 2, 3
Use private list-servs and channels such as Slack to encourage
communication outside of team meetings.
Ongoing 1, 2, 3
Team meeting to troubleshoot collaboratively and for additional
training.
Weekly 1, 2, 3
Encouraging
Collaboration and peer modeling during team meetings. For
example, role playing course planning and production call with
peers.
Weekly 1, 2,
Feedback and coaching from senior course developers. Ongoing 1, 2, 3
Rewarding
CREATING EFFECTIVE EDUCATIONAL CONTENT 109
Performance incentive for course developer when the use of
specific engagement strategies increases and the time to produce
a course decreases.
Quarterly, or
project-based
1, 3
Public acknowledgment, such as a mention at All-Content
meetings, when the team and/or individual performance hits a
benchmark related to the number of engaging learning strategies
used in courses.
Quarterly 1, 2, 3
Monitoring
Senior management can create opportunities at All-Course
Developer meetings to share success stories
Monthly 1, 2, 3
Senior management can ask course developers to self-report their
confidence and self-efficacy as it relates to planning courses with
professors.
Two months
after training
1, 2, 3
Senior course developers can assess the performance of the
course developers. Frequent checks can help the organization
monitor progress and make adjustments if results do not match
expectations at that time.
Monthly 1, 2, 3
CREATING EFFECTIVE EDUCATIONAL CONTENT 110
Organizational support. The organization will need to support the course developers by
setting up accountability structures between sub-groups of the department; creating a culture
where gaining knowledge on research-supported methods for learning are supported and by
supporting the leaders in the group. Kirkpatrick (2016) states that companies that reinforce
knowledge and skills with accountability and support can expect up to an 80 percent success rate
whereas organizations that fail to reinforce training with support can only expect a 15 percent
success rate. Taking action on the required drivers for attaining a specific goal is a big marker of
success for any training program (Kirkpatrick, 2016). In order to support course developers with
regards to accountability structures, checklists and frequent meetings amongst team members can
help to ensure that what was discussed in training is still being implemented. In terms creating a
culture where gaining knowledge on research-supported methods is valued, the organization
should set up reward structures that praise those who work to achieve the goal of producing
courses with engaging learning strategies. Finally, supporting leaders (especially senior course
developers and directors) is imperative to reach the production group as a whole. With a strong
management in the team, employees are much more likely to feel supported and confident in
doing their jobs.
Level 2: Learning
Learning goals. Following the completion of the recommended solutions, most notably
the educational technologies training session, the stakeholders will be able to:
1. Recognize the types of online student engagement strategies that are necessary to produce
engaging online courses. (D)
2. Classify and interpret the types of engaging learning strategies that fit best within the
available tools in the learning management system. (D)
CREATING EFFECTIVE EDUCATIONAL CONTENT 111
3. Correctly attribute the aspects of a course that pertain to specific teaching strategies. (P)
4. Plan and monitor their work to conform to set guidelines that align with engaging
strategies for students. (P, M)
5. Indicate confidence that they pan and produce courses with university faculty within their
given timelines. (Confidence)
6. Value the implementation of online learning strategies in the courses that they produce.
(M)
7. Value the amount of time that they spend planning and producing a course in order to
create it with effective online learning strategies. (Value)
Program. The learning goals listed above will be achieved with a training curriculum
that explores in-depth the use of educational technology to effectively engage online students.
The learners, course developers, will study a wide range of themes that pertain to the planning
and production of online courses. For example, existing research on using video metrics (e.g., the
amount of time a student spends on an instructional video) as a means of identifying engagement
and the use of summative and formative evaluations to gauge student learning. The program is
blended and consists of two learning modules that will be placed in the team’s existing
knowledge hub and a face-to-face workshop. The total time for completion is 10 hours or 600
minutes, but one can be expected to grow as the program and organization evolve.
It is important to note that the online content will be structured with effective learning
experiences for the course developers and will carefully balance materials with the learners in
mind. Kirkpatrick (2016) recommends structuring content that is meant for learning in a way that
is focused on getting the right amount of data to show a program’s effectiveness. Therefore, the
CREATING EFFECTIVE EDUCATIONAL CONTENT 112
design of the job aids and content will be simple and straightforward rather than focused on
adding frills and games.
During the asynchronous portion of the program, learners will be given a job aid of terms
and best practices that relate to the creation of engaging online learning content. Some best
practices that will be listed in the job aid will relate to working with faculty to achieve the best
educational product and how to marry asynchronous and synchronous content for online
students. Another job aid will contain information that gives learners tips on how to plan courses
with professors with the strategies in a manner that is strategic and purposeful. The job aids will
be demonstrated in text and video form. The video form will use authentic scenarios with course
developers and faculty members. The key terms will be defined with examples in text form and
available in one central learning management system. The videos will be built in one of the
existing learning management systems for easy access as well. Additionally, within the system
and in a tool called the Bi-Directional Learning Tool (BLT), the videos will have stop points for
each course developer to answer questions based on what they have learned. Hosting the online
training for course developers in the same learning management systems as the courses that they
produce will help to further foster their learning. Additionally, using the same tools (e.g., the
BLTs) as a means to engage them in the coursework will help to instill the educational practices
that they are learning about for online students.
During the in-person piece of the program, the focus will be on having course developers
apply their new found knowledge to real planning sessions with faculty. Role-playing
discussions will enable more experienced course developers to stand in as university faculty
while new course developers practice their content production and planning skills. Group
discussions following the role-playing activities will give course developers the opportunity for
CREATING EFFECTIVE EDUCATIONAL CONTENT 113
the group as a whole to discuss challenges and solutions to producing engaging online learning
content.
Components of learning. Being able to demonstrate declarative knowledge is a vital
piece of being able to apply knowledge when problems arise. Therefore, the importance of
evaluating learning for declarative and procedural knowledge cannot be overlooked. Learners
should also feel confident, committed and valued (Kirkpatrick & Kirkpatrick, 2016). Table 8 lists
the evaluation and timing methods for these components of learning.
Table 8
Components of learning
Method(s) or Activity(ies) Timing
Declarative Knowledge “I know it.”
Knowledge checks, in the form of assessments,
about engaging learning strategies.
Distributed via Google Forms or any
similar application every week and then
quarterly after that.
Think in pairs and share out with the rest of the
team.
Monthly at first, during All-Course
Developer meetings, and then quarterly
after that.
Procedural Skills “I can do it right now.”
Demonstration in groups and individually of using
the job aids to plan and produce a course with
engaging learning strategies successfully.
During the face-to-face meeting and
course planning role play and then
during monthly All-Course Developer
meetings.
CREATING EFFECTIVE EDUCATIONAL CONTENT 114
Quality of the feedback from peers after the planning
and production role-plays during group sharing
During the workshops and then
quarterly, during All-Course Developer
calls.
Retrospective pre- and post-test assessment survey
asking participants about their level of proficiency in
course planning and production before and after the
training.
At the end of the face-to-face training.
Attitude “I believe this is worthwhile.”
Trainer’s observation of participants’ statements and
actions demonstrating that they see the benefit of
what they are being asked to do in course planning
and production meetings.
During the face-to-face training.
Discussions of the value of what they are being
asked to do in the Course Development process.
During the face-to-face training.
Retrospective pre- and post-test assessment item. After the asynchronous course.
Confidence “I think I can do it on the job.”
Survey items using scaled items Following each module/lesson/unit in
the asynchronous portions of the course.
Discussions following practice and feedback. During the face-to-face training.
Retrospective pre- and post-test assessment item. After the asynchronous course.
CREATING EFFECTIVE EDUCATIONAL CONTENT 115
Commitment “I will do it on the job.”
Discussions following practice and feedback. During the face-to-face training.
Create an individual action plan. During the face-to-face training.
Retrospective pre- and post-test assessment item. After the course.
Level 1: Reaction
The table below lists the methods that will be used to determine how the participants
react to the learning events. According to Kirkpatrick (2016), learning has a positive correlation
to learning so gauging how useful, or not useful, course developers find the training will show
some useful data to guide the program’s evolution.
Table 9
Components to Measure Reactions to the Program.
Method(s) or Tool(s) Timing
Engagement
Data analytics in the learning management system
(whether or not learners watched the content and how
they responded to the questions).
Ongoing during asynchronous portion
of the course.
Actual completion of online the online modules. Ongoing during asynchronous portion
of the course.
CREATING EFFECTIVE EDUCATIONAL CONTENT 116
Observation by instructor/facilitator During the synchronous session.
Attendance During the in-person part of the
training
Course evaluation Two to three weeks after the course
Relevance
Check-ins with course developers via survey (online)
and group discussions (ongoing)
After the completion of the online
modules and after the in-person portion
of the exercises.
1:1 discussions with senior course developers and the
course developers on their teams
After the completion of the online
modules and the in-person training.
Customer Satisfaction
Brief pulse-check with participants via survey
(online) and discussion (ongoing)
After every module/lesson/unit and the
workshop
Course evaluation Two weeks after the course and in-
person training.
Evaluation Tools
Immediately following the program implementation. During the asynchronous portion
of the course, the learning analytics tool in the learning management system (LMS) will collect
data about whether or not the participants completed the training. Their answers in the
asynchronous will also demonstrate how engaged they are with the content. These data will
indicate the engagement with the course material and will also show whether or not the learners
CREATING EFFECTIVE EDUCATIONAL CONTENT 117
are answering questions correctly. The creator of the asynchronous training will also distribute
brief surveys after each online course (see Appendix A) to request answers from the participants
about how relevant they feel the material is or is not to their job (Level 2). According to
Kirkpatrick (2016) the use of learner-focused surveys is imperative to gauge the personal
experiences rather than on the trainer. And, as mentioned previously, Kirkpatrick (2016) also
acknowledges that satisfaction is correlated to learning. Thus, the survey will also assess the
learners’ confidence in applying the educational techniques into the courses that they plan and
produce (Level 2)
During the in-person session, the instructor will give periodic brief pulse-checks by
asking the participants about the relevance of the content to their work and the organization,
delivery, and learning environment. This next part of the training will also include checks for
understanding using competitive activities games or competition among groups in responding to
questions and scenarios drawn from the content (Level 2).
Delayed for a period after the program implementation. Around a week after the
training, the leadership team will send a survey that contains open responses and like items to
measure, from the learner’s perspective, the satisfaction and relevance of the training. The survey
will also measure their confidence in applying what they have learned to their jobs (Level 2) as
well as the support from senior course developers and directors (Level 3). Finally, the survey
will also measure whether or not they feel as though the courses they are producing and planning
now consist of more learning pedagogy than before.
Data Analysis and Reporting
The Level 4 goal of course developers is measured by the engaging learning strategies
that course developers implement in the courses that they are planning as well as the results from
CREATING EFFECTIVE EDUCATIONAL CONTENT 118
the survey. Each quarter, senior course developers will track the number of learning strategies
placed into courses. Also, the dashboard below will report the data on the measures as a means
of accountability.
CREATING EFFECTIVE EDUCATIONAL CONTENT 119
Figure 19. Courses intentionally produced with research-based engaging learning strategies.
CREATING EFFECTIVE EDUCATIONAL CONTENT 120
Figure 20. Survey Result from course developers.
Summary
The New World Kirkpatrick Model (2016) was implemented to this organizational case
study to plan, structure, and evaluate recommendations for Ed Tech Inc.’s optimization of its
stakeholders’ goal and its organizational goal. The New World Kirkpatrick Model (2016)
encourages thinking about all four levels: Results, Behavior, Learning, and Reaction, in reverse.
Doing so helps to keep what is most important top of mind. That is, results (the fourth level), in
the case of course developers, is related to the outcomes of their knowledge and implementation
of engaging learning strategies to the courses they plan and produce. More importantly, however,
the implementation of the New World Kirkpatrick Model in this case study influenced the human
aspect of the organization. Kirkpatrick (2016) focuses not only on the metrics and results but also
on the need for constant human interaction and collaboration to sustain meaningful results.
This focus on the human aspect of the training is also demonstrated in Kirkpatrick’s
Blended Evaluation approach, which stresses the importance of structuring evaluative questions
on the learner, rather than the instructor. Therefore, evaluation questions were structured to
highlight the personal experiences of the employee and not on the instructors who lead the
asynchronous and synchronous training. The careful attention that was placed on the evaluation
questions are what differentiates this model from other training models because of the
measurable outcomes that can be shown to the leadership team (Kirkpatrick 2016). Using the
results, the leadership team will benefit from having a more knowledgeable and satisfied
department that can successfully contribute to the business outcomes of the organization.
Moving forward, the New World Kirkpatrick Model has the potential to help the
department to continue to evolve into a stronger educational production group. Kirkpatrick’s four
CREATING EFFECTIVE EDUCATIONAL CONTENT 121
levels: reaction, learning, behavior, and results are especially relevant in an educational
technology company with constant growth and ambitions to create content that increases student
outcomes.
CREATING EFFECTIVE EDUCATIONAL CONTENT 122
References
Allen, I. E., Seaman, J., Poulin, R., & Straut, T. T. (2016). Online report card: Tracking online
education in the United States. Babson Park, MA: Babson Survey Research Group.
Anderson, A. H., McEwan, R., Bal, J., & Carletta, J. (2007). Virtual team meetings: An analysis
of communication and context. Computers in Human Behavior, 23(5), 2558-2580.
http://dx.doi.org.libproxy1.usc.edu/10.1016/j.chb.2007.01.001
Arbaugh, J. B., & Benbunan-Fich, R. (2006). An investigation of epistemological and social
dimensions of teaching in online learning environments. Academy of Management Learning
& Education, 5(4), 435-447. 10.5465/AMLE.2006.23473204
Baltes, B. B., Dickson, M. W., Sherman, M. P., Bauer, C. C., & LaGanke, J. S. (2002).
Computer-mediated communication and group decision making: A meta-analysis.
Organizational Behavior and Human Decision Processes, 87(1), 156-179.
http://dx.doi.org.libproxy.usc.edu/10.1006/obhd.2001.2961
Barbour, M. K., & Harrison, K. U. (2016). Teachers’ perceptions of K-12 online. Journal of
Educational Technology Systems, 45(1), 74-92. 10.1177/0047239516637072
Beaudan, E., & Smith, G. (2000). Corporate culture: Asset or liability? Ivey Business Journal,
64(4), 29. Retrieved from
http://search.proquest.com.libproxy2.usc.edu/docview/225367578?accountid=14749
Benson, D. E., Haney, W., Ore, T. E., Persell, C. H., Schulte, A., Steele, J., & Winfield, I.
(2002). Digital technologies and the scholarship of teaching and learning in sociology.
Teaching Sociology, 30(2), 140-57. 10.2307/3211379
CREATING EFFECTIVE EDUCATIONAL CONTENT 123
Borokhovski, E., Bernard, R. M., Tamim, R. M., Schmid, R. F., & Sokolovskaya, A. (2016).
Technology-supported student interaction in post-secondary education: A meta-analysis of
designed versus contextual treatments. Computers & Education, 96, 15-28.
http://dx.doi.org.libproxy1.usc.edu/10.1016/j.compedu.2015.11.004
Bridger, R. S., Day, A. J., & Morton, K. (2013). Occupational stress and employee turnover.
Ergonomics, 56(11), 1629-1639. 10.1080/00140139.2013.836251
Capra, T. (2011). Online education: Promise and problems. Journal of Online Learning and
Teaching, 7(2), 288. Retrieved from
http://search.proquest.com.libproxy1.usc.edu/docview/1497199208?accountid=14749
Charlton, T., Devlin, M., & Drummond, S. (2009). Using Facebook to improve communication
in undergraduate software development teams. Computer Science Education, 19(4), 273-
292. 10.1080/08993400903384935
Chen, K., & Jang, S. (2010). Motivation in online learning: Testing a model of self-
determination theory. Computers in Human Behavior, 26(4), 741-752.
http://dx.doi.org.libproxy1.usc.edu/10.1016/j.chb.2010.01.011
Clark, T., & Muller, P. (2012). Exploiting model driven technology: A tale of two startups.
Software and Systems Modeling, 11(4), 481-493.
http://dx.doi.org.libproxy1.usc.edu/10.1007/s10270-012-0260-1
Collins, D., Weber, J., & Zambrano, R. (2014). Teaching business ethics online: Perspectives on
course design, delivery, student engagement, and assessment. Journal of Business Ethics,
125(3), 513-529. 10.1007/s10551-013-1932-7
CREATING EFFECTIVE EDUCATIONAL CONTENT 124
Cramton, C. D., & Webber, S. S. (2005). Relationships among geographic dispersion, team
processes, and effectiveness in software development work teams. Journal of Business
Research, 58(6), 758-765. http://dx.doi.org.libproxy.usc.edu/10.1016/j.jbusres.2003.10.006
Cummings, S. M., Chaffin, K. M., & Cockerham, C. (2015). Comparative analysis of an online
and a traditional MSW program: Educational outcomes. Journal of Social Work Education,
51(1), 109-120. 10.1080/10437797.2015.977170
DeAngelis, L., Penney, S., & Scully, M. (2014). Leadership lessons from teamwork. Leader to
Leader, 2014(73), 19-25. 10.1002/ltl.20135
Denison, D. R., Haaland, S., & Goelzer, P. (2004). Corporate culture and organizational
effectiveness: Is Asia different from the rest of the world? Organizational Dynamics, 33(1),
98-109. http://dx.doi.org.libproxy2.usc.edu/10.1016/j.orgdyn.2003.11.008
Eneau, J., & Develotte, C. (2012). Working together online to enhance learner autonomy:
Analysis of learners' perceptions of their online learning experience. ReCALL: The Journal
of EUROCALL, 24(1), 3-19.
http://dx.doi.org.libproxy2.usc.edu/10.1017/S0958344011000267
Flatt, S. J., & Kowalczyk, S. J. (2008). Creating competitive advantage through intangible assets:
The direct and indirect effects of corporate culture and reputation. Advances in
Competitiveness Research, 16(1/2), 13-30. Retrieved from
http://search.proquest.com.libproxy2.usc.edu/docview/211408587?accountid=14749
Giardino, C., Unterkalmsteiner, M., Paternoster, N., Gorschek, T., & Abrahamsson, P. (2014).
What do we know about software development in startups? Software, IEEE, 31(5), 28-32.
10.1109/MS.2014.129
CREATING EFFECTIVE EDUCATIONAL CONTENT 125
Gregory, B. T., Harris, S. G., Armenakis, A. A., & Shook, C. L. (2009). Organizational culture
and effectiveness: A study of values, attitudes, and organizational outcomes. Journal of
Business Research, 62(7), 673-679.
http://dx.doi.org.libproxy1.usc.edu/10.1016/j.jbusres.2008.05.021
Grosse, C. U. (2002). Managing communication within virtual intercultural teams. Business
Communication Quarterly, 65(4), 22-38. Retrieved from
http://search.ebscohost.com/login.aspx?direct=true&db=cms&AN=8811938&site=ehost-
live
Guo, P.H., Kim, J. and Rubin, R. 2014. How video production affects student engagement: an
empirical study of MOOC videos. In Proceedings of the first ACM conference on Learning
@ scale conference (L@S '14). ACM, New York, NY, USA, 41-50.
DOI=http://dx.doi.org.libproxy1.usc.edu/10.1145/2556325.2566239
Harasim, L. (2000). Shift happens: Online education as a new paradigm in learning. The Internet
and Higher Education, 3(1), 41-61. 10.1016/S1096-7516(00)00032-4
Hartnell, C. A., Ou, A. Y., & Kinicki, A. (2011). Organizational culture and organizational
effectiveness: A meta-analytic investigation of the competing values framework's theoretical
suppositions. Journal of Applied Psychology, 96(4), 677. Retrieved from
http://search.proquest.com.libproxy1.usc.edu/docview/878997206?accountid=14749
Haydarov, R., Moxley, V., & Anderson, D. (2013). Counting chickens before they are hatched:
An examination of student retention, graduation, attrition, and dropout measurement validity
in an online master's environment. Journal of College Student Retention: Research, Theory
& Practice, 14(4), 429-449. 10.2190/CS.14.4.a
CREATING EFFECTIVE EDUCATIONAL CONTENT 126
Holtzman, Y., & Anderberg, J. (2011). Diversify your teams and collaborate: Because great
minds don't think alike null. Journal of Management Development, 30(1), 75-92.
10.1108/02621711111098389
Huang, H. (2002). Student perceptions in an online mediated environment. International Journal
of Instructional Media, 29(4), 405-422. Retrieved from
http://search.proquest.com.libproxy2.usc.edu/docview/204279480?accountid=14749
Ilgen, D. R., Hollenbeck, J. R., Johnson, M., & Jundt, D. (2005). Teams in organizations: From
input-process-output models to IMOI models. Annual Review of Psychology, 56, 517-543.
Retrieved from
http://search.proquest.com.libproxy.usc.edu/docview/205830055?accountid=14749
Influences on disparity between girls' and boys' achievements in STEM areas. (2010), Summer;
2015/11). 33, 6+. Retrieved from
http://go.galegroup.com.libproxy1.usc.edu/ps/i.do?id=GALE%7CA232175800&v=2.1&u=u
socal_main&it=r&p=AONE&asid=368a7a3ba1b39b8953cb113dd53a4f12
Ionescu, L. (2008). Assessing organizational effectiveness and corporate culture. Economics,
Management and Financial Markets, 3(2), 59-63. Retrieved from
http://search.proquest.com.libproxy2.usc.edu/docview/1032550837?accountid=14749
Jiang, Y., & Kleitman, S. (2015). Metacognition and motivation: Links between confidence, self-
protection and self-enhancement. Learning and Individual Differences, 37, 222-230.
http://dx.doi.org.libproxy2.usc.edu/10.1016/j.lindif.2014.11.025
Kaur, R., & Sidhu, G. K. (2010). Learner autonomy via asynchronous online interactions: A
Malaysian perspective. International Journal of Education and Development using
CREATING EFFECTIVE EDUCATIONAL CONTENT 127
Information and Communication Technology, 6(3), 88-100. Retrieved from
http://search.proquest.com.libproxy2.usc.edu/docview/817561989?accountid=14749
Kessler, L. L. (2014). The effect of organizational culture on it employees turnover intention in
Israel. Annals of the University of Oradea, Economic Science Series, 23(1), 1019-1027.
Retrieved from
http://search.ebscohost.com/login.aspx?direct=true&db=bth&AN=97635660&site=ehost-
live
Kleinsmann, M., Maier, A., Singh, V., Dong, A., & Gero, J. S. (2013). Social learning in design
teams: The importance of direct and indirect communications. Artificial Intelligence for
Engineering Design, Analysis and Manufacturing: AI EDAM, 27(2), 167-182.
http://dx.doi.org.libproxy1.usc.edu/10.1017/S0890060413000061
Kuh, G., & Vesper, N. (2001). Do computers enhance or detract from student learning? Research
in Higher Education, 42(1), 87-102. 10.1023/A:1018768612002
Kumar, S., & Dawson, K. (2014). The impact factor: Measuring student professional growth in
an online doctoral program. Techtrends, 58(4). http://dx.doi.org/10.1007/s11528-014-0773-
2
Kumaran, D., Summerfield, J. J., Hassabis, D., & Maguire, E. A. (2009). Tracking the
emergence of conceptual knowledge during human decision making. Neuron, 63(6), 889-
901. http://dx.doi.org.libproxy2.usc.edu/10.1016/j.neuron.2009.07.030
Larusson, J. A. (2014). In White B. (Ed.), Learning analytics: From research to practice. New
York, NY: New York, NY: Springer.10.1007/978-1-4614-3305-7
CREATING EFFECTIVE EDUCATIONAL CONTENT 128
Lavolette, E., Venable, M. A., Gose, E., & Huang, E. (2010). Comparing synchronous virtual
classrooms: Student, instructor and course designer perspectives. TechTrends: Linking
Research and Practice to Improve Learning, 54(5), 54-61. 10.1007/s11528-010-0437-9
Levy, Y. (2008). An empirical development of critical value factors (CVF) of online learning
activities: An application of activity theory and cognitive value theory. Computers &
Education, 51(4), 1664-1675.
http://dx.doi.org.libproxy1.usc.edu/10.1016/j.compedu.2008.04.003
Luiz, A. J. (2000). Measuring intangible corporate assets. Journal of Intellectual Capital, 1(1),
68-84. 10.1108/14691930010371636
Ma, J., Han, X., Yang, J., & Cheng, J. (2015). Examining the necessary condition for
engagement in an online learning environment based on learning analytics approach: The
role of the instructor. The Internet and Higher Education, 24, 26-34.
http://dx.doi.org.libproxy2.usc.edu/10.1016/j.iheduc.2014.09.005
MacIntosh, E. W., & Doherty, A. (2010). The influence of organizational culture on job
satisfaction and intention to leave. Sport Management Review, 13(2), 106-117.
http://dx.doi.org.libproxy1.usc.edu/10.1016/j.smr.2009.04.006
Marthers, P., Herrup, P., & Steele, J. (2015). Consider the costs of student attrition. Enrollment
Management Report, 19(6), 1-5. 10.1002/emt.30090
Mattick, B., & Miller, D. G. (2006). Teamwork pays. The Journal for Quality and Participation,
29(4), 30-33, 40. Retrieved from
http://search.proquest.com.libproxy.usc.edu/docview/219151541?accountid=14749
CREATING EFFECTIVE EDUCATIONAL CONTENT 129
Mcalpine, I., Koppi, T., Pearson, E., & Mclean, J. (2004). Course Developers as students: A
designer perspective of the experience of learning online. Research in Learning Technology,
12(2)10.3402/rlt.v12i2.11248
McClelland, T. J. (2014). Why do they leave? An exploration of situational, dispositional,
institutional, technological, and epistemological factors on undergraduate student
withdrawal from online studies at an institute of technology in New Zealand (Ed.D.).
Available from ProQuest Dissertations & Theses Full Text, ProQuest Dissertations &
Theses Global. (1656489754). Retrieved from
http://search.proquest.com.libproxy1.usc.edu/docview/1656489754?accountid=14749
Mor, Y., Ferguson, R., & Wasson, B. (2015). Editorial: Learning design, teacher inquiry into
student learning and learning analytics: A call for action. British Journal of Educational
Technology, 46(2), 221-229. 10.1111/bjet.12273
Myers, J. (2015). In Stone-Johnson C., Barba W. and Duran H. (Eds.), Usability in online course
design: Comparative perspectives on ease-of-use, satisfaction, motivation, and teacher
presence ProQuest Dissertations Publishing.
Nandeshwar, A., Menzies, T., & Nelson, A. (2011). Learning patterns of university student
retention. Expert Systems with Applications, 38(12), 14984-14996.
10.1016/j.eswa.2011.05.048
Ortiz-de-Guinea, A., Webster, J., & Staples, D. S. (2012). A meta-analysis of the consequences
of virtualness on team functioning. Information & Management, 49(6), 301-308.
http://dx.doi.org.libproxy.usc.edu/10.1016/j.im.2012.08.003
CREATING EFFECTIVE EDUCATIONAL CONTENT 130
Pearson, A. F. (2010). Real problems, virtual solutions: Engaging students online.
10.1177/0092055X10370115
Rice, J., Rojjanasrirat, W., & Trachsel, P. (2013). Attrition of online graduate nursing students
before and after program structural changes. Journal of Professional Nursing, 29(3), 181-
186. http://dx.doi.org.libproxy2.usc.edu/10.1016/j.profnurs.2012.05.007
Rice, J., Rojjanasrirat, W., & Trachsel, P. (2013). Attrition of online graduate nursing students
before and after program structural changes. Journal of Professional Nursing, 29(3), 181-
186. http://dx.doi.org.libproxy2.usc.edu/10.1016/j.profnurs.2012.05.007
Roberts, T. S., & McInnerney, J. M. (2007). Seven problems of online group learning and their
solutions. Journal of Educational Technology & Society, 10(4), n/a. Retrieved from
http://search.proquest.com.libproxy1.usc.edu/docview/1287042951?accountid=14749
Robillard, P. N., Lavallée, M., Ton-That, Y., & Chiocchio, F. (2014). Taxonomy for software
teamwork measurement. Journal of Software: Evolution and Process, 26(10), 910-922.
10.1002/smr.1641
Schafer, J. A., Heiple, E., Giblin, M. J., & Burruss Jr., G. W. (2010). Critical incident
preparedness and response on post-secondary campuses. Journal of Criminal Justice, 38(3),
311-317. http://dx.doi.org.libproxy2.usc.edu/10.1016/j.jcrimjus.2010.03.005
Schein, E. H. (2010). Jossey-bass business and management: Organizational culture and
leadership (4th edition). Hoboken, NJ, USA: Jossey-Bass.
Seiler, S., Lent, B., Pinkowska, M., & Pinazza, M. (2012). An integrated model of factors
influencing project managers' motivation — findings from a Swiss survey. International
CREATING EFFECTIVE EDUCATIONAL CONTENT 131
Journal of Project Management, 30(1), 60-72.
http://dx.doi.org.libproxy1.usc.edu/10.1016/j.ijproman.2011.03.002
Şendağ, S., & Ferhan Odabaşı, H. (2009). Effects of an online problem-based learning course on
content knowledge acquisition and critical thinking skills. Computers & Education, 53(1),
132-141. http://dx.doi.org.libproxy1.usc.edu/10.1016/j.compedu.2009.01.008
Sørensen, J. B. (2002). The strength of corporate culture and the reliability of firm performance.
Administrative Science Quarterly, 47(1), 70-91. Retrieved from
http://www.jstor.org.libproxy2.usc.edu/stable/3094891
Stanton-Salazar, R. D. (1997). A social capital framework for understanding the socialization of
racial minority children and youths. Harvard Educational Review, 67(1), 1-40. Retrieved
from http://search.proquest.com.libproxy.usc.edu/docview/212284572?accountid=14749
Steven, J. G., & Moore, C. (2014). Patterns of online student enrolment and attrition in
Australian open access online education: A preliminary case study. Open Praxis, 6(1), 45-
54. 10.5944/openpraxis.6.1.95
Stewart, S., Cobb-Roberts, D., & Shircliffe, B. J. (2013). Maintaining pedagogical integrity of a
computer mediated course delivery in social foundations. Journal of Educators Online,
10(2)10.9743/jeo.2-13.2.4
Tan, E., Calabrese Barton, A., Kang, H., & O'Neill, T. (2013). Desiring a career in STEM-related
fields: How middle school girls articulate and negotiate identities-in-practice in science.
Journal of Research in Science Teaching, 50(10), 1143-1179. 10.1002/tea.21123
Toby Marshall Egan, Yang, B., & Bartlett, K. R. (2004). The effects of organizational learning
culture and job satisfaction on motivation to transfer learning and turnover intention. Human
CREATING EFFECTIVE EDUCATIONAL CONTENT 132
Resource Development Quarterly, 15(3), 279-301. Retrieved from
http://search.proquest.com.libproxy2.usc.edu/docview/234904324?accountid=14749
Toby Marshall Egan, Yang, B., & Bartlett, K. R. (2004). The effects of organizational learning
culture and job satisfaction on motivation to transfer learning and turnover intention. Human
Resource Development Quarterly, 15(3), 279-301. Retrieved from
http://search.proquest.com.libproxy2.usc.edu/docview/234904324?accountid=14749
Tompsett, C. (2013). On the educational validity of research in educational technology. Journal
of Educational Technology & Society, 16(3), 179-n/a. Retrieved from
http://search.proquest.com.libproxy1.usc.edu/docview/1462203885?accountid=14749
Tsui, A. S., Zhang, Z., Wang, H., Xin, K. R., & Wu, J. B. (2006). Unpacking the relationship
between CEO leadership behavior and organizational culture. The Leadership Quarterly,
17(2), 113-137. http://dx.doi.org.libproxy1.usc.edu/10.1016/j.leaqua.2005.12.001
Waddock, S. (2004). Creating corporate accountability: Foundational principles to make
corporate citizenship real. Journal of Business Ethics, 50(4), 313-327. Retrieved from
http://www.jstor.org.libproxy.usc.edu/stable/25075200
Wadsworth, M. B., & Blanchard, A. L. (2015). Influence tactics in virtual teams. Computers in
Human Behavior, 44(0), 386-393.
http://dx.doi.org.libproxy.usc.edu/10.1016/j.chb.2014.11.026
Wheeler, S. (2006). Learner support needs in online problem-based learning. Quarterly Review
of Distance Education, 7(2), 175-183. Retrieved from
http://search.proquest.com.libproxy1.usc.edu/docview/231183989?accountid=14749
CREATING EFFECTIVE EDUCATIONAL CONTENT 133
Yukawa, J. (2007). Factors influencing online communication style in LIS problem-based
learning. Journal of Education for Library and Information Science, 48(1), 52-63. Retrieved
from http://search.proquest.com.libproxy1.usc.edu/docview/203227810?accountid=14749
CREATING EFFECTIVE EDUCATIONAL CONTENT 134
Appendix A
Recruitment Email
Hello,
As part of my dissertation for the EdD in Organizational Change and Leadership at the
University of Southern California, I am conducting a research study examining knowledge,
motivational and organizational influences on course developers. If you have developed, or
are in the middle of developing, an online course you are invited to participate in the study. A
link to the survey is provided below and, if you agree, you are invited to also participate in an
interview describing the online Course Development process.
The interview will be held via Zoom, and is anticipated to take 60-90 minutes, and will be video
recorded.
Participation in this study is voluntary. Your identity as a participant will remain confidential
during and after the study. Identifying information will be excluded from transcripts, analysis,
and publication of findings.
CREATING EFFECTIVE EDUCATIONAL CONTENT 135
Appendix B
Survey Protocol
Please complete the following survey. Your participation will help to assist me in completing my
dissertation requirement for an Ed.D. in Organizational Change and Leadership. I understand
that filling out this survey will require an investment of your valuable time. My sincere
appreciation for your cooperation.
Scale
1. Student achievement and engagement is affected by how I plan content with professors.
(strongly agree-strongly disagree)
2. Student achievement and engagement is affected by how I have content built in the
learning management systems. (strongly agree-strongly disagree)
3. I want to continue to learn how I can create content that is engaging to online learners.
(strongly agree-strongly disagree)
4. I feel confident in my ability to identify content that is engaging or not engaging.
(strongly agree-strongly disagree)
5. I understand how online assessments and video lectures work to engage students.
(strongly agree-strongly disagree)
6. I feel that my organization provides me with enough resources to plan engaging content.
(strongly agree-strongly disagree)
7. The organization has provided me enough training to understand how online tools engage
students. (strongly agree-strongly disagree)
8. I want more information that tells me how students have engaged with content in the past.
(strongly agree-strongly disagree)
9. Engaging an online learner requires attention to structuring asynchronous activities.
(strongly agree-strongly disagree)
Yes-No
1. Do you believe that the way that you structure your course affects online student
engagement? (yes-no)
1. Do you try to foster online discussions in the LMS/OC (change terminology based on
program, e.g., MSW@USC calls this the VAC)? (yes-no)
1. Would you like more information on how students engage with the content you create?
(yes-no)
1. What, if any of the items below, drives the means you use to plan and produce content?
a. The goals of the department
b. The goals of the organization
c. Learning experience for students
d. Teaching experience for professors
e. My team
f. Other:_______
1. How do you think, if at all, you can gain more insight into how students engage with the
content you’ve produced?
a. Surveys
CREATING EFFECTIVE EDUCATIONAL CONTENT 136
b. Student focus groups
c. Training
d. Other:_______
1. In thinking about future Course Development, which of the following most influence
your decision on the content you produce?
a. Timeline
b. Pedagogy
c. Student engagement
d. Past professor feedback
e. Other:_______
1. How do you think, if you are interested, you can be provided with more information on
how to build and plan content that engages online students?
a. Wiki
b. In-person training
c. Attending conferences on educational media
d. Other:_______
Open-Ended
1. How do you define student engagement?
1. How, if at all, do you know if you’ve done good work in producing a course? Knowledge
a. Do you have any evidence of success being achieved?
How do you think, if at all, courses designed to require online students to answer questions about
their asynchronous content influence student engagement and motivation?
End
Thank you for taking the time to complete this survey.
CREATING EFFECTIVE EDUCATIONAL CONTENT 137
Appendix C
Interview Protocol
1. How, if at all, do you know if you’ve done good work in producing a course?
2. Do you have an ideal metric?
3. What types of feedback do you get?
4. Does anyone give you feedback about what students say?
5. What does good flow mean to you?
6. What do you use, if anything, as evidence to show if you’ve done good work?
7. How do you define student engagement?
8. What factors, if any, influence your confidence in your ability to plan engaging
online content?
9. Do you solicit feedback from coworkers that are in programs?
10. How many courses have you had to plan?
11. How do you define course planning? Can you identify stages in course planning?
12. What are your biggest challenges in planning?
13. What types of notes for live sessions do you usually make
14. Do you find that profs often know what they want?
15. How often do live session plans tie to asynchronous work?
1. Do you believe that the way that you structure your course affects online student
engagement?
2. What is your approach, if any, for structuring content that will be engaging to students?
3. How often do you talk to co-worker students?
4. What types of questions do you ask?
1. What do you perceive as a factor in how students engage with what you create? Is this
more than one?
2. How, if at all, does the university affect how you plan courses?
1. How, if at all, does the team affect how you plan courses?
2. How, if at all, does Course Planner affect how you plan your courses?
3. How does Course Strategy/CSR affect how you plan courses?
4. How does Course Development leadership affect how you plan courses?
1. Can you give me examples of what you’ve learned from workshops/seminars?
2. Would you want more information as it relates to design or theory?
3. How much time are you devoting to planning?
4. How do you feel about the balance?
5. Can you tell me about your planning process with professors?
6. Do you try to foster online discussions in the LMS/OC?
7. What if anything do you do to make sure these asynchronous discussions tie into live
sessions?
8. Do you get resistance from professors when planning LS?
1. If you were trying to think about increasing student engagement in online courses, what
would you do?
2. How often do professors ask, “What works best for students?”
3. How do you usually answer that?
CREATING EFFECTIVE EDUCATIONAL CONTENT 138
Appendix D:
Informed Consent/Information Sheet
Effective Online Engagement Strategies in Planning Online Courses
You are invited to participate in a research study conducted by Erika Maldonado and Dr.
Anthony Maddox (faculty advisor) at the University of Southern California, because you are a
member of the Content team. This study is not funded. Your participation is voluntary. You
should read the information below, and ask questions about anything you do not understand,
before deciding whether to participate. Please take as much time as you need to read the consent
form. You may also decide to discuss participation with your family or friends. If you decide to
participate, you will be asked to sign this form. You will be given a copy of this form.
PURPOSE OF THE STUDY
The purpose of the study is to understand the knowledge, motivation and organizational
influences that affect how the content team plans and produces online coursework.
STUDY PROCEDURES
If you volunteer to participate in this study, you will be asked to participate in an online survey.
You will also be asked to participate in a one on one semi-structured interview. Your interview
will be conducted via Zoom and will be recorded. However, you may request to not be recorded
if you wish. The interviews will not last longer than one hour, and the survey should take no
more
than twenty minutes
POTENTIAL RISKS AND DISCOMFORTS
You may become uncomfortable answering questions that pertain to your job and perceptions
about online learning. However, your responses will be anonymous. You also may feel that
interview times are inconvenient for you. Therefore, any interviews will occur outside of normal
working hours. There are no financial risks to participating or not.
POTENTIAL BENEFITS TO PARTICIPANTS AND/OR TO SOCIETY
The anticipated benefit is that we will have a deeper understanding of the perceptions of the
department that is responsible for planning and producing online content. As the organization
continues to scale, focusing efforts on research-based practices in developing content will
become more. As this is a research study, the benefits are contingent on the results. There is no
guarantee that they will, in fact, benefit the department as a whole.
PAYMENT/COMPENSATION FOR PARTICIPATION
You will not be compensated in any form for your participation in this research study.
CONFIDENTIALITY
Any identifiable information obtained during the course of the study will remain confidential.
Surveys used during the course of the study will not collect identifiable information including
name or address. Recordings used during the interviewing process will be coded allowing
CREATING EFFECTIVE EDUCATIONAL CONTENT 139
anonymity of the participants. All data gathered will be kept for one year after the conclusion of
the study and then destroyed.
The members of the research team and the University of Southern California’s Human Subjects
Protection Program (HSPP) may access the data. The HSPP reviews and monitors research
studies to protect the rights and welfare of research subjects.
When the results of the research are published or discussed in conferences, no identifiable
information will be used.
PARTICIPATION AND WITHDRAWAL
Your participation is voluntary. Your refusal to participate will involve no penalty or loss of
benefits to which you are otherwise entitled. You may withdraw your consent at any time and
discontinue participation without penalty. You are not waiving any legal claims, rights or
remedies because of your participation in this research study.
ALTERNATIVES TO PARTICIPATION
Your alternative may be to submit written responses to interview questions. Please contact Erika
Maldonado for further information.
INVESTIGATOR’S CONTACT INFORMATION
If you have any questions or concerns about the research, please feel free to contact Erika
Maldonado, Dr. Anthony Maddox or Dr. Ilya Goldin. Erika Maldonado may be reached at (818)
726-1752, Dr. Anthony Maddox may be reached at (213) 675-8100, and Dr. Ilya Goldin may be
reached at: igoldin@2u.com .
RIGHTS OF RESEARCH PARTICIPANT – IRB CONTACT INFORMATION
If you have questions, concerns, or complaints about your rights as a research participant or the
research in general and are unable to contact the research team, or if you want to talk to someone
independent of the research team, please contact the University Park Institutional Review, Board
(UPIRB), 3720 South Flower Street #301, Los Angeles, CA 90089-0702, (213) 821-5272 or
upirb@usc.edu
Abstract (if available)
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Online graduate-level student learning and engagement: developing critical competencies for future leadership roles: an evaluation study
PDF
Will school-based online faculty development be an effective tool for their professional growth?
PDF
Student engagement in online education: an evaluation study
PDF
Student engagement in online learning: examining undergraduate student engagement in online learning communities to improve instruction
PDF
A comparative study of motivational predictors and differences of student satisfaction between online learning and on-campus courses
PDF
Success in the sticky: exploring the professional learning and instructional practices that are sticky for distinguished secondary STEM educators of students historically…
PDF
Full-time distance education faculty perspectives on web accessibility in online instructional content in a California community college context: an evaluation study
PDF
Mind, motivation, and meaningful learning: A cognitive science approach to learning how to learn
PDF
Instructional designers in the new learning paradigm: COVID-19’s impact on roles and responsibilities of instructional designers
PDF
The importance of rigor and engagement on student achievement and mastery: an evaluation study
PDF
Perspectives of learning in synchronous online education
PDF
Effective course design for improving student learning: a case study in application
PDF
Understanding student persistence in massive open online courses (MOOCs): an evaluation study
PDF
Prior learning assessment portfolios: an evaluation study
PDF
Effective engagement of minority alumni: an innovation study
PDF
Digital portfolios for learning and professional development: a faculty development curriculum
PDF
Examining student beliefs and behaviors in online and on-campus courses: measuring openness to diversity, voluntary peer collaboration, and help seeking
PDF
Instructional technology integration in a parochial school district: an evaluation study
PDF
Adaptive learning in higher education: an evaluation study
PDF
An evaluation of general education faculty practices to support student decision-making at one community college
Asset Metadata
Creator
Maldonado, Erika Christine
(author)
Core Title
Creating effective online educational content: an evaluation study of the influences affecting course developers’ abilities to create online content with engaging learning strategies
School
Rossier School of Education
Degree
Doctor of Education
Degree Program
Organizational Change and Leadership (On Line)
Publication Date
08/14/2018
Defense Date
08/14/2018
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
Bloom's,course development,Ed Tech,education technologies,instructional design,learn,learning science,media,OAI-PMH Harvest,online,online learning,teach,video metrics
Format
application/pdf
(imt)
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Maddox, Anthony (
committee chair
), Goldin, Ilya (
committee member
), Hirabayashi, Kimberly (
committee member
)
Creator Email
ecmaldon@usc.edu,erikacmaldo@gmail.com
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c89-69391
Unique identifier
UC11671927
Identifier
etd-MaldonadoE-6746.pdf (filename),usctheses-c89-69391 (legacy record id)
Legacy Identifier
etd-MaldonadoE-6746.pdf
Dmrecord
69391
Document Type
Dissertation
Format
application/pdf (imt)
Rights
Maldonado, Erika Christine
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
Bloom's
course development
Ed Tech
education technologies
instructional design
learn
learning science
media
online
online learning
teach
video metrics