University of Southern California Dissertations and Theses

82nd Training Wing technical training evaluation: an innovation study

(USC Thesis Other)

82nd Training Wing technical training evaluation: an innovation study

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Running head: 82nd TRAINING WING TECHNICAL TRAINING EVALUATION 1

82ND TRAINING WING TECHNICAL TRAINING EVALUATION:
AN INNOVATION STUDY

Shelley L. Bard

A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION

May 2019

Copyright 2019 Shelley L. Bard

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 2

Acknowledgments
I am grateful to the Airman who contributed to every part of this study, from the senior
leaders who encouraged me to examine training evaluation to the painfully young E-2’s anxious
to finish training and start working for the World’s Greatest Air Force. I received nothing but
encouragement, support, and offers of assistance at every turn. Thank you.
I will be forever thankful for the faculty and members of Cohort 6. The USC OCL
program is an amazing opportunity did not exist a short time ago. My sincerest appreciation to
Dr Seli and the faculty and staff who got us over, through and beyond every hurdle. Trust the
process. Their patience as I struggled to describe an environment few are familiar with and their
feedback through draft after draft was priceless. To Cohort 6, I hope we provided each of you
peer support. Here is where I also add “Theresa, you are magic. Thank you.”
My family encouraged me to take this final step in my formal education and fulfill a life-
long goal. I love all of you and I hope you are as proud of your roles in making this possible as I
am of you. I do not often do so within your hearing, but everyone else knows how much I brag
about each of you. Really. They either ask me to adopt them or beg me to shut up. This is
especially true for the MOMD, who too often had to work too hard to keep me on track. For
you, words are not enough.

“In no other profession are the penalties for employing untrained personnel so appalling or so
irrevocable as in the military.” General Douglas MacArthur

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 3

Abstract
Organizations rarely conduct impact-level training evaluation; thought to be costly, overly
complicated in working environments or unwanted by organizational leaders. The 82d Training
Wing (82 TRW) at Sheppard AFB, TX has never conducted comprehensive training evaluation
and as a result, lacked the means to prove that training outcomes add value to major commands
that employ graduates of aircraft maintenance training. The questions that guided this study
asked if training directors within the 82 TRW had the knowledge, motivation, organizational
support, and resources to design and implement impact-level evaluation and if the results would
have value. The stakeholder group of focus for this study included eleven senior civilian
instructional specialists with the capacity to conduct impact-level evaluation for the 82 TRW.
This study applied Clark and Estes’ (2008) gap analysis to data derived from interviews with the
stakeholders. The findings identified that directors possess a high degree of metacognition and
motivation to design and implement impact-level evaluation, although resource distribution to
support additional evaluation was a concern. There was a lack of conceptual and procedural
knowledge about both general and specific training evaluation models and practices by the
training directors, as well as a need for explicit process guidance. Another gap identified that
directors believe negative perceptions exist for training conducted by the wing and of training in
general by operational commands. Recommendations to address these gaps include training to
provide knowledge of the Kirkpatrick New World Model (2016) and provide directed practice
identifying leading indicators that correlate to learning objectives. The end goal is to prove that
training generates measurable strategic impact.
Keywords: Training evaluation. military training, strategic evaluation

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 4

Table of Contents
List of Tables 5
Introduction to the Problem of Practice 7
Organizational Context and Mission 8
Importance of Addressing the Problem 9
Organizational Performance Goal 11
Stakeholder Group of Focus and Stakeholder Goal 12
Purpose of the Project and Questions 13
Methodological Approach 14
The Gap Analysis Conceptual Framework 16
Review of the Literature 16
The Value of Impact-Level Training Evaluation to an Organization 17
Evaluation Models 18
Evaluation Approach by Sector 23
Training Directors’ Knowledge, Motivation and Organizational Influences 26
Knowledge Influences 27
Motivation Influences 31
Organizational Influences 33
Conceptual Framework: The Interaction of Training Directors’ Knowledge, Motivation, and the
Organizational Context 38
Qualitative Data Collection and Instrumentation 44
Interviews 45
Data Analysis 47
Findings 50
Findings for Declarative, Procedural and Metacognitive Knowledge Influences 52
Training Directors’ Motivation Influences 59
Organizational Culture and Context Influences 66
Summary of the Findings 79
Recommendations for Practice to Address KMO Influences 80
Conclusion 81
References 83
Appendix A: Participating Stakeholders’ Sampling and Recruitment, Interview Strategy, and
Rationale 98
Appendix B: Interview Protocol 101
Appendix C: Credibility and Trustworthiness 105
Appendix D: Ethics 107
Appendix E: Limitations and Delimitations 110
Appendix F: Recommendations for Practice to Address KMO Influences 113
Appendix G: Integrated Implementation and Evaluation Plan 128
Appendix H: Influence to Finding to Recommendation Matrix 146

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 5

List of Tables
Table 1: Organizational Mission, Organizational Global Goal and Stakeholder Goal 12
Table 2: Knowledge Influences 30
Table 3: Motivation Influences 33
Table 4: Organizational Influences 38
Table 5: Training Directors’ Knowledge of Training Evaluation Models and Practices 55
Table 6: Training Directors Procedural Knowledge of Training Evaluation 57
Table 7: Training Directors’ Responses Related to Self-Efficacy 61
Table 8: Training Directors’ Motivation Based on Expectancy-Value Theory Components of
Utility and Cost 65
Table 9: Training Directors’ Responses to Changing Evaluation Practices 69
Table 10: Training Directors’ Feelings About MAJCOMs Understanding of Training
Processes 71
Table 11: Training Directors’ Feelings About MAJCOMs Perception of 82 TRW Training 74
Table 12: Training Directors’ Feelings About Procedural Guidance as a Cultural Setting 76
Table 13: Training Directors Feelings About Resources as a Cultural Setting 78
Table 14: Summary of Knowledge Influences and Recommendations 115
Table 15: Summary of Motivation Influences and Recommendations 118
Table 16: Summary of Organization Influences and Recommendations 122
Table 17: Metrics and Methods for External and Internal Outcomes 131
Table 18: Critical Behaviors, Metrics, Methods, and Timing for Evaluation 133
Table 19: Required Drivers to Support Critical Behaviors 134
Table 20: Evaluation of the Components of Learning for the Program. 138
Table 21: Components to Measure Reactions to the Program 140
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 6

List of Figures
Figure 1: Conceptual framework. 41

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 7

Introduction to the Problem of Practice
Organizational performance dependent on complex work relies on training designed and
conducted to meet specific learning needs. However, when the evaluation of technical training
fails to provide meaningful metrics to the organization, senior leaders can only assume that
training added some value to their bottom-line concerns. According to a survey conducted by
the Association of Talent Development (ATD, 2016), a primary organization in the field of
human resource training, fewer than 37% of 199 organizations evaluate their training to
determine any benefit to their organization. No more than 15% evaluated training to measure for
a return on investment (ROI). In the same report, nearly 90% of organizations reported they
regularly assessed how students felt about training as well as student acquisition of knowledge
and skills. While many training elements assess students, instructors, and curriculum, the
majority lack the knowledge, motivation, organizational interest or means to prove that
investments in training result in a measurable benefit. In fiscally austere times, these training
departments are at risk of disappearing if they cannot prove what they add to their organization’s
success.
The number of training evaluation models designed to provide evidence of organizational
benefit or ROI are growing (Aguinis & Kraiger, 2009; Baldwin & Ford, 1988; Bates, Holton, &
Hatala, 2012; Kirkpatrick, 1994; Nafukho, Graham, & Kacirek, 2009; Phillips, 2012;
Stufflebeam, 2003). However, training professionals generally assert that post-training
evaluation, while critical, is too complicated, costly or, conversely, of little value because
training will continue regardless of evaluation outcomes to their organization (Brewer, 2007;
Kirkpatrick & Kirkpatrick, 2016). In high-risk environments, in situ evaluation presents
unacceptable levels of risk from harsh weather conditions, military combat or work that is
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 8

dangerous even without actively hostile conditions (Bell & Reigeleuth, 2014; Kennedy, Chyung,
Winiecki, & Brinkerhoff, 2013; Pulichino, 2007; Shobe & Curtis, 2007). There are more choices
of models available today than in the past 50 years for how and where to evaluate, what to
measure, and who should see the results. Additionally, organizations are coming to feel that
without evidence of impact, training is a questionable use of limited investment funds.
Although training evaluation is problematic in many environments, there is increased
expectation among organizational leadership for proof that a training investment made an impact
on organizational goals or to mission accomplishment (Palmer, 2008; Williams & Nafukho,
2015). Expenditures for training across sectors are expected to grow from an annual $165 billion
reported by the ATD (2015a). According to the Industry Training Report (ATD, 2015b), 777
respondents reported that training consumed 5% of their organization’s budget and an average of
54 hours of employee time annually. As competitive organizations look for cost savings,
training departments need to prepare to prove how much value they contributed to the
organization and that training is not an unavoidable expenditure but an investment, by
establishing impact or a return on expectations (ROE).
Organizational Context and Mission
The United States Air Force (USAF) is organized into ten major commands based on
functional or geographic areas of responsibility. The Air Education and Training Command
(AETC) is one of these major commands and provides much of the technical training and
education for the other nine for both entry-level and advanced skills training (Harrington 2017;
USAF, 2017c). The AETC is further subdivided into numbered air forces: Second Air Force
(2AF) and Nineteenth Air Force (19AF). The 2AF is responsible for career technical and
education training for enlisted personnel.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 9

Six regional commands subordinate to AETC are assigned responsibility for training:
basic military training and advanced combat skills, medical, space and missile, intelligence and
cryptography, communications and weather, and aircraft maintenance. Entry-level and advanced
aircraft maintenance and support courses are primarily conducted by the 82nd Training Wing (82
TRW) and constitute 46% of all USAF technical training. Responsibility for aircraft
maintenance training is shared between three training groups and a Wing Training Operations
department (82 TRW, 2017). A typical training group is comprised of two to four training
squadrons, and each squadron manages from 50 to 500 courses that support a particular type of
aircraft or a common skill set shared across several types of aircraft. Air Force Specialty Codes
have enlisted career field identifiers that indicate general and specific skills and levels of
expertise.
The 82 TRW is located at Sheppard Air Force Base in Wichita Falls, Texas, and
graduates over 60,000 students each year from more than 1,000 courses (82 TRW, 2017). In
addition to its role as a military training center, 82 TRW is also one of one hundred and twelve
affiliated campuses of the Community College of the Air Force (CCAF; AETC, 2014). The
training wing maintains all course and faculty qualifications standards required to maintain
Southern Association of College’s and School Commission on Colleges accreditation for 990
courses and 1,933 civilian and uniformed military instructors who teach them (AETC, 2014; 82
TRW, 2017).
Importance of Addressing the Problem
After decades of assessing students, curriculum, and instructors, it has become evident
that it is also vital to assess training evaluation conducted by the 82 TRW at an organizational
level in relation to the mission goal to train and inspire warriors for several reasons. Training is
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 10

required for nearly every enlisted or officer airman in uniform, from fundamental skills to
advanced training and education (Harrington et al., 2017). The training is specialized and few if
any, external sources offer it (AETC, 2015, 2017b; Shobe & Curtis, 2007; USAF, 2017a).
Military training is also time sensitive; for example, a newly trained avionics technician who will
report for duty at Lakenheath Air Force Base in 2019 will encounter a requirement for a seat in
training established in 2016, well before that young man or woman ever spoke with an Air Force
recruiter (AETC, 2015; Alliger, Beard, Bennett, Symons, & Colegrove, 2013; Harrington et al.,
2017). Personnel rotations for an airman after a first assignment are also programmed well in
advance to allow for additional training time or attrition, particularly in high-risk fields (USAF,
2017b). Maintaining personnel manning levels and mission capability is a perpetual challenge
for operational commands and relies significantly on training programming and availability.
Personnel management practices for the USAF also generate constraints to available
trained and qualified human resources. Once trained personnel fulfill their contracted time, they
may choose to reenlist and compete for promotion, separate from the service, or laterally transfer
to another career field or location where cross-training may be required (Phillips & Foster, 2008;
USAF, 2017b). If an airman is injured, he or she becomes non-deployable, and another airman
must be found to fill emergent requirements after yet more requisite training. Maintaining fully
qualified instructional staff is also an on-going challenge in the face of mandatory personnel
rotation policies and practices.
Hardware, software, and process upgrades also generate new and immediate training
needs to meet new USAF mission requirements (Bell & Reigeleuth, 2014; Harrington et al.,
2017; Shobe & Curtis, 2007). The Secretary of the Air Force (2017) requested $132.4 billion to
operate in the fiscal year 2018. The investment the USAF made in training required a significant
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 11

portion of that funding, and compelling proof of the effectiveness of that training required
documentation and justification as training needs continue to compete with other military and
national demands (Harrington et al., 2017; Salas, Tannenbaum, Kraiger, & Smith-Jentsch, 2012).
While useful within the schoolhouse, training evaluation results of individual student outcomes
were not the data needed by leaders and stakeholders in USAF technical training for strategic
decision-making (AETC, 2017a; USAF, 2017c). The 82 TRW needs strategic, impact-level,
descriptive and accurate training evaluation data meaningful to, the supported major commands
(MAJCOMs) to support training and education investments and guide technical training
direction for today as well as in 10, 20 and 30 years into the future.
Organizational Performance Goal
The USAF Air Education and Training Command Instruction 36-2640, Technical and
Basic Military Training Evaluation (AETC, 2015) mandates training evaluation practices within
the 82 TRW and requires an evaluation process that closely follows three levels of Kirkpatrick’s
(1994) four levels model. The instruction calls for evaluating student reaction to training (Level
1), student acquisition of learning (Level 2), and student transfer of knowledge and skills from
classroom to work environment (Level 3). An equivalent process to measure for the
organizational benefit of training (Level 4) is not included nor required by USAF, AETC or 82
TRW guidance. There was no expectation at the time of this study to measure or analyze metrics
to determine any impact-level value of training (Harrington et al., 2017). The USAF and AETC
were aware of the need to fill this data void because the 2017 2AF Strategic Plan addressed
improvements and innovations that will require impact-level evaluation processes to measure the
impact of training. Table 1 captures the organizational mission, global goal and one stakeholder
goal to provide targeted metrics and support training impact data.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 12

Table 1
Organizational Mission, Organizational Global Goal and Stakeholder Goal
Organizational Mission
The mission of the 82nd Training Wing (TRW) is to train and inspire warriors.
Organizational Global Goal
By December 2022, the 82 TRW will have a comprehensive training evaluation system
in place that provides for impact-level results as well as assessment of training transfer,
learning acquisition, and student reaction.
Stakeholder Goal
By December 2020, 100% of training directors within the 82 TRW will identify leading
indicators of an impact-level training evaluation system to measure and analyze
organizational benefits of training.

Stakeholder Group of Focus and Stakeholder Goal
The USAF training organization is one of the largest in the world (CCAF, 2017). It is
also well-defined, with a structured organizational chart that identifies strategic, tactical, and
operational levels of responsibility to conduct more than 2,700 technically challenging courses
for more than 150,000 Airmen each year (AETC, 2017b; Harrington et al., 2017). There are
stakeholders throughout the USAF and AETC chain of command who make essential
contributions to achieving the organizational goal and overall mission of the 82 TRW. At a
strategic level, the MAJCOM Functional Managers (FMs) are the stakeholder group most
concerned with the impact of training on mission accomplishment. While each stakeholder
group is critical to the process of developing and delivering technical training, the key
stakeholder group of focus for this study was 11 training directors within the training wing
organization with day to day responsibility to implement decisions that support strategic, long-
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 13

term training planning, delivery, and evaluation. The training directors are group- and squadron-
level leaders who plan, lead, and manage both long-term strategic and day-to-day tactical
training requirements in response to USAF future needs (Deis, 2010).
Purpose of the Project and Questions
The purpose for this study was to collaborate with training directors within the 82 TRW
to recommend an innovative implementation of impact-level training evaluation that will provide
strategic data to inform and influence MAJCOM perceptions and decision-making for the
training conducted by the 82 TRW. In fiscal year 2017, the wing conducted 46% of the USAF
enlisted technical training, offered 1,009 different courses delivered by 1,933 instructors, and
graduated 62,724 students (82 TRW, 2017). The graduates of training conducted by the wing are
subsequently assigned to other MAJCOMs in the USAF, but the primary benefactors of training
are the Air Combat Command, the Air Force Global Strike Command and the Air Mobility
Command who provide most of the funding and resources that support training. These three
MAJCOMs track performance metrics based on their mission effectiveness, but there are
currently no correlations made between training metrics and mission accomplishments. This
innovation proposes an implementation to provide an essential link between MAJCOM mission
outcomes.
Training evaluation is a mandatory process directed by instructional guidance in the
United States Air Force Instruction 36-2201, Military Training (USAF, 2010) and AETCI 36-
2640, (AETC, 2015). Like many organizations, the 82 TRW regularly evaluates student
knowledge and skill acquisition and can assess the transfer of learned skills into military work
environments. As is found in civilian organizations, technical training groups of the 82 TRW
also struggle to conduct and produce training evaluation results that prove that training delivered
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 14

was precisely the training needed, at the same pace as rapidly developing technology (Bell &
Reigeleuth, 2014; Giangreco, Carugati, & Sebastiano, 2010; Harrington et al., 2017; Pulichino,
2007; Salas et al., 2012). This study examined knowledge, motivation and organizational return
on expectations or impact-level evaluation system. Four research questions guide this inquiry:
1. What are the training directors’ knowledge and motivation needs related to measuring
impact-level evaluation?
2. What USAF cultural influences affect implementation of a system to measure impact-
levels?
3. What USAF resources are needed to implement an impact-level evaluation system?
4. What recommendations will bridge knowledge, motivation or organizational culture gaps
related to impact-level training evaluation within the 82 TRW?
Methodological Approach
This study was designed as a qualitative study, with data developed from interviews with
the 11 training directors who are the sole source within the 82 TRW with the professional
knowledge and experience to understand training evaluation needs, processes, and results. The
reasoning behind selecting a qualitative study was to apply phenomenologically and capture the
lived experiences of a select group of participants (Creswell, 2014). A document review of
previous training evaluation expectations, processes or results other than what is detailed in the
current instructions and process guides was not possible as no research on this specific topic has
been conducted within the 82 TRW to date, and no documentation or artifacts were found that
examined impact-level evaluation practices or results. The sample size for data collection for
this study was limited to the 11 training directors within the wing, a population too small to
support a quantitative approach. Interviews are an effective means of collecting data not readily
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 15

apparent from observation, or for which there is little documentary evidence (Patton, 2002). The
interviews conducted provided both explicit comments related to the research questions and as
importantly, the context for those responses from non-verbal reactions or responses accompanied
by certainty or hesitation before a tentative response (Patton, 2002). Qualitative methodology
allowed the researcher to explore the knowledge, motivation, and experiences of each training
director and to add depth to their initial responses.
A qualitative approach examines the beliefs, knowledge, perceptions and motivation of
stakeholders in context (Maxwell, 2013; Merriam & Tisdell, 2016). For this study it, allowed the
researcher to examine knowledge, motivational and cultural influences within the 82 TRW.
How a squadron conducts day-to-day training evaluation of students and instructors is directed in
detail by USAF instructional policy and procedures (AETC, 2014), but experience and deep
understanding of what strategic assessment is by senior leadership throughout the 82 TRW
varied widely due to the differences in previous assignments, formal education, and the cultural
contexts of individual training directors. Interviews were conducted with each training director
to determine levels of knowledge and motivation as well as the cultural perceptions and beliefs
about training evaluation that would need to be addressed to achieve organizational and
stakeholder goals.
Locke, Silverman, and Spirduso (2010) stated that the value of qualitative data is in the
flexibility in the design of the research that allows a researcher to provide vivid descriptions and
context. Quantitative data were readily available from the training groups that show that
students succeed in the training provided, but these were not the kind of data needed to provide a
sufficiently detailed and comprehensive story of training evaluation within the training wing. As
Creswell (2014) and Malloy (2011) identified, a qualitative investigation may collect meaningful
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 16

data from a face-to-face, social connection between investigator and interviewees and add depth
to the picture created. Therefore, to collect data related to the knowledge and motivation of each
training director and their perceptions of the organizational influences on the conduct of training
evaluation, a qualitative approach was selected as the preferred research methodology for this
study.
The Gap Analysis Conceptual Framework
The Clark and Estes (2008) gap analysis model was followed to direct the design and
implementation of this study as well as the recommendations that resulted. Within the design
was the search to identify gaps in knowledge, motivation and organizational shortfalls that could
interfere with the training director’s ability to identify leading indicators of training evaluation.
The implementation of the study maintained the focus of the research questions, the interview
protocol, and the findings on the gap analysis design. Finally, the gap analysis framework
focused recommendations on data-proven gaps in knowledge, motivation, and organizational
influences to ensure the stakeholder goal was achieved.
Review of the Literature
This section presents an analysis of current research on training evaluation. The review
included results that described leadership perceptions of training evaluation, current training
evaluation models in use, and how training evaluation was used in different industries. The
purpose of this literature review was to determine if there was empirical support for the premise
of this study that training evaluation results supportive to organizational planning by senior
leaders need to be measured and presented strategically.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 17

The Value of Impact-Level Training Evaluation to an Organization
Impact-level evaluation to ascertain organizational benefits and shortfalls of training has
a broad and diverse history in the literature. There is evidence that training professionals believe
that impact-level evaluation is burdensome, unnecessary, and unwanted by organizational
leadership (Guerci & Vinante, 2011; Pineda, 2010). There is also evidence that organizational
leaders from across sectors feel that training evaluation results presented in strategic language
and against strategic goals were found to be of significant interest (Palmer, 2008; Pineda, 2010;
Preston, 2010; Salas et al., 2012). This research indicated that senior leaders were aware that
effective training reduced costly mishaps, was the best method to keep employees current on
technology upgrades and contributed to maintaining a competitive edge, all areas defined as
strategic goals (Palmer, 2008; Percival, Cozzarin, & Formancek, 2013; Salas et al., 2012). The
importance of impact-level training evaluation exists but needs to be fully recognized as
significant to different groups of stakeholders by the training professionals responsible for
evaluation design and conduct.
The value of impact-level training evaluation to an organization’s senior leaders was in
direct proportion to how the evaluation was designed, analyzed, and reported. Impact evaluation
models that used leading indicators aligned to stakeholder values. Additionally, the effectively
measured at an organizational level what and how training contributed to the mission, vision, and
goals (Kirkpatrick & Kirkpatrick, 2016; Preston, 2010; Salas et al., 2012). How to determine
and communicate the efficiency and effectiveness of training is highly dependent upon
organizational attributes and expectations (Percival et al., 2013). A growing number of
organizations have become aware that impact-level training evaluation results based on the
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 18

organizations’ strategic metrics increased competitiveness, reduced inefficiencies, and improved
their bottom line when training was designed to include impact-level evaluation from the start.
Evaluation Models
Current training evaluation models are not one-size-fits-all; each category of model
manifested strengths and weakness. An approach like that of Kirkpatrick’s (1994) four levels
worked well enough for some organizations but did not provide ROI results, addressed by the
Phillips (1998) model. Some designs are broadly useful if shallow, and some are deep and
narrowly specific to a field or a single specialized training evaluation question (Giangreco et al.,
2010; Holton, 1996). The literature review examined a cross-section of current models used for
technical training and education. It included Kirkpatrick’s four levels and variations including
the Phillips model, results- and decision-based models like Stufflebeam’s (2003) context-input-
process-product (CIPP) model. The training evaluation models selected were not comprehensive
as a multitude of models were variations of each of a model identified. These examples were
selected to represent unique characteristics or approaches that current research indicated were
used for more than a single study.
The Kirkpatrick New World Model and the Kirkpatrick taxonomy. Published in
2016, the Kirkpatrick’s brought the original Kirkpatrick Four Levels into the 21
st
century by
adding processes that defined it as a model. The New World Model presented methodology by
which organizations could meaningfully assess impact-level training evaluation by starting the
design of raining with Level 4 – to measure the value of training to the organization, and
maximize the results of Level 3, 2 and 1 evaluation. Recommendations from this study relied on
the New World Model to design and implement recommendations to address the problem of
practice.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 19

The forerunner to the New World Model, Kirkpatrick’s (1959a, 1959b, 1960a, 1060b)
four levels approach remains one of the most commonly used training evaluation constructs.
The four levels generated a shift in how organizations and training specialists thought about,
discussed, and conducted training evaluation. According to Thalheimer (2018), Kirkpatrick’s
thinking was influenced by Raymond Katzell’s 1950’s hierarchy of steps model. Katzell (1952)
identified distinctive stages of knowledge and skill acquisition from training that was seminal if
less elegant in a training context than Kirkpatrick’s descriptions. However, as Katzell’s focus
was from an industrial psychology perspective, Kirkpatrick was free in the field of training
development to revise the steps to levels and simplified the definitions of each without losing the
intent, making it a useful and popular tool with which to explain training evaluation to non-
trainers, and seminal in the field of training.
More accurately described as a taxonomy, it became a generic eponym for a category of
training evaluation models (Holton, 1996). One of the significant strengths was in its use as a
clear and straightforward way to explain training evaluation goals in successive terms to
organizational leadership (Kirkpatrick & Kirkpatrick, 2016; Salas et al., 2012). It was simple,
adaptable, and beneficial for organizations interested in outcomes but not in the details of how
training evaluation was planned or conducted (Phillips, 1998). It was also subject to
misunderstandings and misuse when organizations failed to implement a comprehensive
evaluation program, fractured the Kirkpatrick approach or misunderstood the intent of the four
levels (Giangreco et al., 2010; Holton, 1996; Kirkpatrick, 1994; Williams & Nafukho, 2015).
The four levels remained substantially unchanged while also subject to suggested
updates, revisions, and modifications that attempted to reflect more modern needs and the goals
of new research directions in training evaluation (Pineda, 2010; Singh, 2013). Kirkpatrick and
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 20

Kirkpatrick’s (2016) model’s refinements incorporated the identification and use of leading
indicators: critical behavior points an organization can regularly sample and assess to determine
if employees applied what they were trained to do. Additionally, the New World Model
emphasized beginning training evaluation design at Level 4 and working downward to Level 1.
This ensured that impact-level training evaluation did not get left behind or left out altogether.
Integrated models. A review of training evaluation models does not lend itself to an
easy categorization of types of models, but existing research does provide a means of identifying
types of models with more similarities than differences (Singh, 2013). The commonality of these
models is in the importance of recognizing parity between student and organizational gains from
training, although the influence of Kirkpatrick (1994) was evident throughout (Alvarez, Salas, &
Garofano 2004; Salas et al., 2012). Integrated models include those models that contained a
notable variation from elements or levels of Kirkpatrick’s approach.
Integrated models differ from the Kirkpatrick-based models by setting aside the
progressive approach of levels and equalizing training evaluation emphasis throughout the
design and delivery of training. The integrated models viewed training evaluation as a balance
between an organizational acceptance of training and individual student’s performance outcomes
to determine whether training was considered successful (Salas et al., 2012; Singh, 2013).
Criticism also arose that integrated models failed to fully account for influences within an
organization’s culture that impacted training evaluation results (Alvarez et al., 2004). While
there remained traces of Kirkpatrick’s influence in integrated models, most integrated models
emphasize factors that the four levels did not examine.
Decision- and results-based models. The decision- and results-based category of
training evaluation model included those that emphasized both student outcomes and
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 21

organizational gains. The Stufflebeam (2003) CIPP model grounded the theoretical, perfect
world approach of earlier models into real-world working environments and additionally
expected the evaluator to be explicitly trained in evaluation techniques (Stufflebeam, Madaus, &
Kellaghan, 2000). The CIPP model is an example of decision- and results-based design
specifically to provide strategic-level training evaluation results to senior leadership within
organizations (Miller, King, & Mark, 2008). More than any other approach or strategy for
training evaluation, CIPP was a departure from the graduated, progressive design of Kirkpatrick-
based models but continued to emphasize the need for each component in a continuous process.
With the focus of evaluation on the expertise of the evaluator as much as on the training to be
evaluated, the CIPP model was a forerunner of authentic evaluation methods, particularly when
assessing in complex environments.
Swanson and Holton (2009) presented another results-based model that provided an
example of a model that emphasized student results and the financial demand of training on an
organization. However, as with other models in this category, the design was limited in
application and quickly became unwieldy and complex in application. Singh (2013) concluded
that no one evaluation model provided all of the training outcome data needed or desired by an
organization.
Twitchell, Holton, and Trott (2000) conducted a broad study of technical training
evaluation with an expectation that technical training professionals would conduct Level 3 and 4
evaluations more frequently than any other sector training. This study found that there was little
difference between the conduct of technical and non-technical training evaluation, the
percentages of organizations that conducted Levels 1 through 4 were about the same as 40 years
earlier, and there was a notable overall decline in the use of each type of progressive level of
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 22

assessment, regardless of the specific model used. The study also identified the top three reasons
offered for why Level 4 evaluation was not conducted: evaluation was not required by the
organization, training to conduct evaluation was lacking, and training evaluation was too costly
to undertake.
Singh (2013) presented a series of less familiar and more complex or specialized models.
The discrepancy model was criterion-based and process-oriented and possibly suitable for
technical training specifically with additional details established. However, it required that goals
for evaluation be identified well in advance of training delivery, making it a challenge when end-
goals for training were never identified before training was designed. The transaction model was
both objective and subjective and relied on an iterative process loop using both quantitative and
qualitative data but became costly due to its demand for resources. Also, the transaction model
failed to address the collection of metrics relevant to organizational interests. The goal-free
model is client-driven and objective, with no predetermined outcomes. It also lacked standards
to define what was successful about the training. The systemic model examined points of
intersectionality within a whole system of training, was organization-oriented and cost-benefit
sensitive. This model was like taking a temperature at set points and assessing current readings
against baseline results. The systemic model has several elements that are like the New World
Model (Kirkpatrick & Kirkpatrick, 2016). Cervero‘s (1985) continuing education evaluation
model was comprehensive and somewhat complicated for the environment for which it was
intended. It consisted of seven categories, from program design through outcomes. However, it
failed to address a ROI for training.
Various models have been used in attempts to address perceived shortcomings in current
training evaluation practices and to mature beyond Kirkpatrick’s (1994) four levels concept. The
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 23

models added layers of complexity that fixed a shortcoming but constrained broader
understanding, acceptance or adoption and created shortfalls of their own (Singh, 2013).
Another consideration evident from this review was that, while many of these models were
useful for human resources soft skills training, only the CIPP model explicitly acknowledged or
identified the unique nature of hard skills training evaluation needs.
Evaluation Approach by Sector
There was insight into the value of training evaluation to an organization for both hard
and soft skills training evaluation models used by organizations. This literature review included
an examination of how businesses, healthcare organizations, public agencies, and military use,
measure, and value technical training evaluation. Research in this area revealed identification of
barriers and benefits as well as an understanding of why some organizations were successful at
impact-level training evaluation and some were not.
The business sectors. The purpose of this section was to review the literature for
evidence of how business organizations generally think of training and evaluation. The business
sector was an umbrella label that extended over large and small organizations with a variety of
approaches to training and training evaluation. The first study examined a small logistics
company interested in two ROI results: return on monetary investment and a return of social
capital (Curado & Teixeira, 2014). The second study considered hotel training evaluation by
managers and results based on both Kirkpatrick’s and Phillip’s approaches (Ho, Arendt, Zheng,
& Hanisch, 2016). The third organization was not interested in ROI specifically but succeeded
in satisfying leadership of the effectiveness of training to the organization (Baraldi & Cifalino,
2015). The fourth study examined a larger company with a chief financial officer and questions
whether ROI was the correct measure for training evaluation (Andru & Botchkarev, 2011). In all
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 24

cases, findings indicated that leadership valued results that were directly useful to how they were
required to tell the story of the organization’s accomplishments, not by the the number of
students trained but in how the training improved the organization’s ability to sell, produce or
meet goals.
The healthcare sectors. Within the healthcare sector, training evaluation research
provided abundant detailed studies and results. The same limitations of time, cost, and the lack
of expertise persistently obstructed organizational efforts to conduct the ROI or impact-level
training evaluation that other sectors reported (Attree, 2006; Hill, 1999; Preston, 2010). It also
appeared that, within the healthcare sector, subject matter experts who conducted training were
primarily medical service providers who train as opposed to professional trainers and educators
who support healthcare providers (Baraldi & Cifalino, 2015). For example, Preston (2010)
identified several factors that attenuate training evaluation results: poor metrics with little
meaning to leaders, incomplete data and half-told stories, unmemorable reports with no
connection to the people within the organization and training data that failed to align with the
organization’s interests. As a result, while there was current research available for the sector,
there was little substance found that directly discussed the results of training evaluation.
The public sector. A review of the literature of training evaluation in the public sector
identified a wide variety of research on training evaluation effectiveness. Included were two
surveys of the training professionals’ membership of the ATD: one quantitative and one
qualitative (Brewer, 2007; Phillips, 2003). There was also a longitudinal 6-year study of a law
enforcement training agency’s training evaluation processes and utilization (Burnett, 2012).
Finally, there were two studies of training mandated by the policies of government agencies: one
in the United States and the other in India (Hypes, 2016; Kodwani, 2017). What became
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 25

apparent from public sector training evaluation was that, while training evaluation was regularly
conducted, the focus of research was on skills acquisition with emphasis on student learning and
not on an organizational advantage. It was unclear if organizational benefits were defined or
recognized as an element in training evaluation.
The military sector. Unlike other industries included in this study, the military
environment emphasized unique challenges and considerations when evaluating training (Bell &
Reigeleuth, 2014; Salas, Milham, & Bowers, 2003; Shobe & Curtis, 2007). Ineffective training
for soft skills may deprive an organization a customer or a sale, but ineffective training in
technical fields can result in injury or loss of life (Williams & Nafukho, 2015). For example, it
was noted as necessary to consider the complexity and risks associated with real-time combat
behaviors, explosive ordinance disposal learning transfer, or aircraft and munitions maintenance
practices performed in-flight (Aaberg & Thompson, 2012; Harrington et al., 2017). Despite
inherent hazards, there was also evidence of a growing awareness at the executive leadership
level of military organizations of the importance of conducting training evaluations at all levels
to understand the value of training within the military sector, at the strategic or long-term
decision-making level.
The military and other challenging training environments were not stringently assessed in
the past when other sectors were beginning to seek evidence of effectiveness and efficiency. It
has only recently become apparent that the unique evaluation requirements associated with
specific fields such as combat-related requirements within the military services demand a
different approach to produce useful results (Harrington et al., 2017; Williams & Nafukho,
2015). It was problematic to assess in specific fields comprehensively, but there was also a
growing requirement driving the search for an active process that could be widely applied, if not
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 26

across business sectors, then within the spectrum of military training and similarly challenged
organizations.
The review of current literature on training evaluation models, practices and results
provided a general idea of what types of models exist and of the differences and similarities
between them. It provided a lens through which to examine senior leadership perceptions of the
value of training and training evaluation results. It also captured some of the similarities and
differences between currently used training evaluation models. It considered how training
evaluation was used in different fields, with divergent results and conclusions of usefulness and
utility. Finally, it served to highlight, by comparison, some of the potential differences that high-
risk training environments and fields must consider when designing Levels 3 and 4 training
evaluation.
Training Directors’ Knowledge, Motivation and Organizational Influences
The training director’s stakeholder goal was that, by December 2020, all training
directors within the 82 TRW would identify leading indicators of an impact-level training
evaluation system to measure and analyze training. This section examines the knowledge and
motivational influences of the key stakeholders related to this goal, and the organizational
influences that support or interfere with achieving the goal. Only with data-driven evidence of
what exists and what is needed for success may an effective plan be designed to implement this
innovation.
More than talent or even aptitude, the active learning of targeted knowledge and skills,
the transfer of that learning to a workplace, and the motivation to apply it are the influences that
best support change within an organization (Clark & Estes, 2008). Further, when gaps exist in
workplace knowledge, skills, or motivation or organizational support, employees will manifest
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 27

frustration and a lack of commitment that inhibits and obstructs effective and lasting change
(Rueda, 2011). The influences needed to support the stakeholder goals identified for this study
were examined in the following paragraphs.
Knowledge Influences
Four types of knowledge influences are critical to achieving a stakeholder goal: factual,
conceptual, procedural, and metacognitive (Clark & Estes, 2008; Krathwohl, 2002; Rueda,
2011). According to Anderson (2001)), factual and conceptual together are grouped under
declarative knowledge, as facts are basic and stand alone, but have no value without
applicability. Conceptual knowledge describes the links and relationships needed to make sense
of facts (Hiebert & Lefevre, 1986). Procedural knowledge includes how to use or apply facts
and conceptualizations; steps or processes that result in actions taken (Krathwohl, 2002).
Metacognition identifies an opportunity to reflect on the earlier stages of knowledge and how to
use it.
For training directors to successfully identify and correlate leading indicators to training
objectives to produce impact-level training evaluation results, they must first know what impact-
level evaluation is, how it is useful and how it is designed to function. Training directors must
also know how to identify, correlate, and regularly measure leading indicators. Finally, to learn
and apply these facts, concepts, and procedures, and lead in implementing impact-level
evaluation, training directors must practice metacognition and reflect and understand how they
learn and think before they, in turn, disseminate training evaluation knowledge throughout their
squadron or group. Knowing where gaps in knowledge and skills exist in a workforce identifies
where to apply training with the greatest effect. These influences provide a lens through which
to identify stakeholder knowledge or lack of knowledge.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 28

Knowledge of impact-level evaluation and the purpose of correlating and measuring
training objectives and leading indicators. The knowledge of training evaluation needed by
training directors before achieving impact-level evaluation includes knowledge of the definitions
of impact-level evaluation and leading indicators, examples of declarative knowledge. This
knowledge includes the necessary application of facts that provide an essential foundation for
more complex learning of why, when and how-to use the knowledge (Krathwohl, 2002).
Further, a stable foundation based on declarative knowledge provides a common language to
support the propagation of the desired concepts and procedures.
Aaberg and Thompson (2012) conducted a case study that demonstrated the importance
of knowledge influences in implementing impact-level training evaluation in a military
environment. They fused an evaluation design from both the Kirkpatrick and CIPP models to
evaluate training of a lengthy technical training course conducted by the U.S. Navy. They
reported positive results, in part by explicitly finding declarative, procedural, and metacognitive
knowledge gaps throughout the command. Instruction and evaluation were aligned and uniform
and consistent training provided to close the knowledge gaps. The result was sustained learning
by students, sustained assessment practices by instructors and increased awareness of the value
of training evaluation by training command leadership.
Knowledge of procedures to identify, correlate, and measure training objectives and
leading indicators. The 82 TRW is highly experienced with student, course, and instructor
evaluations and how results influence can revise training. What is missing is supporting data that
indicates that training provided to students results in technicians with knowledge and skills of
use to the MAJCOMs. Implementing impact-level evaluation practices provides the training
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 29

directors a significant method of refining training to meet the needs of the MAJCOM
stakeholders.
In a mixed-methods study to examine the intersectionality of evaluation levels and
techniques used in technical training, Guerci and Vinante (2011) identified that different groups
of stakeholders have different needs from training evaluation and that what satisfied one group
was not necessarily valued by other groups. A psychometric measurement of Air Force training
mission essential competencies established that, while the identification of competencies is
relevant, it is more useful to ascertain that what is taught is what needs to be taught (Alliger,
Beard, Bennett, Symons & Colegrove, 2013).
Reflecting on capacity to design and implement impact-level training evaluation.
Training for simple topics is not simple, and training for advanced topics is significantly more
complex and benefits from metacognitive reflection and regulation of training (Baker, 2006). It
is critical that as training directors design and implement impact-level evaluation, they reflect on
the similarities and differences of their own experiences with training evaluation, and with the
variations among stakeholder groups expectations and cooperation, and adjust their strategies
accordingly (Clark & Estes, 2008). Finally, training directors are the group of stakeholders with
capacity to directly influence the success of changes made in training evaluation; their decision-
making in the design and implementation requires metacognition of their knowledge, motivation,
and organizational support requirements. As the identification and use of leading indicators has
not been used by the 82 TRW, exploration of training director metacognitive practices was
designed to be broadly assessed and supported.
Table 2 provides the declarative, procedural, and metacognitive knowledge elements
considered necessary to the achievement of the stakeholder goal for this study. Training
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 30

directors must understand what impact-level evaluation is, and how it can be used and valued at
a strategic level for decision-making. They must also grasp leading indicators as metrics of
interest to MAJCOMs and FMs, instead of traditionally reported quantitative data of how many
students were trained or how many courses exist to support MAJCOM needs. Of interest to the
MAJCOM is training evaluation data that indicates whether the MAJCOM is effectively and
efficiently meeting or exceeding its own goals. One example of this is with a MAJCOM
tracking the number of safety mishaps that occur with MAJCOM aircraft operations each month.
MAJCOM leadership closely monitors the number of mishaps over time, to identify and address
trends that indicate emergent problems. If schoolhouse safety training improves, then safety
violations and mishap reporting data reflect improvements as well.
Table 2
Knowledge Influences
Knowledge Type
Conceptual

Procedural

Metacognitive

Knowledge of what impact-level evaluation
and leading indicators are, and what they can
be used to measure. (Aaberg & Thompson,
2012; Krathwohl, 2002; Kirkpatrick &
Kirkpatrick, 2016; Rueda, 2011).

Knowledge of procedures to identify,
correlate, and measure training objectives and
leading indicators (Kirkpatrick & Kirkpatrick,
2016; Krathwohl, 2002; Rueda, 2011).

Reflecting on capacity to design and
implement impact-level training evaluation
(Baker, 2006; Clark & Estes, 2008;
Krathwohl, 2002; Rueda, 2011).

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 31

Motivation Influences
Motivation influences describe what can drive people to think about what can be done,
consciously choose to act and then continue to act in the interests of the organization or
themselves (Clark & Estes, 2008). Insufficient motivation influences are another reason that
organizations fail to achieve change (Clark & Estes, 2008). Motivation is the degree of
enthusiasm, and effort one exercises to accomplish a goal or purpose and results from active
choice, persistence, and mental effort (Clark & Estes, 2008; Mayer, 2011). Clark and Estes
(2008) go on to assert that active choice occurs in conjunction with intention and action,
persistence is focus and concentration in the face of distractions, and mental effort is a
combination of energy and enthusiasm. The motivation influences identified as most influential
to 82 TRW training directors included self-efficacy theory and expectancy-value theory. Table 3
shows the essential motivation influences related to the training director’s stakeholder goal
accomplishment.
Efficacy about being able to identify, correlate, and use leading indicators for
impact-level training evaluation. Self-efficacy theory describes one’s belief in their own
ability to overcome obstacles and accomplish a task or goal, and explains the dedication and
focus an individual applies toward that task or goal (Bandura, 1994). Additionally, self-efficacy
is the manifestation of a strong internal locus of control or agency, eschews luck or external
influences, and leads to accomplishment (Pajares, 2006). Individual or collective self-efficacy
can increase the desire or intent to act as well as the persistence one or a group are willing to
invest (Bandura, 1994; Pajares, 2006; Rueda, 2011). Finally, to close motivation influence gaps,
Pintrich (2003) recommends designing training to increase self-efficacy among other
motivational influences.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 32

Training directors are training specialists successful and confident in their field, with an
earned sense of self-worth based on well-established and effective skills and records of the
improvements they generated to benefit the 82 TRW. Grossman and Salas (2011) researched
characteristics that predict the successful transfer of learning and identified sixteen significant
traits or elements that relate to success. Self-efficacy was second to cognitive ability as an
indicagtor of the successful transfer of learning. Provided with evidence that identifying leading
indicators of training that correlate to learning objectives will result in improved training, the
training directors are expected to possess a strong belief in their capacity to learn, train and
implement impact-level evaluation within their squadrons.
Value for implementing impact-level training evaluation. When an individual
perceives the importance of a task and has determined that they are capable of the work needed
to achieve the task, that individual is more likely to both begin and persevere in the task (Eccles,
2006; Mayer, 2011). According to Eccles (2006), expectancy value is a transactional motivation
theory of four components; utility, cost-benefit, attainment, and value. The utility of an action is
measured based on usefulness and application with the anticipation of something advanced or
gained. The cost-benefit component is a judgment of worth, a determination of whether the
action will deliver juice that is worth the squeeze. Attainment is the element that weighs one’s
knowledge and skills and produces confidence and perseveration. The final component is a
decision that the action is a source of intrinsic pleasure or external worth as another part of the
internal transactional assessment that becomes an action performed with the expectation of an
outcome that will generate enough value (Wigfield & Cambria, 2010).
The 82 TRW and military technical training throughout the Department of Defense has a
long and successful history of the application and pragmatic truth in expectancy-value theory in
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 33

any learning environment. Most training courses developed for use by the military follow the
Analysis, Design, Development, Implementation, and Evaluation (ADDIE) instructional design
model that intentionally provides a reason to learn, an award system, assurance of the utility of
learning objectives and clearly defined cost-benefit analysis (Allen, 2006). Given proof of the
value of impact-level training evaluation to all stakeholders with an interest in technical training
results, training directors will be motivated to accomplish the stakeholder goal because they
practice the elements of expectancy-value theory.
Table 3
Motivation Influences
Motivation Influences
Self-efficacy Theory

Expectancy-Value Theory

Efficacy about being able to identify, correlate and use
leading indicators for impact-level training evaluation
(Bandura, 1994; Pajares, 2006; Pintrich, 2003; Rueda,
2011.

Value for implementing impact-level training evaluation
(Eccles, 2006; Mayer, 2011; Rueda, 2011; Wigfield &
Cambria, 2010).

Organizational Influences
In addition to knowledge and motivation influences, those influences rooted in an
organization's culture as beliefs, traditions and unwritten rules, and as published policy, work
aids, and process guides can make or break an organization’s goal achievement (Clark & Estes,
2008). Even when employees possess the requisite knowledge and are highly motivated to
accomplish an end goal if they do not receive active and conscious support of the organization
they are unlikely to achieve or sustain success (Clark & Estes, 2008; Rueda, 2011). Gallimore
and Goldenberg (2001) describe cultural influences as intangible models and tangible settings
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 34

that are shared schema that define an environment of two or more people. Cultural influences
are frequently unconscious and no matter how ridiculous they can appear to an outsider, applied
without question by members of the organization (Schein, 2004). How an organization will
respond to and support change will determine the success and longevity of change.
Cultural models. Cultural model influences are those unseen drivers that stem from the
study of how the world works and how organizational beliefs and practices contribute to
successful change or the successful resistance to change (Schein, 2004). Clark and Estes (2008)
assert that effective change is dependent on including the consideration of organizational beliefs.
Because they are frequently unconscious beliefs, cultural model influences are often overlooked
or underestimated for the damage they can wreak on the best of change planning. During this
study, AETC rescinded instructional guidance that explicitly mandated processes and
requirements for training evaluation, leaving only unquestioned, non-directive process guides
based on habit and residual habits. As a result, the following influences are considered
intangible cultural models and not settings.
There are three cultural model influences to consider within the 82 TRW identified in
Table 4 considered in this study. The first is how training directors think about impact-level
training evaluation, an innovative idea that requires approaching training evaluation from a
different direction than current practices dictate. The second influence is support for or
resistance to change in training evaluation practices at every level up to the MAJCOM, as
training directors are only one of several stakeholder groups interested in the delivery of
technical training. The third influence is the external perception of the value of training; if the
MAJCOMs are unable to prioritize this innovative practice, they may overlook significant
leading indicators without which impact-level evaluation is useless.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 35

Organizational openness to adopting processes that identify leading indicators of
impact-level training effectiveness. An effective leader understands that change is hard,
alternations to routine are inefficient, and people need to be motivated by a purpose or by what
the benefits of change will look like (Schein, 2004). The research also indicates that change is
more likely to succeed when leaders make it a point to communicate and motivate their
personnel (Gilley, Gilley, & McMillian, 2009). The adoption of a different approach to the
implementation of training evaluation and a refocusing away from student-centric metrics and
towards MAJCOM metrics will require a change in the current cultural model of training
evaluation for reaction, learning, and transfer.
Organizational acceptance of the value of leading indicators as a strategic planning
tool. The second cultural model exists both internally and externally to the 82 TRW, in the
traditional identification, use, and value of training evaluations based on student-centric data.
The 82 TRW currently tracks and reports student, course and instructor metrics, and the
MAJCOMs who need trained technicians acknowledge the numbers of student that graduate and
report to USAF flight lines. However, traditional data does not tell the MAJCOMs s anything
useful to MAJCOM leadership. It fails to capture data that correlates to and benefits MAJCOM
successes, trends, or challenges. If a cultural model gap is identified from this study, it will be
necessary to convince the MAJCOMs to provide access to their reporting data and support
impact-level evaluation.
Organizational value for education and training missions as highly as operational
missions. The third cultural model necessary to successful implementation of identifying
leading indicators that correlate to training objectives is that technical training must be
recognized as professionally equal and necessary by the MAJCOMs who are supported by the
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 36

schoolhouses. The competition for resources and recognition between military units is as old as
the military, and operational missions typically support stronger arguments than training
missions, particularly after nearly a decade of budget shortfalls. The justification to support
operational missions first and training missions with anything remaining is deeply embedded in
the USAF. It is necessary that every MAJCOM remembers that every Airman is a product of
formal training and that improvements in training directly benefit the MAJCOMs.
Cultural settings. Cultural models are beliefs expectations rarely thought of once people
belong to a culture, but settings are the concrete artifacts produced to conceptualize and support
the application and continuation of those models (Gallimore and Goldenberg, 2001). As models
must be addressed to affect change, so must cultural settings (Clark & Estes, 2008). When
employees contribute to and are rewarded for contributing to change, the likelihood of successful
change increases (Clark & Estes, 2008). Cultural setting are guideposts that provide consistency
for large or growing organizations and complex processes; a method to ensure what one knows,
all know.
Organizational provision of explicit guidance to identify, correlate and analyze
training objectives and leading indicators for training evaluation to determine the impact
of training to external stakeholders. Between late 2018 and early 2019, AETC rescinded most
of the written directive guidance used for technical training administration, development,
delivery, and evaluation, and replaced the instructions with general memos, good for one-year,
that assign only roles and responsibilities. This decision created an opportunity for technical
training specialists to directly contribute to the creation or revision of living process guides that
replace fossilized guidance in place since the 1970’s and 1980s. The timing is fortuitous, as one
cultural setting influence necessary to the innovation of this study is to provide consistent and
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 37

uniform processes that Airman may follow to identify leading indicators that will correlate to
training objectives. Process guides generalize declarative and procedural knowledge and
motivational influences throughout large organizations and will contribute to the expeditious
success of the recommendations of this study.
Organizational provision of the necessary resources to fully conduct impact-level
training evaluation. The final cultural setting influence foundational to this innovation is the
need for reallocation of resources to support impact-level evaluation. To implement the
recommendations from this innovation, training directors will reallocate manpower and time
from obsolete and less effective training evaluation practices to the identification and
measurement of leading indicators and correlated training objectives. Manpower will be needed
to investigate and coordinate what leading indicators are available from MAJCOM metrics of
interest, and to establish measurement cycles and data collection, analysis and reporting
practices. The reallocation of time as a resource is important, as existing evaluation practices are
time sensitive and considered inspectable items, subject to negative results that affect squadron,
group, and wing status reports, that in turn impact personnel appraisals, promotions, and rewards.
Every aspect of how resources are assigned and used throughout the USAF is closely monitored
and justified to prevent unacceptable fraud, waste, or abuse.
This section addressed the knowledge and motivation of training directors and the
cultural model and settings within the 82 TRW with potentially significant impact on influencing
organizational performance and achieving the organizational goal. The types of knowledge that
training directors must possess to organize and conduct training evaluation is foundational for
implementing impact-level technical training evaluation and were identified as conceptual,
procedural, and metacognitive. The motivation influences considered here included the self-
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 38

efficacy of training directors to adopt training evaluation practices and what expectation of value
that stakeholders could anticipate in return for their work in establishing and conducting training
evaluation. Finally, the cultural model and cultural setting influences needed to successfully
conduct impact-level training evaluation were discussed, based on the general literature of the
influences of change and the specific needs of the 82 TRW.
Table 4
Organizational Influences
Organizational Influences
Cultural Model Influence 1 Organizational openness to adopting processes that
identify leading indicators of impact-level training
effectiveness (Clark & Estes, 2008; Kirkpatrick &
Kirkpatrick, 2016).
Cultural Model Influence 2 Organizational acceptance of the value of leading
indicators as a strategic planning tool (Clark & Estes,
2008; Kirkpatrick & Kirkpatrick, 2016).
Cultural Model Influence 3 Organizational value for education and training
missions as highly as operational missions (AFI, 2010;
Clark & Estes, 2008).
Cultural Setting Influence 1 Organizational provision of explicit guidance to
identify, correlate and analyze training objectives and
leading indicators for training evaluation to determine
the impact of training to external stakeholders (AETC,
2015; Clark & Estes, 2008; USAF, 2010).
Cultural Setting Influence 2 Organizational provision of the necessary resources to
fully conduct impact-level training evaluation (Clark &
Estes, 2008; Harrison, 2016).

Conceptual Framework: The Interaction of Training Directors’ Knowledge, Motivation,
and the Organizational Context
A conceptual framework provides a method to consider in context the assumptions made
about interactions between selected factors chosen for a study (Maxwell, 2013). The framework
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 39

provides a platform on which theory, prior research, experience and thought experiments are
used to build a structure and to provide the means to rotate the resultant structure and examine
intersections from different angles and altitudes (Merriam & Tisdell, 2016). If the structure
proves to have integrity, knowledge about the study may be advanced for others to use in turn
(Merriam & Tisdell, 2016). In addition to providing a working structure with checks and
balances available through the early stages of a study, the conceptual framework can also reveal
the methodology best suited to the research (Maxwell, 2013). Finally, the conceptual framework
may be helpful for future readers unfamiliar with the specific context of the subject under
examination (Maxwell, 2013). The conceptual framework provides an outline within which the
scale and complexity of an unknown organization are aligned, much like placing a ruler in a
photograph next to an unknown object helps a viewer see that the object is relatively large or
small. Seen without a ruler for scale, a viewer is left to guess if the object is one-inch or one-
mile wide.
The primary philosophical approach of this study, that added meaningful color to a
grayscale structure, was the pragmatic worldview. The pragmatic view worked well in
addressing specific problems and potential and functional solutions (Creswell, 2014).
Pragmatism allowed for flexibility in this research as an innovation study, desirable because,
during the evaluation process, new avenues to explore appeared. Pragmatism for this study was
considered the most appropriate worldview because the USAF relies on tradition and specific,
well-documented processes that are slow to impel change until all positive and negative
ramifications have been reviewed (Deis, 2010; USAF, 2017b). There are social and political
influences within the USAF as well, but, for the purposes of this study of training evaluation,
pragmatism was the most applicable to evaluating this problem of practice.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 40

This study recommended the knowledge, motivation, and organizational influences
within the 82 TRW to implement the identification and use of leading indicators correlated to
training objectives and in turn provide metrics to support long-range planning of technical
training conducted by the 82 TRW. However, gaps were found in the intersections of the whole,
at the confluence of knowledge influence with motivation influence with organizational norms or
traditions. More significantly, problem-solving, and potentially useful recommendations were
identified by stepping back and seeing with an enhanced perspective where there were no
intersections (Schwandt & Marquardt, 2000). In using Clark and Estes (2008) knowledge,
motivation, and organizational gap analysis framework with a pragmatic worldview, the
stakeholder goal of 100% of the 82 TRW’s training directors identify leading indicators with
which to measure the organizational benefit of training by December of 2020 was realistic and
achievable. Figure 1 provides a graphic representation of the conceptual framework.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 41

Figure 1. Conceptual framework.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 42

The large blue outer ring represents the organization of the USAF. The smaller green
ring inside of the blue organization ring represents the 82 TRW and includes the training
directors as the stakeholder group of focus. The stakeholders exist within the 82

TRW,
completely immersed in the organization model and setting influences. The knowledge and
experience of every training director as a key stakeholder began with the cultural influences of
the USAF and the technical training environment (AETC, 2015; Deis, 2010) which were
codified in the 1970s and changed little since (Allen, 2006; Deis, 2010).
Culture guides the USAF approach to the support and maintenance of national security
interests by keeping weapons systems in the air, an inherently dangerous activity bounded by
effective risk assessment and management practices (Harrington, et al., 2017). Innovation was
deliberately discouraged in favor of developing a culture that was risk averse and failure
intolerant. Another cultural paradigm within the USAF was that education and training were
viewed as lesser and supporting missions instead of a valued operational activity among the
operational missions of other MAJCOMs. The third cultural influence within the USAF was the
reliance on highly detailed instructions and mandatory checklists that detailed every aspect of
every airman’s daily routine. A resulting artifact from these cultural biases was that executive
leadership, while always a product of the training environment, had no conceptualization or
experience with what was required to design, develop, implement, and evaluate training at any
level. Evidence of these cultural model influences exists in the lack of instructional guidance on
when, how, or why to conduct impact-level evaluation (AETC, 2015) and in the dearth of
specifically identified evaluation specialists with the skills and knowledge to conduct strategic
evaluation.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 43

Identified within the green ring are the significant knowledge and motivation influences
identified from general literature on learning and motivation as well as specific literature on the
knowledge requirements and motivational elements needed to implement impact-level evaluation
(Clark & Estes, 2008; Kirkpatrick & Kirkpatrick, 2016; Phillips & Phillips, 2012; Shobe &
Curtis, 2007). Stakeholders understand what ROI evaluation is, why it is useful and how to
design and implement it using leading indicators (Kirkpatrick & Kirkpatrick, 2016). In addition
to conceptual and procedural knowledge of ROI leading indicators, training directors understand
more precisely how impact-level evaluation could be conducted within the more challenging,
high-risk environment of the USAF. Next, training directors are motivated to discover and apply
the benefits to their teams and squadrons’ decision-making process from the acquisition of sound
data to make strategic, long-term training decisions (Harrington et al., 2017). Finally, training
directors clearly identify resources needed to support the design and implementation of impact-
level evaluation (Kirkpatrick & Kirkpatrick, 2016).
The arrow below the rings in Figure 1 represents the process flow to goal achievement
that results when the intersections of knowledge, motivation with cultural models and settings
are identified and addressed. The stakeholder goal identified in this innovative study is
represented by the green rectangle and is a necessary component of ultimately achieving the
larger organizational global goal that, by December of 2022, the 82 TRW will have an impact-
level training evaluation system in place. The organizational goal is outside of the blue ring of
the organization but connected because goal achievement will benefit the training group and
training wing but will also prove throughout USAF training as a larger organization that impact-
level evaluation is practicable.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 44

The intent of this study was to discover and understand what will enable the 82 TRW to
implement impact-level training evaluation. The study was constructed to identify knowledge,
motivational and organizational attributes, and constraints from the training directors. The
graphic in Figure 1 identified these elements to support a gap analysis of training evaluation.
Qualitative Data Collection and Instrumentation
This was a descriptive and detailed qualitative study that explored the environment,
knowledge, and behaviors of the technical training directors at the squadron and group level
within the 82 TRW. Qualitative data were collected through interviews and personal
observations by the researcher (Maxwell, 2013; Merriam & Tisdell, 2016). There are challenges
when collecting and analyzing qualitative data in establishing and maintaining credibility and
trustworthiness of the data and results (Merriam & Tisdell, 2016). Interviews were the source of
data collected for analysis. Appendix A provides the stakeholder sampling and recruitment
considerations as well as the interview strategy and rationale. Appendix B provides the
interview protocol, and Appendix C addresses the credibility and trustworthiness of this study,
particularly the hazardous use of a sole data collection method. Appendix D discusses the ethical
concerns and requirements related to conducting interviews, analyzing, and securing data and
researching within the researcher’s personal or professional environment. Appendix E discusses
the limitations and delimitations associated with this study and the potential influence each had
on the findings that it generated. Appendix F lists the knowledge, motivation, and organization
(KMO) recommendations for practice, and Appendix G provides details of the integrated
implementation and evaluation plan.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 45

Interviews
Useful qualitative data collection relies on well-designed protocols that provide data to
answer research questions with a depth of detail that describes how research participants
perceive their world and organization (Patton, 2002). Interviews were selected for qualitative
data collection, with 11 key stakeholders as the focus group of this study. Semi-structured
interviews are an effective and efficient way to acquire details of the knowledge and motivation
that the stakeholders of focus possess as well as of their experiences with and perceptions of their
organization (Merriam & Tisdell, 2016). Appendix B provides the interview protocol that
guided each interview, ensured consistency, and provided a guide to adhere to, for what could
otherwise become wide-ranging conversations. Initial interview questions typically collect
demographic data, establish the background of a participant, and permit a rapport to develop
(Merriam & Tisdell, 2016).
Open-ended questions were used to facilitate conversational interviews while allowing
flexibility to adapt to where the interviewees’ responses led. Probes were added for selected
questions to allow for the pursuit of more complex or unexpected responses. Each question was
based on the research questions for this study and provided the data needed to evaluate for
knowledge, motivation, or organizational gaps that, if addressed, would correct, or attenuate the
problem of practice for this study.
Gaining access to where the data resides is the initial challenge of collection (Bogdan &
Biklen, 2007). Interviewing provides access to events or processes and to participants’ thoughts
and feelings otherwise unavailable (Weiss, 1994). Surveys were not the appropriate method for
this study due to the small sample size and the environment in which the collection of data
occurred. The effective method to gain insight on the thoughts and experiences of the training
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 46

directors was by interviews. Interviews allow a researcher to establish and maintain an open line
of communication with interviewees and adapt throughout the interview to what needs to asked
and to understand context (Bogdan & Biklen, 2007; Patton, 2002; Rubin & Rubin, 2012).
Interviews were selected as the means to access what a training director really thought and felt
about the potential for conducting training evaluation required trust and rapport.
The questions developed for the protocol for this study were designed to reflect
recommendations for six types of questions identified by Patton (2002). The questions are about
experience and behavior, opinion, and values, how participants feel about specific things, what
they know of the subject matter, and demographics questions to establish the relative levels of
experience that inform each training directors responses. The protocol was used as a semi-
structured interview guide, as the participants were senior managers and expected to provide rich
detail without the need for additional and more explicit probes. As Merriam and Tisdell (2016)
asserted, the circumstances would dictate if specific questions were needed.
Glesne (2011) recommended that interviews be recorded, with the participants’
knowledge and concurrence, to accurately capture nuanced responses and to maintain ethically
accurate records. Therefore, the interviews for this study were recorded with the participants’
permission. Note-taking during interviews was another practice that supported interviewer
impressions, added details, and maintained the fidelity of the interview context and responses
(Weiss, 1994). With both recorded and written notes based on the semi-structured interview
protocol, the resulting data supported the research questions of this study.
Interview procedures. I planned and conducted in-person interviews with the 11
training directors in their offices or work environments. I first contacted the training directors by
email and explained that I was a doctoral student and explained the topic and intent of my study.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 47

I included that participation in the interviews was voluntary on their part, and that they were
welcome to any results or recommendations generated by the study. I began making
appointments by email to conduct the interviews once I received institutional review board (IRB)
approval. Training directors frequently travel; I anticipated that interview appointments might
not be possible for up to six weeks. I reserved a period of seven weeks for interview
appointments and possible follow-up appointments for member checking. Additionally, Patton
(2002) pointed out that amplifying data may be desired once initial interviews are complete, and
a follow-up appointment with the earliest participants may be desired. A request for the
opportunity to follow up in person or by telephone was included in the initial meeting request.
Informal interviews were the preferred approach as participants needed to be comfortable
exploring ideas for which they may not possess extensive knowledge. The technical training
wing is a professional but informal environment that is open to wide-ranging conversations
related to improving training. The researcher had professionally engaged with many of the
participants previously and felt the interviewees were more forthcoming about what they did and
did not know about impact-level evaluation practices beyond what they experienced throughout
their careers in an informal discussion.
Data Analysis
Data analysis has complex requirements that begin during the initial planning of a study
and include a determination of the best ways to access and organize relevant data and how to
maintain the credibility and validity of both the data and subsequent data analysis (Bogdan &
Biklen, 2007; Merriam & Tisdell, 2016). The collection of qualitative data should take an
iterative approach to produce both a learning process and a continuous improvement and
refinement of the data from the first interview (Bogdan & Biklen, 2007; Merriam & Tisdell,
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 48

2016). Best practices for qualitative data collection from interviews include planning and
maintaining consistency of the study and maintaining the context and accuracy of interview
transcriptions with detailed note-taking before, during and after each interview to document
impressions, nuances, and ancillary for follow-up.
For this study, each interview was prepared for by providing the interviewees with the
purpose of the study and the intended areas of interest for data collection. The interviews were
each recorded, and notes were written by the researcher to supplement the recorded responses.
Post-interview notes captured impressions made by the interviewee responses and highlighted
the potential for a follow-up line of inquiry that might provide richer data useful to the study
analysis. The recordings were transcribed using TEMI, a third-party application and downloaded
in text format. Texts of each interview were then imported into NVIVO, a software program
useful for coding and organizing coded qualitative responses and converting anecdotal inputs
into data for analysis and trend identification.
Within NVIVO, nodes were first selected to flag factual, conceptual, procedural, and
metacognitive knowledge related responses. These nodes were paired with the terms “positive”
and “negative” to create queries that indicated trends in knowledge influences. Positive results
correlated to assets manifested by training directors that the anticipated influence existed and
was satisfactory to support accomplishment of the stakeholder goal. Trends that developed from
influence and negative paired coded nodes were identified as needs, and ultimately generated
recommendations that conclude the study. Trends that appeared from positive coded pairs of
nodes were identified as assets.
Nodes were then included for the motivation theory influences: self-efficacy and
expectancy-value. I scanned for comments related to self-efficacy, value, and motivation and
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 49

coded them with the terms “positive” and “negative” in data queries to isolate the presence or
lack of training director motivation to identify leading indicators. Correlations were collected,
and both positive and negative paired trends were added into the design of a program to leverage
training directors’ assets and include targeted training to close the gaps disclosed. Nodes were
also added for each of the organizational cultural influences identified in the design of this study
that were expected to directly relate to the stakeholders’ ability to identify leading indicators.
Positive and negative modifiers were again included in the queries run on these nodes as well.
The nodes included the terms “cultural model,” “cultural setting” and “resource.”
Finally, the notes taken during and after the interviews were transcribed by the researcher
and imported into NVIVO. The nodes identified for coding included the following terms:
statement, impression, diversion and aside. Those notes that related to a statement indicated
interviewee emphasis on a given response that indicated a passion or strongly held belief that the
researcher found to be particularly convincing over responses that were more indicative of
alignment with what the organizational stand on the subject was. The term “impression” was
intended to yield support for any distinction between a personal and professional belief related to
the value of training evaluation but failed to yield any identifiable trend or query results. The
terms “diversion” and “aside” were used to annotate subtopics not relevant to the study that
interviewee responses indicated could be pursued with additional research.
The identification of reliable trends and accurate analysis of what those trends may
indicate rests on a researcher’s ability to correctly connect what information from the data
collection is relevant (Merriam & Tisdell, 2016). The credibility and reliability a researcher
achieve is dependent on the how clearly the data are collected, developed, and reported as well as
on whether those results are reproducible by another researcher (Maxwell, 2013).
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 50

Findings
The purpose of this study was to determine if the 82 TRW possessed the knowledge,
motivation and organizational openness to transition from a limited training evaluation process to
an innovative evaluation process to measure impact-level outcomes of technical training. The
study was designed to answer the following research questions:
1. What are the training directors’ knowledge and motivation needs related to measuring
impact-levels?
2. What USAF cultural influences affect implementation of a system to measure impact-
levels?
3. What USAF resources are needed to implement an impact-level evaluation system?
4. What recommendations will bridge knowledge, motivation or organizational culture gaps
related to impact-level training evaluation within the 82 TRW?
In response to these questions, analysis provided the following key findings of the gaps
or assets in knowledge and motivation.
• Training directors’ responses indicated a need for additional declarative and procedural
training for themselves. Many of the directors were unfamiliar with the vocabulary,
concepts, and procedures needed to conduct impact-level evaluation;
• Training directors manifested a high level of metacognition. Throughout the interviews
every director demonstrated self-awareness while thinking and describing how they
acquire and apply knowledge;
• Training directors’ responses exhibited high levels of self-efficacy and recognized value
in conducting impact-level evaluation. However, some felt constrained by
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 51

intersectionality between high motivation and uncertainty of support for the
organizational distribution of resources.
There were also five organizational cultural factors explored by this innovation study that
produced additional key findings. The first three influences examined cultural model influences
and found the following and the final two are cultural setting influences.
• The 82 TRW training wing is open to change and innovation.
• There is a cultural belief within the 82 TRW that the MAJCOMs do not understand how
training is designed or delivered.
• There is a cultural belief within the 82 TRW that the status of 82 TRW training among
the MAJCOMs is limited by a devaluation of training compared to the value of
operational missions. The first cultural model influence was identified as an asset in
achieving the stakeholder goal and is utilized in the Program designed to implement this
innovation. The second and third influences were found to be a gap, and are included as
an objective in the Program, to provide useful results meaningful to the MAJCOMs, and
prove direct cause and effect from changes made in training.
• The 82 TRW values written processes to ensure uniform processes.
• The 82 TRW’s ability to obtain and provide additional resources to the squadrons and
training groups is restricted.
This study generated these findings based on 11 interviews with senior civilians assigned
to the 82 TRW and in supervisory positions responsible for the design, delivery, and evaluation
of technical training throughout the wing. The stakeholder group of focus central to achieving
the goal of identifying leading indicators for the 82 TRW included squadron and group training
directors: a small and uniquely qualified population responsible for implementing and
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 52

maintaining improvements and innovations to training. The title of training director was used for
every participant in the study, as actual position titles would too closely identify specific
individuals and create confusion for anyone without direct experience with USAF training
hierarchies. The training directors are federal civil service employees at the GS-13 and -14
levels, possess graduate degrees or equivalent educational credentials and 15 to 40 years of
professional experience in multiple positions within the USAF technical training community.
No other demographic data were reported as this population is small, and individuals could be
easily identified. The eleven participants were each identified by codes as TD1 through TD11.
Qualitative data were collected through face-to-face, semi-structured interviews with the
11 82 TRW training directors. The interviews were audio recorded, transcribed, coded, and
analyzed for common themes and trends related to the research questions that directed this study.
Notes were also taken during each interview to collect observational data that added richness of
detail to the responses. An analysis of data trends and patterns from the interview data and
observations then directed the design of recommendations that conclude this study to improve
training evaluation conducted by the 82 TRW, as the training directors will lead others in the
wing to design and implement comprehensive, impact-level evaluation.
Findings for Declarative, Procedural and Metacognitive Knowledge Influences
Managers and supervisors who lead successful change in an organization are competent
and knowledgeable (Clark & Estes, 2008; Rueda, 2011). This portion of the study sought to
determine if training directors possessed knowledge of training evaluation practices beyond the
AETC mandated evaluation practices for end-of-course critiques, student summative
assessments, and limited field evaluation questionnaires. While effective at measuring student
reaction, learning and transfer, USAF and AETC evaluation at the time of the study did not
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 53

measure impact-level data on training outcomes for strategic planning. The interviews explored
training directors’ level of conceptual knowledge of training evaluation models from formal
training, perceptions of the utility of any current models, and their individual experiences with
how to design and conduct evaluation practices and generate organizational changes to training
using impact-level results. Finally, this section examined training directors’ metacognitive
knowledge to determine if they had an awareness of their skills to lead the organizational change
needed to implement impact-level evaluation.
Training directors lacked declarative knowledge of impact-level training evaluation.
To effect successful change, leaders must be able to communicate the conceptual outcomes of
change; the intent, direction, and goal of the change (Clark & Estes, 2008). An informed and
educated staff working with a knowledgeable leader significantly increases the odds of
overcoming inertia and producing lasting change (Clark & Estes, 2008; Rueda, 2011). Analysis
found eight training directors were unfamiliar with strategically applicable evaluation practices.
While every director demonstrated knowledge and experience with training evaluation conducted
by the USAF and AETC, only three exhibited recall of the taxonomy and use of Kirkpatrick’s
four levels, knowledge of impact-level evaluation components or uses specifically, or of the
Kirkpatrick New World Model (2016) and the potential use of leading indicators.
Each participant provided anecdotal experiences with training evaluation practices and
outcomes and primarily described student reactions in the form of end-of-course surveys and
summative learning assessments. TD7 paused for thought and then related, “We collect a great
deal of data, but we do not really use it for anything. The MAJCOMs don’t want it, and it does
not influence their decisions about the training they want.” TD3 was uncertain of training
evaluation practices beyond those the USAF dictated: “I vaguely recall something about
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 54

Kirkpatrick’s work. Maybe as we talk, it will jog my memory.” TD4 began to speculate what
impact-level evaluation might look like and brought up safety mishap trends and the potential to
correlate safety training that could be revised specifically based on the trend results. He or she
also added, “But we don’t do that.”
Further analysis of the responses indicated that TD10 possessed the vocabulary of
training evaluation practices and expectations, but had no personal experience in conducting it.
He or she was knowledgeable of Kirkpatrick’s four levels and what leading indicators are and
how they are used. Two directors had both deeper knowledge and experience with impact-level
evaluation practices and were implementing it. TD10 stated, “I think there is going to be a lot of
value to strategically useful data.” The remaining eight demonstrated little to no knowledge of
training evaluation outside of USAF practices.
Additional responses to knowledge questions that demonstrated the lack of conceptual
knowledge from the interviews are listed in Table 5. Given a description of the Kirkpatrick New
World Model (2016) and of the use of leading indicators in training evaluation, seven directors
with no prior knowledge of the Kirkpatrick New World Model (2016) expressed interest in how
impact-level evaluation methods could work for the training they are responsible for. This
interest in the utility of impact-level evaluation revealed an intersection between knowledge and
the motivation influences discussed in the following section on motivational influence findings.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 55

Table 5
Training Directors’ Knowledge of Training Evaluation Models and Practices
Interview Question Participants’ Responses
Tell me about your
knowledge and experience
with training evaluation.
“I can't recall any specific models, although I did a lot of evaluation
working with the Inspector General.” TD11

“I know we evaluate students and the courses, and then we survey some
supervisors and new maintainers six months after they graduated by long-
distance field questionnaires, but I don’t know much more about formal
evaluation, in general.” TD6

“I used to go out and conduct interviews face to face with supervisors and
former students, and that gave us really good data about the effectiveness
of our courses, but that costs a lot of money and we don’t do that
anymore.” TD5

Training directors lacked procedural knowledge of impact-level training evaluation.
Krathwohl (2002) identified procedural knowledge as the information that tells a learner how to
do something. This knowledge influence was explored and data indicated that training directors
had little knowledge of how to identify and measure leading indicators. Only the three training
directors who did possess conceptual knowledge of impact-level training evaluation were able to
describe the procedures used to conduct it as well. Two of the three had attended formal training
and understood what a leading indicator was and how it was used to evaluate training. TD9
discussed attending formal New World Model training and recalled that it was immediately
obvious that the New World Model would take training evaluation beyond using student
statistics to something really meaningful. TD1 responded with more pragmatism and identified
the challenges he or she experienced in trying to explain to others what to look for and the intent
behind it.
Of the remaining eight training directors, TD2 represented all the directors without the
advantage of current evaluation training when he or she said “I’ve had no additional evaluation
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 56

training after college,” but then demonstrated an immediate understanding of the utility to and
the resistance from MAJCOMs and the 82 TRW to identify leading indicators and correlate them
to training objectives in current courses:
In my experience, we have never looked at leading indicators, but wouldn’t that be great?
If we are going to, it seems we need a common vocabulary with the MAJCOMs first. We
know they are concerned about the length of time training takes, and they do not know
why we insist on taking that time. But when they have a major mishap, the first thing
they say is that we didn’t provide enough safety training.
Table 6 lists other comments in response to interview questions to assess training
director’s procedural knowledge. It was evident that there was a tremendous amount of
evaluation experience among the directors and that it was deep, but not wide enough to include
impact-level evaluation. It was necessary to provide an explanation of impact-level evaluation
procedures to continue the interviews.
TD2, with no formally acquired conceptual knowledge, began to speculate on the need to
make a connection between a leading indicator and training objectives that would correlate, so
that a change in course content would produce a change to the leading indicator. However, even
TD2 was unsure of how to identify a leading indicator or how to correlate it with a training
objective. TD9 then speculated that he or she could really see how revising an objective or set of
objectives would show a linked result to those things that functional managers track, but added
that it was unclear as to how to identify leading indicators. TD8 confirmed the lack of
procedural knowledge during his or her interview and noted that the command has frequently
noted an inability to connect the dots for the functional managers (FMs),
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 57

Table 6
Training Directors Procedural Knowledge of Training Evaluation
Interview Questions Participants’ Responses
A leading
indicator was
identified because
it was found that
safety mishaps
declined at several
operational bases.
The decreased
number of mishaps
correlated to
revisions made in
training. What
other leading
indicators might
be identified from
metrics that the
MAJCOMs track?
“We always get told they don’t care what the data shows about obsolete
skill or task, they want that task to remain in the course, and we have
to keep it. They don’t listen to us.” TD6

“I have no idea how we would start to do that, but the results would be
really interesting.” TD11

“I can’t think of anything that the FM or the MAJCOM asks about,
once new training is implemented. At [Specialty Training
Requirements Team] (STRT) meetings and [Utilization & Training
Workshops] (U&TWs), the MAJCOMs tell us what tasks they want
training designed for, but there was never any discussion about follow
up for the results, at least, not until we conduct a course review after
two or three years.” TD4

“We are very bureaucratic, and we have such a compliance mentality
that I think it will be hard to get the MAJCOM representatives I have
worked with to share even simple information. We have to get smart
about this kind of evaluation, but so do they.” TD10

How can we
identify and use
leading indicators?
“The MAJCOM cares about how long training takes. And they do not
care what the data we collect now indicates. Only after something like
a safety mishap occurs do they look at training, and then, it is always
been to place blame and not to fix or prevent it from happening
again.” TD2

Intersectionality appeared because of this knowledge-based section of the interview
questions, between the training directors’ knowledge and uncertainty of the utility of impact-
level evaluation, a constraint to both motivational influences. TD5 revealed one intersection
knowledge and resource in response to a question about procedural knowledge with, “If the
instructions or the MAJCOMs don’t ask for that kind of evaluation, I honestly don’t have the
time to think about it.” TD6 pointed out, “they don’t get asked about that by the functional
managers, confirming that there is currently no expectation of impact-level evaluation practices
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 58

or value, and added, “I’ve never heard them talk about the impact of our training on their
metrics.” TD4, with more experience, pointed out that thinking about the utility of impact-level
evaluation was a strategic skill that younger training professionals don’t think strategically yet,
but they need to.
The data also indicated that training directors lack experience with impact-level
evaluation, but that they possessed the capacity to learn, understand and implement it, if trained.
Once provided with a broad description of how to design and use impact-level evaluation, TD11
stated, “If we conducted impact-level evaluation after getting everyone who makes decisions
trained, we would have an unprecedented opportunity to improve training.” Seven training
directors responses demonstrated a growing awareness of the utility of impact-level evaluation as
the researcher conducted successive interviews. TD9 commented on the current lack of value to
evaluation practices and noted the collection of so much data that is never used. He or she
mused that it would be a real advancement to collect and use data to produce changes that made
a difference in training outcomes. TD4 supported this as well with, “I am a believer in data-
driven decisions, but we don’t do that often. I don’t know if we’ve been pulling the wrong data,
but it's been hard to use it. Maybe this is the way we need it.”
Training directors demonstrated the ability to reflect on the design and conduct of
training and implementation for impact-level training evaluation. The final knowledge
influence examined in this section of the study assessed the training directors understanding of
their own knowledge and ability to learn and to lead the acquisition of new knowledge. Baker
(2006) defined this level of self-awareness as metacognition. When asked what they thought
they would need to understand and direct impact-level evaluation processes, all interviewees
demonstrated a rich awareness of metacognitive thinking. TD9 responded that the entire wing
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 59

contained performance-based learners who prefer to learn by hands-on training. He or she added
that to identify an actual leading indicator from the MAJCOM he or she supported and correlate
it to one of his or her own courses would be an optimal use of training. TD3 concurred with
performance-based training by preference, particularly with time so limited. He or she added
that any training that could be immediately put to use was the best training strategy when they
had so little time for anything else. TD2 described the level or degree of metacognition that
prevails in the wing by saying, “We are efficient learners. We learned how to learn quickly
when we all taught and wrote curriculum. We don’t waste time. We don’t need three iterations
or multiple practice sessions.” Finally, TD11 stated, “We are going to need training. Keep it
organized, make it fast and to the point, and we will get it, and use it.” All the directors indicated
they felt that training was needed, but that once trained, impact-level evaluation knowledge and
skills would rapidly propagate in the wing because the wing is a learning organization.
The training directors’ responses revealed a strong metacognitive consciousness on which
to base the design of the program. The trend from these responses indicated the design of the
training in the program must be efficient and immediately functional if the training directors
were to reiterate it within their areas of responsibility, benefit from it, and use it to train their
own personnel in turn. A second theme was also apparent that training delivery should leverage
the training directors’ metacognitive abilities to lead and think and adapt learning and training
with intersectionality with the limited resource of time discussed further in the findings.
Training Directors’ Motivation Influences
Change initiatives frequently fail because of a lack of motivation (Clark & Estes, 2008;
Rueda, 2011). The interview items probed for training directors’ motivation influences for
possible gaps to address in the program. The motivation influences assumed for this study were
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 60

self-efficacy and expectancy-value theories. For self-efficacy, this study asked training directors
to discuss their feelings about their capacity and confidence to identify leading indicators and
conduct impact-level evaluations. Self-efficacy can be a powerful motivator, but motivation
from the characteristics related to expectancy-value theory was also explored. Therefore, the
interview incorporated questions related to the confidence and value each had in learning about
and leading how to identify leading indicators and produce impact-level results.
Training directors’ belief in their ability to lead the identification and correlation of
leading indicators was influenced by a lack of knowledge and intersectionality with
organizational culture. Self-efficacy is one’s personal belief in their own capability to function
or act (Bandura, 1994). Interview results provided evidence that, while all participants were
confident in their ability to successfully effect change in their units and in the 82 TRW, they also
felt they lacked either the knowledge or an expectation of organizational support for impact-level
evaluation. Eight directors were doubtful that resources would be provided even before the
protocol reached that set of questions. TD3 identified needed evaluation training as a resource
and somewhat hesitantly responded, “I could do this, but I think my present ability is lacking; I
need very specific training.” TD8 answered similarly: “Of course, we are going to do what the
wing directs us to, but in this case and even though it sounds effective, I have to get a lot smarter
about what it will take to do.” TD6, with little recall of training evaluation fundamentals, also
identified training as a needed resource and that in order to prepare his or her staff, he or she
would need education and training and to know better what to expect and do.
Five directors indicated that, based on their experiences, they felt capable of leading
impact-level evaluation if other organizational expectations were reduced or eliminated, or
conversely, if additional resources were provided in support. Their reluctance to state they felt
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 61

fully confident was based on competing demands on their time and staff, rather than on a
perception that they lacked the personal ability to conduct impact-level evaluation. TD5 stated
unequivocally, “They can ask me to do it, but unless they also make it a high priority it will stay
at the bottom of my to-do pile, behind everything else we are tasked with.” In another interview,
TD7 was emphatic in his or her view that it was frustrating because for all the data regularly
collected, little ever changed. TD7 added a concern among directors that this innovation could
also become another wasteful initiative. This trend in responses also indicated intersectionality
between self-efficacy as a motivator and the upcoming analysis of organizational resources of
time and manpower as an influence. Further examples of the qualified responses to interview
questions on self-efficacy are included in Table 7.
Table 7
Training Directors’ Responses Related to Self-Efficacy
Interview Question Participants’ Responses
How do you feel about your
ability to implement
measurement of the impact of
training at an institutional
level?
“I could do this, but I think my present ability is lacking; I need very
specific training.” TD6

“We absolutely have the capacity, I have it for my unit, but there is
already so much bureaucracy that gets in the way.” TD4

“I think any of our training directors could do this successfully, but our
operational tempo is still so high and that takes priority.” TD5

“I think I have the skill, the ability to do this and I see the value, well
there is a lot of potential I think I see. But a lot of people in the field,
they see things one of two ways: either they don’t care and don’t see
the point of any training evaluation or they don’t want to get anyone in
trouble so you don’t get honest information from them. It’s difficult to
get honest responses.” TD1

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 62

Table 7, continued
Interview Question Participants’ Responses
How do you feel about your
ability to implement
measurement of the impact of
training at an institutional
level?
“I can see the use, the value of doing this, and I have great folks that
work for me that would get excited about this, but I don’t know exactly
what would we would need until I get some training.” TD3

Training directors were aware that their behavior and confidence in a task or action is
influential to their subordinates; anything they take seriously will be taken seriously by others.
Conversely, anything training directors do not wholly believe in will receive minimal attention
and effort. The trend from the interview responses indicated that training directors were
confident of their professional abilities but not fully convinced that organizational support exists,
in general. The program designed from the results of this intersectionality of self-efficacy and
organizational support addresses the need to identify and acquire adequate resources to
implement impact-level evaluation process.
Training directors recognized value in implementing impact-level training
evaluation. Employees are motivated by four considerations: what professional characteristics
they value, satisfaction experienced from a job well-done, usefulness of a task and the cost in
effort or time (Eccles, 2006; Mayer, 2011). In addition to interview questions, these
characteristics were assessed using notes from observations made during the interviews, as the
characteristics of this theory are subjective. Table 9 provides some of the responses training
directors provided for analysis of this influence. The resulting findings from this section of the
interviews consistently indicate that training directors are professionally knowledgeable and take
pride in acting in the best interests of their customers: the USAF, AETC and the wing. Every
training director also stated at some point during their interview that the utility of impact-level
evaluation ranged from having potential at the least to assert that it was, in the words of TD2
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 63

“going to produce some amazing results” for the wing and the MAJCOMs. Finally, most
managers within an organization will consider the utility and cost of any action or plan
concurrently. Nine training directors shared this characteristic and, regardless of where they
estimated the costs to be, indicated that the potential results justified those costs. The analysis of
this section of the interview indicated a high level of motivation from the training directors’
perceptions of themselves, from the satisfaction they feel from effecting positive change, and
from the perceived utility of this innovation, even with a cost increase in terms of time and
manpower.
As with most organizations, the higher a person moves within a military hierarchy, the
more often they are expected to demonstrate the ability to recognize and implement practices that
benefit the organization (Deis, 2010). An awareness of and actions taken to contribute to that
greater good are valued throughout the military hierarchy (Deis, 2010). Within the 82 TRW, this
professional attribute was observed throughout all 11 interviews. Each training director spoke
with an awareness of the potential value that leading indicators and impact-level evaluation have
for their squadrons, training groups,and the wing. As TD3 commented, “I feel like I would use
it. I’d be interested to see what techniques are used, because I can see the importance of it.”
TD9 also agreed with the potential for value and said it sounded like a great way to see where the
rubber meets the road, which was not something currently accomplished by the wing. He or she
mentally reviewed the potential results of impact-level evaluation and became animated as the
benefits to training became more apparent through the course of the interview.
The second component of expectancy-value theory is a reward for an accomplishment.
There are few external rewards available for senior civil servants: salaries are capped, merit and
time-off awards are limited and the possibility of promotion is unlikely unless a training director
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 64

is prepared to move to another location. Intrinsic rewards, however, are highly prized
throughout the organization. Many employees prepare and apply for, and stay in their career
field because they derive intrinsic satisfaction from influencing the training and preparation of
the next generation of aircraft maintainers. TD4 commented that the one regret of his or her
career to date was the inability to find a better way to assess the success or failures of training.
TD9 noted the frustration he or she experienced from the incomplete current practices and said
that all the rest of evaluation is lip service when an organization fails to evaluate at all four levels
and later added “I would feel a lot better making recommendations on data I trust, that was
designed from the start to be collected for this use.”
The third component in expectancy-value theory is the perceived utility of a task (Eccles,
2006). Table 8 lists additional responses to questions about the potential utility and the fourth
component of cost to implement impact-level evaluation from the participants. Responses on the
two components of utility and cost were consistently intertwined and are presented together to
maintain context of the data. Each of the training directors (100%) felt that there was potential
for utility in conducting impact-level evaluation of the training for which they are responsible.
Additionally, even those training directors with little knowledge or experience with training
evaluation quickly noted the value to the wing and the MAJCOMs. TD10 asserted, “I am certain
that the MAJCOMs and functional managers will be surprised and delighted when we can
provide evidence that this training improved that MAJCOM tracked metric. They might not
even mind adding resources for it. Much.” TD7 supported this clearly with, “The MAJCOMs
need to see the effects when we improve training and give them the results that they want.” In
response to a follow-on question, TD2 explored the benefits from individual course
improvements all the way up to MAJCOM utility and listed them:
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 65

We could individualize instruction and teach only what a specific student needed. We
would have better data to review and refine the Career Field Education and Training Plan
(CFETP) for each specialty. Then the MAJCOMs would get a much better idea of what
we produce for them, because right now, none of them have the background to
understand what it takes to deliver training. They believe that they can do it just as well
as we can, but faster and with fewer resources. It would be worth a great deal.
Table 8

Training Directors’ Motivation Based on Expectancy-Value Theory Components of Utility and
Cost

Interview Question Participants’ Responses
How would you
describe the value,
utility, and cost of
measuring impact-
level evaluation?
“I think that if we evaluated based on what the MAJCOMs value,
they would trust AETC and what we deliver for them. That would be
worth a lot to us.” TD2

“I think the time is right for shifting gears because we feel like this is
the wild west right now. With policies rescinded and innovations
encouraged, it is a great time to put our fingers on data that will be
used instead of data collecting just to collect data. The MAJCOMs
need to see the effects when we improve training and give them the
results that they want.” TD7

“The General said that the information was nice, but how did we
really know we were cranking out quality graduates and we were
dumbfounded at the realization that we don’t have an instrument in
place to gauge that kind of success. We do a great job of
determining results of Level one, we need to do the same for Level
four. It would mean so much more at and above the wing level.”
TD4

“If I evaluate a graduate or question their supervisor, I am not going
to get level 4 data. I need someone going to weekly MAJCOM sortie
meetings to ask if training ever comes up. And if so, is training
contributing to the success or failure of whatever your mission is?”
TD1

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 66

Table 8, continued
Interview Question Participants’ Responses
How would you
describe the value,
utility, and cost of
measuring impact-
level evaluation?
“Of course, I would do it if I was directed to, but unless I was
told to make it a top priority, I probably wouldn’t get to it just
because of everything we have on our plate right now. We
function at about 80% manning, and I would really have to see
a huge result.” TD5

Cost is the fourth and final element of expectancy-value theory. For training directors,
cost generally equates to money, manpower and time. Nine training directors (82%) felt that
additional time and manpower would be needed to conduct impact-level evaluation, but only one
director (9%) felt that the cost would be too high to regularly adopt impact-level evaluation.
TD5 commented that, “In theory, it would be great, but I won’t get extra manpower, and I won’t
exchange this task for another, so I could not afford to adopt impact-level and build it from the
bottom up.” Conversely, TD2 paused to think about the implementation of impact-level
evaluation and disagreed regarding need for an increase in resources or cost: “Maybe I am
missing something, but I think a lot of leading indicators are out there and a matter of record. I
don’t think it would take that much to find the data.” The training directors demonstrated that
they are sufficiently motivated by expectancy-value, and no additional actions are recommended
for the Program to address motivational influences if resources are available.
Organizational Culture and Context Influences
Cultural models and settings produce the underlying beliefs of an organization and drive
decision-making within that organization (Gallimore & Goldenberg, 2001). Further, Schein
(2004) added that cultural models contribute to the degree of successful change attempted by an
organization. This study repeatedly found evidence in interview data that the cultural model of
embedded beliefs of the Air Force and the 82 TRW certainly intersect with knowledge and
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 67

motivational influences. The training directors’ responses reflected that they felt the 82 TRW
and the MAJCOMs they support can significantly benefit from impact-level evaluation, if the
influences of the cultural model were acknowledged. Three cultural model influences were
analyzed in this study: organizational capacity for change, support from leadership at the
MAJCOM level, and cultivation of culture of respect in MAJCOM and AETC relationships.
Two cultural settings were also assumed to be necessary to implement impact-level evaluation:
publication of explicit directive guidance in the form of an AETC instruction, and organizational
resourcing for manpower and time to support the tasking.
The 82 TRW demonstrated a capacity for cultural change. Interview responses from
the training directors varied broadly to questions about how they judge the wing’s capacity for
change, but in summary, the training directors felt that change was not only possible but highly
encouraged. Organizational capacity for change prevents stagnation and improves the
organization’s ability to adapt and survive (Senge, 1990). It was an interesting time to examine
cultural change in Air Force training, as AETC leadership paved the way to change throughout
the entire command and at the same time, blocked attempts to retreat that unraveled change in
the past (Harrington et al., 2017). In early 2018, AETC (2017) cancelled the directive guidance
that mandated detailed processes and invited every employee to make and forward improvements
for training administration, design, implementation, and evaluation. Five respondents referenced
this fundamental shift in the culture throughout the interviews. It is worth noting, however, that
from the directors who commented on change, positive comments were followed by a caveat.
TD2 described the environment: “It’s the wild west here, now and there are no laws, and the
policies were rescinded, and we are being encouraged to innovate. Everything is fair game.”
TD2 then added, “Our biggest problem is our middle management who have been around a long
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 68

time and have a lot of experience but cannot change.” In response to the question of whether we
can change, TD7 said, “I think we absolutely have the capacity for change.” Later TD7 added
that change for AETC was usually temporary and that as soon as the leadership changed, the
policies reverted to what they had been. Yet another director, TD9, commented positively on the
change, but with caution, added:
I think this is an amazing time for us, with all the changes at AETC. They made room for
fresh thinking and new ideas. What they did not do was shake up middle management
enough and that’s where a lot of resistance to change occurs for us. If they disagree, they
drag their feet until the pendulum reverses.
Additionally, the training directors identified where change was needed. TD3 reflected
that MAJCOMs understanding what a training organization does as one area ripe for change,
“No disrespect to any leadership, but to understand some things without having experience in
that position is something that needs to change.” TD4 pointed out that change requires the will
to do more than collect data and wondered about the frustrations of change, “We will have 1000
students complain that a classroom is too hot, but no one acts on that.” At the same point in
another interview, TD7 noted, “Humans are basically lazy, and change is hard. Even the things
we could have changed last year are still wrong, still waiting.” These comments and others listed
in Table 9 served to highlight the point where motivation and organizational change intersect and
where change efforts must focus. Training directors appeared to believed that leadership and
followers were aligned with change and innovations, change was already occurring in systems
processes and approaches, there was an acceptance of innovations and accountability, but they
were hesitant to believe there was permanence.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 69

Table 9
Training Directors’ Responses to Changing Evaluation Practices
Interview Question Participants’ Responses
How do you feel
about change here in
the wing and up
through AETC?

“In the past, leaders would come to the schoolhouse once and put eyes on the
classroom environments they have not seen since they attended training here.
The impressions they leave with are the only input they’ve had, and we need
to change that.” TD5

“No disrespect to any leadership, but to understand some things without
having experience is something that needs to change.” TD3

“It’s time to shift gears. It’s the wild west and there are laws, and the policies
were rescinded, and we are being encouraged to innovate. Everything is fair
game.” TD2
“Leadership is supporting more professional development and wants us to
focus on learning innovations, and they are talking about manning and
creating research positions that will impact how we create courses and apply
strategies. This was unheard of even five years ago” TD6

“Locally, change is hard, but I think we will get there. I tell my folks they are
only limited by what you can imagine.” TD1
How do you feel
about change here in
the wing and up
through AETC?
“I wonder about change. We will have 1000 students complain that a
classroom is too hot, but no one acts on that. Who is doing anything with
the data we already collect?” TD4

“If the change you are talking about is after the fact, then what happens is
we will change when we are told to, but once that leader leaves, we
backslide into our old habits.” TD7

“If the change you are talking about is after the fact, then what happens is
we will change when we are told to, but once that leader leaves, we
backslide into our old habits.” TD7

“I think we absolutely have the capacity for change.” TD7

“The pilot mentality that shapes so much about the Air Force also influences
our perceptions about change. Right now, because of our old rules, that
perception is that AETC and we are missing the boat in writing and
delivering training.” TD10

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 70

Table 9, continued
Interview
Questions
Participants’ Responses
How do you feel
about change here in
the wing and up
through AETC?
“I think this is an amazing time for us, with all the changes at AETC. They
made room for fresh thinking and new ideas. What they didn’t do was shake
up middle management and that’s where a lot of resistance to change occurs
for us.” TD9

There is a cultural belief within the 82 TRW that the MAJCOMs do not understand
how training is designed or delivered. Air Force policy states, “AETC establishes academic
and procedural guidance for executing training (SECAF, 2015). There are multiple groups of
stakeholders at every level of training, from the Airman Basic just learning fundamental skills to
the senior leaders at the Pentagon. The training directors felt that there is a lack of
communication and understanding between their policy-driven responsibilities in the wing and
the MAJCOMs they ultimately serve. When TD1 said, “These are the people we serve” as in
part, a protest of MAJCOM resistance to training professionals. The training directors
unanimously responded to this section of the interviews in the negative, as seen in Table 11.
However, MAJCOM perceptions were attributed to a broad spectrum of causes and the reasons
varied by director. TD3 explained that if the 82 TRW could show the FMs the value of the
training provided, the FMs would have more trust in the training delivered at Sheppard AFB.
The lack of trust was echoed by other directors’ responses. TD5 expanded on that in his or her
interview and said, “I think that trust and perceptions are two-way. We are not 100% sure that
we meet their training needs and they just assume that we always miss the mark and waste time.”
Most training directors stated at some point in during each interview that they have never been
asked to provide proof of the effectiveness of the training they delivered. TD7 captured this
when he or she stated, “I have never been asked about other evaluation results besides student
comments, and those are requested and then dismissed when they see there is no use for those, at
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 71

their level.” TD5 confirmed this belief again with “I have never been asked for proof of results.
Specialty Training Requirement Teams (STRTs) generally tell us what to train, period.” In
analyzing the responses in Table 10, it became apparent that one of the reasons for this
organizational gap was cultural; a perception of subdued hostility from MAJCOMs was an
unquestioned expectation. Every training director noted it without direct awareness in many
cases and proceeded to discuss it as if it were immutable, the very definition of cultural model.
Table 10
Training Directors’ Feelings About MAJCOMs Understanding of Training Processes
Interview Question Participants’ Responses
Describe what you
feel are the
perceptions the
MAJCOMs have for
training and AETC.
Do you think they
trust us?
“I think that if we show them the value of what we do, they would
trust us more.” TD3

“I think that trust and perceptions are two-way. We are not 100%
sure that we met their training needs and they just assume that we
always miss the mark and waste time.” TD5

“I always wondered about that. We are not treated like a MAJCOM;
we don’t act like one. You know what else we need with this
innovation? We need some PR, a newsletter. A way to tell them that
we are here for you.” TD1

“I think if their perception of us was better, they would be more
inclined to give us what we need to train.” TD4

“I think we are missing the boat. We need to provide them strategic
level data to convince them of the value of formalized training that
AETC presents. They need experts instead of trying to do train
themselves.” TD7

“I think we have perception problems in the wing, not just with the
MAJCOMs. We get together in a room and everyone is so busy
fighting for their own rice bowl, they can’t see a bigger picture.” TD2

“I think that of all the MAJCOMs, they see us at the bottom.”
TD8
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 72

The belief that MAJCOMs lack respect for training functions and personnel is not unique
to the military. Many instructional professionals report feeling devalued when compared to
those units that generate income or produce quantifiable results (ATD, 2015). The most
effective method of counteracting the lack of parity is through communication. A goal
developed in the program to address perception issues is to require training personnel regularly
speak with MAJCOM reps to identify, measure and report on leading indicators. Regular
communications that produce useful results that in turn are reflected in MAJCOM metrics of
interest may also provide non-training leadership with the reality of the work that goes on behind
the curtain of training design and delivery.
There is a cultural belief within the 82 TRW that the status of 82 TRW training
among the MAJCOMs is limited by a devaluation of training compared to the value of
operational missions. The key identifier of a cultural model influence is that people are not
even aware of beliefs they hold in common with others in their group (Schein, 2004). Of the
three cultural model influences included for examination in this study, this interview topic
generated the highest number of responses from the participants. Seven training directors
believed that they work against an invisible but certain bias against training and two appeared to
share that bias and assumed that shortfalls in the training organization were immutable because it
is just training is and always been.
TD11 considered the question of the MAJCOM division of responsibilities and said, “I
always wondered why AETC is a MAJCOM, it’s not at all organized like the others.” AETC
exists to provide centralized training services to other MAJCOMs in the USAF. Seven directors
(64%) felt that there is not only a lack of support for training, but there was also a perceptual
difference throughout the USAF for what AETC and the schoolhouses did. TD2 discussed the
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 73

continuing difficulty in filling instructional staff positions with active military members, “They
have physically demanding jobs and they work awful hours under a lot of stress when they are
on the flight lines.” TD2 brought up the difficulty of recruiting volunteer instructional staff from
operational commands and continued, “They come here and finally spend time with family,
know what their schedule is, take college classes. Why don’t they want to spend three or four
years here? It’s weird. They think it hurts their career.”
TD8 noted that there is a perception in the field that training lacked authenticity and was
immediately undermined once graduates reported to their jobs. He or she quoted a familiar
refrain, “Forget what they taught you at school. This is how we do it here!” This resulted in the
appearance for the first time of a bitterness evident from the responses in Table 11. In
observation notes from interviews, there was a subdued tone to the responses to this section of
questions: a lack of animation from the directors. Additionally, the researcher noted that for
three participants, body language changed as well in posture and voice tone that indicated the
respondent felt uncomfortable with the questions. TD4 changed the topic and provided an
example of something unrelated to MAJCOM perceptions about training. TD9 lost his or her
train of thought and was unable to recall what he or she was going to say. Whether a lack of
understanding was due to competition between MAJCOMs for funding and other resources, or
for attention and status or is due to an even more deeply embedded perception of teachers and
trainers versus those who go out in the world and do things, this cultural model influence exists
and was felt in the 82 TRW.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 74

Table 11
Training Directors’ Feelings About MAJCOMs Perception of 82 TRW Training
Interview Question Participants’ Responses
What level of respect
do the MAJCOMs have
for the training you
produce?
“Sequestration hurt us. It hit so hard and fast, so we cut where it
was easiest, not where it was best. We made poor decisions, we
ignored the data, and currently their satisfaction levels with us are
way down.” TD4

“We hear them at conferences, they say we are not adaptable and
we are too bureaucratic. And they are right.” TD7
“With the change in policies and processes, I have the chance to
look through the eyes of the folks in the MAJCOMs and see how they
do what they do. I think the authenticity will then be in our training,
and the they will see it, and trust us more. Right now, our graduates
go out and some Airman on the flight line says to them “Forget
what they taught you at Sheppard. This is how it’s really done.”
TD7

“I think we don’t entirely trust ourselves. We reluctantly accept the
enterprise changes but when the next turn-over comes we can’t
finish what we started, we are off on a whole new thing.” TD9

“I’m not sure. They ask us how hard it would be to just send new
Airmen directly to them instead of getting fundamentals here. They
say that what we do is not that hard.” TD2

“They do not know what they don’t know. Most of them have never
been in AETC, or in training development on the scale that we
conduct it. All’s they know is that support work can’t compare with
operational missions.” TD8

“With few exceptions, those who make decisions know a lot about
their own organization and none about ours. They make a lot of
assumptions, but there isn’t a foundation of trust. Nearly all of us
come from other MAJCOMs, and our active military will return to
the MAJCOMs at the end of their tour, but they still don’t
understand. It’s almost like they can’t or do not want to take what
they learned with them across the border.” TD6

The 82 TRW did not provide guidance for uniform processes to conduct impact-
level evaluation. At the start of this study, there were several active AETC instructions that
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 75

explicitly described how technical training was to be planned, resourced, conducted, and
evaluated (AETC, 2014, 2015, 2017). However, in early 2018, AETC leadership rescinded those
detailed instructions and replaced them with guidance memoranda that detailed roles and
responsibilities, but not processes. Instead, processes were provided for the Department of
Defense (DoD) via a collection of online social business tools under milSuite. Anyone within
the military domain (.mil) can retrieve process guides and recommend updates and
improvements using milSuite. The processes found in milSuite for course design and
development also do not address establishment of impact-level evaluation.
Analysis of this influence revealed another intersection between organizational resources
and the first cultural model or belief of these findings: 82 TRW willingness to adapt to change in
training evaluation practices without directive guidance. In the past, directive guidance clearly
identified what procedures were mandatory. Now, procedural steps found in milWiki are good
practices but are not mandatory. As seen in Table 12 with other participants responses on the
question of guidance as well, the implications of these changes and this intersection were
identified by TD7: “We have always been such a compliance culture. It’s almost like we are
working without a safety net, now. We have to rely on our own judgment, and hope our
judgment is good.” TD7 continued, “It’s ironic, of course. For years we complained that our
innovation was throttled by regulations, and now the regulations are gone we act like we have no
idea what to do.”
Regardless of the change in how procedures are published, 64% of the training directors
identified the need for uniform processes to identify, correlate and track leading indicators. “We
have seen this before, when everyone had different ideas of how to do something great” laughed
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 76

TD11. He or she continued, “If we were told to make pudding, we won’t even stop and ask what
flavor! We just go and do it the way we think best.”
Table 12
Training Directors’ Feelings About Procedural Guidance as a Cultural Setting
Interview Question Participants’ Response
What is needed to
establish uniform
processes throughout the
82 TRW?
“It’s like the lawless old west suddenly. No instructions, just a lot
of good intentions and the processes captured in milWiki, but with
no directive if someone decides to go in another direction. So, we
wanted freedom to operate, but now we fear mayhem.” TD2

“When we gave up on written instructions, we also kind of
abandoned standardization. That means if a course gets taught in
two places, Class #1 will do really well because location #1 does a
good and conscientious job. Location #2 just wants to get it done,
so it’s the same curriculum, but it sure isn’t the same course. It’s
dangerous not to have guidance and procedures.” TD9

“We used to have instructions that included words like you must
and the training director shall, and for optional things we used the
words may, as in the training director may do this or that. It was
great to know exactly what to do for complex tasks. But for simple
things, we simply wanted to be trusted to do the right thing, right.
Now the instructions are gone and we have to put our money where
our mouth is.” TD8

“It’s ironic, of course. For years we complained that innovation
was throttled by regulations, and now they are gone we act like we
have no idea what to do.” TD7

Training directors do not expect the organization to be able to dedicate the
necessary resources. Change typically generates a demand for resources (Clark & Estes, 2008,
Harris, 2016; Lewis, 2011). Time, manpower and training are the three significant resources that
training directors felt they would need to design and conduct impact-level evaluation. Seven
directors responded that they would need time to train their staff to identify leading indicators,
but they were unable to identify any other resource needs without more knowledge of the
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 77

requirements that the innovation would levy. TD3 thought that adequate resources existed
within his or her area of responsibility, if he or she understood the process of impact-level
evaluation correctly. Additional quotes from the participants about the need for and distribution
of resources are included in Table 13. The trend identified from this section of the interviews
indicated that resources have been limited for so long it is an embedded belief or cultural model
influence that resources are unlikely to be provided, or will not be enough. TD2 stated the issue
simply, when he or she rhetorically asked, “Has there ever been change that didn’t cost us??”
TD10 put it succinctly as well, “It’s an endless loop. With more resources we can provide better
training or training evaluation but until we show improvements we won’t get the resources, and
without the resources…” TD7 was more specific about the allocation of resources, with “This is
an innovation, and with innovation you have to talk about technology and security, and we
haven’t figured how to do all that yet.” TD7 continued, “I think in terms of resources, we are
going to have to higher, to the MAJCOMs at least, to provide them. It’s an enterprise solution
we need.” This intersection of cultural model and setting influence gap occurred with most of
the training directors spontaneously, without specific questions or follow-on probes, as training
resources fall directly under their responsibilities.
This was the single area of the interview responses where one director’s response was the
opposite of all others, when TD3 felt that there were enough resources available to identify
leading indicators and correlate training objectives to support impact-level evaluation. The
possible reasons for such a contrary response may be due to the directors’ level of experience
with training evaluation, the status of the personnel manning for his or her training unit, and or
the director’s perception of how to delegate. A last explanation for an outlier response could be
in the description of impact-level evaluation processes provided by the researcher.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 78

Table 13
Training Directors Feelings About Resources as a Cultural Setting
Interview Question Participants’ Response
What resources to
you see will be
needed to identify
leading indicators
and conduct impact-
level evaluation?
“When I started, we had so many more people here, and then those
billets were cut a little here and a little there and at the end of the day,
I am still working on finishing the things that must be done today. I
will believe we will get resources when they show up on my doorstep.”
TD5

“This is an innovation, and with innovation you have to talk about
technology and security, and we haven’t figured out how to do all that
yet. I think in terms of resources, we are going to have to look higher,
to the MAJCOMs at least, to provide them.” TD7

“It is an endless loop. With more resources, we can provide better
training or training evaluation but until we can show improvements we
won’t get the resources, and without the resources…” TD10

“I would add training to the needed resources. I keep coming back to
that in nearly every answer I give you, but I think that this is an idea, a
process with real potential but I need to understand it better.” TD3

“I would say that this sounds like it is not as resource intensive as I
expected. We have evaluation staff for the manpower, and what they
need they can get via phone calls and conferences they already
participate in. The time they have is dictated by the evaluation and site
visit demands we put on them and we can provide that if the end
justifies it. I think we have the means.” TD11

“Has there ever been change that didn’t cost something? We will have
to give something up, or give someone up. The wing, the training
groups and even the squadron commanders will have to decide what
can be sacrificed if this is going to be a required tasker for all of us.”
TD2

Another training director identified time as the resource needed most, “As an
organization we are tired. We have been understaffed for a long time and there isn’t enough time
in the day to add something voluntarily.” A third director identified training as the specific
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 79

resource need for him or herself as well as for staff members. The director who provided a
contrary response asserted, “We have evaluation staff for the manpower, and what information
they need they can get via phone calls and conferences they already participate in.” This director
felt that as he or she could determine, there might not be a huge resource demand and existing
resources would be enough.
Summary of the Findings
Analysis of interview responses indicated shortfalls with training directors’ knowledge,
motivation, and organizational environment. Analysis of interview data indicated that training
directors require training to broaden their knowledge of training evaluation beyond the training
evaluation practiced by the USAF and to include the Kirkpatrick New World Model (2016) in
particular. There was a dearth of both conceptual and procedural knowledge that, if not
addressed would prevent the training directors from designing and accomplishing the stakeholder
goal to identify leading indicators to measure and analyze organizational the benefits of training.
Analysis of the motivational influences of the theories of self-efficacy and expectancy-
value was conducted in this study as well. The results revealed that training directors exhibited a
high degree of self-efficacy and the traits of expectancy-value theory, but concerns about support
and organizational resources diminished their confidence in implementing impact-level training
evaluation. Perhaps training directors were responding pragmatically by indicating caution in
assuming additional work without acquiring resources. Experience in identifying and measuring
leading indicators may reduce the negative perceptions that training directors have that
undermines their confidence. The overall results show that the traits associated with expectancy-
value theory are present and that training directors possess the motivation to design and lead the
innovation recommended by this study.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 80

Of the organizational culture influences of interest in the study, the results indicated that
the 82 TRW is positioned to adopt changes in training evaluation practices and processes. One
concern, however, is that training directors do not have trust or communication processes with
MAJCOMs needed to support the identification and use of leading indicators, and the
MAJCOMs do not value training. The MAJCOMs participation in the identification and
measurement of leading indicators are critical to the stakeholder’s goal.
Recommendations for Practice to Address KMO Influences
The findings from interviews of the training directors indicated a need for impact-level
evaluation knowledge and skill training as well as for updated training evaluation paradigms and
resources common to the culture of the 82 TRW. Appendix F describes the recommendations in
detail of training seminars designed to increase the knowledge and motivation influences that fall
short of optimal for successful implementation of impact-level evaluation. Success breeds
achievement and success from the program will also support success in altering organizational
expectations and the wing’s capacity to evaluate training more effectively. Program training
does not end with the end of the seminar, however. Instead, it establishes how squadrons and
groups can use impact-level evaluation practices to spread the broader use of impact-level
evaluation practices. Appendix G presents an integrated implementation and evaluation plan
based on the Kirkpatrick New World Model approach.
The New World Model (Kirkpatrick & Kirkpatrick, 2016) directs the design of training,
to begin with the highest or fourth level of evaluation, with the end goal of training maintained as
the organizational raison d’etre for the design, development, and implementation of training.
Level 4 asks the extent to which the organization benefitted from training at every point along
the analysis, design, development, implementation, and evaluation cycle of training development
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 81

and expects Levels 3, 2 and 1 evaluation to link to that Level 4 question, as well. The section
also provides specific evaluation tools for the four levels of evaluation and a data analysis and
reporting dashboard template for use at wing staff meetings that shows consolidated progress of
each level of evaluation implementation.
Finally, this section addresses the cultural models and settings that support
implementation of this program as well as the needed resources to accomplish impact-level
evaluation. The 82 TRW collects data on students, courses, instructors, effectiveness, and
efficiency of training on a daily, weekly, monthly, and annual basis, but little of it is used to
improve training. The program presents a different type of evaluation data and different methods
for its collection and evaluation. The wing is also culturally dependent on written policy and
guidance to ensure consistency and uniform compliance in collection and reporting requirements,
and the program provides the outlines for the creation of that guidance. Communication of the
purpose and value of implementing impact-level evaluation is a critical component to
successfully achieve the stakeholder goal of identifying leading indicators and, ultimately, the
organizational goal of impact-level evaluation throughout the wing. The program is designed to
communicate the necessary steps in a logical format and allow for knowledge and skill
acquisition, modeling, guided practice and finally independent application within squadrons and
groups.
Conclusion
Organizations, including the military and the 82 TRW specifically continue to struggle
with training evaluation that includes evidence of the impact of training to the organization’s
bottom line (Bell & Reigeleuth, 2014; Harrington et al., 2017; Kennedy, Chyung, Winiecki, &
Brinkerhoff, 2013; Pulichino, 2007; Shobe & Curtis, 2007). The literature indicates that this is a
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 82

wide-spread area of concern to any organization that invests in training but is challenged to
reconcile the cost of training and bottom line successes or failures (Aguinis & Kraiger, 2009;
Baldwin & Ford, 1988; Bates, Holton, & Hatala, 2012; Kirkpatrick, 1994; Nafukho, Graham, &
Kacirek, 2009; Phillips, 2012; Stufflebeam, 2003). The Clark and Estes (2008) gap analysis
model was used to examine what knowledge, motivational and organizational influences that
were or were not present to accomplish impact-level evaluation as an innovation goal.
One-on-one interviews were conducted with members of the stakeholder group of focus
to collect rich data on how training directors felt and how they perceived their experiences with
training evaluation and the organizations perceptions of value of training. This study found gaps
in the knowledge of the stakeholders best positioned to implement this innovation, and both
assets and gaps in stakeholder motivational influences. Additionally, there were gaps identified
in the organizational culture of the 82 TRW and other stakeholder groups perceptions, of both
embedded beliefs and in actual cultural settings and the distribution of resources.
The Kirkpatrick New World Model (2016) was used to select recommendations and
design a program to address identified gaps and leverage training director assets to implement
impact-level evaluation, and achieve the stakeholder goal and prepare for the organizational goal
identified from the statement of the problem of practice. Internal recommendations included a
provision for training of declarative and procedural knowledge and the identification of resources
needed in the 82 TRW. External recommendations were also provided to address the perception
of gaps in the relationships between training professionals and the organizations they support.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 83

References
82nd Training Wing. (2017a). 82nd Training Wing, Sheppard Air Force Base, TX: CCAF
annual report for 2016. Sheppard Air Force Base, TX: Author.
Aaberg, W., & Thompson, C. J. (2012). Infusing two models of evaluation into a military
environment: A case study. Performance Improvement, 51(1), 17–25. https://doi.org/10.
1002/pfi.21237
Agocs, C. (1997). Institutionalized resistance to organizational change: Denial, inaction, and
repression. Journal of Business Ethics, 16(9), 917–931. https://doi.org/10.1023/A:101793
9404578
Aguinis, H., & Kraiger, K. (2009). Benefits of training and development for individuals and
teams, organizations and society. Annual Review of Psychology, 60(1), 451–474.
https://doi.org/10.1146/annurev.psych.60.110707.163505
Air Education and Training Command. (2014). Faculty development and master instructor
programs (AETCI36-2202). San Antonio, TX: Author.
Air Education and Training Command. (2015). Technical and basic military training evaluation
(AETCI36-2640). San Antonio, TX: Author.
Air Education and Training Command. (2017a). Fact sheet. Retrieved from http://www.aetc.af.
mil/Units/
Air Education and Training Command. (2017b). Technical and basic military training
development (AETCI36-2641). San Antonio, TX: Author.
Alkin, M. C. (2011). Evaluation essentials: From a to z. New York, NY: The Guildford Press.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 84

Allen, W. C. (2006). Overview and evolution of the ADDIE training system. Advances in
Developing Human Resources, 8(4), 430–441. https://doi.org/10.1177/152342230629
2942
Alliger, G. M., Beard, R., Bennett, W., Jr., Symons, S., & Colegrove, C. (2013). A psychometric
examination of mission essential competency (MEC) measures used in air force
distributed mission operations training needs analysis. Military Psychology, 25(3), 218–
233. https://doi.org/10.1037/h0094964
Alper, S., Tjosvold, D., & Law, K. (2000). Conflict management, efficacy, and performance in
organizational teams. Personnel Psychology, 53(3), 625–642.
https://doi.org/10.1111/j.1744-6570.2000.tb00216.x
Alvarez, K., Salas, E., & Garofano, C. M. (2004). An integrated model of training evaluation and
effectiveness. Human Resource Development Review, 3(4), 385–416.
https://doi.org/10.1177/1534484304270820
Anderson, L. W. et al. (2000). A taxonomy for learning, teaching and assessing: a revision of
Bloom’s taxonomy of educational objectives. Abridged ed. New York , NY: Allyn &
Bacon
Andru, P., & Botchkarev, A. (2011). Return on investment: A placebo for the Chief Financial
Officer…and other paradoxes. Journal of Multidisciplinary Evaluation, 7(16), 201–206.
Retrieved from http://journals.sfu.ca/jmde/index.php/jmde_1/article/view/322
Association for Talent Development. (2015a). 2015 state of the industry. Alexandria, VA: ATD
Press.
Association for Talent Development. (2015b). Industry training report. Alexandria, VA: ATD
Press.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 85

Association for Talent Development. (2016). Evaluating learning: Getting to measurements that
matter. Alexandria, VA: ATD Press.
Attree, M. (2006). Evaluating healthcare education: Issues and methods. Nurse Education Today,
26(8), 640–646. https://doi.org/10.1016/j.nedt.2006.07.014
Baker, L. (2006). Metacognition. Retrieved from http://www.education.com/reference/
article/metacognition/
Baldwin, T. T., & Ford, K. J. (1988). Transfer of training: A review and directions for future
research. Personnel Psychology, 41(1), 63–105. https://doi.org/10.1111/j.1744-
6570.1988.tb00632.x
Bandura, A. (1994). Self-efficacy. In V. S. Ramachaudran (Ed.), Encyclopedia of Human
Behavior (Vol. 4, pp. 71–81). New York, NY: Academic Press.
Bandura, A. (2000). Exercise of human agency through collective efficacy. Current Directions in
Psychological Science, 9(3), 75-78. https://doi-org.libproxy1.usc.edu/10.1111%2F1467-
8721.00064
Baraldi, S., & Cifalino, A. (2015). Delivering training strategies: The balanced scorecard at
work. International Journal of Training and Development, 19(3), 179–198.
https://doi.org/10.1111/ijtd.12055
Bates, R. A., Holton, E. F., III, & Hatala, J. P. (2012). A revised learning transfer system
inventory (LTSI): Factorial replication, item reduction and validation. Human Resource
Development International, 15(5), 549–569. https://doi.org/10.1080/13678868.2012.
726872
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 86

Bell, H. H., & Reigeleuth, C. M. (2014). Paradigm change in military training and education.
Educational Technology, 54(3), 52–57. Retrieved from https://www.researchgate.net/
publication/271587508_Paradigm_Change_in_Military
Bogdan, R. S., & Biklen, S. K. (2007). Qualitative research for education: An introduction to
theory and methods (5th ed.). Boston, MA: Allyn and Bacon.
Brewer, T. K. (2007). Use of Phillips five level training evaluation and return on investment
framework in the United States non-profit sector (Doctoral dissertation). Retrieved from
ProQuest Dissertations and Theses Database. (3288246)
Buckingham, M., & Coffman, C. (1999). First, break all the rules: What the world’s greatest
managers do differently. New York, NY: Simon & Schuster.
Bunch, K. J. (2007). Training failure as a consequence of organizational culture. Human
Resource Development Review, 6(2), 142–163. https://doi.org/10.1177/15344843072
99273
Burnett, L. D. (2012). Professional development: A six-year data evaluation of HIDTA law
enforcement task force training programs (Doctoral dissertation). Retrieved from
ProQuest Dissertations and Theses Database. (3513727)
Cevero, R. M. (1985). Continuing professional education and behavioral change: A model for
research and evaluation. The Journal of Continuing Education in Nursing, 16(3), 85-88.
Chapman, D. D. (2004). Preferences of training performance measurement: A comparative
study of training professionals and non-training managers. Performance Improvement
Quarterly, 17(4), 31–49. https://doi.org/10.1111/j.1937-8327.2004.tb00319.x
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 87

Chiaburu, D. S., & Lindsay, D. R. (2008). Can do or will do? The importance of self-efficacy
and instrumentality for training transfer. Human Resource Development International,
11(2),199-206. https://doi-org/10.1080/13678860801933004
Clark, R. E. (2005). Research-tested team motivation strategies. Performance Improvement, 4(1),
13-16.
Clark, R. E., & Estes, F. (2008). Turning research into results: A guide to selecting the right
performance solutions. Charlotte, N.C.: Information Age.
Community College of the Air Force. (2017). 2017-2019 CCAF general catalog. Maxwell AFB,
AL: Author.
Creswell, J. W. (2014). Research design: Qualitative, quantitative and mixed-methods
approaches. Thousand Oaks, CA: SAGE.
Curado, C., & Teixeira, S. M. (2014). Training evaluation levels and ROI: The case of a small
logistics company. European Journal of Training and Development, 38(9), 845–870.
Retrieved from www.emeraldinsight.com/2046-9012.htm https://doi.org/10.1108/EJTD-
05-2014-0037
Daly, E. (2006). Behaviorism. Retrieved from http://222.education.com/reference/article/
behavior/
Deis, M. R. (2010). A phenomenological study of United States Air Force civilian squadron
director leadership development. (Doctoral dissertation). doi:10.1146/annurev.psych.60.1
10707.16505
Denler, H., Wolters, C., & Benzon, M. (2014). Social cognitive theory. Retrieved from:
http://www.education.com/reference/article/social-cognitive-theory/
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 88

Denning, S. (2005). The leaders guide to storytelling: Mastering the art and discipline of
business narrative. New York, NY: Wiley and Sons.
Eccles, J. (2006). Expectancy-value theory. Retrieved from http://www.education.com/reference/
article/expectancy-value-theoryEdge,+Beyond+Data.pdfEducation_and_Training_172
Foster, R. E., & Fletcher, J. D. (2013). Toward training transformation. Military
Psychology,25(3), 308–317. https://doi.org/10.1037/h0094971
Galanis, G., & Sottilare, R. (2013). Effective measurement is essential for effective training. In
C. Best, G. Galanis, J. Kerry, & R. Sottilare (Eds.), Fundamental issues in defense
training and simulation (pp. 253–256). Boca Raton, FL: CRC Press.
Gallimore, R., & Goldenberg, C. (2001). Analyzing cultural models and settings to connect
minority achievement and school improvement research. Educational Psychologist,
36(1), 45-56. https://doi.org/10.1207/S15326985EP3601_5
Giangreco, A., Carugati, A., & Sebastiano, A. (2010). Are we doing the right thing? Food for
thought on training evaluation and its context. Personnel Review, 39(2), 162–177.
https://doi.org/10.1108/00483481011017390
Gilley, A., Gilley, J. W., & McMillian, H. S. (2009). Organizational change: Motivation,
communication, and leadership effectiveness. Performance Improvement Quarterly,
21(4), 75-94. https://doi.org/10.1002/piQ.20039
Glesne, C. (2011). But is it ethical? Considering what’s right. In Becoming qualitative
researchers: An introduction (4th ed; pp. 162–183). Boston, MA: Pearson.
Grohmann, A., & Kauffeld, S. (2013). Evaluating training programs: Development and
correlates of the questionnaire for professional training evaluation. International Journal
of Training and Development, 17(2), 135–155. https://doi.org/10.1111/ijtd.12005
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 89

Grossman, R., & Salas, E. (2011). The transfer of training: What really matters. International
Journal of Training and Development, 15(2), 103–120. https://doi.org/10.1111/j.1468-
2419.2011.00373.x
Guerci, M., & Vinante, M. (2011). Training evaluation: An analysis of the stakeholder’s
evaluation needs. Journal of European Industrial Training, 35(4), 385–410.
https://doi.org/10.1108/03090591111128342
Harrington, L. M., Reedy, K., Ausink, J. A., Bennett, B. E., Bicksler, B., Jones, D. D., & Ibarra,
D. (2017). Air Force non-rated technical training: Opportunities for improving pipeline
processes. Santa Monica, CA: RAND. Retrieved from https://www.rand.org/pubs/
research_ reports/RR2116.html
Harrison, T. (2016). Analysis of the FY2017 defense budget. Report of the CSIS Defense Outlook.
Center for Strategic Studies. Washington, D. C.: Rowman & Littlefield.
Harter, J. K., Schmidt, F. L., Kilham, E. A., & Asplund, J. W. (2006). Q12 meta-analysis.
Technical Paper, The Gallup Organization, Omaha, NE.
Hattaway, D., & Henson, J. (2013). It can be smart to dumb things down. Stanford Social
Innovation Review Online. Retrieved from http://ssir.org/articles/entry/it_can_be_smart_
to_dumb_things_down#sthash.TRePfJgS.dpuf
Hessler, K. L., & Henderson, A. M. (2013). Interactive learning research: Application of
cognitive load theory to nursing education. International Journal of Nursing Education
Scholarship,10(1), 133-141.
Hiebert, J. & Lefevre, P. (1986). Conceptual and procedural knowledge: An introductory
analysis. In J. Hiebert (Ed.), Conceptual and procedural knowledge: The case of
mathematics (pp. 1-27). Hillsdale, NJ: Lawrence Erlbaum Associates.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 90

Hill, D. R. (1999). Evaluation of formal, employer-sponsored training in the United States
healthcare industry (Doctoral dissertation). Retrieved from ProQuest Dissertations and
Theses Database. (9947255)
Ho, A. D. D., Arendt, S. W., Zheng, T., & Hanisch, K. A. (2016). Exploration of hotel managers
training evaluation practices and perceptions utilizing Kirkpatrick’s and Phillip’s models.
Journal of Human Resources in Hospitality & Tourism, 15(2), 184–208. https://doi.org/
10.1080/15332845.2016.1084861
Holton, E. F., III. (1996). The flawed four-level evaluation model. Human Resource
Development Quarterly, 7(1), 5–21. https://doi.org/10.1002/hrdq.3920070103
Hypes, B. R. (2016). Training programs in the federal government: A systematic review of how
effectiveness is evaluated (Doctoral dissertation). Retrieved from ProQuest Dissertations
and Theses Database. (10252416)
Katzell, R. A. (1952). Can we evaluate training? A summary of a one day conference for
training managers. A publication of the Industrial Management Institute, University of
Wisconsin, April, 1952.
Kennedy, P. E., Chyung, S. Y., Winiecki, D. J., & Brinkerhoff, R. O. (2013). Training
professional’s usage and understanding of Kirkpatrick’s level 3 and level 4 evaluations.
International Journal of Training and Development, 18(1), 1–21. https://doi.org/10.1111/
ijtd.12023
Kester, L., Kirschner, P. A., & van Merrienboer, J. J. G. (2006). Just-in-time information
presentation: Improving learning a troubleshooting skill. Contemporary Educational
Psychology, 31(2), 167-185. https://doi.org/10.1016/j.cedpsych.2005.04.002
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 91

Kirkpatrick, D. L., (1956). AT+D Classic: How to start an objective evaluation of your training
program. An excerpt from the May-June 1956 Journal of the American Society of
Training Directors. Training and Development, May 2004, 58(5), 1-3.
Kirkpatrick, D. L. (1959a). Techniques for evaluating training programs. Journal of the
American Society of Training Directors, 13(11), 3-9.
Kirkpatrick, D. L. (1959b). Techniques for evaluating training programs: Part 2—Learning.
Journal of the American Society of Training Directors, 13(12), 21-26.
Kirkpatrick, D. L. (1960a). Techniques for evaluating training programs: Part 3—Behavior.
Journal of the American Society of Training Directors, 14(1), 13-18.
Kirkpatrick, D. L. (1960b). Techniques for evaluating training programs: Part 4—Results.
Journal of the American Society of Training Directors, 14(2), 28-32.
Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels. San Francisco, CA:
Berrett-Koehler.
Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Four levels of training evaluation. Alexandria,
VA: ATD Press.
Kirschner, P., Kirschner, F., & Paas, F. (2006). Cognitive load theory. Retrieved from
https://www.eduction.com/reference/article/information-processing-theory/
Knowles, M. S. (1980). Modern practice of adult education: From pedagogy to andragogy.
Chicago, IL: Follett.
Kodwani, A. D. (2017). Decoding training effectiveness: The role of organizational factors.
Journal of Workplace Learning, 29(3), 200–216. https://doi.org/10.1108/JWL-05-2016-
0038
Kotter, J. P. (2007). Leading change. Harvard Business Review, 86(1), 97-103.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 92

Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory into
Practice,41(4), 212–218. https://doi.org/10.1207/s15430421tip4104_2
Lewis, L. K. (2011). Organizational change: Creating change through strategic communication.
Chichester, West Sussex, UK: Wiley-Blackwell.
Locke, L. F., Silverman, S. J., & Spirduso, W. W. (2010). Reading and understanding research
(3rd ed.). Thousand Oaks, CA: SAGE.
Malloy, C. L. (2011). Moving beyond data: Practitioner-led inquiry fosters change. EDge, 6(4),
3-19. Retrieved from https://ou-eds-faculty.wikispaces.com/file/view/PDK,+
Maxwell, J. A. (2013). Qualitative research design: An interactive approach. Thousand Oaks,
CA: SAGE.
Mayer, R. E. (2011). How learning works. Applying the Science of Learning. Boston, MA:
Pearson Education.
Merriam, S. B., & Tisdell, E. J. (2016). Qualitative research: A guide to design and
implementation (4th ed.). San Francisco, CA: Jossey-Bass.
Miller, R. L., King, J., & Mark, M. (2008). The oral history of evaluation: The professional
development of Daniel L. Stufflebeam. The American Journal of Evaluation, 29(4), 555–
571. https://doi.org/10.1177/1098214008321995
Nafukho, F. M., Graham, C. M., & Kacirek, K. (2009). Education service agency audits:
Reinforcing the need for systematic evaluation. International Journal of Vocational
Education and Training, 17(2), 19–38.
Pajares, F. (2006). Self-efficacy theory. Retrieved from http://www.education.com/reference/
article/self-efficacy-theory
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 93

Palmer, P. C. (2008). Differences in training stakeholders’ investment, expectations, and
measurement preferences for training evaluation data (Unpublished doctoral
dissertation). University of Tulsa, Tulsa, OK.
Patton, M. Q. (2002). Qualitative research & evaluation methods (3rd ed.). Thousand Oaks, CA:
SAGE.
Pekrun, R. (2006). Emotions as drivers of learning and cognitive development. In R. A. Calvo
and S. K. D’Mello (eds.), New Perspectives on Affect and Learning Technologies,
Explorations in the Learning Sciences (pp. 23–39). New York, NY: Springer.
Percival, J. C., Cozzarin, B. P., & Formaneck, S. D. (2013). Return on investment for workplace
training: The Canadian experience. International Journal of Training and Development,
17(1), 20–32. https://doi.org/10.1111/ijtd.12002
Phillips, H. L., IV, & Foster, T. C. (2008). Toward a naval aviation training quality feedback
system. Performance Improvement, 47(3), 36–45. https://doi.org/10.1002/pfi.192
Phillips, J. J. (1998). Systematic evaluation: Trends and practices. In J. Phillips (Ed.), In action:
Implementing evaluation systems and processes (pp. 1–12). Alexandria, VA: ASTD.
Phillips, J. J. (2012). Moving from evidence to proof (training evaluation). Human Resource
Management International Digest, 20(1). https://doi.org/10.1108/hrmid.2012.04420
aaa.014
Phillips, P. P. (2003). Training evaluation in the public sector (Doctoral dissertation). Retrieved
from ProQuest Dissertations and Theses Database. (3103677)
Phillips, P. P., & Phillips, J. J. (Eds.). (2012). Measuring ROI in learning & development: Case
studies from global organizations. Alexandria, VA: ASTD Press.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 94

Pineda, P. (2010). Evaluation of training in organizations: A proposal for an integrated model.
Journal of European Industrial Training, 34(7), 673–693.
https://doi.org/10.1108/03090591011070789
Pintrich, P. R. (2000). An achievement goal theory perspective on issues in motivation
terminology, theory, and research. Contemporary Educational Psychology, 25(1), 92-
104. https://doi.org/10.1016/ceps.1999.1017
Preston, K. F. (2010). Leadership perceptions of results and return on investment training
evaluations (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses
Database. (3419081)
Pulichino, J. P. (2007). Usage and value of Kirkpatrick’s four levels of training evaluation
(Unpublished doctoral dissertation). Pepperdine University, Malibu, CA.
Rubin, H. J., & Rubin, I. S. (2012). Qualitative interviewing: The art of hearing data (3rd
ed.).Thousand Oaks, CA: SAGE.
Rueda, R. (2011). The three dimensions of improving student performance: Finding the right
solutions to the right problems. New York, N.Y.: Teacher’s College Press.
Salas, E., Milham, L. M., & Bowers, C. A. (2003). Training evaluation in the military:
Misconceptions, opportunities, and challenges. Military Psychology, 15(1), 3–16.
https://doi.org/10.1207/S15327876MP1501_01
Salas, E., Tannenbaum, S. I., Kraiger, K., & Smith-Jentsch, K. A. (2012). The science of training
and development in organizations: What matters in practice. Psychological Science in the
Public Interest, 13(2), 74–101. https://doi.org/10.1177/1529100612436661
Schraw, G., & McCrudden, M. (2006) Information processing theory. Retrieved from
https://www.eduction.com/reference/article/cognitive-load-theory/
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 95

Schwandt, D. R., & Marquardt, M. J. (2000). Organizational learning: From world-class
theories to global best practices. New York, NY: St. Lucie Press.
Secretary of the Air Force. (2017). Air Force presents fiscal year 2018 budget. Retrieved from:
www.af.mil/News/Article-Display/Article/1191344/af-presents-fiscal-year-2018-budget
Senge, P. (1990). The fifth discipline: The art and practice of the learning organization. New
York, NY: Currency Doubleday.
Schein, E. H. (2004). The concept of organizational culture: Why bother? In E. H. Schein (Ed.),
Organizational Culture and Leadership (3
rd
ed., pp3-24). San Francisco, CA: Jossey-
Bass.
Shobe, K. K., & Curtis, B. G. W. (2007). Training assessment challenges in a military
environment: A U. S. Navy example. Performance Improvement, 46(3), 42–46.
https://doi.org/10.1002/pfi.116
Simon, M. K., & Goes, J. (2013). Scope, limitations, and delimitations. Retrieved from
http://dissertationrecipes.com/wp-content/uploads/2011/04/limitationscope
delimitation1.pdf
Singh, M. (2013). Training evaluation: Various approaches and applications. The IUP Journal of
Soft Skills, 7(1), 27–34.
Sirkin, H. L., Keenan, P., & Jackson, A. (2005). The hard side of management change. Harvard
Business Review, October, 2005, Reprint R0510G, 99-108. Retrieved from
http://www.HBR.org
Strebel, P. (1996). Why do employees resist change? Harvard Business Review, 74(3), 86-92.
https://doi.org/10.1109/EMR.2009.5235497
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 96

Stufflebeam, D. L. (2003, October). The CIPP model for evaluation: An update, a review of the
model’s development, a checklist to guide implementation. Paper presented at the 2003
Annual Conference of the Oregon Program Evaluators Network (OPEN) Portland, OR.
https://doi.org/10.1007/978-94-010-0309-4_4
Stufflebeam, D. L., Madaus, G. F., & Kellaghan, T. (2000). Evaluation models: Viewpoints on
educational and human services evaluation (2nd ed.). Boston, MA: Kluwer Academic.
Swanson, R., & Holton, E. (2009). Foundations of human resource development. San Francisco,
CA: Berrett-Koehler.
Tanner, G., & Otto, K. (2016). Superior-subordinate communication during organizational
change: Under which conditions does high-quality communication become important?
The International Journal of Human Resource Management, 27(19), 2183-2201.
Thalheimer, W. (2018, January 30). Donald Kirkpatrick was NOT the originator of the four-level
model of learning evaluation. [Web log post]. Retrieved from
https://www.worklearning.com/2018/01/30/donald-kirkpatrick-was-not-the-originator-of-
the-four-level-model-of-learning-evaluation/
Twitchell, S., Holton, E. F., III, & Trott, J. W., Jr. (2000). Technical training evaluation practices
in the United States. Performance Improvement Quarterly, 13(3), 84–109. https://doi.org/
10.1111/j.1937-8327.2000.tb00177.x
United States Air Force Doctrine. (2015). AFD 1-1, Core Doctrine, Vol. II. Washington, DC:
Author.
United States Air Force. (2010). Instruction. AFI36-2201, Training evaluation. Washington, DC:
Author.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 97

United States Air Force. (2017a). Air Force demographics. Retrieved from
http://www.afpc.af.mil/Air-Force-Demographics
United States Air Force. (2017b). Fact sheet. Retrieved from: http://www.af.mil/
United States Air Force. (2017c). Second Air Force 2017 strategic plan. Randolph Air Force
Base, TX: Author.
United States Southern Command. (2018, June 6). U.S. Air Force to lift six Guatemalan children
injured in volcanic explosion in humanitarian to Galveston. Air Force Online. Retrieved
from https://www.af.mil/News/Article-Display/Article/1543018/us-air-force-to-airlift-
guatemalan-children-injured-by-volcanic-eruption/
Vilchez, V., Darnall, N., & Aragon Correa, A. (2016). Stakeholder influences on the design of
firms’ environmental practices. Journal of Cleaner Production, 142, 3370–3381.
https://doi.org/10.1016/j.jclepro.2016.10.129
Vogel-Walcutt, J. J., Fiorella, L., & Malone, N. (2013). Instructional strategies framework for
military training systems. Computers in Human Behavior, 29(4), 1490-1498.
https://doi.org/10.1016/j.chb.2013.01.038
Weiss, R. S. (1994). Learning from strangers: The art and method of qualitative interview
studies. New York, NY: The Free Press.
Wigfield, A., & Cambria, J. (2010). The decade ahead: Theoretical Perspectives on motivation
and achievement. Advances in Motivation and Achievement, 16, 35-70. https://doi-
org/10.1108/S0749-7423(2010)000016A005
Williams, R. C., & Nafukho, F. M. (2015). Technical training evaluation revisited: An
exploratory, mixed-methods study. Performance Improvement Quarterly, 28(1), 69–93.
https://doi.org/10.1002/piq.21187
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 98

Appendix A: Participating Stakeholders’ Sampling and Recruitment, Interview Strategy,
and Rationale
This study was intended to evaluate the perceptions of the stakeholders of focus
regarding why and how training evaluation was conducted, assessed, and used to support long-
term, strategic decision-making within the training wing. According to both Creswell (2014) and
Maxwell (2013), there are several ways of collecting qualitative data: observation, interview,
document review, and audio or visual artifacts. Further, Maxwell (2013) identified convenience
sampling as a type of purposive sampling useful when the population to be assessed has unique
characteristics. The training directors interviewed for this study were the total community within
the training wing with the knowledge and experience of evaluation to provide the data for this
study.
Interviews were conducted face-to-face with the training directors to ascertain their
knowledge, experience, thinking and motivation to implement impact-level evaluation and to
understand their perceptions of organizational models and settings as either supports or barriers.
The need for better data-based decisions related to student outcomes and value of the training
conducted by the 82 TRW was identified at the Faculty Development Executive Council (FDEC)
meeting of 2016. The director of the 82 TRW Training Operations department was a member of
the FDEC, approved this study, and supported participation by members of the training wing.
Participation could not be mandated by the training director, but the potential value of the results
can benefit every participant and provided motivation that encouraged full participation.
Criterion 1. Each interviewee held a senior civilian management position with oversight
responsibility for numerous technical training courses within the 82 TRW. The designation of
senior civil service management position was defined by both a four-year degree with education-
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 99

related coursework and a significant depth of experience in technical training. This criterion was
important because the research questions explored the possible gaps in each director’s current
knowledge and motivation as well as the training wing organizational culture and settings. The
existence of gaps had influence on the identification of leading indicators, correlated training
objectives and achievement of the stakeholder goal.
Criterion 2. Each interviewee held a general schedule 1750 series position at the 13 or
14 grade that indicated a minimum of a four-year degree with 24 credit hours in education
specific courses and 10 to 40 years of experience in positions that span technical training
instruction, training management, curriculum development, policy and evaluation, training
director and management positions. The selected interviewees also typically earned graduate
level degrees, and completed professional and continuing education from the Air Command and
Staff College, the Air War College, or other Air University programs. The significance of this
criterion was that it identified the best possible participants with broad levels of experience in
technical training positions that directly contribute to achievement of the stakeholder goal.
Criterion 3. While training directors were invited to participate in this study, it was
voluntary for the selected interviewees. A requirement of these positions included potential for
temporary duty assignments away from 82nd TRW. Interviewees were not always available
with the timeframe for the interviews. Participants were requested to refrain from discussing the
interview process until after this study was complete. The significance of this criterion was that
interviews were face-to-face and open-ended. Interviewees were made comfortable with
considering and exploring questions they initially found unfamiliar or challenging to their
organizational beliefs and long-standing practices. Telephone interviews would have been
restrictive; if any potential interviewee was out of the area, it was because they have additional
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 100

responsibilities that limited the time and attention they had available to minimize anxiety and
achieve full and complete responses.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 101

Appendix B: Interview Protocol
Thank you for agreeing to meet with me and participate in my study. I know that you are busy,
and I hope that you also find value in these questions and in any recommendations from my
dissertation. I appreciate your taking the time to participate. This should take about an hour, but
we have allotted an hour and a half in case we need extra time.
I am a doctoral candidate at the University of Southern California, and this is an innovation study
of impact-level technical training evaluation within the wing. For the purpose of this study, I am
here as a student researcher to collect data and not as a member of the wing, and this study will
not judge either your performance or the performance of your squadron or group. Any
information you share with me in the course of this interview may be considered for this study,
but it will remain anonymous. This interview is confidential and neither your name, position or
title will be disclosed. No one outside of this study will have access to this information, nor is
anyone from the U.S. government, the DoD, the USAF or the 82 TRW associated with this
study. I may use a direct quote from you but I will not provide your name specifically or any
other identifying data. I will gladly provide you a copy of my final publication upon request.
Your participation is entirely voluntary. You may skip any questions you do not want to
answer and you may stop this interview at any time.
With your consent, I will be using this recorder throughout the interview in order to capture your
responses accurately. This recording and the subsequent transcripts will not be shared with
anyone outside the scope of this study except a transcription service. It will be maintained on my
personal computer using files protected by two-factor authentication and stored on a separate,
password protected flash drive dedicated solely to my interview materials. I will be using a
third-party transcription service. All the files will be returned to me once the final interviews are
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 102

transcribed. All files and the flash drive will be destroyed three years after the date my
dissertation defense is approved. The information sheet provides all of this in writing and is
yours to keep.
Before we begin, do you have any questions about the purpose or intent of this interview or my
study?
Thank you. For the record, do I have your consent to record this conversation?
1. Tell me what you know about impact-level evaluation. (Conceptual Knowledge)
Probe: We track student production metrics that have value to 2AF, AETC and others.
How could other impact-level metrics show the impact of training?
2. What key indicators would show the impact of training at an institutional level? (Conceptual
Knowledge)
Probe: What impact-level questions, if any, have you been asked about the training
we deliver?
Probe: During a Specialty Training Requirements Team (STRT) or Utilization &
Training Workshop (U & TW), have functional or career field managers asked questions
about training impact that you have been unable to fully answer?
3. How could you measure key indicators? (Procedural Knowledge)
Probe: At what points of an airman’s continuum of learning could we measure key
metrics?
4. How would you describe the value of measuring the impact of training at the impact-level?
(Expectancy-Value Theory Motivation)
5. What could be gained if we measured impact? (Expectancy-Value Theory)
Probe: Could you identify some ways that impact metrics could be used?
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 103

6. What might be some reasons why the Air Force does not currently measure impact of training
at the impact-level? (Organizational)
7. How would impact metrics influence our supported commands? (Cultural Model)
Probe: Do you feel the major commands would or would not include metrics of training
impact in their own reporting?
Probe: What would it look like if the major commands could easily point to the impact of
training as a factor in their own successes or failures?
8. How do you feel about your ability to implement measurement of the impact of training at an
impact level? (Self-efficacy)
Probe: What do believe are your strengths in considering impact-level impact?
(Metacognitive)
Probe: What might be some areas of challenge for you?
9. What could motivate the 82 TRW to adopt the practice of evaluating for impact at the
institutional level? (Cultural Model)
10. What would leadership need to learn about implementing and using impact metrics?
(Cultural Model)
Probe: What could overcome resistance to impact-level evaluation?
11. If impact metrics were stated in terms meaningful to other Air Force major commands, how
would it affect the degree to which they value training? (Cultural Model)
Probe: To what degree do they currently value training impact data?
Probe: To what degree would they value more explicit training impact data?
12. What kinds of resources would we need to implement impact evaluations? (Cultural
Setting)
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 104

Probe: Would resources be provided if we produced impact metrics? (Cultural Setting)
13. Do you have any additional thoughts about the value or use of impact metrics?

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 105

Appendix C: Credibility and Trustworthiness
The credibility and trustworthiness of the results of any study and its usefulness to
advance knowledge in a field is at the heart of research if it is to be meaningful (Merriam &
Tisdell, 2016). Because qualitative research relies on rich and detailed description instead of
quantitative data, the data used and subsequent analysis must be believable, reproducible, and
consistent (Merriam & Tisdell, 2016). Validity is a challenging concept to define and may be
based on selective criteria chosen to match with the purpose of the study (Merriam & Tisdell,
2016). The characteristics intended to maximize trust in the results of this study included a
review of an implicit quantitative component, comparison or responses and the search for
discrepant responses, recommended by Maxwell (2013).
While quantitative studies explicitly rely on the numeric prevalence of like responses,
qualitative data are useful to determine implied trends of responses (Maxwell, 2013). In the
analysis of the eleven sets of responses, believable conclusions were possible based on the
consistency between responses. Where a participant’s response differed, it was evident from that
the difference was one of degree and not of the participants’ conclusion. There was one
exception to this in the data analysis, when one director felt that there was no need for additional
resourcing to conduct impact-level evaluation. The possible reasons for this difference in
perspective by one director was specifically addressed in the analysis section.
The second method of assessing credibility in this study was a comparison of the
responses. Each interview took between 45 and 60 minutes to conduct, and each produced a rich
amount of detail that was based on the unique backgrounds and experiences of each participant.
A review of the coded responses revealed that while some responses were coded as a negative
response and others were coded as a positive response, in the end, the responses aligned with the
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 106

identified trends that appeared, with the single exception noted in the previous paragraph. The
differences between any training directors’ responses were more a matter of degree than of
opposing answers.
The third strategy used to establish credibility of the study was a specific search for
discrepant responses. As already noted, there was only one response to a single influence that
completely differed from all others. It was apparent from the beginning of this study that this
population was particularly homogenous; the directors share similar backgrounds and they have
filled many of the same positions as they promoted into their current assignments. Each
interviewee was conducted with the intent to extract rich, descriptive detail and full answers, but
it was also intended to listen closely for deviations in responses that might indicate a different
and unanticipated perspective.
The credibility and validity of this study was important to maintain because the findings
should have credibility to impact and improve the delivery of technical training in the 82 TRW.
Technical training provided by the Aircraft University is highly specialized and, in many cases,
unavailable from any other source, but the training wing continues to be committed to providing
not only the best possible training but training that best meets the needs of the supported
commands of the USAF.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 107

Appendix D: Ethics
Ethical research requires specific protections for the human subjects of the research and a
commitment to do no harm (Glesne, 2011). Additionally, institutional review board (IRB)
principles guide the conduct of data collection, including a respect for the interviewee’s time,
experience, and knowledge (Rubin & Rubin, 2012). This study included the use of human
subjects, the stakeholder group of focus was small and homogenous and the location and context
for this study lay within the researcher’s work environment. Ethical considerations for these
facts dictated that potential ethical risks and considerations were addressed.
This qualitative study was designed to collect data by interviewing a small and unique
group of training specialists. To protect the rights of the participants, the purpose and design of
the study and the interview protocol that directed the questions used were submitted to the
University of Southern California’s IRB process. The study and the protocol were approved and
the interviews were subsequently conducted in accordance with the rules and guidelines designed
to protect the rights of the participants of the study.
Training director is used as a generic identification, as the specific job titles would reveal
personally identifying information. Training directors are federal civil service employees,
typically in general schedule (GS) 13 and 14 positions (Deis, 2010). There are a limited number
of such senior civilian positions in any USAF or AETC training organization and the ethical
challenge was the small population available for the study and eventual risk to maintaining
anonymity. This drove the decision to eliminate demographic data on the director’s years of
experience or even gender, as it became too easy to identify individual responses.
The researcher is also a federal civil service employee at the GS-12 level with 17 years of
experience. This is a position junior to that of all the participants. At the time of the study, there
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 108

was also no direct reporting relationship with any of the members of the stakeholder group of
focus. There existed no possibility of influence by the researcher on any of the directors by
coercion or potential gain. Additionally, due to the federal civil service rules and guidelines, the
researcher cannot benefit from the conduct or the results of this study; there is no vested interest
on the part of the researcher apart from providing recommendations to solve this problem of
practice. Similarly, the participants did not directly benefit from participating in this study nor
were they penalized if they declined to answer interview questions or strongly disagreed with the
conclusions of the study.
The purpose of this research was provided during the initial contact with each respondent,
and the potentially beneficial uses of the final study results explained, as advised by Merriam and
Tisdell (2016). Information sheets were provided to each one prior to the scheduled interview to
ensure that interviewees were fully-informed of the intent of the research and the use to which it
may be put. Informed consent is ethically necessary to confirm that respondents fully understand
what data will be collected and to what uses it may be put by the researcher (Glesne, 2011). The
information sheet specifically explained that participation was voluntary, could not be coerced,
data would remain protected and anonymous and the participants could end the interview at any
time without penalty. Additionally, the form clearly stated that responses were kept confidential,
and anonymity maintained throughout and after the study (Merriam & Tisdell, 2016). Each
participant was asked for and provided consent to be recorded during the interview sessions to
maintain fidelity of the information collected.
The data collected will continue to be maintained securely until it is destroyed at the end
of this study. Subsequent transcripts were available for review and validation of accuracy. All
data collected will continue to be maintained in digital format under two-factor authentication
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 109

protection on the researcher’s own personal computer, within an encrypted digital file vault.
There is an extremely low probability of data incursion or corruption.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 110

Appendix E: Limitations and Delimitations
This section identifies the limitations and delimitations of this study. According to
Simon and Goes (2013), limitations are the uncontrollable influences that constrain the design
and construction of a study. Delimitations are the intentionally selected boundaries that restrain
a study to only address the research questions identified with the problem of practice.
Limitations and delimitations are addressed as one component of the overall credibility and
trustworthiness of this study.
The first limitation of note in this study was that 11 interviews were planned, a small
population from which to collect data and infer trends and patterns. At the time of this study,
there were only 11 GS-14 and -13 supervisory training specialists in the entire population of the
82 TRW with the ability to implement changes to training evaluation practices. There are few
such positions available throughout the Air Force at any location, and with three training groups,
the 82 TRW is unique in having more training directors available than any other training wing.
An effective way to minimize the impact of this limitation would be to repeat this study within
another Air Force training group or wing that conducts enlisted technical training and collect
additional qualitative data from more training directors. An opportunity to repeat the interview
protocol in Appendix A outside of the relatively isolated and homogenous 82 TRW could
significantly increase the reliability and validity findings of this study.
The second limitation identified was the limited availability of research related to training
evaluation within the United States military. Additionally, there was no documentation or
artifacts related to impact-level evaluation found from the training wing, as it has not been
required nor conducted, except for one training group. Near the end of this study, the researcher
found that a limited amount of formal training was conducted, and some of the study participants
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 111

were more familiar with the concepts and procedures of impact-level evaluation. Additionally,
2AF is pursuing additional training and research into the value of the Kirkpatrick New World
Model (2016). These intentions are currently limited due to funding constraints.
There were also delimitations of note for this study that intentionally restricted its scope.
The first delimitation was due to the researcher’s intent to conduct a qualitative and descriptive
study of a specific problem of practice related to training evaluation within the 82 TRW, and not
expand to the larger community of USAF training in multiple locations. The research questions
and the physical availability of 11 training directors provided the boundaries and established the
requirements for the data collection method of semi-structured interviews and the specific
protocol questions used.
This innovation study was designed to recommend the 82 TRW implement impact-level
evaluation by learning to identify leading indicators that would, in turn, support strategic
decision-making for MAJCOM future training needs. A second delimitation was in restricting
the study to the 82 TRW training directors of the training groups and squadrons as the
stakeholders of focus and concentrating on what would be recommended internally to the wing.
Restricting the size of the implementation was intended to keep the innovation manageable, and
provide a prototype for applicability throughout other training units within AETC.
A third delimitation was in selecting the design of this study as a qualitative
investigation. Merriam and Tisdell (2016) noted that a characteristic of action research is that it
focuses on a problem of practice, and as action research, it is typically conducted as a qualitative
study. A quantitative or mixed methods study may have discovered additional data that was not
found in the results of interview-based data.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 112

Finally, the recommendations from this study were designed for implementation using
the Kirkpatrick’s New World Model (2016) and did not consider or include other possible
approaches other than leading indicators correlated to training objectives and measured on a
regular basis. This was not to imply that the New World Model is the only methodology for the
implementation of impact-level evaluation to improve or ensure the effectiveness and efficiency
of the training provided to supported commands. AETC as the MAJCOM responsible for
technical training is currently introducing innovative practices that may intersect with and
support the intentions of this impact-level evaluation, but time and the timing of these
implementations prevented inclusion or consideration of them.
In 2018 AETC rescinded the written instructional guidance that dictated the design,
delivery, and evaluation of technical training conducted by AETC (AETC, 2014, 2015, 2017b).
To encourage innovation in how the Air Force designs and delivers training, senior leadership
removed compliance issues, restrictions, and bureaucratically dense requirements and
encouraged instructional professionals who know best to do what needs to be done with
efficiency and effectiveness. One result of this is that 2AF and AETC are exploring the
applicability of the Kirkpatrick New World Model, once funding becomes available for training.
The intent and result of this may have influenced the specific recommendations in the Program,
designed for the findings of this study.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 113

Appendix F: Recommendations for Practice to Address KMO Influences
This section discusses recommendations generated by this study and how to implement
and evaluate them for this innovation. The findings from interviews of the training directors
indicated a need for impact-level evaluation knowledge and skill training as well as for updated
training evaluation paradigms and resources common to the culture of the 82 TRW. The
following paragraphs describe training seminars recommended to increase the knowledge and
motivation influences that fall short of optimal for successful implementation of impact-level
evaluation. Success breeds achievement and success from the program will also support success
in altering organizational expectations and the wing’s capacity to evaluate training more
effectively. Program training does not end with the end of the seminar, however. Instead, it
establishes how squadrons and groups can use impact-level evaluation practices to spread the
broader use of impact-level evaluation practices.
The New World Model (Kirkpatrick & Kirkpatrick, 2016) directs the design of training,
to begin with the highest or fourth level of evaluation, with the end goal of training maintained as
the organizational raison d’etre for the design, development, and implementation of training.
Level 4 asks the extent to which the organization benefitted from training at every point along
the analysis, design, development, implementation, and evaluation cycle of training development
and expects Levels 3, 2 and 1 evaluation to link to that Level 4 question, as well. The section
also provides specific evaluation tools for the four levels of evaluation and a data analysis and
reporting dashboard template for use at wing staff meetings that shows consolidated progress of
each level of evaluation implementation.
Finally, this section addresses the cultural models and settings that support
implementation of this program as well as the needed resources to accomplish impact-level
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 114

evaluation. The 82 TRW collects data on students, courses, instructors, effectiveness, and
efficiency of training on a daily, weekly, monthly, and annual basis, but little of it is used to
improve training. The program presents a different type of evaluation data and different methods
for its collection and evaluation. The wing is also culturally dependent on written policy and
guidance to ensure consistency and uniform compliance in collection and reporting requirements,
and the program provides the outlines for the creation of that guidance. Communication of the
purpose and value of implementing impact-level evaluation is a critical component to
successfully achieve the stakeholder goal of identifying leading indicators and, ultimately, the
organizational goal of impact-level evaluation throughout the wing. The program is designed to
communicate the necessary steps in a logical format and allow for knowledge and skill
acquisition, modeling, guided practice and finally independent application within squadrons and
groups.
Knowledge Recommendations
The knowledge influences identified in Table 14 present, with a high degree of
probability, knowledge gaps that impaired achievement of the organizational goal identified for
this study according to the Clark and Estes (2008) framework for gap analysis. Data derived
from stakeholder interviews validated the identified knowledge influence gaps. The influences
in the knowledge dimension included three types of knowledge identified by Krathwohl (2002):
declarative, procedural, and metacognitive. Krathwohl’s framework allows for a regular
examination of knowledge acquisition or need from simple to complex and usefully provides
delineated points at which knowledge gaps may be assessed. Recommendations to bridge these
knowledge gaps include the provision of fundamental information, determining a reason the
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 115

learning matters, and an opportunity to practice and to work independently (Schraw &
McCrudden, 2006).
Table 14
Summary of Knowledge Influences and Recommendations
Principle and Citation Context-Specific
Recommendation
Knowledge of what
impact-level evaluation
and leading indicators
are, and what they can
be used to measure
(Krathwohl, 2002;
Rueda, 2011)
(D)eclarative knowledge includes
factual and conceptual knowledge that
supports the application,
differentiation, and discrimination of
differing theories (Krathwohl, 2002).
Education also allows individuals to
explore the application and usefulness
of new knowledge by collective
agency (Clark & Estes, 2008)
Provide training that
includes declarative
knowledge of impact-level
evaluation practices.
(Kirkpatrick & Kirkpatrick,
2016)
Knowledge of
procedures to
identify, correlate,
and measure training
objectives and
leading indicators
(Krathwohl, 2002;
Rueda, 2011)
(P)rocedural knowledge includes
knowledge of how and when to act
or process (Krathwohl, 2002).
Job aids provide strategies and
processes to reduce cognitive load
and support schema construction
and rule automation (Kirschner,
Kirschner, & Paas, 2006)
Provide performance-
based training that
includes procedures to
identify leading indicators
and correlate them to
training objectives.
(Denler, Wolters, &
Benzon, 2009).
Reflecting on
capacity to design
and implement
impact-level training
evaluation (Clark &
Estes, 2008;
Krathwohl, 2002;
Rueda, 2011; USAF,
2010).
(M)etacognitive knowledge is self-
awareness and self-assessment of
one has acquired education and
training, and the ability to act on
what was learned, from both
familiar and newly acquired
knowledge (Krathwohl, 2002;
Mayer, 2011).
The more one applies new
knowledge and skills; the more one
can apply that knowledge and those
skills (Baker, 2006; Clark & Estes,
2008).

Continually support
training directors’
metacognition. Provide a
schedule of collaborative
meetings for training
directors to meet and
review how they each
came to and identified
leading indicators
measurements and results,
and how they intend to
prepare stakeholder
reports of the analysis of
leading indicators.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 116

Provide training that includes declarative knowledge of impact-level evaluation
practices. The data demonstrated that 73% of the training directors had no recent, formal
training on current impact-level evaluation practices. To improve their knowledge and skills,
training based on social cognitive theory will be developed. Specifically, Krathwohl (2002)
found that conceptual knowledge supports the use and discrimination of differing theories, and
the opportunity to apply factual knowledge.
The recommendation for this declarative knowledge influence incorporates the necessary
factual knowledge and targeted training using a checklist to facilitate the directors’ exploration
of the uses and potential for application of the Kirkpatrick New World Model (Kirkpatrick &
Kirkpatrick, 2016). Clark and Estes (2008) introduce the use of a job aid for somewhat common
knowledge that needs context as opposed to new learning. Additionally, a job aid can be used to
ensure efficient as well as effective training for knowledgeable professionals.
Provide performance-based training that includes procedures to identify leading
indicators and correlate them to training objectives. Data analysis from the interviews
indicated that training directors lacked experience on how to identify and measure leading
indicators for organizational-level training evaluation. Krathwohl (2002) defined procedural
knowledge as the how and when to employ a process. Clark and Estes (2008) suggest the use of
job aids to provide cognitive assistance in the form of a map or memory aid needed for
performance on the job. Denler, Wolters and Benson (2009) social cognitive theory asserted
that modeling strategies or behaviors improved the capacity to learn and perform. Learning that
is also perceived as credible and authentic also supports learning.
The recommendation for this influence is to provide uniform and consistent hands-on
training on the processes training directors will learn and in turn lead in their training units.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 117

Additionally, data from training directors supported the use of real examples from their own
work units. Training will incorporate actual leading indicators used by participating MAJCOMs
and the training objectives with which they correlate.
Continually support training directors’ metacognition. Krathwohl (2002) identified
metacognition as the self-awareness and self-assessment of one’s own education and training and
ability to act using that acquired knowledge and skill. Metacognitive reflection refers to how
more one learns, the more one can learn (Mayer, 2011; Rueda, 2011). Training directors’
responses evinced engagement in self-reflective practice both in terms of their learning and
leadership skills. This demonstrates that they had a foundation for self-reflective practice. It
will be important to continue to support the self-reflective nature of their professional practice as
use of leading indicators is introduced and promulgated throughout the squadrons and training
groups.
The recommendation to specifically support this knowledge influence is to construct a set
of scheduled meetings for directors to collaborate and review the leading indicators each
identified for his or her respective units. Training directors will then determine how the resulting
measurements will be reported as valuable information to each MAJCOM as impact-level data
that support the bottom line of that command. With use and experience in identifying leading
indicators and looking for data results meaningful to an operational command, directors will
achieve mastery of the process. The more one practices, the more one can practice (Baker, 2006;
Clark & Estes, 2008).
Motivation Recommendations
The motivational influences identified in this study are collected in Table 15 and have a
high probability of impacting achievement of the goal to implement an organizational-level
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 118

training evaluation innovation. Three elements relate to motivation: choice, persistence, and
mental effort (Clark & Estes, 2008). Choice is the element that can move an individual from
intention to action (Rueda, 2011). Persistence is the element that impels one to continue to
pursue a goal in the face of hardship or distraction (Rueda, 2011). Mental effort is the final
element and describes the continued search for knowledge, skill, and understanding needed to
perform a new requirement (Rueda, 2011). The motivational influences in Table 15 also include
context-specific recommendations that will support goal accomplishment.
Table 15
Summary of Motivation Influences and Recommendations
Assumed Motivation
Influence
Principle and Citation Context-Specific
Recommendation
Efficacy about being
able to identify,
correlate and use
leading indicators for
impact-level training
evaluation. (Alper,
Tjosvold, & Law, 2000;
Bandura, 1994; Pajares,
2006; Rueda, 2011).
Self-efficacy theory

Self-efficacy is strongly correlated to
the successful transfer of learning.
One’s judgment of his or her own
ability to organize and execute a
course of action and achieve a goal.

Pintrich (2000) said to design to
motivate, to increase self-efficacy
and a belief in one’s competence.

Denler, Wolters, and Benzon, (2014)
suggest modeling to increase learning
and performance.

Pekrun (2006) said success breeds
achievement and Bandura (1994) said
an individual’s beliefs are essential,
and choice, persistence, and effort.
Grossman &Salas said there is a high
correlation to learning success.

(Bandura, 1994; Grossman & Salas
(2011); Pajares, 2006; Pekrun, 2006)
Identify the resources
training directors feel are
needed to implement the
identification and
measurement of leading
indicators. Provide
evidence and examples from
MAJCOM leadership and
functional career managers
of the type of training
evaluation results they would
value, and could apply in
their reporting.
Use models that build self-
efficacy.
Provide goal-directed
practice.
Model, feedback practice.
In a study that examined the
value of transfer, Grossman,
and Salas (2011) found the
characteristics with the
highest correlation to transfer
were cognitive ability
followed by self-efficacy
(Chiaburu & Lindsay, 2008;
Grossman & Salas, 2011)
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 119

Table 15, continued
Assumed Motivation
Influence
Principle and Citation Context-Specific
Recommendation
Value for
implementing impact-
level training
evaluation. (Aaberg &
Thompson, 2012;
Eccles, 2006;
Kirkpatrick &
Kirkpatrick, 2016;
Rueda, 2011).
Expectancy-value theory
(Eccles, 2006; Rueda, 2011; Wigfield
& Cambria, 2010)
Based on the utility value of work or
learning. Activating personal interest
by providing choice and control.
Strong correlation to higher
expectations and perceptions of
confidence. Belief is the driver
(Wigfield & Cambria, 2010). Utility
as intrinsic or extrinsic motivator.
Also, Pekrun (2006) talked about the
emotional value.
Support the continued
recognition of the
personal, professional, and
organizational value of
impact-level evaluation,
and acknowledge the
costs. Assess and follow up
on the identification of
leading indicators, the
schedule of repeating
measurements of the leading
indicators, and with sharing
results with peer training
directors. Demonstrate
choice and success, model
and provide feedback of the
process of identifying and
measuring leading indicators.
Assess the training director’s
expectancy for success and
utility.

Identify the resources training directors feel are needed to implement the
identification and measurement of leading indicators. Interview results indicated that training
directors were confident of their own ability to achieve the stakeholder goal. However, the
directors’ self-efficacy was moderated by uncertainty based on cultural insecurity over resources
needed to conduct impact-level evaluation. Self-efficacy is the motivational influence of one’s
judgment of one’s own ability to achieve a goal (Bandura, 1994). The second principle that
applies to this intersectionality was identified by Lewis (2011) who recommends that to increase
the effectiveness of change, ensure needed resources are made available.
The recommendation for this influence is to identify and provide the manpower, time and
training that training directors feel are needed to institute impact-level evaluation, and enhance
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 120

the level of self-efficacy they feel. Grossman and Salas (2011) conducted a study to assess
correlations among 16 individual characteristics related to the success of transfer of technical
training and found that self-efficacy rated as the second highest correlated trait, just behind
cognitive ability. It is critical to support the provision of resources to achieve change, and nearly
as essential that change leaders feel confident of their ability to make changes.
Support the continued recognition of the personal, professional, and organizational
value of impact-level evaluation, and acknowledge the costs. Training director data indicated
they each recognized value in implementing organizational-level training evaluation for their
unit as well as for the USAF organization as a whole. The four elements of expectancy-value
include attribution, intrinsic pleasure, utility, and cost. Employees or learners who value
accomplishments and recognize the utility of either work or learning are more likely to activate
both personal interest and persistence, particularly to achieve complex learning demands
(Wigfield & Cambria, 2010). The recommendation to support this influence is to share the
initial results of the first leading indicators identified as a training objective and demonstrate the
value of the results to strategic planners within the USAF.
Choice, persistence, and mental effort are the three elements that Clark and Estes (2008)
identified as affecting motivation. The positive impact of attainment, the achievement of
intrinsic or extrinsic gains, the judgment of usefulness or the balance of cost in time or effort will
enhance a learner’s choice and persistence (Eccles, 2006; Rueda, 2011). Grossman and Salas
(2011) identified the expectation of use of learning content as the fourth highest indicator of
effective and efficient transfer of learning. To activate choice, persistence, and the mental effort
to learn and employ a new process, training directors need to see direct evidence of the utility
and value of strategic assessment.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 121

Organization Recommendations
The USAF is structured for effectiveness based on uniformity and the ability to function
using written guidance when time critical communication with leadership may be impaired or
impossible. This efficient approach in wartime, however, limits agility and adaptability even if
the need for change is widely recognized. The Kirkpatrick New World Model (Kirkpatrick &
Kirkpatrick, 2016) identifies the importance of Level 4 evaluation to determine impact-level, but
the USAF will need to codify the processes to identify and measure leading indicators. Further,
the MAJCOMs who benefit strategically from training evaluation data based on leading
indicators need to be convinced of the value of those data (Kirkpatrick & Kirkpatrick, 2016).
Another necessary change within the organizational culture of the USAF is in the perception that
training is less important for the organization than operational commands. A perceptual shift is
possible when operational commands are provided direct proof of the value that professional
training directly contributes to operational success. The two cultural artifacts needed to support
this innovation and the identified stakeholder and organizational goals are (a) written guidance to
articulate processes to identify, measure and analyze leading indicators, and (b) the identification
and assignment of resources, such as human resources and time to implement impact-level
training evaluation.
Table 16 provides a list of assumed organizational influences. The influences are in the
order in which they will be addressed to achieve the organizational goal, and each was found to
be highly probable because of interviews with the stakeholders, who can take direct action to
affect change in training evaluation. The recommendations derived from current research and
USAF practices include capturing and promulgating both broad policy and specific processes for
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 122

both up and down the chain of command and communicating and advertising the progress and
benefits of organizational-level training evaluation implementation.
Table 16
Summary of Organization Influences and Recommendations
Assumed Organization Influence Principle and
Citation
Context-Specific
Recommendation
CM1:
Organizational openness to adopting
processes that identify leading indicators of
impact-level training effectiveness. (Clark
& Estes, 2008; Kirkpatrick & Kirkpatrick,
2016).
Effective
organizations establish
organizational policies
and procedures to
support organizational
goals and values
(Clark & Estes, 2008).
The 82 TRW will direct
the development of
processes to measure
training results that
correlate between Career
Field Managers data of
interest and use and
measured leading
indicators that correlate
to the training provided
by the wing.
CM2: Organizational acceptance of the
value of leading indicators as a strategic
planning tool (Aaberg & Thompson, 2012;
Brewer, 2007; Bunch, 2007; Clark & Estes,
2008; Foster & Fletcher, 2013; Grossman &
Salas, 2011; Guerci & Vinante, 2011;
Harrington et al., 2017; Kennedy, Chyung,
Winiecki, & Brinkerhoff, 2013; Kirkpatrick
& Kirkpatrick, 2016; Palmer, 2008;
Phillips, 2012).
Effective change
efforts ensure key
stakeholders’
perspectives inform
the design and
decision-making
processes of the
change.

Organizational
effectiveness increases
when leaders identify,
articulate, focus the
organization’s efforts
and lead from the
“Why” (Knowles,
1980; Wigfield &
Eccles, 2000).
The wing will train,
support, and advertise
the use and value of
leading indicators as a
strategic tool of benefit
to MAJCOMs.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 123

Table 16, continued
Assumed Organization
Influence
Principle and Citation Context-Specific
Recommendation
CM 3: Organizational value for
education and training missions
as highly as operational missions
(Agocs, 1997; Aguinis &
Kraiger, 2009; Chapman, 2004;
Clark & Estes, 2008; Galanis &
Sottilare, 2013; Grohmann &
Kauffeld, 2013; Kodwani, 2017;
Preston, 2010; Vilchez, Darnall,
& Aragon Correa, 2016).
Organizational effectiveness
increases when leaders
continuously build
relationships: employees
need to feel that they matter
and their efforts make a
difference to organizational
outcomes (Buckingham &
Coffman, 1999).
Continued organizational
support for innovation that
improves the design and
delivery of training to
MAJCOMs with
operational missions. The
wing will provide the
MAJCOMs the training
evaluation evidence of the
return on expectations from
the professional and
standardized training
offered within AETC and
the 82 TRW.
CS 1: Organizational provision
of explicit guidance to identify,
correlate and analyze training
objectives and leading indicators
for training evaluation to
determine the impact of training
to external stakeholders (AETC,
2015; Clark & Estes, 2008;
USAF, 2010).
Effective leaders use
effective communication
during times of
organizational change
(Denning, 2005; Hattaway &
Henson, 2013; Lewis, 2011).
Effective change efforts
ensure everyone has
resources aligned with
priorities (Clark & Estes,
2008)
Provide training directors with
explicit guidance on the
processes to conduct impact-
level evaluation. The 82 TRW
will design and conduct
specific guidance and training
to design and implement
strategic-level evaluation and
ensure that analysis results
directly address whether
training adds and continues to
add value to the MAJCOMs
mission accomplishment.
CS 2: Organizational provision
of the necessary resources to fully
conduct impact-level training
evaluation (Clark & Estes, 2008;
Harrison, 2016).
Effective change ensures the
right resources are in place
and aligned with needs and
organizational priorities
(Clark & Estes, 2008)
Identify and provide base-line
resources to implement the
identification and correlation
of leading indicators and
training objectives. The 82
TRW will provide trained
personnel to each squadron to
implement and conduct
strategic-level training
evaluation and provide results
that the 82 TRW can, in turn,
provide the MAJCOM.

Continued organizational support for innovation that improves the design and
delivery of training to MAJCOMs with operational missions. The 82 TRW needs policies
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 124

that identify and use leading indicators specific to stakeholders throughout supported MAJCOMs
to prove that effective training directly correlates and contributes to the strategic goals of those
commands. Senge (1990) identified characteristics needed to build a learning organization:
systems thinking for intersections, interrelationships and trends, communication between
stakeholders at varying levels, and the need to build a shared vision that gets beyond blame or
adversarial positioning. Effective organizations establish change policies to support
organizational goals and values (Clark & Estes, 2008; Lewis, 2011). The first recommendation
is to support the adoption of leading indicators to measure the impact of training conducted by
the 82 TRS. The second recommendation is to communicate the results of impact-level training
evaluation to the MAJCOMs as proof of the influence that changes in training have on metrics of
MAJCOM interest. The third recommendation is to leverage the ability to prove there is direct
cause and effect from training objectives to MAJCOM metrics to improved MAJCOM mission
capabilities so that the mission of training is as highly valued as other operational missions.
Salas et al. (2012) identified that one critical purpose of training evaluation is to tell an
organization what training to keep and what to improve or cancel. Kodwani (2017) studied the
training evaluation results for public sector employees (n=123) in India for factors that
influenced outcomes and found that Level 1 and 2 results are perceived to be within the control
of individual students while Levels 3 and 4 results are under organizational control and are
proportional to how processes are defined and communicated. Additionally, Hill (1999) found
that training evaluation practices increased when written policies and procedures existed. Well-
planned and communicated practices support the adoption and value of training observations and
measurements and produce long-term and reliable results that correct the course of training and
increase the value of training to the organization as well as the individual.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 125

Support and promote the use of leading indicator measurements as a strategically
significant planning tool. The 82 TRW also needs to support the use and value of leading
indicators as a strategic planning tool. Effective change must be based on stakeholder needs as
well as organizational capabilities, and the effectiveness and sustainability of change increase
when leaders identify and articulate expectations, processes, and the reason for the change (Clark
& Estes, 2008; Knowles, 1980; Wigfield & Eccles, 2000). Lewis (2011) also identified that, to
achieve organizational change, stakeholders must assert the power to control processes,
demonstrate legitimacy and hold the moral high ground, and establish urgency of the need for
change. Visible support for the implementation and uses of leading indicators of training
evaluation by the wing will improve the probability of successfully establishing strategic-level
evaluation results.
Communicate valid and reliable evidence of the effectiveness of training to
MAJCOMs. The MAJCOMs of the USAF must value the education and training provided by
the AETC as highly as the contributions of any operational command. Organizational
effectiveness increases when leaders work to build relationships and recognize that there is
intersectionality in different groups and the needs of stakeholders within an organization (Clark
& Estes, 2008). The wing will provide evidence of correlations between training content and
metrics valued by supported MAJCOMs on a regular and recurring basis to maintain an accurate
understanding of the value of AETC training.
Provide training directors with explicit guidance on the processes to conduct
impact-level evaluation. Standardized practices support squadron ability to identify, collect and
analyze specific data correlated to leading indicators used by the supported commands and
functional managers. Effective leadership maintains communication during times of change
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 126

(Denning, 2005; Hattaway & Henson, 2013; Lewis, 2011). Additionally, effective change
initiatives maintain alignment with priorities, resources, and the end goal (Clark & Estes, 2008).
The 82 TRW will design and conduct training for squadrons to provide the policies, procedures,
and organizational goal to communicate a uniform understanding of the reason for, and
expectations from, change practices.
Tanner and Otto (2016) found that, within an organization, communication to achieve
unified understanding of change intentions and goals, the utility of intended change, and intra-
and inter-organizational validity of training are significant to the achievement of lasting change.
Planned, repeated and consistent discussions of the purpose, value, and practices of
implementing strategic-level evaluation will minimize resistance to change among squadrons by
ensuring a common understanding of the larger organizational goal and specific squadron
contributions.
Identify and provide base-line resources to implement the identification and
correlation of leading indicators and training objectives. Training directors were confident of
their own ability to achieve the stakeholder goal, but were unsure of certain success due to
intersectionality with an organizational insecurity over the need or requirements for resources
needed to conduct impact-level evaluation. The wing will provide resources to thoroughly
conduct strategic-level training evaluation. Effective change ensures that the right resources are
in place at the right times and align with organizational needs and priorities (Clark & Estes,
2008). The 82 TRW will assign trained personnel to provide squadron training and a timeline to
prepare them to implement and monitor strategic-level evaluation of training metrics correlated
to supported commands leading indicators. Additionally, the wing will offer the functional
managers of supported commands training on the effective use and value of leading indicators
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 127

for strategic planning. Finally, the wing will ensure training and access to sources of MAJCOM
metrics to identify those leading indicators that may correlate with training outcomes.
In a study of training evaluation conducted within the federal government, Hypes (2016)
identified eight barriers to successful training evaluation, five of which apply directly to the
resourcing needed to achieve strategic-level training evaluation: time, training, funding, human
resources, and technology. Two of the remaining three barriers occur because of constrained
resources due to poor managerial support and low employee interest, while only the eighth
barrier is unrelated to resources: the fear of published results.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 128

Appendix G: Integrated Implementation and Evaluation Plan
The effectiveness of training measures organizational progress against expected results of
that training (Kirkpatrick & Kirkpatrick, 2016). Designed first to address the organizational
value of training, Level 4 of the Kirkpatrick New World Model ensures that the design and
implementation of training will result in an accurate assessment of the value of that training to
the organization (Kirkpatrick & Kirkpatrick, 2016). Level 3 is then achieved through the
measurement of desired behaviors in working environments, accompanied by reinforcements,
encouragement, and rewards for success. Level 2 goes beyond student acquisition of knowledge
and skills and includes assessment of student self-efficacy and motivation while Level 1
measures student reaction to training, and questions student engagement, the perception of
training relevance and satisfaction with training quality (Kirkpatrick & Kirkpatrick, 2016).
Implementation and Evaluation Framework
The 82 TRW has long measured the equivalents of Kirkpatrick’s Levels 1, 2 and 3 to
assess students in technical training. Evaluation of training influences every step in the analysis,
design, development, implementation, and evaluation model directed for use by the AETC in
production and delivery of every formal course that has been taught since the early 1960s.
However, there has never been an equivalent to Kirkpatrick’s Level 4 evaluation practice that
asks whether training delivered contributed, failed to contribute, or continues to contribute to the
successes or failures of the organizations that students report to at the end of training.
Organizational Purpose, Need, and Expectations
Within the USAF, the AETC is one of 10 MAJCOMs. The mission of AETC is to
provide unique and critical training needed by the USAF to fly, fight, and win. Every airman,
military or civilian in the USAF is a product of AETC training, and AETC is unique among all
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 129

the MAJCOMs in that it strives to meet the needs of every MAJCOM, a broad and challenging
mandate. There is a need to provide the USAF and its MAJCOM leadership evidence that
training unequivocally contributes to the mission success of every other command.
The purpose of this innovative study is to identify leading indicators and the training
results that correlate to those indicators to derive strategic-level evidence of the effectiveness of
training to the leadership of supported commands. The lack of impact-level training evaluation
is a current problem of practice as AETC and the 82 TRW currently provide only anecdotal
evidence that is ineffective to support needed resource acquisitions or overcome institutional
resistance to implementation of training improvements. With Level 4 training evaluation in
place, the leadership of MAJCOMs will receive on-going proof that training directly affects each
command’s mission accomplishment rate, as well as the overall achievement of the USAF goal
to fly, fight and win.
The innovation recommended by this study is to implement Level 4 measurement at the
development of every training course designed and taught by the 82 TRW. This will entail
educating and training every level of leadership within the wing to ensure that knowledge,
motivation, and organizational practices support implementation and that results are
communicated to functional leaders within each MAJCOM. Specific recommendations to
address knowledge, motivation and organizational gaps were developed based on the results of
semi-structured interviews with the stakeholder group on focus who are most likely to
successfully implement Level 4 evaluation practices for the 82 TRW.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 130

Level 4: Results and Leading Indicators
The goal of strategic-level training evaluation is to prove to the MAJCOMs that the
training they defined and requested is the training delivered and is training that directly improves
the functions that generate their reporting metrics. Both external and internal outcomes
contribute to the achievement of this goal, as coordination and communication with other
stakeholders are required. Table 18 shows two external and four internal outcomes expected
from the successful implementation of the recommendations from this study.
External outcomes. The mission of the USAF is to fly, fight and win. The successful
implementation of the innovation examined in this study will support that at both the training
squadron and MAJCOM levels without creating additional layers of bureaucracy and
complicated process delays. The two external outcomes expected from implementation of
recommendations from this study are improved communications concerning training needs and
outcomes by providing data meaningful to MAJCOM end-users and improved end-user
perceptions of value and effectiveness of AETC and the training groups within the 82 TRW.
Internal outcomes. Training directors within every squadron but one has only limited
knowledge of strategic-level evaluation. Before achieving external outcomes, internal wing
policy and specific processes need to be modified to ensure standardization of how squadrons
will collect and use leading indicators to revise the curriculum. Internal Level 4 outcomes will
result from effectively preparing squadrons to collaborate with their supported commands,
matching leading indicators to correlated training objectives and establishing the means to
monitor the results of curriculum revisions on leading indicators regularly. Table 17 also
provides the metrics and methods to achieve Level 4 results.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 131

Table 17
Metrics and Methods for External and Internal Outcomes
Outcome Metric(s) Method(s)
External Outcomes
Improved MAJCOM
leadership perceptions of
AETC/82 TRW training.
Periodic measurement of the
effects of training revisions on
MAJCOM metrics: MAJCOM
satisfaction with leading
indicators, training outcomes,
new LI, etc.
Regularly scheduled
and reviewed post-
implementation Level 4
survey Appendix A
Increased communications
between MAJCOMs and
the training managers
responsible for training
analysis, development, and
delivery.
Number of courses reviewed
and revised based on leading
indicator deltas
Training manager
progress reporting at
weekly squadron staff
meetings and results of
Appendix A
Internal Outcomes
Increased policy and
process guidance to
conduct Level 4
evaluation.
Written policy and uniform
process identifying who is
responsible for finding
correlations between training
and leading indicators
Periodic meetings at the
squadron, group, or
wing level to
communicate findings
and progress and results
of Appendix A
Increased knowledge
among squadron-level
training personnel of Level
4 evaluation.
Compliance with policy and
processes directed in guidance
Periodic meetings at the
squadron, group, or
wing level to
communicate findings
and progress and results
of Appendix A

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 132

Table 17, continued
Outcome Metric(s) Method(s)
Internal Outcomes
Increased number of
identified correlations
between Leading Indicators
and training outcomes.
Number of correlations to
Leading Indicators and
number of courses reviewed
and revised based on analysis
of Leading Indicators
Periodic meetings at the
squadron, group, or wing
level to communicate
findings and progress and
results of Appendix A
Decreased time to review
and revise training.
Number of weeks between
review and revision and
number of weeks from
revision and impact on
Leading Indicators
Pre- and regularly
scheduled and reviewed
post-implementation
surveys and results of
Appendix A

Level 3: Behavior
Critical behaviors. Creating a vision of change and planning for it are functions
predicated on perfect circumstances in an ideal environment. Conversely, every step taken to
effect change reveals unforeseen challenges. What makes success possible is each gain was
planned for and monitored, every requirement exists for a reason, and every obstacle is met with
a prepared and effective counter-action.
The critical behaviors identified in Table 19 detail actions training directors will take to
demonstrate the implementation of strategic-level evaluation practices successfully. The table
includes the actions, measures, and methods of those actions, and the recommended timing for
the measurements. These behaviors include communicating with Functional Managers (FMs)
who are most concerned with the connection between desired training and training outcomes.
This communication is necessary to identify leading indicators valued by the MAJCOM to
correlate to specific training objectives, and, in turn, drive curriculum revisions. The
communication loop that began with FMs will then be completed when changes to the
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 133

curriculum are shown to improve training outcomes evident in the change in MAJCOM metrics.
Another critical behavior is identifying meaningful correlations between leading indicators and
training objectives. Table 18 shows the critical behaviors and what, how, and when these
behaviors will be measured.
Table 18
Critical Behaviors, Metrics, Methods, and Timing for Evaluation
Critical Behavior Metric(s) Method(s) Timing
1. Research,
communicate and
identify with
Functional Managers
(FM) at the MAJCOM,
leading indicators as
metrics of interest
Number of leading
indicators identified for
each MAJCOM
1. Report by squadron
evaluators of
implementation
progress and lessons
learned and results of
Appendix B
Monthly
2. Connect and correlate
specific training
objectives with a
probable impact on
leading indicators
Number of training
objectives correlated to
leading indicators
2. Report by squadron
evaluators of
implementation
progress and lessons
learned and results of
Appendix B
Monthly
3. Revise training
objectives, methods, or
delivery in curriculum
Number of
courses/objectives
revised to meet specific
leading indicators
3. Report by squadron
evaluators of
implementation
progress and lessons
learned and results of
Appendix B
Monthly
4. Monitor leading
indicators
The weekly metrics
each MAJCOM tracks
(safety, qualifications,
fully mission capable
aircraft)
4. Shared MAJCOM
Statement of Health
(SoH) of the Fleet
briefing and results of
Appendix B
Monthly

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 134

Table 18, continued
5. Communicate with
FMs at MAJCOMs to
reinforce the
connections between LI
metrics and training
delivered
The deltas from leading
indicators, the
correlation to training
objectives and the
actions taken that made
those changes possible
5. Report by squadron
evaluators of
implementation
progress and lessons
learned shared with
FMs and results of
Appendix B
Quarterly

Required drivers. Even with a strong motivation to adopt and apply strategic-level
evaluation practices, training directors will benefit from drivers designed to reinforce, encourage,
reward, and monitor their use of the knowledge and skills acquired to support this innovation.
Kirkpatrick and Kirkpatrick (2016) refer to these as required drivers, essential in supporting
behavioral changes. Table 19 identifies the specific drivers, how often each should be used and
which critical behavior or behaviors it supports.
Table 19
Required Drivers to Support Critical Behaviors
Method(s) Timing Critical Behaviors
Supported
1, 2, 3 Etc.
Reinforcing
Checklist that supports the communication
with supported units or commands and
identification of leading indicators used by
the MAJCOM
On-going 1,5
Progress reporting among peers for leading
indicators identified, and subsequent
correlation to training objectives in the
existing or planned curriculum
Monthly 2

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 135

Table 19, continued
Method(s) Timing Critical Behaviors
Supported
1, 2, 3 Etc.
Encouraging
Documented use of leading indicator as a
driver of curriculum revisions and the
effect on subsequent measurements of that
leading indicator
Monthly 1, 2, 3, 4, 5
Feedback from squadron and group peers Monthly 2, 3, 4
Rewarding
Recognition from the training group and
wing for successes in identifying and
correlating leading indicators and learning
objectives to monitor and act upon
Quarterly 2, 3, 4

Monitoring
Required reporting of the challenges and
successes of identifying, regularly
measuring, and correlating leading
indicators to training objectives
Monthly 2, 3, 4

Organizational support. Fewer than 50% of organizational change efforts succeed, and,
while some may succeed, their success is often only partial (Kotter, 2007; Strebel, 1996). One of
the common reasons that organizations fail in part or in whole is when change is decreed and but
follow up with those who implement it is neglected (Sirkin, Keenan, & Jackson, 2005).
Kirkpatrick and Kirkpatrick (2016) contend that change has a better chance of succeeding when
it is scaffolded by regular peer support, status checks, recognition for progress and an awareness
that the organization continues to care. The continuation of interest in the process by
organizational leadership is a change more likely to become integrated into the organization’s
culture.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 136

Reinforcement of this innovative change by the 82 TRW include the use of a checklist of
steps useful to identify leading indicators and the learning objectives that correlate directly to
them. Support will also be generated by members in every squadron assigned the responsibility
of identifying leading indicators, as strong correlations can be shared and learned from during
monthly meetings. Finally, feedback on the utility and effectiveness of the results to the
MAJCOMs will reinforce the value and the conduct of impact-level evaluation.
Encouragement has an amplified effect when effort pays off in positive results. When
leading indicators are successfully correlated to teaching objectives and the curriculum is revised
to improve student outcomes of those learning objectives, there should ultimately be positive
changes in the leading indicators closely monitored by the FMs at the MAJCOM. Some
curriculum revisions will challenge organizational patience as results will not be seen nor be
measurable for many months for longer training pipelines. However, it would be feasible to
insert data collection points throughout longer pipelines to measure gains or losses in knowledge
and skill retention. Such a solution may also provide training directors valuable data for more
than strategic-level assessment, as knowledge retention and decay of technical skills over long
periods are considered a pernicious problem (Aaberg & Thompson, 2012; Bell & Reigeleuth,
2014).
There are a variety of rewards used by the USAF to recognize employees whose efforts
produce positive results. The rewards range from personal awards that may include financial
merit awards or time-off, recognition at command-wide events, higher annual appraisal ratings
and possibly enhanced competitiveness for promotion. Additionally, the USAF cultivates a
culture of intrinsic reward that values innovation and a mastery mindset.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 137

Level 2: Learning
Learning goals. Upon completion of the training recommended, the training directors
will:
1. Identify the elements of strategic-level evaluation and what defines a leading indicator
(D)
2. Identify leading indicators that may correlate to technical training learning objectives (D)
3. Monitor the effects of training objective outcomes on leading indicators (P)
4. Monitor leading indicators for indications of a need to examine and revise training
objectives (P)
5. Reflect on their ability to implement and improve the process of correlating and using
data from leading indicators to improve training (K)
6. Value the use of leading indicators and training objective correlations to generate data-
driven curriculum revisions (M)
7. Demonstrate the confidence to identify leading indicators and correlated training objects,
and use them to identify needed revisions in training (M)
8. Communicate with peer training directors, the training group and wing, and MAJCOM
FMs when changes to curriculum affects correlated objectives (CS)
Program
Training sessions or seminars are familiar territories for USAF and Department of the Air
Force employees, and the design used for training is an effective one that presents declarative
knowledge, followed by conceptual and procedural knowledge. The performance objectives are
then reiterated and students participate in guided practice. The completion of training requires
every graduate to demonstrate a degree of proficiency in using what was learned. This design
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 138

provides the model for the training needed to successfully implement the innovation
recommended in this study and identified by the results of the interviews that revealed a shortfall
of knowledge but a high degree of motivation to learn and implement impact-level evaluation.
Evaluation of the components of learning. In the environment of technical training, the
design for learning begins with questions: What should the student be able to do at the end of
training? What knowledge is needed to acquire a skill? What degree of proficiency is desired?
How should the student be assessed? The second, third and fourth questions close a loop in
learning with the first question, as the students’ final performance is a summative assessment of
outcomes that logically follows formative assessments of the enabling components of learning.
In conjunction with a student’s acquisition of knowledge and skills, their surety and self-
confidence, and the value they hold for the importance of what they demonstrated they can do all
enhance the likelihood that what was gained will be used (Daly, 2006; Mayer, 2011). Table 20
shows the specific components of learning to measure throughout the program, along with when
each component is measured.
Table 20
Evaluation of the Components of Learning for the Program.
Method(s) or Activity(ies) Timing
Declarative Knowledge “I know it.”
Training directors identify the levels of
evaluation found in the Kirkpatrick New World
Model, in facilitated discussions.
Before, during and at the end of each
facilitated discussion during the
seminar.
Training directors describe the elements of each
level of evaluation used by the Kirkpatrick New
World Model
Before, during and at the end of each
facilitated discussion during the
seminar.
Training directors design a plan for their group
or squadron training for how to identify leading
indicators and correlated training objectives.
At the end of the training seminar and
assessed quarterly throughout the next
12 months.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 139

Table 20, continued
Method(s) or Activity(ies) Timing
Procedural Skills “I can do it right now.”
Training Directors apply the Kirkpatrick New
World Model and identify a Level 4 Leading
Indicator to be measured and how often.
During and at the end of the training
seminar, and to be used in
demonstrations as they carry the
training back into their own group and
squadron.

Attitude “I believe this is worthwhile.”
Training directors participate in facilitated
discussions on the utility and value of impact-
level training evaluation.
Assessed throughout the seminar by
the degree of participation
demonstrated
Training directors identify what they found of
value and what they did not find of value in the
material presented in the seminar.
Assessed at the end of the seminar by
an End of Course survey

Confidence “I think I can do it on the job.”
Training directors reflect on the knowledge and
skills they acquire, and the use to which each
can be put in support of the organizational goal
to implement impact-level evaluation, and their
degree of confidence.
Throughout and after the seminar, as
they pass on the training to their
groups and squadrons. Training
director feedback will be collected
when their seminar students are
assessed.

Commitment “I will do it on the job.”
Training directors demonstrate the correlation of
a training objective to a leading indicator along
with their plan to regularly measure and analyze
the results and communicate the degree and
direction of impact on the leading indicator.
Post-training, at quarterly wing
meetings for all seminar attendees to
communicate and share the results of
identifying leading indicators and the
training objectives that correlate to
them.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 140

Level 1: Reaction
The stakeholder members for whom the program was designed are experienced trainers
themselves and senior leaders. A facilitated seminar was selected as the most effective method
to deliver requisite knowledge and performance elements, encourage engagement, and maximize
participation. Table 21 describes how and when trainee reactions are measured and timing of
assessments of training directors’ reactions to the program. Engagement is one indicator that
signals interest and indicates the student identified utility in the learning. Relevance focuses
general interest in the subject matter by a student for a specific application or use. Customer
satisfaction is a measure of confidence a student feels that the training delivered as promised and
that, at the end of training, they have the tools they need to apply what they learned.
Table 21
Components to Measure Reactions to the Program
Method(s) or Tool(s) Timing
Engagement
Observation by seminar facilitator On-going
Follow up discussions with group and squadron
evaluations specialists
On-going
End of Course Survey At the end of the seminar
Relevance
Facilitated discussions in class and End of Course
Survey at the end of the seminar
Start of the seminar, end of a
topic, and end of the seminar
Introductory discussion for each segment of the
seminar that presents what will be learned and how it
will be used after the end of training
On-going
Customer Satisfaction
End of Course Survey At the end of the seminar

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 141

Evaluation Tools
In alignment with the Kirkpatrick New World Model (Kirkpatrick & Kirkpatrick, 2016),
the following evaluation tools for the program were designed with the end in mind. The intent
was to present the final, or Level 4, assessment of the overall initiative first for the training
directors to consider. The tools to assess Levels 3, 2 and 1 then follow to keep in sight that the
program’s purpose was to identify leading indicators. Achieving Level 4 evaluation, while the
most distant from program completion, is dependent on the other levels, but impact-level
evaluation is designed first and must not be lost or set aside because it is not immediate
(Kirkpatrick & Kirkpatrick, 2016). Throughout the training wing, when the other training
evaluation Level 1 and 2 results are strongly positive, “good enough” results diminish the
urgency of any more comprehensive evaluation process. This section was intended for
implementation by senior training directors and designed to emphasize that all roads lead to
impact-level evaluation as well as to prevent successful Level 1, 2 and 3 results from eroding the
momentum needed to achieve the stakeholder goal.
Level 4 evaluation of the program. Donald Kirkpatrick (1994) stated that, for training
evaluation to be useful, all four levels of evaluation must be measured. Typically, Levels 1 and 2
are addressed first, as they assess immediate student responses to training, followed by Level 3
and then 4. However, as this innovation recommends implementing impact-level evaluation to
experienced training professionals, it began with the critical but delayed Level 4 tool. Appendix
A is intended for use by the training directors for a period between 6 to 8 weeks following
internal training using the program and initial implementation of impact-level evaluation within
their units. In addition to measuring the effectiveness of the program, it continues to be useful at
intervals after the program to continuously measure progress toward accomplishment of the
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 142

stakeholder goal and encourage the training wing to move on to the organizational goal.
Appendix A includes six questions about a unit’s accomplishment of the internal outcomes and
two external outcomes as well as to confirm the value of the impact-level evaluation innovation.
Finally, it provides an opportunity for training directors to include additional feedback and
comments on the progress of impact-level evaluation practices and needs and supports the
continuous process improvement data inherent in the model.
Level 3 evaluation of the program. Appendix B was designed to assess the transfer of
what was learned during the program and subsequently use in the working environments of the
squadrons. Like the Level 4 evaluation tool, it is delayed until after the end of the program and
then requested and completed approximately 3 to 6 weeks after seminar completion. It was
designed to determine the progress of impact-level evaluation implementation and how
knowledge and skills from the program were utilized by ascertaining if graduates experienced
encouragement, reinforcement, monitoring and rewards for correctly using what was learned.
The results of Appendix B are instrumental in determining if program delivery was effective and
efficient as well as the degree of application of student learning from the program.
Immediately after the training: Level 2 and Level 1 evaluation of the program.
Attached as Appendix C is the end of training survey intended to assess student acquisition of
knowledge and skill (Level 2) and student reaction to the training process itself (Level 1). Of
interest from the surveys was feedback on student engagement, a perception of relevance and
overall satisfaction with the training. Salas et al. (2012) asserted that effective training is
training that gets used. Therefore, considered assessment of students’ reaction to training may
indicate their intent to transfer what they learned. The training designed for the program was
exclusively intended for senior trainers, and it was, therefore, deemed most efficient to combine
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 143

Level 2 with Level 1 responses. The consolidated responses from the training directors directly
affect revisions to program content and delivery for use in training throughout each squadron and
group.
Data Analysis and Reporting
Data collection throughout the 82 TRW is a full-time responsibility for nearly every
instructional specialist employed. As several training directors observed in the interviews, a
massive amount of data is collected and reported at every level but never used to change or
improve any processes. The data collected from the use of Appendices A, B and C is used to
track the implementation and continuation of the program. At the wing level, the metric of
interest is the impact of training on leading indicators. This is followed by consolidated data
analysis. Reporting from the data based on the evaluation tools is simple and understood at a
glance using Appendix D. The training directors are the stakeholders with vested interest in the
success of the program and familiarity with the meaning of the data being reported, so there is no
value for more complex reporting at a higher level within the command. The primary analysis
reported at the wing level will be condensed to simply indicate the degree to which, at each level
of evaluation, each squadron and group accomplished, made progress toward, or failed to
accomplish the enabling goals necessary to achieve the stakeholder goal. Once the stakeholder
goal of identifying and correlating leading indicators is accomplished, the same process and tools
are applicable and available as the 82 TRW works toward the 2022 organizational goal of fully
implementing impact-level evaluation throughout the wing.

82nd TRAINING WING TECHNICAL TRAINING EVALUATION 144

Summary
Implementation of training evaluation based on the Kirkpatrick New World Model
(Kirkpatrick & Kirkpatrick, 2016) requires training organizations to plan from the height of
Level 4 evaluation and work downward through Level 3, then Level 2 and Level 1. The design
for implementing the recommendations from this program began with the end goal of Level 4
and the identification of leading indicators and worked down through Levels 3, 2 and 1.
Appendix H was developed to ensure that alignment of influence, finding and recommendation
was adhered to as the Program developed. The advantage of using the Kirkpatrick New World
Model (2016) was that it maintained the end goal of establishing impact-level training evaluation
from the beginning of the Program and prevented drift back to the previous practices of
evaluating only students and curriculum. It is the gradual deviation from the change plan that
derails many change evolutions (Bunch, 2007).
In using the Kirkpatrick New World Model (2016) to direct the implementation of the
stakeholder’s goal, training directors will gain first-hand knowledge and experience with the
same process that will direct accomplishment of the organizational goal. This will specifically
provide training directors with the knowledge and skills to close the gaps identified in the
findings. The design of training for this innovation also includes support for the assets the
training directors revealed in their metacognition and in their motivation to successfully
implement change that they value. Next, the training program supports the progressive
identification of resources that prove to be necessary, and an opportunity to share the resources
that already exist, to support this innovation. Finally, the training program targets the
organizational challenges that were found in this study, and seeks to improve communications
between the members of the 82 TRW and the MAJCOM FMs, to identify leading indicators and
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 145

correlated training, and provide the FMs with functional value. What makes the Kirkpatrick
New World Model (2016) so compelling is that it provides an efficient means to improve
training evaluation and the most effective method to improve training and training evaluation.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 146

Appendix H: Influence to Finding to Recommendation Matrix
Knowledge
Conceptual

(I) Knowledge of what impact-level evaluation and leading
indicators are, and what they can be used to measure.

(F) Training directors lacked conceptual knowledge of
impact-level training evaluation.

(R) Provide training that includes conceptual knowledge of
impact-level evaluation practices.
Procedural (I) Knowledge of procedures to identify, correlate, and
measure training objectives and leading indicators.

(F) Training directors lacked procedural knowledge of
impact-level training evaluation.

(R) Provide performance-based training that includes
procedures to identify leading indicators and correlate
them to training objectives.
Metacognitive

(I) Reflecting on capacity to design and implement impact-
level training evaluation.

(F) Training directors demonstrated the ability to reflect on
the design and conduct of training and implementation for
impact-level training evaluation.

(R) Continually support training directors’ metacognition.
Motivation
Self-efficacy Theory

(I) Efficacy about being able to identify, correlate and use
leading indicators for impact-level training evaluation.

(F) Training directors’ belief in their ability to lead the
identification and correlation of leading indicators was
influenced by a lack of knowledge and intersectionality
with organizational culture.

(R) Identify the resources training directors feel are
needed to implement the identification and measurement
of leading indicators.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 147

Expectancy-Value Theory

(I) Value for implementing impact-level training
evaluation.

(F) Training directors recognized value in implementing
impact-level training evaluation.

(R) Support the continued recognition of the personal,
professional, and organizational value of impact-level
evaluation, and acknowledge the costs.
Organization
Cultural Model Influence 1 (I) Organizational openness to adopting processes
that identify leading indicators of impact-level
training effectiveness.

(F) The 82 TRW demonstrated a capacity for cultural
change.

(R) Continued organizational support for innovation
that improves the design and delivery of training to
MAJCOMs with operational missions.
Cultural Model Influence 2 (I) Organizational acceptance of the value of leading
indicators as a strategic planning tool.

(F) There is a cultural belief within the 82 TRW that the
MAJCOMs do not understand how training is designed
or delivered.

(R) Support and promote the use of leading indicator
measurements as a strategically significant planning
tool.
Cultural Model Influence 3 (I) Organizational value for education and training
missions as highly as operational missions.

(F)There is a cultural belief within the 82 TRW that the
status of 82 TRW training among the MAJCOMs is
limited by a devaluation of training compared to the
value of operational missions.

(R) Communicate valid and reliable evidence of the
effectiveness of training to MAJCOMs.
82nd TRAINING WING TECHNICAL TRAINING EVALUATION 148

Cultural Setting Influence 1 (I) Organizational provision of explicit guidance to
identify, correlate and analyze training objectives and
leading indicators for training evaluation to determine
the impact of training to external stakeholders.

(F) The 82 TRW did not provide guidance for uniform
processes to conduct impact-level evaluation.

(R) Provide training directors with explicit guidance on
the processes to conduct impact-level evaluation
Cultural Setting Influence 2 (I) Organizational provision of the necessary resources
to fully conduct impact-level training evaluation.

(F) Training directors do not expect the organization to
be able to dedicate the necessary resources.

(R) Identify and provide base-line resources to
implement the identification and correlation of leading
indicators and training objectives.

Abstract (if available)

Abstract Organizations rarely conduct impact-level training evaluation

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Manager leadership skills in the context of a new business strategy initiative: an evaluative study

PDF

An evaluative study on implementing customer relationship management software through the perspective of first level managers

PDF

Anti-bias training in community colleges: an exploratory study

PDF

Developing physician trainees leadership skills: an innovation study

PDF

Preventing excessive force incidents by improving police training: an evaluation study of a use-of-force training program

PDF

The integration of medicine and compassionate care: an evaluation study

PDF

Pre-deployment training effectiveness for Army National Guard units mobilized to deploy in support of combat operations in Iraq and Afghanistan: an evaluation study

PDF

Development of intraorganizational post-merger collaboration plan: an evaluation study

PDF

Relationship between employee disengagement and employee performance among facilities employees in higher education: an evaluation study

PDF

Aligned leadership attributes and organizational innovation: an evaluation study

PDF

Adaptability characteristics: an evaluation study of a regional mortgage lender

PDF

The board fundraising challenge after nonprofit mergers: an evaluation study

PDF

The role of professional development and certification in technology worker turnover: An evaluation study

PDF

Customer satisfaction with information technology service quality in higher education: an evaluation study

PDF

Principals’ impact on the effective enactment of instructional coaching that promotes equity: an evaluation study

PDF

Structured leadership development in the judicial system to enhance public service: an executive dissertation evaluation study

PDF

The impact of faculty interactions on online student sense of belonging: an evaluation study

PDF

Preparing university faculty for distance education: an evaluation study

PDF

Improving the evaluation method of a military unit: a gap analysis

PDF

An application of Clark and Estes gap analysis model to improve training transfer at a midsize pharmaceutical company

Asset Metadata

Creator Bard, Shelley Louise (author)

Core Title 82nd Training Wing technical training evaluation: an innovation study

School Rossier School of Education

Degree Doctor of Education

Degree Program Organizational Change and Leadership (On Line)

Publication Date 03/21/2019

Defense Date 03/21/2019

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag Air Force training,leading indicators,Level 4 evaluation,military training,OAI-PMH Harvest,technical training evaluation,training evaluation

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Seli, Helena (committee chair), Kim, Samuel (committee member), Yates, Kenneth (committee member)

Creator Email shelley.bard@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c89-135640

Unique identifier UC11676901

Identifier etd-BardShelle-7172.pdf (filename),usctheses-c89-135640 (legacy record id)

Legacy Identifier etd-BardShelle-7172.pdf

Dmrecord 135640

Document Type Dissertation

Format application/pdf (imt)

Rights Bard, Shelley Louise

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA