University of Southern California Dissertations and Theses

Improving instructor skills (IIS): a Needs analysis

(USC Thesis Other)

Improving instructor skills (IIS): a Needs analysis

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content i Improving Instructor Skills (IIS): A Needs Analysis by David Hawkins A Dissertation Presented to the FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF EDUCATION December 2017 Copyright 2017 David Hawkins ii Abstract This dissertation evaluates Instructor’s practical knowledge in effectively delivering instruction in Federally Funded Research and Development Centers (FFRDC). Through semi- structured interviews, observation, and document analysis this investigation explored the skill sets of seven space technical instructors conducting highly technical instruction to the organization’s engineering staff in order to answer two research questions: What are the knowledge, motivation, and organizational elements that interfere with instructors achieving 100% of its goal of applying effective learning engagement strategies during technical training classes? What are the recommended knowledge, motivation, and organizational solutions? The purpose of this study was not to place blame on the instructor’s lack of adult learning knowledge, but rather to identify their needs so that the organization can provide them better support. It is not uncommon for Subject Matter Experts (SMEs) to be sought after for their technical expertise and without any formal instructional design training. Findings indicated that an instructor improvement program can provide instructors the knowledge, motivation, and organizational influences needed to keep their students engaged, to better manage their classrooms, and to increase knowledge transfer to their students. Implications for an instructor improvement program include the redevelopment of the current instructor workshop to include topics such as adult learning theory, student engagement strategies, and classroom management best practices. Moreover, to establish processes and procedures to ensure that the identified critical behaviors are reinforced, encouraged, and rewarded. Lastly, to implement an evaluation strategy to enable the organization to make adjustments as needed. With these improvements, the organization can meet the instructor’s needs to help them effectively delivering instruction in federally funded research and development centers. iii Keywords: Learning and Development; Training and Development; Trainer; Instructor, Teacher, Facilitator; Train The Trainer; Instructor Workshop; Adult Learning; Student Engagement; Knowledge Transfer; Classroom Management; NEEDS Analysis; GAP Analysis; Knowledge; Motivation; Organizational Influences; KMO iv Dedication This dissertation is proudly dedicated to… All of my Family To My Wife “Gabrielle”, my Son “Grant” Who have given me stamina and inspiration! To My mother “Rita”, my father “Gary” Who have given me strength and courage! - David - v Acknowledgements I started down the doctoral road well before the start of class back in January of 2015. It started with conversations with my dad on the front porch of our house in 2005. Well before the Marines and Iraq, well before marriage and children. I had just graduated with my Masters and was struggling with finding a job. I was considered over educated and under experienced. He didn’t necessarily want me to join the service and thought that while I was looking for a job it wouldn’t hurt to continue with my education and pursue my doctorate. After military service, after marriage and children, my father and I were back on the porch talking about the future and I told him, I still wanted to pursue my doctorate. He told me that, “Son, at some point you have to be happy with who you are and enjoy what you have”. At the time I didn’t know how to take that. Today, after sacrificing years to military service, and time dedicated to school, I now know what he means. Dad, I’m happy with who I am and I’m enjoying my family! I wish you were here to see my achievements, and to experience the joy of your grandson! I love you! I have been tested many different ways over the years and every time it feels like I’ve reached my threshold. Once again I’ve been tested though this process and I would not have been able to endure without the tough love and support from my wife and the inspiration from my son! Gabi, thank you for your patience and support though this program, Grant, I promise once this is done we will play more Halo! I love you! To my mom. I am so grateful that you have brought us home! Thank you for your vision and generosity for the future for our family! I love you! vi Table of Contents Contents Abstract ........................................................................................................................................... ii Keywords: ...................................................................................................................................... iii Dedication ...................................................................................................................................... iv Acknowledgements ......................................................................................................................... v Table of Contents ........................................................................................................................... vi List of Tables ................................................................................................................................. ix List of Figures ................................................................................................................................ ix CHAPTER 1: INTRODUCTION .................................................................................................. 1 Introduction to the Problem of Practice ...................................................................................... 1 Organizational Context and Mission ........................................................................................... 2 Organizational Performance Goal ............................................................................................... 2 Related Literature ........................................................................................................................ 3 Importance of the Evaluation ...................................................................................................... 3 Stakeholder of Study and Stakeholder Performance Goal .......................................................... 4 Purpose of the Project and Questions .......................................................................................... 6 Methodological Framework ........................................................................................................ 6 Definitions ................................................................................................................................... 7 Organization of the Study ........................................................................................................... 8 CHAPTER 2: REVIEW OF LITERATURE .................................................................................. 9 Introduction ................................................................................................................................. 9 Organizational Learning .............................................................................................................. 9 Adult Learning ......................................................................................................................... 9 Classroom Management ........................................................................................................ 10 Student Engagement .............................................................................................................. 11 Knowledge Transfer .............................................................................................................. 12 Learning and Motivation Theory .............................................................................................. 13 Knowledge and Skills................................................................................................................ 14 Knowledge Types .................................................................................................................. 15 Stakeholder Knowledge Influences ....................................................................................... 15 Declarative knowledge influences ......................................................................................... 16 vii Procedural knowledge influences .......................................................................................... 16 Metacognitive knowledge influences .................................................................................... 16 Motivation ................................................................................................................................. 18 Expectancy Value Theory ..................................................................................................... 18 Self-Efficacy Theory Defined ............................................................................................... 19 Instructors Assumed Motivational Influencers...................................................................... 19 Organizational Influences ......................................................................................................... 22 Assumed Cultural Influences................................................................................................. 22 Cultural Models ..................................................................................................................... 23 Cultural Settings .................................................................................................................... 23 Summary ................................................................................................................................... 25 CHAPTER 3: METHODOLOGY ................................................................................................ 26 Purpose of the Project and Questions ........................................................................................ 26 Methodological Framework ...................................................................................................... 26 Participating Stakeholders ......................................................................................................... 27 Observation Sampling Criterion and Rationale ..................................................................... 27 Observation Sampling (Access) Strategy and Rationale ....................................................... 27 Data Collection and Instrumentation......................................................................................... 28 Interviews .............................................................................................................................. 29 Observation ............................................................................................................................ 29 Documents and Artifacts ....................................................................................................... 31 Data Analysis ............................................................................................................................ 31 Credibility and Trustworthiness ................................................................................................ 32 Validity and Reliability ............................................................................................................. 32 Ethics ......................................................................................................................................... 33 Limitations and Delimitations ............................................................................................... 35 CHAPTER 4: FINDINGS ............................................................................................................ 37 Motivational Influence not Verified ...................................................................................... 37 Stakeholder Goals & Organization of Chapter ...................................................................... 37 A Description of the Instructor role ....................................................................................... 38 Interview and Observation Analysis ...................................................................................... 40 Findings ..................................................................................................................................... 43 viii Results ................................................................................................................................... 44 Knowledge Results Discussion ............................................................................................. 44 Motivation Results Discussion .............................................................................................. 50 Organizational Results Discussion ........................................................................................ 54 Summary ................................................................................................................................... 57 CHAPTER 5: FINAL RESULTS ................................................................................................. 58 Recommendations for Practice to Address KMO Influences ................................................... 58 Declarative Knowledge Solutions, Needs, or Assets. ........................................................... 61 Procedural Knowledge Solutions, Needs, or Assets.............................................................. 61 Metacognitive Knowledge Solutions, Needs, or Assets ........................................................ 62 Motivational Recommendations ............................................................................................ 63 Process and Procedures ......................................................................................................... 68 Cultural Support .................................................................................................................... 69 Integrated Implementation and Evaluation Plan ....................................................................... 69 Level 4: Results and Leading Indicators ............................................................................... 70 Level 3: Behavior .................................................................................................................. 73 Level 2: Learning ................................................................................................................... 78 Level 1: Reaction ................................................................................................................... 84 Evaluation Tools .................................................................................................................... 86 Summary ................................................................................................................................... 90 Strengths and Limitations.......................................................................................................... 91 Conclusion ................................................................................................................................. 91 APPENDIX A ............................................................................................................................... 93 Interview Protocol ..................................................................................................................... 93 APPENDIX B ............................................................................................................................... 96 Observation Protocol ................................................................................................................. 96 APPENDIX C ............................................................................................................................. 100 Consent Information Sheet ...................................................................................................... 100 APPENDIX E ............................................................................................................................. 102 Consent Information Sheet ...................................................................................................... 102 APPENDIX F.............................................................................................................................. 103 Classroom Setup ...................................................................................................................... 103 ix REFERENCES ........................................................................................................................... 109 List of Tables TABLE 1 STAKEHOLDERS AND STAKEHOLDERS’ PERFORMANCE GOALS ......................................... 5 TABLE 2 KNOWLEDGE INFLUENCES ............................................................................................... 17 TABLE 3 MOTIVATION INFLUENCES ............................................................................................... 21 TABLE 4 ORGANIZATIONAL INFLUENCES ....................................................................................... 24 TABLE 5 KNOWLEDGE RESULTS .................................................................................................... 49 TABLE 6 MOTIVATION RESULTS .................................................................................................... 53 TABLE 7 ORGANIZATIONAL RESULTS ............................................................................................ 56 TABLE 8 SUMMARY OF KNOWLEDGE INFLUENCES AND RECOMMENDATIONS ............................... 59 TABLE 9 SUMMARY OF MOTIVATION INFLUENCES AND RECOMMENDATIONS ............................... 64 TABLE 10 SUMMARY OF ORGANIZATION INFLUENCES AND RECOMMENDATIONS ......................... 67 TABLE 11 OUTCOMES, METRICS, AND METHODS FOR EXTERNAL AND INTERNAL OUTCOMES ...... 71 TABLE 12 STRATEGIES FOR TRAINING OF INSTRUCTOR-LED TRAINING ......................................... 74 TABLE 13 REQUIRED DRIVERS TO SUPPORT CRITICAL BEHAVIORS ............................................... 76 TABLE 14 COMPONENTS OF LEARNING FOR THE PROGRAM ........................................................... 81 TABLE 15 COMPONENTS TO MEASURE REACTIONS TO THE PROGRAM........................................... 84 TABLE 16 EVALUATION TOOLS ...................................................................................................... 87 TABLE 17 BLENDED EVALUATION ................................................................................................. 89 List of Figures FIGURE 1 CLASS ACTIVITIES COMPARISON .................................................................................... 47 1 CHAPTER 1: INTRODUCTION Introduction to the Problem of Practice The educational problem this dissertation addresses is the training and preparation of professional corporate instructors or Subject Matter Experts (SMEs) and their ability to transfer knowledge to their students. It is not uncommon for Subject Matter Experts to be asked to teach technically challenging classes because of their content knowledge and not for their pedagogical knowledge. It is necessary for the organization to identify and meet the instructor’s needs to provide quality instruction. The quality of Instructors and their performance in corporate classrooms is important because research shows that the training function has a positive impact on the performance of individuals, teams, and ultimately the organization (Aguinis, 2009). The Business school, University of Colorado, Denver study determined training affect declarative knowledge or procedural knowledge, but also improves strategic knowledge (Aguinis, 2009). Strategic knowledge is defined as knowing when to apply specific knowledge or skill (Kozlowski et al. kraiger et al. 1993). In the following chapters the knowledge, motivation, and organizational enablers, along with the necessary instructor skills to be successful in the classroom will be defined. This dissertation will present a road map towards addressing how to improve Instructor skills, first, by evaluating current skill levels and establish the current baseline. Second, by assessing the company’s baseline with like organizations. Third, by conducting a Needs analysis and make recommendations for change. Evaluating instructor’s abilities and skills in adult learning will allow organizations to be able to measure the quality of their training program and make changes for improvement. 2 Organizational Context and Mission The name of the proposed organization project site is Spartan Flight Systems (SFS). SFS was founded in the early1960s in Southern California. Spartan Flight Systems’ headquarters, has a broad engineering and laboratory facility in Southern California. SFS has a network of regional offices maintained throughout the United States to provide customers with dedicated on- site support. SFS has been successful at independent technical and scientific research, development, and advisory services to national security space (NSS) programs, the United States Air Force’s Space and Missile Systems Center (SMC), the National Reconnaissance Office (NRO), and national interest for civil agencies like NASA, the National Oceanic and Atmospheric Administration, commercial companies, universities, and international organizations. Spartan Flight Systems dedicates themselves to assuring mission success and solving the Nation’s most complex technical problems. SFS employs approximately 5,000 employees, 70% of whom are technical experts; Among SFS’s technical staff 70 percent holds advanced degrees, with close to 1,000 holding a Ph.D. One third of the SFS population resides in the California Campus. Organizational Performance Goal Spartan University’s (SU) goal is that by the fourth quarter of 2016, SU will have implemented an Instructor Quality Management Program. The Associate General Manager of SU established this goal after a weeklong leadership and program review with the leadership team outlining several key areas that need enhancement in order for SU to improve Spartan Flight Systems learning and development programs. The achievement of SU’s goal will be 3 measured by observation from instructional designers and Kirtpatrick’s level 1, 2 and 3 evaluations. Related Literature The related literature defines an adult learner as a biologically, psychologically, legally, or sociocultural person with a wealth of life experiences and learning habits (Malik, 2016; Mahan, 2014; Longenecker, 2013). Additionally, adult learners need instructors that have the skills necessary to manage a classroom to succeed (Blackburn, 2014). Moreover, students need to be engaged in order for knowledge to successfully transfer. Student engagement is typically explained as how involved or interested students appear to be in their learning and how connected they are to their classes, their institutions, and each other (Zepke, 2015; Parson, 2014). Student engagement is a pre-requisite to knowledge transfer. Knowledge transfer is a complex cognitive exercise that requires the learner to create connections between corresponding elements in different representations (Piksööt, 2014). Importance of the Evaluation The problem of U.S. Companies not implementing Instructor Quality Management programs is important to solve for a variety of reasons. First, from an organizational perspective, for Spartan Flight Systems to remain in business, it needs to develop and prepare their work force to meet the demanding needs of the Nation and their customers. In order to maintain the necessary workforce skills, Spartan Flight Systems (SU) needs to improve its Learning and Development processes and meet its training goals to keep both their internal and external customers from perusing alternative means of training. Second, from a community perspective, U.S. learning and development programs prepares employees and ensures their skills remain relevant and stimulates both the local and 4 national economy (Aguinis, 2009). The consequences to SU if the problem is not solved can be devastating and result in further lack of confidence from both executive management and students or the dismantling of SU and Spartan Flight Systems moving to a decentralized Learning and Development model. Stakeholder of Study and Stakeholder Performance Goal The combined efforts of all stakeholders will contribute to the achievement of the overall organizational goal to provide quality learning and development products that enables our stakeholders to solve the nation’s most complex challenges. It was important to evaluate Spartan University’s instructor’s performance goals. Therefore, the stakeholder focus for this dissertation was the space technical instructors. The stakeholders’ goal, supported by Spartan University’s Leadership, was by January 2017, 100% of instructors will apply effective learning engagement strategies during technical training classes. Success involves instructors improving classroom management, student engagement and knowledge transfer, SU scheduled instructor workshops, and observation evaluations. Failure to accomplish this goal would lead to continued mediocracy. Noncompliance would lead to a loss in funding, which adversely impacts the organization’s ability to provide sustain corporate knowledge by means of knowledge transfer to the next generation of engineers and the organization’s overall goal of providing quality learning and development products. The following table outlines the organization’s mission and performance goals. It also identifies the stakeholder of study’s performance goal. 5 Table 1 Stakeholders and Stakeholders’ Performance Goals Organizational Mission To provide quality learning and development products that enables our stakeholders to solve the nation’s most complex challenges. Organizational Performance Goal By August 2017, 100% of participants will be satisfied with technical trainings. Instructors By January 2017, 100% of instructors will apply effective learning engagement strategies during technical training classes. 6 Purpose of the Project and Questions The purpose of this project was to evaluate the effectiveness of the instructor improvement program. The purpose is not to place blame on the instructors, but rather to identify their needs in order for the organization to better support them and to further develop instructor’s skills to engage students effectively. Student engagement extends to creating and managing an environment conductive to adult learning which enables knowledge transfer. The research questions that guided this study were: 1. What are the knowledge, motivation, and organizational elements that interfere with instructors achieving 100% of its goal of apply effective learning engagement strategies during technical training classes. 2. What are the recommended knowledge, motivation, and organizational solutions? Methodological Framework The methodological framework that was utilized was a modified Clark and Estes’ (2008) Gap analysis. Instead of identifying gaps the researcher identified needs. The Needs analysis is a systematic, analytical method that helps to clarify organizational goals and identify the needs between the actual performance level and the ideal performance level within an organization. This methodology takes into account interfering elements that are generated based on personal knowledge and related literature. These elements will be validated by using observations, document analysis, interviews, literature reviews, and content analysis. Research-based solutions will be advocated and assessed in an all-inclusive manner. 7 Definitions Adult learning: Adult learning is the entire range of formal, non-formal and informal learning. Activities which are undertaken by adults after a break since leaving initial education and training, and which results in the acquisition of new knowledge and skills. Knowledge Transfer: Is a complex cognitive exercise that requires the learner to create connections between corresponding elements in different representations. Student Engagement: Student engagement is how involved or interested students appear to be in their learning environment. Classroom Management: Instructors governing of the classroom which directly affect the student learning STEM: Science, Technology, Engineering & Math SFS: Spartan Flight Systems SU: Spartan University SGO: Spartan General Offices IQM: Instructor Quality Management FFRDC: Federally Funded Research and Development Center KMO: Knowledge, Motivation, Organization OSG: Operational Support Group TOD: Training Operations Department TTDD: Technical Training Development Department ISD: Instructional Systems Designer SME: Subject Matter Expert 8 Organization of the Study Five chapters are used to organize this dissertation. This chapter provided the reader with the key concepts and terminology commonly found in a discussion about adult learning and student engagement. The organization’s mission, goals and stakeholders as well as the initial concepts of needs analysis were introduced. Chapter Two provides a review of current literature surrounding the scope of the study. Topics of adult learning, student engagement, classroom management, and knowledge transfer will be addressed. Chapter Three details the assumed causes for this study as well as methodology when it comes to choice of participants, data collection and analysis. In Chapter Four, the data and results are assessed and analyzed. Chapter Five provides solutions, based on data and literature, for closing the perceived needs as well as recommendations for an implementation and evaluation plan for the solutions. 9 CHAPTER 2: REVIEW OF LITERATURE Introduction This literature review will examine factor related to the implementation of an Instructor Quality Management Program at Spartan Flight Systems. The literature review begins with the importance of instructor skills and continued development of instructor skills throughout their teaching careers at SFS. Next, this review will provide an overview of Instructor Quality Management and the challenges associated with improving instructor skill nationally, in industry, and specifically in a federally funded research and development center. The review will then present an in-depth discussion on systematically based instructional practices and characteristics of excellent instructors. This section includes current research on knowledge of instructors, instructor preparation, and professional development practices in the U.S. Finally, the review highlights research on factors affecting fidelity to instructional best practices and provides insights into the assumed organizational causes contributing to implementation needs. The literature review concludes with a description of the Instructor Quality Management Program at The Space Corps, designed to improve the quality of instruction delivered by the instructors; a discussion of the need for further research; and a summary of the assumed knowledge, motivation, and organizational needs. Organizational Learning Adult Learning Adult learning to those that do not study education can sound complicated and people may believe that there is more to it than there really is. To their benefit there are professionals that focus their time in this field of study to shed light on the unknown. We have a group of highly educated people that study how adults learn and create programs for adults to learn new 10 things. The term adult learner can be defined as a biologically, psychologically, legally, or sociocultural person with a wealth of life experiences and learning habits (Malik, 2016; Mahan, 2014; Longenecker, 2013). The basis for how adults’ best learn can be expressed as the physiological foundation of best methods to effectively teach and facilitate their learning (Malik, 2016; Mahan, 2014; Longenecker, 2013). Life experience and past learned habits and behaviors affect the adult learner and the adult learner is able to comprehend, process, and apply information, based on their ability to comprehend, process, and apply information being taught (Mahan, 2014). Adult learners are individuals that are of the age of those that are graduates of a four-year degree. These learners are primarily in the workforce applying what is learned in their daily activities, and their role as a student is secondary (Malik, 2016). Adult learning is defined as the “the entire range of formal, non-formal, and informal learning activities which are undertaken by adults after a break since leaving initial education and training, and which results in the acquisition of new knowledge and skills.” (www3.hants.gov.uk/adult-learning-definition.pdf). Crucial to successful adult learning is applying the knowledge of how adults learn to how instructors deliver course material. The following section will explore how instructors manage the classroom. Classroom Management Research offers varying definitions of classroom management, but generally include actions taken by the instructor to establish order, engage students, elicit cooperation, prepare activities and anticipate for unforeseen and potentially sticky predicaments (Jala, 2016; Blackburn, 2014; Emmer 2001). Instructors influence students and how they respond to subjects 11 in the classroom (Jala, 2016); therefore, instructors have a lot of influence over their students. Consequently, even if instructors are given a pre-approved curriculum they ultimately make decisions regarding the governing of the classroom which directly affect the student learning outcome (Jala, 2016). Obtaining the skills necessary to manage a classroom is crucial for instructors to succeed (Blackburn, 2014). A classroom setting is built for the learning experience; instructors should ensure that the classroom environment is conducive to learning (Jala, 2016). Physical classroom management issues may range from technical difficulties like troublesome computers, downed databases, and Internet access (Blackburn, 2014). Similar to Facebook, which is built on and for capturing and holding people’s interest, the classroom should be built and aligned for learning and engaging activities to meet the learning objectives and the interest of the students (Jala, 2016; Blackburn, 2014). Adult learning and classroom management strategies are two considerations for the instructor to contemplate before facilitating a class, with special attention to capturing and maintaining the students’ interest in the learning content. The instructor must consider how to keep the students engaged before setting foot in the classroom. Student Engagement The literature has different definitions of student engagement. Parsons (2014) defines student engagement as engagement influenced by specific context and situation. Zepke (2015) describes student engagement as a set of student and institutional behaviors in a classroom and as a social-cultural environment in which engagement is the glue joining the classroom, students, instructors, stakeholders, and contributors to learning. The theme of the literature describes student engagement as fluid in nature, susceptible to change depending on the instructor, the design of delivery, and the tasks that can encourage or discourage student engagement (Zepke, 12 2015; Parson, 2014). Moreover, student engagement typically is explained as how involved or interested students appear to be in their learning and how connected they are to their classes, their institutions, and each other. The researcher recognize that student engagement is more complex than just observable behaviors (Zepke, 2015; Parson, 2014). Educators strive to design engagement experiences for students that utilize cognitive engagement and the use of metacognitive and self-regulated strategies because engagement is explicitly associated with student achievement (Zepke, 2015; Parson, 2014; Howard, 1996). The combination of classroom management and student engagement leads to the instructor’s ability to transfer knowledge to the student, which will be discussed in the follow section. Knowledge Transfer Reviewing the literature it became apparent that organizations whose core business function is of a complex scientific nature have similar challenges with learning and development and knowledge transfer (Curran, 2014). Knowledge transfer is a complex cognitive exercise that requires the learner to create connections between corresponding elements in different representations (Piksööt, 2014). The objective of learning is to prepare students for the world outside of the classroom. It is the goal of the instructor to employ rigorous, thought-provoking activities for the students to help them develop formal thought processes that they can apply to their everyday life (Day, 2012). Day (2012) continues on to say that education is about providing an environment where students gain knowledge that they can apply in a relevant way to new situations. 13 Learning and Motivation Theory This study will be applying a modified version of the Clark and Estes' (2008) Gap analysis framework. However, this study will focus on needs and continue to use the gap analysis six-step problem-solving approach designed to identify performance needs, uncover root causes, validate the causes, and formulate research-based solutions to improving the performance of Learning and Development instructors. The Needs analysis framework utilized in this dissertation will focus on knowledge, motivation and organization (KMO) influences. This study uses a modified Clark and Estes' (2008) Gap analysis (needs analysis) which uses KMO to categorize performance problems in manageable and understandable terms, starting with knowledge. For this dissertation instead of identifying gaps, needs were identified and with the use of the KMO framework. There are four types of knowledge, described as being factual, conceptual, procedural, and metacognitive. The second area of Needs analysis, motivation, is addressed using four motivational theories from the literature: Expectancy Value Motivational Theory, Attribution Theory, Self-Efficacy Theory, and Interest. The study shows that all four can be applied in the context of learning and development at Spartan Flight Systems. The third component of the KMO framework, organization, applies to several areas in the case of this study. The literature describes several organizational factors that are influences; three in particular have been observed in this study: (1) Failed, outdated technology, or technology being use inappropriately; (2) outdated processes and procedures; (3) unsafe or inadequate work space. This study shows how the Needs analysis framework around knowledge, motivation, and organizational influences is used to identify needs, evaluate current practices, and address performance issues. 14 Knowledge and Skills Spartan Flight Systems (SFS) is a federally funded research and development center supporting the United States Air Force and other government agencies. SFS provides technical expertise concerning space (rockets, satellites, etc.), communications, imagery, cyber (a term encompassing cybersecurity, cyberwarfare, and other cyber technology concepts), and national intelligence. SFS also supports civilian agencies (SpaceX), non-military agencies (NASA), and the international space community in the disciplines described above. SFS shares knowledge and expertise in different ways; for example, by having assigned personnel to project teams, performing research and development, contracting services, and designing, developing, and delivering training courses. The purpose of this paper is to review relevant literature that focuses on knowledge- related influences that are pertinent to the achievement of the stakeholder goal being, “By January 2017, 100% of instructors will apply effective learning engagement strategies during technical training classes”. Spartan University (SU) is the learning and development division within SFS and is a directorate within the Organizational Support Group (OSG). SU is made up of three internal departments, which includes the Education Program and Information Center (EPIC), Technical Training Development and Delivery (TTDD), and the Training Operations Department (TOD). The leadership team is comprised of an associate general manager and three directors, one for each department. One manager supports the west coast TOD, while a director leads the east coast TOD. The leadership team has two level-four technical subject matter experts (SMEs) in an advisory capacity. The leadership team and the SU as a whole experienced many changes over the past 10 months. This paper will analyze the reviewed literature in terms of the type of knowledge that is being used by instructors. 15 Knowledge Types There are four types of knowledge discussed in the literature reviewed and in the course content. Krathwohl (2002) identified these types in a taxonomy table as Factual, Conceptual, Procedural, and Metacognitive. Factual is described as discrete, isolated content elements (Krathwohl, 2002). Conceptual is described as complex, organized forms of knowledge (Krathwohl, 2002). Procedural is described as how to do something. Metacognitive is the awareness of and the knowledge about one’s own cognition (Krathwohl, 2002). Employees may improve their knowledge and skills through training programs. Learning alone is not sufficient for training to be considered effective. Using these knowledge types in the taxonomy table, the operational group will be able to decide where and how to enhance the planning of curriculum and the delivery of instruction to improve our instructor’s teaching skills (Krathwhol, 2002; Grossman, 2011). Applying the knowledge types and the taxonomy table with the stakeholder influences will give insight and understanding about the SFS’s instructors’ level of teaching ability and what is needed to improve their abilities. Stakeholder Knowledge Influences This section will examine literature that is relevant to the leadership team’s goal to improve trainer skills. A Grossman (2011) article explains that training inputs are thought to influence conditions of transfer, both directly and indirectly, through their impact on training outputs. With that said, learning is contingent, partly, on the use of cognitive processes such as memory and attention, the activation of background knowledge, and the utilization of cognitive strategies to achieve particular goals, in our case the improvement of trainer skills (Baker, 2006; Grossman, 2011). This section is organized by three sub subdivisions; Declarative Knowledge, Procedural Knowledge, and Metacognitive knowledge influences. 16 Declarative knowledge influences The literature indicated that the use of the taxonomy table provides an insight into complex types of knowledge and cognitive processes. Further laboratory studies showed that metacognitive knowledge and control were linked to successful cognitive performance, and applied research confirmed the practical importance of metacognition in the classroom (Baker, 2006; Krathwohl, 2002). Applying the taxonomy table to SU’s stakeholder’s goal of improving trainer skills, we can say that a declarative factual influence is that instructors need knowledge of their students’ performance after instruction, and that conceptually instructors need to know details of methods of their specific subject, for example, science, technology, engineering, and math (STEM) instruction to improve their ability to teach their students these subjects. See the knowledge worksheet in Table 1. Procedural knowledge influences The utilization of the taxonomy table allows developers at SU to identify missed learning opportunities. For example, to influence procedural knowledge learning, SU can utilize various job/teaching aids that can include informational aids, procedural aids, and decision-making and coaching aids (Grossman, 2011; Krathwohl, 2002). Job aids enable learning by providing important instructions and other reference materials, thereby reducing the mental workload required to apply new skills to the workplace (Grossman, 2011). Metacognitive knowledge influences To improve our instructors’ skills through metacognitive influences we can make them aware of their metacognitive activity. So they can use this ability to alter the ways they think and operate, a commonly used approach to assess both metacognitive knowledge and metacognitive control is 17 to ask what they know or what they do while engaging in particular cognitive activities (Baker, 2006; Krathwohl, 2002). For example, in Table 1 the Instructors are asked to perform a self- assessment based on course feedback to determine strengths and weaknesses. Table 2 Knowledge Influences Organizational Mission To provide quality learning and development products that enable our stakeholders to solve the nation’s most complex challenges. Organizational Global Goal By August 2017, 100% of participants will be satisfied with technical trainings. Stakeholder Goal By January 2017, 100% of instructors will apply effective learning engagement strategies during technical training classes. Assumed Knowledge Influence Knowledge Influence Assessment Declarative: (Factual): Instructors need knowledge of what effect engagement strategies are in instruction What strategies do you use to keep your students engaged and motivated? Procedural: Instructors need to know how to implement engagement strategies. What teaching methods do you find to be the most effective to keeping students engaged? Metacognitive: Instructors need to know how to reflect on their own effectiveness in the classroom. How do you prepare for class? What do you think about before your instruction? 18 Motivation SU, because of the recent changes to the organization, has developed new goals and is in the process of creating short-term performance objectives. Before these performance goals can be framed, SU needs to review literature that focuses on motivation-related influences that are pertinent to the achievement of developing an approach for improving trainer skills. SU leadership team handles both the strategic and tactical learning and development strategies. The following pages will define motivation, and its impact, based on the sources from the literature. Pintrich (2003) says that investigators ask questions about how and why some students seem to learn and thrive in school contexts, and others don’t, which leads us to consider the role of motivation as an influence in learning. In the following pages two motivational theories will be discussed. They are Expectancy Value Theory, and Self-Efficacy Theory. Expectancy Value Theory Since Expectancy Value Theory is relevant in the study of motivational factors at SU, it has been identified as a core influence in this study. It is important to understand this theory and how it applies in the organization context. The significance of student motivation has varied from incidental to central in psychological and educational research. Expectancy Value Theory creates linkages to cultural norms, experiences, aptitudes, and to those personal beliefs and attitudes that are commonly assumed to be associated with achievement-related activities (Eccles et al., 1998; Pintrich, 2003). Expectancy Value Motivational Theory says that student motivation is a cognitive process based on the idea that people believe there are associations between the effort they put at work, the performance they achieve from that effort, and the rewards they receive from their effort and performance. Simply said, people will be motivated if they believe that a good performance will lead to rewards (Eccles et al., 1998; Simone, 2015). 19 Self-Efficacy Theory Defined Self-Efficacy Theory has been identified as one of the assumed motivational influences of this stakeholder group. Self-efficacy can be defined as the judgments that individuals hold about their capabilities to learn or to perform courses of action at designated levels (Pajares, 2006). Restated, Self-Efficacy beliefs are the self-perceptions that individuals hold about their capabilities (Pajares, 2006). Igbaria (1995) uses self-efficacy theory to identify the number of factors affecting computer usage. The acceptance and use of computers by individuals appear to be limited due to fear of computers and their confidence in using them. Applying this theory in every department in the organization, SU can help identify instructors’ self-perceptions about their abilities and skills to develop course material, instruct, and use classroom technology. Instructors Assumed Motivational Influencers This section will discuss the assumed motivational influences of SU instructors. Expectancy Value and Self-Efficacy theories were identified as major influences on SU instructors. Linkages between the theories mentioned above and SU instructors will be presented for consideration. SU instructors are subject matter experts (SME’s) who typically have a Ph.D. in a STEM academic background. SU has a baseline set of course offerings aligned with the strategic thrusts of the organization; however, the majority of the course offerings are courses recommended by the instructors. Instructors have full-time engineering positions and teach part- time for SU with the approval of their management. This arrangement has created an environment that directly affects the instructors’ behavior and actions. For example, during the development stage of a course it is hard for SU to keep instructors on a schedule. SU often experiences schedule slippage, which adversely affects the SU support staff. The trainer’s 20 attitude towards SU support staff, when asked for updates or products that are due, is negative and unconcerned. These very same instructors do not have problems with meeting deadlines in their other day-to-day duties. This variance in performance occurs because SU has no real authority over them. SU asks their instructors to not only perform in their normal position as an engineer, or scientist, but also as an instructor. Course development, traveling, cost and time are things that instructors at SU have to consider before they commit to teaching a class. These SMEs do not instruct at SU for a bonus, a raise, or an award. Buzz around the company is that these instructors take on this extra duty because they enjoy their work, they want to pass on knowledge, and they enjoy teaching. We can link Expectancy-value theory because SU’s instructors’ self-concept involves their beliefs about their abilities to deliver quality instruction to their students. Instructors do not receive a bonus, raise, or award, but they exert extra effort, and are high performers. So why do they do this? Because of other internal/intrinsic rewards. For one, it is seen as prestigious to be an instructor at SU. Being an instructor allows for more visibility and credibility with upper management and peers. Not to mention many of them enjoy and desire to pass along knowledge which is enough for them. We can then tie Self-efficacy in this context because it includes the trainers’ beliefs about their abilities and competence, which are specific to an individual domain, for example, space/technical knowledge, STEM, etc. 21 Organizational Mission To provide high quality learning and development products that enable our stakeholders to solve the nation’s most complex challenges. Organizational Global Goal By December 2016, SU will implement a formal assessment plan for all technical training. Stakeholder Goal By May 2016, the Leadership Team to develop approach to improve trainer skills. Assumed Motivation Influences How Will It Be Assessed? Self-efficacy Expectancy Value Interviews, observation and Artifacts & document analysis Table 3 Motivation Influences 22 Organizational Influences The “O” in the KMO framework is Organization. In this section cultural influences, models, and settings will be reviewed. Culture is shared assumptions, values, and beliefs, which inform how people behave in their organization. These shared values influence employees in how they dress, act, and perform their jobs. Like any other organization, SU has a unique culture, which includes unwritten yet understood guidelines and boundaries for behavior. Assumed Cultural Influences The first of the three areas discussed is cultural influences. Two specific influences were identified as being key to the problem of practice. The first influence is that SU does not have a good reputation and instructors do not see SU as relevant to meeting the strategic goals of the organization and the needs of the SFS’s customers. Therefore, instructors do not respect or value the SU staff, SU rules and regulations, and processes and procedures. The population of instructors at SU has been instructing for the better part of 20 years. They have seen SU change from a valuable and efficient partner to a mismanaged organization with unnecessary rules. The second influence is that SU is under new management, and processes and procedures are changing. Without a strong organizational change management strategy to ease the transition, instructors are resisting changes in classroom processes and procedures. These changes are not just on the administrative side of the organization, but also in the classroom and in how training is developed and delivered. Instructors are resistant to these changes, wondering whether they will endure, and questioning the expertise of the new SU leadership team. 23 Cultural Models Holland and Quinn (1987) define cultural models as being ‘presupposed, taken-for- granted models of the world that are widely shared (although not necessarily to the exclusion of other alternative models) by the members of a society, and that play an enormous role in their understanding of that world and their behavior in it.’ The second of the three areas that this paper discusses is cultural models. At SFS, instructors do not trust the SU staff, hindering the institutional goal of integrating the instructor quality management program. As noted earlier, a majority of the instructors have been at SFS for over 20 years and have Ph.Ds. They feel that they know more than the SU staff. Moreover, the organizational level, skill level, knowledge, expertise and the high salaries of some of the instructors, compared with the SU staff, make collaborating awkward, and oftentimes instructors simply bypass SU staff to execute their agenda. Cultural Settings Cultural settings reflect beliefs, opinions and other aspects of a certain location. Two specific setting influences were identified as being key to the problem of practice. The first setting influence is that instructors see that SU staff has a high operational tempo and that they do not have the time to spend on the instructor quality management program. The second influence is that instructors know the SU staff, and know that there is no experience within the staff with regard to implementing an instructor quality management program. Again, the instructors have been around SFS and instructing at SU for over 20 years. They may have even implemented a quality management system in their own department, and may be reluctant to engage in a program that requires additional time and effort to their already full schedules. 24 Table 4 Organizational Influences Organizational Mission To provide quality learning and development products that enable our stakeholders to solve the nation’s most complex challenges. Organizational Global Goal By June 2017, The Space Institute will implement a formal assessment plan for all space technical training. Stakeholder Goal By October 31 2016, The Space Institute will improve the quality of instruction provided by our space technical instructors. Assumed Organizational Influences Organizational Influences Assessments Cultural Influences: Organization is not providing enough support or training; there is a lack of training and support for engagement practices Interview questions that identify needs and requirements of the instructors. Document and artifact review that identifies needs and requirements Cultural Model Influence: There is a culture of working in silos rather than collaboratively. Interview questions about whether instructors trust SU staff, and whether they feel that the SU function is relevant to the company. 25 Summary This literature review examined research on instructor skills utilizing the KMO framework to understand instructor’s knowledge motivation and organizational influences. Framing SU within the KMO framework reveled themes that guided the initial literature review. The literature review focused on the importance of instructor skills and continued development of instructor skills throughout their teaching careers, a standard that has been lost from SU for several years. The literature reviewed provided an overview of adult learning, classroom management, student engagement, and knowledge transfer, all lost by many instructors, at no fault of their own, because it is a standard that SU does not have in effect. Finally, this review covered highlights of research on factors affecting fidelity to instructional best practices and provided insights into the assumed organizational causes contributing to implementation Needs. The following chapter will outline the approach and methodology of the evaluation of SU instructor skills. 26 CHAPTER 3: METHODOLOGY Purpose of the Project and Questions The purpose of this project was to evaluate the effectiveness of the instructor improvement program, and to further develop instructors’ skills to engage students effectively. To reiterate, the intent of this dissertation is not to place blame on the instructors for having poor instructor skills, but rather to identify their needs in order for the organization to better support them and to further develop instructor’s skills to engage students effectively. Effective student engagement extends to creating and managing an environment conductive to adult learning which enables knowledge transfer. The research questions that guide this study are: 1. What are the knowledge, motivation, and organizational elements that interfere with instructors achieving 100% of the goal of applying effective learning engagement strategies during technical training classes? 2. What are the recommended knowledge, motivation, and organizational solutions? Methodological Framework The methodological framework to be utilized is a modified version of Clark and Estes’ (2008) Gap analysis. However, this study is looking at needs and not gaps. Needs analysis is a systematic, analytical method that helps to clarify organizational goals and identify the Needs to improve from the actual performance level and the ideal performance level within an organization. Elements that interfere with ideal performance were generated based on personal knowledge and related literature. These elements were validated by using surveys, document analysis, interviews, literature reviews, and content analysis. Research-based solutions are advocated and assessed in an all-inclusive manner. 27 Participating Stakeholders The stakeholder population of focus for this study is the instructors who work at The Space Corps (SFS) and teach Space Technical courses part-time at Spartan University (SU) in the Space General Offices (SGO) in El Segundo, California. The following pages will identify the particular criteria associated with the stakeholder group that will be considered for the purpose of sampling the study’s participants, and will explain the rationale for the criteria. Observation Sampling Criterion and Rationale Criterion 1. Live instructor-led space technical classes in the El Segundo training facility, specifically the main classroom N402 in building D8 at SGO. The purpose of this study is to observe the instructors teaching in a live environment to determine their mastery of student engagement, classroom management, and their use of tools to enhance the likelihood of successful knowledge transfer. Criterion 2. Space Technical Instructors–SU has three different categories of classes: Space Technical, Computing Skills, and Business and Leadership. The focus of this study is Space Technical courses. Space Technical courses cover complex technical topics where knowledge transfer techniques and student engagement strategies would increase the students’ ability to learn, retain information, and apply what they learned not only to their work but to conceptual frameworks other than what was specifically taught. Observation Sampling (Access) Strategy and Rationale To observe Space Technical Instructors, the researches had to gain access to the Space Technical classrooms in the El Segundo facilities. A letter was drafted and sent to the Associate General Manager of SU requesting access and approval to conduct the study within the facilities. 28 The letter requested access to training material for all Space Technical classes offered in GSO. Observations of live instructor-led classes were conducted throughout the entire life cycle of the data collection process. There are 120 different space technical courses offered in SGO, with just over 220 instructors who teach space technical courses in a fiscal year (FY). The selection of instructors and courses that will be observed was made randomly, based on the training being conducted during the study. Observations took place within the first quarter (Q1) of the fiscal year 2017 (FY17). The researcher was limited to how many courses were offered in the Q1 of FY17; however, the researcher will not observe more than five space technical courses. Data Collection and Instrumentation This study used interviews, observations, and document artifact analysis to answer the research questions. The researcher collected and analyzed documents and artifacts to determine what influences are present in the SU environment, specifically, whether there are incentives to encourage instructors to utilize student engagement strategies, classroom management techniques, and other best practices that lead to successful knowledge transfer. Document and artifact analysis allowed the researcher to determine whether there are any organizational influences that restrict instructors from using the above-mentioned strategies, techniques, and best practices. The researcher observed live instructor-led classes to determine whether instructors are utilizing student engagement strategies, classroom management techniques, and other best practices that lead to successful knowledge transfer. Though observation does not determine the underlining reason why an instructor is behaving in a particular way, it did tell the researcher if the instructors were using student engagement strategies, classroom management techniques, and other best practices that lead to successful knowledge transfer. 29 Interviews The interviews were conducted in a private setting, utilizing a SU operations conference room. Some participants wished to conduct the interviews in their office or over the phone, and the researcher accommodated these requests. The interviews were informal in nature and started with a short period of discussion that introduced the topic and background of the study. The semi-structured interviews were guided by an interview protocol (Appendix B). Each interview was scheduled for an hour. If more time was required, a follow-up interview was scheduled. The researcher took written notes during the interviews in order to document the interviewee’s story. To ensure that nothing was missed and to help with reflection, written notes were later transcribed and saved in a digital form. The researcher acquired approval from SU so that interviews could be conducted during working hours. The researcher accommodated requests from those participants who preferred to meet after hours. The interview questions derived from the KMO framework and literature review outlined in Chapter 2. The K questions were designed to answer the instructor’s knowledge of what adult learning, student engagement, classroom management, and knowledge transfer is, how to implement it, and how they prepare for instruction. The M questions were designed to answer the instructor’s confidence in the application of adult learning. Those who were interviewed were also observed, allowing the researcher to align the instructor’s answers to the interview questions with what could be observed in the classroom. Observation Observations were limited to the availability of courses being taught during Q2 FY17, not to exceed five total courses. Space technical courses vary in instructor size, and the number of instructors being observed varied depending on the courses being offered. These courses range 30 from having as few as three instructors to as many as 28 instructors. Each instructor was observed through the entirety of the modules that they were teaching. Each instructor was observed once. As previously stated, the purpose of the observation was to determine how effectively instructors used student engagement strategies, classroom management techniques, and other best practices that lead to successful knowledge transfer. The researcher was mindful of the inherent risk of his presence influencing the instructor’s behavior, as well as the behavior of those attending the class. To mitigate that risk, the researcher restrained himself to capturing observations during the class, and not interrupting or interfering with activities, students, the instructors, or the SU staff. The researcher’s actions were guided by the SU policies and procedures of conduct for site visitors. Methods used to capture observations:  Digital pictures of classrooms and classroom artifacts and facilitation tools  Observation notes capturing class sessions, and select breakout groups The researcher arrived before the beginning of the class to set up the audio recording device and the work area where the researcher could take observational notes. The researcher stayed after each class to recover the recording device and to clean up the work area. The main focus of the observation was the instructor and his or her ability to engage the students, manage the classroom, and to facilitate knowledge transfer; however, the researcher was also observing the students to determine whether they were engaged in the learning activity. The class sizes were no less than 12 students to a maximum of 36 students, per SU class scheduling policies. Class sizes vary depending on the time of year, the class being offered, and other external influences. See Appendix A for the observation protocol. 31 Documents and Artifacts The researcher read documents and watched videos of previous SU class offerings to supplement live instructor-led class observations. Documents included the SU instructor class agenda, course materials, memoranda, development material, emails, student evaluations, class evaluations, website content, etc. SFS employee climate surveys and other ad-hoc culture surveys were reviewed. The collection of this data allowed the researcher an understanding of SU instructors and the environment, which helped provide rich analysis. The collection of these artifacts produced relevant data beyond what can be observed in the classroom; therefore, the examination of these documents was an integral method of data collection for this study. Data Analysis Interviews, observations, and data analysis began during data collection. The researcher wrote analytic memos after each interview and each observation to document thoughts, concerns, and initial conclusions about the data in relation to the conceptual framework and research questions. Interviews were transcribed and coded. In the first phase of analysis, the researcher opened coding, looking for empirical codes and applying a priori codes from the conceptual framework. A second phase of analysis was conducted where empirical and a priori codes were aggregated into analytic/axial codes. In the third phase of data analysis the researcher identified pattern codes and themes that emerged in relation to the conceptual framework and study questions, followed by analysis of the documents and artifacts for evidence consistent with the concepts in the conceptual framework. 32 Credibility and Trustworthiness Qualitative research must take into account the environment, the participants’ biases, and even the researcher’s biases in order to produce accurate conclusions (Onwuegbuzie, 2007). For this study, the researcher reviewed SU documents and artifacts to paint a picture of the environment in which the instructors operate. Moreover, the collection of these documents and artifacts ensured accurate and comprehensive data to protect against critiques about the level of credibility and trustworthiness of the research. The researcher also observed live, instructor-led classes, which provided an opportunity to triangulate the different methods of data collection, further increasing the trustworthiness and credibility of the findings. Although methods and procedures do not guarantee validity they are nonetheless essential to the process of ruling out threats to validity and increasing the credibility of the conclusions (Maxwell, 2013). The trustworthiness of the data is tied directly to the trustworthiness of those who collect and analyze the data and their demonstrated competence (Merriam, 2016, p. 260). The researcher worked to identify his own biases before the collection of data, and presented them as limitations for this study. Moreover, the researcher took an all-inclusive approach and provided context of the research and the environment in which it occurred for the targeted audience. In order to achieve this, the researcher has provided a sizeable amount of detail from observations and data collected from rich descriptive documents and artifacts to set the stage for the audience and to provide the context and background of the study. Validity and Reliability To maintain accuracy, the researcher must set controls to ensure that the investigation is valid and reliable. Validity and reliability in qualitative research addresses questions of how well research findings match reality (Merriam, 2016). Validity and reliability are concerns that 33 can be approached through careful attention to a study’s conceptualization and the way in which the findings are presented (Merriam, 2016). To achieve validity and reliability, the researcher analyzed the results based on collected documents, artifacts, and live, instructor-led observations at SU facilities and from SU document archives. The researcher used triangulation and presented limitations to ensure reliability. To ensure trustworthiness, the researcher provided the data collection methodology in order for the study to be duplicated. Ethics As the researcher conducting this qualitative research and as an employee of the organization, I acted as a doctoral student and researcher, required to make ethical decisions throughout the course of the study. In no way will the results of this study be used as a performance improvement or corrective action tool for the organization against any participants. As the researcher, I acted as a researcher and not as an employee or a manager of SU. I worked with management to ensure there would be no risk of retaliation or reprimand to employees participating in this study. In order to ensure that I separated my role as a graduate student from my role as an employee, I announced to all staff members involved in the study that I was acting as a researcher and not as an employee of the organization, and as such would defer research- based discussions, comments, questions, etc., to SU employees and/or management. By participating in this study employees helped identify and solve issues within the organization and thus improve organizational performance. As the researcher and an employee of the organization, I will only release what I learned after the finalization of the study and publication of this paper. At that time the paper will be made available for those interested in the topic for download or email distribution. There will be no preliminary discussion of findings prior to the publication of this paper. 34 This study’s data consists of observational notes, course documents and artifacts. Participants were asked to sign informed consent forms at the beginning of the study. By signing the informed consent participants have allowed the researcher (David Hawkins) to act as an observer of the following: 1) the setup of the class (starting 1 hour before the official start of class); 2) the class itself, including the students, instructors, and all activities associated with the class; 3) the after-class shut down procedures up to one hour; and 4) all courseware documents and artifacts. Read-access to the courseware documents and artifacts was permitted via the informed consent form. According to Glesne (2011), informed consent is necessary to ensure the participants are aware that their participation is voluntary, all the collected data will be kept confidential, and participants can withdraw at any time without penalty. As part of the measures to minimize the risk that individual rights and welfare concerns are negatively affected, I will submit my study to the University of Southern California Institutional Review Board (IRB) and have followed their rules and guidelines regarding the protection of the rights and welfare of the participants in this study. Doing so ensures that the IRB reviews and concurs that I practiced due diligence in order to reduce risk to the extent it is possible to reduce risk. All of the participants have signed consent forms and have been reminded that this study is voluntary, and their identity is kept confidential. Confidentiality is extremely important for this study, and no names or links to individuals have been associated to observational notes. To further ensure confidentiality, data collected has been codified and categorized to avoid personally identifiable information or the linking of a response to a question or observation of a specific individual. Prior to the observations, I obtained permission to record observations, and provided the participants with copies of notes taken during the observations to allow them the opportunity to 35 ensure I did not misrepresent the operations of their class. I reminded the participants I will not provide any incentives, so as not to coerce them; however at the conclusion of the study I will send them a letter of appreciation for participating in the study. The lack of incentive is a way to minimize the possibility that participants will feel under any pressure to participate. Limitations and Delimitations Like any study there are limitations and draw back to how we collect data, and this one is no different. The targeted instructors’ pool consists of engineers and scientists who have been in the aerospace field for over 20 years, and most of them have been instructing students for just as long. Half of the identified participants have Ph.Ds. in a scientific field of study, and tend to have a type A personality (i.e., aggression, autonomy, extroversion, and impulsiveness) (Chesney, 1981). The data collection method of interviews, observations, and document analysis was designed to mitigate the risk of an instructor’s temptation to skirt the line of fact and fiction when it comes to their capabilities in the classroom. To minimize impression management which is a strategy to alter the public's perception about their legitimacy (Elsbach, 1992), additional data sources will be gathered to allow the investigative team to see past the fog of charisma and personalities that charm the best of us professionals. The instructor pool was limited to the schedule of courses that were being offered during the data collection phase. The instructor pool was further limited to courses offered in Southern California, and instructors at lower position levels in the organization needed management approval to participate. Moreover, instructors needed to participate in both the interview and observation phase of the study. SU staff were instrumental in helping identifying courses and instructors to interview and observe. 36 The investigative team took great care in developing the conceptual framework. The interview questions link directly to the KMO framework, are focused on the instructors’ self- assessment, and are designed in a manner to allow validation from observing the instructor during a live class. What cannot be validated through observation can be validated by the investigative team using document analysis as a validation source. 37 CHAPTER 4: FINDINGS “Teaching is a lot like entertaining I prepare as if I’m going up on stage like a comedian does for their act,” said one of the instructors interviewed in this study. In the following sections a review of the instructor’s answers to questions like, “what strategies do you use to keep your students engaged and motivated”, and “what can the organization provide you to be more effective in the classroom”. This chapter will also provide information gathered from observation, documents, and artifacts collected from the organization. Motivational Influence not Verified During the planning of this study the researcher originally alleged that there were more motivational influences that affected the instructors, specifically trust. The researcher believed that trust between SU and the instructors was a key component into the instructor’s perceived behavior. However, in the field, the study focused on observable behavior in the classroom, and the interview questions focused on the instructor’s self-assessment of their knowledge, motivation, and what they believe are organizational influences as applied to adult learning, student engagement, classroom management, and knowledge transfer. Documentation that was reviewed did not reveal trust as an issue as it applies to this study. In the future a broader interviewing scope to include SU staff, organizational leadership and key stakeholders will provide better context as to how trust between the instructors and SU affects course design/ delivery and the instructor’s performance in the classroom. Further research in these areas would be beneficial. Stakeholder Goals & Organization of Chapter The purpose of this qualitative study was to understand how effective SU’s instructor staff are in a federally funded research and development center’s learning and development 38 division in a space technical classroom. As stated earlier the purpose of this dissertation is not to place blame or hold instructors accountable for not having a strong instructional skills. On the contrary, it is to identify instructor needs so that the organization can better meet their needs. The following are two research questions that guided this investigation: 1. What are the knowledge, motivation, and organizational influences that are related to instructors achieving 100% of the goal to apply effective learning engagement strategies during technical training classes? 2. What are the recommended knowledge, motivation, and organizational solutions? In order to answer these questions, the researcher interviewed and observed seven SU instructors who are currently teaching space technical classes. Talking with each of the instructors, the researcher learned how they became involved in teaching classes, their experiences as instructors, and how they felt about the dual roles that they play at the organization; being a full time engineer and a part time engineering instructor. While their stories were distinctive and in many cases extraordinary, several common themes emerged from the analysis of the data collected during the interviews with them. The following is a recount of the significant themes. The first section, description of the instructor’s role, provides background information and an overview of the data collection that took place for this study, followed by a presentation of the research findings using the KMO framework. Each KMO influence section closes with a summary, and the chapter concludes with an overall recap of the findings. A Description of the Instructor role The Subject Matter Expert (SME) is an engineer that works in the business. SMEs are “selected” or “volunteer” to participate in the creation of a technical course. Creation is 39 misleading yet is used to officially describe the ISD process. Often times it is the SME that approaches SU with an idea about a new class. Which leads to courses that are not aligned with business needs or result in increased organizational performance. The SME’s are the primary content developers of the courses, and instructional designers are involved to ensure that proper instructional design is applied, therefore ISD’s also act as the primary point of contact for the instructors to interface with SU. There are multiple SME’s assigned to a class; anywhere from 2 to 12+ SME/ instructors are involved in the design and delivery of a space technical course. When the course is ready for delivery, a training coordinator is assigned to the course to manage the logistical needs of the delivery such as the tasks associated with setting the course up on the learning management system, room logistics, coordination with the Information Technology department, logistical coordination with facilities and any last minute coordination with the instructors. The researcher coordinated with SU’s Tech Lead to identify classes that fell within the time frame of the data collection phase of this study. There were three classes being offered at the same time as the data collection phase. For the purposes of this study they were coded as The Blue Class, The Kilo5 class, and The Noble class. These are not the classes’ real names; they are taken from the researcher’s favorite teams in his favorite science fiction novel. The Blue Class is a two-day class, lasting for six hours each day. The Blue Class has six instructors assigned to both the design and delivery team. The Kilo5 class is a three-day class, lasting six hours each day. The Kilo5 class has seven instructors assigned to both the design and delivery team. The Noble class is a half-day class, lasting three hours. The Noble class has one instructor assigned to the development team and five instructors assigned to the delivery team. 40 Once these three classes were identified as possible classes to be observed, the researcher coordinated with the senior Tech Lead to collect all relevant documents associated with each class. The researcher used the class instructor roster as a tool to identify instructors to both interview and observe. The researcher sent out an email to all 18 instructors that were assigned to these three classes to solicit their participation in the study. Seven of the 18 instructors agreed to participate in the study. The organization’s general council required that instructors categorized as organizational level 3 and below and were not in a management role and would require verbal management approval to participate. Only two fell in this category, and the researcher called their management to receive approval. Interview and Observation Analysis The researcher coded the instructors as follows: John, Kelly, Frederic, Linda, Emile, Carter, and Jorge. These are not the instructors’ real names. The names are the researcher’s favorite characters from his favorite science fiction novel. John and Kelly instructed several modules from the Blue Class. Normally an instructor would teach only one of the modules in the course, because they are considered to be a subject-matter expert (SME) in that module’s topic. However, for this iteration of the Blue class one of the instructors was unable to make it, so the lead instructor (John) had to pick up the slack. The researcher only observed John and Kelly for an hour each. John was interviewed in person before the class, and Kelly was interviewed in person after the observation period. The researcher held the interviews in the instructor’s respective offices, with John’s interview lasting just over an hour, and Kelly’s lasting for an hour. The next set of instructors, Frederic, Linda, Emile, and Carter, were all part of The Kilo5 class. All of these instructors were interviewed after the researcher observed them in the 41 classroom and the interviews were all conducted over the phone. The nature of the work and the demanding needs of their customers and their location made it hard for the researcher to schedule in-person interviews. The researcher observed each instructor for an hour and each interview except for one lasted between 45 minutes and an hour. Emile’s interview lasted for an hour and 30 minutes. Emile interview last longer than the rest because he offered longer stories and took longer to get through all of the interview questions. Jorge is the lead instructor for The Noble class and was the last instructor that the researcher interviewed and observed. Jorge has a hectic schedule that required a phone interview, which was conducted after the observation of the instructor’s class. The researcher observed Jorge for an hour and the interview lasted just over an hour. As mentioned in Chapter 3, the researcher also reviewed SU artifacts. The researcher reviewed several organizational documents, process and procedure documents, charters, statement of works, etc. The researcher also reviewed internal websites, released emails, SharePoint sites, and calendars. For every course the researcher spent two hours reviewing the following documents: course material, course instructor roster, course assignments and course evaluations. Please see Chart 5 for a visual comparison of Duration breakdown of data collection methods. Based on the collection of information, the researcher was able to determine that there is a misalignment between the different departments within SU and of the needs of the corporation as a whole. Processes and procedures in the technical department are out dated and do not match current documents in the operations department. The list of Learning Initiative Proposals (LIP) does not match the development schedule, or the course schedule that operations manages. Which means that courses are being created outside of the agreed upon process without proper vetting from the management team to ensure that the classes being created are 42 relevant or aligned to the needs of the corporation. This is important evidence showing that the departments are working in silos rather than collaboratively. In of itself this discovery does not provide further insight into SU’s instructor’s skills or abilities. However, it does provide evidence into the organizations approach to learning and development, and begs the question, how does this dysfunction affect the instructors? Further investigation showed that classes that should reside in Professional Training and Development was being developed and or purchased from a vendor by the Technical Training Department. Management made an effort to further define the roles and responsibilities between the two departments as described in emails that were reviewed. 43 Findings The researcher is privileged to represent the opinions of the instructors here, and thanked them for their time and honesty during their respective interview and for allowing the researcher to observe them. It was the intention of the interviewer to do them justice when interpreting their stories and experiences. In sharing their exceptional thoughts and understandings about what it was like for them to teach for SU, perhaps this effort can inform SU training practices and increase support for SU instructors in order to help them reach the next level of success. The following sections are the major themes from the interviews, observations, and document analysis conducted for this study. Knowledge 1. Instructors do not know what effective engagement strategies are in instruction. 2. Instructors do not know how to implement engagement strategies. Motivation 1. Instructors do not have confidence in applying adult learning theory and applying student engagement methods. 2. Instructors do not value the importance of adult learning principles. Organizational Influences 1. The organization is not providing training; there is a lack of training in adult learning principles, student engagement practices, knowledge transfer, and classroom management techniques. 2. There is a culture of working in silos rather than collaboratively 44 Results This qualitative study used data gathered from semi-structured interviews, observations, and document analysis. Interviews and observations were conducted with each of the seven participants in order to understand how effective the instructor staff is in a space technical classroom. This data helped the researcher determine how instructor training programs maybe leveraged to better support instructors and give them the knowledge, motivation, and organizational support to better perform in the classroom. The following is an exposition of the research findings using the KMO framework, and the chapter ends with an overall summary of findings. Knowledge Results Discussion The first major theme that emerged in data analysis shows that instructors do not know what effective engagement strategies are in instruction. When asked during the interview what engagement strategies they use, the common answer instructors gave was that they use eye contact and real-world examples to keep students engaged. Moreover, they also feel that they keep the students engaged by asking them questions to get them to participate in the class. Participants reported that they are effective in keeping their students engaged in the classroom based on their own understanding of what engagement strategies are. They talked about being able to keep students engaged by bringing examples from work and using real-time student feedback via Q&A. One instructor, Emile, expressed her efforts to keep students engaged by using show business: Teaching is a lot like entertaining, I prepare as if I’m going up on stage like a comedian does for their act. You have to keep your students engaged like a comedian keeps his 45 audience interested in his story… I rehearse what I’m going to say three days before my class, and I go over what I’m going to say until I get it right. According to Emile, “Humor, being active on stage and having specific eye contact with the students is effective to keep them interested in what you have to say. ” Another instructor, Kelly, explained that sequencing is important to helping students follow along: “I introduce the concept, and then I give them an example of how that concept relates to work”. The thoughts, ideas, and perceptions these instructors had in response to the interview questions showed an inconsistent level of sophistication. John and Kelly used real-world examples and eye contact as student engagement, whereas Emile used “Show Business” and teaching to “general concepts.” Emile also said “I focus on concrete examples even to the loss of technical accuracy as a way to keep my students engaged.” Linda, Frederic, Carter, and Jorge relied on eye contact, using storytelling techniques and asking probing questions as their method of choice as key student engagement techniques. However, when observed, Emile was the only instructor to hold true to his approach to keeping his students engaged. During the other observations the researcher observed students getting distracted with technology or glazed over with the long periods of lectures in-between breaks. Imagine sitting in a classroom where only half of the lights are turned on, so that you can better see the main screen displaying the PowerPoint presentation. The instructor is at the front of the room and he or she has been talking for 15 or 20 minutes, flipping through charts with diagrams with math equations. Every 3 or 4 minutes a different student gets up and leaves to use the rest room. A random instructor at the back of the room speaks to another instructor about what they need to do next and all of a sudden the instructor at the front of the class asks the class if they have any questions. Before you could respond he or she flips to the next slide and 46 continues on with their lecture. Now, repeat for seven more instructors, for six hours a day over three days. There is one group quiz on day one, and one group activity on day two. This approach is not conducive to learning or achievement of the overall goals of the SU. When it comes to presentation skills, the instructors were observed using good eye contact, incorporated real-world examples, and they often introduced concepts and followed with real-world application of how those concepts apply in the work place. However, the instructors did not ask probing questions. John asked his students if they had a question on the topic he was lecturing on, but he did not give the class enough time to respond before continuing on with his lecture. As the observations continued, this failure to provide time for questions showed as a trend across all of the instructors observed. Comparing the observation field notes and the class agenda to the Pace Methods to Avoid Boredom Methods Scale, and the Training/Facilitation vs. Presenting Chart as seen in appendix B, it became clear that what these instructors are really doing is presenting information. For a total of 33 hours of classroom instruction there was an hour long activity where the students performed an exercise in the parking lot (not observed; however, was listed in the course agenda); a 15-minute writing assignment and a 5-minute, 1- question group quiz that the instructor never went over because he ran out of time. What does this mean? What’s the impact of this? The students that attended this class spent most of their time watching someone stand in front of the classroom talking at them. There was a missed engagement opportunity on the instructor’s part. The instructor could have asked more questions, had the students work in groups on a problem, the instructor could have shown a video. Moreover, there is a missed learning opportunity for the students because there was no opportunity for them to work with or use the knowledge that the instructor was trying to transfer to them. Please see Chart 6 for a visual comparison of class activities. 47 Figure 1 Class Activities Comparison 3% 1% 0% 96% Class Activity Comparison Group Activity 1 Group Activity 2 Group Quiz Lecture 48 The second major theme that emerged in the data analysis indicates that instructors do not know how to implement engagement strategies. As described above instructors have their own idea as to what effective engagement techniques are. However, as observed, instructors were not consistent in implementing their own engagement techniques. All of the instructors interviewed said that they have only received on-the-job training, and that the only classes they have to prepare them for teaching are classes on how to give good presentations, and professional briefing classes. Emile said, “I received no formal training from the organization.” Jorge had a similar response, following up with a short story about his first teaching job, “I was told that, you’re a smart guy! You’ll figure it out.” Clearly, this is a group of instructors who have received training on how to be good briefers and how to give good presentations, which is why, when asked “What engagement strategies do you use?” They give answers that one would hear from people trained in how to be good presenters, rather than answers given by professionally trained instructors. By their own admission, these instructors have not been taught what good engagement strategies are, nor have they been trained or shown how to properly implement them. The following table gives an overview of the knowledge results as discussed in the previous paragraph. 49 Table 5 Knowledge Results Organizational Mission To provide quality learning and development products that enable our stakeholders to solve the nation’s most complex challenges. Organizational Global Goal By August 2017, 100% of participants will be satisfied with technical training. Stakeholder Goal By January 2017, 100% of instructors will apply effective learning engagement strategies during technical training classes. Verified Knowledge Influence Knowledge Influence Evidence Instructors do not know what effective engagement strategies are in instruction (D). Interview: “I use eye contact and real world examples” “I Engage the students by asking them questions” Observed: Students were not engaged; often times their eyes were glazed over by the long hours of lecture. The instructors often asked “any questions?” but did not allow time for students to respond. Instructors do not know how to implement engagement strategies (P). Interview: Instructors only gave two techniques: eye contact and using real-world examples 50 Observed: The two strategies that they used had limited effectiveness Motivation Results Discussion The major motivation themes that emerged in data analysis indicate that instructors do not have confidence in applying adult learning theory and applying student engagement methods, and that they do not value the importance of adult learning principles. When asked during the interview, “How confident are you using adult learning strategies?” Instructors gave an overwhelming response that they are very confident in teaching. Kelly said that “everyone who is a parent is a teacher.” He went on to say, “My confidence comes from talking about a subject that I have been doing for a long time.” Frederic had a similar response saying, “I think my confidence comes from grad school; I taught several different subjects.” All of the instructors made it clear that the experience they have, both in their jobs as engineers and in the time they have spent in the classroom, has given them the confidence to stand up in front of a room full of students and teach them the subjects about which they are considered world experts. Regardless of their confidence, the impression given to the researcher during the data analysis was one of overconfidence. The instructors’ observed behavior in the classroom is different than what they say they are doing. When asked the question, “What would make you more confident?” a majority of the instructors struggled with the answer and finally settled on “not much,” because they are very experienced: 51 I am not sure…I am very knowledgeable about the topic, I have a lot of experience working in this subject area and I keep current on what is going on. A few of the instructors said that they would like to have a better understanding of the students’ current knowledge; for example Linda said, “I would like a better understanding of the students; I would then be able to determine what I should focus on. I would talk about some things less and some things more, depending on how well I knew my students’ knowledge of the topic.” A valid response about what content to deliver students, but not the answer we would expect when talking about if she felt confident using adult learning theory and applying student engagement methods. Additionally, when asked where their confidence comes from, the majority of the instructors said it comes from their professional experience as engineers, and knowledge of the subject matter. Emily said, “It comes with the knowledge of the material and the experience teaching the material.” Linda had a similar response, saying, “I have confidence in my knowledge of the material and I have learned how to be self-aware during my class and I ask myself if I’m sending the right message.” Linda went on to say, “If I know the content I feel confident; I feel less confident with material I don’t know as well.” This answer became a theme among the instructors’ responses; it is an important concept going forward. The fact that the instructors as a whole feel that they are confident with content that they know, and less confident with material that they do not know, is a construct that we can use to describe their confidence with adult learning, student engagement, classroom management, and ultimately knowledge transfer. The instructor confidence construct as they applied it to course material: 52 If I know the “content” I feel confident; I feel less confident with “material” I don’t know as well. Based off the instructor’s responses of not having been trained in adult learning, student engagement, etc., and based off of the observations we can infer that these instructor’s knowledge of these concepts is weak. Therefore, we can apply the construct mentioned earlier and say that they feel less confident in adult learning, student engagement, classroom management, and ultimately knowledge transfer, because they don’t know that material well. Thus, they will not be as confident in applying those adult learning concepts as they are in applying their presentation skills. The instructor confidence construct applied to adult learning: I know how to give a presentation so I feel confident; I fell less confident with adult learning concepts, because I don’t know them as well. From the interviews we can say that the instructors are not aware of formal adult learning principles. Therefore, we can conclude that the instructors do not value the importance of adult learning principles, because they don’t know that they need to, and they do not know the benefits of these methods, because they have not been trained. The following chart gives an overview of the motivation results as discussed in the previous paragraph. 53 Table 6 Motivation Results Organizational Mission To provide high quality learning and development products that enable our stakeholders to solve the nation’s most complex challenges Organizational Global Goal By December 2016, SU will implement a formal assessment plan for all technical training. Stakeholder Goal By May 2016, the Leadership Team to develop an approach to improve trainer skills. Verified Motivation Influences Influence Evidence Instructors do not have confidence in applying adult learning theory and applying student engagement methods. Interview: “If I know the content I feel confident; I feel less confident with material I don’t know as well.” Observation: Instructors are not using student engagement methods. Instructors do not value the importance of adult learning principles. Interview: “I am not sure, I am very knowledgeable about the topic, I have a lot of experience working in this subject area and I keep current” Observation: Adult Learning Principles not currently used. 54 Organizational Results Discussion This section will discuss the data collected from all three sources – interviews, observations, and document analysis – as they relate to the organizational influences. The findings from this study indicate that there are two organizational influences that need to be improved. First, the analysis of the collected data indicates that the organization is not providing training; there is a lack of training in adult learning principles, student engagement practices, and knowledge transfer and classroom management techniques. During the interviews several of the instructors indicated that they received no training from the organization to be instructors. In fact, one of the instructors was told, “You’re a smart guy; you’ll figure it out.” Moreover one of the instructors said, “I haven’t received feedback saying that I need training.” The topic of training for the instructors is split, but not evenly and leaning towards more training. For example, one of the instructors said, “I’d like to see the organization develop a train-the-trainer course or workshop.” But it is not just a class that these instructs want; it’s also reference material. One instructor said, “A workbook would be good, so we could self-teach or formal training would be a good idea.” After analyzing the collected data a revelation occurred to the researcher. Giving these instructors training, job aids and exposure to adult learning, engagement techniques and other learning delivery options other than lecture, will increase student engagement and ultimately knowledge transfer. The second organizational influence affecting the organization in meeting its goals is that there is a culture of working in silos rather than collaboratively. A review of SU documents shows that department documents are not aligned and/or are outdated. The researcher reviewed several organizational documents, process and procedure documents, charters, statement of works, etc. The researcher also reviewed internal websites, released emails, SharePoint sites, 55 and calendars that SU has to support this conclusion. These artifacts are important because they show the direction of the departments and how they interact with each other. They describe what SU’s mission and goals are, and they describe how the different departments support those goals. These artifacts describe what the work is, who is doing the work, how the work is being done, and how the different departments are working together. 56 Table 7 Organizational Results Organizational Mission To provide quality learning and development products that enable our stakeholders to solve the nation’s most complex challenges. Organizational Global Goal By June 2017, The Space Institute will implement a formal assessment plan for all space technical training. Stakeholder Goal By October 31, 2016, The Space Institute will improve the quality of instruction provided by our space technical instructors. Verified Organizational Influences Organizational Influences Evidence Organization is not providing training; there is a lack of training in adult learning principles, student engagement practices, knowledge transfer, and classroom management techniques Interview: “I’d like to see the organization develop a train- the-trainer course or workshop.” “A workbook would be good, so we could self- teach or formal training for employees.” Observation: Application of adult learning principles, student engagement practices, knowledge transfer, and classroom management techniques were not consistent across the instructors observed. Document Analysis: 57 Summary Part of the analysis process is to synthesize the data collected from the three different methods of collection: instructor interview, classroom observation, and document analysis. As stated earlier, analysis of the collected data indicates that the organization is not providing training; there is a lack of training in adult learning principles, student engagement practices, knowledge transfer, and classroom management techniques. During the interviews one instructor said they received no formal instructor training, and a few of them recommended that the organization provide it. The interviews and document analysis showed inconsistencies, and implies that there is a culture of working in silos rather than collaboratively. Although the instructors may have had a lack of formal education in instruction they have excellent presentation skills and they try different strategies to keep their students engaged. The researcher also, has to give them credit for putting themselves in a situation where they have not been formally trained and are excited to participate. Reviewed documents do not support that 100% of the instructor staff has taken the instructor workshop. 58 CHAPTER 5: FINAL RESULTS Recommendations for Practice to Address KMO Influences The knowledge influences in Table 8 represent the complete list of knowledge influences, based on the most frequently mentioned knowledge influences to achieving the stakeholders’ goal. During the informal interviews and supported by the literature review, Clark and Estes (2008), suggest that declarative knowledge is often necessary to know before applying it. This holds true to the teaching realm and it is necessary for instructors to have the appropriate knowledge to ensure the achievement of their stakeholder goals. Table 8 shows the recommendations for these influences based on theoretical principles. It is not uncommon for subject matter experts to be asked to become an instructors because of their technical expertise even if they have not had formal training in instructional skills. The following recommendations are offered to better support the instructors and to help them achieve their goals. Any organization would be lucky to have a team of instructors, like the one the researcher observed that are as enthusiastic to participate as instructors. 59 Table 8 Summary of Knowledge Influences and Recommendations Assumed Knowledge Influence: Cause, Need, or Asset* Validated Yes, High Probability, or No (V, HP, N) Priority Yes, No (Y, N) Principle and Citation Context-Specific Recommendation Instructors do not know what effective engagement strategies are in instruction (D) V Y Knowledge about something is necessary before you apply it. (Clark and Estes, 2008) Provide training on effective engagement strategies Instructors do not know how to implement engagement strategies. (P) V Y Job aids enable learning by providing important instructions and other reference materials, thereby reducing the mental workload required to apply new skills to the workplace (Grossman, 2011). Provide training on how to implement effective engagement strategies. Instructors do not reflect on their own effectiveness in the classroom. (M) V N They can use this ability to alter the ways in which they think and operate. A commonly used approach 1. Provide training on how to reflect on their own 60 to assess both metacognitive knowledge and metacognitive control is to ask what they know or what they do while engaging in particular cognitive activities (Baker, 2006; Krathwohl, 2002). effectiveness in the classroom, 2. Institute a self- evaluation (Management to ensure the necessary reflection is taking place.) * Note: (D) Declarative Knowledge, (P) Procedural Knowledge, and (M) Motivation 61 Declarative Knowledge Solutions, Needs, or Assets. Declarative knowledge refers to factual knowledge and information that a person knows (n.d., 2017). Instructors at SU need to know the fundamentals of classroom instruction for adult learners. For example, the instructor of a technical course that covers the history of scientists who specialize in astronomy will need to know different student engagement strategies to keep the attention of their students than an instructor of a structure mechanics course that covers material core density. Student engagement is not the only skill that SU instructors need to master. Instructors who are successful at keeping their students engaged will increase the amount of knowledge transfer to their students. SFS employs highly educated engineers and scientists who are considered domain experts. The nature of the work at SFS exposes employees to a variety of different projects, allowing them to develop knowledge and skills outside of their expertise. These employees take SU courses in these subjects to enhance their capability in the workplace. However, oftentimes employees will take a SU courses and find what they learned in the field is different than what is being taught, and will challenge the instructors. This scenario is common, and SU instructors need to know how to manage these difficult students, and keep the class on schedule. Procedural Knowledge Solutions, Needs, or Assets Procedural knowledge is knowing how to perform certain activities as described in Clark and Estes (2008). For example, an instructor can lecture for 30 minutes and then ask their students questions to check on knowledge, or break the students into small groups to discuss a topic and ways to apply the content in their role at work. SU instructors also need to know how to manage their classrooms, not just what classroom management is. Among other things, instructors need to know how to respond to a 62 difficult student, when to change the pace of the class depending on the verbal and nonverbal cues from the students, how to plan the ratio of lecture to learning activities and practice exercises, when to take breaks, etc. Instructors also need to know how to keep the class moving so that they are able to cover the entire required topic within the allotted time frame. Metacognitive Knowledge Solutions, Needs, or Assets Metacognition is “cognition about cognition,” “thinking about thinking,” or “knowing about knowing,” and higher order thinking skills. In the case of SU instructors we are talking about reflection on performance. SU instructors need to reflect after each class, not only to improve their teaching capability, but also to improve the learning potential for their next course offering. It is important for an instructor to reflect on their class. An instructor can write down the different questions they receive during the instruction so that they can add or change course content. They can identify when students get lost or frustrated, and make changes to examples or exercises. Reflection provides instructors an opportunity to go over what worked and what did not work so they can change their strategy and approach to increase the next classes learning potential. Training the instructors could involve train-the-trainer classroom instruction on student engagement, with activities and opportunities for each instructor/student to obtain feedback from fellow instructor/students; this training should include a Kirkpatrick level 2 measurement to determine the individual instructor’s success or need for remediation. Once classroom training is complete, each instructor should return to training, and receive feedback from observations and individual coaching with an experienced mentor or master trainer. Student feedback could also be used to give the instructor another point of view on the effectiveness of the instruction. 63 Motivational Recommendations The motivation influences in Table 9 represent the complete list of validated motivational influences. The most frequently mentioned motivation influences on achieving the stakeholders’ goal, during informal interviews and supported by the literature review and the review of motivation theory. Clark and Estes (2008) suggest that there are three indicators of motivation in task performance – choice, persistence, and mental effort. Choice is going beyond intention to start something. Persistence is continuing to pursue a goal in the face of distractions, and mental effort is seeking and applying new knowledge to solve a novel program or perform a new task. As all instructors have chosen to deliver training, the assumed causes of lower quality instruction appear to suggest instructors lack persistence and mental effort in their execution of adult learning, student engagement, and classroom management. However, adding the additional duties of an instructor doesn’t mean you’re automatically prepared for it. Perhaps the lack is not in persistence and mental effort, but in knowledge and understanding of basic instructional techniques. Many people see teaching as something they could easily do, not realizing that it involves so much more than standing in front of a group of people and talking. As such, as indicated in Table 7, some motivational influences have been validated and have a high priority for achieving the stakeholders’ goal. Table 7 also shows the recommendations for these influences based on theoretical principles. 64 Table 9 Summary of Motivation Influences and Recommendations Assumed Motivation Influence: Cause, Need, or Asset* Validated Yes, High Probability, No (V, HP, N) Priority Yes, No (Y, N) Principle and Citation Context-Specific Recommendation Instructors do not have confidence in applying adult learning theory and applying student engagement methods. V Y Provide instructional support (scaffolding) early on, build in multiple opportunities for practice and gradually remove supports (Pajares, 2006). Provide interactive training Instructors do not value the importance of adult learning principles V Y Individuals are more likely to engage in an activity when it provides value to them. (Eccles, 2009). Provide recognized industry best practices during training 65 SU instructors do not have confidence applying adult learning theory and student engagement methods during their classes. Ambrose (2010) suggests that organizing and rehearsing desired behavior overtly is effective at achieving observational learning. Moreover, Bandura (1997) explains that self-efficacy is increased as individuals succeed in a task. This would suggest that providing instructors with classroom instruction in which they can practice adult learning and student engagement methods will increase their self-efficacy. The recommendation then is for the organization to provide training on adult learning and student engagement techniques and provide opportunities for them to practice these techniques in a safe environment: the classroom. Clark and Estes (2008) state that “beliefs are (almost) everything” suggesting that when individuals have positive beliefs about their ability to do something they are more likely to pursue the goal and increase performance. Bandura (1997) stated that teachers’ perceptions of efficacy depend on more than their ability to teach subject matter. The author goes on to state that teachers’ effectiveness is, in part, determined also by their efficacy beliefs in maintaining classroom discipline that establishes an environment of learning, in using resources, and in supporting stakeholders efforts to help their students learn. SU instructors do not value the importance of adult learning principles, simply because they are not aware of them. Eccles (2009) suggests that people are more likely to engage in an activity when it provides a value to them. Applying Eccles’ concept and showing SU instructors that there is value applying adult learning principles should provide motivation for the instructors to apply those principles to their classes. The recommendation then is to provide SU instructors training that demonstrates improved student engagement and knowledge transfer resulting from careful planning and execution of adult learning principles, student engagement methods, and 66 classroom management techniques. This could be followed by subsequent classroom data collection that shows their own increase in effectiveness. Eccles et al. (1983) defined and measured expectancies for success as beliefs about how well they will do on upcoming tasks, either in the immediate or longer term future. Eccles (2009) stated that individuals’ choice, persistence, and performance can be explained by their beliefs about how well they will do on the activity and the extent to which they value the activity. The author goes on to state that individuals are influenced by their own perceptions from previous experiences and a variety of socialization influences. The organizational influences in Table 10 represent the complete list of assumed organizational influences and their probability of being validated, based on the most frequently mentioned organizational influences in achieving the stakeholders’ goal during informal interviews and supported by the literature review and the review of organization and culture theory. Clark and Estes (2008) suggest that organization and stakeholder goals are often not achieved due to a lack of resources, most often time and money, and stakeholder goals that are not aligned with the organization’s mission and goals. Gallimore and Goldenberg (2001) propose two constructs about culture: 1.) cultural models, or the observable beliefs and values shared by individuals in groups; and 2.) cultural models, or the settings and activities in which performance occurs. Both constructs about culture are known as cultural models. Thus, both resources and processes, and cultural models and settings, must align throughout the organization’s structure to achieve the mission and goals. As such, as indicated in Table 10, some organizational influences have a high probability of being validated and have a high priority for achieving the stakeholders’ goal. Table 8 also shows the recommendations for these influences based on theoretical principles. 67 Table 10 Summary of Organization Influences and Recommendations Assumed Organization Influence: Cause, Need, or Asset* Validated Yes, High Probability, No (V, HP, N) Priority Yes, No (Y, N) Principle and Citation Context-Specific Recommendation Organization is not providing training V Y Clark and Estes (2008) state that when policies and procedures are aligned and communicated from the top with all stakeholders, organizational performance increases. 1. Establish an instructor train the trainer policy 2. Establish procedures for instructional designers and instructors to follow There is a culture of working in silos rather than collaboratively. V Y Organizational performance increases when individuals communicate constantly and candidly to others about plans and processes (Clark & Estes, 2008) 3. Use collative collaborative project management tools 68 Process and Procedures SU is not providing training to their instructors; this includes a lack of training in adult learning principles, student engagement practices, knowledge transfer, and classroom management techniques. Clark & Estes (2008) suggest that organizational performance increases when individuals communicate constantly and candidly to others about plans and processes. This suggests that SU needs to establish a policy that has instructors attend a train- the-trainer program that covers adult learning principles, student engagement practices, knowledge transfer, and classroom management techniques. Moreover, this suggests that procedures need to be established for instructional designers and instructors to follow pre and post class discussions (what worked and did not work) focusing on best practices for adult learning principles, student engagement practices, knowledge transfer, and classroom management techniques. Clark and Estes (2008) state that when policies and procedures are aligned and communicated from the top with all stakeholders, organizational performance increases. Lee (2010) explains that training solutions are expensive, and there are several factors that influence why some students learn and some do not; the instructor can play a major role in effective learning transfer and subsequent job performance. The author suggests that if organizations do not require instructors to continually improve and refine their skills, they risk putting unprepared, ill-equipped, and unknowledgeable employees in the workforce. Lee (2010) also suggests that when employees receive unclear instruction about job expectations, productivity can suffer. As such, it appears that the literature would support the necessity for establishing processes and procedures to ensure that SU students receive quality instruction. 69 Cultural Support SU has a culture of working in silos rather than collaboratively. Clark & Estes (2008) suggest that organizational performance increases when individuals communicate constantly and candidly to others about plans and processes. This suggests that SU needs to use collaborative project management tools, SharePoint workspaces and other internal social media to reinforce communication outside of team meetings. In a study of medical staff, Blum et al. (2005) examined the nature of communication and strategies to increase situational awareness. Blum et al. (2005) explains that effective team communication and collaboration improve problem solving and increase performance. By implementing a debriefing communication strategy they found that encourages work-based learning, collaboration, teamwork, and improved communication. The literature suggests that effective communication is essential for highly productive teams. An approach would be to have a monthly or quarterly trainers’ roundtable to discuss classroom challenges; ah-ha moments; and best practices in design, development, delivery, and effective classroom management. Integrated Implementation and Evaluation Plan The New World Kirkpatrick Model (Kirkpatrick & Kirkpatrick, 2016) was used as reference for this evaluation and implementation plan, which is based on the original Kirkpatrick Four Level Model of Evaluation (Kirkpatrick & Kirkpatrick, 2006). The New World model suggests that evaluation plans start with the goals of the organization and work backwards. With this approach, the “leading indicators” that connect recommended solutions to the organization’s goals are both easier to identify and more closely aligned with organizational goals. Moreover, this “backwards planning” of the New World Kirkpatrick Model allows for a sequence of three other actions: first, the development of solution outcomes that focus on assessing work 70 behaviors; next, the identification of indicators that learning occurred during implementation; and finally, the emergence of indicators that organizational members are satisfied with implementation strategies. Drafting the evaluation and implementation plan in this method forces connections between the tactical solutions and the overall strategic goal and solicits “buy in” to ensure success (Kirkpatrick and Kirkpatrick, 2016). It is important to talk about the significance of the relationship between the evaluation and implementation and the solutions mention earlier. As described in Kirkpatrick (2016), the evaluation of the program that is being implementation needs to be built into the program from the onset. If not, it is added as an afterthought and key leading indicators of success are often not identified, as well as critical behaviors (Kirkpatrick, 2016). Using Kirkpatrick’s 4 level model, identifying the level 4 results, and working backwards ensures that the solutions that we desire are focused and that the program is measurable to indicate success. Level 4: Results and Leading Indicators Table 11 shows the proposed Level 4: Results and Leading Indicators in the form of outcomes, metrics and methods for both external and internal outcomes for SU. If the internal outcomes are met as expected, as a result of the training and organizational support for an instructor improvement program, then the external outcomes should also be realized. 71 Table 11 Outcomes, Metrics, and Methods for External and Internal Outcomes Outcome Metric(s) Method(s) External Outcomes Increase the number of classes using engagement strategies available to the organization The number of classes using engagement strategies available at a given time Instructors will self-evaluate, students will provide an evaluation, and the ISD will observe and provide an evaluation. Increase the knowledge transfer rate for students participating in SU courses 1. The average achievement rating (Kirkpatrick level 2). 2. The use of new knowledge and skill on the job (Kirkpatrick level3). 1. Student instructors will conduct no-credit quizzes before, during, and after class. 2. SU will follow up with managers to determine whether new knowledge and skills are being used once students return to work after training. Internal Outcomes Instructors capable of delivering an engaging class using adult learning strategies 1. Student assessment of instructor’s knowledge, preparedness, and skill in 1. Collect Kirkpatrick level 1 metrics for every class; compile and document changes in students’ reaction to training over time, as 72 providing an engaging classroom experience (Kirkpatrick level 1). instructors put new engagement behaviors into practice. Instructors capable of delivering an engaging class using adult learning strategies 1. Observation and evaluation of instructor’s use of engagement strategies and tactics during class. 1. Random observation and evaluation by SU management/course leads of instructors delivering training. Increased student satisfaction Positive/ negative feedback survey from students and course leads Compare course survey results 73 Level 3: Behavior Critical behaviors. The stakeholders of focus are the SU instructors conducting Space Technical courses. The first critical behavior is that instructors must engage students with questions, exercises, and activities. For example, having 3 to 4 questions to ask the students after each module, having at least one activity for each module (group or individual) and having a capstone at the end of the class that demonstrates the students’ gained knowledge of the courses learning objectives. The second critical behavior is that they must properly manage the classroom. What does that mean? The instructor needs to be able to handle a challenging student effectively, whether that student is disruptive or not engaged. The instructor also needs to know how to approach support instructors to ensure that they are aligned with what is being taught, but also to ensure that they are not a distraction or disruptive to the class. Finally, the instructor needs to manage the tempo of the class, which includes managing the class time and agenda to ensure that the students stay on track. The third critical behavior is that they must engage the students by establishing eye contact and adjust course activities when engagement starts to slip. The specific metrics, methods, and timing for each of these outcome behaviors appears in Table 12. Required drivers. Instructors require the support of SU and the organization to reinforce what they learn in the training and to encourage them to apply what they have learned to deliver quality instruction for Space Technical courses. Therefore, a number of actions immediately following the conclusion of the program and in the months to follow were put into practice to drive and maximize on-the- job application and results. 74 Table 12 Strategies for Training of Instructor-Led Training Critical Behavior Metric(s) Method(s) Timing 1. 1. Engage students with questions, activities, and exercises linked to the learning objectives of the course. The ratio of time spent asking and answering questions compared to the time spent in lecture The course lead shall inject questions that the instructors will ask in the course material. When the instructor comes to that slide the instructor will ask the class that question. The instructor will ask the class at least 3 questions for every 15 minutes of lecture 2. The instructor will give the class appropriate breaks and lunches. The number of breaks in the class. (one break of at least 10 minutes per hour [or 90 minutes] of instruction The course lead will design into the course agenda breaks and lunches The instructor will give 2 breaks in the morning, 1 lunch at noon, and 2 breaks in the afternoon. 75 3. The Instructor will engage the students by establishing eye contact and adjusting class activities when engagement starts to slip. Number of students not paying attention in class. Instructors and co-instructors will watch students and communicate with each other when students start to get lost or become distracted. Continues throughout the class 76 Table 13 Required Drivers to Support Critical Behaviors Method(s) Timing Critical Behaviors Supported: 1. Engaging students with questions, exercises, and activities 2. Taking appropriate breaks 3. Establishing eye contact and adjusting activities when engagement slips Reinforcing Team meetings with experienced Instructors to design or redesign course Quarterly 1, 2 Job aid including checklist to prepare for class Ongoing 1, 2 Use SharePoint and other internal social media to reinforce communication outside of the team meeting Ongoing 1, 2, 3 77 Team meeting to troubleshoot collaboratively and for additional training Quarterly and as needed 1, 2, 3 Encouraging Feedback and coaching from course lead and ISD Ongoing 1, 2, 3 Rewarding Random observations using checklists and a performance dashboard. Ongoing 1, 2, 3 78 Monitoring Three strategies could be used to ensure that the Required Drivers occur: a) SU can create opportunities at All-Hands meetings and/or appreciation luncheons to share success stories and present awards for outstanding achievements and most improved; b) after completion of an instructor improvement program workshop provide the instructor an opportunity to practice what he/she has learned. Training should always be just-in-time, to improve the learning curve and reduce the forgetting curve. SU can ask instructors to self-report their confidence and self- efficacy in instructor-related tasks; and c) the instructional designer can assess the performance of the instructor during the delivery of a course. Frequent, quick checks can help the organization monitor progress and make adjustments if results do not match expectations at that time. Level 2: Learning Learning goals Following completion of the recommended solutions, most notably the Instructor Improvement Program’s workshop, the instructors will be able to: 1. Demonstrate knowledge of general adult learning theory with 90% accuracy. (D) 2. Demonstrate knowledge of general engagement strategies with 90% accuracy. (D) 3. Demonstrate knowledge of general classroom management principles with 80% accuracy. (D) 4. Demonstrate basic instructional systems design (ISD) principles with 90% accuracy. (D) 5. Classify and interpret body language to determine if a shift in classroom activity is needed to re-engage the students. (P) 79 6. Create learning objectives and learning outcomes for each course that they are instructing. (P) 7. Create questions linked to course learning objectives and outcomes (D, P) 8. Create an appropriate course agenda for a 1-day, 6-hour course; a 2-day, 12-hour course; and a 3-day, 18-hour course. (D, P) 9. Indicate confidence level (i.e. 90 % confident) that they can apply what they learned accurately and within the prescribed time limits. (Confidence) 10. Value the accuracy and timeliness of the instructional design process. (Value) Program The learning goals listed in the previous section will be achieved through an instructor improvement program that addresses adult learning theory, student engagement, knowledge transfer, and classroom management principles. The instructor will study a broad range of topics pertaining to the design and delivery of the technical courses. The program is blended, consisting of e-learning modules, one face-to face application workshop, and mentoring and coaching from instructor/designers and senior instructors. The instructor improvement program is a continuous improvement program and active instructors will always be a part of the program; however, the total time to complete the initial eLearning and workshop is 22 hours. During the asynchronous e-learning modules, learners will be provided an online workbook of key terms and references pertaining to adult learning theory, engagement strategies, knowledge transfer, and classroom management principles, as well as a chart of different types of exercises and activities they can build into their courses. Another job aid will contain a decision flow chart for recognizing unengaged students, and options to get them re-engaged. The job aids will be demonstrated on video using authentic SU classroom scenarios, and key 80 terms will be defined with examples. The e-learning module will have built-in questions to check learners’ understanding throughout the learning activity. Following the e-learning module, the learners will be provided the opportunity to practice using the job aids, and to receive feedback from peer review, and review by the workshop instructors. The demonstrations, practice, and feedback approach will also be used to train engagement strategies and classroom management principles. During the synchronous in-person sessions, the focus will be on applying what learners have learned asynchronously to design and deliver a course module using adult learning theory, engagement strategies, knowledge transfer, and classroom management principles in training groups, role-playing, discussions, peer modeling, and teaching back to each other. More experienced reviewers will also discuss the value and benefits of being engaging and knowing when to switch up engagement strategies to ensure that knowledge transfer is taking place. Components of learning Demonstrating declarative knowledge is often necessary as a precursor to applying the knowledge to solve problems. Thus, it is important to evaluate learning for both declarative and procedural knowledge being taught. It is also important that learners value the training as a prerequisite to using their newly learned knowledge and skills on the job. However, they must also be confident that they can succeed in applying their knowledge and skills and be committed to using them on the job. As such, Table 14 lists the evaluation methods and timing for these components of learning. 81 Table 14 Components of Learning for the Program Method(s) or Activity(ies) Timing Declarative Knowledge “I know it.” Knowledge checks using multiple choice. In the asynchronous portions of the course, throughout the e- learning module Knowledge checks through discussions, “pair, think, share” and other individual and group activities Periodically during the live instructor-led workshop and documented via observation notes Procedural Skills “I can do it right now.” During the asynchronous portions of the course using scenarios with multiple-choice. In the asynchronous portions of the course, at the end of each module/lesson/unit Demonstration in groups and individually using the job aids to successfully perform the skills. During the in-person training workshop 82 Using feedback from peers during group sharing in order to make immediate adjustments to improve delivery During the workshop and during dry runs for live courses Individual application of the skills with authentic SU scenarios. Immediately at the end of the workshop Retrospective pre- and post-test assessment survey, asking participants about their level of proficiency before and after the training. At the beginning, during, and at the end of the workshop Attitude “I believe this is worthwhile.” Data showing value of engagement techniques, as well as case studies from actual SFS learning scenarios. (This helps answer the “why” that adult learners need to understand before accepting new knowledge and integrating new attitudes.) In the asynchronous portions of the course, throughout the e- learning module, and periodically during the instructor-led training workshop Instructor's observation of the students’ statements and actions demonstrating that they see the benefit of what they are being asked to do on the job. During the workshop 83 Discussions of value of what they are being asked to do on the job, and how they can apply these techniques to improve their own performance. During the workshop Pre- and post-test assessment item At the beginning, during, and end of the workshop Confidence “I think I can do it on the job.” Survey items using scaled items Following each of the modules/lessons/units in the asynchronous portions of the course Discussions following practice and feedback During the workshop and dry runs for live courses Retrospective pre-and post-test assessment item At the beginning, during, and end of the workshop Commitment “I will do it on the job.” Discussions of key concepts and soliciting feedback (from the instructor and fellow learners) During the workshop and dry runs for live courses Create individual action plans During the Workshop Retrospective pre-and post-test assessment At the beginning, during, and end of the workshop 84 Level 1: Reaction Table 15 shows the proposed Level 1: in the form of components to measure reactions to the program for both method(s) or Tool(s) and timing. Table 15 Components to Measure Reactions to the Program Method(s) or Tool(s) Timing Engagement Data analytics that track students’ information and course information in the Learning Management System. Ongoing during the asynchronous portion of the course. Completion of online modules/ lessons/ units. Ongoing during asynchronous portion of the course. Observation by instructor. During the workshop. Attendance. During the workshop. Course evaluation. After the workshop. Relevance Targeted questions about the workshop, built into the check on learning via survey (online) and discussion (ongoing). After every module/lesson/ unit and the workshop. 85 Course evaluation After the workshop. Customer Satisfaction Targeted questions built in the check on learning via survey (online) and discussion (ongoing). After every module/lesson/ unit and the workshop. Course evaluation After the workshop 86 Evaluation Tools Immediately following the program implementation. During the asynchronous portion of the course, the reporting tool in the learning management system (LMS) will collect data about the start, duration, and completion of modules by the students. This report will indicate the level of engagement with the course material. The e-learning modules administer brief surveys after each section requesting the student to indicate the relevance of the material to their job performance, and their overall satisfaction with the content and delivery of the online course. For Level 1, during the in-person training workshop, the instructor will conduct periodic checks by asking the students about the relevance of the content to their work as an instructor and the organization, delivery, and learning environment. Level 2 will include checks for understanding effective use of exercises and activities, or competition among groups in responding to questions and scenarios drawn from the content. 87 Table 16 Evaluation Tools Evaluation Tool Level 1 and 2 during and immediately following the program implementation Declarative Knowledge Item Know what student engagement is. Given multiple choices, be able to select the right answer. Know the definition of Adult Learning Theory. Given multiple choices, be able to select the right answer Know and understand the learning objectives. The content was relevant and easy to follow Procedural Knowledge Keep students engaged. Be able to demonstrate student engagement methods by reacting to students in a classroom during a period of instruction. Attitude Knowledge gained will help with their job. Rate on a scale their ability to do their job as a result of taking this workshop Confidence 88 Demonstrate confidence in what was learned. Declaration that they are clear with what is expected of them as a result of taking this workshop Understand where they can find additional help. Declaration that they know where to go for additional resources and help Commitment Demonstrate a commitment to use strategies learned in the workshop Explain the first thing they are going to implement from what they learned in the workshop 89 Table 17 Blended Evaluation Blended Evaluation Scale Highly Disagree > Highly Agree Evaluation Item L1: Reaction Survey: What I learned in the workshop has been very valuable in my ability to keep students engaged L2: Learning Survey: I was able to keep students engaged during my course after the workshop than before the workshop. L:3 Behavior I ask questions and vary activities during my class when I see that my students are not engaged. L4: Results I am able to keep students engaged for each of the different courses that I teach. 90 To ensure the success of SU instructors it is necessary to gauge their progress through all four levels of the Kirkpatrick model. The first two levels as depicted in table 15 shows above will utilize a survey that is targeted to gain buy-in to the program and to determine if the instructors see the information given to them will add value to the classes that they teach. It also allows them to reflect and self-evaluate and asses whether or not they are applying what they learned in the workshop. Levels 3 and 4 will use observation to gauge if there is a behavior change and to ultimately see if the instructor’s behaviors meets the overall desired outcomes that are expected from the program. Through random observation SU leadership will observe each instructor to evaluate how they react to students that are not engaged, and to see if the instructor keeps their students engaged with questions, activities, and exercises. The Level 4 goal of increasing the number of engaging courses is measured by instructor self-evaluation, student evaluation, and independent SU observation evaluations. For, each course, SU will track the number of evaluations processed, analyze the evaluation results, and make recommendations for improvement if necessary to the course ISD and report findings to the SU leadership team. A dashboard report of these measures will be generated as a monitoring and accountability tool. Similar dashboards will be created to monitor Levels 1, 2 and 3. See table xxx on the following page. Summary The book the New World Kirkpatrick Model was used to plan and develop the above recommendation with a built in evaluation for the IIS program. When developing these recommendations the researcher started at level 4 results and worked backwards. The above recommendations are optimize to achieve the stakeholder goal and the organizational goal as 91 seen in Table 15. This study was an evaluation of the instructors to confirm that a train-the- trainer program is needed. It is expected that using the KMO framework with a Needs analysis and then using the New World Kirkpatrick Model that the recommendations will allow the organization to meet its goals and improve the quality of instruction provided by the instructors, ultimately providing the corporation the added value of a workforce with the skills and competencies needed to meet the needs of the corporation’s customers. Strengths and Limitations One strength of this study is the rigorous use of an in-depth descriptive qualitative method used to examine the mix of gathered data from interviews, observations, documents and artifacts. However, this study has some limitations. The most significant limitation was that the study was conducted at a single training location and included instructors from that location. Moreover, only one program offered at the organization was the focus of this study. Thus, there is a need for additional investigations expanding the scope of interviews beyond that of the instructors. Interviews with students, Instructional Designers, and organizational leadership would add significantly to the understanding of the dynamics at play within the university. Conclusion This study showed Spartan University’s instructor’s improvement program has been ineffective since its implementation. The purpose of this study was not to place blame on the instructor’s lack of adult learning knowledge, but rather to identify their needs so that the organization can provide them better support. This study identified through interviews, observations, and document analysis is that SU’s instructors need to be formally trained on adult learning, student engagement, classroom management, and knowledge transfer. Based off the findings, it is recommended that the current instructor improvement program needs to be 92 overhauled using professional development and mixed learning best practices. This approach effectively teaches instructors the skills that they need to be successful in the classroom. This includes knowing the who, what, where, when’s of adult learning, student engagement, classroom management, and knowledge transfer. The proposed instructor program creates the best learning outcomes. Future research should expand the scope of programs at the university to get a better understanding of skills needed across the portfolio. It is further recommended that research be expanded to include instructional design and curriculum development. 93 APPENDIX A Interview Protocol The researcher will read the non-colored portion of the text below. Pretext discussion and disclosures The purpose of this interview is for me to gain a better understanding of your experience as a SU Instructor. I am particularly interested adult learning, student engagement, classroom management, and transfer of knowledge. I want to hear your story, and perspective of SU classes. During the interview, I will limit my comments because I want to center my attention on your story. I may ask clarifying questions from time to time in order to explore a particular subject. That being said, my plans to listen to you as you describe your experiences. And, if you have a question at any point, do not hesitate to ask. With your permission, I would like to record this interview. The purpose of recording is so that I can actively listen to you and not get overwhelmed taking notes. Although, I might make small notes to capture my own thoughts. And as described in the consent form, everything covered here today is confidential. After I review the transcriptions of the interviews, is it ok to contact you to ask clarifying questions. [Thank the interviewee should permission be granted]. Do you have any questions? [Answer all questions posed.] Can I offer you something to drink? I also have cookies If you like. KMO Framework  K o Instructors need knowledge of what effect engagement strategies in instruction o Instructors need to know how to implement engagement strategies. o Instructors need to know how to reflect on their own their instruction  M o Self-efficacy 94 o Expectancy Value  O o Organization is not providing enough support or training; there is a lack of training and support for engagement practices o Instructors do not trust SU Staff. K-Questions: 1. How do you prepare for class? a. What do you think about before your instruction? 2. What strategies do you keep your students engaged and motivated? 3. What teaching methods do you find to be the most effective to keeping students engaged? M-Questions: 1. How confident are you using various engagement strategies while teaching? a. Where does that confidence come from? b. What do you think will make you more confident? 2. How important do you think implementing engagement strategies are in technical instruction Probes/Prompts—[To be drawn upon as necessary throughout the interview] i). Could you please tell me more about…? ii). Could you give me some examples? iii). Can you give me an example of a moment when…? iv). How do you do that? v). How did you feel when…? vi). What do you mean by…? vii). This is what I heard, is that what you said? [Paraphrase what I think was said] viii). If I were watching you do this, what would I see? ix). What were other people doing? 95 x). What was significant about this for you? xi). How did that come about? xii). Is there a metaphor or image that best describes this? [Notion/idea/moment/feeling] xiii). Is there another way to describe this…? Concluding Remarks: Before I let you go, do you have any questions? I want to thank you time and patience with me today. Please do not hesitate to email me with any questions or additional information that you feel may help me better understand what we discussed today. Again, thank you. 96 APPENDIX B Observation Protocol Observing First priority is the Process, the students and the safety of the students. The investigation is secondary to that. As the researcher I have to be mindful of the impact my presence can have on the Process. We want to minimize this impact.  Interaction with the Process Teachers o Minimal o Give them space to do their jobs o Direct all questions through the instructor or SU staff member  Invisibility o Be on time for sessions—It is not ok to come in late o Quietness, No paper shuffling o Neutral expression o Be natural if your eyes meet students’ eyes, but do not seek them out  Awareness of where students are o Not engaged (i.e... Reading their email, doing other work related activities, playing a game on their phone, talking to a coworker other than that of what the teachers is talking about) o Awareness of “inside jokes”  Being in the classroom o When in the classroom attention is ALWAYS on what is going on in the Process o No side conversations , No joking 97 o Be a ninja, be Invisible o You are there as a witness to the learning experience o Always assume students and instructors eyes are open and on you  If students approach you o Be natural, and pleasant, but maintain a distance o Do not talk to them about what you observe about them, o Answer simple questions…where do you live, where’s the bathroom. Any other questions, direct them to an instructor. As the researcher I’m looking to document the following while in the classroom. The purpose of observing these instructors is to document what the instructor actually does and compare that to their answers to their interview questions. The observational team is focusing on the instructors and how well they incorporate adult learning in their classrooms and what their behaviors are during instruction. K-Observations: 1. Are the instructors evaluating their students to measure knowledge transfer or learning? And how? a. Do you give you student’s pre and post surveys or questions to gauge learning? b. Check on knowledge? 2. Are the instructors using adult learning strategies to keep students engaged and motivated? 3. What teaching methods are the instructors using? Are they effective? Running head: IIS 98 The Following chart will be referenced to determine if the proper adult learning strategies is used. Running head: IIS 99 The Following chart will be referenced to determine if the proper adult learning strategies is used. Running head: IIS 100 APPENDIX C Consent Information Sheet University of Southern California Rossier School of Education University of Southern California, Los Angeles, CA 90089 INFORMATION/FACTS SHEET FOR EXEMPT NON-MEDICAL RESEARCH Instructor Skills Assessment You are invited to participate in a research study. Research studies include only people who voluntarily choose to take part. This document explains information about this study. You should ask questions about anything that is unclear to you. PURPOSE OF THE STUDY The purpose of this study is to understand how well instructors are engaging students in the classroom and how effective knowledge transfer and adult learning techniques are being utilized. PARTICIPANT INVOLVEMENT If you agree to take part in this study, you will be asked to participate in a 30 minute interview and be observed during a class session. You do not have to answer any questions you don’t want to; you do not have to be observed if you do not want to; if you don’t want to be interviewed or observed, you cannot participate in this study. PAYMENT/COMPENSATION FOR PARTICIPATION You will not be compensated for your participation; however, you will receive a letter of appreciation with a carbon copy sent to your supervisor. ALTERNATIVES TO PARTICIPATION Your alternative is to not participate. Your relationship with your employer will not be affected whether you participate or not in this study. CONFIDENTIALITY Any identifiable information obtained in connection with this study will remain confidential. Your responses will be coded with a false name (pseudonym) and maintained separately. The data will be stored on a password protected computer in the researcher’s office for three years after the study has been completed and then destroyed. The members of the research team, and the University of Southern California’s Human Subjects Protection Program (HSPP) may access the data. The HSPP reviews and monitors research studies to protect the rights and welfare of research subjects. IIS 101 When the results of the research are published or discussed in conferences, no identifiable information will be used. RESEARCHER CONTACT INFORMATION Principal Researcher Principle Researcher via email at Principle_Researcher@usc.edu or phone at (xxx) xxx-xxx IRB CONTACT INFORMATION University Park Institutional Review Board (UPIRB), 3720 South Flower Street #301, Los Angeles, CA 90089-0702, (213) 821-5272 or upirb@usc.edu Running head: IIS 102 APPENDIX E Consent Information Sheet <DATE> Spartan Flight Systems Employee “Name” 1300 Pennsylvania Ave. Washington DC, 20229 Re: Instructor Skills Assessment; Principle Researcher Dear The Space Corps Employee: I am writing to let you know about an opportunity to participate in a research study about Instructor skills. This study is being conducted by Principle Researcher at the University of Southern California, Rossier School of Education. The purpose of this study is to understand student engagement and knowledge transfer techniques being utilized in The Spartan University classroom. Agreement to be contacted or a request for more information does not obligate you to participate in any study. If you would like additional information about this study, please call Principle Researcher at xxx-xxx-xxx or email at Principle_Researcher@usc.edu Thank you again for considering this research opportunity. Principle Researcher Running head: IIS 103 APPENDIX F Classroom Setup Main SU Classroom view from the back of the room to the front of the room. IIS 104 Main SU Classroom view from the front of the room to the back of the room. IIS 105 Main SU Classroom view from the instructors view. IIS 106 Main SU Classroom view of the instructor’s podium from the back corner of the classroom. IIS 107 Main SU Classroom view of the instructor’s “stage” and podium. Running head: IIS 108 Main SU Classroom Observers view Running head: IIS 109 REFERENCES Bono, J. E., Foldes, H. J., Vinson, G., & Muros, J. P. (2007). Workplace emotions: The role of supervision and leadership. Journal of Applied Psychology, 92(5), 1357–1367. Borgogni, L., Russo S. D., & Latham, G. P. (2011). The relationship of employee perceptions of the immediate supervisor and top management with collective efficacy. Journal of Leadership and Organizational Studies, 18(1), 5–13. Brown, S. M., & Seidner, C. J. (Eds.). (2012). Evaluating corporate training: Models and issues (Vol. 46). Springer Science & Business Media. Blackburn, H., & Hays, L. (2014). Classroom Management and the Librarian. Education Libraries (Online), 37(1/2), 23. Chesney, M. A., Black, G. W., Chadwick, J. H., & Rosenman, R. H. (1981). Psychological correlates of the Type A behavior pattern. Journal of Behavioral Medicine, 4(2), 217- 229. Curran, M. K. (2014). Examination of the teaching styles of nursing professional development specialists, Part I: Best practices in adult learning theory, curriculum development, and knowledge transfer. The Journal of Continuing Education in Nursing, 45(5), 233-240. Day, S. B., & Goldstone, R. L. (2012). The import of knowledge export: Connecting findings and theories of transfer of learning. Educational Psychologist, 47(3), 153-176. DeJoy, D. M. (1994). Managing safety in the workplace: An attribution theory analysis and model. Journal of safety research, 25(1), 3-17. Elsbach, K. D., & Sutton, R. I. (1992). Acquiring organizational legitimacy through illegitimate actions: A marriage of institutional and impression management theories. Academy of management Journal, 35(4), 699-738. IIS 110 Eccles, J. (2009). Expectancy value motivational theory. Retrieved from http://www.education.com/reference/article/expectancy-value-motivational-theory/ Emmer, E. T., & Stough, L. M. (2001). Classroom management: A critical part of educational psychology, with implications for teacher education. Educational psychologist, 36(2), 103-112. Gentner, D. 1983. Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7: 155–170. Goette, L., Huffman, D., Meier, S., & Sutter, M. (2012). Competition between organizational groups: Its impact on altruistic and antisocial motivations. Management Science, 58(5), 948–960. Grossman, R., & Salas, E. (2011). The transfer of training: What really matters. International Journal of Training and Development, 15(2), 103–120. Hale, J. (2011). Performance-based certification: How to design a valid, defensible, cost- effective program. John Wiley & Sons. Howard, B. C. (1996). Cognitive Engagement in Cooperative Learning. Igbaria, M., & Iivari, J. (1995). The effects of self-efficacy on computer usage. Omega, 23(6), 587-605. Jalal, J., Sistla, K., & Mathews, N. M. (2016). CLASSROOM MANAGEMENT LESSONS FROM FACEBOOK. International Journal of Current Research and Review, 8(5), 37. Longenecker, C., & Abernathy, R. (2013). The eight imperatives of effective adult learning: Designing, implementing and assessing experiences in the modern workplace. Human Resource Management International Digest, 21(7), 30-33. IIS 111 Malik, M. (2016). Assessment of a Professional Development Program on Adult Learning Theory. Portal: Libraries and the Academy, 16(1), 47-70. Mahan, J. D., & Stein, D. S. (2014). Teaching adults—best practices that leverage the emerging understanding of the neurobiology of learning. Current problems in pediatric and adolescent health care, 44(6), 141-149. (n.d.). Retrieved October 18, 2017, from http://lrs.ed.uiuc.edu/students/sbarnett/edpsy399/declarative.html Onwuegbuzie, A. J., & Leech, N. L. (2007). Validity and qualitative research: An oxymoron?. Quality & Quantity, 41(2), 233-249. Parsons, S. A., Nuland, L. R., & Parsons, A. W. (2014). The ABCs of student engagement. Phi Delta Kappan, 95(8), 23-27. Piksööt, J., & Sarapuu, T. (2014). Supporting Students' Knowledge Transfer in Modeling Activities. Journal of Educational Computing Research, 50(2), 213-229. Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95(4), 667–686. Salas, E., Tannenbaum, S. I., Kraiger, K., & Smith-JenSFSh, K. A. (2012). The science of training and development in organizations: What matters in practice. Psychological science in the public interest, 13(2), 74-101. Schraw, G., Veldt, M., & Olafson, L. (2009). Knowledge. Retrieved from http://www.education.com/reference/article/knowledge/ Simone, (2015). Expectancy Value theory: motivating Healthcare workers, National Research Council of Italy, http://www.aijcrnet.com/journals/Vol_5_No_2_April_2015/3.pdf Strother, J. B. (2002). An assessment of the effectiveness of e-learning in corporate training programs. The International Review of Research in Open and Distributed Learning, 3(1). IIS 112 Tzeng, G. H., Chiang, C. H., & Li, C. W. (2007). Evaluating intertwined effects in e-learning programs: A novel hybrid MCDM model based on factor analysis and DEMATEL. Expert systems with Applications, 32(4), 1028-1044. Tarique, I. (2014). Seven Trends in Corporate Training and Development: Strategies to Align Goals with Employee Needs. Pearson Education. Zepke, N. (2015). Student engagement research: thinking beyond the mainstream. Higher Education Research & Development, 34(6), 1311-1323.

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Strategic support for philanthropic fundraising: a needs analysis of development officers in higher education

PDF

Application of professional learning outcomes into the classroom: an evaluation study

PDF

Increasing institutional retention: a gap analysis

PDF

Mitigating low employee engagement through improved performance management: an evaluation study

PDF

Service provider compliance with regional center service agreement: a gap analysis

PDF

The implementation of a multi-tiered system of support at Downtown Unified School District: an analysis of teacher needs

PDF

Graduation rates in college of students with disabilities: an innovation study

PDF

Violence experienced by registered nurses working in hospitals: an evaluation study

PDF

Building teacher competency to work with middle school long-term English language learners: an improvement model

PDF

The use of differentiation in English medium instruction in Middle Eastern primary schools: a gap analysis

PDF

Factors contributing to student attrition at a healthcare university: a gap analysis

PDF

Evaluation of thesis completion in a graduate blended learning program

PDF

Creating "excellent" learning experiences: a gap analysis of a university extension program

PDF

The impact of advanced technologies on the workplace and the workforce: an evaluation study

PDF

Leadership readiness: evaluating the effectiveness for developing managers as coaches

PDF

Collaborative instructional practice for student achievement: an evaluation study

PDF

Employment rates upon MBA graduation: An evaluation study

PDF

Learners’ perceptions of the microlearning format for the delivery of technical training: an evaluation study

PDF

The role of professional development and certification in technology worker turnover: An evaluation study

PDF

Winning the organizational leadership game through engagement: a gap analysis

Asset Metadata

Creator Hawkins, David Wallace (author)

Core Title Improving instructor skills (IIS): a Needs analysis

School Rossier School of Education

Degree Doctor of Education

Degree Program Organizational Change and Leadership (On Line)

Publication Date 11/14/2017

Defense Date 10/24/2017

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag adult learning,classroom management,gap analysis,instructor workshop,instructor, teacher, facilitator,KMO,Knowledge,knowledge transfer,learning and development,Motivation,needs analysis,OAI-PMH Harvest,organizational influences,student engagement,train the trainer,trainer,training and development

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Hirabayashi, Kimberly (committee chair), Polite-Wilson, Anita (committee member), Seli, Helena (committee member)

Creator Email dwhawkin@usc.edu,dwhawkins@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c40-454319

Unique identifier UC11264309

Identifier etd-HawkinsDav-5909.pdf (filename),usctheses-c40-454319 (legacy record id)

Legacy Identifier etd-HawkinsDav-5909.pdf

Dmrecord 454319

Document Type Dissertation

Rights Hawkins, David Wallace

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA