University of Southern California Dissertations and Theses

Improving pilot training by learning about learning: an innovation study

(USC Thesis Other)

Improving pilot training by learning about learning: an innovation study

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Running head: IMPROVING PILOT TRAINING OUTCOMES 1 Improving Pilot Training by Learning About Learning: An Innovation Study by Regan J. Patrick A Dissertation Presented to the FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF EDUCATION May 2018 IMPROVING PILOT TRAINING OUTCOMES 2 IMPROVING PILOT TRAINING OUTCOMES 3 DEDICATION It is my distinct privilege to provide this dedication to the three most important people in my life; powerful, purposeful, and magnificent women without whom I would be adrift amongst the clouds. To my wife, Carolyn, thank you for always supporting and encouraging me to dream big and go farther, faster, and higher than I could ever have imagined. Your faith eclipsed my own on more than one occasion, and I am so grateful to you for believing I could make this dream real. Thank you a thousand times over! To my daughter, Elleanor, thank you for inspiring me to always seek excellence and to persevere when I started feeling crushed. You’re my favorite study buddy! One day, I hope you will forgive me for all the missed events and the times I could not share with you. It is my life’s highest honor to watch you spread your own wings, dry your feathers, and see you fly where I could only dare. To my mother, Mary, thank you so much for showing me the value of hard work and commitment. You have always inspired me to believe in myself when others did not. You have lifted me up so many times, and it is with great pride and joy that I present this to you, as humble acknowledgement of your profound wisdom and grace as a parent, teacher, and mentor. “You'll be bothered from time to time by storms, fog, snow. When you are, think of those who went through it before you, and say to yourself, 'What they could do, I can do.'” - Antoine de Saint-Exupéry, Aviation Pioneer, 1939 IMPROVING PILOT TRAINING OUTCOMES 4 ACKNOWLEDGEMENTS A work of this magnitude would not be possible without the guidance and support of many kind, patient, and benevolent souls. To my dissertation chair, Dr. Melora Sundt thank you for your steadfast faith and encouragement, and for tightening my scope on this project from laughable to achievable. To my committee members, Dr. Monique Datta and Dr. Mary Niemczyk, mahalo and thank you for agreeing to join me on this journey of discovery. It has been a privilege learning from true thought leaders and visionaries, and I am honored to pay your inspired example forward. To my friend and mentor, Dr. Steven Tourville, thank you for showing me the awesome power of applied thinking. I would still be stuck in the chocks without your cheerful support and encouragement. I look forward to our conversation on philosophies of learning. To my awesome editors, Mr. Mark Bracich and Ms. Becca Moore, thanks for the quick turns, the unyielding attention to the smallest details, and for your enduring sense of humor as you attempted to decipher my intent through bad grammar and worse punctuation. To those who inspired and encouraged me to undertake this amazing experience, my eternal thanks and continuing admiration. Dr. Cassie Barlow (Col, USAF, Ret), Brigadier Gen- eral Dagvin R.M. Anderson, and Dr. Seth Gordon stood on my behalf and proclaimed I could do this. What a blessed thing to prove them right. To my fellow USC Cohort 3 members, thank you and FIGHT ON! IMPROVING PILOT TRAINING OUTCOMES 5 TABLE OF CONTENTS List of Tables 7 List of Figures 8 Abstract 10 Chapter One: Introduction of the Problem of Practice 11 Background of the Problem 12 The Looming Pilot Shortage 14 Importance of the Organizational Innovation 17 Organizational Context and Mission 18 Organizational Performance Goal 20 Description of Stakeholder Groups 21 Purpose of the Project and Questions 22 Conceptual and Methodological Framework 22 Definitions 23 Organization of the Proposal 23 Chapter Two: Review of the Literature 25 Developing Learning Strategies 27 Stakeholder Knowledge, Motivation, and Organization Influencers 29 Knowledge and Skills 29 Motivation 42 Organization 50 Conclusion 57 Chapter Three: Methodology 59 Conceptual and Methodological Framework 59 Assessment of Performance Influences 65 Knowledge Assessment 66 Motivation Assessment 68 Organization/Culture/Context Assessment 70 Participating Stakeholders and Sample Selection 72 Data Collection and Instrumentation 73 Surveys 73 Interviews 76 Observation 80 Documents and Artifacts 85 Reliability and Validity 85 Credibility and Trustworthiness of Data 86 Role of Investigator 87 Ethics 87 IMPROVING PILOT TRAINING OUTCOMES 6 Chapter Four: Results and Findings 89 Validation 90 Participating Stakeholders 91 Survey Sampling 91 Interview Sampling 94 Observation Sampling 97 Documentation and Artifact Sampling 97 Results and Findings for Knowledge Influences 98 Results and Findings for Motivation Influences 109 Results and Findings for Organization Influences 119 Chapter Five: Recommendations 129 Recommendations for Practice to Address KMO Influences 129 Knowledge Recommendation 129 Motivation Recommendation 135 Organization Recommendations 135 Integrated Implementation and Evaluation Plan 137 Level 4: Results and Leading Indicators 138 Level 3: Behavior 139 Level 2: Learning 144 Level 1: Reaction 147 Evaluation Tools 148 Data Analysis and Reporting 149 Limitations and Delimitations for Future Research 150 Recommendations for Future Research 151 Conclusion 151 References 153 Appendices 167 Appendix A: Survey Instrument Protocol Email 167 Appendix B: Survey 168 Appendix C: Interview Introduction 177 Appendix D: Interview Protocol 178 Appendix E: Informed Consent 180 Appendix F: Observation Protocol, Classroom 185 Appendix G: Observation Protocol, Simulator/Flight 190 Appendix H: Documents and Artifacts Collection Protocol 195 Appendix I: Level 1 and Level 2 Evaluation Instrument 196 Appendix J: Level 1, 2, 3, and 4 Blended Evaluation Model 198 IMPROVING PILOT TRAINING OUTCOMES 7 LIST OF TABLES Table 1. Estimated costs to achieve individual flight ratings at State University 20 Table 2. Organization Mission, Global Goal, and Stakeholder Performance Goal 21 Table 3. Summary of Knowledge Influences on Faculty and Related Literature 41 Table 4. Summary of Motivation Influences on State University’s Ground and 49 Flight Instructors Table 5. Summary of Organization Influences on State University’s Ground 57 and Flight Instructors Table 6. Summary of Assumed Knowledge, Motivation, and Organizational 65 Influences on Faculty Table 7. Summary of knowledge influences and assessment methods 67 Table 8. Summary of motivation influences and assessment methods 69 Table 9. Summary of organization influences and assessment methods 71 Table 10. Interview Results from State University Academic Faculty 78 Table 11. Interview Results from State University Flight Training (contractor) Faculty 78 Table 12. Interview Participants, broken out between academic and flight line faculty 97 Table 13. Validated Influencers Table 126 Table 14. Summary of Knowledge Influences and Recommendations 130 Table 15. Summary of Organization Influences and Recommendations 135 Table 16. Outcomes, Metrics, and Methods for External and Internal Outcomes 138 Table 17 Critical Behaviors, Metrics, Methods, and Timing for Faculty 140 Table 18. Required Drivers to Support Faculty’s Critical Behaviors 141 Table 19. Components of Learning for the Program 146 Table 20. Components to Measure Reactions to the Program 147 IMPROVING PILOT TRAINING OUTCOMES 8 LIST OF FIGURES Figure 1. Typical civilian pilot progression to achieve an Airline Transport Pilot 13 certification Figure 2. Age 65 Retirements from Major U.S. Airlines 16 Figure 3. Schraw and McCrudden’s Information Processing Model 32 Figure 4. Coutinho and Neuman’s Model of Goal Orientation, Learning Styles, 34 Self-Efficacy, and Metacognition Figure 5. Oaks Organizational Effectiveness Model 36 Figure 6. Development Model of Pedagogical Content Knowing (PCKg) 46 Figure 7. Conceptual framework for stakeholder relationships 61 Figure 8. The Clark and Estes Performance Gap Analysis Model 64 Figure 9. Survey participant employment affiliation (n=39) 91 Figure 10. Survey participant employment alignment (n=36) 92 Figure 11. Amount of training in adult education reported by survey participants (n=46) 92 Figure 12. Number of years survey participants have worked in adult education (n=39) 93 Figure 13. Number of years survey participants have worked in the professional flight 93 program (n=39) Figure 14. Amount of time total time respondents reported serving in the active duty 94 military, Guard, or Reserves (n=38) Figure 15. Amount of time total time respondents reported serving in the airline 94 industry or commercial aviation (n=39) Figure 16. Interview demographics from State University Academic Faculty 95 Figure 17. Interview figures from State University Flight Training (contractor) Faculty 95 Figure 18. Survey Question 2 – “How much can you influence decisions related to 99 curriculum in the Professional Flight Program?” Figure 19. Survey Question 3 – “If you wanted to asses a student’s knowledge of a 99 particular topic, which of the following tools would you most prefer to use” Figure 20. Survey Question 4 – “Which of the following do you believe is most 100 important to increase a student’s retention of what they have been taught in previous lessons?” Figure 21. Survey Question 5 – “I believe instructors should make a priority of finding 110 new and innovative ways to help students develop a mastery focus (a focus on constant learning and improvement) of the rules of aviation.” Figure 22. Survey Question 6 – “Please indicate where you think a teacher’s priority 111 should be on this continuum by sliding the bar to the value that corresponds with your beliefs.” Figure 23. Survey Question 7 – “My students would benefit from learning how to 113 enhance their self-perception of their capabilities, motivation, and ability to influence events affecting their lives.” Figure 24. Survey Question 9 – “Teaching strategies designed to increase student 114 confidence in their ability to learn aviation-specific concepts will increase their performance in flight.” Figure 25. Survey Question 10 – “Using the scale bar below, please rate your 114 confidence in accomplishing the following tasks.” IMPROVING PILOT TRAINING OUTCOMES 9 Figure 26. Survey Question 11 – “To what extent is the instructor responsible 118 for a student’s strong or weak performance?” Figure 27. Survey Question 12 – “I believe my academic program provides 120 adequate education and training to faculty on practices related to teaching student reflection and self-motivation.” Figure 28. Survey Question 14 – “To what extent does your collegiate flight 121 program provide the resources you need to improve a student’s understanding of reflection and self-motivation?” Figure 29. Survey Question 13 – “I believe my academic program rewards 125 innovation and continuous improvement in syllabus development.” Figure 30. Survey Question 15 – “To what extent do you believe your collegiate 125 flight program is open to new ideas about instruction?” IMPROVING PILOT TRAINING OUTCOMES 10 ABSTRACT Numerous forecasting studies indicate that commercial aviation will soon face a shortage of qualified pilots. Research is showing that university flight training programs can increase stu- dent knowledge, task performance, and graduation rates to help fill this performance gap by im- proving student metacognition and enhancing their sense of self-efficacy. Central to the im- provement of student outcomes is the aviation faculty member. This study applies an innovation model to examine the knowledge, motivation, and organizational needs required for faculty to improve student self-efficacy and metacognition, and provides recommended solutions to those needs. This convergent parallel mixed-methods case study, utilizing descriptive surveys, inter- views, observations, and artifact gathering examined faculty knowledge and use of self-efficacy and metacognitive behavioral strategies with their students to improve performance and gradua- tion rates. The stakeholder of focus for this study is the faculty at State University (pseudonym), one of approximately 100 university flight training programs accredited by the Aviation Accredi- tation Board International (AABI). The guiding methodological framework for this study is Clark and Estes’ (2008) performance gap analysis model, a systematic, analytical problem- solving process for identifying causes of performance gaps, clarifying organizational goals, and recommending improvements. Among the study’s findings, four knowledge influences and one organizational influence were identified as affecting current practice. Recommendations to ad- dress the problems of practice were assembled using the Kirkpatrick and Kirkpatrick (2016) New World Model as the bases for an evaluation strategy. IMPROVING PILOT TRAINING OUTCOMES 11 CHAPTER ONE Introduction of the Problem of Practice Commercial aviation will soon face a shortage of qualified pilots that will require a delib- erate effort from government, industry, and academia to solve. Numerous forecasting studies have predicted a worldwide shortage of airline pilots in the coming decades, with estimates of up to 85,000 domestically and 200,000 globally (Bjerke & Malott, 2011; Higgins, Lovelace, Bjerke & Lounsberry, 2013; ICAO, 2011; Lovelace & Higgins, 2010). Boeing Aircraft Corporation, one of the world’s largest manufacturers of commercial aircraft, recently cited growing market demands for qualified, skilled, and experienced pilots driving a need for industry to train more than 637,000 new airline pilots by 2036 (Boeing, 2017). European aircraft manufacturer Airbus predicts a global requirement for over 524,000 new pilots internationally (Global Market Fore- cast, 2017). Pilot production concerns are not limited to the civilian airline industry. Lt Gen Darryl Roberson, commander of the U.S. Air Force’s Air Education and Training Command commented, “Pilot production is a national level problem” (Roberson, 2017). Air Force Chief of Staff General David L. Goldfein recently commented, “We are in a crisis. We’re 1,500 pilots short, and if we don’t find a way to turn this around, our ability to defend the nation is compro- mised” (Air Force leaders, 2017). If the nation’s aviation industry expects to remain viable, it must rethink how it trains and qualifies new pilots. To help address this issue, research indicates that civilian pilot training programs can in- crease graduation rates and improve in-flight performance by improving student metacognition skills and enhancing their understanding of learning strategies. Central to the improvement of student performance are the academic and flight training instructors, who provide education, mentorship, inspiration, and guidance. This project explores the knowledge, motivation, and or- IMPROVING PILOT TRAINING OUTCOMES 12 ganizational resources needed by aviation faculty to improve performance and graduation rates by helping aviation students learn how to learn and acquire better self-regulation skills. It ex- plores the use and evaluation of instructional design strategies to optimize learning, and exam- ines what faculty need to execute these approaches as ways to increase the number and quality of student pilots. This study provides strategies to help faculty improve retention and performance, reduce time and cost to train, and deliver higher quality graduates to aviation consumers as quickly as possible. These approaches can play an important role in an overall strategy for meet- ing the increasing demands for more commercial pilots. Background of the Problem Commercial aviation, which includes civilian pilot training, charter, corporate, cargo, and regional and major airlines relies heavily on two sources for new pilots – retiring/separating U.S. military members and civilian flight training center graduates (McGee, 2015). Several recent surveys and reports highlight an impending shortage of qualified pilots to fill airline cockpits in near and long terms (Boeing, 2017; Higgins et al., 2013; USGAO, 2014). In a recent survey, Geoff Murray, an aerospace sector lead for consulting firm Oliver Wyman, found that the num- ber of graduates from collegiate professional pilot degree programs fell 23% between 2001 and 2012, and he expects the trend to continue (Murray, 2017). In their Chapter 11 Bankruptcy fil- ings, Republic Airlines noted a nationwide pilot shortage and “a lack of pilot resources” had forced the airline to restructure (Republic Airways Holdings, 2016). Two recent RAND studies have predicted an increase in the number of pilots required, with a decrease in the number of qualified applicants available to fill open cockpit seats (McGee, 2015; Sweeney, 2014). A recent Forbes article lamented, “U.S. airlines are in the early stages of a pilot shortage that could boost labor costs and even constrain growth” (Prentice & Gouel, 2016, p. 1). IMPROVING PILOT TRAINING OUTCOMES 13 The path to becoming a commercial airline pilot is closely regulated by the Federal Avia- tion Administration (FAA). Individuals can become a qualified commercial pilot through sever- al different programs, including military service, private training programs, and accredited uni- versities and colleges. In all cases, aspiring pilots must follow a training progression that builds knowledge and skills sequentially through increasingly complex aircraft and flying environ- ments. In the United States, civilian aviators begin their flying career by becoming FAA- certified Private Pilots. After acquiring a specified minimum number of flight hours, training, and experience, pilots add additional ratings to their certifications, such as instrument flight qual- ification, operating multi-engine aircraft, and becoming an instructor in single or multi-engine aircraft and in clear and adverse weather conditions. The minimum certification required to become an airline pilot is the Airline Transport Pi- lot (ATP) rating, though some commercial carriers will accept applicants with a restricted ver- sion of the ATP (known as the R-ATP), which includes fewer flight hours (FAA, 2013). Becom- ing a certified flight instructor is not required for an ATP certification, though most pilots add this certification to enable them to build required flight time while being paid. Though FAA rules allow waivers to some requirements based on military experience or other specific criteria, Figure 1. provides an example of a typical progression for civilians to become airline pilots. Figure 1. Typical civilian pilot progression to achieve an Airline Transport Pilot certification. Shaded boxes are certifications and un-shaded boxes represent additional ratings to the previous certification. Source: IACRA, 2017 The Code of Federal Regulations (CFR) provides general and permanent rules related to all aspects of flight, including training, maintenance, air traffic control, and operations. A com- IMPROVING PILOT TRAINING OUTCOMES 14 plete list of FAA pilot rating and certification training requirements can be found at 14 CFR § 61 (Certifications, 2017). According to these rules (referred to as “Part 61”), the basic requirements for becoming a Private Pilot include being at least 16 years old, reading, speaking and under- standing the English language, passing a medical examination, receiving instruction from an FAA-certified flying training program (governed by 14 C.F.R. § 141, or “Part 141”) conducted by a Certified Flight Instructor (CFI), passing a written examination, and passing a flying evalua- tion ( “check ride”) delivered by an FAA-approved examiner. Training rules and procedures for all ratings and certifications are broken into three phases: ground training, flight training, and evaluations. While the FAA permits flying training programs flexibility in determining course content and delivery, all evaluations are conducted using FAA-approved tests and evaluators. The Looming Pilot Shortage There are several causes for the impending pilot shortage, generated by a confluence of issues that aviation training programs must address as they seek to meet future production de- mands. Military recruiting, FAA retirement rules, and new federal laws affecting pilot training and certification are three of the biggest concerns driving doubts about future pilot manning. Decrease in military pilots. Military to civilian pilot accessions, once the mainstay for sourcing new commercial pilots have begun to dry up. Both the Air Force and the U.S. Navy are experiencing aviator retention issues that are affecting current readiness and future capabilities, as well as reducing the source pool for future commercial aviators. In his 2017 testimony before the House Armed Services Committee, Vice Admiral Robert P. Burke, Chief of Naval Personnel, stated, “Navy aviator retention…has fallen below the three, five, and ten year average continua- tion rates, a trend that is expected to continue for the foreseeable future” (Military Pilot Shortage, 2017). According to a 2014 Government Accountability Office (GAO) report, former military IMPROVING PILOT TRAINING OUTCOMES 15 pilots, either separated or retired from active duty or on reserve status, are no longer a significant supply source for commercial carriers (Aviation Workforce, 2014). The GAO noted that prior to 2001, approximately 70% of airline pilot hires came from the military; in 2014 only 30% came from the military, with rates continuing to fall. Airlines can no longer count on the military to provide sufficient sourcing for future recruiting needs. In an unusual move, General Goldfein, the Air Force Chief, brought together members of the military and regional and mainline airline leaders in 2017 to discuss this issue, noting, “Today’s aviation enterprise doesn’t adequately meet the needs for national defense and national commerce” (McRae, 2017). Retirements. Federally mandated retirements are also adversely affecting pilot rosters in commercial aviation. Following the 2007 adoption of Public Law 110-135, the Fair Treatment for Experienced Pilots Act, FAA regulations require pilots to retire from active flying by their 65 th birthday (H.R. 110-135, 2008). As a result of past hiring practices, a steadily growing exo- dus of age-limited pilots in mainline air carriers is now occurring, and will peak in the early 2020s (see Figure 2.). Wolfe (2005) found the combination of current age demographics in air- line rosters and the FAA retirement age rules are partially responsible for creating the forecasted pilot shortage. IMPROVING PILOT TRAINING OUTCOMES 16 Figure 2. Age 65 Retirements from Major U.S. Airlines. Requirements for new pilots are esti- mated to continue growing steadily through 2021 and remain at historically high levels through 2028. Source: McGee, 2015 Legislation. Recent congressional legislation establishing new qualification standards for becoming an airline pilot is severely affecting both universities and students. The Aviation Safe- ty and Federal Aviation Administration Extension Act of 2010, known as Public Law (PL) 111- 216, was passed as a result of the loss of Colgan Air Flight 3407 in 2009. Following the acci- dent, a National Transportation Safety Board (NTSB) report listed five probable causes, four of which were pilot training-related (NTSB, 2010). PL 111-216 made sweeping changes to how commercial airline and corporate pilots receive their qualifications, including a new requirement for pilots to hold an ATP before applying for employment with an airline (Depperschmidt, 2013). Applicants must now have 1,500 hours of total flight time (a 700 hour increase over pre- vious qualification minimums), including 500 hours of specialized cross-country time not previ- ously required (Bjerke & Malott, 2011). PL 111-216 was intended to increase the amount of experience pilots have before they can apply to an airline, but it has also created a significant challenge for young aviators and the schools training them. In a 2013 survey of 59 collegiate flight-training institutions, 67% felt the new law would be detrimental to aviation training programs (Depperschmidt, 2013). One re- spondent noted that once prospective students realize how limited their employment opportuni- ties will be without an ATP, there will be so few students pursuing a degree in professional pilot- ing that many schools will have to close. Another believed fewer students will be able to “see the light at the end of the tunnel,” and unable to cope with “the crushing burden of student loans” while trying to make a living as a pilot (p. 12). IMPROVING PILOT TRAINING OUTCOMES 17 If collegiate training programs cannot adapt to the new FAA requirements, if they cannot compete with new private competitors, or if they fail to provide access to cost-effective public programs, a decrease in the number of qualified pilots produced per year could easily occur at a time when production should be increasing (Fullingim, 2011). Pilot training sources must be- come more effective, efficient, and agile to meet the growing demands of commercial carriers. Importance of the Organizational Innovation Civil aviation, which includes pilot training, regional and major airline operators, charter, and corporate carriers, has a significant impact on economies and industries worldwide. A 2016 industry report from the Air Transport Action Group (ATAG) shows that aviation supports over 62.7 million jobs worldwide, moves over 3.3 billion passengers and 50.4 million tons of freight annually, and produces over $2.7 trillion in global economic impact (Aviation Benefits, 2016). Civil aviation has a significant impact on the U.S. economy in terms of production, operation, and service. Domestically, the FAA states that U.S. air carriers transported 837.2 million pas- sengers in 2014, moved more than 61.2 billion revenue ton-miles (RTM) of freight, and enabled $262.8 billion of visitor expenditures on goods and services (FAA, 2014). Civil aircraft manu- facturing is the top net exporter in the U.S., with a positive trade balance of $54.3 billion. Com- mercial aviation plays a critical part in America’s economic might, supporting 11.8 million jobs domestically, accounting for $1.5 trillion in total economic activity, and contributing 5.4 percent to U.S. Gross Domestic Product (GDP) (FAA, 2014). Commercial pilots play a vital part in America’s economic might and in the viability of global trade. To remain viable, civilian air car- riers must have a steady flow of qualified pilots entering the industry. Closing the gap between operational requirements and pilot production requires a comprehensive assessment of needs and capabilities, with both aircraft manufacturers and IMPROVING PILOT TRAINING OUTCOMES 18 operators fearing a looming pilot shortage will negatively impact the industry. In their 2015 forecast, Boeing warned that, “instructors will need to have cross-cultural and cross-generational skills to engage tomorrow's increasingly diverse aviation workforce” (Boeing, 2015, p. 5). The Airline Pilots Association (ALPA) agrees, stating, “unless we work to make the airline pilot profession an attractive career, providing livable wage, benefits, and career-progression opportunities, we will be faced with a shortage of qualified pilots” (Keep America Flying, 2016, p. 6). Facing an estimated 400% increase in airline pilot hiring in the next 10 years, the Department of Defense is reassessing training and retention programs in an effort to maintain readiness as they continue to provide hundreds of pilots annually to commercial carriers through retirements and separations (Military Pilot Shortage, 2017). The U.S. Air Force was 1,544 pilots short of its operational requirements for Fiscal Year 2017, which represents a near-term crisis for the service as well as a longer-term concern for future commercial operators (McRae, 2017). The consequences of not training sufficient numbers of qualified pilots has already been seen through a reduction in commercial service to smaller communities (McLaughlin, 2015). Regional airlines, the current primary feeding source to the major air carriers, have faced consolidation and in some cases reduced capacity. Major air carriers draw heavily from Regionals for replacement pilots, and they will soon feel the strain as flights between city pairs are reduced and ticket prices increase. The relative ease and access that the public and businesses enjoy through air travel will eventually be strained, and larger economies may suffer until more pilots can be sourced and trained to FAA standards. Organizational Context and Mission The civilian flying training community represents a large, diverse, and loosely integrated collection of stakeholders interested in the causes and effects of this problem of practice and in IMPROVING PILOT TRAINING OUTCOMES 19 potential solutions. As one of many organizations seeking to understand and address the im- pending commercial pilot shortage, the Aviation Department at State University (a pseudonym), a large U.S. university, is providing training system leadership and expertise. The flying training program was established in the latter half of the twentieth century and is accredited through the Aviation Accreditation Board International (AABI). It offers Bachelor’s through Master’s de- gree programs in Air Traffic Control, Aviation Management and Human Factors, Flight Educa- tion, and Unmanned Aerial Systems (Collegiate Aviation Guide, 2016). The Aviation Training Program is certified under FAA Part 61 and 141, and offers pilot certifications including Private, Commercial, Instrument, Multi-Engine, Certified Flight Instructor (CFI) airplane and instrument, and Restricted ATP (R-ATP). A contractor conducts all flight instruction at a nearby airport, providing FAA-certified instructors for flying training. At graduation, students will achieve at least 400 hours of flight time, and internships are available to students while they are enrolled in the university. In addition to tuition, living, and school expenses, students must pay for flight instruction as well as flight time. Table 1 shows the estimated costs associated with these flight ratings. Table 1 Estimated costs to achieve individual flight ratings at State University Private Pilot Instrument Commercial CFI $23,000 $33,000 $24,000 $19,000 State University’s aviation training program’s focus is to prepare students for a career as a professional aviator. They provide baccalaureate and graduate degree options to regional students as well as the national and international air transportation community (organization’s website, 2016). State University’s professional flight training program student body is made up IMPROVING PILOT TRAINING OUTCOMES 20 primarily of undergraduate students, which includes both native-born U.S. citizens as well as international students able to pass an FAA Class 1 flying physical examination. State University enrolls approximately 60 new students each year to its flight training program, of which approximately 20 complete the course and move on to professional flying positions. State University’s program goals include national recognition as a leading academic program in aviation by providing students high quality training and education in flight, air traffic and aviation management areas at the undergraduate level, and in aeronautical management and aviation human factors at the graduate level (organization’s website, 2016). State University has a stated goal of supporting and encouraging the next generation of aviators and measures success by an increase in the number of students who complete their four-year program. Organizational Performance Goal By January 7, 2019, State University will ensure all instructors (faculty and contractors) are aware of and practicing effective principles of instructional design, specifically those that increase student self-efficacy and metacognition practices to improve student outcomes (more, better performing graduates). This goal involves awareness training as well as implementation and execution strategies for faculty and flight instructors to improve student metacognitive understanding and support. The initiative will be coordinated with State University’s Aviation Training Program Director. Goal achievement will be measured through an increase in faculty support of self-efficacy training and an improvement in student performance on the ground (academic and simulator training) and in flight, as well as improvements in graduation rates. Table 2 Organization Mission, Global Goal, and Stakeholder Performance Goal Organization Mission The mission of this organization is to achieve national recognition as a leading academic pro- IMPROVING PILOT TRAINING OUTCOMES 21 gram in aeronautical management technology by providing high quality education in flight, air traffic and aviation management, aeronautical management, and aviation human factors. Organizational Global Goal Provide students with the knowledge and skills critical to success, while delivering content that is not only current, but also forward looking. Success will be measured by an increase in the number of students who complete their program in four years. Stakeholder Goal By January 7, 2019, all aviation instructors (faculty and contractors) will be aware of and prac- ticing effective principles of instructional design, specifically those that increase student self- efficacy and metacognition practices to create more, better qualified graduates. Description of Stakeholder Groups There are five stakeholder groups associated with State University’s performance goal. The first is the faculty at State University’s Aviation Training Program, a collection of approximately 25 full- and part-time teachers and faculty members. The second stakeholder group is the flying training contractor associated with State University’s program, employing approximately 46 full- and part-time FAA-rated flight instructors and evaluators. The third stakeholder group consists of the program leadership at State University, responsible for the recruitment, oversight, training, and strategic guidance of the Aviation Training Program. The Aviation Accreditation Board International (AABI) is the fourth, an organization dedicated to advancing quality aviation education worldwide through accreditation and leadership of collegiate aviation training programs. AABI provides regulatory guidance and accreditation standards to all participating schools and universities. The fifth stakeholder group is made up of the regional and mainline airlines who will employ graduates of State University’s Aviation Training Program. IMPROVING PILOT TRAINING OUTCOMES 22 Additional stakeholders such as the Regional Airline Association, the U.S. military, and aviation education providers in industry and academia all share interest in scoping the nature of the problem and applying innovative solutions to address any impending pilot crisis. Purpose of the Project and Questions The purpose of this project was to apply an innovation model to examine knowledge, mo- tivation, and organizational resources necessary to help aviation faculty address a growing short- age of qualified pilots in U.S. commercial aviation. The analysis examined strategies and meth- odologies needed by faculty to improve training system performance. This study also examined the knowledge, motivation, and organizational factors influencing the ability of the aviation training program faculty, both ground and flight instructors, to improve student performance and graduation rates. The questions that guided this study were: 1. What are the knowledge, motivation, and organizational needs required for faculty to im- plement and evaluate sound instructional design to improve student self-efficacy and metacognition? 2. What are the recommended knowledge, motivation, and organizational solutions to ad- dress those needs? Conceptual and Methodological Framework Clark and Estes’ (2008) gap analysis, a systematic, analytical method that helps to clarify organizational performance goals and identify the gap between the actual performance level and the preferred performance level within an organization, was adapted for needs analysis as the conceptual framework. The methodological framework was a mixed-methods case study with descriptive statistics, interviews, observations, and artifacts. Assumed knowledge, motivation IMPROVING PILOT TRAINING OUTCOMES 23 and organizational needs were generated based on personal knowledge and related literature. The needs were validated using findings and results from surveys, interviews, observations, a literature review, and content analysis. Research-based solutions were recommended and evalu- ated in a comprehensive manner. Definitions FAA: Federal Aviation Administration. The U.S. federal agency responsible for the safety and regulation of all aviation activities within U.S. airspace boundaries. UAA: University Aviation Association. A non-profit association of collegiate aviation programs across the United States, consisting of approximately 105 accredited colleges and universities. AABI: Aviation Accreditation Board International. An advocate and accreditation agency for aviation education programs around the world. CFI: Certified Flight Instructor. An FAA-granted flight certification authorizing an individual to provide instruction to others on how to safely operate an aircraft. Specific restrictions and rules governing flight instructors can be found at 14 CFR § 61.195 - Flight instructor limitations and qualifications. Organization of the Proposal This study is organized into five chapters. Chapter one provides the reader with the key concepts and terminology commonly found in a discussion about commercial flight training, the issues surrounding the impending commercial pilot shortage, and the impact of aviation on the U.S. and global economies. The mission, goals and stakeholders of State University’s flying training program were introduced, as well as the problems of practice associated with improving student outcomes. The initial concepts of gap analysis adapted to needs analysis were also introduced. Chapter Two provides a review of current literature surrounding the scope of the IMPROVING PILOT TRAINING OUTCOMES 24 study. Topics including learning and motivation theory, knowledge and skills, goal orientation, self-efficacy theory, and cultural models and settings will be addressed. Chapter Three details the assumed needs for this study as well as the methodology employed to choose participants, collect data, and conduct analysis. Chapter Four provides the results of the study, and Chapter Five concludes with recommendations for closing the performance gaps and evaluating progress. IMPROVING PILOT TRAINING OUTCOMES 25 CHAPTER TWO Review of the Literature This chapter explores the literature regarding knowledge, motivation, and organizational influencers necessary to help university faculty address a growing shortage of qualified pilots in U.S. commercial aviation. It is assembled in four general parts, beginning with an examination of the literature describing fundamental issues affecting commercial aviation training and some of the variables. Next, an overview of the Clark and Estes (2008) Gap Analytic Conceptual Framework is provided, followed by a review of literature on learning and motivation theory, and knowledge and skills influences. Finally, this chapter presents research that identifies knowledge, motivation, and organizational influencers affecting flying training faculty as they learn about and apply self-efficacy and metacognition instruction in their curriculum design to improve student performance and graduation rates. Developing Aviation Faculty Aviation training faculty in university programs around the country face many challenges with finding ways to inspire, motivate, and support students as they seek to become successful aviators. These schools must comply with FAA rules and regulations and be responsive to the learning styles and needs of college-age students who face unique educational challenges. Fly- ing training involves knowledge and performance mastery in the cognitive, affective, and psy- chomotor learning domains. It requires detailed understanding of fundamental concepts, as well as application, analysis, and synthesis of different knowledge groups demonstrated in classroom, simulated flight, and actual flight environments. Training systems built with an understanding of how students approach learning are better able to capture their interest and improve the odds they will graduate, as, “the thrill of flight alone does not necessarily carry students through difficult IMPROVING PILOT TRAINING OUTCOMES 26 content” (Niemczyk, 2008, p. 19; Niemczyk & Ulrich, 2009). Curriculum developers and facul- ty must find ways of not only engaging students through respect, connection, involvement, and empowerment, but also by helping them understand fundamental metacognitive skills to adapt the ways in which they think and operate (Berbary & Malinchak, 2011; Krathwohl, 2002). As Clark and Estes (2008) point out, instructors in technical fields, including aviation, are often experienced practitioners, but they frequently lack a solid background in instructional de- sign. Their cultural expectations may also differ from those of their students. Providing instruc- tion and guidance on metacognitive practices and learning strategies to curriculum developers and faculty enhances student engagement by building trust and commitment. Refining Programs in Lieu of PL 111-216 Commercial pilot training is heavily regulated and structured through FAA rules and guidelines (Certification, 2017). Recent changes to these rules, enacted through The Aviation Safety and Federal Aviation Administration Extension Act of 2010, or PL 111-216, have rewrit- ten the requirements governing minimum qualification standards for becoming a commercial pi- lot. These new rules require applicants to hold an ATP rating before they can apply to become an airline pilot, a process involving a series of sub-ratings and a minimum of 1,500 hours of total flight time (Depperschmidt, 2013). University aviation training programs are challenged to pro- vide an effective and efficient conduit for students to gain academic experience, requisite flight ratings, and actual flight time before they can apply to become commercial pilots. Supporting student learning needs in aviation training programs extends from the ground- based classroom into the air. Whatever is taught academically needs to translate to a complex, dynamic, and potentially high-stress flight environment. While PL 111-216 was intended to in- crease the amount of flight experience pilots achieve before they can apply to an airline, it also IMPROVING PILOT TRAINING OUTCOMES 27 created unanticipated challenges for young aviators and the schools training them. As university programs adapt to the updated FAA requirements, supporting student learning needs becomes essential, or a decrease in the number of pilots produced per year could easily occur at a time when production should be increasing (Fullingim, 2011). Developing Learning Strategies Identifying and implementing learning strategies designed specifically to improve student performance is a critical step in improving flying training outcomes. Answering the question, “What learning strategies are appropriate to enhance knowledge retention for students?” is im- portant in describing instructional design for an optimized learning system. Obtaining a basic understanding of cognitive load theory can help pilot training faculty build and communicate ef- fective learning strategies. Kirschner (2002) describes three load variables that shape the capaci- ty for transferring knowledge and skills into long-term memory – intrinsic, extraneous, and ger- mane cognitive loading. Intrinsic cognitive loading is a result of simply performing a task based on the level of complexity. Extraneous cognitive loading reflects the impact of instructional de- sign efficiency. Germane cognitive loading refers to engaging the learner in meaningful learning tasks, and requires careful schema construction to facilitate effective learning. Research indi- cates that good instructional design will control cognitive loading and increase learning by pre- senting information in a manner that optimizes student performance (Kirschner, 2002). Control- ling cognitive loading is an essential element in optimizing instructional design and in faculty development and education. To improve their approaches to instruction, aviation faculty must understand and apply self-awareness and self-regulation practices in their curricula. Research indicates that metacog- nitive knowledge and control is associated with successful cognitive performance in any activity IMPROVING PILOT TRAINING OUTCOMES 28 that involves thinking (Baker, 2006). Academic self-regulation, or the ability to control the fac- tors affecting learning, can lead to achievement improvements and an increased sense of self- efficacy (Dembo & Eaton, 2000). This includes setting goals, applying prior knowledge, consid- ering alternative strategies, developing an action plan, and considering contingencies when stu- dent pilots face obstacles. Student self-awareness and self-regulation has been shown to improve both resiliency and knowledge transfer (Rueda, 2011). As Bandura (2005) noted, “the content of courses is perishable, but self-regulatory skills have lasting functional value” (p. 17). Clark and Estes (2008) identified three specific causes of performance gaps that must be examined independently and holistically to identify factors that should be addressed to achieve organizational performance goals. These factors include the stakeholder’s knowledge and skills, their motivation to achieve the performance goal, and organizational barriers that keep stake- holders from achieving their goals. Clark and Estes (2008) noted that, “increasing knowledge, skill, and motivation – and focusing those assets on organizational goals – are the keys to suc- cess” (p. 2). Following 30 years of study, Locke (1996) reported that high commitment to organ- izational goals is obtained when individuals are convinced that the goal is important and attaina- ble, and feedback is provided showing progress towards the objective. The gap analysis model examines how conceptual and procedural knowledge can be aligned to define what is needed and inspire high faculty commitment. Finally, the Clark and Estes model links motivation and organ- izational influences to help close performance gaps and realize goals. In the context of this study, aviation training faculty need to know the organizational goals, and make these goals relevant and important to students to increase the likelihood they will succeed. Faculty must also understand the importance of feedback in student learning, and they need to know how to give effective feedback. Feedback includes the broader interactions IMPROVING PILOT TRAINING OUTCOMES 29 between teachers, students, and colleagues within the school and the wider subject community (Foster, Burn, McConnell, & Counsell, 2016). Stakeholder Knowledge, Motivation, and Organization Influencers To address problems of practice such as the one facing the commercial aviation training community, it is essential to identify the barriers and influences that create gaps in system per- formance. Narrowly defined, a performance gap is the difference between an existing level of performance and a desired level (Chevalier, 2010). Clark and Estes (2008) defined the “Big Three” causes of performance gaps in most organizations as individual knowledge and skill, mo- tivation to achieve goals, and organizational barriers (p. 43). Closing gaps between current and desired levels of performance takes a deliberate effort from stakeholders, typically beginning with education and training. For this study, the Clark and Estes’ (2008) Gap Analysis Model was used to examine self-efficacy and metacognitive instruction and its relevance to commercial pilot training. This model provides an analytic, systematic methodology to clarify organizational goals and identify gaps between actual performance and desired performance levels. Assumed causes for perfor- mance gaps were identified based on personal knowledge and related literature. These causes were validated using literature reviews, surveys, interviews, and document analysis described in Chapter 3 of this study. Research-based solutions are evaluated and recommended in Chapter 5. Knowledge and Skills This section of the literature review identifies key knowledge and skills faculty must pos- sess to effectively understand and apply self-efficacy and metacognitive strategies to help stu- dents succeed during flight training. IMPROVING PILOT TRAINING OUTCOMES 30 Knowledge Influences. Training professional aviators is a multi-dimensional exercise, requiring motivated students to achieve a mastery-level orientation in requisite knowledge and skills of safely operating an aircraft. Key elements of flight instruction can be identified to es- tablish an aviation learning taxonomy. A learning taxonomy is as a way of describing desired learning behaviors and characteristics and identifying specific stages of learning (O’Neill & Murphy, 2010). One of the most frequently applied learning taxonomies was created by Dr. Benjamin Bloom in 1956. His “Taxonomy of Educational Objectives. Handbook 1: Cognitive Domain” served as one of the earliest frameworks for classifying expectations for what students should learn as a result of instruction (Krathwohl, 2002). The cognitive domain can be defined as the recall or recognition of knowledge and the development of understanding and intellectual abilities and skills (Reigeluth, 1999). Subsequent publications from Krathwohl, Bloom, and Masia (1964), Simpson (1966), and Harrow (1972) further expanded Bloom’s original work into the affective and psychomotor domains. The affective domain encompasses decision-making abilities, situational awareness, attitudes, beliefs, values, feelings, emotions, and adaptability to stressful situations (Krathwohl et al., 1964). The psychomotor domain involves well-organized and coordinated skills and activi- ties requiring manual manipulation of things, movement, or language (Atienza, 2010). To be successful, aviation program instructional designers and faculty utilize a blended approach for each of these domains to inspire effective, lasting change in student behaviors. Krathwohl (2002) revised and expanded on Bloom’s original work, describing four cog- nitive knowledge dimensions. Factual knowledge includes discrete, isolated content elements that form the basics students must know to understand a discipline or solve problems within it. The conceptual knowledge dimension involves complex, organized forms of knowledge students IMPROVING PILOT TRAINING OUTCOMES 31 use to build an understanding of the interrelationships among basic structural elements that ena- ble them to function together. Procedural knowledge describes knowing how to do something, and includes methods of inquiry and the criteria for using a particular skill or technique. Meta- cognitive knowledge is an understanding and comprehension of cognition in general, and specifi- cally an awareness and knowledge of one’s own thinking (Krathwohl, 2002). University avia- tion programs must ensure their faculty internalize each of these knowledge dimensions and build systems to support them as they educate aspiring pilots. To ensure their programs achieve desired knowledge and skills transfer levels, organizations must carefully examine their teaching strategies and the tools they use. Learning alone does not ensure training is effective – it is the positive transfer of training that leads to changes and improvement in student performance (Grossman & Salas, 2011). Aviation training programs found in universities, the military, and commercial training ventures must understand and support the cognitive learning processes associated with flying training. As Kirschner (2002) pointed out in his discussion on cognitive load theory, human cognitive architecture should be a major consideration when designing instruction. In their writ- ings on information processing theory, Schraw and McCrudden (2013) described a three-part Information Processing Model (IPM), a useful guide for explaining how human beings transfer knowledge from input to long-term memory (Figure 3.). To generate consistent, repeatable per- formance outcomes in pilots, it is essential that faculty understand this transfer process as they seek to optimize their curricula to support student mastery of pilot-specific cognitive, affective, and psychomotor skills. IMPROVING PILOT TRAINING OUTCOMES 32 Figure 3. Schraw and McCrudden’s Information Processing Model. Applying Bloom’s Taxon- omy domains for Aviation Training contexts. Source: Schraw and McCrudden, 2013 Grossman and Salas (2011) organized training system inputs into three main categories; trainee characteristics, training design, and work environment. A study of 108 collegiate avia- tion students by Niemczyk and Ulrich (2009) indicated that many pilot training programs face significant challenges to provide a coherent, detailed instructional design. The IPM helps pro- cess owners better understand the cognitive learning process and provides a tool to refine train- ing design and improve student outcomes. University aviation programs can apply the IPM model to define student cognition and learning objectives while comparing how metacognition and self-efficacy are related to success. Assumed declarative conceptual knowledge causes. Aviation training faculty need to understand the relationship between student metacogni- tion, self-efficacy, and student success and understand their role in improving learner outcomes. Dr. John Flavell formally introduced the term metacognition in 1976, which he referred to as knowledge and control of the process of cognition in the transfer of learning (Baker, 2006). In- formally referred to as “thinking about thinking,” metacognition involves higher-order mental learning processes, including making plans for learning, using appropriate skills and strategies to solve a problem, making estimates of performance, and calibrating the extent of student learning Sensory Memory (Register) Working Memory (consciousness) Long-Term Memory INPUTS -  Cognitive -  Affective -  Psycho-motor Attention Retrieval Storage Memory Loss Memory Loss Elaboration (Rehearsal) 5 Senses (hearing, sight, touch, smell, and taste) Connect incoming information with existing experiences IMPROVING PILOT TRAINING OUTCOMES 33 (Coutinho & Neuman, 2008). It is the awareness of and knowledge about one’s own cognition and the cognitive process itself, allowing an individual to know when and why to do something, and to understand the contextual aspects of a particular problem (Krathwohl, 2002; Rueda, 2011). Successful instructors must also know and understand the value of self-efficacy in the learning process. Bandura (1991) described self-efficacy as a person’s belief that they can exe- cute a task at a certain level of performance. It refers to higher-order thinking involving active control over the cognitive processes as students engage in learning (Livingston, 1997). As a principle element of Social Cognitive Theory, self-efficacy provides the foundation for motiva- tion, well-being, and accomplishment, and helps create desired outcomes for individuals (Pajar- es, 2006). Substantial evidence supports the relationship between self-efficacy and higher levels of motivation, expectations, and academic performance (Chemers, Hu, & Garcia, 2001; Coutinho & Neuman, 2006; Davis et al., 2000; Pintrich & De Groot, 1990). In a survey of 629 first-year college students, Chemers et al. (2001) found that academic self-efficacy is, “significantly and directly related” to academic expectations and performance (p. 61). While the literature paints a clear picture of the relation of self-efficacy and metacogni- tion to student academic achievement, faculty must clearly understand their role in inspiring and supporting these behaviors in their classrooms. Knowing that enhancing these two learning ele- ments, which tend to improve with age, can lead to improved student performance is an im- portant first step in faculty education (Coutinho, 2008). Coutinho and Neuman (2006) created a theoretical framework to help explain how college-age students learn, describing the relationship between metacognition and self-efficacy as they attempt to improve performance and realize mastery goals (Figure 4.). Their model aims to link concepts of achievement goals (performance IMPROVING PILOT TRAINING OUTCOMES 34 and mastery orientations), learning styles (deep processing, surface processing, and disorganiza- tion), metacognition, and self-efficacy to student academic performance. Figure 4. Coutinho and Neuman’s Model of Goal Orientation, Learning Styles, Self-Efficacy, and Metacognition. Source: Coutinho and Neuman, 2013 Goal orientations are an important instructional element aviation faculty must understand as they seek to improve student outcomes. They fall into two general categories; mastery and performance. A mastery orientation, also referred to as a learning orientation, involves interest in self-improvement and a desire to increase competence (Kaplan & Maehr, 2007; Phillips & Gully, 1997). In a study of 405 undergraduate students, Phillips and Gully (1997) found that having a learning orientation had a positive effect on self-efficacy. Performance orientations are intended to demonstrate competence, managing the impression others may have, and reflect a desire to be positively evaluated by others (Kaplan & Maehr, 2007; Phillips & Gully, 1997). Mastery and performance orientations are further broken down into approach and avoidance goals. Approach orientations reflect a student’s focus on the possibility of achieving success, while avoidance refers to a focus on the possibility of failure and attempts to avoid it (Kaplan & Maehr, 2007). Student Performance Metacognition Self-Efficacy Disorganization Surface Processing Deep Processing Mastery Approach Mastery Avoidance Performance Approach Performance Avoidance Goal Orientation Learning Styles IMPROVING PILOT TRAINING OUTCOMES 35 For aviation faculty, a mastery goal “structure” is an environment where instructional practices and norms convey to students that learning and understanding is important, that effort and improvement are valued, and that all students are valued as well (Skaalvik & Skaalvik, 2013). To support students seeking a mastery orientation in complex subjects such as aviation, Coutinho and Neuman (2006) suggested teachers encourage students to foster an enduring sense of self-efficacy, form deep connections between new and old information, and adopt rote re- hearsal and memorization practices when appropriate. Aviation training faculty must know the performance goals related to providing self- efficacy and metacognition instruction to their students. An important step in closing any per- formance gap is defining clear, concise, measurable goals for individuals and organizations. Ef- fective goals direct attention, effort, and action towards desired outcomes, and can be assigned by others, set jointly by participants, or be self-set (Locke & Latham, 2006). In the context of Social Cognitive Theory, goals represent anticipated, desired, or preferred outcomes to demon- strate that people not only learn, they use forethought to see a desired future outcome and gener- ate actionable plans to achieve them (Denler, Wolters, & Benzon, 2014). As Rueda (2011) pointed out, goal setting is a critical first step in providing direction while creating a mechanism for determining when to change the current course of action. It also provides a benchmark against which performance should be evaluated. A “global goal,” which can take one to five years to accomplish, is often found in an organizational mission statement to help individuals focus their personal visions (Rueda, 2011). Once an organization decides on a global goal, all members must align their individual efforts to help accomplish the goal to achieve success. Understanding the relationship between student metacognition, self-efficacy, and suc- cessful learner outcomes becomes an important initial step in the formation and acceptance of IMPROVING PILOT TRAINING OUTCOMES 36 organizational goals. Figure 5 applies Oaks’ (2015) Organizational Effectiveness Model as a framework for communicating goal attainment. Institutional, or strategic goals are decomposed into lower echelon goals, which become points of action and outcomes. Outcomes become indi- cators of organizational performance and data for goal assessment, and both elements inform unit and individual reflection and suggestion for change. To be effective, a global goal of engaging faculty to provide self-efficacy and metacognitive instruction to students must be clearly under- stood by all stakeholders, achieve concurrence of faculty, mastery-focused, and include con- sistent reflection and feedback throughout the process. Figure 5. Oaks Organizational Effectiveness Model. Mapping organizational goals to perfor- mance, followed by reflection. Source: Oaks, 2015 Outcomes-Based Assessment Institutional Goals Departmental Goals Major Activities Performance Indicators Expected Out- comes Reflection and Intentional Change IMPROVING PILOT TRAINING OUTCOMES 37 Assumed metacognitive knowledge causes. Aviation training faculty need to know how to apply metacognitive and self-efficacy prac- tices in their own instruction and reflect on how they are performing relative to their goals. Un- derstanding how to apply metacognitive and self-efficacy practices in the classroom is an ena- bling step in improving aviation training. Çubukçu (2012) described the value of creating stu- dent-centered learning environments in building self-efficacy behaviors, involving the learner in decision making and curriculum development to improve critical-thinking skills, problem- solving abilities, and reflective thinking. Challenging students with difficult tasks and high ex- pectations has been shown to be an effective method of building self-efficacy, improving learn- ing performance, and enhancing satisfaction with college life (Chemers et al., 2001). Bandura (1986) described differences between general and domain-specific self-efficacy, and how focus- ing on context-specific self-efficacy education and evaluation can positively influence effort, persistence, and perseverance. As Pajares (1996) noted, self-perception of competence is an im- portant predictor of goal attainment and successful outcomes. Helping aviation faculty gain con- fidence in implementing self-efficacy instruction in their classrooms provides a critical bridge to help students adopt a mastery orientation, engage in more metacognitive activity and cognitive strategy use, and more effectively manage their efforts (Pintrich & De Groot, 1990). To effectively learn and master basic aviation fundamentals both on the ground and in the air, student pilots must understand their own cognitive processes (Baker, 2009). Metacognitive instruction is an important component in any educational program, including aviation if schools want to help their students learn to think like professional pilots (Mayer, 2011; Tanner, 2012). Developing an awareness of student metacognitive needs and establishing support initiatives rep- resents strategic thinking, responsive to changing student needs, and nurtures a shared vision of IMPROVING PILOT TRAINING OUTCOMES 38 how they want their students to learn complex material on the ground and in the air (Senge, 1990). Metacognitive training, even when provided for a short time to academically weak stu- dents, has been shown to significantly improve learner performance (Coutinho, 2008). To build an academic culture grounded in metacognitive strategies and principles, aviation faculty need to understand the value and merits of metacognition and know how students think about aviation subjects (Tanner, 2012). As shown in Figure 5, reflection is an essential element in successful goal attainment. In his seminal work, How We Think, Dewey (1910) described the process of reflective thinking: “active, persistent, and careful consideration of any belief or supposed form of knowledge in the light of the grounds that support it, and the further conclusions to which it tends, constitutes re- flective thought” (p. 6). He later stated that we learn more from reflecting on our experiences than from the experience itself (Dewey, 1933). Schön (1987) referred to the need for teachers to reflect “in” and “on” action as they make adaptations to their instruction. Reflection-in-action involves restructuring one’s understanding of a situation in the face of new information to invent a new strategy of action and generate new knowing (Schön, 1995). Reflection-on-action refers to after-the-event evaluation, adding to experiences already acquired in practice and establishing a conceptual congruence between behavior and thinking (Bengtsson, 1995; Tillema, 2000). Focused reflection on training system effectiveness is an essential element of instruction- al design. Cochran, DeRuiter, and King (1993) explored the idea that multiple opportunities for teaching and reflection are important components of effective teacher preparation. For aviation faculty, reflection is an essential tool in deciding how to achieve educational goals, evaluate pro- grams, and determine the factors that help or hinder goal attainment (Valli, 1997). Developing reflection in education involves elements of purpose, process, focus, and ultimately judgment, IMPROVING PILOT TRAINING OUTCOMES 39 which Procee (2006) described as the power to determine which rules, concepts, and theories best align with a particular situation or problem. Learning and applying reflective practices in aviation training links organizational goals with student performance. Assumed procedural knowledge causes. Aviation training faculty must know how to structure learning experiences to increase students’ self-efficacy and metacognition abilities in their programs, how to give effective feed- back, how to communicate clear performance goals, and how to evaluate their teaching relative to student performance. Strong self-efficacy beliefs are directly related to a student’s ability to apply effective problem-solving and decision-making strategies, organize their resources, hold positive expectations about outcomes, set higher goals, and ultimately see difficult situations as challenges rather than threats (Chemers et al., 2001). As aviation faculty learn and apply strate- gies to help students realize these objectives, student performance will improve. Borgogni, Dello Russo, Petitta, and Vecchione (2010) suggested that training directed at increasing self-efficacy should include strategies targeted at enhancing “the main sources of self- efficacy beliefs,” which include mastery experiences, vicarious experience built on peer and team coaching, and verbal persuasion involving feedback, support, and encouragement to build commitment (p. 290). Writing on efforts to enhance classroom effectiveness, Tinto (2012) en- couraged faculty to employ assessment techniques regarding the learning environment, work with support staff to develop early warnings and interventions where appropriate, and to acquire and refine pedagogical skills to engage students in learner-centric environments. These elements provide a framework for aviation instructors seeking to improve self-efficacy and metacognition instruction in their ground and flight training programs. IMPROVING PILOT TRAINING OUTCOMES 40 Providing timely, relevant feedback is an enabling step in applying these forms of behav- ioral instruction in an aviation context. To be effective, student and instructor feedback must be informed by a clear understanding of learning objectives and the plan for getting there (Foster et al., 2016). Feedback must be specific and clear, focused on the task rather than the learner, and focused on improvement rather than simply verifying performance (Deans for Impact, 2015). Effective feedback, as well as actual success with challenging tasks, positively enhances an indi- vidual’s perception of academic or performance competence (Borgogni et al., 2010). Students are more likely to achieve academic success when their performance is assessed through frequent feedback in ways that enable the student, faculty, and school administration to adjust their behav- iors accordingly (Tinto, 2012). Kluger and DeNisi (1996) argued that effective feedback must be timely, task-focused (rather than person-focused), concrete, and clear. Flight instructors typical- ly provide feedback at the point of instruction, through comprehensive preflight or post-flight debriefings, and through scoring and examination grading. These essential components form an integrated evaluation and feedback system designed to improve student performance in the air. Clear performance goals must be accompanied by a deliberate evaluation strategy that measures an organization’s progress towards closing performance gaps. One of the most im- portant challenges facing an aviation training program is creating and implementing appropriate systematic evaluation methods (Reio, Rocco, Smith, & Chang, 2017). Kirkpatrick’s New World training evaluation model (Kirkpatrick & Kirkpatrick, 2016) is a comprehensive, widely used methodology that examines four sequential levels of instruction – reaction, impact, transfer, and bottom line results. It provides a useful framework for examining aviation training systems in particular. In examining Kirkpatrick’s model, Reio et al. (2017) compared seven competing and complimentary evaluation methodologies and concluded that while there is no universally ac- IMPROVING PILOT TRAINING OUTCOMES 41 cepted process or tool, evaluations can and should be performed regularly to ensure objectives and goals are being met. They encouraged all organizations to consider Kirkpatrick’s model to identify unique critical levels of performance, eliminate or modify ineffective programs, ensure training recourses are allocated wisely, and enhance the impact of the organization (Reio et al., 2017). As aviation training faculty seek to improve self-efficacy and metacognition instruction in their programs, feedback and evaluation methods become essential parts of the university’s overall strategy for organization and student success. Table 3 Summary of Knowledge Influences on Faculty and Related Literature Assumed needs of Successful Instructors General Literature Declarative (Conceptual) Aviation training faculty need to under- stand the relationship between student metacognition, self-efficacy, and student success and understand their role in im- proving learning outcomes (Baker, 2009; Bandura, 1991; Chem- ers, Hu, & Garcia, 2001; Coutinho, 2008; Coutinho & Neuman, 2006; Davis, Fedor, Parsons, & Herold, 2000; Kaplan & Maehr, 2007; Krathwohl, 2002; Livingston, 1997; Pajares, 2006; Phillips & Gully, 1997; Pintrich & De Groot, 1990; Rueda, 2011; Skaalvik & Skaalvik, 2013) Aviation training faculty must know the performance goals related to providing self-efficacy and metacognition instruc- tion to their students (Deans for Impact, 2015; Denler, Wolters, & Benzon 2014; Locke, 1996; Locke & Latham, 2006; Rueda, 2011; Oaks, 2015) Metacognitive Aviation training faculty need to know how to apply metacognitive and self- efficacy practices in their own instruction and reflect on how they are performing relative to their goals (Baker, 2009; Bandura, 1986; Dew- ey, 1910; Dewey, 1933; Chemers, Hu, & Garcia, 2001; Cochran, De- Ruiter, & King, 1993; Coutinho, 2008; Çubukçu, 2012; Mayer, 2011; Pajares, 1996; Pintrich & De Groot, 1990; Procee, 2006; Schön, 1983; Senge, 1990; Tanner, 2012; Valli, 1997) IMPROVING PILOT TRAINING OUTCOMES 42 Procedural Aviation training faculty must know how to structure learning experiences to in- crease students’ self-efficacy and meta- cognition abilities in their programs, how to give effective feedback, how to com- municate clear performance goals and how to evaluate their teaching relative to student performance (Borgogni, Dello Russo, Petitta, & Vecchione, 2010; Chemers, Hu, & Garcia, 2001; Foster, Burn, McConnell, & Counsell, 2016; Kirk- patrick, 2006; Kirkpatrick & Kirk- patrick, 2016; Reio, Rocco, Smith, & Chang, 2017; Tinto, 2012) Motivation Motivation is a critical element in the development of skills-based training programs such as flying training. Knowledge and motivation are equally important and tightly intertwined fac- tors in performance improvement programs (Clark & Estes, 2008). Clark and Estes (2008) noted that increased motivation links with knowledge, skills, and work processes to achieve goals, and they highlight three motivational processes central to increasing performance: active choice, per- sistence, and mental effort. Active choice refers to a decision to replace intention with action, persistence involves steadfast commitment in the face of adversity, and mental effort is a re- sponse to confidence to generate new knowledge or learning (Clark & Estes, 2008; Rueda, 2011). Recent research studies indicate that self-determined motivation resulting from personal values will create better, longer-term learning outcomes than controlled, or reward/punishment motivation (Deans for Impact, 2015). Teaching self-regulatory strategies such as metacognition and self-efficacy to students can prevent boredom and inspire self-motivation, thus helping them set goals, develop positive beliefs in their ability, and maintain these beliefs when facing difficul- ty (Dembo & Eaton, 2000). Goal Orientation Theory. Yough and Anderman (2006) described goal orientation the- ory as a method to examine the reasons students engage in academic work. Setting personal goals can affect student effort, persistence, attention, and planning. They can be arranged in two IMPROVING PILOT TRAINING OUTCOMES 43 types: mastery and performance. A mastery goal orientation motivates students to master the task at hand in a task-focused, task-involved intrinsic learning framework. A performance orien- tation motivates students to demonstrate higher competence than other students when comparing relative ability in an egocentric, extrinsic framework (Rueda, 2011; Yough & Anderman, 2006). To refine this model, Pintrich (2000) built a matrix to differentiate between mastery and perfor- mance orientations, as well as approach and avoidance focuses. He notes that while mastery is generally seen as preferable in educational settings, performance goals are not necessarily mala- daptive or in opposition to mastery goals. Aviation instructors must understand the differences and their relationship to flight-specific subjects to encourage and inspire student internal motiva- tion. The relationship between mastery and performance-goal orientations, self-efficacy and metacognition, and student performance can be seen in Figure 4. Goal Orientation in Aviation. Research indicates that pilot training programs must help students set and achieve an appropriate goal orientation consistent with the skill or task-specific objective. Ferguson and Johnson (2007) noted that professional aviation students face unique challenges not typified by a majority of four-year degree programs. With the FAA requirements for medical screening, extensive ground and air training, and a minimum of 1,500 flight hours before becoming a commercial pilot, flight training represents a significantly higher cost to stu- dents in terms of time, money, and effort than many other undergraduate degree programs (Dep- perschmidt, 2013; Ferguson & Johnson, 2007). These steep barriers to completion present con- siderable challenges to aviation students, and training programs must provide guidance and focus to help define individual goal orientations. While performance orientations may be adaptive in some situations in aviation, student pilots should initially focus on learning, gaining new compe- tencies, and successfully accomplishing challenging tasks (Rueda, 2011). Yough and Anderman IMPROVING PILOT TRAINING OUTCOMES 44 (2006) noted that mastery goals are closely related to adaptive outcomes and improvement, as opposed to performance and ability differences. High grades and rewards that do not translate into reliable, repeatable performance in the air can lead to disaster. State University can assist the aviation training community overall by examining their faculty orientations to ensure support and guidance consistent with a mastery-student focus are in place, robust, and well-supported. Assumed Goal Orientation causes. Aviation training faculty should desire to find new and innovative ways to help their stu- dents achieve a mastery orientation of the rules of aviation. Finding new and innovative ways to enhance student self-efficacy and metacognition begins with an understanding of a knowledge or skill deficiency, often achieved through personal reflection on individual practice. Valli (1997) described reflective teaching as thinking back on what is seen and heard, examining obstacles to student motivation and learning, finding ways to make the curriculum more interesting and en- gaging, and understanding how to help students help each other to learn. It is an outward mind- set, linking theory to practice while designing and modeling specific teaching strategies (Valli, 1997). Aviation faculty can use these reflective practices to discover new ways to educate stu- dents on how to enhance expectancy beliefs, which are important components of academic moti- vation (Pajares, 1996). Despite years of subject matter expertise in their chosen area of specialty, many college instructors are not trained in pedagogy, curriculum, assessments, or the fundamental elements of self-efficacy and metacognitive behaviors (Tinto, 2012). As Clark and Estes (2008) pointed out, instructors in technical fields, including aviation, are often experienced practitioners but they frequently lack a solid background in instructional design. Instruction in learning strategies is seldom incorporated into aviation training curricula; most educators focus on content, not on IMPROVING PILOT TRAINING OUTCOMES 45 how to learn it (Niemczyk, 2008). Tinto (2012) highlighted contemporary challenges of imple- menting successful classroom innovations at scale, finding promising examples that transfer to larger settings, and breaking through “silos” of innovative practice that occur in the margins of education (p. 7). Çubukçu (2012) provided a model of student-centered teaching as a method of overcoming these and similar performance barriers, improving “learning to learn” behaviors (metacognition) and critical thinking, problem solving, and reflective thinking skills related to self-efficacy. When faculty understand that self-efficacy and metacognitive beliefs do not hap- pen automatically, and learn what influences the development of these behavioral traits, they can begin to help students become more familiar with their own capabilities without reducing their confidence, optimism, or drive (Schunk & Pajares, 2009). Cochran et al. (1993) provided an integrative conceptual model to help faculty under- stand the difference between being a teacher in a subject area and being a subject area expert. They state that the transformation of subject matter for teaching occurs as instructors reflect crit- ically on and interpret subject matter to find ways to represent the information, adapt material to student abilities, and tailor material to student needs. Further, they offered a development model, referred to as Pedagogical Content Knowing, to conceptualize the fundamental components as- sociated with helping students achieve a mastery orientation in aviation (Figure 6.). This model synthesizes four elements – knowledge of pedagogy, subject matter, students, and environmental contexts – to shape how flight instructors see their roles and responsibilities in improving student learning (Cochran et al., 1993). IMPROVING PILOT TRAINING OUTCOMES 46 Figure 6. Development Model of Pedagogical Content Knowing (PCKg). Source: Cochran, et al., 2015 Self-Efficacy Theory. As a predictive component of an individual’s capacity for learn- ing and motivation, self-efficacy refers to beliefs and expectations about a person’s ability to re- alize a specific outcome (Pajares, 2009). These beliefs are cognitive, goal-referenced, context- specific, malleable, and future-oriented judgments of competence (Schunk & Pajares, 2009). Bandura (2000) argued that self-efficacy affects behavior through its impact on “goals and aspi- rations, outcome expectations, affective proclivities, and perception of impediments and oppor- tunities in the social environment” (p. 75). Research indicates that individuals with high self- efficacy, belief in their own competence, and elevated expectations for positive outcomes will be more motivated to engage and persist at difficult tasks such as professional aviation (Rueda, 2011). In his work introducing Social Cognitive Theory, Bandura (1986) described self-efficacy theory as rooted in self-regulation and reflection, helping individuals make sense of their experi- ences, explore cognitive assumptions and beliefs, evaluate their actions, and adjust their thinking and behaviors (Schrunk & Pajares, 2009). Self-Efficacy Education for Faculty. Research suggests that pilot training faculty must support student self-efficacy as an important component in shaping student pilot success. In Knowledge of Pedagogy Knowledge of Environmental Contexts Knowledge of Students Knowledge of Subject Matter Pedagogical Content Knowing IMPROVING PILOT TRAINING OUTCOMES 47 their study of aviation students, Niemczyk and Ulrich (2009) noted that helping students develop self-efficacy is positively related to their course grades and is a highly desired trait in aviation as a measure of success and effectiveness. Understanding how self-efficacy education relates to in- flight performance will help faculty better align their instructional design to improve learning. While examining the predictors of self-efficacy among 297 volunteer aviation students, Davis et al. (2000) found that high levels of self-efficacy were associated with persistence in the face of failure and stressful job situations; common occurrences in aviation. A student pilot’s initial self-efficacy during the early phases of training is a likely predictor of success in later phases of training, and efforts to improve self-efficacy may improve performance throughout the training and education program (Davis et al., 2000). Educating faculty on the value of self-efficacy and how they can inspire this form of self-regulating behavior in their learning environments is an essential element in improving student outcomes. Assumed self-efficacy causes. Aviation training faculty need to be confident that self-efficacy and metacognition in- struction will improve student outcomes, and in their ability to provide this instruction as a way of improving student outcomes and performance. Teachers must inspire and promote the cogni- tive learning and confidence of their students as they tackle the demanding nature of powered flight. Effective instructors know that self-efficacy and metacognitive instruction can improve a student’s emotional state, correct false beliefs and habits of thought, raise academic and perfor- mance skills, improve self-regulation, and alter academic structures (classroom, simulator, air- craft) to ensure student success (Schrunk & Pajares, 2009). Metacognitive practices extend be- yond simple study skills; they have been shown to improve thinking processes overall, and when practiced regularly they become essential elements for creating lifelong learners in a student’s IMPROVING PILOT TRAINING OUTCOMES 48 chosen discipline (Tanner, 2012). As noted by Grossman and Salas (2011), even though “self- efficacy may not be an obvious component of training programs, organizations could greatly benefit from understanding its significant role in the transfer of training” (p. 109). Believing that they can provide quality instruction in these behaviors is essential for aviation faculty as they seek to develop a sense of collective efficacy in themselves and their students. While the literature on the effectiveness of metacognition and self-efficacy is compelling and rich, knowledge of these learning strategies is not enough to guarantee achievement; students must also be motivated to use them and to regulate their effort (Pintrich & DeGroot, 1990). When students achieve an understanding and control over their own cognitive process in any domain, learning is enhanced (Baker, 2006). Teachers have an essential role in inspiring not on- ly knowledge but also motivation to improve performance while building mastery-oriented life- long learners in the science and profession of aviation. Aviation training faculty need to attribute student success or failure, in part, to their skill as an instructor. Tinto (2012) described the attributes of effective classrooms as having clear expectations, support, assessment and feedback mechanisms, and faculty involvement. He states that student performance is determined, in part, by faculty expectations of their learners; high expectations are a necessary condition for academic success (Tinto 2012). Bandura (1986) be- lieved that self-efficacy judgments that exceed what a person believes they can do are desirable as a method of raising effort and persistence (Schunk & Pajares, 2009). Faculty must find ways of not only engaging students through respect, connection, involvement, and empowerment, but also helping them understand fundamental metacognitive skills to adapt the ways in which they think and operate (Berbary & Malinchak, 2011; Krathwohl, 2002). IMPROVING PILOT TRAINING OUTCOMES 49 If instructors are willing to attribute a student’s success or failure to their instructional skills, they can begin to understand the influence they have over student behavior and motiva- tion. Attribution theory provides a useful framework for examining both student and teacher motivation in the classroom and in flight. People seek to understand their environments and why things happen as they do, and understanding the causes of past events will influence their ability to control what happens in the future (Anderman & Anderman, 2006). Prawat, Byers, and An- derson (1983) found in both motivational and instructional situations that instructors can feel a student's success on an important exam or activity is also a teacher's success, especially if the instructor can claim some credit for the result. When teachers provide specific feedback that promotes attributions to effort or appropriate strategy use, students are more likely to be motivat- ed to try harder or change their learning strategies in the future (Anderman & Anderman, 2006). Teachers must be willing to accept that student success is dependent, in part, on the quality and consistency of their motivational and instructional abilities. Table 4 Summary of Motivation Influences on State University’s Ground and Flight Instructors Assumed needs of Successful Instructors General Literature Goal- Orientation Aviation training faculty should desire to find new and innovative ways to help their students achieve a mastery orienta- tion of the rules of aviation (Clark & Estes, 2008, Cochran, De- Ruiter, & King, 1993; Çubukçu, 2012; Depperschmidt, 2013; Fergu- son & Johnson, 2007; Niemczyk, 2008; Pajares, 1996; Pintrich, 2000; Rueda, 2011; Schunk & Pajares, 2009; Tinto, 2012; Valli, 1997; Yough & Anderman, 2006) IMPROVING PILOT TRAINING OUTCOMES 50 Self-Efficacy Aviation training faculty need to be con- fident that self-efficacy and metacogni- tion instruction will improve student out- comes, and in their ability to provide this instruction as a way of improving student outcomes and performance (Baker, 2006; Bandura, 1986; Ban- dura, 2000; Davis, Fedor, Parsons, & Herold, 2000; Grossman & Salas, 2011; Niemczyk & Ulrich, 2009; Pajares, 2009; Pintrich & DeGroot, 1990; Rueda, 2011; Schunk & Pajar- es, 2009; Tanner, 2012) Faculty need to attribute student success or failure, in part, to their skill as an in- structor (Anderman & Anderman, 2009; Bandura, 1986; Berbary & Ma- linchak, 2011; Krathwohl, 2002; Prawat, Byers, & Anderson, 1983; Schunk & Pajares, 2009; Tinto, 2012) Organization The third of Clark and Estes’ (2008) performance gap measurements is organizational influencers, which include processes, values, and culture. This section examines the literature regarding organizational influences on commercial aviation training and student learning. While there are many organizational issues surrounding the optimization of commercial pilot training curricula, this study focuses on university aviation faculty as they seek to understand, support, and improve student learning strategies - learning how to learn and belief in their abilities as stu- dents. When combined with knowledge and motivation influencers, examining organizational influences can guide universities in designing learning strategies to help students achieve success in aviation. It can also inform practices to better support faculty as they work to ensure knowledge transfer occurs and their students adopt a mastery goal orientation. These strategies can play an important role in closing organizational performance gaps found in collegiate flight training programs seeking to remedy the impending pilot shortage. Organizational influencers represent a change in existing aviation training programs and are designed to improve graduation rates and increase student flight knowledge and skill. IMPROVING PILOT TRAINING OUTCOMES 51 Cultural Models and Cultural Settings In their study of organizational change, Clark and Estes (2008) pointed out that change begins by examining motivational influences, followed by policy and/or equipment upgrades, which lead to changes in training. These influencers are all driven by organizational culture. Gallimore and Goldenberg (2001) first introduced to ideas of cultural models and cultural set- tings as a method of thinking about improving achievement in education. They describe cultural models as “tools for the mind”; a shared mental schema or normative understandings of how the world works (or ought to work), incorporating behavioral, cognitive, and affective activities (Gallimore & Goldenberg, 2001; Rueda, 2011). Cultural settings have been variously defined as social contexts where organizational policies and practices are enacted (Rueda, 2011). They are visible, concrete manifestations of cultural models that can be used to assess and improve organ- izational performance. Examined independently and together, cultural models and settings are useful tools for identifying organizational performance variables and learning. Examining or- ganizational influences on performance gaps provides a mechanism for change agents to uncover process, value, or cultural barriers. Stakeholder-specific influences. For aviation training programs struggling to produce a sufficient number of pilots to meet industry needs, examining organizational processes and culture is instructive in identifying and fixing production barriers (Clark & Estes, 2008). State University currently has approximately 200 full time students enrolled in their Aviation Training Program, many with aspirations to be- come commercial pilots. To understand how they, and other programs across the country can increase the quality and quantity of their graduates, these flight schools must closely examine several variables. These include understanding how students “learn to learn,” what universities IMPROVING PILOT TRAINING OUTCOMES 52 can do to support faculty in developing and executing relevant, effective learning strategies, how recent laws have affected program requirements, and how schools can react in response to im- prove self-efficacy and metacognitive behaviors in their students. Understanding how federal law has affected the certification process and how schools might fail to support student learning needs provides important clues for improving both the training process and the quantity and competence of graduates. Changing the culture of aviation training systems will involve enhanc- ing intrinsic student motivation. Although they arrive with a desire to succeed, aviation students often lack an understanding of how to learn the knowledge and skills necessary to fly. Establishing faculty understanding of self-efficacy. Applying targeted learning strategies to support student learning within new FAA guide- lines moves collegiate aviation courses closer to meeting aviation industry needs. University aviation programs have been challenged to adapt to FAA requirement changes, and supporting student learning needs has become an essential component of closing performance gaps (Fullingim, 2011). As Rueda (2011) pointed out, producing self-regulated learners who can transfer their newly-acquired knowledge and skills to real-world problems should be the main goal of any aviation curricula. While the content of education and training is perishable, self- regulatory skills have lasting functional value (Bandura, 2005). Assumed cultural model causes. The organization values experimentation and having faculty understand cognitive learn- ing processes by providing learning opportunities and requiring their participation in these ac- tivities. Once university aviation program leaders and designers understand the value and im- portance of metacognitive and self-efficacy concepts in flight training, they must provide a sup- portive environment for their faculty to experiment with and apply these learning elements in IMPROVING PILOT TRAINING OUTCOMES 53 practice. Clark and Estes (2008) found that successful organizations align policies and proce- dures with organizational culture to ensure goals are understood and met. Schools are thus ex- pected to set challenging goals in introducing and assessing metacognitive and self-efficacy be- haviors to realize greater effort and persistence from their faculties (Locke & Latham, 2006). Providing opportunities for guided self-monitoring, self-assessment, and debriefing/reflection on the thinking process is also important as new initiatives are undertaken to strengthen these be- haviors in students (Baker, 2006). As Valli (1997) noted, designing reflective cultures for schools is extremely difficult, involving issues of theory and execution, but it is essential for cre- ating open, supportive communities of learning. Effective communications between organizational leadership, instructional designers, and teachers is a critical step in helping faculty become more productive and feel included and em- powered (Berger, 2014). Changing educational practice requires an understanding of the organi- zation’s present performance and learning systems capabilities, which evolve into a continuous process of organizational learning (Schwandt & Marquardt, 2000). In their book on corporate management, Buckingham and Coffman (2014) encouraged organizational change leaders to fo- cus their time and energy on meeting employees’ basic needs of opportunity, recognition, caring, and encouragement. To build a strong, vibrant educational environment, the same construct can be applied to meeting basic student needs. Developing an awareness of student metacognitive needs and establishing support initiatives aligns organizational policy to changing student re- quirements and nurturing a shared vision of how they want their students to learn complex mate- rial, both on the ground and in the air (Senge, 1990). For pilot training, encouragement includes developing an understanding of the best ways to learn a subject while establishing a foundation from which to build and synthesize future knowledge and skills. IMPROVING PILOT TRAINING OUTCOMES 54 The organization provides the resources necessary for aviation training faculty to adopt student self-efficacy and metacognition programs in their pilot training classrooms. Universities must provide adequate resources to help faculty integrate metacognition and self-efficacy into their practices, which include curriculum support and professional development opportunities. Clark and Estes (2008) stated that it is crucial to focus on people and what they need to succeed by first examining motivation, then policy/equipment requirements, then knowledge and training issues. Training systems built with an understanding of how students approach learning are bet- ter able to capture their interest and improve the odds they will graduate (Niemczyk, 2008). Aviation faculty need access to resources designed specifically to develop self-efficacy and metacognitive behaviors in their students to meet organizational goals. Where skills and knowledge gaps exist in understanding and applying behavioral education to students, flight training organizations must fully invest in methods to improve teacher abilities. Extensive litera- ture has demonstrated that teacher professional development is a key enabler for improving classroom instruction and student achievement (Ball & Cohen, 1999; Cohen & Hill, 2000; Cor- coran, Shields, & Zucker, 1998; Darling-Hammond & McLaughlin, 1995; Elmore, 1997; Little, 1993; National Commission on Teaching and America’s Future, 1996; Yoon, Duncan, Lee, Scar- loss, & Shapley, 2007). A recent study from the U.S. Department of Education found that teach- ers who receive substantial professional development boosted student achievement in their re- gion by approximately 21 percentage points (Yoon et al., 2007). Fifield and Kedzior (2004) de- scribed 10 characteristics of high-quality teacher professional development programs, including content-focus, enduring, collaboration, a part of daily work, continuous, coherent and integrated, inquiry-based, teacher-driven, informed by student performance, and self-evaluated. To improve IMPROVING PILOT TRAINING OUTCOMES 55 student metacognition and self-regulation instruction, university aviation programs must ensure adequate resources, time, and support are available to faculty as they refine their curricula. Assumed cultural setting causes. The organization recognizes and rewards innovation and continuous improvement by providing an evaluation framework that values innovative practices for teachers to measure their performance against the goal. Building adaptive-learning organizations requires educational leaders to help their instructors see the underlying causes of problems and new possibilities for shaping the future (Senge, 1990). They must ensure employees are empowered, aligned, and en- gaged in the vision to successfully accomplish goals (Burke, 2002). Leaders must also be persis- tent in building frameworks to recognize innovation and improvement in the classroom, simula- tor, and inflight environments. University aviation instructors need to examine their own prac- tices and policies, link theory to practice, improve performance, analyze problems, and apply evidence and evaluation to assess decisions (Valli, 1997). Building and communicating evalua- tion methods that recognize and reward improvement and progress towards those goals increase the probability of organizational success. Evaluation frameworks can also serve as a motivational tool for school administrators. Motivation consists of two subsystems: intrinsic, or internal thoughts and feelings, and extrinsic, which comes from the work environment itself (Silverman, 2004). For rewards to be effective, they must clearly identify the target population(s), establish a clear purpose for the program, en- list support, design assessment measures and schedules, communicate, and evaluate fairly (Han- sen, Smith, & Hansen, 2002). Often, non-financial rewards can provide an effective means of providing tangible and meaningful recognition for individuals or teams to meet goals or objec- tives (Silverman, 2004). Recognition of innovation and continuous improvement in aviation in- IMPROVING PILOT TRAINING OUTCOMES 56 struction is an effective tool for leaders to inspire learning organizations to grow and adapt to changing environments. The organization provides an organizational culture that is accepting of change and able to respond to it effectively. Organizational culture can be defined as, “the firmly implanted be- liefs and values of organizational members” that capture the subtle psychology of their work- place (Schneider, Brief & Guzzo, 1996, p. 11). Schein (1984) referred to culture as a pattern of basic assumptions a group invents, discovers, or develops to cope with external and internal problems. These assumptions have worked well enough in the past to be taught to new group members as the correct way to behave in their new environment (Schein, 1984). Cultural under- standing is desirable for all members of an organization but is critical for leaders as they seek to build and sustain an environment that is agile and responsive to change (Schien, 1984). Building a strong, resilient organizational culture capable of adapting to changes such as PL 111-216 be- gins with clear communications, and that starts with the organization’s leadership. Transforma- tional leadership is essentially a communications process to achieve specific goals, requiring competence in listening skills to earn trust and affect behavior (Berger, 2014). To inspire change to take root and produce intended results, leaders must focus on the climate of their organization (Schneider et al., 1996). University aviation programs must continue to learn and grow if they intend to survive in competitive marketplaces and environments where change occurs often (Senge, 1990). IMPROVING PILOT TRAINING OUTCOMES 57 Table 5 Summary of Organization Influences on State University’s Ground and Flight Instructors Assumed needs of Successful Instructors General Literature Cultural Model Influence 1 The organization values experimenta- tion and having faculty understand cog- nitive learning processes by providing learning opportunities and requiring their participation in these activities (Baker, 2006; Bandura, 2005; Ber- ger, 2014; Bjerke & Malott, 2011; Buckingham & Coffman, 2014; Clark & Estes, 2008; Fullingim, 2011; Locke & Latham, 2006; Pajar- es, 2009; Rueda, 2011; Schwandt & Marquardt, 2000; Senge, 1990; Valli, 1997) Cultural Model Influence 2 The organization provides the resources necessary for aviation training faculty to adopt student self-efficacy and meta- cognition programs in their pilot train- ing classrooms Ball & Cohen, 1999; Clark & Estes, 2008; Cohen & Hill, 2000; Corcoran, Shields, & Zucker, 1998; Darling- Hammond & McLaughlin, 1995; Elmore, 1997; Fifield & Kedzior, 2004; Little, 1993; National Com- mission on Teaching and America’s Future, 1996; Niemczyk & Ulrich, 2009; Niemczyk, 2008; Yoon, Dun- can, Lee, Scarloss, & Shapley, 2007 Cultural Setting Influence 1 The organization recognizes and re- wards innovation and continuous im- provement by providing an evaluation framework that values innovative prac- tices for teachers to measure their per- formance against the goal (Burke, 2002; Hansen & Hansen, 2002; Senge, 1990; Silverman, 2004; Valli, 1997) Cultural Setting Influence 2 The organization provides an organiza- tional culture that is accepting of change and able to respond to it effec- tively (Berger, 2014; Schein, 1984; Schien, 2010; Schneider, Brief & Guzzo, 1996; Senge, 1990) Conclusion This chapter has examined the literature regarding knowledge, motivation, and organiza- tional influencers affecting university aviation training programs as they seek to support faculty in providing self-efficacy and metacognitive instruction to students. Improving student perfor- mance and graduation rates is a critical component in addressing the larger community problem IMPROVING PILOT TRAINING OUTCOMES 58 of creating a larger pool of qualified pilots to fill a growing shortage of commercial aviation cockpits. A strong foundation in self-efficacy and metacognitive understanding has been shown to improve student learning, motivation, and persistence in complex training systems. By apply- ing the Clark and Estes (2008) Gap Analytic Conceptual Framework to examine knowledge, mo- tivation and organizational influences, the literature provides a framework to help flying training program leaders and faculty understand the relationship between goal orientations, learning styles, metacognition, and self-efficacy practices. It also demonstrates how to apply student self- efficacy and metacognition instruction to improve learner performance and meet organizational goals. This study examined this problem of practice from the perspective of a large university flight training program. The validation process for this study will be detailed in the Methodolo- gy section in the next chapter. IMPROVING PILOT TRAINING OUTCOMES 59 CHAPTER THREE Methodology This study examined knowledge, motivation, and organizational barriers facing universi- ty faculty as they seek to improve training system performance in response to a growing shortage of commercial pilots. The stakeholder of focus was the faculty, both ground and flight instruc- tors in the Aviation Training Program at State University. Specifically, this project examined faculty’s knowledge and use of self-efficacy and metacognitive behavioral strategies with their students to improve academic and in-flight performance and graduation rates. This study used purposeful sampling of State University aviation instructors as key stakeholders, and included input from AABI member programs as general stakeholders in the problem of practice. Under- standing the underlying knowledge, motivation, and organizational influences impacting the key stakeholder’s ability to apply self-efficacy and metacognition education is a critical step towards producing more graduates with the necessary skills to become safe, competent commercial pi- lots. The research questions that guided this study were: 1. What are the knowledge, motivation, and organizational needs required for faculty to im- plement and evaluate sound instructional design to improve student self-efficacy and metacognition? 2. What are the recommended knowledge, motivation, and organizational solutions to ad- dress those needs? Conceptual and Methodological Framework A conceptual framework is defined as a theory that presents the ideas and beliefs a re- searcher holds about a phenomenon under study, providing a roadmap as they seek to examine IMPROVING PILOT TRAINING OUTCOMES 60 and understand a project (Maxwell, 2013). It outlines the information necessary to infer appro- priate research questions, methods, and goals. Conceptual frameworks also focus on methodolo- gy and operational approaches to data gathering, measurement, and descriptions of findings (Merriam and Tisdell, 2016). For this study, a conceptual framework helps explain the relation- ship of the key stakeholder, the State University Aviation Training Program faculty, to the broader commercial aviation training community in the United States, represented by the Aviati- on Accreditation Board International (AABI), the nation’s largest college and university flying training accreditation organization. AABI’s goals are to stimulate collegiate aviation program excellence and self-improvement, establish uniform minimum educational quality criteria, and increase the credibility, integrity, and acceptance of collegiate aviation programs within institu- tions of higher education and all aspects of the aviation community, including industry, govern- ment and the public-at-large (About AABI, 2017). Figure 7 describes the conceptual framework for this study, showing the relationship between stakeholders and organizational performance goals and objectives. IMPROVING PILOT TRAINING OUTCOMES 61 Figure 7. Conceptual framework for stakeholder relationships. AABI establishes and evaluates standards for accreditation at over 30 schools around the world, including State University. Pro- gram goals are executed through university faculty, which include academic and contractor flight instructors. Source: Author. The relationship between the knowledge and motivation influencers to the organization influencers are explored as State University seeks to improve its pilot training program. State University’s goal for the aviation training program is to increase the number of graduates per year and improve the quality of in-flight performance of students seeking jobs as commercial pilots. State University’s program is accredited through the AABI, which represents over 30 col- lege and university flight training programs. Self-efficacy and Metacognitive practices Increase the number of students who successfully graduate State University's Aviation Training Program, and improve their performance during their check ride. Aviation Accreditation Board International (AABI) Cultural Settings and Cultural Models: understanding cognitive learning processes, supporting student learning needs, impact of PL 111-216, refining ISD approaches State University Aviation Training Program Cognitive learning processes, self- awareness/regulation practices, mastery orientation, self-efficacy, metacognition, impact of PL 111-216, refining ISD State University Aviation Training Program Faculty Academic & Flightline IMPROVING PILOT TRAINING OUTCOMES 62 Aviation training program faculty are typically functional experts, but often lack an equivalent level of expertise in adult education. Air traffic controller instructors, for example typically develop air traffic controller training programs and material. However, as Clark and Estes (2008) point out, they are rarely experts in learning domains or educational practice. Fur- ther, while program developers and instructors are motivated to provide the best training possi- ble, they may lack an understanding of how to best support their students attempting to master aviation-specific content (Tinto, 2012). Students will often rely on learning approaches that were successful for them in other, non-aviation contexts, but these approaches are not necessarily optimal for mastering the requirements of professional aviation. Flight training requires compe- tence in cognitive, psychomotor, and affective skills domains, and educators should provide learning strategies, metacognitive practices, and self-regulation skills to help students achieve a mastery orientation in each of these domains. This behavioral instruction is essential to help stu- dents organize, assimilate, and synthesize content into the flying environment. Research suggests that aviation training program faculty can improve student outcomes, measured by an increase in the number of graduates and their performance in the air, by improv- ing student self-efficacy and metacognition (Davis, Fedor, Parsons, & Herold, 2000). Knowledge influencers outline the importance of faculty understanding of the cognitive learning processes associated with flying training, self-awareness, and self-regulation practices. Motiva- tion influencers describe how faculty should find new and innovative ways to help their students achieve a mastery orientation of aviation rules and regulations, and understand their role in sup- porting student self-efficacy to improve performance. Organizational influencers include the processes, materials, policies, and procedures State University uses to encourage or inhibit edu- cation and training by understanding cognitive learning processes, supporting student learning IMPROVING PILOT TRAINING OUTCOMES 63 needs, and metacognition. Research indicated that Public Law 111-216, which brought about substantial changes to commercial pilot training standards, is forcing these programs to reex- amine their processes and driving a need to refine their instructional design to better support stu- dent self-efficacy (Bjerke & Malott, 2011; Depperschmidt, 2013; NTSB, 2010). The guiding methodological framework for this study was Clark and Estes’ (2008) per- formance gap analysis model. It is a systematic, analytical problem-solving process for identify- ing causes of performance gaps, clarifying organizational goals, and recommending improve- ments. This organizational process model, shown in Figure 8, begins by identifying and defining key organizational goals. Next, individual performance goals are assessed, and gaps between current and desired performance are identified. Clark and Estes (2008) identify three primary categories of these gaps: knowledge and skills, motivation to achieve goals, and organizational barriers. Each is examined separately and holistically to arrive at suggested performance im- provement areas. Proposed solutions are executed, refined, revised, and finally evaluated against the initial organizational goals. This model represents a continual cycle of examination, plan- ning, learning, execution, and reflection to align intent with action and increase the probability of successful change and growth. IMPROVING PILOT TRAINING OUTCOMES 64 Figure 8. The Clark and Estes Performance Gap Analysis Model. Source: Clark and Estes, 2008 Clark and Estes’ (2008) performance gap analysis model provides researchers with a wide variety of tools to gather and analyze data. Qualitative approaches, such as face-to-face interviews, observations, or document analyses, can be used to determine and define perfor- mance problems. Quantitative research methods, including surveys, tests, and correlational stud- ies generalizable to a wide audience, are also useful in defining areas of opportunity. Combining the two in a mixed-methods approach can be particularly effective in triangulating findings that yield recommendations for organizational change and improvement. The methodological framework for this study was a convergent parallel mixed-methods approach with descriptive surveys, interviews, observations, and artifact gathering (Creswell, 2014). Assumed knowledge, motivation, and organizational needs were generated based on per- sonal knowledge and related literature. These needs were validated using surveys, interviews, STEP 1 STEP 2 STEP 3 STEP 4 STEP 5a STEP 5b STEP 5c STEP 7 Identify Key Goals Identify Individual Performance Goals Determine Performance Gaps Analyze Gaps to Determine Causes Identify Knowledge/Skill solutions and implement Identify Motivation solutions and implement Identify Organizational Process and Material solutions and implement Evaluate Results, Tune Systems and Revise Goals IMPROVING PILOT TRAINING OUTCOMES 65 observations, a literature review, and content analysis. Research-based solutions are recom- mended and evaluated in a comprehensive manner. Assessment of Performance Influences In Chapter 2 of this study, four knowledge influencers, three motivation influencers, and four organizational influencers were identified that are potentially affecting State University fac- ulty’s ability to implement and evaluate student self-efficacy and metacognition practices in their programs. Table 6 provides a summary of each area. Tables 7, 8, and 9 align knowledge, moti- vation, and organizational (KMO) influencers with survey, interview, observation, and artifact inquiries. Researcher-developed survey, interview, and observational protocols were used with document analysis to determine factors influencing the key stakeholder goal of ensuring all avia- tion instructors (faculty and contractors) are aware of and practicing effective principles of in- structional design, specifically those that increase student self-efficacy and metacognition prac- tices to create more, better qualified graduates by January 7, 2019. Table 6 Summary of Assumed Knowledge, Motivation, and Organizational Influences on Faculty Assumed KMO Needs of Successful Aviation Instructors Knowledge Influences Declarative Conceptual Knowledge Aviation training faculty need to understand the relationship be- tween student metacognition, self-efficacy, and student success and understand their role in improving learning outcomes Aviation training faculty must understand performance goals re- lated to providing self-efficacy and metacognition instruction to their students Metacognitive Knowledge Aviation training faculty need to know how to apply metacogni- tive and self-efficacy practices in their own instruction and reflect on how they are performing relative to their goals Procedural Knowledge Aviation training faculty must know how to structure learning experiences to increase students’ self-efficacy and metacognition abilities in their programs, how to give effective feedback, how to communicate clear performance goals, and how to evaluate their teaching relative to student performance IMPROVING PILOT TRAINING OUTCOMES 66 Motivation Influences Goal-Orientation Aviation training faculty should desire to find new and innovative ways to help their students achieve a mastery orientation of the rules of aviation Self-Efficacy Aviation training faculty need to be confident that self-efficacy and metacognition instruction will improve student outcomes, and in their ability to provide this instruction as a way of improv- ing student outcomes and performance Faculty need to attribute student success or failure in part to their skill as an instructor Organization Influences Cultural Model 1 The organization values experimentation and having faculty un- derstand cognitive learning processes by providing learning op- portunities and requiring their participation in these activities Cultural Model 2 The organization provides the resources necessary for aviation training faculty to adopt student self-efficacy and metacognition programs in their pilot training classrooms Cultural Setting 1 The organization recognizes and rewards innovation and continu- ous improvement by providing an evaluation framework that val- ues innovative practices for teachers to measure their perfor- mance against the goal Cultural Setting 2 The organization provides an organizational culture that is ac- cepting of change and able to respond to it effectively Knowledge Assessment The literature review of knowledge issues associated with this problem of practice re- vealed four assumed knowledge influencers, including two declarative (conceptual), one meta- cognitive, and one procedural, as shown in Tables 3 and 6. Declarative conceptual knowledge involves the understanding of and ability to explain relationships between categories or principles, or the semantics of actions that take place (Krathwohl, 2002; McGill & Volet, 1997). A semi-structured interview process with open-ended questions that permit probes is an excellent way to assess this type of knowledge. Qualitative research is an inductive process, which allows researchers to gain understanding by focusing on a subject’s meaning and under- standing as they interpret it (Merriam & Tisdell, 2016). Using a survey question to examine be- lief in using specific behavioral tools and strategies can also provide important insight into de- IMPROVING PILOT TRAINING OUTCOMES 67 clarative conceptual knowledge. Metacognitive knowledge involves knowledge about cognition in general as well as awareness of and knowledge about one's own cognition (Pintrich, 2002). In reflecting on one’s own thinking processes, assessing metacognitive knowledge can best be done through open-ended interview questions. Krathwohl (2002) defines procedural knowledge as knowing how to do something. Interview questions are ideally suited for exploring procedural understanding, as it allows researchers to understand phenomenon of interest from the partici- pant’s perspective while uncovering what knowledge exists and what is missing (Merriam & Tisdell, 2016). Table 7 Summary of knowledge influences and assessment methods Assumed Knowledge Influence Assessment Declarative Conceptual Knowledge Aviation training faculty need to understand the relationship between student metacognition, self-efficacy, and student success to understand their role in improving learning outcomes Survey: How much can you influence decisions re- lated to curriculum in the Professional Flight Program? If you wanted to asses a student’s knowledge of a particular topic, which of the following tools would you most prefer to use? Which of the following do you believe is most important to increase a student’s re- tention of what they have been taught in previous lessons? Interview: Would you please describe the learning skills you feel are important for students to improve their performance? IMPROVING PILOT TRAINING OUTCOMES 68 Aviation training faculty must know the per- formance goals related to providing self- efficacy and metacognition instruction to their students Interview: Can you describe your school’s goals relat- ed to the aviation training program? Metacognitive Knowledge Aviation training faculty need to know how to apply metacognitive and self-efficacy practices in their own instruction and reflect on how they are performing relative to their goals Interview: Would you please describe your experience with teaching students “how to learn”? What strategies can you use as an instructor to increase a student’s memory of what they have been taught in previous lessons? Observation: Classroom and Simulator/Flight Procedural Knowledge Aviation training faculty must know how to structure learning experiences to increase stu- dents’ self-efficacy and metacognition abilities in their programs, how to give effective feed- back, how to communicate clear performance goals, and how to evaluate their teaching rela- tive to student performance Interview: What do you feel is your most effective technique for helping students when they struggle to learn a difficult subject? How has your education and experience prepared you to provide metacognition and self-efficacy instruction to your students? Can you give me an example? Observation: Classroom and Simulator/Flight Motivation Assessment Three assumed motivation influencers were identified through the literature review, in- cluding one Goal-Orientation and two Self-Efficacy influencers. Each of these influencers, de- scribed in Table 3, was examined through survey and interview protocols. Goal-orientation ex- amines why individuals engage in a particular activity (Anderman, 2015). Self-efficacy influ- ences, or self-perceptions that individuals hold about their own capabilities, explore belief in one’s ability (Pajares, 2006). As Bandura (2005) notes, a visualized future serves as a current IMPROVING PILOT TRAINING OUTCOMES 69 guide, as well as a motivator of behavior. As with knowledge assessments, survey questions and open-ended interview questions in a semi-structured format with periodic probing questions pro- vided excellent methods for examining motivation influences. Table 8 Summary of motivation influences and assessment methods Assumed Motivation Influence Assessment Goal-Orientation Aviation training faculty should desire to find new and innovative ways to help their students achieve a mastery orientation of the rules of aviation Survey: I believe instructors should make a priority of finding new and innovative ways to help students develop a mastery focus (a focus on constant learning and improvement) of the rules of aviation. Please indicate where you think a teacher’s priority should be on this continuum by sliding the bar to the value that corresponds with your beliefs Interview: What motivates you, personally to find new or innovative ways to help students achieve a mastery orientation of aviation skills? Self-Efficacy Aviation training faculty need to be confident that self-efficacy and metacognition instruction will improve student outcomes, and in their ability to provide this instruction as a way of improving student outcomes and performance Survey: My students would benefit from learning how to enhance their self-perception of their capabilities, motivation, and ability to influence events affecting their lives. Teaching strategies designed to increase student confidence in their ability to learn aviation-specific concepts will increase their performance in flight. Using the scale bar below, please rate your confidence in accomplishing the following tasks: IMPROVING PILOT TRAINING OUTCOMES 70 - Keep students on task during difficult assignments - Increase a students' understanding of what they have been taught in previous lessons - Get students to believe they can do well in school work - Positively influence the performance of students who are struggling - Help students master difficult tasks or subjects - Motivate students who are struggling. Interview: Can you provide an example of how the design of your course(s) can be improved to help teach students how to learn the content more effectively? In your opinion, how important is instruc- tion on self-efficacy and metacognition to improving student graduation rates? Faculty need to attribute student success or failure in part to their skill as an instructor Survey: To what extent is the instructor responsible for a student’s strong or weak performance? Interview: When you see students struggling with the program, what factors do you consider to explain that poor performance? Organization/Culture/Context Assessment The literature review identified four organization influencers associated with this problem of practice, including two cultural models and two cultural setting influences, outlined in Table 3. As mentioned in Chapter 2, cultural models refer to a shared understanding of how the world works (or should work), and cultural settings indicate a visible, concrete manifestation of cultural models used to assess and improve organizational performance. The principle methods for as- sessing cultural model and setting influences are surveys and open-ended interview questions. IMPROVING PILOT TRAINING OUTCOMES 71 Documents and artifacts, which can be considered a quantitative component of a research study, provide valuable insight into codified organizational expectations, as well as important infor- mation and perspective prior to other data collection methods (Merriam & Tisdell, 2016). Table 9 Summary of organization influences and assessment methods Assumed Organization Influence Assessment Cultural Model Influence 1 The organization values experimentation and having faculty understand cognitive learning processes by providing learning opportunities and requiring their participation in these activi- ties Survey: I believe my academic program provides adequate education and training to faculty on practices related to teaching student re- flection and self-motivation. Documents and Artifacts: Faculty PD/Training Program Information Cultural Model Influence 2 The organization provides the resources neces- sary for aviation training faculty to adopt stu- dent self-efficacy and metacognition programs in their pilot training classrooms Survey: To what extent does your collegiate flight program provide the resources you need to improve a student’s understanding of re- flection and self-motivation? Interview: How do you think the University can better support faculty in helping students achieve subject mastery in aviation? Documents and Artifacts: Faculty PD/Training Program Information Cultural Setting Influence 1 The organization recognizes and rewards inno- vation and continuous improvement by provid- ing an evaluation framework that values inno- vative practices for teachers to measure their performance against the goal Survey: I believe my academic program rewards innovation and continuous improvement in syllabus development. IMPROVING PILOT TRAINING OUTCOMES 72 Cultural Setting Influence 2 The organization provides an organizational culture that is accepting of change and able to respond to it effectively Survey: To what extent do you believe your colle- giate flight program is open to new ideas about instruction? Interview: I’d like to ask your recommendations for how the University can improve the current instructional design of the aviation training program to improve student learning Participating Stakeholders and Sample Selection The stakeholder population of focus for this study is the academic and flight faculty from State University’s Aviation Training Program. The academic faculty consists of approximately 21 full-time and part-time State University professors. The flight faculty consists of approxi- mately 46 FAA-certified part-time and full-time contractor instructors who provide both simula- tor and flight training at a nearby airport. Collectively, these instructors form the primary sample group for this study. State University faculty have noted expertise and experience in many aspects of profes- sional flight, including airline, military, and civilian flight education and training, and they con- tinue their involvement with industry and in research and scholarly activities. Since State Uni- versity is accredited through the AABI and has met membership requirements for the UAA, the potential for realizing generalizable results applicable outside of the State University context is improved. All members of the faculty were provided a link to the online survey, increasing re- sponse reliability for this study. Identical surveys were provided to all respondents, and the same interview question guides were used for all interviews (See Appendices A through D). IMPROVING PILOT TRAINING OUTCOMES 73 Data Collection and Instrumentation To assess and validate faculty understanding and use of metacognitive and self-efficacy strategies in aviation training required a combination of survey, interviews, observations, and data analysis methods. Each method provided unique insight into different elements of the re- search questions. Surveys offered a method of collecting generalizable information across the entire faculty, as well as other member organizations of the AABI. They are instructive for de- termining the current, or “as is,” state of an organization. Interviews provided detailed, personal- ized responses and specific information on respondent’s perspectives. Observation, which oc- curs in the setting where the phenomenon of interest naturally takes place, represents a firsthand encounter with evidence, providing emotional context to the topic while adding a layer of depth and richness to the data (Merriam & Tisdell, 2016). It represents a triangulation step, where dif- ferent data sources are examined to build a coherent justification for themes used to compare survey responses with observed behavior (Creswell, 2014; Maxwell, 2013). Finally, documents and artifacts were collected to provide an understanding of current approaches without intruding upon or altering the setting (Merriam & Tisdell, 2016). Surveys An online survey was provided to all members of State University’s Aviation Training faculty, including academic and flight training instructors. The survey link, hosted by Qualtrics (www.qualtrics.com) was also shared with 60 program leaders of flight training programs ac- credited by AABI, to expand the generalizability of the findings. Surveys provide an ideal method for obtaining consistent, accurate data and a broad understanding of current practices, perceptions, and processes related to the research questions. For this study, survey links were sent to 127 individuals (N=127), including all 21 State University academic faculty members, all IMPROVING PILOT TRAINING OUTCOMES 74 46 contract flight instructors, and 60 members of the AABI. A total of 65 responses were rec- orded (n=64), representing a 51% response rate for this nonrandom convenience sampling. All participants were educators in UAA-member, AABI-accredited aviation training programs either at State University, or in Canada or the United States. Although convenience sampling is not typically considered optimal for obtaining generalizable results, the homogeneity of respondent occupations and affiliation provides a consistent sampling base with common interest in the problem of practice (Christensen & Johnson, 2014). Survey sampling criteria. The following criteria were used to determine eligibility for survey participants: Criterion 1. All participants must be existing employees of State University, the flying training contractor, or academic faculty from UAA-member, AABI-accredited university flight training programs. Consultants or advisors from different university departments may not have equity in the outcomes of these programs, and may lack sufficient knowledge or understanding of existing efforts to improve student self-efficacy and metacognition at State University. Criterion 2. All participants must be considered “faculty.” For the purposes of this study, faculty includes Aviation Training Program instructors on staff with State University, con- tracted CFI’s directly supporting flying training activities (simulator or flight) for the State Uni- versity program, and flight training educators from UAA-member, AABI-accredited schools. Criterion 3. Flight training instructors must be FAA-certified and hold a minimum of a CFI-single engine rating. Instructors who have not been or who are not FAA-certified may pro- vide inaccurate or false information regarding flight instruction rules, procedures, or guidelines. Survey instrumentation. Due to the large cluster sample size (approximately 127), indi- vidual surveys are appropriate to identify and measure general performance gaps in current fly- IMPROVING PILOT TRAINING OUTCOMES 75 ing training programs. Questions will explore the extent to which faculty and contract instruc- tors prioritize, emphasize, and conduct actions and/or programs to support student self-efficacy and metacognition. The survey was tailored to evaluate pilot training methodology as it relates to the accomplishment of Air Transport Pilot (ATP) ratings required for employment as a com- mercial pilot. Self-reported data from program instructors at accredited aviation training pro- grams like State University can be used to understand and expose potential KMO influencers that fail to support student metacognition and learning. This survey attempted to examine the current state of student support in aviation-specific instruction domains at State University (the “as is” condition). The survey was written exclusively in English, since Amendment 164 to the Interna- tional Civil Aviation Organization Annex 1 designated English as the official language for all pilots and ground controllers as of March 2008 (Personnel Licensing FAQ, 2012). The survey items were derived from the specific KMO influencers identified in Chapter 2, and were designed to uncover faculty awareness of, comfort and experience with, and belief in metacognitive and self-efficacy practices designed to improve student learning outcomes and performance in an aviation context. The survey questions used nominal, ordinal, interval, and ratio scales of measurement to provide insight into whether metacognition and self-efficacy prac- tices are deemed desirable in current instructional design. A copy of the survey instrument, as viewed on the Qualtrics website link, can be found at Appendix B. Survey responses from UAA-member, AABI-accredited program educators provided generalizable perspectives on metacognition and self-efficacy understanding and practice from different programs around the country. This, in turn helped inform understanding of the issues affecting State University’s program on a broader contextual level. IMPROVING PILOT TRAINING OUTCOMES 76 Survey data collection procedure. Program instructors at State University’s Aviation Training Program were provided a voluntary online survey hosted on the Qualtrics website. The survey was opened and invitation emails and links were sent via State University’s email service on October 20, 2017, which was two months into the fall term, allowing teachers time to settle in for the semester while deconflicting from mid-term and final examinations. The survey open season also provided respondents time to return their responses before the Thanksgiving holiday. The survey was closed on November 10, 2017. Survey release and distribution was coordinated with program administrators to request approval, gain access, and encourage maximum partici- pation. Survey administration, originally planned prior to interviews and observation, had to be postponed due to logistics issues and refinements to the survey questions. Interviews Formal and informal interviews were conducted with twelve members of State Universi- ty’s aviation training program faculty--six academic faculty and six contract flight instructors. These targeted interviews provided a deeper understanding of State University’s current practic- es, helped identify performance gaps, and illuminated potential solutions. Interviews enabled a rich, contextual understanding of what instructors knew about self-efficacy and metacognition and the behavioral strategies they use in their current program design and execution. Merriam and Tisdell (2016) note that interviews allow researchers to enter into the other person’s perspec- tive and expand understanding through follow-up and clarifying questions. Further, they provid- ed an opportunity to gain deeper perspective on teacher comfort and understanding in applying these behavioral strategies towards organizational goal attainment. Interview sampling criteria. The following criteria were used to determine eligibility for interview participants: IMPROVING PILOT TRAINING OUTCOMES 77 Criterion 1. All participants must be existing employees of State University or the flying training contractor. Consultants or advisors from different departments may not have equity in the outcomes of these programs, and may not have sufficient knowledge or understanding of ex- isting efforts at State University to improve student self-efficacy. Criterion 2. All participants must be considered “faculty.” For the purposes of this study, faculty includes State University-employed Aviation Training Program instructors or CFI’s contracted by State University directly supporting flight or simulator training. Criterion 3. Flight training instructors must be FAA-certified and hold a minimum of a CFI-single engine rating. Instructors who have not been or who are not FAA-certified may pro- vide inaccurate or false information regarding flight instruction rules, procedures, or guidelines. Interview instrumentation. A series of researcher-developed interview questions were created to examine the KMO influencers affecting State University faculty’s understanding of self-efficacy and metacognitive behavioral strategies. Interview questions were derived from Patton’s (2002) six types, each designed around the research questions. Questions were config- ured to gather awareness and meaning through the interview participants’ perspective, and pro- vide depth and breadth of understanding of the current faculty/instructor efforts to enhance stu- dent self-efficacy and metacognition (Maxwell, 2013). Interview data collection procedure. Interviews were conducted during the week of October 2-6, 2017, at State University’s flying training facility. Six academic instructors (one faculty associate and five lecturers; see Table 10), and six contractor flight instructors (see Table 11) were interviewed. All interviews lasted a minimum of one hour, with some going longer based on participant willingness and interest. Table 10 IMPROVING PILOT TRAINING OUTCOMES 78 Interview Results from State University Academic Faculty State University Academic Faculty Interview Matrix On Staff Interviewed Percentage Interviewed Faculty Associate 11 1 9% Lecturer 8 5 63% Associate Professor 2 0 0% Total: 21 6 28.6% Table 11 Interview Results from State University Flight Training (contractor) Faculty State University Flight Training Faculty (contractor) Interview Matrix On Staff Formal Interview Percentage Interviewed CFI 46 5 11% All available academic faculty during the interview period were solicited for participa- tion, and all agreed. Experience, age, sex, and race were not considerations in the selection crite- ria. All interview participants were male, over 40 years old, and had prior U.S. military or airline training and experience. Academic instructors were interviewed with the permission of State University’s Aviation Training Program leadership. At the time of the interviews, the makeup of the academic instructional staff was approximately 76% male, 24% female. All interviews took place in the faculty member’s personal offices, and were recorded and transcribed for accuracy and content. The contractor flight instructors were selected in consultation with program management and chosen by the site director to minimize the impact to scheduling and operational require- ments. Experience, age, sex, and race were not considerations in the selection criteria. At the IMPROVING PILOT TRAINING OUTCOMES 79 time of the interviews, the makeup of the contractor’s staff was approximately 87% male, 13% female, with over 80% of all instructors below age 30. All interview participants were male, less than 35 years old, FAA-certified single and multi-engine evaluators, and considered experienced aviators by their contractor employer. Not all participants had undergraduate degrees, and some were current State University students. Interviews took place individually in flight briefing rooms at the contractor’s flight line facility, and recorded and transcribed for accuracy and con- tent. These rooms are very small (approximately 10’ x 10’), with no windows and a single table and two to three chairs. They are designed for quiet, private, and uninterrupted instructor- student training or debriefing sessions, making them ideal for these interviews. For all interviews, once the recording was started, a standardized introduction was read aloud to all participants (Appendix C) before the first question was presented. Each was given a copy of the Informed Consent form (Appendix E), which they signed and returned to the re- searcher. Once all questions had been asked and answered, participants were given an oppor- tunity to ask questions of their own. At the conclusion of the interview, the researcher provided his contact information to each participant. All recordings were transcribed using on online tran- scription service that converted the audio files into Microsoft Word documents for coding and detailed data analysis. A researcher-designed Interview Protocol (Appendix D) was assembled to ensure align- ment with the research questions, consistency between lines of inquiry with each participant, and accuracy and integrity of the lines of inquiry (Patton, 2002). Probes were used to obtain a deeper understanding of subject areas where necessary. Several additional informal interviews and con- versations occurred during the data-gathering period with interview and observation participants, without coding to follow up on open lines of questioning or to dig deeper into areas of interest. IMPROVING PILOT TRAINING OUTCOMES 80 The State University Flight Training Program Director was interviewed to generate a more complete interpretation of the program’s goals and objectives, history, filtering and training practices for instructors, and contractor selection and evaluation processes. Observation Observations of live student training at State University’s aviation training program were conducted to collect information on how current instructional methodology supports or does not support using metacognitive or self-efficacy strategies in class. A total of three academic class- room training periods, two simulator training sessions, two pre-and post-flight briefings, and one ground training session were observed during the week of October 2-6, 2017, on-site at State University’s academic and flying training facilities. Each of these observations gathered infor- mation on current practices designed to support student self-efficacy and metacognition. Observation sampling criteria. The following criteria were used to determine eligibil- ity for observation opportunities: Criterion 1. All classroom, simulator, and flight training observation sessions must be taught by an existing employee of State University or of the flying training contractor. Consult- ants or advisors from different departments may not have equity in the outcomes of these pro- grams, and may not have sufficient knowledge or understanding of existing efforts at State Uni- versity to improve student self-efficacy and metacognition. Criterion 2. All classroom, simulator, and flight training observation sessions must be taught by primary instructors considered “faculty.” Substitute or guest lecturers were not ob- served. For the purposes of this study, faculty includes State University-employed Aviation Training Program instructors or CFI’s contracted by State University directly supporting flight or simulator training activities. IMPROVING PILOT TRAINING OUTCOMES 81 Criterion 3. Flight training instructors must be FAA-certified and hold a minimum of a CFI-single engine rating. Academic instructors who are not FAA-certified may provide inaccu- rate or false information regarding flight instruction rules, procedures, or guidelines. Observation instrumentation. Direct observations were appropriate for this study to gather an understanding of potential flying-training gaps during classroom instruction, simulator rehearsal, and during in-flight operations. In all observation sessions, the researcher was solely an observer, not a participating member of the class or a member of a simulated or actual flight crew. Merriam & Tisdell (2016) note that to be effective, observations must be systematic, ad- dress a specific research question, and be subjected to checks and balances to produce trustwor- thy results. To that end, researcher-created protocols were used for classroom, simulator, and pre- and post-flight observations. Each protocol contained three sections: pre-assessment, activi- ties, and reflections and interpretations. The protocols created a framework for the researcher to capture essential elements of the environment, and served as a prompt for participants to reflect on their general and specific impressions. A protocol for both classroom and simulator/flight observations is included in Appendices F and G. Observation data collection procedure. Observations occurred over a five-day period during the fall semester. They provided an enhanced perspective into how flying training, both on the ground and in the air, is conducted, focusing on how self-efficacy and metacognitive prac- tices are introduced, practiced, and evaluated. Access to classrooms, simulators, and aircraft was coordinated through State University’s Aviation Training Program administration and the local contractor site manager to ensure safety, school guidelines, and FAA requirements were met. Both group and individual learning events were observed after conducting a review of syllabus and lesson plans. The observations were not used to evaluate effectiveness, but to obtain an un- IMPROVING PILOT TRAINING OUTCOMES 82 derstanding of teacher understanding, comfort, and use of self-efficacy and metacognitive strate- gies during their training sessions. During classroom observations, the researcher was seated in the back of the room and remained quiet for the duration of class. Only one instructor made mention of the researcher’s presence and purpose. The researcher took handwritten notes to record observations, impres- sions, and perspectives. No audio or video recordings of the class sessions were made. Other than one equipment-specific laboratory, all classrooms were multi-use rooms, with a podium, computers, a projector and screen, tables aligned in rows, no windows, and no assigned seating. All rooms were well lit, at a comfortable temperature, and free from distraction from outside. A reliable Wi-Fi signal was available in all classrooms. Two simulated flight training sessions were observed. During training, the researcher stood behind the students and the instructor, out of sight from the students but connected to the audio system and able to hear all communications between students, instructor, and the simulated air traffic control during the training period. Prior to the start of training, the instructor informed the students what the researcher was doing, but made no mention of the ongoing observation dur- ing the training sessions. Simulation training periods are scheduled as classes for students in the Flying Training Program, with two students supported by a single faculty member who serves as the simulator operator, lesson instructor, and all external communications (ATC, other aircraft, weather advisory, etc.). Pre-flight ground training, the simulated flight period, and the debrief- ings were all observed. State University uses several ground-based, fixed-wing aircraft flight simulators for FAA-approved instruction in their Flight Training Program. Simulation and mission rehearsal devices have been an essential component of flying training for almost as long as powered flight IMPROVING PILOT TRAINING OUTCOMES 83 itself. They are an effective tool for learning and applying behavioral strategies vital to profes- sional pilots. As Rolfe and Staples (1988) noted, ground-based simulation provides five specific benefits to flight training programs: 1. Increased efficiency 2. Increased safety 3. Lower training costs 4. Ability to practice situations which cannot be rehearsed in actual flight 5. Reduction in operational and environmental disturbances As described in the previous literature review, each of these elements are impacted by student understanding and application of self-efficacy and metacognitive behaviors. Simulator training is an ideal setting for rehearsing these learning characteristics. For this study, the observed simulator periods were conducted in simulators classified by the FAA as Flight Training Devices (FTD) and configured as a dual engine turboprop aircraft with control loading, visual, and audio inputs representative of actual flight conditions. This FTD was a fixed base, non-motion device powered by a desktop PC supported by several linked and connected computer screens, each approximately 3’ x 4’. The FTD was housed in a quiet, dimly lit classroom of a school building on campus with no windows, set to a comfortable tem- perature, and free from distraction from outside. A reliable Wi-Fi signal was available and used during the training period for student tablet computer-based procedural training, which is also conducted in flight. For safety and contractual reasons, observation of actual flight training was not possible during this study. However, with the consent and support of State University’s contract program manager and site director, pre- and post-flight procedural and safety briefings, as well as sylla- IMPROVING PILOT TRAINING OUTCOMES 84 bus-directed ground training sessions were observed. These training events were held on the flight line in contractor-provided and -maintained facilities. During these observations, the re- searcher was seated in the back of the room and remained quiet for the duration of training peri- od. The contract flight instructors did not mention or reference the researcher at all during train- ing. The researcher took handwritten notes to record observations, impressions, and perspec- tives. No audio or video recordings were made. Observations occurred in the same briefing rooms as the interviews. One pre-flight and one post-flight briefing were observed, specifically to examine if self- efficacy and metacognitive practices were outlined in either the contract or the performance standards of the contractor. Pre-flight briefings are standard operating procedure in both civil and military flight training, and are used to ensure critical preflight elements such as weather, weight and balance, aircraft status, and flight planning are obtained and understood. They are also often used to define training objectives prior to walking out to the aircraft. Post-flight de- briefings are used to review flight events from takeoff to landing, uncover lessons learned, ac- complish gradebook and logbook entries, and conduct syllabus reviews. Ground training ses- sions are syllabus-driven events to train operational procedures such as traffic patterns, instruc- tional techniques, or weather radar operations. They can be taught in classrooms, briefing rooms, or at the aircraft itself, depending on content and instructor preference. Pre-flight, post- flight, and ground training sessions are of varying lengths and intensity, but they are a critical part of the Elaboration process and connecting information within the IPM model (see Figure 3), moving information from working to long-term memory. Total observation time for all events did not exceed a single class, simulator, or ground training period (approximately 50 minutes), or one pre- or post-flight debriefing period (between IMPROVING PILOT TRAINING OUTCOMES 85 10-45 minutes). Observation data was used to gain further insight and understanding of the KMO influencers surrounding these classes, and to obtain a clearer understanding of the cultural influencers working within State University’s aviation instruction protocols and methodologies. Documents and Artifacts Documents and artifacts provide insight into cultural and organizational influencers with- in State University’s Aviation Training Program. Current syllabi, instructor guides, lesson plans, and texts were collected and examined for evidence of support for metacognition and self- efficacy instruction in this program. Access was requested through State University’s Admin- istration and faculty. The results of the document and artifact review were used to inform and focus survey and interview questions related to the overall research effort. A list of required documents can be found in Appendix H. Reliability and Validity To ensure reliability and validity in this study, interviews, surveys, observations, and document/artifact gathering methods were all used to answer the research questions. Using mul- tiple methods to answer the same research questions (triangulation), and combining the results improves internal validity by providing different perspectives on the use of metacognitive and self-efficacy practices in aviation training. The mixed-methods approach also allows the re- searcher to examine how survey responses align to interview responses, and how those elements align in practice during classroom, simulator training, and flight training. The results of each research method were examined independently and holistically to arrive at a blended conclusion, providing a deeper, richer understanding of context and applicability to aviation education. Raw data, in the form of field notes, sampling returns, and records have been preserved and included in the final report. IMPROVING PILOT TRAINING OUTCOMES 86 Credibility and Trustworthiness of Data Four strategies were used to ensure credibility and trustworthiness of the data. First, data obtained through surveys, interviews, observations, and document and artifact examination was triangulated, mixed, and compared to the research questions. Data triangulation, where different converging sources of data are examined to build themes, is a powerful strategy for increasing internal validity of a study (Creswell, 2014; Merriam & Tisdale, 2016). Second, survey items and protocols, interview questions and protocols, and observation protocols were peer, faculty, and committee reviewed, pilot tested, and ultimately approved through the Institutional Review Board process of two universities (USC and State University) prior to use in this study. Survey and interview questions were assembled using Patton’s six rules of questions (Patton, 2015). These instruments were subjected to a review and approval process involving academic rigor and testing to ensure reliability. Third, survey anonymity and interview confidentiality were careful- ly and strictly maintained throughout the data gathering process. All primary and backup copies of recordings, transcriptions, notes, and responses were kept in a confidential, password- protected computer system. Finally, a member-checking strategy was employed during data analysis. Creswell (2014) describes the importance of bringing the results of data gathering back to a survey, interview, or observation participant to determine whether or not they feel the con- clusions are accurate. This may involve follow up interviews or informal discussions to make sure assumptions and inferences are consistent with the intent of the participants. Role of Investigator The researcher was a U.S. Air Force Lieutenant Colonel and a command pilot with over 20 years of aviation experience in single and multi-engine, rotary wing and fixed wing aircraft, and over 2,600 hours of flight time. He is an Air Force Master Instructor, having served in fly- IMPROVING PILOT TRAINING OUTCOMES 87 ing training operations, acquisition, management, and leadership positions for the past 10 years, including commanding an Air Force Training Squadron. This experience enabled him to com- municate clearly with high levels of understanding on complex aviation-specific topics, as well as student training issues related to aviation. The Air Force adheres to the same Federal Aviation Regulations as commercial, non-military flying operators, so a shared understanding of aviation rules and procedures existed before the study began. The researcher was not affiliated with State University in any way, past or present. His role was to conduct a problem-solving investigation as a method of improving the organization’s performance. Ethics This mixed-methods research study focused on meaning and understanding to answer the research questions. The importance of making ethical choices when conducting this study was acknowledged, since much of the data collection entailed interview and conversation. Informed consent forms were provided to and signed by all participants at the commencement of the study (see Appendix H). Glesne (2011) described the importance of informed consent to ensure partic- ipants are aware their participation is voluntary, understand that all discussions will be kept con- fidential, and know that they can withdraw at any point without penalty. To ensure the safety of the participants, the researcher submitted his study to the University of California Institutional Review Board (IRB) and followed their rules and guidelines regarding the protection of the rights and welfare of the participants. State University’s IRB accepted the results of the Univer- sity of California’s IRB results and approved the data collection strategy this study. The predominant research location for this study was at State University. The researcher was not affiliated with this program or school in any way other than this study, which was con- ducted under the approval of the Aviation Training Program department chair. The researcher IMPROVING PILOT TRAINING OUTCOMES 88 received prior permission to audio record all interviews. Participants were reminded that the re- searcher will not provide any incentives, which minimized the possibility that participants would feel coerced to participate in any way. IMPROVING PILOT TRAINING OUTCOMES 89 CHAPTER FOUR Results and Findings Chapter four presents the findings and analysis of this convergent, parallel, mixed- methods research study, investigating the knowledge, motivation, and organization resources (KMO) needed by aviation training program faculty to improve instructional strategies aimed at enhancing student performance and graduation rates. Chapter one outlined the problem of prac- tice and defined key organizational performance goals, including State University’s intent, by January 7, 2019 to have all aviation instructors aware of and practicing effective principles of instructional design, specifically those that increase student self-efficacy and metacognition. Two research questions were identified to guide this study: 1. What are the knowledge, motivation, and organizational needs required for faculty to im- plement and evaluate sound instructional design to improve student self-efficacy and metacognition? 2. What are the recommended knowledge, motivation, and organizational solutions to ad- dress those needs? Chapter two introduced the Clark and Estes (2008) Performance Gap Analysis model, used to define the key KMO elements for this study. Table 6 outlined and organized each as- sumed KMO influencer. A comprehensive literature review was conducted to explore current practice and understanding, while refining the influences and barriers affecting university flying training programs. Chapter three established the data collection methodology for this study, cre- ating a conceptual framework (Figure 7) and aligning the collection efforts with the research questions. IMPROVING PILOT TRAINING OUTCOMES 90 This study used purposeful sampling of State University aviation instructors as key stakeholders, and gathered additional quantitative input from program leaders in the AABI net- work to increase the generalizability of the results. A triangulation strategy was applied during data collection and analysis to improve reliability and validity of the study findings. Each of the 11 assumed KMO influences outlined in Table 6 were examined through surveys, interviews, observations, and artifact analysis to assess the validity of the assumptions and produce findings that informed suggestions for closing performance gaps. Chapter four concludes with a sum- mary of the results (Table 13), which drove the recommended solutions described in Chapter five. Validation When examining assumed influences for this problem of practice, the term “validated” indicates that the research data has uncovered a KMO performance gap that must be addressed for faculty to effectively implement self-efficacy and metacognitive strategies to achieve the stated stakeholder goal. Assumed influences were determined to be either validated, not validat- ed, or could not be determined. Assumed KMO influence were considered validated if 50% or more of responses across all data sources agreed or indicated that the influencer was necessary. An assumed influence was considered not validated if these conditions were not met, indicating the performance measure is working and no changes are necessary to achieve organizational goals. An assumed influence was considered could not be determined if the results of data collection were insufficient or inad- equate to determine whether a performance gap exists, indicating further study is needed. IMPROVING PILOT TRAINING OUTCOMES 91 Participating Stakeholders The stakeholder group for this study was State University’s flying training program aca- demic and contractor flight instruction faculty. Additional quantitative data was obtained by surveying program leaders from Aviation Accreditation Board International (AABI) member programs across the United States. Survey Sampling Nonrandom convenience sampling was conducted through a researcher-designed online survey, distributed to participants from State University and program leaders within the AABI. A total of 127 (N=127) links were sent, and a total of 65 responses (n=65) were recorded, for a 51% response rate. Respondent demographics indicated a wide degree of training and experi- ence in adult education. The majority of survey participants affiliated as academic faculty (64%, n=25, Figure 9), with a most considering themselves full-time (56%, n=20, Figure 10). Figure 9. Survey participant employment affiliation (n=39). Source: Author. IMPROVING PILOT TRAINING OUTCOMES 92 Figure 10. Survey participant employment alignment (n=36). Source: Author. Approximately 34% indicated they had received between one and six months of formal training in adult education, with the majority of respondents (63%) stating they had less than two years (Figure 11). The number of years respondents had worked in adult education varied wide- ly, with 21% (8 of 39) participants stating one year, 18% (7 of 39) reporting more than 20 years, and several clustered groups between these two values (Figure 12). Figure 11. Amount of training in adult education reported by survey participants (n=46). Source: Author. IMPROVING PILOT TRAINING OUTCOMES 93 Figure 12. Number of years survey participants have worked in adult education (n=39). Source: Author. Approximately 41% (16 of 39) had been with their respective flight training programs less than three years, with results recorded out to over 20 years (Figure 13). Figure 13. Number of years survey participants have worked in the professional flight program (n=39). Source: Author. IMPROVING PILOT TRAINING OUTCOMES 94 The majority of survey respondents (66%, or 25 of 38) had no prior military experience (Figure 14), while 64% (25 of 39) had served in the airline industry or in commercial aviation in some capacity (Figure 15). Figure 14. Amount of time total time respondents reported serving in the active duty military, Guard, or Reserves (n=38). Source: Author. Figure 15. Amount of time total time respondents reported serving in the airline industry or commercial aviation (n=39). Source: Author. Interview Sampling Nonrandom, purposeful sampling was used to conduct in-person interviews to gather contextually-specific data on State University’s aviation training program. All available academ- ic faculty during the October 2017 data collection period were interviewed, and all available IMPROVING PILOT TRAINING OUTCOMES 95 flight instructors were interviewed within operational and time constraints established by the contractor. Table 10 (Chapter three) described the sampling pool from State University’s aca- demic faculty, and Table 11 (Chapter three) showed the number of certified flight instructors (CFIs) employed and interviewed. In total, approximately 29% of the academic faculty and 16% of the flight training faculty were interviewed formally (recorded, using the interview protocol) and informally. Informal interviews were used to gather insight into current operations, issues, barriers, and procedures used in flying training at this site and elsewhere. Figure 16 provides interview demographics from State University faculty participants, and Figure 17 provides the same information for flight training (contractor) faculty. Figure 16. Interview demographics from State University Academic Faculty. Source: Author. Figure 17. Interview figures from State University Flight Training (contractor) Faculty. Source: Author IMPROVING PILOT TRAINING OUTCOMES 96 According to State University’s Academic Affairs Manual, faculty positions, which in- clude the flying training program fall into three general categories (organization’s website, 2018). Faculty Associates are in non-tenure accruing positions qualified by training and experi- ence to teach university-level courses and appointed by the university on semester or annual terms. Lecturers are also in non-tenure and non-tenure-track positions employed to teach under- graduate or clinical courses, with appointments varying in length from one to three years. Asso- ciate Professors are traditional tenure-track appointments, serving as members of the school’s Academic Assembly and eligible for promotion and tenure review. Nearly all contracted flight instructors were under 30 years of age, with less than 1,500 hours of flight time and less than two years as a certified flight instructor. Most reported they were attempting to build flight time and experience required to achieve their R-ATP rating so they could submit applications for commercial air carriers. The contractor site lead reported that once CFIs reach the minimum qualifications for obtaining their R-ATP rating they “almost al- ways immediately move on to an airline job” (contractor site lead, personal communication, Oc- tober 2, 2017). Airlines pay their pilots substantially more money, provide a more stable, pre- dictable lifestyle, and offer the ability to select routes and work days. Airline pilots typically af- filiate with profession- or company-specific labor unions, who provide privileges as well as pro- tections from potential furloughs based on the amount of time an employee has been with the company and in the union. There is strong incentive, therefore, for young aviators to obtain the minimum necessary FAA ratings and join an airline as quickly as possible. Table 12 shows the breakout between the academic and flight line faculty formal Inter- view Participants (IP) at State University’s aviation training program. Number assignments were randomly applied by the researcher during coding. IMPROVING PILOT TRAINING OUTCOMES 97 Table 12 Interview Participants, broken out between academic and flight line faculty Interview Participant (IP) Breakouts State University Academic Faculty IP 1 IP 3 IP 6 IP 8 IP 9 IP 10 State University Flight Training (contactor) Faculty IP 2 IP 4 IP 5 IP 7 IP 11 Observation Sampling Direct observations were made of three academic classes, two simulator training sessions, two pre-and post-flight briefings, and one ground training session. Researcher-developed proto- cols were used for all observation periods to create descriptive and reflective field notes. These field notes were then used to obtain findings related to the research questions and the problem of practice, minimizing internal threats to validity (Creswell, 2014). No audio or video recordings were made during these observations to minimize impact on student training. Similarly, no eval- uations, either in academics or on the flight line were observed. The site lead for the contract flying training program authorized observations of ground- based flying training only. No in-flight observations were conducted due to operational and safety reasons. The dynamic nature of the flying training environment kept the scheduling pro- cess fluid and random, but the findings provided important perspectives that would not have been available through other research methods. These findings were essential in validating as- sumptions on KMO influences in State University’s overall flying training program. Documentation and Artifact Sampling Documents and artifacts describing and affecting State University’s flying training pro- gram were obtained during this study as a key ingredient of the study’s triangulation strategy. IMPROVING PILOT TRAINING OUTCOMES 98 Syllabi, instructor guides, and lesson plans were obtained from academic faculty; lineup cards, briefing guides, and flight profiles were obtained for simulator training; and lineup cards, brief- ing cards, text references, and lesson plans were obtained from flight line instructors. Results and Findings for Knowledge Influences The assumed declarative conceptual, metacognitive, and procedural knowledge elements presented in Chapter three were evaluated during this study by examining survey, in-person in- terviews, and observation findings. Table 7 shows the assumed knowledge and skill influences with the assessment methods used to determine validity. Assumed Knowledge Influence 1: Aviation training faculty need to understand the rela- tionship between student metacognition, self-efficacy, and student success and understand their role in improving learning outcomes (Declarative Conceptual). This influence gap was validated. Data indicated that aviation training faculty need to, but do not, understand the relationship between self-efficacy and metacognition, nor their role in applying these behavioral strategies to improve student learning outcomes. Survey Results. To help assess this assumed influence, three Likert-scale survey ques- tions were created. The first question asked faculty how much influence they had over curricu- lum decisions. Approximately 63% of respondents felt they had moderate to no influence over decisions related to curriculum in their programs, indicating a lack of agency amongst faculty that may affect their understanding of their role in improving learner outcomes (Figure 18). IMPROVING PILOT TRAINING OUTCOMES 99 Figure 18. Survey Question 2 – “How much can you influence decisions related to curriculum in the Professional Flight Program?” Source: Author The second question asked participants to select their preferred knowledge assessment method from a list of options. Two-thirds selected written or verbal examination options over metacognitive techniques such as pre-assessments, journaling, and demonstration (Figure 19). Figure 19. Survey Question 3 – “If you wanted to asses a student’s knowledge of a particular topic, which of the following tools would you most prefer to use?” Source: Author The third question asked participants to select from a list what they believed was the most important element in increasing student retention. Only 18% of survey participants believed that metacognitive practices such as journaling and reflecting on previous experience was the best option, with 56% indicating a preference for adjusting to individual learning styles (Figure 20). IMPROVING PILOT TRAINING OUTCOMES 100 Figure 20. Survey Question 4 – “Which of the following do you believe is most important to in- crease a student’s retention of what they have been taught in previous lessons?” Source: Author Interview Findings. Interview participants were asked to describe the learning skills they felt were important for students to improve their performance. Responses from the 11 in- terview participants (IPs) fell into four general categories; basic study skills, mechanics of learn- ing, understanding learner responsibilities, and attitudes and behaviors. When asked, “Are you providing, or describing what these study skills are to these students?” one participant responded: No…There's nothing that says, “Hey, how are you going to study this?” I usually have what works for me, like, “Hey, make flashcards or stuff like that.” But there's nothing that says we have to do that. They’re kind of left to their own devices. (IP 2) This response reflects a compliance orientation, or, “Compliance and adhering to stand- ards and expectations” (IP 10), focused on doing whatever minimum effort is required. Compli- ance perspectives contrast with a student-centric orientation, designed around optimizing strate- gies for helping students learn in the most efficient and effective way. Several self-efficacy strategies were mentioned. IP 12 suggested a strategy for under- standing correlation, or, “How does it all apply?”. Self-initiated problem solving, or becoming a “self-directed problem solver” was recommended by IP 6. IP 2 believed in the importance of IMPROVING PILOT TRAINING OUTCOMES 101 building effective support networks, referencing a, “relationship between being well liked and being successful”. Metacognitive strategies such as time management were described by IP 7, who stated students should be willing, “to put in a little bit extra to get done” during breaks in training, since, “they don't realize I think how much time it's going to take to catch back up to where they were in the first place.” IP 8 believed note taking was an important learning skill for young pilots, stating: I think one of the big parts is, they may read the chapter but then they don't take notes. They don't highlight things. They just read it and call it. You know, they might skim the chapter so to speak, and call it good. (IP 8) Rehearsal strategies were discussed by IP 6, especially in the instrument phase. He re- layed the following example: I just had a student that looked up forward slips. Instead of just being told what forward slips is, he looked at the video, got in the plane, and it's like he'd been taught that already. He performed as the video was. (IP 6) One faculty member believed that fundamental studying skills were the most important learning skills students need for success in flying, stating: Study skills. One hundred percent. That's really the number one answer. So many people come so grossly ill-prepared. They're either smart enough that they can just skate through high school. There's really just no motivation but to really read the material and become familiar with it. (IP 2) While participants appeared to understand the relationship between self-efficacy and metacognition, and successful learner outcomes, they varied in their understanding of faculty’s role in applying them. When asked for what he felt were important learning skills, IP 3 respond- IMPROVING PILOT TRAINING OUTCOMES 102 ed that it “depends on the student.” He continued, stating, “I think they just need to study the textbooks more than they generally do, just judging by the way they do on the tests.” Assumed Knowledge Influence 2: Aviation training faculty must understand performance goals related to providing self-efficacy and metacognition instruction to their students (De- clarative Conceptual). This influence gap was validated. Interview data indicated that the aviation training pro- gram should have defined organizational performance goals that are clearly communicated to their faculty. Faculty were not aware of any stated organizational performance goals, specifical- ly those related to providing self-efficacy and metacognition instruction to their students. Interview Findings. Interview participants were asked to describe State University’s aviation training program’s goals. Only one interviewee was able to clearly articulate a stated organizational goal of any kind. All other respondents provided assumed program or context- specific personal objectives that were not consistent between respondents. IP 6 and IP 9 were not aware of any goals. Some respondents identified competency objectives for students, includ- ing taking “anyone who wants to be a pilot, and make[ing] them a pilot, and with the certificates that they promise on the website” (IP 4). Other respondents stated, “my goal is to make sure the student has the knowledge and skill demonstration in order to pass the check ride” (IP 7). IP 5 believed the organizational goal was to make “a professional qualified pilot, while at the same time keeping them at a relatively quick and efficient pace.” Sharing a focus on program out- comes, IP 8 stated, “We're gonna produce high [sic] qualified graduates that can assimilate into the military or general aviation, or more specifically assimilate into airline aviation. I'm pretty much airline-focused, but our graduates go everywhere.” When asked whether he was aware of State University’s program goals, IP 11 responded, “A little bit.” IMPROVING PILOT TRAINING OUTCOMES 103 The inconsistencies between responses, both from the academic and flight line sides of the flying training program, combined with a lack of understanding of program-level objectives in student performance or behavioral instruction, exposed an opportunity for State University to establish objectives as a method of improving performance. Assumed Knowledge Influence 3: Aviation training faculty need to know how to apply metacognitive and self-efficacy practices in their own instruction and reflect on how they are performing relative to their goals (Metacognitive). This influence gap was validated. Interview and observation data indicated that academic and flight line faculty lacked a clear, consistent understanding of how to apply self-efficacy and metacognitive strategies in their instruction. Faculty appeared to reflect on assumed personal goals and not on organizational goals, which were less-well defined. Interview Findings. Interview participants were asked to describe their experience with teaching students “how to learn.” Several participants had over 20 years in adult education (IPs 3, 8, and 9) while others had less than two years of experience. Younger instructors, particularly those on the flight line, relied on the FAA Instructor Handbook, mentorship from more senior instructors, and “life experiences in general” (IP 4) for their understanding of self-efficacy and metacognitive instructional strategies. Many of these CFIs reported feeling, “like people don't have study skills that are conducive to them being successful” (IP 4), and that “it all depends on the student” (IP 11), or “every student is different” (IP 2). Standardized faculty practices or pro- cesses for helping students acquire self-regulation or metacognitive skills did not appear to be available or used. Faculty participants were also asked what strategies they would use to increase a stu- dent’s memory of what they had been taught in previous lessons. Some faculty members provid- IMPROVING PILOT TRAINING OUTCOMES 104 ed clear examples of how they applied self-efficacy and metacognitive instruction in their curric- ulum. IP 9 used a correlational study of aircraft accidents and pilot behaviors to illustrate situa- tions that young pilots will likely face in their flying careers. IP 7 stated, “The strategies [sic] that I have for the retention of learning, is quizzing… oral or writing down,” which was a com- mon response. IP 8 used hands-on training opportunities and field trips, while IP 10 believed, “one of the best ways to learn something is to teach it. I think by flight instructing, they're rein- forcing everything they've learned.” IP 4 provided a list of options, including repetition, spot checks, oral exams, scenario-based verbal checks, lecture, video, and pictures, while IP 3 com- mented, “Well, other than testing, I don't really know.” Many of these behavioral learning strat- egies can be effective in improving learner outcomes, but they do not appear to be consistently applied by all instructors as a method of achieving organizational goals. Observation Findings. Academic classroom observations revealed that instructors were generally unaware of and did not extensively use specific self-efficacy or metacognitive practices during their instruction to improve performance. While instructors occasionally discussed learn- ing goals for the day or class, mastery-orientation learning strategies were not discussed in any observed classes. “Foot stomping,” a term occasionally used in military flight training to high- light important subjects, was applied in one class session, though recommended methodologies for remembering, understanding, or applying the content were not provided. Academic learning objectives tended to be data-centric, relying almost purely on rote memorization. Students were not required to take notes. No discussions were observed between instructors and students out- lining potential learning strategies for this level of learning, and none were found in the syllabus reviews. Faculty were generally clear about the learning expectations, commenting “That is all I want you to know for this,” and asking students questions to validate understanding while con- IMPROVING PILOT TRAINING OUTCOMES 105 necting lecture items to upcoming examinations. All observed faculty displayed a genuine en- thusiasm and passion for their subjects and an honest interest in student success. Simulator observations occurred during training “missions,” where students were given a plan for taxi, take off, en route navigation, approach, landing, and taxi. At the start of class, the instructor provided mission details and several learning goals, including a review of previous les- sons learned for this phase of training. The simulator instructor acted as a humble, approachable, and credible Instructor Pilot, creating a low stress learning environment while allowing the stu- dent crew members to brief, make critical mission decisions, and run the flight profile as they wished. Following individual flight phases, he asked, “Any questions or comments on that one?” The instructor created his own syllabus, lesson plan, and mission guidelines, as the school’s program guidance was limited. Suggestions to students for improving in-flight perfor- mance were discussed extensively throughout the training session. These simulator events showed very effective use of self-efficacy and metacognitive strategies, despite a lack of guid- ance from syllabus or lesson plans. During observations of ground training sessions on the flight line, instructor pilots were seen using many different self-efficacy and metacognitive strategies to improve student perfor- mance in flight, though sporadically and inconsistently. Lesson plans or instructor guidelines correlating learning strategies to knowledge-level expectations were not observed. All instructor pilots showed a genuine enthusiasm and passion for their subjects and an honest interest in stu- dent success, as long as students were perceived as making a reciprocal effort to learn. While several flight line faculty were observed providing positive reinforcement to students, such as “You’re only on Ride 3, so you’re doing fine,” and, “You’re doing good,” Kluger and Denisi (1996) argue that generalized, non-specific praise can actually have a debilitating effect on per- IMPROVING PILOT TRAINING OUTCOMES 106 formance. They go on to state that, “in specific situations, feedback interventions can yield a large and positive effect on performance” (p 278). In most cases, instructors provided clear les- son objectives for the mission during the observed briefings. Observations of academic, simulator, and flight line instructional periods indicated that most faculty were aware of the importance of student self-efficacy and how metacognitive strat- egies can support learning, but few were aware of how to build and apply these practices consist- ently in their instruction. Lacking clear performance goals, faculty were unable to reflect on their instructional practice or student performance relative to any objective other than those that were immediate or self-determined. Assumed Knowledge Influence 4: Aviation training faculty must know how to structure learning experiences to increase students’ self-efficacy and metacognition abilities in their programs, how to give effective feedback, how to communicate clear performance goals, and how to evaluate their teaching relative to student performance (Procedural). This influence gap was validated. Interview and observation data indicated that State University faculty did not have a clear understanding of how to incorporate self-efficacy and metacognitive practices into their learning structures, how to communicate goals, or how to evaluate their own teaching effectiveness. Interview Findings. This assumed influence was assessed by two semi-structured, open- ended interview questions. The terms metacognition and self-efficacy were first defined for par- ticipants and used in context. The first question asked participants what they felt was their most effective technique for helping students when they struggle to learn a difficult subject. Empathy was the most common theme amongst responses. IP 10 stated, “I think it's very important that we connect with our students so that we bring examples of our own struggles that we've had. I IMPROVING PILOT TRAINING OUTCOMES 107 think misery enjoys company.” A shared sense of “the fact that I have gone through all of this that they're going through,” (IP 1) drove other responses, such as, “I find that really helps them with their confidence going into their check ride. I think that's half the battle,” (IP 2). In terms of describing their techniques for structuring learning experiences and com- municating performance goals through feedback, responses were less clear. Learning strategies such as repetition (IP 4), visualization (IP 11), and varying the instructors during training (IPs 5 and 7) showed an understanding of the value of structures that support learning, but they were inconsistently described and understood across interview participants. One instructor noted, “I do think some instructors come in, punch in, punch out, go home.” Flight instructors appeared to understand the value of post-flight debriefings as a tool for enhancing learning, but many felt challenged to provide adequate attention to students due to tight schedules. IP 7 mentioned, “The most effective way to get them to correct, in my opinion, is having a thorough debrief. It's the amount of time you spend with your student. If you get back from the plane and you talk to them for 30 seconds, that's not enough.” While faculty stated that debriefings are an important part of learning, the quality of those events may not be as effective as they either think or wish. The second interview question asked participants how their education and experience prepared them to provide metacognition and self-efficacy instruction to their students, and if they could provide an example. Participants shared personal stories from their own education and training experiences, and how that informed their current instructional practices. Findings re- vealed that not all instructors applied behavioral learning strategies in their instructional design. In fact, some didn’t believe in their effectiveness in improving learning. For example, seven par- ticipants stated they relied on previous experiences as their primary source for how to succeed as a teacher. IP 6 stated, “I think my experience is probably what helps me to teach.” IP 11 com- IMPROVING PILOT TRAINING OUTCOMES 108 mented, “I think through my experience... having a high workload at one point in my life.” Some referenced previous military or airline experience, stating, “My experience in the Air Force, training hundreds of young people on some pretty complex tasks. I think I'm pretty good about getting feedback from them and explaining it” (IP 3). Very few interviewees stated they had received any training on strategies for improving student learning, and none referenced any organizational or program goals in their responses. When asked if he had specific training in self-reflective behaviors, IP 6 stated, “No.” One re- spondent commented, “We walked in with our instructor (pilot) certificate. I [took] the written test that allowed me to be a ground instructor and, basically, they gave me a class and said, ‘Teach.’” While five participants described specific metacognitive strategies they used with their students, none credited the organization for providing education and training in them, or whether policy and procedure required them. Only one respondent (IP 1) said he had attended a specific course on learning behaviors, which he then used to build his own lesson on self- efficacy and metacognitive strategies for students. Observation Findings. Academic classroom observations revealed that while some in- structors were able to effectively structure learning environments to increase student self- confidence and self-regulation, others were not. One instructor applied several behavioral learn- ing strategies, with comments such as, “I’m preparing you for when I’m not in there (in the air- craft) with you,” “Come on, you can do it,” and “Our focus today is on doing the right steps in the traffic pattern.” Other instructors started class without stating any learning objectives for the day, and with minimal engagement from the class. Minimal feedback was requested or received from students, and learning strategies were not discussed. Other than formal post-semester in- structor reviews, in most cases no feedback was provided to faculty members from students. IMPROVING PILOT TRAINING OUTCOMES 109 As IP 7 noted previously, post-flight debriefings are a time-honored, almost ritualistically applied learning tool in aviation training. In military contexts, flight debriefings are often exten- sive, lasting over an hour with detailed frameworks and specific learning objectives. U.S. Air Force flight briefings and debriefings are conducted in closed door sessions, in an open, some- times brutally honest learning environment. Civilian flying training programs, as represented by State University’s faculty and their prior experiences, differ in several ways. Preflight briefings were observed as extremely short, in one case less than 5 minutes. They were conducted casual- ly, light on details or specifics, and with an assumed understanding that critical flight planning tasks, including learning objectives and goals were accomplished and understood by instructor and student. They were also unstructured, typically only covering a few main points the instruc- tor chose to highlight. While many pre-and post-flight briefing tasks seem mundane or administrative, they form the cognitive foundation that all subsequent learning will be built on during the flight. These briefings provide opportunity for motivation and inspiration (self-efficacy), and align thinking for the day’s mission prior to stepping to the aircraft (metacognition). They are the first and final opportunities an instructor has to establish and reinforce the learning structures a student will re- ly on in the dynamic, loud, chaotic, and sometimes turbulent flight environment. Given the compressed, rapid pace of training observed and reported by the flight instructors, these briefings were often compressed and general in focus. They were not consistently observed as essential elements of a learning structure, or a dedicated environment for feedback on performance goals. Results and Findings for Motivation Influences The assumed goal-orientation and self-efficacy motivation elements presented in Chapter three were evaluated during this study by an online survey, in-person interviews, observations, IMPROVING PILOT TRAINING OUTCOMES 110 and document and artifact reviews. Table 8 shows the assumed motivation influences with the assessment methods used to determine validity. Assumed Motivation Influence 1: Aviation training faculty should desire to find new and innovative ways to help their students achieve a mastery orientation of the rules of aviation (Goal-Orientation). This influence gap was not validated. Survey and observation data indicated that aviation training faculty already desire to find new and innovative ways to help students. Survey Results. To help assess this assumed influence, one Likert-scale and one sliding scale question was created for the survey. This question asked participants if they believed in- structors should make a priority of finding new and innovative ways to help students. Of the 42 responses, 79% (33 of 42) Agreed to Strongly agreed (Figure 21). Figure 21. Survey Question 5 – “I believe instructors should make a priority of finding new and innovative ways to help students develop a mastery focus (a focus on constant learning and im- provement) of the rules of aviation.” Source: Author For the second question, participants were asked to use a sliding scale to indicate where they thought a teacher’s priority should be between applying what they already know how to do something and learning and applying innovative approaches. The average (mean) for all re- IMPROVING PILOT TRAINING OUTCOMES 111 sponses was 65.97 and the mode was 70, indicating a preference for “Learn and Apply Innova- tive Approaches” (Figure 22). Apply What I Already Know How to Do Learn and Apply Innovative Approaches 0 10 20 30 40 50 60 70 80 90 100 Slide the bar to reflect your belief (1) Field N Min Max Mean (µ) Std Dev (∂) Var Slide the bar to reflect your belief 40 27 94 65.97 17.37 301.87 Figure 22. Survey Question 6 – “Please indicate where you think a teacher’s priority should be on this continuum by sliding the bar to the value that corresponds with your beliefs.” Source: Author Interview Findings. Interview participants were asked what motivates them to find new or innovative ways to help students achieve a mastery orientation of aviation skills. While some respondents described internal motivators such as a sense of “self-pride” or “self-improvement” (IP 1), wanting to be a “top notch human being” (IP 4), or preserving their own sense of success (“I have a four and zero pass rate right now. Everyone's passed,” IP 4), nearly everyone agreed that they wanted to do their best to help students master the art and science of aviation. One re- spondent summarized this sentiment by noting: IMPROVING PILOT TRAINING OUTCOMES 112 I like teaching, I enjoy it. That's a good enough drive, and seeing a kid on the other side, holding a certificate just makes me want to get the next guy through, you know, learn to teach him, especially when you come against a kid who just doesn't learn that way that you've gotten so much success with, and you want to see him get through. You just start talking to other instructors trying to figure out a different way around. (IP 5) IP 6 noted, “a lot of the motivation comes from the students themselves.” Others com- mented that they enjoyed teaching as a profession, with statements like, “I'm into this stuff, for starters. I find it inherently interesting. Every lecture, every year” (IP 9). Participants noted, “I mean, I want them to succeed” (IP 2) and, “We are successful or unsuccessful together, as a team, when I'm working with a student. If they fail their check ride, I have failed” (IP 4). A leg- acy perspective, where respondents wanted to make things better than they had it was also de- scribed. IP 11 noted, “What motivates me is just thinking to myself, ‘Okay, how would I have wanted it? What kind of instructor would I have wanted?’” All IPs interviewed demonstrated interest and investment in their student’s success, expressing a willingness to experiment and reach out to others as methods for improve learning. Assumed Motivation Influence 2: Aviation training faculty need to be confident that self- efficacy and metacognition instruction will improve student outcomes, and in their ability to provide this instruction as a way of improving student outcomes and performance (Self- Efficacy). This influence gap was not validated. Findings indicated that aviation training faculty surveyed and interviewed currently have confidence that self-efficacy and metacognitive educa- IMPROVING PILOT TRAINING OUTCOMES 113 tion and training will improve student performance, and generally believe they are capable of delivering it. Survey Results. To assess this assumed influence three survey questions were created, including two Likert-scale and one sliding scale. For the first question, participants were asked if they felt their students would benefit from learning metacognitive strategies such as self- perception, motivation, and learning how to influence events affecting their lives. Seventy-one percent (30 of 42) of respondents Agreed to Strongly agreed (Figure 23). Figure 23. Survey Question 7 – “My students would benefit from learning how to enhance their self-perception of their capabilities, motivation, and ability to influence events affecting their lives.” Source: Author For the second question, participants were asked to agree or disagree with the statement that teaching self-efficacy strategies designed to increase student self-confidence will improve performance in flight. Seventy-eight percent (32 of 42) of respondents Strongly agreed to Agreed (Figure 24). IMPROVING PILOT TRAINING OUTCOMES 114 Figure 24. Survey Question 9 – “Teaching strategies designed to increase student confidence in their ability to learn aviation-specific concepts will increase their performance in flight.” Source: Author The third survey question used a sliding-scale response to examine faculty confidence in accomplishing six discrete tasks related to improving student self-efficacy and metacognition. The averages (mean) of each subtask were 74 or higher, indicating an average rating between “Moderately Can Do” and “Highly Certain Can Do” (Figure 25). Cannot Do At All Moderately Can Do Highly Certain Can Do 0 10 20 30 40 50 60 70 80 90 100 Keep students on task during difficult as- signments Increase a students' understanding of what they have been taught in previous lessons Get students to believe they can do well in school work Positively influence the performance of stu- dents who are struggling Help students master difficult tasks or sub- jects Motivate students who are struggling IMPROVING PILOT TRAINING OUTCOMES 115 Field N Min Max Mean (µ) Std Dev (∂) Var Keep students on task during difficult as- signments 41 30 100 74.78 15.64 244.66 Increase a students' understanding of what they have been taught in previous lessons 41 50 100 82.24 11.45 131.01 Get students to believe they can do well in school work 41 47 100 78.34 15.63 244.22 Positively influence the performance of students who are struggling 41 40 100 77.15 13.89 193.05 Help students master difficult tasks or sub- jects 41 60 100 80.88 10.00 100.06 Motivate students who are struggling. 41 37 100 74.44 18.09 327.17 Figure 25. Survey Question 10 – “Using the scale bar below, please rate your confidence in ac- complishing the following tasks.” Source: Author Interview Findings. Interview participants were asked two questions to explore faculty confidence that self-efficacy and metacognitive instruction will improve learner outcomes, and in their ability to deliver it. The first question asked participants to provide an example of how the design of their course could be improved to help teach students how to learn content more effec- tively. Responses fell into three general categories; course structure, behavioral learning strate- gies, and everything was working well (no change needed). Course structure suggestions included changes to syllabus or structure of the learning en- vironment. For example, IP 1 noted that better simulators (FAA approved) would improve train- ing capability and capacity. He commented, “We'd be able to actually log flight time on them with a flight instructor. I mean, simulated flight time. But it would just offer a better experience for the students that are going through that are learning instruments.” IP 5 felt that more time for each phase of flight training would improve learning, though it would negatively affect the speed at which students received their ratings and graduated. When asked if he felt the program was too fast now, he commented, “To make a safe pilot? No. To make a great pilot? Yes.” IMPROVING PILOT TRAINING OUTCOMES 116 Behavioral learning strategies were also suggested by respondents. IP 4 mentioned that teaching others was an effective learning strategy for students, stating, “I feel like [it] really helps people if you can teach something.” IP 6 stated that an inability to perform basic math skills without the use of computers and or knowing fundamental pilotage skills (navigation and basic flight planning, for example) were significant challenges for students. Other suggestions to improve learning effectiveness included increased use of case studies (IP 7), video training (IP 8), game-based scenario training (IP 10), and increased social interaction (IP 11). Respondents who felt things were working well provided comments such as, “Well, I think I'm doing a pretty good job as it is. I don't know what else I could possibly use” (IP 3), “With the type of classroom setups we've got right now, I don't know of other ways to improve things much more” (IP6), and “I’m pretty happy with where we are” (IP 9). The second interview question asked participants how important they felt instruction on self-efficacy and metacognition was to improving graduation rates. With only three exceptions, State University’s aviation training faculty believed that this type of behavioral instruction would be an important element in improving learning outcomes. IP 1 stated, “I think it's extremely im- portant in this program,” while IP 8 commented this type of training was “key to being a success- ful graduate.” IP 9 stated, “That's part of what you should be doing in a four-year institution. This isn't just a flight school.” Highlighting his belief in the importance of self-efficacy and metacognitive instruction in the flying training curriculum, IP 11 said, “Oh, it's huge. It's abso- lutely huge. I mean the thing is, it's not just instruction at that point, it's a friendship. It's a bond that you have. It's more, ‘Hey what can I do to be there for you?’” When asked if he felt this type of training would be beneficial to his fellow instructors and students, IP 7 replied in the af- firmative, commenting, “I don’t see how it could be detrimental.” IMPROVING PILOT TRAINING OUTCOMES 117 Only three interview participants expressed uncertainty or disagreement. One comment- ed: I think it would help a ton but that's not really what I'm paid to do. That's some- thing that there's an expectation that they already have. They're already supposed to have developed that skill. Ninety percent of the time that's not true, right? (IP 2) IP 7 did not appear convinced of the value of behavioral training for faculty, stating, “A lot of people would probably take some of this stuff to heart, and that's gonna vary person to per- son.” IP 4 was the most skeptical, stating, “I do this with every student, what you're talking about right now. I tell them, ‘Do some introspection. Ask yourself, did I learn today?’ I don't see a drastic improvement in performance.” The findings from both interview questions appear to refute this assumed motivation in- fluence. When asked how the instructional design of their courses could be improved to help students learn more effectively, faculty provided several options directly related to enhancing student self-efficacy and metacognition. Asked how important they felt this type of instruction would be, the majority of faculty (8 of 11) responded enthusiastically in the affirmative. Assumed Motivation Influence 3: Faculty need to attribute student success or failure in part to their skill as an instructor (Self-Efficacy). This influence gap could not be validated. While survey responses indicated fairly con- sistent results showing agreement with the assumed influence, interview responses were mixed and varied, with only three of eleven respondents mentioning instructor skill specifically as at- tributable to student success or failure. Survey Results. To assess this assumed motivation influence one Likert-scale question was developed, asking respondents to what extent they felt an instructor was responsible for a IMPROVING PILOT TRAINING OUTCOMES 118 student’s performance. Of the 41 responses, 97.56% (40 of 41) of respondents felt that the in- structor was responsible Sometimes or Most of the time (Figure 26). Figure 26. Survey Question 11 – “To what extent is the instructor responsible for a student’s strong or weak performance?” Source: Author Interview Findings. Interview participants were asked what factors they considered to explain a student’s poor performance. Findings were mixed, with poor attendance and study skills as the most reported responses. IP 7 asked, “Do they have the study skills? That’s big.” The high cost of the program was also listed as a significant potential cause by three respondents (IP 1, 8, and 10). IP 1 stated, “I do know that money is probably the biggest factor in students leaving.” He also mentioned parental influence in deciding to attempt this program, stating, “a lot of students whose parents are in the airlines, are the ones that are struggling.” IP 4 implicated a “lack of desire to learn.” IP 10 shared that “sometimes students are overwhelmed,” and IP 5 believed that age was a potential factor, specifically “like guys that are over 40.” Only three par- ticipants (IPs 2, 6, and 11) mentioned instructors specifically as potential factors. One IP noted: “Part of it's the student, but you also have to question yourself. Am I getting the information to the student? Are they receiving what I'm transmitting? Am I using the wrong terms. And so, I try to sit down with them individually, given I've got somebody that has the desire to learn.” (IP 6) IMPROVING PILOT TRAINING OUTCOMES 119 However, the majority of respondents did not mention instructors as causal to student problems in the program, so the assumed influence cannot be fully verified based on the inter- view findings alone. Results and Findings for Organization Influences The assumed cultural model and cultural setting organization elements presented in Chapter three were evaluated during this study by an online survey, in-person interviews, obser- vations, and documents and artifact reviews. Table 9 shows the assumed organization influences with the assessment methods used to determine validity. Assumed Organization Influence 1: The organization values experimentation and having faculty understand cognitive learning processes by providing learning opportunities and requiring their participation in these activities (Cultural Model 1). This influence gap could not be validated. A wide variance in survey responses and a lack of evidence supporting organizational support for cognitive learning processes meant more data was required to confirm or refute this assumption. Survey Results. To assess this assumed organization influence, one Likert-scale ques- tion was created. Participants were asked if they believed their academic program provided ade- quate education and training to faculty on behavioral learning practices such self-efficacy and metacognition. As shown in Figure 27, responses to this question were varied across the answer spectrum, showing an almost even split between agree options (19 of 39, or 49%) and disagree options (20 of 39, or 51%). IMPROVING PILOT TRAINING OUTCOMES 120 Figure 27. Survey Question 12 – “I believe my academic program provides adequate education and training to faculty on practices related to teaching student reflection and self-motivation.” Source: Author Document Findings. Document analysis of faculty training programs was unable to re- veal data that would confirm or refute this organizational influence assumption. No aviation training behavioral learning programs were mandated by either academic or flight line leader- ship, nor did any faculty member express that one existed. Professional development programs for faculty emphasizing experimentation in instructional design or training and education on learning strategies, beyond FAA-mandated initial training, were also not found. Assumed Organization Influence 2: The organization provides the resources necessary for aviation training faculty to adopt student self-efficacy and metacognition programs in their pilot training classrooms (Cultural Model 2) This influence gap was validated. Survey, interview, and document analysis appeared to indicate that the organization does not provide dedicated resources in the form of training, educa- tion, or instruction that State University faculty would need to adapt and apply student self- efficacy and metacognition programs in their curriculum. Survey Results. To assess this assumed motivation influence, one Likert-scale question was developed. Participants were asked to indicate to what extent their program provided the IMPROVING PILOT TRAINING OUTCOMES 121 resources needed to improve student understanding of self-efficacy and metacognition. Re- sponses showed that 63% of respondents (24 of 38) believed their programs provided either none or only some of faculty’s resource needs (Figure 28). Figure 28. Survey Question 14 – “To what extent does your collegiate flight program provide the resources you need to improve a student’s understanding of reflection and self-motivation?” Source: Author Interview Findings. This assumed influence was assessed by one semi-structured, open- ended question, asking participants how they thought the university could better support faculty in helping students achieve subject mastery in aviation. Many instructors were happy with the current level of perceived support from the university, stating, “I think we do an awesome job on this side and I think they do an awesome job on that side,” (IP 2) and, “The university itself I think is doing a good job,” (IP 6). Others felt that students needed more time with faculty to de- velop and grow as aviators, as noted in comments such as, “There's just not enough time in this fast-paced program to do all the instruction yourself. I think these students need more time with, basically a teacher, a focused teacher,” (IP 5), and, “just did not have the time to … make the changes that he wanted to make in individual classes” (IP 6). One participant expressed a desire for State University to hold faculty more accountable for student performance, noting: If there was accountability in the fact I would lose my job if I didn't pass three out of four people every check ride, and then was held to it, I was going to lose my IMPROVING PILOT TRAINING OUTCOMES 122 job. Or a teacher, if they were going to lose their job if 90% of the kids didn't get an A, I'll bet you they would stay later. I'll bet you they would teach better. I'll bet you they would dance in front of that class if they had to, in order to get those students to learn. They would do whatever it takes, but we don't hold people to those standards. (IP 4) Clarifying expectations between the organization and faculty was expressed as an oppor- tunity for the university to provide better support. IP 11 stated, “I think a better understanding of the expectations” was needed to ensure continuity of effort between the academic and flight line training elements. IP 9 responded, “What is the university's expectation? What constitutes mas- tery? I have my view and I think I do a pretty good decent job of meeting my criteria for mas- tery.” This view highlights a gap in organizational expectations regarding goals and objectives, while revealing a lack of consistent feedback to faculty. Document Findings. Both the academic program lead and the contractor flight training site lead indicated that there were no ongoing faculty professional development programs specif- ically targeted at improving instructional design, curriculum development, or teaching strategies. Mandatory recurrent training was provided by State University on items such as diversity, infor- mation assurance, and sexual harassment. Current and qualified instructor pilots are also re- quired to take annual check rides to ensure proficiency and competence. However, professional development sessions designed to improve faculty understanding and application of behavioral strategies were not found. Perhaps the most significant document gathered during this data collection phase was the FAA Aviation Instructor’s Handbook, or FAA-H-8083-9A. Often referred to as “8083,” this 228-page instructor pilot training book was last revised in 2008, and is mandatory reading for all IMPROVING PILOT TRAINING OUTCOMES 123 aspiring civilian instructor pilots. It is the source document for most of their duties and training standards, and was cited frequently by all CFIs interviewed in this study as the primary reference for becoming a good flight instructor. It discusses many of the concepts described in Chapter two of this study in varying levels of detail, including learning theory and structures, the Infor- mation Processing Model (IPM, Figure 3), and knowledge acquisition principles. While Appendix A provides a reference list of studies used to inform the chapters on adult learning and teaching processes, 8083 contains several assumptions and assertions that lack a solid foundation in academic research and literature. Behaviorist E.L. Thorndike’s three Laws of Learning are noted as “universally accepted” (p. 2-10), though this statement cannot be vali- dated and his views on educational “law” are far from universally accepted in contemporary ped- agogy or andragogy. The Handbook’s statement that, “According to research on the human brain, people have a preferred side of the brain to use for understanding and storing infor- mation,” and “it is a preference, not an absolute” (p 2-18) is tied to a concept known as hemi- sphericity, which has been repeatedly disputed by neuroscientists and brain researchers through extensive study (Nielsen, Zielinski, Ferguson, Lainhart, & Anderson, 2013; OECD, 2002; Pinto, de Haan, & Lamme, 2017). The discussion on learning styles, beginning on page 2-17, noted that, “Learning styles are simply different approaches or ways of learning based on the fact that people absorb and process information in different ways,” and “everyone has an individual style of learning.” A growing volume of research does not support these learning style assumptions, yet they continue to be taught as foundational elements of instructional design in flight training (Deans for Impact, 2015; Düvel, Wolf, & Kopiez, 2017; Gilmore, McCarthy, & Spelke, 2007; OECD, 2002; Pashler, McDaniel, Rohrer, & Bjork, 2008). IMPROVING PILOT TRAINING OUTCOMES 124 While 8083 provides well supported and relevant information on many adult learning traits and characteristics in an aviation-specific context, these unsubstantiated statements ex- posed limitations on this dated handbook’s reliability as an accurate guide on adult learning pro- cesses. Further, although it discusses some essential elements related to the understanding and use of self-efficacy and metacognitive practices in improving student pilot performance, im- portant gaps exist in explanation and application. The terms self-efficacy and metacognition are not mentioned or defined anywhere in the handbook. Assumed Organization Influence 3: The organization recognizes and rewards innovation and continuous improvement by providing an evaluation framework that values innovative practices for teachers to measure their performance against the goal (Cultural Setting 1). This influence gap was not validated. Survey data analysis seemed to indicate that re- spondents generally agreed that their organizations are providing an evaluation framework that values innovation and continuous improvement to measure against performance goals. Survey Results. To assess this assumed organization influence, one Likert-scale ques- tion was developed, asking participants if they believed their program rewards innovation and continuous improvement in syllabus development. Sixty-four percent (25 out of 39) of respond- ents agreed at some level that their programs rewarded innovation and continuous improvement in syllabus development. However, nearly 36% of respondents chose “Somewhat agree” (the highest selected response), and nearly 36% disagreed to some level that their programs adequate- ly reward innovation and continuous improvement. IMPROVING PILOT TRAINING OUTCOMES 125 Figure 29. Survey Question 13 – “I believe my academic program rewards innovation and con- tinuous improvement in syllabus development.” Source: Author Assumed Organization Influence 4: The organization provides an organizational culture that is accepting of change and able to respond to it effectively (Cultural Setting 2). This influence gap was not validated. Survey data and interview findings appeared to show that State University’s culture is accepting of new ideas in curriculum design and instruc- tion. On average, faculty appear satisfied with how their program is running. Survey Results. To assess this assumed motivation influence, one Likert-scale survey question was created asking participants to what extent they believed their programs were open to new ideas about instruction. Responses showed that 84% (32 of 38) of respondents believe their flight training program is open to new ideas about instruction either Some or Substantially. Figure 30. Survey Question 15 – “To what extent do you believe your collegiate flight program is open to new ideas about instruction?” Source: Author IMPROVING PILOT TRAINING OUTCOMES 126 Interview Findings. Interview participants were asked for their recommendation on how State University can improve the current instructional design of the aviation training program to improve student learning. Overall, responses tended towards general satisfaction with the cur- rent program. Comment such as, “I think both sides do an awesome job is the thing. I really think it's incredible” (IP 2), “I'm pretty happy with the program and compared to what I've seen in other places. I think we're a competitive program” (IP 6), and “I think it's set out fine” (IP11) were common and appeared heartfelt. IP 2 stated, “So I really don't see very much improvement as how to get more students through.” Recommendations for improvement included faculty training and professional development (IP 1, 6, and 9), improved communication between aca- demic and flight line facilities (IP 7), and improved advertising and recruiting (IP 8). IP 6 went on to say, “I have the freedom to pretty much teach what I want to teach, I like that,” reflecting a cultural sense of accepting change in curriculum development. Table 13 Validated Influencers Table Knowledge Influences Declarative Conceptual Knowledge Aviation training faculty need to understand the rela- tionship between student metacognition, self-efficacy, and student success and understand their role in im- proving learning outcomes. Validated Aviation training faculty must understand perfor- mance goals related to providing self-efficacy and metacognition instruction to their students. Validated Metacognitive Knowledge Aviation training faculty need to know how to apply metacognitive and self-efficacy practices in their own instruction and reflect on how they are performing rel- ative to their goals. Validated Procedural Knowledge Aviation training faculty must know how to structure learning experiences to increase students’ self-efficacy and metacognition abilities in their programs, how to give effective feedback, how to communicate clear performance goals, and how to evaluate their teaching relative to student performance. Validated IMPROVING PILOT TRAINING OUTCOMES 127 Motivation Influences Goal-Orientation Aviation training faculty should desire to find new and innovative ways to help their students achieve a mas- tery orientation of the rules of aviation. Not Validated Self-Efficacy Aviation training faculty need to be confident that self-efficacy and metacognition instruction will im- prove student outcomes, and in their ability to provide this instruction as a way of improving student out- comes and performance. Not Validated Faculty need to attribute student success or failure in part to their skill as an instructor. Could Not Be Validated Organization Influences Cultural Model 1 The organization values experimentation and having faculty understand cognitive learning processes by providing learning opportunities and requiring their participation in these activities. Could Not Be Validated Cultural Model 2 The organization provides the resources necessary for aviation training faculty to adopt student self-efficacy and metacognition programs in their pilot training classrooms. Validated Cultural Setting 1 The organization recognizes and rewards innovation and continuous improvement by providing an evalua- tion framework that values innovative practices for teachers to measure their performance against the goal. Not Validated Cultural Setting 2 The organization provides an organizational culture that is accepting of change and able to respond to it effectively. Not Validated Summary This chapter presented the findings of data collected via online survey and on-site inter- view, observation, and document review to assess the knowledge, motivation, and organization influences outlines in Chapter three. Of the 11 assumed influences, five were validated, four were not validated, and two could not be validated based on the data collected. In the next chap- ter, these findings will be used to inform recommendations for program improvement aligned with State University’s stated organizational goal of having all faculty aware of and practicing effective principles of instructional design, specifically those designed to increase student self- IMPROVING PILOT TRAINING OUTCOMES 128 efficacy and metacognition by January 7, 2019. Chapter five will present solutions for the vali- dated influences. IMPROVING PILOT TRAINING OUTCOMES 129 CHAPTER FIVE Recommendations Chapter one of this innovation model study introduced a current problem of practice, which was to examine knowledge, motivation, and organizational resources necessary to help aviation program faculty address a growing shortage of qualified pilots in U.S. commercial avia- tion. Instructional practices, specifically those related to increasing student self-efficacy and metacognitive practices were introduced as a potential method for improving graduation rates and student performance in the air. Chapter two presented a review of relevant literature on the importance and application of behavioral learning strategies in improving student outcomes, and introduced Clark and Estes (2008) Performance Gap Analysis Model as the methodological framework for this study. Chapter three framed the methodology and conceptual framework that guided data collection for this mixed-methods study, and Chapter four provided a detailed analy- sis of the findings and responses from the four collection methods used to inform the recommen- dations. Chapter five presents those recommendations, using the KMO framework, and offers a comprehensive evaluation strategy for program leaders seeking to improve flying training pro- gram outcomes. Recommendations for Practice to Address KMO Influences Knowledge Recommendation The knowledge influences shown in Table 14 are derived from the findings and results from Chapter four of this study. They are presented with references from the literature study in Chapter two, along with context-specific recommendations to address performance gaps. Three primary recommendations emerged from this review. State University must establish and com- municate performance goals aligned with their global organizational goals and mission, and IMPROVING PILOT TRAINING OUTCOMES 130 communicate those goals clearly to all faculty and contractors. Second, relevant job aids, which Clark and Estes (2008) refer to as self-help information for achieving performance goals, should be created and distributed to faculty. Third, organization leaders should develop and deliver Pro- fessional Development (PD) sessions to faculty that provide awareness and understanding of the value of self-efficacy and metacognition in education, and furnish support for enhancing lesson and curriculum design. Table 14 Summary of Knowledge Influences and Recommendations KMO Knowledge Influences and Recommendations Assumed Knowledge Influence Validated? Principle and Citation Context-Specific Recom- mendation Declarative Conceptual Aviation training facul- ty need to understand the relationship be- tween student metacog- nition, self-efficacy, and student success and understand their role in improving learning out- comes. Yes Learning and motiva- tion are enhanced when learners have positive expectancies for success (Pajares, 2006). Provide information on metacognition and self- efficacy, including the im- pact on student performance improvements. Aviation training facul- ty must know the per- formance goals related to providing self- efficacy and metacogni- tion instruction to their students. Yes Rationales that include a discussion of the im- portance and utility value of the work or learning can help learn- ers develop positive values (Eccles, 2006; Pintrich, 2003). Goals motivate and di- rect students (Pintrich, 2003). Establish clear organiza- tional performance goals, aligned with global goals and mission Provide a job aid that clear- ly defines organizational performance goals related to the application of self- efficacy and metacognition instruction. Metacognitive Aviation training facul- ty need to know how to apply metacognitive and self-efficacy prac- Yes Self-regulatory strate- gies, including goal set- ting, enhance learning and performance Provide opportunities for faculty to reflect on their own self-regulatory strate- gies, including the im- IMPROVING PILOT TRAINING OUTCOMES 131 tices in their own in- struction and reflect on how they are perform- ing relative to their goals. (APA, 2015: Dembo & Eaton, 2000; Denler et al., 2009). The use of metacogni- tive strategies facilitates learning (Baker, 2006). portance and use of reflec- tive practices relative to their stated goals. Procedural Aviation training facul- ty must know how to structure learning expe- riences to increase stu- dents’ self-efficacy and metacognition abilities in their programs, how to give effective feed- back, how to communi- cate clear performance goals, and how to eval- uate their teaching rela- tive to student perfor- mance. Yes Increasing germane cognitive load by en- gaging the learner in meaningful learning and schema construction facilitates effective learning (Kirshner et al., 2006). Feedback that is private, specific, and timely enhances performance (Shute, 2008). Provide job aids to faculty on steps for improving cur- riculum and course design, learning schema, and evalu- ation strategies (Kirkpatrick, 2016). Provide faculty training/PD on feedback techniques and application. Declarative Conceptual knowledge solutions, or description of needs or assets. Avia- tion training faculty need to understand the relationship between student metacognition, self- efficacy, and student success and understand their role in improving learning outcomes. Accord- ing to Pajares (2006), learning and motivation are enhanced when learners have positive expec- tancies for success. This suggests that university aviation training programs should provide in- formation on metacognition and self-efficacy to their instructors, including the impact of these behavioral learning strategies on improving student performance. Instructors play a critical role in shaping student expectations for learning and understanding how metacognition and self- efficacy are related to success is an important element in curriculum design. IMPROVING PILOT TRAINING OUTCOMES 132 In a study of 173 undergraduate students, Coutinho (2008) discovered that self-efficacy and metacognition positively influence student performance. This supports the concept that teaching these behavioral strategies and enhancing these beliefs in adult learners are important elements for improving performance. As such, providing information to faculty on the value of these elements and how they form the foundation for student motivation, well-being, academic performance, and accomplishment becomes an essential element in improving overall program outcomes, measured by graduation rates and in-flight performance (Pajares, 2006). Aviation training faculty must know the performance goals related to providing self- efficacy and metacognition instruction to their students. In expectancy value motivational theo- ry, Eccles and her colleagues (2009) described how knowing the importance and utility value of learning topics can help develop positive values. Further, Pintrich (2003) found that goals are a powerful way to motivate and direct students. This suggests that knowing and understanding how performance goals relate to organizational success is critical to improving learning out- comes. Providing faculty a job aid clearly defining the organizational performance goals related to the application of self-efficacy and metacognition instruction without requiring guided prac- tice can be an effective way to benchmark, represent desired outcomes, and generate individual action plans. Locke and Latham (2006) argued that effective goals direct attention, effort, and action towards desired goal-relevant outcomes. Further, they note that learning goals enhance meta- cognition. As Rueda (2011) indicated, goal setting represents a critical first step in providing direction, course corrections, and for providing a benchmark for performance evaluation. In his study on goal theory, Locke (1996) suggested that specific and difficult goals lead to higher per- formance, that commitment is the most critical attribute in goal attainment, and that higher com- IMPROVING PILOT TRAINING OUTCOMES 133 mitment occurs when individuals believe a goal is important and attainable. Linking these ele- ments to organizational goal job aids provides a framework for instructors to align their lesson design and execution. Metacognitive knowledge solutions, or description of needs or assets. Aviation train- ing faculty need to know how to apply metacognitive and self-efficacy practices in their own in- struction and reflect on how they are performing relative to their goals. The American Psycho- logical Association (2015), Dembo and Eaton (2003), and Denler et al. (2009) all suggested that self-regulatory strategies such as goal setting enhance learning and performance in adult educa- tion. Further, Baker (2006) argued that applying metacognitive strategies facilitates improved learning. Aviation training programs are therefore encouraged to provide opportunities for facul- ty to reflect on their own self-regulatory strategies, including the importance and use of reflective practices relative to their stated organizational goals. As a metacognitive strategy, Çubukçu (2012) described the value of creating student- centered learning environments to instill self-efficacy behaviors in students, while involving them in decision making and curriculum development to improve critical thinking skills, prob- lem-solving abilities, and reflective thinking. Pintrich and De Groot (1990) found that when fac- ulty have confidence in implementing self-efficacy instruction in their classrooms, it provides a critical bridge that helps students adopt a mastery orientation towards important concepts and better manage their study efforts. Finally, Schön (1987) described a need for teachers to reflect “in” and “on” action to adapt their instruction to student needs, while Cochran, DeRuiter, and King (1993) argued that teaching and reflection are both important components of effective teacher preparation. For aviation faculty, reflection becomes an essential tool in deciding how to IMPROVING PILOT TRAINING OUTCOMES 134 achieve educational goals, evaluating programs, and determining the success factors in goal at- tainment (Valli, 1997). Procedural knowledge solutions, or description of needs or assets. Aviation training faculty must know how to structure learning experiences to increase student self-efficacy and metacognition abilities in their programs, how to give effective feedback, how to communicate clear performance goals and how to evaluate their teaching relative to student performance. Kirshner et al. (2006) found that increasing germane cognitive load by engaging the learner in meaningful learning and schema construction facilitates effective learning. Likewise, Shute (2008) suggested that private, specific, and timely feedback enhances organizational perfor- mance. Providing job aids to faculty describing steps for improving curriculum and course de- sign, learning schema, and evaluation strategies, as well as providing training on feedback tech- niques and application is an effective strategy for achieving organizational performance goals. Kirshner et al. (2006) suggested that human cognitive architecture should be an important consideration in the design and construction of any instructional program. Constructing a sche- ma to control cognitive load variables so learners are not overwhelmed represents an important challenge to curriculum designers. Tinto (2012) advocated employing learning-environment as- sessment techniques, developing early warnings and interventions where appropriate, and acquir- ing and refining pedagogical skills to engage students in learner-centric environments as effec- tive methods for enhancing classroom effectiveness. Foster et al. (2016) warned that effective student and instructor feedback must be informed by a clear and specific understanding of that at which students should get better, while Kluger and DeNisi (1996) suggested that to be effective, feedback must be timely, task-focused, concrete, and clear. In his book on learning science, Meyer (2011) suggested that learning assessments must be valid, reliable, objective, and refer- IMPROVING PILOT TRAINING OUTCOMES 135 enced. Ultimately, effective feedback is an essential element in evaluating performance relative to organizational goals, and in structuring learning experiences to increase students’ self-efficacy and metacognition abilities. Motivation Recommendations Since no motivation influencers were validated, no recommendations have been made. Organization Recommendations The organization influences shown in Table 15 are derived from the findings and results from Chapter four of this study. They are presented with references from the literature study in Chapter two, along with context-specific recommendations to address performance gaps. Two primary recommendations for addressing organization gaps emerged. First, State University should conduct Professional Development (PD) sessions and provide time during regular meet- ings to focus on learning processes and strategies as well as instructional design. Second, pro- gram leadership must ensure faculty have adequate training and access to resources they need to provide instruction on self-efficacy and metacognition in their curriculum. Table 15 Summary of Organization Influences and Recommendations KMO Organization Influences and Recommendations Assumed Organization Influence Validated? Principle and Citation Context-Specific Rec- ommendation Cultural Model The organization values ex- perimentation and having faculty understand cognitive learning processes by providing learning opportu- nities and requiring their participation in these activi- ties. (Cultural Model 1) Could not be validated N/A N/A The organization provides Yes Insuring staff’s re- Program leadership en- IMPROVING PILOT TRAINING OUTCOMES 136 the resources necessary for aviation training faculty to adopt student self-efficacy and metacognition programs in their pilot training class- rooms. (Cultural Model 2) source needs are being met is correlated with increased student learn- ing outcomes (Waters, Marzano & McNulty, 2003). sures faculty has ade- quate training and access to resources needed to provide instruction on self-efficacy and meta- cognition in their class- rooms. Cultural Setting The organization recognizes and rewards innovation and continuous improvement by providing an evaluation framework that values in- novative practices for teachers to measure their performance against the goal. (Cultural Setting 1) No N/A N/A The organization provides an organizational culture that is accepting of change and able to respond to it ef- fectively. (Cultural Setting 2) No N/A N/A Cultural Models. Cultural Models represent shared mental schema or normative under- standings of how the world works (or ought to work), incorporating behavioral, cognitive, and affective activities (Gallimore & Goldenberg, 2001). To help improve faculty understanding and application of self-efficacy and metacognitive practices in their classrooms, Clark and Estes (2008) emphasized the need for organizational policies and procedures to align with culture. Baker (2006) pointed out that successful organizations provide opportunities for employee self- monitoring, self-assessment, and reflection any time new initiatives are undertaken, and Berger (2014) described the importance of effective communications in helping faculty become more productive, and to feel included and empowered. Aligning organizational policy and resources to assist faculty in developing processes to address student self-efficacy and metacognitive needs and apply cognitive learning processes in their instruction, helps cultivate a shared vision and IMPROVING PILOT TRAINING OUTCOMES 137 expectation (Senge, 1990). To improve student outcomes, aviation training program leadership must provide both academic and flight line faculty with resources designed specifically to en- hance behavioral instruction in their classrooms and in the air. Integrated Implementation and Evaluation Plan Implementation and Evaluation Framework The new world Kirkpatrick evaluation model (Kirkpatrick & Kirkpatrick, 2016), was used to inform the recommendation and evaluation strategies for this study. Based on the semi- nal work of Dr. Donald Kirkpatrick’s original four level model of evaluation (Kirkpatrick & Kirkpatrick, 2006), the new world evaluation model provides a structured framework for align- ing goals and outcomes. The new world evaluation model examines and evaluates four levels of behavior as keys to understanding and evaluating progress towards a goal. Level 1 examines re- actions, or the degree to which participants find training events favorable, engaging, and rele- vant. Level 2 looks at learning, which is the degree to which participants acquire the intended knowledge, skills, attitudes, confidence, and commitment. Level 3 focuses on behavior, or the degree to which participants apply what they learn, and Level 4 examines results, which is the degree to which desired outcomes occur as a direct result of training (Kirkpatrick & Kirkpatrick, 2016). The Kirkpatrick model uses leading indicators and required drivers to shape and evaluate effort. The process works in reverse, beginning with the desired outcome and working back to- ward student satisfaction with the training events. By aligning and targeting results, critical be- haviors, learning, and reactions, the model builds an evaluation strategy organizational stake- holders can use to ensure education and training is effective at the time and place of need. It also ensures that results are consistent with goals and useful in closing performance gaps. IMPROVING PILOT TRAINING OUTCOMES 138 Organizational Purpose, Need, and Expectations The purpose of State University’s aviation training program is to prepare students for a career as a professional aviator in the military or commercial aviation industry. They offer bac- calaureate and graduate degree options to students as they earn their commercial aviation ratings, providing a critical source of new pilots to an industry facing a potentially crippling pilot short- age in the immediate future. With estimated shortages of up to 85,000 domestic and 200,000 international pilots, the need to produce as many safe, competent pilots as possible has never been more urgent in commercial aviation. The proposed solution, focused on improving aviation training program faculty awareness and application of student self-efficacy and metacognitive strategies, should provide an increase in overall graduation rates as well as an increase in student performance, both academically and in flight. Level 4: Results and Leading Indicators Table 16 shows the outcomes, metrics, and methods for the desired external and internal outcomes at State University. If the internal outcomes are met as a result of the PD and Job Aids, then the external outcomes should also be achieved. Table 16 Outcomes, Metrics, and Methods for External and Internal Outcomes Outcome Metric(s) Method(s) External Outcomes 1. Stakeholder airlines in- dicating high quality per- formance of new hires from State University’s professional flight program based on improved instruc- tional design processes Number of reports from airlines indicating satisfaction with State University professional flight program graduates Annual review by the student placement office examining cus- tomer (airline) satisfaction with graduate quality and compe- tence Feedback comments from air- lines related to student self- efficacy and metacognition be- havioral skills and competencies 2. State University aviation Number of State University Annual review by student IMPROVING PILOT TRAINING OUTCOMES 139 training program graduates hired by airlines aviation training program grad- uates hired by airlines placement office 3. State University aviation training program faculty recognized for improve- ment in graduate numbers and quality Number of offers State Univer- sity aviation training program faculty receive to present or speak on program successes Invitations for State University aviation training program facul- ty to speak/write on new in- structional design construct Number of articles and papers written on instructional design improvements in the aviation training program Annual review by aviation train- ing program leadership Internal Outcomes 4. State University aviation training program student comments showing satis- faction with course content and instructional design Number of student reports indi- cating favorable responses to faculty instructional design and performance Student surveys from State Uni- versity aviation training pro- gram administration 5. State University aviation training program faculty participates in regular pro- fessional development (PD)/ programming train- ing focused on organiza- tional goals Number of PD/programming meetings scheduled Bi-annual reviews of faculty PD/programming meetings scheduled and completed Number of syllabus and lesson plan revisions to incorporate self-efficacy and metacognition into instructional design Annual reviews of faculty sylla- bi and lesson programming Level 3: Behavior Critical behaviors. There are four critical behaviors that State University stakeholders must be aware of and practicing to meet the organizational performance goal. First, job aids must be provided to faculty to improve understanding and application of self-efficacy and meta- cognitive instruction. Second, professional development (PD) training and regularly scheduled meetings must be conducted to support faculty application of these practices and improve in- structional design. Third, faculty must be afforded consistent, specific, and timely feedback on their understanding and use of self-efficacy and metacognitive approaches in their curriculum and course design. Fourth, faculty must incorporate elements of self-efficacy and metacognition IMPROVING PILOT TRAINING OUTCOMES 140 in their syllabi and in how they provide instruction to their students. Metrics, methods, and tim- ing for each of these critical behaviors are shown in Table 17. Table 17 Critical Behaviors, Metrics, Methods, and Timing for Faculty Critical Behavior Metric(s) Method(s) Timing 1. Provide job aids to faculty on steps for improving curriculum and course design, learning schema, feedback, and evalua- tion strategies Number of syllabi altered post- instruction State University avia- tion training program chair will assign com- mittee to review syllabi updates Quarterly during first six months of program start Thereafter – annually, so long as previously successful Number of job aids created and provided to faculty Syllabi and curriculum committee review of products and distribu- tion Quarterly during first six months of program start Thereafter – annually, so long as previously successful 2. Faculty attend regu- lar PD/ programming sessions and apply lessons learned in their curriculum de- sign and lesson plans Number of PD/ pro- gramming training sessions completed in the first year State University avia- tion training program chair will assign com- mittee to coordinate and conduct PD/programming train- ing Quarterly during first six months of program start Thereafter – annually, so long as previously successful Number of syllabi and lesson plans al- tered post- instruction State University avia- tion training program chair will assign com- mittee to review syllabi and curriculum updates Quarterly during first six months of program start Thereafter – annually, so long as previously successful 3. Administration provide feedback to faculty on self- efficacy and metacog- nition practices in the classroom Number of syllabi and lesson plans al- tered post- instruction directing feedback options and intervals Syllabi and curriculum committee review of syllabi and lesson plans Quarterly during first six months of program start Thereafter – annually, so long as previously successful 4. Faculty incorporate self-efficacy and met- Number of syllabi altered post- Syllabi and curriculum committee review of Annually, and during subsequent program IMPROVING PILOT TRAINING OUTCOMES 141 acognitive behavioral instruction in their syllabi and instruc- tional design instruction syllabi and instruction- al design reviews Required drivers. Kirkpatrick and Kirkpatrick (2016) outline four categories of re- quired drivers to ensure students assimilate what is learned and to encourage application to job- specific contexts–reinforcing, encouraging, rewarding, monitoring. For the key stakeholders in this aviation training program, several required drivers have been identified and described in Ta- ble 18, which also provides timing factors and connections between supported critical behaviors. Table 18 Required Drivers to Support Faculty’s Critical Behaviors Method(s) Timing Critical Behaviors Supported 1, 2, 3 Etc. Reinforcing Provide job aid on metacogni- tion and self-efficacy, includ- ing the relationship to student performance improvements Ongoing 1, 4 Provide job aid that defines organizational performance goals related to the application of self-efficacy and metacog- nition instruction Ongoing 1, 4 Provide job aid on steps for improving curriculum and course design, learning sche- ma, and evaluation strategies Ongoing 1 Provide faculty training/PD on feedback techniques and application Monthly for first year, then Semi-annually 1, 2, 3 Team meeting to troubleshoot collaboratively and for addi- tional training Weekly 1, 2, 4 Create challenging, yet achievable, instructional de- sign objectives for instructors. Provide clear and accurate Semester 2, 4 IMPROVING PILOT TRAINING OUTCOMES 142 feedback regarding compe- tence and faculty self- efficacy, focused on develop- ing expertise and skill. Provide opportunities for fac- ulty to choose and control content and delivery Encouraging Collaboration and peer model- ing during team meetings Weekly 2 Feedback and coaching from Program Chair or Center Di- rector Ongoing 2 Provide opportunities for fac- ulty to reflect on their own self-regulatory strategies, in- cluding the importance and use of reflective practices relative to their stated goals Ongoing 2, 4 Establish organization and management structures to en- courage personal responsibil- ity for self-efficacy and meta- cognitive practice Semester 1, 3, 4 Rewarding Public acknowledgement, such as a mention at All- Hands meetings, when student performance and graduation rates improve Quarterly 4 Monitoring Program Chair or Center Di- rector creates opportunities at All-Hands meetings to share success stories Quarterly 4 Supervisors ask faculty to self-report their understanding and confidence in applying self-efficacy and metacogni- tive instruction in their classes Quarterly 1, 2, 4 Program Chair provides feed- back to faculty, stressing the process of learning and the importance of effort, strate- gies, and self-control associat- Weekly 1, 2, 4 IMPROVING PILOT TRAINING OUTCOMES 143 ed with learning. Provide opportunities for fac- ulty to choose and control content and delivery. Build supportive and caring person- al relationships between facul- ty and administration Organizational support. As aviation training program leadership organizes and infuses job aids, PD sessions, and behavioral education into their faculty culture, they must provide sup- port to sustain the effort and ensure these changes endure. Reinforcing behaviors, including clear goal-setting and meaningful feedback, form the foundation for faculty development and must be a regular part of institutional practice. State University must continue to encourage fac- ulty as they seek to understand and apply concepts of self-efficacy and metacognition in their curriculum design through collaborating, reflecting, and enhancing a sense of agency and per- sonal accountability. They must recognize and reward successful application of these concepts, indicated by improvements in student achievement as well as increased rates of retention and graduation. Finally, stakeholders must consistently monitor progress by recognizing successes, providing forums for sharing and learning, and enhancing administration and faculty relation- ships. Level 2: Learning Learning goals. Following completion of the recommended solutions, aviation training program faculty will be able to: • describe the relationship between student metacognition, self-efficacy, and stu- dent success (Declarative); • understand faculty’s role in improving learning outcomes (Declarative); • identify the performance goals related to providing self-efficacy and metacogni- IMPROVING PILOT TRAINING OUTCOMES 144 tion instruction to their students (Declarative); • list the steps required to: • structure learning experiences to increase students’ self-efficacy and meta- cognition abilities in their programs • give effective feedback • communicate clear performance goals • evaluate individual teaching relative to student performance (Procedural); • know how to apply metacognitive and self-efficacy practices in their own instruc- tion and reflect on how they are performing relative to their goals (Metacogni- tive); • describe how to apply self-efficacy and metacognitive instruction in all of their classes and instruction sessions (Declarative); • articulate the training goals they will complete to close their performance gaps (Declarative); • attribute student success or failure in part to their skill as an instructor (Self- Efficacy); • understand how faculty can affect and improve a student’s cognitive learning pro- cesses (Cultural Model). Program. These learning goals will be achieved through a training program that ex- plores the fundamental components of learning to help faculty understand how instruction on self-efficacy and metacognition contributes to improved student outcomes. This program will provide specific, targeted job aids and professional development sessions to modify instructional design and course instruction. IMPROVING PILOT TRAINING OUTCOMES 145 Faculty job aids will be developed to provide education and training on key terms, in- struction on how to effectively apply behavioral education through curriculum development and course delivery, and examples related to existing course content. These job aids will: describe the relationship between student metacognition, self-efficacy, and student success; help faculty understand their role in improving learning outcomes; and solidify organizational performance goals related to providing self-efficacy and metacognition instruction to their students. Further, they will provide guidance for how to give effective feedback, communicate clear performance goals, and evaluate individual teaching relative to student performance. The faculty professional development program will include initial and recurring training on improving understanding and use of behavioral strategies such as self-efficacy and metacog- nition, as well as curriculum and lesson design and execution. Training will also establish, dis- tribute, and review organizational goals related to improving student performance in class and in flight, and increasing the number of graduates per year. These mentoring/coaching sessions pro- vide enduring opportunities for enhancing faculty performance and a forum for establishing per- formance goals and strategies to achieve them. Training will establish new processes within ex- isting structures to align personal preferences and routines with improvement goals. Components of learning. Establishing an evaluation framework for Level 2 requires an understanding of goals, objectives, and strategies that ensure the desired content was delivered and received. In their new world model, Kirkpatrick and Kirkpatrick (2016) describe four com- ponents of learning—knowledge, attitude, confidence, and commitment—that measure the de- gree to which training has succeeded. Knowledge evaluations are conducted through practice, discussion, observation, or re- hearsal, though it typically cannot be done effectively by asking questions. Evaluating skill can IMPROVING PILOT TRAINING OUTCOMES 146 also be conducted through practice and rehearsal, requiring participants to show mastery of a new process or task in as real, or authentic, a manner as possible. Application to performance problems is essential. To evaluate attitude, Kirkpatrick and Kirkpatrick (2016) noted the im- portance of ensuring participants understand and see benefit in applying training to their individ- ual contexts. Failure here ensures failure of the overall effort, so leaders must ensure this forma- tive process accurately captures participant opinions through observation, discussion, and direct questioning. Evaluating confidence and commitment involves examining faculty readiness to apply what they have learned to their academic or flight line courses. It addresses questions of overcoming barriers to application, listening to concerns and fears, and honestly evaluating plans. It is conducted through honest discussion, action planning, and feedback. Table 19 pro- vides practical approaches for evaluating these components of learning. Table 19 Components of Learning for the Program. Method(s) or Activity(ies) Timing Declarative Knowledge “I know it.” Knowledge checks through discussions and individu- al/group activities Periodically during the in-person training sessions and documented via observation notes Procedural Skills “I can do it right now.” Using an existing syllabus as an example of how to apply self-efficacy/metacognitive strategies in current practice During the training course Individual application of instructional skills from job aids/PD with existing syllabi During the training course Demonstration of applying job aids to make changes to existing syllabi During the training course Attitude “I believe this is worthwhile.” Instructor’s observation of participant’s statements and actions demonstrating that they see the benefit of what they are being asked to do on the job During the training course Discussions of the value of what they are being asked to do on the job During the training course Retrospective pre- and post-test assessment item After the training course IMPROVING PILOT TRAINING OUTCOMES 147 Confidence “I think I can do it on the job.” Retrospective pre- and post-test assessment item After the training course Commitment “I will do it on the job.” Discussions following practice and feedback During the training course Create an individual action plan During the training course Retrospective pre- and post-test assessment item After the training course Level 1: Reaction Evaluating Level 1, or the degree to which participants find training useful to their con- text, is the last level to be addressed. Kirkpatrick and Kirkpatrick (2016) suggest this should be done quickly and efficiently, as a measure of simply confirming the quality of the program and instructor. This formative method involves surveys provided immediately after training, obser- vation by the instructor, periodic pulse-checks from participants, and course evaluations provid- ed shortly after training has concluded. Level 1 evaluations examine engagement, relevance, and student satisfaction with the training. Table 20 provides examples of how these evaluations can support State University’s faculty support initiative. Table 20 Components to Measure Reactions to the Program Method(s) or Tool(s) Timing Engagement Observation by instructor/facilitator During the training sessions Attendance During the training sessions Course evaluation Two weeks after the training sessions Relevance Brief pulse-check with participants via survey (online) and discussion (ongoing) Following the training sessions Course evaluation Two weeks after the course Customer Satisfaction Brief pulse-check with participants via survey (online) and discussion (ongoing) Following the training sessions Course evaluation Two weeks after the course. Evaluation Tools IMPROVING PILOT TRAINING OUTCOMES 148 Immediately following the program implementation. During the initial PD session, program leadership and the instructors will collect data from the faculty participants about their willingness to participate, their understanding of the material and its significance, and their con- fidence with using their new skills and understanding in their current coursework and instruc- tional design. Brief surveys will be provided to faculty attendees to measure satisfaction with the course materials, application to their course of instruction, and understanding of organizational goals (Appendix I). For Level 1 evaluation, instructors will conduct periodic brief pulse-checks by asking faculty participants about the relevance of the content to their courses and learning environments. Level 2 evaluation will include checks for understanding, syllabus reviews, and organizational goal reviews at follow-on PD sessions. Delayed for a period after the program implementation. Approximately six weeks af- ter the initial PD session, and at the recurring follow-on PD sessions, program leadership will administer another brief survey using a blended evaluation approach containing open and scaled items (Appendix J). This survey will measure faculty satisfaction and determine relevance of the training (Level 1), confidence and understanding of value towards applying their training and job aids (Level 2), application of the training elements to achieve organization goals and support from peers and program administration (Level 3), and the extent to which their approaches to instructional design are contributing to improving student outcomes (Level 4). Data Analysis and Reporting Since faculty PD sessions will occur quarterly, faculty understanding and use of behav- ioral strategies will be discussed and evaluated during each of these meetings. To examine sup- port for the organizational goal of increasing student graduation rates, graduation statistics and IMPROVING PILOT TRAINING OUTCOMES 149 student performance reviews will be provided and discussed annually during the annual fall PD sessions. This affords faculty the opportunity to establish a baseline from previous years, com- pare successes related to organization goals, and map plans for the coming year. To examine support for the organizational goal of increasing student performance in the classroom and in the air, trend data from the flight instruction contractor will be given to State University’s Professional Flight Program for analysis and discussion at PD sessions. This data will also be used for analysis and performance discussion and guidance with the contractor in- structor pilots. Notes from the PD sessions, including goals, objectives, and tools, will be cap- tured and distributed to faculty through email and reviewed quarterly at follow-on PD sessions. During the annual spring PD session, instructional design strategies, curriculum, and course changes will be shared, and student performance and graduation rates will be examined and compared to previous years, in light of current organizational goals. Planning will begin for the next year’s performance goals, including instructional design planning for academic and flight elements. Reviews, discussions about behavioral instruction, and regular goal and metrics re- views must become enculturated into the organization for this change to effectively take hold. Limitations and Delimitations for Future Research As with any research program, there were limitations to this study that must be acknowl- edged. While a great deal has been written about the application of self-efficacy and metacogni- tion in academic (classroom) settings, little has been written on the application of these behav- ioral traits in the combined cognitive, psychomotor, and affective domains found in other con- texts—such as aviation. Medicine, in particular surgery and trauma care, provides similar refer- IMPROVING PILOT TRAINING OUTCOMES 150 ences, but more study is warranted as a method of improving learner outcomes measured through increased performance, resilience, confidence, and successful completion of training rates. All survey, interview, and observation protocols and instruments were developed specifically for this study, and, while each was pilot-tested prior to use at State University, none have been rigorous- ly field-tested in other schools or contexts. Survey and interview data were all self-reported, which made results subject to biases in the form of selective memory, telescoping, attribution, and exaggeration. Finally, all instruments in this study will be used only once, with no oppor- tunity for longitudinal comparisons. While this study examines a large university aviation training program within the AABI portfolio, further research is recommended at smaller school settings and in different regions of the country to compare results for greater generalizability and scaling. Other potential issues of delimitation include a deliberate focus on teachers and instructors rather than program leadership or administration, choices of faculty/instructors to interview, and the choice of which classes and training sessions to observe for data collection purposes. Recommendations for Future Research While numerous forecasting studies indicate an imminent shortage of commercial pilots, organizations such as the Airline Pilots Association and the Government Accountability Office continue to express doubt on the timing, intensity, and estimated duration of potential shortages. In their Aviation Workforce analysis, the GAO noted, “studies GAO reviewed examining wheth- er the future supply of pilots will be sufficient to meet this need had varying conclusions” (Avia- tion Workforce, 2014). Further research into exactly how many pilots need to be produced an- nually, and at what level of skill and competence, is warranted. While the quantity and composi- tion of faculty feedback to students was not specifically noted in the list of assumed influences, IMPROVING PILOT TRAINING OUTCOMES 151 responses and observations indicated a variety of techniques between individual instructors. Work by Kluger and Denisi (1996) and others on feedback interventions indicates there is poten- tial for further research, and for discussion at faculty PD sessions. Conclusion Training student pilots requires a detailed understanding of adult learning behaviors and the learning domains in which they operate. While there are similarities between flying and driv- ing a car or sailing a boat, the complexities of operating in three dimensions simultaneously while contending with the machine, the elements, and other human beings makes the exercise daunting at first, but ultimately fulfilling and satisfying to many participants. Instructing others on how to safely navigate the skies requires patience, persistence, and compassion, as demon- strated every day by the aviation training faculty at State University. As with any human system, there are always opportunities for improving the art and science of instruction, and this study ex- amined behavioral aspects designed to enhance a student pilot’s competence and confidence in flight. This study used Clark and Estes (2008) methodological framework to demonstrate how faculty can accomplish this by applying self-efficacy instruction and metacognitive practices in their instructional design. By examining knowledge, motivation, and organization influences, State University can identify and pursue strategies for closing the gap between the number and quality of graduates today and what is needed in the future. As the research has shown, this may require faculty to improve their personal competence as teachers, mentors, and coaches. A com- prehensive evaluation strategy that links goals to practice is essential to ensure the success of this change effort. The Kirkpatrick new world evaluation model provides an excellent template for State University to evaluate progress towards their performance objectives. Through reflective IMPROVING PILOT TRAINING OUTCOMES 152 goal setting and continuous faculty development, this program stands ready to meet the challenge for producing more, better qualified aviators today and tomorrow. IMPROVING PILOT TRAINING OUTCOMES 153 References AABI Mission. (2016). Retrieved July 19, 2017, from http://www.aabi.aero/mission.html About AABI. (2017). Retrieved January 09, 2018, from http://www.aabi.aero/about-aabi/ Air Force leaders address aircrew crisis. (2017, September 22). Retrieved November 25, 2017, from http://www.af.mil/News/Article-Display/Article/1321906/air-force-leaders-address- aircrew-crisis/ Anderman, E. (2015). Goal Orientation Theory. Retrieved from http://www.education.com/reference/article/goal-orientation-theory/. Anderman, E. & Anderman, L. (2006). Attributions. Retrieved from http://www.education.com/reference/article/attribution-theory/. Atienza, M.A. (2010). Teaching and learning in the health sciences. E. A. Sana (Ed.). Diliman, Quezon City: University of the Philippines Press. ASU Professional Flight Mission Statement. (2016). Retrieved November 6, 2016, from http://poly.engineering.asu.edu/aviation/professional-flight-mission/ Aviation Benefits Beyond Boarders (Rep.). (2016). Geneva: Air Transport Action Group. Aviation Workforce: Current and Future Availability of Airline Pilots (Rep. No. GAO-14-232). (2014). Retrieved November 25, 2017, from https://www.gao.gov/products/GAO-14-232 Aviation Instructor’s Handbook. (2008). Washington, D.C.: FAA. Baker, L. (2006). Metacognition. Retrieved from http://www.education.com/reference/article/metacognition/. Ball, D. L., & Cohen, D. K. (1999). Developing practices, developing practitioners: Toward a IMPROVING PILOT TRAINING OUTCOMES 154 practice-based theory of professional development. In G. Sykes & L. Darling-Hammonds (Eds.), Teaching as the learning profession: Handbook of policy and practice (pp. 30– 32). San Francisco, CA: Jossey-Bass. Bandura, A. (1986). Social foundation of thought and action: A social-cognitive view. Englewood Cliffs. Bandura, A. (1991). Social cognitive theory of self-regulation. Organizational behavior and human decision processes, 50(2), 248-287. Bandura, A. (2000). Exercise of human agency through collective efficacy. Current directions in psychological science, 9, 75-78. doi: Bandura, A. (2005). The evolution of social cognitive theory. Great minds in management, 9-35. Bengtsson, J. (1995). What is reflection? On reflection in the teaching profession and teacher education. Teachers and Teaching, 1(1), 23-32. Berbary, D. F., & Malinchak, A. A. (2011). Connected and Engaged: The Value of Government Learning. Public Manager, 40(3), 55. Berger, B. K. (2014). READ MY LIPS: Leaders, Supervisors, and Culture Are the Foundations of Strategic Employee Communications. Research Journal of the Institute for Public Re- lations, 1(1), 2-17. Bjerke, E., & Malott, D. (2011). Impacts of public law 111-216: Will the flight instructor career path remain a viable option for aspiring airline pilots? Collegiate Aviation Review, 29(1), 1. Boeing: 2015 Pilot and Technician Outlook. (2015). Retrieved from http://www.boeing.com/resources/boeingdotcom/commercial/boeing- edge/assets/brochure/edge-flight-services/pilottechnicianoutlook.pdf IMPROVING PILOT TRAINING OUTCOMES 155 Boeing: 2017 Pilot and Technician Outlook. (2017). Retrieved from http://www.boeing.com/commercial/market/pilot-technician-outlook/2017-pilot-outlook/ Borgogni, L., Dello Russo, S., Petitta, L., & Vecchione, M. (2010). Predicting job satisfaction and job performance in a privatized organization. International Public Management Journal, 13(3), 275-296. Buckingham, M., & Coffman, C. (2014). First, break all the rules: What the world's greatest managers do differently. Simon and Schuster. Burke, W. W. (2002). Organization change: Theory and practice. Sage Publications. Certification: Pilots, Flight Instructors, and Ground Instructors, 14 C.F.R. § 61 (2017). Chemers, M. M., Hu, L., & Garcia, B. F. (2001). Academic self-efficacy and first year college student performance and adjustment. Journal of Educational Psychology,93(1), 55-64. Chevalier, R. (2010). Gap analysis revisited. Performance Improvement, 49(7), 5-7. Christensen, L. B., & Johnson, B. (2014). Research methods, design, and analysis (Fifth ed.). Thousand Oaks, CAc: SAGE Publications. Clark, R. E., & Estes, F. (2008). Turning research into results: A guide to selecting the right performance solutions. Charlotte, NC: Information Age Publishing, Inc. Cochran, K. F., DeRuiter, J. A., & King, R. A. (1993). Pedagogical content knowing: An integrative model for teacher preparation. Journal of teacher education, 44(4), 263-272. Cohen, D. K., & Hill, H. C. (2000). Instructional policy and classroom performance: The mathematics reform in California. Teachers College Record, 102(2), 294–343. The 2016 Collegiate Aviation Guide. (2016). Retrieved February 06, 2017, from https://issuu.com/universityaviationassociation/docs/cag_2016_online_rev6 Corcoran, T. B., Shields, P. M., & Zucker, A. A. (1998). The SSIs and professional development IMPROVING PILOT TRAINING OUTCOMES 156 for teachers. Menlo Park, CA: SRI International. Coutinho, S. (2008). Self-efficacy, metacognition, and performance. North American Journal of Psychology, 10(1), 165. Coutinho, S. A., & Neuman, G. (2008). A model of metacognition, achievement goal orientation, learning style and self-efficacy. Learning Environments Research, 11(2), 131-151. Creswell, J. W. (2014). Research design: qualitative, quantitative, and mixed methods approaches. Los Angeles: SAGE Publications. Çubukçu, Z. (2012). Teachers' evaluation of student-centered learning environments. Education, 133(1), 49-66. Darling-Hammond, L., & McLaughlin, M. W. (1995). Policies that support professional development in an era of reform. Phi Delta Kappan, 76(8), 597–604. Davis, W. D., Fedor, D. B., Parsons, C. K., & Herold, D. M. (2000). The development of self- efficacy during aviation training. Journal of Organizational Behavior, 857-871. Deans for Impact (2015). The Science of Learning. Austin, TX: Deans for Impact. Dembo, M. H., & Eaton, M. J. (2000). Self-regulation of academic learning in middle-level schools. The Elementary School Journal, 473-490. Denler, H., Wolters, C., & Benzon, M. (2014). Social cognitive theory. Retrieved from http://www.education.com/reference/article/social-cognitive-theory/. Depperschmidt, C. L. (2013). Public Law 111-216: Effects of new legislation on collegiate aviation flight training programs. Collegiate Aviation Review, 31(1), 1. Dewey, J. (1910). How we think. Retrieved June 26, 2017, from http://www.gutenberg.org/files/37423/37423-h/37423-h.htm Dewey, J. (1933). How we think: A restatement of the reflective thinking to the educative IMPROVING PILOT TRAINING OUTCOMES 157 process. Heath. Düvel, N., Wolf, A., & Kopiez, R. (2017). Neuromyths in Music Education: Prevalence and Pre dictors of Misconceptions among Teachers and Students. Frontiers in Psychology, 8. Elmore, R. F. (1997). Investing in teacher learning: Staff development and instructional improvement in Community School District #2, New York City. New York, NY: National Commission on Teaching & America’s Future. Federal Aviation Administration (2013). Advisory Circular 61-139, Institution of Higher Education’s Application for Authority to Certify its Graduates for an Airline Transport Pilot Certificate with Reduced Aeronautical Experience. Retrieved from https://www.faa.gov/documentLibrary/media/Advisory_Circular/AC_61-139.pdf Federal Aviation Administration. (2014). The Economic Impact of Civil Aviation on the U.S. Economy. Retrieved from https://www.faa.gov/air_traffic/ publications/ media/2014- economic-impact-report.pdf . Ferguson, M. D., & Johnson, J. A. (2007). Cost and perceived value in obtaining a bachelor's degree in aviation professional flight: Will collegiate aviation price themselves out of the market with technologically advanced aircraft? Journal of Aviation/Aerospace Education & Research, 17(1). Fifield, S., & Kedzior, M. (2004). Teacher professional development. Retrieved from http://dspace.udel.edu/bitstream/handle/19716/245/v15_May.pdf?sequence=1 Fink, A. (2012). How to conduct surveys: A step-by-step guide: A step-by-step guide. Sage Publications. Foster, R., Burn, K., McConnell, T., & Counsell, C. (2016). Feedback. Teaching History,164, 2. IMPROVING PILOT TRAINING OUTCOMES 158 Fullingim, J. F. (2011). The marketability of higher education aviation graduates as perceived by regional airline pilots. Collegiate Aviation Review, 29(1), 28. Gallimore, R., & Goldenberg, C. (2001). Analyzing cultural models and settings to connect minority achievement and school improvement research. Educational Psychologist, 36(1), 45-56. Gilmore, C. K., McCarthy, S. E., & Spelke, E. S. (2007). Symbolic arithmetic knowledge with out instruction. Nature, 447(7144), 589-591. Glesne, C. (2011). Chapter 6: But is it ethical? Considering what is “right.” In Becoming qualitative researchers: An introduction (4 th ed.) (pp. 162-183). Boston, MA: Pearson Global Market Forecast (Rep.). (2017, April). Retrieved November 25, 2017, from Airbus web site: http://www.airbus.com/content/dam/corporate-topics/ publications/ backgrounders/ Airbus_Global_Market_Forecast_2017-2036_Growing_Horizons_full_book.pdf Grossman, R., & Salas, E. (2011). The transfer of training: what really matters. International Journal of Training and Development, 15(2), 103-120. Hansen, F., Smith, M., & Hansen, R. B. (2002). Rewards and recognition in employee motivation. Compensation & Benefits Review, 34(5), 64-72. Harrow, A. J. (1972). A taxonomy of the psychomotor domain: A guide for developing behavior- al objectives. Addison-Wesley Longman Ltd. Henry, J. M. (1997). Gaming: A teaching strategy to enhance adult learning. Journal of Continu- ing Education in Nursing, 28(5), 231. Higgins, J., Lovelace, K., Bjerke, E., Lounsberry, N., Lutte, R., Friedenzohn, D., ... & Craig, P. (2013). An Investigation of the United States Airline Pilot Labor Supply. Grand Forks, ND, University of North Dakota, University of Nebraska Omaha, Embry-Riddle Aeronau- IMPROVING PILOT TRAINING OUTCOMES 159 tical University, Southern Illinois University, LeTourneau University, Middle Tennessee State University, 35. H.R. 110-135, 110th Cong., 153 (2008) (enacted). Integrated Airman Certification and Rating Application (IACRA). (2017, May 1). Retrieved July 17, 2017, from https://iacra.faa.gov/IACRA/default.aspx. IACRA Version 8.14.5 International Air Transport Association. (2014). Aviation benefits beyond borders. www. avia- tionbenefits beyondborders. org, 2-8. International Civil Aviation Organization. (2011). ICAO Study Reveals Strong Demand for Qualified Aviation Personnel up to 2010. Retrieved from http://www.icao.int/Newsroom/News%20Doc/PIO.03.11.EN.pdf. Kaplan, A., & Maehr, M. L. (2007). The contributions and prospects of goal orientation theory. Educational psychology review, 19(2), 141-184. Keep America Flying: A Flight Plan for Safe and Fair Skies. (2016, April 1). Airline Pilot's As- sociation, International. Retrieved from https://www.alpa.org/~/media/D376EB61112D48E38350A58FE899C3BA.ashx Kerr, P. (2016). Adaptive learning. ELT Journal, 70(1), 88-93. doi:10.1093/elt/ccv055 Kirkpatrick, D. L. (2006). Seven keys to unlock the four levels of evaluation. Performance Improvement, 45(7), 5-8. Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick's four levels of training evaluation. Association for Talent Development. Kirschner, P. A. (2002). Cognitive load theory: Implications of cognitive load theory on the design of learning. Learning and instruction, 12(1), 1-10. Kluger, A. N., & Denisi, A. (1996). The effects of feedback interventions on performance: A IMPROVING PILOT TRAINING OUTCOMES 160 historical review, a meta-analysis, and a preliminary feedback intervention theo- ry. Psychological Bulletin,119(2), 254-284. Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into practice, 41(4), 212-218. Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1964). Taxonomy of educational objectives: The classification of educational goals. Handbook II: The affective domain. New York: David M Krueger, R. A., & Casey, M. A. (2009). Focus groups: a practical guide for applied research (Fourth ed.). Los Angeles: SAGE. Little, J. W. (1993). Teachers’ professional development in a climate of educational reform. Educational Evaluation & Policy Analysis, 15(2), 129–151. Livingston, J. (1997). Metacognition: An Overview. Retrieved May 29, 2017, from http://gse.buffalo.edu/fas/shuell/cep564/metacog.htm Locke, E. A. (1996). Motivation through conscious goal setting. Applied and preventive psychology, 5(2), 117-124 Locke, E. A., & Latham, G. P. (2006). New directions in goal-setting theory. Current directions in psychological science, 15(5), 265-268. Lorains, M., MacMahon, C., Ball, K., & Mahoney, J. (2011). Above real time training for team invasion sport skills. International Journal of Sports Science and Coaching, 6(4), 537- 544. doi:10.1260/1747-9541.6.4.537 Lovelace, K., & Higgins, J. (2012). US pilot labor supply. In US/Europe International Aviation Safety Conference, Cleveland Ohio USA June (Vol. 12, p. 14). Maxwell, J. A. (2013). Qualitative research design: an interactive approach (Third ed.). IMPROVING PILOT TRAINING OUTCOMES 161 Thousand Oaks, CA: SAGE Publications. Mayer, R. E. (2011). Applying the science of learning. Boston, MA: Pearson/Allyn & Bacon. McGee, M. (2015). Air transport pilot supply and demand: Current state and effects of recent legislation (Doctoral dissertation, Pardee Rand Graduate School). RAND Cooporation. McGill, T. J., & Volet, S. E. (1997). A conceptual framework for analyzing students’ knowledge of programming. Journal of research on Computing in Education, 29(3), 276-297. McLaughlin, K. (2015, May 27). Strapped for pilots, Republic asks partners to reduce flights. Retrieved April 27, 2016, from http://www.ibj.com/articles/53343-strapped-for-pilots- republic-asks-partners-to-reduce-flights McRae, J. (2017, May 19). Goldfein meets with airline executives. Retrieved November 27, 2017, from http://www.af.mil/News/Article-Display/Article/1187913/goldfein-meets- with-airline-executives/ Merriam, S. B., & Tisdell, E. J. (2016). Qualitative research: a guide to design and implementation. San Francisco, CA: Jossey-Bass. Military Pilot Shortage: Hearings before the Subcommittee on Military Personnel of the House Armed Services Committee Mar. 29, 2017. 115th Cong. 2 (2017) (statement of Vice Ad- miral Robert P. Burke, Chief of Naval Personnel, United States Navy). Murray, G. (2017, October 24). Aviation Week & Space Technology: Geoff Murray on US Pilot Shortage. Retrieved November 25, 2017, from http://www.oliverwyman.com/media- center/2017/oct/aviation-week---space-technology--geoff-murray-on-us-pilot-short.html National Commission on Teaching and America’s Future (1996). What matters most: Teaching for America’s future. Summary Report, New York, 9 IMPROVING PILOT TRAINING OUTCOMES 162 National Transportation Safety Board. (2010). Loss of Control on Approach Colgan Air, Inc. Operating as Continental Connection Flight 3407. Retrieved from http://www.ntsb.gov/investigations/AccidentReports/Reports/AAR1001.pdf . Nielsen, J., Zielinski, B., Ferguson, M., Lainhart, J., & Anderson, J. (2013). An Evaluation of the Left-Brain vs. Right-Brain Hypothesis with Resting State Functional Connectivity Mag- netic Resonance Imaging. PLoS ONE, 8(8), E71275. Niemczyk, M. (2008). Student Approaches to Learning in Aviation Contexts. Journal of Avia- tion/Aerospace Education & Research, 18(1). Retrieved from http://commons.erau.edu/jaaer/vol18/iss1/7 Niemczyk, M., & Ulrich, J. W. (2009). Motivation and learning strategies influencing perfor- mance in an aviation course. Collegiate Aviation Review, 27(1), 65. O’Neill, G., & Murphy, F. (2010, January 20). Guide to Taxonomies of Learning [PDF]. Dublin: University College Dublin. Oaks, D. A. J. (2015). Mapping to Curricular and Institutional Goals. New Directions for Institutional Research, 2014(164), 51-60. Organisation for Economic Co-operation Development. (2002). Understanding the brain: To wards a new learning science. Paris: Organisation for Economic Co-operation and De- velopment. Pajares, F. (1996). Self-efficacy beliefs in academic settings. Review of educational research, 66(4), 543-578. Pajares, F. (2006). Self-efficacy theory. Retrieved from http://www.education.com/reference/article/self-efficacy-theory/. Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: Concepts and IMPROVING PILOT TRAINING OUTCOMES 163 evidence. Psychological science in the public interest, 9(3), 105-119. Patton, M. Q. (2002). Qualitative research and evaluation methods (Third ed.). Thousand Oaks, CA: Sage Publications. Patton, M. Q. (2015). Qualitative research and evaluation methods (Fourth ed.). Thousand Oaks, CA: Sage Publications. Personnel Licensing FAQ, ICAO. (2012, August 7). Retrieved July 19, 2017, from https://www.icao.int/safety/airnavigation/Pages/peltrgFAQ.aspx#10 Phillips, J. M., & Gully, S. M. (1997). Role of goal orientation, ability, need for achievement, and locus of control in the self-efficacy and goal-setting process. Journal of applied psy- chology, 82(5), 792-802. Pinto, Y., de Haan, E. H., & Lamme, V. A. (2017). The Split-Brain Phenomenon Revisited: A Single Conscious Agent with Split Perception. Trends in Cognitive Sciences, 21(11), 835- 851. Pintrich, P. R. (2000). An achievement goal theory perspective on issues in motivation terminology, theory, and research. Contemporary educational psychology, 25(1), 92-104. Pintrich, P. R. (2002). The role of metacognitive knowledge in learning, teaching, and assessing. Theory into practice, 41(4), 219-225. Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of educational psychology, 82(1), 33. Pilot Schools, 14 C.F.R. § 141 (2017). Prawat, R. S., Byers, J. L., & Anderson, A. H. (1983). An attributional analysis of teachers’ affective reactions to student success and failure. American Educational Research Jour- nal, 20(1), 137-152. IMPROVING PILOT TRAINING OUTCOMES 164 Prentice, B., & Gouel, P. (2016, January 28). Pilot Shortage Threatens To Slow U.S. Airline Growth. Forbes. Retrieved October 16, 2016, from http://www.forbes.com/sites/oliverwyman/2016/01/28/pilot-shortage-threatens-to-slow-u- s-airline-growth/#2b4d404bb6ec Procee, H. (2006). Reflection in education: A Kantian epistemology. Educational Theory, 56(3), 237-253. Reigeluth, C. M. (1999). Instructional-design Theories and Models. Mahwah N.J.: L. Erlbaum. Reio, T. G., Rocco, T. S., Smith, D. H., & Chang, E. (2017). A Critique of Kirkpatrick's Evaluation Model. New Horizons in Adult Education and Human Resource Development, 29(2), 35-53. Republic Airways Holdings Inc., Cooperate Communications. (2016, February 25). Republic Airways Holdings Inc. and Subsidiaries File for Chapter 11 Reorganization [Press re- lease]. Retrieved November 25, 2017, from http://www.rjet.com/Press_Releases.aspx Roberson, Darryl. “Comments at Commander’s Call.” 30 January 2017. Joint Base San Antonio-Randolph, TX. Rolfe, J. M., & Staples, K. J. (Eds.). (1988). Flight simulation (No. 1). Cambridge University Press. Rueda, R. (2011). The 3 Dimensions of Improving Student Performance: Finding the Right Solutions to the Right Problems. Teachers College Press. 1234 Amsterdam Avenue, New York, NY 10027. Schein, E. H. (1984). Coming to a new awareness of organizational culture. Sloan management review, 25(2), 3-16. Schneider, B., Brief, A. P., & Guzzo, R. A. (1996). Creating a climate and culture for sustainable IMPROVING PILOT TRAINING OUTCOMES 165 organizational change. Organizational dynamics, 24(4), 7-19. Schön, D. A. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. Jossey-Bass. Schön, D. A. (1995). Knowing-In-Action: The New Scholarship Requires a New Epistemology. Change: The Magazine of Higher Learning, 27(6), 27-34. Schraw, G., & McCrudden, M. (2013). Information processing theory. Information Processing Theory, 420-433. Schunk, D. H., & Pajares, F. (2009). Self-efficacy theory. Handbook of motivation at school, 35- 53. Schwandt, D., & Marquardt, M. J. (1999). Steps and strategies for brining organizational learning to your organization. Organizational learning from world-class theories to global best practices. New your, NY: St. Lucie Press, 227-249 Senge, P. M. (1990). The leader’s new work: Building learning organizations. Sloan Management Review, 32(1), 7-23 Silverman, M. (2004). Non-financial recognition: The most effective of rewards. Institute for Employment Studies Research Network. Simpson, B.J. (1966). The classification of educational objectives: Psychomotor domain. Illinois Journal of Home Economics, 10(4), 110-144. Skaalvik, E. M., & Skaalvik, S. (2013). Teachers’ perceptions of the school goal structure: Relations with teachers’ goal orientations, work engagement, and job satisfaction. Inter- national Journal of Educational Research, 62, 199-209. Sweeney, N. (2014). Predicting active duty Air Force pilot attrition given an anticipated increase in major airline pilot hiring. The Pardee RAND Graduate School. IMPROVING PILOT TRAINING OUTCOMES 166 Tanner, K. D. (2012). Promoting student metacognition. CBE-Life Sciences Education, 11(2), 113-120. Tillema, H. (2000). Belief change towards self-directed learning in student teachers: immersion in practice or reflection on action. Teaching and Teacher Education, 16(5-6), 575-591. Tinto, V. (2012). Enhancing student success: Taking the classroom success seriously. The International Journal of the First Year in Higher Education,3(1). United States, Government Accountability Office, Aviation Workforce. (2014, February 28). Retrieved August 21, 2016, from http://www.gao.gov/products/GAO-14-232 Valli, L. (1997). Listening to other voices: A description of teacher reflection in the United States. Peabody journal of Education, 72(1), 67-88. Wolfe, K. (2005). Older pilots find not-so-friendly skies. CQ Weekly, 218. Yoon, K. S., Duncan, T., Lee, S. W. Y., Scarloss, B., & Shapley, K. L. (2007). Reviewing the Evidence on How Teacher Professional Development Affects Student Achievement. Is- sues & Answers. REL 2007-No. 033. Regional Educational Laboratory Southwest (NJ1). Yough, M., & Anderman, E. (2006). Goal orientation theory. Retrieved from http://www.education.com/reference/article/goal-orientation-theory/. IMPROVING PILOT TRAINING OUTCOMES 167 APPENDIX A SURVEY INSTRUMENT PROTOCOL EMAIL 1/11/18, 22'38 University of Southern California Mail - Improving Pilot Training Study Page 1 of 4 https://mail.google.com/mail/u/0/?ui=2&ik=76c31d7d51&jsver=pkG7bi…iml=15f3acdaf86013f4&siml=15f3acdc2db85964&siml=15f3acdc9ae1de12 Regan Patrick <rjpatric@usc.edu> Improving Pilot Training Study 3 messages Regan Patrick <rjpatric@usc.edu> Fri, Oct 20, 2017 at 12:21 PM To: paul.cline@asu.edu, captdbluemist@gmail.com, hjdelugt@asu.edu, gregory.files@asu.edu, awgaglia@mainex1.asu.edu, michawl.hampshire@asu.edu, carol.hannah@asu.edu, james.hartman@asu.edu, jimmy.kimberly@asu.edu, verne.latham@asu.edu, mike.mccarthy@asu.edu, Mary Meeker <MARY.SCRANTON@asu.edu>, robert.nullmeyer@asu.edu, Marc O'Brien <MARC.OBRIEN@asu.edu>, sandra.l.park@asu.edu, michael.person@asu.edu, richard.sauer@asu.edu, david.sperling@asu.edu, kathy.wallmueller@asu.edu, anthony.j.wende@asu.edu Cc: Mary Niemczyk <Mary.Niemczyk@asu.edu> Bcc: Regan Patrick <rjpatric@usc.edu> Hello! I am doctoral candidate at the University of Southern California conducting a research project to explore ways to improve pilot training instruction. I am seeking volunteers to participate in a brief survey (approximately 15 minutes), and I'm asking for help from the ASU Professional Flight Team. All information obtained will be securely maintained to preserve confidentiality and anonymity, and the analysis and reporting of any data will ensure the identity of participants cannot be determined. Participation is voluntary and you can withdraw your involvement at any time. All data obtained will be destroyed at the completion of the study. Please call me at 505-206-4748 if you have any questions, or email me at rjpatric@usc.edu. If you'd like to participate in this study, please complete the online survey at the following link: https://usceducation.az1.qualtrics.com/jfe/form/SV_a9jPjQfHh9Az1Ah Thank you very much! Regan J. Patrick Doctoral Candidate Rossier School of Education University of Southern California ------------------------------------------- (505) 206-4748 rjpatric@usc.edu Mail Delivery System <MAILER-DAEMON@mx0b-00164701.pphosted.com> Fri, Oct 20, 2017 at 12:21 PM To: rjpatric@usc.edu This is the Email Security Gateway at host bcnetw01.asu.edu. 1/11/18, 22'38 University of Southern California Mail - Improving Pilot Training Study Page 1 of 4 https://mail.google.com/mail/u/0/?ui=2&ik=76c31d7d51&jsver=pkG7bi…iml=15f3acdaf86013f4&siml=15f3acdc2db85964&siml=15f3acdc9ae1de12 Regan Patrick <rjpatric@usc.edu> Improving Pilot Training Study 3 messages Regan Patrick <rjpatric@usc.edu> Fri, Oct 20, 2017 at 12:21 PM To: paul.cline@asu.edu, captdbluemist@gmail.com, hjdelugt@asu.edu, gregory.files@asu.edu, awgaglia@mainex1.asu.edu, michawl.hampshire@asu.edu, carol.hannah@asu.edu, james.hartman@asu.edu, jimmy.kimberly@asu.edu, verne.latham@asu.edu, mike.mccarthy@asu.edu, Mary Meeker <MARY.SCRANTON@asu.edu>, robert.nullmeyer@asu.edu, Marc O'Brien <MARC.OBRIEN@asu.edu>, sandra.l.park@asu.edu, michael.person@asu.edu, richard.sauer@asu.edu, david.sperling@asu.edu, kathy.wallmueller@asu.edu, anthony.j.wende@asu.edu Cc: Mary Niemczyk <Mary.Niemczyk@asu.edu> Bcc: Regan Patrick <rjpatric@usc.edu> Hello! I am doctoral candidate at the University of Southern California conducting a research project to explore ways to improve pilot training instruction. I am seeking volunteers to participate in a brief survey (approximately 15 minutes), and I'm asking for help from the ASU Professional Flight Team. All information obtained will be securely maintained to preserve confidentiality and anonymity, and the analysis and reporting of any data will ensure the identity of participants cannot be determined. Participation is voluntary and you can withdraw your involvement at any time. All data obtained will be destroyed at the completion of the study. Please call me at 505-206-4748 if you have any questions, or email me at rjpatric@usc.edu. If you'd like to participate in this study, please complete the online survey at the following link: https://usceducation.az1.qualtrics.com/jfe/form/SV_a9jPjQfHh9Az1Ah Thank you very much! Regan J. Patrick Doctoral Candidate Rossier School of Education University of Southern California ------------------------------------------- (505) 206-4748 rjpatric@usc.edu Mail Delivery System <MAILER-DAEMON@mx0b-00164701.pphosted.com> Fri, Oct 20, 2017 at 12:21 PM To: rjpatric@usc.edu This is the Email Security Gateway at host bcnetw01.asu.edu. IMPROVING PILOT TRAINING OUTCOMES 168 APPENDIX B SURVEY Welcome Thank you for participating in our research study! It should take about 15 minutes to complete. This study aims to understand your perceptions and attitudes regarding the application of self- efficacy and metacognitive instruction to improve student outcomes, measured by an increase in the quality and quantity of graduates. You do not have to answer any questions you don’t want to. Your responses will re- main confidential and anonymous. When the results of the research are published no identifiable information will be used. For questions, please contact the Principal Investigator, Regan Patrick via email at rjpatric@usc.edu, or the University Park Institutional Review Board at upirb@usc.edu. Click the Next arrows to get started! Q1 Please state the amount of training (in years and months, if any) you’ve received in adult ed- ucation – specifically, how to teach adults, build syllabi, lesson plans, and/or course develop- ment. o None o 1 - 6 months o 7 - 11 months o 1 - 2 years o 3 - 5 years o 6 - 10 years o More than 10 years IMPROVING PILOT TRAINING OUTCOMES 169 Q2 How much can you influence decisions related to curriculum in the Professional Flight Pro- gram? o None at all o A little o Moderate amount o A lot o A great deal Q3 If you wanted to asses a student’s knowledge of a particular topic, which of the following tools would you most prefer to use? o Open-ended written examination o Pre-assessments, journaling, and demonstration o Standardized or open-ended verbal examinations o Written multiple choice examinations Q4 Which of the following do you believe is most important to increase a student’s retention of what they have been taught in previous lessons? o Well rested and nourished before class sessions o Frequent repetition of previous lessons o Journaling and reflecting on previous experiences o Relate topics to individual learning styles (visual, auditory, kinesthetic) IMPROVING PILOT TRAINING OUTCOMES 170 Q5 I believe instructors should make a priority of finding new and innovative ways to help stu- dents develop a mastery focus (a focus on constant learning and improvement) of the rules of aviation. o Strongly disagree o Disagree o Somewhat disagree o Somewhat agree o Agree o Strongly agree Q6 Please indicate where you think a teacher’s priority should be on this continuum by sliding the bar to the value that corresponds with your beliefs Apply What I Already Know How to Do Learn and Apply Innovative Ap- proaches 0 10 20 30 40 50 60 70 80 90 100 Slide the bar to reflect your belief IMPROVING PILOT TRAINING OUTCOMES 171 Q7 My students would benefit from learning how to enhance their self-perception of their capa- bilities, motivation, and ability to influence events affecting their lives. o Strongly disagree o Disagree o Somewhat disagree o Somewhat agree o Agree o Strongly agree Q8 I believe it is important to attribute student success or failure in part to a teacher’s skill as an instructor. o Strongly disagree o Disagree o Somewhat disagree o Somewhat agree o Agree o Strongly agree IMPROVING PILOT TRAINING OUTCOMES 172 Q9 Teaching strategies designed to increase student confidence in their ability to learn aviation- specific concepts will increase their performance in flight. o Strongly disagree o Disagree o Somewhat disagree o Somewhat agree o Agree o Strongly agree Q10 Using the scale bar below, please rate your confidence in accomplishing the following tasks: Cannot Do At All Moderately Can Do Highly Certain Can Do 0 10 20 30 40 50 60 70 80 90 100 IMPROVING PILOT TRAINING OUTCOMES 173 Keep students on task during difficult as- signments Increase a students' understanding of what they have been taught in previous lessons Get students to believe they can do well in school work Positively influence the performance of students who are struggling Help students master difficult tasks or subjects Motivate students who are struggling Q11 To what extent is the instructor responsible for a student’s strong or weak performance? o Not at all o Rarely o Sometimes o Most of the Time o Always IMPROVING PILOT TRAINING OUTCOMES 174 Q12 I believe my academic program provides adequate education and training to faculty on prac- tices related to teaching student reflection and self-motivation. o Strongly agree o Agree o Somewhat agree o Somewhat disagree o Disagree o Strongly disagree Q13 I believe my academic program rewards innovation and continuous improvement in sylla- bus development. o Strongly agree o Agree o Somewhat agree o Somewhat disagree o Disagree o Strongly disagree Q14 To what extent does your collegiate flight program provide the resources you need to im- prove a student’s understanding of reflection and self-motivation? o Not at all o Meets some of my needs o Meets my needs o Exceeds my needs IMPROVING PILOT TRAINING OUTCOMES 175 Q15 To what extent do you believe your collegiate flight program is open to new ideas about in- struction? o Not at all o Little o Some o Substantial Q16 Please select the number of years you have worked in adult education. Years ▼ 1 (0) ... 20+ (20) Q17 Please select the number of years you have worked in the Professional Flight program. Years ▼ 1 (0) ... 20+ (20) Q18 Are you a member of the University faculty or a contract flight instructor? o University Faculty o Contractor Instructor o Neither IMPROVING PILOT TRAINING OUTCOMES 176 Q19 Please select the choice that best describes your position in this program. o Full Time Faculty o Adjunct Faculty o Part Time Faculty o Contractor Instructor o Other Q20 How much total time have you served in the active duty military, Guard, or Reserves? o None o Less than 5 years o 5-10 years o 11-15 years o 16-20 years o More than 20 years Q21 How many years have you worked in the airline industry, or in some segment of commer- cial aviation? o None o Less than 5 years o 5-10 years o 11-15 years o 16-20 years o More than 20 years IMPROVING PILOT TRAINING OUTCOMES 177 APPENDIX C Interview Introduction Before we start, I want to thank you for taking the time to meet with me today about learning behaviors in this program. I’m a doctoral student at the University of Southern Califor- nia studying supporting faculty as they apply self-efficacy and metacognitive instruction to stu- dents and how this can improve graduation rates and student performance. Your experience and responses to my questions during this interview will help [State University] improve its instruc- tion to future generations of commercial aviators. All information that you share during this interview will remain confidential and anony- mous. Your comments will not be attributed to you by name. You can choose to skip any ques- tion and may end the interview at any time. We are scheduled for one hour, but I will do my best to provide at least 10 minutes at the end for any questions you might have. I would like your permission to record our interview. This allows me to give my com- plete focus and attention to you and your responses during our conversation today. All record- ings are destroyed once the study is completed. May I record our interview? Before we begin, what questions do you have for me? IMPROVING PILOT TRAINING OUTCOMES 178 APPENDIX D Interview Protocol Knowledge 1. Would you please describe your experience with teaching students “how to learn”? 2. Can you describe your school’s goals related to the aviation training program? 3. Would you please describe the learning skills you feel are important for students to im- prove their performance? 4. What do you feel is your most effective technique for helping students when they strug- gle to learn a difficult subject? 5. How has your education and experience prepared you to provide metacognition and self- efficacy instruction to your students? Can you give me an example? 6. What strategies can you use as an instructor to increase a student’s memory of what they have been taught in previous lessons? Motivation 7. What motivates you, personally to find new or innovative ways to help students achieve a mastery orientation of aviation skills? 8. Can you provide an example of how the design of your course(s) can be improved to help teach students how to learn the content more effectively? 9. When you see students struggling with the program, what factors do you consider to ex- plain that poor performance? 10. In your opinion, how important is instruction on self-efficacy and metacognition to im- proving student graduation rates? IMPROVING PILOT TRAINING OUTCOMES 179 Organization 11. How do you think the university can better support faculty in helping students achieve subject mastery in aviation? 12. I’d like to ask your recommendations for how the university can improve the current in- structional design of the aviation training program to improve student learning. IMPROVING PILOT TRAINING OUTCOMES 180 APPENDIX E Informed Consent University of Southern California Rossier School of Education Los Angeles, CA 90089 INFORMED CONSENT FOR NON-MEDICAL RESEARCH IMPROVING STUDENT OUTCOMES IN COMMERCIAL AVIATION TRAINING THROUGH LEARNING ABOUT LEARNING: AN INNOVATION STUDY You are invited to participate in a research study conducted by Regan Patrick and Melora Sundt, PhD at the University of Southern California, because you are either a faculty member at Arizo- na State University or a flight/simulator instructor at ATP Flight School in Mesa supporting ASU’s Aviation Training Program. Your participation is voluntary. You should read the infor- mation below, and ask questions about anything you do not understand before deciding whether to participate. Please take as much time as you need to read the consent form. You may also de- cide to discuss participation with your family or friends. If you decide to participate, you will be asked to sign this form. You will be given a copy of this form. PURPOSE OF THE STUDY This research study aims to understand your perceptions and attitudes regarding the application of self-efficacy and metacognitive instruction in your curriculum design to improve student out- comes, measured by an increase in the quality and quantity of program graduates. Findings from IMPROVING PILOT TRAINING OUTCOMES 181 this study will contribute to the growing body of literature on improving instruction in pilot train- ing programs. STUDY PROCEDURES If you volunteer to participate in this study, you will be asked to complete an online survey, which is anticipated to take about 15 minutes. You do not have to answer any questions you don’t want to, and you may click “next” or “N/A” in the survey to move to the next question. POTENTIAL RISKS AND DISCOMFORTS There are no anticipated risks associated with this research study. POTENTIAL BENEFITS TO PARTICIPANTS AND/OR TO SOCIETY This study is being conducted to expand understanding of optimized instructional design in avia- tion training programs – specifically, the application of self-efficacy and metacognition instruc- tion. Findings from this study can be used to improve organizational practices at university avia- tion training programs around the country to help address an anticipated domestic commercial pilot shortage. CONFIDENTIALITY We will keep your records for this study confidential as far as permitted by law. However, if we are required to do so by law, we will disclose confidential information about you. The members of the research team and the University of Southern California’s Human Subjects Protection IMPROVING PILOT TRAINING OUTCOMES 182 Program (HSPP) may access the data. The HSPP reviews and monitors research studies to pro- tect the rights and welfare of research subjects. Any identifiable information obtained in connection with this study will remain confidential. Your responses will be coded with a false name (pseudonym) and maintained separately. The data will be stored on the researcher’s password-protected computer and will be kept until the study is completed, at which time it will be destroyed. When the results of the research are pub- lished or discussed in conferences, no identifiable information will be used. PARTICIPATION AND WITHDRAWAL Your participation is voluntary. Your refusal to participate will involve no penalty or loss of ben- efits to which you are otherwise entitled. You may withdraw your consent at any time and dis- continue participation without penalty. You are not waiving any legal claims, rights or remedies because of your participation in this research study. INVESTIGATOR’S CONTACT INFORMATION If you have any questions or concerns about the research, please feel free to contact Principal Investigator Regan Patrick via email at rjpatric@usc.edu or phone at (505) 206-4748 or Disserta- tion Chair Dr. Melora Sundt at sundt@usc.edu or (310) 403-6671. RIGHTS OF RESEARCH PARTICIPANT – IRB CONTACT INFORMATION If you have questions, concerns, or complaints about your rights as a research participant or the research in general and are unable to contact the research team, or if you want to talk to someone IMPROVING PILOT TRAINING OUTCOMES 183 independent of the research team, please contact the University Park Institutional Review Board (UPIRB), 3720 South Flower Street #301, Los Angeles, CA 90089-0702, (213) 821-5272 or upirb@usc.edu SIGNATURE OF RESEARCH PARTICIPANT I have read the information provided above. I have been given a chance to ask questions. My questions have been answered to my satisfaction, and I agree to participate in this study. I have been given a copy of this form. Name of Participant Signature of Participant Date IMPROVING PILOT TRAINING OUTCOMES 184 SIGNATURE OF INVESTIGATOR I have explained the research to the participant and answered all of his/her questions. I believe that he/she understands the information described in this document and freely consents to partic- ipate. Name of Person Obtaining Consent Signature of Person Obtaining Consent Date IMPROVING PILOT TRAINING OUTCOMES 185 APPENDIX F Observation Protocol, Classroom PRE-OBSERVATION DATA Teacher ________________________________ Date __________________ School ________________________________ Grade/Level _____________ Observer ______________________________ Program _________________ 1. Class period or time of class: 2. Topic or topics: 3. Placement of class or lesson within the unit of study: 4. Purpose (objectives): 5. Intended outcomes: IMPROVING PILOT TRAINING OUTCOMES 186 6. How students will be assessed (for this lesson): IMPROVING PILOT TRAINING OUTCOMES 187 CLASSROOM ACTIVITIES Start Time: __________ Duration ______________ 1. Description of the classroom: (Describe how the seating is arranged, number and kind of windows and lights, any special equipment or materials. Describe what is on the walls, especially bulletin board displays. Give an overall general description of the size of the room, e.g., ‘large’ is sufficient.) 2. Teaching aids/materials (per activity/task if appropriate): (All materials including chalkboard, overhead projector, teacher-made handouts, textbook, etc. should be listed.) IMPROVING PILOT TRAINING OUTCOMES 188 3. Assessment strategies used (per activity/task if appropriate): (If during the observation the teacher uses some form of assessment strategies, record them. For example, a teacher may circulate among students doing work in small groups and make nota- tions on a check-sheet.) 4. Nature of class activity, teacher/instructor demeanor, interactions: IMPROVING PILOT TRAINING OUTCOMES 189 REFLECTIONS AND INTERPRETATIONS 1. Impressions of Teacher support for student Self-Efficacy: (A person’s belief about their capabilities to produce designated levels of performance and exer- cise influence over events that affect their lives. It determines how they feel, think, motivate themselves, and behave through cognitive, motivational, affective and selection processes.) 2. Impressions of Teacher support for Metacognition: (The processes a person uses to plan, monitor, and assess their understanding and performance, including a critical awareness of one’s thinking and learning and oneself as a thinker and learn- er.) IMPROVING PILOT TRAINING OUTCOMES 190 APPENDIX G Observation Protocol, Simulator/Flight PRE-OBSERVATION DATA Instructor/Evaluator _______________________________ Date ___________ School ________________________________ Grade/Level _______________ Observer ______________________________ Program __________________ 1. Class period or time of class: 2. Topic or topics: 3. Placement of class or lesson within the unit of study: 4. Purpose (objectives): 5. Intended outcomes: IMPROVING PILOT TRAINING OUTCOMES 191 6. How students will be assessed (for this lesson): IMPROVING PILOT TRAINING OUTCOMES 192 SIMULATOR/FLIGHT ACTIVITIES Start Time: __________ Duration ______________ 1. Description of the learning environment (sim/aircraft): (Describe device/aircraft type, number and kind of windows and lights, any special equipment or materials. Describe what is on the walls, especially bulletin board displays.) 2. Teaching aids/materials (per activity/task if appropriate): (All materials including chalkboard, overhead projector, teacher-made handouts, checklists, in flight guides, textbook, etc. should be listed.) IMPROVING PILOT TRAINING OUTCOMES 193 3. Assessment strategies used (per activity/task if appropriate): (If during the observation the instructor/evaluator uses some form of assessment strategies, rec- ord them. If observation is during an evaluation, obtain a copy of the evaluation form.) 4. Nature of class activity, teacher/instructor demeanor, interactions: IMPROVING PILOT TRAINING OUTCOMES 194 REFLECTIONS AND INTERPRETATIONS 1. Impressions of Instructor/Evaluator support for student Self-Efficacy: (A person’s belief about their capabilities to produce designated levels of performance and exer- cise influence over events that affect their lives. It determines how they feel, think, motivate themselves, and behave through cognitive, motivational, affective and selection processes.) 2. Impressions of Instructor/Evaluator support for Metacognition: (The processes a person uses to plan, monitor, and assess their understanding and performance, including a critical awareness of one’s thinking and learning and oneself as a thinker and learn- er.) IMPROVING PILOT TRAINING OUTCOMES 195 APPENDIX H Documents and Artifacts Collection Protocol Document Source Date Requested Date Received Course Syllabi, Academics Course Lead Course Syllabi, Simulator/Flight Course Lead Instructor Guides, Academics Course Lead Instructor Guides, Simulator/Flight Course Lead Lesson Plans, Academics Course Lead Lesson Plans, Simulator/Flight Course Lead Texts Course Lead Faculty PD/Training Pro- gram Information Program Administra- tion IMPROVING PILOT TRAINING OUTCOMES 196 APPENDIX I Level 1 and Level 2 Evaluation Instrument Please complete the following survey to help us assess the success of this training course and to improve future development sessions. Your feedback is important to make sure the organization is achieving its desired goals. Thank you! 1. Following this training session, I understand how self-efficacy strategies are important ele- ments of improving my student performance Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 2. Following this training session, I understand how metacognition strategies are important el- ements of improving my student performance Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 3. My learning was enhanced by the knowledge of the instructor Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 4. The practical examples in this training program can be applied to my own course planning Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 5. I can apply the concepts in this training program and the job aids provided to improve learn- er outcomes in my courses. Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 6. Following this training, I have a better understanding of our program’s goals. IMPROVING PILOT TRAINING OUTCOMES 197 Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 7. Following this training, I have a better understanding of how I can contribute to meeting our program’s goals. Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 8. I understand the steps required to structure learning experiences to increase a student's’ self- efficacy abilities in my program. Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 9. I understand the steps required to structure learning experiences to increase a student's’ met- acognition abilities in my program. Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 10. I understand how to evaluate individual teaching relative to student performance. Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 11. Following this training session, I understand how to evaluate individual teaching relative to student performance Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree IMPROVING PILOT TRAINING OUTCOMES 198 APPENDIX J Level 1, 2, 3, and 4 Blended Evaluation Model 1. I am able to provide self-efficacy strategies to my students to improve their learning out- comes (L4) Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 2. I am able to provide metacognitive strategies to my students to improve their learning out- comes (L4) Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 3. I am able to apply the course content I learned in designing and conducting my classes (L3). Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree 4. What additional help or support could increase your success in applying what you learned? (L3) IMPROVING PILOT TRAINING OUTCOMES 199 5. I was able to apply what I learned from job aids and PD sessions to improve my student’s performance in class or in the air (L2) Strongly Disagree Strongly Disagree Strongly Dis- agree Strongly Dis- agree Strongly Disagree Strongly Disa- gree 6. Which job aids supported your effectiveness in applying the skills you learned? (L2) 7. I was able to apply what I learned from job aids and PD sessions to improve my student’s performance in class or in the air (L2) Strongly Disagree Strongly Disagree Strongly Dis- agree Strongly Dis- agree Strongly Disagree Strongly Disa- gree 8. What I learned from job aids and PD sessions has been valuable in improving my curricu- lum (L1) Strongly Disagree Strongly Disagree Strongly Dis- agree Strongly Dis- agree Strongly Disagree Strongly Disa- gree 9. What information from the training session was most relevant to you? (L1) IMPROVING PILOT TRAINING OUTCOMES 200 10. What information from the training session was least relevant to you? (L1) 11. What would increase the training session’s relevance for you at your school? (L1)

Abstract (if available)

Abstract Numerous forecasting studies indicate that commercial aviation will soon face a shortage of qualified pilots. Research is showing that university flight training programs can increase student knowledge, task performance, and graduation rates to help fill this performance gap by improving student metacognition and enhancing their sense of self-efficacy. Central to the improvement of student outcomes is the aviation faculty member. This study applies an innovation model to examine the knowledge, motivation, and organizational needs required for faculty to improve student self-efficacy and metacognition, and provides recommended solutions to those needs. This convergent parallel mixed-methods case study, utilizing descriptive surveys, interviews, observations, and artifact gathering examined faculty knowledge and use of self-efficacy and metacognitive behavioral strategies with their students to improve performance and graduation rates. The stakeholder of focus for this study is the faculty at State University (pseudonym), one of approximately 100 university flight training programs accredited by the Aviation Accreditation Board International (AABI). The guiding methodological framework for this study is Clark and Estes’ (2008) performance gap analysis model, a systematic, analytical problem-solving process for identifying causes of performance gaps, clarifying organizational goals, and recommending improvements. Among the study’s findings, four knowledge influences and one organizational influence were identified as affecting current practice. Recommendations to address the problems of practice were assembled using the Kirkpatrick and Kirkpatrick (2016) New World Model as the bases for an evaluation strategy.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

A gap analysis of course directors’ effective implementation of technology-enriched course designs: An innovation study

PDF

Developmental education pathway success: a study on the intersection of adjunct faculty and teaching metacognition

PDF

Increasing workplace training transfer

PDF

Strategic support for philanthropic fundraising: a needs analysis of development officers in higher education

PDF

Inclusionary practices of leaders in a biotechnology company: a gap analysis innovation study

PDF

Opening interpretations: visual literacy and teaching for inclusion, first-year studio faculty innovate for improvement

PDF

Graduation rates in college of students with disabilities: an innovation study

PDF

Using mastery learning to address gender inequities in the self-efficacy of high school students in math-intensive STEM subjects: an evaluation study

PDF

Sustained mentoring of early childhood education teachers: an innovation study

PDF

Supporting faculty in preparing entrepreneurship: an exploration in the context of active learning

PDF

Examining the faculty implementation of intermediate algebra for statistics: An evaluation study

PDF

Efficacy of non-formal education programs in educational outcomes of marginalized Filipino children: an evaluation study

PDF

Raising cultural self-efficacy among faculty and staff of a private native Hawaiian school system

PDF

Civic learning program policy compliance by a state department of higher education: an evaluation study

PDF

College and career readiness through independent study: an innovation study

PDF

Modern corporate learning requires a modern design methodology: an innovation study

PDF

Violence experienced by registered nurses working in hospitals: an evaluation study

PDF

An examination of factors that affect online higher education faculty preparedness

PDF

The continuous failure of Continuous Improvement: the challenge of implementing Continuous Improvement in low income schools

PDF

Knowledge, motivation, and organizational influences within leadership development: a study of a business unit in a prominent technology company

Asset Metadata

Creator Patrick, Regan John (author)

Core Title Improving pilot training by learning about learning: an innovation study

School Rossier School of Education

Degree Doctor of Education

Degree Program Organizational Change and Leadership (On Line)

Publication Date 03/05/2018

Defense Date 05/01/2018

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag Aviation,aviation faculty,behavioral learning approaches,behavioral learning strategies,collegiate aviation,faculty development,flight training,instructional design,metacognition,OAI-PMH Harvest,performance goals,pilot shortage,pilot training,professional development,resilience,self-efficacy,self-regulation

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Sundt, Melora (committee chair), Datta, Monique (committee member), Niemczyk, Mary (committee member)

Creator Email cre3tx@gmail.com,rjpatric@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c40-479114

Unique identifier UC11267345

Identifier etd-PatrickReg-6071.pdf (filename),usctheses-c40-479114 (legacy record id)

Legacy Identifier etd-PatrickReg-6071.pdf

Dmrecord 479114

Document Type Dissertation

Rights Patrick, Regan John

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA