University of Southern California Dissertations and Theses

Exploring the effectiveness of continuing medical education for physicians enrolled in an MBA program

(USC Thesis Other)

Exploring the effectiveness of continuing medical education for physicians enrolled in an MBA program

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Exploring the Effectiveness of Continuing Medical Education for Physicians Enrolled in an MBA Program by Jessica L. Walter Rossier School of Education University of Southern California A Dissertation Submitted to the faculty in partial fulfillment of the requirements for the degree Doctor of Education May, 2021 © Copyright by Jessica L. Walter 2021 All Rights Reserved The Committee for Jessica L. Walter certifies the approval of this Dissertation Alison Keller Muraszewski Courtney Lynn Malloy Bryant Adibe, Committee Chair Rossier School of Education University of Southern California 2021 iv Abstract Program evaluation to ensure the transfer of knowledge to the workplace is critical, especially in the understudied area of developing physicians’ leadership capacity. Previous research points toward myriad factors influencing physician learning. These consist of choices made by individual physicians and their motivations, educational practices, and the physicians’ workplaces. This study sought to understand how to improve transfer of knowledge for a CME- bearing, graduate-level course. Physician-students who completed the health systems course in the previous 12 months and claimed CME were interviewed using a qualitative approach. Of key interest were what physicians remembered about the major concepts, their motivation for creating change in their organizations, and the barriers and facilitators they encountered while working toward their initiatives. This study found that physicians recalled the basics of the key concepts. Furthermore, physicians identified themselves as being self-efficacious in using this knowledge to make system-level changes, expressing beliefs about the utility of their recommendations and the morality of working toward the aims. Physicians reported success in making changes closest to their practice with increased challenges as goals became broader and involved additional stakeholders. They identified personal and workplace factors that influenced their success. Creating alignments and alliances for the change and strong communication skills supported progress, where lack of knowledge and experience limited progress. Within the workplace, supportive leadership, interdisciplinary teams, and alignment with organizational goals and strategy promoted progress. Competing priorities, changes in strategy, and external mitigating forces impeded progress. The findings will be used toward program improvement. Keywords: continuing medical education, transfer of learning, program evaluation, self- efficacy, motivation, organizational change, leadership development, qualitative research. v Acknowledgements The dissertation is the culminating document of an individual’s formal education, yet this dissertation would not have been possible without the support of faculty, peers, family, and colleagues. To my dissertation committee, I have great appreciation for your patience with my questions and evolving ideas: Dr. Alison Muraszewski, who taught our first course in the program and inspired me to be a better writer and teacher; Dr. Courtney Malloy, who improved my inquiry process and dispelled the mystery of qualitative research; Dr. Bryant Adibe, my committee chair, who instilled the importance of a confident step, one after another; and Dr. Marc Pritchard, whose feedback and support helped shape this document and propel it forward. I never dreamed of the amazing peers I would meet and friendships that would form. First to our reading group, formed in that first term walking across the campus, without whom there would have been fewer laughs as we pulled and prodded each other along. Second, the various pods of group chats that brought wisdom and motivation, keeping the finish line in sight. Laura Cardinal and Jenn Wells, Tom Hurtado and Jonathan Eldredge, and Danette Nelson and Lisa Bagby, thank you for your inspiration. I have learned much from you all. Many supported these efforts outside of the classroom. First and foremost, for my husband and parents, who supported this journey in numerous ways both in terms of giving and tolerating my absences (and absent-mindedness). Second, to my colleagues at work who never complained when I passed along (too many) readings and who cheered from the sidelines. A special note to Christine Flores who planted this seed nearly five years ago with the simple question, “Have you ever thought about applying for CME credit for your courses?” Of course, this would not have been possible without the generosity of the participants who took the time to share their experiences and ideas as we work toward our futures. vi Table of Contents Abstract .......................................................................................................................................... iv Acknowledgements ......................................................................................................................... v List of Tables ................................................................................................................................ viii List of Figures ................................................................................................................................ ix Chapter One: Overview of the Study .............................................................................................. 1 Context and Background of the Problem ................................................................................... 1 Purpose of the Project and Research Questions ......................................................................... 2 Importance of the Study ............................................................................................................. 3 Overview of Theoretical Framework and Methodology ............................................................ 4 Definition of Terms .................................................................................................................... 5 Organization of the Study ........................................................................................................... 8 Chapter Two: Literature Review ..................................................................................................... 9 Acquisition of Knowledge .......................................................................................................... 9 Motivational Factors ................................................................................................................. 19 Progress Toward Goals ............................................................................................................. 28 Conceptual Framework ............................................................................................................ 32 Summary ................................................................................................................................... 35 Chapter Three: Methodology ........................................................................................................ 37 Research Questions .................................................................................................................. 37 Overview of Design .................................................................................................................. 37 Research Setting ....................................................................................................................... 39 The Researcher ......................................................................................................................... 39 Data Sources ............................................................................................................................. 40 Validity and Reliability ............................................................................................................ 46 vii Ethics ........................................................................................................................................ 47 Chapter Four: Results and Findings .............................................................................................. 50 Participants ............................................................................................................................... 50 Research Question 1: What do Physicians Remember as the Key Learning from the CME Activity? ................................................................................................................................... 54 Research Question 2: What Are the Physicians’ Motivations to Identify and Make Recommendations for System-Level Areas of Improvement? ................................................ 64 Research Question 3: What are the Barriers and Facilitators Physicians Perceive to Implementing Change? ............................................................................................................. 78 Chapter Five: Recommendations and Discussion ......................................................................... 98 Discussion of Findings and Results .......................................................................................... 98 Recommendations for Practice ............................................................................................... 101 Limitations and Delimitations ................................................................................................ 107 Recommendations for Future Research .................................................................................. 109 Conclusion .............................................................................................................................. 110 References ................................................................................................................................... 113 Appendix A: Interview Questions by Concept ........................................................................... 131 Appendix B: Interview Protocol ................................................................................................. 134 Appendix D: Information Sheet .................................................................................................. 139 Appendix E: Invitation Email ..................................................................................................... 141 viii List of Tables Table 1 Data Sources 38 Table 2 Summary of Physician Demographics 51 Table 3 Reported Self-efficacy Scores for Applying the Quadruple Aim Framework 72 Table 4 Reported Self-efficacy Scores for Applying the Gap Analysis Framework 73 Table 5 Summary of Internal Factors 80 Table 6 Summary of Organizational Factors 87 Appendix A: Interview Questions by Concept 131 ix List of Figures Figure 1 Motivation Framework 20 Figure 2 Factors Influencing Effectiveness of CME 33 1 Chapter One: Overview of the Study Continuing medical education (CME) lacks efficacy in producing substantive changes in physician practices. In 2018, CME activities generated $2.8B of income and provided 1.2M hours of instruction (Accreditation Council for Continuing Medical Education [ACCME], 2019). However, participation in these activities does not lead to consistent change in physician practice (Légaré et al., 2015, 2017; Sargeant, 2009; Schostak et al., 2010). If the purpose of education is to learn and the purpose of learning is to increase knowledge to behave in ways that achieve goals (Alexander et al., 2009), then the research suggests a gap between knowledge acquisition and behavior change. Previous research describes barriers to learning which are internal and external to the physician-learner. Internal factors include selection of CME (Atwood et al., 2016; Cook, Price, et al., 2017; Schroedl et al., 2020), retention of knowledge (Flett et al., 2018; Gooding et al., 2017), and motivation to adapt behaviors (Luconi et al., 2019; Williams et al., 2014, 2015). External factors include education providers not fully availing themselves of modern learning practices (Davis & McMahon, 2018; Légaré et al., 2017; Ramani et al., 2019; Sargeant, 2009; Stevenson & Moore, 2018) and lack of support within the workplace (Haley et al., 2012; Jeong et al., 2018; Schostak et al., 2010). This study examines physician response to a specific CME activity. An extensive search of the literature reveals that the intersection of graduate education, CME, and organizational change has yet to be explored. Context and Background of the Problem Since 2017, the Department of Medical Management Education (DMME) at Cascadia Health Sciences University (CHSU), both pseudonyms, has awarded CME credit for their graduate course in healthcare systems. This is the first course students complete in the Master of Business Administration (MBA). The DMME is unique in the region as the only management 2 and leadership graduate program housed within a School of Medicine. The focus on management and leadership within the healthcare delivery environment supports the notion that well organized, financed, and managed organizations are more likely to have improved clinical and population health outcomes (Stevenson & Moore, 2018). Furthermore, there is little research about using CME to build leadership capacity and systems-thinking (Allen et al., 2019; McMahon, 2017; Stevenson & Moore, 2018). While DMME assesses learning for academic purposes, no assessment has occurred as to the application of academic learning to the workplace or the extent to which the course influences physicians’ beliefs about their ability to effect organizational change. Purpose of the Project and Research Questions The purpose of this research was to explore how to increase the transfer of knowledge in physician-students who earn CME for their graduate coursework. Transfer of knowledge, in this context, refers to applying learning within the workplace (Ambrose et al., 2010). Physicians operate in a complex system, influenced by internal and external factors (Coiera, 2011; Lavoie et al., 2017; Mazmanian & Mazmanian, 1999). This study explored how personal factors such as retention of knowledge acquisition and motivation interact with the environmental considerations (both the educational environment and workplace) to encourage behavior change. Results of this project will be used for programmatic improvements. This study was guided by the following three questions: 1. What do physicians remember as the key learning from the CME activity? 2. What are the physicians’ motivations to identify and make recommendations for system-level areas of improvement? 3. What are the barriers and facilitators physicians perceive to implementing change? 3 Importance of the Study Preparing physicians to lead complex health systems has the potential to improve healthcare delivery. Physician leadership has varied dramatically over the years. In 1935, 35% of hospitals were led by physicians, decreasing to 4% in 2008 (Gunderman & Kanter, 2009). This has increased slightly with 5% of hospital leadership in the U.S. in 2014 (Angood & Birk, 2014). Yet evidence suggests that physician-led hospitals are associated with higher quality rankings (Goodall, 2011). In response, hospitals are seeking to increase the number of physician executives (Robeznieks, 2014). Hospitals and health systems are complex organizations (Coiera, 2011; Lavoie et al., 2017; Stoller et al., 2016) that require good management practices to achieve their goals (Bloom et al., 2014; Stevenson & Moore, 2018). Previous research suggests that successful management and leadership positively impact hospital outcomes (Bloom et al., 2012, 2014; Gupta, 2019), as well as increasing employee satisfaction and decreasing employee burnout (Shanafelt et al., 2015). However, physicians often lack the education to navigate these complexities (Angood & Birk, 2014; Schwartz & Pogge, 2000). Physicians need preparation to assume these responsibilities (Steinert et al., 2012). A study of plastic surgeon program directors found that nearly 90% of those surveyed viewed business and leadership education as important for residents (Zarrabi et al., 2017). Physicians are increasingly seeking additional education to prepare themselves for leadership (Turner et al., 2018). The mission of the DMME is to develop passionate healthcare leaders (Department of Medical Management Education, n.d.). Between 2011 and 2019, the number of physicians enrolling in the MBA program quadrupled and currently comprises 40% of the cohort admitted in fall 2019. Validating the efficacy of education and identifying areas of improvement are mission-centric. This is compounded by external recognition: Physicians who complete the 4 healthcare systems course were eligible for up to 44 hours of AMA PRA Category 1 Credit™. This comprises 44% to 100% of the requisite bi-annual CME for physicians, depending on state of licensure (Federation of State Medical Boards, 2019). Given this substantial weight, it is important to ensure that the education is impactful. The current study explores the effectiveness of an educational offering that teaches a systems-perspective and focuses on application to the workplace, a gap in the current literature (Allen et al., 2019; McMahon, 2017; Stevenson & Moore, 2018). This study examines knowledge retention, motivation, and progress toward goals (including successes and challenges), which furthers curriculum improvement. Failing to study the application of learning is a disservice to the students and the larger healthcare community. Overview of Theoretical Framework and Methodology Bandura’s social-cognitive learning theory (SCT) (Bandura, 1977, 1989, 2005) explores the interactions between the person, their environment, and their behavior, thus embracing the complexity of learning (Alexander et al., 2009). This allows for evaluation of the multi- influencing interactions at play when exploring the efficacy of CME. SCT predicts behavior where choice is difficult (Williams et al., 2014) and the person exists within a complex system (Atwood et al., 2016). This study explores the specific concepts of person, behavior, and environment within the social-cognitive model. Within the construct of “person,” the study explores physicians’ knowledge retained from the CME activity and their motivation to effect change in their workplace. Behavior is operationalized as progress toward goals set at the completion of the course. Finally, the study queries physicians’ perceptions about barriers and facilitators to implementation of recommendations, including factors within their workplace and the education environment. 5 This qualitative study targets nine physicians who completed the healthcare systems course as part of their MBA degree offered by the DMME. While based in a conceptual framework, qualitative research allows for an inductive process, which informs the model and, thus, program improvement (Creswell & Creswell, 2018; Merriam & Tisdell, 2016). The study uses criterion sampling, a form of non-probability sampling (Merriam & Tisdell, 2016), as this targets only physicians who claimed CME credit for the healthcare systems course in fall 2019. All possible participants receive an invitation to participate in a 90-minute interview with the researcher. The interview consists of structured and semi-structured questions, allowing for comparability across participants as well as exploration of themes (Patton, 2002). Questions derive from a combination of validated measures (Bandura, 2006; Légaré et al., 2014) and the research questions. Resulting data is analyzed for themes and compared to the conceptual model, ultimately resulting in concrete recommendations for program improvement. Definition of Terms The following definitions for words and terms provide a universal understanding for their use throughout this study. Citations offer additional insights and origins for the defined terms. • ADKAR is an organizational change model (Prosci, n.d.). It is comprised of the following five aspects: (a) awareness, (b) desire, (c) knowledge), (d) ability, and (e) reinforcement. • AMA PRA Category 1 Credit™ is defined as A specific type of CME credit that physicians earn by participating in certified activities sponsored by CME providers accredited by either the ACCME or an ACCME-recognized State/Territory Medical Society; by participating in activities recognized by the AMA as valid educational activities and awarded directly by the AMA; and by participating in certain international activities recognized by the 6 AMA through its International Conference Recognition Program. (American Medical Association & Accreditation Council for Continuing Medication Education, 2017, p. 3) • Commitment to Change (CTC) is the behavioral manifestation of an intention to change consisting of three parts: (a) identify the specific, intended change; (b) designate the level of commitment to that goal; and (c) reflect about goal attainment at a specified point in the future (Mazmanian & Mazmanian, 1999). • Continuing Medical Education (CME) is the third component of physician education, coming after undergraduate medical education and graduate medical education (i.e., residency, fellowship). The purpose of CME is for physicians to close professional practice gaps (Stevenson & Moore, 2018). • Environmental Considerations within the context of this study include the impact of the education environment (teaching and assessment methods) and the professional environment (e.g., resources, social support, and structures) (Bandura, 2005). • Expectancy-Value Theory (EVT) identifies four key values, subjectively defined: (a) attainment value, the satisfaction of doing well in a task; (b) intrinsic value, the joy of engaging in the activity; (c) utility value, how this activity assists in achieving larger goals; and (d) cost value, that the cost associated with the activity will not be higher than the benefit (Wigfield et al., 2017). • Intention for Change describes an internal readiness for a specified change (Ajzen, 1991). • Learning is defined as a long-term and sustained change in a person including how they view and interact with the world. Learning exists within a multi-influencing exchange 7 between the learner and their environment within a place and time (Alexander et al., 2009). • Moral Norms are obligations (Luconi et al., 2019) that guide behavior separate from the values set forth by EVT. They are important actions regardless of cost or utility (Krones et al., 2010). • Motivation is a multi-faceted, internal construct that affects an individual’s choice, persistence, and expended effort (Wigfield et al., 2017). • Professional Practice Gap is cognitive and/or behavioral differences between current state and desired future state (Stevenson & Moore, 2018). • Quadruple Aim is a four-prong model assessing and improving healthcare delivery at a systems level. This builds on the three prongs of the Triple Aim (access, cost, and quality of care) (Berwick et al., 2008) with the addition of provider wellness, recognizing that for providers to take care of others, they must be viewed as integral to the system (Bodenheimer & Sinsky, 2014). • Self-Efficacy is the belief in one’s capacity to achieve a specific result (Bandura, 2005). • Social Cognitive Theory (SCT) is a tritactic approach to explain human learning and behavior (Bandura, 1986). This model explores the connections among the person’s internal state (e.g., knowledge, motivation), their behavior, and their environment. • SMART is an acronym for effective goal setting. S represented specific, M is measurable, A is achievable, R is realistic, and T is time-bound (Reed et al., 2012). • Theory of Planned Behavior (TPB) is a motivation theory created by Ajzen (1991) that describes how attitude, subject norms, and perceived control influence intention and, ultimately, behavior. 8 Organization of the Study This dissertation contains five chapters. Chapter One introduces the challenge of CME’s impact on producing behavior change in physicians, the use of social-cognitive theory as the research framework, and the study’s purpose and research questions. Chapter Two provides a review of current literature relevant to this topic, including the importance and evolution of CME and previous research into barriers and facilitators to changes in practice. Chapter Three reviews the study’s methodology. Chapter Four presents the findings and data analysis. Chapter Five provides recommendations for improvements based on the findings. 9 Chapter Two: Literature Review This literature review consists of four sections. The first section summarizes the role of the physician as an individual, exploring both the cognitive sphere of knowledge acquisition and the emotional aspects of intention to change and motivation. The second section reviews aspects of behavior change, including commitment to change, goal alignment, and perceived facilitators and barriers to change. The third section reviews the impact of the organizational environment (workplace) and learning environment on the effectiveness of CME. The final section outlines the conceptual model, based on findings from the literature, to explain factors that contribute to the effectiveness of CME. Acquisition of Knowledge An assumption in both the literature and in CME is that learning should change behavior. Alexander et al. (2009) defined learning as, a multidimensional process that results in a relatively enduring change in a person or persons, and consequently how that person or persons will perceive the world and reciprocally respond to its affordances physically, psychologically, and socially. The process of learning has as its foundation the systemic, dynamic, and interactive relation between the nature of the learner and the object of the learning as ecologically situated in a given time and place as well as over time. (p. 186) This definition aligns with the inter-influential model of SCT (Bandura, 1986) by articulating the connections within the learner as well as the mutually influencing interactions between the learner and their environment. Similar to Bandura (1986), Alexander et al. (2009) emphasize the impact learning has on behavior. Furthermore, this aligns with the goals of CME as a mechanism for physicians to keep pace with advances in the field (Vandergrift et al., 2018) through 10 maintaining and augmenting knowledge, skills, and performance (Davis et al., 2008), with the ultimate intention to improve patient outcomes (Moore et al., 2009) and population health (Stevenson & Moore, 2018). This section explores factors that influence a physician’s knowledge acquisition, including selection of CME activities and the education environment. Selection of CME Activities Most state licensing boards across the United States require physicians to participate in CME to maintain licensure. Requirements vary greatly, with most states requiring 20 to 50 hours per year with some expectations of content (e.g., opioid prescribing or opioid abuse) and accreditation (e.g., American Medical Association) (Federation of State Medical Boards, 2019). Within those requirements, physicians have latitude for selection of CME activities, for a premise of CME is that choice allows physicians to close unique practice gaps (Jeong et al., 2018; Schroedl et al., 2020). A professional practice gap is defined as cognitive and behavioral differences between current state and desired future state (Stevenson & Moore, 2018). This assumes that physicians engage in careful analysis through which they identify their professional practice gaps and that allowing physicians to tailor their requisite CME hours will result in better integration of learning (Jeong et al., 2018; Schroedl et al., 2020). However, internal and external factors mediate this assumption. This section discusses the intersection of selection of CME activities and knowledge acquisition. Mandated CME The mandate of CME increases broad clinical knowledge. Continuing medical education is a relatively new requirement, thus allowing for a natural experiment: Does required CME correlate with board exam scores? Vandergrift and colleagues (2018) found that rigorous CME requirements were associated with higher test scores. This finding was especially strong for 11 states that adopted CME requirements during the study period (2006–2013). In other words, while physicians in states with existing CME requirements continued to show growth in test scores, physicians in states that adopted CME requirements since the individual’s previous exam showed larger gains. However, this does not necessarily translate to practice (Légaré et al., 2015; Schostak et al., 2010; Stevenson & Moore, 2018). Additional research demonstrates that CME can advance specific clinical knowledge and skills. As policies continue to evolve and change, research has found CME to be effective in updating clinical knowledge and behaviors. In response to the state of New Mexico requiring content on pain management, educators created a 5-hour course that increased knowledge and some initial behavior change in attendees (Katzman et al., 2014). This concept holds true in an organizational context. When a hospital identified non-indicated use of a specific, expensive medication, the organization created CME targeting clinical decision-making for this medication. This resulted in improved quality and a reduction in costs (Vuković et al., 2015). In these two examples, a clear need was identified: a higher than national average of opioid-related overdoses and misuse of medication, respectively. In the latter example, organizational alignment of practices and support for change resulted in tangible improvements to patient health and the organization’s financial stability. Overall, CME as a general requirement increases clinical knowledge. When combined with societal or organizational goals, it can change behavior. However, physicians mostly self-select CME activities. Self-assessed Professional Practice Gaps Selecting CME based on self-assessment can enhance knowledge acquisition (Schroedl et al., 2020). Participants who voluntarily chose a CME activity targeting integrative medicine strategies demonstrated greater knowledge gains and reported more changes to practice than 12 those who were required to attend (Atwood et al., 2016). Furthermore, self-selection allows participants to gain repeated exposure to content. In exploring the effectiveness of CME on the topic of cancer survivorship, 28% of participants noted attending similar CME activities previously (Brothers et al., 2015). Researchers found that knowledge increased significantly for all, including those with prior exposure. Supporting this idea of multiple exposure and self- directed learning, participants requested additional, specific topics for further learning, suggesting that physicians selecting similar content may be doing so to increase depth of knowledge. While most research in this area is quantitative and survey-based, Young et al. (2011) conducted focus groups with academic and community physicians. This qualitative study offered the insight that physicians want more control over the content in terms of both breadth and depth in addition to time (Young et al., 2011). Physicians demonstrate a desire for autonomy and choice in their learning. This appears to have a positive impact on building clinical knowledge. Self-directed learning has weaknesses. Physician choice of CME does not guarantee closure of practice gaps. To close practice gaps, physicians must be able to make accurate assessments of their learning needs. Physicians report focusing their learning on their patient needs (Cook, Price, et al., 2017). However, studies report that physicians are ill-equipped to make accurate assessments of their own knowledge gaps (Davis et al., 2006; Schroedl et al., 2020) and are reluctant to ask others for mentorship in this area (Cook, Price, et al., 2017; Wood et al., 2019). Moreover, there is no accepted, systematic process to identify knowledge gaps (Kitto et al., 2013). This means that physicians have their own, often unrecognized, practice gaps and tend to choose sessions that do not significantly advance their practice. A large-scale survey of physicians found that their definition of learning includes engaging information that confirms existing practice (Schostak et al., 2010). 13 Selecting CME to close practice gaps is a noble goal. Before this can be reached, there must first be a reliable way to identify these gaps. Additionally, it is not just limited insight that hinders physicians in choosing CME that best addresses practices gaps. Practical Considerations Practical concerns affect selection and attendance. Physicians highlight content and quality as key considerations when selecting CME. They desire credible information (Salinas, 2016; Young et al., 2011) that is of high quality (Salinas, 2016; Wood et al., 2019) and relevant to practice (Young et al., 2011). A physician may desire high quality content that is relevant to practice; however, most professional development is conducted on the physician’s own time (Cook, Price, et al., 2017). Time and concerns about cost influence CME selection (Cook, Price, et al., 2017). Thus, physicians opt for convenience in selecting activities (Cook, Price, et al., 2017; Salinas, 2016). Wood et al. (2019) found that 75% of respondents rated convenience as a top priority in selecting CME. Selection of CME is fraught with decision points due to the distinction among desire, the ideal, and the realities of decision making. Once a physician decides upon a specific CME activity, the qualities within the education environment can serve to facilitate or inhibit learning. Education Environment’s Impact on Knowledge Acquisition Continuing medical education lags in the adoption of education theory. Research-based practices can guide educators in selecting practices to enhance learning. Yet, providers of CME have not widely applied nor researched the effectiveness of learning theories and practices (Légaré et al., 2017). Leaders in the field increasingly call for the application of learning theory in the development of CME (Davis & McMahon, 2018; Légaré et al., 2017; Ramani et al., 2019; Stevenson & Moore, 2018), and the limited available research supports their call. This section 14 explores the current literature about teaching techniques, aligning learning needs and opportunities and lack of leadership content. Teaching Techniques Most CME development and delivery relies on subject matter experts presenting to their peers, resulting in variability of quality across activities (Ratelle, Bonnes et al., 2017; Schostak et al., 2010). Research suggests that rather than relying on experts, content should be created in collaboration with experts (who may or may not be the presenters), and presenters should be selected based on their teaching effectiveness (Ratelle, Bonnes et al., 2017). Presenters were more effective when they engaged their audience in content that challenged audience knowledge and required reflection on practices considering this information. These presenters were more likely to have clinical teaching experience, use practices to enhance learning such as relevant examples and audience response systems, and present content clearly and thoughtfully (Wittich et al., 2013). In a follow-up study, Ratelle, Wittich et al. (2017) similarly found active learning techniques that required participants to compare session content with current professional practice to increase learning. The evidence supports the use of skilled educators to deliver CME. Rethinking the structure of CME activities can increase the opportunity for active learning. Knowing that time, convenience, and relevance are highly important to physicians in selecting CME, educators are employing technology to help address these concerns. In the oft- cited and classic work, Mazmanian et al. (2009) found that CME can improve clinical outcomes and that the use of multiple media, instructional techniques, and multiple exposures increases effectiveness. Research demonstrates increased learning through combining online and in-person content. Using the flipped classroom model may address concerns about time away from clinic, cost of attendance, and scheduling conflicts (Komarraju et al., 2018). Benefits of the flipped 15 classroom model include that it encourages self-directedness and allows for increased focus during the live session, and participants reported higher levels of engagement compared to other sessions (Komarraju et al., 2018). Participants who completed the online module, overall reported it to be effective and valuable. However, only 11% of attendees participated in the pre- work. Those who did not complete the pre-work cited some of the very factors this was to mitigate, namely, time and uncertainty if they would choose that session until the last minute (Komarraju et al., 2018). This finding is consistent with results from Wilkes et al. (2017). Their study found that online modules aimed at primary care physicians were evaluated as enjoyable. Moreover, the participants in the interactive-online section demonstrated significant factual learning and retention. Using technology to provide greater flexibility of when physicians complete CME and to allow for just-in-time learning may increase physician satisfaction with the experience and knowledge. Once learned, individuals must retain and recall content appropriately to be effective. The use of research-based instructional design increases retention of knowledge. In a study of retention of resuscitation skills, researchers found that both knowledge and skills decline steadily over six months without the refreshing of knowledge or opportunity to practice (Anderson et al., 2012). Research shows that the use of spaced learning can increase retention of information. Spaced learning is an instructional technique in which learners recall content multiple times over a period of time (Agarwal & Bain, 2019). This process makes use of two important psychological constructs in learning: recall and time. Research suggests improved memory formation when learners recall information through test questions (Carpenter, 2012), and this recall happens repeatedly over time (Cepeda et al., 2008). Flett et al. (2018) developed a focused, time-compressed activity consisting of 10 true-false test questions about the use of 16 antibiotics delivered over the course of one month. They found that by asking two true/false questions a day (thus delivering all questions over five days), repeating questions physicians answered incorrectly, and retiring questions after two successful responses, physicians demonstrated knowledge retention throughout six months. Gooding et al. (2017) used spaced recall to supplement a 1-hour, live, interactive lecture on the topic of screening for eating disorders. Participants received two questions per week, with questions repeating if answered incorrectly. Those who participated in the spaced learning reported screening for eating disorders at a higher rate than those who received the traditional learning, reading of print material. This research suggests delivering content that effectively engages the audience, increases active learning, and requires recall of information across time can improve knowledge acquisition. Furthermore, it is important that the content matches identified gaps. Aligning Knowledge Gaps and Activities Matching the learner with the appropriate activities challenges both education providers and learners. In 2008, Miller and colleagues advocated for highly specific education with associated knowledge and behavior outcomes, and leaders in the field have called for CME to be based on a needs assessment (Stevenson & Moore, 2018). Just as physicians struggle to identify their practice gaps (Cook, Blachman, et al., 2017), CME providers struggle to match content with audience need (Salinas, 2016). Physicians consistently rank the topic of a CME activity as highly important (e.g., Salinas, 2016; Wood et al., 2019). Beyond that, physicians vary in their preferences, with 45% preferring didactic content and 55% preferring interactive case-based activities (Salinas, 2016). They have a strong preference for individual work (69%) and choose online versus live activities at similar rates, 38% and 43%, respectively (Salinas, 2016). Research by Wood et al. (2019) further demonstrates this challenge. Their research found that physicians 17 in academic centers identified their learning needs by first reviewing their patient populations and focusing on disease-specific knowledge gaps within their current population. The study further identified that physicians prefer formal learning (e.g., online course, meeting/symposium, tumor board at hospital). However, CME activities may not be available or accessible at the time of need. Thus, 75% of participants cited convenience of activities as an important consideration. As Komarraju et al. (2018) noted in their study of flipped classroom design, many attendees did not complete the pre-work because they did not know whether or not they would attend the activity. Matching learner needs and CME activities may be challenging, but effective matching produces gains in knowledge. Education providers have applied the concept of specificity to create meaningful CME activities. Rather than create content to appeal to broad audiences, some CME providers narrowly define need and focus content for learners. For example, interactive online education offerings are proving to be effective. Using interactive, multimedia, case-based content, learners were able to customize their education in terms of breadth and depth and showed significant gains in knowledge while reporting high rates of satisfaction (Salinas, 2015). Alternatively, public policy may determine need based on regional health concerns. Katzman et al. (2014) found that a brief CME activity about opioid addition offered in response to a public health crisis and tailored to a specific geographic region resulted in gains in knowledge and initial changes in behavior. Similarly, a hospital in Serbia targeted education at ineffective medication use within specific units, which resulted in increased knowledge, improvement in medication use, increased quality of care, and financial savings (Vuković et al., 2015). Content that is both aligned with the learners’ needs and designed using research-based practices may expedite learning. Overall, this research suggests that the process of teaching and learning warrants equal attention as the 18 content. Physicians, however, are part of a complex system. Limiting CME to clinical knowledge and skills ignores the system issues that impact healthcare delivery (Kitto et al., 2013). Lack of CME to Build Leadership Skills Less is known about how CME builds leadership capacity, but research suggests that, while more challenging to study, it is an important area of physician development. Healthcare organizations are unique, complex systems that produce results based on their unique design (Coiera, 2011). Therefore, to change the results, the system must also change. As the practice of medicine becomes more complex, physicians need to be prepared to step into leadership roles and responsibilities (Steinert et al., 2012). Stevenson & Moore (2018) note that focus on management and leadership within the healthcare delivery environment supports the notion that well organized, financed, and managed organizations are more likely to have improved clinical and population health outcomes. With outcomes that are less tangible and more abstract, they are also harder to measure. Due to these complexities, research into this area is stymied (Allen et al., 2019). The use of CME to develop leadership capacity and influence the system of healthcare delivery, while deemed important, is not yet widely defined nor understood (Balmer, 2013; Stevenson & Moore, 2018). Overall, CME increases knowledge and has capacity for improvement by matching learner needs with appropriate content to close gaps and by developing learning activities that employ research-based teaching practices. Providers of CME have an opportunity to broaden offerings to include leadership and system-thinking content. In addition to cognitive aspects, motivational factors are important considerations when exploring the link between knowledge acquisition and behavior. 19 Motivational Factors Between learning (cognitive) and behavior (action) lies the emotional component of motivation: An individual must want to engage the action for it to occur. Furthermore, knowledge, or lack thereof, does not necessarily translate into action (Ajzen et al., 2011). Motivation is multi-faceted and affects an individual’s choice, persistence, and expended effort (Wigfield et al., 2017). As motivation increases, so does one’s internal readiness for a specified change (e.g., Légaré et al., 2014; Williams et al., 2014, 2015). This internal readiness is often referred to as intention, which precedes action or behavior change (Ajzen, 1991). This section summarizes current literature about motivation factors in relation to CME and education practices that advance motivation in attendees. Theory of Planned Behavior Prochaska and Velicer (1997) noted “[n]o single theory can account for all of the complexities of behavior change” (p. 41). Of the myriad theories of motivation based in SCT, the theory of planned behavior (TPB) (Ajzen, 1991) has long-standing roots in the CME literature. According to the TPB (Ajzen, 1991), the concept of intention operationalizes the concept of motivation. Intention is defined as the amount of effort an individual is willing to exert to perform a behavior (Ajzen, 1991). The theory describes three antecedents to intention: (a) attitude; (b) subjective norm; and (c) perceived control. Additional models provide greater depth of understanding. Specifically, this review draws upon Eccles’ expectancy-value theory (EVT) to explore influences of attitude (Wigfield et al., 2017) and the concept of self-efficacy (Bandura, 1977, 1986) as a key component of perceived control. Researchers in the field of CME also highlight moral norms as an important construct for healthcare providers. This section describes each of element of TPB and summarizes relevant and current literature. 20 Figure 1 Motivation Framework Note. Adapted from Ajzen (1991), Bandura (1986), Luconi et al. (2019), and Wigfield et al. (2017). Intention to Change When learners approach learning, they do so with unique levels of pre-existing intention to apply their knowledge, to change. In their seminal work Mazmanian et al. (1998) found intention to change to have a positive correlation to actual behavior change. Unsurprisingly, those who reported coming to the CME activity with an intention to change were most likely to report subsequent behavior change. Those who did not report intention to change prior to 21 attending the session, but subsequently reported an intention to change at the end, were more likely to report behavior change during follow-up. Légaré and colleagues (2017) replicated this finding: Those who made behavior changes showed higher levels of intention to change both before and after a CME activity. This demonstrates the need to include motivational factors when considering the effectiveness of CME. Motivational theories and concepts can help frame the discussion to better understand elements that increase intention to change relationships. Attitude Attitude refers to whether a person thinks an activity is worth pursuing (Ajzen, 1991). EVT further clarifies the concept of attitude by articulating common beliefs about consequences of actions (Wigfield et al., 2017). This, in turn, influences the degree to which an individual will choose a particular action, persist in their efforts, and the amount of effort exerted (Wigfield et al., 2017). The four key values linked to this theory are: (a) attainment value, the satisfaction of doing well in a task; (b) intrinsic value, the joy of engaging in the activity; (c) utility value, how this activity assists in achieving larger goals; and (d) cost value, that the cost associated with the activity will not be higher than the benefit. Most of the CME literature does not parse out attitude to this level. To complicate understanding, some researchers use the term attitude in association with confidence (Pelayo et al., 2011) or familiarity and comfort (Wittich et al., 2016). Other researchers are beginning to focus on better understanding the connection between attitude and expectations as part of TPB. Research suggests that CME can change attitude. Most studies focus on utility value and cost value. Beginning with selection of CME, Hadadgar, Changiz, Masiello, et al. (2016) found strong connection between attitude and intention of adopting eCME (online CME). Further analysis revealed high-cost value (Hadadgar, Changiz, Dehghani, et al., 2016) for participants: 22 The reduction in travel and expense contributed toward a favorable view of the eCME, which led to intention to adopt. In exploring the impact of CME, Luconi et al. (2019) specifically asked participants to rate attitude based on utility and benefit (cost value). This study demonstrated a strong connection between a favorable attitude and intention to implement quality improvement practices. Reed and colleagues (2012) found that the act of briefly reflecting on the importance of a change (utility value) increased intention to change. Connecting CME content to an individual’s values regarding benefit and cost appears to influence intention to change. According to TBP, individuals additionally internalize external expectations that influence their behavior. Subjective and Moral Norms Subjective norms influence or pressure individuals to engage in certain behavior, typically portrayed as group norms or social pressure (Ajzen, 1991; Krones et al., 2010). Within the CME literature, subjective norms do not have strong predictive ties to intention and behavior (Hadadgar, Changiz, Dehghani, et al., 2016; Hadadgar, Changiz, Masiello, et al., 2016; Krones et al., 2010; Luconi et al., 2019). On the other hand, moral norms appear to be particularly influential for physicians whose professional oath conveys a level of moralism (Parsa-Parsi, 2017). Moral norms are viewed as obligations (Luconi et al., 2019). They guide behavior separate from values set out by EVT, as these are actions physicians believe are important to do regardless of cost and utility (Krones et al., 2010). At this time, researchers consider moral norms to be related to, yet distinct from, subjective norms (Hadadgar, Changiz, Dehghani, et al., 2016; Hadadgar, Changiz, Masiello, et al., 2016; Krones et al., 2010). However, most literature discusses moral norms philosophically rather than empirically (e.g., Katrova, 2017; Welie, 23 2012). Additional internal drivers have been shown to be influential in developing an intention to change. Perceived Control and Self-Efficacy Finally, perceived control describes the assumptions an individual has about the ease or difficulty of completing the activity (Ajzen, 1991). This aspect of TPB has received the most attention in the CME literature. Perceived control is often operationalized using Bandura’s concept of self-efficacy (Ajzen, 1991), the belief in oneself to achieve a desired outcome (Bandura, 1977). Self-efficacy influences a person’s intention to change (Mazmanian et al., 1998; Williams et al., 2015). This connection to intention is stronger than past experiences (Godin et al., 2008; Williams et al., 2015), social influence (Godin et al., 2008; Luconi et al., 2019), and knowledge (Allen et al., 2019; Godin et al., 2008). This begs the question whether CME can effectively build task-specific self-efficacy as it does knowledge. Research finds that CME can build and further self-efficacy in participants. Active learning increases self-efficacy of participants. In comparing multiple offerings of lifestyle medicine education, Dacey et al. (2013) found that active learning offers significantly increased self-efficacy in participants. Additionally, the research team noted that those participants who arrived to the session reporting lowest knowledge and confidence showed the most gain in the post-session survey. Furthermore, Williams et al. (2015) found self-efficacy to be a mediating variable between acquisition of knowledge and intention to change. They found that participants with high self-efficacy regarding the topic reported higher levels of intention to apply knowledge after the session, indicating a positive correlation between self-efficacy and intention to change. Of note in this study, a physician’s personal self-efficacy bore greater influence on motivation than beliefs derived from past experiences. These finding suggests that educators and learners 24 can use CME to increase self-efficacy and, therefore, strengthen the likelihood of behavior change. Overall, utility value, cost value, and self-efficacy appear to be the most salient features in understanding how CME increases motivation. Motivation theories aid in the understanding of under which conditions knowledge acquisition leads to an intention to change and, ultimately, to new behavior. Additional research suggests that the education environment can foster motivational factors. Education Environment’s Impact on Motivation Just as research-based teaching practice can increase knowledge acquisition, alignment teaching practices can enhance motivational factors. This section reviews current findings that increase intention to change. Specifically, providers of CME can foster motivation through aligning activities with an individual’s readiness for change. During the active learning phase, educators can provide opportunities to practice. Finally, as a transition to practice, participants can develop goal statements. Aligning Motivation and Activities Earlier elements of this review focused on matching content and knowledge gaps. Similarly, researchers found that matching education activities with level of motivation proved effective in fostering change. Individuals who have high motivation to change are more likely to do so (Williams et al., 2015). The learning environment can increase the likelihood of behavior change as demonstrated by Shirazi et al. (2013). The researchers created two learning environments: knowledge-building and planning and practice. They further divided learners based on levels of readiness for change (low and high). In this experimental design, learners were assigned to one of the two environments. Learners with low readiness for change who were 25 assigned to the knowledge-building group showed greater gains in knowledge than those in the planning and practice session. Learners with high readiness for change who participated in the planning and practice group saw greater behavior change after completing the CME activity compared to those in the knowledge-building group. Providers of CME may not be able to predict their audience as clearly. In CME designed to increase skills of general practitioners in having existential conversations with their patients diagnosed with cancer, educators designed a full-day event comprised of three distinct steps (Hvidt et al., 2018). First, educators presented information to build knowledge. Second, the participants engaged in self-reflection. The third segment included training and practice. By building from cognitive to motivational factors, this activity increased knowledge, perception of the value of the conversations (attitude), and self- efficacy (perceived control) in participants within a broad audience. In follow-up interviews, participants highlighted the importance of practice, identifying this as an effective tactic to build self-efficacy. Repetition and Practice Incorporating practice into CME activities increases self-efficacy related to personal effectiveness. Walker et al. (2019) found that combining knowledge building didactic content with case studies to practice resulted in increased confidence in participants to change their practice. Furthermore, multiple exposure to content may continue to positively impact self- efficacy. Berrett-Abebe et al. (2019) found that participants who had previous exposure to the content showed greater gains in self-efficacy. Additionally, 43.5% of participants cited additional information and skills as a mechanism to continue to increase self-efficacy. This is consistent with other research that found higher self-efficacy prior to a session led to increased levels of self-efficacy and higher intention to change after the sessions (Légaré et al., 2017; Williams et 26 al., 2015). Together, these studies show that combining knowledge acquisition with practice increases participants’ self-efficacy. Moreover, it may take multiple exposures to content and practice to do so. Once a physician has the knowledge and confidence to act, committing to the change solidified intent. Commitment to Change: Individual Goal Setting Commitment to change (CTC) is the behavioral manifestation of an intention to change (Mazmanian & Mazmanian, 1999). SCT discusses the importance of cognitive rehearsal as a facilitator in acquiring and honing skills and behaviors (Bandura, 1986). Commitment to change is a specific formulation of cognitive rehearsal, moving attention from the internal, implicit realm to the external, explicit environment yet prior to actual action. In constructing a CTC, an individual completes a three-step process: (a) identify the specific, intended change; (b) designate the level of commitment to that goal; and (c) reflect about goal attainment at a specified point in the future (Mazmanian & Mazmanian, 1999). Should a goal not be reached, the learner is asked to reflect on reasons why (Mazmanian & Mazmanian, 1999). In the TPB model, CTC creates the bridge between intention and action. Since being described by Mazmanian and Mazmanian in 1999, researchers have used variations of CTC in researching the effectiveness of CME. Learners who set goals as part of their continuing education activity tend to report higher levels of knowledge application. When physicians can clearly describe what the change will look like and how it will work, they are more likely to attempt and complete their goal (Haley et al., 2012; Kogan et al., 2017). Reed et al. (2012) applied the use of SMART goals to conference- style CME and found that those who reported being “very” or “extremely” close to their goals were more likely to have set well-developed SMART goals. After three months, just over half 27 (54%) reported being close to goal attainment (Reed et al., 2012). Without the benefit of a control group, the difference between those who achieved goals compared to those who did not may be a result of higher motivation (Kogan et al., 2017; Rodriguez et al., 2010), or they may have attended the event to confirm practice rather than to close a learning gap (Schostak et al., 2010). Aghera and colleagues (2018) addressed some of these concerns through an experimental method. With a focus on medical residents, who have less choice over content, researchers found that the use of SMART goals does not impact knowledge acquisition, but does increase completion of post-educational activities within the next two weeks (Aghera et al., 2018). Interestingly, research cautions against an overreliance on goals. A study of general practitioners found that, while clear goals precipitated behavior change, an increased number of goals decreased successful implementation (Haley et al., 2012). Furthermore, physicians are less likely to complete goals perceived to be difficult. Kogan et al. (2017) noted that goals that involve cooperation of others, a specific complexity in achieving goals, are less likely to be completed. Planning for action appears to increase the likelihood that physicians make changes in their clinical practice. However, effecting change beyond the patient-physician dyad proves more challenging. Once a plan is in place, additional internal factors influence whether a physician changes their behavior, especially in regard to larger goals. The final component of CTC asks learners to reflect on why, if appropriate, they did not make their intended change. This question provides insights into what physicians perceive as barriers to their change. Bandura (1989) discusses both expected outcomes and extraneous (i.e., external) variables, which can enhance or disrupt self-regulation of behavior. Both internal beliefs and the external environment impact behavior change. The next section considers progress toward goals after attending a CME activity. 28 Progress Toward Goals Continuing medical education is built upon the assumption that the education event itself is enough to inspire changes in physician behavior (Kitto et al., 2013; Parochka & Paprockas, 2001). However, outcomes are inconsistent (Cervero & Gaines, 2015), and the literature does not support the assumption that knowledge translates to behavior change (Allen et al., 2019). In a foundational review of the literature, Bordage et al. (2009) found that 79% of CME activities resulted in some increased knowledge. Drawing upon a review of the same studies, Mazmanian et al. (2009) found that less than 40% of CME activities demonstrated long-term beneficial effects. A person’s behavior does not exist in a vacuum. When a physician returns from a CME activity, the wider environment influences perspective on newly acquired knowledge and gains in motivational factors. As a physician attempts to engage new ways of behaving, their environment provides feedback. This feedback, in accordance to SCT, thus further shapes the individual’s motivation as well as their behavior (Bandura, 1986). Physicians face organizational barriers of practice expectations and policies (Schostak et al., 2010) and complicated and dynamic political landscapes (Schroedl et al., 2020). Physicians do not operate in isolation, and internal and external forces influence a physician’s likelihood of affecting change. Internal Factors Prioritizing Change Individual physicians report a variety of internal and external reasons that limit implementation of learning activities. Initially, as established above, physicians must believe that the change is useful (Rodriguez et al., 2010), a finding consistent with SCT (Bandura, 1986, 1989, 2005). Furthermore, when physicians prioritize a change, they are more successful. Haley and colleagues (2012) found the combination of giving the activity a high priority and setting 29 aside dedicated time facilitated action. Priorities exist within a larger context. While a physician may believe that an activity or goal is important, competing priorities may hinder progress. Upon return to practice, a physician may recalibrate their priorities (Kogan et al., 2017; Rodriguez et al., 2010) or perceive lack of organizational support through allocation of time (Jeong et al., 2018). Just as physicians engage CME activities on their own time (Cook, Price, et al., 2017), physicians rely on personal time to work toward their goals (Jeong et al., 2018). When this happens, their family’s perception of work-life balance results in renegotiation of priorities (Haley et al., 2012). A physician draws upon internal beliefs about the importance of the change and is influenced by the priorities of others in their environment. Further, physicians may question their abilities upon returning to their workplace. Fear of Failure While physicians report increased self-efficacy at the completion of CME, it may not be sustained. In a six-month follow-up survey, Luconi et al. (2019) found the confidence related to CME content waned. Physicians report feeling overwhelmed when contemplating a change once they have returned to their workplace, thus reducing their likelihood of making progress toward their goal regardless of their beliefs about the importance of their goal (Haley et al., 2012). Additionally, physicians report concern about failure. This fear is defined as judgement by self and others associated with failing to reach a goal (Haley et al., 2012). Other research found physicians to be concerned with appearing less than competent when practicing new behaviors (Jeong et al., 2018). As knowledge and motivation increase during CME activities, the effects appear to be short-term. Environmental factors influence internal perceptions, which further impacts progress toward goals. 30 External Factors Organizational Resources Projects with adequate resources are more likely to be successful. However, what constitutes resources is unclear in the literature. A physician may simply not have the appropriate tools or supports. Rodriguez et al. (2010) found that some physicians reported lacking the requisite knowledge or skills to effect the change, having appropriate access or authority. Additionally, physicians who thought that their endeavor would be difficult to accomplish were less likely to make progress toward their goal (Kogan et al., 2017). While this study did not identify how physicians defined difficult, the research points to the issue of complexity. Practice expectations, policies, and required change on the part of others in the care team may impede physician behavior change (Schostak et al., 2010). Supporting this, Jeong et al. (2018) found the lack of a cohesive and reliable team (e.g., financial support, turnover, poor communication) and insufficient infrastructure impede change. Alternatively, research by Haley et al. (2012) and Kogan et al. (2017) found peer and social support for implementing change activities proved to be a positive influence on progress toward goals. Viewing goals within the larger context and planning to mitigate barriers is a popular theme in the discussion of the literature (e.g., Kogan et al., 2017; Reed et al., 2012). Organizationally sponsored CME shows some success with this. In their research about reducing non-indicated use of an expensive medicine, a Serbian hospital aligned organizational financial goals, patient safety, and quality goals (Vuković et al., 2015), resulting in improvements in all three areas. The alignment of clinical, patient, and organization goals facilitates change. Similarly, an organization’s culture impacts success of change. 31 Organizational Culture There is no single setting or culture that defines the healthcare workplace, and individuals may work across a variety of contexts, facing competing priorities. Healthcare organizations are unique, complex systems that produce results based on their unique design (Coiera, 2011). When a physician returns from a CME activity with new knowledge, they return to a complex system that may thwart their change efforts. Clinicians face not only their own clinical inertia but system inertia as well, an organization’s inability to adapt to change (Coiera, 2011; Lavoie et al., 2017). Lack of change may reflect the bias of the environment rather than the desire of the individual physician (Coiera, 2011; Lavoie et al., 2017). Research suggests incorporating an organization’s culture and setting into planning for change. Organizational culture, however, is difficult to assess and understand the specific mechanism by which it impacts behavior (Jeong et al., 2018). The environment of the physician’s workplace, including expectations, mitigates practice changes. The strongest evidence points toward collaborative environments as effective means to change behavior. In an exploration of the complexities of disease management for multiple sclerosis, researchers found that by providing an opportunity for a collaborative team to come together and develop their own project and learning, they achieved greater alignment in their treatment plans and communication (Ravyn et al., 2019). In addition to the goal alignment discussed earlier and building on the concept of collaboration, researchers found further evidence to support team learning. Focused on family medicine, a systematic review found that a collaborative team-based approach most facilitated practice changes (Chauhan et al., 2017). Notably, teams were most effective when they included physicians, nurses, and pharmacists. This supports the concept of self-directed team learning when supported and resourced 32 appropriately and aligned with organizational goals and population health (Davis & McMahon, 2018). Just as knowledge alone will not lead to behavior change in physicians, the research suggests that it takes a larger team to affect change within the healthcare organizations. Conceptual Framework Social cognitive theory (Bandura, 1986) forms the broad framework for the current study. Bandura describes this model as one of triadic reciprocity (Bandura, 1986), exploring the interactions among a person, their behavior, and their environment. At its heart, SCT is a learning theory (Bandura, 1986) that embraces the idea that the ultimate goal of learning is to change behavior (Alexander et al., 2009). This aligns with the goals of CME: to keep current with advances in science and industry as an effort to improve patient outcomes (Moore et al., 2009) and population health (Stevenson & Moore, 2018). It recognizes that knowledge is not enough to change behavior (Allen et al., 2019; Kitto et al., 2013). Motivational factors create the impetus to act (Ajzen, 1991; Williams et al., 2014, 2015). Furthermore, both knowledge acquisition and motivational factors may be influenced by the environment (Hvidt et al., 2018; Kogan et al., 2017; Rodriguez et al., 2010). Environmental factors further influence whether an intention or commitment to change results in change within the workplace (Aghera et al., 2018; Haley et al., 2012; Reed et al., 2012). This study explores the translation of learning from a graduate course in Healthcare Systems (as part of the student’s MBA program) into their workplace through the lens of SCT. This section reviews the model as applied toward the current study. 33 Figure 2 Factors Influencing Effectiveness of CME Knowledge Acquisition From within the construct of person, the current study examines the degree to which individuals retain key learning objectives from the course. Research demonstrates that CME can increase knowledge acquisition (Katzman et al., 2014; Vandergrift et al., 2018). Furthermore, the subject matter of this course, healthcare systems, teaches concepts advocated for by leaders in CME (Balmer, 2013; McMahon, 2017; Stevenson & Moore, 2018) yet not widely researched (Allen et al., 2019). Additionally, from the perspective of the environment construct, the study explores factors within the learning environment that enhance and detract from knowledge acquisition. Providers of CME can enhance knowledge acquisition and retention using research- based educational practices such as active learning (Ratelle, Bonnes, et al., 2017; Wilkes et al., 2017), spaced learning (Flett et al., 2018; Gooding et al., 2017), and combining online and in- person sessions (Komarraju et al., 2018; Wilkes et al., 2017). This combination explores the Person Knowledge aquisition Motivational factors: beliefs about consequences, self-efficacy, intention Behavior Commitment to change Progress toward goals Perceived barriers & facilitiators Environment Workplace culture & policies Learning theory 34 interplay between the learner and the learning environment. However, knowledge acquisition alone will not result in change. Motivational Factors For a physician to apply their knowledge, they must have the motivation to do so. Within the construct of person, this study explores motivational factors found to be impacted by CME activities and salient to creating intention to change (Haley et al., 2012; Kogan et al., 2017; Reed et al., 2012). The research suggests that an individual’s attitude impacts intention to change; specifically, they must believe that the action will be useful and that the benefit will outweigh the cost (Luconi et al., 2019; Wigfield et al., 2017). Additionally, the individual must be confident that they can achieve the desired outcome, to have self-efficacy (Bandura, 1986; Williams et al., 2014, 2015). Increases in these factors lead toward an intention to change (Légaré et al., 2014; Luconi et al., 2019; Williams et al., 2014, 2015). This study explores levels of motivation six months after completion of the course to assess longer-term influences to motivation. Progress Toward Goals The current study, within the construct of behavior, examines how effective participants are in making change in their organizations. Research suggests that physicians operate within a complex system (Mazmanian & Mazmanian, 1999) that is stymied with inertia (Coiera, 2011; Lavoie et al., 2017) and lacks time and resources (Haley et al., 2012; Jeong et al., 2018; Rodriguez et al., 2010). Physicians may also lack knowledge and skills to influence the larger system (Jeong et al., 2018; Rodriguez et al., 2010). Thus, within the construct of environment, this study further seeks to explore perceived barriers and facilitators as mitigating aspects of progress toward goals. This, in turn, informs needed changes to the education environment. 35 Summary Continuing medical education provides activities to keep physicians current with science, practice, and the industry (Cook, Price, et al., 2017; Schostak et al., 2010) and has capacity to increase knowledge (Schostak et al., 2010; Vandergrift et al., 2018) and improve healthcare outcomes (Katzman et al., 2014; Vuković et al., 2015). The effectiveness of CME is influenced by the tritactic relationships among the individual physician, their environment, and their behavior. Within the learning environment, CME can increase knowledge (e.g., Brothers et al., 2015; Gooding et al., 2017) and motivation (e.g., Luconi et al., 2019; Williams et al., 2014, 2014), resulting in greater likelihood of behavior change (e.g., Aghera et al., 2018; Légaré et al., 2017; Reed et al., 2012). Furthermore, a physician’s pre-existing levels of motivation (Shirazi et al., 2013) and prior exposure to the content (Atwood et al., 2016; Brothers et al., 2015) influence how they approach the CME activity. When physicians return to work, their ability to change practice depends on the workplaces’ culture and structures (Jeong et al., 2018). Collaborative teams (Chauhan et al., 2017; Ravyn et al., 2019) and alignment with organizational goals (Vuković et al., 2015) increase practice changes. However, many efforts do not result in change due to lack of resources (Haley et al., 2012; Rodriguez et al., 2010) and knowledge and skills to influence the larger system (Jeong et al., 2018; Rodriguez et al., 2010). Activities related to systems’ perspective and leadership skills are not well studied, nor are activities associated with formal education. The current research employs SCT as a framework to examine retention of acquired knowledge, motivation to change, and progress toward goals, specifically drawing out facilitators and barriers within the education environment and workplace. The study targets physicians who earned CME for Healthcare Systems course as part of the MBA degree. The 36 results will inform curriculum design and add to the body of research into the effectiveness of CME. 37 Chapter Three: Methodology The purpose of this research was to explore how to increase the transfer of knowledge in physician-students who earn CME for their graduate coursework. This chapter provides an overview of the current study’s methodology. The first section reviews the research questions. The second and third sections describe the overview of the research design and research setting, respectively. The fourth section discusses the researcher. The fifth section, data sources, outlines the participants, instruments, and data collection and analysis procedures. The sixth section reviews validity and reliability concepts applicable to the current research. Relatedly, the seventh section discusses the ethics of the study. Research Questions This study was guided by the following three research questions, derived from the conceptual model: 1. What do physicians remember as the key learning from the CME activity? 2. What are the physicians’ motivations to identify and make recommendations for system-level areas of improvement? 3. What are the barriers and facilitators physicians perceive to implementing change? Overview of Design This qualitative study sought to understand the impact of CME associated with a graduate-level management course has on physician knowledge, motivation, and behavior in addition to perceived facilitators and barriers to application of knowledge in the workplace. The researcher was the key instrument, gathering information directly from participants through the use of interviews and analyzing the resulting data for themes (Creswell & Poth, 2018). Table 1 summarizes the research questions and data sources. Participant selection occurred using a 38 criterion-based sampling, a form of nonprobability sampling where participants must meet certain criteria (Merriam & Tisdell, 2016). In this case, participants were physicians who completed a graduate course in healthcare systems as part of their MBA degree within the last 12 months and who claimed CME for the course. Each of these aspects is discussed in more detail below. Only those who met both criteria were invited to participate. Table 1 Data Sources Research Questions Interview What do physicians remember as the key learning from the CME activity? X What are the physicians’ motivations to identify and make recommendations for system-level areas of improvement? X What are the barriers and facilitators physicians perceive to implementing change? X 39 Research Setting Cascadia Health Sciences University (CHSU) is the academic health university for a state in the western United States. The Department of Management Education is an education unit within the School of Medicine and serves professionals in the healthcare industry through graduate programs in healthcare management and leadership (unduplicated headcount was approximately 200 students). Beginning in fall 2017, the Center for Professional Development (CPD) at CHSU annually approved the healthcare systems for AMA PRA Category 1 Credits™. This study targeted physicians pursuing their MBA degree in healthcare management and leadership who claimed CME for completion of the healthcare systems course within the last 12 months. The MBA admits approximately 40 students a year, totaling an average of 120 students in the MBA across the cohorts. Typically, clinicians comprise 50% of an MBA cohort (e.g., physicians, nurses, physical therapists, psychologists, dentists, pharmacists, etc.). Of the clinicians, the majority are physicians (approximately 16 per cohort) from across a wide variety of specialties. This research specifically focused on the effectiveness of CME awarded for the healthcare systems course. Findings from this study will inform program improvement. The Researcher The researcher held a leadership role within the Department and was highly visible to students. While active in assurance of learning efforts and administrative aspects of CME for the healthcare systems course, the researcher was not a faculty of record for the course and had no authority or oversight of grades for the course. Neither the researcher nor participants received compensation associated with the research. 40 Data Sources The primary data source was an interview. These interviews took place 12 months after participants completed their healthcare systems course as part of their MBA degree. The intention of a delay between completion of the course and interview was to give participants the opportunity to work toward their recommendations set at the end of the course (Mazmanian & Mazmanian, 1999; Reed et al., 2012) and verify long-term retention of learning (Anderson et al., 2012; Flett et al., 2018; Gooding et al., 2017). Furthermore, prior research shows that confidence begins to wane with the passage of time (Luconi et al., 2019), thus providing a longer-term measure of confidence. The primary data source was an interview with each participant. Interviews allow the researcher access to inner thoughts, feelings, and experiences that may not otherwise be observable nor captured through standardized surveys (Weiss, 1994). The interview contained a mix of structured and semi-structured interview questions (Merriam & Tisdell, 2016), and this combined format allowed for the use of their unique strengths (Patton, 2002). The structured aspects of the interview allowed for assessment of concepts such as self-reported self-efficacy scores, while the semi-structured portion allowed for greater use of prompts and follow-up questions to better understand a person’s experiences and perceptions. The interview protocol consisted of 19 questions, combining open and closed-ended questions. Six closed questions were scale-based questions assessing concepts such as self- efficacy and intention to change. The open-ended questions allowed for more rich data to be collected (Robinson & Leonard, 2018). At the end of the interview, basic demographic data relating to professional practice was collected. The researcher piloted the initial interview questions with two individuals who completed the course but who did not claim CME. Based on 41 these pilot interviews, interviews were scheduled for 90 minutes, allowing time for consenting and closure, building rapport, and allowing for fuller answers to the questions. This is in accordance with guidance from Weiss (1994), who suggests that interviews of no more than 90 minutes can be completed in a single sitting. Interviews ranged from 56 minutes to 73 minutes. Participants This research used criterion sampling (Merriam & Tisdell, 2016), otherwise known as purposeful sampling (Maxwell, 2013). This form of non-probability sampling matches possible participants based on specific criteria as reflected in the research questions. The interviews targeted physicians who claimed CME for a graduate course in healthcare systems in fall 2019. This population of nine students (out of the 34 enrolled students) represented those who were both physicians and requested CME as part of this course. The remaining 25 students included physicians who did not request CME for the course, other clinicians (e.g., nurses, social workers, dentists) and non-clinicians (e.g., individual contributors, managers, administrators, etc.). While any student may request CME credit for this course, the criterion to focus exclusively on physicians was to explore the intersection of the lack of CME for developing leadership skills in physicians specifically (e.g., Kitto et al., 2013), the education innovation of offering CME for MBA course work, and the unique position physicians hold within healthcare delivery. Furthermore, physicians who did not request CME were excluded because requesting CME brings an inherent assumption that one will take their learning back to their place of practice which may influence expectation and motivation. All possible participants who met the definition of the study population were invited to participate. The invitations described the importance of the project to improve the learning experience and best support students in connecting learning to application. The CPD and the 42 education coordinator within the DMME sent emails to students who met the criteria. A total of six email invitations were sent over a period of four weeks. Participants scheduled their time using an online scheduler (e.g., YouCanBook.me). Of the nine possible participants, six scheduled interviews, and five participated (one missed the appointment and did not respond to attempts to reschedule). Instrumentation This section describes the origin of the interview protocol. See Appendix A for a list of the questions arranged by concept; Appendix B provides the interview protocol. The interview protocol used questions both adapted from validated surveys and those that pulled directly from the research questions and learning activity. The interview protocol was structured for a conversational flow and order effect (Robinson & Leonard, 2018). Order effect primarily affects knowledge questions: Participants answered the knowledge questions before the self-efficacy assessment because the self-efficacy questions embed the answer to the knowledge questions. However, the discussion of the interview items below is organized by the conceptual framework. Regardless of origin, all questions were based on best practices. Questions related directly to the research questions and conceptual model. As questions referred to specific periods of time (during the course, after the course, looking forward), individual questions oriented the participants to the particular time of interest, and questions were ordered for logical flow (Patton, 2002). Each question asked about one idea (Patton, 2002) and used language familiar to the participants (Robinson & Leonard, 2018). Knowledge questions flowed directly from the learning activity. These two open-ended questions asked participants to recall the four aims of the Quadruple Aim (Bodenheimer & Sinsky, 2014) and to define a gap analysis. Research tends to favor recognition memory as more 43 rigorous than recall memory (Haist et al., 1992). To help participants recall knowledge from six months prior, contextual cues such as reminders about the course title, specific assignment, and faculty grounded their responses (Robinson & Leonard, 2018). While open-ended questions may pose a challenge for coding data (Robinson & Leonard, 2018), these specific questions had concrete answers, allowing ease of interpretation. Knowledge declaration, while a lower-level outcome (Krathwohl, 2002; Stevenson & Moore, 2018), is foundational to further exploration of self-efficacy and behavior. The self-efficacy section employed a Likert scale and was based on Bandura’s validated methodology (Bandura, 2006). While the original scale span was 0–100, Bandura noted that a simplified 0–10 scale may also be used (Bandura, 2006). For this survey, the reduced 0–10 scale was used. The endpoints and midpoint descriptors remained the same, with a zero indicating “cannot do at all,” five indicating “moderately can do,” and 10 indicating “highly certain can do.” Each item asked about a specific concept, aligning with Bandura’s definition that self- efficacy is task-specific rather than global (Bandura, 2006). Therefore, key elements from both the Quadruple Aim (Bodenheimer & Sinsky, 2014) and gap analysis (Clark & Estes, 2008) were separated to explore self-efficacy in discrete areas. Furthermore, respondents were asked about both their confidence in analyzing an organization using each element and their confidence in creating recommendations, with a recognition that these are two different skills. An open-ended question asked participants to consider how their perspective about their work has changed. This provided a descriptive element to the assessment of self-efficacy and allowed for participants to surface observations not directly asked about in the self-efficacy scales. Exploration into attitude used two open-ended questions that targeted beliefs about consequences and values. While TPB persists in the literature as a popular motivation 44 framework, little is known about physicians’ beliefs about consequences. Previous research focused more on beliefs about one’s own abilities (self-efficacy) and beliefs about values, whether an activity would be useless/useful and harmful/beneficial (Luconi et al., 2019). These questions sought to provide additional insight into motivational processes. Intention to change was assessed with two questions adapted from Luconi et al. (2019). These questions use a four-point Likert response (strongly disagree/strongly agree) and were based on a subset of questions from Légaré and colleagues (2014), who developed a 12-item validated survey to measure intention to change directly after a CME activity. Williams et al. (2014, 2015) used a separately created yet similar question in their research. Progress toward goals was measured using a single, ordinal-based question that asked participants to consider the recommendation for which they have made the most progress. The options were “not started,” “in progress,” and “complete.” Two open-ended questions asked participants about how they have otherwise applied their learning. One question asked about application of the Quadruple Aim, and the other asked about application of the gap analysis tool. These questions recognized that while someone may not have made progress toward their original recommendations, they may have applied their knowledge within their workplace to emergent issues. This is especially important as participants completed their coursework just before the onset of COVID-19, which disrupted healthcare operations. Questions about the environment included separate questions about the workplace and the education environment. All questions used an open-ended format. Participants were asked, separately, about the barriers and facilitators to implementation of their recommendations within their workplace. Barriers were addressed first, and participants were asked a follow-up question about how they had attempted to mitigate these barriers before asking about facilitators. While 45 previous research indicates that lack of time and resources are a chief barrier to behavior change (e.g., Haley et al., 2012; Jeong et al., 2018; Rodriguez et al., 2010; Schostak et al., 2010), it is important to understand the facilitators and barriers for this unique population in order to make improvements in the education environment. Questions about the education environment focused on how to improve learning about analyzing an organization separately from creating recommendations. Finally, participants were asked about which aspects of the learning environment were most effective. This aided in understanding what to keep, remove, or change when presenting recommendations. While this created numerous questions, separation of environment and context kept the questions focused (Mahmood, 2017). Finally, a short form collected general demographic information about the participants and their workplace. Basic queries about the participants’ employment and workplace included the number of years practicing medicine, type employment (i.e., employed, contracted), and setting (i.e., clinic vs hospital, single vs multisite organization). Personal demographic information was not collected as this information was not the focus of the study. This information assisted in contextualizing interview responses. Data Collection Procedures Interviews occurred 12 months after completion of the CME activity. This time lag allowed for exploration of retained knowledge, long-term impacts on self-efficacy (Luconi et al., 2019), and progress toward recommendations created upon completion of the activity (Reed et al., 2012). The interviews were scheduled for 90 minutes. This allowed time for general introduction of the study, permission to record, and to build rapport (Bogdan & Biklen, 2007). The interviews were via Zoom video conferencing software, allowing for voice and video recording as well as automated transcription. An MP3 recording served as a backup. The 46 researcher took notes throughout the interview using a protocol and coding sheet (Creswell & Creswell, 2018). Automated transcriptions required subsequent manual cleanup and further coding for analysis. Data Analysis The primary tool for data analysis was ATLAS.ti. This software program allowed for coding and indexing qualitative data to then analyze and interpret the results. Transcripts of the interviews were inputted into ATLAS.ti and then coded using open coding (Gibbs, 2018). Open codes were further categorized into themes (Gibbs, 2018) and connected to the research questions. Reponses to self-efficacy questions were compiled in Microsoft Office Excel and analyzed through basic descriptive statistics, primarily the mean (Salkind & Frey, 2019). Validity and Reliability Validity and reliability are important elements of any research. Validity is concerned with whether the research measures intended targets, while reliability focuses on consistency of findings (Creswell & Creswell, 2018; Merriam & Tisdell, 2016). Within qualitative research, the terms of credibility and trustworthiness can be considered synonymous. This section discusses both methodological and interpretive factors to increase rigor (Merriam & Tisdell, 2016). First, and as previously discussed, instrument creation adapted validated tools and question creation using best practices. Second, the researcher was an individual who can use the information to improve learning, and the objectives of the study were clearly stated. This pragmatic approach can increase the sense of sincerity and meaningfulness of the project (Merriam & Tisdell, 2016). To maximize responses, invitations specified the purpose of the project and included reminders to participate (Robinson & Leonard, 2018). As current students, the hope was that participants had a vested interest in participating to improve their education and professional experiences as 47 well as the experiences of future students. Moreover, with a small population, it was important to hear from as many voices as possible (Merriam & Tisdell, 2016). As data was collected, member-checking and reflexivity provided more intimate strategies. The researcher offered to share summaries of the interviews with each participant (only their summary). No participant requested a summary of their interview; however, two asked if the researcher would share a summary of findings at the end of the project. Should a participant have wished to revise or add to their summary, the researcher would have included this information in the data results. This form of member-checking allowed for additional data validation by the participants (Creswell & Creswell, 2018; Merriam & Tisdell, 2016). Finally, the researcher engaged in the practice of reflexivity, where the researcher reflects on how they impact the research and are impacted by the research (Merriam & Tisdell, 2016). This practice focused on participant responses as well as how responses compared with what might be expected based on the conceptual model and prior research. Finally, the purpose of this study was grounded in current research and was designed to provide insights into how to improve a specific learning experience for a particular audience. The transferability of the information, therefore, is intentionally limited. However, the conceptual framework and methods may be transferable to other settings. Ethics Ethics begins with the study’s purpose and rigorous development with an eye toward validity and reliability (Merriam & Tisdell, 2016). This section explores the protection of participants in the current study. This study was being mentored by qualified faculty as partial fulfillment of a doctor of education degree (Glesne, 2016). Furthermore, the study was approved 48 by the IRB at which the researcher is a student (Glesne, 2016). The researcher’s place of employment waived oversight, deeming that the institution had no involvement. Special care must be taken during recruitment and data collection, as the researcher was an administrator in the academic department where participants were enrolled as students. While the research was not a faculty of record for the course being assessed (nor a faculty of record for any current courses the participants will be taking at the time of data collection), this added a power element which can be ameliorated through careful design (Bogdan & Biklen, 2007; Robinson & Leonard, 2018). During recruitment, personalized emails were sent by the CPD office and the education coordinator within the DMME. This allowed potential participants to easily opt-out without engaging the researcher directly. Participants interacted with the researcher using the researcher’s student email (vs. email associated with place of employment). Four email invitations were sent over a multi-week span (Robinson & Leonard, 2018). Finally, the researcher explicitly addressed that the research goal was to improve the education experience and that participation in the research would not impact their student status. During recruitment and data collection, participants were informed about the purpose of the study through written and oral exchanges (Glesne, 2016; Merriam & Tisdell, 2016). The researcher distinguished between confidentiality and anonymity, as within a small department anonymity may not be possible (Robinson & Leonard, 2018), and the researcher clearly outlined how the information generated from the study would be used. Participants were reminded that they may withdraw their participation at any time and without penalty, and there was no compensation nor reciprocity associated with participating (Merriam & Tisdell, 2016). During data collection, interviews started with answering any questions the participant had and a verbal request for permission to record (Bogdan & Biklen, 2007). All participants agreed to be 49 recorded. During transcription, pseudonyms were used as needed. Data was stored electronically in secure cloud storage. Any physical artifacts (e.g., interview notes) were stored in a locked file cabinet and destroyed upon completion of the project. 50 Chapter Four: Results and Findings This qualitative research explored how physicians who claimed CME credit for a graduate course within their MBA applied their learning within their workplace and identified areas of improvement within the educational environment. Areas of interest included retention and impact of key concepts, motivations to effect change, and barriers and facilitators to implementing their recommendations. The study employed a combination of structured and semi-structured interview questions based on the conceptual model, rooted in social cognitive theory (Bandura, 1986). This chapter presents a description of the participants, followed by each research question’s findings. Participants Of the nine individuals who met the inclusion criteria, six responded to the invitation, and five participated in the interviews (one missed the interview and did not respond to the invitation to reschedule; 56% participation rate). Each chose a pseudonym, and references to specific organizations and individuals were obfuscated. All participants were physicians working in hospitals and clinics along the West Coast, with an average of 20.2 years of professional practice. The four physicians with more than 20 years of professional practice also held formal leadership roles within their organizations. Four of the five were specialists, and one was a general practitioner. Two worked in an urban setting, and three practiced in rural settings. Below is a brief biography of each participant, with all participants summarized in Table 2. 51 Table 2 Summary of Physician Demographics Jack Monica Grant Blodgett Joe Practice type Emergency department physician Specialist, out-patient, in-patient, procedures General Practitioner, mainly out- patient Surgeon Surgeon Years of practice 6 years 22 years 27 years 25 years 21 years Patient location Out-patient Out-patient, some in- patient Out-patient, some in- patient Out-patient Out-patient, some in- patient Employer Multi-site system Multi-site system Independent Practice Multi-site system Independent practice Practice location Single site Multiple sites Single site Single site Multiple sites Employment type Employed Employed Owner, partner Employed Contract Formal leadership role None Service-line director Owner, ACO board member Service-line director Vice chief of staff Jack was an emergency department (ED) physician with six years of experience as a physician (including residency), with two and a half years at his current organization, a large academic medical center along the West Coast also designated as a level 1 trauma center. While the organization is multi-site, Jack works at one location. Throughout his interview, Jack returned to the themes of social disparities and social determinants of health and focused on educating and empowering the next generation of physicians. Monica was a foreign-trained physician with 22 years of experience in the U.S. Monica and Jack worked at the same academic medical center and started working for the organization 52 about the same time. Previously, Monica worked in a variety of settings, both larger quaternary care hospitals and smaller community hospitals. Unlike Jack, Monica had her current clinical practice split between two sites. At one site, she was the inaugural service-line director for her specialty, charged with developing onsite and coordination of services with the other service-line directors across the system. Within the integrated care center, she did not have a leadership role. Her work included seeing patients primarily in the ambulatory setting and procedural units. She spent less time with hospitalized patients (in-patients). Monica’s examples demonstrated awareness of the complexities and interdependencies within healthcare delivery. She often questioned what something meant to and for patients, those providing the care, and the organization’s position relative to the community and external pressures. Grant was a general practitioner in a rural area with 27 years of experience as a physician. At the time of the interviews, he had two positions, both with leadership responsibilities. He is a partial owner in a small independent private practice. Additionally, he served on the board of the regional accountable care organization (ACO). An ACO is a regional response to help improve the health of the population as part of Medicaid Expansion (Centers for Medicare and Medicaid Services, 2020). Grant requested to answer interview questions using these two perspectives. He realized the differentiation between a small practice where he had greater authority than within a large coordinating body that monitored, informed, and sought to influence. Blodgett was a surgeon with 25 years of professional experience. He has worked in a variety of settings and for several organizations. At the time of the interviews, Blodgett worked for an organization that spanned the nation and practiced at a single site that, while located in a small city, served both the local and regional, rural community. He saw patients in the out- 53 patient setting. As a service-line director, he described the organization as being mission-centric, bureaucratic, and political. In his responses, Blodgett spoke to a desire of developing skills to navigate a large system effectively and to positively influence it. Joe, also a surgeon, was internationally trained and had 21 years of practice. He saw patients in clinic (out-patient) and in hospital (in-patient) in a rural environment. Like Grant, Joe was active in his regional ACO (a different ACO than Grant), but did not currently serve on the board. Joe was employed by an independent practice that contracts with the local hospital for services. Within the hospital, Joe has had several leadership positions. He reported that he had recused himself from all leadership roles except that as vice chief of staff over the last year. Of his concerns, those that took precedent were clinician wellness and developing collaborating relationships among independent hospitals and clinics. He expressed concern throughout the interview of a “healthcare desert” for rural areas should the system remain resistant. The physicians interviewed came from a wide range of experiences, from the front lines of healthcare delivery of general practice and emergency medicine to specialty care and surgery. They worked for organizations of different sizes and in different settings (urban vs. rural). Most (80%) came to the interviews with more than 20 years of experience as physicians and with some leadership experience that informed their perspectives. The next sections describe what physicians remember from their coursework, their motivations to make change using this information, and barriers and facilitators to doing so. 54 Research Question 1: What do Physicians Remember as the Key Learning from the CME Activity? Learning results in long-term change in knowledge that influences how a person perceives and interacts with the world (Alexander et al., 2009). This research question addressed what physicians recalled from key course concepts, how the course changed their perception about their work, and how they have applied what they have learned within their workplace. This question also explored what physicians found effective in, and what would have improved, their learning. Many studies conduct pre-/post-test measures to report on knowledge acquisition (Allen et al., 2019), and some studies conduct 6-month follow-ups (e.g., Anderson et al., 2012; Reed et al., 2012). In this research, interviews were conducted 12 months after completing the course, which allowed for assessing long-term retention and application of knowledge. To analyze knowledge acquisition, physicians recalled two key aspects of the course at the beginning of the interview. They responded to the following prompts: “Name the four aims of the Quadruple Aim,” and “Define ‘gap analysis.’” Answers were verified against definitions provided during the course. To assess the application of knowledge, physicians replied to the following open-ended questions: • How, if at all, have you used the framework of the Quadruple Aim in your work since last December? • How, if at all, have you used the gap analysis framework since last December? While the question, “Since the class ended, how, if at all, has your perspective about your work changed?” was designed to explore motivational factors, physicians shared changes in their 55 cognition in addition to the affective components of motivation. The responses related to cognition were included here. Similarly, to assess the effectiveness of the learning environment to support knowledge acquisition and application of learning, physicians responded to the following open-ended questions: • Thinking back to your time in the course, what would have made the course better in terms of learning how to analyze your organization? • What would have made the course better in terms of learning how to create recommendations for your organization? • What do you remember being the most effective in learning this content? (asked after each of the above questions) The interviews suggested that the course deepened their understanding of the Quadruple Aim and resulted in practice changes. Physicians had stronger recall for the details of the Quadruple Aim than the gap analysis. Overall, physicians described a broadening of their perspective of the healthcare system beyond their clinical areas as a result of the course and exposure to interdisciplinary perspectives. Furthermore, all interviewees reported progress toward the goals set forth at the end of the course and application of content to emergent issues. Physicians also provided recommendations to improve their learning. Physicians Had Stronger Recall for the Quadruple Aim Physicians recalled all the elements of the Quadruple Aim. The Quadruple Aim comprises access (or patient-centeredness), quality (or experience) of care, cost of care, and clinician wellness (Bodenheimer & Sinsky, 2014). Participants spontaneously recalled quality of care, cost of care, and provider wellness using consistent language. Two of the five participants 56 labeled access to care in those words, while two provided variations on this aim, describing the aim as “patient-centeredness” and “improved health.” All of the physicians volunteered that they had previously learned about the Quadruple Aim either through their place of work or continuing education. Jack indicated that this concept was “not a new idea to me” coming into the course. Grant has kept the Quadruple Aim active in his thinking, sharing, “I look at [it] all the time. So, I keep I keep reiterating. I keep going over in my head because I use it a lot in my work at all care.” This framework was a clear presence in the daily lives of these physician. Physicians struggled to recall all elements of a gap analysis. The gap analysis definition consists of two levels. The first level is the overarching process of identifying the current state, creating a vision for the future state, and identifying the gaps between those two states (Clark & Estes, 2008; Moore et al., 2018). The second level, based on the work of Clark and Estes (2008), describes three types of gaps as knowledge gaps, motivation gaps, and organizational gaps. All participants identified the first level, the current state, the future state, and the gap therein. None of the participants were able to recall the types of gaps. In probing about the types of gaps, four replied similarly to Blodgett, “No, I don’t remember.” Monica, on the other hand, did not recall the types of gaps but remembered their additional nuance and where to locate this information: “Don't remember them, but I meant to I meant to go back and look at it because I want to do a gap analysis of some of our current services.” Physicians offered insight into the variations in their ability to recall these key frameworks. Where the course reinforced the Quadruple Aim, the course introduced the gap analysis. Furthermore, they also noted that the elements of the Quadruple Aim were broken down into their respective components. In class, physicians had multiple opportunities to consider the connection between course content and their workplaces. None of the physicians recalled 57 practicing the gap analysis prior to the final paper. The course reinforced knowledge of the Quadruple Aim and introduced the basics of a gap analysis to physicians as a path to creating change, including changes to their perspectives about the healthcare delivery in the U.S. Physicians Developed a Broader View of the System Interviewees identified that having a framework aided understanding of and ability to adopt a systems perspective. Joe described how the course “made concrete for me observations I have had for a long time.” Jack specified how frameworks provide “a better vocabulary to what a lot of people can feel, and trying to…explain for a long time.” He continued that the frameworks additionally pointed toward solutions, “how to pick a direction,…the direction we think we should push the ship in for now.” This increased understanding led Monica to develop “confidence in being able to understand data and understand healthcare delivery more effectively.” Blodgett further expanded on how this connected to action, “If I can…separate one thing from another, I could at least know who to talk to about it.” Taken together, these suggest that frameworks and vocabulary can change how physicians conceived of and navigated their work. Furthermore, the assignment required students to apply the framework to their organization. Specifically, the requirement to apply the framework beyond the physician-patient dyad generated new perspectives about the healthcare system. Grant reflected that “in medical school, you're not trained to promote health. You're trained to find disease and treat it.” He described value in “taking a step back and being able to talk about things from a different perspective and a global perspective and putting it in writing.” As a seasoned physician with experience working at a variety of organizations, Monica spoke to her burgeoning awareness of her patients’ care beyond their time with her: “I think it's a combination of this whole year of being able to look at 58 things from the hundred-foot view. …Being able to identify what are the patient’s needs and how can we provide it.” This demonstrated her widening perspective of how the system impacted her practice and patients. As physicians develop a broader perspective, they realized the complexities therein. Understanding the complexity of the system is not without challenges. The broadening of perspective required physicians to reconceive their roles and place within the system. Monica reflected on how she was learning to interact differently with her non-physician colleagues, “I'm learning to recognize the ‘why’ of other team members, looking at it from their perspective.” Rather than independent pieces operating together, she saw greater interdependence. This growing awareness also brought a sense of frustration in how the system prioritizes and allocates resources. Jack, a newer physician who works in a high-volume ED, lamented, “We ignore that 50% of really largely socio-economic factors. …It's almost self-defeating, but we're focusing a large amount of effort toward…the wrong areas.” Jack added how the course “emphasized how much we need to do that has nothing to do with medical care.” The use of frameworks brought clarity and frustration as physicians grappled with the magnitude of the healthcare delivery system. This brought new appreciation for the interdisciplinary work needed to change a multi- faceted challenge. Interdisciplinary Points of View Fostered Learning Physicians cited dialog with colleagues from diverse roles across healthcare as instrumental in their learning. Dialog took place in three contexts: the synchronous class time, asynchronous forum posts, and interviews for their final paper. Joe highlighted the impact that a story from a social worker classmate during the live session: “I never thought about how I’d access care if I was living in a tent under a bypass and hadn't showered in three months and 59 somebody wouldn't let me into the ED.” As a specialist, Joe’s view of healthcare focused on his patient population. Hearing from others broadened his perspective of the healthcare system and his definition of population health. This level of discussion did not translate into the asynchronous forum posts. Jack described the focus posts as a way for him to “reflecting on where we're really working, who you're working for, who you're working with was particularly impactful just on a practical day-to-day kind of basis,” rather than exchanging in dialog. Physicians appreciated hearing from classmates from across the healthcare industry to increase their systems understanding. Three specified that the synchronous conversations were more effective than asynchronous forum posts. Many took additional advantage to converse with leaders and peers within their workplace. Interviewing leaders within their organizations, an expectation of the assignment, provided specific insights and professional opportunities. Learners were encouraged but not required to interview leaders within their workplace. Physicians were divided in their behavior. Three of the five physicians volunteered that they interviewed leaders and found it informative and career-enhancing. Grant “interviewed about 10 or 12 people. …I got to know a number of people in the organization, got personal time with them.” He specified how “it was really helpful to understand the organization that I'm considering a future with” as he considered his career trajectory. For Monica, the interviews resulted in opportunity. Two of the people she reports to are alumni and previously completed the same assignment. They took an active interest in her analysis, asking about what she was learning and provided opportunities for her to apply her learning. Interviewing leaders enhanced physicians’ understanding of and interactions within their workplaces, which they found valuable. However, not all physicians conducted interviews. 60 Physicians who did not conduct interviews believed it would have been helpful to do so. Blodgett did not remember conducting any interviews as part of the assignment and posited that if “they [faculty] give you permission to talk to the executives…or require you to connect with people in your organization,” he would have. He further shared that some instruction and guidance about setting up and conducting an interview would help him be more confident in his request, suggesting a worksheet with sample questions and tips for interviewing. Taken together, this suggested that physicians saw the value of interviewing individuals in their organizations regardless of their behavior and would like additional instruction on conducting interviews. The third component of learning (Alexander et al., 2009) and goal of CME (Davis et al., 2008; Moore et al., 2009; Vandergrift et al., 2018) is to change of behavior. Physicians Applied Learning in Planned and Unplanned Ways All interviewees made progress on their recommendations. Four deemed their recommendations to be “in progress,” and one completed his primary recommendation. All interviewees described the need for flexibility and adaptability in progress toward larger goals and questioned how one knows when larger-scale projects are finished. This appeared to be a significant shift in mindset connected to the role of physician-leader. Grant spoke of his shift in mindset from “I took care of an illness…or this person” to “it was all just… ‘in process’ all the time.” COVID-19 specifically interrupted all interviewees’ plans. As an example, Joe focused on his recommendation targeting the fourth aim of wellness. He raised support for a physician wellness program that would bring speakers and workshops to the rural hospital and clinics. Once travel and event bans went into effect, the team paused and shifted to virtual offerings. Physicians succeeded in making at least some progress toward their goals, adapting to emergent events as needed. Additionally, they used the Quadruple Aim framework beyond the assignment. 61 Physicians provided examples of applying the Quadruple Aim beyond the classroom requirements that extended from organizational strategy to personal decision-making to teaching others. In the last year, Grant used the Quadruple Aim framework to consider an integrated group practice (joint management with other independent practices), and questioned “When an organization is deciding on any given strategy or implementation of a program, does it address one of the aims? If it doesn't, I'm not sure it should be a high priority.” Grant used the Quadruple Aim as a decision-making litmus test. Monica used it to balance trade-offs. While setting up her schedule for her clinical practice at one site, she noted, “In my previous practices, I had always overbooked…. And I was very, very disciplined, this [adding more time slots] is not going to happen until the scribe comes.” She also attended to the balance of appointment types for current patients, new patients, and emergent needs. She consciously balanced two of the four aims: access to care and clinician wellness. Having the opportunity to work with interns and residents, Jack focused on teaching these concepts forward. In working with these learners, Jack encouraged them to identify gaps in quality and access specifically. Application of the Quadruple Aim appeared to bring clarity of decision-making, including priorities, within a physician’s control. Physicians reported less application of the gap analysis framework, yet provided examples throughout the interviews. This was best captured by Monica who shared, “it’s been very organic, it’s not been intentional. I have not sat down and said, ‘Let me do a gap analysis.’” Yet, the anecdotes she provided during the interviews suggest she applied the framework. In setting up her practice schedule, she spoke of how “studying the schedule and analysis each month has shown me these are the these are the areas we need to work more on.” This suggested an internalization of a concept, if not a systematic discipline, for conducting a gap analysis. Jack 62 differentiated between identification and action: “I think our department leadership has done a good job describing kind of issues with access, [but] haven’t done a ton with it.” This may be due in part to the pace of healthcare delivery at an operational level and the scope of healthcare delivery at the macro level. This concept was further explored in the third research question addressing facilitators and barriers to effecting change. Having had 12 months of practice, physicians reflected on aspects of the learning experience that enhanced learning and areas of improvement. Learners Requested Models, Tools, and Practice Physicians shared recommendations to improve their learning experience. Two physicians requested hearing from more thought leaders who have successfully navigated change in their workplaces. Blodgett noted that in hearing from senior leaders, “I found that stuff really revealing, like they're actually pretty smart about some of this stuff.” Monica sought guidance about their thought process, “how they sit down [at] a drawing board…and how they choose.” Both Blodgett and Monica sought guidance about where and how to make change; Blodgett wondered how leaders turn ideas into action. Blodgett summarized it as “how would you actually make these changes in an incremental way or how would you go about it?” Physicians perceived having seasoned leaders share their stories and to serve as models would further enhance their learning and application of learning within their workplace. Their focus of questions for thought leaders suggested the need for tools in analysis and creating recommendations. Interviewees sought tools to help them generate better analyses. Four spoke specifically to benchmarks. Blodgett questioned how to know “how well you are doing” and where “I am aiming towards.” Grant and Monica shared that they knew where to find quality benchmarks and wondered if there were benchmarks for the other aims. Jack added his curiosity about “how to 63 pick goals that are both aspirational and achievable.” He described his shift in perspective from setting goals for himself to setting goals for and with teams. He disclosed his strategy of “not necessarily meeting goals [if] you set them high enough, as long as you're making improvement” and recognized that not everyone had “the same comfort with what is technically failure.” Similarly, Blodgett identified a classroom team-building activity as a tool he wanted to use in his workplace to identify shared goals and develop alignment. He experienced this as a powerful tool and wanted the experience to be supplemented by instructional materials. Overall, physicians desired some more specifics on creating recommendations both for the purpose of their paper and as skills to take forward. Physicians offered two concrete ideas they believe would improve their papers: milestones and examples. While the final paper was introduced early in the course, there were few milestones during the quarter. Joe emphasized the need to work on the paper throughout the term and remembered it as “I think we were expected to build upon it sequentially during the course, and I did… But I don't think there was an actual sort of check-in type of thing.” He thought they might have submitted an outline, but he did not remember receiving feedback. As Grant gathered information and began to write his paper, he wished for some models or “some real-life examples of here was an analysis of an organization…, here's the way this consultant went about bringing those together, and organizing them into these groups.” Grant’s perspective brought together two crucial concepts. Physicians are both learning about the system of healthcare and changing their perspective about how they interact with the system. As they move from the physician-patient dyad to exploring their organization and the larger system, they realized the need for new skills as well as perspective. 64 Summary The impact of the course and assignment persisted over the intervening 12 months. Physicians demonstrated growth in multiple areas of Alexander et al.’s (2009) definition of learning, including recall, changes of perception in the world around them, and application of knowledge. They demonstrated strong recall for the four aims of the Quadruple Aim and partial recall for the definition of a gap analysis, struggling to identify the types of gaps. Most substantially, physicians expressed how exploring the Quadruple Aim and discussion with their classmates broadened their perspective about healthcare delivery from a focus on the physician- patient relationship to the broader system in which that relationship sits. Finally, the interviewees spoke to the application of these frameworks. All participants made at least some progress on their recommendations, with one reaching completion. Several applied the concepts to emergent decisions within their practices and organizations with the Quadruple Aim more explicitly applied and the gap analysis more intuitively. Physicians also provided recommendations to enhance learning, including hearing from thought-leaders, access to tools to create recommendations, and practice. The next research question explored why this learning was important to physicians and their motivations for system change. Research Question 2: What Are the Physicians’ Motivations to Identify and Make Recommendations for System-Level Areas of Improvement? Motivation bridges knowledge to action (Ajzen, 1991; Prochaska & Velicer,1997). This research question explored physician motivation for enacting system-level improvements. The analysis explored physicians’ personal drivers for change (attitude) through two open-ended interview questions: • What are the biggest drivers for you in making these changes in your work? 65 • Since the class ended, how, if at all, has your perspective about your work changed? Physicians rated their self-efficacy for components of both the Quadruple Aim and gap analysis frameworks using a 10-point Likert scale. Physicians reflected on their intention directly after the course and 12 months after the course using a 4-point agreement scale to assess intentions to change. The combination of discrete answers and open reflection provided insight into aspects of physician motivation. Overall, physicians are motivated and self-efficacious in making changes to the system. They expressed the desire to improve the healthcare system for their patients for themselves. They reported stronger self-efficacy for quality of care and moderate self-efficacy for influencing the cost of care and provider wellness. Finally, they intend to further their recommendations or work on recommendations that emerged over the last 12 months. This section explored each of these three findings in more detail. Physicians Envisioned Big Change and Acted Locally Before behavior changes, physicians must find the potential change important and valuable (Ajzen, 1991). Three themes emerged during the interviews: delivering high-quality care, increasing access and health of populations, and improving the system for themselves as a means to improving care for their populations. Physicians Prioritized Delivering High-Quality Care Physicians cited a desire to help their patients. When asked to describe their reasons for the recommendations they made in their papers, all discussed the ultimate impact on patients. Monica described the mutually reinforcing relationship between high-quality care and her identity as a physician: “To me, it is it is intensely gratifying to deliver patient care in a meaningful way that not only gets the patients what they want, but also gives me satisfaction as a 66 physician.” Monica identified the positive impact she can make for her patients, the hopeful outcome of improved health for the patient, which aligned with moral norms. However, she also identified a deeper importance that she accomplished her work in a way that she finds satisfying, which links to attainment value, the value of performing well. Physicians redefined high-quality healthcare when viewed systemically. Grant noted his shift from the medical school paradigm toward his desire to “improve health as opposed to just treat disease.” This influenced his conception of what high-quality care looked like: “Quality shouldn't be the outcome; it should be the process.” He described a possible scenario of a patient at risk for a heart attack where a “patient may say, ‘I don't really want to take another medicine.’…[Giving] the patient the autonomy…. You could reduce your risk for heart attack. Is that valuable to you? If it's valuable to you, then we can do this. If it's not, if you'd rather not be on a meds…” It is the dialog with the patient, to enable them to live their definition of a quality life, that motivated Grant. Viewing healthcare from a systems perspective clarified Grant’s moral norms of what it means for him to be a physician and to help his patients. The framework provided a utility value to arrive at a solution that met the needs of his patients and aligned with his identity. However, physicians kept patients at the center of their decision making. Even as he spoke of his work with legislative committees, Jack focused his intention on the individual patient: “It's just at the end of the day, I want to make sure that I stay in that patient relationship.” Through these perspectives, the physicians appeared to embody the moral norms of their profession, to take care of their patients. Within this, they described the attainment and utility values they derived from delivering care and improving the health of their communities. 67 Physicians Desired to Influence the System to Improve Access to Care All the interviewees identified access as a system-level challenge. They recognized that achieving the Quadruple Aim requires work on improving the system, specifically access to care. During the interviews, three key examples emerged: access to care within an organization, access to care in rural environments, and access to care for non-English speakers. Physicians wanted to ensure that patients have access to the appropriate care. Jack provided an example of how the current system limits the ED in furthering the Quadruple Aim. He spoke passionately about access to care and preventive care, sharing his vision of “10, 15, 30, 40 years from now, where you don't need emergency doctors” with the caveat that “they're always going to be accidents.” He juxtaposed this with his recent week: [T]he system failed these people, and we set them up to end up with this heart attack or stroke. It’s impossible for them to go see [their] primary care doctor so they come to the ER because it’s easy, and we’re always open. Especially this last week it’s been particularly difficult. Your patients come in because it’s 30 degrees and they’re homeless, and they don’t have a medical problem. How does this happen? Jack recognized that without changes to access to primary and preventive care, the ED would continue to accept all patients because that is where the patients can access care. Jack described access within the context of his organization; however, this issue spans across healthcare delivery in the U.S. Physicians in rural areas worried that patients may not have proximal care. For Joe, access to care meant ensuring healthcare services in rural areas. He shared the vision, “I see that we could do so much more as a rural system of care.” He spoke of this being at risk and 68 predicted dire consequences if individual practices and independent hospitals refuse to collaborate: And if that falls apart. …what happen[ed] in Montana and the Dakotas. A 200-mile or a 400-mile round trip drive for care. And that care might be a precipitated labor, that care may be a myocardial infarction, may be a Level 1 trauma with expanding bleeding. Each one of those three things I just outlined [has] a 90% mortality at a 200-mile trip. Like Jack, Joe saw access to care as a critical component to achieving the Quadruple Aim. These two examples highlight the limitation of an individual’s motivation for change. These examples require system-level initiatives and supports beyond the decisions and actions of an individual physician. Conversely, change can percolate up from a service-line to have broader impact. While care may be proximal, patients may not be able to access care due to cultural barriers. Monica spoke of access through the lens of increased cultural awareness. In beginning a new position as a service-line medical director, she learned that a quarter of the community’s population identified as Spanish-speaking. Initially focusing on her subset of patients, she wondered, “Do we have enough interpreters? Do we know to give after-visit summary in Spanish? Are schedulers able to schedule patients when they’re calling in a different [language]?” She then considered the campus more broadly: “How accessible is it for them?... What kind of directions do they get? … Do we have enough signs in Spanish?” As she spoke, her speech quickened with the excitement of the impact of what she deemed as achievable goals that may improve access and quality of care to a quarter of their patient population. She saw the opportunity for piloting change within her service-line that could lead to broader change throughout the site, “Now I'm looking at it from the whole system.” None assigned metrics to 69 their ideas. Instead, Jack, Joe, and Monica articulated the morality of their work and the utility value in small changes to improve the health of the populations they serve. Physicians Sought to Improve the System for Peers Physicians viewed the fourth aim of clinician wellness as critical to being successful in the other three aims, recognizing burnout as a national challenge for clinicians, physicians in particular. All physicians identified this as a motivating force. Blodgett spoke specifically in his investment as, “I’m not doing this to advance my career any more than would naturally occur. But I'm doing this to help my department, to help doctors.” Jack spoke to the system impacts on “my co-workers, and basically do what I can to make life better and easier on that level.” There was a pragmatic utility value expressed in the responses and a recognition that communicating physician needs and sharing their stories required different skills. Blodgett summarized this, “We’re very proud of what we do. We just can’t explain it to other people.” He clarified that other people were non-clinician administrators unfamiliar with the clinical environments but who make decisions impacting clinical environments. Monica similarly focused on communicating about and addressing the challenges faced by physicians, “…I also want…to be a big advocate for them…. I think streamlining the process and organizing teams in a way that allows for this efficient, impactful delivery of care is what I'm very passionate about.” Monica and Grant recognized the trade-offs, the costs. Placing limits in some areas allowed for progress toward the fourth aim of wellness and burnout prevention. In discussing improvements for clinicians, the physicians linked improving clinician wellness with improving their other areas of focus: access to and quality of care. In this context, provider wellness has utility toward achieving patient- focused goals. 70 Summary As medical students transition to their role of a physician, they recite the Physician’s Pledge. The first pledge states, “I SOLEMNLY PLEDGE to dedicate my life to the service of humanity” (The World Medical Association, 2017, “The Physician’s Pledge,” line 2). As evidenced by the physicians’ focus on improving the health of their populations, moral norms motivated much of their thoughts and behaviors, which aligns with the idea of service to humanity. From this lens, physicians discussed possible paths leading to improving the system through the Quadruple Aim, namely through the quality of care, access to care, and clinician wellness. Their motivation notably lacked mention of controlling the cost of care. These themes continued when exploring self-efficacy. Physicians Were Self-Efficacious in Applying Both Frameworks Self-efficacy describes the confidence an individual has to engage in a specific task and achieve the desired outcome (Bandura, 1986). Due to the task specificity component, physicians reported their self-efficacy for each aspect of the models on a 10-point scale. A response of 0 indicated that they “could not do at all,” a 5 indicated “can moderately do,” and a 10 indicated “highly certain can do.” The Quadruple Aim was broken down into the four aims. Each aim was assessed for one’s self-efficacy in identifying system-level areas of improvement and in creating recommendations to improve the system. The gap analysis was broken into six components: (a) accurately describe the current state, (b) develop a vision of the future, (c) accurately articulate knowledge gaps, (d) accurately articulate motivation gaps, (d) accurately articulate organizational gaps, and (e) use the information obtained to create appropriate recommendations. Overall, physicians rated themselves as moderately to highly self-efficacious in effecting changes via the two frameworks. Results are summarized for Quadruple Aim in Table 3 and gap 71 analysis in Table 4. Specific to the Quadruple Aim, most felt more self-efficacious in making recommendations than analyzing the system for areas of improvement. Consistent with this, physicians were most confident in their ability to develop a vision for the future with slightly lower ratings for accurately describing the current state when using the gap analysis. Furthermore, while physicians could not recall the aspects of the three types of gaps, all reported at least moderate self-efficacy in articulating the three types of gaps within their organization (M = 6.8). This finding noted a seeming disconnect between knowledge recall and perceived self-efficacy to apply said knowledge. Participants may have considered their ability to be able to identify types of gaps in the future, projecting a future level of self-efficacy upon being reminded of the types of gaps, which would be consistent with prior research (Williams et al., 2015). Together this suggested that physicians can envision a future but may not know how to achieve their vision. Additionally, variations of self-efficacy within and between physicians across the two frameworks are noteworthy. In discussing their ratings, physicians offered insights into factors that impact their self-efficacy in effecting system-level change. Key themes included the influence of complex problems and physicians’ perceived strengths in addressing quality. 72 Table 3 Reported Self-efficacy Scores for Applying the Quadruple Aim Framework Patient Access Quality of Care Cost of Care Provider Wellness M Identify Systems-Level Areas of Improvement Jack 7.0 8.0 3.0 3.0 5.3 Monica 7.5 7.0 8.0 6.0 7.1 Grant Independent Practice 9.0 9.0 9.0 9.0 9.0 ACO Board 6.5 8.0 9.0 8.0 7.9 Blodgett 8.0 7.0 5.0 6.0 6.5 Joe 7.0 9.0 5.0 7.0 7.0 M 7.5 8.0 6.5 6.5 7.1 Recommendations to Improve Systems Jack 5.0 8.0 5.0 3.0 5.3 Monica 8.0 7.0 8.0 7.0 7.5 Grant Independent Practice 8.0 9.0 8.0 9.0 8.5 ACO Board 7.0 8.0 6.5 8.0 7.4 Blodgett 9.0 7.0 5.0 7.0 7.0 Joe 9.0 9.0 9.0 7.0 8.5 M 7.7 8.0 6.9 6.8 7.4 Table 4 Reported Self-efficacy Scores for Applying the Gap Analysis Framework Current State Future State Knowledge Gaps Motivation Gaps Organiza- tional Gaps Create Recommend- ations M Jack 8.0 8.0 6.0 6.0 6.0 7.0 6.8 Monica 7.0 7.0 6.0 7.0 8.0 8.0 7.2 Grant Independent Practice 8.0 9.0 6.0 7.0 7.0 8.0 7.5 CCO Board 8.0 7.0 6.0 7.0 7.0 8.0 7.2 Blodgett 6.0 8.0 6.0 7.0 9.0 7.0 7.2 Joe 8.0 10.0 8.0 8.0 6.0 8.0 8.0 M 7.5 8.2 6.3 7.0 7.2 7.7 7.3 Note. The full prompts for the first six columns are as follows: (a) Accurately describe current state; (b) Develop a vision for the future state; (c) Accurately articulate knowledge gaps; (d) Accurately articulate motivation gaps; (e) Accurately articulate organizational gaps; (f) Use the information obtained to create appropriate recommendations. 73 74 Complexity of Issues Decreases Self-Efficacy Organizational context influenced self-reported self-efficacy scores. In considering his self-efficacy, Grant requested to provide two ratings, one for his role as a physician-owner of an independent private practice and another for his board role with the regional ACO. He consistently rated his self-efficacy for applying the Quadruple Aim higher within the context of his private practice than in his work on the board of the ACO. This discrepancy was particularly evident in two areas. Grant assessed his self-efficacy to identify systems-level areas of improvement for patient access as a 9 for his practice but a 6.5 for his work with the ACO. The same pattern emerged in creating recommendations for system improvement for the cost of care with rating his self-efficacy within his practice as an 8 and his work with the ACO as a 6.5. Using access to care as an example, he explained, “In our own office, as far as access, I would say that is fairly concrete. Either patients can see us, or they can’t.” He contrasted that with this work with the ACO: “It’s a little bit more nebulous because then you’re talking about access for a population of 50,000 people…dealing with a lot of independent offices, small groups, the health centers, larger hospital systems.” Grant’s reflection about his differentiated self-efficacy across contexts suggested that physicians have greater confidence in achieving their goals when the context is smaller with fewer variables and where they have greater control. Supporting this idea, Grant replied, “I’m not in charge of everything.” Grant’s examples highlighted the intersection of complexity and control. All of the physicians noted the cost of care as unique among the four aims, mainly for the complexity and lack of control. Joe identified the “multi-dimensionality” of system-level change as a mitigating factor for self-efficacy in controlling healthcare costs, which was his lowest reported self-efficacy rating. He cited two distinct areas of concern. First, he noted making cost- 75 conscious decisions in selecting between items of equal efficacy; however, he referred to his fear of lawsuits as a reason for adopting a new, expensive option over traditional and effective “clamping and cutting.” Second, he noted not being privy to the “administrative costs” that are factored into the cost of care. Monica shared this deficit, finding herself unprepared to understand and act upon information in the area controlling costs in healthcare delivery. In response, she recently supplemented her course learning with weekly executive meetings at her worksite “where we discuss finance; we discuss quality, patient safety, marketing, strategy, leadership, all of those. And so, hearing all about current budget, current staffing, future projections on a monthly basis weekly basis.” She noted that this combination of class learning and application in her career has increased her confidence in this area to one of her higher scores. The physicians reported being more self-efficacious when they have complete information and can take personal action. Physicians cited clinician-wellness as necessary yet rated this as the area of lowest self- efficacy. This appeared to differentiate between their individual choices, as all spoke to decisions they have personally made to improve wellness and that of the system. Joe elucidated the role control plays in this dichotomy. He initially rated his self-efficacy in analyzing the system and making system-level recommendations for provider wellness as a 10 for each. He adjusted his scores to a 7 when considering improvements at a system level. His initial rating was based on his personal success in this area. The strain COVID-19 placed on him as a physician and a father required that he change his perception and behavior: “I will not take on more, and I will not do more. And I used to look at those providers as being selfish and bad doctors, …as not understanding the romantic nature of healthcare.” Joe focused on his own choices. He shared that the system pressures him to increase his hours, requiring active resistance. He posited that “if all 76 of my business goes to this other doctor, sooner or later, they're going to burnout. And business will come back.” The pressure from the system and physicians’ individual choices decreased Joe’s self-efficacy in improving physician wellness at a broader level. These examples indicated insight and awareness about how an organization’s size and the level at which one is attempting to impact outcomes interact with our perceived self-efficacy to achieve desired results. Physicians are Self-Efficacious in Improving Quality Physicians rated their ability to identify areas of improvement in quality and to create recommendations to improve quality as areas of highest self-efficacy. Grant noted that of the four aims, quality is unique in that “usually not we are usually not defining quality; we're responding to quality.” On the larger scale of the ACO, he pointed to state mandates for quality and the star rating for the Medicare Advantage plan, also noting, “It's not always easy to bring up the rates,” and that having the target set for an organization removes a layer of uncertainty. For Monica opting into accreditation standards removed some of the ambiguity. When it came to improving quality at a system level, she felt, “There are so many quality tools and so many resources, I will need to lean on somebody with more experience.” Quality of care was embedded into care delivery environments, providing the support to Monica’s assertion. Jack specifically highlighted the organizational supports for quality, “We have a pretty good QI— quality improvement—system and reporting system. … It at least gets things on the radar.” He caveated this with, “How much we can fix the systemic issues depends on what that root cause is.” Organizational supports (e.g., QI systems, staff), external expectations, and cultural awareness of quality improvement may mean that physicians entered this course with pre- existing self-efficacy in this area. 77 Summary Physician self-efficacy was most robust in areas where they had more control, such as their own choices, their practice, and areas in which they have authority. As found in quality improvement, self-efficacy was bolstered by organizational supports, which are further discussed in the exploration of environmental factors. They reported less self-efficacy in controlling the cost of care, citing the multi-dimensionality of complexity in understanding cost. As the system became more complex, such as involving multiple, independent practices and organizations, physicians reported being less, yet still at least moderately, confident in their ability to effect change. Higher self-efficacy is a precursor to intention to change yet does not guarantee an intention to change. Physicians Reported Strong Intention to Change Between motivation and action lies an intention to change (Mazmanian et al., 1998). Within the final paper, individuals created recommendations for their organization, including guidance on which they could take action. During the interview, physicians were asked whether they agreed or disagreed with the statements: • At the end of the course, I planned to take action on my recommendations. • I plan to take action on my recommendations in the future. Reflecting to when they crafted their recommendations, all physicians agreed that they planned to take action on their recommendations at that time. Four of the five agreed or strongly agreed with the statement that they intended to continue working on their recommendations in the future. Blodgett was the exception and shared that he completed his primary recommendation. Of the five, he also was the only interviewee who did not intend to continue work on his recommendations, citing both completion of his primary goal and organizational changes that 78 precluded progress toward the others. This underscored the dynamic context in which physicians work. Physicians may be motivated to improve the system, yet they operate within a complex environment, discussed in more detail in the next section. These mutually influencing factors mitigate action, as explored in the next section. Summary Physicians were motivated to and perceived themselves as capable of making some change within the healthcare system. They found high value in improving the system to offer high-quality care and to improve the lives of their peers. They rated their self-efficacy moderately-high in all areas with the lowest related to controlling the cost of care. Interestingly, physicians rated their self-efficacy moderately high in areas they could not recall, namely types of gaps found within a gap analysis. They noted less confidence as challenges increased in complexity and ambiguity. Yet, they expressed a continued intention to work toward their original and emergent recommendations related to achieving the Quadruple Aim. Physicians identified barriers to and facilitators for their endeavors. Research Question 3: What are the Barriers and Facilitators Physicians Perceive to Implementing Change? This research question explored facilitators and barriers to physicians implementing the recommendations they created as the culminating assignment in the health systems course. During the interviews, physicians responded to open-ended questions. Additionally, information shared when reflecting on the knowledge and motivation questions that pertained to this question was included in the analysis. Physicians were asked the following questions as part of their interviews: 79 • Considering all of the recommendations you made in the Quadruple Aim paper, what factors in your workplace have most facilitated your progress toward implementing your recommendations? • What have been the most significant barriers in your workplace in implementing your recommendations? • What have you been able to do to mitigate these barriers? Barriers and facilitators arose from a combination of internal factors and organizational factors. Internal factors are those more in control of the individual, while organizational factors reflect the individual’s setting and context. Within each of these broad categories, multiple themes emerged. Internal factors consisted of control, skills in creating alignments and alliances, communication, and knowledge and experience. Organizational structures were those facilitators and barriers external to the individual, such as organizational strategy and culture, competing priorities, teams, leadership and mentoring, and the external environment. Table 5 and Table 6 summarize these themes. Within the education environment, physicians discussed areas of strengths and areas of improvement for course content and activities. Details of each theme are further described below. Internal Factors Physicians had most successes in changes to their clinical practice. These were local, concrete changes focused mainly on the aim of clinician wellness. Changes to clinical practice involved change on the part of others. For both local change and broader change, developing alignment and alliances facilitated change efforts, as did strong communication skills. Challenges came mainly from lack of knowledge and experience, especially when outcomes and processes are less clear. 80 Table 5 Summary of Internal Factors Category Facilitators Barriers Practice changes • Decisional control • Setting priorities • Understanding the trade-offs Alignment and alliances • Pulling together themes to establish a shared vision • Openness to ideas • Small wins • Differing motivations • Conflicting ideas Communication • Creating a key message • Influencing • Speaking to stakeholders • Vulnerability and humility • Listening • Lack of access to decision- makers • Hierarchy of medicine Knowledge and experience • Increased exposure • Difficult to choose a direction • System complexity • Social determinants of health Access to information • Lack of experience Physicians Created Change in Their Clinical Practices Overall, the physicians described more success in making changes where they had control. In considering where they made the most change, four physicians identified their clinical practice. Three specifically concentrated on setting limits on hours spent working as an example of how they have worked toward achieving the Quadruple Aim. Monica spoke about prioritizing change in setting up her new practice, “In my previous practices, I had always overbooked and added. And I was very, very disciplined, this [adding more appointments] is not going to happen until the scribe comes.” Knowing that meeting demand for her services was insurmountable, she described balancing her own wellness needs with quality and access. Using her practice data, she considered how to structure time and who was the best person to see the patient. Only after she 81 had additional resources would she consider adding patients. Similarly, Joe knew that he could not change the amount of documentation required after each visit. He could control the number of patients he saw in a week. Through reverse engineering of the dilemma, he determined the number of patients he could see in a week while staying within his goal of total hours worked. Grant spoke of similar trade-offs in his private practice group. The practice owners, which include Grant, set limits on access: “We would probably do better for our patients if we offered Saturday or Sunday hours, [but] we just have not…from a provider wellness perspective, we have not decided—for the most part—not to work weekends.” Physicians were successful in creating change where they had control of decisions. However, they also spoke of successes in efforts that involve others and noted skills that lead to productive action. Developing Alignment and Alliances Cleared the Path for Change Physicians spoke of the importance of creating and articulating a shared vision. From the perspective of the paper, Grant shared, “They certainly weren't my unique ideas, but they were things that the organization was moving in the direction of, and I sort of put them into words.” In Grant’s case, he summarized conversations and ideas discussed both formally and casually over time into a single document. He attended to themes and areas of agreement, which he synthesized and shared with concrete recommendations. In developing new teams and service- lines, this historical record does not exist. It can take time to find the best direction and a willingness to adapt, as articulated by Monica: But at the same time being open-minded because maybe the idea that you're proposing is not the best. … Sometimes, you may not get the entire change implemented… recognizing that you are not so vested in the outcome of that process that you lose some of the buying-in of the stakeholders. 82 This suggested that a series of smaller actions that ultimately lead to a grander shared vision and larger change can build alignments. Jack supported this in speaking about smaller experiments and that “easy wins every now and again are a good idea and keep momentum going.” As momentum builds, physicians found themselves needing to keep the focus on the goal. Monica noted that this takes concerted and consistent communication “so everybody feels heard…and everybody has heard the same thing, and, has agreed upon the same workflow, strategies, or whatever the discussion.” With this, Monica began shifting her focus from creating a shared vision to putting ideas into action. Where alignment provides the direction, alliances provided the political will for action. Grant specified that this required “rather than just intuitively, but in…a concrete strategic manner.” Joe added that to create change, shared vision was insufficient and found himself “advocating strongly for…the fourth theme [clinician wellness]. That's really what’s important to me.” This combination of being a champion of a shared need allowed him to launch a physician wellness program. Not all alliances are as easily formed. Alliances involve understanding individuals more deeply. Monica described, “It’s often, sometimes it’s a bit of a power thing, or people have different motivations, and why a certain decision needs to be laid out, and to be curious and having those conversations and understanding those motivations.” She continued, “to build allies and build alliances and make sure that you have read the room and have allies that the conversation before the conversation, and the conversation after the conversation.” These efforts have helped her mitigate barriers to change within organizations. Physicians offered their strategies for overcoming individual resistance to change. 83 Communication Cited as a Critical Behavior All physicians spoke to the role that communication played in building alignments and alliances. Blodgett identified “the hardest part was getting people on board with making changes. People don't like change.” He used the metaphor “instead of peeling the onion, I have to build layers,” which he further described as being “able to explain it to other people and show them what is needed to fill the gaps.” His approach to building alliances was to develop a critical mass with a clear message, “…we need to plead for something else… So, I gotta get them going. Gotta get their language right.” He described his tactic as “shuttle diplomacy.” “I actually go around and stand in people's offices…. I don't try and force it on them, but I try and convince them that it’s important, that they should think about it.” He found a challenge in finding “linchpin people,” those who will be heard by executives. When prompted about who he thought would be a good advocate, he replied: “I think it’s their friends. So, I know who their friends are. I’m friends with their friends,…maybe they’ll listen. Because it’s not gonna be slides and graphs.” Monica affirmed this sentiment, and recognizing the complexity of these conversations, framed her goals for communication as a message “articulated in a way that it resonates with all the different stakeholders,” noting that “I'm still polishing.” Physicians saw their presentation skills as a foundational requirement that girds their ability to effect change. They additionally acknowledged that communication required opportunities for providing input. Hearing from others is especially important in the traditional hierarchy of medicine, where many have learned to defer and acquiesce to the physician. Within the ED, the teams that Jack works with varied shift to shift. He spoke to the importance of creating a work environment where people “are not afraid to speak up.” Jack focused on his daily interactions while Grant reflected on the importance of multi-directional communication in change efforts. He defined 84 communication as “listening and also communicating, plans, etc., rationale for why you’re doing something.” He believed this behavior builds trust, a critical factor in change efforts by validated people’s thoughts, “They're more likely to trust that you are making a decision that, though they may not necessarily agree with it, but they would appreciate that you consider their input.” Afterward, he amended, “A bit of vulnerability helps as well so that people understand that…you have some humility.” Physicians demonstrated how they practice creating space for others and the positive impact this had on leading change efforts. This concept resurfaced in the context of teamwork through bottom-up solutions, as discussed in the next section. Physicians identified forming alignments and alliances as strategies that aid their efforts. Furthermore, interviewees believed that attuning efforts with organizational strategy, incorporating perspectives, and understanding underlying motivations strengthen the likelihood of success. As projects and initiatives proceed, physicians described the need for continued teamwork. Lack of Knowledge and Experience Stymied Physicians As physicians engaged in broader systems-level change, they reported increased challenges in applying their recommendations. This wider scope hindered direction-setting, as described by Blodgett with some frustration, “The chasm is so big that it's hard to know where we're heading…I…feel like I reach out into the chasm….” Grant echoed this in his challenges associated with increased complexity when attempting system-level changes. Grant focused on the increased breadth at the ACO board level, “We’re a lot more removed from the direct access.” He described the challenges associated with maintaining alignment across organizations rather than being able to offer concrete solution for a defined population. He lacked knowledge in how to manage these alignments as well being able to understand the unique issues of access 85 across organizations. This suggested a difference from practice-level changes where they saw concrete needs and paths forward, encouraging those who do the work to create solutions. The interviewees identified the complexity of making system-level improvements as a Sisyphean task. Throughout the interview, Jack’s focus returned to social determinants of health and closing health disparities, lamenting, “It’s definitely a helpless feeling when you’re trying to help a patient who needed help, you know, 10 years ago.” When considering what is within his domain, he added the caveat, “Obviously, one person can't change the system. You can only push so hard in one direction.” Blodgett captured the tension as, “I think the system just rules too much. I don’t think I can change it. …I can change little things, …but the [electronic medical record] is that something I can’t change.” Jack and Blodgett divided ideas into areas they can change or control and those they cannot; Joe saw the system as entrenched and enmeshed. Trained overseas, Joe was newer to the U.S. healthcare system. Through this lens, he described, “The more [I learn] about the U.S. system of care, I don’t see how we take 80 or 90 years of layers of bureaucracy and get to the point where I could say, ‘I've seen a patient. … This is the bill.’” Joe pointed to regulatory and external pressures that impact organizational goals, strategies, and tactics, ultimately impacting clinicians and patients. Cost of care emerged as a specific challenge for physicians. Physicians rated their self- efficacy in this as their lowest overall, and all physicians interviewed spoke explicitly to their challenge in controlling cost in healthcare from a system perspective. Monica focused on challenges associated with expanding her expertise, “Cost of care is…all this [is] so complex, no matter how much you learn about it.” After over 20 years as a practicing physician, she found herself in the role of novice and feeling the need to develop depth in this topic. She added, “Cost is not something that you can immediately effect,” indicating that these broader changes have a 86 longer timeline and may be more difficult to measure. To mitigate this, Monica requested regular meetings with administrators and executive leadership and included financial aspects as a key component on the agendas. Joe similarly cited the multi-dimensionality of costs, specifically administrative costs and external pressures over which he has no oversight, as the reason for his lower self-efficacy ratings. Physicians conveyed the feeling that creating recommendations for and taking action on controlling the cost of healthcare was beyond their current capacity. The overall lack of knowledge and experience coupled with complexity and ambiguity overwhelmed physicians at the broader level. The intersection of organizational and external factors complicated change efforts, which are discussed in the next section. Summary Physicians recognized the validity of working toward the Quadruple Aim and made successful changes in areas within their control. They made varying levels of progress toward larger goals through the use of alignments and alliances and through strong communication skills. However, they felt that they lacked knowledge, control, and time to influence change at the broader levels, citing challenges associated with organizational and external forces. Organizational Factors Physicians exist within the larger microcosm of their organization, and organizations are said to achieve the results for which they are designed (Coiera, 2011). As physicians reflected on their progress, they highlighted key facilitators and barriers to the implementation of their recommendations. Facilitators included alignment with organizational goals and values, team building, and supportive leadership and mentorship. Physicians were challenged in navigating the ambiguity of broad system-scale challenges, competing and shifting organizational priorities, the historical culture of medicine, and dysfunctional leadership. Just as physicians exist within 87 the microcosm of their organizations, organizations exist within the larger context of legislation, regulation, and current events. Given the enormity of the external impacts of 2020, physician experiences are shared here. Table 6 Summary of Organizational Factors Category Facilitators Barriers Organizational goals and values • Clear organizational goals • Shared values • Experiments, iteration • Access to resources • Rogue individuals • Lack of awareness team successes Team building • Interdisciplinary teams, inclusion • People who speak up • Solution-focused attitude • “Heroic” physician myth • Individuals working collectively, but not as a team Competing priorities and shifts of direction • Clear vision • Exploring possible unintended consequences • Physician champion • Disparate departmental needs/populations • Silo • Resource competition • Fear of impact on self • Change fatigue Leadership • Clearly stating goals • Encouraging bottom-up solutions • Resource support/protection • Assistance navigating change • Mentorship • Clear decision-making • Lack of clear leadership structure • Unclear decision-making • Factors beyond the control of leadership External environment • Shifting regulation • Current political uncertainty • Unknowns and restrictions associated with COVID-19 88 Alignment with Organizational Goals and Values Facilitated Change Aligning with the goals and values of an organization appeared to facilitate change. This was an extension of creating alignment within teams. Physicians spoke of this in two ways: setting goals and aligning with goals. Three physicians spoke of setting clear organizational goals and allowing ideas to percolate. Grant shared an example from his private practice. He noted that had worked with his fellow physician-owners for over 20 years and that they “have similar values.” He attributed this to fostering an openness to running experiments about how to improve an organizational goal of increasing employee engagement. His practice was to “have bottom-up solutions rather than top-down solutions” and to implement “change in a small pilot project…and then passing on best practices to other areas of the office…” He referred to the success of the extended huddles as an example. Jack described making changes within his teams that align with his department’s and organization’s larger goals as, “We set a super ambitious goal, but it’s just on a very relatively small, small scale.” Jack noted that his colleagues “find solutions and realize they don’t work, and find different solutions.” While the goal or objective may come from senior leaders, both Jack and Grant spoke of successful solutions coming from those closest to the work. In the opposite direction, aligning with organizational goals can benefit teams. Blodgett described his organization as hierarchical and noted the importance of working within the system. Blodgett found that for individuals who are “shooting [from] the hip, seat of the pants. Making things come and go to suit themselves,” the organization “will eventually find you, route you out, and throw you away.” This idea evoked responses of both self-preservation and a direction forward. Blodgett spoke of his desire to “get us [the department] aligned” with organizational goals, seeing this as a path that “allows us to invoke resources appropriately.” 89 Blodgett articulated how strategically selecting goals within his area can increase their chance of success and build credibility throughout the organization “to make my department shine.” Earlier, physicians spoke to the importance of aligning local teams. As change ideas extended beyond their current sphere of influence, physicians found aligning with organizational goals and initiatives effective. Yet, operating at a system-level proved to be more challenging and required teamwork to achieve goals. Physicians Focused on Building Teams as a Driver of Change When considering what makes projects successful, physicians highlighted the importance of teams. They also noted this as a change in their perspective. Physicians reflected that the historic culture of healthcare had not always encouraged teamwork. In considering their decisions to become physicians, Joe described the “romantic” nature of being a doctor. Jack spoke plainly, “Everyone wants to be called a hero.” Similar to the difference between treating disease and improving health as identified by Grant earlier, Monica discerned the difference between a medical team and teamwork: “Medicine, in general, is not really teamwork. I mean, we have teams, but we really are doing our own individual learning and applying one to one to our patients.” As physicians faced challenges associated with population health management and leadership roles that require managing additional perspectives, the importance of teamwork increased. Three physicians explicitly endorsed an interdisciplinary approach to teamwork. Jack cited nurses and social workers as “…the biggest asset in the department and probably [hospital- wide].” He further described them as people who identify problems “because they're rolling around in the mud on a daily basis…going that extra step for our patients and to just raise the flag and say, ‘Hey, listen, we need to do better at this.’” Jack described a patient- and solution- 90 focused attitude toward work that suggested a belief that improvement can happen. However, it required a place and confidence to raise issues, as discussed earlier, which can be built intentionally. When Grant saw fragmentation within the staff of his private practice, he felt responsibility to act. To create a sense of belonging and increase engagement, Grant described a recent change: We’re actually doing a little huddle each day. …I think it gives a little more voice to people who may be on the frontlines of answering phones and may be frustrated with feeling overwhelmed or feeling lack of support. Having a five-minute check-in each day gives them a place to say, “Hey, I saw this yesterday,” or “here’s a problem that came up.” Whereas if you don’t set aside that time, you just, you’re just processing stuff all day long and busy and not really in building a team mode. In the interviews, physicians identified those who desire to improve the situation as an essential factor in creating change. Their examples noted that these ideas come from across roles such as other physicians, nurses, staff, and technicians. Monica further supported developing interdisciplinary teams that include administrators. She described the variety of teams she has formed: care delivery team, a quality team, a service- line team, a peer-leadership team. Each served their respective goals. For example, in her role as a medical director for a service line, she said, “I feel physicians, nurses, and administrators should work as a triad.” This triad connects to project teams within the service line and monthly with the hospital’s executive team. In reply to a query about how she keeps her teams moving forward, she laughed and attributed it to her organization skill and noted “We communicate a lot.” She stated, “The biggest resistance to change has been cultural,” referring to competing priorities. 91 Competing Priorities and Shifts of Direction Hampered Change Efforts Competing priorities created a unique challenge for larger organizations. Jack spoke to balancing the needs of patient populations across different departments within organizational planning, “I think it’s hard to support the missions of individual departments…when the population of patients is pretty drastically different. For example, between the trauma surgery department and the plastic surgery department.” Reflecting on how this impacts him and his team, he described, “I’m working in a large institution, with many moving parts, which is relatively siloed.” He continued, “You’re fighting for resources,” and it “makes it hard to align any goals really.” Aligning goals at organizational and system levels requires individuals to change. Physicians drew attention to the entwined competing priorities within individuals and their organizations. Monica identified that system changes “may threaten their current position or the current role or the current territory in certain ways.” Monica highlighted the need to explore all of the facets of change and to understand unintended consequences. For a large-scale change, the identity of organizations may be at risk in addition to the identity of individuals. Joe stepped back from his clinical practice to examine healthcare delivery in his region as part of his work with the regional ACO. He described a vision to “get all of the facilities…to work as one big rural system of care.” Currently, private practices and small, community hospitals send patients to larger cities 85-115 miles away rather than the local tertiary hospital where Joe worked “because they are afraid if their patients come to us, care nearby, then they [will] come here for everything else anyway.” He noted two barriers to improving regional care. First, he focused on his physician peers not wanting “to take the impetus to be the outreach providers for those areas.” Second, he expressed confusion about why the executive leadership had not pursued this 92 as a regional goal and wondered if “each individual fiefdom has its own competing interests,” including job security for senior leaders. He recognized that scaling these efforts would require executive sponsorship and support and physician champions. Yet, energy to champion initiatives is a finite resource. When organizations change strategy and initiatives, physicians working on the previous priorities can feel like they failed. In speaking about an aborted project that he thought would improve workflows, Blodgett shared wearily, “Oh, I had good intentions…to help the health care providers, and I was going to try and reduce alerts, but I’ve done a bad job with that. I just didn’t do that at all. We went a different direction at work….” As Blodgett saw the impact of these changes on his own motivations, Monica saw the effect on teams. She worried about change fatigue seeing both shifting priorities and that too many initiatives “overwhelm…all the staff, the nursing staff, the physicians, the financial teams.” Physicians expressed awareness and concern for the volume and seemingly arbitrary nature of shifting priorities with their organizations. They also identified leadership as having the potential to mitigate these challenges. Supportive Leadership Aided Change Efforts Supportive leadership was identified as a key facilitator for change. Working in a large, complex healthcare system, three interviewees focused on the importance of leadership in effecting change. Monica has spent the last nine months building a new service-line. She credited “a supportive administrative team, and empowering me to lead in the way that allows me to implement a lot of these tools.” She specified that they offered “support from the team to navigate some of the barriers that come up.” Resources comprised a specific barrier. As the initiative started, the healthcare system began limiting resources to redeploy as needed in response to COVID-19. The executive team “protected the resources that have been allocated for 93 this work. … It was delayed a bit, but it wasn't halted, and it’s complete now.” For Monica, supportive leadership included the freedom for her to lead and apply her coursework, lending authority to navigate barriers and protecting resources. Responsive executive leadership supported change efforts. Similar attributes were found at the department level. Department leadership provided access to resources and mentorship. Jack described how his department chair modeled behavior earlier described by the physicians as advantageous. She had a “broad view of what where we are, where we need to be. But does a really good job of accounting for…personal aspects…making sure people feel heard and actually doing something about it.” The department chair modeled the effective top-down behaviors of creating a vision. She balanced this with actions that support bottom-up leadership, listening and attending to the pain-points of those closest to the work. Similarly, Jack noted that their director of quality maintained a sense of curiosity, helping the team connect to the larger, systemic issues. The quality director asks of every case reviewed, “How can we do better for our patients? What did we do that affected our patient negatively…? And what do we do in our process…to set us up for this failure or this improvement opportunity.” Jack connected good leadership with good systems and implied that good leaders create good systems. These systems can emerge quickly in response to emergent needs. In responding to a crisis, leadership can effectively marshal resources and align organizations. Blodgett spoke to the speed and efficiency in which leadership responded to COVID-19, “Incident Command and was a big rush to get all this stuff done back in the spring, early summer.” The response plan was in effect at the time of the interview; however, Incident Command has been dismantled. Blodgett expressed concern about how decisions will be made going forward as the pandemic evolves, specifying a lack of clear leadership. When leadership is 94 aligned with the goals of the project, it helps secure resources and navigate barriers, and system- level projects can happen quickly. The lack of such clarity sowed seeds of concern. Poor leadership disenfranchised physicians. In talking about barriers to implementing his recommendations, Joe shared, “This place is run by morons,” and recognized that “our hospital and our medical center are just microcosm reflections of the wider healthcare system and U.S. government.” This observation of the cascading impact of leadership echoes what was shared by Monica and Jack, however, in reverse. A consequence is that over the last year, Joe has resigned from all but one of his leadership roles. He furthered emphasized his desire to distance himself from what he saw as poor role models, absenting himself from board meetings where he witnessed “absolute crazy decision-making.” In creating this distance, Joe reported that he had shifted his focus from trying to improve the broader system to creating a professional domain that provides him work-life balance and engagement in personally meaningful projects. Leadership has the power to inspire, empower, and align resources to accomplish goals. Leaders of healthcare organizations find themselves buffeted by external factors that impact their effectiveness. The External Environment Confounded Change Efforts Physicians saw challenges associated with working in a complex and regulated environment. Monica acknowledged that organizational leadership had limits to its authority in supporting system-level change at the broadest level. She spoke of the limits of making large changes due to policy, reimbursement structures, and regulation. She offered a specific example of telemedicine: “If you do a telephone visit on a new patient, you spend the same amount of time, and you really don't get paid anything for it, like $20….” While this may increase access and reduce costs, the organization cannot support this at scale and survive financially. These 95 regulations were beyond the control of physicians and organizations yet influenced decisions and direction setting. Organizational priorities can change due to changes in the external environment. At the time of these interviews, the results of U.S. Presidential elections were being challenged in the courts, and the electoral college had yet to ratify the results. Four of the five physicians interviewed specifically spoke to the political landscape and possible policy change. Grant’s belief that he could effect change on a larger scale was tempered, “It’s so dependent on a lot of things out of control out of your control—political things, trends….” Joe expressed concern about the sustainability of the Medicaid Expansion Plan and the financial devastation that would create in his community. He predicted that would decrease access to care significantly for his community. Simultaneous with the political uncertainty, the U.S. was facing the second surge of COVID-19 cases, nearing 300,000 cumulative deaths nation-wide. Healthcare organizations were preparing for an influx of patients, launching clinical trials, and planning for initial vaccine distribution. These plans were changing weekly in response to evolving information and in efforts to address historically marginalized communities and structural racism within medicine. In considering barriers to implementing recommendations made pre-COVID-19, Monica summarized, “I think in this pandemic is not a huge area where you can make a big change right now. Recognizing that and…tabling it [for] the future.” Similarly, Jack observed, “For last nine months it has been incredibly easy to get sucked up by kind of the day-to-day literal operations of things.” He described psychic toll COVID-19 has taken in his department: “They’re worrying, ‘Am I going to get COVID today?’ …[Y]ou’re having 40- and 50-year-olds who are coming in and dying. … At this point it’s taken up more of our cognitive efforts, our attention.” The 96 response to COVID-19 catalyzed organizations out of necessity and intensified the focus on the present. Blodgett reflected to his time on the COVID-19 incident command team. He noted the speed and efficiency which occur when an organization focuses. He juxtaposed this with concern that policy will not keep up with current needs and development, “They’ve dismantled the incident command. So, I have no authority, but the chief of staff is interim and who knows what will happen.” COVID-19 has consumed healthcare, restricting resources, and narrowing the focus of organizations. Physicians appeared both frustrated and resigned to what they perceive as barriers beyond their organizations, yet they were not without hope. Physicians planned to continue their involvement. Grant and Joe planned to continue to be involved in their regional ACO and local professional groups. Monica planned to increase her involvement with national associations aligned with her specialty. In response to the course, Jack, already involved with his professional association, joined the legislative arm of the local chapter to help influence policy. He voiced optimism that the conversation, while difficult, may produce change: “It seems that people are more willing to speak out about access and discrimination and just really these issues that have been buried in health care for a long time.” The combination of COVID-19 and political transitions created a disruptive environment where information changed daily and created uncertainty in paths forward. Despite the challenges, physicians continued to plan for the future. Summary As physicians worked toward implementing their recommendations, they found supports and detractors within their organizations. Similar to work within their clinical practice, aligning recommendations with organizational goals supported efforts. However, progress in these efforts was limited by the complexity that accompanies larger scale change. Furthermore, as 97 organizations responded to the environment, the result was competing or changing priorities, which adversely affected progress. This was especially true in the landscape of 2020 with a global pandemic, local and national rallies for social justice raising questions long unasked, and a turbulent national election. Supportive leadership and mentorship mitigated some of these challenges through the protection of resources and supporting individuals. Having explored the facilitators and barriers within the workplace, physicians also reflected on the third environmental factor of the education environment and how the experience better prepares change leaders. 98 Chapter Five: Recommendations and Discussion This qualitative research explored how physicians applied their learning from the health systems course within their workplace with the intention to improve the learning experience. This chapter describes the findings as they related to prior research followed by research-based recommendations. Next, the limitations and delimitations of the current study are discussed. The chapter concludes with recommendations for future research. Discussion of Findings and Results Physicians experienced growth in their knowledge, reported high levels of motivation, and made progress on their recommendations despite external challenges. The strengths and weaknesses within knowledge acquisition align with previous research about spaced-learning and repeated content. Furthermore, the current study sheds additional light on an area not well articulated in the CME literature, building capacity for systems-thinking. Physicians expressed the moral implications of their work and saw utility in their learning and recommendations, indicating high motivation. As they made progress on their goals, areas in which they have control, solid leadership, and alignments and alliances fostered success. Lack of experience and knowledge, especially related to complex and ambiguous challenges, stymied action. Each of these is discussed below. Knowledge Acquisition Previous literature found connections between repeated content and learning. This includes between events (Berrett-Abebe et al., 2019) and within a single learning event (Flett et al., 2018; Gooding et al., 2017). Additionally, participants’ learning increased when asked to compare session content to their current practice (Ratelle, Wittich et al., 2017). Retention of knowledge aligned with these previous findings. All interviewees entered the course with 99 previous exposure to the Quadruple Aim, and they completed regular reflective exercises related to each aspect of the Quadruple Aim prior to the culminating final analysis. Conversely, participants participated in one, live 60-minute interactive session on conducting a gap analysis using a case study. A shorter lecture, worksheets, and supplemental material were provided as optional, on-demand content through the learning management system. However, there was no formal practice with the gap analysis leading up to the final paper, and physicians did not report exposure to this concept prior to the course. Repeated exposure to content, both before the course and during, and multiple opportunities to practice influenced retention of knowledge, consistent with previous findings. The current study generated novel findings as well. The requirement to apply the framework beyond the physician-patient dyad generated new perspectives about the healthcare system. According to Kitto et al. (2013), this is an understudied area of CME. However, it is acknowledged that healthcare needs physician leadership (Steinert et al., 2012), which requires physicians to understand management, leadership, and finance (Stevenson and Moore, 2018). Within the context of this study, physicians reported increased capacity for system-level perspective. Furthermore, in discussing implementation of their recommendations, they spoke to practicing key leadership activities, including creating alignments, communication, and teamwork (McMahon, 2017). This activity expanded leadership capacity in learners. Motivational Factors Physicians were motivated to make changes aligned with the Quadruple Aim. Previous research suggested that utility value (Luconi et al., 2019) and moral norms (Luconi et al., 2019; Parsa-Parsi, 2017) are particularly influential motivational factors. Both of these constructs were reported by physicians in this study along with attainment value, the satisfaction of doing their 100 work well. Furthermore, physicians rated themselves as self-efficacious in applying frameworks to analyze organizations and create recommendations. Of note, Williams et al. (2015) found that personal self-efficacy influenced motivation more than past experiences, which was born out in this study. While physicians could not recall the details of a gap analysis and reported not using the gap analysis consciously within their work, they reported strong self-efficacy in applying the tool. The second trend in self-efficacy was the connection between years of practice and leadership roles with reported self-efficacy. The literature lacked studies exploring these variables, yet prior research identified lack of authority as a perceived barrier to change (Rodriquez et al., 2010), lending support to this concept. Progress Toward Goals Physicians reported their greatest success in areas closest to their direct practice. This aligned with previous findings that lack of access and authority presented a barrier to change (Rodriguez et al., 2010). Three physicians focused on the most elemental aspect of their practice, their availability. This came in the form of setting limits to the number of patients they could see and resigning from leadership roles in an effort to work on projects and achieve work-life balance. That physicians widely chose this as a goal is unsurprising given that previous research found that lack of time thwarted efforts or impinged on personal time (Haley et al., 2012; Jeong et al., 2018). For projects with a broader scope, all physicians interviewed spoke to the importance of alignment and alliances as ways to mitigate barriers of access and authority. Unlike previous studies, physicians did not report fear of failure (Haley et al., 2012) or fear of appearing less than competent (Jeong et al., 2018) as challenges. However, similar to Rodriguez et al. (2010), they identified lack of experience and knowledge as challenges to effecting change 101 in their organizations, most frequently when discussing cost-control and direction-setting in ambiguous and complex environments. Organizational structures and cultures can intensify or mitigate some of the physicians’ challenges. This study supported previous research (e.g., Kogan et al., 2017; Reed et al., 2012; Vuković et al., 2015) that found physicians have more success when aligned with organizational goals. Furthermore, all physicians spontaneously spoke to the necessity of interdisciplinary teamwork in effecting change, consistent with previous findings (Chauhan et al., 2017; Davis & McMahon, 2018; Jeong et al., 2018). Finally, physicians identified supportive leadership as a key ingredient, including formal and informal mentorship, which supports previous research by Haley et al. (2012) and Kogan et al. (2017). Most of the findings in this study supported previous research and provided specific areas of improvement for the educational environment. Recommendations for Practice The recommendations for practice centered on changes within the education environment with the goal of better preparing physicians to apply their learning and make progress on their goals. The three key recommendations below addressed the findings of improving knowledge acquisition for conducting a gap analysis and supporting learners in creating their analysis, creating achievable recommendations, and building skills in navigating ambiguity. The recommendations used research-based practices supported in the literature. Embed Additional, Research-Based Teaching Practices Research-based teaching practices can enhance knowledge acquisition, support self- efficacy, and increase the successful application of content. First, add practice with the gap analysis framework as part of the spaced-learning already occurring for the Quadruple Aim framework. This can be accomplished by having students frame their forum responses to distinct 102 areas of the Quadruple Aim using a gap analysis framework. Spaced-learning has been shown to increase knowledge acquisition (Agarwal & Bain, 2019; Berrett-Abebe et al., 2019; Flett et al., 2018; Gooding et al., 2017) and self-efficacy (Dacey et al., 2013; Williams et al., 2015). This practice would allow physicians to integrate complementary frameworks, which may mitigate some of the challenges they expressed in navigating ambiguous environments. Finally, this activity can support goal-directed practice and targeted feedback (Ambrose et al., 2010). Learners receive feedback from faculty and peers throughout the weeks leading up to their final paper. This provides an opportunity to practice course content and deepens understanding and thinking with a focus on mastery. Additional support can help learners focus by removing extraneous work. The second recommended research-based practice is scaffolding. Interviewees specifically requested two forms of scaffolding: model papers and job aids. Model papers, including annotated outlines and exemplars, provide mental models to constructing a consulting report, a form of writing new to most physician-learners. Learners can focus on the content rather than determining the format. The second scaffolding element requested was worksheets, a form of job aid. Job aids can help in practicing new skills or refresh knowledge when a task is periodically performed (Clark & Estes, 2008). To aid with the former, a worksheet about how to set up and conduct interviews as part of their assignment may enrich the information gathered by students as part of their paper. In regards to the latter, learners requested worksheets (known as enduring materials within CME) about how to replicate certain class activities (e.g., techniques to determine existing alignments) so that they could use these activities within their professional settings. Including specific elements of scaffolding can help learners focus within the context of the course and apply their learning within their workplaces. 103 Develop Content to Teach Development of Recommendations Findings from the current study and previous research demonstrated that appropriate and achievable recommendations are linked with goal attainment. Course content should consider the following four aspects: alignment with organizational mission and goals, benchmarking, stakeholders and team, and required resources. Previous research into the effectiveness of CME shows that goals aligned with organizational mission and strategic goals have high success rates (Katzman et al., 2014; Vuković et al., 2015). Second, benchmarking can help organizations monitor change over time and compare metrics to other groups, thus providing direction to focus efforts (Ozcan, 2008). Third, as seen in the research, interdisciplinary teams (Chauhan et al., 2017; Davis & McMahon, 2018; Jeong et al., 2018) and supportive leadership (Haley et al., 2012; Kogan et al., 2017) aid in goal completion, and the current study underscored the importance of including stakeholders. Therefore, it is recommended that consideration of these two groups be incorporated into recommendations. Finally, under-resourcing has been cited in the literature as a barrier to change following a CME activity (Rodriguez, 2010). While alignment with organizational mission and strategic goals may assist with this, developing clarity about required resources can assist in the creation of achievable goals. Broaden Guest Speakers Who Model Behavior The final recommendation is to add guest speakers to serve as models for learners. Modeling behavior has been shown to be effective in moral development (Bandura, 1986) and in the development of managerial skills (Bandura, 1997). Learners shared how the shift in mindset from physician-patient dyad to physician-leader impacting a system required an extended timeline and lacked the same feeling of completion. Furthermore, they requested hearing from leaders about setting direction and navigating ambiguous environments. Hearing from leaders 104 may assist emerging leaders to build efficacy in selecting goals (choice) and developing coping capabilities and persistence, key components to sustained effort (Bandura, 1989). Integrated Recommendations A change management plan improves the likelihood of success. Kotter (2007) notes that most change efforts fail due to lack of structure that acknowledges the process and pacing of change. Previous research demonstrates the effectiveness of using the ADKAR model to effect complex organizational change (Wong et al., 2019). The ADKAR acronym stands for: awareness, desire, knowledge, ability, and reinforcement (Prosci, n.d.). This section explores organizational factors and resources required in each of these areas to implement the recommendations from this study. Awareness comprises the first phases of the ADKAR model. This begins with communication of the problem from credible sources, thus establishing the desired future environment. The study’s findings support themes from data not included in the study, including faculty meetings, assurance of learning processes, and academic program review. Additionally, recommendations were formed using research-based practices and recommendations from students, both with their unique form of credibility. Of note, the recommendations from students align with research-based practices, and provide guidance as to which practices to prioritize. These form the basis of information, which can be supported by departmental leadership. However, as seen in this study, knowledge alone does not lead to change. One must desire a different future. Desire follows awareness, linking information to the organizational context and personal motivations. The department’s mission is to develop passionate healthcare leaders (DMME, n.d.), which is heartily endorsed by the faculty and staff. Furthermore, the department is 105 modernizing its curricula to keep pace with the market, evolving technology, and both challenges and opportunities presented by COVID-19. As a result, this information further informs change spurred by additional influences and which are supported by leadership. For these specific recommendations, connecting to the personal motivation of individual faculty and staff helps ensure adoption. The modified TPB (Ajzen, 1991) described in Chapter 2 of this study can be applied to adoption of change efforts. The TPB begins with attitude, well operationalized using the EVT model (Wigfield et al., 2017). Faculty express satisfaction from knowing their courses have positive impacts on students, a form of attainment value. Seeing improved student work from the course and hearing from students about their success can bolster attainment value. Moreover, linking these recommendations to wider program goals can harness the power of utility, how a change in one area can improve success at wider change. Finally, cost can be mitigated by engaging teamwork. Where a faculty member may not feel self-efficacious in changing instructions or supplemental materials, other faculty and staff can lend support, increasing collective self-efficacy (Bandura, 2000), the second aspect of the TPB model, perceived control. The final aspect of the TPB, commitment, requires additional bolstering. Similar to the findings of this study, building alliances and alignments are critical activities. If those respected by the faculty endorse the recommendations and act themselves, other faculty members will be more likely to adopt them. While not part of the ADKAR model, Ajzen (1991) noted that a commitment to change increases action. Pausing to draw out commitment to specific action can help prepare faculty for the activities required in changing their courses. Subsequently, individuals may be attracted to the vision and committed to act, but they may not have the requisite skills. 106 Increasing knowledge and skills comprise the third aspect of the ADKAR model. For the recommendations in this study, the following two inter-connected activities are critical. First, and as mentioned previously, individuals have varying levels of knowledge and experience using research-based teaching practices and technical knowledge that applies to the course. Those who have knowledge and skills can help create materials. For example, one person may be able to assist in creating rubrics that both help students develop their work and faculty in providing better feedback. Another person may have specific content knowledge about how to form recommendations as it taught in a subsequent course, which can be introduced in a preceding course. This requires the second component of teamwork and coaching each other based on individual strengths. These set the stage to increase the penultimate aspect of the model. Ability adds practice, feedback, and resources to the above. Presenting this as an experiment, one that uses an iterative process, open to adaptation based on feedback, can help individuals adjust to new ways of doing. This approach has served the team well in the past especially when combined with coaching from those who have previously adopted practices successfully. The lack of time or other emergent distractions may derail the project. Leadership’s intervention to individuals to protect time, set clear milestones, and reinforce priorities can help the team stay focused. Once the change effort is underway, efforts must be in place to maintain the change. Reinforcing change prevents organizations from returning to the previous state. However, one must remain open to iterative change based on feedback and outcomes of the change initiative. Course feedback and post-course interviews with students can further clarify what aspects assisted their learning and transfer of knowledge and what needs refinement. Individual faculty can share their learning with the larger group to build best practices and model behavior. 107 Finally, leadership can publicly acknowledge those who have made changes and the impact of those changes on the course and program outcomes. This can happen informally at the individual and group level and formally through nomination for teaching awards. At completion of a study, it may be clear to those intimately involved what change is needed and why. However, this must be compellingly conveyed to the larger community involved in adopting the recommendations. The use of a change model, like ADKAR, can facilitate these efforts, though not completely remove barriers and resistance. Additionally, limitation and delimitations of the study should be weighed in assessing organizational change. Limitations and Delimitations Limitations and delimitations are weaknesses in a study that influence interpretations of the findings, which are mitigated by placing the research within its context and qualifying the outcomes (Ross & Bibler Zaidi, 2019). Limitations are factors beyond the researcher’s control. Delimitations are factors inherent to the study design and research choices that may limit the findings. The main limitations of this study centered around participation. Of the nine students who met the inclusion criteria, 5 completed the interview. These five practiced mainly in the outpatient setting and procedural areas. Therefore, these findings may not be relevant for physicians practicing as hospitalists or intensivists in the inpatient setting. The use of multiple invitations (total of six email invitations) and follow-ups to those who missed their initial interview increased participation (Robinson & Leonard, 2018), but competing factors may have limited participation. At the time of the interviews, the national landscape was in disarray as the U.S. presidential election results were being challenged by the sitting president. Clinics and hospitals were experiencing the post-Thanksgiving surge of COVID-19 cases. This surge came 108 nine months into the global pandemic which further strained an already overextended healthcare delivery system. The nature of the pandemic created extra stress as each day a physician entered their hospital or clinic, they put themselves at risk. Concurrently, healthcare organizations were deploying individuals to support clinical trials and vaccine dissemination for COVID-19 to their communities. This rapidly shifting environment may have precluded availability to participate or otherwise influenced participation. During an interview, a physician may be challenged in responding to questions. They may have responded in ways that they deemed socially desirable (Robinson & Leonard, 2018), or their memory of the course may have changed over time. Based on the variety of responses offered by participants, social desirability was not obvious. During two of the interviews, participants commented on not having a clear memory about an aspect of the course and confabulated an experience in another course. Alternatively, the delay between completion of the course and recalling key concepts provided a measure of enduring knowledge (Anderson et al., 2012). Finally, 90 minutes was a substantial request for a practicing physician. While three interviews were rescheduled from their original date and time and two interviews were temporarily interrupted by calls from the hospital, participants were able to respond to all questions. In addition to limitations, the study design required choices that impact transferability. Delimitations included the decision to focus on physicians who claimed CME and the focus of the interview questions. The decision to focus on physicians who requested CME was in response to assessing a new initiative. While the academic program has existed for more than 10 years, awarding CME for the course began in 2017. While it is likely that the findings are translatable to most of the learners in the degree program, the findings may not be translatable to larger contexts. Moreover, as this is a new line of inquiry, topics may arise that were not 109 predicted by the conceptual framework. The combination of structured and semi-structured format helped maintain reliability while providing freedom to explore lines of inquiry set out by the conceptual framework and research questions through follow-up questions and probes. Overall, the findings supported the prior research and unearthed an interesting line of inquiry, the connection between longevity and leadership with the ability to effect organizational change. Recommendations for Future Research The study generated two surprising elements. First, the study found a disconnect between knowledge and self-efficacy. Physicians were unable to recall the types of gaps and did not report using the gap analysis in their work, yet they rated the self-efficacy in its application as moderately-high. Exploration of this perplexing contradiction augments research by Williams et al. (2015) that found personal self-efficacy outweighs beliefs derived from past experiences. Second, years in practice and having a formal leadership role aided in goal creation and completion. This is of interest because CME as a form of leadership development is understudied (e.g., Kitto et al., 2013). Furthermore, previous research found access to and lack of authority as barrier to goal completion (Rodriguez et al., 2010), which longevity and leadership roles may mitigate. Taken together, this suggests a possible line of future inquiry. Other future studies are more incremental in nature. The current study offers concrete recommendations for the education environment based on the findings and research-based practices. Upon implementation of these practices, follow-up interviews can ascertain whether the recommendations improved application of learning. An iterative and persistent approach may help keep the education environment responsive to changes in the workplace environment and to the needs of current learners. Similarly, physicians comprise a portion of total students, and a degree program consists of multiple courses. The 110 current line of inquiry could be expanded to all learners and multiple courses. The result would be a more robust data set that would lead to greater transferability. Conclusion The purpose of this research was to explore how to increase the transfer of knowledge in physician-students who earn CME for their graduate coursework. Social-cognitive theory (Bandura, 1986) formed the foundation for the conceptual model with specific factors deriving from the CME literature. The study explored knowledge acquisition and motivational factors, commitment to change and progress toward goals, perceived barriers and facilitators to change, and areas of strength and improvement for the education environment. The findings were used to create research-based recommendations to improve the educational environment. Overall, physicians showed strong retention of knowledge even after 12 months. They showed stronger retention for content that was not new to them and that was reinforced throughout the course. They showed partial recall for new content. However, they reported moderate to high levels of self-efficacy in applying their knowledge to identify and solve system- level challenges. They also indicated both progress toward previously state goals, application of the Quadruple Aim framework to emergent issues, and intention to continue working toward their recommendations. Physicians demonstrated adapting to their environment throughout their interviews. Findings from the current study support previous research about the effectiveness of CME to change behavior. Namely, physicians were successful in creating change where they had control and less so as recommendations became more complex. Lack of knowledge and experience navigating complex and ambiguous environments was a key limitation for individuals. Alignment of individuals and of goals with organizational mission and strategy 111 supported change as did supportive leadership. This study added to the literature perspective on building capacity for developing system-perspective, a key leadership skill. From these findings, key recommendations concentrated on improvement to the learning environment. The three key recommendations below address the findings related to improving knowledge acquisition for conducting a gap analysis and supporting learners in creating their analysis, creating achievable recommendations, and building skill in navigating ambiguity. The first recommendation is to embed additional space-learning and practice activities related to the gap analysis alongside of the existing practices for the Quadruple Aim. Additionally, provide scaffolding for students in conducting consulting types of activities such as alignment exercises, interviews, and crafting consulting reports. The second recommendation is to develop content specific to developing recommendations that consider the facilitators and barriers to change. Finally, focus guest speakers on how to effect change in organizations. This serves to increase knowledge and vicarious experience and to model behaviors. These practices can help facilitate application through increased knowledge and motivation. Successful implementation of these recommendations has the possibility to not only increase the transfer of knowledge, but to impact healthcare delivery systems. Physicians who pursue the degree and complete this course are practitioners, many with current leadership roles or who have been identified as emerging leaders. Yet change in one course may not be sufficient. Therefore, future lines of inquiry should assess impact of these changes and extend beyond this subset of students and to multiple courses. The findings of the current study pointed toward other areas of research specific to CME. Future studies may want to examine an apparent contradiction and a novel area of exploration. While physicians only had partial recall for gap analysis, they all reported 112 moderately high levels of self-efficacy in applying to framework. While this may be a discrepancy between recall and recognition, it could be worth exploring as previous research (Williams et al., 2015) found a similar split between beliefs and experience in regards to self- efficacy. Additionally, continuous improvement research regarding changes to the education environment may result in program improvement and provide direction for effective delivery of leadership education in CME. While developing leadership capacity through CME is an understudied area (e.g., Kitto et al., 2013; Steinert et al., 2012; Stevenson & Moore, 2018), this study demonstrated success in developing system-level perspectives in physicians engaged in graduate education. 113 References Accreditation Council for Continuing Medical Education. (2019). ACCME data report: Growth and diversity in continuing medical education—2018. http://www.accme.org/sites/default/files/2019-07/818_20190716_2018_Data_Report.pdf Agarwal, P. K., & Bain, P. M. (2019). Powerful teaching: Unleash the science of learning. John Wiley & Sons. Aghera, A., Emery, M., Bounds, R., Bush, C., Stansfield, R. B., Gillett, B., & Santen, S. A. (2018). A randomized trial of SMART goal enhanced debriefing after simulation to promote educational actions. Western Journal of Emergency Medicine: Integrating Emergency Care with Population Health, 19(1). https://doi.org/10.5811/westjem.2017.11.36524 Ajzen, I., Joyce, N., Sheikh, S., & Cote, N. G. (2011). Knowledge and the prediction of behavior: The role of information accuracy in the theory of planned behavior. Basic and Applied Social Psychology, 33, 101–117. https://doi.org/10.1080/01973533.2011.568834 Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T Alexander, P. A., Schallert, D. L., & Reynolds, R. E. (2009). What is learning anyway? A topographical perspective considered. Educational Psychologist, 44(3), 176–192. Allen, L. M., Palermo, C., Armstrong, E., & Hay, M. (2019). Categorising the broad impacts of continuing professional development: A scoping review. Medical Education, 53(11), 1087–1099. https://doi.org/10.1111/medu.13922 Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: Seven research-based principles for smart teaching. John Wiley & Sons. 114 American Medical Association, & Accreditation Council for Continuing Medication Education. (2017). Accreditation Council for Continuing Medical Education (ACCME(c)) and American Medical Association (AMA) glossary of terms and definitions. http://www.accme.org/sites/default/files/2018-04/011_20170421_Glossary_of_Terms.pdf Anderson, G. S., Gaetz, M., & Statz, C. (2012). CPR skill retention of first aid attendants within the workplace. Prehospital and Disaster Medicine, 27(4), 312–318. https://doi.org/10.1017/S1049023X1200088X Angood, P., & Birk, S. (2014). The value of physician leadership. Physician Executive, 40(3), 6. Atwood, K. A., Shamblen, S. R., Gaudet, T., Rindfleisch, A., Collins, D. A., Milovani, C., Greenfield, R., Kiefer, D., & Rakel, D. (2016). Impact of a clinical educational effort in driving transformation in health care. Family Medicine, 48(9), 711–719. Balmer, J. T. (2013). The transformation of continuing medical education (CME) in the United States. Advances in Medical Education and Practice, 4, 171–182. PubMed. https://doi.org/10.2147/AMEP.S35087 Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191–215. http://dx.doi.org.libproxy1.usc.edu/10.1037/0033-295X.84.2.191 Bandura, A. (1986). Social foundations of thought & action: A social cognitive theory. Prentice- Hall. Bandura, A. (1989). Human agency in social cognitive theory. American Psychologist, 44(9), 1175–1184. https://doi.org/DOI:10.1037/0003-066X.44.9.1175 Bandura, A. (1997). Self-Efficacy: The exercise of control. Freeman. 115 Bandura, A. (2000). Exercise of human agency through collective efficacy. Current Directions in Psychological Science, 9(3), 75–78. Bandura, A. (2005). The evolution of social cognitive theory. Great Minds in Management, 9– 35. Bandura, A. (2006). Self-efficacy beliefs of adolescents. Guide for Constructing Self-Efficacy Scales, 5, 307–337. Berrett-Abebe, J., Cadet, T., Nekhlyudov, L., Vitello, J., & Maramaldi, P. (2019). Impact of an interprofessional primary care training on fear of cancer recurrence on clinicians’ knowledge, self-efficacy, anticipated practice behaviors, and attitudes toward survivorship care. Journal of Cancer Education, 34(3), 505–511. https://doi.org/10.1007/s13187-018-1331-y Berwick, D. M., Nolan, T. W., & Whittington, J. (2008). The Triple Aim: Care, health, and cost. Health Affairs, 27(3), 759–769. https://doi.org/10.1377/hlthaff.27.3.759 Bloom, N., Sadun, R., & Reenen, J. V. (2012). Does management really work? Harvard Business Review, November. https://hbr.org/2012/11/does-management-really-work Bloom, N., Sadun, R., & Van Reenen, J. (2014). Does management matter in healthcare. Boston, MA: Center for Economic Performance and Harvard Business School. Bodenheimer, T., & Sinsky, C. (2014). From Triple to Quadruple Aim: Care of the patient requires care of the provider. The Annals of Family Medicine, 12(6), 573–576. https://doi.org/10.1370/afm.1713 Bogdan, R., & Biklen, S. K. (2007). Qualitative research for education: An introduction to theories and methods (5th ed.). Pearson. 116 Bordage, G., Carlin, B., & Mazmanian, P. E. (2009). Continuing medical education effect on physician knowledge: Effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest, 135(3), 29S-36S. https://doi.org/10.1378/chest.08-2515 Brothers, B. M., Carpenter, K. M., Shelby, R. A., Thornton, L. M., Frierson, G. M., Patterson, K. L., & Andersen, B. L. (2015). Dissemination of an evidence-based treatment for cancer patients: Training is the necessary first step. Translational Behavioral Medicine, 5(1), 103–112. https://doi.org/10.1007/s13142-014-0273-0 Carpenter, S. K. (2012). Testing enhances the transfer of learning. Current Directions in Psychological Science, 21(5), 279–283. Center for Professional Development. (n.d.). Who can accept AMA PRA Category 1 Credit TM ? https://www.CHSU.edu/sites/default/files/2018- 12/Who%20can%20accept%20AMA%20PRA%20Category%201%20Credit.pdf Cepeda, N. J., Vul, E., Rohrer, D., Wixted, J. T., & Pashler, H. (2008). Spacing effects in learning: A temporal ridgeline of optimal retention. Psychological Science, 19(11), 1095– 1102. https://doi.org/10.1111/j.1467-9280.2008.02209.x Cervero, R. M. R. M., & Gaines, J. K. J. K. (2015). The impact of CME on physician performance and patient health outcomes: An updated synthesis of systematic reviews. Journal of Continuing Education in the Health Professions, 35, 131–138. https://doi.org/10.1002/chp.21290 Chauhan, B. F., Jeyaraman, M., Mann, A. S., Lys, J., Skidmore, B., Sibley, K. M., Abou-Setta, A., & Zarychanksi, R. (2017). Behavior change interventions and policies influencing 117 primary healthcare professionals’ practice-an overview of reviews. Implementation Science, 12. https://doi.org/10.1186/s13012-016-0538-8 Clark, R. E., & Estes, F. (2008). Turning research into results: A guide to selecting the right performance solutions. IAP. Coiera, E. (2011). Why system inertia makes health reform so difficult. BMJ, 342. https://doi.org/10.1136/bmj.d3693 Cook, D. A., Blachman, M., Price, D., West, C., Berger, R., & Wittich, C. (2017). Professional development perceptions and practices among U.S. physicians: A cross-specialty national survey. Academic Medicine, 92(9), 1335–1345. https://doi.org/10.1097/ACM.0000000000001624 Cook, D. A., Price, D. W., Wittich, C. M., West, C. P., & Blachman, M. J. (2017). Factors influencing physicians’ selection of continuous professional development activities: A cross-specialty national survey. Journal of Continuing Education in the Health Professions, 37(3), 154–160. https://doi.org/10.1097/ceh.0000000000000163 Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications. Creswell, J. W., & Poth, C. N. (2018). Qualitative inquiry & research design: Choosing among five approaches (4th ed.). SAGE. Dacey, M., Arnstein, F., Kennedy, M. A., Wolfe, J., & Phillips, E. M. (2013). The impact of lifestyle medicine continuing education on provider knowledge, attitudes, and counseling behaviors. Medical Teacher, 35(5), e1149–e1156. https://doi.org/10.3109/0142159X.2012.733459 118 Davis, D. A., Mazmanian, P. E., Fordis, M., Van Harrison, R., Thorpe, K. E., & Perrier, L. (2006). Accuracy of physician self-assessment compared with observed measures of competence: A systematic review. JAMA, 296(9), 1094–1102. Davis, D. A., & McMahon, G. T. (2018). Translating evidence into practice: Lessons for CPD. Medical Teacher, 40(9), 892–895. https://doi.org/10.1080/0142159X.2018.1481285 Davis, N., Davis, D., & Bloch, R. (2008). Continuing medical education: AMEE education guide No 35. Medical Teacher, 30(7), 652–666. https://doi.org/10.1080/01421590802108323 Department of Medical Management Education [DMME]. (n.d.). About Us. www.CHSU.edu/DDME/about Federation of State Medical Boards. (2019). Continuing medical education: Board-by-board overview. http://www.fsmb.org/siteassets/advocacy/key-issues/continuing-medical- education-by-state.pdf Flett, K. B., Bousvaros, A., Carpenter, J., Millrinen, C. E., Martin, P., & Sandora, T. J. (2018). Reducing redundant anaerobic therapy through spaced education and antimicrobial stewardship interventions. Journal of the Pediatric Infectious Diseases Society, 7(4), 317–322. https://doi.org/10.1093/jpids/pix090 Gibbs, G. R. (2018). Analyzing qualitative data (Vol. 6). Sage. Glesne, C. (2016). Becoming qualitative researchers: An introduction. Pearson. Godin, G., Bélanger-Gravel, A., Eccles, M., & Grimshaw, J. (2008). Healthcare professionals’ intentions and behaviours: A systematic review of studies based on social cognitive theories. Implementation Science, 3. https://doi.org/10.1186/1748-5908-3-36 Goodall, A. H. (2011). Physician-leaders and hospital performance: Is there an association? Social Science & Medicine, 73(4), 535–539. 119 Gooding, H. C., Cheever, E., Forman, S. F., Hatoun, J., Jooma, F., Touloumtzis, C., & Vernacchio, L. (2017). Implementation and evaluation of two educational strategies to improve screening for eating disorders in pediatric primary care. Journal of Adolescent Health, 60(5), 606–611. https://doi.org/10.1016/j.jadohealth.2016.12.002 Gunderman, R., & Kanter, S. L. (2009). Perspective: Educating physicians to lead hospitals. Academic Medicine, 84(10), 1348–1351. https://doi.org/10.1097/ACM.0b013e3181b6eb42 Gupta, A. K. (2019). Physician versus non-physician CEOs: The effect of a leader’s professional background on the quality of hospital management and health care. Journal of Hospital Administration, 8(5). Hadadgar, A., Changiz, T., Dehghani, Z., Backheden, M., Mirshahzadeh, N., Zary, N., & Masiello, I. (2016). A theory-based study of factors explaining general practitioners’ intention to use and participation in electronic continuing medical education. Journal of Continuing Education in the Health Professions, 36(4), 290–294. https://doi.org/10.1097/CEH.0000000000000123 Hadadgar, A., Changiz, T., Masiello, I., Dehghani, Z., Mirshahzadeh, N., & Zary, N. (2016). Applicability of the theory of planned behavior in explaining the general practitioners eLearning use in continuing medical education. BMC Medical Education, 16(1), 1–8. https://doi.org/10.1186/s12909-016-0738-6 Haist, F., Shimamura, A. P., & Squire, L. R. (1992). On the relationship between recall and recognition memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(4), 691–702. https://doi.org/10.1037/0278-7393.18.4.691 120 Haley, M., Lettis, A., Rose, P. M., Jenkins, L. S., Glasziou, P., & Rose, P. W. (2012). Implementing evidence in practice: Do action lists work? Education for Primary Care, 23(2), 107–114. https://doi.org/10.1080/14739879.2012.11494085 Hvidt, E. A., Ammentorp, J., Søndergaard, J., Timmermann, C., Hansen, D. G., & Hvidt, N. C. (2018). Developing and evaluating a course programme to enhance existential communication with cancer patients in general practice. Scandinavian Journal of Primary Health Care, 36(2), 142–151. https://doi.org/10.1080/02813432.2018.1459235 Jeong, D., Presseau, J., ElChamaa, R., Naumann, D. N., Mascaro, C., Luconi, F., Smith, K. M., & Kitto, S. (2018). Barriers and facilitators to self-directed learning in continuing professional development for physicians in Canada: A scoping review. Academic Medicine, 93(8), 1245. Katrova, L. (2017). Leadership or obligation: Value aspects of the professionalization of doctors in global society. EC Dental Science, 7, 106–114. Katzman, J. G., Comerci, G. D., Landen, M., Loring, L., Jenkusky, S. M., Arora, S., Kalishman, S., Marr, L., Camarata, C., Duhigg, D., Dillow, J., Koshkin, E., Taylor, D. E., & Geppert, C. M. A. (2014). Rules and values: a coordinated regulatory and educational approach to the public health crises of chronic pain and addiction. American Journal of Public Health, 104(8), 1356–1362. https://doi.org/10.2105/AJPH.2014.301881 Kitto, S., Bell, M., Goldman, J., Peller, J., Silver, I., Sargeant, J., & Reeves, S. (2013). (Mis)perceptions of continuing education: Insights from knowledge translation, quality improvement, and patient safety leaders. Journal of Continuing Education in the Health Professions, 33(2), 81–88. https://doi.org/10.1002/chp.21169 121 Kogan, J. R., Conforti, L. N., Yamazaki, K., Iobst, W., & Holmboe, E. S. (2017). Commitment to change and challenges to implementing changes after workplace-based assessment rater training. Academic Medicine, 92(3), 394–402. https://doi.org/10.1097/ACM.0000000000001319 Komarraju, A., Bartel, T. B., Dickinson, L. A., Grant, F. D., & Yarbrough, T. L. (2018). Implementation of a flipped classroom for nuclear medicine physician CME. Journal of Continuing Education in the Health Professions, 38(3), 198–204. https://doi.org/10.1097/CEH.0000000000000204 Kotter, J. (2007). Leading change: Why transformation efforts fail. Harvard Business Review, 86, 97-103. Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory into Practice, 41(4), 212–218. https://doi.org/10.1207/s15430421tip4104_2 Krones, T., Keller, H., Becker, A., Sönnichsen, A., Baum, E., & Donner-Banzhoff, N. (2010). The theory of planned behaviour in a randomized trial of a decision aid on cardiovascular risk prevention. Patient Education and Counseling, 78(2), 169–176. https://doi.org/10.1016/j.pec.2009.06.010 Lavoie, K. L., Rash, J. A., & Campbell, T. S. (2017). Changing provider behavior in the context of chronic disease management: Focus on clinical inertia. Annual Review of Pharmacology and Toxicology, 57(1), 263–283. https://doi.org/10.1146/annurev- pharmtox-010716-104952 Légaré, F, Freitas, A., Thompson-Leduc, P., Borduas, F., Luconi, F., Boucher, A., Witteman, H. O., Jacques, A., Légaré, F., Freitas, A., Thompson-Leduc, P., Borduas, F., Luconi, F., Boucher, A., Witteman, H. O., & Jacques, A. (2015). The majority of accredited 122 continuing professional development activities do not target clinical behavior change. Academic Medicine, 90, 197–202. https://doi.org/10.1097/ACM.0000000000000543 Légaré, F., Freitas, A., Turcotte, S., Borduas, F., Jacques, A., Luconi, F., Godin, G., Boucher, A., Sargeant, J., & Labrecque, M. (2017). Responsiveness of a simple tool for assessing change in behavioral intention after continuing professional development activities. PLoS ONE, 12. https://doi.org/10.1371/journal.pone.0176678 Légaré, F., Borduas, F., Freitas, A., Jacques, A., Godin, G., Luconi, F., & Grimshaw, J. (2014). Development of a simple 12-item theory-based instrument to assess the impact of continuing professional development on clinical behavioral intentions. PLoS One, 9(3). Luconi, F., Boillat, M., Mak, S., Chartrand, D., Korah, N., Daly, M., Teferra, M., & Gutberg, J. (2019). Patient safety and quality of care are everybody’s business: evaluating the impact of a continuing professional development program beyond satisfaction. MedEdPublish, 8(1). https://doi.org/10.15694/mep.2019.000046.1 Mahmood, K. (2017). Reliability and validity of self-efficacy scales assessing students’ information literacy skills: A systematic review. The Electronic Library, 35(5), 1035– 1051. https://doi.org/10.1108/EL-03-2016-0056 Maxwell, J. A. (2013). Qualitative research design: An interactive approach (Vol. 41). Sage publications. Mazmanian, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., & Kantrowitz, M. P. (1998). Information about barriers to planned change: A randomized controlled trial involving continuing medical education lectures and commitment to change. Academic Medicine, 73(8), 882–886. 123 Mazmanian, Paul E., Davis, D. A., & Galbraith, R. (2009). Continuing medical education effect on clinical outcomes. Chest, 135(3, Supplement), 49S-55S. https://doi.org/10.1378/chest.08-2518 Mazmanian, Paul E., & Mazmanian, P. M. (1999). Commitment to change: Theoretical foundations, methods, and outcomes. Journal of Continuing Education in Health Professions, 19(4), 200–207. McMahon, G. T. (2017). The leadership case for in continuing professional development. Academic Medicine, 92(8), 1075–1077. https://doi.org/10.1097/ACM.0000000000001619 Merriam, S. B., & Tisdell, E. J. (2016). Qualitative research: A guide to design and implementation. John Wiley & Sons. Miller, S. H. M., Thompson, J. N. M., Mazmanian, P. E., Aparicio, A. M., Davis, D. A., Spivey, B. E. M., & Kahn, N. B. J. (2008). Continuing medical education, professional development, and requirements for medical licensure: A white paper of the Conjoint Committee on Continuing Medical Education. Journal of Continuing Education in the Health Professions Spring 2008, 28(2), 95–98. https://doi.org/10.1002/chp.164 Moore, D. E., Green, J. S., & Gallis, H. A. (2009). Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. Journal of Continuing Education in the Health Professions, 29, 1–15. https://doi.org/10.1002/chp.20001 Ozcan, Y. A. (2008). Health care benchmarking and performance evaluation. Springer Science+ Business Media. 124 Parochka, J., & Paprockas, K. (2001). A continuing medical education lecture and workshop, physician behavior, and barriers to change. Journal of Continuing Education in the Health Professions, 21(2), 110–116. https://doi.org/10.1002/chp.1340210208 Parsa-Parsi, R. W. (2017). The revised declaration of Geneva: A modern-day physician’s pledge. JAMA, 318(20), 1971–1972. https://doi.org/10.1001/jama.2017.16230 Patton, M. Q. (2002). Chapter 7: Qualitative interviewing. In Qualitative research and evaluation methods (3rd ed., pp. 339–380). SAGE. Pelayo, M., Cebrián, D., Areosa, A., Agra, Y., Izquierdo, J. V., & Buendía, F. (2011). Effects of online palliative care training on knowledge, attitude and satisfaction of primary care physicians. BMC Family Practice, 12(1), 1–11. https://doi.org/10.1186/1471-2296-12-37 Prochaska, J. O., & Velicer, W. F. (1997). The transtheoretical model of health behavior change. American Journal of Health Promotion, 12(1), 38–48. https://doi.org/10.4278/0890- 1171-12.1.38 Prosci. (n.d.). The Prosci ADKAR Model. https://empower.prosci.com/the-prosci-adkar-model- ebook-bundle Ramani, S., McMahon, G. T., & Armstrong, E. G. (2019). Continuing professional development to foster behaviour change: From principles to practice in health professions education. Medical Teacher, 1–8. https://doi.org/10.1080/0142159X.2019.1615608 Ratelle, J. T., Bonnes, S. L., Wang, A. T., Mahapatra, S., Schleck, C. D., Mandrekar, J. N., Mauck, K. F., Beckman, T. J., & Wittich, C. M. (2017). Associations between teaching effectiveness and participant self-reflection in continuing medical education. Medical Teacher, 39(7), 697–703. https://doi.org/10.1080/0142159X.2017.1301655 125 Ratelle, J. T., Wittich, C. M., Yu, R. C., Newman, J. S., Jenkins, S. M., & Beckman, T. J. (2017). Relationships between reflection and behavior change in CME. Journal of Continuing Education in the Health Professions, 37, 161–167. https://doi.org/10.1097/CEH.0000000000000162 Ravyn, D., Goodwin, B., & Lowney, R. (2019). Continuous learning in multiple sclerosis care: A qualitative study of the expanded learning model for systems. International Journal of Medical Education, 10, 122–128. https://doi.org/10.5116/ijme.5cfa.29cb Reed, V. A., Schifferdecker, K. E., & Turco, M. G. (2012). Motivating learning and assessing outcomes in continuing medical education using a personal learning plan. Journal of Continuing Education in the Health Professions, 32(4), 287–294. https://doi.org/10.1002/chp.21158 Robeznieks, A. (2014, May 10). Hospitals hire more doctors as CEOs as focus on quality grows. Modern Healthcare. https://www.modernhealthcare.com/article/20140510/MAGAZINE/305109988/hospitals- hire-more-doctors-as-ceos-as-focus-on-quality-grows Robinson, S. B., & Leonard, K. F. (2018). Designing quality survey questions. Sage Publications. Rodriguez, E., Marquett, R., Hinton, L., McBride, M., & Gallagher-Thompson, D. (2010). The impact of education on care practices: An exploratory study of the influence of “action plans” on the behavior of health professionals. International Psychogeriatrics, 22(6), 897–908. https://doi.org/10.1017/S1041610210001031 Ross, P. T., & Bibler Zaidi, N. L. (2019). Limited by our limitations. Perspectives on Medical Education, 8(4), 261–264. https://doi.org/10.1007/s40037-019-00530-x 126 Salinas, G. D. (2015). CME effectiveness: Utilizing outcomes assessments of 600+ CME programs to evaluate the association between format and effectiveness. Journal of Continuing Education in the Health Professions, 35, S38–S39. Salinas, G. D. (2016). Engaging learners in CME in 2016: Understanding CME preferences to increase future participation. Journal of Continuing Education in the Health Professions, 36. https://doi.org/10.1097/CEH.0000000000000089 Salkind, N. J., & Frey, B. B. (2019). Statistics for people who (think they) hate statistics. Sage Publications, Incorporated. Sargeant, J. (2009). Theories to aid understanding and implementation of interprofessional education. Journal of Continuing Education in the Health Professions, 29(3), 178–184. https://doi.org/10.1002/chp.20033 Schostak, J., Davis, M., Hanson, J., Schostak, J., Brown, T., Driscoll, P., Starke, I., & Jenkins, N. (2010). Effectiveness of continuing professional development project: A summary of findings. Medical Teacher, 32(7), 586–592. Schroedl, C., McMahon, G. T., & McGaghie, W. C. (2020). Mastery learning, continuing professional education, and maintenance of certification. In W. C. McGaghie, J. H. Barsuk, & D. B. Wayne (Eds.), Comprehensive Healthcare Simulation: Mastery Learning in Health Professions Education (pp. 331–349). Springer International Publishing. https://doi.org/10.1007/978-3-030-34811-3_18 Schwartz, R. W., & Pogge, C. (2000). Physician leadership: Essential skills in a changing environment. The American Journal of Surgery, 180(3), 187–192. 127 Shanafelt, T. D., Gorringe, G., Menaker, R., Storz, K. A., Reeves, D., Buskirk, S. J., Sloan, J. A., & Swensen, S. J. (2015). Impact of organizational leadership on physician burnout and satisfaction. 90(4), 432–440. Shirazi, M., Lonka, K., Parikh, S. V., Ristner, G., Alaeddini, F., Sadeghi, M., & Wahlstrom, R. (2013). A tailored educational intervention improves doctor’s performance in managing depression: A randomized controlled trial. Journal of Evaluation in Clinical Practice, 19(1), 16–24. https://doi.org/10.1111/j.1365-2753.2011.01761.x Steinert, Y., Naismith, L., & Mann, K. (2012). Faculty development initiatives designed to promote leadership in medical education. A BEME systematic review: BEME Guide No. 19. Medical Teacher, 34(6), 483–503. https://doi.org/10.3109/0142159X.2012.680937 Stevenson, R., & Moore, D. E. (2018). Ascent to the summit of the CME pyramid: Defining outcomes for CME activity. JAMA, 319(6), 543–544. https://doi.org/10.1001/jama.2017.19791 Stoller, J. K., Goodall, A., & Baker, A. (2016, December 27). Why the best hospitals are managed by doctors. Harvard Business Review. https://hbr.org/2016/12/why-the-best- hospitals-are-managed-by-doctors The World Medical Association. (2017). WMA Declaration of Geneva. https://www.wma.net/policies-post/wma-declaration-of-geneva/ Turner, A. D., Stawicki, S. P., & Guo, W. A. (2018). Competitive advantage of MBA for physician executives: A systematic literature review. World Journal of Surgery, 42(6), 1655–1665. https://doi.org/10.1007/s00268-017-4370-3 128 Vandergrift, J. L., Gray, B. M., & Weng, W. (2018). Do state continuing medical education requirements for physicians improve clinical knowledge? Health Services Research, 53(3), 1682–1701. Vuković, M., Gvozdenović, B. S., Ranković, M., McCormick, B. P., Vuković, D. D., Gvozdenović, B. D., Kastratović, D. A., Marković, S. Z., Ilić, M., & Jakovljević, M. B. (2015). Can didactic continuing education improve clinical decision making and reduce cost of quality? Evidence from a case study. Journal of Continuing Education in the Health Professions, 35(2), 109–118. https://doi.org/10.1002/chp.21272 Walker, R., Bennett, C., Kumar, A., Adamski, M., Blumfield, M., Mazza, D., & Truby, H. (2019). Evaluating online continuing professional development regarding weight management for pregnancy using the new world Kirkpatrick model. Journal of Continuing Education in the Health Professions, 39(3), 210–217. https://doi.org/10.1097/CEH.0000000000000261 Weiss, R. S. (1994). Learning from strangers: The art and method of qualitative interview studies. The Free Press. Welie, J. V. M. (2012). Social contract theory as a foundation of the social responsibilities of health professionals. Medicine, Health Care and Philosophy, 15(3), 347–355. https://doi.org/10.1007/s11019-011-9355-7 Wigfield, A., Rosenzweig, E., & Eccles, J. (2017). Achievement values. Handbook of Competence and Motivation: Theory and Application, 116–134. Wilkes, M. S., Day, F. C., Fancher, T. L., McDermott, H., Lehman, E., Bell, R. A., & Green, M. J. (2017). Increasing confidence and changing behaviors in primary care providers 129 engaged in genetic counselling. BMC Medical Education, 17(1), 163. https://doi.org/10.1186/s12909-017-0982-4 Williams, B. W., Kessler, H. A., & Williams, M. V. (2014). Relationship among practice change, motivation, and self-efficacy. Journal of Continuing Education in the Health Professions, 34(S1), S5–S10. https://doi.org/10.1002/chp.21235 Williams, B. W., Kessler, H. A., & Williams, M. V. (2015). Relationship among knowledge acquisition, motivation to change, and self-efficacy in CME participants. Journal of Continuing Education in the Health Professions, 35(S1), S13–S21. https://doi.org/10.1002/chp.21291 Wittich, C. M., Szostek, J. H., Reed, D. A., Kiefer, J. L., Mueller, P. S., Mandrekar, J. N., & Beckman, T. J. (2013). Measuring faculty reflection on medical grand rounds at Mayo Clinic: Associations with teaching experience, clinical exposure, and presenter effectiveness. Mayo Clinic Proceedings, 88(3), 277–284. https://doi.org/10.1016/j.mayocp.2012.11.014 Wittich, C. M., Wang, A. T., Fiala, J. A., Mauck, K. F., Mandrekar, J. N., Ratelle, J. T., & Beckman, T. J. (2016). Measuring participants’ attitudes toward mobile device conference applications in continuing medical education: Validation of an instrument. Journal of Continuing Education in the Health Professions, 36(1), 69–73. https://doi.org/10.1097/CEH.0000000000000031 Wong, Q., Lacombe, M., Keller, R., Joyce, T., & O’Malley, K. (2019). Leading change with ADKAR. Nursing Management, 50(4), 28–35. https://doi.org/10.1097/01.NUMA.0000554341.70508.75 130 Wood, M., George, T. J., Manochakian, R., Polansky, M., Baer, A., Grupe, A., Shyr, Y., Wang, C., & Horn, L. (2019). Quantitative assessment of learning behaviors for oncology providers. Journal of Cancer Education. https://doi.org/10.1007/s13187-019-01593-4 Young, K. J., Kim, J. J., Yeung, G., Sit, C., & Tobe, S. W. (2011). Physician preferences for accredited online continuing medical education. Journal of Continuing Education in the Health Professions, 31(4), 241–246. https://doi.org/10.1002/chp.20136 Zarrabi, B., Burce, K. K., Seal, S. M., Lifchez, S. D., Redett, R. J., Frick, K. D., Dorafshar, A. H., & Cooney, C. M. (2017). Business education for plastic surgeons: A systematic review, development, and implementation of a business principles curriculum in a residency program. Plastic and Reconstructive Surgery, 139(5), 1263–1271. https://doi.org/10.1097/PRS.0000000000003290 131 Appendix A: Interview Questions by Concept Person Question Level of Measurement Response options Knowledge Name the four aims of the Quadruple Aim. Nominal Open-ended Define “gap analysis.” Nominal Open-ended Self-Efficacy Please rate how certain you are that you can identify system-level areas of improvement (vs. individual patient-physician interactions) for the following: a. Patient access b. Quality of care c. Cost of care d. Provider wellness (Adapted from Bandura, 2006) Interval 0 – 10 Scale with… 0 = cannot do at all 5 = moderately can do 10 = highly certain can do Please rate how certain you are that you can create recommendations that would address areas of improvement (vs. individual patient-physician interactions) for the following: a. Patient access b. Quality of care c. Cost of care d. Provider wellness (Adapted from Bandura, 2006) Interval 0 – 10 Scale with… 0 = cannot do at all 5 = moderately can do 10 = highly certain can do Please rate how certain you are that you can apply the gap analysis framework to an organizational dilemma or problem: a. Accurately describe current state b. Develop a vision for the future state c. Accurately articulate knowledge gaps d. Accurately articulate motivation gaps e. Accurately articulate organizational gaps Interval 0 – 10 Scale with… 0 = cannot do at all 5 = moderately can do 10 = highly certain can do 132 Question Level of Measurement Response options f. Use the information obtained to create appropriate recommendations (Adapted from Bandura, 2006) Attitude What are the biggest drivers for you in making these changes in your work? Nominal Open-ended Since the class ended, how, if at all, has your perspective about your work changed? Nominal Open-ended Intention to change At the end of the course, I plan to take action on my recommendations. (Adapted from Légaré et al., 2014; Luconi et al., 2019; Williams et al., 2015) Ordinal 4-point Likert scale: • Strongly disagree • Disagree • Agree • Strongly agree I plan to take action on my recommendations in the future. (Adapted from Légaré et al., 2014; Luconi et al., 2019; Williams et al., 2015) Ordinal 4-point Likert scale: • Strongly disagree • Disagree • Agree • Strongly agree Behavior Question Level of Measurement Response options Of the recommendations you created as part of your final paper in MGT 560, consider the one you have made the most progress toward, if at all. Which statement best reflects your progress: Ordinal ¨ Not started ¨ In progress ¨ Completed How have you used the framework of the Quadruple Aim in your work since last December? Nominal Open-ended How have you used the gap analysis framework since last December? Nominal Open-ended 133 Environment Question Level of Measurement Response options Workplace What have been the most significant barriers in your workplace in implementing your recommendations? Nominal Open-ended What have you been able to do to mitigate these barriers? Nominal Open-ended Considering all of the recommendations you made in the Quadruple Aim paper, what factors in your workplace have most facilitated your progress toward implementing your recommendations? Nominal Open-ended Education Thinking back to your time in the course, what would have made the course better in terms of learning how to analyze your organization? Nominal Open-ended Similarly, what would have made the course better in terms of learning how to create recommendations for your organization? Nominal Open-ended What do you remember being the most effective in learning this content? Nominal Open-ended Miscellaneous Question Level of Measurement Response options Anything that you would like to add? Nominal Open-ended 134 Appendix B: Interview Protocol Pseudonym __________________ Date: ________________ OK to record? _____________ Name the four aims of the Quadruple Aim ¨ Access ¨ Quality ¨ Cost ¨ Provider Wellness Define “gap analysis.” ¨ Current state ¨ Future state ¨ Gap between ¨ Knowledge ¨ Motivation ¨ Organization Please rate how certain you are that you can identify system-level areas of improvement (vs. individual patient-physician interactions) for the following: 0 = cannot do at all 5 = moderately can do 10 = highly certain can do 0 1 2 3 4 5 6 7 8 9 10 Patient access Quality of care Cost of care Provider wellness 135 Please rate how certain you are that you can create recommendations that would address areas of improvement (vs. individual patient-physician interactions) for the following: 0 = cannot do at all 5 = moderately can do 10 = highly certain can do 0 1 2 3 4 5 6 7 8 9 10 Patient access Quality of care Cost of care Provider wellness Please rate how certain you are that you can apply the gap analysis framework to an organizational dilemma or problem: 0 = cannot do at all 5 = moderately can do 10 = highly certain can do 0 1 2 3 4 5 6 7 8 9 10 Accurately describe current state Develop a vision for the future state Accurately articulate knowledge gaps Accurately articulate motivation gaps Accurately articulate organizational gaps Use the information obtained to create appropriate recommendations How much do you agree with the following statement: At the end of the course, I plan to take action on my recommendations. ¨ Strongly disagree ¨ Disagree ¨ Agree ¨ Strongly agree 136 What are the biggest drivers for you in making these changes in your work? Since the class ended, how, if at all, has your perspective about your work changed? Of the recommendations you created as part of your final paper in MGT 560, consider the one you have made the most progress toward, if at all. Which statement best reflects your progress: ¨ Not started ¨ In progress ¨ Completed Considering all of the recommendations you made in the Quadruple Aim paper, what factors in your workplace have most facilitated your progress toward implementing your recommendations? What have been the most significant barriers in your workplace in implementing your recommendations? What have you been able to do to mitigate these barriers? How much do you agree with the following statement: I plan to take action on my recommendations in the future. ¨ Strongly disagree ¨ Disagree ¨ Agree ¨ Strongly agree 137 How, if at all, have you used the framework of the Quadruple Aim in your work since last December? How, if at all, have you used the gap analysis framework since last December? Thinking back to your time in the course, what would have made the course better in terms of learning how to analyze your organization? What do you remember being the most effective in learning this content? Similarly, what would have made the course better in terms of learning how to create recommendations for your organization? (Environment – education) What do you remember being the most effective in learning this content? Anything that you would like to add? 138 Demographic Questions 1. Years of practice Years of practice as a physician (including residency and fellowship): ____ years 2. Setting Which workplace settings best describes where you work? ¨ Single-site organization ¨ Multiple-site organization 3. Location Referring to your practice, where do you see most of your patients? If your time is split, select both. ¨ Clinic, outpatient ¨ Hospital, in-patient 4. Employment type Are you considered an employed physician (employed by the organization directly) or contract (employed by a group who contracts with the organization)? ¨ Employed ¨ Contracted 139 Appendix D: Information Sheet INFORMATION SHEET FOR EXEMPT RESEARCH STUDY TITLE: Exploring the Effectiveness of Continuing Medical Education for Physicians Enrolled in an MBA Program PRINCIPAL INVESTIGATOR: Jessica L. Walter, M.A. FACULTY ADVISOR: Bryant Adibe, M.D. You are invited to participate in a research study. Your participation is voluntary. This document explains information about this study. You should ask questions about anything that is unclear to you. PURPOSE The purpose of this study is to understand how physicians apply their learning from the health systems course to their workplace. We seek to learn what barriers and facilitators you experience when applying classroom learning within your workplace. From this, we hope to learn how to improve the education environment to increase application of course content to the workplace. You are invited as a possible participant because you completed the health systems course in Fall 2019, are a physician, and requested CME credit for the course. PARTICIPANT INVOLVEMENT If you decide to take part, you will be asked to complete one 90-minute interview with the researcher, Jessica Walter. This interview is voluntary, and you may choose to not answer questions. The interview will be conducted virtually using Zoom video conferencing software and will be recorded. Recordings will be transcribed, at which time the recording will be destroyed. You may request to not be recorded. PAYMENT/COMPENSATION FOR PARTICIPATION You will not be compensated for your participation. CONFIDENTIALITY The members of the research team, and the University of Southern California Institutional Review Board (IRB) may access the data. The IRB reviews and monitors research studies to protect the rights and welfare of research subjects. Every endeavor will be taken to keep information confidential. Interview notes will be kept in a locked file cabinet and will not contain identifying information. Recorded interviews will be stored in secure cloud storage (Box.com) and will be deleted upon transcription of the 140 interviews. Transcripts will not include identifying information. Audio will be transcribed using Rev.com and manually reviewed and cleaned up by the researcher. Participants may request to review and provide comments on the transcript. When the results of the research are published or discussed in conferences, no identifiable information will be used. INVESTIGATOR CONTACT INFORMATION If you have any questions about this study, please contact: Jessica Walter, M.A. Researcher JessicaW@usc.edu Bryant Adibe, M.D. Faculty Advisor BAdibe@usc.edu IRB CONTACT INFORMATION If you have any questions about your rights as a research participant, please contact the University of Southern California Institutional Review Board at (323) 442-0114 or email irb@usc.edu. 141 Appendix E: Invitation Email SUBJECT: Interview Invitation (sent on behalf of Jessica Walter) Dear <<First Name>>, I would like to invite you to participate in an interview. This interview is part of a research study exploring how learners apply classroom content to the workplace. I am conducting this study, under the supervision of Bryant Adibe, M.D., in partial fulfillment of my doctoral degree at the USC Rossier School of Education. You are invited to participate because you are a physician who requested CME for MGT 560 Organization, Financing, and History of Healthcare Delivery in the U.S. in Fall 2019. Your participation in this study is completely voluntary. Your interviews will be confidential and will have no impact on your role as a student. You may discontinue participation in the study at any time. If you choose to participate, you will be asked to schedule an interview with me. The interview will take 60 – 90 minutes and will be conducted via Zoom. While there are no payments or compensation associated with participating in this study, your participation will assist me in improving our education programs. Sign-up for an interview using this link. Alternatively, you can email me at jessicaw@usc.edu to schedule a convenient time. For more information, please feel free to contact me at jessicaw@usc.edu or Dr. Bryant Adibe at badibe@usc.edu. Thank you for your consideration. Jessica

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Raising women leaders of Christian higher education: an innovation study

PDF

Primary care physicians' experiences working within the patient-portal to improve the quality of patient care

PDF

The impact of advanced technologies on the workplace and the workforce: an evaluation study

PDF

Developing physician trainees leadership skills: an innovation study

PDF

Evaluation of early career exploration interventions in a medical school professional development program

PDF

Succession for success: an evaluation study of corporate strategy to improve employee satisfaction for women of color

PDF

Managing successful organizational change for faculty in higher education: an innovation study

PDF

Readiness factors influencing the ability of institutions of higher education to align the budget with organizational goals: an evaluation study of a college

PDF

Women of color senior leaders: pathways to increasing representation in higher education

PDF

An improvement study of leading a sustainable electric utility future through organizational change effectiveness

PDF

Creating a culture of connection: employee engagement at an academic medical system

PDF

Opening interpretations: visual literacy and teaching for inclusion, first-year studio faculty innovate for improvement

PDF

Manager leadership skills in the context of a new business strategy initiative: an evaluative study

PDF

Gender role beliefs of male senior leaders in retail and the impact on women’s advancement

PDF

A gap analysis of course directors’ effective implementation of technology-enriched course designs: An innovation study

PDF

Using mastery learning to address gender inequities in the self-efficacy of high school students in math-intensive STEM subjects: an evaluation study

PDF

Academic department chair readiness to lead toward equity: a gap analysis

PDF

Collective Impact: a framework to advance health promotion in higher education

PDF

Implementing organizational change in a medical school

PDF

Factors impacting the effectiveness of mentor teachers in a national teacher residency

Asset Metadata

Creator Walter, Jessica L. (author)

Core Title Exploring the effectiveness of continuing medical education for physicians enrolled in an MBA program

School Rossier School of Education

Degree Doctor of Education

Degree Program Organizational Change and Leadership (On Line)

Publication Date 04/09/2021

Defense Date 03/17/2021

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag continuing medical education,leadership development,Motivation,OAI-PMH Harvest,organizational change,program evaluation,qualitative research,self-efficacy,transfer of learning

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Adibe, Bryant (committee chair), Malloy, Courtney Lynn (committee member), Mursaszewski, Alison Keller (committee member)

Creator Email jessicalwalter@gmail.com,jessicaw@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c89-440572

Unique identifier UC11668782

Identifier etd-WalterJess-9433.pdf (filename),usctheses-c89-440572 (legacy record id)

Legacy Identifier etd-WalterJess-9433.pdf

Dmrecord 440572

Document Type Dissertation

Rights Walter, Jessica L.

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA