University of Southern California Dissertations and Theses

Underrepresented minorities in medicine in the United States: an innovation study to develop an effective holistic admissions process for the New School of Medicine

(USC Thesis Other)

Underrepresented minorities in medicine in the United States: an innovation study to develop an effective holistic admissions process for the New School of Medicine

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content UNDERREPRESENTED MINORITIES IN MEDICINE 1

UNDERREPRESENTED MINORITIES IN MEDICINE IN THE UNITED STATES: AN
INNOVATION STUDY TO DEVELOP AN EFFECTIVE HOLISTIC ADMISSIONS
PROCESS FOR THE NEW SCHOOL OF MEDICINE

By

Linda Davis Tolbert

A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION

May 2020

Copyright 2020 Linda Davis Tolbert

UNDERREPRESENTED MINORITIES IN MEDICINE 2
DEDICATION
I dedicate this endeavor to all past, present and future underrepresented minorities in medicine,
especially my parents, Drs Gilbert & Cora Davis, who were true pioneers.

UNDERREPRESENTED MINORITIES IN MEDICINE 3
ACKNOWLEDGEMENTS
I would like to thank everyone who supported me during this process:

1. My family – my beloved husband Will … my rock, and an underrepresented minority in
medicine as well, who unexpectedly passed away a month before my dissertation
defense. He is smiling. And to our children, Bianca, Bridgette & Billy, who were tolerant,
supportive and proud of their Mom.
2. Each of my professors during the program. They gave their heart and soul while
imparting great wisdom.
3. My Committee – Dr. Patricia Tobey (Chair), Dr. Darline Robles and Dr. Ravneet Tiwana,
who had an unwavering belief in me and engulfed me with the Trojan spirit and support
and encouraged me to “Fight On.”
4. The Dean, Associate Deans and Staff of the “New School of Medicine,” who walked the
talk of inclusion by welcoming me into the School, granting me access and trusting my
efforts.
5. The cohort of Admissions File Reviewers who graciously participated in this study.
6. My extended family and friends who supported my goals and understood when my time
was limited.
7. Those who recommended me for the EdD program at one of the finest institutions in the
world.
8. My formatter, Guadalupe Montano.
9. My classmates, who added so much to this process and made it fun!

THANK YOU!!!

UNDERREPRESENTED MINORITIES IN MEDICINE 4
TABLE OF CONTENTS
Dedication 2
Acknowledgements 3
List of Tables 7
List of Figures 9
Abstract 10
Chapter One: Introduction 11
Introduction to the Problem of Practice 11
Organizational Context and Mission 11
Organizational Performance Goal 12
Background of the Problem 13
The Importance of Addressing the Problem 14
Description of Stakeholder Groups 15
Stakeholders’ Performance Goals 16
Stakeholder Group of Focus 16
Purpose of the Study 17
Research Questions 17
Key Definitions 18
Organization of the Paper 18
Chapter Two: Review of the Literature 20
General Review of the Literature 20
The Mission of Medical Schools and Health Care Equity 20
Role of the Association of American Medical Colleges 21
Reasons for the Lack of Underrepresented Minorities in U.S. Medical Schools 22
Less Academic Preparation 22
Less Exposure to and Knowledge of the Medical Field 23
Fewer Financial Resources 23
Lower Academic Performance 23
Racial Discrimination 24
Implicit Bias in Medical School Admissions 24
Less Social Support and Networking Leading to a Lesser Quality of Life 25
Mitigating the Gap of Underrepresented Minorities in U.S. Medical Schools 26
Academic, Social and Financial Mentoring and Support 26
Financial Support 27
The Lack of Underrepresented Minorities in U.S. Medical School Faculty 27
Current and Historical Perspective 27
Impact of Too Few URM Medical Faculty on Recruitment, Mentoring and Retention of
URM Medical Students 28
The Importance of Cultural Competency Teachings and Diversity to Care Delivery 29
Cultural Competence 29
The Clark and Estes (2008) Gap Analysis Conceptual Framework 29
Knowledge Influences 30
Motivation 36
Organization 39

UNDERREPRESENTED MINORITIES IN MEDICINE 5
Conceptual Framework: The Interaction of Stakeholders’ Knowledge and Motivation
and the Organizational Context 47
Chapter Three: Methodology 54
Introduction to the Methodology 54
Participating Stakeholders 54
Interview Sampling (Recruitment) and Rationale 55
Observation Sampling Criteria and Rationale 55
Observation Sampling (Access) Strategy and Rationale 56
Explanation for Choices 56
Data Collection and Instrumentation 57
Qualitative Data Collection and Instrumentation 57
Documents 58
Observation 59
Observation Procedures 60
Interviews 61
Data Analysis 63
Credibility and Trustworthiness 63
Ethics 65
Limitations and Delimitations 68
Chapter Four: Results and Findings 69
Overview of Purpose and Questions 69
Research Questions 69
Participating Stakeholders 69
Knowledge 73
Motivation 80
Organization 84
Chapter Five: discussion and Recommendations 90
Knowledge Recommendations 90
Motivation Recommendations 94
Organization Recommendations 96
Integrated Implementation and Evaluation Plan 101
Implementation and Evaluation Framework 101
Organizational Purpose, Needs and Expectations 102
Level 4: Results and Leading Indicators 102
Level 3: Behavior 103
Level 2: Learning 106
Level 1: Reaction 109
Evaluation Tools 110
Data Analysis and Reporting 111
Summary 111
Future Study 112
Conclusion 112
References 114
Appendix A: Letter to Stakeholder Group From the Associate Dean of Admissions
Introducing the Researcher 127
Appendix B: Interview Protocol 128
UNDERREPRESENTED MINORITIES IN MEDICINE 6
Appendix C: Observation Checklist 132
Appendix D: Immediate Evaluation Instrument 133
Appendix E: Blended Instrument 135
Appendix F: Key Performance Indicators 137

UNDERREPRESENTED MINORITIES IN MEDICINE 7
LIST OF TABLES
Table 1: Organizational Mission, Global Goal and Stakeholder Performance Goals 16
Table 2: Knowledge Influences, Types, and Assessments for Knowledge Gap Analysis 36
Table 3: Motivational Influences and Assessments for Motivation Gap Analysis 39
Table 4: The Organizational Mission and Goal, Stakeholder Goal, Organization Influences
and Assessments 45
Table 5: Summary Table of Assumed Influences on Performance 46
Table 6: Sampling Strategy and Timeline 57
Table 7: Validated Declarative Knowledge Need 73
Table 8: Validated Procedural Knowledge Need 78
Table 9: Validated Metacognitive Knowledge Need 79
Table 10: Validated Motivation Need – Expectancy/ Task value 81
Table 11: Validated Motivation Need: Self-Efficacy 83
Table 12: Validated Organization Need - Trust 84
Table 13: Validated Organizational Need: Shared Belief 86
Table 14: Validated Organization need: Provision of Professional Development Training 87
Table 15: Summary of Knowledge Needs and Recommendations 90
Table 16: Summary of Motivation Needs and Recommendations 94
Table 17: Organization Need and Recommendations – Cultural Model: Trust 96
Table 18: Organization Need and Recommendations – Cultural Model: Shared Belief 98
Table 19: Organization Need and Recommendations -Cultural Setting – Provision of PD
Training 99
Table 20: Outcomes, Metrics, and Methods for External and Internal Outcomes 103
Table 21: Critical Behaviors, Metrics, Methods, and Timing for Evaluation 104
Table 22: Required Drivers to Support Critical Behaviors 105
Table 23: Components of Learning for the Program. 108
UNDERREPRESENTED MINORITIES IN MEDICINE 8
Table 24: Components to Measure Reactions to the Training. 109

UNDERREPRESENTED MINORITIES IN MEDICINE 9
LIST OF FIGURES
Figure 1: Interaction of Stakeholder Knowledge and Motivation within Organizational
Cultural Models and Settings. 49
Figure 2: AFR cohort by gender. 70
Figure 3: AFR cohort by ethnicity. 71
Figure 4: AFR participants by location. 71
Figure 5: AFR cohort by underrepresented minority in medicine – U.S. 72
Figure 6: AFR participants by years as a physician. 72
Figure 7: Important non-academic qualities for medical students to have identified by AFR
cohort. 75

UNDERREPRESENTED MINORITIES IN MEDICINE 10
ABSTRACT
This study addressed the underrepresentation of minorities in medicine in the United
States. The Association of American Medical Colleges defines underrepresented minorities in
medicine (URiM) as “those ethnic and racial populations that are underrepresented in the
medical profession relative to their numbers in the general population.” These are the African
American, Hispanic or Latino, American Indian or Alaska Native, and Hawaiian or Pacific
Islander groups. In 2011, these groups made up 31% of the United States population, 15% of the
United States medical student body, and 8.8% of the total medical faculty. This is a problem
because these racial and ethnic groups are also an increasingly large segment of the population to
receive health care, and physician diversity is integral to the effective delivery of care to all
populations and to the elimination of health care disparities. The index organization of this
innovation study was the New School of Medicine (NSM) which welcomed its inaugural class in
2020. The organizational performance goal was to design a holistic admissions program to
effectively matriculate, retain and graduate a diverse class of students, including
underrepresented minoritized groups who have traditionally had lower matriculation rates and
higher attrition rates. The target was to have all Admissions File Reviewers (AFRs) able to
conduct a holistic review process (HRP) one year prior to the matriculation of the inaugural class
of the New School of Medicine (NSM). The knowledge, motivation and organizational needs of
the AFRs were analyzed and findings published to enable future refinement of HRPs to further
the creation of diverse classes and mitigate both the underrepresentation gap and the health care
disparities gap.
Keywords: Physician diversity; underrepresented minorities in medicine (URiM);
underrepresentation gap; health care disparities gap; medical school admissions; holistic
admissions; holistic review.

UNDERREPRESENTED MINORITIES IN MEDICINE 11
CHAPTER ONE: INTRODUCTION
Introduction to the Problem of Practice
This study addressed underrepresented minorities in medicine (URiM) in the United
States. The Association of American Medical Colleges (2004) defines underrepresented
minorities in medicine (URiM) as “those ethnic and racial populations that are underrepresented
in the medical profession relative to their numbers in the general population” (p. 1). These are
the African American, Hispanic or Latino, American Indian or Alaska Native, and Hawaiian or
Pacific Islander groups. In 2011, these groups made up 31% of the U.S. population, 15% of the
medical student body, and 8.8% of the total medical faculty (Association of American Medical
Colleges, 2012). In 2017, despite a higher absolute number of underrepresented minorities, the
gap widened to nearly 70% for Hispanics, nearly 60% for African American males, and nearly
40% for African-American females. This is a problem because, as the fastest growing segment of
the population projected to constitute the majority by 2050, these racial and ethnic groups are
also an increasingly large segment of the population to receive health care (Kelly, 2015). Also,
because more minority physicians pursue primary care specialties and practice in underserved
communities than non-minority physicians (Derck, Zahn, Finks, Mand, & Sandju, 2016), the
demographic gap is these communities is widened when there are not enough minority
physicians in the workforce. The problem of the underrepresentation of minorities in medicine
was important to address because physician diversity is integral to the effective delivery of care
to all populations and to the elimination of health care disparities (Derck et al., 2016).
Organizational Context and Mission
The index organization of this innovation study was the New School of Medicine (NSM),
a national medical school in California, which welcomed its’ inaugural class of 48 students in
UNDERREPRESENTED MINORITIES IN MEDICINE 12
2020. The School is affiliated with a large hospital system, and health maintenance organization,
integrated and exclusively contracted with medical groups under one umbrella.
The funding for the medical school is derived primarily from community benefit monies
as a result of the non-profit status of the health plan. The organizational performance goal of the
New School of Medicine was to effectively matriculate, retain and graduate a diverse class of
students, including underrepresented minoritized groups who have traditionally had lower
matriculation rates and higher attrition rates. The research performance need of focus was the
design of a holistic admissions program to effectively matriculate, retain and graduate
underrepresented minority students. The target was to have all AFRs able to conduct a holistic
review process by July of 2019, one year prior to the matriculation of the inaugural class, to
facilitate the matriculation of a diverse class each year. This need aligned with the organization’s
mission “… an unwavering commitment to improve the health and well-being of patients and
communities,” and was related to the larger problem of underrepresented minorities in medicine
in the United States.
Organizational Performance Goal
A goal of the New School of Medicine was to matriculate, retain and graduate 100% of a
diverse student body comprised of a higher than the usual 15% percent representation of URiMs.
This goal was established by the medical school leadership in accordance with the usual
expectations and standards of a four-year medical school curriculum set by the American
Association of Medical Colleges, and uniformly sought by medical colleges in the United States.
Benchmarked measures used to track progress towards the goal include the admissions data
percentages of the URiMs, performance on periodic internal examinations and clerkships, as well
as performance on national medical examinations. The organizational performance goal was
UNDERREPRESENTED MINORITIES IN MEDICINE 13
related to solving the larger problem of the disproportionate underrepresentation of specific
minority groups in the United States healthcare workforce, which impedes access to care for
underrepresented minority populations and widens the health care disparities gap.
Background of the Problem
Evidence obtained by the Bureau of Health Professions (2006), showed that
underrepresented minority students from under-resourced K-12 schools frequently lacked
resources, test-taking skills and knowledge, due to years of educational inequalities, and because
competitive scores are usually required for admission to college and medical school, many of
these students are automatically eliminated (Valentine, Wynn, & McLean, 2016). Undergraduate
underrepresented minority students report having inadequate access to information, resources,
mentoring, and assistance with key career decisions, stating that even when information is
obtained through various on-campus venues, it is often obtained too late to act upon (Freeman,
Landry, Trevino, Grande, & Shea, 2015). In addition, undergraduate underrepresented minority
students often have fewer personal funds, more difficulties obtaining loans and less knowledge
about how to fill out financial applications (Valentine et al., 2016), reporting significant financial
stress because of having to pay for admissions tests, books, school applications and clothes for
interviews (Freeman et al., 2015). Once enrolled in medical school, there is an increased risk of
withdrawal, leave of absence or dismissal, and of delay in graduation from school associated
with URiM status, and 18% of URiM students experience at least one of these events by the end
of their fourth year, as compared to 3% of the non-URiM students (Huff & Fang, 1999). Racial
discrimination has also been found to be a significant variable for URiM students, with a multi-
center study showing that 68.4% of Blacks and 40% of Hispanics experienced such
discrimination, versus only 3% of whites (Mangus, Hawkins, & Miller, 1998).
UNDERREPRESENTED MINORITIES IN MEDICINE 14
The Importance of Addressing the Problem
It is important to address the problem of underrepresented minorities in medicine in the
United States to improve clinical care and reduce health disparities (Sullivan & Mittman, 2010).
Physician diversity is integral to the effective delivery of care to all populations and to the
elimination of health care disparities (Derck et al., 2016). Research has shown that when patients
feel culturally and linguistically comfortable with their doctors, they are more likely to follow
medical advice which leads to improved health outcomes (Vollman, 2015) and increased patient
satisfaction (Laveist & Nuru-Jeter, 2002).
Addressing the problem will likely also positively impact the representation of minorities
in academic medicine and foster matriculation of URiM students into an environment which will
allow them to thrive (Deas et al., 2012). It has been shown that having more diverse medical
school classes increases students’ cognitive skills (Gurin, Dey, Hurtado, & Gurin, 2002),
engenders more expansive classroom discussions and better prepares majority students to
function in multicultural settings (Whitla et al., 2003). These factors develop cultural
competency and foster an attitude in the students of equitable access to care for all (Saha, Guiton,
Wimmers, & Wilkerson, 2008). Additionally, available recruitment and mentoring opportunities
would commensurately increase, as would the pipeline of URiM students into medical school
and ultimately, of underrepresented minority physicians into the United States healthcare
workforce (Sullivan & Mittman, 2010).
Cultural transformation in academic medicine is needed to achieve global recognition
that diversity emphasis should not be limited to representation of race and ethnicity of certain
subsets of individuals, but rather to promote overall excellence that abounds in heterogeneous
groups (Fine & Handelsman, 2010). This conceptual reframing acknowledges that diversity
UNDERREPRESENTED MINORITIES IN MEDICINE 15
enhances the experience of all medical students, faculty and patients and becomes a core
ingredient and catalyst for excellence in research, teaching and clinical practice (Nivet, 2009).
The impetus then becomes one of removing barriers to institutional excellence rather than a
cursory tokenistic approach to create diversity, and the opportunity to develop measurable goals
is afforded (Nivet, 2009). To garner such a level of engagement surrounding this “diversity as a
driver of excellence" rationale aligns with the approach of Sullivan (2004) to have appropriate
leadership “push the agenda” and requires the intentional leadership of board members,
presidents, deans and department chairs of academic health centers to hold their institutions
accountable for diversifying the faculty. Notably, the biggest correlate of URiM faculty
representation is the number of URiM medical students ten years prior (Page, Page, & Wright,
2011).
Description of Stakeholder Groups
The index organization for this study was the New School of Medicine (NSM), and the
organizational goal was to embed diversity, equity, and inclusion within the medical school
curriculum and the entire school, with a commitment to improve the well-being of patients and
their communities. There were four stakeholder groups:
1. The NSM Board of Directors who endorsed this organizational goal as part of the
overarching goal of the School “to foster an inclusive culture in alignment with the vision
that the graduates of the NSM will be “a diverse community of compassionate healers…”
The Board held the administration accountable for the achievement of this goal.
2. The Admissions File Reviewers (AFRs) – the group charged with the task to effectively
matriculate a diverse student body in alignment with the organizational goal.
UNDERREPRESENTED MINORITIES IN MEDICINE 16
3. Faculty –significant contributors as diverse representatives themselves by role-modeling
and practice.
4. Students – upon enrollment and in the class environment, students will have the inherent
capacity to generate and perpetuate a culture of inclusivity, the level attained to be
measured after two years of school and then yearly thereafter.
Stakeholders’ Performance Goals
Table 1
Organizational Mission, Global Goal and Stakeholder Performance Goals
Organizational Mission
To provide a unique medical education, embedded in a physician-led health care delivery
system that ignites a passion for learning, a desire to serve, and an unwavering commitment
to improve the health and well-being of patients and communities.
Organizational Performance Goal
To inspire a collaborative culture at the New School of Medicine, committed to embedding
diversity, equity, and inclusion throughout the School, with admissions, student services,
hiring, and data reporting systems in place. The target was to have all Admissions File
Reviewers (AFRs) able to conduct a holistic review process by July of 2019, one year prior to
the matriculation of the inaugural class of NSM, to effectively matriculate a diverse class.
Admissions File Reviewers
(AFRs)
Administration & Faculty NSM Board &
Administration
By July 2019, one year prior
to the matriculation of the
inaugural class of NSM, the
Admissions File Reviewers
(AFRs) will conduct and
iteratively refine, a holistic
review process that supports
admitting a measurably
diverse class each year.
By July 2019, one year prior
to the matriculation of the
inaugural class of NSM, the
Admissions File Reviewers
(AFRs) will conduct and
iteratively refine, a holistic
review process that supports
admitting a measurably
diverse class each year.
Continue to attract and
develop, now and ongoing,
diverse, culturally
competent leaders, faculty,
and staff who themselves
will be representative of
our communities and will
foster an inclusive,
collaborative, and
accountable culture.

Stakeholder Group of Focus
While the joint efforts of the stakeholders contribute to the achievement of the overall
organizational goal, the stakeholder group of focus of this study was the group of 19 Admissions
File Reviewers (AFRs). The rationale was that a significant way to embed diversity into the
UNDERREPRESENTED MINORITIES IN MEDICINE 17
school was to have a diverse student body. The target of the AFRs was to develop an effective
holistic review process with all AFRs able to conduct a holistic review process by July of 2019,
to drive the attainment of a diverse class of matriculants for the inaugural 2020 year and
thereafter. This measure was chosen because it demonstrated learning and effective application/
transfer of the attributes of equity and inclusion. The consequence of not achieving the goal was
the diminished capacity to effectively matriculate a diverse student body impacting effective care
delivery to patients and impeding educational excellence. Faculty and students, significant future
determinants of success for the School, were not integral at the outset to facilitate the stated
organizational goal.
Purpose of the Study
The purpose of this study was to conduct a needs’ analysis, utilizing the Clark and Estes’
(2008) gap analysis model, to ascertain the knowledge and skill, motivation, and organizational
resources necessary to attain the organizational performance goal of effectively matriculating a
diverse student body in numbers proportional to their representation in the general population.
The analysis began by generating a list of assumed needs with systematic validation of those
needs. While a complete needs’ analysis would have focused on all stakeholders, for practical
purposes, the stakeholder group of focus for this study was the group of AFRs, charged with the
initial file review of the applicants to the NSM. The research questions that guided this
innovation study follow.
Research Questions
1. What are the knowledge, motivation and organizational needs required for the New
School of Medicine (NSM) Admissions File Reviewers (AFRs) to develop and
effectively enact a holistic review process to matriculate a diverse class each year?
UNDERREPRESENTED MINORITIES IN MEDICINE 18
2. What is the interaction between the AFRs knowledge and motivation and the NSM
organizational culture and context?
Recommended solutions to the knowledge, motivation and organizational needs will be
elucidated based on the above and discussed in Chapter Five.
Key Definitions
Health Disparities – differences and/or gaps in the quality of health and healthcare across
racial, ethnic, and socio-economic groups. It can also be understood as population-specific
differences in the presence of disease, health outcome, or access to healthcare.
Holistic Review – a flexible, individualized way of assessing an applicant’s capabilities
by which balanced consideration is given to experiences, attributes, and academic metrics and,
when considered in combination, how the individual might contribute value as a medical student
and physician.
Underrepresented Minorities in Medicine - those ethnic and racial populations that are
underrepresented in the medical profession relative to their numbers in the general population.
These are the African American, Hispanic or Latino, American Indian or Alaska Native,
and Hawaiian or Pacific Islander groups.
Organization of the Paper
Five chapters are used to organize this study. This chapter provided the reader with the
key concepts and terminology commonly found in a discussion related to underrepresented
minorities in medicine and ramifications of same. In addition, the index organization’s mission
and goals were introduced along with key stakeholders and their aligned goals, particularly the
development of a holistic review admissions process. Chapter Two provides a review of current
UNDERREPRESENTED MINORITIES IN MEDICINE 19
literature surrounding the root causes of underrepresentation of minorities in medicine and
examines relevant policies and potential interventions. Chapter Three details the assumed needs
for this study as well as methodology regarding choice of participants, data collection and
analysis. Chapter Four details the findings of the study and Chapter Five provides solutions
based on data and literature to address the needs and recommendations for an implementation
plan and ongoing evaluation.

UNDERREPRESENTED MINORITIES IN MEDICINE 20
CHAPTER TWO: REVIEW OF THE LITERATURE
Chapter Two examines the root causes of the representation gap of URiMs in medicine in
the U.S. The review begins with research showing disproportionately fewer URiMs in medical
school than their non-URiM counterparts, vastly attributed to less academic preparation, less
exposure to and knowledge of the medical field, fewer financial resources and an uneven playing
field in the admissions process. This is followed by an overview of literature showing that, of
those URiMs admitted to medical school, a disproportionate number compared to non-URiMs
experience graduation delays or failure, attributed to factors including lower academic
performance, social isolation and racial discrimination in the learning environment, and to a
lesser sense of support and quality of life. The review then presents an in-depth discussion of the
underrepresentation of minorities in academic medicine. Following the general research
literature, the review turns to the Clark and Estes (2008) Gap Analysis Conceptual Framework
and, specifically, knowledge, motivation and organizational influences on medical school
admission review committees’ ability to implement holistic processes to level the playing field
for the matriculation of underrepresented minorities.
General Review of the Literature
The Mission of Medical Schools and Health Care Equity
Medical schools have a comprehensive research, educational, patient-care and social
mission to create culturally competent physicians who have the knowledge, skill, behavior and
attitude to provide the best possible care for all people (Nivet, 2010; Pololi, Cooper, & Carr,
2010). In the context of underrepresentation of minorities in medicine, health disparities in the
United States are among the highest in the developed world and reducing them is a major priority
(AHRQ, 2004). However, as Beltran (2003) noted, although health equity remains an often-
UNDERREPRESENTED MINORITIES IN MEDICINE 21
discussed ideal, it has not found resonance nor support with the United States public. Health
reform and a more equitable health system requires a central, transformational role for diversity
and inclusion as a solution, alongside strategies of how we finance, deliver, and evaluate health
care (Nivet, 2015). In addition to the principles of equity and social justice promoting equal
access to health professions for all, the United States will create a richer culture by developing
the talent, creativity and potential of diverse people in all segments of society (Sullivan &
Mittman, 2010).
Role of the Association of American Medical Colleges
The Association of American Medical Colleges (AAMC) has been committed to
increasing diversity in the health workforce for decades (Terrell, 2006). In 1991, the AAMC
launched Project 3000 by 2000, a national campaign endorsed by the deans of the U.S. medical
schools to increase the number of URiMs matriculating annually to medical school from then
1,584 to 3000 by the year 2000. At the inception of the campaign, the U.S. was operating under
the holding of the 1978 Supreme Court case, Regents of the University of California v. Bakke,
which allowed schools to take race into account in admissions and which spurred on affirmative
action policies in higher education. Indeed, the AAMC reported significantly increased minority
enrollment until 1996: 6.4% to 9% for Blacks and 3.9% to 7.2% for Hispanics. In 1996,
however, affirmative action was attacked by the 5th Circuit Court ruling in Hopwood v Texas
and jurisdictions from taking race into account in their admissions policies. The impact was
massive as more than half of Hispanics in the U.S. live in Texas and California and it is
estimated that California’s “social clock” was set back 25 years (Tienda, 2001). Minority
enrollment declined from 2,340 to 1,922 between the 1995-96 and 2000-01 school years and thus
the goal of Project 3000 by 2000 was not met (AAMC Data Book, 21). Despite the 2003
UNDERREPRESENTED MINORITIES IN MEDICINE 22
Supreme Court affirmative action decision in Grutter v. Bollinger et al, in which the compelling
state interest of promoting diversity was upheld, the climate for using affirmative action as a tool
to promote diversity remains tenuous. Another core strategy of the Project 3000 by 2000
initiative, still functioning today despite massive Federal budget cuts, was the creation of 26
collaborative sites with health professions and K-12 schools. The goal of the initiative was to
mitigate the unequal access to educational opportunities in primary and secondary schools for
low-income minority students, thought to be the root cause of underrepresentation of minorities
in medical schools (Cohen, Gabriel & Terrell, 2002).
Reasons for the Lack of Underrepresented Minorities in U.S. Medical Schools
Underrepresentation of minorities in medical school occurs because disproportionately
fewer minority applicants are admitted than their majority counterparts, often because they have
less academic preparation, less knowledge of the medical field and fewer financial resources and,
when enrolled, underrepresented minority students report social isolation, racial discrimination
and less support leading to more graduation delays and attrition than their majority counterparts.
Less Academic Preparation
Evidence obtained by the Bureau of Health Professions (2006), showed that
underrepresented minority students from under-resourced K-12 schools frequently lacked
resources, test-taking skills and knowledge, due to years of educational inequalities, negatively
impacting their matriculation to and performance in college. Unable to perform well in college,
many underrepresented minority students are automatically eliminated from admission
consideration for medical school because competitive scores are usually required. (Valentine et
al., 2016).

UNDERREPRESENTED MINORITIES IN MEDICINE 23
Less Exposure to and Knowledge of the Medical Field
Minority applicants often have less knowledge about and exposure to the medical field as
compared with their majority counterparts In focus groups with undergraduate underrepresented
minority students participating in the 2012-2013 “Tour for Diversity in Medicine,” a program for
minority physicians to have conversations with underrepresented minority students to stimulate
their interest in health care professions, the students reported having inadequate access to
information, mentoring, and assistance with key career decisions (Freeman, Landry, Trevino,
Grande, & Shea, 2015). The students also described limited institutional resources, few clinical
opportunities and stated that even when information was obtained through various on-campus
venues, it was often obtained too late to act upon (Freeman et al., 2015).
Fewer Financial Resources
Another issue that is more prevalent in URiM students than in their non-URiM
counterparts, is the lack of financial resources. Focus group participants at the “Tour for
Diversity in Medicine” reported significant financial stress because of having to pay for
admissions tests, books, school applications and clothes for interviews (Freeman et al., 2015).
More often than majority students, underrepresented minority students have fewer personal
funds, more difficulties obtaining loans and less knowledge about how to fill out financial
applications (Valentine et al., 2016).
Lower Academic Performance
Studies have shown URiM students score lower on standardized licensing exams and are
more likely to fail than non-URiM students (Orom, Semalulu, & Underwood, 2013). In addition,
the researchers found URiM students had lower GPAs, performed less well on periodic
UNDERREPRESENTED MINORITIES IN MEDICINE 24
assessments and in clerkships and were more likely to experience failure and delays than their
majority peers. A study of medical students in 1992 showed that URiM student status was
associated with an increased risk of withdrawal, leave of absence or dismissal from school or, a
delay in graduation (Huff & Fang, 1999). The study showed that 18% of the URiM students
experienced at least one of these events by the end of their fourth year as compared to 3% of the
non-URM students. In addition, URiM students are disproportionately over-represented in
decelerated medical school programs, comprising 37% of the students in the programs (McGrath
& McQuail, 2004).
Racial Discrimination
Compared to majority counterparts, underrepresented minority medical students
experience more racial discrimination, and less satisfaction with their learning environment,
social networking and level of support received (Orom et al., 2013). A multi-center study from
1996 showed that, whereas 3% of whites reported experiencing racial discrimination, 68.4% of
Blacks and 40% of Hispanics reported such discrimination (Mangus et al., 1998). These facts
help explain the underrepresentation of minorities in medicine in terms of discriminatory
treatment impacting academic success and graduation rates.
Implicit Bias in Medical School Admissions
Implicit white race preference has been associated with discrimination in the education,
criminal justice, and health care systems potentially impeding underrepresented minority entry
into the medical profession (Capers, Clinchot, McDougle, & Greenwald, 2017).Because little
was known about unconscious bias in medical school admissions, Capers et al (2017) sought to
determine the presence and extent of unconscious racial bias on their admissions committee by
having all members take the implicit association test (IAT). The authors later surveyed the
committee members to determine their impressions of their individual results and whether
UNDERREPRESENTED MINORITIES IN MEDICINE 25
they thought it was a useful exercise. They also sought to determine whether having admissions
committee members take the IAT would have any impact on URM diversity in the next
incoming class. The authors found that admissions committee members at their school displayed
significant implicit white preference. Most of the committee members thought that the exercise
was valuable and might be helpful to reduce bias and some reported that it did impact their
admissions decisions in the next cycle (Capers et al., 2017).
Less Social Support and Networking Leading to a Lesser Quality of Life
In a 1996 national survey of fourth year medical students, URM students reported more
difficulty establishing support and social networks than non-URM students (25.4% versus
14.5%), as well as more difficulty establishing good peer working relationships (12.9% versus
6%) (Bright, Duefield, & Stone, 1998). In a single-institution study of first-year medical
students, underrepresented minority students were less satisfied than their white counterparts
regarding the overall learning environment, the timeliness of their performance evaluations and
the responsiveness of faculty to students’ concerns. (Robins, Gruppen, Alexander, Fantone &
Davis, 1997). These two studies were further corroborated by research published in 2007 by
Odom, Roberts, Johnson, and Cooper, which determined lack of social support and experiences
with racial stereotyping and discrimination to be among inhibitors of success. These factors
contribute to the lower level of personal satisfaction and quality of life experienced by
underrepresented minority medical students.
In a study evaluating burnout, depression, and quality of life in minority and nonminority
United States medical students, both sets of students had similar degrees of burnout and
depression, but the minority group had a lower sense of personal accomplishment (Dyrbye,
Thomas, Huschka, & Lawson, 2006). The researchers have associated this finding with negative
UNDERREPRESENTED MINORITIES IN MEDICINE 26
self-image and lesser coping mechanisms and suggest that these quality of life differences
experienced by URiM medical students during training contribute to their higher attrition rate.
Mitigating the Gap of Underrepresented Minorities in U.S. Medical Schools
Although these disadvantages exist for URiM medical students, they can be offset by
appropriate interventions, thereby helping to level the playing field. Providing effective
academic and social mentoring, as well as financial support through school can mitigate the
attrition gap between URiM and non-URiM medical students, thus increasing diversity in the
healthcare workforce.
Academic, Social and Financial Mentoring and Support
The identification of high school and college underrepresented minority students
interested in medicine through effective pipeline and post baccalaureate premedical programs
along with the provision of academic, social and financial support, increases their medical school
admission and graduation rates. A student-led initiative at the University of Kentucky College of
Medicine to increase recruitment of underrepresented minority students to their medical school
resulted in the program participants making up 25% of matriculating students over the six years
since inception in 2010 (Achenjang & Elam, 2016). Similarly, a post baccalaureate program at
Wayne State University School of Medicine boasts medical school completion rates between
1969-1992, of 83% for black student graduates and 94% for other racial and ethnic groups
(Cohen & Steinecke, 2006). Additionally, the Premedical Honors College, an eight-year high-
school-through-medical- school program created by Baylor College of Medicine and the
University of California post baccalaureate premedical programs are examples of successful
programs that have improved matriculation of racial and ethnic minorities into medical school
(Cantor, Bergeisen, & Baker, 1998; Grumbach & Chen, 2006; Thompson, Ferry, King,
Martinez-Wedig, & Michael, 2003). The authors note that, among the continuum of educational
UNDERREPRESENTED MINORITIES IN MEDICINE 27
pipeline programs, post baccalaureate interventions are relatively high yield, only a single year
of intervention targeted to students with a demonstrated interest and short timeline to achieve
their results.
Social Mentoring
Mentoring of underrepresented minority students in high school, college and medical
school increases recruitment and retention. A partnership between the University of Toledo and
the local chapter of the Student National Medical Association yielded almost a 50% increase in
medical student enrollment in one year (Rumala & Cason, 2007). In addition, community-based
not-for-profit pre-professional programs such as “Mentoring in Medicine,” which provide
education and mentoring primarily for disadvantaged students and families, have succeeded in
helping thousands of underrepresented minority students achieve their goals (Figueroa, 2014).
Financial Support
Providing financial report to underrepresented minority students in pipeline programs and
in medical school increases the number of URiM students entering and graduating from medical
school. In focus group forums, URiM students mentioned that financial concerns created
additional challenges to achieving their academic and professional pursuits (Odom et al., 2007).
The students also reported feeling that the reliance on student loans versus family support
disadvantaged them as compared to non-URiM medical students and they identified scholarship
opportunities as an important aid to enable attendance and success in medical school.
The Lack of Underrepresented Minorities in U.S. Medical School Faculty
Current and Historical Perspective
The disadvantaged position that minority faculty members find themselves compared
with whites, stems from a systematic and cumulative tradition of segregation, culture and elitism
in academic medicine (Clark & Corcoran, 1986). Just as a gap exists between the 15% URM
UNDERREPRESENTED MINORITIES IN MEDICINE 28
medical students in 2011 relative to the 31% URM in the population at large, the lower
proportion of URM medical faculty at 8.8% in 2011 demonstrates an even more dramatic gap,
likely due in part to vestiges of discriminatory treatment (AAMC, 2012; Mader et al., 2016;
Xierali, Fair, & Nivet, 2016). Historical events which help to explain the genesis of the lack of
underrepresented minorities in academic medicine harken back to Abraham Flexner, an
educational theorist who was charged by the Carnegie Foundation for the Advancement of
Teaching to evaluate all 155 medical schools in the United States and Canada (Flexner, 1910). A
primary recommendation in his report was to insist that all medical schools be integrated into an
established university structure. Although this decision has been credited for raising the quality
of medical education overall, it led to closure of seven of nine historically black medical schools
and this, coupled with segregation, severely limited the attendance of blacks to medical school
(Nivet, 2010; Sullivan & Mittman, 2010). Indeed, until 1964, whites comprised 97% of the
medical student body in the U.S. (Shea & Fullilove, 1985), and the mainstream medical schools
enrolled one African-American student every other year (Cohen, 2003).
Impact of Too Few URM Medical Faculty on Recruitment, Mentoring and Retention of
URM Medical Students
In United States medical schools, faculty diversity is linked to student diversity, cultural
competence and a climate of inclusivity on campus (Page, Castillo-Page, & Wright, 2011; Piercy
et al., 2005). The lack of underrepresented minorities in academic medicine adversely affects
recruitment, mentoring and retention of minority students, ultimately leading to their
underrepresentation in health care (Institute of Medicine, 2003). Research has shown that more
URiM students than white students used faculty diversity as either a “positive” or “very positive”
factor in deciding which medical school to attend (Zhang, Xierali, Castillo-Page, Nivet, &
UNDERREPRESENTED MINORITIES IN MEDICINE 29
Conrad, 2015). By increasing faculty diversity, available recruitment and mentoring of URiM
medical students commensurately increases, as does the matriculation of URiM physicians into
the healthcare workforce (Deas et al., 2012; Zhang et al., 2015; Sullivan & Mittman, 2010).
Inclusion of underrepresented faculty needs to be valued by medical school leaders, as failure to
fully engage their skills and insights impairs the ability to provide the best science, research,
education, and medical care (Pololi et al., 2010).
The Importance of Cultural Competency Teachings and Diversity to Care Delivery
Cultural Competence
Developing cultural competence, the knowledge, skills, attitude, and behavior required of
a practitioner to provide optimal health care services to persons from a wide range of cultural and
ethnic backgrounds, necessitates that education be in non-homogeneous settings, with URiM
faculty, peers and administration reflective of the diverse society (Cohen, Gabriel, & Terrell,
2002). In diverse settings, the quality of medical education is enhanced and there is a driving
force to do relevant research for different ethnic groups (Pololi et al., 2010; Whitla et al,.2003).
These “dividends of diversity” and the knowledge imparted by a diverse faculty are important
components of the framework to effectively deliver care to all populations and integral to the
mitigation of health care disparities in America (Derck et al., 2016)
The Clark and Estes (2008) Gap Analysis Conceptual Framework
The Clark and Estes (2008) gap analysis framework provides a systematic way by which
to clarify organizational and stakeholder performance goals and identify the gaps between the
actual performance level and the performance goal. Once potential gaps are identified, this
framework examines the stakeholder knowledge, motivation and organizational influences that
may underlie performance gaps (Clark & Estes, 2008). Krathwohl (2002) identified four types of
UNDERREPRESENTED MINORITIES IN MEDICINE 30
knowledge and skills used to determine if stakeholders know how to achieve a performance goal,
namely (a) factual, (b) conceptual, (c) procedural, and (d) metacognitive. Motivation influences
include the active choice to consider goal achievement, persistence in working towards a goal
and the mental effort to accomplish the goal (Clark & Estes, 2008; Rueda, 2011). Motivational
principles such as self-efficacy, attributions, values and goals are also considered
when analyzing the performance gap (Rueda, 2011), as are organizational influences such as
workplace culture, processes, and resources which also affect stakeholder performance (Clark &
Estes, 2008).
What follows is the application of the Clark and Estes’ (2008) framework to the assumed
knowledge, motivation and organizational needs of the New School of Medicine Admissions
File Reviewers, to meet their performance goal of learning and effectively utilizing the holistic
admissions process for medical school applicants by July of 2019. The first section discusses
assumed knowledge and skills influences on the stakeholder performance goal. The second
section examines assumed motivational influences on the attainment of the stakeholder goal and
lastly, assumed organizational influences on achievement of the stakeholder goal are examined.
Each of these influences is then examined in Chapter Three through the methodology lens.
Knowledge Influences
This section reviews the literature focusing on knowledge-related influences in general,
followed by specific references to the ones that are pertinent to the achievement of the goal of
the Admissions File Reviewers. The knowledge dimension can be categorized into four types:
factual knowledge, conceptual knowledge, procedural knowledge, and metacognitive knowledge
(Krathwohl, 2002). Factual knowledge includes terminology, specific details and content
elements; conceptual knowledge represents complex, organized forms of knowledge such as
UNDERREPRESENTED MINORITIES IN MEDICINE 31
classifications, theories and models; procedural knowledge is the knowledge of how to do
something; and metacognitive knowledge is the awareness and knowledge about one’s own
cognition, the task, and strategies needed to carry out the task (Krathwohl, 2002; Rueda, 2011).
There are two components of metacognitive knowledge, namely, knowledge about cognition and
regulation of cognition (self-regulation).
What follows is a discussion of the types of knowledge needed by the Admissions File
Reviewers in their quest to first, conceptually understand what the holistic admissions process is
and why it is important to utilize, and second, to procedurally know how to conduct a holistic
admissions process.
Admissions File Reviewers need to know what the holistic admissions process is.
The types of knowledge invoked for Admissions File Reviewers to know what and why a
holistic admissions process is conducted, are the factual and conceptual knowledge types. The
factual knowledge needed is what the term “holistic admissions” refers to and what the specific
elements are that comprise holistic admissions and why this type of process is important to
administer. The conceptual knowledge required to understand why it is important to conduct a
holistic admissions process lies in the understanding of the model and framework comprising the
holistic admissions process. To acquire the knowledge described will require professional
development training for the Admissions File Reviewers by admissions professionals who are
able to give the background “what” and “why” knowledge of the holistic admissions process.
Factually, holistic review is a strategic, mission-driven, evidence-based process that
recognizes diversity as critical to excellence, offers a flexible framework for selecting future
physicians, and facilitates achieving the institutional mission and addressing societal needs
(Conrad, Addams & Young, 2016). Researchers explain that holistic review is designed to help
UNDERREPRESENTED MINORITIES IN MEDICINE 32
universities consider a broad range of factors reflecting the applicant’s academic readiness,
contribution to the incoming class, and potential for success both in school and later as a
professional (Conrad et al., 2016). Holistic review assesses an applicant’s unique experiences
along with the traditional measures of academic achievement such as grades and test scores, and,
when used in combination with a variety of other mission-based practices, constitutes a “holistic
admission” process (Ono, 2016). Acknowledging that quantitative assessments of applicants
yield valuable information, researchers note that these assessments fail to convey the full story of
an applicant, and that selection is as much art as science (American Association of Medical
Colleges (AAMC), 2013; Conrad et al., 2016; Ono, 2016). By incorporating mission-based,
diversity-aware processes and using human judgment and evidence-based practice, holistic
review provides a framework for marrying the art with the science without sacrificing the unique
value that each brings (AAMC, 2013; Conrad et al., 2016).
In 2003, the U.S. Supreme Court officially described the strategy as a “highly
individualized, holistic review of each applicant’s file, giving serious consideration to all the
ways an applicant might contribute to a diverse educational environment” (Grutter v. Bollinger,
2003). Recognizing that the desired outcomes of a holistic admission process will vary
depending on each institution’s mission and goals, one core goal of a holistic process is the
assembly of a diverse student body — diverse not only in race, ethnicity, and gender, but also in
experience, socioeconomic status, and perspective (Ono, 2016). A key tenet of holistic review is
the recognition that a diverse learning environment benefits all students and provides teaching
and learning opportunities that more homogenous environments do not (Milem, 2003).
Addressing the conceptual knowledge requirements of the holistic review process,
candidates are evaluated by criteria that are institution-specific, broad-based, and mission-driven
UNDERREPRESENTED MINORITIES IN MEDICINE 33
and that are applied equitably across the entire candidate pool (AAMC, 2013; Conrad et al.,
2016). There are four core principles constituting the framework of a holistic admission process,
namely, (a) Selection criteria are broad-based, clearly linked to school mission and goals, and
promote diversity as an essential element to achieving institutional excellence; (b) A balance of
applicant experiences, attributes, and academic metrics is used to assess applicants with the
intent of creating a richly diverse interview and selection pool and student body; (c) Admissions
staff and committee members give individualized consideration to how each applicant may
contribute to the school learning environment and to the profession, weighing and balancing the
range of criteria needed in a class to achieve the outcomes desired by the school; (d) Race and
ethnicity may be considered as factors when making admission-related decisions only when such
consideration is narrowly tailored to achieve mission-related educational interests and goals
associated with student diversity, and when considered as part of a broader mix of factors, which
may include personal attributes, experiential factors, demographics, or other considerations
(AAMC, 2013; Ono, 2016). When admissions file reviewers apply these concepts, it is an
acknowledgement that medical student selection is responsive to and informed by the broader
social context, health and health care needs, educational research and evidence, and state and
federal law and policy.
Admissions File Reviewers need to know how to conduct a holistic admissions
process. The type of knowledge required for Admissions File Reviewers to know how to
conduct a holistic admissions process is the procedural knowledge type. To acquire the
procedural knowledge will require professional development training for the Admissions File
Reviewers by admissions professionals who are able to give the background “how” knowledge,
and procedurally model how it is done.
UNDERREPRESENTED MINORITIES IN MEDICINE 34
Research by the American Association of Medical Colleges (AAMC) (2013), has shown
that the mastery and application of the holistic review process for medical school admission
produces a far more diverse class than if such a process is not utilized. Because the mission of
the NSM includes diversity at its core, the successful creation of a diverse class fulfills the
stakeholder goal to matriculate a diverse class, in alignment with the organizational goal of
embedding diversity throughout the school. During training, Admissions File Reviewers were
able to build on their prior knowledge of admissions processes, allowing the occurrence of what
Schraw and McCrudden (2006), describe as the meaningful organization and connection of new
knowledge to prior knowledge, to construct meaning. The authors also note that it is important to
help the trainees, here the Admissions File Reviewers (AFRs), develop mastery by allowing
them to practice integrating the component skills they have acquired into their work, and by
helping them understand when to apply their learnings. As espoused by Mayer (2011), it is
important to provide the AFRs with frequent opportunities to practice the administration of the
holistic review process with ongoing feedback to allow transfer. Mayer (2011), has also shown
that it is beneficial to have AFRs identify pre-training, what they do know and do not know
about the holistic admissions process, and to break the content into parts, providing pre-training
for more complex parts of the model to avoid cognitive overload of the team members. In
addition, it is important to provide AFRs with the opportunity to utilize guided self-monitoring
and self- assessment before, during and after training (Baker, 2006).
In addition to the need for a specific goal to matriculate a diverse class (Dembo & Eaton,
2000), the training and learning objectives for the holistic review process also need to be specific
(Daly, 2009). From a procedural perspective, Admissions File Reviewers needed to have time to
observe credible and enthusiastic models who have mastered holistic admissions. After seeing a
UNDERREPRESENTED MINORITIES IN MEDICINE 35
demonstration of the process, Admissions File Reviewers should organize, rehearse, and then
practice what they have learned (Mayer, 2011). It is also important to provide immediate
feedback and reinforcement during the process (Tuckman, 2009), which should be private,
specific, and linked to the use of strategies to improve performance (Shute, 2008).
Admissions File Reviewers need to reflect upon their learning of the holistic review
process. The metacognitive assessment required here of reflection over time will provide
information to iterate and enhance the holistic review process, to stay aligned with the school’s
mission and objectives. The medical school admission process is key to the medical education
continuum, not an isolated event. For holistic review processes to be effective, they need to be
aligned with institutional mission and goals; integrated across outreach, recruitment, financial aid
and curriculum, and consistent with and directly connected to institutional values (AAMC). It
will be important for the AFRs and the admissions department to continually assess and reflect
on the implementation of the holistic review process, ways to improve upon it and fine-tune
calibration across the cohort to ensure equitable application of the holistic review process ,
reduction of individual biases and inter-rater reliability (Kreiter, 2013). Such practice will not
only generate improvement but will also satisfy leaders who would want and need to know the
impact of this practice on incoming student academic qualifications, student retention, and
student performance, especially as relates to achieving desired outcomes in the areas of access,
diversity, and student success in higher education. (Holistic Admissions in the Health
Professions: findings from a national survey Greer Glazer, Karen Bankston, 2014).
Table 2 below shows the organizational mission, the organizational goal and the
stakeholder performance goal, as well as information specific to knowledge influences,
knowledge types, and knowledge influence assessments.
UNDERREPRESENTED MINORITIES IN MEDICINE 36
Table 2
Knowledge Influences, Types, and Assessments for Knowledge Gap Analysis
Organizational Mission

To provide a unique medical education, embedded in a physician-led health care delivery
system that ignites a passion for learning, a desire to serve, and an unwavering commitment to
improve the health and well-being of all patients and communities.
Organizational Goal

To inspire a collaborative culture at the New School of Medicine (NSM), committed to
embedding diversity, equity, and inclusion in each medical class and throughout the School.
Stakeholder Goal

Stakeholder is the group of Admissions File Reviewers (AFRs):
By July of 2019, one year prior to the matriculation of the inaugural class of NSM, all AFRs
will be able to conduct a holistic review process to achieve the matriculation of a diverse
class each year.

Knowledge Influence
Knowledge Type
Declarative (factual and
conceptual); procedural;
metacognitive

Knowledge Influence
Assessment
1. AFRs need to know
what a holistic review
process is

Factual

Ask the AFRs to paraphrase
what the holistic review
process is
2. AFRs need to know
how to conduct a holistic
admissions process.

Procedural

Ask the AFRs to demonstrate
how the holistic review process
would occur.
3. AFRs need to reflect on
their learning of the
holistic review process

Metacognitive Ask the AFRs to reflect on their
learning of the process, any
iterations made, calibrations and
what they believe the outcomes will
be.

Motivation
Motivation is the second component of the Clark and Estes (2008) three-pronged gap
approach used to evaluate performance problems and their solutions. Motivation is a key factor
in the achievement of goals because it impacts whether individuals start, persist and exert effort
to finish a task, and it elucidates their beliefs about themselves as learners (Mayer, 2011; Rueda,
UNDERREPRESENTED MINORITIES IN MEDICINE 37
2011). Motivation is what drives individuals to engage in a process from start to finish to achieve
the desired outcome (Mayer, 2011). Therefore, understanding how motivation contributes to the
AFRs learning and enacting of a holistic review process was critical to determine the needs to
address to accomplish the stakeholder goal (Rueda, 2011).
Recognizing that motivation can be impacted by several factors, this section reviews the
literature focusing on two motivation-related influences specific to the ones that are pertinent to
the achievement of the AFRs goal to conduct a holistic review process to achieve the
matriculation of a diverse class each year. First, AFRs need to believe that learning about and
being able to conduct a holistic review process, is professionally important and valuable. Second,
AFRs need to believe that they are capable of effectively learning how to conduct a holistic
admissions process which, Bandura (1991), describes as self-efficacy in his social cognitive
theory.
Expectancy value theory: Admissions File Reviewers (AFRs) need to believe that
learning about the holistic review process (HRP) is professionally important and valuable.
Eccles’ (2006), explains in the expectancy value motivation theory that individuals are more
likely to be engaged and successfully attain a goal when they place a high value on the goal.
Assessments typically focus on the learner’s expectations of outcomes and the belief that a given
behavior will or will not lead to a given outcome, and that the learner’s belief significantly
influences the actual outcome (Mayer, 2011; Rueda, 2011). In this study, value recognition was
acknowledged during training by competent, experienced and enthusiastic models who shared
their outcomes based on real-world application of the process over time (Eccles, 2006; Pintrich,
2003), and by the demonstrated connection between the matriculation of a diverse class each
year and the mitigation of health care disparities (AAMC, 2013).
UNDERREPRESENTED MINORITIES IN MEDICINE 38
Self-Efficacy: Admissions File Reviewers need to believe that they can learn the
holistic review process. Self-efficacy is an individual’s belief in one’s own ability to
successfully complete the task (Bandura, 1991). The Admissions File Reviewers (AFRs) needed
to have confidence in their ability to complete a task or performance goal and the belief that they
could make a difference in the long term. According to Pajares (2006), the professional
development training with observation of credible, similar models engaging in behavior that has
demonstrated functional value, with goals that are temporally close, concrete and challenging,
make it clear to AFRs that they are capable of learning what is being taught and are able to
perform the process. Additionally, AFRs ought to experience success when they have multiple
opportunities to practice the process with less scaffolded instructional support over time, and
frequent, targeted, private feedback on their performance and progress (Pajares, 2006). Just as
engagement and participation in skills training has been positively correlated with the
strengthening of self-efficacy and performance in medical students (Stegers-Jager, Cohen-
Schotanus, & Themmen, 2012), such is the expectation for the AFRs with robust participation in
skills training for the holistic review process. Rewards can be linked with progress to reinforce
learning and improvement over time (Pintrich, 2003).
The Admissions File Reviewers belief in the value of what they are learning, along with
the belief that they will successfully learn, provide the motivation to persist in the task and exert
the mental effort necessary to achieve (Mayer, 2011). The importance of this, as connected to the
holistic review process (HRP), is the achievement of the goal to conduct an HRP to achieve the
matriculation of a diverse class each year, in alignment with the organizational goal to inspire a
culture at the NSM, committed to embedding diversity, equity, and inclusion in each medical
class and throughout the School.
UNDERREPRESENTED MINORITIES IN MEDICINE 39
Table 3 shows the organizational mission, the organizational goal and the stakeholder
performance goal, and identifies two motivational influences that focus on expectancy value and
self-efficacy. These influences will be used to more fully understand the mindset needed by
AFRs to be successfully trained to utilize the HRP to select students for the NSM.
Table 3
Motivational Influences and Assessments for Motivation Gap Analysis
Organizational Mission
To provide a unique medical education, embedded in a physician-led health care delivery
system that ignites a passion for learning, a desire to serve, and an unwavering commitment to
improve the health and well-being of all patients and communities.
Organizational Goal
To inspire a collaborative culture at the New School of Medicine (NSM), committed to
embedding diversity, equity, and inclusion in each medical class and throughout the School.
Stakeholder Goal
Stakeholder is the group of Admissions File Reviewers (AFRs):
By July 2019, one year prior to the matriculation of the inaugural class of NSM, all AFRs
will be able to conduct a holistic review process to achieve the matriculation of a diverse
class each year.
Assumed Motivation Influences Motivational Influence Assessment
Expectancy Value Theory – AFRs need to
believe that learning about the holistic review
process is professionally important and valuable.
Interview each AFR to ascertain the
extent they believe that the material
they are learning is useful to
accomplish NSM Admissions goals.
Self-Efficacy - AFRs need to believe they can
learn the holistic review process.
Interview each AFR to ascertain the
extent they believe that they can learn
the HRP.
Organization
Clark and Estes (2008) note that because organizations face profound, complex, and
persistent change, increasing knowledge, skills, and motivation with a focus of these influences
towards the performance of organizational goals, is the key to a sustainable competitive
advantage. Knowledge, skills and motivation are inextricably intertwined with organizational
culture which Schein (2017), defines as a pattern or system of beliefs, values, and behavioral
UNDERREPRESENTED MINORITIES IN MEDICINE 40
norms that come to be taken for granted as basic assumptions and eventually drop out of
awareness. Organizations as we know them are the people in them and change is effective only
to the degree that change in the psychology of the people occurs (Schneider, Brief, & Guzzo,
1996). Culture is often the last to change and yet, if ignored, any performance improvements will
likely be short-lived (Sundt, n.d.).
This review of the literature focuses on the organization-related influences that are
pertinent to the achievement of the stakeholder goal to achieve mastery of the holistic review
process by 100% of the Admissions File Reviewers, in furtherance of the matriculation of a
diverse class each year. Here, the importance of the NSM organizational influences to the
admissions file review process is in alignment with the organizational goal to inspire a culture at
the NSM, committed to embedding diversity, equity, and inclusion in each medical class and
throughout the School, to achieve institutional excellence and improve the health and well-being
of all patients and communities.
Organizational culture filters and affects any attempt to improve performance, and
successful performance improvement depends on taking specific organizational culture into
account (Clark & Estes, 2008). This section reviews the literature on organization-related
influences in general, followed by the specific NSM organization influences pertinent to the
achievement of the AFRs stakeholder goal. Gallimore and Goldenberg (2001), describe
organizational culture in terms of cultural models and cultural settings, defining cultural models
as the shared mental schema of how the world works, and defining a cultural setting as occurring
whenever two or more people come together, over time, to accomplish something. Six NSM
organization influences pertinent to the achievement of the AFRs stakeholder goal are
categorized in terms of cultural models and settings below.
UNDERREPRESENTED MINORITIES IN MEDICINE 41
Cultural Model 1: NSM needs the trust of Admissions File Reviewers (AFRs) to
adopt a holistic review process (HRP). Utilizing the holistic review process, candidates are
evaluated by criteria that are institution-specific, broad-based, mission-driven, and applied
equitably across the entire candidate pool (American Association of Medical Colleges (AAMC)
2013; Conrad et al., 2016; Milem, 2003; Ono, 2016). The U.S. Supreme Court officially
described the strategy as a “highly individualized, holistic review of each applicant’s file, giving
serious consideration to all the ways an applicant might contribute to a diverse educational
environment” (Grutter v. Bollinger, 2003). NSM leadership recognizes that, although
quantitative assessments of applicants yield valuable information, the full story of an applicant is
not conveyed, and selection is as much art as science (AAMC, 2013; Conrad et al., 2016; Ono,
2016).
Admissions File Reviewers (AFRs) need to recognize, acknowledge, believe and trust in
the vision of NSM’s leadership that the way to achieve a diverse class each year is with adoption
of a holistic review process. In accordance with the typology of Agocs (1997), to counteract
resistance to change, AFRs need to: (a) defend the credibility of the change message as well as
the credibility of the messengers (NSM leaders); and (b) accept and support the responsibility to
master the HRP (Agocs, 1997). Creating and maintaining a culture of trust amongst AFRs and
between AFRs and NSM leadership, will be critical to achieving the stakeholder goal of HRP
mastery as well as the organizational goal of matriculating a diverse class each year. Having
ongoing open communication, solicitation of feedback, and demonstration of concern by
leadership consistent with organizational policies will facilitate such trust (Agocs, 1997; Berger,
2014; Korsgaard, Brodt, & Whitener, 2002). The engendered trust will enable the holistic review
process to matriculate a diverse class each year, in fulfillment of the NSM organizational goal
UNDERREPRESENTED MINORITIES IN MEDICINE 42
and mission to achieve institutional excellence and improve the health and well-being of all
patients and communities.
Cultural Model 2: NSM leadership and the AFRs need to share the belief that it is
necessary to matriculate a diverse class each year. Members of the NSM community need to
recognize, acknowledge and internalize as a shared belief, that a diverse learning
environment:(a) benefits all students; (b) provides teaching and learning opportunities that more
homogenous environments do not; and that (c) the way to achieve a diverse class each year is
through a holistic review process (Milem, 2003). One core goal of a holistic process is the
assembly of a diverse student body — diverse not only in race, ethnicity, and gender, but also in
experience, socioeconomic status, and perspective (Ono, 2016). Holistic review is a strategic,
mission-driven, evidence-based process that recognizes diversity as critical to excellence, offers
a flexible framework for selecting future physicians, and facilitates achieving the institutional
mission and addressing societal needs (Conrad et al., 2016). The attributes of holistic review
drive the need to use the holistic review process at NSM to achieve the organizational goal of
matriculating a diverse class each year.
Cultural Setting: NSM needs to provide professional development training on the
holistic review process for the AFRs, with set-aside time to participate, as well as the
opportunity to rehearse, receive feedback and calibrate amongst the cohort. Many
performance benefits have been realized by individuals, teams and organizations after
professional development training (Aguinis & Kraiger, 2009). Research by the AAMC, (2013),
has shown that the mastery and application of the holistic review process for medical school
admissions produces a far more diverse class than if such a process is not utilized. To acquire
such mastery requires professional development training for the AFRs by admissions
UNDERREPRESENTED MINORITIES IN MEDICINE 43
professionals able to provide the factual knowledge about the HRP and procedurally model how
it is done. During training, AFRs will be able to build on their prior knowledge of admissions
processes, allowing the occurrence of what Schraw and McCrudden (2006), describe as the
meaningful organization and connection of new knowledge to prior knowledge to construct
meaning. The authors also note that it is important to support the trainee AFRs in their quest to
develop mastery of the HRP by helping them (a) acquire component skills, (b) practice
integrating the skills into their work, and (c) understand when to apply their learnings (Schraw &
McCrudden, 2006). Successful enactment of the HRP by the AFRs fulfills the stakeholder goal,
and subsequent matriculation of a diverse class will fulfill NSM’s goal, in alignment with the
mission of embedding diversity throughout the school.
Because there is factual and procedural knowledge to impart, professional development
training needs to occur. The training process and the subsequent enactment of holistic review
requires ongoing time set aside to allow the AFRs to participate. Two-thirds of schools already
using holistic review report that implementation requires investment of additional resources
(Artinian, 2017). Resources are needed up front for training and workshops, and investing
ongoing resources, such as increasing faculty and staff time for admissions efforts are also
required (Artinian, 2017). Many schools have a three-step review process involving (a)
holistic review of each application; (b) selection of the interview pool, and (c) an admissions
committee meeting to discuss which applicants to offer acceptance to (Monroe, Quinn,
Samuelson, Dunleavy, & Dowd, 2013). The holistic review process takes 15 minutes or longer
per application and the training process is important to achieve common knowledge for AFRs, as
well as calibration and interrater reliability (Artinian, 2017; Kreiter, 2013; Monroe et al., 2013).
UNDERREPRESENTED MINORITIES IN MEDICINE 44
As espoused by Mayer (2011), it will be important to provide AFRs with frequent
opportunities to practice the administration of the HRP with ongoing feedback and the
opportunity to reflect on the process. Mayer (2011) has also shown benefit to have AFRs identify
what they do and do not know about holistic review, and to break the content into parts,
providing pre-training for more complex parts of the model to avoid cognitive overload of the
team members. In addition, it is important to provide AFRs the opportunity to utilize guided self-
monitoring and self- assessment before, during and after training (Baker, 2006). The AFRs need
to be given time to observe credible models who have mastery of holistic admissions. After
seeing a demonstration of the process, AFRs should organize, rehearse, and practice what they
have learned (Mayer, 2011). It is also important to provide immediate feedback and
reinforcement during the process (Tuckman, 2009), which should be private, specific, and linked
to the use of strategies to improve performance (Shute, 2008). The ability to reflect on the
learnings as individual AFRs, as well as collectively as a group, will help boost the individual
self-efficacy of each AFR members and the collective self-efficacy of the AFR group (Bandura,
1977; Bandura, 2000). This will, in turn, facilitate the achievement of the stakeholder goal of
mastery of the HRP by all AFRs, in furtherance of the organizational goal of matriculating a
diverse class each year. It is also important for the NSM to engage in evaluation of the HRP
outcomes over time to demonstrate the value of the HRP (Kreiter, 2013; Monroe et al., 2013).
Table 4 below shows the organizational mission, the organizational goal and the
stakeholder performance goal, as well as information specific to organization influences and
organization influence assessments.
UNDERREPRESENTED MINORITIES IN MEDICINE 45
Table 4
The Organizational Mission and Goal, Stakeholder Goal, Organization Influences and
Assessments
Stakeholder Goal
Stakeholder is the group of Admissions File Reviewers (AFRs). By July 2019, one year prior
to the matriculation of the inaugural class of NSM, all AFRs will be able to conduct a
holistic review process (HRP) to achieve the matriculation of a diverse class each year

diverse class each year.

Assumed Organization Influences Organization Influence Assessment
The NSM leadership needs the trust of
AFRs to adopt a holistic review process
(HRP)
Interview the AFRs to ascertain how much
they trust NSM leaders to facilitate the
development of an HRP.
The NSM leadership and the AFRs need to
share the belief that it is important to
matriculate a diverse class each year
Interview the AFRs to ascertain how
important they believe it is to matriculate a
diverse NSM class each year.
The NSM needs to provide effective PD
training on the HRP for the AFRs with set-
aside time to fully participate, as well as the
opportunity to rehearse, receive feedback
and calibrate amongst the cohort.
Interview the AFRs to ascertain that (a)
effective PD training was provided; (b) there
was adequate time set-aside to be able to
fully participate; (c) there was the
opportunity to rehearse, receive feedback and
calibrate amongst the cohort.

UNDERREPRESENTED MINORITIES IN MEDICINE 46
Table 5 below is a summary of the knowledge, motivation and organizational assumed
influences.
Table 5
Summary Table of Assumed Influences on Performance

Assumed Influences on Performance
Source of
Assumed
Influences
Knowledge Motivation Organization
Learning and
Motivation
and
Organizational
Theory
• Admissions File
Reviewers (AFRs)
need to know what
a holistic review
process (HRP) is
• AFRs need to
know how to
conduct an HRP
• AFRs need to
reflect on their
learning of the
HRP
• AFRs need to
believe that
learning about the
HPR is
professionally
important and
valuable
• AFRs need to
believe that they can
learn the HRP.
• The NSM needs
the trust of AFRs
to adopt an HRP.
• The NSM
leadership and
the AFRs need to
share the belief
that it is
important to
matriculate a
diverse class
each year.
• The NSM needs
to provide
professional
development
training on the
HRP for the
AFRs with set-
aside time to
fully participate,
as well as the
opportunity to
rehearse, receive
feedback and
calibrate amongst
the cohort.

UNDERREPRESENTED MINORITIES IN MEDICINE 47
Conceptual Framework: The Interaction of Stakeholders’ Knowledge and Motivation and
the Organizational Context
A conceptual framework is the system of concepts, assumptions, expectations, beliefs,
and theories that supports and informs a research study (Maxwell, 2013). It is a tentative theory
of phenomena being investigated and is one that is constructed, not found (Maxwell, 2013). It
incorporates ideas presented in the literature (theoretical and empirical), as well as personal
experiential knowledge, and thought experiments (speculative model building), to create an
understanding of the interactions of what is happening and why (Maxwell, 2013). Merriam and
Tisdell (2016) prefer the term “theoretical framework” which is often used interchangeably with
“conceptual framework,” as a broader and more inclusive scaffolding or frame of a research
study, and which includes terms, concepts, models, thoughts and ideas, as well as references to
specific theories. The vehicle of the conceptual framework (a) justifies the research and assists in
identifying the most appropriate methods for framing and exploring the research questions at the
heart of the research design; (b) can be visual or narrative, and (c) has the goal to map
relationships between the knowledge, motivation and organizational influences (Maxwell, 2013;
Merriam & Tisdell, 2016; Rocco, 2009). It defines how you are laying a “stake in the ground”
about what will be researched (Tiwana, n.d.).
This study is informed by two worldviews, constructivism and pragmatism.
Constructivism seeks to “construct” meaning from and generate explanations about phenomena
(Creswell 2018), which this study aims to do regarding the interaction of influences that best
facilitates the group of Admissions File Reviewers achieving an effective holistic review
process. A constructivist researcher looks at “the processes of interaction among individuals”
(Creswell, 2018) and the meaning is subjectively made through personal interaction. A pragmatic
UNDERREPRESENTED MINORITIES IN MEDICINE 48
world view also informs this study, enabling the use of all available approaches to understand the
research problem and questions (Creswell, 2018). Pragmatism “arises out of actions, situations,
and consequences rather than antecedent conditions, and researchers are free to choose the
methods, techniques, and procedures of research that best meet their needs and purposes
(Creswell, 2018). Pragmatism further develops the constructivist world view by allowing for
analysis using a cross-section of methods to help understand the problem from many
perspectives. and generate new knowledge (Creswell, 2018). In this study, a pragmatic
worldview elucidates what will practically work for the Admissions File Reviewers to address
the research problem of developing an effective holistic review process for the NSM.
While each of the assumed knowledge, motivation, and organizational influences were
previously presented as separate entities required for the achievement of an effective holistic
review process, the three entities are interwoven, operating together, not in isolation, and must be
simultaneously addressed to achieve stakeholder and organizational goals (Clark & Estes, 2008).
Therefore, the conceptual framework presented here articulates the theory of practice as
determined by the interaction of the assumed knowledge and motivation work influences within
the cultural and contextual milieu of the NSM, to achieve the stakeholder goal of mastery of the
holistic review process by 100% of the Admissions File Reviewers, in furtherance of the
organizational goal to matriculate a diverse class each year. Figure 1 below illustrates this
conceptual framework.

UNDERREPRESENTED MINORITIES IN MEDICINE 49
Figure 1. Interaction of Stakeholder Knowledge and Motivation within Organizational Cultural
Models and Settings.

The Organization: New School of Medicine (NSM)
Cultural Models: shared belief in diversity, equity and inclusion; willingness and trust in leadership to
adopt a holistic review process (HRP) aligned with mission & goals; accountability. Cultural Settings:
professional development (PD) training for Admissions File Reviewers (AFRs) in HRP with time to
practice, get feedback & reflect; carved-out time for each AFR to participate fully in the process.

Global Goal – Innovation- Development of Holistic Review Process (HRP)
Holistic Admission Devel

Holistic Admissions P

AFR motivation:
Expectancy/Task
Value: Each AFR needs to
believe that training in HRP is
important and has
professional value. Self-
efficacy: each AFR needs to
believe that they are capable
of learning and performing
the tasks required.

AFR knowledge:
Factual: holistic review components
(balance of experience, attributes & metrics
(e,a,m) to screen, interview & select;
unique, contextual definition of diversity).
Procedural: conduct a HRP (broad
selection criteria linked to school mission &
goals, promoting diversity to drive
excellence; individualized; balance of e,a,m;
legal consideration of race & ethnicity)
Metacognitive: self-reflection, self-
regulation and calibration

Achieved Phenomenon: After clear articulation
and understanding of the stakeholder goal with
focused training, engagement & reflection, each
AFR experiences individual mastery of the HRP
leading to calibrated, reliable AFR performance.

Stakeholder Goal
By July 2019, one year prior to the matriculation of the inaugural class of
NSM, 100% of AFRs will be able to conduct a HRP to achieve the
stakeholder goal of matriculation of a diverse class each year, in
furtherance of the organizational goal to embed diversity, equity and
inclusion in each medical class and throughout the school.

m

UNDERREPRESENTED MINORITIES IN MEDICINE 50
Figure 1. Interaction of Stakeholder Knowledge and Motivation within Organizational Cultural
Models and Settings.

LEGEND:

Blue outlined box – the organization – NSM

Red box – Global goal/ innovation – holistic review process development

Yellow boxes – stakeholder knowledge & motivation influences

Gold oval – AFRs – individual and collective experience of phenomenon

Purple bi-directional arrow – simultaneous interaction

White downward arrow – interaction leads to

AFRs stakeholder goal

Broadly, Figure 1 outlines the interactional relationship between knowledge and
motivation influences for Admissions File Reviewers to attain mastery of the HRP, enveloped
within the organizational culture and context of New School of Medicine (NSM).
Deconstructing Figure 1, the blue square represents the organization of study, NSM, and the
cultural models and contextual settings that need to exist within NSM to drive goal attainment
(Gallimore & Goldenberg, 2001; Schein, 2017; Schwandt & Marquardt, 1999; Schneider et al.,
1996; Senge, 1990). These include the cultural model needs of (a) shared beliefs in diversity,
equity and inclusion (Artinian, 2017; AAMC, 2013; Conrad et al., 2016; Harris & Bensimon,
2007; Milem, 2003; Ono, 2016), and (b) willingness to keep an open mind in the face of change,
potential conflict and resistance (Agocs, 1997; Korsgaard et al., 2002), with trust in leadership to
adopt an accountable holistic admission process (HAP) (Kreiter, 2013; Price, et al 2011), aligned
with NSM specific mission & goals (Alper, Tjosvold, & Law, 2000; Burke, 2002; Burke 2018;

UNDERREPRESENTED MINORITIES IN MEDICINE 51
Daly, 2009; Dembo & Eaton, 2000; Kezar, 2001 a; Kezar, 2001 b; O’Neill, 2013), as well as the
cultural setting needs of (a) NSM providing effective professional development (PD) training in
HRP for Admissions File Reviewers (AFRs) (Aguinis, 2009; Pajares, 2006; Shraw &
McCrudden, 2006); (b) set-aside time for each AFR to participate fully in the PD training to
master and enact the HRP (Monroe, 2013), and (c) time to practice, get feedback and reflect on
the training (Mayer, 2011; Shute, 2008; Tuckman, 2009).
Within the organization, designated in Figure 1 as the large red square, there is the
innovation that grounds the subject of the study, namely, the global goal of the implementation
of a holistic review process (HRP). Inside the large red square global goal are two yellow boxes
which represent knowledge and motivation influences of the Admissions File Reviewers. The bi-
directional purple arrow in between the yellow boxes represents the fluid, symbiotic interaction
between the knowledge and motivation influences.
The knowledge influences are Krathwohl’s (2002) factual, procedural and metacognitive
influences, shown here in the yellow box on the left in Figure 1. The factual influence relates to
knowledge of the holistic review balancing of the experience, attributes and metrics components
to enable individualized screening, interviewing and selection of applicants, as well as the
knowledge of the unique, contextual definition of diversity (AAMC, 2013; Conrad et al., 2016;
Ono, 2016). The procedural knowledge component is the ability to conduct an HRP using a
broad selection of criteria linked to school mission and goals, promoting diversity to drive
excellence; an individualized approach with a balance of the experience, attributes and metrics
evaluation applied per candidate, as well as endorsed legal considerations of race & ethnicity
(AAMC, 2013. Grutter v Bollinger, 2003; Milem, 2003; Ono, 2016), The metacognitive
knowledge component is the ability of the AFRs to reflect on their knowledge of the HRP and
UNDERREPRESENTED MINORITIES IN MEDICINE 52
debrief their learnings of the HRP, with a view to making improvements by tweaking or
changing how things are done (Baker, 2006). The metacognitive knowledge component is the
self-reflection and self-regulation of each AFR regarding the how and why of supporting an
HRP, as well as the ability to calibrate and demonstrate inter-rater reliability (Kreiter, 2013;
Monroe, 2013). Notably, this is inextricably linked (bidirectional purple arrow) to the motivation
influence of self-efficacy of each AFR, and to the collective efficacy of the AFR cohort
regarding their belief in themselves that they can accomplish the task of an effective holistic
admissions process as an entire committee (Bandura, 1977; Bandura, 2000).
According to Pajares (2006), the professional development training in HRP provided by
NSM, with observation of credible, similar models engaging in behavior that has demonstrated
functional value, feeds the motivation influence because, by showing that the AFRs are capable
of learning what is being taught and are able to perform the process by setting goals that are
temporally close, concrete and challenging, drives the experience of success. Rewards can be
linked with the progress of the AFRs to reinforce their learning and improvement over time
(Pintrich, 2003).
The other connected motivation influences in the yellow box on the right (Figure 1),
include expectancy value for the AFRs, that they perceive that training in the HRP is important
and has professional value and their belief in the outcome of success plays a significant role in
the actual outcome (Dweck, 1986; Eccles, 2006; Mayer, 2011; Rueda, 2011). The AFRs belief in
the value of what they are learning, along with the belief that they will successfully learn, will
provide the motivation to persist in the task and exert the mental effort necessary to attain
individual and group self-efficacy (Bandura, 1977; Bandura, 2000; Mayer, 2011).
UNDERREPRESENTED MINORITIES IN MEDICINE 53
The importance of the dynamic interplay between the knowledge and motivation
influences (bidirectional purple arrow), soaked in the sea of organizational culture and context,
depicted by the blue square in Figure 1, is the ability to effectively drive the HRP, achieving the
phenomenon of mastery of the HRP by the AFRs (shown in the gold oval in Figure 1), which
then drives (white arrow) the stakeholder goal of an effective HRP, in furtherance of the
organizational goal to matriculate a diverse class each year, and the mission to improve the
health and well-being of all patients and communities (shown as the green pentagon in Figure 1).

UNDERREPRESENTED MINORITIES IN MEDICINE 54
CHAPTER THREE: METHODOLOGY
Introduction to the Methodology
To provide high-quality care to diverse patient populations requires that health care
professionals be educated in environments that value diversity (Artinian et al., 2017). Research
has demonstrated that schools using a holistic review process (HRP) have experienced increases
in the diversity of their incoming classes (Kondo & Judd, 2000; Urban Universities for HEALTH
study, 2014). In furtherance of the mission of the New School of Medicine (NSM), to embed
diversity, equity, and inclusion within the medical curriculum and the entire school, this
innovation study sought to identify the resources necessary to develop an effective holistic
review process for NSM. The study utilized a gap analysis framework with a qualitative design
and this chapter outlines the research design, methodology, data collection and analysis. The
questions guiding this study were:
1. What are the knowledge and motivational needs of the Admissions File Reviewers
(AFRs) of NSM to effectively enact a holistic review process to matriculate a diverse
class each year?
2. What is the interaction between the AFRs knowledge and motivation and the
NSM organizational culture and context?
The recommended knowledge, motivation, and organizational solutions to those needs is
discussed in Chapter 5. This chapter begins with a description of the participating stakeholders
and continues with an explanation of the ds used and sampling criteria.
Participating Stakeholders
The stakeholder group of focus was the cohort of 19 Admissions File Reviewers (AFRs)
at the New School of Medicine (NSM) and the stakeholder goal was that by July 2019, one year
prior to the matriculation of the inaugural class of NSM, all AFRs could conduct a holistic
UNDERREPRESENTED MINORITIES IN MEDICINE 55
review process to drive the attainment of a diverse class of matriculants. The stakeholder goal
was in furtherance of the organizational goal to embed diversity, equity, and inclusion within the
medical curriculum and the entire school.
Interview Sampling (Recruitment) and Rationale
I conducted individual interviews with 11 of the Admissions File Reviewers, thus
approaching maximum variation sampling of the AFR group and the ability to capture core
experiences and the emergence of any common themes (Merriam & Tisdell, 2016). The
interviews used a qualitative informal conversational approach as well as an interview guide
approach, allowing specific topics to be covered with open-ended questions to encourage the
accumulation of rich data (Johnson & Christensen, 2014; Merriam & Tisdell, 2016). I did not see
the need for a highly structured interview which would be used more to obtain standardized,
quantitative data across the group (Merriam & Tisdell, 2016). The interviews were performed by
telephone. From the perspective of Clark and Estes (2008) knowledge and motivation
framework, the interviews captured the Krathwohl (2002), factual and procedural knowledge
influencers of the AFRs, along with the relevant motivational influences related to self- and
group-efficacy (Bandura, 1977; Bandura, 2000; Dweck, 1986), expectancy value (Eccles, 2006)
and goal attribution (Yough & Anderman, 2006).
Observation Sampling Criteria and Rationale
Observation is watching the behavioral patterns of people, important because attitudes
and behavior are not always congruent (Johnson & Christensen, 2014). I engaged in qualitative
observation of the file reviewers in the natural setting of their training to learn the holistic review
process. I observed all potentially relevant phenomena and took field notes, without having
specified in advance what was to be observed (Johnson & Christensen). I noted if they were
UNDERREPRESENTED MINORITIES IN MEDICINE 56
NSM- mission-centric during the process as it is a central tenet of the HRP, to focus on the
unique mission of the organization, known to drive the matriculation of students that would be a
good fit for the school (Ono, 2016). I also noted the organizational resources required to drive
the process, useful as I examined the Clark and Estes (2008) interactional framework of
knowledge, motivation and organizational influences.
Observation Sampling (Access) Strategy and Rationale
The AFR group knew that I was there, endorsed by NSM leadership, to observe the group
as part of my dissertation. Thus, I was not a complete observer, totally hidden from the group
(Merriam & Tisdell, 2016; Johnson & Christensen, 2014). As an observer, I was privy to
generated documents and other visual data work-product along the way (Johnson & Christensen,
2014). From a timing perspective, I observed the one-day professional development training in
June of 2019, allowing firsthand knowledge of alignment with institutional mission, engagement,
questioning, organizational resource investment and needs and opportunities to create solutions
to tweak processes (Conrad et al., 2016).
Explanation for Choices
I chose qualitative design of interviews and observations without quantitative surveys
because my target population is relatively small, allowing for majority assessment.

UNDERREPRESENTED MINORITIES IN MEDICINE 57
Data Collection and Instrumentation
Table 6
Sampling Strategy and Timeline
Sampling Number in Number of Start and
Strategy Stakeholder
Population
Participants End Date

Observation
Observe the AFRs during
their scheduled one-day
in-person training.

19

19

June, 2019

Interviews
Purposeful with
maximum variation
19 11 Oct-Nov 2019

Qualitative Data Collection and Instrumentation
This qualitative innovation model study is designed to deeply probe the interaction of the
Admissions File reviewers’ knowledge, motivation and organizational influences to develop an
effective holistic review process (HRP) that will drive the matriculation of a diverse New School
of Medicine (NSM) class each year. My interests, as guided by Creswell (2014) and Merriam
and Tisdell (2016), and in alignment with my research questions, were to understand, interpret
and attribute meaning to data collected from my research process. I chose to examine documents,
conduct interviews, and to observe the participant group in the field to uncover meaning, develop
understanding and discover insights through research question congruence as shown in my
conceptual framework, allowing an emergent design as well as an inductive analysis (Merriam &
Tisdell, 2016).
The documents were produced independent of my research study and thus nonreactive
and grounded in the context under study, addressing the knowledge and organizational
components of my research questions. I conducted one-on-one interviews with 11 of the
participants engaging in questioning related to my research study and conceptual framework,
facilitating purposeful access to the broad range of perspectives and ideas of each participant
UNDERREPRESENTED MINORITIES IN MEDICINE 58
Admissions File Reviewer (Bogdan & Biklen, 2007; Johnson & Christensen, 2015; Merriam &
Tisdell, 2016; Patton, 2015). As discussed by Roulston (2010), using a neo-positivist
philosophical interview orientation, I was skillful in asking good questions congruent with the
knowledge, motivation and organizational aspects of my research questions, and I minimized
bias through a neutral stance (as cited in Merriam & Tisdell, 2016). I did not do focus group
interviews for this research study because my questions which included topics of diversity,
equity and inclusion were designed to solicit answers of a sensitive and highly personal nature,
not likely to be authentically elicited in a group of strangers (Merriam & Tisdell).
All three of the data collection methods discussed above, namely, document analysis,
observation, and interviews allowed for triangulation of data and an inductive qualitative
process with emergence of themes upon which to help make sense of the data and inform the
development of an effective HRP for the NSM (Bowen, 2009; Merriam & Tisdell, 2016). The
timing of doing document analysis first was to ensure my complete understanding of the mission,
vision and strategic plan of the NSM. The observation of the cohort at the one-day training as
they worked as a team followed by individual interviews allowed for a logical approach to
address my research questions.
Documents
Documents provided the context of the organizational needs and desires vis-à-vis the
HRP (Bowen, 2009). Documents obtained for my study included the mission, vision and
strategic plan of NSM. as well as the Liaison Committee on Medical Education (LCME)
standards. via the administration of NSM, and I will focus on those related to admission and
diversity. These documents enabled accurate and thorough framing of background and
contextual elements, additional questions to be asked and supplementary material to obtain
UNDERREPRESENTED MINORITIES IN MEDICINE 59
(Bowen, 2009). The documents related specifically to the knowledge and organizational
components of my research questions and conceptual framework. They provided the fodder, in
an unobtrusive and non-reactive way, to fully understand the context myself, by initially
skimming, then thoroughly reading and interpreting them (Bowen, 2009). This allowed
subsequent, effective probing and understanding of each AFR’s knowledge of the mission and
goals of the school as they sought to develop an effective HRP within the organizational culture
and context of NSM. I was also privy to all printed and visual media documents used to train the
AFRs.
Observation
Observation Protocol. The most important determinants when deciding what to observe
are the problem of practice, research questions and conceptual framework (Merriam & Tisdell,
2016). The focus of my observation was to watch how the cohort of AFRs worked separately and
together as they formulated what was necessary to develop an effective HRP to drive the
matriculation of a diverse medical school class each year. Therefore, my observation related
directly to my research questions and conceptual framework and explored all the KMO
influences. I looked specifically for evidence of knowledge of the elements of holistic review,
procedural knowledge of how to perform holistic review and evidence of reflection on the
holistic review process. From a motivation standpoint, I looked for evidence of perceived
positive task value and self-efficacy at an individual and collective AFR group level. Regarding
organizational cultural models and settings, I looked for demonstrated evidence of shared belief
in diversity, equity and inclusion, and willingness and trust in leadership to adopt a mission-
aligned HRP, as well as the provision of required resources with a commitment to invest in the
successful enactment of the holistic review process. The choice of these KMO elements from my
UNDERREPRESENTED MINORITIES IN MEDICINE 60
conceptual framework is reflective of those found in other settings to be indicative of a robust
HRP.
Observation Procedures
I observed the cohort as they worked as a team in the classroom during their one-day
formal training on the holistic review process in June. I was present for the full training day,
focusing on the discussions, interactions, training process, and shared documents. I collected
handouts including power-point presentations and capture documentation on boards via cell
phone pictures. I documented in pen to have permanency and better visibility and I used a hard
back, bound notepad for the security of the documentation. I chose a notepad rather than a laptop
in the field because it was less obtrusive and easier and faster for me to quickly and quietly jot
down my notes. I was also not dependent on a battery pack or electric power source.
I did, however, have a lap top available as often there are notes pushed out by e-mail or
potentially a web-Ex or Skype medium of interaction that would be good to follow along, but it
was not needed. I did not record the sessions as I believed it may have chilled the collective
experience and participants would not have been as forthcoming. As per Patton (1987) and
Bogdan & Biklen (2007), I took detailed notes, and verbatim as much as possible, and
paraphrased with direct quotes. I did not draw inferences but kept a summary of my observations
regarding body language and physical actions, as well as my own actions and introspective
reactions, in an observer comments section (Patton, 1987). Immediately after the encounter, I
added more recalled details to my notes and reflected upon what happened in terms of what was
said and done and what my own assumptions, biases and reactions were. I then safely secured
my work-product (Bogdan & Biklen, 2007).

UNDERREPRESENTED MINORITIES IN MEDICINE 61
Interviews
Interview protocol. Semi-structured interviews revealed the experiences and worldviews
of the Admissions File Reviewers related to diversity, equity and inclusion, and their knowledge
and motivation surrounding the creation of an HRP in the context of the NSM milieu. This was
the best interview approach for this qualitative data collection because it afforded a mix of more
and less structured questions, the exact wording nor the order of which was necessarily fixed,
allowing me as the researcher to respond to the situation at hand, to the emerging worldview of
the respondent, and to new ideas on the topic (Merriam & Tisdell, 2016).
My interview protocol (Appendix B) began with introductions and an overview of the
goal of the interview. I then engaged the participant in twelve interview questions linked to all
of the potential knowledge, motivational, and organizational influences on the problem of
practice, gleaned from the literature and my own knowledge base, and inextricably tied to my
research questions as outlined in my conceptual framework.
The interview questions flowed from less intimidating general questions, creating a
comfortable rapport, to more personal and probing yet non-threatening questions, respecting and
purposefully soliciting answers reflective of the life and experiences of each participant. The
questions were direct, open-ended and descriptive, not “test-like” or leading, and were written in
clear, concise and familiar language to encourage elaboration about specific experiences
(Bogdan & Biklen, 2007; Merriam & Tisdell, 2016).
The interview protocol included 12 questions in a semi-structured format with three
scripted probe questions to dig deeper if necessary in coherence with the research questions and
the K, M or O influence in question. After the first two introductory questions, each question
UNDERREPRESENTED MINORITIES IN MEDICINE 62
mapped to a specific K, M or O influence from my conceptual framework in alignment with the
research questions, namely, two knowledge influences (factual and procedural); motivational
influences (task value and self-efficacy) and organizational cultural model influences. Ten of the
15 interview questions were from four of the six Patton (2015) types of questions. Two questions
were background/ demographic questions; six questions were opinion and value questions; one
was a feeling question, and one was a factual knowledge question. Of the remaining five
questions, two were from the ideology of Strauss, Schatzman, Bucher, and Sabshin (1981),
utilizing a factual knowledge ideal position question and a devil’s advocate question. The other
three were general interview questions relating to procedural knowledge and motivational task
value. Overall, the interview protocol had a greater emphasis on questions surrounding what
equity and diversity meant to the participants and their motivation to create an HRP, and the
semi-structured format
allowed for data coherence across interviews as well as flexibility in sequencing as needed
(Bogdan & Biklen, 2007). At the end of the interview, participants were given my contact
information to follow up for any reason. In addition, I offered to share interview transcripts with
them to verify accuracy and invite further input as desired.
Interview Procedures. I conducted the interviews by telephone between the months of
October and November of 2019, after the AFRs initial training and after they had begun the
process of holistic review. Prior to the interviews, I had collected and reviewed relevant
documents discussed above, which greatly informed my interview questioning. Because the
cohort was comprised of only 19 participants, it was appropriate and reasonable for me to
individually interview the majority, providing maximal variation (Johnson & Christensen, 2014).
I allowed 90 minutes for each telephone interview and most took 45 – 60 minutes. I took hand-
UNDERREPRESENTED MINORITIES IN MEDICINE 63
written notes during each interview and the interview was also audio-recorded and transcribed
for further analysis and allowing sharing with participants to verify accuracy and completeness.
After each interview, I reviewed my notes and reflected on the dialogue, the goal being to ensure
consistency in data capture across interviews and systematically seek out emergent themes
(Merriam & Tisdell, 2016).
Data Analysis
Data analysis is the process of making meaning from the data (Merriam & Tisdell, 2016).
For this study, I conducted qualitative data analysis of documents, observations and interviews.
Document review yielded background facts and materials used for directed questioning and
contextual observation of participants. I reviewed transcripts of interviews and my notes jotted
during the observations and interviews. I took additional notes while reviewing transcripts to
capture specific thoughts. To document the entire analysis process, a codebook was developed to
capture and document emerging patterns and themes as well as counter-examples, allowing for
holistic and representative exploration into the research questions.
Credibility and Trustworthiness
In qualitative research, because the researcher human being is the primary instrument for
data collection, it is imperative to ascertain that the researcher is a valid and reliable instrument
(Merriam & Tisdell, 2016). To ascertain validity and reliability in a qualitative study involves
careful attention to the study’s conceptualization, data collection, analysis and interpretation, as
well as ensuring that the investigation is done in an ethical manner (Merriam & Tisdell, 2016).
Although it is acknowledged that qualitative researchers can never capture an objective “truth” or
“reality,” there are strategies that can be used to increase the credibility or internal validity and
enhance the rigor and therefore trustworthiness of the qualitative research findings (Merriam &
UNDERREPRESENTED MINORITIES IN MEDICINE 64
Tisdell, 2016). In the qualitative research context, validity and reliability equate to credibility
(internal validity) of the findings, and trustworthiness applies to both the researcher and the
process used to collect and analyze data (Maxwell, 2013; Merriam & Tisdell, 2016). Patton
(2015) tells us that the trustworthiness of the data is tied directly to the trustworthiness of those
who collect and analyze the data, and their demonstrated competence, and that research design,
analytical techniques and procedures do not ensure rigor, which is manifest in rigorous thinking
about everything including methods and analysis. As guided by Merriam and Tisdell (2016), and
Patton (2015), in my process, the rigorous coherence of the literature review, research questions,
conceptual framework, and the interview protocol, created a trustworthy platform. Of note,
reliability in qualitative research, rather than demanding that outsiders get the same results,
wishes outsiders to concur that, given the data collected, the results make sense – they are
consistent, dependable and congruent with the data collected (Merriam & Tisdell, 2016).
To promote validity and reliability and enhance the rigor, credibility and trustworthiness
of my study, I employed the strategies of triangulation, member checks (respondent validation),
adequate engagement in data collection, and reflexivity.
Triangulation was attained by using the three sources of data and data collection methods
described above, namely, documents, observations and interviews, to confirm emerging findings
(Merriam & Tisdell, 2016). I performed the member checks/ respondent validation by taking my
tentative interpretations/ findings from my interviews and observations underpinned by my
document analysis, back to the participants and asking if they are plausible (Merriam & Tisdell,
2016). I had adequate engagement in my data collection by spending enough time collecting data
to the point of becoming “saturated” without any new emergent themes and having the
experience of uncovering a couple discrepant cases (Merriam & Tisdell, 2016). I also engaged
UNDERREPRESENTED MINORITIES IN MEDICINE 65
in reflexivity/ researcher’s position, which is the notion of understanding and sharing how I
affect the research process and how the research process affects me, a concept related to my
integrity as a qualitative researcher (Merriam & Tisdell, 2016). To be considered a trustworthy
researcher during this process, I clearly articulated my positionality and any biases and
assumptions regarding my research while also reflecting upon the reflexivity created when a
human is the primary instrument of data collection (Maxwell, 2013; Merriam & Tisdell, 2016).
As a minority physician, medical school faculty member and current member of the diversity
committee for the medical school, I have a vested interest in the success of the development of
an effective HRP to matriculate a diverse class, representative of the population at large.
However, I am confident that subjecting my research to the rigor of these checks and balances
ensured credibility and trustworthy results.
Ethics
In qualitative research, the burden of producing a study that has been conducted and
disseminated in an ethical manner lies with the individual investigator (Merriam & Tisdell,
2016). Institutional review boards (IRBs), mandated by the federal government, try to ensure
ethical behavior in research but, the obligations of a qualitative researcher go beyond compliance
and governmental and organizational rules (Krueger & Casey, 2009; Rubin & Rubin, 2012). A
core requirement of the IRBs is informed consent is to ensure that the research participants
understand the nature of the research, the risks that it poses, and that they are not forced or
coerced in any way to participate (Rubin & Rubin, 2012). I submitted this study to my
institution’s IRB and followed their rules and guidelines regarding the protection of the rights
and welfare of my research participants.
UNDERREPRESENTED MINORITIES IN MEDICINE 66
Regarding informed consent, my approach to help interviewee AFRs decide whether to
participate in my research study was to (a) fully explain to them who I was (my background,
investigator role); (b) what the purpose of the research was, and (c) what I was asking of them
(interviews to ascertain their knowledge and attitudes of the NSM mission, the holistic review
process as well as any perceived organizational barriers) (Rubin & Rubin, 2012). I assured the
participants as part of the initial informed consent process, and reminded them periodically
throughout the process, that their participation was voluntary and that they could drop out at any
time during the project or refuse to answer any question (Rubin & Rubin, 2012). Prior to the
interviews I got permission to audio record them and offered participants he opportunity to
review transcripts and verify accuracy and also paraphrased responses during the interview to
ascertain that I was understanding what I was being told (Rubin & Rubin, 2012). In addition, and
especially as the first tenet is always to do no harm, I reassured them that, (a) because they own
the work-product of their interview, requests for retractions from collected data would be
granted, (b) no information would be revealed that could diminish their well-being, and (c) data
would be securely stored in a safe environment with locked or encrypted access (Glesne, 2011;
Rubin & Rubin, 2012). Incentives to participate were not provided so as not to exert coercive
pressure but, at the end of the study, I sent each participant a thank you card with a small
monetary gift card as a token of my appreciation for their participation (Rubin & Rubin, 2012).
As stated in Glesne (2011), participants have a right to expect that when they give you
permission to observe and interview, their confidences will be protected, and their anonymity
preserved. To protect confidentiality, I used pseudonyms in the notes and the transcripts and
results were published in the aggregate without individual attribution (Rubin & Rubin, 2012).
Although the Admissions File Reviewers will be known to the school and other interested
UNDERREPRESENTED MINORITIES IN MEDICINE 67
stakeholders, those interviewed will not be publicly known nor will their work-product, unless
they consent to do so if the need arises, unlikely here (Rubin & Rubin, 2012).
As a medical school faculty member and current member of the Diversity, Equity &
Inclusion Committee for the medical school, I do have a vested interest in the success of the
development of an effective holistic admissions program to matriculate a diverse class,
representative of the population at large. I addressed any potential confusion of my role as a
physician of the medical group and my role as a researcher by explaining the dissertation process
and reiterating my investigator hat in this context. I did not have participants in a subordinate
role because I am not an employee of the medical school and therefore not positionally situated
as their boss. However, I am aware of the hierarchical dynamic that exists amongst physicians at
different stages in their careers, with deference to more senior physicians regardless of context. I
thus thought it important to state that there was equal footing, and respect, reminiscent of
Margaret Mead’s statement that, “anthropological research does not have subjects; we work
with informants in an atmosphere of mutual respect” (Glesne, 2011). It was also imperative that
participants knew that future performance evaluations and/or job advancement would not be
influenced in any way by their involvement (Rubin & Rubin, 2012).
In this study, I gathered and made meaning of the information imparted to me through the
interviews, without injecting my opinions (Merriam & Tisdell, 2016). I acknowledge that there
might have been some assumptions of advocacy (Glesne, 2011), and biases that I may have had,
and/or be presumed to have had because, as a black female, I am part of an underrepresented
minority group whose enrollment we are trying to increase.

UNDERREPRESENTED MINORITIES IN MEDICINE 68
Limitations and Delimitations
There were limitations and delimitations that the researcher was aware of as this study
commenced. The limitations were the factors not in my locus of control. The delimitations were
the decisions I made that may have had implications for the study. Limitations that existed for
this study were (a) the dependence on the truthfulness of the participants; and (b) the study
timeframe as a small window of time before the opening of the new medical school, and thus
with limited context. Delimitations that existed for this study were (a) Data collection from one
stakeholder group of AFRs, all physicians and not administrators nor students; and (b) the study
examined the AFRs mastery of the HRP itself without analysis of whether the outcome was the
matriculation of a diverse class.

UNDERREPRESENTED MINORITIES IN MEDICINE 69
CHAPTER FOUR: RESULTS AND FINDINGS
Overview of Purpose and Questions
Utilizing Clark and Estes’ (2008) gap analysis model, the purpose of this study was to
conduct a needs analysis in the areas of knowledge and skill, motivation, and organizational
resources necessary for the Admissions File Reviewers (AFRs) to achieve the organizational
performance goal. The goal was to effectively matriculate a diverse class with students in
numbers proportional to their representation in the general population, including
underrepresented minoritized groups who have traditionally had lower matriculation rates. The
analysis began by generating a list of possible needs to systematically validate. While a complete
needs analysis would have focused on all stakeholders, for practical purposes, the stakeholder
focused on in this analysis was the cohort of AFRs, charged with the initial review of the
applicants to the NSM. The research questions that guided this innovation study follow.
Research Questions
1. What are the knowledge, motivation and organizational needs required for the New
School of Medicine (NSM) Admissions File Reviewers (AFRs) to develop and
effectively enact a holistic review process to matriculate a diverse class each year?
2. What is the interaction between the AFRs knowledge and motivation and the NSM
organizational culture and context?
Participating Stakeholders
The stakeholder group of focus for this study was the group of 19 Admissions File
Reviewers (AFRs) because, a significant way to embed diversity into the school was to
matriculate a diverse student body. The target of the AFRs was to develop an effective holistic
review process with all AFRs able to conduct a holistic review process by July of 2019, to drive
UNDERREPRESENTED MINORITIES IN MEDICINE 70
the attainment of a diverse class of matriculants for the inaugural 2020 year and thereafter. This
measure was chosen because it demonstrates learning and effective application of the
competencies of diversity, equity and inclusion. The consequence of not achieving the goal
would be a diminished capacity to effectively matriculate a diverse student body which would
impede educational excellence and effective care delivery to patients. All the AFRs were
physicians trained in the holistic review process in June of 2019. Fifty-eight percent of the cohort
was interviewed for this study.

Figure 2. AFR cohort by gender.

11
8
AFR Cohort by Gender
female male
UNDERREPRESENTED MINORITIES IN MEDICINE 71

Figure 3. AFR cohort by ethnicity.

Figure 4. AFR participants by location.

5
14
0 2 4 6 8 10 12 14 16
non-ethnic
ethnic
AFR cohort by ethnicity
5
4
1 1 1
0
1
2
3
4
5
6
Southern CA Northern CA Central CA Oregon Colorado
Number of Participants
Location
AFR Participants by Location
UNDERREPRESENTED MINORITIES IN MEDICINE 72

Figure 5. AFR cohort by underrepresented minority in medicine – U.S.

Figure 6. AFR participants by years as a physician.

4
15
URiM Majority representation
2
9
1
0
1
2
3
4
5
6
7
8
9
10
0-10 >10-20 >20
Number of AFRs
Number of Years
AFR Participants by Years as a Physician
UNDERREPRESENTED MINORITIES IN MEDICINE 73
Knowledge
The assumed knowledge needs suggested by literature review for this study were: that
Admissions File Reviewers (AFRs) need to (a) know what a holistic review process is; (b) know
how to conduct a holistic review process; and (c) reflect upon their learning of the holistic
review process. These needs were validated by data collection. It is the ability to effectively
conduct a holistic admissions process that is the stakeholder (AFRs) goal, paramount to
achieving the organizational goal of matriculating a diverse medical school class each year. The
Krathwohl (2002) framework is used to guide the discussion of the validated knowledge needs,
namely (a) declarative knowledge (factual and conceptual); (b) procedural knowledge, and (c)
metacognitive knowledge. Recommendations for the needs are based on cited theoretical
principles discussed in Chapter 5. Table 7 below summarizes the validated knowledge needs.
Admissions File Reviewers need to know what a holistic review process is.
Table 7
Validated Declarative Knowledge Need
Declarative Knowledge –
validated assertion
Findings
Admissions File Reviewers (AFRs)
need to know what the holistic
review process is
• Some had prior knowledge of holistic review (HR)

• Most gleaned their knowledge from the NSM training
only

• All clearly understood the value of diversity:
background – sociocultural; uniqueness; geographic;
age; nationality; native language; gender; sexual
orientation; experiences; first-generation college student

• Most able to articulate equity and its application to the
HR process; looking at distance traveled – each
person’s journey; leveling the playing field

• Fewer able to articulate what inclusion means in the
context of the class; hadn’t thought that far

UNDERREPRESENTED MINORITIES IN MEDICINE 74
Assertion #1: AFRs need to know what the holistic review process is. The analysis of
this assertion addresses the knowledge component of the first research question regarding the
knowledge, motivation and organizational needs required for the New School of Medicine
(NSM) Admissions File Reviewers (AFRs) to develop and effectively enact a holistic review
process to matriculate a diverse class each year. The findings of this study indicate that all
admissions file reviewers realized the knowledge needs of the holistic review process, namely
(a) what a holistic review process is; (b) how to conduct a holistic review process; and, (c) the
ability to reflect upon their learning of the holistic review process.
Although starting with different levels of awareness of the process, most gleaned in-depth
declarative knowledge about the holistic review process during the training. To fully understand
the considerations underpinning holistic review as well as the rationale and process, it was
imperative that they understood the basic tenet of holistic review which is centered on achieving
diversity and equity in the context of with the mission specific to the school. Although aware of
the notion of looking at the whole person and not just at academic scores, most needed the rigor
of the training to fully conceptualize the process. AFR 6 thought that “they gave us a lot of very
helpful information in a relatively short amount of time.” AFR 1, in the minority stated: “for me,
it was not so much about learning the process, because I do it a lot” but emphasized “the
importance of the mission of NSM as the basis for the holistic review process.” The group
uniformly identified important considerations for review beyond grade point average (GPA) and
Medical College Admission Test (MCAT) scores, shown in Figure 7. AFR 10 stated that “the
benefit of holistic review is that you have more of a potential not to miss somebody who would
do super-well at this career just by discounting them based off of numbers.”
UNDERREPRESENTED MINORITIES IN MEDICINE 75

Figure 7. Important non-academic qualities for medical students to have identified by AFR
cohort.

Regarding diversity, AFR 8 stated that “it is absolutely imperative that we increase the
number of underrepresented minorities in medicine…we can’t take as good care of our patients
unless they are able to see themselves in the physician group….When you have a physician that
is of the same ethnic group as the patient they can connect differently with the patient. They
know more about that culture and the outcomes are better…to me, having a diverse class will
have far reaching outcomes that we can’t even fathom right now. I’m just thinking if in a
community where historically they have never had a doctor that looked like them and suddenly
they do, you are talking about changes in health and outcomes and quality of life that previously
we would never have thought were possible to achieve.”
AFR 11 concurred, stating “I think patients really relate better, no matter how good a
physician you are, to physicians that are similar in background or ethnicity to them.” AFR 2
stated that populating our medical schools with representatives of the overall population brings
awareness. It gives a voice to populations that generally don’t have one. Hopefully, by bringing
that awareness we can start to equalize some of those disparities that we see in health care.” AFR
diversity
inclusivity
humility
good letters
drive
innovation
good work ethic
volunteerism
leadership
advocacy
tenacity
grit
critical thinking skills
resilience
important non-academic qualities for medical
students to have identified by AFR cohort
UNDERREPRESENTED MINORITIES IN MEDICINE 76
2 also stated that “diversity makes your future interactions better, and perhaps erases some of
those biases that you may have come into the situation with.”
Regarding equity, AFR 2 said: “…what opportunities each person had…working so
many jobs that they didn’t have time to volunteer…not being able to afford test prep classes and
maybe that’s why their scores are a little bit lower…first generation or immigrant trying to
assimilate and learn the language. Equity to me is taking all these things into account for the
person to explain their application.” AFR 11 stated: “it’s giving people of different
circumstances and backgrounds an equal shot at going to medical school.” AFR 8 stated that
“equity doesn’t mean that everybody gets the same thing. To me, it means everybody gets the
tools that they need so that they can be successful.” AFR 10 said: “ I had one person who was
homeless who had struggled in high school then went to community college, got decent grades
considering they were still homeless and then transferred to a four-year college to finish… they
had a couple of jobs while doing that and got mostly B pluses…but that’s phenomenal compared
to somebody who never had a job in their life …and spent all their time doing research which is
important and great on its own…But the other applicant may not have been able to do any
research because they were working and they couldn’t study necessarily as much or do as well
because they were just trying to get through. I just think that there is so much value in that
person’s life experience to contribute and they’ve shown that they can still get it done. Which is
all you really need…for the person to be able to succeed through this program. They can’t have
C’s and D’s and stuff like that, but for the most part, these applicants do exceedingly well despite
challenges in their background.” AFR 8 stated: “you want to make sure that your students will be
prepared and be able to pass their examinations, but I don’t think that just looking at that alone is
adequate at all.”
UNDERREPRESENTED MINORITIES IN MEDICINE 77
In response to whether taking people with lower scores would diminish the status of the
school, 95% disagreed. AFR 1 said: “If you define your school by numbers, then that’s probably
true. But we shouldn’t define our school by numbers; we should define our school by our
students and what they accomplish, and what they do.” AFR 4 stated that “the whole belief that
underrepresented minorities historically don’t do as well on standardized testing may be true, but
I think we do just as well as anybody else on our clinical rotations and taking care of patients.”
AFR 2 noted that “in the past when reviewing applications, focus was only on scores and if they
weren’t at a certain level, they went right out the door for us. Now, in retrospect what a terrible
way to evaluate people especially since their job was going to be taking care of patients. I try to
find a theme in the application to kind of show me who this person really is more than just a
single experience or score. ”
Of note, AFR 1 emphasized the importance of “equity between the people who are
screening…making sure that everyone is up to a certain standard in terms of understanding their
own biases, the mission of the school, and are screening applications on an equal basis. ” AFR 1
said: “maybe put everyone through a bias test…put people through some self-reflection…stating
“I’m a brown man, how could I be biased and then with self-reflection doing the bias test I said
“Oh, I didn’t realize I was doing that.” AFR 1 stated: “ I knew I was biased, I just didn’t realize
how much. If someone went to the same college as me, I might give them a plus for that because
I know it was a good school because I went to it. Now I try to look more at who they are and
other factors… It’s all about the person’s journey and their story, not necessarily their scores or
where they went to school.”
As regards inclusion, demonstrated by acceptance and compassion, AFR 10 stated that
“it’s being open to all people no matter their background and celebrating their backgrounds…not
UNDERREPRESENTED MINORITIES IN MEDICINE 78
ignoring the differences … but celebrating the differences and really understanding where
everybody is coming from… acknowledging your biases and really being open to all groups.”
AFR 4 stated about inclusion: “ it’s when people feel valued despite their differences … they feel
that their differences are positive and of great value to the group they’re part of, and that they
don’t feel like there’s anything being held against them because of their background.”
AFR1 noted that it’s important to be open to all but stated that “when folks are given an
opportunity, I see a lot of cases of the so-called imposter syndrome… feelings from them that
they are not comfortable in this new environment that they have suddenly entered in to. If you
don’t provide opportunity and training for each individual to help them in different areas of
perceived weaknesses and strengths, you can have all the diversity you want, but you’re not
going to be able to use it as a strength.” AFR 8 stated that inclusion is making sure everyone has
a seat at the table… a voice… and an opportunity for representation.”
Assertion #2: AFRs need to know how to conduct a holistic review process. From the
procedural knowledge perspective, the results and findings of this study, summarized in Table 8
below, indicated that all AFRs needed procedural knowledge about how to conduct holistic
review for admissions.
Table 8
Validated Procedural Knowledge Need
Procedural knowledge:
Validated Need/Assertion
Findings

AFRs need to know how to conduct a
holistic review process

• Most learned to conduct the HR process by reviewing 2
cases in the training and then by doing actual reviews

• Some were able to partake in monthly call-in follow-up
meetings to ask questions

• Most “hit their stride” after 3-4 weeks of reviewing 5-
10 cases a week
UNDERREPRESENTED MINORITIES IN MEDICINE 79
AFR 1 had prior knowledge enabling seamless application to this process stating: “it was
not so much about learning the process because I do it a lot.” The others learned how to conduct
the HR process during the training. AFR 5 stated that the orientation, practicing and discussions
were very helpful as was having ongoing dialogue with other file reviewers and the admissions
team at monthly meetings. AFR 8 noted that “the system used to type the notes in could be
streamlined…some parts of it are redundant and don’t match up with the way the PDF of the
application comes to us.” Most desired more time to practice before launching in to reviews and
more time to calibrate with the cohort. AFR 2 stated: “They gave us a small amount of time to
rehearse, I’ve gotten feedback a couple of times on my applications though more would me
nicer.” AFR 5 said: “We did not have much of an opportunity to calibrate as a group. I’m hoping
that happens in the future so we (AFRs) can be consistent in our process.” Most AFRs “hit their
stride” a month of reviewing five to ten cases a week.
Assertion #3: AFRs need to reflect upon their learning of the holistic review process.
From the metacognitive knowledge perspective, the results and findings of this study,
summarized in Table 9 below, indicated that AFRs need to reflect about how to conduct holistic
review for admissions.
Table 9
Validated Metacognitive Knowledge Need
Metacognitive knowledge –
Validated Need/Assertion
Findings

AFRs need to reflect upon their
learning of the holistic review process
- Implicit bias training valuable and should be a
defined accountability in order to proceed
- Need more calibration amongst the cohort to
establish inter-rater reliability
- It was helpful for those AFRs who focused on
potential outcomes of their efforts – creating an
ideal “mission-fit” class for NSM

UNDERREPRESENTED MINORITIES IN MEDICINE 80
The results and findings of this study indicated that Admissions File Reviewers needed to
reflect on their learning of the holistic review process. AFR 2 said: “It has been slow for me. I
do think that I am improving with each batch of applications that I do in terms of (a) efficiency
and (b) I think I’m getting a little bit better at detecting things like sustained activities, defining a
theme and seeing who the person is and how I think that they would fit into the med school.”
AFR 5 stated, “the best I could do is to always constantly check my biases. Every time I was
reviewing an application and I was so quick to make a decision I would ask myself why I was
quick to make that decision. I think there has definitely been an increased comfort level for me
over time, but I’m still constantly sitting back and thinking, well, why do I think this way about
this applicant? Is there some bias that I have that is making me think one way or another? As I
learn about each applicant, it’s a learning process for myself as well.” AFR 4 stated that I learned
a lot about my own personal biases, how to recognize them and be cautious of them by doing
different implicit bias tests on the computer before reviewing applications.” AFR 1 said: “maybe
put everyone through a bias test…put people through some self-reflection…stating “I’m a brown
man, how could I be biased and then with self-reflection doing the bias test I said “oh, I didn’t
realize I was doing that. I knew I was biased; I just didn’t realize how much. If someone went to
the same college as me, I might give them a plus for that because I know it was a good school
because I went to it. Now I try to look more at who they are and other factors… It’s all about the
person’s journey and their story, not necessarily their scores or where they went to school.”
Motivation
The assumed motivation needs for this study were as follows: (a) Admissions File
Reviewers (AFRs) need to believe that learning about the holistic admissions process was
UNDERREPRESENTED MINORITIES IN MEDICINE 81
professionally important and valuable; and (b) AFRs need to believe that they can learn the
holistic admissions process.
Assertion # 4: AFRs need to believe that learning about the holistic review process is
important and valuable for them. The analysis of this assertion addresses the motivation
component of the first research question regarding the knowledge, motivation and organizational
needs required for the New School of Medicine (NSM) Admissions File Reviewers (AFRs) to
develop and effectively enact a holistic review process to matriculate a diverse class each year.
The results and findings of this study indicated that all Admissions File Reviewers believed that
the holistic review process (HRP) was professionally important and valuable for them to learn.
Table 10
Validated Motivation Need – Expectancy/ Task value
Motivation – Expectancy/ Task value
Validated Need/ Assertion
Findings

AFRs need to believe that learning about the
holistic admissions process is professionally
important and valuable for them
1. All AFRs believed that learning about
the HR process was important &
valuable for them and recognized the
value of HR itself

2. After reviews started, all but one AFR
believed that learning about the HR
process was necessary for them to
conduct the admissions process
specific to the New School of
Medicine. The outlier thought that her
batches of applicants reviewed could
have matriculated anywhere regardless
of HR

AFR 1 believed that the NSM admissions process could not have been done without
learning the holistic review skill set because at the center of the HRP was “the NSM mission
statement, the whole reason behind the message of the school.” It was the latter point about the
UNDERREPRESENTED MINORITIES IN MEDICINE 82
specific NSM mission that was not acknowledged by AFR 3 who stated that her batches of
applicants could have gotten in anywhere regardless of holistic review. AFR 5 stated “I think
there’s definitely value for me in learning the HRP. I think if it wasn’t holistic, then easily
there’s a couple of candidates that come to mind that would probably have been rated lower and
we would probably miss out on somebody is my feeling. There are a few candidates that I think
of that really had nontraditional applications but really could add something of value to the
medical school…I think the HRP helps bring to light all the potential candidates that can really
be a good mission-fit for the school.” AFR 10 stated “I don’t even know what the point of a file
reviewer would really have been if it wasn’t going to be holistic … I think that it’s really
valuable.” “AFR 9” expressed the majority perspective stating that “a lot of times I look at an
application and I feel like the candidate will probably get into many medical schools but not
necessarily NSM…their goals and mission didn’t align as well with NSM. They may have
excellent grades and research and very good experiences…but it’s like there’s something
missing.” AFR 1 stated that “having the ability to have that impact of finding people who in
many places would have been lost from the system and giving people opportunities was my
driving force.”
Assertion #5: AFRs need to believe they can learn the holistic review process. The
analysis of this assertion addresses the motivation component of the first research question
regarding the knowledge, motivation and organizational needs required for the New School of
Medicine (NSM) Admissions File Reviewers (AFRs) to develop and effectively enact a holistic
review process to matriculate a diverse class each year, paramount to achieving the
organizational goal of matriculating a diverse medical school class each year. This need was
validated by data collection revealing that the Admissions File Reviewers were confident that
UNDERREPRESENTED MINORITIES IN MEDICINE 83
they could learn the process but several acknowledged nervousness in the actual doing of the
process and that it took them longer to master than they thought it would. What AFR 2 stated
about learning the HR process was: “I can do this. I can figure this out. I went to medical
school…I know who should go to medical school.” Then stated: “actually I learned that there’s
more to it than just, I like that guy…drilling down learning a new way to look at things was
really helpful and once learned you say oh yes, this is the way to do it.” AFR 5 stated, “I think
initially I felt a little bit of nervousness and anxiety that I may somehow not do a good enough
job where I might miss something or may not provide an opportunity to a candidate who
deserves one.” AFR 2 stated “there was just this little bit of ongoing apprehension running
through, I think all of us had thoughts of okay well, just let us know if we’re screwing it up.”
AFR 8 stated: “I’ll admit I was a little nervous when I started because I wanted to do a good
job….I felt fairly confident when I was reviewing. I took a long time with each application…I
know I’m not as fast as some of the other file reviewers, but I don’t know how to get faster
without sacrificing quality, but I felt pretty confident with the process.” Table 11 below
summarizes the validated motivation need of self-efficacy.
Table 11
Validated Motivation Need: Self-Efficacy
Motivation – Self-efficacy –
Validated Need/ Assertion Findings
AFRs need to believe they can learn the
holistic review process.
1. Each AFR did believe that they could
learn the HR process
2. However, several:
(a)were nervous to start the reviews
noting a big gap between theory and
doing; and
(b)noted that it took longer for them to
“master” the process than they thought
it would

UNDERREPRESENTED MINORITIES IN MEDICINE 84
Organization
The assumed organizational needs for this study were as follows: (a) NSM leadership
needs the trust of AFRs to adopt a holistic review process (HRP); (b) NSM leadership and the
AFRs need to share the belief that it is important to matriculate a diverse class each year; and (c)
The organization needs to provide professional development training on the holistic admissions
process for the Admissions File Reviewers. These needs were validated by the study and Tables
12, 13 and 14 below summarize the validated organizational needs and findings.
Assertion #6: NSM leadership needs the trust of the Admissions File Reviewers
(AFRs) to create an effective holistic review process (HRP). The analysis of this assertion
primarily addresses the second research question regarding the interaction between the AFRs and
the NSM organizational culture and context. The findings of this study summarized in Table 12
below, validate the need for NSM leadership to have the trust of the AFRs.
Table 12
Validated Organization Need - Trust
Organization: Cultural Model
Validated Need/ Assertion- Trust
Findings
NSM leadership needs the trust of
AFRs to create an effective holistic
review process
1. All AFRs asserted that the New School of
Medicine leadership needs their trust to adopt
and effective HR process and that they, the
AFRs need the trust of the NSM.

2. Factors impeding trust
• open solicitation of AFRs without intentional
tapping in to known existing expertise
• uncompensated training time
• uncompensated time to review (versus other
admissions roles)
• infringement on personal/ family time →
resentment and deterred future participation
• Perceived inequities across the Regions
regarding time and compensation for reviews

UNDERREPRESENTED MINORITIES IN MEDICINE 85
The results and findings of this study indicate that the New School of Medicine
leadership has the trust of most of the Admissions File Reviewers. Impediments to trust were
identified based on perceived inconsistent approaches around the regions of the organization,
lack of compensation for the training and reviews and how the cohort was chosen. AFR 1 noted
that it would have been better to intentionally seek out people who were already familiar with
holistic review to build and engage in the process for the first year, stating: “We have a lot of
graduate medical education people in the organization. I feel like they should have actively
recruited and leveraged people that had the skillsets of holistic review already. Then have those
people teach more people. When they were soliciting, an e-mail went out to every doctor
regardless of your expertise.”
Another perception of AFR 1 was that, “if you were coming from “Region A,” it was
very easy for you to get to meetings and you were also having your time blocked. To participate
in “Region B,’ it was after hours, nights and weekends with no time given for it. AFR 8 stated
that “the chief of my department wrote that she can do it if it never interferes with her clinic
time.” She then got an e-mail from finance saying “oh, we heard you are doing this work and we
should be getting reimbursed for it” and had to inform them that it was a volunteer position. In
addition, because most AFRs were reviewing files after-hours without set-aside times for
reviews, there was infringement on personal and family time leading to resentment and deterring
future participation. When asked if they would be an AFR again, AFR 1 stated: “I don’t know, I
want to do it. If expectations could be made clear, with dates set far in advance, and I didn’t have
to pay my own way to travel in, I would strongly consider it.” AFR 11 stated : “I think I would
have liked to have seen a little more direction and clarity about the expectations of the
application review.
UNDERREPRESENTED MINORITIES IN MEDICINE 86
The interviews validated the need for the NSM leadership to have the trust of the AFRs.
Assertion #7: NSM leadership and the AFRs need to share the belief that it is
important to matriculate a diverse class each year. The analysis of this assertion primarily
addresses the second research question regarding the interaction between the AFRs and the NSM
organizational culture and context. The findings of this study summarized in Table 13 below,
validate the need for NSM leadership and AFRs to share the belief that it is important to
matriculate a diverse class each year.
Table 13
Validated Organizational Need: Shared Belief
Organization – Cultural Model:
Validated Need/ Assertion- shared belief
Findings
NSM leadership and the AFRs need to share
the belief that it is important to matriculate a
diverse class each year
1. All AFRs acknowledged a strong and
consistent shared belief with the New
School of Medicine leadership of the
importance to matriculate a diverse
class each year

2. The AFRs uniformly stated that this
was well-addressed and demonstrated
during training

The findings of this study show that there was a uniformly shared belief of the
importance of matriculating a diverse class each year between the New School of Medicine
leadership and the Admissions File Reviewers. AFR 2 stated: “ I definitely get the sense from
leadership and the AFRs that we share a belief that it is important to matriculate a diverse class
each year. I feel very much that any contact we made, whether e-mail or an actual meeting in
person, that that has 100% been the message.” AFR 8 said: “Oh yes, I definitely think so, ” and
the other AFRs also stated that indeed there was a shared belief.
UNDERREPRESENTED MINORITIES IN MEDICINE 87
Assertion # 8: The New School of Medicine (NSM) needs to provide professional
development (PD) training on the holistic review process for the AFRs, with set-aside time
to participate, as well as the opportunity to rehearse, receive feedback and calibrate. The
analysis of this assertion primarily addresses the second research question regarding the
interaction between the AFRs and the NSM organizational culture and context. The findings of
this study summarized in Table 12 below, validate the need for NSM to provide professional
development (PD) training on the holistic review process for the AFRs, with set-aside time to
participate, as well as the opportunity to rehearse, receive feedback and calibrate.
Table 14
Validated Organization need: Provision of Professional Development Training
Organization – Cultural Setting:
Validated Need/ Assertion – provision of
professional development training
Findings
The organization needs to provide
professional development training on the
holistic admissions process for the AFRs,
inclusive of guided assessment and
monitoring, to enable effective file review.
1. Most found the in-person training
effective for the task. Some found it
to be rushed without enough practice.
2. Many AFRs found the timing of
monthly meetings inconvenient and
thus inaccessible; only some were
able to partake in monthly call-in
meetings to ask questions

3. Many AFRs stated that there was too
little practice time and minimal
calibration of the cohort; some sought
calibration with others outside of
cohort

4. Most AFRs wanted more direct and
timely feedback

The findings of this study show the importance for the New School of Medicine to
provide effective professional development training on the holistic review process to the
Admissions File Reviewers cohort. Most AFRs found the training adequate to be able to do the
UNDERREPRESENTED MINORITIES IN MEDICINE 88
task of application review, but some found it rushed without enough practice time. In addition,
there was assigned pre- and ongoing work (articles and Implicit Association Testing) as well as
monthly virtual follow-up debriefs. However, many could not attend the monthly meetings, nor
did they necessarily do the recommended asynchronous work.
Regarding the in-person training, AFR 6 said: “I think I was given as much training as
you could do without actually going through a bunch of applications to see the breadth and
nuances. So I guess the training wasn’t enough to master the process.” AFR 1 stated: “ I felt it
was rushed and there was a lot of you’ll see this and you’ll see that…I didn’t feel there was
adequate time to actually practice. We could have done a day of training and a day of simulation.
I also felt like the application process timeline was not clear. I just got an e-mail saying…oh
yeah, there’s applications.” AFR 3 said: “…we’ve been really cut loose and have no idea how we
are doing….yeah you can join on Microsoft teams and I have, but sometimes it’s really hard to
do at lunch if you’re not local and have a different schedule, or at seven because I start clinic at
seven.” AFR 1 stated that at seven on Fridays I was busy making lunch, packing boxes and
screaming at the kids to get ready for school. I know that time was probably chosen because
most people could attend, but anyone with kids knows that is the worst time in the world.” AFR
2 stated: “They gave us a small amount of time to rehearse. It would have been nice to have more
time. I’ve gotten feedback a couple of times on my applications though more would be nicer. I
feel like that’s been a major difficulty of the process so far. Knowing whether I was writing
enough in the narrative or whether I needed to write more would have been helpful for the first
handful.” AFR 6 said: “I thought the training was fairly effective. They gave us a lot of very
helpful information in a relatively short amount of time…Maybe for all of us being brand new to
the med school admissions process, having time to do more examples together would be good.”
UNDERREPRESENTED MINORITIES IN MEDICINE 89
All the AFRs stated it would have been great to have had more time to calibrate with the cohort.
AFR 8 said: “It would be nice to not just review, but to actually be able to see in the system how
others are doing it…maybe there’s a step I could eliminate where I could still be thorough but be
more efficient.” AFR 5 stated: “I definitely hope that we incorporate more time to calibrate in the
future.” AFR 6 said: “It would have been nice to calibrate more than the two times during the
training. In the ones we did, the “faults” of the candidate were obvious, but once you start doing
the reviews you realize how small details can make a big difference. So going over a few more
applications together would have been helpful. But I do feel that the ability to get feedback, and
give feedback along the way, and hear other people’s feedback was helpful in the meetings we
have held since.”
The interviews of the AFR cohort validated the organization need to provide effective
professional development training on the holistic review process for the AFRs, inclusive of
guided assessment, feedback and monitoring, to enable effective file review.

UNDERREPRESENTED MINORITIES IN MEDICINE 90
CHAPTER FIVE: DISCUSSION AND RECOMMENDATIONS
Chapter Five outlines recommendations based on the validated knowledge, motivation
and organization needs validated in this study and concludes with a proposed implementation
and evaluation plan for the recommendations.
Knowledge Recommendations
Table 15
Summary of Knowledge Needs and Recommendations
Validated Knowledge Need Context-Specific Recommendation
Admissions File Reviewers
(AFRs) need to know what the
holistic review process is
1. Provide AFRs information on the important
points of the HRP with accompanying
illustrations of alternate scenarios.
2. Pre- and post-test (multiple choice & open-ended
questions) for the AFRs to encompass:
• Mission, vision, values of NSM
• Mission-driven holistic review factors
• Completion of Implicit Associations Test
(IAT) to understand their biases
• Know factors to consider the MCAT and
GPA together to ascertain academic
readiness
• Know the layout of the Admissions
Application Management System
• Understand the layout and content of the
application and recognize “red flags”
• Factors to include when crafting a
comprehensive narrative about each
applicant
3. Validate that any assigned pre-work is done:
(e.g., IAT, articles to read, videos to watch):
It should be an accountability not an option
AFRs need to know how to
conduct a holistic review process
1. Provide AFRs a job aid on each of the steps of
the holistic review process and model strategies
for various scenarios
2. Allow time for practice and calibration amongst
the cohort
3. Create multiple times for follow-up meetings to
enable sustained engagement
UNDERREPRESENTED MINORITIES IN MEDICINE 91
Table 15, continued

Validated Knowledge Need

Context-Specific Recommendation
AFRs need to reflect upon their
learning of the holistic review
process
1. Provide AFRs opportunities to self-monitor and
self-assess their holistic review processes with a
view to making critical alterations and
improvements.
2. Build in reflection time during training and
subsequent meetings:
• Calibration exercises within the cohort
• Tweaking their HR process technique with
iterative application of their learnings
• Keeping qualities of an ideal class top of mind:
applying the concepts of diversity, equity &
inclusion
• Keeping the goal top of mind: to proportionately
increase the number of underrepresented
minorities in medicine and decrease the health
care disparities gap

AFRs need to know what the holistic review process is. The results and findings of
this study indicated that 90% of the admissions file reviewers needed more in-depth declarative
knowledge about the holistic review process. Recommendations rooted in information
processing system theory have been selected to close this declarative knowledge gap. Schraw
and McCrudden (2006) found that to develop mastery, individuals must acquire component
skills, practice integrating them, and know when to apply what they have learned. This would
suggest that helping learners identify and understand important points as well as providing
experiences that help them make sense of the material rather than just focusing on memorization,
would support their learning. The recommendation then is to provide AFRs with information on
the important points of the HRP with accompanying illustrations of alternate scenarios.
The training and learning objectives for the holistic admissions process need to be
specific (Daly, 2009). Building on their prior knowledge of admissions processes with provision
of information about holistic review, AFRs will be able to, as described by Schraw and
UNDERREPRESENTED MINORITIES IN MEDICINE 92
McCrudden (2006), meaningfully organize and connect new knowledge to prior knowledge to
construct meaning. Mayer (2011), has also shown that it will be beneficial to have the
Admissions File Reviewers identify pre-training, what they do and do not know about the
holistic admissions process, and to break the content into parts, providing pre-training for more
complex parts of the model to avoid cognitive overload of the team members. Schraw and
McCrudden (2006), also note that AFRs will develop mastery by acquiring component skills,
practicing the integration of these skills into their work, and understanding when to apply their
learning.
Admission File Reviewers need to know how to conduct a holistic review process.
The results and findings of this study indicated that 90% of the AFRs needed more in-depth
procedural knowledge about how to conduct holistic review for admissions. Recommendations
rooted in information processing theory have been selected to close this procedural knowledge
gap. Schraw and McCrudden (2006) found that modeling effective strategy use, including
“how” and “when” to use different strategies promotes learning and continued practice takes less
capacity in working memory and promotes automaticity and transfer. This would suggest that
providing AFRs with a job aid on each of the steps of the holistic review process and modeling
strategies for various scenarios would support their learning. The recommendations then, are to
provide job aids for each step of conducting holistic review as well as modeling each step for the
AFRs and having them practice each step.
As espoused by Mayer (2011), it is important to provide the AFRs with frequent
opportunities to practice the administration of the holistic review process with ongoing feedback
to allow transfer. Admissions File Reviewers need to be given time to observe credible and
enthusiastic models who have mastered holistic admissions. After seeing a demonstration of the
UNDERREPRESENTED MINORITIES IN MEDICINE 93
process, the AFRs should organize, rehearse, and then practice what they have learned (Mayer,
2011). It is also important to provide immediate feedback and reinforcement during the process
(Tuckman, 2009), which should be private, specific, and linked to the use of strategies to
improve performance (Shute, 2008).
AFRs need to reflect upon their learning of the holistic review process. The results
and findings of this study indicated that Admissions File Reviewers need to reflect on their
knowledge about the holistic review process. A recommendation rooted in information
processing system theory has been selected to help facilitate metacognition. Baker (2006), found
that the use of metacognitive strategies such as providing opportunities for learners to engage in
guided self-monitoring and to debrief the thinking process upon completion of learning task
facilitates learning. This would suggest and it is recommended, that providing the AFRs
opportunities to self-monitor and self-assess their holistic review processes with a view to
making critical alterations and improvements, would support their learning. This
recommendation also embraces analysis of the results of the Implicit Association Test (IAT),
with reflection as to their meaning and application of learnings during subsequent holistic
review.
In addition to the strategies of providing opportunities for AFRs to engage in guided self-
monitoring and self-assessment and for AFRs to debrief the thinking process upon completion of
learning a task (Baker, 2006), the author espouses that AFRs model their own metacognitive
processes by talking out loud and assessing strengths and weaknesses. Mayer (2011), also
promotes the strategy of having learners identify prior knowledge, that is, what they know and
do not know about a topic before learning a task.
UNDERREPRESENTED MINORITIES IN MEDICINE 94
Motivation Recommendations
Table 16
Summary of Motivation Needs and Recommendations
Validated Motivation Need Context-Specific Recommendation
Expectancy/ Task Value Theory –
AFRs need to believe that learning about
the holistic admissions process is
professionally important and valuable for
them.
1. Provide AFRs with relevant materials and
activities about the holistic admissions
process, which connect to their interests and
demonstrate real-world implications of the
holistic admissions process.
2. Assess value to them by:
• observing engagement (pre-work completed;
active participation during training)
• asking (survey, interview) the relevance to
them and what they see as potential
outcomes
• Ascertaining the reason for their
commitment
Self-Efficacy - AFRs need to believe
they can learn the holistic admissions
process.
1. Provide AFRS a respected peer model who
provides guided practice on the holistic
admission process and affords them
immediate targeted feedback.
2. Specifically ask about their belief that they
can learn it and their confidence that they
can do it
3. Provide assurance that they will be asked to
proceed when mutually deemed ready

AFRs need to believe that learning about the holistic admissions process is
important and valuable for them. The results and findings of this study indicate that
Admission File Reviewers need to believe that the holistic review process (HRP) was
professionally important and valuable for them to learn. A recommendation rooted in expectancy
value theory supports this need. Eccles (2006) and Pintrich (2003) found that including a
discussion of the importance and utility value of the work or learning can help learners (here, the
AFRs) develop positive values. This would suggest that having discussions with the AFRs
UNDERREPRESENTED MINORITIES IN MEDICINE 95
concerning the importance and utility of learning about the HRP would increase their expectancy
value. The recommendation then is to provide AFRs with relevant materials and activities about
the HRP, which connect to their interests and demonstrate real-world implications of the HRP.
Eccles’ (2006), explains in the expectancy value motivation theory that individuals are
more likely to be engaged and successfully attain a goal when they place a high value on the
goal. Assessments typically focus on the learner’s expectations of outcomes and the belief that a
given behavior will or will not lead to a given outcome, as well as the learner’s belief that
expected outcome significantly influences the actual outcome (Mayer, 2011; Rueda, 2011). In
this study, the value recognition was acknowledged during effective training by competent,
experienced and enthusiastic models who shared their outcomes of real-world application of the
process over time (Eccles, 2006; Pintrich, 2003). In addition, value recognition was attained by
demonstrating the link between the matriculation of diverse classes of medical students each
year and mitigation of the health care disparities gap (AAMC, 2013).
AFRs need to believe they can learn the holistic admissions process. The results and
findings of this study indicated that the Admission File Reviewers were confident that they could
learn the holistic review process (HRP). Some, however, were not as confident in actually doing
the process. A recommendation rooted in self-efficacy theory has been selected to close this gap.
Pajares (2006) found that modeling and feedback increases self-efficacy and high self-efficacy
can positively influence motivation. This would suggest that providing the AFRs with a
demonstration of what they need to do followed by feedback on their performance would
increase their self-efficacy. The recommendation then is to provide AFRs with a respected peer
model who provides guided practice on the holistic review process and affords them immediate
targeted feedback.
UNDERREPRESENTED MINORITIES IN MEDICINE 96
Clark and Estes (2008) state that “beliefs are (almost) everything” suggesting that when
individuals have positive beliefs about their ability to do something, they are more likely to
pursue the goal and increase performance. Just as engagement and participation in skills training
has been positively correlated with the strengthening of self-efficacy and performance in medical
students (Stegers-Jager, Cohen-Schotamus, & Themmen, 2012), such is the expectation for the
AFRs with robust participation in skills training for the holistic admissions process. Mayer
(2011), notes that if the AFRs have the belief that they will successfully learn, this will provide
the motivation to persist in the task and to exert the mental effort necessary to achieve. From a
theoretical perspective, then, it appears that increasing the self-efficacy of the AFRs through
modeling, practice, and targeted feedback would increase performance in learning the HRP to
enable the matriculation of a diverse class each year.
Organization Recommendations
Table 17
Organization Need and Recommendations – Cultural Model: Trust
Validated
Organization
Need

Context-Specific Recommendation
Cultural Model:
NSM leadership
needs the trust of
AFRs to create an
effective holistic
review process
(HRP)
• NSM leadership needs to communicate constantly and candidly with
the AFRs and solicit feedback in an ongoing fashion.
• Leverage existing expertise to create a unified, calibrated cohort
• Create clear timelines and expectations for application review
• For people in the AFR role outside of the local area, pay their way
for the training
• Create a compensation system, equitable across the Regions, for the
file review work which should be set-aside time and not infringing
on personal/ family time, which is becoming burdensome and for
some, deterring commitment to future service

UNDERREPRESENTED MINORITIES IN MEDICINE 97
Literature review noted that organizational culture filters and affects any attempt to
improve performance (here, the stakeholder AFRs developing and effective holistic review
process), and successful performance improvement depends on taking specific organizational
culture into account (Clark & Estes, 2008), in order to achieve the organizational goal of
matriculating a diverse medical school class each year. The organizational needs are high
priority in that each affects all stakeholders, is feasible to address, and has a high impact
towards the achievement of the stakeholder goals. The frameworks of Gallimore and Goldenberg
(2001), Schein (2004) , Clark and Estes (2008) and Agocs (1997), guide the discussion.
Recommendations for the needs are based on cited theoretical principles.
NSM leadership needs the trust of the Admissions File Reviewers (AFRs) to create
an effective holistic review process (HRP). The findings of this study indicate that the New
School of Medicine leadership needs the trust of the Admissions File Reviewers. A
recommendation rooted in communication theory has been selected to close this organizational
gap. Creating and maintaining a culture of trust is critically important to achieving stakeholder
goals (Clark & Estes, 2008). Ongoing communication, solicitation of feedback and
demonstration of concern by leadership consistent with organizational policies will facilitate
such trust (Agocs, 1997; Clark & Estes, 2008). This would suggest that NSM leadership needs to
communicate constantly and candidly with the AFRs and solicit feedback in an ongoing fashion
in order to garner the trust of all the AFRs.
Agocs (1997), states that to counteract resistance to change, the change message and the
messenger need to be deemed credible and the recipients of the message need to accept and
support the responsibility to master the change. Having ongoing open communication,
solicitation of feedback, and demonstration of concern by leadership consistent with
UNDERREPRESENTED MINORITIES IN MEDICINE 98
organizational policies will facilitate such trust (Agocs, 1997; Berger, 2014; Koorsgaard, Brodt,
& Whitener, 2002). As such, it appears that the literature supports the necessity for creating and
maintaining a culture of trust between AFRs and NSM leadership by bridging the
communication gap and addressing issues of concern to the AFRs (feedback, lack of set-aside
time, compensation, perceived inter-regional cohort inequities), with a view to making critical
alterations and improvements, thereby enabling the matriculation of a diverse class each year.
NSM leadership and the AFRs need to share the belief that it is important to matriculate a
diverse class each year
Table 18
Organization Need and Recommendations – Cultural Model: Shared Belief
Validated Organization Need

Context-Specific Recommendation
Cultural Model:
NSM leadership and the AFRs need to
share the belief that it is important to
matriculate a diverse class each year
1. NSM leadership needs to continue to
communicate candidly with the AFRs and solicit
feedback to specifically ascertain that the AFRs
share the belief that it is important to matriculate a
diverse class each year.

The findings of this study indicate that a shared belief, of the importance of matriculating
a diverse class each year, exists between the New School of Medicine leadership and the
Admissions File Reviewers. Support for this finding is rooted in communication theory. The
commitment of the NSM leadership to clearly and candidly communicate to the AFRs the
importance of matriculating a diverse class each year is a critical success factor to enable the
development of the shared belief of the importance of matriculating a diverse class each year
(Clark & Estes, 2008). The recommendation is to continue candid communication by NSM
leadership to AFRs with solicitation of their feedback to specifically ascertain that the AFRs
share the belief that it is important to matriculate a diverse class each year.
UNDERREPRESENTED MINORITIES IN MEDICINE 99
The New School of Medicine (NSM) needs to provide professional development (PD)
training on the holistic review process for the AFRs, with set-aside time to participate, as
well as the opportunity to rehearse, receive feedback and calibrate.
Table 19
Organization Need and Recommendations -Cultural Setting – Provision of PD Training
Validated Organization Need

Context-Specific Recommendation
Cultural Setting:
To enable effective file review, the New
School of Medicine (NSM) needs to
provide professional development (PD)
training on the holistic review process for
the AFRs, inclusive of set-aside time to
participate, as well as the opportunity to
rehearse, receive feedback and calibrate.
1. NSM leadership needs to provide AFRs with
professional development training on the holistic
file review process as well as consistent
assessment, feedback and monitoring during and
after training by calibrating worked examples with
the rest of the AFR cohort.
• 2. Mandatory pre-training work with accountability
• 3. Longer in-person training: One day training; and
one day practice simulation to go over at least 10
“dummy” applications with time for immediate
feedback and calibration until confident in scoring
and narrative
• 4. Protected time for monthly follow-up meetings –
not always at 7 am when people are “hustling
getting kids ready”; not always at “lunch-time”
which varies for practitioners and is often pre-
committed at local sites
• 5. As reviews progress, provide ongoing and
frequent feedback to AFRs regarding:
• - The quality of each AFR’s reviews
• - Did the Admissions Committee agree with
reviewer’s recommendations?
• - Was an interview granted?
• - How did the student fare?
• - Was there an offer of acceptance?
• - Any and all process issues

The findings of this study indicate that the New School of Medicine provided effective
professional development training to the Admissions File Reviewers cohort on the HRP which
enabled them to fulfill their tasks. There were also monthly follow-up debriefs, but many AFRs
could not attend these nor did they mandate them. In addition, there were articles about holistic
UNDERREPRESENTED MINORITIES IN MEDICINE 100
review and bias to read pre-training and implicit association tests available to take after the
training. Neither the articles nor the IAT work were mandated nor was there verification that
they were done. A recommendation rooted in learning theory has been selected to close this
organizational gap. Many performance benefits have been realized by individuals, teams and
organizations after professional development training along with the opportunity to utilize
guided self-assessment during and after training (Aguinis & Kraiger, 2009; Baker, 2006).
Research by the AAMC, (2013), has shown that the mastery and application of the
holistic review process for medical school admissions produces a far more diverse class than if
such a process is not utilized. To acquire such mastery will require professional development
training for the AFRs by admissions professionals able to provide the factual knowledge about
the HRP and procedurally model how it is done (Pajares, 2006). Schraw and McCrudden
(2006), note that it is important to support the trainee committee members in their quest to
develop mastery of the HRP by helping them (a) acquire component skills, (b) practice
integrating the skills into their work, and (c) understand when to apply their learnings. The
recommendation is that NSM leadership needs to provide AFRs with professional development
training on the holistic admissions file review process with mandated pre-work and ongoing
assessment and monitoring during and after training with practice time and the calibration of
worked examples with the rest of the AFR cohort. This will allow the successful enactment of
the holistic admissions process by the AFRs, facilitating the matriculation of a diverse class each
year.

UNDERREPRESENTED MINORITIES IN MEDICINE 101
Integrated Implementation and Evaluation Plan
Implementation and Evaluation Framework
The model used to design the implementation and evaluation plan for this study is the
New World Kirkpatrick Model (Kirkpatrick & Kirkpatrick, 2016), based on the original
Kirkpatrick Four Level Model of Evaluation (Kirkpatrick & Kirkpatrick, 2006). It is a four-level
model embracing (a) Level 4: Results - the degree to which targeted outcomes occur as a result
of the training, support and accountability demonstrated by leading indicators which are short-
term observations and measurements that suggest that critical behaviors are on track to create a
positive impact on the desired results; (b) Level 3: Behavior - the degree to which participants
apply what they learned during training when they are back on the job with use of processes and
systems that reinforce, encourage and reward performance of critical behaviors on the job
(required drivers); (c) Level 2: Learning - the degree to which participants acquire the intended
knowledge, skills and attitude, as well as confidence and commitment based on their
participation in the training; and (d) Level 1: Reaction - the degree to which participants find the
training favorable, engaging and relevant to their jobs. For maximal impact, consideration will be
given to the five foundational principles upon which the training and evaluation model is based,
namely (a) the end is the beginning - effective training and development begins before the
program even starts; (b) return on expectations is the ultimate indicator of value; (c) business
partnership is necessary to bring about positive return on expectations; (d) value must be created
before it can be demonstrated. By utilizing this model, “leading indicators” that bridge
recommended solutions to the organization’s goals are more easily identified and more closely
aligned with organizational goals. This “reverse order” of the New World Kirkpatrick Model
UNDERREPRESENTED MINORITIES IN MEDICINE 102
generates “buy-in” and creates synergy between the immediate solutions and the larger goals to
ensure success (Kirkpatrick and Kirkpatrick, 2016).
Organizational Purpose, Needs and Expectations
The performance need focus of this study was the design of a holistic review process to
effectively matriculate, retain and graduate underrepresented minority students. This need
aligned with the organization’s mission “… an unwavering commitment to improve the health
and well-being of patients and communities,” and is related to the larger problem of
underrepresented minorities in medicine in the United States which impedes access to care for
underrepresented minority populations and widens the health care disparities gap.
The stakeholder group of focus for this study was the Admissions File Reviewers
(AFRs), whose goal it was to develop an effective holistic review process with all AFRs able to
conduct a holistic review process by July, 2019. The knowledge, motivational and organizational
barriers impeding the achievement of the goal were examined and solutions recommended to
bridge the gaps to achieve an effective holistic review process.
Level 4: Results and Leading Indicators
Table 20 shows the proposed Level 4: Results and Leading Indicators in the form of
outcomes, metrics and methods for both external and internal outcomes for NSM. If the internal
outcomes are met as expected as a result of the training and organizational support for new
Admissions File Reviewers’ performance on the job, then the external outcomes should also be
realized.

UNDERREPRESENTED MINORITIES IN MEDICINE 103
Table 20
Outcomes, Metrics, and Methods for External and Internal Outcomes
Outcome Metric(s) Method(s)
External Outcomes
1.The underrepresented
minorities in medicine gap
is decreasing
Obtain count of URMs in
medicine
Data from AAMC
(Association of American
Medical Colleges)
2. The public is
experiencing improved
health outcomes
Population outcomes data Data from AAMC
3. The health care
disparities gap is lessening
Population outcomes data Data from CDC (Centers for
Disease Control) and IHI
(Institute for Healthcare
Improvement)
4. The pipeline of URMs
into medical school is
increasing as available
underrepresented mentors
increase
Obtain count Undergraduate institution
data and AAMC data
Internal Outcomes
5. Increased knowledge of
AFRs about the holistic
review process
Summary applicant narrative
review
Summary applicant narrative
review
6. Increased AFR
confidence of AFRs to
conduct a holistic review
process
Number of holistic file reviews
conducted by AFRs
Interview
7. Effective execution by
AFRs of a holistic review
process
Production of accurate
narratives for reviewed
applications
Review narratives against
applications
8. Matriculation of a
diverse class to NSM
Statistics from inaugural class
admission
Survey

Level 3: Behavior
Critical behaviors. The stakeholders of focus are the admissions file reviewers
completing training in holistic review process for medical school admissions. The first critical
behavior is that reviewers must understand the layout and content of the application. The second
critical behavior is to recognize gaps in applications. The third critical behavior is to recognize
UNDERREPRESENTED MINORITIES IN MEDICINE 104
“red flags” in applications. The fourth critical behavior is that admissions file reviewers must
holistically review assigned applications with consistency to develop reliability, The fifth critical
behavior is that file reviewers must assess MCAT and GPA together to ascertain academic
readiness. The sixth critical behavior is that AFRs must calibrate with the rest of the cohort to
produce inter-rater reliability. The seventh critical behavior is that AFRs must know how to craft
the final narrative about the applicant. The metrics, methods, and timing for each of these
outcome behaviors appear in Table 21.
Table 21
Critical Behaviors, Metrics, Methods, and Timing for Evaluation
Critical Behavior Metric(s)
Method(s)
Timing
1. Understand the
layout and content of
the medical school
application
Number of thorough
and accurate
completed applicant
narrative summaries
Review completeness and
relevance of narrative
summary of applicant
Weekly: June -
March
2. Recognize gaps in
an application
Number of gaps
documented in
narrative
Review narrative summary
of applicants for documented
gaps
Weekly June -
March
3. Recognize “red
flags” in an
application
Number of red flags
described in narrative
Look for agreed-upon red
flags as files are reviewed

Weekly June -
March
4.. Holistically review
assigned applications
with consistency to
develop reliability
Documentation of
candidate experiences
and attributes in
summary narrative
Utilize the AAMC holistic
admissions checklist to
identify experiences and
attributes
Weekly June -
March
5. Assess MCAT and
GPA together to
ascertain academic
readiness
Summarize findings
in narrative
Use GPAs and validated
MCAT ranges for readiness
Weekly June -
March
6. Calibrate with the
rest of the cohort to
produce inter-rater
reliability
Have each AFR
discuss and rate
shared application
exemplars
Group discussions via List-
Serves shared files
Weekly June -
March
7. Know how to craft
the final narrative
about the applicant
Have each AFR create
a narrative for shared
application exemplars
Holistically review assigned
applications with
consistency to develop
reliability
Weekly June -
March

UNDERREPRESENTED MINORITIES IN MEDICINE 105
Required drivers. The cohort of admissions file reviewers requires the support of their
trainers and the NSM Admissions Department to reinforce what they learned in the training and
to encourage them to apply what they have learned to review medical student applications
consistently, equitably, accurately and timely. Rewards should be established for achievement of
performance goals to incentivize the file reviewers. Table 22 shows the recommended drivers to
support critical behaviors of the admissions file reviewers.
Table 22
Required Drivers to Support Critical Behaviors
Method(s) Timing
Critical Behaviors
Supported
1, 2, 3 Etc.
Reinforcing
Review primer on reading an
MD applicant file holistically
Yearly 1-7
Review checklist for narrative
summary
Yearly 1-7
Private list-serve and team
meetings to engage in
calibration and inter-rater
reliability
Weekly 4, 6
Encouraging
Collaboration during team
meetings
Weekly 1-7
Feedback and coaching from
team lead
Weekly 1-7
Rewarding
Acknowledgement when
performance hits benchmark
Monthly 4, 6
Monitoring
Secondary review of
completed file reviews and
narratives by team lead
Weekly 7

Organizational support. The critical behaviors and required drivers monitored for
performance improvement are premised upon implementation of recommendations at the
organizational level. In this case, for the admissions file reviewers to achieve their goals, NSM
UNDERREPRESENTED MINORITIES IN MEDICINE 106
leadership would need to communicate regularly and candidly with the AFRs, solicit and provide
feedback in an ongoing fashion, and specifically ascertain that the AFRs share the belief that it
is important to matriculate a diverse class each year.
Level 2: Learning
Learning goals. Following completion of the recommended solutions, most notably the
file reviewers’ training course, the stakeholders will be able to:
1. Understand the mission, vision and values of NSM
2. Understand the NSM mission-driven holistic review process
3. Know the value of their work
4. Be aware of their own implicit biases and ways to mitigate them
5. Know how to navigate the medical school admissions Application Management System
6. Understand the layout and content of the applications and recognize any “red flags”
7. Exhibit confidence that they can holistically review applications accurately, timely and
with calibrated reliability
8. Assess MCAT and GPA together to ascertain academic readiness
9. Know how to craft a comprehensive final narrative about each applicant
Program. The learning goals listed in the previous section, will be achieved with an
eight-hour in-person training session that lays out the vision and mission of the medical school
and the responsibilities and value of being a faculty member with the vital role of examining and
rating the medical school applicant files. In addition, there will be ongoing on-line teaching and
discussion groups. The file reviewers will be taught about the holistic review process in the
context of the NSM’s mission and how to reliably review each application in detail with ongoing
inter-rater calibration. There will be specific teaching around ascertaining gaps and “red-flags” in
UNDERREPRESENTED MINORITIES IN MEDICINE 107
an application and secondary review by admissions professionals to enable appropriate coaching
and further education. There will be periodic job aids provided as well as scenarios to discuss as
a group with specific instruction as to understanding and incorporating MCAT and GPA metrics
as a factor in determining applicant readiness and likelihood for success, in the context of the
other experiences and attributes possessed by the applicants. There will be an ongoing available
on-line cohort presence for peer and professional help, encouragement and affirmation, as well
as practice and secondary review by instructors. The demonstrations, practice, and feedback will
also guide time management strategies to model how the review process can be completed in a
timely fashion.
Components of learning. Demonstrating declarative knowledge is often necessary as a
precursor to applying the knowledge to solve problems. Thus, it is important to evaluate
learning for both declarative and procedural knowledge being taught. It is also important that
learners value the training as a prerequisite to using their newly learned knowledge and skills on
the job. However, they must also be confident that they can succeed in applying their knowledge
and skills and be committed to using them on the job. Table 23 lists the evaluation methods and
timing for these components of learning.

UNDERREPRESENTED MINORITIES IN MEDICINE 108
Table 23
Components of Learning for the Program.
Method(s) or Activity(ies) Timing
Declarative Knowledge “I know it.”
Pre-test through multiple-choice and open-ended
questioning
Prior to training
Knowledge checks using open-ended questioning by
instructors
Formative - during lectures and practice
sessions
Knowledge checks through discussions, “pair, think,
share” and other individual/group activities.
Periodically during trainings and email forums.
Procedural Skills “I can do it right now.”
During the asynchronous portions of the course using
scenarios with multiple-choice items.
In the asynchronous portions of the course at the
end of each module/lesson/unit
Demonstration in groups and individually using the
job aids to successfully perform the skills.
During the trainings
Quality of the feedback from peers during group
sharing
During the trainings.
Individual application of the skills with practice
applications.
At the end of the training
Retrospective pre- and post-test assessment survey
asking participants about their level of proficiency
before and after the training. .
At the end of the training.
Attitude “I believe this is worthwhile.”
Instructor’s observation of participants’ statements
and actions demonstrating that they see the benefit of
what they are being asked to do on the job.
During the training and e-mail forums
Discussions of the value of what they are being asked
to do on the job.
During the training
Retrospective pre- and post-test assessment items After the course.

UNDERREPRESENTED MINORITIES IN MEDICINE 109
Table 23, continued
Method(s) or Activity(ies) Timing
Confidence “I think I can do it on the job.”
Survey items using Likert-scaled items Following each module/lesson/unit in the
asynchronous portions of the course.
Discussions following practice and feedback. During the training.
Retrospective pre- and post-test assessment After the training.
Commitment “I will do it on the job.”
Discussions following practice and feedback. During the training.
Create an individual action plan. During the training and after review of some
applications.
Retrospective pre- and post-test assessment item. After the training.

Level 1: Reaction
Table 24 lists the methods to be used to determine how the AFRs react to the learning event(s).
Table 24
Components to Measure Reactions to the Training.
Method(s) or Tool(s) Timing
Engagement
Pre-work completed Prior to training
Participation and questions During synchronous and asynchronous portions
of the training.
Completion of online
modules/lessons/units
Prior and during training and by email
discussions
Observation During the training
Course evaluation Two weeks after the training

UNDERREPRESENTED MINORITIES IN MEDICINE 110
Table 24, continued
Method(s) or Tool(s) Timing
Relevance
Brief pulse-check with participants via
survey (online) and discussion (ongoing)
After every module/lesson/unit and the training

Course evaluation Two weeks after the training
Customer Satisfaction
Brief pulse-check with participants via
survey (online) and discussion (ongoing)
After every module/lesson/unit and the training

Course evaluation Two weeks after the training

Evaluation Tools
Immediately following the program implementation. For Level 1 evaluation, during
the in-person training workshop, the instructor will conduct periodic pulse-checks by asking the
AFRs about the relevance and discussing application of the content to their file review process.
Level 2 evaluation will include formative exercises in the form of periodic questions to assess
understanding, practice file reviews, and a summative quiz at the end. (See Appendix D).
Delayed for a period after the program implementation. Approximately six and 12
weeks after training and implementation of the file review process, leadership will administer a
survey containing scaled and open-ended using the Blended Evaluation approach. The survey
will measure the trainees satisfaction and relevance of the training (Level 1), confidence and
value of applying their training (Level 2), application of the training to the file review process,
and the support from instructors and leadership they are receiving (Level3), and the extent to
which their performance of the file review process is yielding positive results (See Appendix E).

UNDERREPRESENTED MINORITIES IN MEDICINE 111
Data Analysis and Reporting
For the components comprising each of the four Kirkpatrick levels, the following three
key data analysis questions will be asked: i) were expectations met? ii) if not, why not? and iii) if
so, why? For Level 1 (Reaction) and Level 2 (Learning), the trainers will conduct analysis
formatively as they teach and determine the range of acceptable expectations based on their
experience. Key performance indicators will be evaluated against the benchmark of how often
they should be performed. Expectations for Level 4 program results (outcomes) will be evaluated
based on attaining defined leading outcomes. For any Level component that does not meet
expectations, the root cause will need to be identified and corrected before targeted results are
jeopardized. Appendix F shows a dashboard of each of the key performance indicators to be
measured, the target, actual results and the rating.
Summary
The New World Kirkpatrick Model (2016) was used to plan, implement and evaluate the
recommendations for the New School of Medicine, to optimize the stakeholders (admissions file
reviewers) goal to effectively conduct a holistic file review process by July, 2019. Achievement
of the stakeholder goal was in furtherance of the organizational goal to matriculate, retain and
graduate classes of medical students comprised of a higher than the usual 15% representation of
underrepresented minority (URM) students, to increase the percentage of URMs in the U.S.
healthcare workforce, improve access to care and ultimately, narrow the healthcare disparities
gap.
Utilizing the Kirkpatrick and Kirkpatrick (2016) four-level framework allowed for a
systematic approach to drive results and achieve outcomes, as illustrated by leading indicators
UNDERREPRESENTED MINORITIES IN MEDICINE 112
which showed that behaviors critical to achieving the goal were being performed. Continual
monitoring of the learning and transfer of skills during the integrated implementation and
evaluation processes created value and a high return on expectations, as expertise in the holistic
file review process was developed, evaluated and reinforced over time, enabling the
matriculation of diverse classes to the New School of Medicine.
Future Study
Areas for future study include:
1. A longitudinal evaluation of the effectiveness of the holistic review process in attaining
the organizational goal of the NSM.
2. Further definition and refinement of the attributes most important to the mission of NSM
for AFRs to consider.
3. Whether there is successful mitigation of biases held by AFRs
4. An assessment of the longitudinal impact of each class regarding chosen specialties,
geographic and socioeconomic areas of practice.
5. Impact to the number of underrepresented minorities in medicine, including academic
medicine
6. Impact to the health care disparities gap
Conclusion
The index organization in this study was the New School of Medicine (NSM), which
welcomed its’ inaugural class in July of 2020. At NSM, implementing an effective holistic
review process was critical to enable the matriculation of diverse classes to the school.
Admission File Reviewers (AFRs) were selected as the primary stakeholder as they are
responsible for the student application review. The stakeholder goal was for the AFRs to
UNDERREPRESENTED MINORITIES IN MEDICINE 113
have an effective holistic admissions process (HAP) in place by July of 2019, to drive the
attainment of a diverse class of matriculants. This stakeholder goal was in furtherance of the
organizational goal to embed diversity, equity, and inclusion within the medical curriculum
and the entire school, with a commitment to improving the well-being of patients and their
communities. Research has shown that the mastery and application of the holistic review
process for medical school admission produces a far more diverse class than if such a process
is not utilized (AAMC, 2013; Conrad, S.S., Addams, A. N., & Young, G. H. 2016). It is thus
imperative that the AFRs be effectively trained in the holistic review the process to facilitate
the achievement of their goal to matriculate diversity in the classes in proportion to the
general population.
Facilitating closure of this gap was the purpose of this study and aligned with the
mission to improve the health and well-being of all patients and communities, with the
ultimate prize of equitable access to care and the elimination of health care disparities. This
study focused on the knowledge, motivation, and organization related influences that are
pertinent to the achievement of the stakeholder goal. AFRs need knowledge and motivation
to master the holistic review process along with support and trust of the NSM to enact it. The
dynamic, symbiotic interplay between the validated knowledge and motivation influences
soaked in the sea of organizational culture and context, to achieve mastery of holistic review
process by the AFRs was illustrated, all in furtherance of the organizational goal to
matriculate a diverse class each year. The findings of this study will enable the development
of a more refined holistic review process over time and mitigate the gap of underrepresented
minorities in medicine, with the ultimate prize of a reduction in the health care disparities
gap.

UNDERREPRESENTED MINORITIES IN MEDICINE 114
REFERENCES
Achenjang, J. N., & Elam, C.L. (2016). Recruitment of Underrepresented Minorities in Medical
School Through a Student-led Initiative. Journal of the National Medical Association,
106, 147-151.
Agocs, C. (1997). Institutionalized resistance to organizational change: Denial, inaction, and
repression. Journal of Business Ethics, 16, 917-931.
Aguinis, H., Kraiger, K. (2009). Benefits of training and development for individuals, teams,
organizations and society. Annual Review of Psychology, 60: 451-74.
Agency for Healthcare Research and Quality. (2004). National Healthcare Disparities Report:
2004. Washington, DC: US Department of Health and Human Services.
Alper, S., Tjosvold, D., & Law, K. (2000). Conflict management, efficacy, and performance in
organizational teams. Personnel Psychology, 53(3), 625-642.
Artinian, N, T. (2017). Holistic admissions in the health professions: Strategies for leaders. Coll
Univ, 92(2), 65-68.
Association of American Medical Colleges. (2004). Underrepresented in medicine definition.
Retrieved from https://www.aamc.org >initiatives >urm
Association of American Medical Colleges. (2012). Diversity in medical education: Facts &
figures 2012. Number (%) of U.S. medical school graduates by race and ethnicity, 1971-
2010. Table 8, 72.
Association of American Medical Colleges. (2012). Diversity in medical education: Facts &
figures 2012. Percentage and number of U.S. medical school faculty by race and
ethnicity, 1971-2010. Figure 18, 35.
UNDERREPRESENTED MINORITIES IN MEDICINE 115
Association of American Medical Colleges. (2012). Data book. Data and analysis. Retrieved
from https:// www.aamc.org > data > databook
Association of American Colleges. (2013). Roadmap to excellence: Key concepts for evaluating
the impact of medical school holistic admissions. Retrieved from https://store.aamc.org/
downloadable/download/sample/sample_id/198/
Baker, L. (2006). Metacognition. Retrieved from http://www.education.com/reference/article/
metacognition/
Bandura, A. (1977). Self-efficacy. Toward a unifying theory of behavioral change. Psychological
Review, 84, 191-215.
Bandura, A. (2000). Exercise of human agency through collective efficacy. Current Directions in
Psychological Science, 9(3), 75-78.
Beltran, R.A. (2003). Affirmative action in medical school admissions. JAMA, 289(23), 3084
Berger, B. (2014). Read my lips: Leaders, supervisors, and culture are the foundations of
strategic employee communications. Research Journal of the Institute for Public
Relations, 1(1).
Bogdan, R. C., & Biklen, S. K. (2007). Qualitative research for education: An introduction to
theories and methods (5th ed.) Boston, MA: Allyn and Bacon.
Bowen, G.A. (2009). Document analysis as a research method. Qualitative Research Journal,
9(2)
Bright C. M., Duefield C. A., & Stone, V. E. (1998). Perceived barriers and biases in the medical
education experience by gender and race. Journal of the National Medical Association,
90, 681-688.
UNDERREPRESENTED MINORITIES IN MEDICINE 116
Bureau of Health Professions. (2006). The rationale for diversity in the health professions: A
review of the evidence. Rockville, MD; US Department of Health & Human Services.
Burke, W. (2002). Organization change: Theory and practice. Thousand Oaks, CA: SAGE.
Burke, W. (2018). Organization change: Theory and practice. Thousand Oaks, CA: SAGE.
Cantor, J.C., Bergeisen, L., Baker, L.C. (1998). Effect of an intensive educational program for
minority college students and recent graduates on the probability of acceptance to
medical school. JAMA, 280(9), 772-6
Capers, Q., IV, Clinchot, D., McDougle, L., Greenward, A.G.(2017). Implicit racial bias in
medical school admissions. Academic Medicine, 92(5), 571-575.
Clark, R. E. & Estes, F. (2008). Turning research into results: A guide to selecting the right
performance solutions. Charlotte, NC: Information Age.
Clark, S. M., Corcoran, M. (1986). Perspective on professional socialization of women faculty.
Journal of Higher Education, 57, 20-43.
Cohen, J., Gabriel, B., & Terrell, C. (2002). The case for diversity in the health care workforce.
Health Affairs, 21, 90-102.
Cohen, J. J. (2003) The consequences of premature abandonment of affirmative action in
medical school admissions. Journal of the American Medical Association, 289, 1143-49.
Cohen, J. J., & Steinecke, A. (2006). Building a diverse physician workforce. Journal of the
American Medical Association, 296, 1135 -1137.
Conrad, S. S., Addams, A. N., & Young, G. H. (2016). Holistic review in medical school
admissions and selection: A strategic, mission-driven response to shifting societal needs.
Academic Medicine, 91(11), 1472-1474.
UNDERREPRESENTED MINORITIES IN MEDICINE 117
Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods
approaches. Thousand Oaks, CA: SAGE.
Daly, E. (2006) Behaviorism. Retrieved from http://www.education.com/reference/article/
behaviorism/
Deas, D., Pisano, E. D., Mainous, A.G., III, Johnson, N. G., Singleton, M. H., Gordon, L.,
Taylor, W.… & Reves, J. G. (2012). Improving diversity through strategic planning: A
10-year (2002-2012) experience at the Medical University of South Carolina. Academic
Medicine, 87, 1548-1555.
Dembo, M., & Eaton, M. J. (2000). Self-regulation of academic learning in middle-level
schools. The Elementary School Journal, 100, 473–490.
Derck, J., Zahn, K., Finks, J., Mand, S., & Sandju, G. (2016). Doctors of tomorrow: An
innovative curriculum connecting underrepresented minority high school students to
medical school. Education for Health, 29, 259-265.
Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41(10),
1040.
Dyrbye, L. N., Thomas, M. R., Huschka, M. M., & Lawson, K. L. (2006). A multicenter study of
burnout, depression, and quality of life in minority and nonminority US medical students.
Mayo Clinic Proceedings, 81, 1435-42.
Eccles, J. (2006). Expectancy value motivational theory. Retrieved from
http://www.education.com/reference/article/expectancy-value-motivational-theory/
Figueroa, O. (2014). The significance of recruiting underrepresented minorities in medicine: An
examination of the need for effective approaches used in admissions by higher education
institutions. Medical Education Online, 19, 24891.
UNDERREPRESENTED MINORITIES IN MEDICINE 118
Fine, E., Handelsman, J. (2010). Benefits and challenges of diversity in academic settings.
Madison, WI: Women in Science and Engineering Leadership Institute.
Flexner, A. (1910). Medical education in the United States and Canada. Bulletin Number Four.
Retrieved from archive.carnegiefoundation.org.elibrary
Freeman, B. K., Landry, A., Trevino, R., Grande, D., & Shea, J.A. (2016). Understanding the
leaky pipeline: Perceived barriers to pursuing a career in medicine or dentistry among
underrepresented-in-medicine undergraduate students. Academic Medicine, 91, 987-993.
Gallimore, R., Goldenberg, C. (2001). Analyzing cultural models and settings to connect
minority achievement and school improvement research. Educational Psychologist, 36
(1), 45-56
Glesne, C. (2011). Becoming qualitative researchers: An introduction (4th ed.). Boston, MA:
Pearson.
Grumbach, K., Chen, E. (2006), Effectiveness of University of California postbaccalaureate
premedical programs in increasing medical school matriculation for minority and
disadvantaged students. JAMA, 296 (9),1079-1085.
Grutter v. Bollinger, 539 U.S. 306, 2003
Gurin, P., Dey, E. L., Hurtado, S., & Gurin, G. (2002). Diversity and higher education: Theory
and impact on educational outcomes. Harvard Educational Review, 72, 330-366.
Hopwood v. Texas, 78 F.3d 932 (5th Cir. 1996)
Huff, K. L., & Fang, D. (1999). When are students most at risk of encountering academic
difficulty? A study of the 1992 matriculants to U.S. Medical Schools. Academic
Medicine, 74, 454-460.
UNDERREPRESENTED MINORITIES IN MEDICINE 119
Institute of Medicine. (2003). Unequal treatment: Confronting racial and ethnic disparities in
health care. Washington, DC: The National Academies Press.
https://doi.org/10.17226/10260.
Johnson, R. B., & Christensen, L. B. (2015). Educational research: Quantitative, qualitative,
and mixed approaches. (5th ed.). Thousand Oaks: SAGE.
Kelly, R. L. (2015). Health disparities in America. Washington, DC: Office of Congresswoman
Robin Kelly. Retrieved from House.gov>robinkelly>media-center
Kirkpatrick, J.D., & Kirkpatrick, W. K. (2016). Kirkpatrick’s four levels of training evaluation.
Alexandria, VA: ATD Press.
Kondo, D. G., & Judd, V. E. (2000). Demographic characteristics of US medical school
admission committees. JAMA, 284(9): 1111-13).
Harris, F., & Bensimon, E. (2007). The equity scorecard: A collaborative approach to assess and
respond to racial/ethnic disparities in student outcomes. New Directions for Student
Services, 120, 77-84.
Kezar, A. (2001a). Theories and models of organizational change. Understanding and facilitating
organizational change in the 21st century: Recent research and conceptualizations.
ASHE-ERIC Higher Education Report, 28(4), 25-58.
Kezar, A. (2001b). Research-based principles of change. Understanding and facilitating
organizational change in the 21st century: Recent research and conceptualizations.
ASHE-ERIC Higher Education Report, 28(4), 113-123.
Korsgaard, M., Brodt, S., & Whitener, E. (2002). Trust in the face of conflict: The role of
managerial trustworthy behavior and organizational context. Journal of Applied
Psychology, 87(2), 312-319.
UNDERREPRESENTED MINORITIES IN MEDICINE 120
Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory Into Practice,
41(4), 212–218
Kreiter, C. D. (2013). Research methodology: A proposal for evaluating the validity of holistic-
based admission process. Teaching and Learning in Medicine, 25(1), 103-107.
Krueger, R. A., & Casey, M. A. (2009). Focus groups: A practical guide for applied research
(4th ed.). Thousand Oaks, CA: SAGE.
Laveist, T.A., & Nuru-Jeter, A. (2002). Is doctor-patient race concordance associated with
greater satisfaction with care? Journal of Health and Social Behavior, 43, 296-306.
Lett, L.A., Murdock, H.M., Orji, W. U., Aysola, J., Sebro, R. (2019). Trends in racial/ethnic
representation among US medical students. JAMA Network Open, 2(9)
Mader, E. M., Rodriguez, J. E., Campbell, K. M., Smilnak, T., Bazemore, A.W., Petterson, S.,
Morley, C. P. (2016). Status of underrepresented minority and female faculty at medical
schools located within historically black colleges and in Puerto Rico. Medical Education
Online, 21, 29535. http://dx.doi.org/ 10.3402/meo.v21.29535
Mangus, R. S., Hawkins, C. E., & Miller, M. J. (1998). Prevalence of harassment and
discrimination among 1996 medical school graduates: A survey of eight US schools.
JAMA: The Journal of the American Medical Association, 280, 851-853.
Maxwell, J. A. (2013). Qualitative research design: An interactive approach (3rd ed.). Thousand
Oaks, CA: SAGE.
Mayer, R. E. (2011). Applying the science of learning. Boston, MA: Pearson Education.
McGrath, B., & McQuail, D. (2004). Decelerated medical education. Medical Teacher, 26, 510-
513.
UNDERREPRESENTED MINORITIES IN MEDICINE 121
Merriam, S. B., & Tisdell, E. J. (2016). Qualitative research: A guide to design and
implementation (4th ed.). San Francisco, CA: Jossey-Bass.
Milem, J. F. (2003). The educational benefits of diversity: Evidence from multiple sectors.
Retrieved from Researchgate.net.
Monroe, A., Quinn, E., Samuelson, W., Dunleavy, D.M., & Dowd, K. W. (2013). An overview
of the medical school admission process and use of applicant data in decision making:
What has changed since the 1980s? Academic Medicine.
Nivet, M. A. (2009). Striving toward excellence: Faculty diversity in medical education.
Washington, DC: Association of American Medical Colleges.
Nivet, M. A. (2010). Minorities in academic medicine: Review of the literature. Journal of
Vascular Surgery, 51(98), 53S-58S.
Nivet, M. A. (2015). A diversity 3.0 update: are we moving the needle enough? Academic
Medicine, 90(12), 1591-1593.
Odom, K. L., Roberts, L. M., Johnson, R. L., & Cooper, L. A. (2007). Exploring obstacles to and
opportunities for professional success among ethnic minority medical students. Academic
Medicine, 82, 146-153.
O’Neill, L., Vonsild, M. C., Wallstedt, B., Dornan T. (2013). Admission criteria and diversity in
medical school. Medical Education, 47: 557-61.
Ono, S. J. (2016). Holistic admissions: What you need to know. Trusteeship. Retrieved from
https://www.agb.org/trusteeship/2016/marchapril/holistic-admissions-what-you-need-to-
know
UNDERREPRESENTED MINORITIES IN MEDICINE 122
Orom, H., Semalulu, T., & Underwood, W., III. (2013). The social and learning environments
experienced by underrepresented medical students: A narrative review. Academic
Medicine, 88, 1765-1777.
Page, K. R., Castillo-Page, L., Wright, S. M. (2011). Faculty diversity programs in U.S. medical
schools and characteristics associated with higher faculty diversity. Academic Medicine,
286, 1221-1228.
Pajares, F. (2006). Self-efficacy theory. Retrieved from http://www.education.com/reference/
article/self-efficacy-theory/.
Patton, M. Q. (1987). Qualitative research and evaluation methods (4th ed.). Thousand Oaks,
CA: SAGE.
Patton, M. Q. (2002). Qualitative research & evaluation methods (3rd ed.) Thousand Oaks, CA:
SAGE.
Patton, L.D. (2009). My sister’s keeper: A qualitative examination of mentoring experiences
among African American women in graduate and professional schools. The Journal of
Higher Education, 80(5), 510-537.
Piercy, F., Giddings, V., Allen, K., Dixon, B., Meszaros, P., Joest, K. (2005). Improving campus
climate to support faculty diversity and retention: A pilot program for new faculty.
Innovative Higher Education, 30(1):53-66.
Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in
learning and teaching contexts. Journal of Educational Psychology, 95(4), 667–686.
Pololi, L., Cooper, L. A., & Carr, P. (2010). Race, disadvantage and faculty experiences in
academic medicine. Journal of General Internal Medicine, 25(12), 1363–1369.
http://doi.org/10.1007/s11606-010-1478-7
UNDERREPRESENTED MINORITIES IN MEDICINE 123
Regents of the University of California v. Bakke, 438 U.S. 265 (1978)
Robins, L. S., Gruppen, L. D., Alexander, G.L., Fantone, J.C., & Davis, W. K. (1997). A
Predictive model of student satisfaction with the medical school learning environment.
Academic Medicine, 72, 134-139.
Rocco, T. S., & Plakhotnik, M. S. (2009). Literature reviews, conceptual frameworks, and
theoretical frameworks: Terms, functions, and distinctions. Human Resource
Development Review, 8(1), 120-130
Rubin, H.J. & Rubin, I. S. (2012). Qualitative interviewing: The art of hearing data (3rd ed.).
Thousand Oaks, CA: SAGE.
Rumala, B., & Cason, P.R. (2007). Recruitment of underrepresented minority students to
medical school: Minority medical student organization, an untapped resource. Journal of
the National Medical Association, 99, 1000-1009.
Rueda, R. (2011). The 3 dimensions of improving student performance. New York, NY:
Teachers College Press.
Saha, S. (2014). Taking diversity seriously: The merits of increasing minority representation in
medicine. JAMA Internal Medicine, 174, 291-292.
Schein, E. H. (2017). Organizational culture and leadership (5th ed.) San Francisco, CA:
Jossey- Bass.
Schneider, B., Brief, A., & Guzzo, R. (1996). Creating a culture and climate for sustainable
organizational change. Organizational Dynamics, 24(4), 7-19.
Schraw, G., McCrudden, M. (2006). Information processing theory. Retrieved from http://
www.education.com/reference/article/information-processing-theory/
UNDERREPRESENTED MINORITIES IN MEDICINE 124
Shea, S, Fullilove, M.T. (1985). Entry of black and other minority students into U.S. medical
schools: historical perspective and recent trends. New England Journal of Medicine, 212,
933-40.
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78, 153–
189.
Stegers-Jager, K. M., Cohen-Schotanus, J., Themmen, A. P. (2012). Motivation, learning
strategies, participation and medical school performance. Medical Education, 46(7), 678-
688.
Strauss, A., Schatzman, L., Bucher, R., & Sabshin, M. (1981). Psychiatric ideologies and
institutions (2nd ed.). New York, NY: Wiley.
Sullivan, L. W., & Mittman, I. S. (2010). The state of diversity in the health professions a
century after Flexner. Academic Medicine, 85, 246-253
Sundt, M. (n.d.). Understanding organizational culture. Retrieved from
https://2sc.rossieronline.usc.edu/mod/page/view.php?id=138924. Retrieved May 20,
2018.
Terrell, C. (2006). The health professions partnership initiative and working toward diversity in
the health care workforce. Academic Medicine, 81(6), 52–54.
Thompson, W.A., Ferry, P.G., King, J. E., Martinez-Wedig, C. & Michael, L. H. (2003).
Increasing access to medical education for students from medically underserved
communities: one program’s success. Academic Medicine, 78, 454–459.
Tienda, M. (2001). College admission policies and the educational pipeline: Implications for
medical and health professions. In B. D. Smedley (Ed.), The right thing to do; The smart
UNDERREPRESENTED MINORITIES IN MEDICINE 125
thing to do: Enhancing diversity in the health professions. Washington, DC: National
Academy Press.
Tiwana, R. (n.d.). The role of the interactive conceptual framework. Retrieved from
https://2sc.rossieronline.usc.edu/mod/page/view.php?id=138395. Retrieved July 14,
2018.
Tuckman, B. (2006). Operant conditioning. Retrieved from http://www.education.com/
reference/article/operant-conditioning/
Urban Universities for HEALTH. (2014). Holistic admissions in the health professions: Findings
from a National Survey. Washington, DC: Author.
Valentine, P., Wynn, J., & McLean, D. (2016). Improving diversity in the health professions.
North Carolina Medical Journal, 77, 137-140.
Vollman, A. (2015). Health professions schools: Bridging the gap for underrepresented
minorities. Insights into Diversity. Retrieved from http://www.insightsintodiversity.com/
wp-content/media/issues/aprilmay2015.pdf.
Whitla, D.K., Orfield, G., Silen, W., Teperow, C., Howard, C., & Reede, J. (2003). Educational
benefits of diversity in medical school: A survey of students. Academic Medicine, 78,
460-466.
Xierali, I. M., Fair, M.A., Nivet, M.A. (2016). Faculty diversity in U.S. medical schools:
Progress and gaps coexist. Association of American Medical Colleges Analysis in Brief,
16, 6, 1-3. Retrieved from: https://www.aamc.org>download>data
Yough, M., & Anderman, E. (2006). Goal orientation theory. Retrieved from
http://www.education.com/reference/article/goal-orientation-theory/
UNDERREPRESENTED MINORITIES IN MEDICINE 126
Zhang, K., Xierali, I. M., Castillo-Page, L., Nivet, M., Conrad, S.S. (2015). Students’ top factors
in selecting medical schools. Academic Medicine, 90(5), 693.

UNDERREPRESENTED MINORITIES IN MEDICINE 127
APPENDIX A
Letter to Stakeholder Group From the Associate Dean of Admissions Introducing the Researcher

Dear Admissions File Reviewers,
I would like to introduce you to Dr. Linda Tolbert, a physician with the “Medical Group” for the
past 26 years, who is currently pursuing a Doctorate through the Rossier School of Education at
the University of Southern California. The nature of her dissertation is to understand the
knowledge and motivation influences at play for physicians in the role of an Admissions File
Reviewer for the “New School of Medicine” (NSM), and the interplay of any presumed NSM
cultural and contextual organizational needs and barriers.

The questions guiding the research are:

1. What are the knowledge, motivation and organizational needs required for the NSM
Admissions File Reviewers to develop and effectively enact a holistic admissions process to
matriculate a diverse class each year?
2. What is the interaction between the Admissions File Reviewers knowledge and motivation and
the NSM organizational culture and context.

Recommended solutions to the knowledge, motivational and organizational needs will be
developed.

Dr. Tolbert’s role is purely as an investigator and she will gather and make meaning of the
information imparted to her through interviews, without injecting her opinions. She anticipates
an hour for each interview but asks that you allow at least 90 minutes to conduct it from start to
finish. Most interviews will be conducted and recorded by telephone as you are located across
the country. Of note, although the File Reviewer cohort is known, individual participant
responses will not be
traceable. Dr. Tolbert will assign pseudonyms to the interview transcripts which will be kept
confidential and in a secure environment so that your anonymity is preserved.

The File Reviewer group of the Admissions Faculty was chosen because of the recognition of the
great impact of the work you are doing. We believe that this study will provide very useful
information which, when aggregated into themes, will help in the ongoing evaluation and
improvement of our program. As such, we encourage you to wholeheartedly participate.

Many thanks,

Associate Dean for Admissions

UNDERREPRESENTED MINORITIES IN MEDICINE 128
APPENDIX B
Interview Protocol
The purpose of this study, in which you have graciously agreed to participate, is to
develop and effectively enact an effective holistic review process (HRP), to matriculate a
diverse medical school class each year for the New School of Medicine (NSM), which is
slated to open in the Fall of 2020. My name is Dr. Linda Tolbert, and I am a physician with
the “Medical Group” for the past 26 years, currently pursuing my Doctoral degree in
Organizational Change and Leadership from the Rossier School of Education at the University
of Southern California. I am very excited that we will be working together in this endeavor,
and my role here is purely that of an investigator. I will gather and make meaning of the
information imparted to me through the interviews, without injecting my opinions. The nature
of the research is to understand the knowledge and motivation influences at play for
physicians in the role of Admissions File Reviewers, as well as to ascertain the organizational
needs and barriers of the NSM and their interplay with the admissions file reviewers’
knowledge and motivation influences. We will meet again for an interview after your
committee has worked together and has undergone professional training in the holistic
admission process.
As you know from our informed consent previously obtained, your participation is
completely voluntary. I will use a pseudonym for you in both the notes and publication, and all
data obtained will be confidential and securely kept in a safe environment with locked or
encrypted access. You can refuse to answer any question if you so choose and you may also
withdraw from the study at any time. I will periodically remind you of these stipulations during
the process. Please know that in this forum, there is no hierarchy nor judgement and, because
UNDERREPRESENTED MINORITIES IN MEDICINE 129
information obtained is confidential and your anonymity preserved, there is not the possibility of
any repercussions to you for your given answers. Very specifically, future performance
evaluations or job advancement will not be influenced in any way by your involvement.
I appreciate that you have granted me permission to audio record the interviews and I will
provide you with the opportunity for you to review the transcripts and verify accuracy. In
addition, because you own the work-product of the interview, any requests for retractions from
collected data will be granted. Please know that although the Admissions File Reviewers are
known to the school and other interested stakeholders, specific AFRs interviewed for this study
will not be publicly known, nor will their work-product.
Do you have any questions before we begin the interview? None? Okay, then let’s begin!

Transitional language: So, just a couple of background questions to get us started…

1. Background/demographic question (Patton)
“How long have you been a physician?”

2. Background/demographic question (Patton)
“What was your admission process to medical school like?”

Transitional language: Back to the present…

3. Motivation – Task value
“Could you discuss some of your reasons for being a part of the NSM Admissions Committee?”

Transitional language: Knowing what you know now, having been through medical school and
the practice of medicine…

4. Knowledge - conceptual: Ideal position question - (Strauss, Schatzman, Bucher & Sabshin)
“Describe for me what you think the ideal medical school class would look like?”

Transitional language: continuing in that vein…

5. Opinion and values question (Patton)
“What is your opinion as to whether having underrepresented minorities is important in the
context of a medical school class?”
Probe follow up if needed: Opinion and values question (Patton)
UNDERREPRESENTED MINORITIES IN MEDICINE 130
“What is the value of diversity to you?”

Transitional language: shifting to a different, but related topic…

6. Opinion and values question (Patton)
“What does equity in the context of admissions mean to you?”
Probe follow up: Knowledge: Devil’s advocate question (Strauss, Schatzman, Bucher &
Sabshin)
“Some people would say that admitting students from underrepresented minority groups who
historically, often have lower GPAs than their mainstream counterparts, will lessen the status of
the school. What would you tell them?”

Transitional language: expanding further upon the concepts of diversity and equity…

7. Opinion and values question (Patton) “What does inclusion mean to you?”
Probe follow up if needed: Knowledge - conceptual
“If you had to explain inclusion to someone, what would you say?”

Transitional language: Now, regarding admissions processes…
8. Knowledge: factual (Patton)
“Tell me what you know about the holistic admission process.”

Transitional language: The next couple of questions relate to your interest in the holistic
admissions process…

9. Motivation – task value
“What do you see as the value of performing a holistic admission process, if any?

10. Motivation - self-efficacy: Feeling question: (Patton)
“How do you feel about your ability to learn how to perform a holistic admissions process?”

Transitional language: Now…

11. Knowledge - metacognitive
“Tell me about the last time you reflected on what you know about the long-term outcomes of
holistic admissions.”

Transitional language: And finally, shifting to an organizational perspective…

12. Organization – Cultural model
“Describe your thoughts about whether there is a shared belief held by NSM leaders and
Admissions Committee members, that it is important to matriculate a diverse medical school
class each year.”

Thank you so much!

UNDERREPRESENTED MINORITIES IN MEDICINE 131
Feel free to reach out to be if you have anything further and I look forward to sharing
the transcript with you and being able to seek further clarification.

My contact information is:

Email: name@.org

Cell: xxx-xxx-xxxx

UNDERREPRESENTED MINORITIES IN MEDICINE 132
APPENDIX C
Observation Checklist
KNOWLEDGE components:
Evidence of factual & conceptual knowledge of the elements of holistic review:
1) Interest and engagement during training–questions, answers
2) Evidence that pre-work was done

Evidence of procedural knowledge of how to perform holistic review:
(i) Practice
(ii) Feedback

Evidence of reflection on the holistic review process.
(i) Tweaking of the process
(ii) Calibration

MOTIVATION components:
1) evidence of perceived task value:
(i) Application to their professional work
(ii) Development of new skill-set

2) self-efficacy at an individual and collective AFR group level:
a. AFRs believe they can learn the HR process
b. Collaboration amongst the cohort

ORGANIZATIONAL CULTURE: models
(i) demonstrated evidence of shared belief in diversity, equity and inclusion,
(ii) willingness and trust in leadership to adopt a mission-aligned HRP: Trust –NSM
trusts AFRs; AFRs trust NSM

ORGANIZATIONAL CULTURE: settings
(i) provision of required resources by NSM
(ii) commitment by NSM to invest in the successful enactment of the holistic review
process:

UNDERREPRESENTED MINORITIES IN MEDICINE 133
APPENDIX D
Immediate Evaluation Instrument

LEVEL 1: REACTION

Engagement

Your participation as a trainee was encouraged by the trainer

Strongly Agree—Agree—Disagree—Strongly Disagree

The program held your interest

Strongly Agree—Agree—Disagree—Strongly Disagree

Was there anything about your experience that interfered with your learning? If so, what?

Relevance

What you learned from the course will help you to holistically review the Admissions files

Strongly Agree—Agree—Disagree—Strongly Disagree

During the training, you discussed how to apply what you learned

Strongly Agree—Agree—Disagree—Strongly Disagree

What material did you find most relevant to the holistic review of admissions files?
What material was a waste of time?

Customer Satisfaction

The pre-work before your training was valuable

Strongly Agree—Agree—Disagree—Strongly Disagree

I will recommend this program to future Admissions File Reviewers

How could the training be improved?
UNDERREPRESENTED MINORITIES IN MEDICINE 134
LEVEL 2: LEARNING

Knowledge

Measure by:
1. formative exercises during the session and
2. A summative quiz at the end

Ask: What are the major concepts that you learned during the training?

Skills

Measure by practice file review activities during the session

Attitude

You believe that it will be worthwhile for you to apply what you have learned

Strongly Agree—Agree—Disagree—Strongly Disagree

What is the importance of applying what you learned during training to file review?

Confidence

You feel confident applying what you have learned to the file review process

Strongly Agree—Agree—Disagree—Strongly Disagree

You anticipate that you will receive the necessary support to successfully apply what you have
learned

Strongly Agree—Agree—Disagree—Strongly Disagree

What barriers do you anticipate that could limit your success applying what you have learned?

Commitment

You are committed to applying what you have learned to the file review process

Strongly Agree—Agree—Disagree—Strongly Disagree

UNDERREPRESENTED MINORITIES IN MEDICINE 135
APPENDIX E
Blended Instrument

DELAYED LEVEL 1: REACTION

Relevance

I have used what I learned in training when reviewing admissions files

Strongly Agree—Agree—Disagree—Strongly Disagree

The information presented during training continues to be applicable to file review

Strongly Agree—Agree—Disagree—Strongly Disagree

Customer Satisfaction

I continue to espouse the benefits of my training to my colleagues.

Strongly Agree—Agree—Disagree—Strongly Disagree

How could the training have been improved for you?

DELAYED LEVEL 2: LEARNING

I continue to utilize the job-aids provided in training

Strongly Agree—Agree—Disagree—Strongly Disagree

LEVEL 3: BEHAVIOR

I have successfully applied what I learned during training to the file review process

Strongly Agree—Agree—Disagree—Strongly Disagree

I have received support in order to apply what I have learned successfully

Strongly Agree—Agree—Disagree—Strongly Disagree

Describe any challenges you are experiencing in applying what you learned to the file review
process, and possible solutions to overcome them.
UNDERREPRESENTED MINORITIES IN MEDICINE 136
LEVEL 4: RESULTS

Your efforts have contributed to the fulfillment of the mission of the New School of Medicine
Strongly Agree—Agree—Disagree—Strongly Disagree

Because of the holistic file review process training that you received as an Admissions File
Reviewer, you enable the matriculation of a diverse class to the New School of Medicine?

Strongly Agree—Agree—Disagree—Strongly Disagree

UNDERREPRESENTED MINORITIES IN MEDICINE 137
APPENDIX F
Key Performance Indicators
Key Performance indicator Target
LEVEL 1: REACTION
1. Participants engage during the training program 100%
2. Participants recognize the relevance of the training for their role 100%
3. Participants are satisfied with the training 95%
LEVEL 2: LEARNING
1. Participants achieve a post-training test score of at least 90% 100%
2. Participants demonstrate the ability to conduct a file review 100%
3. Participants enjoy applying the new skills learned 95%
4. Participants self-confidence levels regarding file review is
satisfactory
100%
5. Each participant able to regularly conduct their share of file reviews 100%
LEVEL 3: BEHAVIOR
1. Participants regularly document applicant experiences, attributes
and metrics in summary narrative
100%
2. Participants recognize and document red flags in summary narrative

100%
3. Participants apply holistic review consistently

100%
4. Participants demonstrate inter-rater reliability 95%
LEVEL 4 : RESULTS
1. Matriculation of a diverse class to NSM every year 100%

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

The emerging majority in the United States health professions: a gap analysis innovation model for Latinx recruitment planning within higher education

PDF

The evaluation of in-person versus virtual interviews from underrepresented minorities in medicine

PDF

An examination of ethnic minority physician representation in clinical medicine to reduce health disparities among minoritized groups

PDF

The underrepresentation of Latinx in entrepreneurship and the identification of social, societal, and institutional barriers to close the gap

PDF

The factors supporting or inhibiting Teachers of Color to accept and stay in an international school in Southeast Asia

Asset Metadata

Creator Tolbert, Linda Davis (author)

Core Title Underrepresented minorities in medicine in the United States: an innovation study to develop an effective holistic admissions process for the New School of Medicine

School Rossier School of Education

Degree Doctor of Education

Degree Program Organizational Change and Leadership (On Line)

Publication Date 02/25/2020

Defense Date 12/18/2019

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag health care disparities gap,holistic admissions,holistic review,medical school admissions,OAI-PMH Harvest,physician diversity,underrepresentation gap,underrepresented minorities in medicine (URiM)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Tobey, Patricia Elaine (committee chair), Robles, Darline P. (committee member), Tiwana, Ravneet (committee member)

Creator Email ellemed21@gmail.com,ltolbert@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c89-273866

Unique identifier UC11673482

Identifier etd-TolbertLin-8207.pdf (filename),usctheses-c89-273866 (legacy record id)

Legacy Identifier etd-TolbertLin-8207.pdf

Dmrecord 273866

Document Type Dissertation

Rights Tolbert, Linda Davis

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA