Close
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Evaluating collective impact in a local government: A gap analysis
(USC Thesis Other)
Evaluating collective impact in a local government: A gap analysis
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
Running head: EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 1
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT: A GAP ANALYSIS
by
Alexis Renee Munoz
_____________________________________________
A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION
August 2018
Copyright 2018 Alexis Renee Munoz
Running head: EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 2
ACKNOWLEDGEMENTS
None of this would have been possible without the love and support of my late
grandparents Beatriz and Juan Rocha who always wanted me to pursue my dreams, work hard
for something I loved, and make a contribution in this world. They always encouraged me to be
curious, to think beyond delineated boundaries, defy norms, and exceed expectations. No one
ever believed in me more than they have. If you can dream it, then you can achieve it – they’d
say. I know that they’re looking down from heaven in this moment beaming with joy.
I’d like to thank my family, Ralph, Debbie, Mari, Amer and baby Z for supporting me
with love and generosity – I wouldn’t have been able to go into my defense without my mock
dissertation expert panelists. I’d especially like to thank you mom who supported me by
accompanying on long car rides back/forth from LA, talking me through theories and project
ideas, and adventuring with me around the world on yoga retreats. I’d like to thank my
incredibly supportive friends Nicole, Sheilah, Bianca, Danette, Monique, Autumn, Traci and
Sierra - you girls have kept me grounded and smiling even when I carried a cloud of stress over
my head.
I’d like to also thank my dissertation committee Dr. Melora Sundt, co-chair, and Dr. Liz
Hernandez for their encouragement and advice. I’d like to give a special thank you to my boss
and director, Nick Macchione, who has served as the most visionary, ambitious and inspirational
mentor I’ve ever known. My committee has helped me again and again think about academic
studies with practicality in mind. I’d like to offer a special thanks to my committee chair, Dr.
Yates, co-chair, who helped expedite the completion of this dissertation. Writing this paper
pushed me harder than any professional project ever has, which has given me confidence that I
can deliver under pressure. I appreciate you working with me through the program on
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 3
independent studies, which facilitated my reasoning and I would never have been able to have
been successful in my dissertation paper, without his guidance and expertise.
Finally, I’d like to thank my spiritual guru and one of greatest supporters, De Vida Gray
Touré you’ve been a friend/god-mother and angel in my spiritual development and personal
growth – thank you for always providing me unwavering love, support, positivity, and
unconditional love. Thank you for being a source of inspiration, taking my successes as your joy
and motivating me to succeed. Thank you for praying that everything goes not only into my plan
but also into God’s plan. Thank you for seeing my accomplishments as the ones you’ve always
manifested for me.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 4
TABLE OF CONTENTS
ACKNOWLEDGEMENTS .................................................................................................................2
LIST OF TABLES ................................................................................................................................9
LIST OF FIGURES ........................................................................................................................... 10
ABSTRACT ....................................................................................................................................... 11
CHAPTER ONE: INTRODUCTION............................................................................................... 12
Introduction of the Problem of Practice ................................................................................... 12
Context and Mission of the Healthy County Government ....................................................... 17
Organizational Performance Goal ........................................................................................... 18
Related Literature ................................................................................................................... 19
Relationship Between Healthy County Government and Community Health ........................ 19
The Role of Backbone Organizations and Communicating Shared Data and Measurement
Practices ............................................................................................................................... 21
Decision-Making that is informed by the Collective Impact Model within Healthy County
Government ......................................................................................................................... 24
Importance of the Evaluation .................................................................................................. 26
Purpose of the Study ............................................................................................................... 27
Organization and Stakeholders’ Performance Goals ................................................................ 28
Organizational goal .............................................................................................................. 28
Stakeholders and stakeholders’ goals .................................................................................... 29
Stakeholder Performance Goal and Critical Behaviors .......................................................... 31
Purpose of the Project and Research Questions ..................................................................... 35
Methodological Framework.................................................................................................. 36
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 5
Definitions ........................................................................................................................... 36
Organization of the Study ....................................................................................................... 39
CHAPTER TWO: REVIEW OF THE LITERATURE ................................................................... 40
Introduction ............................................................................................................................ 40
Process of Reviewing Literature ........................................................................................... 42
Knowledge, Motivation, & Organization: The Conceptual Framework ................................. 43
Motivation............................................................................................................................ 53
Organization and Culture...................................................................................................... 57
CHAPTER THREE: METHODOLOGY ........................................................................................ 63
Purpose of the Project and Questions ...................................................................................... 63
Conceptual and Methodological Framework ......................................................................... 64
Motivation Assessment ........................................................................................................ 73
Organization/Culture/Context Assessment .............................................................................. 78
Participating Stakeholders and Sample Selection Population................................................... 81
Sampling .............................................................................................................................. 81
Recruitment .......................................................................................................................... 82
Instrumentation....................................................................................................................... 83
Survey .................................................................................................................................. 83
Focus Group and Individual Interviews ................................................................................ 84
Data Collection ....................................................................................................................... 86
Online Surveys ..................................................................................................................... 87
Individual Interviews ............................................................................................................ 87
Data Analysis ......................................................................................................................... 87
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 6
Surveys ................................................................................................................................ 87
Trustworthiness of Data .......................................................................................................... 87
Ethics ................................................................................................................................... 88
Role of Investigator ................................................................................................................ 89
Limitations ............................................................................................................................. 89
CHAPTER FOUR: RESULTS AND FINDINGS ........................................................................... 91
Participating Stakeholders ...................................................................................................... 93
Data Collection .................................................................................................................... 99
Data Validation ...................................................................................................................... 99
Results and Findings for Knowledge Causes ......................................................................... 102
Factual Knowledge ............................................................................................................. 105
Conceptual Knowledge ...................................................................................................... 110
Procedural Knowledge ....................................................................................................... 112
Metacognitive Knowledge .................................................................................................. 115
Results and Findings for Motivation Causes ......................................................................... 117
Value ................................................................................................................................. 121
Self-Efficacy ...................................................................................................................... 123
Mood ................................................................................................................................. 125
Goal Orientation ................................................................................................................. 126
Attribution Theory ............................................................................................................. 128
Results and Findings for Organization Causes ...................................................................... 129
Resources ........................................................................................................................... 133
Policies, Processes & Procedures........................................................................................ 134
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 7
Cultural Models.................................................................................................................. 137
Summary of Validated Influences ......................................................................................... 139
Knowledge ......................................................................................................................... 139
Motivation.......................................................................................................................... 140
Organization....................................................................................................................... 140
CHAPTER FIVE: RECOMMENDATIONS AND EVALUATION ........................................... 142
Purpose of the Project and Questions .................................................................................... 142
Recommendations to Address Knowledge, Motivation, and Organization Influences ............ 144
Organizational Goals .......................................................................................................... 145
Stakeholder Goals .............................................................................................................. 146
Knowledge Recommendations ........................................................................................... 147
Motivation Recommendations ............................................................................................ 161
Organization Recommendations ......................................................................................... 166
Summary of Knowledge, Motivation and Organization Recommendations ........................... 170
Integrated Implementation and Evaluation Plan .................................................................... 174
Organizational Purpose, Need and Expectations ................................................................. 174
Implementation and Evaluation Framework ....................................................................... 176
Level 4: Results and Leading Indicators ............................................................................. 178
Level 3: Behavior ............................................................................................................... 180
Level 2: Learning ............................................................................................................... 184
Level 1: Reaction ............................................................................................................... 189
Evaluation Tools ................................................................................................................ 190
Data Analysis and Reporting .............................................................................................. 192
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 8
Summary of the Implementation and Evaluation ................................................................... 192
Limitations and Delimitations ............................................................................................... 193
Recommendations for Future Research ................................................................................. 195
Conclusion ........................................................................................................................... 196
References ........................................................................................................................................ 198
APPENDIX A .................................................................................................................................. 218
APPENDIX B .................................................................................................................................. 219
APPENDIX C .................................................................................................................................. 225
APPENDIX D .................................................................................................................................. 227
APPENDIX E ................................................................................................................................... 228
APPENDIX F ................................................................................................................................... 229
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 9
LIST OF TABLES
Table 1: Organizational mission, global goal and stakeholder goals 31
Table 2: Summary of Knowledge Influences and Method of Assessment 67
Table 3: Summary of Motivation Influences and Method of Assessment 75
Table 4: Summary of Organization Influences and Method of Assessment 79
Table 5: Information about the HPS who Participated in the Online Survey 95
Table 6: Information about the HPS who Participated in Face-to-Face Interviewees 98
Table 7: Results of the Knowledge Survey by Type of Dimension: Factual 104
Table 8: Results of the Knowledge Survey by Type of Dimension: Conceptual 109
Table 9: Results of the Knowledge Survey by Type of Dimension: Procedural 112
Table 10: Results of the Knowledge Survey by Type of Dimension: Metacognitive 114
Table 11: Results of the Motivation Survey by Types of All Motivational Dimensions 118
Table 12: Results of the Motivation Survey by Mean Score 121
Table 13: Results of the Organizational Survey by Type of Organizational Dimension 130
Table 14: Results of the Organizational Survey by Mean Score 132
Table 15: Summary of Knowledge Influences and Recommendations 148
Table 16: Summary of Motivation Influences and Recommendations 163
Table 17: Summary of Organization Influences and Recommendations 168
Table 18: Outcomes, Metrics, and Methods for External and Internal Outcomes 179
Table 19: Critical Behaviors, Metrics, Methods, and Timing for Evaluation 181
Table 20: Required Drivers to Support Critical Behaviors 183
Table 21: Evaluation of the Components of Learning for the Program 188
Table 22: Components to Measure Reactions to the Program 189
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 10
LIST OF FIGURES
Figure 1: The Gap Analysis Process 65
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 11
ABSTRACT
This dissertation project applies the Gap Analysis framework to evaluate the knowledge,
motivation, and organizational (KMO) factors that impact the role of Health Promotion Staff
(HPS) leading a collective impact initiative. The purpose of this study was to determine the
factors that influence the ability of HPS employed within a local government to communicate
shared health data and measurement practices between Executive Leadership and community
stakeholders. Using a sequential explanatory mixed methods approach, this study is
characterized by the collection and analysis of quantitative data through an online survey of
thirty respondents, followed by a collection and subsequent analysis of qualitative data through
individual interviews with six participants. Findings indicate assumed influences and causes of
the gaps are caused by lack of factual, conceptual and metacognitive knowledge, self-efficacy in
motivation, and lack of organizational policies. Evidence based recommendations include
conducting educational efforts, ongoing trainings, facilitating information sharing and overseeing
an internal audit of policies. This study contributes to the body of research of health promotion
efforts conducted by local governments, especially those guided by collective efforts to promote
population health; and additionally, provides increased understanding of KMO factors among
HPS that impact communication of data between Executive Leadership and community
stakeholders.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 12
CHAPTER ONE: INTRODUCTION
Introduction of the Problem of Practice
Chronic illness such as heart disease, stroke, cancer, diabetes, and respiratory diseases are
the leading causes of death worldwide, accounting for approximately sixty-percent of all deaths
(CDC, 2017). There is increasing evidence that chronic disease places strain on individuals,
health care systems, and government resources (CHCF, 2017). In California, for example, those
with chronic illness account for more than eighty-percent of all health care spending and this
number is expected to continue increasing by over twenty-five percent before the year 2050
(CHCF, 2017; Hung et al., 2007). Other statistics including rising mortality and morbidity at the
regional level have been equally pronounced. In 2010, the County Board of Supervisors (BOS)
from a local government in the Western Region of the United States adopted a population health-
based initiative that they believed would make meaningful, lasting and fiscally responsible
change. They felt that this initiative, which embodied collaborative efforts, would be imperative
to improve health promotion and disease prevention across the region.
The BOS selected a Collective Impact Model (CI) based strategy to unite all County
sectors to accomplish large-scale policy and community impacts (Erwin, Greene, Mays, Ricketts,
& Davis, 2011; Flood et al., 2015; Kania, & Kramer, 2013). This comprehensive, long-term
initiative aimed to redefine the role of the government as a steward of public welfare (Kania, &
Kramer, 2011; Kania, & Kramer, 2013). In their directive, the BOS decided that in order to
tackle the growing problem of preventable disease-related deaths they would need to look at the
drivers of health, which require adaptive and innovative thinking across multiple fronts. The
BOS appointed the internal Local Health Department to lead the local government in this
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 13
initiative. From this point forward, the Local Health Department within the local government
will be renamed “Healthy County Government (HCG).”
Prior to adopting the CI initiative, HCG had adopted a health promotion approach to
achieving health and well-being. This approach is founded on the principles of understanding
and addressing the primary causes of poor health outcomes and developing strategies to prevent
disease and injuries and promote health among populations-at-large (IOM, 2003; Maben, &
Clark, 1995). In 1986, the first international conference on health promotion was held and one
outcome was the Ottowa Charter, which described the tenets of improving health for all (WHO,
1986). The goal was to reach a state of physical, mental and social well-being and to extend
efforts beyond the field of health. Since then, nine global health promotion conferences have
been held, which have promoted public health, health equity, and more recently health in city
planning, which was described in the Shanghai Consensus on Healthy Cities in 2016 (WHO,
2017). The focus has expanded to explore a world where health is at the center of all
conversations, programs and is embodied in all policies (WHO, 2017).
Traditional health promotion models have helped to identify opportunities for integrative
approaches in programs, policies and interventions. However, as time has passed, key
stakeholders continue to advocate and demonstrate an increasing need for multi sector
collaboration, more integrative approaches and a more generative solutions to achieving
improved population health. In approximately 2005, public health researchers started to discuss
disparities in life expectancy across different regions and identified the social gradients in health
impacted by social inequalities or the social determinants of health (Marmot & Wilkinson,
2005). Researchers viewed social ecological strategies to explore environmental factors and
their impact on health status (Glasgow, Lichtenstein, & Marcus, 2003; Green, 2006; Kania, &
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 14
Kramer, 2013; Murray at al., 2013; Webb et al., 2010). Many have suggested and continue to
agree, even today, that it is increasingly important to collaborate across sectors. In order to
improve and establish sustainable improvements in health outcomes – strategies should offer
solutions across multiple levels of intrapersonal, interpersonal, and population health
(Bronfenbrenner, 1992; Brownson, Baker, Deshpande, & Gillespie, 2017; Cooper, 2016; Issel,
2016; Frenk et al., 2010; Frieden, 2010; Parekh et al., 2011). When the BOS adopted the CI
initiative, in 2010, HCG felt that it was both timely and innovative to develop a comprehensive
solution to poor health and increasing costs. This initiative was believed to be a potential
solution because it required such tenets as shared communication, goals, vision, data and
measurement
Similarly, CI initiatives operate under five conditions, which are: (1) coordination
through a backbone organization, (2) unified by a shared agenda, (3) providing shared data and
measurement practices, (4) participating in constant communication and (5) undertaking
mutually reinforcing activities (Kania & Kramer, 2013). When the BOS adopted the vision to
promote health, they stepped forward as the influential champion, or the backbone organization
within the CI model. Literature on CI initiatives outline six responsibilities of backbone
organizations, which are: (1) communicating shared data and measurement practices, (2) guiding
the CI vision and strategy, (3) supporting the aligned activities, (4) building public will, (5)
advancing policy and (6) mobilizing funding efforts (Hanleybrown, Kania, & Kramer, 2012). In
order to accomplish these six responsibilities, the BOS appointed Executive Leadership team
from within HCG to lead CI throughout the region. As lead, HCG took on the oversight of the
developmental efforts, which include connecting networks across sectors, leveraging ideas,
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 15
opportunities and implementing solutions for community driven efforts to maintain the
passionate focus to promote health and wellness.
From 2010 through the present date, HCG has and continues to employ a number of
employees, called Health Promotion Staff (HPS). These health professionals specialize in
implementing CI efforts through community health promotion efforts. During the past ten years,
the CI model initiative has grown to reach more communities, partners, families and individuals
across the County. As needs have expanded and diversified, the number of HPS has increased.
Although HPS work for one organization, HPS span approximately five different departments
within the umbrella of HCG, which means that HPS are tied to more than five funding sources,
project emphases and fall under different leadership teams. Scattered throughout the
organization, there is great variation between HPS roles because staff manage numerous
activities such as public health policies, projects or programs. They participate in outreach,
communication and education to raise awareness of physical and mental health, diet and exercise
efforts, and may participate on emergency or special projects depending on unique local and or
environmental needs.
Due to the fluctuating nature of their roles and diversification across the organization,
performance evaluations of HPS track twenty-two general categories in a five-point scale
(Outstanding, Above Standard, Standard, Improvement Needed and Unsatisfactory). The twenty-
two general categories include: attendance, punctuality, physical fitness, safety practices,
personal neatness, compliance with rules/regulations, cooperation, acceptance of new
ideas/procedures, application of effort, interest in job, accuracy of work, quality of judgment,
public/employee relations, written expression, oral expression, equipment operation, neatness of
work, performance with minimal supervision, promptness in completing work, volume of work
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 16
produced, performance under pressure, and performance in new work situations. However,
performance evaluations of HPS do not evaluate CI responsibilities including how CI roles are
implemented, nor do they evaluate the core areas of CI model initiatives, or the roles of HPS
within the backbone organization. In this study, Executive Leadership aimed to strengthen the
CI efforts by evaluating HPS’ CI performance in a unified manner.
At this time, Executive Leadership within HCG have decided to focus on one of the
backbone organization’s areas of focus, which is communicating shared health data and
measurement practices with community partners. Executive Leadership aims to share
comprehensive data from community partners to increase cross-collaborative efforts. HCG has
explicitly outlined the need for accurate and effective information from the field. However, little
is known about the process of how staff communicate shared health data and measurement
practices across community partners. As such, HCG set out to evaluate the effectiveness of
health data and measurement practices communication between community partners and
government systems. Executive Leadership aimed to gain a better understanding of how HPS
can communicate comprehensive information with community partners, which is a key
responsibility of a backbone agency.
The problem of practice is that the HCG is unclear about how employees are
communicating shared health data and measurement practices with community partners.
Executive Leadership needs comprehensive health data to make informed decisions that will help
reduce the preventable, disease-related deaths among local inhabitants. Executive Leadership
within HCG need employees to exchange information effectively with community partners,
health providers and health systems, health-related organizations, media, academia, and
community-based organizations that are pursuing similar objectives (Laverack, 2007; Kania, &
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 17
Kramer, 2011; Manning, 2014; McDermott, 2010). Related literature indicates that the
percentage of preventable disease-related deaths improves when health professionals effectively
communicate information to Executive Leadership (Savel, Foldy, & CDC, 2012). Healthy
County Government believes the preventable disease related deaths will decrease, or improve,
when they are able to fully implement a CI model for population health.
This model is dependent on timely, accurate and comprehensive exchange of health data.
When CI is properly implemented, it aims to eliminate duplicative efforts and enhance impacts
of an issue by working from multiple perspectives simultaneously (Laverack, 2007; McDermott,
2010). In order to better understand the process of how HCG is working with community
partners to communicate shared health data and measurement practices, this study explored how
data is communicated to determine potential ways to improve effectiveness and quality of
outreach efforts.
Context and Mission of the Healthy County Government
The mission of the HCG is that essential services must be family focused, community-
based and delivered in a cost-effective, outcome-driven fashion. Healthy County Government
aims to protect the health of residents by providing essential health services, especially for those
who are least able to help themselves. Driven by commitment to establish a service delivery
system that is regionalized and accessible, community-based and customer oriented, HCG
utilizes the, person and family centered approach to health care delivery, thereby improving
positive outcomes for all. Services are offered in a no wrong door style, or strategically across
the County, through various geographic service regions. Additionally, HCG operates a service
delivery system that reflects a whole person wellness and emphasizes cross collaborative
partnerships between public-private entities to meet the needs of local communities.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 18
Healthy County Government oversees over eighteen internal departments, nearly two
hundred programs and more than six thousand employees, which facilitate the delivery of
integrated health services among several million inhabitants and a several billion-dollar budget
(CDC, 2014). Healthy County Government employs public health professionals including public
health nurses, mental health, social and eligibility workers who serve clients in an integrated
fashion. Employees across HCG treat families and individuals in need, working across
departments and sharing data to ensure customer needs are met. Departments allocate regional
teams to plan and manage the delivery of government health and social services.
Since the BOS adopted the CI initiative, they have helped HCG to engage with
community leadership in outcome-driven partnerships to meet the unique needs of the diverse
communities and neighborhoods. Healthy County Government utilizes multiple theoretical
approaches to providing health care services and resources including the socio-ecological model,
life-course theory, and collective impact model. Progress is tracked via statistics such as health
data (e.g. mortality, morbidity, and indicators such as life expectancy, and quality of life).
Organizational Performance Goal
The BOS as well as the local government’s Executive Leadership team and HCG believe
that the role of government is a steward of health and social wellness. While initially focused on
improving health or the mortality percentages, HCG is interested in taking a systems-wide
approach to improving health. Healthy County Government aims to not only impact the health,
safety and social welfare of residents, they aim to impact the broader range of social
determinants of health by working with community partners, community stakeholders, and local
municipalities to exchange resources and information (e.g. active transportation routes, public
safety, health, human services, probation and juvenile delinquency services).
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 19
The objective of HCG is to reduce the percentage of preventable disease related-deaths
and improve the health of residents through CI efforts. Other variables under consideration
include improving life expectancy to over 85 years (in 2017, it is 82.3 years), and improving
quality of life (in 2017 it is 94.9 percent) to over 96 percent. Executive Leadership measures
progress and track effectiveness of reaching these objectives using health data compiled by the
(internal) Statistics Unit (SU). The SU reports health data annually and incorporates additional
findings from outreach efforts among HCG staff. Executive Leadership would like to strengthen
the use of the CI model by measuring their utilization and impact of the shared agenda, shared
measurement system, mutually reinforcing activities, continuous communication, and a central
infrastructure, which comply with the five components outlined in CI models (Kania, & Kramer,
2013).
Related Literature
Relationship Between Healthy County Government and Community Health
Dannenberg (2016) suggests that the role of health professionals working for the
government is challenging because preventing disease and improving health status among
communities is interdependent on multiple existing systems such as the economic, social, and
physical environments, health care services, local, state and federal policies, societal social
systems and political leadership. These inter-related systems impact the distribution of money,
power and resources at national state and local levels, and the cost, access to and quality of
health care systems (Gebbie, Rosenstock, & Hernandez, 2003; Glanz, & Bishop, 2010; Golden,
McLeroy, Green, Earp, & Lieberman, 2015; McLeroy, Bibeau, Steckler, & Glanz, 1988). Health
outcomes reflect a complex number of components within a larger ecosystem, and the
reinforcing and reciprocal nature of these systems (Webb et al., 2010).
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 20
Some relationships reinforce positive or negative health outcomes (Gebbie et al., 2003).
For example, public policies addressing safety, public open spaces, access to healthy and
affordable food and high-quality housing can impact the protective or risk factors and contribute
to the cumulative effects of physical and mental health outcomes (Bronfenbrenner, 1992;
Dubowitz et al., 2016; Geronimus, 2000). According to the social ecological model, health
outcomes are the result of dynamic relationship between individuals, groups and health systems
or sometimes called reciprocal determinism (Bronfenbrenner, 1999; Glanz, & Bishop, 2010).
Environmental conditions impact behavioral patterns such as tobacco use, consumption
of unhealthy foods, and sedentary behavior. For instance, these behavioral patterns are more
prevalent in environments with large scale advertising and price incentives for tobacco products,
affordable and easy access to unhealthy food products, which are compounded because of
unaffordable and otherwise limited access to healthy alternatives, and lack of infrastructure for
active living (Golden et al., 2015). In one series of studies, called the Adverse Childhood
Experience (ACEs) studies, researchers evaluated the relationship between living in a socially
advantaged or disadvantaged area as a child and the risk for poorer health outcomes in adulthood
(Felitti et al., 1998). These studies found that early-onset of exposure including the repeated and
chronic activation or inactivation of physiological stress processes are highly related to the life
course, or lifelong health of individuals (Elder, 1998; Felitti et al., 1998).
While behavior choices are critical to the increased risk of illness, disease, and death,
political factors strongly impact societal, environmental elements of individual choices. Federal,
state and local policies have the capacity to influence health outcomes of a community. Pollutant
regulation policies that address water and air, access to healthy foods, the design of road, the
specifications of vehicle emissions, the elimination of lead and exposure to asbestos, and
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 21
regulation of seat belt use impact health outcomes in a population (Golden et al., 2015; Frieden,
2010). Policies that encourage increased use of active transit systems, physical activity, smoke-
free housing, and buildings, have been associated with increased social health and wellness
(Frieden, 2010; Jayasinghe, 2015). Moreover, as procedures, policies and leadership roles move
into the public or governmental sphere, issues related to health behaviors such as taxing alcohol,
tobacco, unhealthy foods and sugar-sweetened beverages are associated with improved clinical
and community health outcomes (Frieden, 2010).
Some argue that the way to improve population health is to work with community
stakeholders, politicians, local municipalities, and the local health departments to formulate
systems-wide approaches that address multiple, dynamic interrelated causal factors (Jayasinghe,
2015). In short, socio-economic, cross-sector and integrative approaches that work in
conjunction with the support of local government are more often becoming a central component
creating lasting, sustainable and improved population health (Frieden, 2010; Brownson, Allen,
Duggan, Stamatakis, & Erwin, 2012).
The Role of Backbone Organizations and Communicating Shared Data and Measurement
Practices
Within the CI model, backbone organizations are responsible for leading sustainable,
long-term, systemic changes throughout a system or community (Turner et al., 2012). Current
health efforts require coordinated and comprehensive strategy among traditional and
nontraditional partners. Backbone organizations must help guide strategy, support aligned
activities, establish shared measurement, cultivate community engagement and ownership,
advance policy, and mobilize resources (Kania & Kramer, 2013). When backbone organizations
complete these activities, they are more likely to develop and implement effective systems and
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 22
community actions (Turner et al., 2012). In this study, HCG serves as an example of the
backbone organization in the CI initiative to promote health among local inhabitants.
Without relying on implementing new policies, procedures and regulations in a singular
effort, backbone organizations in the field of health must help a variety of organizations systems,
to collaborate, share data and measurement practices and to work towards the common goal,
even despite philosophically different approaches (e.g. public, private, government). In the
scope of this study, sharing data and measurement practices is the activity of focus. Executive
Leadership must evaluate the process of collecting data, interpreting data, and informing
decision-making. Healthy County Government is in charge of leading work with partners to
ensure that everyone understands how they will participate in the shared measurement system
using a common set of indicators in a timely and consistent manner.
Effective communication strategies are integral in evaluating health data and improving
the health outcomes of communities (Gilson, Doherty, Loewenson, & Francis, 2007; Wallerstein,
2006). One strategy, which has been adopted by Healthy County Government is the open
government directive described by President Obama in 2010 (McDermott, 2010). This approach
highlights the importance of information sharing for addressing policy issues such as public
safety in the face of natural disasters, anti-terrorism responses and public health situations.
Information sharing is a complex task given interpersonal, intra-organizational, and inter-
organizational factors (Chun, Shulman, Sandoval, & Hovy, 2010). The open government
directive incorporates three principles: transparency, participation and collaboration. The first
principle, transparency means that the government provides citizens with information about what
they are doing to promote accountability. The second principle, participation is meant to
encourages public engagement by increasing opportunities for the public to participate in
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 23
policymaking and provides the government with the collective knowledge, ideas, and expertise
of the populations thought to improve the Government's effectiveness and improves the quality
of its decisions. The third principle is collaboration or to foster partnerships and cooperation
among the federal government agencies, across all levels of government and with nonprofit
organizations, businesses, and individuals to improve the effectiveness of government
(McDermott, 2010).
While Healthy County Government has adopted an open systems procedural approach to
communicate data and measurement practices within their system, these processes have not been
formally evaluated. As the backbone organization, Healthy County Government is responsible
for communicating data and measurement practices consistently across all partners to ensure
efforts remain aligned and participants hold each other accountable (Hanleybrown et al., 2012).
Healthy County Government is charged with evaluating efforts to better understand the extent to
which and how the initiative’s ultimate outcomes have been achieved and the extent to which the
CI efforts have contributed to these outcomes (Turner et al., 2012). When communities share
information, it can inform the learning process by uniting causes, which contributing to long-
term and sustainable improvements (Kania, & Kramer, 2013). This type of data sharing requires
consistent and open communication is between HPS and community partners. Data sharing
typically requires a secure level of trust to assure mutual objectives, and create common
motivation (Turner et al., 2012). While HCG believes that HPS are sharing data and
measurement practices, they would like to better understand this process. Executive Leadership
seeks timely, high-quality data and measurements that enables reflection and informs strategic
and tactical decision-making and this dissertation study supports these ongoing efforts.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 24
Decision-Making that is informed by the Collective Impact Model within Healthy County
Government
Local governments have the potential to impact many lives through the provision and
allocation of potential programs, services, resources, of funding which can support or challenge
efforts to promote health, safety and well-being. Researchers have demonstrated that
government decisions regarding social determinants of health (e.g. sectors such as: housing,
transportation, public safety, education, sustainability, climate change, parks, air and water
quality, criminal justice, and economic development and food access) impact social, educational,
welfare and health outcomes of individuals, families and communities. The CI model helps
inform important decisions over the lifetime of a CI initiative. In some cases, decisions
regarding the CI structure, strategic direction, choices regarding resource allocation, and
communications are impacted by ongoing evaluation efforts. The goal of measurement and
evaluation is to provide organizations with the specific information they need at a given point in
time. This information aims to help decision makers understand the effectiveness of an initiative
and make well-informed evaluations about its strategy and activities.
In the field of public health, backbone organizations have a unique role, which has been
described as especially hard to manage (Turner et al., 2012). Relationships between activities
are often not directly associated, which make these relationships challenging to measure. For
example, increasing health promotion activities among a set of individuals does not directly
cause individuals to experience improved health outcomes in an immediate sense (Kania, &
Kramer, 2013). Moreover, there is a shift away from providing volume to value of care.
Researchers argue that coordinating government and community efforts is necessary to set up a
strong foundation for continued efficiency and innovative collaborative approaches with external
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 25
partners (Yasnoff, O'carroll, Koo, Linkins, & Kilbourne, 2001). Studies have shown that when a
Local Health Department establishes a common agenda and shared strategies with the public,
there are improvements in community social capital, strengthened partnerships, multi-sector
approaches to promote public health and innovative policy changes (Marmot, 2005).
Researchers have indicated that more than fifty-percent of economic growth in the U.S.
during the last century was the result of improvements in public health efforts to support
advances in population health (Homer & Hirsch, 2006). Policies that make communities
healthier have seen significant improvements in lowering health care cost, creating jobs and
increasing tax revenue (Green, 2006). As the understanding of health outcomes increases,
decision makers recognize the direct and indirect nature, as well as the complexity and
interdependency of policy decisions (Marmot, 2005). Since adopting the CI model strategy,
HCG has been working to ensure decisions are being made with health promotion practices in
mind, such as prioritizing the health and well-being of communities at-large, which have been
described in previous literature (O'carroll, Cahn, Auston, & Selden, 1998). While Executive
Leadership believes that decision-making must be collective to be sustainable, therefore it must
be driven by local community needs, they maintain that decisions must be results-oriented,
focused yet adaptive, and consider long-term ramifications. Executive Leadership believes in
leveraging acquisition and communication of health data to improve population health goals. By
using the CI model, Healthy County Government is invested in the alignment of effort between
departments, staff and the community to improve community health through effective policies,
which is described in the literature as a comprehensive approach to improving population health
(Trochim, Cabrera, Milstein, Gallagher, & Leischow, 2006).
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 26
In order to transform complex problems such as fragmented, inefficient health systems
into unified delivery systems, backbone organizations have to pursue concurrent activities and
goals, such as communicating shared data and measurements. While these activities are
complex, time consuming and labor intensive, they have been shown to improve social outcomes
through across. Researchers have found that the multifaceted relationship between politicians,
policies, health systems, health care providers, community organizations and the individuals
served is intricate and requires sophisticated strategies (Wallerstein, 2006). One consistent
finding has been that collective approaches have been shown to promote the public health
(House, Landis, & Umberson, 1988). Additionally, integrative and aligned efforts have resulted
in the reduction of risk factors for disease and injury. Having a unified vision, common
activities, shared data, strong public will and policies and available public funding foster success
among backbone organizations. In the case of HCG, evaluating and improving in these indicator
areas helps leaders to understand and incorporate the health promotion vision into decision-
making. With a broad range of public health employees, ranging from clinical, to field workers
and policy advisors, it is essential to collaborate shared concerns for cost effective delivery of
resources and services, prevention and community health in effective and efficient ways.
Importance of the Evaluation
It is important to evaluate an organization’s organizational goals and performance goals
to better understand how an organization like HCG is achieving community-level health
improvements and contributing to improved social outcomes. CI models are successful when all
participants are aware of changes in context, conditions and circumstances, which is why
evaluation should take place periodically during the implementation of efforts. Additionally,
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 27
these results can provide Executive Leadership with strategies for more integrative and effective
use of future communication (Glasgow, Lichtenstein, & Marcus, 2003).
When there is continuous and comprehensive communication, there is a greater
likelihood for improved health service delivery, collective impact and improved community
health status of all inhabitants improves (including life expectancy, infant mortality,
homelessness, health-related quality of life), and more there are greater levels of commitment
and engagement among employees (Lau, Hagens, & Muttitt, 2006). Employees significantly
influence the outcome of work projects (Bong, 2001). When there is strategic and purposeful
communication, there is a higher likelihood that all employees working towards a common
purpose and towards organizational goals (Ginsberg, 2005). Improving communication
strategies mitigates risks, such as less effective strategic planning, ineffective use of government
spending and the misallocation of human capital, uncommitted and disengaged employees.
When risks are not averted, they contribute to static or worsening health outcomes among
community members.
Purpose of the Study
The purpose of this study was to examine the knowledge, motivation, and organizational
(KMO) factors that influence the ability of HPS to achieve their role of communicating shared
data and measurement practices that can be used to evaluate the implementation the CI model.
Evaluating the CI model enables Executive Leadership to know whether or not the purpose of
adopting the CI model is being realized. Thus, to better serve the community and further the
organization’s mission, an in-depth evaluation of HPS CI activities illuminated the status of HPS
KMO needs of previous CI efforts. By performing an evaluation of existing activities and
strategies among health promotion staff, Executive Leadership gained a greater understanding of
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 28
the current status of health of the community. Moreover, Executive Leadership team is now
better informed to implement ongoing evaluations of HPS performance. Executive Leadership
additionally, is better able to identify strategies for integrative and effective CI efforts to
ultimately facilitate community-based health service delivery among communities within the
County.
Organization and Stakeholders’ Performance Goals
A stakeholder group is a group of individuals who directly contribute to and benefit from
the achievement of the organizational goals (Clark, & Estes, 2008). In this section, the
organizational goal will be described, followed by the three stakeholders from the HCG and their
role as it contributes to the achievement of the organization’s goal.
Organizational goal
The organizational goal of Healthy County Government is to offer disease prevention and
health promotion activities to improve the health status and quality of life, and social well-being
of residents (via mortality and quality of life indicators). Indicators of mortality are measured by
the regional Medical Examiner’s Office. Quality of life is measured by the SF-12, which
measures eight domains of health: physical functioning, bodily pain, general health, vitality
health, vitality, social functioning, mental health, emotional health and physical health in an
aggregate score, and has been validated with a Cronbach alpha of .89 (Ware Jr., Kosinski, &
Keller, 1996).
More specifically, the organizational goal is to serve as the backbone organization and
redefine the role of government as a steward of innovative health services and resources by using
the CI model. The goal of HCG is to run its operations in a way that supports a region that
combines the collective strengths of community stakeholders, transforms policies and systems
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 29
and integrates delivery of services and resources. Healthy County Government sees the value in
being transparent and facilitating collaboration and participation to the health and well-being of
the millions of local residents in order to reach the mission’s goals of fostering a community
where residents are healthy, safe and thriving.
Stakeholders and stakeholders’ goals
The first stakeholder is the BOS. Each member of the board is elected to four-year terms
in their respective districts. The critical behaviors of the BOS include fulfilling executive,
legislative and quasi-judicial roles such as the delegation of certain authorities, the execution and
agreements and procurement of activities to County officials and designees. Their goal is to
make decisions regarding the allocation of federal, state and local taxes and funding aimed at
impacting the public’s health and safety of local residents in a in an efficient and effective
manner. The BOS’ goal is to make timely decisions with respect to the health and safety of local
residents.
The second stakeholder are the several million local residents served by HCG. Residents
reflect diversity in age, race and ethnicity, with the population being shaped by diverse ethnic
groups, military, refugee, Mexican and Native American influences; and the average person is
considered more well-educated and wealthier than the average person from neighboring regions.
The critical behaviors of this group are to work, live, learn, and to participate in leisure,
community and faith-based activities. The goal of this group is to be safe and healthy. The goal
of local inhabitants is to take responsibility for their own health and safety and use health data to
make positive health behavior decisions.
The third stakeholder are the public health professionals, specifically the Health
Promotion Staff (HPS) that interact with individuals, families and communities and serve as the
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 30
liaisons between the local government and the community. HPS represent less than ten percent
of the staff within the Healthy County Government. HPS are responsible for implementing the
CI initiative efforts with community partners on a daily basis. Executive Leadership admits that
HPS are sometimes described as the “face of the local government by communities, and in turn
the face of the community by the local government.” Responsibilities of HPS include spending
time in the field, also called the community, facilitating health services and resources. The staff
is responsible for implementing the CI model and serving as the liaison between the backbone
organization and communities sharing data and measurement practices. Some HPS are less
successful at communicating shared data effectively because they lack a thorough understanding
of CI model, the value of information, how to communicate shared data, how this data informs
progress of the CI initiative and how data relates to the goals of Healthy County Government;
which are essential components in the successful implementation of the CI model initiative
(Kania, & Kramer, 2011). Failure to fulfill roles and responsibilities begins with HPS that lack
comprehension of the expected tasks, mechanisms, processes, motivation and organizational
support. When HPS fail to communicate shared data and measurement practices, this results in
an inability to reach goal of improving health status, quality of life and life expectancy.
In this study, HPS activities and implementation practices of communicating shared data
were explored. Executive Leadership aimed to learn more about the acquisition and
communication strategies of health data. Executive Leadership from Healthy County
Government believed that an evaluation would be the best way to learn about their processes. A
summary of the organization’s mission, goal and the stakeholder goals are shown in Table 1.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 31
Table 1.
Organizational mission, global goal and stakeholder goals
Organizational Mission
The hallmark of Healthy County Government is its commitment to a service delivery system
that is regionalized and accessible, community-based and customer oriented.
Organizational Goal
In 2010, the local mortality rates were caused by chronic disease, and over sixty-five percent
were caused by preventable risk factors. Executive Leadership team from HCG and the BOS
from adopted a comprehensive, long-term initiative to redefine the role of government as a
steward of health, safety and wellness. Specifically, HCG established a goal to reduce the
percentage of preventable disease to below fifty percent, and improve indicators including life
expectancy and quality of life.
Stakeholder 1 Goal:
Board of Supervisors
Given accurate and complete
information from the HCG,
the BOS make timely policy
decisions with respect to
improving the health and
safety of local residents.
Stakeholder 2 Goal:
Inhabitants of the County
To take responsibility for
their own health and safety.
Stakeholder 3 Goal:
Health Promotion
Specialists
To communicate shared data
and measurements in order to
help inform policy decisions
about the health and well-
being of local residents.
Stakeholder Performance Goal and Critical Behaviors
The stakeholder of focus for this study was Health Promotion Staff (HPS), which are a
subset of employees from the HCG that specialize in public health fieldwork. They were
selected because they serve as the bridge between the HCG and local community residents. HPS
are primarily tasked with promoting disease prevention and health promotion efforts among all
local inhabitants (Battel-Kirk, Barry, Taub, & Lysoby, 2009; Edelman, Mandle, & Kudzma,
2013; Naidoo, & Wills, 2016; Olds, 2006). HPS often spend a large portion of their time in local
communities or in the field implementing and promoting populational health efforts including
planning and coordinating events, programs, and campaigns (Sharma, 2016). When they are in
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 32
the field, HPS must effectively work with community stakeholders to help implement the CI
vision.
As Healthy County Government’s aim for health promotion has evolved into systems-
wide changes, the role of HPS evolved as well. While in the past, and prior to this study, HPS
focused their efforts on health conditions in singular efforts, the emerging role of HPS utilizes
socio-ecological or collective models to improve health (Auerbach, 2016). Instead of reacting to
conditions, HPS’ efforts are more proactive in nature (Catford, 2009). HPS in Healthy County
Government are employed in multiple departments, and while they focus on different health
topics they primarily are responsible for facilitating education, participation, and sustained
engagement in community outreach efforts. HPS spend a majority of their time in communities.
They attend town hall and community collaborative meetings, working with community
stakeholders on specific initiatives such as school district systems, obesity prevention, and
behavioral health, and provide support and integrate cross-sector efforts. Health professionals,
specifically the HPS, have many competing priorities and obligations due to the nature of
working in the field. Fluctuating meeting schedules, commuting to a variety of locations,
spending less time at their desk are several differences compared to workers who do not
primarily work in the field. However, HPS working in the field have greater access to partner
organizations and community stakeholders. Given their extensive time and energy working with
community partners, HPS have the potential to acquire and communicate comprehensive health
data that impact their community and contribute to the Healthy County Government’s goal of
improving the population health.
The performance goal of the HPS field staff is to utilize the core competencies as a
strategy to communicate comprehensive shared data and measurement practices about local
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 33
communities that leadership requires making informed decisions about the health and well-being
of local residents. The processes and systems that were examined in this study evaluated
systems-wide mechanisms that reinforce, encourage, and reward CI performance efforts
behaviors on the job. These efforts evaluated supportive and challenging factors, which facilitate
communicating comprehensive information in a way that is: clear and concise, timely,
forthcoming and continuous, objective, and measured for effectiveness. Behaviors include those
related to how HPS acquire shared data and measurement practices pertaining to the three core
competencies. The performance goal in this study is that HPS will collect and communicate one-
hundred percent of the shared health data and measurement practices required by Executive
Leadership by December 31, 2018. The current study contributed to this goal by providing a
baseline assessment of current practices of communication of shared data and measurement
practices, which will help Executive Leadership evaluate the implementation of the CI model.
After the initial assessment is completed for this study, the long-term goal of HPS is to
demonstrate increased responsiveness to community needs and better understand the population
they serve and better address community needs.
Government employed public health field workers are sometimes considered gatekeepers
of information, with eyes and ears in the community when it comes to learning the candid
opinions and attitudes about health policies and politics (Love, Gardner, & Legion, 1997). As
gatekeepers, field workers, are asked to consider multiple factors including the cultural and
political acumen, role of policies and government, the conditions and environments in which
policies and governments function before acknowledging, agreeing or disagreeing while in the
field. Field workers are the recipients of emerging information that may not yet be available in
reports, including community residents and stakeholders’ attitudes, opinions, current events, and
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 34
opportunities that directly or indirectly contribute to individual and community health behavior
changes (Love et al., 1997; Sallis, Owen, & Fisher, 2008; Rosenthal et al., 2010; Witmer, Seifer,
Finocchio, Leslie, & O’Neil, 1995). HPS as field workers have the capacity to collect
information, which may not be readily available in surveys or traditional collection methods.
Field staff may be tasked with suggesting or facilitating changes within existing systems,
such as policy development or policy amendments, these changes are multi-pronged and involve
political, physical, social or organizational systems. Systems changes require dynamic
partnerships to sustainably impact conditions of health (Glanz, & Bishop, 2010; Golden et al.,
2015; Green & Allegrante, 2011; IOM, 2003). As field staff health professionals are one
component in the process of executive decision making within HCG, it is important to
understand how field workers acquire and communicate health data to executives as well as
community partners.
Critical behaviors define the stakeholder activities that most influence the stakeholder
and organizational outcomes (Kirkpatrick, & Kirkpatrick, 2016). Critical behaviors are the
necessary activities that HPS field staff must complete to achieve performance goals. The
leading indicators are short-term observations and measurements suggesting that critical
behaviors are on track to create a positive impact on desired results. Leading indicators provide
the measures and methods to track the stakeholders’ progress in achieving their goals, and the
organization’s general goal (Kirkpatrick, & Kirkpatrick, 2016). In this study, three critical
behaviors were measured related to communicating shared data and measurement practices to
determine whether one-hundred percent of health data and measurement is being acquired and
transferred to Executive Leadership. The three critical behaviors are to:
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 35
1. List the activities related to acquiring and communicating activities that improve delivery
of health services and resources through clinical delivery and IT systems (e.g. improving
their systems-wide internal or external approaches).
2. List the activities related to programs or projects that facilitate cultural models
characterized by which the institutional, social and physical environment support the
mindset that healthy behaviors are the norm (e.g. activities that facilitate holistic health in
a normalizing manner).
3. List the activities related to policies and environmental procedures/regulations that have
been adopted or reinstated to support how physical, human and intellectual resources are
generated and allocated, including their geographic and needs-based allocation (e.g.
activities that facilitate holistic health in a normalizing manner).
The three leading indicators are whether HPS are (1) connecting on a routine basis with
CI partnering organizations, (2) collecting the shared data and measurement practices in a
systematic way, and (3) reporting the shared health data and measurement practices to
supervisors on an ongoing and accurate way, so that Executive Leadership can participate in
informed decision-making.
Purpose of the Project and Research Questions
The purpose of this study was to examine the KMO factors that influence the ability of
HPS to achieve their role as a participant in the backbone organization, of communicating shared
data and measurement practices that can be used to evaluate the implementation the CI model.
Without evaluating the CI model, there is no way for Executive Leadership to know whether the
purpose of adopting the CI model is being realized. While a complete performance evaluation
would focus on all stakeholders, for practical purposes this study selected only one of the
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 36
stakeholders, which was HPS field staff. The research questions that guided the evaluation study
were:
1. What are the KMO factors that challenge HPS field staff in their communication of
shared health data and measurement practices of one-hundred percent comprehensive
information required by Executive Leadership?
2. What are the KMO opportunities that exist to facilitate HPS field staff’s ability to
communicate shared health data and measurement practices of one-hundred percent
comprehensive information required by Executive Leadership?
Methodological Framework
The gap analysis framework (Clark, & Estes, 2008) was adapted as an evaluation model
to review the data and measurement practices among HPS. The data that was collected was
qualitative and quantitative data regarding HPS knowledge (factual, conceptual, procedural and
metacognitive) of their responsibilities, motivation (value and confidence) in acquiring and
communicating these data to leadership, and the organizational factors (resources, policies and
procedures, and the cultural model) that support communication practices. The method used to
collect data was through an online survey and a subsequent individual interviews. The analytical
techniques adopted include coding of qualitative data using the KMO framework and theme
development and descriptive statistics for quantitative data.
Definitions
Collective Impact (CI) initiative: The movement and commitment of a group of important
actors/organizations from different sectors to a common agenda for solving a specific social
problem that follows the tenets of a Collective Impact Model (Kania, & Kramer, 2012).
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 37
Collective Impact Model: A theoretical model, which addresses broad cross-sector coordination.
Key tenets of the model include: a centralized infrastructure, a dedicated staff, and a structured
process that leads to a common agenda, shared measurement, continuous communication, and
mutually reinforcing activities among all participants (Kania, & Kramer, 2012).
Community (Population): Unified body of individuals who share the same geographic region; or
with common characteristics; which contribute to a common or shared group of interests.
Community Based Health: The idea that a group of people, who are categorized by such
characteristics as neighborhood, zip code, region, age, gender, or ethnic/racial status share
similar health status or outcomes.
Community Health Statistics Unit (CHSU): an internal department within the Healthy County
Government, which provides health statistics that describe health behaviors, diseases and injuries
for specific populations, in addition to health trends and comparisons to national targets.
Critical Behaviors of HPS: (1) Building better delivery systems: listing activities that the CI
partner organizations are participating in to improve delivery of health services and resources
through their systems; (2) Supporting positive choices: listing programs or projects that facilitate
cultural models characterized by which healthy behaviors are the norm, and the institutional,
social and physical environment support this mindset; (3) Pursuing policy and environmental
changes: listing policies and environmental procedures/regulations that have been adopted or
reinstated to support how physical, human and intellectual resources are generated and allocated,
including their geographic and needs-based allocation.
Executive Leadership Team: The twenty executive staff that manage or direct employees in
Healthy County Government to influence and guide these individuals. Those leading executive
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 38
leadership processes typically oversee activities and fulfilling organizational goals, strategic
planning development and overall decision-making. Sometimes called Executive Leadership.
Field Worker/ Field staff: Staff members that spend the majority of their time facilitating the
delivery of health services between the Healthy County Government and the community while in
community settings or environments outside of traditional work offices.
Health Data: Data including health status, and health outcomes such as behavioral factors,
particularly diet and physical activity patterns, tobacco use, alcohol consumption, sexual
behavior, avoidable injuries, mortality percentages, life expectancy, and quality of life.
Health Promotion Staff: The public health promotion staff who practice of promoting and
protecting the health of populations using knowledge from public health sciences.
Local Health Department: A government agency in the United States on the front lines of public
health. They may be local or state government agencies and often report to a mayor, city council,
County BOS, or County commission. There are approximately 2,800 Healthy County
Government across the United States.
Local Government: A form of public administration, which, in a majority of contexts, exists as
the lowest tier of administration within a given state. The term is used to contrast with offices at
state level, which are referred to as the central government, national government, or (where
appropriate) federal government and also to supranational government, which deals with
governing institutions between states. Local governments generally act within powers delegated
to them by legislation or directives of the higher level of government.
Population Health: The health outcomes of a group of individuals, including the distribution of
such outcomes within the group from specific geographic populations, or are useful to describe
activities limited to clinical populations and a narrower set of health outcome determinants.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 39
Public Health: Public health promotes and protects the health of people and the communities
where they live, learn, work and play. Public health as a field aims to prevent people from
illness and injury, while promoting wellness through healthy behaviors.
Shared Data and Measurement Practices: A common set of measures that monitor performance,
track progress towards outcomes that all CI partnering organizations utilize.
Organization of the Study
The dissertation was organized as five chapters. Chapter One provides the reader with
the key concepts and terminology commonly found in a discussion about a Healthy County
Government goal to improve the percentages of preventable disease, and leading health
indicators such as life expectancy and quality of life. The organization’s mission, goals and
stakeholders as well as the initial concepts of gap analysis are introduced. Chapter Two provides
a review of current literature surrounding the scope of the study. The topics in Chapter Two
explored the relationship between the Healthy County Government and community health, the
role of a backbone organization’s communication of data and measurement practices, and the
relationship between communication and decision-making within a Healthy County Government.
Chapter Three details the assumed causes for this study as well as methodology of recruitment
efforts, data collection and analysis. In Chapter Four, the data and results are assessed and
analyzed. Chapter Five provides solutions, based on data and literature, for closing the perceived
gaps as well as recommendations for an implementation and evaluation plan for the solutions.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 40
CHAPTER TWO: REVIEW OF THE LITERATURE
Introduction
As mentioned in Chapter One, chronic illness is the leading causes of death worldwide,
accounting for approximately sixty-percent of all deaths (CHCF, 2017). Current literature on the
topic suggests that this statistic is further pronounced by the fact that the number of persons with
chronic illness is rapidly growing given the burgeoning aging population or people living with
multiple chronic conditions. People are living longer and with more prevalence of disease,
disability and conditions, chronic illness, which causes greater strains on individual, health care
system, and government resources (CHCF, 2017). Literature on the topic of chronic illness,
ironically suggests that these are oftentimes preventable and related to behaviors such as tobacco
cessation, improved physical activity, and reductions in poor eating habits. Yet studies show that
while individuals are ultimately responsible for their own health behaviors, there are
opportunities for the government and other large stakeholders to employ systems wide changes
to support better individual choices.
In response to the concerns related to rises in chronic illness, the BOS in a local region in
the Western United States stepped forward to address this concern – recognizing that systems
wide changes are imperative for sustainable change. The local government agreed that
collaborative efforts would be essential to improve health promotion and disease prevention
efforts. The government cited research and best practices, which demonstrated that multi-sector
approaches have touched health systems, government, communities and non-traditional partners,
greater and more sustainable efforts in health promotion are possible.
Healthy County Government (HCG) adopted evidence-based practices based on the
Collective Impact (CI) Model (Kania, & Kramer, 2011). As mentioned in Chapter One, this
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 41
approach is based on the idea that long-term commitments by a group of important actors from
different sectors for a common agenda will contribute to sustainable and popular solutions to
complex social problems. CI models outline that actions are supported by a shared measurement
system, mutually reinforcing activities, and ongoing communication and are led by an
independent backbone organization. Unlike other efforts that invite collaboration, Kania and
Kramer (2011) argue that the CI model solicits integration and generativity across sectors to
rigorously and collectively address complex problems. The CI initiative under review is aims to
accelerate health and wellness by addressing a comprehensive, systematic and environmental
approaches. However, little if any literature exists on CI initiatives which have been undertaken
at a government level and among a region as large as HCG oversees.
In order to better understand the process of how HCG is working with community
partners to communicate shared data and measurement practices, this study explored the role of
one subset of HCG employees, called Health Promotion Staff (HPS). HCG felt that in order to
formulate innovative approaches to communicate data, they must evaluate current knowledge,
motivational and organizational (KMO) factors among HPS that influence the process of
collecting data, interpreting data, and informing decision-making. HCG also felt that they
needed to better understand how HPS communicate this information with community
stakeholders. Executive Leadership acknowledge that HPS must understand how to
communicate shared measurements using a system or a common set of indicators in a timely and
consistent manner. The manner in which data is communicated to determine potential ways to
improve effectiveness and quality of outreach efforts and is vital to health and well-being of
local residents. As HCG continues in nearly a decade of utilizing these efforts, Executive
Leadership felt that at present, it is important to evaluate the initiative’s effectiveness and impact.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 42
Literature on the topic of collective suggest the importance of evaluation, which will be explored
in section.
There are many competing issues to consider including needs of stakeholders and
capacities of staff. The task of recommending solutions along with evaluation and
implementation plans become more complex. This study utilizes the Gap Analysis Framework
(Clark, & Estes, 2008), and the New World Model of Training and Evaluation (Kirkpatrick, &
Kirkpatrick, 2006) to ensure that HCG remains accountable and committed to achieving
measurable health outcomes.
Process of Reviewing Literature
This Chapter aims to explore existing literature on the role of CI initiatives, with a
particular concentration on communicating shared data and shared measurements by a
government stakeholder, the role of HPS, public health, and systems-wide approaches impacts
health. Google Scholar, USC Libraries, JStor, ERIC, and Psycinfo were search databases used to
conduct advanced searches of successful communication strategies resulting from collective
impact models. The review of the literature presented positive communication strategies from
backbone organizations that are similar in context to those described by developers of the
collective impact (CI) model. The results from the searches using keywords: “public health,”
“collective impact,” “government,” “population health,” “community health,” “cross sector,”
“public health professional,” “communicating data,” “health information,” “assessment,”
“analysis,” “assessment,” “evaluation,” and, “effectiveness,” presented limited research,
indicating a need for more studies quantifying evaluation and communication strategies of
backbone organizations.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 43
Knowledge, Motivation, & Organization: The Conceptual Framework
Prior to this study, Executive Leadership from within HCG have been unclear about how
employees acquire and communicate shared health data and measurement practices. Little is
known about the process of how they communicate shared health data and measurement
practices between community partners and Executive Leadership. If workers are not
communicating and acquiring comprehensive or one-hundred percent of health data to Executive
Leadership it is valuable to discern whether proper Knowledge, Motivation or Organizational
(KMO) factors are needed to help improve public health field workers capacities (Glanz, &
Bishop, 1988). The purpose of this chapter is to review the literature and framework around the
evaluation of workers and the impacts on health outcomes of the population served.
This chapter provides background information on KMO factors that Executive
Leadership considered as they evaluated the performance of HPS. Literature in this chapter
explores how communicating shared data and measurement practices impact individual and
organization goals. This chapter also addresses how the learning and motivation literature
explains performance assets and barriers using the Clark and Estes (2008) framework. Chapter
One detailed the impact, influence and implications of decision-making from Executive
Leadership, HCG’s role of being a backbone organization, and the relationship with the health
and well-being of the inhabitants they serve. Chapter One also examined the role of HPS
communicating shared health data and measurement practices for effective decision-making.
This chapter will summarize and synthesize existing publications and discuss the role of HPS,
public health communication strategies that adopt systems wide approaches and processes and
systems that may challenge, or reinforce performance, in order to support positive performance.
The literature review will serve as a foundation and support new insights and generativity of
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 44
ideas. Through reviewing empirical based evidence, Chapter Two seeks to explain how
organizations may support continued acquisition and communication of health data by HPS.
The purpose of this study was to examine the KMO factors that influence the ability of
HPS to achieve their role to obtain the knowledge, skills, motivation and utilize organizational
factors that aid in communicating shared data and measurement practices that can be used to
evaluate the implementation the CI model. The gap analysis model was used as a guide to help
evaluate communication practices by identifying whether a problem existed and was caused by
lack of knowledge, motivation or culture. Goals and measurement indicators will then guide
HCG in their goal of improving communication of shared data and measurement practices. In
addition, the use Kirkpatrick’s (2016) four levels of evaluation provided an implementation and
evaluation framework to this study. The research questions that guided the evaluation study
were:
1. What are the KMO factors that challenge HPS field staff in their communication of
shared data and measurement practices of one-hundred percent comprehensive
information required by Executive Leadership?
2. What are the KMO opportunities that exist to facilitate HPS field staff’s ability to
communicate shared data and measurement practices of one-hundred percent
comprehensive information required by Executive Leadership?
Knowledge and Skills.
In order for HPS to communicate shared data with community partners and Executive
leaders, they need to possess the knowledge and skills when of the role of a health promotion
worker, organizational goals initiatives, the roles of backbone organizations and the shared
health data and measurement practices, or factual, procedural, conceptual and metacognitive
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 45
knowledge. The role of the Health Promotion Staff, or also referred to in the literature as a
Community Health Workers (CHW), is relatively new profession, as the first pieces of literature
published account for them as recent as sixty year ago. This may be why there is less literature
on their role, and potentially some ambiguity around the expectations of performance goals
(Love et al., 1997). As described by some, the primary role of a CHW or HPS has been to
improve engagement between communities and the US healthcare system through health
promotion practices (Balcazar et al., 2011). They oftentimes work as frontline healthcare
professionals, sometimes indigenous to the communities they serve (Love et al., 1997). Even
when they are not native to their region, they more often than not serve geopolitically challenged
populations or other high-risk populations in the role of an educator, and translator of
information, and promoter of health information and resources. Communities that they reach
include minority populations like youth, elderly, racial or ethnically underserved, homeless, drug
users, HIV positive, single young mothers, impoverished, and those who are struggling to meet
basic health, social, financial or physical needs (Balcazar et al., 2011).
Depending on the funding stream, project or program funding their target audience and
target message, typically, health promotion workers are tasked with communicating public health
or medical messages into a way that is both understandable and meaningful to the population of
focus. Essentially the performance goal of their work is tied to their organization’s focus and
may vary considerably depending on where the HPS works, being for a non-profit organization
promoting government aid such as Medicaid eligibility, or for a business promoting employee
wellness programs. Tasked with speaking directly with community stakeholders and the people
from populations through events, HPS are involved in outreach and educational events providing
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 46
information, resources, and other opportunities to communities with the goal of improving health
outcomes (Balcazar et al., 2011).
As previously mentioned, health promotion workers often work in different health topic
areas such as disease prevention, or health promoting activities, and aim to offer community
perspectives and priorities into the process of improving health care systems. In a dynamic way,
the role of HPS works equivalently in the opposite direction– offering health care systems
perspectives to the community stakeholders people they serve. As the middle man going
between the community and the local health department within a local government, the health
promotion staff person is sometimes described as a broker of health, neither providing immediate
clinical care or services, but as the conduit of health information. Their role is to increase
awareness of public health and population health strategies. Some consider the role of health
promotion staff as valuable to connecting with hard-to-reach community members. Others
suggest that the numerous possibilities of health promotion workers as educators and teachers to
the community, with the impact to indirectly effect improved outcomes at the health care
systems level (Atun et al., 2015; Wallerstein, 2006). At times, the role of a health promotion
staff is tied to specific projects or programs, the role of a government health promotion workers
may be more of a generalist role, and often is tasked with promoting all health, housing and or
human resources available to communities. Often time health promotion and disease prevention
activities, which are the primary role of HPS, has very little large scale immediate outcomes, and
thereby relies on leading indicators to denote performance improvements (Kirkpatrick &
Kirkpatrick, 2016).
In this study, HPS must additionally have factual knowledge about what collective
impact efforts are and how this impacts their specific role, given that HCG is the backbone
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 47
organization is in this model (Kania, & Kramer, 2011). In general, collective impact is
considered the “commitment of a group of important actors from different sectors to a common
agenda for solving a specific social problem.” From a factual perspective, HPS must know what
the role what it means to have a primary role in the initiative and be tasked with communicating
shared data and measurement practices. Overall, HPS must have strong factual knowledge and
skills in order to understand unique needs of the community of focus while at the same time the
big picture of community health outcomes. From a conceptual knowledge perspective – HPS
must also understand the link to individual behavioral patterns and practices.
One specific theory that help inform health promotion staff in how to undertake their
work, or via procedural knowledge, are though community based participatory research
approaches. Community based participatory research is founded on the assumption that many of
today’s complex health problems may be studied and addressed through approaches that
emphasize collaboration with communities (Israel, Schulz, Parker, & Becker, 1998; Minkler,
Blackwell, Thompson, & Tamir, 2003). This type of research describes that there is a need for
an anthropological approach that outlines the health promotion staff’s role in laying a
foundational level of trust between outside entities and the community under study to support
community building and capacity development. Over time, the health promotion staff must
spend time in the community as an observer, learning the social networks, dynamics, and values.
Sometimes this step can take a considerable amount of time.
Next, the researcher gains trust and support of the community, and learns about the
community at a deeper level. They may better be able to understand and explore topic areas of
importance to the community with the aim of combining knowledge and action for social change
to improve community health and eliminate health disparities (Wallerstein, 2006). Relationships
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 48
are at the center of every step in the process of evaluation, and implementation of approaches.
Long term strategies are defined by a partnership between the health promotion staff, researcher
or academic role, and the community as an equal partner where co-learning, mutual benefit on
ongoing support increases the community’s participation in health promoting efforts. Over time,
communities gain respect, trust and utilize the health worker to facilitate health resources since
information comes from a knowledgeable and trained health professional (Chen, et al., 2004).
Once HPS engage with the community, they also have to understand how to communicate data
to them via open government practices (Chun et al., 2010; McDermott, 2010).
In this study, HPS must additionally have procedural knowledge about how to
accomplish the performance goals required in collective impact initiatives (Kania, & Kramer,
2011). From a procedural perspective, HPS must know regarding specific steps to communicate
shared data and measurement practices (Hanleybrown et al., 2012). For example, they need to
know who the stakeholders are that they have to communicate with, and the “rules for
engagement” as Kania & Kramer (2011) describe.
A distinguishing feature of community-based collaboration is broad community
engagement, which suggests that sustainable community-based approaches must incorporate a
unique dynamic mix of individual and community approaches. From a conceptual knowledge
level, HPS must understand that approaches like community based participatory research suggest
that strategies should incorporate the perspectives of diverse individuals and groups who
represent the concern and/or geographic area or population, and then work towards shared
interests make consensus among the prospective partners possible (Argote & Fahrenkopf, 2016;
Gebbie & Turnock, 2006). A community is less willing to positively accept or receive a
suggested solution to a problem that they do not see or agree exists, nor comply with a long-term
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 49
solution that does not align with their existing philosophies and additionally may be difficult to
implement. In contrast, when solutions are integrated with the community’s perspective in mind,
resolutions can be highly impactful and meaningful to the long-term health of a population. For
example, in a 2014 study, Cacari-Stone, Wallerstein, Garcia, & Minkler, evaluated the
application of bridging evidence with policy to help eliminate racial and ethnic health inequities.
In San Diego, California, this study evaluated the formation and adoption of policies to enhance
procedural justice, which helped to enable the active participation of traditionally marginalized
residents in the policymaking process. In addition to policies, community based participatory
research can also improve the ability to achieve improved levels of cultural sharing on topics
where there may have been cultural sensitivities. When a participant actively and candidly
engages with a researcher, it improves community participation and thereby the validity of
validity of measurement through access to high quality data which helps to uncover lay
knowledge and increased levels of sharing around sensitive topics (Minkler et al., 2003).
Knowing the importance of working in sync with communities, it becomes essential to
consider the strategic alignment of health outcomes for communities at large through conceptual
knowledge. Health promotion staff are often trained in procedural knowledge or “how to”
knowledge of the process of both obtaining and then communicate health information. In the
field of public health, procedural knowledge and skills are complemented by the conceptual
understanding of learning about health behavior changes models. Theories such as the diffusion
of innovations – the stages of adoption of a novel idea by a population, Transtheoretical model –
advancing over multiple stages, actions and adaptations over time, Social Cognitive Theory –
studying human behavior in terms of a three-way, dynamic, reciprocal model in which personal
factors, environmental influences and behavior continually interact, and Health Belief Model –
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 50
the people’s beliefs about whether they are at-risk for a disease or health problem and their
perceptions of the benefits of taking action to avoid it, influence their readiness to take action;
these help explain whether populations are likely to change their knowledge, attitudes and
behaviors (Elder, 1998; Glanz, & Bishop, 2010). Glanz, & Bishop (2010) suggest that health
behavior theories help explain why people do or do not practice health-promoting behaviors,
help identify what information is needed to design an effective intervention strategy and provide
insight into how to design a program so that it is successful. This conceptual or larger scaled
purview of knowledge helps health promotion staff to understand research studies and statistics,
and at the same time understand how to translate that information to community health
behavioral changes.
In this study, HPS must additionally have conceptual knowledge about what collective
impact efforts are and what this means for the HPS in relation to community stakeholders, people
who are served by community organizations and HCG (Kania, & Kramer, 2011). For example,
HPS must understand the relationship between their role of communicating shared data and
measurement practices with population or community level health outcomes. HPS need to
understand the balance between intentionality (that comes with the development of a common
agenda) and emergence (that unfolds through collective seeing, learning, and doing), so as to be
mindful of how to lead in communication efforts.
Those who are in community health, often spend a considerable amount of time in
community settings. This immersion helps them to serve as connectors between healthcare
consumers and providers to promote health among groups that have traditionally lacked access to
adequate care (Witmer et al., 1995). By spending so much time in community settings, this
workforce spends a lot of time working on tasks in an independent way. This is one of the key
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 51
attributions of HPS is that by working in the field, health promotion staff are better able to ensure
children, youth and families have access to health resources (Chen, et al., 2004; Vidich, &
Bensman, 1954). Given the need for HPS to promote health and wellbeing in measurable and
goal-oriented ways, HPS must know how to reflect on their roles and responsibilities in strategic
and objective ways. Additionally, they must understand the relevance of prioritizing tasks in
order to acquire, reach, and monitor goals for health data acquisition and communication using a
strategic plan. One of the key components of their role is communication with both community
stakeholders, and within their home organizations. Thereby HPS are highly reliant on mobile
work systems in order to communicate progress towards performance goals. Many of the
systems that health promotion staff utilize are mobile-based, dependent on and driven by
information. HPS must understand how to effectively use mobile based technology to regulate
their performance. Kunsting, Kempf, & Wirth (2013) suggest that metacognitive approaches
(orienting, clarifying, planning, executing, monitoring, controlling, and checking finally) can
improve learning. In their 2013 study, Kunsting et al. found that metacognitive approaches
improved simulation-based scientific discovery learning (SDL) (involves stating and testing
hypotheses in a self-regulated cycle of planning, conducting, and evaluating scientific
experiments) among 9th graders. They evaluated conceptual knowledge gains and found that the
learners retained conceptual knowledge three weeks later. Another study that looked at
developing nurses as knowledge workers in health found that reflective nursing practice or
metacognition to access, learn and understand lived experiences increased learning and that
positive dynamics between expert/novice nursing also improved the retention of learning.
Dinsmore, Alexander, & Loughlin (2008) found that approaches in metacognition, self-
regulation, and self-regulated learning are beneficial for learners to engage in and should not be
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 52
confused with one another as they vary in their application. Metacognition is “thinking about
thinking” and operationalized into four areas: metacognitive knowledge, metacognitive
experience, goals, and the activation of strategies. Self-regulation is the reciprocal determinism
of the environment on the person, mediated through behavior, whereas the self-regulated learner
is the intrinsic and identified motivation of a learner who is able to manifest and manage the
learning experience. Some argue that the goal is to develop self-regulated learners, which is
when learners can translate mental abilities into academic-related skills (Pintrich, 2000; Pintrich
& de Groot, 1990). Findings from these studies may suggest that HPS would benefit from
practicing metacognitive or self-regulated learning strategies in order to reflect on
communication strategies, and the utilization of communication tools, like the internet and
mobile laptops, and phones. Furthermore, they may increases the likelihood of worker
productivity resulting in fewer workers to support the work of administrators, professionals, and
technical staff (Gebbie & Turnock, 2006). Other such strategies to improve reflective learning
include problem-based learning (Hung, 2011), which describes learning which is initiated and
consequently driven by a need to solve poorly defined, real-world type of problems.
In sum, this section explains the knowledge and skills needed among community health
workers, or HPS. It further describes how the role of HPS is part of a public health model to
facilitate health services, resources, programs and services with underserved populations. HPS’
role is essential in advancing population health, which is more of an integrative process than
clinical work and describes the use of anthropological approaches such as Community Based
Participatory Research, which have demonstrated improvements in health outcomes. Literature
describes that it is important for HPS to understand their role in the conceptual process of leading
health behavior change and the ability to self-reflect on strategies. In the subsequent section, this
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 53
paper will explore the role of motivation among public health workers, how their self-efficacy,
mood, and values impacts community health, health systems and HCG. Additionally, it will
discuss the role of health promotion and the importance of communicating shared health data and
measurement practices.
Motivation
Research shows that motivation, or the role of mental effort, active choice and
persistence play almost an equal part in human performance (Clark & Estes, 2008). The
following section will explore potential causes of high and low motivation (or evidence of high
and low motivation). Theories of human motivation and their applications among HPS field
staff, to specifically describe, explain, and predict motivation will be listed. Motivation theories
will be compared and help determine how this relates to the literature on the public health
professional workforce. Literature on intrinsic and extrinsic rewards, value, mood, self-efficacy,
goal orientation, attribution and expectancy-value theory will be incorporated in this
examination.
Literature has shown that high levels of affective influences, or mood and value can
contribute strongly to goal-oriented cognition and behavior. Published studies examining the
role of HPS describe that generally health promotion workers feel positive that the health data
and measurement practices that they communicate, feel that their work is valuable, and express
high levels of intrinsic goal orientation. This would suggest that HPS have high levels of
motivation to solve problems. One such example of the high levels of mood and value that HPS
place on their work is the compensatory rates (Godshalk & Sosik, 2003; Rosenthal et al., 2011).
Health promotion workers make relatively meager salaries in comparison to other health
professions such as epidemiologists, biostaticians, environmental health workers, physicians or
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 54
clinical providers, earning on average between $25,000 and $50,000 a year, compared to those
others who earn over $75,00 or for clinical care providers who earn well over $100,000 a year
(Bureau of Labor Statistics, 2018; Rosenthal et al., 2011). Similar to other educational and
service-oriented professions such as teachers, and social workers, public health workers report to
pursue careers this field because they feel high levels of intrinsic motivation, integration or value
assigned to the purpose of their work, reflecting congruence between the self and values and
goals (Deci, Koestner, & Ryan, 1999). As opposed to extrinsically motivated workers, which are
externally regulated or controlled by others or by external constraints such as rewards. As such,
community health workers often display high levels of commitment towards identifying
community problems, developing innovative solutions, and translating them into practice
(Roussos & Fawcett, 2000). Some suggest that given their motivation and access, they have an
untapped capacity to health improve population health outcomes (Witmer et al., 1995). Health
promotion staff have the ability to improve health outcomes by (1) increasing access to health
care, (2) improving the quality of care, (3) reducing the cost of care, and (4) improving
community relationships. In a 2003 study, Godshalk, & Sosik, describe that the role of learning
goal orientation in mentoring relationships were positively related, suggesting that proteges who
possessed high levels of learning goal orientation similar to their mentor were associated with the
highest levels of psychosocial support. This suggests by aligning mentors with similar protégés,
the novice learners have the ability to establish higher levels of career development, idealized
influence, enacted managerial aspirations, desired managerial aspirations, and career satisfaction.
In addition to valuing their work, HPS, must be confident using evidence-based practices
to communicate comprehensive shared health data and measurement practices required by
Executives Leadership. In his seminal work, Bandura (1977) describes self-efficacy as a
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 55
unifying theory of behavioral change. Bandura discusses the impact that expectations of
personal efficacy are derived from principal sources of information including, performance
accomplishments, vicarious experience, verbal persuasion, and physiological states all of which
correlate with self-efficacy. Studies on self-efficacy describe its predictive ability for such
outcomes as varied as: academic achievements, behaviors such as smoking cessation, correlation
with athletic performances, professional decisions, assertiveness, coping with feared events,
recovery from heart attack, and sales performance (Bandura, 1977). According to another author
on the topic of self-efficacy, Schunk (1991) felt that people who have a low sense of self efficacy
for performing a task are more likely to avoid it compared to those with high efficacious levels,
which have a tendency to work harder and persist for longer periods of time. In order to build
higher levels of self-efficacy, learners have to accomplish tasks through personal and
observational experiences, as failing at tasks lowers self-efficacy. This would suggest that high
levels of self-efficacy by HPS to be able to communicate shared data and measurement practices
within the CI model in order to spark higher levels of motivation to attempt and persist in this
endeavor.
Attribution is the process by which individuals explain the causes of behavior and events
and the development of models to explain these processes is called attribution theory (Weiner,
1972). In this study, HPS need to understand that their roles and responsibilities impact of
communicating data and shared measurement practices impact the health and wellbeing of
community residents and therefore impact the health outcomes of populations served. In this
study, HPS are often in the field, working with multiple stakeholders in dynamic work situations
including and it is important for them to arrive at causal explanations for event based on their
work, to motivate their performance efforts (Weiner, 1972). Weiner suggests that causal
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 56
attributions such as accolades, acknowledgement, punishment or shame influence the likelihood
of undertaking achievement activities, the intensity of work at these activities and the degree of
persistence in the face of failure contribute to efforts. HPS must feel that the comprehensive and
successful communication of comprehensive shared health data and measurement practices
directly relates to the health outcomes of populations served. On a related issue of motivation,
HPS must also believe that they have effective or control over how to successfully complete a
task within the CI model. These expectancies may be learned through model, cultural or
specifically taught. The HPS may then attribute importance, novelty or expectations based on
beliefs regardless of accuracy. Expectancy values may not accurately predict success or
effectiveness however they do contribute to motivation to engage in the task. In a 2012 study,
Johnson & Sinatra used different types of task values including: utility value, attainment value,
intrinsic value, and cost to gauge engage in the learning task (Johnson & Sinatra, 2012). The
found that participants adopted an approach in their reading task in a manner with the task value
being induced, suggesting that their competence related beliefs and values predicted different
kinds of outcomes (Wigfield & Eccles, 2000).
The CANE model of work motivation describes a two-stage model of commitment and
necessary mental effort (Clark, 1999). This model suggested that work motivation first
influences commitment to persist at a specific work goal and then determines the amount of
mental effort required to achieve the goal. Commitment is hypothesized to relate to goal value,
emotions and personal agency suggesting that completing a task requires a desire participate.
The way that this may potentially be interpreted in the current study is that HPS field staff need
to be motivated to communicate shared data and measurement practices. Following the Clark &
Estes (2008) framework, research shows that when knowledge and motivation are not the cause
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 57
for performance gaps, organizational components are likely the cause. The role that
organizations play is examined in the next section.
Organization and Culture
Organizational factors such as resources, federal, state and local policies and cultural
models contribute to the potential to communicate comprehensive shared health data and
measurement practices. Multi-sector collaborations are one of the key approaches to addressing
public health topics. Foundational works on ecological models of health behavior change (Sallis
et al., 2008) describe that there are multiple levels including interpersonal, intrapersonal,
community, organizational and policy levels of influence on specific health behaviors. What
they describe is the potential that ecological approaches to public health problems can extend
identifying that the people don’t operate in domains, instead the “nature of people’s transactions
with their physical and sociocultural surroundings, that is, environments” are the holistic
environments that health exists (Bronfenbrenner, 1992; Jacob, Allen, Ahrendt, & Brownson,
2017; Stokols, 1992). Further reading of Stokols (1996) explored social ecological theory and
guidelines for community health promotion. In this text, Stokols outlines that behavior change,
environmental enhancements, and social ecological models are the core principles of social
ecological theory used to derive practical guidelines for designing and evaluating community
health promotion programs. He argues that most all of public health challenges to influence
health behavior changes, are “too complex to be understood adequately from a single level of
analysis, and instead require a much more comprehensive approach that is ecological and
psychological, organizational, cultural, community, and regulatory from the society.” Stokols
(1996) furthers that environmental enhancement strategies have long-term effects; but when
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 58
activities are undertaken in conjunction with health behavior change and lifestyle change
programs the results are “even greater.”
Fast forwarding twenty years and researchers are still arguing the benefits of systems
wide changes. For example, in 2014, Davies et al. suggested that the new wave in public health
improvements are conditional on a health-promotion across a community or in a societal context.
They emphasize that incorporating cultural models into the approach is essential to shifting
cultural norms, environmental practices and shifts in policies. They characterized shifts where
the society promotes a culture in which healthy behaviors are the norm and in which the
institutional, social, and physical environment support this mindset. Two years later, Martin et
al. (2016) describe very similar thinking about shifting health, suggesting that in order to
establish long lasting health improvements - strengthening the integration of health services and
systems is worth exploring. They further that in order to coordinate and combine strengths
across sectors, the traditional roles of individual attempts are not as effective, as the more diverse
approaches. Aligning stakeholders from multiple levels and across diverse interests helps
maneuver populations towards improved health and well-being (Marin et al, 2014). Just as
Davies et al. (2014) explained, achievements in improved health outcomes requires a “positive,
holistic, eclectic, and collaborative effort, involving a broad range of stakeholders.”
Others agree that when policies and processes align, health outcomes improve across all
inhabitants. In 2015, Atun et al., evaluated Latin America health-system reforms, which
produced a distinct approach to universal health coverage, underpinned by the principles of
equity, solidarity, and collective action to overcome social inequalities. They found that shifting
policies and regulations, managing financing and managing resources as well as delivering
health that was both clinical and population focused impacted the health status of residents.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 59
They shifted ambiguous statements into clear objectives that resulted in improved equity,
efficiency, and effectiveness in their approaches. Davies and colleagues suggest that amending
policies, procedures and processes allows for the maximization of the value of health and
incentives for healthy behavior; promotion of healthy choices as default; and minimization of
factors that create a culture and environment which promote unhealthy behavior (2014).
Furthermore, integrative and inclusive health systems that combine efforts of medical
care, public health, and social services are demonstrating greater health improvements in
preventing disease and improving health outcomes (Martin et al., 2016). At its conception, their
model suggests that combining efforts, or shifting cultural models will produce a more effective,
equitable, higher-value whole that maximizes the production of health and well-being for all
individuals. Shifting cultural models to look at access, balance and integration, and consumer
experience and quality mean that person centered care, or in the case of this study, HPS bringing
tailored approaches to communities. Van Wave, Scutchfield, & Honoré (2010) explored
advances in public health systems and highlighted network analysis and public health practice–
based research networks to understand of how systems and infrastructure influence population
health. They looked at published systems research in areas such as public health finance,
performance standards, infrastructure measurement, information systems, and health system
capacity. The found that engaging state and local health practitioners in systems research at
every level of the public health system is effective and practical. Improving health systems
research is recognized as essential to improving health system performance.
In this study, the CI model was adopted to similarly shift cultural models, so as to be
more inclusive and collaborative than previous approaches to health delivery. According to
Kania & Kramer (2013), allocating appropriate resources, materials and supplies are key
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 60
components to the success of CI initiatives. However, they suggest that these appropriate
resources are not housed in one sector, but often are considered complementary and compounded
in their benefits when working in unison with partners. They argue that multi-sector approaches
are more adept at solving complex social problems compared to singular, or siloed efforts. They
further that social problems require systems wide changes in cultural models, policies,
procedures and processes as well including sharing a common agenda or purpose, sharing
processes or procedures via measurements, participating in mutually reinforcing activities which
indicate that cultural models must in some ways align, undertaking continuous communications
or infrastructural and again cultural setting changes, and having a key leader backbone support or
again shifting cultural components.
In 2015, Roberto et al., described an obesity prevention effort to advance policies and
environmental changes by emphasizing the reciprocal nature of the interaction between the
environment and the individual through regulatory actions in South Australia. They adopted a
“health-in-all-policies” approach, which was a government objective for a healthy population by
combining health and wellbeing as a key component of policy development across all sectors.
Some of the policies they adopted included prevention approach in disadvantaged areas through
local governments including: demanding that new housing and commercial developments adhere
to activity or transit-oriented development guidelines, increase physical activity access through
bike lanes, green space, complete streets (designed for all users, including pedestrians, bicyclists,
motorists, and public transit vehicles), safe routes to school, or slow-speed zones.
Literature on health informatics suggests that the relationship between data and
information can be transformed into the creation of knowledge that is applied to make decisions
and solve problems through methods employed in informatics (Dalrymple, 2011; Eng, 2004).
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 61
Friede, Blum, & McDonald (1995) discussed then emerging opportunities in the field of public
health informatics as an opportunity to bridge the needs of practitioners and researchers. Health
informatics then and today is seen as a way to connect systems with reliable, timely information
to make information-driven decisions by enabling the public health community to communicate
more effectively via human-to-human, human-to-machine; and machine-to-machine processes.
Combining public health informatics with systems wide approaches to population health in this
study have the potential to improve practices of the flow of information. In some instances,
health information organizations have been established to help facilitate integrated data sharing
across numerous stakeholders also called health information exchanges (Shapiro, Mostashari,
Hripcsak, Soulakis, & Kuperman, 2011).
One way to address improved health is to use health informatics. Nearly a decade ago, a
project that was adopted in 2010 by Chun and colleagues, called Government 2.0, adopted the
use of interactive approaches to allow the public to access existing and new information, to
communicate and to perform transaction-related interactions with stakeholders through channels
including the internet. This and subsequent approaches have modernized the way that the
government interacts with the public, which have pushed information to be easier to access
related to resources, programs and services, afforded higher quality interactions between
consumers and the government, and improved the ability to access information conveniently
(McDermott, 2010). McDermott furthered as established by President Obama issued a
Memorandum on Transparency and Open Government calling on his administration to develop
recommendations that would “establish a system of transparency, public participation, and
collaboration” called an “Open Government Directive.” (Ubaldi, 2013). McDermott and Ubaldi
describe that when data is relevant, easily accessible, usable and reusable by all is becomes
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 62
useful to public and allow for continuous improvement. Ubaldi suggests that enhanced data
accessibility can enable higher collaboration within governments, as well as between government
agencies and the wider society, including the private sector, civil society organizations and
citizens (2013). In sum, this section explains that HPS must have policies, programs, and
services in place to support their roles and responsibilities. Both internal policies within the
Healthy County Government, and external policies that impact the region where people live
contribute to and influence physical, social, and economic environments. They also create
opportunities and incentives through policies and procedures that influence local resident’s
behaviors.
Overall, HPS must embody an organizational model that encompasses values and
behaviors that contribute to the unique social and psychological environment of the organization.
This is reflected by collective values, beliefs, and principles or organizational members and is a
produce of such factors as history, policy, market, technology, strategy, education, management
style, nation and climate. The cultural model includes the organization’s vision, values, norms,
systems, language, assumptions, environment, location, beliefs and habits. The cultural models
will contribute to the successful acquisition and communication of health data among HPS.
Moreover, HPS field staff need to have the organizational support for their efforts to
communicate health data and measurement practices to Executive Leadership within the HCG.
Research shows that when knowledge and motivation are not the cause for performance gaps,
organizational components are likely the cause (Clark & Estes, 2008).
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 63
CHAPTER THREE: METHODOLOGY
Purpose of the Project and Questions
There is a need for comprehensive communication of shared health data and
measurement practices for optimum decision-making in Local Health Departments (O'carroll et
al., 1998). Executive Leadership are expected to create proactive approaches that improve the
health and well-being of the people they serve. As such, it is essential that executives have one-
hundred percent of the information needed to make these necessary decisions (Frieden, 2010).
Moreover, public health executives need to have comprehensive information about building
better delivery systems, supporting positive choices and pursuing policy and environmental
changes in a timely manner. If not, Executive Leadership run the risk of being less prepared to
make informed decisions to develop and implement strategic, integrative, effective and proactive
approaches (Birkhead, Klompas, & Shah, 2015; Witmer et al., 1995). The purpose of this study
was to examine the Knowledge, Motivation, and Organizational (KMO) factors that influence
the ability of Health Promotion Staff (HPS) to achieve their role as a participant in Collective
Impact (CI) model. As the backbone organization, Healthy County Government is responsible
for communicating shared data and measurement practices. The two questions that guided this
GAP analysis evaluation study were:
1. What are the KMO factors that challenge HPS field staff in their communication of
shared health data and measurement practices of one-hundred percent comprehensive
information required by Executive Leadership?
2. What are the KMO opportunities that exist to facilitate HPS field staff’s ability to
communicate shared health data and measurement practices of one-hundred percent
comprehensive information required by Executive Leadership?
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 64
Conceptual and Methodological Framework
The conceptual and methodological framework that guided this study was the Clark and
Estes (2008) systemic problem-solving approach, which aims to improve the performance and
achievement of organizational goals through evaluating KMO factors. The Clark and Estes’
model first identifies an organizational achievement goal and assesses the current status against
the stated goal. This is called the gap. The next step is to evaluate the challenges, which prevent
the achievement of the organizational goal using the KMO framework. This step is completed
through data collection and a review of the literature. The framework also emphasizes
implementation and evaluation strategies to ensure success of the recommendations. This study
emphasized the evaluation component of this framework, as such, this study completed an
evaluation of existing communication strategies of shared health data and measurement practices
among Health Promotion Staff (HPS). At completion of this study, Executive Leadership gained
a more comprehensive understanding of the current status of the health data of local populations.
The study provided Executive Leaders with more information about the challenges of achieving
one-hundred percent, or comprehensive information of shared health data and measurement
practice; and helped to identify strategies for integrative and effective communication to
ultimately increase responsiveness to local needs, better understand the population served and
better address community needs across the County.
HPS shared their knowledge and skills to communicate shared health data measurement
practices that leadership requires, the motivation (value and confidence) to communicate data to
leadership, and the organizational factors that challenge and support their efforts including
resources, cultural models, policies and procedures. The methodology in this study included
collecting data that was qualitative and quantitative in nature to capture this information in a
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 65
comprehensive manner. Figure 1 demonstrates the gap analysis as a model to help organizations
identify goals with the purpose of gaining a deeper understanding of any possible causes that
could impede the organization from achieving its goal. A number of key steps are demonstrated,
which highlight potential breakdown in the process of goal attainment to understand why goals
are not being achieved.
Figure 1. The gap analysis process, a comparison of the current organizational performance and
the desired performance.
In Chapter Two, the review of literature explored the KMO factors of HPS. The review
identified strategies for sharing health data and measurement practices across studies revealing
clear, continuous, objective and effective strategies. Examining the literature helped to provide a
basis to discuss how the communication of shared health data and measurement practices
impacts decision making processes within HCG. In Chapter Three, the methods for the study are
described and demonstrate the shared health data and measurement practices communication
strategies among HPS through an online questionnaire and subsequent face-to-face interviews
during the fall of 2017.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 66
Up to this point in time of this study, it had been unclear what KMO factors challenge or
reveal opportunities to facilitate the communication of shared health data and measurement
practices to Executive Leadership (Glanz, & Bishop, 1988). The purpose of this chapter is to
discuss how this study measured the KMO components that support and challenge HPS field
workers as they communicate health data.
Knowledge Assessment.
According to Clark and Estes (2008), performance challenges can sometimes be
attributed to lack of knowledge and/or skills. Essentially this means that certain individuals may
not have the specific knowledge and skills to accomplish a certain task or goal. For the purpose
of this project, lack of knowledge and/or skills may be attributed to HPS not knowing what or
how to communicate comprehensive shared health data and measurement practices. When HPS
lack knowledge about health data needed, organizational goals or performance goals, they may
not know what health data and measurement practices to collect, why to collect it, how to
respond proactively to changes in it, or how to reflect on their progress on accomplishing goals.
In these examples, specific knowledge and skill deficits may exist, which prevent HPS from
communicating comprehensive shared health data and prevent effective decision-making by
Executive Leadership.
According to Krathwohl (2002), there are four different types of knowledge: factual,
procedural, conceptual, and metacognitive. Factual knowledge is knowledge of discrete bits or
pieces of information, including elements and details. An example of factual knowledge refers
to being able to identify the various approaches for communicating comprehensive shared health
data and measurement practices to Executive Leadership. Conceptual knowledge considers more
abstract information and consists of categories, schema, theories, ideas, abstract terms and
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 67
principles. In this project, conceptual knowledge was explored through HPS’s ability to
articulate performance goals through following theories: Social Ecological Model, Life Course
Theory, and Collective Impact Model. Procedural knowledge involves the knowledge of “how-
to” do something. An example of procedural knowledge involves knowledge of how to
complete steps in a sequence, or a criterion for making decisions within contexts and domains,
e.g. if x then y. The fourth type of knowledge is metacognitive, which refers to one’s ability to
reflect or be aware of their own cognition. An example of metacognition in this project is when
HPS reflected on the how the information that they communicate is transferred from immediate
line level managers upwards to Executive Leadership; or when HPS reflect on the strengths and
limitations of the communication processes. See Table 2 for measures of knowledge, including
how an item was evaluated via individual interviews and individual surveys.
Table 2
Summary of Knowledge Influences and Method of Assessment
Assumed Knowledge
Influences
Survey Items Interview Items
Declarative Factual (terms,
facts, concepts)
HPS can identify the goals of
Collective Impact models.
Within the Collective Impact
models, what do you think the
goals of a backbone
organizations are? (Mark all
that apply):
• A centralized in
infrastructure (backbone
organization)
• A dedicated staff
• A common agenda
• Shared data and
measurement practices
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 68
• Continuous
communication
• Mutually reinforcing
activities among all
participants.
HPS can identify the goals of
the backbone organization in
CI model initiatives.
What do you think the goals of
Collective Impact initiatives
are? (Mark all that apply):
• Guiding the CI vision and
strategy.
• Supporting the aligned
activities.
• Communicating shared
data and measurement
practices.
• Building public will.
• Advancing policy.
• Mobilizing funding efforts.
Please describe the role of
the Healthy County
Government as a
backbone organization in
the design and
implementation of the CI
initiative.
As a backbone
organization, the Healthy
County Government is
tasked with facilitating
communicating shared
data and measurement
practices with partnering
organizations. What is
your role in
communicating shared
data and measurement
practices with partnering
organizations? (PROBE:
How has your time at the
County impacted your
understanding of this
task? How has your time
in your present position
impacted your
understanding of this
task?)
HPS can identify what shared
data and measurement
practices are necessary to
learn how community
partners are supporting
positive choices.
In order to communicate
shared data and measurement
practices with community
partners to support positive
choices, you share... (Mark all
that apply):
• Past, current and future
interventions/programs
• Communication practices
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 69
• Shifts in organizational or
performance goals
• Shifts in administration
systems
• Shifts in technology and
IT
• Shifts in production
system
HPS can identify what shared
data and measurement
practices are necessary
pursuing policy and
environmental changes.
In order to share data and
measurement practices with
community partners to pursue
policy and environmental
changes, what kinds of
information do you
share/communicate (Mark all
that apply):
• Efforts adopting policies
and environmental
procedures/regulations
• CI initiative
• Efforts to convene multi-
sector stakeholders
• CI initiative areas of
influence
• CI initiative indicators
• Shift in external
guidelines/policies
• Policy enforcement
approaches
• Shift in internal
guidelines/policies
• Participate in health
education events
• Shift in information
systems
HPS can identify what shared
data and measurement
practices is related to how
community partners are
building better delivery
systems among community
partners.
In order to communicate
shared data and measurement
practices with community
partners to build better
delivery systems, you share...
(Mark all that apply):
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 70
• Decision making practices
(shared and internal
practices)
• Electronic record keeping
practices
• Information about formal
and informal partners
• Whole person care models
• Communication practices
Declarative Conceptual
HPS can articulate various
approaches in connecting
with CI partnering
organizations.
Which of the CI initiative
approaches (below) help you
communicate shared data and
measurement practices? (Mark
all that apply)
• Building better delivery
systems
• Supporting positive
choices
• Pursuing policy and
environmental changes
HPS can articulate the
significance of their role to
communicate shared data and
measurement practices.
In order to facilitate the CI
initiative, my role within the
Healthy County Government
is to communicate shared
health data shared and
measurement practices
because, it (Mark all that
apply):
• Delivers accurate,
accessible, and actionable
health information
• Provides new
opportunities to connect
with diverse populations.
• Supports shared decision-
making
• Builds social support
networks
• Enables fast and informed
responses to health issues
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 71
HPS can articulate why
collecting the data and
measurement practices in a
systematic way is related to
the ongoing progress of the
CI model initiative.
Please indicate which answer
choice(s) below best
describe(s) why collecting
shared data and measurement
practices in a systematic way
is related to the ongoing
progress of the CI initiative.
Collecting shared data and
measurement practices ...
• Improves accountability
among all participants
• Supports the exchange of
work between partnerships
(new and evolving ones)
• Improves the exchange of
knowledge
• Enables effective decision-
making
• Facilitates integrative and
diverse organizational
practices
• Improves the facilitation
towards whole person care
• None of the above
HPS can articulate the
significance of
communicating shared data
and measurement practices.
What does it mean to
communicate shared data and
measurement practices?
• Easily accessible (common
language, easily
measurable)
• Dissemination is effective
• Contributes to outcomes
• Informs decisions
• Dynamic exchange
• Accurate
• Timely
• Everyone has a unique role
Procedural
HPS know how to
communicate shared data and
measurement practices in
alignment with the CI
In order to align your work
with the CI initiative, please
indicate how you connect with
What steps do you take to
communicate shared data
and measurement practice
to Executive Leadership?
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 72
initiative in their work with
community partners.
partner organizations (Mark
all that apply):
• Attend meetings such as
community collaboratives,
and advocacy groups with
partner organizations
• Exchange emails with
partner organizations
• Participate in education
and outreach efforts with
partner organizations
• Meet with
individual/groups in
scheduled one-on-one
meetings with partner
organizations
• Schedule routine telephone
calls with partner
organizations
(probe: what works well,
what could be improved)
Metacognitive
HPS need to self-reflect on
the effectiveness of their own
communication strategies.
As HPS, I self-reflect by…
(Mark all that apply):
• Think about the
effectiveness of my
strategies
• Evaluate my priorities
• Monitor my
communication
approaches
• Consider options when
balancing the needs of
HCG and community
partners
• All of the above
• None of the above
How do you evaluate the
effectiveness of your
communication
strategies? (PROBE: How
do you monitor a need for
change in your
communication?)
Individual HPS develop a
strategic plan demonstrating
how they will accomplish
goals of community health
workers as public health
professionals (i.e. how to
acquire, reach, and monitor
Please explain how people in
my role should report
comprehensive shared health
data and measurement
practices to supervisory staff
(open space for each
question):
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 73
goals for health data
acquisition and
communication).
• Efficiency: What would
make this process
efficient? Open space:
• Accuracy: How would you
check your information for
accuracy before you report
it?
• Frequency - How often per
day - who and how, per
week - who and how, and
per month who and how?
• Compliance - What rules
would you follow?
Motivation Assessment
According to Clark and Estes (2008), performance gaps may also be attributed to lack of
motivation. For example, Clark and Estes suggest that there are three indicators of motivational
processes that impact performance - active choice, persistence and mental effort. Active choice
is the active pursuit of a goal. Indicators of problems with active choice are demonstrated when
a solution is chosen but not implemented, thus, there is intention but no action. Persistence is the
continued pursuit of a goal despite distractions. Indicators of problems are exemplified when a
task has been started but the person is distracted by other goals and interests, therefore, does not
devote time to pursuing the goal. Mental effort is described as devoting energy to achieve a goal
and is influenced by confidence. Indicators of mental effort problems arise when people use
inaccurate or inappropriate but familiar knowledge to solve a new problem that requires a new
approach, making mistakes and projecting responsibilities externally. Optimal performance
towards a goal is contingent upon these three facets of motivation. Studies have demonstrated
that choice or active engagement and persistence increase when there is increased value and self-
efficacy (Bandura, 1997). When learners value what they are learning, they are more likely to
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 74
choose to get involved and persist over time. When individuals feel that they have the ability or
efficacy to learn and apply information, they are more likely to persist and chose to participate
(Clark, 1999). Difficulty levels of a task can influence mental efforts and when learning is too
challenging or too easy, mental efforts decreases. Moderately challenging learning goals and
tasks tend to increase mental efforts (Bandura, 1977). Mental efforts can be measured by asking
one about their value or choice to achieve a work goal, positive emotion to commit or persist
towards a goal, or the personal agency or the self-efficacy to achieve the goal (Clark, 1999).
Mayer (2011) describes that lack mental effort can be manifested by not fully
comprehending what information needs to be communicated, or that an action necessitates
overcoming too many barriers or too much mental energy or effort. In this study, lack of mental
effort manifested when HPS use inaccurate or inappropriate but familiar knowledge to solve new
problems that may require new approaches. Another way it is manifested is when HPS made
mistakes and attribute them to external issues. HPS may have perceived the tasks related to
communicating shared health data and measurement practices as too difficult and this may have
impacted their beliefs about the amount of control they have over this problem and the goals. In
this study, mental effort was explored through determining how much value they see in the
process of communicating comprehensive shared health data and measurement practices. HPS
may not realize that communicating this kind of information helps inform Executive Leadership
decision-making, which thereby helps to improve the health and well-being of County residents.
In 2011, Mayer (2011) describes several factors of motivation, including: interest, self-efficacy,
attribution and goal orientation, which contribute to motivation. Interest includes the value or
importance of doing a particular task of achieving a goal. Self-efficacy is the confidence that
stakeholders have in their ability to complete a task or goal (mastery or performance), and their
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 75
belief that their actions will have long-term impacts on the performance goals. Goals can be
concrete, current or challenging. Emotions or mood can contribute to success in self-efficacy
areas. Attribution is when stakeholders attribute outcomes to the result of something external
from themselves. An example in this study, is when HPS believe that external factors contribute
to challenges or opportunities that impact their capacity to reach performance goals. Table 3
provides a summary of influences and methods of assessment among HPS in their motivation to
communicate shared health data and measurement practices, including how it was evaluated
during individual interviews and individual surveys.
Table 3
Summary of Motivation Influences and Method of Assessment
Assumed Motivation
Influences
Survey Items Interview Items
Value
HPS see value in reporting
comprehensive shared
health data and
measurement practices to
Executive Leadership for
decision making.
How much do you agree with
the following statement, “As a
Health Promotion Staff, I know
that the shared data I report is a
valuable part of the decision
making among Executive
Leadership?”
• Strongly disagree
• Somewhat disagree
• Somewhat agree
• Strongly agree
What is the value in
communicating shared
data and measurement
practices with community
partners (PROBE: How
does this differ when the
aim is to build better
delivery systems-related,
support positive choices
or pursue policy and
environmental changes)?
Self-Efficacy
HPS needs to be confident
connecting with partnering
organizations to share the CI
initiative.
How much confidence do you
have connecting with
community partners to
implement the CI initiative?
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 76
• No confidence at all
• Not very much confidence
• A little confidence
• A lot of confidence
HPS needs to be confident
communicating shared data
and measurement practices
with community partners.
How much confidence do you
have connecting with
community partners to
communicate shared data and
measurement practices through
the CI initiative?
• No confidence at all
• Not very much confidence
• A little confidence
• A lot of confidence
HPS needs to be confident
communicating shared data
and measurement practices
with supervisors.
How much confidence do you
have reporting shared data and
measurement practices to
supervisors (frequency,
compliance, accuracy,
efficiency)?
• No confidence at all
• Not very much confidence
• A little confidence
• A lot of confidence
Mood
HPS need to feel positive
about achieving the goals of
the backbone organizations
in CI model initiatives.
Please indicate agreement with
the following statement, "As a
Health Promotion Staff, my
organization is supportive about
my role communicating shared
data and measurement practices
with community organizations."
• Strongly Disagree
• Somewhat Disagree
• Somewhat Agree
• Strongly Agree
HPS need to feel positive
about internal levels of
motivation, expectations
Please indicate agreement with
the following statement, “My
performance as a Health
Promotion Staff is impacted by
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 77
and engagement with
community partners.
my motivation, expectations
and engagement with
community partners.”
• Strongly Disagree
• Somewhat Disagree
• Somewhat Agree
• Strongly Agree
Goal Orientation
The goals of HPS in
communicating shared
health data measurement
practices are mastery
oriented (mastery -
individual improvement,
learning, and
demonstrations of progress
not performance - want to
do the bare minimum and
show to the administration
that they are meeting the
requirements of data
management).
I communicate shared health
data and measurement practices
because I want to (choose one):
• Individual improvement,
and learning.
• Demonstrations of progress.
Describe your goals for
communicating shared
health data and
measurement practices?
(PROBE: How do your
goals influence your
motivation?)
Attribution Theory
HPS do not attribute
challenges/opportunities to
communicate shared health
data and measurement
practices to influences
outside of their control.
Please indicate agreement with
the following statement, “There
are reasons beyond my control
that contribute to my capacity
to communicate shared data and
measurement practices.”
• Strongly Disagree
• Somewhat Disagree
• Somewhat Agree
• Strongly Agree
Can we play the devil’s
advocate and pretend that
factors are ideal to
support your ability to
communicate shared data
and measurement
practices with community
partners, what else would
still prevent you from
communicating shared
data and measurement
practices with community
partners, including:
developments in
communities, factors
within HCG, such as your
peers, your management
or other leadership?
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 78
Expectancy Value
HPS need to know that their
role of communicating
shared data is well received.
How much do you agree with
the following statement, “As a
Health Promotion Staff, I feel
that I am a part of a culture
where communicating shared
health data and measurement
practices is well-received?”
Organization/Culture/Context Assessment
According to Clark and Estes (2008), performance gaps may also be attributed to
obstacles that exist within organizations or more specifically be caused by insufficient resources,
misaligned policies and procedures, or inconsistent cultural models. When individual
knowledge, skills and/or motivation factors are not causing a performance problem, there is often
a problem at the organizational level (Clark, & Estes, 2008). In the current study, for example,
there were organizational gaps that prevented HPS from communicating comprehensive shared
health data and measurement practices. These gaps were the result of inefficient policies,
procedures and lack of tools and other necessary resources, which contribute to the misalignment
of value chains and value streams.
Performance problems within an organization are also associated with the organizational
culture. An organization’s culture serves as the intangible environmental components, which
contribute to how work is accomplished, the attitudes of the staff including core beliefs, and
values of the organization itself. During the evaluation of culture, this study helped to determine
whether HPS are part of a culture in which the communicated health data is well-received and
without recourse or punishment. Table 4 lists the items that evaluated organizational influences,
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 79
including resources, policies, procedures and cultural models that may impact how HPS are able
to communicate shared health data and measurement practices.
Table 4
Summary of Organization Influences and Method of Assessment
Assumed Organization
Influences
Survey Items Interview Items
Resources (time; finances;
people)
HPS need to have access to
resources, which will enable
them to effectively
communicate
comprehensive shared health
data and measurement
practices?
Please indicate agreement with
the following statement, "As a
Health Promotion Staff, my
organization is offers resources
to support my role
communicating shared data
and measurement practices
with community
organizations."
• Strongly Disagree
• Somewhat Disagree
• Somewhat Agree
• Strongly Agree
Tell me about the
materials, supplies or
resources that your
workplace provides for
you to communicate
comprehensive shared
health data and
measurement practices.
(PROBE: What about
mentors or shadowing
opportunities? How does
this impact your
understanding of your
role?)
HPS need appropriate
professional development
and training to learn how to
communicate
comprehensive shared health
data and measurement
practices.
Please indicate agreement with
the following statement, “My
workplace provides me with
the materials, supplies and
resources to help me in my
role.”
• Strongly Disagree
• Somewhat Disagree
• Somewhat Agree
• Strongly Agree
What trainings (online,
conferences, or other in-
person trainings) you have
received related to
communicating
comprehensive shared
health data and
measurement practices?
(PROBE: How have these
impacted your role?)
Policies, Processes, &
Procedures
There are policies in place
that guide the design and
In your opinion, are there
policies in place that guide the
Ideally, what
organizational components
(e.g. policies/procedures,
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 80
implementation of the CI
model initiative.
design and implementation of
the CI initiative?
• Yes
• No
• I don’t know
cultural components, pay
structures) would allow
you to implement
communicating
comprehensive shared
health data and
measurement practices?
There are policies in place
that guide the design and
implementation of the
backbone organizations in
CI model initiatives.
Please mark the box(es) below,
if you believe there are
policies that help you design
and implement the CI
initiative.
• Policies related to
communication practices
• Policies related to
community partnerships
roles and responsibilities
• Policies related to
electronic record keeping
practices
• Policies that are addressed
in your annual
performance review
There are policies in place
that allow for HPS to work
with mentors to observe and
learn how to communicate
shared health data and
measurement practices.
Please indicate agreement with
the following statement, “My
workplace provides me with
the opportunity to work with a
mentor on the topic of
communicating shared health
data and measurement
practices.”
Cultural Models
The HPS receive incentives
such as acknowledgements
and recognition within the
organizational culture and
encourage HPS efforts to
communicate shared data
and measurement practices.
How much do you agree with
the following statement, “As a
Health Promotion Staff, I am
incentivized (with recognition,
promotion or
acknowledgement) to
communicate shared data and
measurement practices?”
• Strongly Disagree
• Somewhat Disagree
Do you feel that you have
incentives such as
acknowledgement and
recognition to
communicate
comprehensive shared
health data and
measurement practices?
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 81
• Somewhat Agree
• Strongly Agree
The HPS are allotted
adequate time to support
HPS efforts to communicate
shared data and
measurement practices.
Please indicate agreement with
the following statement, "As a
Health Promotion Staff, my
organization facilitates time
for me to communicate shared
data and measurement
practices.”
• Strongly Disagree
• Somewhat Disagree
• Somewhat Agree
• Strongly Agree
Participating Stakeholders and Sample Selection Population
There are approximately seventy health promotion specialist (HPS) staff working at
Healthy County Government. HPS work in the different regional service areas facilitating health
services and resources based on unique population needs. Some HPS work at the centralized
Healthy County Government headquarters on specific health topics and others work within
specific departments on specific programming outreach efforts. HPS interact with communities
via face-to-face interactions, preparing programs and policies, or analyzing intervention
effectiveness. HPS can work in a variety of settings including hospitals, communities,
neighborhoods, prisons, schools and workplaces. HPS can work on specific health related issues
or specialize in one area such as prescription drug abuse, smoking, healthy eating, mental health,
or sexual health. Other HPS can be focused on a specific section of the community, such as
elderly or disabled persons or an ethnic minority group.
Sampling
The stakeholder of focus for this study consisted of a purposive sample of approximately
forty percent or thirty of the seventy HPS employees from Healthy County Government, which
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 82
meets data saturation requirements for thematic findings in the qualitative analyses (Corbin, &
Strauss, 2008; Guest, Bunce, & Johnson, 2006). HPS were selected because they specialize in
health promotion CI efforts with community partners. All HPS perform professional public
health duties in community settings. HPS serve urban and rural populations to assess and
identify individual and community health needs. This position is heavily involved in conducting
outreach, including making community program presentations and collaborating with local
schools, community clinics, public health nurses and eligibility services. HPS serve three
geographic regions, and several departments within the Healthy County Government. The study
sample of HPS represent five departments from across Healthy County Government have at least
three HPS represented in the participation of this study.
Recruitment
All HPS employed within the HCG in this study were entered into a database by the
Human Resources Department and divided by the department that they represent. Next, staff
from each department were randomly selected and contacted for recruitment. See Appendix A
for Recruitment Email letter. Randomly identified participants received an email message
inviting them to participate in a brief survey and invited to follow an internet website link to
participate. At the end of the survey, there was a link to submit the survey. Following the
submission, another box replied to the respondent with an expression of gratitude for
participating and offered an additional link in case the respondent would like to participate in the
upcoming focus group. If the participant was interested they indicated one of two dates and
times they were available and provided their contact information (e.g. first and last name, work
phone and work email). This approach allowed for data triangulation so that the participants
would have been able to participate in both segments of the data collection.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 83
Instrumentation
This study utilized a sequential explanatory mixed methods approach, characterized by
the collection and analysis of quantitative data followed by a collection and analysis of
qualitative data. The purpose of this methodology was to use qualitative results to assist in
explaining and interpreting the findings of the quantitative component of the study. Mixed
methods were employed in this research design to combine qualitative and quantitative approach
to provide better interpretation than one method alone.
Survey
An online survey was administered to better understand how HPS communicate shared
health data and measurement practices through evaluating KMO factors (See Appendix B).
Surveys were selected as the first method to collect data because the anonymity of surveys
allows respondents to answer candidly compared to other survey techniques (Merriam, &
Tisdell, 2016). It is important in such a small sample size for respondents to be as open and
honest as possible in their answers and surveys are one data collection method that allows for
more honest and unambiguous responses (Merriam, & Tisdell, 2016). The survey was offered
online so that participants could complete it in their preferred manner e.g. tablet, desktop, laptop
and in a private manner.
The questionnaire was designed to first provide questions about the background of each
participant, including the duration of time that they have been HPS staff, and what types of
previous roles they have worked in, which may provide context for their current process of
health acquisition and communication strategies. Next, the survey combined a mix of structured
and unstructured questions and explore how HPS describe KMO factors that are related to their
responsibility of data acquisition and communication. An open-ended knowledge question
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 84
asked, “Please explain how you generally report comprehensive shared health data and
measurement practices. How often do you report: (open space); What rules do you follow: (open
space); How do you check for accuracy: (open space); How efficient is your process: (open
space). A closed motivation question might address, “How much do you agree with the
following statement (insert Likert scale 1-4), “As HPS field worker, I see the value in connecting
with CI partnering organizations.”
Focus Group and Individual Interviews
In the initial study design, focus groups were chosen as a secondary data collection
method. This method was selected to triangulate findings that were collected from the
aforementioned anonymous survey data collection method (Bogdan, & Biklen, 2007; Patton,
2002). Qualitative data collection is considered beneficial in studies to offer richer and more
complete understanding of the information seeking and processes of communicating shared data
and measurement practices that may not have been possible using quantitative research methods
(Lichtman, 2014; Maxwell, 2013). Qualitative method triangulation is advocated as a strategy to
achieve a greater comprehension of a phenomena, and a richness in the depth of inquiry (Patton,
2002), and as KMO factors are not easily observed (Clark, & Estes, 2008), collecting qualitative
data was essential to learning about this KMO factors that impact the communication of shared
data and measurement practices. Additionally, focus groups are helpful in order to solicit a large
amount of data on a topic in a short time and gain a deeper understanding of findings by
collecting information from numerous participants simultaneously (Merriam, & Tisdell, 2016).
Focus groups were designed using a semi-structured questionnaire format to provide
opportunities for the participants to hear other responses and make additional comments beyond
their own original survey responses (See Tables 2-4) (Merriam, & Tisdell, 2016). Literature
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 85
describes semi-structured focus groups as beneficial because they allow for more flexibly
worded questions than structured question formats. Multiple types of questions were prepared
including descriptive, devil’s advocate, interpretive and idealistic types, so as to capture a wide
range of perspectives from the participants (Merriam, & Tisdell, 2016).
While the principle investigator recruited all of the online surveyed participants to join
subsequent focus groups, only several were willing to participate. In fact, after four separate
email solicitations and three weeks of recruitment, only four participants had signed up to
participate. With such a low number of participants willing to engage in focus groups, the
principle investigator had to consider a separate option to triangulate the online quantitative data
collection. The principle investigator consulted with the dissertation committee and decided to
move forward with scheduling individual interviews instead of the previously described focus
groups. Triangulation of methods is a research strategy anticipated to contribute to a more
thorough exploration of a topic, and it was important for this researcher to utilize mixed methods
in this dissertation study. One of the distinct features of focus-groups is the group dynamics, and
the social interaction of the group provide a range of ideas and feelings on specific issues and
identify differences in perspectives. However, because focus groups require individuals to self-
disclose, and many of the study participants work with each other, some indicated mixed feelings
about participating in focus groups. Privately, some HPS admitted to the principal investigator
feeling lack of trust and comfort in the idea of sharing candid perspectives and preferred to
participate in individual interviews. HPS indicated feeling that their knowledge, motivation and
perspectives about organizational factors was in fact private and shared not wanting to discuss
personal experiences with peers, which could potentially have negative implications in the future
of their employment. Thus, the principle investigator chose to administer individual interviews
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 86
to collect details accounts of participants’ thoughts, ideas, attitudes, perceptions and knowledge
pertaining to a given phenomenon; under the assumption that participants answer stated
questions (Patton, 2002). Interviewing is a qualitative research method that is used to understand
one’s perspective, and experiences directly from the participant (Merriam, & Tisdell, 2016). In
this study, interviewing was chosen as a method to understand the process of how HPS
communicate information, which would not have been possible to solicit via observation alone
(Patton, 2002). A semi-structured interview format was selected to allow for individual
interpretation and flexibility in response options as this approach combines a set of
predetermined set of questions with particular themes and offered a number of probes with open-
ended questions that would prompt discussion (Harrell, & Bradley, 2009). Specifically, the
wording and order of questions was not determined ahead of time. This format allowed the
interviewer to respond to changes in topics, or emerging issues during the interviews (Merriam,
& Tisdell, 2016).
Data Collection
Following University of Southern California Institutional Review Board (IRB) approval
and the approval of Healthy County Government leadership, participants were solicited by email
and invited to participate in the online survey and following submission of the survey, invited to
participate in the focus group and/or individual interview. The director of human resources and
department directors were notified of the letter and provided assent for their staff to participate.
All data collection occurred between January 29, 2018 and February 28, 2018. Before
participating in the survey, participants were asked to review the overview of the study
acknowledge risks and provide assent.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 87
Online Surveys
The surveys were sent out with between January 29, 2018 and February 28, 2018. The
survey started with a short number of background questions and then move to the semi-
structured questions (see Appendix B). In total, the survey was designed to last no more than
thirty minutes.
Individual Interviews
The individual interviews were completed between March 5, 2018 and March 9, 2018.
The survey started with a short number of background questions and then move to the semi-
structured questions (see Appendix C). In total, the interview was designed to last no more than
sixty minutes.
Data Analysis
Surveys
Quantitative data were analyzed using descriptive statistics including frequency, means,
standard deviations, and medians. For all statistical tests, SPSS statistical software (version 21.2;
IBM Corporation, Armonk, NY) were used to perform statistical analyses. This data analysis
was performed to identify items that should be further explored during the subsequent focus
groups.
Trustworthiness of Data
In order to increase the trustworthiness of this study and to substantiate the information
collected, the project researcher used two data collection methods (surveys and focus groups)
and obtain quantitative and qualitative data approaches (Maxwell, 2013; Miles at al., 2013). A
second approach that ensured internal validity or credibility was member checks, or respondent
validation (Harding, 2013; Merriam, & Tisdell, 2016), and this was done in the individual
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 88
interviews. During the individual interviews, the project researcher flagged certain terms or
phrases (as the participant is speaking), so that when the participant finished with their thoughts,
the researcher asked additional questions to clarify the meaning of phrases and terms. This
approach strengthened the project researcher’s codebook and code definitions. This helped to
confirm understanding of participant descriptions and reduced subjective potential
misinterpretation or biases associated with the data collected. The project researcher also took
one approach in the interviews, which was to use negative, or discrepant case analysis, which
purposefully sought to disconfirm and challenge expectations, reflexivity or emerging findings
(Maxwell, 2013; Merriam, & Tisdell, 2016; Miles et al., 2014).
Ethics
There were some ethical concerns related to the confidentiality and anonymity of the
participant interviews. To minimize this risk to the participant, the project researcher reviewed
the USC consent form with participants prior to participation to communicate that the study was
voluntary in nature, confirm the participant’s right to privacy, discussed the notion of informed
consent and explain that findings de-identified the participants identify (Merriam, & Tisdell,
2016).
In order to demonstrate the context of the dissertation project, the project researcher
discussed the purpose of the individual interviews, surveys and the role of the dissertation within
the doctoral program. Another step that was taken to ensure ethical procedures was to articulate
that there were no desired answers or outcomes and through using empathetic neutrality, the
researcher affirmed that the participant should not attempt to appease interviewer (Patton, 2002).
Prior to the interview, participants were told that they could ask questions at any time and
decline to answer any questions. In order to maintain privacy from colleagues, interviews were
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 89
held in a private location and information was solicited in a professional manner. The participant
gave consent prior to note taking and recording. Regardless of the information disclosed during
the interview, the project researcher explained that findings would not be referenced using the
participant’s name, or any other distinguishing characteristics that would identify them. By de-
identifying the participants from being named, there was an increased likelihood that participants
would be forthcoming and honest (Patton, 2002). Coercion was avoided because special care
was taken to avoid all instances of overt or implicit threat of harm is intentionally by any one
person to another in order to obtain compliance. Under no circumstances was there [u]ndue
influence, by contrast through an offer of an excessive or inappropriate reward or other overture
in order to obtain compliance (Klitzman, 2013).
Role of Investigator
The investigator in this study, was a colleague of the HPS team, working in the Executive
Offices of Health County Government. The investigator developed the survey questionnaire and
facilitated the individual interviews.
Limitations
The study has several limitations. The first is that the researcher has limited experience
and expertise to resolving discrepancies between different types of data and interpreting data.
The second limitation is that the principal investigator of the study is in a highly visible position
in the Healthy County Government and there may have been lack of honest and truthful
communication with participants. The third is that the survey was completed by those who self-
select to participate. Therefore, results of the survey and subsequent individual interviews only
include HPS who have an interest to share their opinion and may not have included those who
were less inclined to participate. In regard to the survey, one limitation is that participants may
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 90
have briefly skimmed the questions and responded without reading the questions. The survey
was a combination of KMO questions, therefore was not a validated instrument and could have
questions that were not scientifically tested as reliable or valid. Additionally, some of the
responses obtained from the individual interviews may have been censored by the HPS in order
to make themselves look favorable. Finally, because the scope of the study is narrow in focus,
the findings reflect more of a case study and are not generalizable. It may be challenging to
extrapolate how comparable the HPS experience would be among other staff within Healthy
County Government, or even among HPS at another local health departments and public health
organizations (Merriam, & Tisdell, 2016).
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 91
CHAPTER FOUR: RESULTS AND FINDINGS
The purpose of this chapter is to describe the results and findings of knowledge,
motivation, and organizational (KMO) factors that contribute to HPS’ ability to communicate
shared data and measurement practices. The Clark and Estes (2008) Gap Analysis Model served
as the framework for the project. This model helps to identify whether the gaps are caused by a
lack of knowledge, motivation or organizational (KMO) barriers. Initially, the performance
goals are identified, and an evaluation of the current performance is assessed. Given the gap
between the desired and actual performance, solutions are proposed to minimize and ideally to
alleviate the gap. In this section, the evaluation of HPS’ current performance is discussed.
To provide context, in Chapter Three, a mixed method approach was described, which
used data to identify and validate the self-perceived causes of potential KMO gaps relevant to
Health Promotion Staff (HPS). The instruments used were designed to validate the assumed
causes. In order to increase the trustworthiness of this study, and to substantiate the information
collected, the project researcher used two data collection methods (online quantitative surveys,
and face-to-face interviews) to triangulate emerging findings (Maxwell, 2013; Miles at al.,
2014). Triangulation was used to facilitate validation of data through cross verification from the
application and combination of several research methods in the study of the same phenomenon.
In this study, triangulation of methods meant that participants were contacted to complete online
survey data then subsequent in-person interviews. The purpose of triangulating data was to help
recognize and identify inconsistencies in the data set and to provide data sets that complement
one another (Harding, 2013; Merriam, & Tisdell, 2016).
A sequential explanatory mixed method was utilized in this study to collect data in the
form of numbers, statistics, words and narrative. The purpose of this methodology is to use
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 92
qualitative results to assist in explaining and interpreting the findings of the antecedent
quantitative component of the study. The process of evaluating HPS’ ability to communicate
shared data and measurement practices and examine whether this is occurring in a
comprehensive way, which is both objective, and subjective.
Descriptive data such as frequencies, means, and standard deviations were collected to
determine participant’s attitude towards the topic in the online surveys that collective quantative
data. When a KMO construct resulted in a wide range of responses e.g. range of scores from 1,
strongly disagree to 4, strongly agree; compared to ranges of scores from 3, somewhat agree to 4,
strongly agree; or had a very high score (> 3.50), or a very low mean scores (<3.00), or were
missing many responses (>40%), then the item construct was reviewed for clarification. The
principal investigator reviewed the online survey data based and confirmed descriptive statistics.
Next, the question construct was reworded so that it would be appropriate to ask during
the in-person script interview and the item was added to the interview questionnaire script to
further understand the meaning behind this data finding. For example, participants that reported
a very high self-efficacy construct mean score, within the motivation construct (m=3.71,
st.dev=0.464, range 3.00 to 4.00), were reviewed, and added to the in-person interviews script.
During interviews, questions responses on self-efficacy validated the quantitative high scores,
because all six of the participants reiterated their confidence in communicating data and shared
measurements with both community partners and executive leadership. On the other hand,
participants who reported very low organizational, or the cultural model construct mean scores
related to receiving incentivizes for their work (m=2.46, st.dev=0.779, range 1.00 to 4.00) via the
survey, reported ambivalence or indifference related to incentives during individual interviews,
citing that although incentives were infrequent, they were not important. Specifically, during in-
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 93
person interviews, four of the six confirmed that they were not incentivized with recognition,
awards or accolades for their work. Of the four, one person felt like this was problematic and the
other three said that external incentives are not important and that they do their work with
integrity, to the best of their ability not for external validation or incentives, instead for intrinsic
motivating factors. Two said that they receive a lot of positive feedback and recognition for their
work. Therefore, given conflicting answers about the importance, or value of extrinsic
validation, the lower mean scores on the online-survey related to incentives did not confirm a
potential cause of a gap in this framework nor whether this contributes to an organization gap.
Further analysis was pursued via in-person interviews to better understand how to interpret low
mean scores and address complex responses via recommendations and solutions.
Data saturation, or the point at which no new information or themes are observed in the
data, is a tool used for ensuring that adequate and quality data are collected to support the study
(Guest et al., 2006). Guest et al. (2006) operationalized saturation and found that saturation
occurred within the first twelve interviews, although basic elements for meta themes were
present as early as six interviews. In this study, data saturation occurred at five interviews. Thus,
interviews were concluded following the sixth participant.
Both the survey and interviews helped answer the project’s question on identifying
potential gaps in motivation, knowledge and organizational factors. In the next section, the
overview of data collection is described. In Chapter Five, potential solutions to address the
validated causes of the gaps will be presented and discussed.
Participating Stakeholders
Thirty HPS participated in an online survey that helped identify whether HPS lack
knowledge, lack motivation, and/or face organizational barriers. The target population consisted
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 94
of 67 HPS (Health Services Project Coordinators, Community Health Program Specialists I & II,
Health Information Specialists I & II, Aging Program Administrator, Aging Program Specialist I
& II) from HCG employed during the 2017-2018 fiscal year. The response rate of the survey
(45%) is considered average for individual responses, which in a series of meta analyses found
that between 33% and 55% is the expected number (Baruch, & Holtom, 2008).
The online survey included five demographic questions during the beginning of the
questionnaire, and included questions pertaining to work experience for the local government,
work experience in the participant’s current work setting within Healthy County Government
(HCG), and aspirations for future career trajectory. For a breakdown of the demographic data,
see Table 5. Of the 30 survey participants, 22 of the participants have worked in the field of
public health before this role, and 20 had worked in community health promotion, while six had
worked in public health policy/government. Participants’ tenure at the HCG were distributed as
follows: one had worked at the HCG for less than one year; nine had worked at the HCG for
between one and three years; and 20 had worked at the HCG for more than three years.
Participants’ tenure in their current position were distributed as follows: seven had worked in
their current role for less than one year; 12 had worked in their current role for between one and
three years; and 11 had worked in their current role for more than three years. Lastly, the
distribution or ten-year vision for the field that they anticipate their careers moving towards was
as follows: 12 saw themselves working in the field of community health promotion, and 12 saw
themselves working in public health policy/government. When it came to prior experience to CI
initiatives 25 were aware of collective impact models but not from firsthand experience or had
read about collective impact models in academic coursework, and five had designed or
implemented a (or more than one) collective impact related project(s) or had managed a (or more
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 95
than one) collective impact related project(s). All participants in these job classifications are
required to hold a master’s degree in Public Health. The online survey contained a final item
requesting consent for participation in a focus group (See Table 5).
Table 5
Information about the HPS who Participated in the Online Survey
Partici
pant #
Prior
Experi
ence in
Public
Health
(Y/N)
Previous
Experien
ce
Commun
ity
Health
Promotio
n (Y/N)
Previous
Experience
Health
Policy/
Government
(Y/N)
Duration
working
at HCG
Duration
working
as HPS
team
10-Year
Career
Goals
Commu
nity
Health
Promoti
on
(Y/N)
10-Year
Career
Goals
Health
Policy/
Governm
ent (Y/N)
Previous
Experience
Working with
CI Initiatives:
(Aware/ Read
about or
Designed/
Implement/
Manage)
S01 Yes Yes No
1 - 3
Years
Less than
1 Year No Yes
Aware/Read
about
S02 Yes Yes No
More
than 3
Years
More than
3 Years No Yes
Aware/Read
about
S03 No Yes No
1 - 3
Years
1 - 3
Years No No
Designed/Imple
ment/Manage
S04 Yes No Yes
1 - 3
Years
Less than
1 Year No Yes
Aware/Read
about
S05 Yes Yes No
1 - 3
Years
1 - 3
Years No No
Designed/Imple
ment/Manage
S06 Yes Yes No
More
than 3
Years
More than
3 Years Yes No
Aware/Read
about
S07 Yes Yes No
More
than 3
Years
More than
3 Years Yes No
Designed/Imple
ment/Manage
S08 Yes No Yes
More
than 3
Years
1 - 3
Years Yes No
Aware/Read
about
S09 Yes Yes No
More
than 3
Years
More than
3 Years No Yes
Aware/Read
about
S010 No Yes Yes
More
than 3
Years
Less than
1 Year No Yes
Aware/Read
about
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 96
S011 No No No
1 - 3
Years
1 - 3
Years No Yes
Designed/Imple
ment/Manage
S012 No No No
More
than 3
Years
1 - 3
Years No No
Aware/Read
about
S013 No No No
More
than 3
Years
More than
3 Years No No
Aware/Read
about
S014 Yes Yes No
More
than 3
Years
More than
3 Years Yes No
Aware/Read
about
S015 Yes Yes No
More
than 3
Years
More than
3 Years Yes No
Aware/Read
about
S016 No No No
More
than 3
Years
1 - 3
Years No Yes
Aware/Read
about
S017 Yes Yes Yes
More
than 3
Years
1 - 3
Years No Yes
Aware/Read
about
S018 Yes Yes No
1 - 3
Years
1 - 3
Years Yes No
Aware/Read
about
S019 Yes No Yes
More
than 3
Years
More than
3 Years Yes No
Aware/Read
about
S020 Yes No No
1 - 3
Years
1 - 3
Years Yes No
Aware/Read
about
S021 Yes Yes No
More
than 3
Years
Less than
1 Year Yes No
Designed/Imple
ment/Manage
S022 No No No
1 - 3
Years
1 - 3
Years No No
Aware/Read
about
S023 Yes Yes Yes
More
than 3
Years
Less than
1 Year Yes No
Aware/Read
about
S024 Yes Yes No
More
than 3
Years
Less than
1 Year Yes No
Aware/Read
about
S025 Yes No No
More
than 3
Years
More than
3 Years No No
Aware/Read
about
S026 No Yes No
More
than 3
Years
More than
3 Years No No
Aware/Read
about
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 97
S027 Yes Yes No
More
than 3
Years
1 - 3
Years No Yes
Aware/Read
about
S028 Yes Yes No
Less than
1 Year
Less than
1 Year No Yes
Aware/Read
about
S029 Yes Yes No
More
than 3
Years
More than
3 Years No Yes
Aware/Read
about
S030 Yes Yes No
1 - 3
Years
1 - 3
Years Yes No
Aware/Read
about
Next, six of the aforementioned online survey respondents consented to participate in
face-to-face interviews to assess whether the root cause of non-completion was lack of
knowledge, motivation and/or organizational barriers. These interviews were conducted to
triangulate data and further validate the assumed causes (Patton, 2002) and to ensure the
emergence of metathemes, as recommended by Guest et al (2006). Of the six interviewees, five
were female and one was a male. Interviewees were offered multiple days and times before or
after work, or during a regularly scheduled lunch break that worked best for their schedules. The
individual interviews lasted approximately one hour each and a point of redundancy or data
saturation was reached after the fifth interview when no new information was forthcoming, and
interviewees began to duplicate answers (Fusch, & Ness, 2015). The setting for each of the
individual interviews was a location of most convenience for the study participant and included a
public coffee shop, vacant office space, employee cafeteria and private conference rooms. Care
was taken to provide a quiet area with ample table space, where interviewees and researcher
could have open discussion without interruption and that would be comfortable for the
interviewee.
Face-to-face interviewed participants were asked four demographic questions at the
beginning of their interview meeting. When asked where they were in the continuum of their
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 98
career, the distribution was as follows: all six felt that they were in the mid-career stage.
Participants’ tenure at the HCG were distributed as follows: one had worked at the HCG for less
than one year; three had worked at the HCG for between one and three years; and two had
worked at the HCG for more than three years. Participants’ tenure in their current position were
distributed as follows: two had worked in their current role for less than one year; three had
worked in their current role for between one and three years; and one had worked in their current
role for more than three years. Finally, participants were asked what they anticipate doing in
their career in ten years from now and their answers were distributed as follows: four anticipate
staying at HCG but anticipate growing into a new role through promotions and other career
advancement opportunities; one said that they anticipate staying in public health but is not sure
where that will lead, and one anticipated working in consulting or a for-profit role, that would
offer more a “competitive pay salary.” Interviews ranged in duration between 39 and 60
minutes. See Table 6 for interviewee demographics.
Table 6
Information about the HPS who Participated in Face-to-Face Interviewees
HPS Stage in
Your
Career
Duration
working at
HCG
Duration
working as
HPS team
10-Year
Career Goals
Minutes
of
Interview
Location
of
Interview
Minutes
“Small
Talk” Post
Interview
IO1
Mid-
Career
1 - 3
Years
Less than 1
Year Stay at HCG 60
Vacant
Office
3
IO2
Mid-
Career
1 - 3
Years
Less than 1
Year
Working in
Consulting 42
Coffee
Shop
2
IO3
Mid-
Career
More than
3 Years 1 - 3 Years
Staying at
HCG 39
Conferen
ce Room
2
IO4
Mid-
Career
1 - 3
Years 1 - 3 Years
Staying at
HCG 53
Conferen
ce Room 5
IO5
Mid-
Career
Less than
1 Year 1 - 3 Years
Staying in
Public Health 54 Cafeteria 4
IO6
Mid-
Career
More than
3 Years
More than 3
Years
Staying at
HCG 52
Conferen
ce Room 3
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 99
Due to the nature of the role of the investigator in the HCG, during the piloting of the
survey, participants discouraged the researcher from asking additional demographic questions to
encourage participation in the study, and to minimize risk of breach of confidentiality among
participants.
Data Collection
The online survey included the previously described demographic and 30 items for a total
of 36 items. Online survey data was collected from January 30 through February 28, 2018. The
survey took on average 21 minutes and 54 seconds to complete. The invitation to participate
came in the form of an email from the principal investigator’s work email address, and was sent
out on four dates: January 30, 2018, February 6, 2018, February 12, 2018 and February 26, 2018.
There was a large spike of online survey participants and participants interested in face-to-face
interviews on February 26, when two of four departmental directors who manage wrote
supporting emails of invited HPS in their department to participate, with a link to the
researcher’s original email and web link to participate.
Data Validation
Of the 30 items, 13 were administered on a four-point ordinal (Likert scale); 12 were
multiple answer; three were open-ended; and two were dichotomous (yes/no). The questionnaire
was modeled to incorporate Gap Analysis Framework constructs. Knowledge questions aimed
to determine what and how HPS identified as goals, why specific information should be
communicated and how to determine effectiveness of the process. This section primarily used
counts to determine potential causes, so each type of knowledge construct utilized different
methods of interpretation. Knowledge constructs that offered multiple answer choices were
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 100
categorized so that any correct answer that was selected by 80% or more of respondents was not
considered a validated cause; answers with between 50% and 80% did indicate familiarity, and
not a significant deficiency in knowledge; whereas, answers with less than 50% of correct
responses, indicated a lack in knowledge. These cut off points are based on the Knowledge,
Attitude and Practices (KAP) Survey (du Monde, 2011). This survey is comprised of a series of
predefined questions that collect quantitative and qualitative information to reveal
misconceptions or misunderstandings that facilitate or prevent individuals from attaining stated
goals. This survey was chosen because similar to the study of focus, this study aims to record
the perspectives based on declarative statements, but not necessarily the performance of
respondents. The cut points from the KAP survey reference Benjamin S. Bloom’s original cut
off points for KAP or, 80-100% for high levels, 60-80% for moderate levels or less than 59% for
low levels of KAP (Abdullahi, 2016; du Monde, 2011). The study of focus has slightly greater
variation in the moderate and low levels of knowledge (50% compared to 60%), and given the
smaller sample size, low power and larger margin for error, there is less likelihood of finding a
real true effect, which is why the variation for these cut points is greater, so as to be more
inclusive (Kenward & Roger, 1997; Krejcie & Morgan, 1970). Answers that were open ended
were not as easily categorized because responses were much lower per unique item, but those
answers that received more than 25% of responses were identified as the primary response for
that item and were explored in greater depth in the in-person interviews. For the Likert scale
items, HPS indicated whether they strongly disagreed (1), somewhat disagreed (2), somewhat
agreed (3) or strongly agreed (4) with the statements presented. The survey results that used the
Likert scale were analyzed around the means, which is the average of the total of all the scores
(Salkind, 2007). The means and standard deviations and ranges were reported for motivation
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 101
items and organizational items. It is important to note that these gaps (any mean less than 4.00)
are not absolutes. However, the closer a mean is to a 4.00, the smaller the gap, or the less
likelihood there is that challenges exist in communication of data and shared measurement
practices. A mean that is less than 4.00 may indicate that HPS perceptive a challenge.
Additionally, standard deviations were noted, as a low standard deviation indicates that the data
points close to the mean (also called the expected value) of the set, while high standard deviation
indicates that the data points are spread out over a wider range of values. As this survey was
generated by the principal researcher, and the survey has not been psychometrically validated,
therefore there are no established benchmarks means, standard deviations, or standardized scores
for this assessment.
In regard to the content, twelve questions evaluated knowledge, including: five questions
that focused on factual knowledge; four questions that focused on conceptual knowledge; two
questions that focused on metacognition knowledge and one question that focused on procedural
knowledge. Ten questions evaluated motivation including: including: three that focused on self-
efficacy; two that focused on mood; two that focused on value; one that focused on attribution
theory; one that focused on goal orientation; and one that focused on expectancy value. Eight
questions evaluated organizational factors, including: four questions that focused on policies,
processes and procedures; two that focused on resources and two that focused on cultural
models.
In order to determine the feasibility of accurately describing potential gaps, this study
collected online survey and in-person data to validate findings. Using a mixed methods
approach allows this study to examine KMO instead of just K, M or O, as using multiple
methods allows for the collection of a larger breadth of data. Mixed methods are an approach
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 102
that can answer broader research questions using multiple methods (Creswell, & Creswell, 2018;
Onwuegbuzie, & Johnson, 2006). Previous literature on the topic of mixed methods describes
that the strengths of one method can overcome the weaknesses of another. Furthermore, this on
type of data may also validate the opposite type of data and provide stronger evidence for a
conclusion or capture information about something that may have otherwise been overlooked. It
also increases the generalizability of the results (Creswell, & Clark, 2018; Creswell, & Creswell,
2018). Online surveys are considered a systematic way to gather data from a target audience,
that is inexpensive, has a low in burden for both the researcher, and the respondent, can be
completed anywhere. However, if items are not worded clearly, can be misunderstood,
mistrusted or may yield answers that lack quality or depth beyond offering descriptive. In-
person interviews have the capacity to capture verbal and non-verbal cues, if they are semi-
structured, can go in the direction that the participant feels most compelled to discuss, can
capture emotional behaviors, can clarify any misunderstood questions and can solicit examples
of specific constructs. However, interviews required a strong interviewer, one that does not ask
guiding questions, that offers a safe environment to speak candidly and displays accurate and
effective listening skills.
Results and Findings for Knowledge Causes
In order to assess whether there is a gap in knowledge it is important to differentiate
between the types of knowledge that HPS may potentially lack. Anderson and Krathwohl
described there are four different types of knowledge: factual, procedural, conceptual, and
metacognitive (2001). Factual knowledge includes statistics, details and terminology. This type
of knowledge is necessary in order to explain what is of relevance to communicate, in this case
being shared data and measurement practices. HPS must be knowledgeable of the goals of the
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 103
communication, what their role entails and what information needs to be collected and how to
best communicate the information. The next type of knowledge is conceptual knowledge
(Anderson, & Krathwohl, 2001; Krathwohl, 2002), which includes knowledge of schemas,
classification and categories. This type of knowledge represents an individual's ability to
organize information. HPS need to understand that in order to communicate shared data and
measurement practices they need to know who would have the data and understand the best
processes (efficient, effective, timely, specification level) to communicate. The subsequent type
of knowledge is procedural and includes knowing what the procedures are for how to solicit, and
then send out data and shared measurement practices. The final type of knowledge is
metacognitive knowledge, which includes knowledge about self-regulation, reflection and self-
knowledge. This type of knowledge includes when HPS take time to reflect how effective their
strategies are, and knowing what people to consult with for data, or how urgent data is to
communicate.
According to Clark and Estes (2008), performance challenges can sometimes be
attributed to lack of knowledge and/or skills. For the purpose of this project, lack of knowledge
and/or skills may be attributed to lack of factual, procedural, conceptual, and metacognitive
knowledge. In this study, gaps in four assumed knowledge influences, declarative factual,
declarative conceptual, procedural, and metacognition were validated through surveys and
interviews. Tables 7-10 presents an overview of the results of the assessment of each assumed
knowledge influence. In this section the results of the application of this methodology is
presented. Recommendations to improve each validated cause will be discussed in Chapter Five.
When knowledge gaps are identified, Anderson and Krathwohl (2001) provide a useful
framework to help validate the cause of the gaps and then provide solutions for the type of
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 104
training needed. A self-report online survey was administered along with an interview question
to provide the greatest level of validity (Salkind, 2007; Kurpius, & Stafford, 2005). Using the
Anderson and Krathwohl (2001) taxonomy for four knowledge dimension questions, a series of
questions were addressed within each dimension. The competencies with the greatest gaps in
knowledge and skills are discussed in the findings section.
Table 7
Results of the Knowledge Survey by Type of Dimension: Factual
Within the Collective Impact models, what do you think the goals of a backbone
organizations are? (Mark all that apply) Count %
Guiding the CI vision and strategy 29 97%
Supporting the aligned activities 28 93%
Communicating shared data and measurement practices. 28 93%
Mobilizing funding efforts 20 67%
Building public will 18 60%
Advancing policy 17 57%
What do you think the goals of Collective Impact models are? (Mark all that apply) Count %
A common agenda 27 90%
Shared data and measurement practices 27 90%
Continuous communication 25 83%
Mutually reinforcing activities among all participants. 24 80%
A centralized in infrastructure (backbone organization) 23 77%
A dedicated staff 17 57%
In order to communicate shared data and measurement practices with community
partners to support positive choices, you share... (Mark all that apply) Count %
Past, current and future interventions/programs 21 70%
Communication practices 16 53%
Shifts in organizational or performance goals 15 50%
Shifts in administration systems 9 30%
Shifts in technology and IT 9 30%
Shifts in production system 6 20%
In order to share data and measurement practices with community partners to
pursue policy and environmental changes, what kinds of information do you
share/communicate (Mark all that apply): Count %
Efforts adopting policies and environmental procedures/regulations 19 63%
CI initiative (building better health, living safely, thriving) 18 60%
Efforts to convene multi-sector stakeholders 17 57%
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 105
CI initiative areas of influence 17 57%
CI initiative indicators 17 57%
Shift in external guidelines/policies 13 43%
Policy enforcement approaches 11 37%
Shift in internal guidelines/policies 10 33%
Participate in health education events 8 27%
Shift in information systems 6 20%
In order to communicate shared data and measurement practices with community
partners to build better delivery systems, you share... (Mark all that apply) Count %
Decision making practices (shared and internal practices) 16 53%
Electronic record keeping practices 13 43%
Integrative/team-based practices 12 40%
Information about formal and informal partners 9 30%
Whole person care models 8 27%
Communication practices 7 23%
Factual Knowledge
Influence 1. HPS participants need to be knowledgeable about what and why to
communicate shared data and measurement practices and know how their role fits within the role
of a backbone organization and the CI initiative.
Survey Results. As shown in Table 7, more than half of HPS participants correctly
indicated all six components of a CI initiative: a common agenda (n=90%), shared data and
measurement practices (n=90%), continuous communication (n=83%), mutually reinforcing
activities among all participants (n=80%), a centralized infrastructure (n=77%), and dedicated
staff (n=57%). Additionally, more than half of HPS participants correctly indicated all six
components of a backbone organization: guiding the CI vision and strategy (n=97%), supporting
the aligned activities (n=93%), communicating shared data (n=93%) and measurement practices
(n=67%), mobilizing funding efforts (n=67%), building public will (n=60%), and advancing
policy (57%). In order to support building better deliver systems, most HPS respondents
indicated that it was important to “share decision making practices” (n=53%). In order to
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 106
support positive choices or performance goals most HPS respondents indicated “sharing past,
current and future interventions/programs” internally and externally (n=70%). In order to pursue
policy and environmental changes, most HPS indicated communicating efforts to “adopt policies
and environmental procedures/regulations” (63%).
Overall, HPS generally had a strong level of factual knowledge, scoring more than the
threshold of 80% correct suggesting they are not lacking factual knowledge about the goals of
HCG and of CI initiatives. However, HPS identified lacked knowledge about what information
to communicate with partners to build better delivery systems with the range of correctly
identified factors between 23% and 53%, as well as not being clear what information to
communicate with community partners to support positive choices, only correctly identifying
factors among 30%-70% of the items. HPS were asked about what information is shared to
“pursue policy and environmental changes,” only slightly more than half of respondents felt that
sharing “efforts to convene multi-sector stakeholders” (n=57%), and less than half felt that
discussing “shifts in external guideline/policies” (n=43%). Given a goal of having no less than
50% of participants able to correctly identify information on this and several other items, there is
factual knowledge that is lacking among HPS.
Interview findings. Current HPS were asked to describe the role of the role of HCG in
the CI initiative; and all discussed that the HCG is the backbone organization in the design and
implementation of the CI initiative, to which all correctly were able to identify. When asked
what the HPS role is in facilitating communication of shared data and measurement practices,
five disclosed that the role of health promotion is dynamic and involves an “interplay” between
what, when, where and how to exchange information so that it is not only well-received but
relevant, valuable and relates to the bottom line. They suggested there are more stakeholders
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 107
than just executives and community partners, there are the “people that the community partners
serve” “community stakeholders - like a school district” or large health system and hospital like
“Kaiser Permanente” as well as the Executive Leadership within the HCG. Information that the
HPS disclosed during dynamic interactions include, “the CI initiative vision”, “CI formalized
partnerships”, “CI areas of influence”, “measurement indicators”, “shifts in internal and external
guidelines and policies”, “health outreach and education events,” and current resources on “hot
topics” like Hepatitis A, grant opportunities, influenza, housing, and homelessness. When HPS
were asked about where these exchanges of communication take place and how many times,
every HPS indicated that these occur in numerous channels but primarily during in-person
meetings and via emails and is usually driven by stakeholder needs and questions. One person
said, “My job is to manage relationships with partners, and provide excellent customer services,
so it’s easy to be the face, meanwhile connecting with my team to request specific data for
them.” Five expressed the desire to improve “health outcomes,” but did not feel confident in the
priorities of executive leadership in a more detailed sense. They described that they know what
the overarching organizational goal but admitted that “when projects aren’t tied to funding
streams there is less clarity about what everyone is reporting.” “On my team, we report
deliverables with consistency,” but on other teams, they indicated knowing about “different
approaches.” But one HPS suggested that diversity is “necessary” because the community is so
large, diverse, and needs vary from “place to place” and “project to project.” “We shouldn’t have
one prescribed way to collect data,” one HPS indicated. Four HPS acknowledged that they were
not clear about what information HCG or the internal statistics unit collected and did not always
know what information would be available to communicate, how to access or when it would be
appropriate to solicit it. These types of admission led one HPS to say that they felt “skeptical”
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 108
that their approaches were “100% comprehensive or effective.” Three of the HPS indicated
sometimes duplicating efforts of team members within HCG, or sometimes not communicating
under the assumption that a colleague would be reaching out. Two of the HPS reported that
during an annual January planning meeting the data team presents to community stakeholders
about local trends and statistics, but the response isn’t generally to look into solutions or devise
solutions immediately connected to the data unless the data drives their professional organization
goal, and personal performance goals forward.
Summary. The majority of HPS know that they should be communicating shared data
and measurement practices. While respondents did not report between 80% and 100% of correct
identification of items for every item the online survey, the responses indicated that they were
generally knowledgeable about what information to communicate to community partners and
executives. Interviews revealed more insightful understanding about the discrepancies that exist,
as one respondent indicated, “lacking clarity” about what specifically needs to be communicated,
including what “supervisors” and “upper management” want to know, and “how can we most
effectively communicate it” to them in addition to community stakeholders, and the people who
are served by our community stakeholders, which weren’t previously mentioned in the survey
item wording. Additionally, face-to-cafe interviews revealed that there were significant lacks in
factual knowledge about what information the HCG statistics and data have to offer community
partners and organizations. The findings reveal that HPS are torn between the desire to have a
unified way to communicate to shared data and measurements, but also being offered the ability
to report what is tailored to the unique needs of the populations they work with. The HPS also
suggest that there are so many untapped opportunities to connect with community stakeholders
and they do not always know what colleagues within the HCG are doing with those same
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 109
community stakeholders. Given duplication of efforts or lack of effort to complete some tasks,
the reports in this section also indicate that their approach may at times lack effectiveness or
efficiency. Finally, when data is communicated to community stakeholders, for example in the
annual data presentations, action steps that may respond to the data may seem feasible to the data
team within HCG, but unless the data supports the community organization’s or personal
professional goals - are received with interest, but not actionable responses. Overall, the results
of the online survey and interviews validated that HPS lacked factual knowledge and offer areas
for further exploration, and analysis (See Table 7).
Table 8
Results of the Knowledge Survey by Type of Dimension: Conceptual
Which of the CI initiative strategic approaches (below) help you communicate
shared data and measurement practices? (Mark all that apply) Count %
Building better delivery systems 25 83%
Supporting positive choices 17 57%
Pursuing policy and environmental changes 17 57%
In order to facilitate the CI initiative, my role within HCG is to communicate
shared health data shared and measurement practices because, it (Mark all that
apply): Count %
Delivers accurate, accessible, and actionable health information 22 73%
Provides new opportunities to connect with diverse populations. 20 67%
Supports shared decision-making 19 63%
Builds social support networks 14 47%
Enables fast and informed responses to health issues 12 40%
Please indicate which answer choice(s) below best describe(s) why collecting
shared data and measurement practices in a systematic way is related to the
ongoing progress of the Collective Impact model initiative. Collecting shared data
and measurement practices ... Count %
Improves accountability among all participants 20 67%
Supports the exchange of work between partnerships (new and evolving ones) 19 63%
Improves the exchange of knowledge 19 63%
Enables effective decision-making 19 63%
Facilitates integrative and diverse organizational practices 18 60%
Improves the facilitation towards whole person care 13 43%
None of the above 0 0%
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 110
What does it mean to communicate shared data and measurement practices? Count %
Easily accessible (common language, easily measurable) 8 27%
Dissemination is effective 7 23%
Contributes to outcomes 4 13%
Informs decisions 4 13%
Dynamic exchange 4 13%
Accurate 3 10%
Timely 2 7%
Everyone has a unique role 1 3%
Conceptual Knowledge
Influence 2. HPS participants need to clearly understand why communicating shared
data and measurement practices is important to the HCG as the backbone organization in the CI
initiative.
Survey results. In order to facilitate the CI vision, HPS indicated that their role within
HCG is to facilitate communicate shared data and measurement practices for the following
reasons: to “deliver accurate, accessible, and actionable health information” (n=73%), to
“provide new opportunities to connect with diverse populations” (n=67%), to “support shared
decision making” (n=63%), to “build social networks” (n=47%), and “enable fast and informed
responses to health issues” (n=40%). When asked why collecting shared data and measurement
practices in a systematic way is related to the ongoing progress of the CI initiative, over half
indicated that it “improves accountability among all participants” (n=67%), it supports the
exchange of work between partnerships (new and evolving ones) (n=63%), it improves the
exchange of knowledge (n=63%), enables effective decision-making (n=63%), and facilitates
integrative and diverse organizational practices (n=60%); however less than half agreed that it
“improves the facilitation towards whole person care” (n=43%). There was less agreement
among HPS when asked what it means to communicate shared data and measurement practices.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 111
For example, most respondents identified that it means that information should be “easily
accessible, or that “communication should be done with a common language, be easily
measurable to all stakeholders” (n=27%), others suggested that “dissemination” strategies need
to be “effective” (n=23%), that the information should contribute towards effective “outcomes”
(n=13%). Overall, HPS did not score high in conceptual knowledge on the online surveys,
scoring generally below the 80% threshold, although they did agree that it has the capacity to
deliver accurate, accessible, and actionable health information across stakeholders (n=73%).
Interview findings. Current HPS were asked to describe their understanding of the task of
communicating shared data and measurement practices. Five indicated that their understanding
comes from the “expectations from their supervisors” or managers, and three indicated that it
comes from what “other HPS” staff in their region or department are doing. As this item was
one of the lower scoring item in the online survey, the researcher probed to better determine
HPS’ understanding of the role of communicating shared data and measurement practices. HPS
described that when data is not objective or “specifically required as tied to a contract of funding
stream”, that this construct is impacted by value (a motivational construct), and cultural models
(organizational construct) given the level of importance that supervisors/peers consider an item is
to be. One participant additionally described that “there wasn’t a formal training to explain how
the larger “theoretical” CI “vision” “translates into everyday understanding of processes that
should take place… I otherwise understand it contributes, but I’m not always sure why.” One
HPS offered, that communicating this information has the potential to save lives and get to
“zero” lives afflicted with “HIV status in the community,” while two others explained that
communicating data should “improve relationships” and “streamline processes.” Overall, all of
the interviewed HPS said that collecting data and demonstrating results is an essential part of
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 112
achieving the vision and mission of the CI vision, but the HPS’s performance approach to get
there was not as “clear.”
Summary. The majority of HPS believe that it is important to communicate shared data
and measurement practices. However, all respondents scored less than 75% correctly on the
items, in several questions, less than 50% could offer specific communication strategies that
ultimately contribute to improved health outcomes. Interviews magnified this issue, confirming
that there is inconsistency between supervisors, management and executive leadership about
what information is both of value, and a cultural norm to communicate. The results of the online
survey and validated that HPS lacked conceptual knowledge (See Table 8).
Table 9
Results of the Knowledge Survey by Type of Dimension: Procedural
In order to align your work with the CI initiative, please indicate how you connect
with partner organizations (Mark all that apply): Count %
Attend meetings such as community collaboratives, and advocacy groups with
partner organizations 25 83%
Exchange emails with partner organizations 22 73%
Participate in education and outreach efforts with partner organizations 21 70%
Meet with individual/groups in scheduled one-on-one meetings with partner
organizations 19 63%
Schedule routine telephone calls with partner organizations 10 33%
Procedural Knowledge
Influence 3. HPS participants need to be knowledgeable about the processes and steps to
undertake as they communicate shared data and measurement practices.
Survey results. When asked how HPS communicate shared data and measurement
practices, there was a high level of responses to this survey item. Most of HPS participants
indicated connecting with partnering organizations to obtain up to date information was to
“attend meetings such as community collaboratives, and advocacy groups with partner
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 113
organizations” (n=83%), via “exchanging emails with partnering organizations” (n=73%), by
“participating in education and outreach efforts with partner organizations (n=70%) and
“meeting with individual/groups in scheduled one-on-one meetings with partner organizations
(n=63%). Fewer than half indicated that they “schedule routine telephone calls with partner
organizations” (n=33%). Overall, participants scored between 50% and 80% on procedural
knowledge questions.
Interview findings. When asked what steps are taken to communicate shared data and
measurement practices, HPS acknowledged that often steps are outlined based on the
requirements of “funding” source or by supervisors and managers. As this was further
explained, each promotion staff is funded by a specific allocation of money and programs from
federal, state or local authorities have varying requirements for what information is being
communicated. One respondent suggested that the collected information rests heavily on what is
mandated including process data and outcomes. This is “helpful to document for continuing or
expanding funding, but may not necessarily be what is insightful, which is where the needs of
leadership comes in.” Four respondents indicated that although there are sometimes competing
requests from leadership staff, they feel very competent and capable of soliciting and providing
information to community partners, and to leadership. Four of the HPS indicated that there
could be more “structure” in place or ways to help “standardize” the collection of information
that is not tied specifically to a funding stream that could be important to Executive Leadership.
All HPS indicated that they are confident sharing information about data and shared
measurement practices that was considered “public-friendly” or “not confidential” when
corresponding with external stakeholders. All six HPS indicated sharing they know how to
communicate information with supervisors, typically in “regularly scheduled meetings” (e.g.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 114
regularly scheduled one-on-one weekly or monthly meetings), four indicated sharing information
in “email updates” and three said that share information with colleagues in “internal meetings”.
Summary. The majority of HPS use multiple approaches to communicate shared data and
measurement practices and typically the information that is reported or communicated is tied to
funding streams. Responses were high and generally above 50% among surveyed respondents,
which meant they could offer a consistent process that information is retrieved and disseminated.
Interviews validated the confidence that HPS feel in the process of communicating shared data
and measurement practices. The results of the online survey and validated that HPS do not lack
procedural knowledge (See Table 9).
Table 10
Results of the Knowledge Survey by Type of Dimension: Metacognitive
As a HPS, I self-reflect by… (Mark all that apply): (n=25) Count %
All of the above 20 80%
Think about the effectiveness of my strategies 6 24%
Evaluate my priorities 5 20%
Monitor my communication approaches 4 16%
Consider options when balancing the needs of HCG and community partners 4 16%
None of the above 1 4%
Please explain how people in my role should report comprehensive shared health
data and measurement practices to supervisory staff (open space for each question)
(n=17) Count %
● Efficiency: What would make this process efficient? Open space:
Standardization of guidelines (simple, and easy to use) 9 53%
Better understanding of needs of supervisor/manager/leadership 4 13%
Alignment with work plans/performance evaluation 3 10%
Increasing time to receive feedback 1 3%
● Accuracy: How would you check your information for accuracy before you
report it? Open space:
Fact Check with other sources 8 47%
Check in with our internal data team 6 35%
Follow supervisors guidelines 5 29%
Follow template to enter 4 24%
Follow colleagues lead 2 12%
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 115
Check with community partners, but this could be sensitive 1 6%
● Frequency - How often per day - who and how, per week - who and how, and
per month who and how? Open Space:
Wait until I meet with manager/supervisor 7 41%
As Needed 4 24%
Weekly 4 24%
Depends on the type 3 18%
Depends on the purpose/urgency 3 18%
Monthly 2 12%
During staff meetings 2 12%
● Compliance - What rules would you follow? Open space:
County guidelines (template provided) 7 41%
Relevant rules are consulted based on the type 5 29%
Confidential 2 12%
I anticipate needs so as to be thorough 2 12%
Metacognitive Knowledge
Influence 4. HPS participants need to understand how to self-reflect on opportunities or
challenges and to understand how to improve the process if necessary given self-directed goals
towards communicating shared data and measurement practices.
Survey results. Most HPS (n=80%) indicated reflecting on their progress, processes and
practices, in multiple ways including “thinking about effectiveness of personal strategies”,
“evaluating priorities”, “monitoring communication approaches”, and “considering options when
balancing the needs internally and externally”. This is a relatively high score indicating that HPS
do spend time reflecting on their goals, strategies, and efforts. However, much lower agreement
was evident in a subsequent question, which asked to self- reflect in their role, HPS suggested
that communication occurs during “meetings with supervisors” (41%), “follow HCG guidelines”
and is “fact checked” by HPS before communicating, but could be more efficient, if there was a
“standardized process” to communicate.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 116
Interview findings. When asked how they evaluate the effectiveness of their
communication strategies, five HPS reported that they look to their managers or supervisors to
provide feedback about the effectiveness of their strategies. Three indicated that struggling with
how to effectively reflect on their strategies, and two felt like they sometimes look to external
peers in their field to confirm strategies and approaches. One indicated that “thinking about
thinking” is time consuming but often very rewarding and suggested that when HPS meet it is a
productive time to “think through” obstacles and opportunities. Another item asked participants
to rank their “effectiveness communicating shared data and measurement practices” from
1=poor, 2=needs improvement, 3=meets expectations, 4=exceeds expectations, 5=exceptional.
Three ranked themselves as a score of 3, or average at communicating, and three ranked
themselves as a 4. The subsequent question asked HPS what it would take to increase their score
to a level 5. Four HPS reported needing more time, three suggested more clarification about
what executives wanted so as not to provide unnecessary information, and three said that they
did not get regular feedback to confidently say that they were doing well at their job.
Summary. Most HPS (n=80%) indicated reflecting on their progress, processes and
practices, in multiple ways to evaluate their performance, needs and to generate strategies to
tackle challenges which is a strong response level. The assumed metacognition knowledge of
HPS not knowing how to communicate shared data and measurement practices was validated.
While the survey showed that HPS lacked some metacognitive skills, the interviews unpacked
the idea that maybe having time to think about these issues would be helpful or having structured
time to think with peers would also prove to be beneficial. In the literature regarding community
health promotion professional one of the most important skills is possessing metacognitive
knowledge (Baker, L. (2006). When HPS are working in self-motivated professional
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 117
environments it essential to access one's knowledge, find the appropriate resources when need
and prepare for interaction with community stakeholders. Every interviewed HPS had indicated
struggling with how to effectively reflect about their own strategies. The results of the online
survey and interview did not validate that HPS lacked metacognition knowledge, instead
revealed that there may be improved processes or structured time allowing for metacognition
type activities to occur. The results also indicated that HPS lack factual and conceptual
knowledge based on self-reflective scores averaging 3.5 out of a maximum score of 5 for
exceeding expectations of communicating effectively (Table 10).
Results and Findings for Motivation Causes
HPS need to be motivated to communicate shared data and measurement practices,
feeling that it is valued, having self-efficacy to do so, feeling positive about collecting it, and
anticipating that the process will result in a positive experience. The gap analysis methodology
follows a path from performance gap to assumed causes to validated causes to solutions. In
Chapter 3, the assumed causes of the performance gap were outlined in Tables 2-4. The assumed
causes of this project revolved around the constructs of motivation and included: value, self-
efficacy, mood, attribution, goal orientation, and expectancy value. The survey served to assess
HPS’ perceptions of whether they had the motivational variables such as commitment and value,
to carry out specific tasks needed to implement comprehensive communication of shared data
and measurement practices. The items shown in Table 11 are sorted by sorted by major
motivation constructs and represent the survey items used to validate the assumed motivation
causes. The table includes the type of motivation construct, the item name, the mean, standard
deviation score, minimum and maximum scores. Descriptive statistics were used to analyze the
results. The subsequent Table 12, summarizes the results for statements measuring motivation
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 118
sorted by the highest means scores, and also contains the type of motivation construct, the item
name, the mean, standard deviation score, minimum and maximum scores.
Table 11
Results of the Motivation Survey by Types of All Motivational Dimensions
M-Value
How much do you
agree with the
following statement,
“As a Health
Promotion Staff
employed within the
County, I know that the
shared data I report is a
valuable part of the
decision making among
Executive Leadership.” Count % Mean
Standard
Deviation Minimum Maximum
Strongly Agree 9 30% 3.39 0.499 3 4
Somewhat Agree 14 47%
Somewhat Disagree 0 0%
Strongly Disagree 0 0%
M-Self-
Efficacy
How much confidence
do you have connecting
with community
partners to implement
the Collective Impact
Initiative? Count % Mean
Standard
Deviation Minimum Maximum
A lot of confidence 17 57% 3.71 0.464 3 4
A little confidence 7 23%
Not very much
confidence 0 0%
Not confident at all 0 0%
M-Self-
Efficacy
How much confidence
do you have connecting
with community
partners to
communicate shared
data and measurement
practices through the
Collective Impact
Initiative)? Count % Mean
Standard
Deviation Minimum Maximum
A lot of confidence 7 23% 3.46 0.658 2 4
A little confidence 14 47%
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 119
Not very much
confidence 3 10%
Not confident at all 0 0%
M-Self-
Efficacy
How much confidence
do you have reporting
shared data and
measurement practices
to supervisors
(frequency,
compliance, accuracy,
efficiency)? Count % Mean
Standard
Deviation Minimum Maximum
A lot of confidence 13 43% 3.39 0.499 3 4
A little confidence 9 30%
Not very much
confidence 2 7%
Not confident at all 0 0%
M-Mood
Please indicate
agreement with the
following statement,
"As a Health Promotion
Staff employed within
HCG, my organization
is supportive about my
role communicating
shared data and
measurement practices
with community
organizations." Count % Mean
Standard
Deviation Minimum Maximum
Strongly Agree 10 33% 3.38 0.576 2 4
Somewhat Agree 13 43%
Somewhat Disagree 1 3%
Strongly Disagree 0 0%
M-Mood
Please indicate
agreement with the
following statement,
“My performance as a
Health Promotion Staff
is impacted by my
motivation,
expectations and
engagement with
community partners.” Count % Mean
Standard
Deviation Minimum Maximum
Strongly Agree 12 40% 3.54 0.509 3 4
Somewhat Agree 11 37%
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 120
Strongly Disagree 0 0%
Somewhat Disagree 0 0%
M-
Attribution
Theory
Please indicate
agreement with the
following statement,
“There are reasons
beyond my control that
contribute to my
capacity to
communicate shared
data and measurement
practices.” Count % Mean
Standard
Deviation Minimum Maximum
Strongly Agree 8 27% 3.13 0.797 1 4
Somewhat Agree 12 40%
Somewhat Disagree 3 10%
Strongly Disagree 1 3%
M-Goal-
Orientation
I communicate shared
health data and
measurement practices
because I want to
(choose one): Count % Mean
Standard
Deviation
A. Advance my skills
and training as a HPS
through individual
improvement and
learning. 14 58% 1.58 0.508
B. Demonstrate that I,
as a HPS, am
competent and capable
of development and
progress. 10 42%
M-
Expectancy
Value
How much do you
agree with the
following statement,
“As a Health
Promotion Staff, I feel
that I am a part of a
culture where
communicating shared
health data and
measurement practices
is well-received." Count % Mean
Standard
Deviation Minimum Maximum
Strongly Agree 7 23% 3.21 0.588 2 4
Somewhat Agree 15 50%
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 121
Somewhat Disagree 2 7%
Strongly Disagree 0 0%
Table 12
Results of the Motivation Survey by Mean Score
Descriptive Statistics Mean
Std.
Deviation Minimum Maximum
M-Self-
Efficacy
My confidence communicating with
community partners CI initiative
3.71 0.464 3 4
M-Mood
My performance as a HPS is
impacted by my motivation,
expectations and engagement
3.54 0.509 3 4
M-Self-
Efficacy
My confidence communicating with
community partners shared data &
measurement
3.46 0.658 2 4
M-Self-
Efficacy
My confidence to report shared data
and measurement practices to
supervisors
3.39 0.499 3 4
M-Value
The shared data I report is a valuable
part of the decision making
3.39 0.499 3 4
M-Mood
Organization is supportive about my
role
3.38 0.576 2 4
M-
Expectancy
Value
I am a part of a culture where
communicating is well-received
3.21 0.588 2 4
M-Attribution
Theory
Reasons beyond my control that
contribute to my capacities.
3.13 0.797 1 4
M-Goal-
Orientation
I communicate shared health data and
measurement practices because of
intrinsic/extrinsic factors
1.58 0.504 1 2
Value
Influence #5. HPS need to be motivated to communicate shared data and measurement
practices, feeling that it is important and of value to do so.
Survey results. In order to facilitate the CI vision, HPS were asked if, they agreed that the
shared data and measurement practices that they report are a valuable part of the decision-
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 122
making process of Executive Leadership. The mean score of this response was 3.39 (sd=0.499)
with a range of 3.00-4.00 (minimum-maximum), or somewhat agree and strongly agree
responses. Overall, this score indicated that HPS feel like communicating this information is of
value. The shared data they report is a valuable part of the decision making, which may indicate
that there may be higher levels of motivation to report something that supervisors and
management value.
Interview findings. Current HPS were asked to describe the value of communicating
shared data and measurement practices with community partners and stakeholders. Six indicated
value towards communicating shared data and measurement practices, and that their value comes
from the idea of “engaging” community stakeholders in collaborative endeavors. Two indicated
that community organizations, and stakeholders are the people whose attitudes and actions have
an “impact” on the “reception and success” of projects in the community. Given the variety of
stakeholders in the community, with different interest, philosophies and priorities, “effective
communication ensures that there is a dynamic collaboration of providing information that is
relevant to their needs and builds positive rapport, which is extremely valuable, it’s what I call
‘social capital’.” HPS staff indicated that community stakeholders must “understand what we
are trying to achieve by communicating messages”, and they must “immediately find value in
our messaging”. If we want to build dialogue to gain a better understanding of our stakeholders’
interests and attitudes, having “strong levels of trust, with accurate, and timely communication
are essential.” One participant explained that the when she meets with community stakeholders
they want to be informed of any new information including updates on policies and health topics,
so that the organization they represent can inform the people they serve, e.g. “clients and the
public with timely information that has to do with health, safety and well-being.” Regarding
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 123
value of communication, one participant described that “there wasn’t a formal way to explain
how the larger “theoretical” CI “vision” “translates into everyday understanding of the value of
processes that should take place… I today understand how it contributes, but I’m not always sure
why.” Another HPS offered, that communicating this information has the potential to save lives
and get to “zero” lives afflicted with “HIV status in the community,” while two others explained
that communicating data should “improve relationships” and “streamline processes.”
Summary. Most HPS feel that communicating shared data and measurement practices
internally with supervisors, and communicating this to external community partners, and
stakeholder is valuable average mean and standard deviation. Most of the interviews confirmed
that HPS hold a high level of value for communicating shared data and measurement practices.
Literature has shown that value affects an individual’s choice of activities, effort, and persistence
(Schunk, 1991). People with a low level of value for accomplishing a task might delay or avoid
the task, while those with high levels of value show persistence though obstacles (Schunk, 1991).
The surveys and interviews indicate that HPS see value in communicating shared data and
measurement practices and therefore this construct did not validate that HPS lack value.
Self-Efficacy
Influence 6. HPS need to be motivated to communicate shared data and measurement
practices, having confidence and self-efficacy in their own ability to engage and achieve this
task.
Survey results. The highest mean scores were reported in the self-efficacy constructs of
motivation, which were higher than any other part of the online survey. For example, current
HPS expressed high levels of confidence connecting with community partners to implement the
CI initiative (m=3.71, sd=0.464, range=3.00-4.00). In accordance, HPS indicated strong levels
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 124
of confidence connecting with community partners to communicate shared data and
measurement practices (m=3.46, sd=0.658, range=2.00-4.00). HPS had slightly less levels of
confidence reporting shared data and measurement practices to supervisors (m=3.39, sd=0.499,
range=3.00-4.00). Moreover, statements measuring self-efficacy ranged from a mean of 3.39 to
3.71.
Interview findings. Current HPS were asked to describe the self-efficacy of
communicating shared data and measurement practices with community partners and
stakeholders. All six HPS indicated that they have high levels of confidence communicating this
information. Three mentioned that this is due to factors such as their “educational” focus or
“masters level degrees” in public health or in health promotion, and years of training or “past
experience” working with community partners. One indicated that communicating information
to the community in meetings, town halls and in larger groups is my “bread and butter” and
wished he could spend more time “promoting HCG efforts, programs and resources.” He
furthered, “this is the main reason I feel that I was hired, which is to promote health and wellness
through appropriate channels and with clear messaging.” In contrast, five HPS indicated feeling
less confident and “at times unclear because of mixed messaging” about when or how to
communicate information to supervisors. I’ve received feedback indicating that “if you know
one situation, then you know one situation” explaining just because you’ve done something one
way and learned one lesson, you have only learned one situation and one lesson. She explained,
“some scenarios of data need to be elevated to my supervisor immediately,” and “sometimes
those heuristic rules of thumb aren’t obvious to me”.
Summary. Most HPS feel strong levels of confidence communicating shared data and
measurement practices internally with supervisors and communicating this to external
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 125
community partners and stakeholders. Literature has shown that an individual’s belief in their
efficacy to regulate their own abilities to accomplish an activity determines their aspirations,
level of motivation, and academic accomplishments (Bandura, 1977). Similar to the
aforementioned value construct, people with high levels of self-efficacy show persistence though
challenges and are more likely to have higher expectations for future performance (Martin, &
Downson, 2009). The surveys and interviews indicate that HPS feel very confident in
communicating shared data and measurement practices with community stakeholders but during
interviews indicated room for improved levels of confidence when communicating to
supervisors. Thus, there is a lack in this motivational construct.
Mood
Influence 7. HPS need to be motivated to communicate shared data and measurement
practices, feeling positive about their role in the process.
Survey results. The mean scores reported in the mood construct of motivation were
again, very high in the online survey. For example, current HPS stated the strongest level of
agreement with the statement implying their role is impacted by motivation, expectations and
engagement with community partners (m=3.54, sd=0.509, range=3.00-4.00). Additionally, HPS
expressed strong agreement feeling that HCG is supportive in the HPS role to communicate
shared data and measurement practices with community organizations (m=3.38, sd=0.576,
range=2.00-4.00). Moreover, statements measuring mood ranged from a mean of 3.38 to 3.54.
Interview findings. Current HPS were asked to describe the mood towards
communicating shared data and measurement practices with community partners and
stakeholders. All six HPS indicated that they feel positive communicating shared data and
measurement practices, additionally feel positive communicating information with supervisors.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 126
Four HPS explained that they feel positive towards this task is because it is “easy” and typically
yields positive outcomes. Often when the communicate information they receive “positive
feedback” or am acknowledged in a positive way and that is “makes me feel like I’m doing good
at my job”. One participant shared, “when I communicate something to a community
stakeholder that benefits them, it elicits an overall positive response,” furthering that
stakeholders are more trusting of the HPS and tend to want to work more with the HPS in the
future.
Summary. Most HPS feel strong positively about communicating shared data and
measurement practices internally with supervisors and communicating this to external
community partners and stakeholders. Literature has shown that an individual’s mood impacts
self-motivation and for example, using the questionnaire, the Profile of Mood States and self-
motivation, literature has demonstrated that elevated global mood disturbance is correlated with
unsuccessful performance (Raglin, Morgan, & Luchsinger, 1990). The surveys and interviews
indicate that HPS feel positive in communicating shared data and measurement practices with
community stakeholders and supervisors and do not lack the mood construct.
Goal Orientation
Influence 8. HPS need to be motivated to communicate shared data and measurement
practices, feeling their it aligns with personal goals.
Survey findings. The survey item asked HPS respondents about their goal orientation to
determine whether the HPS were aligned with mastery or performance. HPS were asked, why
they want to communicate shared data and measurement practices. The first option was mastery
oriented, “to advance my skills and training as an HPS through individual improvement and
learning.” The second option was performance oriented, “to demonstrate that I, as a HPS, am
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 127
competent and capable of development and progress.” Slightly more than half of the
respondents (n=58%) indicated that they were mastery oriented, and 42% indicated being more
performance oriented. Overall, these data would suggest that the HPS had a mastery performance
orientation.
Interview findings. Current HPS were asked to describe their goals towards
communicating shared data and measurement practices with community partners and
stakeholders. Four HPS indicated that their goals are related to improve the health and wellbeing
of local residents. One indicated, “I came to work for the government to serve… I work hard at
what I do because it is my passion.” Two HPS acknowledged that although they are not salaried
employees, they work on their personal time, because as one HPS said, sometimes “I can’t get
everything I want to get to during the workdays,” and another said, “it’s is a labor of love.” One
participant shared her personal journey with drug abuse and experience in and out of
rehabilitation centers, admitting that her struggles with the “system” have been a “driving force”
behind her “commitment to this role”. All of the HPS shared they are motivated to improve
communication efforts and experience satisfaction they are making a difference in the health and
wellbeing of the community. HPS enjoyed seeing communities experience increases in
participation of health promotion events, adopting of policies, and implementation of
environmental changes.
Summary. Slightly more than half of the surveyed HPS identified themselves as mastery
oriented and in the interviews reported examples of intrinsic and mastery-oriented descriptions of
their motivation. Literature on the topic of goal-orientation suggests that those who are mastery
orientation are more likely to have higher levels of self-efficacy, performance and knowledge,
whereas performance orientation was negatively related to performance only.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 128
Attribution Theory
Influence 9. HPS need to be motivated to communicate shared data and measurement
practices, feeling that the process will result in a positive experience regardless of competing
environmental circumstances.
Survey findings. Most HPS respondents reported somewhat agreement towards the
statement that there are reasons beyond their control that contribute to their capacity to
communicate shared data and measurement practices (m=3.13, sd=0.797, range=1.00-4.00).
This question is worded in such a way as to determine if the role of communicating shared data
and measurement practices is depended on the HPS or on environmental factors. The responses
indicate that HPS indicated that there are indeed varying perspectives about whether the HPS
performance is impacted by environmental causes or is the sole byproduct of the individual.
Although there is consistently somewhat of an agreement, this is the lowest mean scoring item
compared to all motivational constructs, however it still scores within the acceptable range for
not declaring a lack in motivation. Albeit, this item received the widest range of responses, with
some participants strongly agreeing, and others strongly disagreeing. This item was explored in
the interview survey to reveal attributed outcomes to reasons beyond their control.
Interview findings. Current HPS were asked to describe reasons beyond their control that
contribute to their capacity to communicate shared data and measurement practices. All six HPS
indicated reasons including “Executive Leadership” “supervisor’s” decisions, “funding sources”,
“extemporaneous circumstances.” Some referenced the recent Hep A crisis, or the union strike as
competing circumstances. Regardless of the environmental factors, HPS reported willingness to
work with dynamic conditions, and unforeseeable challenges because they felt there were
making an impact on the health and wellbeing of the community. Two HPS also expressed
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 129
enjoying the ways that environmental conditions offer a “unique and unconventional work
environment” and appreciate the lack of a routine, explaining this is the nature of our role, which
some reported as “exciting” admitting things are “never boring”.
Summary. HPS surveyed and interviewed agreed that there are reasons beyond their
control that contribute to their capacity to communicate shared data and measurement practices.
HPS indicated both positive and negative external factors that would contribute to successes or
failures of this task. The reasons cited include supervisorial support, funding and staffing
opportunities/limitations. The HPS reported feeling that the processes suppress or support their
capacities and impact their motivation to communicate shared data or measurement practices.
Overall, the results for attribution suggested that HPS attribute outcomes to reasons beyond their
control but do not indicate a lack in the attribution construct.
Results and Findings for Organization Causes
HPS need to have organizational factors in place such as policies, processes and
procedures, resources and cultural models that support HPS as they communicate shared data
and measurement practices. The third and final area of the gap analysis aims to evaluate are the
HPS perceptions of the organizational factors contributing to the communication of shared data
and measurement practices across community stakeholders and management within HCG. In
Chapter 3, the assumed causes of the performance gap were outlined and follow the Clark and
Estes (2008) framework. The authors outline key aspects of an organization that are potential
sources of performance gaps including: resources, work processes, policies, procedures, and an
organizational cultural model. This section explores these three potential causes in these areas.
The items shown in Table 13 are sorted by sorted by major organizational constructs and
represent the survey items used to validate the assumed organizational causes. The table
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 130
includes the type of organizational construct, the item name, the mean, standard deviation score,
minimum and maximum scores. Descriptive statistics were used to analyze the results. The
subsequent Table 14, summarizes the results for statements measuring organizational constructs
sorted by the highest means scores, the type of organizational construct, the item name, the
mean, standard deviation score, minimum and maximum scores.
Table 13
Results of the Organizational Survey by Type of Organizational Dimension
O-
Resources
Please indicate
agreement with the
following statement, "As
a Health Promotion Staff
employed within HCG
my organization is offers
resources to support my
role communicating
shared data and
measurement practices
with community
organizations." Count % Mean
Standard
Deviation Minimum Maximum
Strongly Agree 8 27% 3 0.933 1 4
Somewhat Agree 10 33%
Somewhat Disagree 4 13%
Strongly Disagree 2 7%
O-
Resources
Please indicate
agreement with the
following statement,
“My workplace provides
me with the materials,
supplies and resources to
help me in my role.” Count % Mean
Standard
Deviation Minimum Maximum
Somewhat Agree 14 47% 3.29 0.69 1 4
Strongly Agree 9 30%
Somewhat Disagree 1 3%
Strongly Disagree 0 0%
O-
Policies,
Processes,
In your opinion, are
there policies in place
that guide the design and Count %
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 131
&
Procedure
implementation of the CI
initiative?
Yes 21 84%
I don't know 3 12%
No 1 4%
O-
Policies,
Processes,
&
Procedure
Please mark the box(es)
below, if you believe
there are policies that
help you design and
implement the CI
initiative: Count %
Policies related to
communication practices 20 67%
Policies related to
community partnerships
roles and responsibilities 12 40%
Policies related to
electronic record
keeping practices 8 27%
Policies that are
addressed in your annual
performance review 7 23%
O-
Policies,
Processes,
&
Procedure
Please indicate
agreement with the
following statement,
“My workplace provides
appropriate professional
development in my
role.” Count % Mean
Standard
Deviation Minimum Maximum
Strongly Agree 9 30% 3.17 0.816 1 4
Somewhat Agree 11 37%
Somewhat Disagree 3 10%
Strongly Disagree 1 3%
O-
Policies,
Processes,
&
Procedure
Please indicate
agreement with the
following statement,
“My workplace provides
me with the opportunity
to work with a mentor
on the topic of
communicating shared
health data and
measurement practices.” Count % Mean
Standard
Deviation Minimum Maximum
Strongly Agree 3 10% 2.54 0.977 1 4
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 132
Somewhat Agree 12 40%
Somewhat Disagree 4 13%
Strongly Disagree 5 17%
O-Cultural
Models
How much do you agree
with the following
statement, “As a Health
Promotion Staff
employed within HCG, I
am incentivized (with
recognition, promotion
or acknowledgement) to
communicate shared
data and measurement
practices.” Count % Mean
Standard
Deviation Minimum Maximum
Strongly Agree 1 3% 2.54 0.779 1 4
Somewhat Agree 12 40%
Somewhat Disagree 8 27%
Strongly Disagree 3 10%
O-Cultural
Models
Please indicate
agreement with the
following statement, "As
a Health Promotion Staff
employed within the
HCG, my organization
facilitates time for me to
communicate shared
data and measurement
practices.” Count % Mean
Standard
Deviation Minimum Maximum
Strongly Agree 7 23% 3.04 0.806 1 4
Somewhat Agree 12 40%
Somewhat Disagree 4 13%
Strongly Disagree 1 3%
Table 14
Results of the Organizational Survey by Type of Mean Score
Descriptive Statistics Mean
Std.
Deviation Minimum Maximum
My organization offers materials, supplies and
resources to help me in my role
3.29 0.69 1 4
My organization offers appropriate professional
development in my role
3.17 0.816 1 4
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 133
My organization facilitates time for me to
communicate shared data and measurement practices
3.04 0.806 1 4
My organization offers resources to support my role
communicating with community partners
3.00 0.933 1 4
My organization offers me a mentor on the topic of
communicating shared health data and measurement
practices
2.54 0.977 1 4
I am incentivized (with recognition, promotion or
acknowledgement)
2.46 0.779 1 4
Resources
Influence 10. HPS need to have organizational factors in place such as resources,
materials and supplies that support HPS as they communicate shared data and measurement
practices.
Survey results. In order to facilitate the CI vision, HPS were asked how strongly they
agreed that the organization offers resources to support their role communicating shared data and
measurement practices with community partners. The mean score of this response was 3.00
(sd=0.33) with a range of 1.00-4.00 (minimum-maximum), or strongly disagree and strongly
agree responses. The second survey question asked for agreement to the whether their
workplace provides them with the materials, supplies and resources to help them in their role.
The mean score of this response was 3.29 (sd=0.690) with a range of 1.00-4.00 (minimum-
maximum), or strongly disagree and strongly agree responses. Overall, this score indicated that
HPS feel like the organization offers resources to support their role.
Interview findings. Current HPS were asked to describe the materials, supplies or
resources that their workplace provides for them to communicate comprehensive shared data and
measurement practices. Five indicated that they have laptop computers, which enables them to
work no matter where they are, which one described as “essential” because “the nature of the
role is to be in the community.” Four shared that they use a shared database to “work with
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 134
colleagues” from the same team, which encourages them to view, edit and review peers work
and projects. One HPS responded that it is important to have “mobile access to files with
updated information.” Three HPS expressed the desire to have access to LHD vehicles, as the
demands of the job are to always be on the move. On HPS admitted, I “put on a lot of mileage in
this job... and the wear and tear on my car” since being in this job is one of the constraints of the
job.
Summary. HPS surveyed and interviewed agreed that they have the materials, supplies,
and other general resources needed to perform the roles in their job, including communicating
shared data and measurement practices. While they indicated that the resources currently on
average 3.0 or “somewhat” meet these needs, during the interviews, they indicated that it would
be nice to have other resources, like privileges to use the work cars, or have a mileage allowance
in their wages. Overall, the results for resources available to HPS, suggested that there is a not a
significant lack in the resources construct, except for the use of a county car or mileage
allowance.
Policies, Processes & Procedures
Influence 11. HPS need to have organizational factors in place such as policies, processes
and procedures that support HPS as they communicate shared data and measurement practices.
Survey findings. In order to facilitate the CI vision, HPS were asked if in their opinion,
there were policies in place that guide the design and implementation of the CI initiative. HPS
strongly agreed that “yes”, there are policies in place (n=84%) that the organization offers
resources to support their role communicating shared data and measurement practices with
community partners, while n=4% indicated “no” there were not policies in place, and 12%
responded that they “don’t know” if there are policies in place. Another item asked HPS to
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 135
indicate what types of policies are in place that help to design and implement the CI initiative.
Most HPS indicated that there are policies related to communication practices (n=67%), fewer
than half felt that there are policies related to community partnerships roles and responsibilities
(n=40%), approximately a quarter felt there are policies related to electronic record keeping
practices (n=27%), and policies that are addressed in their annual performance review (n=23%).
Although there was not a strong level of agreement about what specific policies exist to facilitate
communication of shared data and communication practices, there was a strong agreement that
policies are generally in place. The next survey item asked HPS their level of agreement to
whether their workplace provides appropriate professional development in their role. The mean
score of this response was 3.17 (sd=0.816) with a range of 1.00-4.00 (minimum-maximum), or
strongly disagree and strongly agree responses. This indicates that there was strong agreement
with having appropriate professional development. HPS felt less agreement with whether they
felt that their workplace provides them with the opportunity to work with a mentor on the topic
of communicating shared data and measurement practices, (m=2.54; std dev=.977; range=1-4).
The average score indicates that HPS do not strongly agree or disagree with this statement.
Overall, these items indicated that HPS feel like the organization has policies to support their
role of communicating shared data and measurement practices but did not necessarily feel like
they had people in the workplace that they could call their mentors.
Interview Findings. Current HPS were asked to discuss the policies, processes or
procedures exist within their workplace that support their efforts to communicate shared data and
measurement practices. All six of interviewed participants listed numerous policies in place that
support the larger CI initiative, however were not as clear about whether there are processes or
procedures that provide a standardized communication stream of information. One HPS
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 136
indicated that without specific responsibilities or procedures held as a “guideline with a
deadline”, there is less “accountability”. Two indicated that there are policies in place that guide
processes and procedures that help streamline what is communicated between HCG and the
community partners directed by Executive Leadership within HCG, but this is “different
depending on what department or region you work in.” Three reported wanting to have these
communicated in ways that are “convenient” and “easily accessible” in order to “translate” into
everyday practices. Another HPS remarked that on a larger scale, that HCG adopted the CI
initiative in the strategic plan suggesting that the goals, and overarching plans support the CI
vision. They furthered, that the direction of funding, resources, and ongoing programming is
defined by the HCG Executive Leadership Team, and that the operational vision and “direction
comes from the top”. They implied that the impacts of this adoption “trickles down” to all staff
to support and implement. When the CI initiative was first adopted ten years ago, three
participants conceded that it was “not popular” nor “quickly adopted”, but today, one remarked
that the CI vision feels “omnipresent” and continues to “grow and mature”. Additionally, as the
expectations for growing the CI vision expands and incorporates more community partners in the
integration of work standards, and processes, four HPS expressed that finding great “colleagues
is hard, but it’s even harder to keep people working here when they are overworked.” “There
should be policies in place that have recommendations or about what is expected of each staff, so
that it is clear what needs to take place to be successful.” Three of the HPS suggested that there
are “not enough staff” to keep up with the “growth of the CI vision”.
Summary. HPS surveyed and interviewed agreed that there were policies in place that
guide the design and implementation of the CI initiative that start from the top level. HPS
additionally indicated that this information does cascade down and impact policies, and
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 137
procedures, however, these are not standardized across the HCG. While they indicated that there
are policies in place that guide communication practices, some indicated that it would be nice to
have these communicated in a way that is convenient and easily accessible in order to translate
into everyday practice, and that they would like to have a category to highlight this responsibility
on their annual performance review so as to track over time and gauge improvements or to
provide feedback. HPS also indicated that the feel somewhat in agreement that they have
appropriate professional development opportunities (mean=3.17) but did not consistently feel
that they have someone that they can call a “mentor” in their workplace (mean=2.54).
Additionally, as the expectations for growing the CI vision expands there could be clear policies
in place that have recommendations or about what is expected of each staff and hiring
opportunities to keep up with the growth of the CI vision. With resource constraints, there could
be new policies that offer mutually beneficial opportunities where employees can better gauge
performance excellence through the use of indicators. Overall, the results for policies, processes
and procedures available to HPS suggested that there is a lack in this construct.
Cultural Models
Influence 12. HPS need to have organizational factors in place such as cultural models
that support and collaborate with HPS as they communicate shared data and measurement
practices.
Survey results. This survey item asked HPS respondents about their level of agreement
about whether they feel incentivized with recognition, promotion or acknowledgement to
communicate shared data and measurement practices. HPS felt less agreement with this item
and reported a mean score of 2.54 (std dev=.779; range=1-4). The average score indicates that
HPS do not strongly agree or disagree with this statement. The next survey item that looked at
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 138
cultural models asked HPS their level of agreement to whether they believe that their
organization facilitates time for them to communicate shared data and measurement practices.
HPS felt more agreement with this item than the previous item and reported a mean score of 3.04
(std dev=.806; range=1-4). The average score indicates that HPS “somewhat agree” with this
statement. Overall, these items indicated that HPS feel like the cultural models within the HCG
organization support their role of communicating shared data and measurement practices.
Interview findings. Current HPS were asked to describe whether they are allotted time to
communicate shared data and measurement practices, all six indicated that they feel that their
management and organization “in general” facilitates time to do so. Three HPS indicated feeling
incentivized to communicate comprehensive shared data and measurement practices. Others
indicated feeling intrinsically rewarded for doing their job well as public health and public
services is rewarding. Out of all six participants, only one reported getting a promotion as a
reward for their work in health communications on the CI initiative project accomplishments.
Summary. Slightly more than half of the surveyed HPS felt that they were working in a
culture that supported their role of communicating shared data and measurement practices by
facilitating time to do so. While HPS felt less agreement with feeling incentivized and indicated
this on both the survey and during interviews. Literature on the topic of cultural models
indicates that an organization can help to spearhead ideas and practices that are incorporated in
everyday situations (Fryberg, & Markus, 2007). In this regard, facilitating time to communicate
this information seems more commonplace than not, however, the cultural models behind giving
praise seems less present. Overall, the results for cultural models suggested that there is a not
lack in this construct.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 139
Summary of Validated Influences
Knowledge
Five gaps were identified in this study. The first gap was in the area of factual
knowledge. During in-person interviews, HPS indicated that they experienced discrepancies in
understanding about what information exists that can be communicated as shared data and
measurement practices, including what “supervisors” and “upper management” want to know,
and “how can we most effectively communicate it”. Additionally, face-to-cafe interviews
revealed that there were significant lacks in factual knowledge about what information the HCG
statistics and data have to offer community partners and organizations. Thus, the results of the
online survey and interviews validated that HPS lacked factual knowledge.
The second gap was in the area of conceptual knowledge. During in-person interviews,
HPS indicated that they believe that it is important to communicate shared data and measurement
practices. However, all respondents scored less than 75% correctly on the items describing what
to communicate in reaction to this goal. In several questions, less than 50% correctly identified
specific communication strategies that ultimately contribute to improved health outcomes. In-
person interviews magnified this issue, confirming that there is inconsistency between what
information is both of value, and a cultural norm to communicate. The results of the online
survey and validated that HPS lacked conceptual knowledge.
The third gap was in the area of metacognitive knowledge. As HPS participated in online
surveys, most HPS (n=80%) indicated reflecting on their progress, evaluate their performance,
and generating strategies to tackle challenges which is a strong response level. However, during
in-person interviews, HPS indicated having the desire to have more designated time to think
about their role or having structured time to think “aloud” with peers would also prove to be
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 140
beneficial. In the literature regarding community health promotion professional one of the most
important skills is possessing metacognitive knowledge (Baker, 2006). When HPS are working
in self-motivated professional environments it essential to access one's knowledge, find the
appropriate resources when need and prepare for interaction with community stakeholders. The
results of the online survey and interview revealed that there may be improved processes or
structured time allowing for metacognition type activities to occur.
Motivation
The fourth gap was in the area of self-efficacy motivation. During the online interviews,
most HPS reported feeling strong levels of confidence communicating shared data and
measurement practices internally with supervisors and communicating this to external
community partners and stakeholders. However, during face-to-face interviews indicated
desiring increased levels of confidence when communicating to supervisors. HPS indicated that
they have between above average confidence in their ability to communicate shared data and
measurement practices effectively with supervisors, indicating that they are not clear about what
Executive Leadership wants to see and in what format. Literature has shown that an individual’s
self-efficacy to regulate their own abilities to accomplish an activity determines their aspirations,
level of motivation, and academic accomplishments (Bandura, 1977). Thus, people with high
levels of self-efficacy show persistence though challenges and are more likely to have higher
expectations for future performance (Martin et al., 2016). In this case, HPS report an
opportunity to increase their self-efficacy, therefore, there is a lack in this motivational construct.
Organization
HPS surveyed and interviewed agreed that there were policies in place that guide the
design and implementation of the CI initiative that start from the top level, however, they
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 141
acknowledged that these are not standardized across the HCG. Additionally, while they
indicated that there are policies in place that guide communication practices, some indicated that
it would be nice to have these communicated in a way that is convenient and easily accessible in
order to translate into everyday practice. Furthermore, they admitted wanting a category to
highlight this responsibility on their annual performance review so as to track over time and
gauge improvements or to provide feedback. HPS indicated that the feel somewhat in agreement
that they have appropriate professional development opportunities (mean=3.17) but did not
consistently feel that they have someone that they can call a “mentor” in their workplace
(mean=2.54). Additionally, as the expectations for growing the CI vision expands HPS reported
wanting clear policies about what is expected of each staff and allow for additional hiring
opportunities to manage the growth of the expanding CI vision. With resource constraints, there
could be new policies that offer mutually beneficial opportunities where employees can better
gauge performance excellence through the use of indicators. Overall, the results for policies,
processes and procedures available to HPS suggested that there is a lack in this construct.
In order to evaluate the solutions implemented at Healthy County Government,
Kirkpatrick and Kirkpatrick’s (2016) four levels of evaluation will be used to determine if the
solutions are in fact leading to the desired goal. Future research implications of this study
include pursuing longitudinal or cohort studies, which would explore the use of potential training
or tools to improve learning strategies among HPS staff. With greater understanding the how to
effectively support HPS staff, there would likely be an increase in HPS capacities to facilitate
improved health outcomes of local inhabitants. This framework will be presented in further
detail in Chapter Five.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 142
CHAPTER FIVE: RECOMMENDATIONS AND EVALUATION
Purpose of the Project and Questions
There is a need for comprehensive acquisition and communication of shared data and
measurement practices for optimum decision-making in local health departments (O'carroll et al.,
1998). Health professionals, from Executive Leadership to Health Promotion Staff (HPS) are
expected to create proactive approaches that improve the health and well-being of people they
serve. It is essential for them to have comprehensive information to make appropriate decisions
(Frieden, 2010). When professionals do not have comprehensive information about community
health statistics, status and activities in a timely manner, they are less confident to make
informed decisions and develop or implement organization-wide strategic, integrative, effective,
or proactive approaches (Witmer et al., 1995). Furthermore, when professionals face
deficiencies in knowledge, motivation, or organizational (KMO) areas, it contributes to lower
and less profitable results for an organization (Stewart, & Ruckdeschel, 1998). In order to
understand the performance of employees, organization have to ask questions, viewing
employees as capital and fostering investments in their performance (Clark, & Estes, 2008).
In Chapter 4, health professionals working at Health County Government (HCG)
described lacking several KMO factors, which they felt “leads to poor decisions.” They
described their work conditions of routine commuting, remotely working, being reliant on
technologically-based systems, and part of a team dynamic, all of which they admitted can be
stressful when they are “not prepared.” Literature has shown that lack of communication
contributes to unmet expectations such as misinterpretation of requests, missed deadlines,
misunderstanding of project deliverables which lead to inefficiencies. In fact, for the last twenty
years, researchers have argued that there is a distinct economic value tied to inefficiencies in
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 143
work performance and poor communication (Baldwin, & Ford, 1988; Clark, & Estes, 2008;
Crook et al., 2011; Senge et al., 1999; Smith, et al, 1994; Stewart, & Ruckdeschel, 1998). While
HCG adopted an open systems procedural approach to communicate data and measurement
practices within their system, based on interviews and survey questionnaires, staff feel that HPS
have not fully adopted the key tenets of an open systems approach to communication within the
CI scope of their work.
Information sharing is a complex task given interpersonal, intra-organizational, and inter-
organizational factors (Chun et al., 2010). As the backbone organization, HCG is responsible for
communicating data and measurement practices consistently across all partners to ensure efforts
remain aligned and participants hold each other accountable (Hanleybrown et al., 2012). The
Clark and Estes (2008) Gap Analysis Process Model was used as a guide and framework to help
Healthy County Government (HCG) reach its goal of evaluating communication of shared data
and measurement practices among HPS. Following the gap analysis model, HCG identified the
goal, as the current performance of HPS in the following three areas: knowledge, motivation and
organizational factors (KMO). The research questions that guided this study approach were:
1. What are the KMO factors that challenge HPS in their communication of shared
health data and measurement practices of one-hundred percent comprehensive
information required by Executive Leadership?
2. What are the KMO opportunities that exist to facilitate HPS’ ability to communicate
shared health data and measurement practices of one-hundred percent comprehensive
information required by Executive Leadership?
In order to address these questions, existing best practice models and published literature
were referenced in the design of online survey items, and in-person interviews questions. These
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 144
items aimed to help to identify any potential causes of gaps. Mixed methods were employed to
confirm the causes, which were validated in Chapter Four. The five validated causes included
lack in knowledge under the constructs of 1) factual, 2) conceptual and 3) metacognitive
knowledge, 4) lack in motivation under the construct of self-efficacy and lack of organizational
factors under the construct 5) policies, processes and procedures.
The purpose of this chapter is to identify potential recommendations that HCG can
employ in order to reach their goal of evaluating and improving communication of shared data
and measurement practices. Additionally, this chapter aims to support HCG in the process of
implementing the solutions, as well as evaluation suggestions in order to ensure that the solutions
are helpful in closing the identified gap.
Recommendations to Address Knowledge, Motivation, and Organization Influences
As discussed in Chapter One, HCG is responsible for communicating shared data and
measurement practices. Moreover, HCG must run its operations in a way that supports a region
that combines the collective strengths of community stakeholders, transforms policies and
systems and integrates delivery of services and resources. HCG sees the value in advocating for
the health and social well-being of all residents. They believe that it is essential to facilitate
collaboration and participation across community stakeholders in order to foster a community
where residents are healthy and empowered. As such, this study aims to complement stated
organizational goals through evaluating the performance goals of HPS as the stakeholder of
focus. In this previous chapters, activities and implementation practices of communicating
shared data were discussed, in the present chapter recommendations and solutions to improve
communication among HPS will be explored.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 145
Chapter Five is divided into three components that influence the ability of HPS to achieve
their role as a participant in Collective Impact (CI) model: Knowledge, Motivation, and
Organizational (KMO). Each subsequent section contains an overview, including a rational for
the need, if any, to prioritize validated causes that contribute to the gap. Additionally, each
section will include a table with the KMO cause, the priority, the evidence-based principles that
support the recommendation and the recommendation for each cause based on applying the
principle. Following the table, each high priority cause is discussed, as well as the principle and
support for the solution based in the literature.
Organizational Goals
The interpretation of findings will complement the organizational goals of HCG, which
aims to redefine the role of the government as a steward of public welfare by reducing
preventable disease-related deaths and improving health and social wellness. As such, they aim
to adopt cross-sector, collaborative approach to achieve large-scale policy and community
impacts (Flood et al., 2015; Kania, & Kramer, 2013). Executive Leadership believes that
decision making must be collective to be sustainable, therefore it must be driven by local
community needs. HCG believes that decisions must be results-oriented, focused yet adaptive,
and consider long-term ramifications in order to be enduring. Executive Leadership believes in
leveraging acquisition and communication of health data to improve population health goals. By
using the CI model, HCG is invested in the alignment of effort between departments, staff and
the community to improve community health through effective policies, which is described in
the literature as a comprehensive approach to improving population health (Trochim et al.,
2006).
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 146
Stakeholder Goals
HPS interact with individuals, families and community stakeholders and serve as the
liaisons between the local government and the community. Given their extensive time and
energy working with community partners, HPS have the potential to acquire and communicate
comprehensive health data that impact their community and contribute to the HCG’s goal of
improving the population health. Following this study, Executive Leadership aims to gain a
greater understanding of the current status of health of the community.
Moreover, in this chapter we explore how HPS communication is an essential component
to connecting health opportunities, resources, and programs with the general population. Using
the gap analysis model, the difference or gap between the performance goal, and the current
performance of HPS was established and possible causes for the gap were determined (Clark, &
Estes, 2008). Following a mixed methods approach, this chapter describes validated and
triangulated causes, and proposes solutions to close the gap (Creswell, & Clark, 2018; Creswell,
& Creswell, 2018). The solutions in this chapter aim to close the validated gap, and includes
addressing three knowledge constructs including: conceptual, factual, and metacognitive.
Besides knowledge, HPS must close the validated cause in motivational, or self-efficacy.
Furthermore, the validated cause is the organizational construct, or policies practices, which
suggests that the modification of existing policies such as performance review guidelines,
staffing guides, and clear communication guidelines to better support HPS within HCG. At the
conclusion of this study, Executive Leadership team will be better informed to improve
performance, by following best practices of related organizations, and literature on this topic.
Conjointly, Executive Leadership will be better able to identify strategies for integrative and
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 147
effective CI efforts to ultimately facilitate community-based health service delivery among
communities within the County.
Knowledge Recommendations
Introduction. The knowledge influences described in Table 15 represent the complete
list of assumed knowledge influences. These influences were validated based of the perspectives
and beliefs of HPS and because they were mentioned as an area that could be improved. Three
assumed knowledge influences that were validated: factual, conceptual and metacognition. The
table includes the goals of HPS role and the approaches to communicate data and shared
measurement practices. The principles related to this influence are the information processing
model by Schraw and McCrudden (2013). They indicate that information learned meaningfully
and connected with prior knowledge is stored more quickly and remembered more accurately.
Furthermore, new information that is integrated and elaborated with prior learning, combined
with feedback and successful completion of tasks improves self-efficacy, and expectancy value
(Brewer, & Brewer, 2010).
As indicated in Table 15, these influences were validated and have a high priority for
achieving the stakeholders’ goal. Table 15 also shows the recommendations for these highly
probable influences based on theoretical principles. This section will explain the research from
the learning theories knowledge types and create a subsection for each knowledge type.
Declarative, conceptual, and metacognition knowledge will be defined, explained and the
application to the stakeholder goal and the assumed influence will be explored. Moreover, the
literature about this domain/topic will be detailed as it relates to the multiple types of knowledge.
Table 15 lists the knowledge causes, priority, principle and recommendations. Following the
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 148
table, a detailed discussion for each high priority cause and recommendation and the literature
supporting the recommendation is provided.
Table 15
Summary of Knowledge Influences and Recommendations
Assumed Knowledge
Influence
Priority
High
Low
Principle and Citation Context-Specific
Recommendation
Factual
HPS know the various
approaches for obtaining
appropriate information in
order to communicating
shared data and
measurement practices to
community partners and to
executives. (Declarative
Factual)
High Encourage elaboration
and organization by
creating concept maps,
mnemonics, advance
organizers and
analogies (Aguinis
& Kraiger, 2009).
How individuals
organize knowledge
influences how they
learn and apply what
they know (Schraw &
McCrudden, 2006).
Provide HPS with
information via
information sharing
sessions to show them
how to utilize
knowledge and skills
that to communicate
information, which
could occur in-person
or via the eLearning
platform, LMS.
Job aids such as a data
map or visual organizer
would offer HPS to
engage in self-help
information to perform
a task.
Training would be
helpful to provide how-
to practice examples to
obtain and
communicate shared
data and measurement
practices. Trainings
could occur via the
eLearning platform,
LMS, or in person.
Conceptual
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 149
HPS are able to
understand the intersection
of dynamic information
and utilize multiple
dissemination and use of
knowledge and skill. HPS
should be able to transfer
their knowledge and skills
across tasks and
responsibilities in order to
communicate effectively
and meet the needs of
Executive Leadership and
community partners.
(Declarative Conceptual)
High Offer learners
sufficient tools and add
scaffolding to facilitate
understanding,
learning and
performance, then
gradually withdraw
scaffolds as learning
progresses and
performance improves
(Scott & Palinscar,
2013).
Provide HPS with
education to increase
their ability to establish
connections between
dynamic information.
Tasks that require job-
transferable skills, and
knowledge will be used
as exemplars.
Metacognitive
HPS reflect on the how the
information that they
communicate to Executive
Leadership and
community stakeholders
in order to identify
strengths and limitations
of their communication
processes and strategies.
(Metacognitive)
High Provide opportunities
for learners to debrief
the thinking process
upon completion of
learning task (Dori,
Mevarech, & Baker,
2017)
Training through
guided practice and
feedback will allow
HPS to practice
reflecting on strengths
and limitations of their
communication
processes and
strategies.
Declarative Factual Knowledge Solutions. Health Promotion Staff need to know what
shared data and measurement practices that HCG collects, what departments manage it, how to
access it, and how long the process takes in order to communicate this information to community
stakeholders and to executive leadership. HPS who felt that they effectively communicated
information reported feeling more “knowledgeable” when they had a “system” or “structure” to
classify or “organize information.” Aguinis and Kraiger (2009) suggest that organizing
information in a meaningful way, may help an individual quickly and easily locate and reference
it. They further, creating concept maps, mnemonics, advanced organizers and analogies helps an
individual to improve elaboration and organization. Other researchers agree suggesting the way
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 150
an individual organizes knowledge will influence how they apply what they know (Schraw, &
McCrudden, 2006). This would suggest that providing learners with an information, which is
presented in a data map or data organizer may provide a structured or schematic approach to
managing data. These are typically very helpful for people who are experts in an area and are
being asked to learn a new approach, but do not need training (Clark, & Estes, 2008). However,
when combined with training, learners are able to immediately put to use their newly acquired
knowledge and skills. Visually organized or displayed data helps learners to classify, store and
manage information and they typically contain self-help information to help learners perform a
task offering visual organizers as a recommendation in this instance, therefore, would help HPS
to better understand what data is available and where in HCG it is kept. One such platform to
help learners gain information could occur on the Learning Management Site (LMS), which is a
training system that HCG utilizes. HCG is a large organization, and as such, offers e-learning
through a centralized web platform. HCG has the ability to assign modules to staff based on
their job classification, department, and can track how long the learner participated in the
training, if they scored incorrectly or correctly on any quizzes, and track who has completed the
trainings to-date. One HPS indicated that they like using the LMS platform, because it helps to
communicate clearly and succinctly, and it occurs at the most convenient time and location for
the learner. Offering information sharing, and practice using the data, will help facilitate the use
of existing and available data. Some would argue that training results in high impact learning in
any context (Clark, & Estes, 2008). One benefit to training is that it is not context specific and
can be offered in any number of locations. It also may help HPS become more familiar with the
department(s) that manage it, and how to request it with previous experiences would support
their learning. Additionally, training would offer HPS time to practice using the visual data
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 151
organizer or data map as well as provide an opportunity for HPS learners to visualize the
important points to remember in a schema.
Findings from Chapter Four reveal that HPS do not understand how best accomplish their
role of communicating shared data and measurement practices. During online surveys, and in-
person interviews, HPS felt that they did not clearly understand what information to share. Next,
they indicated feeling like they should be prepared to offer information on a wide range of
competencies to optimize health outcomes among community stakeholders but indicated they
don’t consistently feel prepared (Rosenthal et al., 2011). Researchers have indicated that a
learner’s existing knowledge influences their learning. When existing knowledge increases by
incorporating new information, then learning occurs. Graphic organizers are argued to facilitate
the process by providing learners with a meaningful framework, which helps learners connect
their existing knowledge to the new information (Kim, Vaughn, Wanzek, & Wei, 2004). One of
the main benefits described by using visual organizers, is that they are designed to facilitate
teaching and learning of textual materials using “lines, arrows, and spatial arrangement” that
describes content, structure and key conceptual relationships (Kim et al., 2004). Additionally,
graphic organizers include semantic maps, semantic feature analysis, cognitive maps, story
maps, framed outlines, and Venn diagrams.
After the information sharing and review of the visual aids, HPS would be allotted time
practicing the use of this information. Allowing HPS to master by completing steps, then
organize the heuristics in a clear way, will help learners apply what they know (Schraw, &
McCrudden (2006). Similarly, Schraw and McCrudden (2006) explained that to develop
mastery, individuals must acquire component skills, practice integrating them, and learn when to
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 152
apply what they have learned. During professional development, factual knowledge
achievement was increased when new knowledge was practiced (Schraw, & McCrudden, 2013).
As such, the recommendation being made in this case study, is that HPS use a data map
or similar visual organizer, which would offer learners with the opportunity to organize
information. It will allow learners to clearly compartmentalize old and new information with
previous experiences to support their learning. In some instances, it will offer opportunities for
HPS to reference the goals of the organization as the HPS implements position specific goals.
Professional development, specifically a training opportunity, would provide HPS time to learn
information and then to gain practice working with the visual organizer to organize knowledge
and to work through the example scenarios. Time would be allocated to model, offer guidance,
coach, and scaffold learning (Kirschner, 2002; Kirschner, Sweller, J., & Clark, 2006).
Previous studies have discussed the priorities of government health promotion workers.
Researchers identified the need to focus on communication of information, or what is called
health informatics (Yasnoff et al., 2000). They reasoned the need to evaluate and improve
vulnerable points in the causal chain of communication and should consider the role of the
government in the context in which public health is practiced as a key stakeholder in the keeper
of information. The field of health informatics evaluates the information systems and with the
growing movement to use technology, one study identified a model to understand and
communicate more effectively and they suggested the need for “enhancing, speeding, and
simplifying the flow of information, whether that information is held by a human being or a
machine, and whether the information is a number, a picture, a sound, or another representation
of a fact or idea.” They identified three domains that the public health community should
consider in order to communicate more effectively: 1) human-to-human; 2) human-to-machine;
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 153
and 3) machine-to-machine (Friede et al., 1995). They explained the widely used information
system is that the Centers for Disease Control and Prevention called the AIDS Reporting System
used for AIDS surveillance data allows state and local health departments to input, verify, and
prepare reports via menus. They described lessons learned, and recommended developing and
implement standards for individuals, interfaces and equipment. This allows different
stakeholders to quickly retrieve and communicate data. In this study, HCG may consider
adopting better processes in mobile access to dashboards posting updated information; better
communications between channels between data teams and HPS and Executive Leadership and
the development menus with options to request shared data and measurement practices. In the
long term, HCG could consider methods to track how often specific data is being retrieved and
by whom to consider expanding standard shared data to HPS in their regions.
Shared measurement is essential to the ongoing efforts of HPS, and collaborative efforts
will remain superficial without integrating information across individuals and teams (Friede et
al., 1995). HPS field staff need to know the extent and capacities of shared data and
measurement practices, and how to acquire and communicate it, also called health informatics
(Barton, 2005; Eysenbach, & Jadad, 2001). When HPS are able to better understand what data
they may retrieve, they may be better equipped to share it; which would encourage more
collaborative problem-solving, and for an ongoing learning community that gradually increases
the effectiveness of all participants (Coiera, 2015).
Declarative Conceptual Knowledge Solutions. HPS should understand what shared
data and measurement practices are important to communicate to executives, as well as how to
organize knowledge influences how they learn and apply what they know so that it can be
communicated comprehensively but efficiently. Additionally, HPS should know how to transfer
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 154
learning across tasks and responsibilities in order to communicate effectively. This will facilitate
HPS’ ability to better understand and meet the needs of Executive Leadership and community
partners. In this study, HPS indicate lacking the ability to perceive how to incorporate the goals
of community partners and executives into their communication of shared data and measurement
practices, especially if they do not see the relationship between the goals of each party and
relevance to each other.
Scott and Palincsar, (2013) found that facilitating transfer promotes learning and
connecting individual’s independent performance level and their level of assisted performance
promotes optimal learning. Additionally, Schraw and McCrudden (2013) found that if
information is learned meaningfully and connected with related knowledge is stored more
quickly and remembered more accurately because it is elaborated with prior learning. Principles
behind these researchers would suggest providing learners with educational opportunities. As
Clark and Estes (2008) describe, knowledge and skill enhancement are required when people do
not know how to do a specific task, or when they anticipate future challenges during novel
problem solving. In this case, HPS are learning how to transfer existing knowledge into future
situations. Education chosen as the solution in scenarios when there is no need to provide “how
to” information, instead it is adopted when learners need research-based knowledge about the
causes for, and reasons why things happen. Similar to trainings, education can be implemented
in any location that is comfortable and easily accessible for both the learner and teacher.
In this study, HPS anticipate reviewing the goals of executives in combination with their
own performance goals in order to establish ways to communicate shared data and measurement
practices, that aligns with both goals. Education, which will offer learners with the conceptual
knowledge, thereby connecting HPS with goals and potential future challenges. They would also
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 155
suggest solutions such as referencing tools (i.e. organizational guides) to facilitate understanding
via schemas and models, and also providing scaffolding opportunities like working with
colleagues to solve problems to contribute towards understanding, learning and performance.
The next step would be to gradually withdraw scaffolds as learning progresses and performance
improves (Scott, & Palinscar, 2013; Schraw, & McCrudden, 2013).
As a component within the educational piece, a visual organizer could be added to
compliment the learning structure. The visual aid could be a communal spreadsheet, or online
database system as mentioned in the factual knowledge section above, would provide HPS the
opportunity to more clearly report, organize and interpret data. The next step would offer HPS
time to practice, during a training, using the visual organizer aid that shows, testing out the use of
different types of data and anticipate queries that may corresponding with specific information
from executives. Next, an interactive workshop would provide HPS with practical experience
communicating the data and solicit feedback and experience actual successes in the process.
During the training, HPS would work in teams and provide immediate feedback with
guidance, modeling, and coaching during their performance with reinforcement to reduce the
negative ideas that learners may experience when attempting a difficult task without the direction
they need to complete it (Scott, & Palinscar, 2013; Tuckman, 2007). Studies of learning
demonstrate that when learners gain feedback and experience successes on challenging tasks, it
positively influences the learner’s perceptions of competence, which helps reinforce what they
learn (Kirschner et al., 2006). Kirshner explains that providing learners with opportunities to
practicing communicating information derived from executive goals and receiving feedback
would foster opportunities to support their learning acquiring and communicating health data.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 156
HPS next need to know how to take the practiced, existing knowledge and transfer it into
similar occurring situations. Specifically, HPS need to practice the process of utilizing existing
information, combining it with new pieces or newly acquired information and communicating
information to community stakeholders, peers, and Executive Leadership. Given the role of
HPS, it is clear that they serve as frontline employees, working directly with clients, community
stakeholders, and community leaders. As such, they are responsible for communicating with a
variety of stakeholders in the field and are often required to apply existing knowledge to new
situations in real-life, immediate situations (Balcazar et al., 2011; Love et al., 1997). Together,
the education and training would offer HPS the sufficient tools with scaffolding to facilitate
understanding, learning and performance, then gradually withdraw scaffolds as learning
progresses and performance improves (Scott, & Palinscar, 2013).
To build out this example HPS may be familiar with health indicator, quality of life
(QOL) statistics for a population of Severe Mentally Ill (SMI). During a behavioral health
workgroup team meeting, HPS may be asked how to interpret QOL scores among those SMI
who are attempting suicides and where to obtain more information about the data. HPS would
need to know how to evaluate and analyze the QOL data frequency distribution scores among
SMI populations. Next, they would have to interpret the information for the workgroup along
with an anecdotal description for why the score is different compared to the average scores of
QOL among the SMI population. Even if this is the first time that an HPS is seeing the data,
they would have to understand QOL, what causes that it to go up/down, how to incorporate
previous knowledge about SMI populations, what they can infer about those attempting suicide,
and where to go to find additional information that would leverage the HPS’ conclusions.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 157
Previous studies have discussed the use of leveraging emerging health information and
technology systems to scaffold and reinforce job knowledge and skills and improve competency
needs (Barr-Walker, 2017; Blumenthal, 2009; Jacob at al., 2017; Joshi, & Perin 2012; Savel, &
CDC, 2012). Although HPS span different departments within HCG, these studies discuss the
fact that there are common information needs of HPS and there is tremendous diversity in public
health information, needs, and availability among public health practitioners; and diversity in
education and training of professionals. Proposed solutions suggest identifying a common
language with which to organize data, and next provide staff with practice piloting the common
language and use of the visual organizers (e.g. spreadsheets) in different worked examples.
Initially, HPS would be given the purpose of the learning activity, directions to follow and
learning goals expected to achieve. During trainings, HPS would engage with each other to
discuss a concept, problem or process, such as the unique needs of executives and community
partners to ensure understanding. Next, HPS would be given a simplified version of a task and
then gradually increase the complexity, difficulty or sophistication over time. Common
scenarios would be explored as a group beginning with worked examples and moving into
smaller work groups with unworked examples via problem-based learning (Hung, 2011;
Kirshner, 2002; Kirschner et al., 2006). Time would be allocated to provide performance
feedback before, during and after the learning with executives (Kirschner et al., 2006).
As conceptual knowledge (Anderson, & Krathwohl, 2001) includes knowledge of
classification and categories, schemas and mental models that individuals use to organize
information. HPS from HCG must understand that in order communicate comprehensive shared
data and measurement practices, they need to understand what data is the most urgent, relevant
and of value to executives’ goals, so that they can provide it, and learn the most effective and
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 158
efficient way to do so. Some HPS admitted that although they have factual knowledge of what
they need to communicate, they did not rank them self as extremely effective in the
communicating data and shared measurement practices. Because of this, HPS may have omitted
communicating information about certain topics, however communicating too much information
about other topics without realizing that the appropriate information to share between community
partners and executives.
HPS need to know what information is available to communicate and practice effectively
and efficiently translating it when needed into new areas, and then communicating it to
community stakeholders and to Executive Leadership. In this study, HPS explained not
understanding what information should be transferred to their executives, and it is important for
them to understand how best fulfill their roles as communicating shared data and measurement
practices and as health promotion staff (Rosenthal et al., 2011). Scott and Palinscar (2013)
suggest that offering learners sufficient tools and add scaffolding to facilitate understanding,
learning and performance, then gradually withdraw scaffolds as learning progresses and
performance improves. Therefore, during trainings, conceptual knowledge achievement is likely
to increase when new knowledge is practiced along with appropriate feedback. As such, the
recommendation being made in this case study, is that HPS may benefit from using education to
provide conceptual, theoretical and strategic information, a visual organizer, which would offer a
way to structure organizational goals, executive goals, and community partners’ goals, and a
training to provide feedback. Training would provide HPS time to transfer existing knowledge
into new or slightly different scenarios. HPS would have time in a training to practice working
through scenarios, beginning with worked examples moving into unworked examples via
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 159
problem-based learning (Hung, 2011; Kirshner, 2002; Kirschner et al., 2006). Time would be
allocated to model, offer guidance, coach, and scaffold learning (Kirschner et al., 2006).
Metacognitive Knowledge Solutions. Metacognitive knowledge refers to one’s ability
to reflect or be aware of their own cognition (Baker, 2006). An example of metacognition in this
project, is when HPS reflect on the how the information that they communicate is transferred
from HPS to Executive Leadership; or on the strengths and limitations of the communication
processes. HPS should be able to effectively reflect on the how the information that they
communicate to Executive Leadership and community stakeholders is strong, or where there are
limitations in their personal communication processes and strategies. Moreover, HPS need to
have the time and resources in place to allow them to identify prior knowledge, including what
they know and what they do not know about a topic before a learning task, next having
opportunities to engage in guided self-monitoring and self-assessment (Dori, Mevarech, &
Baker, 2017). Schraw & McCrudden (2013) found that information learned meaningfully and
connected with prior knowledge is stored more quickly and remembered more accurately
because it is elaborated with prior learning. Additionally, Dori et al. (2017) found that providing
opportunities for learners to debrief the thinking process upon completion of learning task will
help reinforce what they learn. This would suggest that providing learners with opportunities to
practice new skills, integrating them, knowing when to apply what they have learned, and finally
offering learners the opportunity to debrief and discuss the thinking process with others upon
completion of learning task will to support their learning communicating shared data and
measurement practices. The solution then for HPS might be to provide them with a baseline
questionnaire to evaluate their own prior knowledge by creating a list of concepts and skills they
are expected to learn and have them assess their own level of understanding on those concepts
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 160
and skills; and then conduct a pre/post assessment using different levels of knowledge,
incorporating connections to prior knowledge and applications of knowledge. At the conclusion,
HPS would be able to talk about their self-evaluation or the HPS’s judgments of learning are
compared to actual performance, potentially using a questionnaire such as the Schraw and
Dennison (1994) Metacognitive Awareness Inventory. A solution could include that during a
professional development session, HPS would be allotted time to break down complex tasks and
encourage individuals to think about content in strategic way (Baker, 2006). Strategies should
combine the process of communication shared data and measurement practices between
community partners, peer HPS staff and executives, as a function of community practice and
informatics competencies. HPS would spend time designing ideal procedural information,
analyze the process including the number of steps, challenges, and opportunities and the
prerequisite knowledge (Van Merriënboer, & Kirschner, 2017).
In studies evaluating learning, metacognitive activities have been shown to correlate with
knowledge acquisition, skilled performance on a transfer task, and self-efficacy (Ford, Smith,
Weissbein, Gully, & Salas, 1998). Implications of these findings for research and practice
indicate that metacognitive activities promote self-control of learning or self-directed learning.
Therefore, participating in training would provide time for metacognitive activities could
improve the learning, self-efficacy and skilled performance of task transfer, among HPS. This
could be beneficial to HPS who often have unique and different responsibilities from colleagues
and have to engage in self-directed learning. In a 2015 systematic review, self-regulated learning
strategies were identified including the correlation with time management, metacognition, effort
regulation and critical thinking, which predicted performance outcomes (Broadbent & Poon).
Additional research evaluating metacognition and the retention of science text comprehension
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 161
suggested that that metacognitive self-regulation components enhances students’ performance
when cognitive strategy training was not effective by itself (Leopold, & Leutner, 2015). Overall,
findings indicate that metacognition is linked to performance achievement, and self-regulated
learning. However, an important finding was that traditional in-person, and peer driven
platforms were more effective than online learning platforms (Broadbent & Poon).
As such, the recommendation being made in this case study, is that HPS may benefit
from tracking baseline knowledge to evaluate their own prior knowledge and have them assess
their own level of understanding on those concepts and skills towards the end of a training
curriculum. As a measure of implementation and performance outcomes, HPS would be able to
talk with peers and self-evaluate their performance. HPS must understand and practice how to
evaluate their knowledge, how to implement communication strategies, and identify areas for
improved performance, as well as identify accomplishments.
Motivation Recommendations
Introduction. The motivation influences in Table 16 represent the assumed motivation
influence that was validated based on the most frequently mentioned motivation influences to
achieving the stakeholders’ goal during the online survey, in face-to-face interviews and
supported by the literature review of motivation theory. The assumed motivational influence that
was validated is the self-efficacy construct of motivation. The table includes the goals of the
HPS role and the approaches to communicate data and shared measurement practices. The
principles related to these influences surround the topic self-efficacy, which have been explored
by Pajares & Schunk (2001). They indicate that self-efficacy is related to the theory of personal
and collective agency that operates in correlation with the self-beliefs, the processes that self-
beliefs operate and models that can be strengthened. Previous studies have found that self-
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 162
efficacy is related to clinical problems including phobias, addiction, depression, social skills,
assertiveness, stress, smoking behaviors, pain management, and health and athletic performance
(Pajares, 1997). In the field of learning, self-efficacy has also been found to correlate with
student’ academic performance and achievement, and constructs including: attribution, goal
setting, modeling, reward contingencies, self-regulation, problem solving, expectancy and
strategy training (Pajares, 1997). Self-efficacy is described as the byproduct of self-beliefs,
experiences and perceptions, suggesting that interpretations of an event, situation or experience
helps establish beliefs about one’s capability, knowledge, and skills to undertake ensuing
behaviors in similar domains. As a result, future attempts are in part the byproduct of what the
individual has come to believe that they have achieved and can achieve. Researchers have
suggested that these kinds of self-beliefs may play a mediational role in cognitive engagement
and that increasing them may lead to increased performance (Zhao, Seibert, & Hills, 2005). One
such solution is to use models that build self-efficacy and enhance motivation (Pajares, 1997).
This could potentially look like exchanges where the learner receives feedback and modeling to
increase self-efficacy (Pajares, 1997). Providing learners with feedback on progress in learning
and performance can greatly enhance learning and performance (Pajares, 1997). Furthermore,
setting initial assessments at current levels of self-efficacy, then gradually increasing challenges
of tasks combined with feedback and successful completion of tasks improves self-efficacy
(Brewer & Brewer, 2010).
This section will explain the relationship between the goals of HPS as learners and
motivation. One index will be evaluated, which is related to motivation, self-efficacy, as it
applies to HPS, in that it helps to describe, explain, and predict motivation. Clark and Estes
(2008) suggest that there are three indicators of motivation in task performance – choice,
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 163
persistence and mental effort, and when one or more of these components are missing, there is a
lower probability for a learner to learn new information. As such, as indicated in Table 16, some
motivational influence, self-efficacy among HPS were validated and have a high priority for
achieving the stakeholders’ goal. Table 16 also shows the recommendations for the motivational
influence based on theoretical principles.
Table 16
Summary of Motivation Influences and Recommendations
Assumed Motivation
Influence
Priority
High
Low
Principle and Citation Context-Specific
Recommendation
Self-Efficacy
HPS needs to be
confident about their
ability to communicate
shared data and
measurement practices
among community
partners, peer HPS and
Executive Leadership
(Self-efficacy).
High Use models that build
self-efficacy and
enhance motivation
(Pajares, 2005).
Feedback and
modeling increases
self-efficacy (Pajares,
2005).
Training and routine
check-ins with peer or
mentors to offer HPS
with goal-directed
practice with frequent,
targeted and private
feedback on progress and
performance.
Self-Efficacy Solutions. HPS are not confident about their ability to that they can
communicate shared data and measurement practices among community partners, peer HPS and
Executive Leadership. In 2005, Pajares and Schunk found that goal-directed practice with
frequent, accurate and private feedback on learning and performance, can positively influence
self-efficacy. Additionally, Pajares and Schunk (2005) found that providing instructional
support via scaffolding early on and building in multiple opportunities for practice while
gradually removing supports increases self-efficacy. This would suggest that providing learners
with similar HPS peers and designating a schedule to routinely check-in and discuss current
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 164
projects would increase their self-efficacy. An example of self-efficacy in this study, is that HPS
did not strongly agree to being confident in their ability to communicate with Executive
Leadership. This suggests that is HPS have opportunities to increase their confidence in
communicating with Executives. Literature would suggest that potentially HPS could participate
in activities where they learn something that is being taught or are capable of performing a task,
and then check in frequently with peers to discuss progress (Dori, Mevarech, & Baker, 2017).
Bandura (1977) stated that self-efficacy is personal judgments of one’s capabilities to
organize and execute courses of action to attain designated goals, suggesting that when
individuals have positive beliefs about their capability to do something they are more likely to
pursue and achieve the goal. Zimmerman, Bandura, & Martinez-Pons, (1992) described that
higher levels of self-efficacy may predict higher levels of achievement, and choice, effort, and
persistence. Ajzen (2002) emphasized that self-regulated learners require a number of
characteristics such as the ability to goal set, plan, select strategies, self-monitor and self-
evaluate, and that underlying these is the need for high self-efficacy for learning. Ajzen (2002)
furthered that self-efficacy is a good indicator of learner performance and may foster
independent thinking for problem solving in professional practice. Ajzen then reported that in
order to facilitate student self-efficacy, the teacher should provide praise focused on student
efforts (not just the outcomes), and attribute successes to level of effort and use of effective
strategies, rather than abilities. From a theoretical perspective, then, it would appear that
increasing self-efficacy among HPS would increase performance and benefit health outcomes of
related community stakeholders and partners working with them on projects, programs, practices
and policies.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 165
Manojlovich (2005) studied self-efficacy among other variable to examine the
relationship predictors of professional nursing behaviors. They found that self-efficacy partially
mediated the relationship between structural empowerment and professional practice, suggesting
that the importance of increasing self-efficacy. Manojlovich concluded that leadership should
provide opportunities to enhance self-efficacy through communicating with colleagues via
regular check-ins or in role modeling, and by engaging in positive verbal feedback and
persuasion in order to improve practice behaviors. In a 2007 study evaluating motivation on
student achievement and taking responsibility for learning and teaching, Eccles studied the
importance of doing well on a task (Eccles, 2007). Eccles discussed that when someone is
interested or has a high level of self-efficacy and believe that they can do a task, as well as feel
that the task has utility and find the task enjoyable, the learners is more likely to achieve
academic performance. In a study evaluating motivation, Deci, Koestner, & Ryan, (1999)
explain that as motivation is translated into practice, several components help increase
performance: increasing self-efficacy, providing accurate feedback, creating realistic
expectations of competence, discussing the utility of work, modeling value and interest of work
and activating learners’ personal interest through opportunities for choice and control. From a
theoretical perspective, then, it would appear that increasing self-efficacy among public HPS
would increase performance.
During professional development via training and then routine check-ins with partners
and or mentors, HPS will gain experience exchanging experiences, knowledge and skills, and
become more confident in their ability to communicate information and become more confident
communicating shared data and measurement practices. The recommendation is for HCG, as the
organization, to provide instructional support to HPS, then offer goal-directed, small peer groups
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 166
or mentors, in order to provide HPS with ongoing support and frequent feedback on their
performances.
HPS identify and utilize appropriate acquisition and communication strategies of shared
data and measurement practices. The training, peer groups, or mentorship experiences, will offer
learners with demonstrations that individuals are capable of learning what is being taught or are
capable of performing a task. Private peer groups, or mentorships with more senior colleagues
would also provide HPS time to practice working through common scenarios and receive
confidential, and tailored evaluations of work (Hung, 2011; Kirshner, 2002; Kirschner et al.,
2006). Time would be allocated to schedule and participate in an annual training, then connect
with a small peer group or a role model, exchange insight, guidance, coach, and support through
goal-directed activities (Kirschner et al., 2006).
Organization Recommendations
Introduction. The organizational influences in Table 17 represent the complete list of
assumed organizational influences that were that were validated based on the most frequently
mentioned organizational influences to achieving the stakeholders’ goal during the online survey,
in face-to-face interviews and supported by the literature review. One assumed organizational
influence that was validated is the lack of policies as a construct of organizational influences.
The table includes the goals of HPS role and the approaches to communicate data and shared
measurement practices.
In this study, HPS surveyed and interviewed agreed that there were policies in place that
guide the design and implementation of the CI initiative that start from the top level. HPS
additionally indicated that this information does cascade down and impact policies, and
procedures, however, these are not standardized across the HCG. Specifically, HPS explained
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 167
that these policies are not in place to allow communication to be convenient and easily accessible
in order to translate into everyday practice, and that it is not listed as a category on their annual
performance review so as to track over time and gauge improvements or to provide feedback.
Clark & Estes (2008) suggest that a key element for success is when there is a connection
between a compelling vision, business practices to reach the goal, clear roles and responsibilities,
motivational support and assessment.
In this study, HPS also indicated that they do not have strong agreement that they have
appropriate professional development or mentorship opportunities. Clark & Estes (2008) suggest
that organizational performance increases when organizational messages, rewards, policies and
procedures that govern the work of the organization are aligned with or are supportive of
organizational goals and values (Clark & Estes, 2008). This suggests that when HPS processes
are aligned with organizational processes, the performance among staff such as HPS would
likely increase. Additionally, Clark and Estes (2008) state that when policies and procedures are
aligned and communicated from the top leadership with all stakeholders, then organizational
performance increases, suggesting that top management should continually be involved in the
improvement process.
In the case of HPS and community health outcomes, organizational performance has high
human stakes. Hicks, Larson, Nelson, Olds, & Johnston (2008) describe that stakeholders must
feel secure that all involved in the process have equal opportunity to directly influence decisions
and that decisions are likely to have some impact on the root problem the participants are
addressing. Friede at al., (1995) explain that if public health organization’s effectiveness and
profile are to grow, practitioners and researchers will need reliable, timely information with
which to make information-driven decisions, better ways to communicate, and improved tools to
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 168
analyze and present new knowledge. They continue taking time to clearly define the
organizational problem, set target goals, establish a plan for reaching those goals, and reaching
agreement as to the roles each stakeholder will take in executing a plan. As such, it appears that
the literature would support the necessity for improving communication for the benefits of both
HPS and community stakeholders by conducting an informal audit of policies, procedures and
messages to check for alignment or interference with goals.
As indicated in Table 17, these influences were validated and have a high priority for
achieving the stakeholders’ goal. Table 17 also shows the recommendations for these highly
probable influences based on theoretical principles. Policies will be defined, explained and the
application to the stakeholder goal and the assumed influence will be explored. Moreover, the
literature about this domain/topic will be detailed as it relates to the multiple types of
organizational influence. Table 17 lists the organizational causes, priority, principle and
recommendations. Following the table, a detailed discussion for policy, or the priority
organizational cause and recommendation and the literature supporting the recommendation is
provided.
Table 17
Summary of Organization Influences and Recommendations
Assumed Organization
Influence
Priority
High
Low
Principle and Citation Context-Specific
Recommendation
Policy
HPS need to have
policies, tools and
resources to accomplish
performance goals.
(Policy)
High Effective
organizations ensure
that organizational
messages, rewards,
policies and
procedures that
govern the work of the
HCG will need to
conduct an informal audit
of existing
policies, procedures and
messages
to check for alignment or
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 169
organization are
aligned with or are
supportive of
organizational goals
and values (Clark and
Estes, 2008).
interference with
organizational and
stakeholders goal, as
effective organizations
have clear vision, goals,
and ways to measure
progress.
Policy Solutions. HPS feel that there are policies in place that support communicating
shared data and measurement practices, however, they do not feel that these are standardized
across the HCG, or reflected in the annual performance reviews, nor that there are opportunities
for training development and mentorship opportunities. Clark & Estes (2008) suggest that
effective organizations have clear vision, goals, and ways to measure progress. They further that
evidence based, collective solutions are adapted to improve the organization’s policies, culture,
and embrace improved communication that is candid and thorough. This suggests that HPS need
to be a part of an organization that has clear vision, goals, and ways to measure progress in
addition to policies which support and encourage improved performance through open and
standardized communication practices.
Thompson (2010) indicated that managing organizational change within public health
organizations is important because appropriately and systematically managing change is linked
to improved organizational performance. As such, it appears that the literature would support the
necessity for addressing existing policies of HPS as the CI initiative has changed the
organizational structure and system so as to benefits of HPS, community partners and health
outcomes among residents across the region. In a publication on innovation in public service
sectors, the drivers were described as improving service performance, efficiencies and public
value (Hartley, 2005). These improvements were advanced through collaboration with staff and
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 170
comparable organizations, and facilitating the sharing, or spreading of ideas, ideals, good
practice, and attempting to translate best practices into the organization. Clark and Estes (2008)
suggest that organization and stakeholder goals are often not achieved when stakeholder goals
are not aligned with the organization’s mission and goals. These studies suggest that policies
and processes must align throughout the organization’s structure to achieve the mission and
goals. Furthermore, Meyer, Davis, & Mays (2012) define organizational capacity for public
health services and systems research as a critical determinant of performance and is necessary for
understanding systematic effectiveness, sustainability, or generalizability in public health
services and systems research. As such, as indicated in Table 17, one organizational influence
was validated and has a high priority for achieving the stakeholders’ goal. Table 17 also shows
the recommendations for these influences based on theoretical principles.
Summary of Knowledge, Motivation and Organization Recommendations
Five gaps were identified in this study. The first gap was in the area of factual
knowledge. During in-person interviews, HPS indicated that they experienced discrepancies in
understanding about what information exists that can be communicated as shared data and
measurement practices, including what “supervisors” and “upper management” want to know,
and “how can we most effectively communicate it”. Additionally, face-to-cafe interviews
revealed that there were significant lacks in factual knowledge about what information the HCG
statistics and data have to offer community partners and organizations. Thus, the results of the
online survey and interviews validated that HPS lacked factual knowledge. The assumed
knowledge influence is that HPS need to know the various approaches for obtaining information
in order to communicating shared data and measurement practices to community partners and to
executives. The principles that would help overcome this gap by Aguinis & Kraiger (2009) is to
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 171
encourage elaboration and organization by creating concept maps, mnemonics, advance
organizers and analogies (Aguinis & Kraiger, 2009), as individuals organize knowledge
influences how they learn and apply what they know (Schraw & McCrudden, 2006). The
recommendation to close this gap is to provide an information sharing session with HPS to
demonstrate how to utilize knowledge and skills that to communicate information, which could
occur in-person or via the eLearning platform, LMS. Additionally, job aids such as a data map
or visual organizer would offer HPS to engage in self-help information to perform a task.
Finally, a training would be helpful and provide “how-to” practice examples to obtain and
communicate shared data and measurement practices. Trainings could occur via the eLearning
platform, LMS, or in person.
The second gap was in the area of conceptual knowledge. The assumed knowledge
influence is that HPS are able to understand the intersection of dynamic information and utilize
multiple dissemination and use of knowledge and skill. HPS should be able to transfer their
knowledge and skills across tasks and responsibilities in order to communicate effectively and
meet the needs of Executive Leadership and community partners. For example, when HPS can
make connections between organizational goals, the needs of executives, and individual goals,
then this transfer promotes an environment of transparency, which promotes optimal learning
(Scott, & Palincsar, 2013). Offering learners sufficient tools and add scaffolding to facilitate
understanding, learning and performance, then gradually withdraw scaffolds as learning
progresses and performance improves (Scott, & Palinscar, 2013). The recommendation to close
this gap is to offer HPS with education to increase their ability to establish connections between
dynamic information, then have them participate in tasks that require job-transferable skills, and
knowledge base.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 172
The third gap was in the area of metacognitive knowledge. The assumed knowledge
influence is that drawing connections that facilitates transfer will ultimately promote learning.
The principles that would help overcome this gap include using metacognitive strategies
facilitates learning among HPS (Baker, 2006). For example, when HPS have opportunities to
reflect on their performance and evaluate progress in meaningful ways, this promotes optimal
learning (Scott, & Palincsar, 2013). The recommendation to close this gap is to offer HPS
training through guided practice and feedback will allow HPS to practice reflecting on strengths
and limitations of their communication processes and strategies. This will offer learners s place
to discuss and engage in a peer think “aloud” process upon completion of learning task (Dori et
al., 2017). Professional development will allow HPS to break down complex tasks and
encourage individuals to think about content in strategic ways (Schraw, & McCrudden, 2013).
The fourth gap was in the area of self-efficacy motivation. The assumed motivational
influence is that HPS needs to be confident about their ability to communicate shared data and
measurement practices among community partners, peer HPS and Executive Leadership. The
principles that would help overcome this gap include using models that build self-efficacy and
enhance motivation (Pajares, & Schunk, 2005). Feedback and modeling has been shown to
increase self-efficacy (Pajares, & Schunk, 2005), so this may be one such approach. Training
and routine check-ins with peer or mentors to offer HPS with goal-directed practice with
frequent, targeted and private feedback on progress and performance. The recommendation to
close this gap is to offer HPS peer feedback, with regular check-ins, and guided by goal-directed
practice to demonstrate that individuals are capable of learning what is being taught or are
capable of performing a task (Dori et al., 2017). Practice with feedback will encourage
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 173
individuals to increase attempts to troubleshoot and gain valuable lessons from expert peers
(Schraw, & McCrudden, 2013).
The fifth gap was in the area of policies in the organization. The assumed organizational
influence is that HPS needs to HPS need to have policies in place to accomplish performance
goals. The principles that would help overcome this gap include that effective organizations
ensure that organizational messages, rewards, policies and procedures that govern the work of
the organization are aligned with or are supportive of organizational goals and values (Clark, &
Estes, 2008). For example, when HPS have policies in place that recognize them in their annual
performance review, this increases the level of accountable for accomplishing specific tasks.
The recommendation to close this gap is to have HCG conduct an informal audit of existing
policies, procedures and messages to check for alignment or interference with organizational and
stakeholders goal, as effective organizations have clear vision, goals, and ways to measure
progress (Schraw, & McCrudden, 2013).
To summarize, solutions for the most critical KMO gaps identified in the research
questions call for HPS to be provided with information, job aids such as visual organizers (data
map, training e.g. guided practice), feedback and completing an audit. The suggested solutions
are based on principles of learning, motivation and organizational-related theories. The
implementation plans incorporate project specific solutions to close the gaps, which is then
managed with evaluation. Kirkpatrick’s Four Levels of Evaluation (2016) will be explored in
the next section, as an effective process by which to evaluate the implementation plans.
Combining the literature review, which suggests solutions to close the gaps in knowledge,
motivation, and organizational influencers, and through the validation for the causes, specific
recommendations for implementation at HCG are made.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 174
Integrated Implementation and Evaluation Plan
Organizational Purpose, Need and Expectations
For the last decade, HCG has been implementing the CI model to promote collaboration
towards improving health promotion and disease prevention efforts. This initiative aimed to
promote health, safety and social well-being. In their role as a backbone organization, they
stepped forward as the influential champion within the CI model. The mission of HCG is its
commitment to a service delivery system that is regionalized and accessible, community-based
and customer oriented. Healthy County Government aims to protect the health of residents by
providing essential health services, especially for those who are least able to help themselves.
The CI initiative has helped Healthy County Government to engage with community leadership
in outcome-driven partnerships to meet the unique needs of the diverse communities and
neighborhoods. The goal of HCG is to redefine the role of government as a steward of health,
safety and wellness and reduce the percentage of preventable disease. HCG is interested in
taking a systems-wide approach to improving health, through the CI initiative by impacting the
range of social determinants of health by working with community partners, community
stakeholders, and local municipalities to exchange resources and information (e.g. active
transportation routes, public safety, health, human services, probation and juvenile delinquency
services). Executive Leadership would like to strengthen the use of the CI model by measuring
their utilization and impact of the shared agenda, shared measurement system, mutually
reinforcing activities, continuous communication, and a central infrastructure, which comply
with the five components outlined in CI models (Kania, & Kramer, 2013). Moreover, HCG has
explicitly outlined the need for accurate and effective information from the field.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 175
HCG employs a number of HPS, of the stakeholder of choice in this study, who
specialize in community health promotion and CI efforts. Their goal is to communicate shared
data and measurements as outlined in the CI backbone organization in order to help inform
policy decisions about the health and well-being of local residents. This ties to the
organizational goal because HCG believes the preventable disease related deaths will decrease or
improve when they are able to fully implement a CI model for population health. This model is
dependent on timely, accurate and comprehensive exchange of health data. When CI is properly
implemented, it aims to eliminate duplicative efforts and enhance impact of an issue by working
from multiple perspectives simultaneously (Laverack, 2007; McDermott, 2010). In order to
better understand the process of how Healthy County Government is working with community
partners to communicate shared health data and measurement practices, this study will explore
how data is communicated to determine if there is potential to improve effectiveness and quality
of outreach efforts.
The problem of practices is that little is known about the process of how staff from HCG
communicate shared health data and measurement practices among community partners. As
such, HCG has set the goal to evaluate the effectiveness of health data and measurement
practices communication between community partners and government systems. Executive
Leadership from Healthy County Government would like to gain a better understanding of how
HPS can communicate comprehensive information with community partners, which is a key
responsibility of a backbone agency. Today, Executive Leadership would like to strengthen the
CI efforts and evaluate HPS’ CI efforts in a unified manner and evaluate existing
communication.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 176
Within the CI model, backbone organizations are responsible for leading sustainable,
long-term, systemic changes throughout a system or community (Turner et al., 2012). Current
health efforts require coordinated and comprehensive strategy among traditional and
nontraditional partners. The expectations for the desired outcomes or results of the
aforementioned recommendations for the stakeholder include that HPS advance progress in the
systems-wide approach to improving health, and impact the broader range of social determinants
of health by working with community partners, community stakeholders, and local
municipalities to exchange resources and information (e.g. active transportation routes, public
safety, health, human services, probation and juvenile delinquency services). By improving the
communication of data and shared measurement, they will be able to more objectively reduce the
percentage of preventable disease related-deaths and improve the health of residents through CI
efforts.
Implementation and Evaluation Framework
In order to close the gaps, this section will describe the implementation and evaluation
plan using the New World Kirkpatrick Model (Kirkpatrick, & Kirkpatrick, 2016), which calls for
recommendations to be tailored to the unique needs of the organization, supported by
organizational drivers, and evaluated for ongoing progress. Using this framework, HCG will
have the opportunity to implement and evaluate solutions that impact HPS goals, and advance
towards meaningful change. The goal is for HPS to transfer what they learn into their current
roles and improve job performance. The framework has four levels - the first starts with the end
goal, or the results in mind since the return on expectation is the key indicator of value. The
results should demonstrate a compelling chain of evidence, or leading indicators that contribute
to the bottom line, which in this study, is the mission of HCG. The second level is the behavior,
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 177
which describes the degree to which participants apply what they learned during training and are
back on the job; which can be evaluated by critical behaviors and required drivers, which have
the biggest impact on desired results. The third level is the learning, or the degree to which
participants acquire the intended knowledge, skills, attitude, confidence, and commitment based
on their participation in the training. The final level is the reaction, or the degree to which
participants find the training relevant to their jobs and beneficial to their work.
The main nuance between the New World Kirkpatrick Model and the historic model is
that the New World model starts with measures of success, or results as the first level;
additionally, other constructs are added to improve clarity and interpretation, such as leading
indicators, required drivers, engagement and relevance. In the older model, the levels were
ordered in a different sequence starting with reaction, then moving to learning, behavior and
finally results. While the four levels are essentially the same between both versions, again the
order is in a different progression. In the newer model, the emphasis is placed on return on
expectations of stakeholders. Additionally, there are small components added to new model,
including: engagement and relevance adding to customer satisfaction in the Level 1: reaction;
confidence and commitment adding to knowledge, skill and attitude in the Level 2: learning;
required drivers adding to reinforce, encourage, monitor, and reward in Level 3: behavior; and
leading indicators adding to desired outcomes in Level 4: results. The new model was developed
to correct misuse and misinterpretation of the model, as well as update the application in a
modern context (Kirkpatrick, & Kirkpatrick, 2016). The New World Kirkpatrick Model
framework was selected to close the gap because it incorporates evaluation strategies that require
the HCG mission, goals, and problems be integrated into the intervention plan. Next, it allows
program designers to consider for restrictions such limited resources and offers suggested
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 178
approaches that help bring about the most impact (Kirkpatrick, & Kirkpatrick, 2016).
Recommendations in this study include four components: (1) information, (2) visual aids
organizers, (3) professional development, via training, scaffolding, offering monitoring and
feedback, and (4) completing an audit. This framework will guide the design and evaluation of
these four components to be value driven and results oriented.
Level 4: Results and Leading Indicators
The first level in the new Kirkpatrick model (2016) is the Level 4 result, which is
described by Kirkpatrick and Kirkpatrick as the targeted program outcomes that should occur as
a result of the intervention and contribute to the organization’s highest-level result, or Return on
Expectations, ROE. Since the true Level 4 result can take time to manifest, leading indicators
are employed to help to keep initiatives on track and to reassure stakeholders that the program is
contributing to key organizational goals. The leading indicators are considered the short-term
observations and measurements. There are two types of leading indicators - internal and external,
and if one were to climb up a mountain, each indicator would move one step closer to the peak,
or the desired result. Internal indicators are related to individual, team, department or
organizational outcomes and include: employee satisfaction, quality or errors on the job,
compliance or audit findings, cost, production volume, safety on the job and efficiency to
complete tasks. External indicators are related to customer, client, market or industry response
to the intervention or training and include: customer response, customer satisfaction, and market
response. See the Table 18 for examples of outcomes, metrics and methods for external and
internal outcomes that contribute towards HCG’s organizational goal of improving health, safety
and well-being for local residents, while reducing the percentage of preventable disease related-
deaths through the existing CI efforts.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 179
Table 18
Outcomes, Metrics, and Methods for External and Internal Outcomes
Outcome Metric(s) Method(s)
External Outcomes
Improve community
stakeholders' satisfaction
with availability of shared
data and measurement
practices.
(Customer Satisfaction)
The customer satisfaction
surveys that stakeholders
complete, at a given time.
Solicit customer satisfaction
data from community partners
in the field.
Improve existing
community stakeholders’
use of shared data and
measurement practices.
(Customer Response)
The number of community
stakeholders who request use of
shared data and measurement
practices, at a given time.
Track the number of returning
users requesting for shared data
and measurement practices.
Increase the number of
new community
stakeholders that use
shared data and
measurement practices.
(Market/Industry
response)
The number of community
stakeholders who request use of
shared data and measurement
practices, at a given time.
Track the number of requests
for shared data and
measurement practices.
Improve community
stakeholders’ perception
of the CI initiative and
attitude towards HCG’s
shared data and
measurement practices.
(Customer Satisfaction)
The customer satisfaction
surveys that stakeholders
complete, at a given time.
Solicit customer satisfaction
data from community partners
about the CI initiative and
attitude towards HCG’s role.
Improve community
stakeholders’ perception
of the HPS as resourceful
HCG staff that can
provide shared data and
measurement practices.
(Customer Satisfaction)
The customer satisfaction
surveys that stakeholders
complete, at a given time.
Solicit customer satisfaction
data from community partners
about the CI initiative and
attitude towards HPS’ role.
Internal Outcomes
Improve number of
completed requests for
shared data and
measurement
practices.(Volume)
The number of completed
requests for shared data and
measurement practices.
Aggregate data tracking the
number of requests for shared
data and measurement practices.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 180
Improve HPS efficiency
in processing new
requests for shared health
data and measurement
practices. (Efficiency)
The number of days between the
request for data and provision of
data.
Aggregate data from HPS
regional managers via self-
report and supervisor
confirmation collected via
visual organizing system.
Increase employee
satisfaction/confidence in
delivering shared health
data and measurement
practices to community
stakeholders, and
executive leadership.
(Employee Satisfaction)
Positive/negative feedback from
employee on the topic of
satisfaction and confidence in
delivering shared health data and
measurement practices to
community stakeholders and
executive leadership.
Responses on monthly one-on-
one meetings to discuss
feedback about employee
performance with supervisors.
Level 3: Behavior
Critical behaviors. Level 3 behavior is “the most important level” in the model,
according to Kirkpatrick and Kirkpatrick (2016). They suggest that participants should be held
accountable for following through in activities learned and practices in trainings. These
activities are called critical behaviors, which is considered a comprehensive, continuous
performance monitoring system, that assists people in performing tasks. The “critical behaviors”
are important because if they are performed reliably, they have the capacity to have the biggest
impact on the targeted program outcomes. Post training, training outcomes are evaluated
including, what resources are employed and whether the training encourage new behaviors and
skills. Critical behaviors should be defined in terms that connect the activities to outcomes. An
example of a critical behavior is to attend monthly community stakeholder’s meetings and
provide reports to document current projects, current data, and future projects, so as to
potentially anticipate data needs.
In this study, critical behaviors are the necessary activities that HPS field staff must
complete to achieve performance goals. The three critical behaviors will be measured related to
communicating shared data and measurement practices to determine whether shared data and
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 181
measurement practices are being communicated between community stakeholders and Executive
Leadership. The three critical behaviors are to:
1. Prepare monthly status report with shared data and related information needs of
community stakeholders and executive leadership.
2. Conduct monthly team meetings that include all reports documenting shared data
and related information needs of community stakeholders and executive
leadership with current status and require actions.
3. Conduct monthly group discussions for HPS to provide time to think “aloud”
about current process and potential areas for improvement pertaining to HPS
goals and HCG’s goal.
See Table 19 for examples of critical behaviors, metrics, methods and timing for
evaluation that contribute towards HCG’s organizational goal of improving health, safety and
well-being for local residents, while reducing the percentage of preventable disease related-
deaths through the existing CI efforts.
Table 19
Critical Behaviors, Metrics, Methods, and Timing for Evaluation
Critical Behavior Metric(s)
Method(s)
Timing
Prepare monthly
status report with
shared data and
related information
needs of community
stakeholders and
executive leadership.
Descriptive data
quantifying shared
data and related
information needs of
community
stakeholders and
executive leadership.
HPS manager shall track
for completeness the
data
tracking system, looking
for errors or feedback
from other reviewers
about major flaws.
Report to HPS
managers –
monthly.
Conduct monthly
team meetings that
include all reports
documenting shared
data and related
information needs of
Meeting minutes and
data tracking of
monthly discussion
including status and
action steps to
quantify shared data
HPS manager shall
facilitate team
discussion on the topic
of data and related
information needs of
stakeholders facilitating
Report to HPS
managers –
monthly.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 182
community
stakeholders and
executive leadership
with current status
and require actions.
and related
information needs of
community
stakeholders and
executive leadership.
feedback on the status
including action steps.
Conduct monthly
group discussions
for HPS to provide
time to think
“aloud” about
current process and
potential areas for
improvement
pertaining to
documenting shared
data and related
information needs of
stakeholders, HPS
goals and HCG’s
goal.
Meeting minutes and
data tracking of
monthly discussion
including overview of
current processes and
potential areas for
improvement.
HPS team shall
participate in team
discussions on the topic
of data and related
information needs in a
“think aloud approach”
to engage in current
process and potential
areas for improvement
pertaining to HPS
performance, HPS goals
and HCG’s goal.
Team Meeting –
monthly.
Required Drivers. Required drivers are processes and systems that reinforce, monitor,
encourage, and rewards performance of critical behaviors on the job that can reinforce,
encourage, reward or monitor (support and improve accountability). Required drivers are not
mutually exclusive (e.g. a training could address 4 of them together). In previous sections of this
chapter, KMO recommendations were described in detail. In summary, the three approaches that
can be used are to 1) develop and utilize visual organizers that can be accessed by HPS across
HCG in some level of standardization, 2) offer professional development, 3) update policies to
support performance achievements. In order to accomplish the Level 4 results, the following
required drivers related to the performance of critical behaviors on the job. In order to reinforce
performance, visual aids will be developed, and dashboards will monitor progress of HPS
performance of critical behaviors including the communication of shared data and measurement
practices among stakeholders. In order to encourage performance, ongoing feedback will be
offered to HPS, and rewarding drivers will include public and personal acknowledgement when
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 183
HPS reach benchmark performances. See the Table 20 for examples of required drivers to
support critical behaviors.
Table 20
Required Drivers to Support Critical Behaviors
Method(s) Timing
Critical Behaviors Supported
1, 2, 3 Etc.
Reinforcing
Job Aid including checklist
for shared data, measurement
practices and other related
information needs with
contents and details of
different data available.
Ongoing 1,2,3
Encouraging
Feedback and coaching from
HPS manager.
Ongoing 1,2,3
Feedback and coaching from
HPS team.
Ongoing 2,3
Rewarding
Public acknowledgement,
such as a mention at All-
Staff department meetings, in
updates when team
performance hits a
benchmark.
Ongoing 1,2,3
Personal acknowledgement,
In performance review to
highlight when HPS hits a
benchmark.
Annual 1,2,3
Monitoring
Dashboard to track progress
of HPS performance of
critical behaviors including
the communication of shared
data and measurement
practices among
stakeholders.
Ongoing 1,2,3
Organizational Support. Based on the required drivers and related recommendations,
HCG as the larger organization where HPS will be tasked with providing the following steps.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 184
HCG will review the review the current approach to communicating shared data and
measurement practices between HPS, internal data teams, external community stakeholders and
Executive leadership. Next, HCH will collaboratively develop a plan to redesign the
communication process, which will take into account the organization's goals and additional time
Executive Leadership will need to develop strategies while continuing to oversee HPS
management. Additionally, time will be set-aside in regularly scheduled monthly department
specific HPS meetings to allow HPS and managers the opportunity to openly communicate their
ideas, plans, progress, self-reflect, and support to promote an environment of transparency and
trust. Across the HCG organization, Executive Leadership will provide monthly updates and
assessment on the current communication strategies and outcomes to track and increased rates of
data requests and use that connect the vision and goals with the HPS’s accomplishments.
Moreover, the organization will review the progress of improved health outcomes among those
community stakeholders who are utilizing the shared data to see if there are correlational
impacts.
Level 2: Learning
Level 3 learning is the “degree to which participants acquire the intended knowledge,
skills, attitude, confidence, and commitment based on their participation in the training”,
according to Kirkpatrick and Kirkpatrick (2016). Many of the activities during the intervention
can be evaluated for Level 2 evaluation, and if they are evaluated in an ongoing way, there may
be less of a need to perform a summative evaluation on a delayed basis.
Learning goals. Upon completion of the recommended solutions the HPS will be able
to:
1. Describe the approaches to efficiently navigate and communicate shared data and
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 185
measurement practices to appropriate stakeholders. (Knowledge-Factual)
2. Translate knowledge of existing organizational goals in HPS specific goals. (Knowledge-
Conceptual)
3. Empower HPS to think “aloud” with peers and supervisory staff about stakeholder data
and measurement needs through a collaborative dialogue. (Knowledge-Metacognition)
4. Utilize visual organizers to manage shared data and measurement practices to increase
efficient communication with stakeholders such as community partners and Executive
Leadership. (Knowledge-Conceptual, Motivation-Self-Efficacy)
5. Demonstrate self-confidence that they can effectively and efficiently communicate shared
data and measurement practices (Motivation-Self-Efficacy)
6. Discuss how to standardize processes and policies among HPS that are convenient and
easily accessible in order to translate into everyday practice and are appropriate for teams
with less than full staffing capabilities. (Organizational-Policy)
Program. The learning goals provided in the previous section will be achieved through
visual organizers, professional development, training and exercises that will increase the
knowledge and motivation of the HPS, work through perceived barriers to developing and
recommending strategies to comprehensive communication of shared data and measurement
practices. To develop HPS’ knowledge and skills they will be provided with training, develop
job aids, have time to reflect, and participate in peer team building sessions. Since HPS stay in
their positions for about three to six years, the program will be ongoing.
Each quarter, HPS will work together in workgroups discussing key experiences where
community stakeholder needs were assessed and met which, allowed the HPS to provide
comprehensive shared data and measurement practices. Sessions will emphasize results that will
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 186
be of value to community stakeholders, Executive Leadership within HCG and HPS. HPS will
converse about current use of shared data and measurement practices, and brainstorm about
innovative ways to communicate shared data with each other and community partners; and how
IT and innovative technological approaches, such as systems-wide databases can clearly store
comprehensive that is easy to access and understand. Ideas for improving shared data and
measurement practices will be linked with community stakeholders’ needs and organization
including the mission, vision, strategic plan and operational plan.
HPS, will expand their knowledge and skills, specifically through declarative and factual
knowledge, gain practice using metacognitive skills, and together this will increase their levels of
self-efficacy. The outcome, will be that HPS are more capable of providing data, they not
discouraged by perceived barriers, and are able to develop and recommending strategies to
stakeholders to improve communication of shared data and measurement practices. Feedback
methods will be employed, as well as positive encouragement and successes from
communicating effectively will help increase HPS’ knowledge and motivation about their ability
to provide the community stakeholder, or Executive Leader with comprehensive shared data and
measurement practices. HPS will participate in interactive discussions at the beginning and end
of the meetings in order to highlight improvements in knowledge and demonstrate completion of
tasks. As skills, knowledge and motivation increase, meetings will emphasize the Cognitive
Task Analysis learning method, which captures “cognitive” knowledge and translates that into
objectives, cognitive processes, performance standards and procedural knowledge for learners
(Clark, & Estes, 1999; Clark et al., 2008). HPS will have the opportunity to utilize components
of problem-based learning (English, & Kitsantas, 2013; Hung, 2011) to activate prior knowledge
(gained from previous meetings), to process information with a small group, and promote self-
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 187
directed learning using the goal of acquiring and communicating comprehensive health data to
executive leadership.
Towards the end of each session, metacognitive, or reflective component, and a more
formal evaluation of peer group projects. These reflective think “aloud” sessions will occur with
peers and offer opportunities to discuss current processes, challenges, and successes with each
other. HPS will participate in sessions with peers and HPS managers working through
hypothetical problems with colleagues and receiving feedback. Furthermore, the HPS will be
asked to reflect on their communication effectiveness. On an annual basis, monthly reflections
will be incorporated into annual performance reviews, and will be used by the supervisor, to
assess how often and well the HPS is communicating shared data with stakeholders.
Many theorists suggest that adopting new strategies and learning on the job require a
variety of factors including knowledge, motivational and organizational components (Choi, Van
Merrienboer, & Paas, 2014; Frenk et al., 2010; Kirschner et al., 2006). According to the CANE
model (Clark,1999), personal agency, emotion, value, and environmental/cultural contribute to
one’s motivation or task commitment to a project. In this one-unit course, it will be important to
address these factors while reaching the goal of improving the knowledge acquisition and
communication strategies among HPS.
The program will be as interactive and engaging as possible and prepare HPS to
successfully accomplish critical behaviors while preventing cognitive overload (Kirschner et al.,
2006). Overall, the program is set for meaningful learning to occur by engaging HPS so that
they move through Bloom’s taxonomy of learning stages to reach the highest levels of learning
and use cognitive process dimension (Anderson, & Krathwohl, 2001). There will be an emphasis
on activities that promote “retrieval” to reinforce long-term learning as well.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 188
Evaluation of the Components of Learning. In order to demonstrate learning that will
aid in problem-solving and meet performance goals, HPS must have the knowledge, skills,
motivation and organizational support to achieve their goals. It is important to evaluate learning
for factual and conceptual knowledge as well as track the self-efficacy of HPS. As such, Table
21 lists the evaluation of the components of learning for the program.
Table 21
Evaluation of the Components of Learning for the Program
Method(s) or Activity(ies) Timing
Declarative Knowledge “I know it.”
Knowledge checks using multiple choice. Periodically during the training, such as before,
during and after training demonstrations.
Knowledge checks through discussion
during workgroups.
Periodically during monthly workgroup
meetings and documented via job aid chart.
Report out on table discussions. Throughout the training tracking to ensure that
all attendees are participating and reporting
out.
Procedural Skills “I can do it right now.”
Demonstration in groups and individually of
using the job aids to successfully perform the
skills.
During the monthly workgroup sessions.
Feedback from peers as HPS work through
scenario examples communicating shared
data and measurement practices.
After the learning event.
Use real scenarios in workgroups and role
play.
During the learning event.
Attitude “I believe this is worthwhile.”
Discussions of the self-efficacy of what they
are being asked to do on the job.
During the workgroup sessions.
Pre- and Post-test assessment survey to
determine if the self-efficacy has increased.
At the end of workgroup sessions.
Brainstorm the positive and negative
outcomes of open, transparent and innovative
communication strategies to facilitate the flow
of shared data and measurement practices.
During the learning event.
Confidence “I think I can do it on the job.”
Discussions of the self-efficacy of what they
are being asked to do on the job.
During the workshop sessions.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 189
Feedback from peer and supervisors during
workgroup sessions.
At the end of workgroup sessions.
Self-reflection time when HPS thinks “aloud”
with peers about communication strategies.
At the end of workgroup sessions.
Commitment “I will do it on the job.”
Create an individual action plan. During the workshop session.
Peer discussions following observations. At the end of workgroup sessions.
Ask the attendees to write down and share
how they will implement.
At the end of workgroup sessions.
Level 1: Reaction
Level 1 reaction is the customer satisfaction measurement of an intervention or training,
according to Kirkpatrick and Kirkpatrick (2016). The model describes this level as the degree to
which participants find the training favorable and of significance to their jobs. This level along
with level two provides data related to the training and may be useful to evaluate the training
function and measure the quality of the intervention as well as the effectiveness of the program
delivered. There are three components in level 1 - engagement, relevance and customer
satisfaction and these can be evaluated in formative methods, during implementation and
summative implementation. As such, Table 22 lists the components to measure reactions to the
program.
Table 22
Components to Measure Reactions to the Program
Methods or Tools Timing
Engagement
Checklist rating observation completed by peers
and supervisor.
After the workshop session.
Relevance
Brief pulse-check with participants via survey
(online) and discussion (ongoing).
After every workshop session.
Workshop evaluation. Two weeks after every workshop session.
Customer Satisfaction
Pulse check with supervisor via satisfaction survey. After each workshop session.
Workshop evaluation. Two weeks after every workshop session.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 190
Evaluation Tools
Immediately following the Program implementation. Immediately following learning
event, the workshop participants will complete an evaluation (Appendix D) and a checklist
(Appendix E) for what they believe was addressed in the training, and complete a survey indicate
relevance of the material to the job, participant satisfaction, commitment, attitude, and
confidence in applying what has been learned. This survey will evaluate the Level 1 reaction
and Level 2 learning. After the workshop, the supervisor will complete a checklist that rates the
effectiveness of the HPS in several areas of working through scenario examples, and then will
provide feedback. HPS will participate in reports outs at the end of the workshop sessions, and
HPS supervisors will track progress and store for reference during annual performance
evaluations. During in person learning events the instructor will conduct pulse checks by asking
the participants if the content is relevant to their work and addressing realistic issues. The
instructor will ask about the environment and if it is creating any barriers to the participants’
learning and encourage HPS to report barriers in their written evaluations. Level 2 will include
checks for understanding what is was presented. Level 2 will also use workgroup training
discussions and report outs on the topics being discussed to gauge participant understanding.
Delayed for a period after the program implementation. Approximately one month
after the learning event, and days before the subsequent monthly workshop training session the
organization will administer a survey (Appendix F) containing seven Likert scaled items and two
open ended questions. This approach uses the Kirkpatrick ® Blended Evaluation Plan to
measure from the HPS’ perspective, satisfaction and relevance of the workshop session training
(Kirkpatrick, & Kirkpatrick, 2016). Additionally, HPS would provide data on challenges, or
perceived barriers and offer solutions to facilitate improved communication of shared data and
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 191
measurement practices (Level 1), knowledge, skills, confidence, attitude, commitment and value
of applying their training (Level 2), application of the learning event to the HPS’s ability to
assess community stakeholder and Executive Leadership’s needs and their ability to
communicate shared data and measurement practices (Level 3), and the extent to which they are
able to provide comprehensive data on a regular basis (Level 4).
Biannually, the reviewer will compile the data from HPS staff about communicating data
and shared measurement practices and track the number of data requests coming in. Over the
fiscal year, the reviewer will track the perceptions of HPS staff to gauge increases in transfer
communicating shared data and measurement practices. To monitor the progress and hold the
HPS accountable the dashboard below will report the data on these measures. Similar
dashboards will be created to monitor Levels 1, 2, and 3.
Dashboard Goal Fall 2018
Spring
2019
Annual
2018-2019
Totals
HPS are knowledgeable about what sources of
shared data and measurement practices are
available. 100% XX XX XX
HPS are knowledgeable about how to
communicate shared data and measurement
practices. 100% XX XX XX
HPS feel confident describing data to
colleagues. 100% XX XX XX
HPS feel confident describing data to people in
the community. 100% XX XX XX
HPS feels confident that they have access to the
data matters to people in the community. 100% XX XX XX
HPS feels that their professional data needs are
met. 100% XX XX XX
HPS feels that data needs of people in the
community are met. 100% XX XX XX
HPS feels that data needs of Executive
Leadership are met. 100% XX XX XX
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 192
Data Analysis and Reporting
The Level 4 goal for HCG is to serve as the backbone organization and redefine the role
of government as a steward of innovative health services and resources by using the CI model.
Additionally, it is to run its operations in a way that supports a region that combines the
collective strengths of community stakeholders, transforms policies and systems and integrates
delivery of services and resources. HPS are responsible for implementing the CI initiative
efforts with community stakeholders on a daily basis, and serving as the liaison between the
backbone organization, HCG, and community stakeholders by sharing data and measurement
practices. HPS must have the knowledge, skills, motivation and organizational influences to
communicate shared data and measurement practices and to recommend appropriate information
to community stakeholders and Executive Leadership. Once a month, data will have compiled
from HPS and track the current communication strategies and outcomes to track and increased
rates of data requests and use that connect the vision and goals with the HPS’s accomplishments.
Additionally, the organization will review the progress of improved health outcomes among
those community stakeholders who are utilizing the shared data to see if there are correlational
impacts. In order to encourage performance, HPS feedback will be documented by HPS
managers from month to month, as well as public and personal acknowledgement when HPS
reach benchmark performances.
Summary of the Implementation and Evaluation
The New World Kirkpatrick Model informs the implementation and evaluation plan of
this study (Kirkpatrick, & Kirkpatrick, 2016). The four levels of training and evaluation are used
to ensure that HPS have the knowledge, motivation, and organizational support to provide shared
data and measurement practices to stakeholders to improve health and wellbeing. The
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 193
implementation and evaluation begin with the Level 1: the reactions to the intervention, or
workshop session trainings determine the participants’ satisfaction, engagement, and the
relevance of the training. The next step is Level 2, or the learning outcomes are identified, and
the participants are evaluated on their learning and knowledge, attitude, commitment, and
confidence during the training, which address how to reinforce, encourage, reward or monitor
support and improve accountability among participants. In Level 3 the critical behaviors and
required drivers that assess if the participants are using what they have learned once they are
back on the job. Finally, in Level 4, the results are evidence, based on an evaluation or the
identification of the outcomes, metrics and method to measure the results of the targeted
outcomes that align with the overarching organizational goals.
Evaluations are incorporated into this process to create and demonstrate value to
organizations and stakeholders. The purpose of evaluation is to track what is learned from the
trainings is then applied on the job, which improves performance and then impacts key
organizational outcomes. Effective evaluation strategies will ensure that valuable, limited
resources are dedicated to the programs and services that bring about the most impact
(Kirkpatrick, & Kirkpatrick, 2016).
Limitations and Delimitations
There are several limitations in this study, which are characteristics of the research design
or methodology that impacted the interpretation of the findings. These will impact
generalizability, applications to practice, and/or utility of findings. The limitations are primarily
linked to the process, lack of communication with target population, the role of the principal
investigator and the evaluation instruments or the data collected and reported on those
instruments.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 194
The survey was open for one-month and this limited the collection of data to include HPS
who happened to be working during this month. Due to lack of responses, the data collection
period had to be extended from two-weeks as initially indicated, to one month to meet sample
needs. The issue of confidentiality leads to the next limitation, which is apprehension to
participate or lack of trust to openly and candidly participate the study, and to understand that the
study was supported by Executive Leadership within HCG. For example, the initial study design
aimed include online survey questionnaires and two focus groups. Participants were asked to
write their names and contact information at the conclusion of the online survey, however, only
two participants wrote their names there. After speaking with several HPS, the principal
investigator learned that HPS wanted to retain their anonymity on the survey, and by reporting
their name, they were distrustful of the confidentiality of their responses. Because so few
responded, the research dissertation committee chair suggested modifying the study to collect
individual interviews instead. Thus, six individual interviews with HPS were completed. After
the focus group option transitioned to an interview option, the principal investigator learned that
some HPS felt that although they held a job classification that identified with the role of HPS,
many did not self-identify themselves as HPS and felt that the invitation was sent in error.
Perhaps the language to invite participants, could have omitted the HPS role in the survey
invitation. As there was a lack of support about the purpose of the researcher, there could have
been more communication with Executive Leadership encouraging more emails that would have
supported the target population so as to increase willingness to participate among potential
participants.
Another limitation is the role of the principal investigator, who was at one point a HPS,
but moved into a more visible position in the Executive Offices at HCG during the dissertation
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 195
process. Some HPS seemed less trustful of the integrity, purpose or use of data collected during
the study. In order to minimize this risk, the principle investigator minimized the amount of
demographic information collected so as to keep participants identify confidential.
Finally, as the online survey and interview questions were not validated, the evaluation of
the findings may not have construct validity, although pilot testing was completed prior to
administering the survey. Self-report bias is one last limitation, as self-reported questionnaires
are primarily completed by individuals who are self-motivated to either acknowledge a gripe or
praise something positive. For this reason, some of the responses obtained from the survey and
the interviews could have been censored by the HPS in order to make themselves look more of
less favorable.
Recommendations for Future Research
There are two possibilities for future research. The first option being to look at other job
classifications within HCG or the larger local government organization that are tasked with
communicating between community stakeholders and executive leadership. An evaluation could
be performed on their activities to determine if there are potential KMO gaps. The
recommendation may align similarly with the HPS, but there could be other assumed causes for
gaps, in which case the solutions may vary. The second option being to work with community
stakeholders to evaluate their experiences and perspectives working with shared data and
measurement practices. An evaluation could be performed on their activities to determine if
there are potential KMO gaps. Communities stakeholders may report gaps that require different
approaches to reducing gaps, yet it would be in the best interest of HCG to integrate the HPS role
in these approaches. Overall, there is a need to further pursue evaluative research to identify
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 196
areas for continued improvements and advancement in strategies that are innovative and
generative.
Conclusion
Evaluation is an essential element of success as it is the only way to explore and make
connections between performance gaps, improvement programs and cost effectiveness. In this
study, evaluative efforts were undertaken to better understand the role of communicating shared
data and measurement practices. HPS were identified as stakeholders because they are at the
forefront of working with community partners, stakeholders external to the local government,
and also connect with Executive Leadership to share data and practices. HPS work with
organizations, churches, nonprofits, businesses to promote health as opposed to individuals in the
community. Often HPS are not engaging with stakeholders who are looking at singular clinical
changes, but as organizations looking to tackle larger scale population health topics and are often
looking to partner with the local government. This study was able to evaluate existing
communication efforts to determine gaps between performance goals and current achievement
levels. Findings revealed that there is more work to be done. HPS can benefit from increasing
their factual knowledge so that they are more aware of existing data both within the HCG and the
local government and have a better understanding of how to effectively communicate it and
reflect on improvements or challenges. Gaining this knowledge may support increases in self-
efficacy to communicate shared data and measurement practices. One additional piece is that
HCG might benefit from conducting an internal audit to compare what existing policies,
guidelines and performance reviews track the accountability of their CI responsibilities.
Tracking their efforts will help HCG better understand strengths and opportunities to improve.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 197
The purpose of this project was to use the gap analysis model as a framework to help
identify gaps in knowledge, motivation and organizational factors that could prevent HCG from
reaching its goal (Clark, & Estes, 2008). Using the model, gaps were identified and validated
through surveys and interviews. After gaps were identified, solutions were suggested that could
help HCG reach its goal. The project offered solutions for how HCG can evaluative efforts
implemented using Kirkpatrick’s (2006) Four Levels of Evaluation. HPS communicate with
community stakeholders, and Executive Leaders on a regular basis. Therefore, it is essential to
keep in mind each HPS unique needs and deficiencies such as lack of metacognition skills and
learning strategies. These HPS may need additional support in order to reach performance goals.
In the future, combining the gap analysis framework (2008) and the Kirkpatrick and Kirkpatrick
(2016) four levels of training evaluation may be excellent approach to continue to evaluate and
implement theoretically sound solutions.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 198
References
Abdullahi, A., Hassan, A., Kadarman, N., Saleh, A., Baraya, Y. U. S. A., & Lua, P. L. (2016).
Food safety knowledge, attitude, and practice toward compliance with abattoir laws
among the abattoir workers in Malaysia. International Journal of General Medicine, 9,
79–87.
Aguinis, H., & Kraiger, K. (2009). Benefits of training and development for individuals and
teams, organizations, and society. Annual Review of Psychology, 60, 451-474.
Ajzen, I. (2002). Perceived behavioral control, self-efficacy, locus of control, and the theory of
planned behavior. Journal of Applied Social Psychology, 32(4), 665-683.
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and
assessing: A revision of Bloom’s taxonomy of educational objectives. New York, New
York: Longman.
Argote, L., & Fahrenkopf, E. (2016). Knowledge transfer in organizations: The roles of
members, tasks, tools, and networks. Organizational Behavior and Human Decision
Processes, 136, 146-159.
Atun, R., De Andrade, L. O. M., Almeida, G., Cotlear, D., Dmytraczenko, T., Frenz, P., ... & De
Paula, J. B. (2015). Health-system reform and universal health coverage in Latin
America. The Lancet, 385(9974), 1230-1247.
Auerbach, J. (2016). The 3 buckets of prevention. Journal of Public Health Management and
Practice, 22(3), 215-218.
Baker, L. (2006). Observation: A complex research method. Library Trends, 55(1), 171-189.
Balcazar, H., Lee Rosenthal, E., Nell Brownstein, J., Rush, C. H., Matos, S., & Hernandez, L.
(2011). Community health workers can be a public health force for change in the United
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 199
States: three actions for a new paradigm. American Journal of Public Health, 101(12),
2199-2203.
Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future
research. Personnel Psychology, 41(1), 63-105.
Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change.
Psychological Review, 84, 191-215.
Baruch, Y., & Holtom, B. C. (2008). Survey response rate levels and trends in organizational
research. Human Relations, 61(8), 1139-1160.
Barr-Walker, J. (2017). Evidence-based information needs of public health workers: a
systematized review. Journal of the Medical Library Association, 105(1), 69.
Barton, A. J. (2005). Cultivating informatics competencies in a community of practice. Nursing
Administration Quarterly, 29(4), 323-328.
Battel-Kirk, B., Barry, M. M., Taub, A., & Lysoby, L. (2009). A review of the international
literature on health promotion competencies: identifying frameworks and core
competencies. Global Health Promotion, 16(2), 12-20.
Birkhead, G. S., Klompas, M. D., & Shah, N. R. (2015). Public health surveillance using
electronic health records: rising potential to advance public health. Frontiers in Public
Health Services and Systems Research, 4(5), 25-32.
Blumenthal, D. (2009). Stimulating the adoption of health information technology. New England
Jjournal of Medicine, 360(15), 1477-1479.
Bogdan, R. C., & Biklen, S. K. (2007). Qualitative research for education: An introduction to
theories and methods (5th ed.). Boston, Massachusetts: Allyn and Bacon.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 200
Bong, M. (2001). Between and within-domain relations of academic motivation among middle
and high school students: self-efficacy, task-value, and achievement goals. Journal of
Educational Psychology, 93, 23-34.
Brewer, P. D., & Brewer, K. L. (2010). Knowledge management, human resource management,
and higher education: a theoretical model. Journal of Education for Business, 85(6), 330-
335.
Broadbent, J., & Poon, W. L. (2015). Self-regulated learning strategies & academic achievement
in online higher education learning environments: A systematic review. The Internet and
Higher Education, 27, 1-13.
Bronfenbrenner, U. (1992). Ecological systems theory. In R. Vasta (Ed.), Six theories of child
development: Revised formulations and current issues (pp. 187-249). London, England:
Jessica Kingsley Publishers.
Brownson, R. C., Allen, P., Duggan, K., Stamatakis, K. A., & Erwin, P. C. (2012). Fostering
more-effective public health by identifying administrative evidence-based practices: a
review of the literature. American Journal of Preventive Medicine, 43(3), 309-319.
Brownson, R. C., Baker, E. A., Deshpande, A. D., & Gillespie, K. N. (2017). Evidence-based
public health (3rd ed.). New York, New York: Oxford University Press.
Bureau of Labor Statistics, United States Department of Labor. (2017). May 2017 National
Occupational Employment and Wage Estimates United States. Retrieved from:
https://www.bls.gov/oes/current/oes_nat.htm
Cacari-Stone, L., Wallerstein, N., Garcia, A. P., & Minkler, M. (2014). The promise of
community-based participatory research for health equity: a conceptual model for
bridging evidence with policy. American Journal of Public Health, 104(9), 1615-1623.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 201
Catford, J. (2009). Advancing the ‘science of delivery’ of health promotion: not just the ‘science
of discovery’: Health Promotion International, 24(1), 1-5.
Centers for Disease Control and Prevention (CDC). (2014). “The 10 Essential Public Health
Services: An Overview.” Retrieved from
https://www.cdc.gov/stltpublichealth/publichealthservices/essentialhealthservices.html.
Chen, L., Evans, T., Anand, S., Boufford, J. I., Brown, H., Chowdhury, M., ... & Fee, E. (2004).
Human resources for health: overcoming the crisis. The Lancet, 364(9449), 1984-1990.
Choi, H. H., Van Merriënboer, J. J., & Paas, F. (2014). Effects of the physical environment on
cognitive load and learning: towards a new model of cognitive load. Educational
Psychology Review, 26(2), 225-244.
Chronic Health Care Foundation (CHCF). (2017). Chronic Diseases in California: Facts and
Figures. October 2006. Retrieved from
http://www.chcf.org/publications/2006/10/chronic-disease-in-california-facts-and-figures.
Chun, S. A., Shulman, S., Sandoval, R., & Hovy, E. (2010). Government 2.0: Making
connections between citizens, data and government. Information Polity, 15(1), 1.
Clark, R. E. (1999). The CANE model of motivation to learn and work: A two-stage process of
goal commitment and effort. In J. Lowyck (Ed.), Trends in Corporate Training. Leuven,
Belgium: University of Leuven Press.
Clark, R. E., & Estes, F. (2008). Turning research into results: A guide to selecting the right
performance solutions. Charlotte, North Carolina: Information Age Publishing, Inc.
Coiera, E. (2015). Guide to health informatics (3rd ed.). Boca Raton, Florida: CRC press.
Cooper, A. (2016). Governing global health: challenge, response, innovation. New York, New
York: Routledge.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 202
Corbin, J., & Strauss, A. (2008). Techniques and procedures for developing grounded theory
(3rd ed.). Los Angeles, California: Sage Publications, Incorporated.
Creswell, J. W., & Clark, V. L. P. (2018). Designing and conducting mixed methods research
(3rd ed.). Los Angeles, California: Sage Publications, Incorporated.
Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed
methods approaches (5th ed.). Los Angeles, California: Sage Publications, Incorporated.
Crook, T. R., Todd, S. Y., Combs, J. G., Woehr, D. J., & Ketchen Jr, D. J. (2011). Does human
capital matter? A meta-analysis of the relationship between human capital and firm
performance. Journal of Applied Psychology, 96(3), 443.
Dalrymple, P. W. (2011). Data, information, knowledge: The emerging field of health
informatics. Bulletin of the American Society for Information Science and Technology,
37(5), 41-44.
Dannenberg, A. L. (2016). A Brief History of Health Impact Assessment in the United States.
Chronicles of Health Impact Assessment, Chronicles of Health Impact Assessment. 1(1).
Davies, S. C., Winpenny, E., Ball, S., Fowler, T., Rubin, J., & Nolte, E. (2014). For debate: a
new wave in public health improvement. The Lancet, 384(9957), 1889-1895.
“Deaths and Mortality.” Centers for Disease Control and Prevention National Center for Health
Statistics (CDC, NCHS). 23 May 2017. https://www.cdc.gov/nchs/fastats/deaths.htm.
Accessed August 2017.
Deci, E.L., Koestner, R., & Ryan, R.M (1999). A meta-analytic review of experiments
examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin,
125, 627-668.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 203
Dinsmore, D. L, Alexander, P. A., & Loughlin, S. M., (2008). Focusing the conceptual lens on
metacognition, self-regulation, and self-regulated learning. Educational Psychology
Review, 20, 391-409.
Dori, Y. J., Mevarech, Z. R., & Baker, D. R. (Eds.). (2017). Cognition, Metacognition, and
Culture in STEM Education: Learning, Teaching and Assessment (Vol. 24). Cham,
Switzerland: Springer International Publishing.
Dubowitz, T., Orleans, T., Nelson, C., May, L. W., Sloan, J. C., & Chandra, A. (2016). Creating
Healthier, More Equitable Communities by Improving Governance and Policy. Health
Affairs, 35(11), 1970-1975.
du Monde, M. (2011). The KAP survey model (knowledge, attitudes, & practices). See: spring-
nutrition. org/publications/tool-summaries/kap-survey-model-knowledge-attitudes-and-
practices.
Eccles, J. A. (2007). Motivational perspective on school achievement: Taking responsibility for
learning and teaching. In. R. J. Sternberg and R. F. Subotnik (Eds.), Optimizing student
success in schools with the new three Rs (pp. 199–202). Charlotte, NC: Information Age.
Edelman, C. L., Mandle, C. L., & Kudzma, E. C. (2013). Health Promotion Throughout the Life
Span-E-Book (8th ed.). Elsevier Health Sciences.
Elder, G. H. (1998). The life course as developmental theory. Child Development, 69(1), 1-12.
Eng, T. R. (2004). Population health technologies: emerging innovations for the health of the
public. American Journal of Preventive Medicine, 26(3), 237-242.
English, M C., & Kitsantas, A. (2013). Supporting student self-regulated learning in problem-
and project-based learning. Interdisciplinary Journal of Problem-Based Learning, 7(2),
128-150.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 204
Erwin, P. C., Greene, S. B., Mays, G. P., Ricketts, T. C., & Davis, M. V. (2011). The association
of changes in local health department resources with changes in state-level health
outcomes. American Journal of Public Health, 101(4), 609-615.
Eysenbach, G., & Jadad, A. R. (2001). Evidence-based patient choice and consumer health
informatics in the Internet age. Journal of Medical Internet Research, 3(2).
Felitti, V. J., Anda, R. F., Nordenberg, D., Williamson, D. F., Spitz, A. M., Edwards, V., ... &
Marks, J. S. (1998). Relationship of childhood abuse and household dysfunction to many
of the leading causes of death in adults: The Adverse Childhood Experiences (ACE)
Study. American Journal of Preventive Medicine, 14(4), 245-258.
Flood, J., Minkler, M., Hennessey Lavery, S., Estrada, J., & Falbe, J. (2015). The collective
impact model and its potential for health promotion: overview and case study of a
Healthy Retail Initiative in San Francisco. Health Education & Behavior, 42(5), 654-668.
Ford, J. K., Smith, E. M., Weissbein, D. A., Gully, S. M., & Salas, E. (1998). Relationships of
goal orientation, metacognitive activity, and practice strategies with learning outcomes
and transfer. Journal of Applied Psychology, 83(2), 218.
Frenk, J., Chen, L., Bhutta, Z. A., Cohen, J., Crisp, N., Evans, T., ... & Kistnasamy, B. (2010).
Health professionals for a new century: transforming education to strengthen health
systems in an interdependent world. The Lancet, 376(9756), 1923-1958.
Friede, A., Blum, H. L., & McDonald, M. (1995). Public health informatics: how information-
age technology can strengthen public health. Annual Review of Public Health, 16(1), 239-
252.
Frieden, T. R. (2010). A framework for public health action: the health impact pyramid.
American Journal of Public Health, 100(4), 590-595.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 205
Fryberg, S. A., & Markus, H. R. (2007). Cultural models of education in American Indian, Asian
American and European American contexts. Social Psychology of Education, 10(2), 213-
246.
Fusch, P. I., & Ness, L. R. (2015). Are we there yet? Data saturation in qualitative research. The
Qualitative Report, 20(9), 1408.
Gebbie, K., Rosenstock, L., & Hernandez, L. M. (Eds.). (2003). Who will keep the public
healthy?: Educating public health professionals for the 21st century. Washington, DC:
National Academies Press.
Gebbie, K. M., & Turnock, B. J. (2006). The public health workforce, 2006: New challenges.
Health Affairs, 25(4), 923-933.
Geronimus, A. T. (2000). To mitigate, resist, or undo: addressing structural influences on the
health of urban populations. American Journal of Public Health, 90(6), 867.
Gilson L, Doherty J, Loewenson R, Francis V. (2007). Challenging inequity through health
systems: final report of the Knowledge Network on Health Systems, WHO Commission
on Social Determinants of Health. Geneva: World Health Organization. Retrieved from
http://www.who.int/social_determinants/publications/healthsystems/en/index.html
Ginsberg, M. B. (2005). Cultural diversity, motivation, and differentiation. Theory into Practice
44, 218-225.
Glanz, K., & Bishop, D. B. (2010). The role of behavioral science theory in development and
implementation of public health interventions. Annual Review of Public Health, 31, 399-
418.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 206
Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003). Why don’t we see more translation of
health promotion research to practice? Rethinking the efficacy-to-effectiveness transition.
American Journal of Public Health, 93(8), 1261-1267.
Godshalk, V. M., & Sosik, J. J. (2003). Aiming for career success: The role of learning goal
orientation in mentoring relationships. Journal of Vocational Behavior, 63(3), 417-437.
Golden, S. D., McLeroy, K. R., Green, L. W., Earp, J. A. L., & Lieberman, L. D. (2015).
Upending the social ecological model to guide health promotion efforts toward policy
and environmental change. Health Education & Behavior, 42(1 suppl), 8S-14S.
Green, L. W. (2006). Public health asks of systems science: to advance our evidence-based
practice, can you help us get more practice-based evidence?. American Journal of Public
Health, 96(3), 406-409.
Green, L. W., & Allegrante, J. P. (2011). " Healthy People" 1980-2020: Raising the Ante
Decennially or Just the Name from Public Health Education to Health Promotion to
Social Determinants?. Health Education & Behavior, 38(6), 558-562.
Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment
with data saturation and variability. Field Methods, 18(1), 59-82.
Hanleybrown, F., Kania, J., & Kramer, M.R. (2012). Channeling change: Making collective
impact work. Stanford Social Innovation Review, blog entry. Retrieved from
http://www.ssireview.org/blog/entry/channeling_change_making_collective_impact_wor
k
Harding, J. (2013). Qualitative data analysis from start to finish. Thousand Oaks, California:
Sage Publications, Incorporated.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 207
Harrell, M. C., & Bradley, M. A. (2009). Data collection methods. Semi-structured interviews
and focus groups. Santa Monica, California: Rand National Defense Research Institution.
Hartley, J. (2005). Innovation in governance and public services: Past and present. Public Money
and Management, 25(1), 27-34.
Hicks, D., Larson, C., Nelson, C., Olds, D. L., & Johnston, E. (2008). The influence of
collaboration on program outcomes: The Colorado nurse—family partnership. Evaluation
Review, 32(5), 453-477.
Homer, J. B., & Hirsch, G. B. (2006). System dynamics modeling for public health: background
and opportunities. American Journal of Public Health, 96(3), 452-458.
House, J. S., Landis, K. R., & Umberson, D. (1988). Social relationships and health. Science,
241(4865), 540.
Hung, W. (2011). Theory to reality: A few issues in implementing problem-based learning.
Educational Technology Research and Development, 59, 529-552.
Hung, D. Y., Rundall, T. G., Tallia, A. F., Cohen, D. J., Halpin, H. A., & Crabtree, B. F. (2007).
Rethinking prevention in primary care: applying the chronic care model to address health
risk behaviors. The Milbank Quarterly, 85(1), 69-91.
Institute of Medicine (IOM) (US). (2003). The Future of the Public's Health in the 21st Century:
Committee on Assuring the Health of the Public in the 21st Century. Washington, DC:
National Academy Press.
Israel, B. A., Schulz, A. J., Parker, E. A., & Becker, A. B. (1998). Review of community-based
research: assessing partnership approaches to improve public health. Annual Review of
Public Health, 19(1), 173-202.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 208
Issel, M. (2016). Health program planning and evaluation: A practical, systematic approach for
community health. Burlington, Massachusetts: Jones & Barlett Learning.
Jacob, R. R., Allen, P. M., Ahrendt, L. J., & Brownson, R. C. (2017). Learning about and using
research evidence among public health practitioners. American Journal of Preventive
Medicine, 52(3), S304-S308.
Jayasinghe, S. (2015). Social determinants of health inequalities: towards a theoretical
perspective using systems science. International Journal for Equity in Health, 14(1), 71.
Johnson, M. & Sinatra, G. M. (2012). Use of Task-Value Instructional Inductions for Facilitating
Engagement and Conceptual Change. Contemporary Educational Psychology, 30(1), 51-
63.
Joshi, A., & Perin, D. M. P. (2012). Gaps in the existing public health informatics training
programs: a challenge to the development of a skilled global workforce. Perspectives in
Health Information Management/AHIMA, American Health Information Management
Association, 9(Fall), 1c.
Kania, J. & Kramer, M. (2011). Collective impact. Stanford Social Innovation Review, 9(1): 36–
41
Kania, J., & Kramer, M. (2013). Embracing emergence: How collective impact addresses
complexity. Stanford Social Innovation Review, 1–8.
Kenward, M. G., & Roger, J. H. (1997). Small sample inference for fixed effects from restricted
maximum likelihood. Biometrics, 983-997.
Kim, A. H., Vaughn, S., Wanzek, J., & Wei, S. (2004). Graphic organizers and their effects on
the reading comprehension of students with LD: A synthesis of research. Journal of
Learning Disabilities, 37(2), 105-118.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 209
Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick's four levels of training evaluation.
Alexandria, Virginia: Association for Talent Development.
Kirshner, P. A. (2002). Cognitive load theory: Implications of cognitive load theory on the
design of learning. Learning and Instruction,12(1), 1-10.
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction
does not work: An analysis of the failure of constructivist, discovery, problem-based,
experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.
Klitzman, R. (2013). How IRBs view and make decisions about coercion and undue influence.
Journal of Medical Ethics, 39(4), 224-229.
Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into Practice,
41(4), 212-218.
Krejcie, R. V., & Morgan, D. W. (1970). Determining sample size for research
activities. Educational and Psychological Measurement, 30(3), 607-610.
Kunsting, J., Kempf, J., Wirth, J. (2013). Enhancing scientific discovery learning through
metacognitive support. Contemporary Educational Psychology, 38, 349-360.
Kurpius, S. E. R., & Stafford, M. E. (2005). Testing and measurement: A user-friendly guide.
Thousand Oaks, California: Sage Publications, Incorporated.
Lau, F., Hagens, S., & Muttitt, S. (2006). A proposed benefits evaluation framework for health
information systems in Canada. Healthcare Quarterly (Toronto, Ont.), 10(1), 112-6.
Laverack, G. (2007). Health promotion practice: building empowered communities. Berkshire,
England: McGraw-Hill Education (UK).
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 210
Leopold, C., & Leutner, D. (2015). Improving students’ science text comprehension through
metacognitive self-regulation when applying learning strategies. Metacognition and
Learning, 10(3), 313-346.
Lichtman, M. (2014). Qualitative research for the social sciences. Thousand Oaks, CA: Sage
Publications, Incorporated.
Love, M. B., Gardner, K., & Legion, V. (1997). Community health workers: who they are and
what they do. Health Education & Behavior, 24(4), 510-522.
Maben, J., & Clark, J. M. (1995). Health promotion: a concept analysis. Journal of Advanced
Nursing, 22(6), 1158-1165.
Manning, T. T. (2014). Attitudes and preferences in mentoring among African American women
in human service leadership (Doctoral dissertation, Capella University). Retrieved from
Google Scholar (Accessed October 2017).
Manojlovich, M. (2005). Promoting Nurses' Self-efficacy: A Leadership Strategy to Improve
Practice. Journal of Nursing Administration, 35(5), 271-278.
Marmot, M., & Wilkinson, R. (Eds.). (2005). Social determinants of health. OUP Oxford.
Marmot, M. (2005). Social determinants of health inequalities. The Lancet, 365(9464), 1099-
1104.
Martin, A. J., & Dowson, M. (2009). Interpersonal relationships, motivation, engagement, and
achievement: Yields for theory, current issues, and educational practice. Review
Educational Research, 79(1), 327-365.
Martin, L. T., Plough, A., Carman, K. G., Leviton, L., Bogdan, O., & Miller, C. E. (2016).
Strengthening Integration of Health Services and Systems. Health Affairs, 35(11), 1976-
1981.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 211
Maxwell, J. A. (2013). Qualitative research design: An interactive approach: An interactive
approach (3rd ed.). Los Angeles, California: Sage Publications, Incorporated.
Mayer, R. E. (2011). Applying the science of learning. Boston, Massachusetts: Pearson.
McDermott, P. (2010). Building open government. Government Information Quarterly, 27(4),
401-413.
McLeroy, K. R., Bibeau, D., Steckler, A., & Glanz, K. (1988). An ecological perspective on
health promotion programs. Health Education & Behavior, 15(4), 351-377.
Merriam, S. B., & Tisdell, E. J. (2016). Qualitative research: A guide to design and
implementation (4th ed.). San Francisco, California: Jossey-Bass.
Meyer, A. M., Davis, M., & Mays, G. P. (2012). Defining organizational capacity for public
health services and systems research. Journal of Public Health Management and
Practice, 18(6), 535-544
Miles, M. B., Huberman, A. M., & Saldana, J. (2013). Qualitative data analysis: A methods
sourcebook (3
rd
ed.). Thousand Oaks, California: Sage Publications, Incorporated.
Minkler, M., Blackwell, A. G., Thompson, M., & Tamir, H. (2003). Community-based
participatory research: implications for public health funding. American Journal of
Public Health, 93(8), 1210-1213.
Murray, C. J., Abraham, J., Ali, M. K., Alvarado, M., Atkinson, C., Baddour, L. M., ... &
Bolliger, I. (2013). The state of US health, 1990-2010: burden of diseases, injuries, and
risk factors. Journal of American Medical Association, 310(6), 591-606.
Naidoo, J., & Wills, J. (2016). Foundations for Health Promotion-E-Book. Elsevier Health
Sciences.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 212
O'carroll, P. W., Cahn, M. A., Auston, I., & Selden, C. R. (1998). Information needs in public
health and health policy: results of recent studies. Journal of Urban Health, 75(4), 785-
793.
Olds, D. L. (2006). The nurse–family partnership: An evidence-based preventive intervention.
Infant Mental Health Journal, 27(1), 5-25.
Onwuegbuzie, A. J., & Johnson, R. B. (2006). The validity issue in mixed research. Research in
the Schools, 13(1), 48-63.
Pajares, F. (1997). Current directions in self-efficacy research. Advances in Motivation and
Achievement, 10(149), 1-49.
Pajares, F., & Schunk, D. (2001). The development of academic self-efficacy. In A. Wigfield & J.
Eccles (Eds.), Development of achievement motivation. San Diego, California: Academic
Press.
Pajares, F., & Schunk, D. (2005). Self-efficacy and self-concept beliefs. New Frontiers for Self-
Research, March H. Craven R, McInerney D (eds.). Greenwich, Connecticut: Academic
Press.
Parekh, A. K., Goodman, R. A., Gordon, C., Koh, H. K., & HHS Interagency Workgroup on
Multiple Chronic Conditions. (2011). Managing multiple chronic conditions: a strategic
framework for improving health outcomes and quality of life. Public Health Reports,
126(4), 460-471.
Patton, M. Q. (2002). Chapter 7: Qualitative Interviewing. In Qualitative research & evaluation
methods (3rd ed.) (pp. 339-380). Thousand Oaks, California: Sage Publications.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 213
Pintrich, P. R. (2000). An achievement goal theory perspective on issues in motivation
terminology, theory, and research. Contemporary Educational Psychology, 25(1), 92-
104.
Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components
of classroom academic performance. Journal of Educational Psychology, 82(1), 33-40.
Raglin, J. S., Morgan, W. P., & Luchsinger, A. E. (1990). Mood and self-motivation in
successful and unsuccessful female rowers. Medicine & Science in Sports & Exercise,
22(6), 849-853.
Roberto, C. A., Swinburn, B., Hawkes, C., Huang, T. T., Costa, S. A., Ashe, M., ... & Brownell,
K. D. (2015). Patchy progress on obesity prevention: emerging examples, entrenched
barriers, and new thinking. The Lancet, 385(9985), 2400-2409.
Rosenthal, E. L., Brownstein, J. N., Rush, C. H., Hirsch, G. R., Willaert, A. M., Scott, J. R., ... &
Fox, D. J. (2010). Community health workers: part of the solution. Health Affairs, 29(7),
1338-1342.
Rosenthal, E. L., Wiggins, N., Ingram, M., Mayfield-Johnson, S., & De Zapien, J. G. (2011).
Community health workers then and now: an overview of national studies aimed at
defining the field. The Journal of Ambulatory Care Management, 34(3), 247-259.
Roussos, S. T., & Fawcett, S. B. (2000). A review of collaborative partnerships as a strategy for
improving community health. Annual Review of Public Health, 21(1), 369-402.
Salkind, N. J. (2007). Encyclopedia of measurement and statistics (Vol. 1). Thousand Oaks,
California: Sage Publications, Incorporated.
Sallis, J. F., Owen, N., & Fisher, E. B. (2008). Ecological models of health behavior. Health
Behavior and Health Education: Theory, Research, and Practice, 4, 465-486.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 214
Savel, T. G., Foldy, S., & Centers for Disease Control and Prevention. (2012). The role of public
health informatics in enhancing public health surveillance. The Morbidity and Mortality
Weekly Report Surveillance Summaries, 61(Suppl), 20-4.
Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary
Educational Psychology, 19(4), 460-475.
Schraw, G., & McCrudden, M. (2013). Information processing theory. Information Processing
Theory, 420-433.
Schunk, D. H. (1991). Self-efficacy and academic motivation. Educational Psychologist, 26(3-
4), 207-231.
Scott, S., & Palincsar, A. (2013). Sociocultural theory. Education.com.
Senge, P., Kleiner, A., Roberts, C., Ross, R., Roth, G., Smith, B., & Guman, E. C. (1999). The
dance of change: The challenges to sustaining momentum in learning organizations. New
York, New York: Penguin Random House.
Shapiro, J. S., Mostashari, F., Hripcsak, G., Soulakis, N., & Kuperman, G. (2011). Using health
information exchange to improve public health. American Journal of Public
Health, 101(4), 616-623.
Sharma, M. (2016). Theoretical foundations of health education and health promotion. Sudbury,
Massachusetts: Jones & Bartlett Publishers.
Smith, K. G., Smith, K. A., Olian, J. D., Sims Jr, H. P., O'Bannon, D. P., & Scully, J. A. (1994).
Top management team demography and process: The role of social integration and
communication. Administrative Science Quarterly, 412-438.
Stewart, T., & Ruckdeschel, C. (1998). Intellectual capital: The new wealth of organizations.
New York, New York: Doubleday Dell Publishing Group, Inc.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 215
Stokols, D. (1996). Translating social ecological theory into guidelines for community health
promotion. American Journal of Health Promotion, 10(4), 282-298.
Thompson, J. M. (2010). Understanding and managing organizational change: implications for
public health management. Journal of Public Health Management and Practice, 16(2),
167-173.
Trochim, W. M., Cabrera, D. A., Milstein, B., Gallagher, R. S., & Leischow, S. J. (2006).
Practical challenges of systems thinking and modeling in public health. American Journal
of Public Health, 96(3), 538-546.
Tuckman, B. W. (2007). The effect of motivational scaffolding on procrastinators’ distance
learning outcomes. Computers & Education, 49(2), 414-422.
Turner, S., Merchant, K., Kania, J., & Martin, E. (2012). Understanding the value of backbone
organizations in collective impact: Part 2. Stanford Social Innovation Review. Retrieved
from http://www.ssireview.org.libproxy2.usc.edu/blog/entry/understanding_the_value_
of_backbone_organizations_in_collective_impact_2.
Ubaldi, B. (2013). Open government data: Towards empirical analysis of open government data
initiatives. OECD Working Papers on Public Governance, 22, 0-1.
Van Merriënboer, J. J., & Kirschner, P. A. (2017). Ten steps to complex learning: A systematic
approach to four-component instructional design. New York, New York: Routledge.
Van Wave, T. W., Scutchfield, F. D., & Honoré, P. A. (2010). Recent advances in public health
systems research in the United States. Annual Review of Public Health, 31, 283-295.
Vidich, A., & Bensman, J. (1954). The validity of field data. Human Organization, 13(1), 20-27.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 216
Wallerstein, N. (2006). What is the evidence on effectiveness of empowerment to improve
health?. In World Health Organization [WHO] (Eds.), Health Evidence Network Report.
Copenhagen, Denmark: Health Evidence Network.
Ware Jr, J. E., Kosinski, M., & Keller, S. D. (1996). A 12-Item Short-Form Health Survey:
construction of scales and preliminary tests of reliability and validity. Medical Care,
34(3), 220-233.
Webb, J. C., Mergler, D., Parkes, M. W., Saint-Charles, J., Spiegel, J., Waltner-Toews, D., ... &
Woollard, R. F. (2010). Tools for thoughtful action: the role of ecosystem approaches to
health in enhancing public health. Canadian Journal of Public Health/Revue Canadienne
de Sante'e Publique, 439-441.
Weiner, B. (1972). Attribution theory, achievement motivation, and the educational process.
Review of Educational Research, 42(2), 203-215.
Wigfield, A., & Eccles, J. S. (2000). Expectancy–value theory of achievement motivation.
Contemporary Educational Psychology, 25(1), 68-81.
Witmer, A., Seifer, S. D., Finocchio, L., Leslie, J., & O'neil, E. H. (1995). Community health
workers: integral members of the health care workforce. American Journal of Public
Health, 85(8_Pt_1), 1055-1058.
World Health Organization (WHO). (1986). Health promotion: Ottawa charter. Geneva: World
Health Organization. Retrieved from
http://www.who.int/healthpromotion/conferences/previous/ottawa/en/.
World Health Organization (WHO). (2017). Shanghai Consensus on Healthy Cities 2016.
Retrieved from http://www.who.int/healthpromotion/conferences/9gchp/healthy-city-
pledge/en/
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 217
Yasnoff, W. A., O'carroll, P. W., Koo, D., Linkins, R. W., & Kilbourne, E. M. (2001). Public
health informatics: Improving and transforming public health in the information age.
Topics in Health Information Management, 21(3), 44-53.
Zhao, H., Seibert, S. E., & Hills, G. E. (2005). The mediating role of self-efficacy in the
development of entrepreneurial intentions. Journal of Applied Psychology, 90(6), 1265.
Zimmerman, B. J., Bandura, A., & Martinez-Pons, M. (1992). Self-motivation for academic
attainment: The role of self-efficacy beliefs and personal goal setting. American
Educational Research Journal, 29(3), 663-676.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 218
APPENDIX A
Recruitment Letter to Participate
You are being invited to take part in a research survey through USC and part of a larger study to
learn about the way that we are implementing our Collective Impact initiative. In this study, we
are looking at several strategic approaches including building a better delivery system,
supporting positive choices, and pursuing policy and environmental changes. As Health
Promotion Staff, you are at the forefront of these efforts and it is important to learn your first-
hand perspectives implementing these approaches. If you are interested in participating - all of
the information collected in this survey will remain completely confidential and utilized only for
the purpose of this research project. Data will be analyzed in an aggregate manner and will not
be traceable to individual participants. Participation is anonymous, voluntary and you are able to
opt-out at any time for any reason. The survey should take you approximately 25 minutes to
complete. If you agree to these terms, click "Next" to begin the survey. Thank you!
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 219
APPENDIX B
Online Survey Questions
Online Survey Item
Have you worked in the field of public health/health promotion before working for HCG?
Yes
No
Please indicate the kinds of positions in public health that you held prior to your
current position at the HCG?
Community health promotion
Public health research/academia
Public health policy/government
Biostatistics/Epidemiology
Health communications
Environmental health
Global health
How long you have been working at HCG?
Less than one year
One year to three years
Three years or more
How long you have been in your present position?
Less than one year
One year to three years
Three years or more
Where do you see your career going in 10 years? I see myself working in...
Community health promotion
Public health research/academia
Public health policy/government
Biostatistics/Epidemiology
Health communications
Environmental health
Global health
I don't
Other
What is your previous experience (prior to working for the HCG) with the Collective Impact
Models? Prior to working at the HCG, I...
Was aware of collective impact models but not from firsthand experience
Read about collective impact models in academic coursework
Designed or implemented a (or more than one) collective impact related project(s)
Managed a (or more than one) collective impact related project(s)
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 220
What do you think the goals of Collective Impact models are? (Mark all that apply)
A common agenda
Shared data and measurement practices
Continuous communication
Mutually reinforcing activities among all participants.
A centralized in infrastructure (backbone organization)
A dedicated staff
Within the Collective Impact models, what do you think the goals of a backbone organizations
are? (Mark all that apply)
Guiding the CI vision and strategy.
Supporting the aligned activities
Communicating shared data and measurement practices.
Mobilizing funding efforts.
Building public will.
Advancing policy.
What does it mean to communicate shared data and measurement practices?
Easily accessible (common language, easily measurable)
Dissemination is effective
Contributes to outcomes
Informs decisions
Dynamic exchange
Accurate
Timely
Everyone has a unique role
In order to facilitate the CI initiative, my role within the HCG is to communicate shared health
data shared and measurement practices because, it (Mark all that apply):
Delivers accurate, accessible, and actionable health information
Provides new opportunities to connect with diverse populations.
Supports shared decision-making
Builds social support networks
Enables fast and informed responses to health issues
Which of the CI initiative strategic approaches (below) help you communicate shared data and
measurement practices? (Mark all that apply)
Building better delivery systems
Supporting positive choices
Pursuing policy and environmental changes
In order to communicate shared data and measurement practices with community partners to
build better delivery systems, you share... (Mark all that apply)
Decision making practices (shared and internal practices)
Electronic record keeping practices
Integrative/team-based practices
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 221
Information about formal and informal partners
Whole person care models
Communication practices
In order to communicate shared data and measurement practices with community partners to
support positive choices, you share... (Mark all that apply)
Past, current and future interventions/programs
Communication practices
Shifts in organizational or performance goals
Shifts in administration systems
Shifts in production system
Shifts in technology and IT
In order to share data and measurement practices with community partners to pursue policy
and environmental changes, what kinds of information do you share/communicate (Mark all
that apply):
Efforts adopting policies and environmental procedures/regulations
CI initiative components
Efforts to convene multi-sector stakeholders
CI initiative five areas of influence (health, knowledge, standard of living, community, social)
In order to align your work with the CI initiative, please indicate how you connect with partner
organizations (Mark all that apply):
Attend meetings such as community collaboratives, and advocacy groups with partner
organizations
Exchange emails with partner organizations
Participate in education and outreach efforts with partner organizations
Meet with individual/groups in scheduled one-on-one meetings with partner organizations
Schedule routine telephone calls with partner organizations
Please indicate which answer choice(s) below best describe(s) why collecting shared data and
measurement practices in a systematic way is related to the ongoing progress of the Collective
Impact model initiative. Collecting shared data and measurement practices ...
Improves accountability among all participants
Supports the exchange of work between partnerships (new and evolving ones)
Improves the exchange of knowledge
Enables effective decision-making
Facilitates integrative and diverse organizational practices
Improves the facilitation towards whole person care
None of the above
In your opinion, are there policies in place that guide the design and implementation of the CI
initiative?
Yes
No
I don't know
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 222
Please mark the box(es) below, if you believe there are policies that help you design and
implement CI initiative:
Policies related to communication practices
Policies related to community partnerships roles and responsibilities
Policies related to electronic record keeping practices
Policies that are addressed in your annual performance review
How much confidence do you have connecting with community partners to implement the CI
initiative?
A lot of confidence
A little confidence
Not very much confidence
Not confident at all
How much confidence do you have connecting with community partners to communicate shared
data and measurement practices through the CI initiative?
A lot of confidence
A little confidence
Not very much confidence
Not confident at all
How much confidence do you have reporting shared data and measurement practices to
supervisors (frequency, compliance, accuracy, efficiency)?
A lot of confidence
A little confidence
Not very much confidence
Not confident at all
How much do you agree with the following statement, “As a Health Promotion Staff employed
within HCG, I know that the shared data I report is a valuable part of the decision making
among Executive Leadership.”
Strongly Agree
Somewhat Agree
Somewhat Disagree
Strongly Disagree
How much do you agree with the following statement, “As a Health Promotion Staff employed
within HCG, I am incentivized (with recognition, promotion or acknowledgement) to
communicate shared data and measurement practices.”
Strongly Agree
Somewhat Agree
Somewhat Disagree
Strongly Disagree
Please indicate agreement with the following statement, "As a Health Promotion Staff employed
within HCG, my organization facilitates time for me to communicate shared data and
measurement practices.”
Strongly Agree
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 223
Somewhat Agree
Somewhat Disagree
Strongly Disagree
Please indicate agreement with the following statement, "As a Health Promotion Staff employed
within HCG, my organization is supportive about my role communicating shared data and
measurement practices with community organizations."
Strongly Agree
Somewhat Agree
Somewhat Disagree
Strongly Disagree
Please indicate agreement with the following statement, "As a Health Promotion Staff employed
within HCG, my organization is offers resources to support my role communicating shared data
and measurement practices with community organizations."
Strongly Agree
Somewhat Agree
Somewhat Disagree
Strongly Disagree
Please indicate agreement with the following statement, “There are reasons beyond my control
that contribute to my capacity to communicate shared data and measurement practices.”
Strongly Agree
Somewhat Agree
Somewhat Disagree
Strongly Disagree
As a HPS, I self-reflect by… (Mark all that apply):
All of the above
Think about the effectiveness of my strategies
Evaluate my priorities
Monitor my communication approaches
Consider options when balancing the needs of HCG and community partners
None of the above
Please indicate agreement with the following statement, “My performance as a Health
Promotion Staff is impacted by my motivation, expectations and engagement with community
partners.”
Strongly Agree
Somewhat Agree
Strongly Disagree
Somewhat Disagree
Please indicate agreement with the following statement, “My workplace provides me with the
materials, supplies and resources to help me in my role.”
Somewhat Agree
Strongly Agree
Somewhat Disagree
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 224
Strongly Disagree
Please indicate agreement with the following statement, “My workplace provides appropriate
professional development in my role.”
Strongly Agree
Somewhat Agree
Somewhat Disagree
Strongly Disagree
Please indicate agreement with the following statement, “My workplace provides me with the
opportunity to work with a mentor on the topic of communicating shared health data and
measurement practices.”
Strongly Agree
Somewhat Agree
Somewhat Disagree
Strongly Disagree
I communicate shared health data and measurement practices because I want to (choose one):
A. Advance my skills and training as a HPS through individual improvement and learning.
B. Demonstrate that I, as a HPS, am competent and capable of development and progress.
How much do you agree with the following statement, “As a Health Promotion Staff, I feel that
I am a part of a culture where communicating shared health data and measurement practices is
well-received."
Strongly Agree
Somewhat Agree
Somewhat Disagree
Strongly Disagree
Please explain how people in my role should report comprehensive shared health data and
measurement practices to supervisory staff (open space for each question).
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 225
APPENDIX C
Individual Interview Guide
Interview Questions
THE PROBLEM OF PRACTICE the problem of practice is that the HCG is unclear about the
extent to which HPS employees are connecting on a routine basis with partnering
organizations, collecting the shared data and measurement practices in a systematic way and
reporting this to supervisors in an ongoing and accurate way, so that Executive Leadership can
participate in informed decision-making. Executive Leadership need one-hundred percent of
required health data to make informed decisions that will help reduce the preventable, disease-
related deaths among local inhabitants.
Purpose of the STUDY: 1) Examine the KMO factors that influence the ability of HPS to
achieve individual performance goals.
2) Develop solutions to achieve individual performance goals.
Q1: Please describe the role of the HCG as a backbone organization in the design and
implementation of the CI initiative.
Q2: As a backbone organization, HCG is tasked with facilitating communicating shared data
and measurement practices with partnering organizations. What is your role in communicating
shared data and measurement practices with partnering organizations? (PROBE: How has your
time at the County impacted your understanding of this task? How has your time in your present
position impacted your understanding of this task?)
Q3: What steps do you take to communicate shared data and measurement practice to Executive
Leadership? (probe: what works well, what could be improved)
Q4: How do you evaluate the effectiveness of your communication strategies? (PROBE: How
do you monitor a need for change in your communication?)
Q5: What is the value in communicating shared data and measurement practices with
community partners?
Q6: Can we play the devil’s advocate and pretend that factors are ideal to support your ability to
communicate shared data and measurement practices with community partners, what else would
still prevent you from communicating shared data and measurement practices with community
partners, including: developments in communities, factors within the HCG, such as your peers,
your management or other leadership?
Q7: Describe your goals for communicating shared health data and measurement practices?
(PROBE: How do your goals influence your motivation?)
Q8: Tell me about the materials, supplies or resources that your workplace provides for you to
communicate comprehensive shared health data and measurement practices. (PROBE: What
about mentors or shadowing opportunities? How does this impact your understanding of your
role?)
Q9: What trainings (online, conferences, or other in-person trainings) you have received related
to communicating comprehensive shared health data and measurement practices? (PROBE:
How have these impacted your role?)
Q10: Ideally, what organizational components (e.g. policies/procedures, cultural components,
pay structures) would allow you to implement communicating comprehensive shared health
data and measurement practices?
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 226
Q11: What incentives do you have, such as acknowledgement and recognition, to communicate
comprehensive shared health data and measurement practices?
Q12: Going back to an earlier question, given your description of your role communicating
shared data and measurement practices with partnering organizations. How effective are you in
your role (Likert Scale)? 5=“Extremely”, 4=“Very”, 3=“Modestly”, 2=“Fair”, 1= “Poor”
(PROBE: Why did you choose this number?)?
Q13: Based on your score above, what would you need to increase your score?
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 227
APPENDIX D
For each of the questions below, circle the response that best characterized how you feel about
the statement from 1 (Strongly Disagree), 2 (Somewhat disagree), 3 (Somewhat agree), or 4
(Strongly Agree).
Strongly
Disagree
Strongly
Agree
1 The workshop held my interest. 1 2 3 4
2
During workshop we discussed how to apply what was
learned. 1 2 3 4
3 I will recommend this program to other HPS. 1 2 3 4
4
I believe it will be worthwhile for me to better understand
data needs of people I know in the community when I return
to my job. 1 2 3 4
5
I believe it will be worthwhile for me to better understand
data needs of my Executive Leaders when I return to my job. 1 2 3 4
6
The feedback has given me the confidence to apply what I
learned when I return to my job. 1 2 3 4
7
This workshop has positively impacted my interactions with
data. 1 2 3 4
8
I am committed to applying what I learned during my
discussions. 1 2 3 4
9
I found the feedback from colleagues valuable for
empowering and utilizing data communication skills. 1 2 3 4
1
0
I was satisfied with the workshop on communicating shared
data and measurement practices. 1 2 3 4
PLEASE COMPLETE:
Please provide feedback for the following questions:
1
What part of the workshop did you find irrelevant for your communication
needs?
2 What were the major concepts you learned today?
3 Reflecting on this program how could it have been improved?
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 228
APPENDIX E
Context: This is a checklist that supervisors and/or peers will use when observing HPS during
workshop sessions to rate the HPS skills and ability to communicate shared data and
measurement practices.
A score of 1= Effective Use of targeted behavior
A score of 2= Moderately effective use of targeted behavior
A score of 3= Ineffective use of targeted behavior
Feedback comments may include specific observations that will support the rating, as well as
feedback to help the HPS be more effective when assessing shared data and measurement
practice needs among themselves, the people in the community, community stakeholders, and
Executive Leadership, and professional interests of communicating shared data and
measurement practices.
Target Behavior Rating
Feedback
Comments
HPS was able to describe what sources of shared data and
measurement practices are available.
HPS asked open-ended questions about the shared data and
measurement practice needs of community stakeholders.
HPS clarified with follow-up questions about the community
stakeholders' needs.
HPS recommended the appropriate resource based on the
community stakeholders' responses.
HPS worked collaboratively with the community stakeholder
to resolve his/her issues.
HPS was able to resolve the community stakeholder
questions.
HPS asked open-ended questions about the shared data and
measurement practice needs of Executive Leadership.
HPS clarified with follow-up questions about Executive
Leadership's needs.
HPS recommended the appropriate resource based on
Executive Leadership's responses.
HPS worked collaboratively with Executive Leadership to
resolve his/her issues.
HPS was able to resolve Executive Leadership's questions.
EVALUATING COLLECTIVE IMPACT IN A LOCAL GOVERNMENT 229
APPENDIX F
For each of the questions below, circle the response that best characterized how you feel about
the statement from 1 (Strongly Disagree), 2 (Somewhat disagree), 3 (Somewhat agree), or 4
(Strongly Agree).
Strongly
Disagree
Strongly
Agree
1 I have had the opportunity to use what I learned on the job. 1 2 3 4
2
Reflecting back on the workshop, I believe that it was a good
use of my time. 1 2 3 4
3
After the workshop I have successfully applied what I learned
on the job. 1 2 3 4
4
I have received support from my supervisor to apply what I
have learned to my current job. 1 2 3 4
5
I have received support from my peers to apply what I have
learned to my current job. 1 2 3 4
6 I am seeing positive results from the workshop. 1 2 3 4
7
This workshop has positively impacted interactions with
Executive Leadership and community stakeholders. 1 2 3 4
PLEASE COMPLETE:
Please provide feedback for the following questions:
1
Describe any challenges you are facing as you communicate shared data
and measurement practices.
2
What possible solutions will help you overcome the
challenges?
3 Reflecting on this program how could it have been improved?
Abstract (if available)
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
The moderating role of knowledge, motivation, and organizational influences on employee turnover: A gap analysis
PDF
Examining the adoption of electronic health records system in patient care and students’ education using the GAP analysis approach
PDF
Establishing a systematic evaluation of positive behavioral interventions and supports to improve implementation and accountability approaches using a gap analysis framework
PDF
Establishing a systematic evaluation of an academic nursing program using a gap analysis framework
PDF
Examining teachers' roles in English learners achievement in language arts: a gap analysis
PDF
Promoting equity in discipline practices for Latino students: a gap analysis
PDF
A gap analysis of employee satisfaction within the National Park Service: Kailuana National Park
PDF
Gap analysis of employee satisfaction at a national park: Round Hill Park
PDF
A-G course completion percentage upon high school graduation. A gap analysis
PDF
A gap analysis of employee satisfaction for the National Park Service: Camp Moxie site
PDF
Implementation of the Social Justice Anchor Standards in the West Coast Unified School District: a gap analysis
PDF
An examination of the facilitators of and barriers to effective supervision from the perspective of supervisors in a federal agency using the gap analysis framework
PDF
A gap analysis of employee satisfaction for the National Park Service: Picturesque Park
PDF
Winning the organizational leadership game through engagement: a gap analysis
PDF
An examination using the gap analysis framework of employees’ perceptions of promising practices supporting teamwork in a federal agency
PDF
A case study in student retention at a Northern California private Jewish day school: a gap analysis
PDF
A gap analysis of employee satisfaction within the National Park Service: Casa Linda National Park
PDF
Service provider compliance with regional center service agreement: a gap analysis
PDF
An examination of supervisors’ perspectives of teamwork in a federal agency: promising practices and challenges using a gap analysis framework
PDF
A gap analysis of course directors’ effective implementation of technology-enriched course designs: An innovation study
Asset Metadata
Creator
Munoz, Alexis Renee
(author)
Core Title
Evaluating collective impact in a local government: A gap analysis
School
Rossier School of Education
Degree
Doctor of Education
Degree Program
Education (Leadership)
Publication Date
07/26/2018
Defense Date
03/23/2018
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
collective impact,gap analysis,Government,health,human services,Kirkpatrick model,OAI-PMH Harvest,public,Public Health
Format
application/pdf
(imt)
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Yates, Kenneth (
committee chair
), Hernandez, Elizabeth (
committee member
), Sundt, Melora (
committee member
)
Creator Email
alexismunoz12@gmail.com
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c89-33207
Unique identifier
UC11670731
Identifier
etd-MunozAlexi-6518.pdf (filename),usctheses-c89-33207 (legacy record id)
Legacy Identifier
etd-MunozAlexi-6518.pdf
Dmrecord
33207
Document Type
Dissertation
Format
application/pdf (imt)
Rights
Munoz, Alexis Renee
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
collective impact
gap analysis
human services
Kirkpatrick model
public