Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Delineating the processes and mechanisms of sustainment: understanding how evidence-based prevention initiatives are maintained after initial funding ends
(USC Thesis Other)
Delineating the processes and mechanisms of sustainment: understanding how evidence-based prevention initiatives are maintained after initial funding ends
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
DELINEATING THE PROCESSES AND MECHANISMS OF SUSTAINMENT:
UNDERSTANDING HOW EVIDENCE-BASED PREVENTION INITIATIVES ARE MAINTAINED AFTER INITIAL
FUNDING ENDS
by
Sapna J. Mendon, MSW
A Dissertation Presented to the
FACULTY OF THE USC GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
SOCIAL WORK
August 2020
Copyright 2020 Sapna J. Mendon
ii
Dedication
For my Dad.
Through adversity,
Through wilderness,
Through curiosity,
I became—
Through light—Thank you.
iii
Acknowledgements
Every night at 7pm here in NYC, we lean out of our apartment windows and balconies, go up to
our rooftops with pots and pans and instruments and bells; people making deliveries and driving cabs stop
and honk, all to cheer on and give thanks to our invaluable healthcare force. It’s the hour when most
hospitals have a shift change. It’s the hour that breaks my focus, the hour that breaks my obsession of
whatever “big” roadblock I’ve hit in my own work for the day. Like many, I get caught trying to make sense
of the chaos that is all around us, a perilous pandemic, and the greatest civil unrest most of us have ever
experienced. Though earning my PhD has been a part of my journey over nearly two decades, it’s at 7pm
each day when I am reminded of how small this adventure really is. To all of the essential workers tirelessly
working to keep everyone safe and healthy, thank you. To those who continue to fight for social justice,
you have restored my faith in humanity. #Black Lives Matter.
The past five years
To my dissertation committee: Lawrence Palinkas (chair), Peer Fiss, and Michael Hurlburt – There are
advisors who simply tell their students what to do in order to finish, and then there are mentors who will
leave a lasting mark on one’s life. You three are undoubtedly the latter. Thank you.
Larry, I thought the hard part was over! Yet I’m left scrambling to find the words that will even
come close to describing what you mean to me. Many years ago, you shared a story with me that ended
up convincing me to accept the offer at USC. Throughout the years, your stories and willingness to share
your own experiences continued to give me hope, even in the midst of loss. You helped shape my ideas
without ever explicitly telling me what to do. You directed me by encouraging my own curiosity. You
picked up on my energy when things were difficult and took the time to just sit with me. Your confidence
in me has taken me to new heights. You and Terri are an inspiration beyond words. Thank you for taking
me as your student, seeing me as your colleague, and welcoming me as family.
iv
Peer, admittedly, taking org theory with you several years ago was quite intimidating. And I
remember how nervous I was to share my ideas about using QCA. Over the years and throughout several
projects, you were supportive of my ideas; and, perhaps more importantly, you believed in my ability to
carry out that work. You helped me understand the method, the purpose for it, how to apply it, and, most
importantly, how to teach others. For that I am grateful. Thank you for always pausing to ask me about
how my Mom is doing, and for so thoughtfully caring about my health and well-being.
Mike, I don’t think I’ve ever met another professor who values the art of feedback quite like you
do! From stats classes, to my qualifying exam study, to F31 development, and my dissertation, you
challenged my thinking each step of the way. Even when I felt sure of something, the questions you raised
helped me consider the alternative. Thank you for always taking the time to think through my ideas
thoroughly with me. We have been so fortunate to have you as our program chair!
To those without whom this work would not be possible: Our team at the Center for Prevention
Implementation Methodology (Ce-PIM) at Northwestern University Feinberg School of Medicine,
especially C. Hendricks Brown and Juan Villamar, and Suzanne Spear at Cal State University at Northridge.
To our colleagues in the Substance Abuse and Mental Health Services Administration (SAMHSA) for their
help in connecting with their grantees, and for the grantees themselves for the time and information they
provided for this project. A disclaimer: the content herein is solely the responsibility of the author and
does not necessarily represent the official views of the National Institutes of Health or SAMHSA.
For thinking out loud with me and helping me push my ideas forward: Concepcion Barrio, Rinad
Beidas, Alicia Bunger, Julie Cedarbaum, Jeremy Goldbach, Kimberly Hoagwood, Monica Jolles-Perez,
Edward Miech, and Lisa Saldana.
During some of the most challenging times of this program, I started a group, PhDs and PostDocs
in NYC, for doctoral students and early career faculty across disciplines to come together and write. What
v
grew out of this simple idea still amazes me! For those of us who have worked together day and night,
especially Lauren Jessell and Michelle Young, this whole experience would have been much more difficult
without you all – thank you. The only way out is through!
And to good friends I made along the way, Erin Kelly, Cary Klemmer, and Joshua Russow – nothing
beats good conversations over good food. Thank you. And to Deborah Hoe: our Dads may have brought
us together, but as time goes on, I’m convinced that our souls met and had a party somewhere in the
Pacific long before we shook hands. Thank you for your contagious laughter and so much more.
The road here – This dissertation is a product of a journey that began nearly two decades ago. That journey
would have been remarkably different and likely more challenging, and frankly, less joyful without the
immeasurable support of those I have been fortunate to cross paths with.
To my parents, Jay and Sheila Mendon – You both sacrificed so much to bring us here so that we
could open just about any door we wanted to and be successful. Though what I’ve come to realize is that
most of all you wanted us to be happy. Without you I wouldn’t know the meaning of hard work. I wouldn’t
know how to be curious and unafraid. I wouldn’t know the meaning of happiness. I am forever grateful.
Dad, I miss you every single day. Mom, thank you for believing in me every step of the way.
To the rest of my family, especially Ajja and Ajji, Akka, Sandeep, Asha auntie, Puthran uncle, Jyoti
auntie, and Shashi uncle – who all helped raise me and continue to be an important presence in my life,
from near and afar. And to my cousin, Bina: your presence has generated some serious laughter, thought-
provoking conversations, and zen-like calmness when I needed it most. Thank you.
To my dear friends who have become family: words alone cannot express how much you mean
to me. You have each seen the best and worst parts of me and continue to love and support me. I propose
we all buy a gigantic house and grow old together! Alexis Kuerbis and Amy Kapadia-Little – my unofficial
mentors, my friends, my sisters – you both have been such a grounding presence in my life from the
vi
moment we met. You believed in me long before I ever believed in myself. I’m in awe of the both of you,
and grateful for all of the life lessons. Thank you for helping me connect with my own inner light. Meredith
Doherety – I can’t count the times our paths should have crossed long before we met, but I’m so glad we
met when we did. You are truly an inspiration my dear, I can’t wait to see what comes next for us. Kate
Cleary and I-Chin Chiang – because of you, work never felt like work! Thank you for your endless
encouragement over the years. I’m looking forward to many more getaways to escape the chaos. To my
college trifecta – Diana Truong and Tracy Pantig – the two of you have quite literally been there for me
through some of the most difficult and happiest of times. Belly laughs have taken on a new meaning! Di,
I promise, no more school.
To my PI family, where it all started, especially Lisa Dixon and Susan Essock for being strong female
leaders that I continue to look up to. Thank you for believing in me, challenging me, supporting me, and
inspiring me to do better. Lisa, thank you for continuing to entertain my wild ideas and for giving me the
platforms to bring them to fruition. Ellen Lukens, Ilana Nossel, Rufina Lee, Sarah Piscitelli, Emily Grossman,
Nancy Alexander, and Jordan DeVylder – thank you for your ongoing support and positive energy along
the way.
My heartfelt thanks to many others who have been an important part of my journey – Mike
Washington, Jennifer Stewart, Shekenah Campbell, and Hillel Samlan. To the Plasek family, especially Matt
and Allie Wilson Plasek – soon to be 4P(!!) – thank you for seamlessly welcoming me into your family.
vii
The road ahead
To my partner, Aaron Plasek – I’m still not sure how our paths managed to converge, but it is your hand I
will hold wherever these roads lead us. These walls in our home have seen us work until all hours of the
night together, cry together, laugh together, and dance together with or without any music. You are my
go-to tech support, my copy editor, my 3am “let’s talk shop,” my favorite person to dream big with, and
the only person I am happy to be quarantining with! I don’t have to look forward to what the rest of our
lives will be like, I am already living my best life with you each day. Thank you for walking alongside of me.
Thank you for giving me the space to be the version of myself I am meant to be. Two people in academia
under one small NYC roof isn’t easy, but you make it beyond worthwhile. You’re up next!
To all those who live with mental illness and loved ones who continue to be present – it is possible for
calmness to co-exist alongside the storm. I am committed to creating systems of care where you feel
valued and supported.
;
viii
Table of Contents
Dedication.………………………………………………………………………………………………………………………………………………ii
Acknowledgements………………………………………………………………………………………………………………………………..iii
List of Tables....................………………………………………………………………………………………………………………………..x
List of Figures...................……………………………………………………………………………………………………………………...xi
Abbreviations..............................................................................................................................................xii
Abstract......................................................................................................................................................xiii
Chapter One: Introduction………………………………………………………………………………………………………………………1
The Importance of Studying Sustainment…………………………………………………………………………………..2
Challenges in Studying Sustainment…………………………………………………………………………………………..3
Moving Towards a Mechanistic Approach………………………………………………………………………………….4
Dissertation Blueprint………………………………………………………………………………………………………………..6
References……………………………………………………………………………………………………………………………….11
Chapter Two (Study 1): Sufficient Pathways for Maintaining Prevention Programs Long-Term: A
configurational approach using Qualitative Comparative Analysis…………………………………………………………19
Abstract…………………………………………………………………………………………………………………………………….20
Introduction………………………………………………………………………………………………………………………………22
Methods……………………………………………………………………………………………………………………………………27
Results………………………………………………………………………………………………………………………………………34
Discussion…………………………………………………………………………………………………………………………………34
References………………………………………………………………………………………………………………………………..39
Tables and Figures…………………………………………………………………………………………………………………….48
Chapter Three (Study 2): The process of sustaining evidence-based practice: Establishing the role of
program integration………………………………………………………………………………………………………………………………56
Abstract…………………………………………………………………………………………………………………………………….57
Introduction………………………………………………………………………………………………………………………………59
Methods……………………………………………………………………………………………………………………………………64
Results………………………………………………………………………………………………………………………………………67
Discussion………………………………………………………………………………………………………………………………...70
References………………………………………………………………………………………………………………………………..75
Tables and Figures…………………………………………………………………………………………………………………....80
ix
Chapter Four: Conclusion……………………………………………………………………………………………………………………87
Key Contributions…………………………………………………………………………………………………………………….87
Rethinking Implementation of Health Promotion Programs: A research agenda……………………..89
Implications for policy makers and funding agencies………….………………………………………..89
Implications for agency leaders….………………………………………………………………………………..91
Theoretical implications…………….………………………………………….........................................91
Methodological implications….……………………………………………………..................................92
Concluding thoughts……….………………………………….………………………………………………………..93
Dissertation References……………………………………….………………………………………………………………………………..96
Appendix
The Consolidated Framework for Implementation Research (CFIR) ………………………………………114
x
Tables
Table 2.1. Inter-item reliability by construct…..…………………………………………………………………………………….48
Table 2.2. Operationalization and calibration………..……………………………………………………………………………..50
Table 2.3. Case examples for indirect calibration method…………..…………………………………………………………51
Table 2.4. Truth table analysis……………………………………………..……………………………………………………………….52
Table 2.5. Necessity analysis…………………………..…………………………………………………………………………………….54
Table 2.6. Pathways with a negated outcome……….……………………………………………………………………………..55
Table 3.1. Inter-item reliability by construct……….………………………………………………………………………………..80
Table 3.2. Outcome means by SAMSA program………….………………………………………………………………………..81
Table 3.3. Regression analyses on program integration……….……………………………………………………………….84
Table 3.4. Regression analyses on program sustainment…………….………………………………………………………..85
xi
Figures
Figure 2.1. Respondents by SAMHSA program……………………………………………………………………………………...49
Figure 2.2. Pathways to sustainment…………………………………………………………………………………………………....53
Figure 3.1. Respondent roles…………………………………………………………………………………………………………….…..82
Figure 3.2. Program existence…………………………………………………………………………………………………………….…83
Figure 3.3 Pathway to sustainment...........................................................................................................86
xii
Abbreviations
CFIR – Consolidated Framework for Implementation Research
EBP – Evidence-Based Practice
EPIS – Exploration, Preparation, Implementation, Sustainment model
fsQCA – Fuzzy-set Qualitative Comparative Analysis
NIDA – National Institute on Drug Abuse
SAMHSA – Substance Abuse Mental Health Services Administration
xiii
Abstract
Mental health and substance use disorders, including suicide, continue to plague adolescents and
young adults across the U.S. While numerous prevention initiatives are developed, few are implemented
in community-based settings where it can be accessed by those in need. Even fewer are sustained once
initial seed funding ends, threatening the long-term public health impact of such evidence-based practices
(EBP). Numerous studies in recent years have focused on identifying factors that facilitate sustainment
and refining strategies to improve implementation and sustainment of health promotion and prevention
programs in community settings. However, knowledge pertaining to mechanisms of and causal pathways
to sustainment are considerably lacking.
This dissertation draws from a larger parent study funded by the National Institute on Drug Abuse
which sought to understand how evidence-based programs are sustained once seed funding ends. The
current studies benefited from a comprehensive conceptualization of sustainment based on various
prevention initiatives across a variety of delivery settings, and the use of a newly developed tool to
measure sustainment and its determinants. Additionally, both studies in this dissertation are theoretically
guided by the Consolidated Framework for Implementation Research (CFIR) and the Exploration,
Preparation, Implementation, Sustainment (EPIS) conceptual model.
The first chapter highlights the importance of studying sustainment in health services research
and is followed by a detailed description of why studying sustainment has proved to be challenging. Lastly,
a brief history of what we currently know about sustaining EBP is discussed in the context of how the field
of implementation research is progressing, and what remains to be further developed.
Efforts to deepen our understanding of causal mechanisms and improve the selection of
implementation strategies have continued to rely on traditional regression-based models. The first study
(chapter two) uses a different approach to identify causal pathways to sustainment based on a
xiv
configurational approach. The application of fuzzy-set Qualitative Comparative Analysis (fsQCA), a set-
theoretic approach, focused on how determinants of sustainment work interdependently. Of over 30
possible configurations, this study found two causal pathways that are sufficient to achieving sustainment.
Study two (chapter three) sought to understand the process of sustainment. Program-integration, a
largely understudied concept in implementation research, was believed by many to be a driver of
sustainment. This is the first study to empirically assess what elements promote program-integration, as
well as its relationship to sustainment. Determinants of sustainment found to be explained by integration
suggests that integration may be a phase between active implementation and sustainment.
The overall contributions of this dissertation are discussed in chapter four. Subsequently, a
discussion highlighting key findings, such as planning for sustainment, is provided for policy makers and
funding agencies, and pragmatic suggestions for agency leaders are specified, including approaches to
applying newly established knowledge about determinants of sustainment. To move the field of
implementation science forward, implications and detailed recommendations are offered for the
refinement of theoretical frameworks and methodological approaches.
1
Chapter One: Introduction
Over the past two decades the U.S. has witnessed an increase in suicide rates of 35%, increasing
steadily by about 2% per year since 2006 (Center for Disease Control, 2020). In comparison to all other
age groups suicide is the 2
nd
leading cause of death for youth and young adults ages 10-34. Most people
who die by suicide have related mental health conditions (Arsenault-Lapierre et al., 2004). Amongst other
mental health disorders, depression and substance use disorders are major risk factors for suicide
(National Institute of Mental Health, 2017). A nationally representative survey of over 62,000 college
students conducted by Eisenberg and colleagues as part of the Healthy Minds Network (2019) found that
36% reported either moderate or major depression, 31% reported an anxiety disorder, 24% reported
marijuana use, and 17% reported binge drinking 3-5 times in the past two weeks. Of the 14% of students
that reported suicidal ideation in the past year, 6% had a plan, and 2% attempted suicide. In a nationally
representative sample of over 13,500 youth aged 12-17 years, tobacco users were also more likely to
report internalizing problems such as depressive and anxiety symptoms and externalizing problems such
as conduct disorder, oppositional defiant disorder, and attention deficit disorders as compared to non-
tobacco users (Conway et al., 2018). Those using multiple types of tobacco products were also more likely
to have mental health and substance use problems. Comorbidities between substance use and severe
psychiatric disorders such as major depression and anxiety can also be seen amongst non-medical opioid
users (Goldner et al., 2014). Among youth living with HIV, tobacco, alcohol, and illicit drug use also remains
high (Gamarel et al., 2016).
Suicide is preventable (Bailey et al., 2011; Chambers et al., 2005; Ram et al., 2012) as is substance
use (Bernstein et al., 2017; Malick et al., 2018). Several prevention initiatives and treatments such as
cognitive-behavioral therapies, dialectical-behavioral therapies, and technology-based interventions exist
to reduce suicidal thoughts and attempts (Cha et al., 2018). We have also seen an abundance of
interventions to treat alcohol and substance use developed to treat various age groups and populations
2
(Boustani et al., 2015; Das et al., 2016). Recent research indicates, however, that most people that need
such treatment are not receiving it (Lipari et al., 2016). Additionally, when interventions do successfully
transition from a research setting into community settings making services accessible to the general
public, determining the rate at which these programs are sustained over time is challenging (Shelton et
al., 2018).
The Importance of Studying Sustainment
“By using implementation science and implementation strategies, you can help bridge the divide
between research and practice—and bring programs that work to communities in need.”
- National Cancer Institute (2015)
Research that focuses on intervention development often appears not to consider the challenges
involved in implementing and sustaining interventions. Once implemented in the community, the overall
logistics and process of delivering EBP goes beyond the research setting and becomes reliant on the host
organization and its actors. It also concerns a much larger intended target population and involves the
surrounding communities. This newly implemented EBP may also concern policy makers at the state and
federal level, and relevant funding agencies. Actors across these multi-level contexts are an integral part
of the new initiative and are also invested in the successful implementation of it. Without sustaining a
program over time, the return on investment for these various groups of people across contexts is
compromised (Aarons et al., 2014; Cooper et al., 2015; Shelton et al., 2018; Stirman et al., 2012). More
importantly, neglecting to maintain the quality of care provided as part of the initiative poses a threat to
the patients and communities being served, and ceasing certain treatments and/or services also has
implications for health outcomes (Bumbarger & Perkins, 2008).
Most new programs being implemented in community settings are supported by financial support
from government agencies and private foundations. While initial funding is imperative to the launch of
initiatives and related activities, it is not meant to be a permanent fixture of delivering care, and usually
3
lasts anywhere from 2-5 years (Cooper et al., 2015). Due to insufficient measurement and evaluation
protocols, it has been challenging to gauge the number of programs that have been implemented and
sustained post-initial funding compared to those that have completely been discontinued (Shelton et al.,
2018). One study tracking over 100 initiatives originally funded by the Robert Wood Johnson Foundation
between 1986-2001 found that 92% of the programs “survived” the first year after funding dissolved. This
is remarkably high in comparison to other studies that have found programs implemented in community-
based settings to be sustained about half of the time (Aarons et al., 2014; Hodge & Turner, 2016; Scheirer,
2005); suggesting that a large number of programs dissipate without being able to maintain delivery of
services to intended populations and thus impeding long-term public health impact.
Challenges in Studying Sustainment
Inadequate knowledge regarding how health promotion programs are sustained in the
community following the dissolution of initial grant funding can be partly attributed to how sustainment
is defined, how it is measured, and inconsistencies in nomenclature. Challenges in understanding
sustainment can be partly attributed to irregularities in defining sustainment. Studies have focused on the
continued delivery of intervention core components being delivered with fidelity, the amount of
adaptations made to the intervention, the continued health benefits, and/or simply whether services
continued to be delivered or not (Johnson et al., 2019; Lennox et al., 2018; Stirman et al., 2012).
Challenges defining sustainment also have a temporal relationship with measuring sustainment.
Previously developed scales such as the Stages of Implementation Completion (SIC; Chamberlain et al.,
2011) and the Program Sustainability Assessment Tool (PSAT; Luke et al., 2014) have been limited either
due to poor psychometric validation and generalizability across public health programs, and/or the lack
of the capacity to assess sustainment and its determinants (Palinkas et al., 2015).
Lastly, challenges in conceptualizing sustainment are also due to the variability in nomenclature
spanning over three decades. As cited by Goodman and colleagues (1993), several terms were being used
4
to describe similar processes that occur when a program becomes a part of the organization, including
stabilized, sustained, maintained, routinized, incorporated, and continued. Inconsistencies and
misperception of these terms persist. Terms such as penetration and institutionalization also began to
appear in literature referring to the embeddedness of public health programs and were conceptualized
as akin to sustainment. In the widely used taxonomy of implementation outcomes, Proctor and colleagues
(2011) acknowledged the inconsistencies in such terms and recognized this progression as problematic.
The problem, especially as it relates to the latter stages of implementation, is distinguishing concepts
related to the continued delivery of services from the integration of a health promotion program. This
previously unstudied area of inquiry is explored in study 2. Both studies within this dissertation used a
refined conceptualization of sustainment that moves beyond examining predictors of sustainment using
a binary measurement of the continuation or discontinuation of services delivered. The development of
recently constructed measures is further described below.
Moving Towards a Mechanistic Approach
Implementation science has evolved rapidly over the past decade. A taxonomy of implementation
outcomes (Proctor et al., 2011) provided a foundation for thinking about what the overall objective is (e.g.
adoption of EBP, feasibility, maintaining fidelity, etc.). Theoretical frameworks such as the Consolidated
Framework for Implementation Research (CFIR; Damschroder et al., 2009), the Promoting Action on
Research Implementation in Health Services framework (PARIHS; Rycroft-Malone, 2004), the Reach
Effectiveness Adoption Implementation Maintenance framework (RE-AIM; Glasgow et al., 1999), and the
Diffusion of Innovation framework (Greenhalgh et al., 2004) offered theoretically guided approaches to
identifying factors that may facilitate or impede such outcomes, ultimately providing a means to gauge
how successful implementation efforts were. While these frameworks and the numerous studies that
followed gave us an understanding of what influences successful implementation, importantly they did
not demonstrate how changes take place (Nilsen, 2015). In recent years the development and refinement
5
of strategies used to improve the success of implementation outcomes has become central to
implementation science; however, there are still gaps in understanding how a program moves through
various implementation phases (e.g. adoption to active implementation and sustainment). In recognizing
the need to refine implementation strategies, several studies applied innovative techniques to improve
the selection process (Powell et al., 2015; Powell et al., 2017). Perhaps the most useful was the
development of a tool designed for researchers and community stakeholders interested in improving
implementation efforts (Waltz et al., 2019). The interactive tool itself allows the user to select one or more
areas of concern (e.g. knowledge and beliefs about the intervention) and matches selected elements with
expert-recommended strategies to address them (e.g. conduct educational meetings, develop and
distribute educational materials, and create a learning collaborative).
In a review of methodological approaches to studying mechanisms and strategies in
implementation, Lewis and colleagues (2020) noted that little is known about how and why factors
influencing implementation are related. Furthermore, it is important to identify which strategies
correspond to particular implementation outcomes and examining the process between active
implementation and sustainment may help move the field forward. To address this, Lewis and colleagues
(2018) suggest analyses focused on building causal pathways that not only link mechanisms and strategies,
but also make the connection to proximal outcomes. Both studies take a different approach to building
causal pathways. First, innovation processes within organizations are iterative and non-linear by nature
(Anderson et al., 2004). When efforts are made to match factors that impede implementation with
strategies to address such barriers without understanding the interaction of multiple facilitators and
barriers, the use of the strategies designed to address them is limited. From an analytic approach, Lewis
and colleagues (2020) underscore that studies using analyses to test for moderators and mediators may
not give the full picture of mechanisms at play. Study 2 uses a configurational approach to address this
issue. By employing Qualitative Comparative Analysis (QCA), a set-theoretic approach, rather than simply
6
identifying which facilitators produce a desired outcome, I identified configurations of determinants that
are sufficient to implementation success. Study 2 moves slightly away from identifying determinants, and
questions how changes take place in the process of leading up to sustainment. Most process models focus
on the dissemination and early stages of implementation and knowledge translation (Nilsen, 2015).
Conceptual models that recognize sustainment, such as the Exploration, Preparation, Implementation,
Sustainment (EPIS; Aarons et al., 2011), illustrate a transition from the active implementation phase to
the sustainment phase without recognizing the process between the two in spite of numerous
implications that the embeddedness of a program within its host organization drives sustainment
(Scheirer & Dearing, 2011; Wilson & Kurz, 2008). Study 2 demonstrates the usefulness of program
integration as a mechanism of sustainment.
Dissertation Blueprint
Study Context: A Refined Conceptualization of Sustainment
This dissertation is based on data collected as part of a larger parent study. In 2014, the Substance
Abuse and Mental Health Services Administration (SAMHSA), one of the primary federal funding agencies
to support various health promotion and prevention initiatives, raised concern over the longevity of these
programs, vitally asking what transpires once initial seed funding dissolves. The path to understanding
this broader question unveiled another fundamental issue: sustainment had thus far been theoretically
conceptualized and there was little agreement amongst community stakeholders about how to define it
(Palinkas et al., 2015). In partnership with SAMHSA and the NIDA-funded Center for Prevention
Implementation Methodology, the parent study led by Dr. Lawrence Palinkas sought to refine how we
conceptualize sustainment and develop a tool to measure it (NIDA 1R34DA037516-01A1).
Guided by the CFIR, phases 1 and 2 of the parent study included semi-structured interviews, free
list exercises, and checklists to rate the importance of CFIR constructs to sustainment. Data triangulation
methods using three different sources of information found 15 elements required for sustainment:
7
ongoing coalitions, partnerships and networks; infrastructure and capacity to support sustainability;
community need for program; ongoing evaluation and performance outcomes; funding; consistency with
organizational culture; evidence of positive outcomes; development of a plan for implementation and
sustainment; presence of a program champion; institutionalization and integration of programs;
institutional support and commitment; community buy-in and support; program continuity; supportive
leadership; and opportunities for staff training (Palinkas et al., 2020). These findings subsequently
informed the development of the Sustainment Measurement System Scale (SMSS). Comprising 35 items
grouped within 8 factors and 1 outcome, the SMSS is the first psychometrically validated scale to measure
both sustainment as an outcome and its determinants (Palinkas et al., under review). Based on the parent
study, the current studies submitted as part of this dissertation measure sustainment using four
indicators: 1) the extent to which the program continued to operate as described in the original
application; 2) the extent to which the program continued to deliver preventive services to the intended
population; 3) the extent to which the program continued to deliver evidence-based services; and 4) the
extent to which the program periodically measured fidelity of services delivered.
Theoretical Framework and Conceptual Model
Based on a comprehensive synthesis of existing theories, constructs, and frameworks,
Damschroder and colleagues (2009) developed the CFIR, a meta-theoretical framework composed of 39
characteristics categorized across five distinct domains: intervention characteristics, outer setting, inner
setting, individual characteristics, and implementation (see Appendix). A recent systematic review found
that the CFIR has been used across a variety of settings, has been used to study multiple units of analyses,
and has been widely used in health services research (Kirk et al., 2015). More recently, the CFIR has also
been used as a foundation for the refinement and selection of implementation strategies (Waltz et al.,
2019). Aarons and colleagues (2011) asserted that while many studies have used the CFIR as a guiding
framework, the ability for any of the constructs found in determinant frameworks to predict outcomes
8
can vary with stage of implementation, and subsequently developed the EPIS model. While the EPIS was
originally designed to study implementation of EBP in agencies providing social services in child welfare
and mental health care, numerous projects and studies applying the EPIS model have validated its use
amongst a variety of public sector settings (Moullin et al., 2019). Constructs seen in the EPIS resemble
major domains in the CFIR. The EPIS model, however, advances the utilization of those constructs by
linking them to specific implementation phases, underscoring the importance of how those factors
behave and influence outcomes based on the phase of implementation.
The CFIR provided a strong foundation as a determinant framework to guide the methodology of
the parent study and informed the development of several study instruments to assess what elements
are seen as important to community stakeholders in the context of sustaining EBP. The EPIS model helped
to conceptually frame the relationships between contextual elements and the sustainment phase of
implementation across all three studies within this dissertation. Specifically, study 1 applies a
configurational approach to understanding the interdependent nature of inner and outer setting
constructs and their relationship to the sustainment phase; and study 2 explores the process between the
active implementation and sustainment phases.
Overall Objectives and Methods
This dissertation is comprised of two studies and focuses on further understanding how evidence-
based initiatives are sustained in community-based settings once initial seed funding dissolves; specifically
advancing the field by addressing the mechanisms and processes of sustainment. Central to the strength
of both studies is the use of an empirically developed conceptualization and measure of sustainment
(Palinkas et al., 2020; Palinkas et al., under review). Though studies examining determinants of
sustainment are plentiful (Cooper et al., 2015; Lennox et al., 2018), our knowledge is based on
methodological approaches that rely on traditional, inferential statistical analyses in spite of an awareness
that organizations are complex social systems where one cannot assume linear relationships between
9
facilitators of implementation and desired implementation outcomes (Anderson et al., 2004; Nilsen,
2015). The first study applied Qualitative Comparative Analysis, a set-theoretic approach, to examine the
synergistic formation of determinants and their relationship to sustainment. Additionally, in relation to
the process of sustainment, it has been widely accepted that the implementation of a program moves
from active implementation to the sustainment phase (Aarons et al., 2011); however, there is a body of
work across disciplines that recognizes the event by which a program becomes embedded within the
operations of the host organization (Goodman et al., 1993; Pluye et al., 2004), and acknowledges this as
a driver of sustainment (Wilson & Kurz, 2008). The second study explores how program-integration is
related to sustainment of EBP.
The specific aims of each study are as follows:
• Study 1: 1) identify configurations of elements that that are known to positively influence
sustainment of EBP; and 2) determine which of those causal pathways are sufficient to achieving
successful sustainment.
• Study 2: 1) identify what factors are important to integrating EBP within the host organization and
its operations once initial seed funding has been expended or terminated; and 2) determine if
integration and sustainment are linked and identify how they are linked.
Data Source and Sample
Studies 1 and 2 were informed by data collected from the parent study including grantee sites
across 7 SAMHSA prevention initiatives: 1) the Sober Truth on Preventing Underage Drinking Act (STOP-
Act) grant committed to prevent and reduce alcohol use among youth and young adults; 2) the
Implementing Evidence-based Prevention Practices in Schools (PPS) grants designed to address the
prevention of early childhood behavioral disorders; 3) the Strategic Prevention Framework State Incentive
(SPF-SIG) grants to prevent the onset of substance abuse amongst youth; 4) the Garrett Lee Smith (GLS)
10
State and Tribal Youth Suicide Prevention program to address depression, suicidal attempts, and
behavioral health problems linked to suicide; 5) the Substance Abuse and HIV Prevention Navigator
Program for Racial/Ethnic Minorities (Prevention Navigator, PN) serving minority populations including
those in tribal communities at-risk for substance abuse and HIV; 6) Minority Serving Institutions
Partnerships with Community-Based Organizations (MSI-CBOs) serving communities at risk for substance
abuse, HIV, and hepatitis-C infections; and 7) the HIV prevention services for at-risk racial/ethnic minority
youth and young adults (HIV-CBI). One hundred and forty-seven respondents representing 121 grantees
from all 7 programs were included in the analysis. Participants that existed in some form with substantial
adaptations or in much of the same form as it did when funded by this SAMHSA grant at the time of data
collection were included in both studies.
To maximize public health impact, it is necessary to advance research studying the sustainment
of health promotion initiatives (Shelton et al., 2018). Findings from this dissertation will inform the
mechanisms of sustainment, further advancing the study of strategies to improve the implementation of
EBP and thereby bringing value to policy makers, agency leaders, and providers invested in the long-term
success of the treatments they bring to communities in need.
11
References
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based
practice implementation in public service sectors. Administration and Policy in Mental Health and
Mental Health Services Research, 38(1), 4-23. https://doi.org/10.1007/s10488-010-0327-7
Aarons, G. A., Green, A. E., Willging, C. E., Ehrhart, M. G., Roesch, S. C., Hecht, D. B., & Chaffin, M. J. (2014).
Mixed-method study of a conceptual model of evidence-based intervention sustainment across
multiple public-sector service settings. Implementation Science, 9 Article 183.
https://doi.org/10.1186/s13012-014-0183-z
Anderson, N., De Dreu, C. K., & Nijstad, B. A. (2004). The routinization of innovation research: A
constructively critical review of the state‐of‐the‐science. Journal of Organizational
Behavior, 25(2), 147-173. https://doi.org/10.1002/job.236
Arsenault-Lapierre, G., Kim, C., & Turecki, G. (2004). Psychiatric diagnoses in 3275 suicides: A meta-
analysis. BMC Psychiatry, 4, Article 37. https://doi.org/10.1186/1471-244X-4-37
Bailey, R. K., Patel, T. C., Avenido, J., Patel, M., Jaleel, M., Barker, N. C., Khan, J. A., Ali, S. A., & Jabeen, S.
(2011). Suicide: Current trends. Journal of the National Medical Association, 103(7), 614-617.
https://doi.org/10.1016/S0027-9684(15)30388-6
Bernstein, S. L., & D’Onofrio, G. (2017). Screening, treatment initiation, and referral for substance use
disorders. Addiction Science & Clinical Practice, 12, Article 18. https://doi.org/10.1186/s13722-
017-0083-z
Boustani, M. M., Frazier, S. L., Becker, K. D., Bechor, M., Dinizulu, S. M., Hedemann, E. R., Ogle, R. R., &
Pasalich, D. S. (2015). Common elements of adolescent prevention programs: Minimizing
burden while maximizing reach. Administration and Policy in Mental Health and Mental Health
Services Research, 42(2), 209-219. https://doi.org/10.1007/s10488-014-0541-9
12
Bumbarger, B., & Perkins, D. F. (2008). After randomised trials: Issues related to dissemination of
evidence-based interventions. Journal of Children’s Services, 3(2), 55-64.
Cha, C. B., Franz, P. J., M. Guzmán, E., Glenn, C. R., Kleiman, E. M., & Nock, M. K. (2018). Annual
Research Review: Suicide among youth–Epidemiology, (potential) etiology, and
treatment. Journal of Child Psychology and Psychiatry, 59(4), 460-482.
https://doi.org/10.1111/jcpp.12831
Chamberlain, P., Brown, C. H., & Saldana, L. (2011). Observational measure of implementation progress
in community-based settings: The stages of implementation completion (SIC). Implementation
Science, 6, Article 116. https://doi.org/10.1186/1748-5908-6-116
Chambers, D. A., Pearson, J. L., Lubell, K., Brandon, S., O'Brien, K., & Zinn, J. (2005). The science of public
messages for suicide prevention: A workshop summary. Suicide and Life-Threatening
Behavior, 35(2), 134-145.
Conway, K. P., Green, V. R., Kasza, K. A., Silveira, M. L., Borek, N., Kimmel, H. L., Sargent, J. D., Stanton,
C., Lambert, E., Hilmi, N., Reissig, C. J., Jackson, K. J., Tanski, S. E., Maklan, D., Hyland, A. J., &
Compton, W. M. (2017). Co-occurrence of tobacco product use, substance use, and mental
health problems among adults: Findings from Wave 1 (2013–2014) of the Population
Assessment of Tobacco and Health (PATH) Study. Drug and Alcohol Dependence, 177, 104-111.
https://doi.org/10.1016/j.drugalcdep.2017.03.032
Cooper, B. R., Bumbarger, B. K., & Moore, J. E. (2015). Sustaining evidence-based prevention programs:
Correlates in a large-scale dissemination initiative. Prevention Science, 16(1), 145-157.
https://doi.org/10.1007/s11121-013-0427-1
13
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering
implementation of health services research findings into practice: A consolidated framework for
advancing implementation science. Implementation science, 4(1), 1-15.
https://doi.org/10.1186/1748-5908-4-50
Das, J. K., Salam, R. A., Arshad, A., Finkelstein, Y., & Bhutta, Z. A. (2016). Interventions for adolescent
substance abuse: An overview of systematic reviews. Journal of Adolescent Health, 59(4), S61-
S75. https://doi.org/10.1016/j.jadohealth.2016.06.021
Gamarel, K. E., Brown, L., Kahler, C. W., Fernandez, M. I., Bruce, D., & Nichols, S., & Adolescent Medicine
Trials Network for HIV/AIDS Intervention. (2016). Prevalence and correlates of substance use
among youth living with HIV in clinical settings. Drug and Alcohol Dependence, 169, 11-18.
https://doi.org/10.1016/j.drugalcdep.2016.10.002
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health
promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9),
1322-1327. https://doi.org/10.2105/AJPH.89.9.1322
Goldner, E. M., Lusted, A., Roerecke, M., Rehm, J., & Fischer, B. (2014). Prevalence of Axis-1 psychiatric
(with focus on depression and anxiety) disorder and symptomatology among non-medical
prescription opioid users in substance use treatment: Systematic review and meta-
analyses. Addictive Behaviors, 39(3), 520-531. https://doi.org/10.1016/j.addbeh.2013.11.022
Goodman, R. M., McLeroy, K. R., Steckler, A. B., & Hoyle, R. H. (1993). Development of level of
institutionalization scales for health promotion programs. Health Education Quarterly, 20(2),
161-178. https://doi.org/10.1177/109019819302000208
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in
service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4),
581-629. https://doi.org/10.1111/j.0887-378X.2004.00325.x
14
Healthy Minds Network study (2019). Retrieved May 31, 2020 from
https://healthymindsnetwork.org/wp-content/uploads/2019/09/HMS_national-2018-19.pdf
Hedegaard H, Curtin SC, Warner M. Increase in suicide mortality in the United States, 1999–2018. NCHS
Data Brief, no 362. Hyattsville, MD: National Center for Health Statistics. 2020. Retrieved May
31, 2020 from https://www.cdc.gov/nchs/products/databriefs/db362.htm
Hodge, L. M., & Turner, K. M. (2016). Sustained implementation of evidence‐based programs in
disadvantaged communities: A conceptual framework of supporting factors. American Journal of
Community Psychology, 58(1-2), 192-210. https://doi.org/10.1002/ajcp.12082
Johnson, A. M., Moore, J. E., Chambers, D. A., Rup, J., Dinyarian, C., & Straus, S. E. (2019). How do
researchers conceptualize and plan for the sustainability of their NIH R01 implementation
projects? Implementation Science, 14, Article 50. https://doi.org/10.1186/s13012-019-0895-1
Kirk, M. A., Kelley, C., Yankey, N., Birken, S. A., Abadie, B., & Damschroder, L. (2015). A systematic review
of the use of the consolidated framework for implementation research. Implementation
Science, 11, Article 72. https://doi.org/10.1186/s13012-016-0437-z
Lennox, L., Maher, L., & Reed, J. (2018). Navigating the sustainability landscape: A systematic review of
sustainability approaches in healthcare. Implementation Science, 13, Article 27.
https://doi.org/10.1186/s13012-017-0707-4
Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., Walsh-Bailey, C., & Weiner, B.
(2018). From classification to causality: Advancing understanding of mechanisms of change in
implementation science. Frontiers in Public Health, 6, Article 136.
https://doi.org/10.3389/fpubh.2018.00136
15
Lewis, C. C., Boyd, M. R., Walsh-Bailey, C., Lyon, A. R., Beidas, R., Mittman, B., Aarons, G. A., Weiner, B.
J., & Chambers, D. A. (2020). A systematic review of empirical studies examining mechanisms of
implementation in health. Implementation Science, 15, Article 21.
https://doi.org/10.1186/s13012-020-00983-3
Lipari, R. N., Park-Lee, E., & Van Horn, S. (2016). America's need for and receipt of substance use
treatment in 2015. The CBHSQ Report. Center for Behavioral Health Statistics and Quality,
Substance Abuse and Mental Health Services Administration, Rockville, MD. Retrieved May 31,
2020 from https://www.samhsa.gov/data/sites/default/files/report_2716/ShortReport-
2716.html
Luke, D. A., Calhoun, A., Robichaux, C. B., Elliott, M. B., & Moreland-Russell, S. (2014). Peer reviewed:
The program sustainability assessment tool: A new instrument for public health
programs. Preventing Chronic Disease, 11. https://doi.org/10.5888/pcd11.130184
Malick, R. (2018). Prevention of substance use disorders in the community and workplace. Indian Journal
of Psychiatry, 60(Suppl 4), S559-S563.
https://doi.org/10.4103/psychiatry.IndianJPsychiatry_24_18
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019). Systematic review of the
exploration, preparation, implementation, sustainment (EPIS) framework. Implementation
Science, 14, Article 1. https://doi.org/10.1186/s13012-018-0842-6
National Cancer Institute, National Institutes of Health. Implementation science: about IS. 2015.
https://cancercontrol.cancer.gov/IS/html. Accessed May 30, 2020
Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation
Science, 10, Article 53. https://doi.org/10.1186/s13012-015-0242-0
16
Palinkas, L. A., Spear, S. E., Mendon, S. J., Villamar, J., Valente, T., Chou, C. P., Landsverk, J., Kellam, S. G.,
& Brown, C. H. (2015). Measuring sustainment of prevention programs and initiatives: A study
protocol. Implementation Science, 11, Article 95. https://doi.org/10.1186/s13012-016-0467-6
Palinkas, L. A., Spear, S. E., Mendon, S. J., Villamar, J., Reynolds, C., Green, C. D., Olson, C., Adade, A., &
Brown, C. H. (2020). Conceptualizing and measuring sustainability of prevention programs,
policies, and practices. Translational Behavioral Medicine, 10(1), 136-145.
https://doi.org/10.1093/tbm/ibz170
Palinkas, L.A., Chou, C.P., Spear, S.E., Mendon, S.J., Villamar, J., & Brown, C.H. Measurement of
sustainment of prevention programs and initiatives: The sustainment measurement system
scale. Implementation Science. (under review).
Pluye, P., Potvin, L., Denis, J. L., & Pelletier, J. (2004a). Program sustainability: Focus on organizational
routines. Health Promotion International, 19(4), 489-500.
https://doi.org/10.1093/heapro/dah411
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K.,
& Kirchner, J. E. (2015). A refined compilation of implementation strategies: Results from the
Expert Recommendations for Implementing Change (ERIC) project. Implementation
Science, 10(1), 21. https://doi.org/10.1186/s13012-015-0209-1
Powell, B. J., Beidas, R. S., Lewis, C. C., Aarons, G. A., McMillen, J. C., Proctor, E. K., & Mandell, D. S.
(2017). Methods to improve the selection and tailoring of implementation strategies. The
Journal of Behavioral Health Services & Research, 44(2), 177-194.
https://doi.org/10.1007/s11414-015-9475-6
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M.
(2011). Outcomes for implementation research: Conceptual distinctions, measurement
17
challenges, and research agenda. Administration and Policy in Mental Health and Mental Health
Services Research, 38(2), 65-76. https://doi.org/10.1007/s10488-010-0319-7
Ram, D., Darshan, M. S., Rao, T. S., & Honagodu, A. R. (2012). Suicide prevention is possible: A
perception after suicide attempt. Indian Journal of Psychiatry, 54(2), 172-176.
https://doi.org/10.4103/0019-5545.99535
Rycroft-Malone, J. (2004). The PARIHS framework—A framework for guiding the implementation of
evidence-based practice. Journal of Nursing Care Quality, 19(4), 297-304.
Scheirer, M. A. (2005). Is sustainability possible? A review and commentary on empirical studies of
program sustainability. American Journal of Evaluation, 26(3), 320-347.
https://doi.org/10.1177/1098214005278752
Scheirer, M. A., & Deering, J. W. (2011). An agenda for research on the sustainability of public health
programs. American Journal of Public Health, 101, 2059-2067.
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018). The sustainability of evidence-based interventions
and practices in public health and health care. Annual Review of Public Health, 39, 55-76.
Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of
new programs and innovations: A review of the empirical literature and recommendations for
future research. Implementation Science, 7, Article 17. https://doi.org/10.1186/1748-5908-7-17
U.S. Department of Health and Human Services, National Institute of Mental Health. (2017). Suicide
prevention. Retrieved May 31, 2020 from https://www.nimh.nih.gov/health/topics/suicide-
prevention/index.shtml
Waltz, T. J., Powell, B. J., Fernández, M. E., Abadie, B., & Damschroder, L. J. (2019). Choosing
implementation strategies to address contextual barriers: Diversity in recommendations and
future directions. Implementation Science, 14, Article 42. https://doi.org/10.1186/s13012-019-
0892-4
18
Wilson, K. D., & Kurz, R. S. (2008). Bridging implementation and institutionalization within organizations:
Proposed employment of continuous quality improvement to further dissemination. Journal of
Public Health Management and Practice, 14(2), 109-116.
https://doi.org/10.1097/01.PHH.0000311887.06252.5f
19
Chapter Two (Study One)
Sufficient Pathways for Maintaining Prevention Programs Long-Term:
A Configurational Approach Using Qualitative Comparative Analysis
20
Abstract
Background: An understanding of what determines sustainment is critical to identifying strategies
designed to improve implementation and long-term maintenance of evidence-based practice in
community-based organizations. Several elements such as planning for sustainment, program champions,
coalition support, and meeting the needs of the community, have emerged as key factors or facilitators
of sustainment. However, it is unknown how these elements are interrelated within a complex social
system; in other words, how they act when combined with one another to produce desired sustainment.
Methods: Using data from a larger parent-study designed to understand how programs are sustained
once initial funding is expended, the current study included representatives from 145 grantee sites funded
by seven SAMHSA-funded programs focused on reducing or preventing various mental health and
substance-use related disorders among at-risk youth. A set-theoretic approach, fuzzy-set Qualitative
Comparative Analysis (fsQCA), was used to identify configurations of conditions that, when combined in
causal pathways, are sufficient for producing sustainment. Sustainment was assessed by the extent to
which grantees: 1) continued to operate as described in the original application, 2) continued to deliver
preventive services to the intended population, 3) continued to deliver evidence-based services, and 4)
periodically measured fidelity of services delivered. Fuzzy-set QCA was conducted to assess five conditions
derived from the Consolidated Framework for Implementation Research (CFIR): financial stability;
responsiveness to community needs and values; coalitions, partnerships, & networks; organizational
capacity and staff capability; and characteristics of the implementation process. The extent to which a
case had membership in a given set (e.g., non-membership to full membership) and calibration on a 5-
point fuzzy scale was informed by relevant theoretical and substantive knowledge. Following an analysis
of necessity to determine which conditions are a superset of sustainment, sufficiency analysis was
conducted to determine which configurations are a subset of sustainment. Results: Initial minimizations
based on Boolean logic found 14 configurations of a possible 32 to be relevant to sustainment. Standard
21
analysis found two configurational pathways with high consistency and moderate coverage, indicating
that when these particular constructs combine, they are, as a unit, sufficient to produce sustainment: 1)
community responsiveness and organizational capacity when combined with process, and 2) community
responsiveness and organizational capacity when combined with coalitions, networks, and partnerships.
Conclusion: This study moved beyond identifying individual factors that predict sustainment by using a
configurational approach to identify how these factors work in combination. Such knowledge assigns
value to specific determinants and helps to prioritize which aspects can further guide implementation
efforts in practice. Knowing which pathways are sufficient for sustainment will provide guidance on the
selection of implementation strategies and may be a cost-saving approach.
22
Introduction
A key aim of health services research is to not only develop evidence-informed interventions, but
to ensure that they benefit populations for which they were intended. While initial funding may help bring
an evidence-based practice (EBP) into a community, without sustaining the program or initiative, any
considerable public health impact becomes unfeasible (Cooper, 2015). To date, while some work on
fundamental mechanisms link determinants of specific implementation outcomes to active
implementation phases (Lewis et al., 2018), these determinants do not pertain to facilitating the
sustainment of EBP. Additionally, this work has not focused on pinpointing what determinants or clusters
of determinants are sufficient to achieve sustainment. Individual elements facilitating sustainment and
strategies to improve implementation have been identified; however, we have yet to identify mechanisms
that illustrate how those elements synergistically form pathways to sustainment. Identifying pathways
that show how elements that facilitate sustainment interact with one another and which ones are
sufficient to sustaining EBP can inform the selection of strategies to improve implementation in practice.
Among the determinants linked to sustainment identified in previous studies are financial
stability; community responsiveness; coalitions, partnerships and networks; organizational capacity and
staff capability, and implementation process (Cooper et al., 2015; Lennox et al., 2018; Palinkas et al.,
2020). Mental health and prevention initiatives implemented in community settings often rely on grants
and contracts with external agencies and foundations to support activities and costs associated with the
initial launch of a given program. This funding is time limited however, and inevitably raises questions
about an organization’s ability to sustain a given initiative once such initial funding is no longer available.
Indisputably, any organization needs ongoing financial support in order to continue providing services,
and how important funding is to the sustainment of EBP has received much attention in recent years
(Hunter et al., 2015; Scheirer, 2005). In a study of 112 programs that were originally funded by the Robert
23
Wood Johnson Foundation, Stevens and Peikes (2006) found that 92% of the programs lasted at least one
year after funding ended. Several key factors found among the sustaining programs involved maintaining
or acquiring funds through alternate avenues, including strategies to develop revenue-generating
activities, building individual donor pools, and capitalizing on conversations with funders to help increase
fundraising efforts. Over 70% of sustaining programs maintained or increased their budget, and 8% were
able to increase or maintain the scope of their projects even with a decreased budget. More recent studies
have also reported similar results, showing continued funding to be significantly associated with the
continued delivery of full or partial evidence-based treatments (Hunter et al., 2017; Tibbits et al., 2010).
Collectively, these findings demonstrate the importance of securing funding in order to maintain the
activities of a given initiative. However, it is still unclear how this might change given the importance of
other determinants of sustainment.
Community responsiveness, which refers to meeting the needs of the community as well as
responding to what the community values, is one such determinant of sustainment. Stevens and Peikes
(2006) found that nearly three-fourths of project directors (across 112 projects) believed that strong
community support was very important to the sustainment of their projects. Such support is often based
on the willingness and ability to adapt processes and tailor interventions to meet the needs of the
community in order to maintain program effectiveness (Castro et al., 2004; Feinberg et al., 2008) and
sustainability (Aarons et al., 2014; Chamberlain et al., 2011; Chambers et al., 2013; Han & Weiss, 2005).
Linkages between systems, organizations and partnerships involving service system stakeholders
have also been identified as an important component of program sustainment (Greenhalgh et al., 2004;
Stevens & Peikes, 2006). Scheirer and Dearing (2011) suggested that maintaining coalition support
established during initial implementation may be an important component of sustaining practices and
may even contribute to additional future benefits for the community. This was also found to be a key
24
factor in a study of over 240 grantees where various prevention initiatives were sustained more than two
years after initial seed funding was expended (Cooper et al., 2015).
Organizational capacity (Langley et al., 2010; Smith & Donze, 2010) and staff training and
knowledge (Chor et al., 2015; Duncombe, 2018) have been shown to act as both facilitators and barriers
to implementing EBP. Several studies have found that organizational capacity, characteristics of the
individuals involved in implementing the program, planning for sustainment, and quality training efforts
are associated with greater levels of EBP sustainment (Cooper et al., 2015; Forman et al., 2009; Hunter et
al., 2017; Lennox et al., 2018; Stevens & Peikes, 2006). Characteristics of implementation processes such
as having a program champion to advocate for the program when there are difficult barriers to overcome,
regularly monitoring and giving feedback about the progress of program goals, and sharing program
effectiveness or positive client outcomes have also long been believed to influence implementation
(Damschroder et al., 2011). Studies have shown that having a champion in place and implementation
leadership is associated with sustainability over time (Shelton et al., 2018; Stevens & Peikes, 2006).
Monitoring progress over time and demonstrating effectiveness of the program by reporting positive
client outcomes have also been found to be associated with and facilitate sustainment of EBP over time
(Blasinsky et al., 2006; Lennox et al., 2018). These pillars of organizational capacity, however, are dynamic
in nature and may change course over time depending on staff turnover, goals, and resources.
The importance of all of these constructs - organizational capacity, staff capability, planning for
implementation, community responsiveness, coalition support building, implementation leadership,
monitoring and feedback, sustained funding, and training and increasing organizational capacity - also
appear prominently in the use of strategies to improve implementation and sustainment of EBP
(Hailemariam et al., 2019; Powell et al., 2015). However, many of these studies were either purely
descriptive in nature or have exclusively used inferential statistical methods. By assuming that these
associations are additive and linear, these findings overlook how elements required for sustainment may
25
work in conjunction with one another (Nilsen, 2015). In settings where implementation and sustainment
of EBP rely on multi-level components across layers of organizational and systemic complexities, it is
imperative we move beyond a list of variables associated with sustainment and begin identifying the
mechanisms involved (Lewis et al., 2018). While most studies do not focus on evaluating the cost
effectiveness of using implementation strategies (Powell et al., 2019; Vale et al., 2007); there are distinct
costs that may be associated with applying implementation strategies within an agency, such that the
agency is not at full liberty to select numerous strategies to help improve the success of a given program
being implemented within their agency (Hoomans & Severenes, 2014). Maximizing every strategy that
addresses all determinants of sustainment would be costly and ineffective. For instance, improving
responsiveness to community needs alone may not improve sustainment without also improving
organizational infrastructure and capacity. A valuable tool for selecting implementation strategies in a
targeted way would be to identify the most impactful and cost-effective paths to sustainability. Several
studies emphasized the importance of funding to sustainment (Stevens et al., 2006, Tibbits et al., 2010),
however what they actually proceeded to describe is a strong relationship of financial planning to
sustainment of EBPs. This can be observed as funding, but this is also an aspect of planning for
sustainability; which are, in fact, elements belonging to two different constructs, financial stability and
organizational capacity, as found by Palinkas and colleagues (under review). Stirman and colleagues
(2012) found that having a program champion was less important than funding, and others have posited
that while factors such as strategic planning, evaluation, and communication are important, funding may
trump these organizational characteristics (Hunter et al., 2015). These examples illustrate the incomplete
picture of what drives sustainment, underscoring the need for additional information about the dynamics
and interrelationships between factors associated with sustainment (Shelton et al., 2018). Without
studying the mechanisms of these separate elements, we do not actually know how they interact with
one another and facilitate sustainment. Accordingly, the present study aims to: 1) identify configurations
26
of elements that that are known to positively influence sustainment of EBP, and 2) determine which of
those causal pathways are sufficient to achieving successful sustainment.
To identify the mechanisms of sustainment and meet the growing needs of community
organizations seeking to improve implementation in practice, I used a configurational comparative
method known as Qualitative Comparative Analysis (QCA). Developed in the 1980’s to analyze the nature
of interdependent relationships within complex social systems (Ragin, 1987; Ragin, 2014), QCA offers an
important lens to the study of implementation in health service settings (Kane et al., 2014). Relying heavily
on substantive and theoretical knowledge, QCA is a mixed-methods approach and analytic technique
designed specifically to identify sets (or combinations) of characteristics in relation to the desired
outcome. Based on subset relations theory, the key features of QCA are to: 1) identify factors that are
necessary to produce the outcome, 2) identify pathways of combinations for which there may be multiple
possible configurations, known as equifinality (Fiss, 2011), and 3), identify which of the multiple
configurations that may facilitate an outcome are sufficient; in other words, which pathways are the most
common and consistently appear in the context of a given desired outcome. In recent years, the use of
configurational approaches has gained considerable traction in health services and implementation
research (Cragun et al., 2016; Dy et al., 2005; Thygeson et al., 2012; Yakovchenko, 2020). Findings from
these studies, all showing multiple pathways to a given outcome of interest, illustrate the nature of
configurational approaches and the ability to identify a smaller number of pathways (usually reduced from
a much larger number of possible configurations) that are inherently sufficient to achieving successful
implementation.
Methods
Parent Study
The parent study is theoretically grounded in the Consolidated Framework for Implementation
Research (CFIR), a widely used framework illustrating factors from multiple domains (outer setting, inner
27
setting, and intervention, individual and process characteristics) known to influence key implementation
outcomes (Damschroder et al., 2009). To develop a tool that can be used to measure sustainment of
health care programs, the parent study primarily investigated what is meant by the term sustainment and
what is important for sustainment from the perspectives of multi-level stakeholders across a variety of
settings delivering various mental health, substance use, and prevention initiatives funded by SAMHSA.
Based on a triangulation of three data sets (a semi-structured interview, free list exercises, and a CFIR
checklist of importance), fifteen elements were identified as requirements for sustainment and four
elements were found to be indicators of sustainment (Palinkas et al., 2019). This advanced the
conceptualization of sustainment by enhancing the often-used narrow definition that simply asked
whether or not the project continued to deliver services. These findings informed the development of the
Sustainment Measurement System Scale (SMSS). The scale was then assessed for psychometric properties
using a confirmatory factor analysis (Palinkas et al., under review). The current study uses both the new
conceptualization for sustainment (four indicators) as well as the constructs found to be associated with
sustainment as the basis for the analysis.
Representatives from the following seven grants funded by SAMHSA delivering mental health and
substance use prevention initiatives at both the state and community levels took part in the study: 1) the
Sober Truth on Preventing Underage Drinking Act (STOP-Act) grants to prevent and reduce alcohol use
among youth and young adults; 2) the Implementing Evidence-based Prevention Practices in Schools (PPS)
grants designed to address the prevention of early childhood behavioral disorders; 3) the Strategic
Prevention Framework State Incentive (SPF-SIG) grants to prevent the onset of substance abuse amongst
youth; 4) the Garrett Lee Smith (GLS) State and Tribal Youth Suicide Prevention program to address
depression, suicidal attempts, and behavioral health problems linked to suicide; 5) the Substance Abuse
and HIV Prevention Navigator Program for Racial/Ethnic Minorities (Prevention Navigator, PN) serving
minority populations including those in tribal communities at-risk for substance abuse and HIV; 6)
28
Minority Serving Institutions Partnerships with Community-Based Organizations (MSI-CBOs) serving
communities at risk for substance abuse, HIV, and hepatitis-C infections; and 7) the HIV prevention
services for at-risk racial/ethnic minority youth and young adults (HIV-CBI). The total sample included one
hundred eighty-six representatives from 145 grantee sites who agreed to participate. All data were
collected using web-based survey instruments. A 5-point Likert scale ranging from 0 “not at all” to 5 “to a
great extent” was used for all survey items assessing sustainment and factors associated with
sustainment. Additional information about participating organizations and initiatives, analytic methods,
and details pertaining to the development of the SMSS can be found elsewhere (Palinkas et al., 2015;
Palinkas et al., 2020).
Fuzzy-set Qualitative Comparative Analysis (fsQCA)
Qualitative comparative analysis is primarily a case-based approach. Knowledge of cases and
contextual factors drive decision-making throughout the analytic process from the selection of conditions
to set membership and calibration. These aspects replace approaches in quantitative analysis where
decisions and thresholds are based on inferential statistics. While QCA is a tool designed to answer
questions of cause and effect, the approach greatly differs from those aimed at identifying estimates of
the average effect(s) of variables (Vis, 2012). In contrast, QCA concerns configurations, whereby multiple
factors act together to produce an outcome. Rather than competing to explain variance as in regression-
based models (Fiss, 2007), configurational approaches account for selected factors or conditions that,
when combined, may produce a desired outcome. As such, QCA may complement or compete with
findings from regression-based models. Relationships are described using set-theoretic language such as
subsets and supersets. Superset relations are used to describe necessary conditions whereby a condition
X is necessary for Y but does not guarantee the outcome Y. In necessity analysis, the causal condition
should be a superset of the outcome. In sufficiency analysis, the causal condition should be a subset of
the outcome (Kane et al., 2014; Ragin, 2008). Sufficiency analysis may identify one or more causal
29
pathways, a concept known as equifinality (Fiss, 2011). It is also important to understand the relationship
between necessity and sufficiency. The relationship between the two terms can take several forms: a
cause can be both necessary and sufficient, a cause can be sufficient but not necessary, a cause can be
necessary but not sufficient, and it may also be the case that a cause is neither necessary nor sufficient
(Ragin, 2008, pg. 42). Lastly, QCA is based on the assumption that relationships to the outcome are
asymmetrical in nature, such that one configuration consisting of the presence of conditions that explain
the presence of outcome Y does not necessarily mean that the absence of those same conditions in a
given configuration also explain the negation of the same outcome. This is key in the context of complex
social systems where adding or removing a factor (such as implementation leadership) may not
configurationally explain the presence or absence of an outcome (such as adoption of an EBP). A
comprehensive summary showing the mathematical equations and application of procedures can be
found elsewhere (Liu et al., 2017; Mendel & Korjani, 2012). An abbreviated version of the key steps is
provided in the analytic strategy below.
Data and Measurements
Outcome
Using data from the parent study described above, the following four indicators of sustainment
informed the outcome: 1) continue to operate as described in the original application, 2) continue to
deliver preventive services to intended population, 3) continue to deliver evidence-based services, and 4)
periodically measure fidelity of services delivered (Palinkas et al., under review).
Conditions
A confirmatory factor analysis revealing an 8-factor model (Palinkas et al., under review) was the
basis for the selection of conditions in the current study. However, in part due to the diversity in
configurations important to capture in QCA (Marx, 2010), some constructs were combined into superset
30
conditions based on what conditions theoretically fit together (bolded below). This resulted in the
following conditions: 1) financial stability, 2) community responsiveness (including community needs and
values, 3) coalitions, networks and partnerships, 4) organizational capacity (including organizational
capacity and staff capability), and 5) process (including champions, monitoring and feedback, and
outcomes). A full list of items by condition can be found in Table 2.1.
Analytic Strategy
Step 1: Selection of Cases
Prior to beginning formal analysis, several steps were taken to create a data matrix, including
selecting cases, identifying the outcome and conditions, operationalizing the outcome and conditions,
assigning membership scores, and calibrating each set. The selection of cases began with the projects that
continued to exist at the time of data collection in the parent study (n = 184). Thirty-four of those cases
had missing data on all or most of the conditions of interest and were removed, and an additional five
cases did not have sufficient data to inform set membership and calibration (see below). I used listwise
deletion to account for missing data. As such, the current study was based on a total sample of 145 cases
with no missing values that would impact the membership of a case in any given condition. See Fig. 2.1
for percentage of respondents by SAMHSA program type.
Steps 2 and 3: Operationalization; Set membership and Calibration
Operationalization of a condition is based on the items that were considered for a given condition
or superset condition. See Table 2.2 for further details on how the outcome and all conditions were
operationalized. In crisp-set QCA, the outcome and conditions are dichotomized (0/1) and all cases are
assigned to be either fully out or fully in a given condition. In fsQCA, outcomes and conditions are defined
by a fuzzy-set falling between 0 and 1 that allows a researcher to qualitatively assess a case’s membership
in the outcome and conditions (Ragin, 2008). When enough information is known about a given topic
area, fuzzy-set analysis allows for a deeper understanding of causal conditions and their relationship to
31
the outcome of interest. Using a qualitative assessment based on what is known about contextual factors
positively associated with and influencing sustainment (described above), each case was reviewed
individually and assessed on the basis of how much the case was in or out of a given condition. The current
study used an indirect method to calibrate membership sets on a 5-point scale based on qualitative
groupings: 0 (fully out), .2 (more out than in), .4 (highly ambiguous/don’t know), .7 (more in than out),
and 1 (fully in). Of note, these scores differ significantly from, say, a traditional Likert scale variable,
whereby they are not assessed in relation to one another, rather determined their membership in a given
set. As such, one must be careful not to interpret .4, for instance, as having the same meaning as a lower
score on a Likert scale rating. Rather, it is a qualitative assessment in terms of membership set; .4 in this
case indicates high ambiguity and is therefore calibrated in the middle of the fuzzy set. See Table 2.2.
In fsQCA, this process is a qualitative assessment such that decisions about membership and
calibration should be based on knowledge of the case, methodological processes, and contextual factors
known from theoretical and substantive knowledge that help to define the problem you are seeking to
study. In the current study, I first reviewed Likert-scale responses for each item within a given condition.
When all scores were rated as “great extent” or “not at all” the case was given a 1 or 0, respectively for
that condition. In cases when other ratings such as “a little extent,” “some extent,” or “moderate extent”
presented conflicting information and determining whether a case was more in or more out was more
challenging, three steps were taken to ensure the most accurate membership set. First, a review of
qualitative statements from the survey for any relevant information for a given condition, followed by a
review of specific items where conflicting information appeared; if these items were pre-determined to
have more weight in that condition based on substantive knowledge, membership was assigned
accordingly (e.g. a “great extent” rating on all items except a “not at all” or “little extent” rating on a more
heavily weighted item, would result in a .2 or .7 depending on the nature of the grouping). The selection
of pre-determined items that held more weight in a given condition were based on knowledge from
32
previous studies and are highlighted in bold in Table 3.1. Second, I then reviewed additional items from
the survey when relevant to a given condition; for example, “have you talked to your SAMHSA project
director to plan for sustainment” was originally asked as a demographic question and not included within
the capacity construct but is related to planning for sustainability. Scores from such additional items were
taken into consideration in cases where there was uncertainty about membership. Lastly, even after
implementing the aforementioned approaches if there still remained a question as to how fully that case
was in a given condition, the case was given a .4 to indicate high ambiguity. See Table 2.3 for case
examples.
Step 4: Necessity Analysis
Prior to identifying configurations of conditions and sufficient pathways, an analysis of necessity
was performed to identify, for example, the instances that a causal condition may be necessary but not
sufficient, or as described by Ragin (2008), where instances of an outcome constitute a subset of instances
of a cause. Findings were assessed using two indicators to measure the strength of subset relationships,
consistency and coverage. Consistency scores above .9 are generally considered to indicate a condition
that is usually necessary for an outcome. All conditions were retained for further analysis based on
previously described substantive knowledge about elements that facilitate and determine sustainment.
Step 5: Sufficiency Analysis
The aim of fsQCA is to identify configurations of causal conditions, that is, to show the
interdependent relationships of factors that, when occurring together, produce a given outcome. The
sufficiency analysis examined cases with the same causal conditions to see if they also share the same
outcome (Ragin, 2018). Based on the findings of the necessity analysis, all five conditions were included
in the sufficiency analysis. Of the 32 (i.e., 2
5
) possible configurations that the study began with, the truth
table analysis after the first Boolean minimization revealed that there were only 14 possible
configurations that had a relationship to sustainment (see Table 3.4). Proportional reduction in
33
inconsistency (PRI) scores were assessed to avoid simultaneous subset relations and exhibited a close
match between both PRI consistency and raw consistency indicating that these configurations are in fact
consistent with the outcome (Greckhammer et al., 2018). Given the large sample, the threshold for raw
consistency was set to ≥ .9 and PRI consistency was set to ≥ .75, with a case frequency of ≥ 1 to include
more rare configurations (Greckhamer et al., 2013). The outcome was set to 1 for configurations meeting
these thresholds (n = 14). For remaining configurations with no cases, the outcome was set to 0 (n = 18).
See Table 2.4 for all configurations meeting threshold consistencies for case frequency, raw consistency,
and PRI consistency scores. A standard analysis was conducted with the assumption that the presence of
each condition will contribute to sustainment.
Step 6: Asymmetrical Relationships
Since fsQCA does not assume symmetrical relationships, it is always recommended that a further
step to assess causal conditions on the negation of the outcome be conducted (Schneider & Wagemann,
2012). All conditions were selected as being present on the negated outcome sustainment (essentially,
low sustainment) to explore any further explanations of causal conditions that may be different than
configurations that explain the presence of sustainment.
Results
Of the five causal conditions tested, none were indicated as necessary based on consistency
scores (see Table 2.5). Sufficiency analysis detected two pathways with high consistency and moderate
coverage 1) community responsiveness when combined with organizational capacity and partnerships
(consistency = .969); and 2) community responsiveness when combined with organizational capacity and
process (consistency = .962). Both pathways were represented by 19 cases with ≥ .5 membership in their
respective configurations. Two additional pathways with good consistency and coverage included the
absence of a given condition as denoted by the logical operator (~). Take, for instance, the following
pathway: ~funding*partner*capacity; this can be interpreted as a configuration illustrating high
34
functioning and supportive coalitions (partnerships) AND high organizational capacity with plans for
sustainment and well-trained staff (capacity) when combined with not high financial stability (e.g. cases
that had tenuous funding or hadn’t fully sustained funding) as a sufficient pathway for sustaining EBP. See
Figure 2.2 for all pathways sufficient for sustainment. Since QCA does not assume symmetry, to
understand the nature of all set-theoretic relationships (Ragin, 2008; Rixhoux & Ragin, 2009; Schneider &
Wagemann, 2010), the presence of each condition on the absence of high sustainment was also assessed
(see Table 2.6).
Discussion
The Dynamic Sustainability Framework (Chambers et al., 2013) suggests that sustainability is more
than a static outcome but a process as well, and emphasizes the continued learning and evaluation,
problem-solving, and ongoing adaptation of the interventions to enhance their fit with contexts and
populations that differ from those in which the interventions were initially tested. This further highlights
the importance of context and the need to understand how the interaction of elements, rather than
individual associations, may facilitate sustainment. Sustainment within highly variable contexts (i.e.
organizational capacity, staffing, leadership, how the project responds to community needs) is inherently
non-linear; however, previous studies have largely relied on traditional methodological approaches to
identify what elements across inner-organizational and outer contexts facilitate sustainment.
Our analysis of configurational pathways revealed two pathways with high consistency and
moderate coverage: 1) community responsiveness and organizational capacity when combined with
coalitions, networks, and partnerships, and 2) community responsiveness and organizational capacity
when combined with characteristics of the implementation process. Notably, community responsiveness
and organizational capacity appeared in both of these pathways. These two constructs in particular are
consistent with how previous research has identified them as being critical components to sustainment
(Aarons et al., 2014; Castro et al., 2004; Hunter et al., 2017; Lennox et al., 2018; Shelton et al., 2018;
35
Stevens & Peikes, 2006; Wiltsey Stirman et al., 2012). Additionally, in relation to understanding ways to
improve implementation of EBPs, and in this case, ways to sustain EBPs after the initial funding period is
expended, these findings provide an opportunity to methodically select implementation strategies based
on what is sufficient to producing desirable sustainment outcomes. Furthermore, these findings
contribute to the growing interest in linking implementation strategies to specific implementation
outcomes (Lewis et al., 2018), and lowers the need to develop new strategies. In other words, rather than
focusing on refining or reducing the number of existing strategies (Powell et al., 2015; Powell et al., 2017;
Yakovchenko et al., 2020), or building a model for strategies based only on identifying the barriers present
without knowing which implementation outcome they may or may not address, these findings represent
reduced pathways that are sufficient and consistent with achieving long-term sustainment of EBP,
allowing for a more methodical approach when considering strategy selection. Using an interactive tool
(Waltz et al., 2019), researchers and community stakeholders can select a configuration of strategies that
correspond with the pathways found in this study as being sufficient for sustaining an EBP.
While several previous studies have emphasized the importance of financial stability to
sustainment, the current study found that funding does not actually always have to be a key requirement
for sustainment. Notably, it was the presence of having other elements working together, primarily,
community responsiveness, partnerships, and organizational capacity that consistently produced high
sustainment even when cases demonstrated low financial stability. This may be in part due to several
reasons. First, financial stability in the context of this study primarily pertained to whether the project
itself has alternate funding mechanisms, whether the project has both stable and flexible funding, and
ultimately whether the project has been able to sustain funding. Additionally, being a part of a larger
organization and having coalition support may in fact subsidize costs associated with implementing an
EBP (Cooper et al., 2015; Steven & Peikes, 2006). It may also be the case that funding of an EBP is more
important during earlier implementation phases. For example, as the program is being launched, training
36
of staff, costs of acquiring any necessary materials, costs of developing program evaluation and fidelity
monitoring procedures are notably a part of implementing an EBP (Aarons et al., 2014). However, once
staff are trained and such procedures are in place, funding in and of itself may not actually be necessary
for maintaining the initiative over time. Rather, it may be the presence of other elements, especially
pertaining to coalition functioning and support, continuing to adapt and meet the needs of the
community, and having an overall process in place that involves a key program champion along with
demonstrating positive outcomes that overshadow the need for high financial stability.
Regression-based models that rely on normal distributions and bi-variate relationships only show
the symmetrical associations between conditions and the outcome (Vis, 2012). In complex social systems,
it is often not the case that factors are either present or absent – constructs such as the support of a
coalition or the strength of planning for sustainment lie on a spectrum and may wax and wane at various
points in time. As such, studying associations without assuming symmetrical and linear relationships to a
desired outcome, allows social scientists to see a fuller picture.
While QCA is rooted in mixed-methods research, its approach is qualitative in nature and has
some related limitations and should include other components of qualitative studies, especially in regard
to coding and understanding data. There are several decision-making points throughout the analysis that
rely on a researcher’s knowledge of the problem being investigated, theory, and in-depth case knowledge.
When operationalizing the conditions, assigning set membership scores, and calibrating fuzzy set data, it
is imperative that QCA mimic components of qualitative research, especially in regard to coding.
Qualitative coding, for example, includes multiple raters, with an established procedure for discussing
ratings, reaching consensus, and establishing inter-rater reliability (Fereday & Muir-Chochrane, 2006;
Hamilton & Finley, 2019; Palinkas et al., 2019). Such procedures carried out during analysis can strengthen
the value of findings from configurational analysis.
37
To further understand the construct of financial stability as it pertains to sustaining EBP, future
research can turn explore: a) what kinds of costs are associated with earlier implementation phases,
during adoption and in the first year of existence; and b) how does this differ from the costs associated
with implementing EBP in years 3-5 and beyond. Costs associated with different implementation phases
and sustainment may indeed be quite different (Roundfield & Lang, 2017). Training costs, for instance, in
the earlier stages may involve training multiple providers, supervisors, and even some agency leaders
based on the extent of their involvement. And while community-based organizations do experience
considerable rates of staff turnover (Aarons et al., 2009; Beidas, et al., 2016; Mor Barak, et al., 2001), it
may be the case that as providers are slowly replaced over time, staff and supervisors that have already
been trained in the EBP can train newcomers such that they do not need to allocate funds for new staff
to receive external training. Similarly, once the program has been in place for a period of time, it may be
the case that the organization no longer needs to allot funds for logistical necessities (e.g. protocols,
databases) that support implementation activities. Additionally, given this new knowledge of which
pathways are sufficient to producing sustainment, researchers can focus on matching implementation
strategies to these pathways rather than building a model of strategies that is not rooted in such
knowledge. Rather than selecting which strategies to implement purely based on barriers that might be
present, researchers can build on findings from the current study that identified key pathways shown to
be sufficient to producing sustainment. Researchers can further identify which implementation strategies
correspond with pathways identified in this study and study how the configuration of those strategies
combine to improve sustainment of EBP.
38
References
Aarons, G. A., Sommerfeld, D. H., Hecht, D. B., Silovsky, J. F., & Chaffin, M. J. (2009). The impact of
evidence-based practice implementation and fidelity monitoring on staff turnover: evidence for
a protective effect. Journal of Consulting And Clinical Psychology, 77(2), 270–280.
https://doi.org/10.1037/a0013223
Aarons, G. A., Green, A. E., Willging, C. E., Ehrhart, M. G., Roesch, S. C., Hecht, D. B., & Chaffin, M. J.
(2014). Mixed-method study of a conceptual model of evidence-based intervention sustainment
across multiple public-sector service settings. Implementation Science, 9, Article 183.
https://doi.org/10.1186/s13012-014-0183-z
Beidas, R. S., Marcus, S., Wolk, C. B., Powell, B., Aarons, G. A., Evans, A. C., Hurford, M. O., Hadley, T.,
Adams, D. R., Walsh, L. M., Babbar, S., Barg, F., & Mandell, D. S. (2016). A prospective
examination of clinician and supervisor turnover within the context of implementation of
evidence-based practices in a publicly-funded mental health system. Administration and Policy
in Mental Health and Mental Health Services Research, 43(5), 640-649.
https://doi.org/10.1007/s10488-015-0673-6
Blasinsky, M., Goldman, H. H., & Unützer, J. (2006). Project IMPACT: A report on barriers and facilitators
to sustainability. Administration and Policy in Mental Health and Mental Health Services
Research, 33(6), 718-729. https://doi.org/10.1007/s10488-006-0086-7
Castro, F. G., Barrera, M., & Martinez, C. R. (2004). The cultural adaptation of prevention interventions:
Resolving tensions between fidelity and fit. Prevention Science, 5(1), 41-45.
https://doi.org/10.1023/B:PREV.0000013980.12412.cd
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013). The dynamic sustainability framework:
Addressing the paradox of sustainment amid ongoing change. Implementation Science, 8, Article
117. https://doi.org/10.1186/1748-5908-8-117
39
Chamberlain, P., Brown, C. H., & Saldana, L. (2011). Observational measure of implementation progress
in community-based settings: The stages of implementation completion (SIC). Implementation
Science, 6, Article 116. https://doi.org/10.1186/1748-5908-6-116
Chor, K. H. B., Wisdom, J. P., Olin, S. C. S., Hoagwood, K. E., & Horwitz, S. M. (2015). Measures for
predictors of innovation adoption. Administration and Policy in Mental Health and Mental
Health Services Research, 42(5), 545-573. https://doi.org/10.1007/s10488-014-0551-7
Cooper, B. R., Bumbarger, B. K., & Moore, J. E. (2015). Sustaining evidence-based prevention programs:
Correlates in a large-scale dissemination initiative. Prevention Science, 16(1), 145-157.
https://doi.org/10.1007/s11121-013-0427-1
Cragun, D., Pal T., Vadaparamspil S. T., Baldwin J., Hampel, H., & DeBate, R. D. (2016). Qualitative
comparative analysis: A hybrid method for identifying factors associated with program
effectiveness. Journal of Mixed Methods Research. 10(3), 251-272.
https://doi.org/10.1177/1558689815572023
Duncombe, D. C. (2018). A multi‐institutional study of the perceived barriers and facilitators to
implementing evidence‐based practice. Journal of Clinical Nursing, 27(5-6), 1216-1226.
https://doi.org/10.1111/jocn.14168
Dy, S. M., Garg, P., Nyberg, D., Dawson, P. B., Pronovost, P. J., Morlock, L., Rubin, H., & Wu, A. W. (2005).
Critical pathway effectiveness: Assessing the impact of patient, hospital care, and pathway
characteristics using qualitative comparative analysis. Health Services Research, 40(2), 499-516.
https://doi.org/10.1111/j.1475-6773.2005.0r370.x
Feinberg, M. E., Bontempo, D. E., & Greenberg, M. T. (2008). Predictors and level of sustainability of
community prevention coalitions. American Journal Of Preventive Medicine, 34(6), 495-501.
https://doi.org/10.1016/j.amepre.2008.01.030
40
Fereday, J., & Muir-Cochrane, E. (2006). Demonstrating rigor using thematic analysis: A hybrid approach
of inductive and deductive coding and theme development. International Journal
of Qualitative Methods, 5(1), 80-92. https://doi.org/10.1177/160940690600500107
Fiss, P. C. (2007). A set-theoretic approach to organizational configurations. Academy Of Management
Review, 32(4), 1180-1198. https://doi.org/10.5465/amr.2007.26586092
Fiss, P. C. (2011). Building better causal theories: A fuzzy set approach to typologies in organization
research. Academy Of Management Journal, 54(2), 393-420.
https://doi.org/10.5465/amj.2011.60263120
Forman, S. G., Olin, S. S., Hoagwood, K. E., Crowe, M., & Saka, N. (2009). Evidence-based interventions in
schools: Developers’ views of implementation barriers and facilitators. School Mental Health, 1,
Article 26. https://doi.org/10.1007/s12310-008-9002-5
Greckhamer, T., Misangyi, V. F., & Fiss, P. C. (2013). Chapter 3 The two QCAs: From a small-N to a large-
N set theoretic approach. Configurational Theory and Methods in Organizational Research, 38,
49-75. https://doi.org/10.1108/S0733-558X(2013)0000038007
Greckhamer, T., Fumari, S., Fiss, P. C., & Aguilera, R. V. (2018). Studying configurations with qualitative
comparative analysis: Best practices in strategy and organization research. Strategic
Organization, 16(4), 482–495. https://doi.org/10.1177/1476127018786487
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in
service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4),
581-629. https://doi.org/10.1111/j.0887-378X.2004.00325.x
Hailemariam, M., Bustos, T., Montgomery, B., Barajas, R., Evans, L. B., & Drahota, A. (2019). Evidence-
based intervention sustainability strategies: A systematic review. Implementation Science, 14,
Article 57. https://doi.org/10.1186/s13012-019-0910-6
41
Hamilton, A. B., & Finley, E. P. (2019). Qualitative methods in implementation research: An
introduction. Psychiatry Research, 280, Article 112516.
https://doi.org/10.1016/j.psychres.2019.112516
Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health
programs. Journal of Abnormal Child Psychology, 33(6), 665-679.
https://doi.org/10.1007/s10802-005-7646-2
Hoomans, T., & Severens, J. L. (2014). Economic evaluation of implementation strategies in health care.
Implementation Science, 9, Article 168. https://doi.org/10.1186/s13012-014-0168-y
Hunter, S. B., Han, B., Slaughter, M. E., Godley, S. H., & Garner, B. R. (2015). Associations between
implementation characteristics and evidence-based practice sustainment: A study of the
Adolescent Community Reinforcement Approach. Implementation Science, 10, Article 173.
https://doi.org/10.1186/s13012-015-0364-4
Hunter, S. B., Han, B., Slaughter, M. E., Godley, S. H., & Garner, B. R. (2017). Predicting evidence-based
treatment sustainment: Results from a longitudinal study of the Adolescent-Community
Reinforcement Approach. Implementation Science, 12, Article 75.
https://doi.org/10.1186/s13012-017-0606-8
Kane, H., Lewis, M. A., Williams, P. A., & Kahwati, L. C. (2014). Using qualitative comparative analysis to
understand and quantify translation and implementation. Translational Behavioral
Medicine, 4(2), 201-208. https://doi.org/10.1007/s13142-014-0251-6
Kane, H., Hinnant, L., Day, K., Council, M., Tzeng, J., Soler, R, Chambard, M., & Roussel A. & Heirendt, W.
(2017). Pathways to program success: A qualitative comparative analysis (QCA) of communities
putting prevention to work case study programs. Journal of Public Health Management and
Practice: JPHMP, 23(2), 104-111. https://doi.org/10.1097/PHH.0000000000000449
42
Langley, A. K., Nadeem, E., Kataoka, S. H., Stein, B. D., & Jaycox, L. H. (2010). Evidence-based mental
health programs in schools: Barriers and facilitators of successful implementation. School
Mental Health, 2(3), 105-113. https://doi.org/10.1007/s12310-010-9038-1
Lennox, L., Maher, L., & Reed, J. (2018). Navigating the sustainability landscape: A systematic review of
sustainability approaches in healthcare. Implementation Science, 13, Article 27.
https://doi.org/10.1186/s13012-017-0707-4
Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., Walsh-Bailey, C., & Weiner, B.
(2018). From classification to causality: Advancing understanding of mechanisms of change in
implementation science. Frontiers in Public Health, 6, Article 136.
https://doi.org/10.3389/fpubh.2018.00136
Liu, Y., Mezei, J., Kostakos, V., & Li, H. (2017). Applying configurational analysis to IS behavioural
research: A methodological alternative for modelling combinatorial complexities. Information
Systems Journal, 27(1), 59-89. https://doi.org/10.1111/isj.12094
Mendel, J. M., & Korjani, M. M. (2012). Charles Ragin’s fuzzy set qualitative comparative analysis (fsQCA)
used for linguistic summarizations. Information Sciences, 202, 1-23.
https://doi.org/10.1016/j.ins.2012.02.039
Mor Barak, M. E., Nissly, J. A., & Levin, A. (2001) Antecedents to retention and turnover among child
welfare, social work, and other human service employees: What can we learn from past
research? A review and metanalysis. Social Service Review, 75, 625–661.
https://www.jstor.org/stable/10.1086/323166
Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation
Science, 10, Article 53. https://doi.org/10.1186/s13012-015-0242-0
43
Palinkas, L. A., Spear, S. E., Mendon, S. J., Villamar, J., Valente, T., Chou, C. P., Landsverk, J., Kellam, S. G.,
& Brown, C. H. (2015). Measuring sustainment of prevention programs and initiatives: A study
protocol. Implementation Science, 11, Article 95. https://doi.org/10.1186/s13012-016-0467-6
Palinkas, L. A., Mendon, S. J., & Hamilton, A. B. (2019). Innovations in mixed methods
evaluations. Annual Review of Public Health, 40, 423-442. https://doi.org/10.1146/annurev-
publhealth-040218-044215
Palinkas, L. A., Spear, S. E., Mendon, S. J., Villamar, J., Reynolds, C., Green, C. D., Olson, C., Adade, A., &
Brown, C. H. (2020). Conceptualizing and measuring sustainability of prevention programs,
policies, and practices. Translational Behavioral Medicine, 10(1), 136-145.
https://doi.org/10.1093/tbm/ibz170
Paykani, T., Rafiey, H., & Sajjadi, H. (2018). A fuzzy set qualitative comparative analysis of 131 countries:
Which configuration of the structural conditions can explain health better?. International
Journal for Equity in Health, 17, Article 10. https://doi.org/10.1186/s12939-018-0724-1
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K.,
& Kirchner, J. E. (2015). A refined compilation of implementation strategies: Results from the
Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10,
Article 21. https://doi.org/10.1186/s13012-015-0209-1
Powell, B. J., Beidas, R. S., Lewis, C. C., Aarons, G. A., McMillen, J. C., Proctor, E. K., & Mandell, D. S.
(2017). Methods to improve the selection and tailoring of implementation strategies. The
Journal of Behavioral Health Services & Research, 44(2), 177-194.
https://doi.org/10.1007/s11414-015-9475-6
44
Powell, B. J., Fernandez, M. E., Williams, N. J., Aarons, G. A., Beidas, R. S., Lewis, C. C., McHugh, S. M., &
Weiner, B. J. (2019). Enhancing the impact of implementation strategies in healthcare: a
research agenda. Frontiers in Public Health, 7, Article 3.
https://doi.org/10.3389/fpubh.2019.00003
Ragin, C. C. (2008). User's Guide to Fuzzy-Set/Qualitative Comparative Analysis 3.0. Irvine, California:
Department of Sociology, University of California.
Ragin, C. C. (2014). The comparative method: Moving beyond qualitative and quantitative strategies.
University of California Press. (Originally published in 1987.)
Ragin, C. C., & Rihoux, B. (Eds.). (2009). Configurational comparative methods: Qualitative comparative
analysis (QCA) and related techniques. Sage.
Roundfield, K. D., & Lang, J. M. (2017). Costs to community mental health agencies to sustain an
evidence-based practice. Psychiatric Services, 68(9), 876-882.
https://doi.org/10.1176/appi.ps.201600193
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018). The sustainability of evidence-based interventions
and practices in public health and health care. Annual Review of Public Health, 39, 55-76.
https://doi.org/10.1146/annurev-publhealth-040617-014731
Scheirer, M. A. (2005). Is sustainability possible? A review and commentary on empirical studies of
program sustainability. American Journal of Evaluation, 26(3), 320-347.
https://doi.org/10.1177/1098214005278752
Scheirer, M. A., & Deering, J. W. (2011). An agenda for research on the sustainability of public health
programs. American Journal of Public Health, 101, 2059-2067.
Schneider, C. Q., & Wagemann, C. (2010). Standards of good practice in qualitative comparative analysis
(QCA) and fuzzy-sets. Comparative Sociology, 9(3), 397-418.
https://doi.org/10.1163/156913210X12493538729793
45
Schneider, C. Q., & Wagemann, C. (2012). Set-theoretic methods for the social sciences: A guide to
qualitative comparative analysis. Cambridge University Press.
Smith, J. R., & Donze, A. (2010). Assessing environmental readiness: First steps in developing an
evidence-based practice implementation culture. The Journal of Perinatal and Neonatal
Nursing, 24(1), 61-71. https://doi.org/10.1097/JPN.0b013e3181ce1357
Stevens, B., & Peikes, D. (2006). When the funding stops: Do grantees of the Local Initiative Funding
Partners Program sustain themselves?. Evaluation and Program Planning, 29(2), 153-161.
https://doi.org/10.1016/j.evalprogplan.2006.01.009
Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of
new programs and innovations: A review of the empirical literature and recommendations for
future research. Implementation Science, 7, Article 17. https://doi.org/10.1186/1748-5908-7-17
Tibbits, M. K., Bumbarger, B. K., Kyler, S. J., & Perkins, D. F. (2010). Sustaining evidence-based
interventions under real-world conditions: Results from a large-scale diffusion
project. Prevention Science, 11(3), 252-262. https://doi.org/10.1007/s11121-010-0170-9
Thygeson, N. M., Solberg, L. I., Asche, S. E., Fontaine, P., Pawlson, L. G., & Scholle, S. H. (2012). Using
fuzzy set qualitative comparative analysis (fs/QCA) to explore the relationship between medical
“homeness” and quality. Health Services Research, 47(1 Pt 1), 22-45.
https://doi.org/10.1111/j.1475-6773.2011.01303.x
Vale, L., Thomas, R., MacLennan, G., & Grimshaw, J. (2007). Systematic review of economic evaluations
and cost analyses of guideline implementation strategies. The European Journal of Health
Economics, 8(2), 111-121. https://doi.org/10.1007/s10198-007-0043-8
Vis, B. (2012). The comparative advantages of fsQCA and regression analysis for moderately large-N
analyses. Sociological Methods and Research, 41(1), 168-198.
https://doi.org/10.1177/0049124112442142
46
Waltz, T. J., Powell, B. J., Fernández, M. E., Abadie, B., & Damschroder, L. J. (2019). Choosing
implementation strategies to address contextual barriers: diversity in recommendations and
future directions. Implementation Science, 14, Article 42. https://doi.org/10.1186/s13012-019-
0892-4
Yakovchenko, V., Miech, E. J., Chinman, M. J., Chartier, M., Gonzalez, R., Kirchner, J. E., Morgan, T. R.,
Park, A., Powell, B. J., Proctor, E. K., Ross, D., Waltz, T. J., & Rogal, S. S. (2020). Strategy
configurations directly linked to higher Hepatitis C virus treatment starts: An applied use of
configurational comparative methods. Medical Care, 58(5), 31-38.
https://doi.org/10.1097/MLR.0000000000001319
47
Table 2.1
Inter-item Reliability by Condition Mean SD
Financial stability α = 0.808
Project is funded through non-profit, private, and/or non-governmental sources. 1.97 1.335
Project has a combination of stable (i.e., earmarked) and flexible (i.e., discretionary)
funding.
2.43 1.343
Project has sustained funding. 2.7 1.375
Diverse community organizations are financially invested in the success of the project. 2.77 1.428
Project is financially solvent. 3.17 1.434
Community responsiveness α = 0.696
Project meets the needs of the intended target populations. 4.57 0.682
Project addresses the behavioral health needs of the communities/populations being served.
4.39 0.789
Project can be adapted to meet the needs of the communities or populations being served.
4.62 0.601
Project is consistent with the norms, values and guiding principles of participating
organizations.
4.82 0.422
Project fits well with the values of the organization(s) responsible for sustaining it and the
communities where it is being sustained.
4.74 0.525
Coalitions, partnerships and networks α = 0.923
The grantee organization is networked with other organizations committed to sustaining the
project activities.
4.32 0.914
Community members are passionately committed to sustaining the project. 4.1 0.909
Community is actively engaged in the development of project goals. 3.86 1.031
Community has access to knowledge and information about the project. 4.22 0.91
Project is supported by a coalition/partnership/network of community organizations. 4.33 0.994
Coalition/partnership/network members actively seek to expand the network of community
organizations, leaders, and sources of support for this project.
4.05 1.078
Coalition/partnership/network is committed to the continued operation of this project.
4.18 1.054
High level of networking and communication within the organizations responsible for
sustaining the project.
4.1 1.045
Community leaders are actively involved in the project. 3.97 0.972
Organizational capacity α = 0.871
Project exhibits sound fiscal management. 4.52 0.855
The project activities are well integrated into the operations of the organization and its
partners.
4.24 0.865
Plans for implementing and sustaining the project are developed in advance. 3.96 0.935
Project is carried out or accomplished according to those plans. 4.2 0.906
The project has adequate staff to sustain the grant's goals and activities. 3.83 1.094
Project offers sufficient training to agency staff and community members. 4.35 0.919
Staff possesses adequate knowledge and supportive beliefs about the project. 4.71 0.702
Staff feel themselves to be capable of implementing the project. 4.65 0.69
Process α = 0.798
The project has a formally appointed person responsible for coordinating the process of
implementing and sustaining the project.
4.5 0.956
The project is also supported by a champion who is actively engaged in the process of
implementing and sustaining the project.
4.1 0.964
We have a process in place to sustain the project in the event our champion leaves. 3.57 1.144
Ongoing evaluation of progress made towards sustainment. 4 1.003
Sufficient and timely feedback about the project delivery to maintain or improve quality.
4.14 0.961
Evidence of positive outcomes. 4.18 0.77
Global sustainment α = 0.807
Project continues to operate as described in the original application for funding. 4.35 0.801
Project continues to deliver prevention services to its intended population. 4.69 0.689
Project continues to deliver prevention services that are evidence-based. 4.59 0.8
Project periodically measures the fidelity of the prevention services that are delivered. 4.18 1.09
Note. Bolded items indicate items used for additional set-membership assignment in cases of ambiguity.
48
Figure 2.1
Respondents by SAMSHA program
49
Table 2.2
Operationalization and Calibration
Condition
Membership set
points
Indirect calibration method: Decision rules
1-full
membership
*Indicated by a respondent perceiving that the project continued to be operated as originally described, continued providing evidence-based
and preventive services, and measured fidelity of services to a great extent.
.7- more in than
out
Most ratings for all four items are rated to a "moderate" or "great extent"; .7 was still considered if one item (such as delivering evidence-based services) was low.
.4- highly
ambiguous
A mix of ratings where items such as delivery of evidence-based services might be high, but all other items including delivering prevention services, monitoring
fidelity, are low. Qualitative items do not support ratings in one direction or another.
.2- more out
than in
Most ratings for all four items are rated as "little extent"; one or two items may be rated as "some extent."
0- non-
membership
Indicated by ratings of "not at all" for all items within condition
1-full
membership
*Indicated by a respondent perceiving the project to have sustained funding to a great extent (including securing a diverse source of funding)
.7- more in than
out
Most ratings for all items, especially sustained funding and financial solvency are rated as "moderate" or "great extent."
.4- highly
ambiguous
A mix of ratings where items such as diversity of funding, financial stability, and sustained funding are rated both as "little" or "some extent" and "moderate" extent.
Qualitative items do not support ratings in one direction or another.
.2- more out
than in
Most ratings for all items, especially sustained funding and financial solvency" are rated either "little" or "some extent"; some items may be rated "not at all."
0- non-
membership
Indicated by ratings of "not at all" for all items within condition
1-full
membership
Indicated by a respondent perceiving the project as responding to the needs and values of the community, including adapting to meet the needs
of the community, to a great extent
.7- more in than
out
Most ratings for all items, especially adapting to the needs of the community, are rated as "moderate" or "great extent."
.4- highly
ambiguous
A mix of ratings where items such as meeting the needs/behavioral needs of the intended population and the value fit between the project and the organization are
rated as "little extent" to "moderate extent." Qualitative items do not support ratings in one direction or another.
.2- more out
than in
Most ratings for items, including adapting to the needs of the community, are rated as "little" or "some extent."
0- non-
membership
Indicated by ratings of "not at all" for all items within condition
1-full
membership
*Indicated by the respondent perceiving a high level of coalition functioning that was committed to and supportive of sustaining the project
.7- more in than
out
Most ratings for all items, especially support from coalitions & partnerships and a commitment from coalitions & partnerships as it pertains to the continuation of
project services, are rated as "moderate" or "great extent."
.4- highly
ambiguous
A mix of ratings where items such as being networked with organizations and community engagement are moderate to high, but other key items such as suport
and commitment to sustainment from coalitions and partnerships is low. Qualitative items do not support ratings in one direction or another.
.2- more out
than in
Most ratings for items, including support and commitment from coalitions & partnerships, are rated as "little" or "some extent."
0- non-
membership
Indicated by ratings of "not at all" for all items within condition
1-full
membership
*Indicated by a respondent ’s perception that the organization ’s infrastructure had organizational capacity and staff capability (well trained,
knowledgeable staff) to a great extent.
.7- more in than
out
Most items, including key items RE plans for sustainment, adequate staff, and training are rated either as "moderate" or "great extent."
.4- highly
ambiguous
A mix of ratings on all items, especially three key items indicating ambiguity. Qualitative items do not support ratings in one direction or another.
.2- more out
than in
Most items, especially key items RE plans for sustainment, adequate staff, and training are rated as "little extent" or "some extent."
0- non-
membership
Indicated by ratings of "not at all" for all items within condition
1-full
membership
*Indicated by a respondent ’s perception that implementation leadership, program champions, monitoring and feedback, and positive
outcomes were a part of the program ’s existence to a great extent.
.7- more in than
out
Most items are rated as "moderate" or "great extent" including key items: presence of a program champion and evidence of positive outcomes.
.4- highly
ambiguous
Qualitative items do not support ratings in one direction or another.
.2- more out
than in
Most items are rated low as "little extent" or "some extent," especially key items: presence of a program champion and evidence of positive outcomes.
0- non-
membership
Indicated by ratings of "not at all" for all items within condition
Note. Bolded conditions indicate superset conditions. *A full list of items belonging to each condition can be found in Table 1.
Process
Sustainment
Financial stability
Community responsiveness
Coalitions, partnerships, &
networks
Organizational capacity
50
Table 2.3
Case Examples for Indirect Calibration Method
Condition
Membership set
points
Case Examples
1-full
membership
Case ID C013: ratings for all items were "great extent"
.7- more in than
out
Case ID A004: ratings for all items were "great extent" with the exception of monitoring fidelity which was rated as "little extent" ---> .7
.4- highly
ambiguous
Case ID A011: ratings for delivering prevention services, delivering EBP, and fidelity were "moderate extent", "some extent", and "not at all", respectively – giving this case a .7
would indicate more in than out, but a key component of high sustainment, periodically measuring fidelity, is not being practiced ---> .4
.2- more out
than in
Case ID D075: ratings for all items were "little extent" and monitoring fidelity was "not at all" --> .2; Case ID F5: all items were rated "little extent" and fidelity was rated "not at
all" -- qualitative statement "I know little about the ongoing activities, just that at least one environmental component was sustained" ---> .2
0- non-
membership
No cases were identified as having a full non-membership.
1-full
membership
Case ID C030: all items rated "great extent."
.7- more in than
out
Case ID A022: two items were rated as "little" and "some extent," however, sustained financial stability and financial solvency were rated very highly ---> .7; Case ID A086: key
items for sustained funding and financial solvency were rated high, while other items were rated "little" or "some extent." A membership set of .4 was considered, however
responses to additional survey questions were positive, indicating satisfactory initiatives to plan for additional funding ---> .7
.4- highly
ambiguous
Case ID A108: most items were rated as "some extent" and "little extent"; however items to additional survey questions indicated poor planning for additional funding ---> .4.
For maximum ambiguity items, other survey items (not a part of this condition) were reviewed: "Have you had conversations with your SAMHSA project officers regarding plans
to sustain this project once SAMHSA funding ends?" and "Do you plan to pursue funding for this project after funding by the SAMHSA [Name] Program ends?"
.2- more out
than in
Case ID A011: Two items were rated "not at all" and three items including stable funding, sustained funding, and financial solvency were rated as "some extent". A membership
set of .4 was considered, however responses to additional questions reflected poor planning for additional funding ---> .2; Case ID C012: Most items were rated as "little
extent" with the exception of financial solvency rated as "moderate extent"; qualitative data informed membership set rating: "STOP Act project activities are incorporated into
our coalition goals and objectives; our primary funding comes through the County from SAPT block grant funds; if those funds disappeared it seems unlikely we would have
much capacity to continue focusing on community level policy changes to reduce substance use." ----> .2
0- non-
membership
Case ID F7: all items were rated "not at all" with the exception of finacial solvency which was rated as "great extent." A membership set of .2 was considered, however responses
to both additional survey items were rated poorly, indicating no plans to secure additional funding.
1-full
membership
Case ID C044: All items were rated as "great extent" except responding to behavioral health needs, which was rated as "some extent." Full membership in set was still given
based on high ratings on all other items including key item "adapting to community needs."
.7- more in than
out
Case ID C049: All items were rated as "great extent" except responding to behavioral health needs and meeting the needs of the intended target population, both of which were
rated as "moderate extent."
.4- highly
ambiguous
Case ID C108: all items were rated either "moderate" or "great extent," however, the key item RE adapting to community needs was rated as "some extent." Case ID F5 and F8:
Though adapting to the community, and value fit with the organization were rated highly; meeting community needs of the intended population and meeting behavioral health
needs were rated low "little extent" ---> .4
.2- more out
than in
Case ID F1: all items were rated as "some extent."
0- non-
membership
No cases were identified as having a full non-membership.
1-full
membership
Case ID D002: two of nine items were rated low, however all other items including key items RE coalition support and commitment to sustainment were rated very highly. Case
ID A024: one item RE community engagement was rated as "moderate extent," however, all other items including key items were rated very highly.
.7- more in than
out
Case ID C043: Mix of item ratings from "great extent" to "some extent," key items RE support from coalitions & partnerships and coalitions' commitment to the sustainment of
the project were rated highly.
.4- highly
ambiguous
Case ID B02: one item RE how well the organization was networked with other organizations was very highly rated, however all other items were rated either "not at all" or to "a
little extent." Key items were rated "not at all."
.2- more out
than in
Case ID C049: Two of nine items were rated highly, however all other items were rated as "little" or "some extent"; ratings on key items were low.
0- non-
membership
No cases were identified as having a full non-membership.
1-full
membership
Case ID A081: All items were rated as "great extent." Case ID C008 and C11: one item RE fiscal management was rated as "moderate extent," however all other items, including
three key items, were rated as "great extent."
.7- more in than
out
Case ID A005: All items are rated as "great extent" with the exception of key items RE making plans for sustainment and training are rated "moderate extent."
.4- highly
ambiguous
Case ID D036: One item is rated very low and three items RE staff capability are rated very high. Key items RE making plans for sustainment and training staff are rated low and
high, respectively.
.2- more out
than in
Case ID D008: One of eight items is scored as "moderate extent," however all other items are rated as "not at all," "little extent," or "some extent." Key items are rated low.
0- non-
membership
No cases were identified as having a full non-membership.
1-full
membership
Case ID A042: one item RE plans to replace a program champion in case one leaves was rated as "some extent," however, all other items including key items RE the presence of a
program champion and evidence of positive outcomes were rated very high.
.7- more in than
out
Case ID A071: a mix of ratings between "some extent" and "great extent," however key items RE having a champion in place and evidence of positive outcomes were rated high.
.4- highly
ambiguous
Case ID 030: a mix of low and high ratings on all items. Two key items, having a program champion in place and evidence of positive outcomes were rated as "not at all" and
"moderate extent," respectively.
.2- more out
than in
Case ID D080: One item RE having a formally appointed leader in place was rated very high, however, all other items were rated as "not at all", "a little" or "some extent,"
including key items. Case ID B02: all items including key items were rated as "not at all," one exception was "some extent" to evidence of positive outcomes.
0- non-
membership
No cases were identified as having a full non-membership.
Note. Bolded conditions indicate superset conditions. *A full list of items belonging to each condition can be found in Table 1.
Process
Sustainment
Financial stability
Community responsiveness
Coalitions, partnerships, &
networks
Organizational capacity
51
Table 2.4
Truth Table Analysis
Note. Table shows all configurations that met thresholds for case frequency, raw consistency, and PRI consistency.
Financial stability
Community
Responsivness
Coalitions,
partnerships, and
networks
Organizational
capacity
Process Frequency Sustainment Raw consistency PRI consistency
0 1 1 1 1 67 1 0.971 0.96
1 1 1 1 1 47 1 0.99 0.99
0 1 0 1 1 7 1 0.967 0.931
0 1 1 1 0 4 1 0.989 0.977
0 1 0 0 0 3 1 0.941 0.873
0 1 0 1 0 3 1 0.953 0.899
1 1 1 1 0 3 1 0.996 0.992
0 1 1 0 1 3 1 0.987 0.974
0 0 0 0 0 2 1 0.897 0.724
0 0 1 1 1 2 1 0.953 0.885
0 1 1 0 0 1 1 0.963 0.923
0 0 1 1 0 1 1 0.972 0.925
0 1 0 0 1 1 1 0.97 0.93
1 1 0 1 1 1 1 0.995 0.99
52
Figure 2.2
Pathways to Sustainment
Note. Indicates the presence of a condition within a configuration; represents the
absence of a condition within a configurational pathway.
Note. Algorithm: Quine-McCluskey; Model: Sustain = f(financial stability, community
responsiveness, partnerships, capacity, process)
Note. Figure 2.2 is a Fiss configuration chart (Fiss, 2011)
m
Configurations
1 2 3 4
Financial Stability m m
Community Responsiveness
l l l
Organizational Capacity
l l l
Process
l
Partnerships
l l
Consistency 0.912 0.966 0.969 0.962
Raw Coverage 0.646 0.575 0.787 0.788
Unique Coverage 0.037 0.004 0.015 0.005
Overall Solution Consistency 0.913
Overall Solution Coverage 0.872
l
53
Table 2.5
Necessity Analysis
Note. Necessity analysis showing the presence of each condition on the
presence of sustainment
Consistency Coverage
Financial stability 0.454 0.98
Community responsiveness 0.936 0.885
Coalitions, partnerships, and networks 0.846 0.936
Organizational capacity 0.849 0.926
Process 0.818 0.948
54
Table 2.6
Pathways with a Negated Outcome
Note. Logical operators: * represents AND; ~ indicates the absence of an outcome/condition.
Note. Algorithm: Quine-McCluskey. Model: ~ Sustain = f(financial stability, community
responsiveness, partnerships, capacity, process)
Pathway
raw
coverage
consistency
~Financial stability * Community responsiveness 0.89 0.303
~ Financial stability * Partnershpips * Capacity 0.773 0.313
Community responsiveness * Partnerships * Capacity 0.791 0.235
Community responsiveness * Partnerships * Process 0.603 0.607
Solution coverage: .965
Solution consistency: .244
55
Chapter Three (Study Two)
The process of sustaining evidence-based practice:
Establishing the role of program integration
56
Abstract
Background: Refining strategies to improve implementation requires a better understanding of the
mechanisms involved. While previous studies have highlighted the importance of program integration of
an evidence-based practice within an organization, the role that integration plays on the path to
sustainment is currently unknown. Methods: The current study is based on a larger parent-study designed
to understand how programs are sustained once initial funding has been expended. The current analysis
was informed by survey data from one hundred and forty-seven respondents representing 121 grantees
from seven programs initially funded by SAMHSA prevention grant initiatives. Linear regression models
examined the relationship between eight constructs, including 1) financial stability, 2) responsiveness to
community values, 3) responsiveness to community needs, 4) coalitions, partnerships, and networks, 5)
organizational capacity, 6) organizational staff capability, 7) implementation leadership, and 8) monitoring
and evaluation, and the two main outcomes, integration and global sustainment. Additional analyses were
conducted to further assess how integration may mediate the associations between each of the
constructs and global sustainment. Results: The extent to which programs were integrated and sustained
were moderately high. Responsiveness to community values; coalitions, partnerships, and networks;
organizational capacity; and implementation were found to be significantly associated with how well the
program was integrated within the operations of the host organization. Of the 8 constructs, (1)
responsiveness to community values and (2) coalitions, networks, and partnerships were significantly
associated with integration as well as sustainment. In relation to sustainment, both constructs were
significantly explained by integration, over one-third and about one-fourth, respectively. While
organizational capacity and implementation leadership were both associated with integration, they were
not found to be significant predictors of sustainment. The relationship between each construct and global
sustainment was significantly explained by integration. Two constructs, financial stability and
organizational capacity, were considerably mediated by integration, 80% and 53%, respectively.
57
Conclusion: This study was the first to identify the role that integration plays in the path to sustainment.
Establishing that integration is a mechanism for sustainment and distinguishing its determinants from
those that facilitate sustainment builds a foundation to help refine the selection and use of
implementation strategies designed to help agencies improve the likelihood of sustaining EBP over time.
58
Introduction
In recent years, implementation science has moved beyond defining implementation outcomes
and identifying factors that may facilitate or impede those outcomes, and has reached a point of further
investigating the processes and mechanisms that allow an intervention or evidence-based practice (EBP)
to move throughout phases of implementation (Lewis et al., 2018; Powell, et al., 2014). Growing interest
and investment in understanding how EBPs are sustained once initial funding is no longer available has
led to numerous studies seeking to define sustainment and isolating its determinants (Cooper et al., 2015;
Hunter et al., 2017; Shelton et al., 2018). However, how a new program becomes a part of its host
organization has been understudied in implementation research. This concept, termed here as
integration, has not been well defined in the literature. While integration has often been described as
analogous with sustainment (Glasgow et al., 1999; Proctor et al., 2011), others have considered it to be a
discrete outcome (Scheirer & Deering, 2011). In either case, integration has not been the focus of most
studies interested in long-term maintenance of programs, and we have not yet learned what elements
within organizations make integration possible. Assessing the attributes of integration and discovering
the role that it plays in the path to sustainment, can help us refine implementation strategies and improve
the likeliness of sustaining EBP over-time.
In order to understand the mechanisms of sustainment, we need to first determine whether
program integration is distinguishable from sustainment. However, this aim is complicated by the
persistence of a complex nomenclature and an incomplete conceptualization of integration. Historically,
a variety of terms have been used to define and describe how a novel program becomes a part of the
organization as well as how a program is sustained over time—these terms include institutionalization,
maintenance, routinization, continuation, saturation, and integration (Aarons et al., 2014; Goodman,
1993; Scheirer, 2005). Conflating these terms have contributed to a distorted understanding of what
sustainment is. The concept of institutionalization and routinization, for instance, have been used to study
59
how programs are sustained (Pluye et al., 2005; Shediac-Rizkallah & Bone, 1998; Wallin et al., 2004). In
the highly cited taxonomy of implementation outcomes, Proctor and colleagues (2011) point to the
persistence of overlapping terms. The authors acknowledge their own limitations in using
indistinguishable language to describe two different implementation outcomes (i.e. sustainment, and in
this case, penetration which is described as akin to niche saturation). Second, though implementation
research has since evolved and come to understand sustainment as the continuation of services (Lennox
et al., 2018; Johnson et al., 2019), and more recently, has conceptualized sustainment using four indicators
(Palinkas et al., 2020; Palinkas et al., under review), the field has struggled to agree upon what it means
for a program to be integrated within community-based organizations (Wilson & Kurz, 2008). The
ambiguous use of definitions makes it challenging to study how this type of program embeddedness
organizationally drives sustainment of the program itself. Based on conceptualizations of
institutionalization and routinization further described below, integration of a program seems to rely
much more on aspects of what transpires within an organization, including what systems are in place to
treat the program as a permanent fixture within standard operating procedures, whereas sustainment
primarily refers to service continuity. In the absence of instruments to appropriately measure the
indicators of a program becoming a part of the organization, the current study adopts a pragmatic
approach that focuses on how well the program is integrated within the operations of the host
organization.
The Role of Integration in the Path to Sustainment
To further understand the process of sustainment, it is necessary to examine whether integration
is a driver of sustainment. While many have theorized this (Bowman et al., 2008; Scheirer & Dearing, 2011;
Wilson & Kurz, 2008), to fully understand what strategies should be enacted to enhance sustainment, we
must first consider that there may be an “event” that takes place pertaining to how the program and its
related activities have been integrated within the operations of the organization (Pluye, 2005; Wilson &
60
Kurz, 2008). Conceptually we might assume that long-term sustainment of a program is possible only once
a novel program is no longer considered as such: for instance, when the resources it requires are fully
funded and supported by the organization, and when procedures related to program activities (such as
referral processes, supervision, training, etc.) have all aligned with the values of the organization and
become a part its of standard operating procedures.
Though previously unrecognized as such, prior scholarship presents one vital piece of information
which provides impetus for the present study: the intersection where integration and sustainment meet.
Particularly, we must consider the substantial overlap of features identified as indicators of integration
and the factors that have now been established as determinants of sustainment. Take for instance the
defining factors of institutionalization and routinization. Institutionalization was set forth by Goodman
and colleagues (1993) as having four subsystems: (1) product (the presence of plans and quality
improvement procedures), (2) maintenance, (3) support (stable funding and partnerships with the local
social environment), and (4) managerial (leadership that interacts with all other subsystems).
Subsequently, routinization was believed to be characterized by three elements: 1) adequate resources
(financial, human, and material), 2) adapting to the local context, and 3) ensuring that program activities
align with the values and beliefs of the organization (Pluye et al., 2004a). These indicators used to measure
aspects of program integration, previously studied as institutionalization and routinization, overlap with
some of the most prominent elements known to facilitate sustainment, which suggests a potential
temporal relationship between these indicators and facilitators. Such overlapping factors include
evaluation, monitoring and feedback, planning for sustainment and executing those plans, sustained
funding and financial stability, collaborative partnerships and high coalition functioning, adapting to meet
the needs of the community, adequate staff, value fit between the program and the organization, and the
presence of program champions and leaders specifically appointed to ensure the successful
implementation of the project (Cooper et al., 2015; Hunter et al., 2017; Palinkas et al., 2020; Shelton et
61
al., 2018). The overlap of features at the intersection of integration and sustainment further suggests that
the two are not only related, but that integration may be a phase preceding sustainment.
Elements Associated with Program-Integration
If it is the case that a program must be embedded or integrated within an organization in order
for the program to last over-time (Bowman et al., 2008), it is important to first identify what elements are
responsible for integration. However, due in part to the aforementioned ambiguity around the definition
and measurement of program integration, knowledge of what types of factors might facilitate or be
responsible for program integration remains inadequate. Four prominent studies help illustrate
facilitators of integration. Goodman and colleagues (1993) believed that institutionalization is enhanced
by certain characteristics of the organization, including existing values and skills, the norms and values of
the organization, and attributes of the type of change itself. Pluye and colleagues (2005) describe eight
specific events as associated with or that “encourage” whether or not program activities are routinized:
1) resource stabilization (including financial, human, and material resources); 2) risk-taking (e.g. trying
new activities at the organizational level); 3) promotions and incentives; 4) adaptation of activities to meet
the needs of the surrounding environment; 5) objectives fit [with the values of the organization]; 6)
transparent communication between individuals involved; 7) sharing cultural artifacts (e.g. rituals that
symbolize specific values and beliefs of the program and organization); and 8) integration of rules (i.e.
supervision of and planning for program-related activities are a part of the organization’s existing rules).
In their case study of five health centers implementing a heart health program, all but two specific events
(rules and cultural artifacts) were found to be present in program activities that had become routinized
(or a part of) the host organization. More recently, Moullin and colleagues (2018) using an adapted
sustainment model based on the EPIS framework, levels of routinization and levels of institutionalization
were considered as outcomes, and while these were not directly studied, implementation leaders’ roles
combined with facets of inter-organizational and intraorganizational dynamics were emphasized as
62
important to later stages of implementation. Lastly, one of the few published studies using the LoIn scale,
Zakumumpa and colleagues (2018) used a mixed-methods approach to examine the institutionalization
(including all four subsystems) of Anti-retroviral Therapy (ART) in nearly 200 health facilities across
Uganda. Higher institutionalization, indicating saturation in this case, was attributed to sustained
workforce trainings, retention of ART-trained personnel, and the development of intervention-specific
manuals. Well-established facilities that were considered to be “higher-tier” hospitals with the highest-
ranking institutionalization scores.
The Current Study
The possibility that program integration is the link between active implementation and
sustainment (Wilson & Kurz, 2008) underscores the need to further study how sustainment emerges; not
simply by identifying the elements and key factors associated with sustainment, but by understanding the
process that transpires within the organization. The current study had two specific aims. The first aim was
to identify what factors are important to integrating EBP within the host organization and its operations
once initial seed funding has been expended or terminated. Building on previous research (Moullin et al.,
2018; Pluye et al., 2005; Zakumumpa et al., 2018) and consistent with the EPIS model (Aarons et al., 2014),
I expected that community values, such as adapting to community needs, organizational staff capability,
including training, the presence of an appointed implementation leader, and the support of coalitions and
partnerships will be important factors facilitating program-integration. The second aim of this study was
to determine if integration and sustainment are linked, and subsequently addressed how they are linked.
Given the strong beliefs that integration drives sustainment (Bowman et al., 2008; Scheirer & Dearing,
2011; Wilson & Kurz, 2008), I anticipated a strong relationship between integration and sustainment and
additionally expected that many of the determinants of sustainment are, in fact, explained by how well
the program is integrated within the host organization.
63
Methods
The current cross-sectional study is based on a larger parent study funded by the National
Institutes of Drug Abuse (NIDA 1 R34 DA037516-01A1, PI: Palinkas). The parent study primarily sought to
conceptualize the meaning of sustainment (phase 1) and develop a tool to measure it (phases 2 and 3).
See Palinkas and colleagues (2020) and Palinkas and colleagues (2015) for additional details regarding
recruitment, methodology and the development of the Sustainment Measurement System Scale (SMSS).
Participants
The current study drew from the parent study comprising 186 representatives of 145 grantee sites
supported by 7 SAMHSA initiatives: 1) the Sober Truth on Preventing Underage Drinking Act (STOP-Act)
grants committed to preventing and reducing alcohol use among youth and young adults; 2) the
Implementing Evidence-based Prevention Practices in Schools (PPS) grants designed to address the
prevention of early childhood behavioral disorders; 3) the Strategic Prevention Framework State
Incentive (SPF-SIG) grants to prevent the onset of substance abuse amongst youth; 4) the Garrett Lee
Smith (GLS) State and Tribal Youth Suicide Prevention program to address depression, suicidal attempts,
and behavioral health problems linked to suicide; 5) the Substance Abuse and HIV Prevention Navigator
Program for Racial/Ethnic Minorities (Prevention Navigator, PN) serving minority populations including
those in tribal communities at-risk for substance abuse and HIV; 6) Minority Serving Institutions
Partnerships with Community-Based Organizations (MSI-CBOs) serving communities at risk for substance
abuse, HIV, and hepatitis-C infections; and 7) the HIV prevention services for at-risk racial/ethnic minority
youth and young adults (HIV-CBI). One hundred and forty-seven respondents representing 121 grantees
from seven programs initially funded by SAMHSA were included in the current analysis. Participants in
this study included individuals from programs that continued to exist at the time of data collection, either
with substantial adaptations or in much of the same form they were when funded by respective SAMHSA
grants.
64
Measures
The parent study surveyed respondents using a 52-item web-based survey. Respondents were
asked to complete basic demographic data (e.g. role in project, years of funding, plans to pursue additional
funding) in addition to items based on the CFIR (Damschroder et al., 2009) and considered to be elements
important for sustainment (see Palinkas et al., 2015, Palinkas et al., 2020 for details about the
development of all items). All key survey items were assessed on a 5-point Likert scale ranging from “not
at all” to “a great extent.”
Outcome Variables
There are two main outcome variables in the current study: global sustainment and integration.
Global sustainment was based on findings from the parent study (Palinkas et al., 2020, Palinkas et al.,
under review) which found four indicators of sustainment: 1) continue to operate as described in the
original application; 2) continue to deliver preventive services to intended population; 3) continue to
deliver evidence-based services; and 4) periodically measure fidelity of services delivered. A composite
score based on the means of those four items was created to measure global sustainment (Song, 2013).
Integration was measured using one item: [to what extent is] “the project well integrated into the
operations of the organization and its partners.”
Predictor Variables
To develop a scale measuring sustainment, data triangulation methods were used to develop
constructs based on CFIR ratings of importance, semi-structured interviews, and free list exercises
(Palinkas et al., 2020). Subsequently, a confirmatory factor analysis to validate the Sustainment
Measurement System Scale (SMSS; Palinkas et al., under review) identified a nine-factor, 35-item model.
The current study used all eight determinant factors as predictors: 1) financial stability (5 items); 2)
community needs (2 items); 3) community values (3 items); 4) coalitions, networks, and partnerships (8
items); 5) organizational capacity (3 items); 6) organizational staff capability (3 items); 7) implementation
65
leadership (3 items); and 8) evaluation (3 items). Composite scores based on the mean of items were
created to measure each predictor variable (Song et al., 2013). The item measuring project integration
was removed from the organizational capacity construct. As seen in Table 4.1, most constructs had
moderate scale reliability indicated by a Cronbach’s alpha of ≥ .7 (Nunnally, 1978) or ≥ .4, acceptable for
new scales (Nunally & Bernstein, 1994) or scales with less than 10 items (Pallant, 2013). A list of all items
by construct and corresponding descriptive data can be found on Table 3.1.
Analytic Approach
I calculated a correlation matrix to assess all relationships between predictor variables and two
outcome variables. Based on Pearson R measures, all eight predictor variables had positive and significant
relationships with both outcomes. All predictor variables showing moderate correlations were retained
for further assessment post-regression analysis. An examination of test assumptions indicated satisfactory
homoscedasticity, linearity, and normality. Two multivariate ordinary least squares regression models
were conducted to examine the main effects of all predictors on sustainment (model 1a) and on
integration (model 1b). Variance inflation factor analysis showed only minimal to moderate effects of
multicollinearity. Independent predictors showing significant relationships on respective outcomes were
retained for further analysis. Using a model reduction approach, two additional regression analyses were
conducted and set at 95% confidence intervals. Model 2a included examining the main effects of
responsiveness to community needs; responsiveness to community values; coalitions, networks, and
partnerships; and organizational staff capability on global sustainment. Model 2b included examining the
main effects of responsiveness to community values; coalitions, networks, & partnerships; organizational
capacity; and implementation leadership on integration. Variance inflation factor analysis for both models
2a and 2b confirmed that multicollinearity was minimal (< 2) indicating no need for further model re-
assessment. Lastly, independent mediation analyses were conducted for each predictor variable using the
66
Baron and Kenny Method (1986) to assess if integration was responsible for or explained a change in the
relationship between a given predictor and sustainment.
Results
Sample Characteristics
Program representation for the sample of 147 respondents representing 121 grantees includes
STOP-Act (37.4%), PPS (5.4%), GLS (27.2%), SPF-PFS (16.3%), CBI (4.8%), MSI (4.8%), and PNP (4.1%). The
roles of respondents within a given project included project principal investigator, project director, project
coordinator, evaluator, and other (see Figure 3.1). All projects included in the current sample either
continued to exist with substantial adaptations (34.7%) or continued to exist in much the same form as
they did when funded by the SAMHSA program (65.3%), see Figure 3.2. About half of the respondents
reported that the project does not have additional funding in addition to the SAMHSA program (46.9%),
and 32.7% (of 110 respondents) stated that they are actively pursuing other sources of funding for the
particular project.
Extent of Integration and Global Sustainment
The extent to which programs were well integrated into the operations of the organization and
its partners were generally high. While most SAMHSA projects reported at least a moderate extent of
integration within respective organizations, MSI struggled considerably, with five of the seven
respondents reporting that the project was integrated only a little or to some extent. Conversely, the
STOP-Act and PNP initiatives had the highest number of respondents (>90% and 100%, respectively)
stating that the project was well integrated into the organization. Most respondents also considered the
program to meet global sustainment to at least a moderate extent: continued to operate as described in
original application; continued to deliver preventive services to the intended population; continued to
deliver evidence-based services; and periodically measured fidelity of services delivered (collectively
termed global sustainment). Descriptive data on outcomes by program type can be found in Table 3.2.
67
Factors Associated with Integration
All eight factors were moderately and significantly correlated with integration. Organizational
capacity had the highest correlation (r s = .71, p < .001), followed by coalitions, networks & partnerships (r s
= .56, p < .001) and organizational staff capability (r s = .55, p < .001). These findings suggest that these
factors in particular have a strong relationship with and are positively related to integration. All other
factors were correlated with integration in the expected direction. Two multivariate linear regression
models were conducted for integration; model 1b with all eight factors, followed by a reduced and
simplified model with only significant factors (model 2b). All results for both models are shown in Table
3.3. The multiple regression model 2b demonstrated good model fit F(8, 136) = 26.3, p < .001. Adjusted
R
2
indicated that the model accounted for 58.4% of the variance in integration, greater than the variance
accounted for in model 1b. When keeping all other factors constant, four factors were significant
predictors of integration: responsiveness to community values ( = .29, SE = .12, p ≤ .05); coalitions,
networks, & partnerships ( = .23, SE = .07, p ≤ .01); organizational capacity ( = .56, SE = .07, p ≤ .001);
and implementation leadership ( = .15, SE = .07, p ≤ .05). Findings suggest that organizational capacity
was the strongest predictor, such that for every unit increase, the level of how well the project was
integrated within the operations of the organization and partners increased by more than half a point.
Financial stability, community responsiveness to needs, organizational staff capability, and evaluation
were not significant predictors of integration in this sample.
Factors Associated with Global Sustainment
All eight factors with the exception of financial stability were moderately correlated with global
sustainment. Four factors were significantly correlated with global sustainment: organizational staff
capability (r s = .59, p < .001); coalitions, networks, & partnerships (r s = .55, p < .001); responsiveness to
community needs (r s = .54, p < .001); and evaluation (r s = .52, p < .001). All other factors were correlated
with global sustainment, thought these were not significant, all relationships were in the expected
68
direction. Notably, integration was also moderately correlated with global sustainment (r s = .47, p < .001).
Two multivariate linear regression models were conducted for global sustainment; model 1a with all eight
factors, followed by a reduced and simplified model with only significant factors (model 2a). All results for
both models are shown in Table 3.4. Model 2a was retained given an increase in adjusted R
2
, indicating
that the model accounted for 51.4% of the variance in global sustainment and demonstrated good model
fit F(4, 142) = 39.6, p < .001. When keeping all other factors constant, four factors were significant
predictors of global sustainment: responsiveness to community needs (B = .28, SE = .07, p ≤ .001);
responsiveness to community values ( = .29, SE = .10, p ≤ .01); coalitions, networks, & partnerships ( =
.21, SE = .06, p ≤ .001); and organizational staff capability ( = .32, SE = .08, p ≤ .001). Of these,
organizational staff capability was the strongest predictor, such that for every unit increase, the extent to
which the project was sustained increased by about a third of a unit. Financial stability, organizational
capacity, implementation leadership, and evaluation were not significant predictors of sustainment in this
sample.
Mediation Analysis
Lastly, independent mediation analyses were conducted to assess the extent to which the effect
of each factor on the level of global sustainment was mediated by the extent to which the project was
integrated within the operations of the organization. Based on 10,000 bootstrap samples, integration was
a significant mediator for all eight predictors of sustainment. Figure 3.3 shows the path estimates of four
constructs that significantly predicted integration. Integration accounted most for the effects between
three particular constructs: financial stability (80%), organizational capacity (53%), and implementation
leadership (38%). Of the four determinants found to be positively associated with integration; two
constructs that were also significantly associated with sustainment, (1) community values and (2)
coalitions, partnerships, and networks were significantly explained by integration (by 34% and 24.6%,
69
respectively). In summary, these findings support the claim that being integrated within the operations of
an organization is, indeed, a part of the process of global sustainment.
Discussion
While many have highlighted the importance of program integration within the host organization
in the context of long-term sustainment (Bowman et al., 2008; Han & Weiss, 2005; Moullin et al., 2016;
Wilson & Kurz, 2008), to my knowledge this is the first study to identify the role that integration plays in
sustaining EBPs. Notably, findings align with what was expected regarding the role of integration and
brings to surface a previously unexplored concept in identifying the process of sustainment. In line with
existing literature (Aarons et al., 2014; Cooper et al., 2015; Lennox et al., 2018; Shelton et al., 2018;
Stevens & Peikes, 2006), the current study found responsiveness to community needs, responsiveness to
community values, coalitions & partnerships, and organizational staff capability to predict sustainment.
The relationship between each of these constructs and sustainment was explained to a large degree by
the extent to which the program was integrated within the operations of its host organization; indicating
that program integration is a mechanism of sustainment.
The relationship found between responsiveness to community values and integration is
consistent with previous literature underscoring adaptation and fit with the values of the organization
(Pluye et al., 2005), as well as sustainment (Aarons et al., 2014; Han & Weiss, 2005; Lennox et al., 2018).
Consistent with the EPIS model, this highlights the role of adaptation in later stages of implementation
compared with earlier stages of implementation that may be concerned with adherence to treatment and
fidelity (Aarons et al., 2011). While implementation leadership was not significantly associated with
sustainment, contradicting previous literature (Stevens & Peikes, 2006), it was found to be associated with
integration, which aligns with points made by Moullin and colleagues (2016) that highlight the role of
implementation leaders in maintaining partnerships and procuring contracts that are crucial to the later
stages of implementation.
70
Findings suggest that there may be a fifth stage of implementation to consider within the EPIS
framework. This is especially supported by the different factors found to be responsible for facilitating
integration compared to sustaining it over time, and considerable mediation between predictors of
sustainment. Two conceptual models, a sustainment model based on the EPIS framework that suggests
level of institutionalization and level of routinization as outcomes in later stages of implementation
(Aarons et al., 2014; Moullin et al., 2016) and an adapted model by Wilson and Kurz (2008) showing
institutionalization as preceding maintenance, further support implications from my findings. While Pluye
and colleagues (2004b) inherently believe the stage model to be problematic, they also suggest that
implementation and sustainment are concomitant processes whereby sustainment is influenced by
specific events that bridge implementation and sustainability. In the case of the current study, integration
would be such an event. Additionally, there is a considerable overlap between the indicators of integration
(Goodman et al., 1993; Pluye et al., 2004a) and determinants of sustainment, which also suggests that
integration may precede sustainment. Further longitudinal studies are needed to better understand these
two constructs alongside the passage of time.
While previous research provided a foundation of what elements encourage integration of
programs within their host organizations (Moullin et al., 2018; Pluye et al., 2005; Zakumumpa et al., 2018),
these examples are nonetheless limited by context. Specifically, we must consider three points: (1) the
concept of integration has largely only been theoretically construed—institutionalization and
routinization as described in these studies may not align with what actors in community organizations
perceive it to be (Pluye et al., 2004a); (2) to date, there is no widely agreed upon consensus about
integration that merges theoretical knowledge across disciplines, creating a challenge when attempting
to identify associated facilitators and barriers; and (3) studies that identified facilitators of
institutionalization and routinization (Pluye et al., 2005; Zakumumpa et al., 2018) took place either prior
to or without considering what the field has recently defined as sustainment—as such, there is incongruity
71
between previous efforts discussing the two concepts as concomitant or related and current scholarship.
Precisely because prior research was conducted within the historical scope of treating institutionalization
and routinization as analogous with sustainment, further investigation to identify factors that facilitate
program integration, particularly in the context of viewing it as discrete from sustainment, and empirical
evaluation of how the two concepts are related is needed.
Limitations and Future Studies
Some considerations about the methodological constraints of this study must be acknowledged.
First, this study used a simplified measure of integration. However, as seen in extant literature, the
concept of a new program evolving to one that becomes fully a part of the host organization and its
operations is complex (Goodman et al., 1993; Greenhalgh et al., 2005; Pluye et al., 2004). While much of
this research offers theoretical underpinnings of how to measure the institutionalization or routinization
of a new product with its host organization, the ambiguity in concepts and dated application do not
correspond with current knowledge of implementation outcomes and processes. Furthermore,
institutionalization and routinization and their association with integration or sustainment were not
specifically assessed. A narrow definition of program integration made it possible to establish the
important role that this plays in sustaining EBP. Second, while findings confirmed that integration plays a
role in sustaining EBP, the study was not informed by longitudinal data; hence, limiting our interpretations
of the findings. The passage of time is needed to fully understand aspects of niche saturation as well as
routinization (Goodman et al., 1993), and can be also used to understand the process that takes place
between implementation phases (Wilson & Kurz, 2008). Lastly, in relation to construct development, scale
reliability analysis detected a low Cronbach’s alpha (less than .5) for the community responsiveness to
needs construct. In a previously conducted confirmatory factor analysis, only two items were found to fall
within this construct. It has been suggested elsewhere (see Palinkas et al., under review), that further
72
assessment is needed to consider revising this construct, perhaps by combining it with community
responsiveness to values.
This study launches a relatively unexplored program of research. Theoretical and conceptual
knowledge from organization and management literature must be consulted in conjunction with current
implementation frameworks and models in order to consider the inclusion of integration as a stage
preceding sustainment. Future studies can then proceed by developing a conceptualization of what it
means for a program to be integrated within the organization from the perspectives of community
stakeholders within the agency setting (i.e. direct care providers, mid-level supervisors, and agency
leaders). Subsequently, this can inform the development of a tool to measure program integration that
aligns with preceding theoretical frameworks as well as views of community stakeholders. This measure
can then be used to study integration across multiple time points allowing for a closer examination of
elements that facilitate integration and further understand the process between active implementation
of EBP and program-integration.
Conclusion
This study contributes to the field of implementation in two important ways. First, findings imply
that program integration might be a separate phase that falls between active implementation and
sustainment within the EPIS framework. At a time when there are tremendous resources and efforts being
allocated to better understand how to sustain programs once initial funding has been expended, it is not
only important to identify the determinants of sustainment, but equally necessary to understand what
precedes it. Additionally, in recent years the field has moved away from identifying facilitators and barriers
to implementation by addressing how agencies can strategically improve implementation. Identifying the
elements that facilitate integration, and more so, establishing that integration is a mechanism of
sustainment, has the potential to further refine implementation strategies targeting the latter stages of
implementation. Since this study was based on a sample that includes various mental health and
73
substance use prevention initiatives being delivered across the nation in vastly different service settings,
it provided an opportunity to fully tend to the inter- and intra-organizational constructs that influence two
major implementation outcomes, without being attached to the specific components of any one type of
treatment intervention. It was also the first study to consider the role of integration in sustainment using
a comprehensive definition of sustainment that is based on the perspectives of community stakeholders.
74
References
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based
practice implementation in public service sectors. Administration and Policy in Mental Health
and Mental Health Services Research, 38(1), 4-23. https://doi.org/10.1007/s10488-010-0327-7
Aarons, G. A., Green, A. E., Willging, C. E., Ehrhart, M. G., Roesch, S. C., Hecht, D. B., & Chaffin, M. J.
(2014). Mixed-method study of a conceptual model of evidence-based intervention sustainment
across multiple public-sector service settings. Implementation Science, 9, Article 183.
https://doi.org/10.1186/s13012-014-0183-z
Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological
research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social
Psychology, 51(6), 1173-1182.
Bowman, C. C., Sobo, E. J., Asch, S. M., Gifford, A. L., & HIV/Hepatitis Quality Enhancement Research
Initiative. (2008). Measuring persistence of implementation: QUERI Series. Implementation
Science, 3, Article 21. https://doi.org/10.1186/1748-5908-3-21
Cooper, B. R., Bumbarger, B. K., & Moore, J. E. (2015). Sustaining evidence-based prevention programs:
Correlates in a large-scale dissemination initiative. Prevention Science, 16(1), 145-157.
https://doi.org/10.1007/s11121-013-0427-1
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health
promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9),
1322-1327. https://doi.org/10.2105/AJPH.89.9.1322
Goodman, R. M., McLeroy, K. R., Steckler, A. B., & Hoyle, R. H. (1993). Development of level of
institutionalization scales for health promotion programs. Health Education Quarterly, 20(2),
161-178. https://doi.org/10.1177/109019819302000208
75
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in
service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4),
581-629. https://doi.org/10.1111/j.0887-378X.2004.00325.x
Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health
programs. Journal of Abnormal Child Psychology, 33(6), 665-679.
https://doi.org/10.1007/s10802-005-7646-2
Hunter, S. B., Han, B., Slaughter, M. E., Godley, S. H., & Garner, B. R. (2017). Predicting evidence-based
treatment sustainment: Results from a longitudinal study of the Adolescent-Community
Reinforcement Approach. Implementation Science, 12, Article 75.
https://doi.org/10.1186/s13012-017-0606-8
Johnson, A.M., Moore, J.E., Chambers, D.A., Rup, J. Dinyarian, C., & Straus, S. E. (2019). How do
researchers conceptualize and plan for the sustainability of their NIH R01 implementation
projects?. Implementation Science, 14, Article 50. https://doi.org/10.1186/s13012-019-0895-1
Lennox, L., Maher, L., & Reed, J. (2018). Navigating the sustainability landscape: A systematic review of
sustainability approaches in healthcare. Implementation Science, 13, Article 27.
https://doi.org/10.1186/s13012-017-0707-4
Lewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G. (2015). Outcomes for
implementation science: An enhanced systematic review of instruments using evidence-based
rating criteria. Implementation Science, 10, Article 155. https://doi.org/10.1186/s13012-015-
0342-x
Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., Walsh-Bailey, C., & Weiner, B.
(2018). From classification to causality: Advancing understanding of mechanisms of change in
implementation science. Frontiers in Public Health, 6, Article 136.
https://doi.org/10.3389/fpubh.2018.00136
76
Moullin, J. C., Ehrhart, M. G., & Aarons, G. A. (2018). The role of leadership in organizational
implementation and sustainment in service agencies. Research on Social Work Practice, 28(5),
558-567. https://doi.org/10.1177/1049731517718361
Nunnally, J. C. (1978). Psychometric theory (2
nd
ed.). McGraw Hill.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3
rd
ed.). McGraw Hill.
Palinkas, L. A., Spear, S. E., Mendon, S. J., Villamar, J., Valente, T., Chou, C. P., Landsverk, J., Kellam, S. G.,
& Brown, C. H. (2015). Measuring sustainment of prevention programs and initiatives: A study
protocol. Implementation Science, 11, Article 95. https://doi.org/10.1186/s13012-016-0467-6
Palinkas, L. A., Spear, S. E., Mendon, S. J., Villamar, J., Reynolds, C., Green, C. D., Olson, C., Adade, A., &
Brown, C. H. (2020). Conceptualizing and measuring sustainability of prevention programs,
policies, and practices. Translational Behavioral Medicine, 10(1), 136-145.
https://doi.org/10.1093/tbm/ibz170
Palinkas, L.A., Chou, C.P., Spear, S.E., Mendon, S.J., Villamar, J., & Brown, C.H. Measurement of
sustainment of prevention programs and initiatives: The sustainment measurement system
scale. Implementation Science. (under review).
Pallant, J. (2013). SPSS survival manual. McGraw-Hill Education (UK).
Pluye, P., Potvin, L., Denis, J. L., & Pelletier, J. (2004a). Program sustainability: Focus on organizational
routines. Health Promotion International, 19(4), 489-500.
https://doi.org/10.1093/heapro/dah411
Pluye, P., Potvin, L., & Denis, J. L. (2004b). Making public health programs last: Conceptualizing
sustainability. Evaluation and Program Planning, 27(2), 121-133.
https://doi.org/10.1016/j.evalprogplan.2004.01.001
77
Pluye, P., Potvin, L., Denis, J. L., Pelletier, J., & Mannoni, C. (2005). Program sustainability begins with the
first events. Evaluation and Program Planning, 28(2), 123-137.
https://doi.org/10.1016/j.evalprogplan.2004.10.003
Powell, B. J., Proctor, E. K., & Glass, J.E. (2014). A systematic review of strategies for implementing
empirically supported mental health interventions. Research on Social Work Practice, 24(2),
192–212. https://doi.org/10.1177/1049731513505778
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M.
(2011). Outcomes for implementation research: Conceptual distinctions, measurement
challenges, and research agenda. Administration and Policy in Mental Health and Mental Health
Services Research, 38(2), 65-76. https://doi.org/10.1007/s10488-010-0319-7
Scheirer, M. A. (1993). Are the Level of Institutionalization Scales Ready for “Prime Time"? A
commentary on "Development of Level of Institutionalization (LoIn) Scales for Health Promotion
Programs". Health Education Quarterly, 20(2), 179-182.
Scheirer, M. A. (2005). Is sustainability possible? A review and commentary on empirical studies of
program sustainability. American Journal of Evaluation, 26(3), 320-347.
https://doi.org/10.1177/1098214005278752
Scheirer, M. A., & Deering, J. W. (2011). An agenda for research on the sustainability of public health
programs. American Journal of Public Health, 101, 2059-2067.
https://doi.org/10.2105/AJPH.2011.300193
Shediac-Rizkallah, M. C., & Bone, L. R. (1998). Planning for the sustainability of community-based health
programs: Conceptual frameworks and future directions for research, practice and
policy. Health Education Research, 13(1), 87-108. https://doi.org/10.1093/her/13.1.87
78
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018). The sustainability of evidence-based interventions
and practices in public health and health care. Annual Review of Public Health, 39, 55-76.
https://doi.org/10.1146/annurev-publhealth-040617-014731
Song, M. K., Lin, F. C., Ward, S. E., & Fine, J. P. (2013). Composite variables: When and how. Nursing
Research, 62(1), 45-49. https://doi.org/10.1097/NNR.0b013e3182741948
Stevens, B., & Peikes, D. (2006). When the funding stops: Do grantees of the Local Initiative Funding
Partners Program sustain themselves?. Evaluation and Program Planning, 29(2), 153-161.
https://doi.org/10.1016/j.evalprogplan.2006.01.009
Wallin, E., Lindewald, B., & Andréasson, S. (2004). Institutionalization of a community action program
targeting licensed premises in Stockholm, Sweden. Evaluation Review, 28(5), 396-419.
https://doi.org/10.1177/0193841X04264951
Wilson, K. D., & Kurz, R. S. (2008). Bridging implementation and institutionalization within organizations:
Proposed employment of continuous quality improvement to further dissemination. Journal of
Public Health Management and Practice, 14(2), 109-116.
https://doi.org/10.1097/01.PHH.0000311887.06252.5f
Yin, R. K. (1979). Changing urban bureaucracies: How new practices become routinized. Lexington Books.
Zakumumpa, H., Kwiringira, J., Rujumba, J., & Ssengooba, F. (2018). Assessing the level of
institutionalization of donor-funded anti-retroviral therapy (ART) programs in health facilities in
Uganda: Implications for program sustainability. Global Health Action, 11, Article 1523302.
https://doi.org/10.1080/16549716.2018.1523302
79
Table 3.1
Inter-item Reliability by Construct Mean SD
Financial stability α = 0.755
Project is funded through non-profit, private, and/or non-governmental sources. 2.18 1.33
Project has a combination of stable (i.e., earmarked) and flexible (i.e., discretionary)
funding.
4.76 0.72
Project has sustained funding. 2.85 1.46
Diverse community organizations are financially invested in the success of the project. 2.91 1.43
Project is financially solvent. 3.27 1.47
Responsiveness to community needs α = 0.441
Project meets the needs of the intended target populations. 4.64 0.62
Project addresses the behavioral health needs of the communities/populations being
served.
4.36 0.75
Responsiveness to community values α = 0.533
Project can be adapted to meet the needs of the communities or populations being served. 4.62 0.62
Project is consistent with the norms, values and guiding principles of participating
organizations.
4.82 0.39
Project fits well with the values of the organization(s) responsible for sustaining it and the
communities where it is being sustained.
4.84 0.37
Coalitions, partnerships and networks α = 0.929
Community members are passionately committed to sustaining the project. 4.1 0.91
Community is actively engaged in the development of project goals. 3.86 1.03
Community has access to knowledge and information about the project. 4.22 0.91
Project is supported by a coalition/partnership/network of community organizations. 4.33 0.99
Coalition/partnership/network members actively seek to expand the network of community
organizations, leaders, and sources of support for this project.
4.05 1.08
Coalition/partnership/network is committed to the continued operation of this project. 4.18 1.05
High level of networking and communication within the organizations responsible for
sustaining the project.
4.1 1.05
Community leaders are actively involved in the project. 3.97 0.97
Organizational capacity α = 0.649
Project exhibits sound fiscal management. 4.8 0.49
Plans for implementing and sustaining the project are developed in advance. 4.04 1.02
Project is carried out or accomplished according to those plans. 4.38 0.87
Organizational staff capability α = 0.698
Project offers sufficient training to agency staff and community members. 4.24 0.82
Staff possesses adequate knowledge and supportive beliefs about the project. 4.82 0.43
Staff feel themselves to be capable of implementing the project. 4.73 0.62
Implementation leadership α = 0.664
The project has a formally appointed person responsible for coordinating the process of
implementing and sustaining the project.
4.64 0.8
The project is also supported by a champion who is actively engaged in the process of
implementing and sustaining the project.
4.15 0.99
We have a process in place to sustain the project in the event our champion leaves. 3.6 1.15
Evaluation α = 0.522
Ongoing evaluation of progress made towards sustainment. 4.13 0.86
Sufficient and timely feedback about the project delivery to maintain or improve quality. 4.16 0.79
Evidence of positive outcomes. 4.24 0.72
Global sustainment α = 0.712
Project continues to operate as described in the original application for funding. 4.55 0.66
Project continues to deliver prevention services to its intended population. 4.82 0.43
Project continues to deliver prevention services that are evidence-based. 4.55 0.88
Project periodically measures the fidelity of the prevention services that are delivered. 4.18 1.02
80
Table 3.2
Outcome Means by SAMHSA Program
81
Figure 3.1
Respondent Roles
82
Figure 3.2
Program Existence
0
5
10
15
20
25
30
35
40
STOP-Act PPS GLS SPF-PFS CBI MSI PNP
The project continues to exist and it has been substantially
adapted
The project exists in much the same form as it did when
funded by this SAMHSA Program
83
Table 3.3
Regression Analyses on Program Integration
84
Table 3.4
Regression Analyses on Program Sustainment
85
Fig 3.3
Pathway to Sustainment
*** p ≤ .001; ** p ≤ .01; * p ≤ 05
Note. Path estimates with C’.
86
Chapter Four: Conclusion
“[We] have to remember that what we observe is not nature in itself, but nature exposed to our method
of questioning” (Heisenberg, 1958, p.58).
Immense efforts and resources are dedicated to the development of evidence-based practices
addressing some of the most harmful and debilitating substance use and mental health problems plaguing
our society. After having gone through extensive research trials, such programs designed to treat and
prevent the progression of mental health disorders often fail to be implemented in communities where
those in need can access care. Maintaining the programs that do get implemented in community settings
beyond initial years of seed funding can be even more challenging. Failure to sustain such health programs
threatens the possibility of these programs to make a substantial public health impact.
Several challenges have limited the ability to study sustainment, including the context in which
intervention components are studied (Shelton et al., 2018), conceptualization of sustainment (Palinkas et
al., 2015; Shelton et al., 2018), and vague nomenclature inconsistently using different terms as analogous
to sustainment (Goodman et al, 1993; Proctor et al., 2011). The parent study, funded by the National
Institute on Drug Abuse (NIDA 1R34DA037516-01A1; PI: Palinkas), afforded the opportunity to study
sustainment against the backdrop of a national context comprising of treatment and prevention initiatives
addressing an array of substance use and mental health problems for youth at-risk initially funded by
SAMHSA; strengthening the applicability of findings across various settings and delivery systems. Notably,
the parent study also enabled a closer look at sustainment of EBP using an empirically derived and tested
measure for sustainment (Palinkas et al., 2020; Palinkas et al., under review).
Key Contributions
The findings from both studies offer new perspectives about the mechanisms of sustainment and
open a new gateway by which strategies to improve implementation can be further refined.
87
Relative to commonly used cross-sectional methods to study the determinants of sustainment
(Shelton et al., 2018), study 1 offers a vastly different way of understanding what we know about
contextual factors that influence sustainment. While it is generally understood that elements within and
outside of the environment, such as leadership, partnerships, funding, and other resources, interact in
ways that strengthen other elements known to facilitate implementation and sustainment outcomes
(Aarons et al., 2014b; Aarons et al., 2014c; Moullin et al., 2018; Stevens & Peikes, 2006), traditional
methodologies used in health services research are usually not adequate to capture such exchanges. Kane
and colleagues (2014) first introduced the use of Qualitative Comparative Analysis (QCA) in
implementation research, after which several studies applied QCA and other comparative configurational
methods to better understand determinants and strategies to improve implementation (Kahwati et al.,
2016; Kane et al., 2017; Yakovchenko et al., 2020). The current study applied fuzzy-set QCA (fsQCA) and
contributes to this literature by offering additional strategies for making necessary decisions when
operationalizing and calibrating conditions. The application of fsQCA also contributes to the broader study
of sustainment by examining the interdependent nature of elements known to promote sustainment of
EBP. Out of over 30 possible configurations of five conditions that could lead to high levels of sustainment,
this study found two causal pathways in relation to sustainment: 1) community responsiveness and
organizational capacity when combined with implementation process, and 2) community responsiveness
and organizational capacity when combined with high functioning coalitions, partnerships, and networks.
These findings suggest that these elements when formed in the company of one another, are sufficient
and likely to produce sustainment. Notably, understanding determinants in configurational forms gives us
a different perspective about how elements are associated with outcomes, thereby informing
mechanisms of sustainment. At a time when there is intensely growing interest in learning about and
applying strategies to improve the success of implementation efforts (Lewis et al., 2018; Powell et al.,
2015; Waltz et al., 2019), it is necessary to provide alternative perspectives about the strategy selection
88
process. These findings offer a useful solution to such challenges and offer an approach that may be cost
saving to agencies responsible for selecting methods to improve the success of newly implemented
programs.
Study 2 explored a relatively untouched area of implementation science pertaining to the
integration of EBP within its host organization. Two key findings emerged. First, determinants that
predicted program integration were different from those that predicted sustainment. Second, in addition
to finding that integration predicts sustainment, my analysis determined that sustainment was mediated
by program integration. Of the 8 constructs examined, (1) responsiveness to community values and (2)
coalitions, networks, and partnerships were significantly associated with integration as well as
sustainment. In relation to sustainment, both constructs were significantly explained by integration, over
one-third and about one-fourth, respectively. Program integration, commonly addressed by several
names such as institutionalization and routinization, has been often noted as a driver of sustainment
(Scheirer & Dearing, 2011; Wilson & Kurz, 2008). This was the first study to the author’s knowledge to
empirically establish that integration is part of the process in the path to sustainment. Though more
research is needed to examine this longitudinally, findings suggest that there may be a phase that comes
between active implementation and sustainment in the widely used EPIS model. Additionally, these
findings help to explain how certain determinants of sustainment function, thereby offering an alternative
lens to study and exercise strategies that may help agencies adapt protocols that enable full program-
integration.
Rethinking the Study of Implementation of Health Promotion Programs: A Research Agenda
Implications for Policy Makers and Funding Agencies
The parent study was set against a backdrop of answering one key question asked by SAMHSA, a
major federal agency that provided over 3.2 billion dollars in 2014 alone (Substance Abuse and Mental
Health Services Administration, 2020): what happens to programs implemented in the community once
89
initial funding is expended? The question highlights the absence of knowledge that funding agencies have
about the success of programs that they fund after their initial involvement with the initiative ends. As
demonstrated by this dissertation, the importance of planning for implementation and sustainment and
then subsequently carrying out those plans appears time and again across studies (Cooper et al., 2015;
Palinkas et al., 2020). As policy makers point out, planning for sustainment within and across delivery
systems is also vital to sustaining an EBP (Willging et al., 2015).
When an EBP that is implemented in a community-based organization is not sustained, it limits
the potential for long-term public health impact (Cooper et al., 2015). Generally the findings from this
dissertation support previous literature detailing what aspects are important for policy makers involved
with the sustainment of programs in the public-sector, including implementation leadership, program
champions, partnerships and networks, adequate staff who are well trained and knowledgeable, and
working across systems to plan for sustainment (Willging et al., 2015). Additionally, policy makers at the
state level, especially for community-based organizations in the public sector and ones that are governed
by the state office of health or mental health, may have additional interests in the success of sustaining
EBPs as it concerns resources provided. For instance, as the program is being launched, training of staff,
costs of acquiring any necessary materials, resources needed from the local and state levels for fidelity
and quality improvement procedures are notably a part of implementing EBP (Aarons et al., 2014c). Many
of these associated costs and resources may not be the same in the sustainment phase (Roundfield &
Lang, 2017). If programs are implemented, but not sustained over time, and valuable resources continue
to be disbursed to the launch and implementation of new programs, this would not be effective for the
state. The totality of these findings combined suggest that policy makers and funding agencies should be
a continued presence in support of organizations implementing EBP, and that their collaborative and
active engagement may play a crucial role in the successful sustainment of mental health and prevention
initiatives.
90
Implications for Agency Leaders
Findings across both studies underscore the various components that leaders can directly take
part in to facilitate sustainment, especially in regard to building and maintaining partnerships, establishing
resources, and creating a positive culture of organizational change (Aarons et al., 2014a; Aarons et al.,
2016; Moullin et al., 2018). Two findings from this dissertation have further implications for agency
leaders. First, findings from study 1 suggest that the presence of implementation leadership when
combined with responding to community values and organizational capacity is sufficient to producing
desired levels of sustainment. Second, findings from study 2 highlights the role of implementation leaders
and program integration, which was also identified as a key factor in sustaining EBP. As such, if
implementation leaders take a role in ensuring that a program is well-embedded within organizational
proceedings, this in turn also improves sustainment. Consistent with previous literature, these findings
suggest that by formally appointing leaders who are proactive, knowledgeable, supportive, and
perseverant about implementation (Aarons et al., 2014b), agency leaders can improve the likeliness of
sustaining EBP within their programs.
Theoretical Implications
While numerous theoretical frameworks, process models, and evaluation frameworks exist
(Nilsen, 2015), many have noted the challenges of selecting appropriate frameworks (Flottorp et al., 2013)
as well as the limitations of research when theories are either underused, superficially used, or misused
(Birken et al., 2017). Both studies completed as part of this dissertation were guided by well-established
theoretical frameworks, and findings point to further theoretical implications that require additional
research. Study 1 supports a growing body of literature that acknowledges the complexity of interactions
within and between systems that make it difficult to attribute implementation outcomes to specific causal
relationships as if they were linear (Folttorp et al., 2013; Li et al., 2018; Nilsen, 2015) and also
demonstrates the interrelatedness of elements underscored in the EPIS model (Moullin et al., 2018).
91
Additionally, findings from study 1 advances the functionality of the widely used CFIR by assigning value
to each determinant within key constructs and thereby further facilitating how to prioritize determinants.
The main finding of study 2 implies that there may be a fifth phase in the EPIS model that we must
consider; specifically, one that accounts for program integration positioned between the active
implementation phase and sustainment phase. The notion of program integration, while not new, is
relatively understudied in implementation research. As demonstrated by others (Birken et al., 2017;
Greenhalgh, et al., 2004) and as I explore in this study, it is vital to look to theories across disciplines to
inform our work. Specifically, theories coming from the organization and management literature can guide
future examination of what it means to integrate programs within the procedures and practices of a host
organization after a once-novel program is no longer perceived as “new” and is embedded in the
organization. Future studies wishing to explore the concept of program-integration and its possible
inclusion within a conceptual model with EPIS, would greatly benefit from theoretical guidance and works
by Yin (1981), Goodman and Steckler (1989), Pluye and colleagues (2004), and Greenhalgh and colleagues
(2004).
Methodological Implications
Attempting to understand not only the translation of knowledge, but the process of implementing
new programs into community systems inherently involves several actors, some with competing interests,
against a backdrop of complex organizational systems. Yet much of the literature examining sustainment
is limited due to methodological constraints consisting largely of cross-sectional study designs (Shelton et
al., 2018). The methodological implications set forth by this dissertation are two-fold. Study 1 illustrates
the value of a qualitative and configurational approach to studying complex systems. The use of
Qualitative Comparative Analysis (QCA) quantifies the influence of sustainment and its determinants,
notably doing so in a way that recognizes all other elements involved. While there is an acknowledgement
of the interrelatedness of constructs, with one review even placing this in the context of the synergy of
92
elements (Li et al., 2018), many of the methods used to study the implementation of EBP do not
appropriately assess for configurations of elements when testing for causality. Alternatively, QCA
specifically enables the researcher to understand how different elements function when combined,
yielding an alternative perspective from traditional regression-based models (Ragin, 2008; Vis, 2012).
Importantly, this method also identifies the pathways that are sufficient to producing desired outcomes.
This finding can be valuable on its own when considering the limited approaches that may be available for
agency leaders to focus on in order to improve the likeliness of successfully implementing or sustaining a
program. The value that comparative configurational methods, such as QCA, and other mixed-methods
approaches can bring to implementation science is supported by many (Cragun et al., 2016; Kane et al.,
2014; Palinkas et al., 2019) and continues to grow in health services research. To better understand
mechanisms of the change process that can improve the success of implementation efforts, some have
called for the use of mediation analyses to further assess explanations for given relationships (Lewis et al.,
2018). Study 2 applied this approach in order to better understand the path to sustainment. While
determinants found to significantly predict sustainment were largely in accordance with previous
literature, the current study found that the level of which the program is well integrated into the
operations of the host organization is a key factor that helps explain the relationships between factors
such as coalitions and partnerships, responsiveness to community values, organizational capacity,
implementation leadership, and levels of sustaining EBP. Further mediation analysis of determinants and
desired outcomes, in addition to future studies that can explore how the strength of these determinants
change as a result of program-integration will further guide the use of implementation strategies.
Concluding Thoughts
Looking across findings from this dissertation, we see common elements surface as key
determinants to predicting sustainment of EBP. The different analytic approaches complement one
another and facilitate a new way of understanding the process and mechanisms of sustainment and can
93
offer a lens into how we select and use strategies to improve implementation in practice. Centered within
health and mental health services, my research agenda is organized around two domains: (1) pathways
to successful implementation and (2) impact of large-scale policy changes. In recent years implementation
researchers have made significant contributions to the field by developing strategies to improve
organizational practices, and at present there is a strong impetus for further understanding the
mechanisms behind such strategies. Guided by findings from my dissertation concerning pathways to
sustainment, I seek to advance how we utilize implementation strategies by examining how they form
synergistically—thereby answering how they work best together to improve the continued delivery of
EBP. The answers to such questions can offer a cost-effective and pragmatic approach for agency leaders
across community-based organizations. Building from my previous work, I also aspire to study how policy
changes shape organizations. Currently, in states like New York that are going through large-scale policy
changes such as the transition to value-based care, it is imperative that agency leaders put into place
appropriate measures that will facilitate positive client outcomes and quality of care. One approach to
doing so is to understand the service delivery setting in more detail. In recent years the United States has
seen a significant increase in alternative work arrangements, or the contingent workforce model,
spreading across industries and notably rising to 1 in 5 temporary or contract workers in education and
healthcare settings. Disadvantages of this model may include insufficient training, tenuous relationships
within the organization, and a culture that is restrictive to communication, knowledge sharing, and
organizational performance. In community mental health settings, this problematic landscape of relying
on per diem and fee-for-service (FFS) providers may interact with known organizational barriers that
impede successful implementation of EBP. One solution is to design implementation strategies that will
address these organizational barriers despite the use of a contingent workforce model. However, the
prevalence and impact of the contingent workforce on the delivery of EBP in healthcare settings is
currently unknown. In a recently funded study, I take the first step in understanding this phenomenon to
94
inform the optimization of multi-faceted implementation strategies and explore the restructuring of
organizations to support large-scale policy changes.
95
References
Aarons, G. A., Sommerfeld, D. H., Hecht, D. B., Silovsky, J. F., & Chaffin, M. J. (2009). The impact of
evidence-based practice implementation and fidelity monitoring on staff turnover: evidence for
a protective effect. Journal of consulting and clinical psychology, 77(2), 270–280.
https://doi.org/10.1037/a0013223
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based
practice implementation in public service sectors. Administration and Policy in Mental Health
and Mental Health Services Research, 38(1), 4-23. https://doi.org/10.1007/s10488-010-0327-7
Aarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014a). The implementation leadership scale (ILS):
development of a brief measure of unit level implementation leadership. Implementation
Science, 9(1), 45.
Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Sklar, M. (2014b). The role of leadership in creating a
strategic climate for evidence-based practice implementation and sustainment in systems and
organizations. Frontiers in Public Health Services and Systems Research, 3(4), 3.
Aarons, G. A., Green, A. E., Willging, C. E., Ehrhart, M. G., Roesch, S. C., Hecht, D. B., & Chaffin, M. J.
(2014c). Mixed-method study of a conceptual model of evidence-based intervention
sustainment across multiple public-sector service settings. Implementation Science, 9, Article
183. https://doi.org/10.1186/s13012-014-0183-z
Aarons, G. A., Green, A. E., Trott, E., Willging, C. E., Torres, E. M., Ehrhart, M. G., & Roesch, S. C. (2016).
The roles of system and organizational leadership in system-wide evidence-based intervention
sustainment: A mixed-method study. Administration and Policy in Mental Health and Mental
Health Services Research, 43(6), 991-1008. https://doi.org/10.1007/s10488-016-0751-4
96
Aarons, G. A., Sklar, M., Mustanski, B., Benbow, N., & Brown, C. H. (2017). “Scaling-out” evidence-based
interventions to new populations or new health care delivery systems. Implementation
Science, 12, Article 111. https://doi.org/10.1186/s13012-017-0640-6
Anderson, N., De Dreu, C. K., & Nijstad, B. A. (2004). The routinization of innovation research: A
constructively critical review of the state‐of‐the‐science. Journal of Organizational
Behavior, 25(2), 147-173. https://doi.org/10.1002/job.236
Arsenault-Lapierre, G., Kim, C., & Turecki, G. (2004). Psychiatric diagnoses in 3275 suicides: A meta-
analysis. BMC Psychiatry, 4, Article 37. https://doi.org/10.1186/1471-244X-4-37
Bailey, R. K., Patel, T. C., Avenido, J., Patel, M., Jaleel, M., Barker, N. C., Khan, J. A., Ali, S. A., & Jabeen, S.
(2011). Suicide: Current trends. Journal of the National Medical Association, 103(7), 614-617.
https://doi.org/10.1016/S0027-9684(15)30388-6
Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological
research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social
Psychology, 51(6), 1173-1182.
Beidas, R. S., Marcus, S., Wolk, C. B., Powell, B., Aarons, G. A., Evans, A. C., Hurford, M. O., Hadley, T.,
Adams, D. R., Walsh, L. M., Babbar, S., Barg, F., & Mandell, D. S. (2016). A prospective
examination of clinician and supervisor turnover within the context of implementation of
evidence-based practices in a publicly-funded mental health system. Administration and Policy
in Mental Health and Mental Health Services Research, 43(5), 640-649.
https://doi.org/10.1007/s10488-015-0673-6
Bernstein, S. L., & D’Onofrio, G. (2017). Screening, treatment initiation, and referral for substance use
disorders. Addiction Science & Clinical practice, 12, Article 18. https://doi.org/10.1186/s13722-
017-0083-z
97
Birken, S. A., Powell, B. J., Presseau, J., Kirk, M. A., Lorencatto, F., Gould, N. J., Shea, C. M., Weiner, B. J.,
Francis, J. J., Yan, Y., Haines, E., & Damschroder, L. J. (2017). Combined use of the Consolidated
Framework for Implementation Research (CFIR) and the Theoretical Domains Framework (TDF):
Aa systematic review. Implementation Science, 12, Article 2. https://doi.org/10.1186/s13012-
016-0534-z
Blasinsky, M., Goldman, H. H., & Unützer, J. (2006). Project IMPACT: A report on barriers and facilitators
to sustainability. Administration and Policy in Mental Health and Mental Health Services
Research, 33(6), 718-729. https://doi.org/10.1007/s10488-006-0086-7
Boustani, M. M., Frazier, S. L., Becker, K. D., Bechor, M., Dinizulu, S. M., Hedemann, E. R., Ogle, R. R., &
Pasalich, D. S. (2015). Common elements of adolescent prevention programs: Minimizing
burden while maximizing reach. Administration and Policy in Mental Health and Mental Health
Services Research, 42(2), 209-219. https://doi.org/10.1007/s10488-014-0541-9
Bowman, C. C., Sobo, E. J., Asch, S. M., Gifford, A. L., & HIV/Hepatitis Quality Enhancement Research
Initiative. (2008). Measuring persistence of implementation: QUERI Series. Implementation
Science, 3, Article 21. https://doi.org/10.1186/1748-5908-3-21
Bumbarger, B., & Perkins, D. F. (2008). After randomised trials: Issues related to dissemination of
evidence-based interventions. Journal of Children’s Services, 3(2), 55-64.
Castro, F. G., Barrera, M., & Martinez, C. R. (2004). The cultural adaptation of prevention interventions:
Resolving tensions between fidelity and fit. Prevention Science, 5(1), 41-45.
https://doi.org/10.1023/B:PREV.0000013980.12412.cd
Cha, C. B., Franz, P. J., M. Guzmán, E., Glenn, C. R., Kleiman, E. M., & Nock, M. K. (2018). Annual
Research Review: Suicide among youth–Epidemiology, (potential) etiology, and
treatment. Journal of Child Psychology and Psychiatry, 59(4), 460-482.
https://doi.org/10.1111/jcpp.12831
98
Chamberlain, P., Brown, C. H., & Saldana, L. (2011). Observational measure of implementation progress
in community-based settings: The stages of implementation completion (SIC). Implementation
Science, 6, Article 116. https://doi.org/10.1186/1748-5908-6-116
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013). The dynamic sustainability framework:
Addressing the paradox of sustainment amid ongoing change. Implementation Science, 8, Article
117. https://doi.org/10.1186/1748-5908-8-117
Chambers, D. A., Pearson, J. L., Lubell, K., Brandon, S., O'Brien, K., & Zinn, J. (2005). The science of public
messages for suicide prevention: A workshop summary. Suicide and Life-Threatening
Behavior, 35(2), 134-145. https://doi.org/10.1007/s10488-014-0551-7
Chor, K. H. B., Wisdom, J. P., Olin, S. C. S., Hoagwood, K. E., & Horwitz, S. M. (2015). Measures for
predictors of innovation adoption. Administration and Policy in Mental Health and Mental
Health Services Research, 42(5), 545-573. https://doi.org/10.1007/s10488-014-0551-7
Conway, K. P., Green, V. R., Kasza, K. A., Silveira, M. L., Borek, N., Kimmel, H. L., Sargent, J. D., Stanton,
C., Lambert, E., Hilmi, N., Reissig, C. J., Jackson, K. J., Tanski, S. E., Maklan, D., Hyland, A. J., &
Compton, W. M. (2017). Co-occurrence of tobacco product use, substance use, and mental
health problems among adults: Findings from Wave 1 (2013–2014) of the Population
Assessment of Tobacco and Health (PATH) Study. Drug and Alcohol Dependence, 177, 104-111.
https://doi.org/10.1016/j.drugalcdep.2017.03.032
Cooper, B. R., Bumbarger, B. K., & Moore, J. E. (2015). Sustaining evidence-based prevention programs:
Correlates in a large-scale dissemination initiative. Prevention Science, 16(1), 145-157.
https://doi.org/10.1007/s11121-013-0427-1
99
Cragun, D., Pal T., Vadaparamspil S. T., Baldwin J., Hampel, H., & DeBate, R. D. (2016). Qualitative
comparative analysis: A hybrid method for identifying factors associated with program
effectiveness. Journal of Mixed Methods Research. 10(3), 251-272.
https://doi.org/10.1177/1558689815572023
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering
implementation of health services research findings into practice: A consolidated framework for
advancing implementation science. Implementation science, 4(1), 1-15.
https://doi.org/10.1186/1748-5908-4-50
Das, J. K., Salam, R. A., Arshad, A., Finkelstein, Y., & Bhutta, Z. A. (2016). Interventions for adolescent
substance abuse: An overview of systematic reviews. Journal of Adolescent Health, 59(4), S61-
S75. https://doi.org/10.1016/j.jadohealth.2016.06.021
Duncombe, D. C. (2018). A multi‐institutional study of the perceived barriers and facilitators to
implementing evidence‐based practice. Journal of Clinical Nursing, 27(5-6), 1216-1226.
https://doi.org/10.1111/jocn.14168
Dy, S. M., Garg, P., Nyberg, D., Dawson, P. B., Pronovost, P. J., Morlock, L., Rubin, H., & Wu, A. W. (2005).
Critical pathway effectiveness: Assessing the impact of patient, hospital care, and pathway
characteristics using qualitative comparative analysis. Health Services Research, 40(2), 499-516.
https://doi.org/10.1111/j.1475-6773.2005.0r370.x
Feinberg, M. E., Bontempo, D. E., & Greenberg, M. T. (2008). Predictors and level of sustainability of
community prevention coalitions. American Journal Of Preventive Medicine, 34(6), 495-501.
https://doi.org/10.1016/j.amepre.2008.01.030
Fereday, J., & Muir-Cochrane, E. (2006). Demonstrating rigor using thematic analysis: A hybrid approach
of inductive and deductive coding and theme development. International Journal
of Qualitative Methods, 5(1), 80-92. https://doi.org/10.1177/160940690600500107
100
Fiss, P. C. (2007). A set-theoretic approach to organizational configurations. Academy Of Management
Review, 32(4), 1180-1198. https://doi.org/10.5465/amr.2007.26586092
Fiss, P. C. (2011). Building better causal theories: A fuzzy set approach to typologies in organization
research. Academy Of Management Journal, 54(2), 393-420.
https://doi.org/10.5465/amj.2011.60263120
Flottorp, S. A., Oxman, A. D., Krause, J., Musila, N. R., Wensing, M., Godycki-Cwirko, M., Baker, R., &
Eccles, M. P. (2013). A checklist for identifying determinants of practice: a systematic review and
synthesis of frameworks and taxonomies of factors that prevent or enable improvements in
healthcare professional practice. Implementation Science, 8, Article 35.
https://doi.org/10.1186/1748-5908-8-35
Forman, S. G., Olin, S. S., Hoagwood, K. E., Crowe, M., & Saka, N. (2009). Evidence-based interventions in
schools: Developers’ views of implementation barriers and facilitators. School Mental Health, 1,
Article 26. https://doi.org/10.1007/s12310-008-9002-5
Gamarel, K. E., Brown, L., Kahler, C. W., Fernandez, M. I., Bruce, D., & Nichols, S., & Adolescent Medicine
Trials Network for HIV/AIDS Intervention. (2016). Prevalence and correlates of substance use
among youth living with HIV in clinical settings. Drug and Alcohol Dependence, 169, 11-18.
https://doi.org/10.1016/j.drugalcdep.2016.10.002
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health
promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9),
1322-1327. https://doi.org/10.2105/AJPH.89.9.1322
Goldner, E. M., Lusted, A., Roerecke, M., Rehm, J., & Fischer, B. (2014). Prevalence of Axis-1 psychiatric
(with focus on depression and anxiety) disorder and symptomatology among non-medical
prescription opioid users in substance use treatment: Systematic review and meta-
analyses. Addictive Behaviors, 39(3), 520-531. https://doi.org/10.1016/j.addbeh.2013.11.022
101
Goodman, R. M., & Steckler, A. (1989). A framework for assessing program
institutionalization. Knowledge in Society, 2(1), 57-71. https://doi.org/10.1007/BF02737075
Goodman, R. M., McLeroy, K. R., Steckler, A. B., & Hoyle, R. H. (1993). Development of level of
institutionalization scales for health promotion programs. Health Education Quarterly, 20(2),
161-178. https://doi.org/10.1177/109019819302000208
Greckhamer, T., Misangyi, V. F., & Fiss, P. C. (2013). Chapter 3 The two QCAs: From a small-N to a large-
N set theoretic approach. Configurational Theory and Methods in Organizational Research, 38,
49-75. https://doi.org/10.1108/S0733-558X(2013)0000038007
Greckhamer, T., Fumari, S., Fiss, P. C., & Aguilera, R. V. (2018). Studying configurations with qualitative
comparative analysis: Best practices in strategy and organization research. Strategic
Organization, 16(4), 482–495. https://doi.org/10.1177/1476127018786487
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in
service organizations: Sysystematic review and recommendations. The Milbank Quarterly, 82(4),
581-629. https://doi.org/10.1111/j.0887-378X.2004.00325.x
Hailemariam, M., Bustos, T., Montgomery, B., Barajas, R., Evans, L. B., & Drahota, A. (2019). Evidence-
based intervention sustainability strategies: A systematic review. Implementation Science, 14,
Article 57. https://doi.org/10.1186/s13012-019-0910-6
Hamilton, A. B., & Finley, E. P. (2019). Qualitative methods in implementation research: An
introduction. Psychiatry Research, Article 112516.
https://doi.org/10.1016/j.psychres.2019.112516
Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health
programs. Journal of Abnormal Child Psychology, 33(6), 665-679.
https://doi.org/10.1007/s10802-005-7646-2
102
Healthy Minds Network study (2019). Retrieved May 31, 2020 from
https://healthymindsnetwork.org/wp-content/uploads/2019/09/HMS_national-2018-19.pdf
Hedegaard H, Curtin SC, Warner M. Increase in suicide mortality in the United States, 1999–2018. NCHS
Data Brief, no 362. Hyattsville, MD: National Center for Health Statistics. 2020. Retrieved May
31, 2020 from https://www.cdc.gov/nchs/products/databriefs/db362.htm
Heisenberg, W. (1958). Physics and philosophy: The revolution in modern science. New York: Harper
Row.
Hodge, L. M., & Turner, K. M. (2016). Sustained implementation of evidence‐based programs in
disadvantaged communities: A conceptual framework of supporting factors. American Journal of
Community Psychology, 58(1-2), 192-210. https://doi.org/10.1002/ajcp.12082
Hoomans, T., & Severens, J. L. (2014). Economic evaluation of implementation strategies in health care.
Implementation Science, 9, Article 168. https://doi.org/10.1186/s13012-014-0168-y
Hunter, S. B., Han, B., Slaughter, M. E., Godley, S. H., & Garner, B. R. (2015). Associations between
implementation characteristics and evidence-based practice sustainment: A study of the
Adolescent Community Reinforcement Approach. Implementation Science, 10, Article 173.
https://doi.org/10.1186/s13012-015-0364-4
Hunter, S. B., Han, B., Slaughter, M. E., Godley, S. H., & Garner, B. R. (2017). Predicting evidence-based
treatment sustainment: Results from a longitudinal study of the Adolescent-Community
Reinforcement Approach. Implementation Science, 12, Article 75.
https://doi.org/10.1186/s13012-017-0606-8
Johnson, A. M., Moore, J. E., Chambers, D. A., Rup, J., Dinyarian, C., & Straus, S. E. (2019). How do
researchers conceptualize and plan for the sustainability of their NIH R01 implementation
projects? Implementation Science, 14, Article 50. https://doi.org/10.1186/s13012-019-0895-1
103
Kahwati, L., Jacobs, S., Kane, H., Lewis, M., Viswanathan, M., & Golin, C. E. (2016). Using qualitative
comparative analysis in a systematic review of a complex intervention. Systematic Reviews, 5,
Article 82. https://doi.org/10.1186/s13643-016-0256-y
Kane, H., Lewis, M. A., Williams, P. A., & Kahwati, L. C. (2014). Using qualitative comparative analysis to
understand and quantify translation and implementation. Translational Behavioral
Medicine, 4(2), 201-208. https://doi.org/10.1007/s13142-014-0251-6
Kane, H., Hinnant, L., Day, K., Council, M., Tzeng, J., Soler, R, Chambard, M., & Roussel A. & Heirendt, W.
(2017). Pathways to program success: A qualitative comparative analysis (QCA) of communities
putting prevention to work case study programs. Journal of Public Health management and
practice: JPHMP, 23(2), 104-111. https://doi.org/10.1097/PHH.0000000000000449
Kirk, M. A., Kelley, C., Yankey, N., Birken, S. A., Abadie, B., & Damschroder, L. (2015). A systematic review
of the use of the consolidated framework for implementation research. Implementation
Science, 11, Article 72. https://doi.org/10.1186/s13012-016-0437-z
Langley, A. K., Nadeem, E., Kataoka, S. H., Stein, B. D., & Jaycox, L. H. (2010). Evidence-based mental
health programs in schools: Barriers and facilitators of successful implementation. School
Mental Health, 2(3), 105-113. https://doi.org/10.1007/s12310-010-9038-1
Lennox, L., Maher, L., & Reed, J. (2018). Navigating the sustainability landscape: a systematic review of
sustainability approaches in healthcare. Implementation Science, 13, Article 27.
https://doi.org/10.1186/s13012-017-0707-4
Lewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G. (2015). Outcomes for
implementation science: An enhanced systematic review of instruments using evidence-based
rating criteria. Implementation Science, 10, Article 155. https://doi.org/10.1186/s13012-015-
0342-x
104
Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., Walsh-Bailey, C., & Weiner, B.
(2018). From classification to causality: Advancing understanding of mechanisms of change in
implementation science. Frontiers in Public Health, 6, Article 136.
https://doi.org/10.3389/fpubh.2018.00136
Lewis, C. C., Boyd, M. R., Walsh-Bailey, C., Lyon, A. R., Beidas, R., Mittman, B., Aarons, G. A., Weiner, B.
J., & Chambers, D. A. (2020). A systematic review of empirical studies examining mechanisms of
implementation in health. Implementation Science, 15, Article 21.
https://doi.org/10.1186/s13012-020-00983-3
Li, S. A., Jeffs, L., Barwick, M., & Stevens, B. (2018). Organizational contextual features that influence the
implementation of evidence-based practices across healthcare settings: A systematic integrative
review. Systematic Reviews, 7, Article 72. https://doi.org/10.1186/s13643-018-0734-5
Lipari, R. N., Park-Lee, E., & Van Horn, S. (2016). America's need for and receipt of substance use
treatment in 2015. The CBHSQ Report. Center for Behavioral Health Statistics and Quality,
Substance Abuse and Mental Health Services Administration, Rockville, MD. Retrieved May 31,
2020 from https://www.samhsa.gov/data/sites/default/files/report_2716/ShortReport-
2716.html
Liu, Y., Mezei, J., Kostakos, V., & Li, H. (2017). Applying configurational analysis to IS behavioural
research: A methodological alternative for modelling combinatorial complexities. Information
Systems Journal, 27(1), 59-89. https://doi.org/10.1111/isj.12094
Luke, D. A., Calhoun, A., Robichaux, C. B., Elliott, M. B., & Moreland-Russell, S. (2014). Peer reviewed:
The program sustainability assessment tool: a new instrument for public health
programs. Preventing Chronic Disease, 11. https://doi.org/10.5888/pcd11.130184
105
Malick, R. (2018). Prevention of substance use disorders in the community and workplace. Indian Journal
of Psychiatry, 60(Suppl 4), S559-S563.
https://doi.org/10.4103/psychiatry.IndianJPsychiatry_24_18
Mendel, J. M., & Korjani, M. M. (2012). Charles Ragin’s fuzzy set qualitative comparative analysis (fsQCA)
used for linguistic summarizations. Information Sciences, 202, 1-23.
https://doi.org/10.1016/j.ins.2012.02.039
Mor Barak, M. E., Nissly, J. A., & Levin, A. (2001) Antecedents to retention and turnover among child
welfare, social work, and other human service employees: What can we learn from past
research? A review and metanalysis. Social Service Review, 75, 625–661.
https://www.jstor.org/stable/10.1086/323166
Moullin, J. C., Ehrhart, M. G., & Aarons, G. A. (2018). The role of leadership in organizational
implementation and sustainment in service agencies. Research on Social Work Practice, 28(5),
558-567. https://doi.org/10.1177/1049731517718361
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019). Systematic review of the
exploration, preparation, implementation, sustainment (EPIS) framework. Implementation
Science, 14, Article 1. https://doi.org/10.1186/s13012-018-0842-6
National Cancer Institute, National Institutes of Health. Implementation science: about IS. 2015.
https://cancercontrol.cancer.gov/IS/html. Accessed May 30, 2020
Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation
Science, 10(1), 53. https://doi.org/10.1186/s13012-015-0242-0
Nunnally, J. C. (1978) Psychometric theory (2
nd
ed.). McGraw Hill.
Nunnally, J. C., & Bernstein, I. H. (1994) Psychometric theory (3
rd
ed.). McGraw Hill.
106
Palinkas, L. A., Spear, S. E., Mendon, S. J., Villamar, J., Valente, T., Chou, C. P., Landsverk, J., Kellam, S. G.,
& Brown, C. H. (2015). Measuring sustainment of prevention programs and initiatives: A study
protocol. Implementation Science, 11, Article 95. https://doi.org/10.1186/s13012-016-0467-6
Palinkas, L. A., Mendon, S. J., & Hamilton, A. B. (2019). Innovations in mixed methods
evaluations. Annual Review of Public Health, 40, 423-442. https://doi.org/10.1146/annurev-
publhealth-040218-044215
Palinkas, L. A., Spear, S. E., Mendon, S. J., Villamar, J., Reynolds, C., Green, C. D., Olson, C., Adade, A., &
Brown, C. H. (2020). Conceptualizing and measuring sustainability of prevention programs,
policies, and practices. Translational Behavioral Medicine, 10(1), 136-145.
https://doi.org/10.1093/tbm/ibz170
Palinkas, L.A., Chou, C.P., Spear, S.E., Mendon, S.J., Villamar, J., & Brown, C.H. Measurement of
sustainment of prevention programs and initiatives: The sustainment measurement system
scale. Implementation Science. (under review).
Pallant, J. (2013). SPSS survival manual. McGraw-Hill Education (UK).
Paykani, T., Rafiey, H., & Sajjadi, H. (2018). A fuzzy set qualitative comparative analysis of 131 countries:
Which configuration of the structural conditions can explain health better?. International
Journal for Equity in Health, 17, Article 10. https://doi.org/10.1186/s12939-018-0724-1
Peterson, A. E., Bond, G. R., Drake, R. E., McHugo, G. J., Jones, A. M., & Williams, J. R. (2014). Predicting
the long-term sustainability of evidence-based practices in mental health care: Aan 8-year
longitudinal analysis. The Journal of Behavioral Health Services & Research, 41(3), 337-346.
https://doi.org/10.1007/s11414-013-9347-x
Pluye, P., Potvin, L., Denis, J. L., & Pelletier, J. (2004a). Program sustainability: Focus on organizational
routines. Health Promotion International, 19(4), 489-500.
https://doi.org/10.1093/heapro/dah411
107
Pluye, P., Potvin, L., & Denis, J. L. (2004b). Making public health programs last: Conceptualizing
sustainability. Evaluation and Program Planning, 27(2), 121-133.
https://doi.org/10.1016/j.evalprogplan.2004.01.001
Pluye, P., Potvin, L., Denis, J. L., Pelletier, J., & Mannoni, C. (2005). Program sustainability begins with the
first events. Evaluation and Program Planning, 28(2), 123-137.
https://doi.org/10.1016/j.evalprogplan.2004.10.003
Powell, B. J., Proctor, E. K., & Glass, J.E. (2014). A systematic review of strategies for implementing
empirically supported mental health interventions. Research on Social Work Practice, 24(2),
192–212. https://doi.org/10.1177/1049731513505778
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K.,
& Kirchner, J. E. (2015). A refined compilation of implementation strategies: Results from the
Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10,
Article 21. https://doi.org/10.1186/s13012-015-0209-1
Powell, B. J., Beidas, R. S., Lewis, C. C., Aarons, G. A., McMillen, J. C., Proctor, E. K., & Mandell, D. S.
(2017). Methods to improve the selection and tailoring of implementation strategies. The
Journal of Behavioral Health Services & Research, 44(2), 177-194.
https://doi.org/10.1007/s11414-015-9475-6
Powell, B. J., Fernandez, M. E., Williams, N. J., Aarons, G. A., Beidas, R. S., Lewis, C. C., McHugh, S. M., &
Weiner, B. J. (2019). Enhancing the impact of implementation strategies in healthcare: A
research agenda. Frontiers in Public Health, 7, Article 3.
https://doi.org/10.3389/fpubh.2019.00003
108
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M.
(2011). Outcomes for implementation research: Conceptual distinctions, measurement
challenges, and research agenda. Administration and Policy in Mental Health and Mental Health
Services Research, 38(2), 65-76. https://doi.org/10.1007/s10488-010-0319-7
Ragin, C. C. 1987. The comparative method. Moving beyond qualitative and quantitative strategies.
University of California Press.
Ragin, C. C. 2008. User's Guide to Fuzzy-Set/Qualitative Comparative Analysis 3.0. Irvine, California:
Department of Sociology, University of California.
Ragin, C. C., & Rihoux, B. (Eds.). (2009). Configurational comparative methods: Qualitative comparative
analysis (QCA) and related techniques. Sage.
Ragin, C. C. (2014). The comparative method: Moving beyond qualitative and quantitative strategies.
University of California Press. (Originally published in 1987.)
Ram, D., Darshan, M. S., Rao, T. S., & Honagodu, A. R. (2012). Suicide prevention is possible: A
perception after suicide attempt. Indian Journal of Psychiatry, 54(2), 172-176. https://doi.org/
10.4103/0019-5545.99535
Roundfield, K. D., & Lang, J. M. (2017). Costs to community mental health agencies to sustain an
evidence-based practice. Psychiatric Sservices, 68(9), 876-882.
https://doi.org/10.1176/appi.ps.201600193
Rycroft-Malone, J. (2004). The PARIHS framework—A framework for guiding the implementation of
evidence-based practice. Journal of Nursing Care Quality, 19(4), 297-304.
Substance Abuse and Mental Health Services Administration. (2020, April 4). SAMHSA Grant Awards By
State: Fiscal Year 2014. https://www.samhsa.gov/grants-awards-by-state?year=2014
109
Scheirer, M. A. (1993). Are the Level of Institutionalization Scales Ready for “Prime Time"? A
commentary on "Development of Level of Institutionalization (LoIn) Scales for Health Promotion
Programs". Health Education Quarterly, 20(2), 179-182.
Scheirer, M. A. (2005). Is sustainability possible? A review and commentary on empirical studies of
program sustainability. American Journal of Evaluation, 26(3), 320-347.
https://doi.org/10.1177/1098214005278752
Scheirer, M. A., & Deering, J. W. (2011). An agenda for research on the sustainability of public health
programs. American Journal of Public Health, 101, 2059-2067.
https://doi.org/10.2105/AJPH.2011.300193
Schneider, C. Q., & Wagemann, C. (2010). Standards of good practice in qualitative comparative analysis
(QCA) and fuzzy-sets. Comparative Sociology, 9(3), 397-418.
https://doi.org/10.1163/156913210X12493538729793
Schneider, C. Q., & Wagemann, C. (2012). Set-theoretic methods for the social sciences: A guide to
qualitative comparative analysis. Cambridge University Press.
Shediac-Rizkallah, M. C., & Bone, L. R. (1998). Planning for the sustainability of community-based health
programs: Conceptual frameworks and future directions for research, practice and
policy. Health Education Research, 13(1), 87-108. https://doi.org/10.1093/her/13.1.87
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018). The sustainability of evidence-based interventions
and practices in public health and health care. Annual Review of Public Health, 39, 55-76.
https://doi.org/10.1146/annurev-publhealth-040617-014731
Smith, J. R., & Donze, A. (2010). Assessing environmental readiness: First steps in developing an
evidence-based practice implementation culture. The Journal of Perinatal and Neonatal
Nursing, 24(1), 61-71. https://doi.org/10.1097/JPN.0b013e3181ce1357
110
Song, M. K., Lin, F. C., Ward, S. E., & Fine, J. P. (2013). Composite variables: When and how. Nursing
Research, 62(1), 45-49. https://doi.org/10.1097/NNR.0b013e3182741948
Stevens, B., & Peikes, D. (2006). When the funding stops: Do grantees of the Local Initiative Funding
Partners Program sustain themselves?. Evaluation and Program Planning, 29(2), 153-161.
https://doi.org/10.1016/j.evalprogplan.2006.01.009
Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of
new programs and innovations: A review of the empirical literature and recommendations for
future research. Implementation Science, 7, Article 17. https://doi.org/10.1186/1748-5908-7-17
Thygeson, N. M., Solberg, L. I., Asche, S. E., Fontaine, P., Pawlson, L. G., & Scholle, S. H. (2012). Using
fuzzy set qualitative comparative analysis (fs/QCA) to explore the relationship between medical
“homeness” and quality. Health Services Research, 47(1 Pt 1), 22-45.
https://doi.org/10.1111/j.1475-6773.2011.01303.x
Tibbits, M. K., Bumbarger, B. K., Kyler, S. J., & Perkins, D. F. (2010). Sustaining evidence-based
interventions under real-world conditions: Results from a large-scale diffusion
project. Prevention Science, 11(3), 252-262. https://doi.org/10.1007/s11121-010-0170-9
U.S. Department of Health and Human Services, National Institute of Mental Health. (2017). Suicide
prevention. Retrieved May 31, 2020 from https://www.nimh.nih.gov/health/topics/suicide-
prevention/index.shtml
Vale, L., Thomas, R., MacLennan, G., & Grimshaw, J. (2007). Systematic review of economic evaluations
and cost analyses of guideline implementation strategies. The European Journal of Health
Economics, 8(2), 111-121. https://doi.org/10.1007/s10198-007-0043-8
Vis, B. (2012). The comparative advantages of fsQCA and regression analysis for moderately large-N
analyses. Sociological Methods and Research, 41(1), 168-198.
https://doi.org/10.1177/0049124112442142
111
Wallin, E., Lindewald, B., & Andréasson, S. (2004). Institutionalization of a community action program
targeting licensed premises in Stockholm, Sweden. Evaluation Review, 28(5), 396-419.
https://doi.org/10.1177/0193841X04264951
Waltz, T. J., Powell, B. J., Fernández, M. E., Abadie, B., & Damschroder, L. J. (2019). Choosing
implementation strategies to address contextual barriers: Diversity in recommendations and
future directions. Implementation Science, 14, Article 42. https://doi.org/10.1186/s13012-019-
0892-4
Willging, C. E., Green, A. E., Gunderson, L., Chaffin, M., & Aarons, G. A. (2015). From a “perfect storm” to
“smooth sailing” policymaker perspectives on implementation and sustainment of an evidence-
based practice in two states. Child Maltreatment, 20(1), 24-36.
https://doi.org/10.1177/1077559514547384
Williams, N. J., Glisson, C., Hemmelgarn, A., & Green, P. (2017). Mechanisms of change in the ARC
organizational strategy: Increasing mental health clinicians’ EBP adoption through improved
organizational culture and capacity. Administration and Policy in Mental Health and Mental
Health Services Research, 44(2), 269-283. https://doi.org/10.1007/s10488-016-0742-5
Wilson, K. D., & Kurz, R. S. (2008). Bridging implementation and institutionalization within organizations:
Pproposed employment of continuous quality improvement to further dissemination. Journal of
Public Health Management and Practice, 14(2), 109-116.
https://doi.org/10.1097/01.PHH.0000311887.06252.5f
Yakovchenko, V., Miech, E. J., Chinman, M. J., Chartier, M., Gonzalez, R., Kirchner, J. E., Morgan, T. R.,
Park, A., Powell, B. J., Proctor, E. K., Ross, D., Waltz, T. J., & Rogal, S. S. (2020). Strategy
Configurations Directly Linked to Higher Hepatitis C Virus Treatment Starts: An Applied Use of
Configurational Comparative Methods. Medical Care, 58(5), 31-38.
https://doi.org/10.1097/MLR.0000000000001319
112
Yin, R. K. (1979). Changing urban bureaucracies: How new practices become routinized. Lexington,
Mass.: Lexington Books.
Yin, R. K. (1981). Life histories of innovations: How new practices become routinized. Public
Administration Review, 21-28. https://doi.org/10.2307/975720
Zakumumpa, H., Kwiringira, J., Rujumba, J., & Ssengooba, F. (2018). Assessing the level of
institutionalization of donor-funded anti-retroviral therapy (ART) programs in health facilities in
Uganda: Implications for program sustainability. Global Health Action, 11(1), Article 1523302.
https://doi.org/10.1080/16549716.2018.1523302
113
Appendix 1: Consolidated Framework for Implementation Research (CFIR)
CFIR Constructs
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowerty, J. C. (2009)
Construct Short Description
I. INTERVENTION CHARACTERISTICS
A Intervention Source
Perception of key stakeholders about whether the intervention is externally or
internally developed.
B Evidence Strength & Quality
Stakeholders’ perceptions of the quality and validity of evidence supporting
the belief that the intervention will have desired outcomes.
C Relative Advantage
Stakeholders’ perception of the advantage of implementing the intervention
versus an alternative solution.
D Adaptability
The degree to which an intervention can be adapted, tailored, refined, or
reinvented to meet local needs.
E Trialability
The ability to test the intervention on a small scale in the organization, and to
be able to reverse course (undo implementation) if warranted.
F Complexity
Perceived difficulty of implementation, reflected by duration, scope,
radicalness, disruptiveness, centrality, and intricacy and number of steps
required to implement.
G Design Quality & Packaging
Perceived excellence in how the intervention is bundled, presented, and
assembled.
H Cost
Costs of the intervention and costs associated with implementing the
intervention including investment, supply, and opportunity costs.
II. OUTER SETTING
A Patient Needs & Resources
The extent to which patient needs, as well as barriers and facilitators to meet
those needs, are accurately known and prioritized by the organization.
B Cosmopolitanism
The degree to which an organization is networked with other external
organizations.
C Peer Pressure
Mimetic or competitive pressure to implement an intervention; typically
because most or other key peer or competing organizations have already
implemented or are in a bid for a competitive edge.
D External Policy & Incentives
A broad construct that includes external strategies to spread interventions,
including policy and regulations (governmental or other central entity),
external mandates, recommendations and guidelines, pay-for-performance,
collaboratives, and public or benchmark reporting.
114
III. INNER SETTING
A Structural Characteristics The social architecture, age, maturity, and size of an organization.
B Networks & Communications
The nature and quality of webs of social networks and the nature and quality
of formal and informal communications within an organization.
C Culture Norms, values, and basic assumptions of a given organization.
D Implementation Climate
The absorptive capacity for change, shared receptivity of involved individuals
to an intervention, and the extent to which use of that intervention will be
rewarded, supported, and expected within their organization.
1 Tension for Change
The degree to which stakeholders perceive the current situation as intolerable
or needing change.
2 Compatibility
The degree of tangible fit between meaning and values attached to the
intervention by involved individuals, how those align with individuals’ own
norms, values, and perceived risks and needs, and how the intervention fits
with existing workflows and systems.
3 Relative Priority
Individuals’ shared perception of the importance of the implementation within
the organization.
4
Organizational Incentives &
Rewards
Extrinsic incentives such as goal-sharing awards, performance reviews,
promotions, and raises in salary, and less tangible incentives such as increased
stature or respect.
5 Goals and Feedback
The degree to which goals are clearly communicated, acted upon, and fed back
to staff, and alignment of that feedback with goals.
6 Learning Climate
A climate in which: a) leaders express their own fallibility and need for team
members’ assistance and input; b) team members feel that they are essential,
valued, and knowledgeable partners in the change process; c) individuals feel
psychologically safe to try new methods; and d) there is sufficient time and
space for reflective thinking and evaluation.
E Readiness for Implementation
Tangible and immediate indicators of organizational commitment to its
decision to implement an intervention.
1 Leadership Engagement
Commitment, involvement, and accountability of leaders and managers with
the implementation.
2 Available Resources
The level of resources dedicated for implementation and on-going operations,
including money, training, education, physical space, and time.
3
Access to Knowledge &
Information
Ease of access to digestible information and knowledge about the intervention
and how to incorporate it into work tasks.
115
IV. CHARACTERISTICS OF INDIVIDUALS
A
Knowledge & Beliefs about the
Intervention
Individuals’ attitudes toward and value placed on the intervention as well as
familiarity with facts, truths, and principles related to the intervention.
B Self-efficacy
Individual belief in their own capabilities to execute courses of action to
achieve implementation goals.
C Individual Stage of Change
Characterization of the phase an individual is in, as he or she progresses
toward skilled, enthusiastic, and sustained use of the intervention.
D
Individual Identification with
Organization
A broad construct related to how individuals perceive the organization, and
their relationship and degree of commitment with that organization.
E Other Personal Attributes
A broad construct to include other personal traits such as tolerance of
ambiguity, intellectual ability, motivation, values, competence, capacity, and
learning style.
V. PROCESS
A Planning
The degree to which a scheme or method of behavior and tasks for
implementing an intervention are developed in advance, and the quality of
those schemes or methods.
B Engaging
Attracting and involving appropriate individuals in the implementation and use
of the intervention through a combined strategy of social marketing,
education, role modeling, training, and other similar activities.
1 Opinion Leaders
Individuals in an organization who have formal or informal influence on the
attitudes and beliefs of their colleagues with respect to implementing the
intervention.
2
Formally Appointed Internal
Implementation Leaders
Individuals from within the organization who have been formally appointed
with responsibility for implementing an intervention as coordinator, project
manager, team leader, or other similar role.
3 Champions
“Individuals who dedicate themselves to supporting, marketing, and ‘driving
through’ an [implementation]” [101] (p. 182), overcoming indifference or
resistance that the intervention may provoke in an organization.
4 External Change Agents
Individuals who are affiliated with an outside entity who formally influence or
facilitate intervention decisions in a desirable direction.
C Executing Carrying out or accomplishing the implementation according to plan.
D Reflecting & Evaluating
Quantitative and qualitative feedback about the progress and quality of
implementation accompanied with regular personal and team debriefing about
progress and experience.
Abstract (if available)
Abstract
Mental health and substance use disorders, including suicide, continue to plague adolescents and young adults across the U.S. While numerous prevention initiatives are developed, few are implemented in community-based settings where it can be accessed by those in need. Even fewer are sustained once initial seed funding ends, threatening the long-term public health impact of such evidence-based practices (EBP). Numerous studies in recent years have focused on identifying factors that facilitate sustainment and refining strategies to improve implementation and sustainment of health promotion and prevention programs in community settings. However, knowledge pertaining to mechanisms of and causal pathways to sustainment are considerably lacking. ❧ This dissertation draws from a larger parent study funded by the National Institute on Drug Abuse which sought to understand how evidence-based programs are sustained once seed funding ends. The current studies benefited from a comprehensive conceptualization of sustainment based on various prevention initiatives across a variety of delivery settings, and the use of a newly developed tool to measure sustainment and its determinants. Additionally, both studies in this dissertation are theoretically guided by the Consolidated Framework for Implementation Research (CFIR) and the Exploration, Preparation, Implementation, Sustainment (EPIS) conceptual model. ❧ The first chapter highlights the importance of studying sustainment in health services research and is followed by a detailed description of why studying sustainment has proved to be challenging. Lastly, a brief history of what we currently know about sustaining EBP is discussed in the context of how the field of implementation research is progressing, and what remains to be further developed. ❧ Efforts to deepen our understanding of causal mechanisms and improve the selection of implementation strategies have continued to rely on traditional regression-based models. The first study (chapter two) uses a different approach to identify causal pathways to sustainment based on a configurational approach. The application of fuzzy-set Qualitative Comparative Analysis (fsQCA), a set-theoretic approach, focused on how determinants of sustainment work interdependently. Of over 30 possible configurations, this study found two causal pathways that are sufficient to achieving sustainment. Study two (chapter three) sought to understand the process of sustainment. Program-integration, a largely understudied concept in implementation research, was believed by many to be a driver of sustainment. This is the first study to empirically assess what elements promote program-integration, as well as its relationship to sustainment. Determinants of sustainment found to be explained by integration suggests that integration may be a phase between active implementation and sustainment. ❧ The overall contributions of this dissertation are discussed in chapter four. Subsequently, a discussion highlighting key findings, such as planning for sustainment, is provided for policy makers and funding agencies, and pragmatic suggestions for agency leaders are specified, including approaches to applying newly established knowledge about determinants of sustainment. To move the field of implementation science forward, implications and detailed recommendations are offered for the refinement of theoretical frameworks and methodological approaches.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
How organizations adapt during EBP implementation
PDF
Multilevel influences on organizational adoption of innovative practices in children's mental health services
PDF
Implementation of peer providers in integrated health care settings
PDF
Effectiveness and implementation of a maternal depression treatment in a Head Start setting: a mixed method study
PDF
Low-socioeconomic status families: the role of parental involvement and its association with early childhood academic achievement trajectories
PDF
Acculturation team-based clinical program: pilot program to address acculturative stress and mental health in the Latino community
PDF
Mental Health First
PDF
Unto the least of these homeless ministry: ending homelessness within the co-occurring population
PDF
Promoting emotional intelligence and resiliency in youth: S.U.P.E.R. peer counseling program ©
PDF
Broken promises: gaining an understanding of policy-practice and means-ends decoupling within diversity and inclusion
PDF
Inter professional education and practice in the health care setting: an innovative model using human simulation learning
PDF
Immigrants at a loss: the need for services that promote child well-being among Latino families with child welfare contact
PDF
Assessment and analysis of direct care community worker training: addressing social determinants of health in the home care setting
PDF
LGBT+Aging Immersion Experience: an innovative LGBT cultural competency course for healthcare professionals and students
PDF
Kids Cope Primary Prevention Services: evidence-based practices for supporting children impacted by parental incarceration
PDF
How can metrics matter: performance management reforms in the City of Los Angeles
PDF
The influence of contextual factors on the processes of adoption and implementation of evidence-based substance use prevention and tobacco cessation programs in schools
PDF
Exploring the social determinants of health in a population with similar access to healthcare: experiences from United States active-duty army wives
PDF
Conjoint homeless prevention services for older adults
PDF
Incarceration trajectories of mothers in state and federal prisons and their relation to the mother’s mental health problems and child’s risk of incarceration
Asset Metadata
Creator
Mendon, Sapna J.
(author)
Core Title
Delineating the processes and mechanisms of sustainment: understanding how evidence-based prevention initiatives are maintained after initial funding ends
School
Suzanne Dworak-Peck School of Social Work
Degree
Doctor of Philosophy
Degree Program
Social Work
Publication Date
08/07/2020
Defense Date
06/19/2020
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
implementation,Mental Health,mixed-methods,OAI-PMH Harvest,sustainment
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Palinkas, Lawrence A. (
committee chair
), Fiss, Peer C. (
committee member
), Hurlburt, Michael (
committee member
)
Creator Email
sapna.mendon@gmail.com,smendon@usc.edu
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c89-359795
Unique identifier
UC11666342
Identifier
etd-MendonSapn-8897.pdf (filename),usctheses-c89-359795 (legacy record id)
Legacy Identifier
etd-MendonSapn-8897.pdf
Dmrecord
359795
Document Type
Dissertation
Rights
Mendon, Sapna J.
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
implementation
mixed-methods
sustainment