Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Mixed methods investigation of user engagement with a smoking cessation app
(USC Thesis Other)
Mixed methods investigation of user engagement with a smoking cessation app
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
1
Mixed methods investigation of user engagement with a smoking cessation app
by
Christian J. Cerrada
A Dissertation Presented to the
FACULTY OF THE USC GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(PREVENTIVE MEDICINE: HEALTH BEHAVIOR RESEARCH)
May 2019
2
ACKNOWLEDGEMENTS
I owe a deep debt of gratitude to my committee members, Dr. Jimi Huh, Dr. Donna
Spruijt-Metz, Dr. Chih-Ping Chou, and Dr. Ricky Bluthenthal for their guidance throughout the
doctoral program. I owe my accomplishments to my mentor, Dr. Jimi Huh. She has generously
offered her time and wisdom to me throughout every step of my doctoral career. I would
certainly not be where I am today without her. I thank Dr. Bluthenthal and Dr. Chou for
providing me with the skills I needed to complete this dissertation through coursework and
feedback on writing. I thank Dr. Donna Spruijt-Metz for her encouragement and guidance on
data visualization and mHealth.
I would like to extend my gratitude for the MyQuit team and acknowledge the herculean
effort that was required to recruit and enroll our participants over the course of two years. I
would especially like to thank those I have worked closely with on this project: Sheila Yu, Will
Fang, Jonathan Castaneda, Ana Francisco, and Katherine Colao, who have assisted me with
transcribing and coding the interviews.
I’d like to extend warm thanks to my fellow students, especially Sydney O’Connor,
Darryl Nousome, Brooke Bell, Anupreet Sidhu, Christopher Warren, Derek Dangerfield, Kathy
Wojcik, and many others along the way who have a created a warm and supportive environment
at Soto. I am grateful for Marny Barovich, who has been an invaluable resource to the Health
Behavior Research Program. Thank you to Jordan Patricio for being a part of this journey with
me.
Finally, this dissertation is dedicated to my family. I am deeply grateful for my parents,
Eliseo and Judy, who have supported me without question in all of my endeavors and for my
sister, Christine, who is a constant source of encouragement and warmth.
3
Table of Contents
List of Figures and Tables.................................................................................................. 5
Abstract............................................................................................................................... 6
Chapter 1: Introduction....................................................................................................... 8
Background and Significance........................................................................................ 8
User engagement in behavior change interventions...................................................... 8
Popularity and effectiveness of smoking cessation apps............................................... 8
App usage can help explain intervention effectiveness................................................. 10
Defining user engagement: a construct with multiple dimensions................................ 11
Factors theorized to influence UE in behavior change interventions............................ 15
User engagement vs. effective engagement.................................................................. 16
Interacting with a diverse set of features and underlying usage patterns...................... 19
Overview of dissertation studies.................................................................................... 20
Specific aims and hypotheses........................................................................................ 21
MyQuit USC study design............................................................................................. 24
Description of the MyQuit USC (MQU) app features.................................................. 28
Chapter 2: Mixed methods investigation of user engagement with a smoking cessation
app...................................................................................................................................... 30
Introduction................................................................................................................... 30
Specific Aims................................................................................................................ 35
Methods......................................................................................................................... 39
Results........................................................................................................................... 42
Discussion...................................................................................................................... 60
Conclusion..................................................................................................................... 66
Chapter 3: Identifying dynamic user engagement patterns with a quit app using
multilevel latent class analysis........................................................................................... 67
Introduction................................................................................................................... 67
Specific Aims................................................................................................................ 72
Methods......................................................................................................................... 73
Results........................................................................................................................... 77
Discussion...................................................................................................................... 83
Conclusion..................................................................................................................... 90
Chapter 4: Investigation of effective engagement with a quit app and daily lapse during
a quit attempt using mixed methods................................................................................... 91
Introduction................................................................................................................... 91
Specific Aims................................................................................................................ 97
Methods......................................................................................................................... 98
Results........................................................................................................................... 101
Discussion...................................................................................................................... 115
Conclusion.................................................................................................................... 123
Chapter 5: Summary and Conclusions............................................................................... 124
Summary of aims and findings...................................................................................... 124
Implications................................................................................................................... 126
Contributions to the literature........................................................................................ 132
Appendix A. Examples of operationalization of user engagement in digital behavior
change interventions........................................................................................................... 133
4
Appendix B. Semi-structured interview guide................................................................... 135
Appendix C. Study 1 Coding Scheme................................................................................ 136
Appendix D. Study 3 Coding Scheme................................................................................ 137
References.......................................................................................................................... 138
5
List of Figures and Tables
Figure 1-1 User engagement cycle..................................................................................... 13
Figure 1-2 Internet intervention model.............................................................................. 15
Figure 1-3 MQU home screen............................................................................................ 28
Figure 1-4 II reminder........................................................................................................ 29
Figure 1-5 MyProgress....................................................................................................... 29
Figure 2-1 MQU home screen............................................................................................ 38
Figure 2-2 II reminder....................................................................................................... 38
Figure 2-3 MyProgress....................................................................................................... 38
Figure 2-4 Percentage of users interacting with each MQU feature over 28 days............ 44
Figure 2-5 Percentage of users interacting with each MQU feature over 4 weeks............ 45
Figure 2-6a U27.................................................................................................................. 51
Figure 2-6b U6................................................................................................................... 52
Figure 2-6c U21.................................................................................................................. 53
Figure 2-6d U36.................................................................................................................. 53
Figure 2-6e U54.................................................................................................................. 54
Figure 2-7 Behavioral UE trends by perceived usefulness groups.................................... 58
Figure 2-8 Behavioral UE trends by phone type................................................................ 60
Figure 3-1 Example 3-class multilevel latent class model with covariate......................... 76
Figure 3-2 Heat map of most likely day-level UE pattern by user and day in quit
attempt................................................................................................................................ 82
Figure 4-1 Example 3-class multilevel latent class model with covariate......................... 99
Figure 4-2 Heat map of most likely day-level UE pattern by user and day in quit
attempt ordered by lapse outcome...................................................................................... 120
Table 2-1 User-level characteristics................................................................................... 42
Table 2-2 Number of interactions per day with each MQU feature................................... 43
Table 3-1 MLCA indicators and descriptions.................................................................... 74
Table 3-2 MLCA indicators with univariate frequencies .................................................. 77
Table 3-3 Model fit indices................................................................................................ 78
Table 3-4 Multilevel latent class solution for 3-class model with conditional
probabilities for indicators................................................................................................. 80
Table 3-5 Covariates of day-level latent class membership............................................... 81
Table 3-6 Multilevel latent class solution for 3-class model with conditional
probabilities for indicators with covariates........................................................................ 81
Table 4-1 Single level LCA and MLCA fit indices........................................................... 101
Table 4-2 3-class MLCA solution with conditional probabilities for indicators without
lapse covariate.................................................................................................................... 103
Table 4-3 3-class MLCA solution with conditional probabilities for indicators with
lapse covariate.................................................................................................................... 103
Table 4-4 Themes representing mechanisms of lapse avoidance...................................... 115
6
ABSTRACT
Mobile smoking cessation apps have the potential to reduce smoking rates on a large
scale but app abandonment after minimal use remains a significant challenge. Furthermore, little
is known about the diverse ways smokers use quit apps and how usage mediates positive
cessation outcomes. Thus, the overarching aim of this dissertation was to better understand user
engagement (UE) with a recently developed cessation app, MQU, over the course of a quit
attempt. Following a mixed methods approach, user logs of interaction with each of MQU’s
features were combined with semi-structured interviews to examine longitudinal trends in usage,
factors that influenced usage over time, and relevant mechanisms for lapse avoidance. Multilevel
latent class analysis was used to explore underlying day-level UE patterns and to examine their
association with covariates and a lapse outcome.
Our findings illustrated differences in usage trends between “PUSH” and “PULL” app
features over time and highlighted the second week of the quit attempt as an important time point
for changes in usage and perceptions of MQU novelty and usefulness. Multilevel latent class
analysis uncovered three day-level UE patterns within users: Active, Passive, and Low UE
patterns. Relative to Low UE days, Active UE days were less probable among loan phone users
and as days passed in the quit attempt. Contrary to our hypothesis, Active UE days were
associated with greater probability of same-day lapse relative to Low and Passive UE days.
Finally, we elucidated mechanisms for lapse avoidance attributable to MQU usage, such as
behavioral regulation and increased motivation to quit, along with mechanisms not related to
MQU, such as momentary lack of access to cigarettes and availability to smoke.
Based on our findings, we provide recommendations for strategies to promote re-
engagement with quit apps over time, such as dynamically adapting the content and frequency of
7
II reminders to increase their perceived novelty and usefulness. Furthermore, we describe a
strategy to enhance MQU effectiveness by promoting the occurrence of a day-level UE pattern
found to be associated with reduced probability of lapse (Passive UE days). These strategies may
be applicable to other quit apps that combine PUSH and PULL features and after further testing,
could inform the development of future quit apps that users are motivated to use throughout a
quit attempt and provide effective cessation support.
8
CHAPTER 1: Introduction
Background and Significance
User engagement in behavior change interventions
Mobile smoking cessation apps have the potential to make a large impact on public health
because they are capable of providing highly accessible and personalized quit support to a wide
range smokers
1–3
. But for such interventions to be effective in reducing smoking rates on a large
scale, smokers need to use, or interact, with them to the extent that interventionists intended
1
.
Abandonment after minimal use, however, remains a significant challenge for digital behavior
change interventions such as quit apps and limits the potential of such tools to promote health
1,4
.
To address this challenge, behavioral scientists are increasingly interested in understanding user
engagement (UE) with interventions in an effort to design more effective and engaging tools
1
.
UE describes the amount and manner of interaction a user has with a system and thus represents
a key mediator of the effect of digital behavior change interventions on intended outcomes
5–8
.
Although research is emerging on UE in the context of behavior change interventions, there is
currently limited research on UE with quit apps and how usage mediates cessation outcomes
9
.
Thus, the overarching objective of this dissertation was to examine UE with a quit app over the
course of a quit attempt in order to inform the design of future apps that are not only effective,
but also ones that smokers are motivated to use.
Popularity and effectiveness of smoking cessation apps
Smoking cessation mobile applications (quit apps) are popular tools for supporting quit
attempts
10,11
. For example, in a 2012 survey of smokers, 50% of respondents had previously
9
downloaded apps to quit smoking, 75% of whom actually used them to make a quit attempt
11
.
There is also a wide range of quit apps for smokers to choose from and within each app, a variety
of features are designed to provide support for quitting
12
. For instance, text-based cessation
interventions deliver messages that typically provide information about the health risks of
smoking, encouragement to continue quitting, and suggestions for coping strategies during
craving episodes
13–15
. Other mobile apps provide a suite of support features that users can access
on demand (without being prompted by the app), including games to distract users from craving,
self-monitoring features to record lapses, and advice on using medication for quitting
16,17
. In
addition to the aforementioned features, some quit apps also incorporate social networking
capabilities that connect smokers to friends and family or other smokers who can provide social
support during the quit attempt
18,19
. The popularity and variety of available quit apps makes
them a promising technology-based intervention for curbing smoking on a large scale.
With respect to their effectiveness for smoking cessation, there is limited but preliminary
evidence that at least some quit apps facilitate improved cessation outcomes. A meta-analysis of
text-based quit apps suggest that overall, such tools are positively associated with 6-month
cessation outcomes when compared to usual care
20
. While these findings are promising, it has
been noted that only a handful of quit apps are supported by scientific evidence, including those
that were reviewed in the aforementioned meta-analysis. In fact, a recent review of publicly
available quit apps from leading mobile app stores revealed that the majority were not
scientifically supported
21
. In addition, very few apps provided counseling for creating a quit
plan, recommendations for cessation medication, or referral to a quit line, which are considered
evidence-based clinical guidelines for treating nicotine dependence
10
. Given the widespread use
of quit apps among smokers and preliminary evidence for their effectiveness in facilitating
10
quitting, research is needed to identify the specific behavioral and psychological mechanisms
through which quit app usage influences cessation outcomes. Once such mechanisms are
identified, they can be leveraged to create more effective quit apps that are supported by
behavioral science.
App usage can help explain intervention effectiveness
The diverse ways that smokers interact with quit apps over time may help to uncover the
mechanisms through which quit apps influence cessation outcomes during quit attempts. This is
because how frequently or consistently one interacts with a quit app influences one’s exposure to
app contents intended to aid cessation. Take, for example, a 5-cigarette-per-day (CPD) smoker
who logs into her app consistently 5 times a day and is able to gradually decrease her smoking to
1 CPD after 4 weeks. In contrast, a 5 CPD smoker logs in once each day and is abstinent
throughout the entirety of a 4-week quit attempt. Consider yet another 5 CPD smoker who opens
his app only when experiencing craving and remains a 5 CPD smoker by the end of 4 weeks.
These examples illustrate a small sample of various usage patterns smokers could display during
a quit attempt along with possible cessation outcomes. But if individuals are not using the quit
app as intended, they may not be exposed to the behavior change contents in the way that
interventionists have designed
8
.
The ability to examine diverse usage patterns with digital behavior change interventions
is made increasingly easy with user log data. User logs are passively collected records of every
interaction a user has with the system and thus provide researchers with a rich dataset to examine
UE in relation to outcomes
22
. In depth knowledge about how smokers interact with cessation
apps would allow researchers to examine whether specific kinds of usage patterns, such as
11
frequency or consistency of log-ins, lead to quitting and can prompt further exploration of
potential reasons why. Using this knowledge, we can inform the development of strategies aimed
to promote specific usage patterns that empirically lead to more successful cessation for future
versions of MQU or similar cessation apps.
Defining user engagement: a construct with multiple dimensions
UE, which has been discussed extensively in human computer interaction research,
captures how users experience interaction with technology. Broadly, UE refers to the extent to
which users are captivated by and motivated to interact with a given technological system. The
experience of UE is typically described by attributes such as users’ selective attention and
emotional involvement during use of the technology
23,24
. Furthermore, UE is the complex and
dynamic relationship between a user and technological system and can be conceptualized with
respect to multiple dimensions
23,25
. The following sections introduce and define these
dimensions as they pertain to digital behavior change interventions specifically.
Behavioral UE dimension as usage
With respect to digital behavior change interventions such as quit apps, behavioral UE is
the most commonly assessed UE dimension. Behavioral UE reflects “usage” of the intervention
26
. Usage is typically measured using objective, passively collected data such as number on app
features clicked, number of log-ins, duration of interaction, and other such physical interactions
with the intervention
26
. Examples of common ways UE is operationalized in technology-
mediated behavior change research are included in Appendix A. Behavioral UE indicators, such
as login frequency or number of intervention sessions completed, allow us to examine usage
12
patterns with an app. Despite the ease and ubiquity of assessing behavioral UE with behavior
change apps via user interaction logs, behavioral UE on its own does not necessarily capture how
users respond to interaction as intended or whether they mentally process the intervention
content provided
27
. For example, individuals may click on several app features but not
necessarily actively read or incorporate the information into their behavior change process. Thus,
behavioral UE indicators should be viewed as proxies for the extent to which users are captivated
by or motivated to interact with the technology
24
.
Cognitive and Affective UE dimensions as subjective experience
Cognitive and affective dimensions of UE, on the other hand, relate to the psychological
experience of interacting with the technology. These two UE dimensions more closely align with
how UE is typically described in human computer interaction research
23
. Cognitive UE reflects
how focused or concentrated a user is during interaction and the extent to which they enjoy using
it. This dimension is typically assessed via measures of attention, interest, and intentions to
continue using the technology
24,28
. Methodologies for assessing cognitive dimensions of UE
include eye tracking and heart rate monitoring during use of the technology
24
. Affective UE
encompasses the positive or negative emotions and moods that occur during or as a result of
interaction and have been measured via self-reported emotions and facial tracking
25,29
. It is
hypothesized that affective response to interacting with technology is important for motivating a
user to continue interacting with the technology over time
23
. Together, cognitive and affective
dimensions of UE allow us to examine the subjective experience of UE, which is not readily
captured by behavioral UE indicators alone
25
.
13
Temporal UE dimension as long-term usage
Thus far, we have discussed three UE dimensions that reflect usage or subjective
experience during short-term, momentary episodes of interaction with a system. A fourth
dimension of UE relates to how users continue to choose to interact with the technology over
time during multiple interaction
episodes. Common ways in which this
is measured is the number of interactions
per unit of time, the number of
individuals still interacting with a
technology at specific time points, and
the time until which a user no longer
wishes to interact with a system
30
.
O’Brien and Toms proposed a conceptual framework (Figure1-1) to describe UE as a
process by which users continue to interact with technology over time. This framework was
developed based on interviews from users of different technologies, e.g., video games, online
search engines. Conceptualizing UE as a process implies the additional presence of a temporal
dimension of UE
23
. Their proposed user engagement cycle consists of 4 stages of engagement:
1) point of engagement, 2) period of engagement, 3) disengagement, and 4) re-engagement.
O’Brien and Toms additionally describe key cognitive and affective attributes associated with
each stage. Stages of the user engagement cycle and associated attributes are detailed below.
At the “point of engagement” stage, users are motivated to interact with technology in
order to accomplish a goal or task such as learning new information or to have an enjoyable
experience. If the technology holds the user’s interest and provides novelty, a “period of
Point of
engagment
Period of
engagement
Disengagement
Re-engagement
Figure 1-1. User engagement cycle
(O’Brien & Toms 2008)
14
engagement” occurs during which the user’s attention and interest in the technology is
maintained
23
. During this stage, users are concentrated on accomplishing their task and may be
so absorbed during interaction that they perceive time to pass by quickly
23,25
. Temporary
“disengagement” from the technology may occur for a number of reasons. These include
interruptions or distractions from the user’s physical environment or because aspects of the
technology are too difficult or burdensome to interact with
23
. Users may also make conscious
decisions to begin another activity, especially when they feel they have accomplished their initial
task. The “re-engagement stage” after disengagement may occur at a future time point and can
depend on factors such as users’ evaluations of positive past experiences, convenience, and
positive reinforcement for continued engagement
23
.
In summary, UE is a multidimensional and complex construct that captures what users
are doing, thinking, and feeling during interactions over time. Behavioral UE represents usage of
technology and cognitive and affective UE represent the subjective experience of interaction.
Furthermore, the temporal dimension represents UE as a dynamic process of repeated
engagement cycles occurring over time. Consideration of these various UE dimensions
collectively will provide a deeper understanding of how both usage and subjective experience of
quit apps leads to cessation outcomes. Analysis of UE as a multidimensional construct can help
to inform the design of quit apps that not only promote usage, but also cognitive and affective
processes that may be important for driving positive quitting outcomes.
15
Factors theorized to influence UE in behavior change interventions
In the previous section,
we briefly introduced
technology-related attributes
theorized to influence UE in
general (e.g., usability, novelty,
aesthetics)
23
. For digital
behavior change interventions
specifically, additional factors
associated with the behavior
change process itself may
additionally influence UE. Ritterband and colleagues have made efforts to summarize these
factors for interventions delivered via Internet, i.e. Internet Intervention model
5
. This model
outlines characteristics related to the user (e.g., disease state, demographics, cognitive
processing), their environmental contexts (e.g., other’s perceptions of the intervention tool), and
attributes of the intervention tool, which are theorized to influence UE. Intervention attributes
span a wide range of domains, including its appearance, the behavior change approaches used,
burden of using the technology, and extent to which the intervention is personalized to the user
to name a few
5
. Empirical studies are still needed to determine whether these hypothesized
factors influence UE for mobile apps as well, which are more readily accessible to the user
throughout the day compared to web-based interventions
31
.
Research studies exploring factors related to UE with mobile apps are emerging but are
currently limited. Interview data have shed light on the following factors that may influence UE:
Environment
User characteristics
Website Use
Mechanisms of
change
Behavior change
Symptom
improvement
Website characteristics
Appearance
Behavioral prescriptions
Burdens
Content
Delivery
Message
Participation
Assessment
Treatment
Maintenance
Figure 1-2. Internet intervention model
16
burden related to logging information, interest, convenience, boredom, and device fatigue
32–34
.
In addition, the extent to which a user perceives mobile apps to be useful for managing health
outcomes (perceived usefulness) has been shown to predict both intention to use and actual use
of mobile apps
35,36
. Identifying user-related characteristics that influence perceived usefulness
could help to inform the refinement of app design, including personally tailoring access to
specific features found to be most useful for certain individuals and contexts. Additional
quantitative and qualitative studies are needed to identify such factors, as they may be unique to
quit apps and the behavior change processes associated with smoking cessation.
User engagement vs. effective engagement
Perhaps of utmost importance to intervention researchers are the specific behavioral and
psychological mechanisms through which UE is associated with intended intervention effects
37
.
As discussed previously, UE, operationalized as usage of digital behavior change interventions,
has been described as a prerequisite for achieving intended intervention goals
8,30
. A common
assumption is that more frequent usage or interaction with intervention content (and therefore
more exposure) necessarily leads to positive intervention outcomes
38–40
. For example, Ubhi and
co-authors reported that individuals who opened their quit app (SmokeFree28) on more
occasions showed greater odds of self-reported 28-day abstinence
16
. Despite these findings, a
growing body of evidence suggests that more frequent usage of digital interventions is not
consistently associated with desired behavior change
22,40
. Research identifying the specific
intervention contexts and behavior change outcomes for which this relationship holds is still
nascent, especially as there is not currently an agreed upon definition of UE across behavior
change studies
22,26
.
17
One likely explanation for the inconsistent relationship between usage and behavior
change outcomes is that digital interventions such as quit apps typically contain multiple features
designed to deliver intervention content. For quit apps specifically, these features might include
providing advice for quitting smoking and positive reinforcement for avoiding lapse
12
. Of the
multiple app features available to users, it is possible that only specific features are positively
associated with cessation outcomes while others are not
41,42
. In addition, the most frequently
used app features may not necessarily be effective for changing behavior
26,42
. Heffner and
colleagues, for instance, reported that among the ten quit app features most frequently used, only
two were positively associated with cessation
42
. Therefore, operationalizing UE simply as the
sum of interactions with a given app potentially overlooks the unique behavioral and cognitive
mechanisms for quitting that are associated with each different app feature.
In contrast to the assumption that greater levels of UE (more frequent usage) necessarily
lead to intended behavior change outcomes, Yardley and colleagues have recently advocated for
identifying “sufficient engagement with the intervention to achieve intended outcomes”,
hereafter referred to as effective engagement
6
. Effective engagement can thus be considered a
subset of UE that mediates the effects of intervention on the intended behavior change.
Identifying effective engagement is of interest because individuals may not obtain additional
benefits to behavior change outcomes from usage beyond a certain optimal dosage or exposure to
intervention content
26,40
. Continuing to engage individuals who do not need support may also
represent a waste of resources and could lead to user annoyance and apathy toward behavior
change
18
.
Despite growing interest in identifying effective engagement with digital interventions,
there is currently limited guidance on how specifically one establishes “sufficient” engagement.
18
One approach advanced by Ainsworth and colleagues entailed identification of a minimum
threshold of intervention usage that was still predictive of positive behavior change outcomes
43
.
In their evaluation of a web-based hand hygiene intervention, for instance, Ainsworth et al. show
that one educational session was associated with the largest change in hygiene behavior and
subsequent sessions were associated with diminished improvement in outcomes. Although users
who completed all four intervention sessions showed the greatest change in hygiene behavior,
they concluded that one session was the minimum threshold for users to be exposed to key
aspects of the intervention
43
.
While Ainsworth and colleague’s approach helps to uncover usage that is “sufficient”, it
overlooks information about how users interpret intervention content and reasons for interaction
(or discontinuing interaction). In other words, their approach focuses on the behavioral
dimension of UE in relation to behavior change without consideration of cognitive and affective
processes that may further explain intervention effects. Consider an example where usage data
revealed that only certain app features drove behavior change and only up until a particular point
during the intervention period. It is not completely obvious whether observed behavior change
was solely attributable to app usage or because of other factors that are external to the app that
could have emerged during the intervention period. Understandably, Yardley and colleagues
advocate for mixed methods approaches that incorporate insight from focus groups, interviews,
and other qualitative data to explain objective usage data and what constitutes effective
engagement
6
.
19
Interacting with a diverse set of features and underlying usage patterns
Although analysis of interaction frequency with individual app features has yielded
insight into which specific features tend to drive intended intervention outcomes, less is known
about whether interaction with a diverse set of app features similarly predicts behavior change.
For instance, it is possible that the usage of one feature may enhance (or detract) the
effectiveness of other app features and thus the synergistic effect of both may be overlooked
37
.
Preliminary evidence suggests that investigating usage as combinations or patterns of diverse app
features may be a useful strategy for explaining intervention outcomes
37,40
. For instance, in a
web-based depression intervention, the number of diverse activities a user completed per log-in
was associated with improved outcomes, while total number of log-ins or modules completed
over the intervention period was not
40
. Another study demonstrated that the extent to which a
user interacts with all available intervention features (i.e., breadth), was associated with
increased fruit and vegetable consumption, whereas the total amount of time spent interacting
with features (i.e., duration) was not
22
. Both studies illustrate that exploring interaction with a
combination of intervention features may represent an alternative approach for characterizing UE
and may be helpful for elucidating effective engagement.
20
Overview of dissertation studies
The aforementioned literature alludes to research gaps that limit our understanding of the
diverse ways individuals interact with quit apps and the mechanisms through which app usage
drives cessation throughout the course of a quit attempt. Addressing these gaps would allow us to
make informed decisions about refining intervention content in future app versions that smokers
are motivated to use and are effective for smoking cessation. A summary of these research gaps
and how this dissertation addressed them is described below.
First, whereas UE represents a complex, multidimensional construct, it has been assessed
in behavior change research with static, user-level indicators of usage such as total number of
log-ins. Little is known about the subjective experience of interacting with a quit app during a
quit attempt and how psychological processes related to the cognitive and affective dimension of
UE unfold dynamically over time. To address this research gap, Study 1 used a mixed methods
approach to comprehensively describe cognitive, affective, behavioral, and temporal dimensions
of UE with MyQuit USC (MQU). This investigation was accomplished by triangulating
descriptive statistics of usage data with interviews of users’ subjective experience of interacting
with MQU.
Second, beyond sums of interaction with each individual app feature across an
intervention period, we have limited knowledge of the diverse usage patterns with a quit app that
users might display over time. An alternative approach to characterizing usage patterns is to
examine interaction with a combination of diverse app features on a micro-temporal scale (i.e.,
daily patterns). This novel approach may help to extend our understanding of how app usage
leads to intervention outcome. To address this, Study 2 explored latent classes of day-level UE
21
patterns, defined as sets of behavioral UE interactions that co-occur on a given day. This was
accomplished using multilevel latent class analysis with covariates.
The final research gap relates to understanding effective engagement with a quit app.
Although identifying effective engagement has been described as an important research focus for
digital behavior change interventions, little is known about how to empirically establish
sufficient levels of usage needed to mediate intended cessation outcomes. Study 3 addressed this
gap by examining whether daily UE patterns identified in Study 2 were associated with same day
lapse as one approach for exploring “effective engagement” with MQU. Interview data was
additionally examined to uncover cognitive and affective processes related to interaction that
help to explain the relationship between daily UE patterns and cessation outcomes.
Together, these three studies provided a multidimensional investigation of UE with a quit
app and demonstrated the utility of a multivariate approach to explore day-level UE patterns with
respect to cessation outcomes. Insight from these studies can be used to further refine the
delivery of intervention content (e.g., frequency, timing, novelty) in MQU for subsequent
versions. Furthermore, our findings on dynamic day-level UE patterns may help to inform the
development of quit apps that are responsive to smokers’ usage patterns and subjective UE
experience by dynamically employing strategies to promote re-engagement and effective
engagement across a quit attempt.
Specific aims and relevant hypotheses
Study 1: Mixed methods investigation of user engagement with a smoking cessation app
The overarching goal of Study 1 was to conduct in-depth analyses of multiple dimensions
of user engagement with our own cessation app, MyQuit USC “MQU”, using a mixed methods
22
approach. User interaction log data were used to visualize temporal trends in behavioral UE over
the quit attempt and interviews were used to explore from the users’ perspective cognitive,
affective, and behavioral UE dimensions. The specific aims of this study were:
Aim 1. To visually describe behavioral and temporal UE dimensions represented by user log
data for multiple MQU features over the period of a quit attempt.
Aim 2. To explore cognitive, affective, and temporal UE dimensions through thematic analysis
of interview data.
Aim 3. To triangulate quantitative and qualitative data to explore and identify potential factors
that influence temporal changes in UE dimensions, such as perceived usefulness.
Study 2: Identifying dynamic user engagement patterns with a quit app using multilevel latent
class analysis
The overall objective of Study 2 was to explore distinct latent subtypes of day-level UE
patterns using users’ daily interactions with multiple MQU features as latent class indicators in
multilevel latent class analysis. The specific aims and associated hypotheses were as follows:
Aim 1: To explore underlying subgroups of day-level UE patterns during a quit attempt using
MQU.
H1: There may be at least three day-level UE patterns: active, less active, and not active
patterns
Aim 2: To test the association between probability of membership in day-level UE patterns and
day in quit attempt.
23
H2: The probability of day belonging to an “active” relative to a “less active” day-level UE
pattern will decrease as a function of day in quit attempt.
Aim 3: To examine whether probability of membership in day-level UE patterns varies as a
function of loan phone use.
H3: The probability of a day belonging to an “active” relative to a “less active” day-level
UE pattern will be lower for users interacting with MQU via a loan phone.
Study 3: Investigation of effective engagement with a quit app and daily lapse during a quit
attempt using mixed methods
The overall objective of Study 3 was to identify effective engagement with respect to
cessation outcomes. To accomplish this, we extended multilevel latent class analyses (MLCA)
conducted in Study 2 to examine the association between day-level UE patterns and lapse on the
same day. This study additionally incorporated interview data in a mixed methods approach to
investigate psychological processes related to cognitive and affective UE dimensions that help to
explain how specific day-level UE patterns represented effective engagement.
Aim 1: To examine whether certain latent classes of day-level UE patterns associated with
reduced probability of same-day lapse, i.e., effective engagement patterns.
H1: Days characterized by a highly active pattern of UE with MQU (interaction with all
available MQU features), would be associated with reduced probability of same-day lapse,
compared to days characterized by a less active pattern of UE.
24
Aim 2: To describe the behavioral, cognitive, and affective mechanisms cued by UE that
facilitated users’ cessation outcome.
H2: Users would describe varied behavioral, cognitive, and affective mechanisms with
respect to interaction with different app features when describing how MQU helped them
avoid lapse.
MyQuit USC study design
Study procedure common to all dissertation studies
This section describes MyQuit USC (MQU), the smoking cessation app examined within
this dissertation. Data used for this dissertation were drawn from the parent MyQuit USC Study.
In the parent study, the overarching objective was to evaluate MQU, which is a mobile quit app
developed for Korean American young adult smokers aged 18-25 years. Participants interacted
with MQU for four weeks beginning on a self-specified quit date set at least seven days after
study enrollment. The primary intervention feature was delivery of implementation intention
reminders (II reminders), which is detailed further in the following section. II reminders were
delivered via push notifications to the user at least 5 times a day during hour-long blocks in
which participants pre-specified as “high risk smoking situation” (HRSS, i.e., potential contexts
for lapses). Following a microrandomized trial design
44
, the II reminders were delivered only
before 75% of the HRSS; no intervention is sent during the remaining 25% of HRSS.
Momentary lapse was assessed 15 minutes prior to the end of each 1-hour HRSS window. Day-
level predictors and outcomes were also assessed on MQU at the end of each day (between 8pm-
11pm) of the intervention.
25
The primary lapse outcome for this dissertation was measured via responses to ecological
momentary assessments (EMA) delivered at the end of each of the 28 days in the study. EMAs
are used to capture information about an individual and their behavior in real or near-real time
45,46
and has been used in previous smoking behavior studies to minimize recall bias and enhance
ecological validity for outcomes of interest
45
. Responses to EMAs and all interactions with the
app (e.g., accessing MQU features) were automatically recorded in user interaction logs detailing
the frequency and timing of interactions. These user logs were stored in a secure private server
and monitored by research assistants to enhance participant compliance to the protocol. This type
of data reflects behavioral UE interactions and is commonly used to explore how users interact
with technology-based interventions
22
. Aside from completing 80% of their momentary EMAs,
participants were encouraged to use MQU as much as they felt was necessary.
After the main four-week intervention, all participants were invited to participate in a
semi-structured interview lasting approximately 30-60 minutes to discuss their experience using
MQU. Attempts were made to record and transcribe all available interviews. The study was
approved by the IRB at University of Southern California and all participants provide informed
consent prior to enrollment in the study. An additional information sheet was provided to
participants for detailing the purpose of the interview and how data would be used.
Target user population
MQU was initially developed to curb smoking among Korean American (KA) smokers,
an ethnic group in which smoking rates are significantly higher relative to other Asian American
subgroups
47
. For instance, smoking prevalence has been documented to be as high as 30.0% to
36.7%
47,48
among KA men. High prevalence of this avoidable behavior leaves KA
26
disproportionately burdened by lung cancer
49
. KA emerging adults, those aged 18-25 years,
have been identified as a subpopulation that may benefit from smoking cessation efforts early in
their career. Emerging adulthood is described as a developmental period associated with high
rates of substance use, thought to be related to increased independence from parents and the
desire to explore individual social identities
50
. This particular group’s risk for initiating smoking
is compounded with cultural pressure to smoke, indicated by KAEA’s tendency to severely
overestimate smoking prevalence in the KA community
51
. Once KAEA progress to smoking
regularly, a number of cultural barriers hamper cessation efforts among KA; e.g., desires to
conform to social norms, the relationship between smoking and male gender identity, and the
critical role of smoking in social interactions
52
. For these reasons, KAEA are a priority target of
smoking cessation efforts. Due to challenges in enrollment, however, eligibility was opened to
all smokers who identified as Asian or Asian American starting in early 2018. All other
eligibility criteria remained the same.
Development of MQU
During development of MQU, there was little available guidance as to how to design
effective smoking cessation apps for specific cultural subpopulations (e.g., age, ethnicity).
Formative research grounded in qualitative and quantitative methods was conducted in 2015 to
inform the design and development of a quit app tailored specifically for a Korean American
young adult population
53,54
. Quantitative insight was drawn from a 7-day EMA study (N=78)
that compared social, psychological, and contextual factors associated with smoking events
relative to non- smoking events
55
.
27
The study revealed that smoking events were associated with the presence of Korean
American friends relative to all other social contexts, and socializing, commuting, and being
outside relative to all other activities and locations
55
. Qualitative data was collected via semi-
structured interviews (N=8) and a focus group (N=4) to elicit feedback on common smoking
cessation strategies, e.g., distraction or substitution of cigarettes, tracking and reduction of
behavior, social support for quitting
53
. An important finding was that preference for app design
and intervention features was heterogeneous across individuals and context dependent within
individuals. Specifically, respondents suggested that they might need different types of
intervention depending on the situation, i.e. time of day, location, and activity. Notably,
respondents were generally aware of each of the situations in which they were most likely to
smoke, hereafter referred to as high- risk smoking situations (HRSS)
54
Based on these findings, MQU was designed to provide personalized support during user-
specified HRSSs. To facilitate personalization of support, we elected implementation intentions
(IIs) as the primary intervention strategy. IIs are IF-THEN statements that specify when and how
an individual will respond to specific situations in order to achieve a behavioral goal, such as
smoking cessation
56
. Therefore, subjects are asked during app set-up to identify at least 9 HRSS
along with an appropriate, e.g. feasible, II to enact in order to avoid smoking during the HRSS
56
.
Although they are encouraged to input their own HRSS, MQU provides suggested HRSS derived
from formative research
54
. IIs have been used in less intensive, non-lab designs to successfully
reduce smoking and promote cessation in different populations
57,58
. In contrast to previous
interventions where subjects are only asked to develop IIs at baseline, MQU additionally pushes
a reminder of each user’s HRSS-specific II ten minutes before the HRSS is scheduled to occur.
28
These II reminders constitute the primary intervention component of MQU. Mobile
health intervention designs that deliver real-time, personally-adapted support to an individual in
a given context, e.g., HRSS, are known as just-in-time adaptive interventions
59
(JITAIs). MQU
is a JITAI app insofar as II reminders are delivered JIT before a user-specified HRSS occur.
Furthermore, each II reminder is adapted specifically for each given HRSS.
Description of the MyQuit USC (MQU) app features
Figure 1-3 illustrates the main MQU home and the
primary features that are accessible from this screen.
1. MyPlans: This feature allows users to input each of
their HRSS and pair it with a specific II they are willing
to enact during a given HRSS. After setting up each
HRSS with a corresponding II at the beginning of the
study, MyPlans remains accessible and can be edited
throughout the study.
2. MyCalendar: This feature allows users to schedule
approximately when each HRSS is likely to occur and functions similarly to other calendar based
apps. Prior to each scheduled HRSS, II reminders are pushed to the user 10 minutes before the
HRSS occurs (Figure 1-4). To reduce burden associated with scheduling each day, users generate
a “typical” default schedule separately for weekdays and weekends during setup. If MyCalender
is not updated schedule daily, default schedules are used to populate the calendar.
Figure 1-3. MQU home screen
29
3. MyProgress: This feature is a real-time tracker of
number of cigarettes smoked/resisted, money spent from
cigarettes smoked, money saved from cigarettes resisted,
and money earned from completing EMAs (see Figure 1-
5). Information is derived from momentary EMAs and
end of day EMAs throughout the intervention period.
Each user is free to access this information as often as
they choose.
4. MyCrave: "I want to smoke!": Participants can
immediately report unscheduled HRSS by pressing the "I
want to smoke" button. After selecting the HRSS they are experiencing, MQU provides “just in
time” support by displaying the specific II reminder paired with the given HRSS.
5. MySmoke: “I just smoked!”: Participants report lapses by pressing the “I just smoked” button
and are asked to report what they were doing at the time of lapse.
Figure 1-5. MyProgress
Instead of smoking while: “At
a bar with friends”, you said:
“stay inside and drink water”
Figure 1-4. II reminder
30
Chapter 2: Mixed methods investigation of user engagement with a smoking cessation app
Introduction
Usage, or interaction, represents a key mediator for how quit apps influence cessation
outcomes
30,60
. However, understanding of the diverse ways in which quit apps are used,
especially over the course of a quit attempt, remains limited. In behavior change research,
authors primarily report aggregated indicators of usage frequency (e.g., number of logins) as a
proxy for exposure to intervention content. While useful for assessing frequency of use,
aggregated measures such as these can ignore the multiple intervention features available to
users. Without differentiating between the specific app features users interacted with and the
context of these interactions, it is difficult to evaluate whether users were exposed to intervention
components as intended. Furthermore, if positive cessation outcomes were achieved, it would not
be possible to make strong conclusions about which specific app features facilitated lapse
avoidance and the cognitive and affective processes through which those features drove cessation
25,61
. To make informed design decisions for apps that provide the level of support a given user
needs over time, we need nuanced investigation of the diverse ways in which users interact and
experience various app features throughout a quit period.
18
.
User engagement as usage and as subjective experience
One approach to characterizing how individuals interact with smoking cessation apps is
to examine user engagement (UE)
25
. UE is a multidimensional construct encompassing
behavioral, cognitive, and affective aspects of a user’s experience of interacting with the
technological system
25
. With respect to digital behavior change interventions such as quit apps,
31
behavioral UE is most commonly defined as usage of the intervention. Usage is assessed via
objective, passively collected data such as frequency of opening an app and interacting with
content
26
. Despite the relative ease and ubiquity of collecting behavioral UE data via
automatically generated user interaction logs, behavioral indicators like usage by itself do not
capture whether users absorb or understand intervention content
27
.
Cognitive and affective dimensions of UE, on the other hand, describe how users
experience and react to various aspects of the technological system. Cognitive UE includes
mental processes associated with interaction such as the level of focused attention or
concentration that a user demonstrates with a system. Indicators of cognitive UE are typically
assessed via measures of attention, focus, and intentions to continue using the technology
28
.
Affective UE encompasses the positive or negative emotions and moods that occur during or as a
result of interaction
25,29
. Affective responses are considered to be important for motivating a
user to continue engagement over time
23
. Together, cognitive and affective dimensions of UE
tap into the subjective experience of UE not readily captured by behavioral UE
25
.
Dynamic user engagement as a process over time
Thus far, we have described UE as usage or the subjective experience individuals have
with technology without explicitly mentioning time. An additional dimension of UE relates to
engagement as long-term interaction, comprised of repeated interactions occurring over time
23
25,30
. O’Brien and Toms proposed that these repeated interactions can be characterized by an
engagement cycle that includes multiple stages and associated attributes. During the point of
engagement, users initiate interaction with the technology to accomplish a particular task or goal.
The point of engagement gives way to a period of engagement if the technology holds users’
32
interest and is novel, i.e. provides unexpected information. Users subsequently disengage, or
“sign off”, from the system either because of external forces such as being interrupted by other
activities or because of poor usability. Alternatively, disengagement may also occur because of
internal factors. For example, users may make conscious decisions to disengage because their
initial goal could not be completed during a single interaction or because they have already
accomplished it.
Finally, users may or may not re-engage with the technology at a future time point. A
number of factors have been purported to influence the likelihood of re-engagement, including
positive past experiences with interaction, convenience, and incentives such as rewards for
engagement
23
. In addition, others have suggested that whether engagement is initiated may
depend on whether the technology is perceived to be useful for performing a given task
62
. For
example, when applied to digital behavior change interventions, perceived usefulness refers to
the extent to which users believe the tool makes their behavior change goal easier to accomplish
35,36
. Available studies suggest that perceived usefulness of a technology for performing a task is
positively associated with users’ intention to initiate engagement
36
. Furthermore, users will
continue to engage with technology if it is perceived to be useful for accomplishing their
behavior change goal
25
. In summary, UE is a multidimensional and dynamic process of
interaction between a user and digital behavior change intervention that is influenced by
intervention attributes and user characteristics.
Existing literature on UE in behavior change interventions
Despite the multidimensional nature of UE, behavior change researchers have tended to
focus on behavioral UE via user-level measures of usage
63
. Although examples of behavioral
33
UE measurement are more extensive for web-based digital behavior change interventions,
behavioral UE with smartphone-based interventions is measured in similar ways. Common user-
level metrics for usage include login frequency and duration averaged over time, number of
modules used per session, and number of self-monitoring data inputted
64–66
. These measures
reflect the breadth (i.e., extent of interaction with all available content) and depth of intervention
usage (i.e., total usage minutes)
22
.
Existing studies describing temporal UE are limited. A few studies have reported trends
in behavioral UE using repeated measures of usage across the intervention period (e.g., number
of interactions and proportion of people still using the intervention at different time points)
60,67,68
. Although these studies assessed behavioral UE as a function of time, temporal trends in
UE are still averaged across users (user-level). Less attention has been paid to UE at the within
users over time. For example, users may display greater levels of usage (more frequent or more
breadth of interaction) at one time point during an intervention period and lower levels of usage
at another time point
60
. Consequently, current research fails to represent the cyclical nature of
UE within users over time
23
. By additionally analyzing repeated measures of UE sessions within
users, we are better able to explore potential changes and/or fluctuations in usage and experience
longitudinally and reasons why.
Research on the subjective experience of interaction is lacking. Simply viewing
intervention content or being exposed to content may not be enough to change behavior.
Therefore, psychological processes associated with cognitive and affective dimensions of UE are
important aspects of interaction with digital behavior change interventions that are often
hypothesized to influence behavior change. Unfortunately, indicators of focused attention or
positive emotion are often not feasible to measure repeatedly in real time using traditional human
34
computer interaction methods (e.g., eye tracking, physiological reactions)
25
. Even though
cognitive and affective measures of UE are less frequently measured in behavior change
research, motivational and emotional drivers of behavior change cued by app usage can and
should be explored retrospectively, using semi-structured interviews and other self report
methods
6,18
.
Qualitative investigations of UE with quit apps
Given the multidimensional nature of UE, it has been recommended that qualitative
methods be used to achieve in-depth understanding of how users interact with and respond to
digital behavior change interventions
6,26,61
. Qualitative data, such as those obtained from
interviews, are especially useful for assessing the subjective experience of engagement, i.e.,
cognitive and affective UE dimensions, that behavioral UE alone cannot. Although some studies
have incorporated interview data in their investigation of UE with mobile health apps across a
wide range of health behaviors
34,69–72
, only few have focused specifically on UE with quit apps
15,17,18
.
For example, Smith and colleagues drew from interview data to examine UE with their
smoking cessation app, NewLeaf
18
, which provides a social networking feature for users to
exchange stories about their experiences with quitting as well as tips for quitting and
opportunities for distraction from during cravings
18
. The authors described a variety of cognitive
and affective responses that users experienced with NewLeaf and how those responses changed
over the course of the quit attempt. But despite their in-depth analysis, NewLeaf may not be
generalizable to how smokers might engage with other quit apps that provide different features
for cessation support (e.g., providing information resources, providing rewards or positive
35
reinforcement)
12
. Different types of cessation support and even the design of the app may
engender a varied range of subjective experiences of UE
15
. Therefore, in order to gain additional
insight on the cognitive and affective responses that characterize UE and reasons for interacting
with quit apps in particular ways over time, additional qualitative investigations of UE with quit
apps are needed.
Study 1: Specific Aims
There is limited research on the usage patterns involving diverse app features, the factors
that influence usage, and the subjective experiences smokers have while interacting with various
quit app features over time. Thus, the overarching goal of the present study was to conduct in-
depth analyses of behavioral, cognitive, affective, and temporal UE dimensions with a cessation
app, MyQuit USC “MQU”. We accomplish this by triangulating data from user interaction logs
for various MQU features (quantitative data) with semi-structured interviews (qualitative data)
using a mixed methods approach. User logs enabled us to examine behavioral UE and semi-
structured interviews allowed us to explore cognitive and affective UE dimensions.
Both data sources allowed us to examine the temporal dimension of UE. Specifically,
user logs were used to visualize temporal trends in usage over the 4-week quit attempt and
interviews were analyzed to explore cognitions and emotions that influenced observed trends in
usage from the users’ perspectives. In summary, our mixed methods approach allowed us to
investigate 1) smokers’ usage of diverse MQU features and 2) the subjective experience of those
interactions. The specific aims of this study were:
Aim 1. To visually describe behavioral and temporal UE dimensions represented by user log
data over the period of a quit attempt.
36
Aim 2. To explore cognitive, affective, and temporal UE dimensions through thematic
analysis of interview data.
Aim 3. To triangulate quantitative and qualitative data to explore and identify potential
factors that influence temporal changes in UE dimensions, such as perceived usefulness.
Description of MyQuit USC and points of engagement associated with app features
This study focused on users’ experience with MyQuit USC (MQU), a 4-week just-in-time
smoking cessation mobile app designed to provide support during self-specified moments when
smokers needed it the most
73
, i.e., hours containing self-identified high risk smoking situations
(HRSS). At moments leading up to self-identified HRSS, users were provided with reminders of
self-specified implementation intentions (IIs), or action plans, that they were willing to enact
during HRSS to avoid smoking
58
. MQU was developed based on formative mixed method
research with the target population
74
. Users could access six features via the home screen
(Figure 2-1). Interactions with these specific features were the focus of this study and are
described below.
1. Acknowledge implementation intention (II) reminders: Ten minutes prior to the start of a
randomly selected HRSS (75% of all HRSS) users received a personalized reminder to enact an
II that users compiled at baseline. This within-person, microrandomized design, where an
individual repeatedly receives both intervention and control conditions, has been proposed as one
approach for evaluating mobile health interventions
75
. Figure 2-2 presents an example of an II
reminder for smoking during the HRSS: “at a bar with friends”. These plans are set using the My
Plans button at baseline and could be updated throughout the 4-week study period.
37
2. Momentary ecological momentary assessments (EMAs): Forty-five minutes after the start of
all HRSSs indicated, EMAs were delivered and asked the user whether they enacted the II and
whether they lapsed during the HRSS.
3. End of day (EOD) EMAs: At the end of each day, a summary EMA was administered and
participants were asked to report the total number of cigarettes smoked that day, along with
psychological and emotional outcomes.
4. Report lapse: Participants were asked to report lapses in real-time and to specify the specific
HRSS they experienced using an “I just smoked!” button.
5. Report craving: Participants were asked to report instances of craving to smoke using the "I
want to smoke!" button. Upon clicking this button, they were prompted to specify the HRSS they
were experiencing, after which they received the corresponding II reminder.
6. Check progress: At any point during the quit attempt, participants were able to access “My
Progress”, which displayed a graph of the number of cigarettes smoked/resisted (derived from
EMAs and “I just smoked” interactions), the amount of money saved/spent, or the amount of
money earned by responding to EMAs in the past week (Figure 2-3).
Features 1-3 can be described as PUSH (i.e., “pushed” by the system) feature
interactions, which are initiated by MQU at a predetermined schedule. In contrast, features 4-6
are considered PULL feature (i.e., “pulled” by the users) interactions and are initiated by the user
as needed. Such a distinction has been used in previous research on a quit smoking app
76
and is
likely to influence UE influence how frequently these interactions occur
77
. For example, PULL
features would require individuals to be aware of their cravings and lapses and motivated to
check their progress for a point of engagement to occur
2,44
.
38
Figure 2-1. MQU home screen
Figure 2-3. My Progress screen
Instead of smoking while: “At
a bar with friends”, you said:
“stay inside and drink water”
Figure 2-2. II reminder
39
Methods
Sample population
The sample for the present study consisted of self-identifying Asian American young
adults (i.e., 18-25 years). Due to challenges with recruitment, the initial eligibility criterion for
the sample was expanded from Korean Americans to Asian Americans broadly. A total of 57
smokers completed the intervention. Two participants ended 1 week early because of pre-
arranged international travel schedules. Recruitment activities included a combination of social
media posts, word of mouth, and flyer distribution. Additional enrollment criteria included
smoking on at least 3 days in a week and a willingness to make a quit attempt.
Study Protocol
MQU participants completed a 4-week quit attempt in which they were free to interact
with MQU. MQU was developed to operate only on Android devices. Individuals could use their
personal device if they had an Android or were given a study “loan” phone for the study period.
Participants were provided instructions on how to use the Android during phone setup at baseline
and were asked to keep the phone charged and accessible to them throughout the day.
Participants were encouraged to respond to at least 4 momentary EMAs daily along with
the end of day EMA. Otherwise, they could use the remaining features as needed. We note here
that users were compensated for participation in the study based on the number of momentary
EMAs completed, but not by their success in avoiding lapses. Specifically, to receive full
compensation on a given day, users needed to respond to at least 4 momentary EMAs. Each
interaction (e.g., log-ins, EMA completion, checking progress) with MQU was automatically
recorded and time-stamped on user logs in a web-based user management platform. The study
40
team had access to these user logs to monitor and troubleshoot issues that arose with MQU
throughout the study. User logs were updated automatically when participants’ mobile devices
were connected to either their phone service or Wi-Fi.
At the conclusion of the four-week study period, all 57 participants were invited to
participate in semi-structured interviews lasting approximately 30-60 minutes each. All
interviews took place within a month after participants’ last day in the quit attempt. We were
unable to contact one participant for participation in the interview. Interview questions focused
on how users interacted with MQU features and how they responded to and perceived the MQU
app (Appendix B). Interviews were audio recorded and transcribed verbatim by the research
team.
Quantitative Analysis
User logs of interaction with MQU, representing behavioral UE, were examined
descriptively using means and standard deviations of the number of daily interactions across
users for each MQU feature. To explore longitudinal trends in usage, longitudinal plots of the
percentage of users interacting with each particular MQU feature on each day in the quit attempt
were generated and summarized. Individual behavioral UE plots were also created for each user
with data visualization software (Tableau, Seattle, WA).
Subsequently, we triangulated our quantitative and qualitative data in three ways
78
. First,
interview themes were used to explain linear trends or inflection points in the behavioral UE
plots. Second, key quotes were linked to a subsample of users’ own behavioral UE plots to
understand the extent to which subjective experience of UE was reflected in passively collected
41
quantitative data. Finally, UE covariates that emerged in interviews were used as grouping
variables to explore potential differences in behavioral UE trends (e.g., loan phone use).
Qualitative Analysis
An inductive approach consisting of open coding and memoing was performed on the
first seven interviews by CC and KC since little pre-existing theory regarding UE with quit apps
exist
79
. These initial open codes captured the users’ perceptions of MQU features and
social/contextual factors they thought influenced how they interacted with it. Open codes were
then refined and organized according to deductive codes corresponding to the predetermined
dimensions of UE: cognitive, affective, and behavioral. An additional set of deductive codes
corresponding to each MQU feature was coded along with the UE dimension code to specify
which feature the user was referring to when describing their interaction.
The first author generated an initial codebook, which was applied to subsequent
interviews. Inter-rater reliability between two members of the trained research staff, CC and WF,
was then assessed for a random subsample of 6 additional interviews. After inconsistent coding
was discussed and the codebook was refined, all remaining interviews were coded by CC and
WF. Transcript coding was an iterative process. As new concepts emerged in subsequent
interviews, additional codes were created and applied to all interviews. Thematic analysis was
used to identify and analyze patterns or themes in the data
80
using Atlas.ti v7.5. Similar
codes/quotes that represented patterned responses across interviews were grouped together to
form coherent themes reflecting meaningful UE experiences
80
.
42
Results
Descriptive statistics for user characteristics and behavioral UE from user log data
Participants were 44 males and 13 females with mean age 22 years and mean nicotine
dependence score (Fagerström Test for Nicotine Dependence)
81
of 2.56 (Table 2-1). All
expressed interest in using MQU to quit or reduce smoking and reported smoking cigarettes daily
at baseline. Approximately half of users reported at least one lapse during their EOD EMA on
more than 50% of days in the 28-day study period, illustrating the difficulty of lapse avoidance
among our participants.
Table 2-1. User-level characteristics (N=57)
N (%); Mean (SD)
Male 44 (77)
Age 21.72 (.28)
Loan Phone (Android) 41 (72)
Nicotine Dependence (FTND) 2.56 (.27)
Lapsed on <50% of 28 days 29 (51)
Table 2-2 presents the mean number of interactions per day with each MQU feature
across the quit attempt. The two most frequently occurring interactions on average were with
PUSH features: acknowledging II reminders and completing momentary EMAs. Users
acknowledged an average of 2.3 II reminders per day and answered an average of 3.3 momentary
EMAs per day. Only one EOD EMA was available each day and so the mean value presented
indicates that EOD EMAs were completed on 68% of the 28 days in the quit attempt. PULL
features interactions all have mean values less than 1, indicating that users did not interact with
these specific features on a daily basis.
43
Table 2-2. Number of interactions per day with each MQU feature
MQU Features Mean (SD)
PUSH features
Acknowledge II reminders 2.27 (.04)
End of day EMA 0.68 (.01)
Momentary EMA 3.34 (.05)
PULL features
Report craving 0.34 (.02)
Report lapse 0.32 (.02)
Check progress 0.92 (.06)
In contrast to the static representation of behavioral UE frequency presented in Table 2-2,
Figure 2-4 presents longitudinal behavioral UE (usage) plots for each of the 6 MQU features
using data from the user logs. To graphically display temporal trends in behavioral UE, we first
calculated the percentage of users interacting with a given MQU feature at least once on a given
day. Percentages were then plotted for each of the 28 days and separated by PUSH and PULL
features for ease of interpretation.
The behavioral UE plots suggest that there is day to day variation in the percentage of
users interacting each feature. For instance, on the 6
th
day of the quit attempt, about 50% of
individuals completed the EOD EMA (orange line). On the 7
th
and 8
th
day, 70% and 75% of
users, respectively, interacted with the EOD EMA before declining again on subsequent days.
The difference in percentage corresponds to about 10 users. Similar day to day variation can be
observed for other MQU features. Together, these two plots provide a representation of the user
engagement cycle by indicating the extent to which subsets of users dis-engaged and re-engaged
with MQU each day over the course of the 4-week quit attempt.
44
Figure 2-4. Percentage of users interacting with each MQU feature over 28 days (N=57)
PUSH features: EOD EMA, Momentary EMA, Acknowledge II reminders
PULL features: Check progress, report lapse, report craving
To simplify data visualization, the daily percentages presented in Figure 2-4 were then
averaged within each of the 4 weeks in the quit attempt and plotted on the Y-axis (Figure 2-5).
Percentages for weeks 1 and 4 are labeled for convenience of interpretation. For example, the
average daily percentage of users interacting with the EOD EMA was 62% in the first week and
61% in the last week.
0 2 4 6 8 1 0 12 14 16 18 20 22 2 4 26 28 30
D a y i n q u i t at t e m p t
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
1 00%
% o f u s e r s i n t e r a c t i n g
0 2 4 6 8 10 12 1 4 16 18 20 2 2 2 4 26 2 8 30
D a y i n q u i t at t e m p t
0 %
10 %
20 %
30 %
40 %
50 %
60 %
70 %
80 %
90 %
100 %
% of u s e r s i n t e r ac t i n g
45
Figure 2-5. Percentage of users interacting with each MQU feature over 4 weeks (N=57)
As can be seen in figure 2-5, the percentage of users interacting with PUSH features,
interactions that were MQU-initiatied, appeared relatively stable across weeks (orange, green,
and purple lines), i.e. zero slope. These specific features were answering momentary and EOD
EMAs and acknowledging II reminders. The temporal trends indicate that users, on average,
were continuing to re-engage with these particular MQU features over time. The percentage of
users interacting with PULL features on the other hand, appeared to decrease steadily over time
(red, light blue, and blue lines), i.e., negative slope. These PULL features were reporting lapses
or cravings and checking one’s progress. That the percentage of of users interacting with these
features decreased over time indicates that some users were disengaging without re-engaging at
future time points.
0 1 2 3 4 5
We e k i n q u i t a t t e m p t
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
% u s e r s i n t e r ac t i n g
3 9%
3 0%
6 2%
6 1 %
2 5%
1 1 %
3 5 %
4 7 %
2 1 %
4 4%
PUSH
features
PULL
features
46
Themes from interview data used to explain trends in user log data
Of the 57 interviews attempted, 53 were successfully recorded and transcribed verbatim.
Field notes were provided for 2 participants for whom recordings were not available, one due to
technological difficulty and the other due to a request not to be recorded. One participant could
not be contacted for an interview and the audio file of one interview was lost due to
technological issues. Inter-rater reliability for interviews ranged from .83 to .94 and were
considered acceptable given a .80 minimum threshold for strong agreement
82
. Four themes for
UE were identified and are described below: 1) Cognitive UE dimension 2) Affective UE
dimension 3) Temporal changes in UE dimensions and 4) Barriers and facilitators of UE.
Theme 1: Cognitive UE dimension
This theme describes the thoughts and beliefs associated with interaction during periods
of engagement. About a third of the quotes in this theme concerned II reminders and the thoughts
that followed. As intended, II reminders prompted users to think about their HRSS-specific II.
However, for many, the reminders also brought to mind more broad motivations for quitting
smoking, regardless of the reminder message. For example, 15 users characterized II reminders
as “constant” or “general” reminders not to smoke as described by U53 below. Interpreting II
reminders as general reminders typically occurred when users were not enacting or were not able
to enact associated IIs. Beyond cueing momentary thoughts, II reminders also had longer-lasting
cognitive impacts. Users described how the accumulation of reminders led to heightened and
persistent awareness of their intention to quit.
47
“I wouldn’t actually do exactly what it says or that it worked out as a plan, but used it as a
reminder to not smoke. It doesn’t have to be exactly what you wrote down or your plan. Most of
the time it was just me standing there, inhaling secondhand smoke but not smoking” –U53, M
Surprisingly, some users anthropomorphized the II reminders; ten users compared the
reminders to their mother, a friend, or a “little buddy” who “nagged” them to quit. Even the
physical presence of the loan phone and having to interact with something called MyQuit was
enough to remind some users of quitting.
“You know you’re doing something that’s called MyQuit, you know, quitting cigarettes. It’s just
the idea; you’re just constantly reminded. […] Whenever I pick up that phone, oh it’s [about]
quitting.” –U18, M
While rarely mentioned, at least 5 users found that II reminders occasionally cued the
urge to smoke, an unintended but perhaps unsurprising consequence of alerts occurring more
frequently than HRSSs actually happening.
“I wasn’t a frequent smoker, I didn’t smoke everyday, so it was a lot of notifications that I
wouldn’t be reminded of smoking otherwise.” -U46, F
Checking one’s progress and responding to EMAs made users more “aware” and
“cognizant” of their smoking habits and prevented users from “going through the motions” of
48
smoking. While unintended in their design, responding to EMAs at times triggered a cascade of
self-reflective thoughts about one’s smoking frequency, situational cues, and intentions to quit.
“It got me to think more often throughout the day, like hey, this is something I want for me, it’s a
personal goal that I want to quit smoking, and it got me to think more of like why, who I
surround myself with and the environment I put myself in to smoke and if I even enjoy smoking.”
–U46, F
“Because of the app, I’m more conscious, ‘Okay, why am I smoking this? Do I have to smoke a
cigarette?’ I keep asking myself questions, ‘Why are you smoking? You know this is bad for
you.’” –U50, M
Another third of quotes in this theme were related to MQU’s perceived usefulness for
helping users to avoid lapse. Users frequently evaluated MQU’s usefulness for quitting in
relation to increases in their awareness of their smoking habits, their quit progress thus far, and
whether they were reminded of an II or motivation to quit.
Theme 2: Affective UE dimension
This theme captures emotions associated with interaction with MQU features. Compared
to cognitions, this theme was discussed less frequently. Over half of the quotes were related to
EMAs and II reminders, since these were the most frequently occurring interactions. Despite
their perceived usefulness, II reminders and EMAs often evoked “annoyance”. Reasons included
being alerted during times the scheduled HRSS did not occur (erroneous timing of II reminders),
49
the frequency of the alerts, and the perceived pressure to respond in a certain time frame (15
minutes for EMAs).
“Sometimes it was kind of annoying cause it would alert when I already know that I don’t have a
pack of cigarettes on me.” -U42, M
“It got annoying because when the survey popped up and you had to do it in a time limit it got
kind of annoying. I didn’t really want to interact with the app that much.” –U17, M
Despite some feeling annoyed, users, on average, continued to re-engage with these
PUSH features over time. As noted in Figure 2-5, the percentage of users interacting with PUSH
features was fairly consistent across days in the quit attempt. For at least one user, the frequency
of PUSH interactions, such as EOD EMAs, may have even supported long-term engagement.
U36 described how II reminders allowed him to become “engrained” in the app, despite finding
them annoying. His quote further suggests that his perception of the II reminders’ usefulness
may have offset his annoyance, which could have potentially resulted in disengagement with this
feature otherwise.
“The reminders at the end of the study were the biggest things that helped me to limit my
cigarettes. So I’m not going to say the reminders were bad, but they were so annoying and so
much to do, so many questions to do, that it kind of makes you engrained in the app” –U36, M
50
A few users described how guilt influenced whether one re-engaged with particular
features. For example, U6 describes how guilt associated with reporting lapses and cravings
suppressed his willingness to interact with MQU. Positive affect such as “feeling good” was also
described and was primarily limited to checking one’s progress. Users who were able to avoid
lapsing, such as U42, felt proud after viewing their quit progress or the amount of money they
had saved thus far.
“It’s […] guilt right, when you smoke, you don’t think “Man I got to check this, when I’m done
with this”. You don’t think like that, you just want to forget about it, unless it buzzes. So I think
that’s why I didn’t use the “I just smoked!” button a lot.” –U6, M
“When you saw those low bars, how did you feel…what went through your mind?” - Interviewer
“I did feel proud because I went from smoking five or seven a day to smoking one or two or more
than that or sometimes zero […] So it really felt good.” -U42, M
Theme 3: Temporal changes in UE dimensions over the MQU-assisted quit attempt
(triangulation with behavioral UE data from user logs)
This theme describes the extent to which cognitions and affect related to interaction
changed within the quit attempt and how such changes related to behavioral UE. The key quotes
described within this theme correspond to 5 users and were triangulated with their own
behavioral UE plots. These 5 users were selected for illustrative purposes because they explicitly
mentioned specific time points (weeks) in the quit attempt when describing their cognitions and
affect. Importantly, these individual user plots (Figure 6 a-e) illustrated the extent to which users’
51
subjective experience with MQU was reflected in user logs of behavioral UE. Similar to Table 2-
2, these plots represent the number of interactions per day for each of the six MQU features over
the 4-weeks quit attempt.
A notable finding was that for some users, the 2
nd
and 3
rd
weeks in the quit attempt were
key time points during which changes in thoughts and feelings associated with interaction
occurred. Changes in perceived novelty of MQU were related to the repetitive nature of the II
reminder messages and EMA questions as the quit attempt went on. Users described how the II
reminders’ waning novelty led them to acknowledge II reminders without actually reading the
content, as they could already anticipate what the reminder message would be. U27’s quote
below is consistent with a decline in EOD EMA interaction (orange line) following the 3
rd
week.
U6’s quote is consistent with a sharp decline in acknowledging II reminders (purple line)
beginning in week 2.
“I would say the first 2 weeks, I was attendant
to the app and got into the habit of it, but then
the 3rd and 4th week I kind of got tired of it
because it was making me answer so many
questions” –U27, M
Figure 2-6a: U27
1 2 3 4
We e k i n q u i t at t e m p t
0
1
2
3
4
5
A ve r a ge # o f p e r d a y
U 27
Average # of interactions per day
52
“I should have changed the plan after 3
weeks or something. The more I did it, the
more I get senseless. […]. So maybe
participants are encouraged to change or
adjust their plans after second week or
something.” –U6, M
Figure 2-6b: U6
One cognitive factor that likely contributed to changes in behavioral UE was users’
strengthening or weakening motivation to quit. Some users were discouraged by lack of progress
recorded in MyProgress and others felt EMAs became too burdensome, resulting in decreased
motivation to continue interacting with MQU. In contrast, at least 5 users described increased
app usage in relation to renewed or latent motivation to quit and finally feeling ready to cut down
after two weeks. For example, U21 described increased motivation to quit after the 2
nd
week,
which is consistent with sharp increases in number of PUSH interactions beginning at the 3
rd
week (orange and purple lines). U36 described how the accumulation of II reminders over the
first two weeks increased his motivation to quit. This increased motivation to quit is reflected in
an increase in the number of progress checking (PULL feature, red line) during the second half
of the quit attempt. In addition, U36 markedly increased the average number of II reminders he
acknowledged (purple line) during the final week from previous weeks. For these two users at
least, it is plausible that increased in motivation to quit were associated with increased usage
with multiple MQU features.
1 2 3 4
We e k i n q u i t at t e m p t
0
1
2
3
4
5
A ve r age # of i n t e r ac t i on s p e r d ay
U 6
53
“In the first 2 weeks, I was just like whatever,
it’s just an app, but then after it was like you
know what, let’s see if this really works, let’s
see how helpful this will be. So I attempted to
try a little bit harder.” –U21, M
Figure 2-6c: U21
“At the end of the two weeks, it was engrained
in my head that I wanted to quit […] all these
points kind of came together, all these
reminders of quitting made me want to quit.” –
U36, M
Figure 2-6d: U36
Finally, behavioral UE trends may have been associated with reductions in perceived
usefulness of MQU for quitting. Users described time points during the quit attempt in which
they “didn’t need the app” any longer. For example, users felt they no longer needed II reminders
when they were either able to enact them without prompt or as they reduced smoking. Once
users got into the habit of monitoring their smoking themselves, they quickly reduced interaction
with MyProgress. U54’s behavioral UE plot, for instance, displays declines in EOD EMA
1 2 3 4
We e k i n q u i t at t e m p t
0
1
2
3
4
5
A ve r age # of p e r d ay
U 21
1 2 3 4
We e k i n q u i t a t t e m p t
0
1
2
3
4
5
A v e r a ge # of p e r d ay
U 36
Average # of interactions per day Average # of interactions per day
54
interaction (orange), checking progress (red), and acknowledging II reminders (purple) during
the midpoint of the quit attempt.
“By the second week, when I had that moment,
I’m just going to quit, that’s when I started
using it, not less, but more like I didn’t need to
use it […] didn’t want to smoke anymore. At
that point, I didn’t need the app anymore.” –
U54, M
Figure 2-6e: U54
“After probably the 3rd week, the last week I was pretty much done with the app. I did it for […]
whatever I had to finish. But I didn’t even consider myself to be smoking that much during that
time. […] At that point I became less attentive that last week. I don’t want to say the third week
because I was still trying to use the app here and there. But definitely the last week I was
definitely paying very little attention to the app.” –U36, M
While U54’s behavioral UE plot appears to corroborate his perceived reduction in “need”
for MQU during the 2
nd
week, U36’s quote does not correspond well with his behavioral UE plot
(shown in Figure 2-6d). Whereas U36 described “paying very little attention to the app” during
the last week in his quote, his average number of daily interactions in the last week remained
similar to that of the previous week for EOD EMAs and checking progress. His interaction even
increased in the last week for acknowledging II reminders (purple line). This finding illustrates
1 2 3 4
We e k i n q u i t at t e m p t
0
1
2
3
4
5
A ve r a ge # of p e r d ay
U 54
Average # of interactions per day
55
an instance where behavioral UE alone does not provide a full picture of how users interact with
MQU and how interview data can help to enrich our interpretation of temporal trends in usage. A
likely explanation for the discordance between U36’s behavioral UE plot and subjective
experience is that he started to ignore II reminders messages despite clicking to acknowledge
them. This interpretation is consistent with what other users have described about acknowledging
II reminders without mentally processing them.
Theme 4: Barriers and facilitators of behavioral UE
This theme captures MQU-related factors along with social and environmental
constraints that may have limited re-engagement after disengagement. The most commonly
reported barriers to usage were app malfunctions, which anecdotally tended to occur in the loan
phones. Technical difficulties ranged from poor battery life to EMAs closing out before survey
completion. Others noted that MyProgress graphs were difficult to understand and were at times
perceived as inaccurate, which de-incentivized future re-engagement. Among loan phone users,
forgetting the phone/charger at home either by mistake or for fear of losing it presented
additional barriers. Those who remembered to bring the phone with them occasionally left it in
their bags and could not hear alerts.
“If I forget to bring the charger, then usually by the time the end of the day survey came up, the
phone would be dead.” – U11, M
Another set of barriers was related to momentary availability to interact with features,
especially PUSH ones. Delivery of momentary and EOD EMAs did not always match perfectly
56
with users availability or willingness to interact, especially during the school year. Users
reported being too busy to respond. Others noted social barriers to PUSH feature interactions
such as drinking with friends, but shared that they were more attentive when they were at home
or by themselves.
“When I drink, to be honest, I just completely ignored the app. […] When I’m out with my
friends I’d be drinking with them and I don’t want to be doing a survey while they’re all – I don’t
want to seem rude.” -U21, M
“When the surveys come up, you’re too busy talking to your friends so yeah definitely when I go
outside, less of a usage of the app. But then at school, on break or something, I’d always be
checking for surveys.” –U15, M
Users described several facilitators for behavioral UE that appeared to promote consistent
re-engagement. For some, usage became part of a routine that made interaction automatic.
Routines were event-based such as finishing a meal with friends or going outside. A handful of
users were motivated to complete the EMAs because they were linked to study compensation.
“The money helped me to not turn the damn phone off because it gets annoying, there’s a lot of
questions” –U13, M
More broadly, the fact that users were enrolled in a research study also supported
consistent re-engagement with EMAs. Users wanted to be honest in reporting lapses and felt
57
“obligated” to interact with MQU. Altogether, study-related motivations help to explain the
relatively stable, if not increasing, levels of EMA compliance over time (Figure 2-5).
“As an engineering student, I care about data […] So I just thought the more surveys I did, the
more data for you guys.” –U18, M
“This wouldn’t matter so much if it was an app that was not part of a study where there was no
such thing as compliance, so I would just ignore it, but in this case I wanted to […] get that
survey done so because of that I felt a little bit of that conflicted pressure” –U10, M
Data Triangulation: Identifying covariates of behavioral UE from user logs and interviews
In this section, we triangulated interview data with user logs to explore the extent to
which factors described to influence behavioral UE were actually reflected in behavioral UE
plots.
Perceived usefulness of MQU
Based on the interview data, how useful individuals perceived MQU to be for avoiding
lapse was identified as an important driver of usage. To operationalize this construct, users with
at least one quote describing any MQU feature as useful for lapse avoidance (e.g., II reminders
were “helpful”, “needed”) were categorized as “Perceived MQU as useful” (n=44; 77%). All
other users were coded as “Perceived MQU as not useful” (n=13; 23%). Figure 2-7 displays the
same behavioral UE trend plots as Figure 2-5, but additionally splits the users into the two
aforementioned groups to descriptively compare behavioral UE trends.
58
Figure 2-7. Behavioral UE trends by perceived usefulness groups
In the first week, users interacted with MQU at similar rates between the “Perceived
MQU as useful” and “Perceived MQU as not useful” groups for all features except checking
progress and reporting craving (Figure 2-7, red and dark blue lines). To help explain lower levels
of checking progress and reporting craving, we reviewed the interviews of users in the
“Perceived MQU as not useful” group to investigate possible reasons. We noted that one
possible explanation was the use of alternative methods of quitting, i.e., vaping, among members
of this group. While we asked about vaping behavior at baseline, we did not actively screen out
users who vaped from participating in the study. Although vaping occurred in both groups, we
found that half of those in the “Perceived MQU as not useful” group reported vaping on at least
one day during the study compared to only a third in the “Perceived MQU as Useful” group.
P e r c e i v e d M Q U a s N o t U s e f u l ( n = 1 3) P e r c e i v e d M Q U a s U s e f u l ( n = 4 4 )
0 1 2 3 4 5
We e k i n q u i t at t e m p t
0 1 2 3 4 5
We e k i n q u i t a t t e m p t
0 %
1 0 %
2 0 %
3 0 %
4 0 %
5 0 %
6 0 %
7 0 %
8 0 %
9 0 %
1 0 0 %
% u s e r s i n t e r ac t i n g
2 %
1 2 %
1 9 %
3 8 %
5 4 %
6 7 %
1 6 %
1 1 %
2 2 %
3 5 %
4 5 %
6 0 %
59
Substituting cigarette smoking with vaping is a plausible explanation for low levels of reporting
craving and checking progress and not perceived MQU to be useful; one user described how his
vape, which he purchased a few days before his quit attempt began, “pretty much replaced”
cigarettes.
Another notable difference between the two groups was increasing trends in both
momentary and EOD EMA compliance over week in quit attempt among users in the “Perceived
MQU as Not Useful” group. The increase in percentage of users interacting with these features,
especially in the last week, may reflect the influence of external motivations to interact with
MQU such as monetary compensation for EMA compliance. This was described in the “Barriers
and facilitators of behavioral UE” theme. There was no indication from the interviews that
increased interaction with EMAs in the last week was because of increased motivation to quit
among these users. Specifically, whereas users in the “Not Useful” group may not have been
motivated to interact with MQU at the start of the quit attempt, they may have become motivated
to interact with EMAs once they realized they were not getting the full compensation for their
study participation.
Phone Type
Phone type was also explored since multiple users identified it as a barrier to UE in the
interviews. Interestingly, phone type did not appear to influence UE with PULL features (red,
light blue, blue lines) based on visual inspection of Figure 2-8; the usage trends for these PULL
features are similar whether one used a personal or loan phone. In contrast, a lower percentage of
users were interacting with PUSH features at all time points among the loan phone users. This is
60
likely because interacting with PUSH features was dependent on hearing the alert and having the
loan phone accessible, which did not always occur according to some users.
Figure 2-8. Behavioral UE trends by phone type
Discussion
The present study extends current knowledge on UE with quit apps through an
examination of multiple UE dimensions. In particular, we highlighted temporal changes in usage
and subjective experience, which is consistent with conceptualization of UE as a process of
repeated re-engagement over time
23
. With respect to usage, descriptive statistics of user log data
revealed users did not necessarily interact with each MQU feature on every day of the quit
attempt (Table 2-2). This was especially the case for PULL feature interactions, which required
P e r s on a l P h o n e ( A n d r oi d ) n = 16 L oan P h on e ( i P h o n e ) n = 41
0 1 2 3 4 5
We e k i n q u i t at t e m p t
0 1 2 3 4 5
We e k i n q u i t at t e m p t
0 %
10 %
20 %
30 %
40 %
50 %
60 %
70 %
80 %
90 %
100 %
% u s e r s i n t e r ac t i n g
7 %
1 0%
2 7%
4 9%
6 3%
8 1%
15%
11%
19%
30%
40%
53%
61
users to initiate interaction with MQU. Behavioral UE plots (Figure 2-5) further highlighted that
subsets of users continued to re-engage with PUSH features over time but not with PULL
features. Furthermore, user-level behavioral UE plots (Figures 2-6a-e) showed that within users,
temporal trends in usage varied depending on the MQU feature. A user may continue to re-
engage with certain features over time and not others, resulting in within-person variation in the
specific features users are exposed to over time. Day-to-day variation in “breadth” of usage
across app features within users could therefore represent another approach for examining UE
longitudinally and a worthwhile focus for future studies on UE.
Our qualitative data provided in-depth information regarding the subjective experience of
interaction with a quit app with respect to the four themes: cognitive UE dimension, affective UE
dimension, temporal changes in UE dimensions, and barriers and facilitators of behavioral UE.
Quotes within these themes aided us with explaining trends in the quantitative, behavioral UE
plots that were generated. For example, we uncovered various reasons as to why users might
have continued to re-engage with some MQU features and why they disengaged with others.
These included momentary motivations to avoid lapse, perceived usefulness of MQU for
avoiding lapse, loan phone usage, and momentary availability to interact with the app.
The subjective experience of interaction with MQU appeared to be different across users.
This is reflected in the extent to which users “appropriated” or interpreted app features in
different ways according to the level of quit support they needed
15,17,18
. For example, users
interpreted reminders to enact an II in different ways. From most specific to broadest in scope,
interpretations included a cue for the specific II, a directive to not smoke, reinforcement for
motivation to quit, and a trigger for self-reflection on one’s smoking. Another way users
appropriated MQU was through anthropomorphizing, or attributing human qualities, to the II
62
reminders
83
. This was an unanticipated finding given that none of the features were meant to
mimic any human qualities and the II reminders were chosen or created by users themselves.
Despite this, several users suggested that MQU was a person to whom they were accountable,
which possibly supported long-term re-engagement with MQU. This phenomenon has also been
reported for another mobile-based quit tool that delivered cessation support through computer
generated text messages
15
. Future iterations of MQU might explore an “anthropomorphization”
strategy given the success of human-like avatars in maintaining long-term engagement for other
technologies
30
.
Interview data also highlighted the affective experience of interacting with MQU. A
consistent report across interviews was annoyance with II reminders, especially when users were
too busy to acknowledge them, no longer thought they were needed, or when the scheduled
HRSS did not actually occur. Although quantitative data showed that the percentage of users
interacting with II reminders remained consistent over time (purple line in Figure 2-5), it is
important to note that the negative subjective experience of the alerts was not necessarily
reflected in behavioral UE plots. This finding highlights the importance of the users’ subjective
experience of UE beyond what can be gleaned from behavioral UE plots.
Information about users’ affective experiences is informative for re-design of MQU
features. For example, a future version of MQU could identify an acceptable threshold of PUSH
interactions that better matches users’ evolving perception of MQU usefulness so as to limit
negative affect associated with such interactions. Some users described how II reminders were
no longer useful once they got into the habit of II enactment even before the II reminder was
delivered. A future version of MQU might be able to identify time points where II reminders as
an intervention strategy are no longer appropriate and subsequently reduce the number of
63
notifications sent to the user. Although not discussed as frequently as annoyance, pride was
another affective response that was described in relation to usage. Strategies that capitalize on
momentary experiences of pride or other positive affect when checking MyProgress or
responding “no” to questions about lapse, should be considered in future iterations. Some
strategies may include in-app badges or rewards to provide positive reinforcement for continued
engagement.
A notable finding from our triangulation of behavioral UE plots and interview data was
that the midpoint of the study was an important period during which several changes in cognitive
processes occurred (i.e., perceived usefulness, perceptions of novelty). Although the study
midpoint may have been a convenient way for participants to conceptualize and describe time in
the quit attempt when summarizing their experience in the interview, these self-reported changes
appeared to correspond with notable inflections in behavioral UE trends occurring in the 2
nd
and
3
rd
weeks for some users. Furthermore, among users in the “Perceived MQU as not useful”
group, notable declines in interaction for acknowledging II reminders occurred during the study
midpoint relative to users in the “Perceived MQU as useful” group. Despite the convergence of
quantitative and qualitative evidence that the 2
nd
and 3
rd
weeks were associated with changes in
usage and subjective experience of MQU, it is noted that causal relationships between them
cannot be drawn. For example, we cannot conclude that perceiving MQU as no longer useful
necessarily led to decreased usage, since this process likely unfolded recursively over time.
64
Implications
Internet and mobile interventions often report decreased behavioral UE (e.g., log-ins,
clicks) over time
65
60
, but pay less attention to investigating specific factors associated with
decline in usage and key time points. The present study elucidated such factors for our MQU-
supported quit attempt, which can be leveraged to further refine the frequency, content, and
timing of intervention features that smokers access over time. For instance, our findings suggest
that a useful time to reassess whether II reminders, at the frequency delivered by MQU, are
needed or wanted is the second week of a quit attempt. If needed, we might consider tailoring the
frequency of intervention delivery as other smoking cessation interventions have done
13,15,84
. For
instance, Haug and colleagues evaluated a text-based smoking cessation intervention where users
received 2 daily messages in the first two weeks of the quit attempt and only 1 daily message in
the 2
nd
and 3
rd
week
13
. Whether such a strategy promotes continued re-engagement with a quit
app should be empirically assessed.
Several contextual factors emerged as barriers and facilitators to behavioral UE and may
be additionally important for refining MQU design. One such barrier was user availability, or
momentary willingness and ability to interact with MQU, which is a key design consideration for
just in time adaptive interventions (JITAIs)
73
. Although previous studies have suggested that
HRSSs like drinking and socializing with friends may have been ideal times to deliver II
reminders and EMA prompts
74
, findings from the present study suggest that smokers may not be
receptive to intervention via II reminders during these events. Alternative strategies should be
considered, such as sending a reminder farther ahead in advance of these kinds of HRSS. A key
facilitator of usage was related to being enrolled in a research study and the need to respond to
momentary and EOD EMAs. Although EMAs were only designed to assess behavior change
65
outcomes for research purposes, the perceived “pressure” to respond to them appeared to
motivate individuals to continue re-engaging MQU. The 15-minute time frame in which to
respond, coupled with financial incentives, also likely helped to maintain consistent interaction
with EMAs across the quit attempt.
Limitations
Study findings should be considered in light of limitations. The first is related to
participants’ enrollment in a research study. It is difficult to disentangle the extent to which
usage was driven by study protocols, i.e., EMA compliance and compensation, versus internal
motivation to interact with MQU for cessation support. Usage of PUSH features was overall
stable over time with increasing trends for those who perceived the app as not useful, which
could be explained by study-related motivations such as compensation based on EMA response
rates. Secondly, a majority of users had a loan phone to interact with MQU, which is not
representative of real world quit attempts. Despite pre-testing phones, several users reported app
malfunctions and battery life issues, contributing to lower levels of behavioral UE than might be
expected.
With regard to generalizability of our findings, UE with MQU may not be representative
of all smokers. We enrolled Asian American young adults who are often embedded in pro-
smoking social and cultural contexts
51,85
. Furthermore, nicotine dependence was low overall and
our population is still early in their smoking career. Heavier smokers may have different UE
experiences with MQU if they are more addicted, as one participant suggested. We note
however, that our interest was not necessarily to describe a universal experience of UE, as it
66
varied even within our sample, but also because MQU was designed for this particular smoker
subgroup.
Despite these limitations, we provide in depth information about how MQU users
interpreted the II reminders, the various ways they interacted with PUSH and PULL features
over time, and the extent to which quantitative representation of behavioral UE reflects the
subjective experience of UE. We also uncovered important aspects of the UE experience that
may influence usage such as perceived novelty and usefulness, evolving motivations to quit, and
current progress towards quitting goals. These findings are informative for re-designing MQU
features to better provide dynamic and personally useful quit support as long as a user needs
MQU.
Conclusion
The present mixed methods study highlighted the importance of examining UE across
behavioral, cognitive, affective, and temporal dimensions. Specifically, we demonstrated the
utility of combining users’ subjective experience with usage data for evaluating the extent to
which users interpreted and reacted to intervention content as intend. In contrast to research that
characterizes UE as a user-level variable, our analysis of user log data allowed us to explore the
diverse ways individuals interacted with each MQU feature over time throughout their quit
attempt. Our interview data supplemented this analysis by uncovering factors associated with
changes in observed longitudinal trends in usage. Having provided preliminary analysis of
temporal trends in usage across various MQU features, future work will more formally explore
how usage patterns evolve within users over time.
67
Chapter 3: Identifying dynamic user engagement patterns with a quit app using multilevel
latent class analysis
Introduction
Characterizing UE as patterns of feature interaction
A smoking cessation app often includes multiple components or features intended to
support quitting
12
. Common quit app features include text messages providing encouragement,
social networking features such as text buddies
86
, advice on cessation medication, assistance
with setting a quit date, and rewards for abstinence
12
. While each user may choose to interact
with these app features in unique combinations or patterns over time, the way researchers
currently report on app usage (i.e., behavioral user engagement, “UE”), has remained limited
42
.
For example, behavioral UE with interventions has often been operationalized through
aggregated measures of intervention exposure such as number of log-ins, duration of app visits,
or number of modules completed. Such indicators of usage ignore the specific intervention
features that users are interacting with
63
. Analysis of which specific intervention features users
interact with is important because it helps researchers to identify the behavioral and
psychological processes associated with usage that help to explain intervention outcomes.
The ways in which users interact with combinations of diverse app features is not well
understood. For instance, some users may interact with all features equally (e.g., accessing
advice, encouragement, and social networking), while others may interact with a limited subset
of features (e.g., accessing advice only). It is also possible that certain types of interactions could
co-occur more frequently together
60
and represent what we will refer to in this dissertation as
“UE patterns”. For instance, checking one’s quit smoking progress and reporting craving events
68
could indicate a specific UE pattern characterized by self-monitoring, whereas responding to
assessments and recording lapses may indicate another UE pattern characterized by adherence to
research protocols. UE patterns may also be related to how interactions with features are
initiated. For example, previous quit app research has distinguished between interactions that are
system-initiated (PUSH) vs. those that are user-initiated as needed (PULL)
76
. One less obvious
difference between the two is that relative to push features, PULL features may require
individuals to be more aware of their smoking risks and motivated to request support when those
risks arise
44
. Characterizing UE as interaction with a combination of diverse apps features
represents a novel, multivariate approach to describe individuals’ usage and exposure to various
app features that support quitting.
Once UE patterns are identified, intervention researchers might be interested in whether
one UE pattern is more positively associated with behavior change outcomes than another. A few
studies provide preliminary evidence that certain UE patterns are important with respect to
intervention outcomes. For example, researchers have suggested that there may be instances
where interaction with one feature alone may not predict behavior change, but a combination of
features might
37,40
. In a web-based depression intervention, the number of activities completed
per log-in was associated with improved outcomes, while the total number of log-ins or modules
completed over the intervention period was not
40
. Similarly, another study reported that
“breadth” of UE (interacting with all available intervention features) was positively associated
with fruit and vegetable consumption, whereas the total amount of time spent interacting with
features was not (i.e., duration)
22
. In other words, examining the frequency or duration of app
use alone may provide a limited picture of how intervention tools facilitate intended behavior
69
change. The specific combination of features one interacts with might provide additional insight
into this relationship.
Using latent class analysis to explore underlying UE patterns
While one may hypothesize about the existence of distinct UE patterns using
preconceived classifications (e.g., less vs. more mentally involved interactions), data-driven
modeling represents an alternative approach. Latent class analysis (LCA) is particularly useful
for uncovering underlying, unmeasured UE patterns. LCA allows for exploration of such
subgroups of UE patterns within a population using measured variables as “indicators”
87
. These
subgroups are represented by “latent classes” which are related to indicators (see Eq. 1)
87
. An
advantage of using LCA is the ability to estimate the relative prevalence of each subgroup and
the posterior probability of membership in a latent class given a set of indicators
87
. Adjustment
for multiple covariates is also possible within this framework. Importantly, LCA allows us to
explore the number of UE patterns that best describe the data a priori. This is particularly useful
when there is little available guidance regarding which UE features should be used to describe
subtypes of UE patterns
24
.
Only one known study has attempted to explore underlying subtypes of UE patterns using
data driven methods
88
. Sepah and colleagues used factor analysis to group multiple indicators of
behavioral interactions with a technology-delivered diabetes prevention program. Two factors
were identified, one relating to tasks that users were asked to do weekly and another related to
number of logins and comments to a virtual messaging board
88
. The resulting factor variables
effectively represented UE patterns as a user-level attribute; users could be high or low on each
of these UE factors.
70
Users may display multiple UE patterns over time
When assessed at one time point or aggregated per individual, UE indicators are
inherently treated as user-level attributes. As a result, whether and how usage patterns might
fluctuate over time is overlooked. To address UE patterns dynamically, researchers can examine
repeated measures of UE within users, such as daily patterns of interactions over an intervention
period. The available studies that do report UE indicators dynamically demonstrate that counts of
interaction with app features generally declines over time
65,67,68,89
with differences in rate of
decline depending on the particular feature
66
. Rather than describing longitudinal trends in usage
for individual app features, at least one study has examined temporal changes in breadth of
feature interaction
60
. Gouveia and colleagues showed that as users progressed towards their
walking goals, their interactions with a physical activity promotion app were more frequently
characterized by “glances”, which were short in duration (~5 seconds) and had limited feature
interaction
60
. Additional research is needed to examine whether users display multiple, distinct
UE patterns with quit apps and how these patterns may unfold over time. Interventions that
monitor and identify such UE patterns dynamically may be able to implement strategies in near
real time to promote the specific UE patterns that researchers believe are desirable a priori or
based on previous research.
The present study seeks to extend conceptualization of UE in mobile intervention
research by identifying (1) underlying subtypes of UE patterns within individuals (2) over time
across a smoking cessation attempt. Specifically, the current study design allowed for identifying
and classifying UE patterns at the day level, where days were the unit of observation. Consistent
with our conceptualization of UE as a process of repeated interactions over the course of a
cessation attempt
23
, we hypothesized that users would display heterogeneity in day-level UE
71
patterns within themselves. In other words, a user might exhibit more UE “breadth” with respect
to multiple app features on one day and less breadth on another day
22
. As an extension of our
qualitative investigation in Study 1 regarding changes in UE over time, we place emphasis on the
temporal dimension of UE, rather than aggregating UE indicators to a user-level attribute.
Multilevel latent class analysis adjusts for multilevel data
Because the data for this analysis represent days clustered within users, i.e., individual
users contributing multiple days, observations at the day level are not independent. If multilevel
data structure is unaccounted for, correlation between these observations would yield biased
parameter estimates
90
and, thus, single-level latent class analysis would not be appropriate. To
address our research objective with respect to identifying daily UE patterns, our statistical
approach must account for days clustered within users. In contrast to single-level LCA,
multilevel latent class analysis (MLCA) allows us to adjust for such clustering. In addition to the
general objective of LCA, estimating the probability of a day belonging to a particular latent
class of days, MCLA allows us to adjust for the possibility that the probability of day-level UE
belonging to a particular latent class might vary across users. For instance, the probability of a
given day being classified as “actively engaged UE” for one user may be different from another
user’s. Additional details about MLCA are described in our analytic approach.
Covariates of day-level UE patterns
MLCA allows for the simultaneous modeling of covariates at the day- and user-levels.
With respect to day-level covariates of UE patterns, day in quit attempt (measured continuously
from 1-28) is of primary interest. In addition to previously mentioned research demonstrating
72
declining usage of technology-based behavior change interventions over time
65,67,68,89
,
descriptive UE graphs from Study 1 showed similar declines in usage for PULL features in
particular. Interview data from Study 1 additionally suggested that over time, users no longer
perceived II reminders to be useful, either because they were better able to resist lapse without
support from MQU or because MQU became too burdensome to interact with. Given this
finding, we hypothesize that more active daily UE patterns, characterized by interaction with a
diverse set of MQU features, may be less probable as days in the quit attempt pass by.
In terms of user-level covariates, we demonstrated descriptively in Study 1 that users who
interacted with MQU via a study loan phone displayed lower levels of PUSH feature
interactions, but not necessarily with PULL features. Interview data revealed that this
discrepancy was related to the burden of carrying and charging a secondary phone, which
affected users’ ability to attend to PUSH notifications from MQU. Therefore, the influence of
loan phone use is also likely to be associated with lower probability of active daily UE patterns
relative to other patterns and will be adjusted for in MLCA.
Study 2: Specific Aims
The overall objective of the present study was to explore distinct subtypes of UE patterns
at the day level using users’ daily interactions with multiple MQU features as latent class
indicators in MLCA. The specific research questions and associated hypotheses were as follows:
Aim 1: To explore underlying subgroups of day-level UE patterns during a quit attempt using
MQU.
H1: There may be at least three day-level UE patterns: active, less active, and not active
patterns
73
Aim 2: To test the association between probability of membership in day-level UE patterns and
day in quit attempt.
H2: The probability of day belonging to an “active” relative to a “less active” day-level UE
pattern will decrease as a function of day in quit attempt.
Aim 3: To examine whether probability of membership in day-level UE patterns varies as a
function of loan phone use.
H3: The probability of a day belonging to an “active” relative to a “less active” day-level
UE pattern will be lower for users interacting with MQU via a loan phone.
Methods
Data for this study come from automatically recorded logs of MQU interactions and have
been described in detail in Study 1 as “behavioral UE” or usage. Briefly, participants interacted
with MQU during a 28-day quit attempt by acknowledging self-specified reminders for plans to
avoid lapsing, i.e., implementation intentions (II)
56
, during contexts users indicated they were at
high risk of lapsing. Ecological momentary assessments (EMAs) asked about lapse outcomes
and were delivered 45 minutes after each of these high-risk contexts (momentary EMAs) and at
the end of each day (EOD EMAs). In addition, users could report lapses or cravings and check
their quit progress using a tracking feature. Interactions with these MQU features served as latent
class indicators for the MLCA. The six latent class indicators were all binary variables
representing daily interactions for each feature and are operationalized in Table 3-1.
74
Table 3-1. MLCA indicators and descriptions
MQU indicator PUSH vs. PULL Description
1. Acknowledge II
reminders
PUSH Indicates whether at least 3 II reminders were
acknowledged (user pressed “okay”) on a given day
2. Momentary EMAs PUSH Indicates adherence to the study protocol, defined
as completion of at least 4 EMAs on a given day
3. End of day EMAs PUSH Indicates completion of the single EOD EMA that
day
4. Report lapse PULL Indicates whether or not a lapse was reported at
least once on a given day
5. Report craving PULL Indicates whether or not a craving was reported at
least once on a given day
6. Check progress PULL Indicates whether the MyProgress was viewed at
least once on a given day
In addition to the MLCA indicators, covariates identified from Study 1 were also
included in analysis. Loan phone use was a binary user-level variable and day in 28-day quit
attempt was a continuous day-level covariate.
Analytic Approach
Following approaches suggested by Henry and Muthén, a single-level LCA that ignores
clustering of days within users was conducted first
91
. Beginning with a one-class solution,
additional classes were specified to this base model one at a time. Once the best fitting LCA
model was selected, a parametric approach was used to account for the nested structure of the
data. In this approach, random means were introduced in a series of MLCA models, which allow
for the odds of belonging to a class to differ across users. MLCA is similar to a mixed-effect
regression model with categorical outcomes, the difference being that the outcome is a
categorical latent class
91
. Model fit was assessed iteratively throughout the process, starting
75
from a one class solution in single level LCA onwards using the following standard measures of
fit: BIC and entropy
91
. Final models were chosen based on these indices in addition to
substantive meaning of the identified classes
87
.
Mathematical equations relevant for MLCA as described by Henry and Muthén are
presented below
91
. The probability of a day (i) from a user (j) belonging to particular latent
class (t), !(!
!"
= !), is expressed by equation (1), which is similar to a multilevel logistic
regression model but with a latent rather than observed categorical outcome C
ij
. The two-level
logistic random intercept model is expressed in equations (2) and (3) where the log odds of
!
!"
= ! is a function of the random mean for the probability of belonging to a given latent class
!
!!
and the effect of a day-level predictor !
!
!
!!
, e.g. day in quit attempt (day 1-28). In equation
(3), the random mean is expressed as a function of the average log odds of class membership
across users !
!
, the effect of a user-level predictor !
!
!
!
such as loan phone use, and a user’s
random deviation from the average log odds, !
!!
.
(1) !(!
!"
= !)=
!"# (!
!
! !
!
!
!"
! !
!
!
!
! !
!!
)
!! !"# (!
!
! !
!
!
!"
! !
!
!
!
! !
!!
)
(2) !"#$% !
!"
= ! = !
!!
+ !
!
!
!"
(Level 1: Day level)
(3) !
!!
= !
!
+!
!
!
!
+ !
!!
(Level 2: User level)
Equation (4) expresses the probability of a specific indicator response pattern P(!
!"#
=
!
!
), what we have described previously as combinations of MQU feature interactions
91
. The
probability of a particular response pattern is a function of the probability of a day belonging to a
particular class ! !
!"
= ! described above and response probabilities conditional on class
76
membership !(!
!"#
= !
!
|!
!"
= !). !
!"#
represents a binary latent class indicator for day i nested
in user j for indicator k and s
k
expresses the specific response for that indicator. C
ij
represents the
latent class variable and t refers to a specific latent class among T number of latent classes.
(4) ! !
!"!
= !
!
,!
!"!
= !
!
…!
!"#
= !
!
= ! !
!"
= !
!
!!!
!(
!
!!!
!
!"#
= !
!
|!
!"
= !)
Figure 3-1. Example 3-class multilevel latent class model with covariate
Figure 3-1 presents an example 3-class MLCA model where days are nested in users. The
model includes six day-level binary indicators (UE1-UE6). The top half of the figure reflects the
day level where K number of latent classes are denoted by an open circle (“Daily UE”) and two
K-1 filled circles. The two filled circles represent continuous latent variables denoting random
means for probability of a day belonging to a particular latent class across users. These random
means are further represented in the bottom half of the figure (user level) as two open circles
labeled C#1 and C#2. C#1 refers to the log odds of a day belonging to class 1 vs. class 3, while
C#2 refers to the log odds of a day belonging to class 2 vs. class 3. The third class is not pictured
in Figure 1 but serves as the reference class and is set to 0. Residual variances of the random
C#1 C#2
Day level, n=1596 days
User level, N=57 users
Daily
UE
UE 1 UE 2 UE 3 UE 4 UE 5 UE 6
X
77
means are represented by arrows pointing to the open circles at the user level. The magnitude of
the residual variances indicates the extent to which probability of membership in a given latent
class varies across users.
In the final step, we introduced covariates to the final MLCA model. This model was
expanded to include day-level and user-level covariates simultaneously and latent class
membership was regressed on these covariates
91
. Using loan phone as an example of a user level
covariate, we explored whether a day from an individual using a loan phone was more likely to
belong to a “less engaged” UE pattern relative to day from an individual using their personal
phone, holding day in quit attempt constant. All analyses were conducted using Mplus v. 6 with
its default settings: maximum likelihood estimators and robust standard errors (Mplus, Los
Angeles, CA)
92
.
Results
Table 3-2. MLCA indicators with univariate frequencies
MQU feature Latent class indicator
% days where
interaction occurred
PUSH features
II Reminders Acknowledge ≥3 II reminders
37%
Momentary EMAs
End of day EMAs
Completed ≥4 momentary EMAs
45%
Completed 1 EOD EMA 62%
PULL features
Lapse button Report lapse at least once 19%
Crave button Report craving at least once
19%
MyProgress Check progress at least once
29%
Users provided a maximum of 28 days of data, yielding a total of 1596 observations
(N=57*28=1596 days). 14 days were removed from analysis due to two participants ending the
study early for planned international travel. Table 3-2 presents univariate frequencies for each of
78
the six indicators used in the MLCA. For example, EOD EMAs were completed on 62% of the
days and reports of craving occurred on 19% of days. 1,134 days (72%) were attributed to
interactions on loan phones (vs. personal devices).
Model selection
Model fit indices from traditional single-level LCA are presented in Table 3-3. Bayes
Information Criterion (BIC) declined steadily from 1- to 4-class solutions, indicating improved
model fit as additional classes were specified. BIC leveled off when a fifth class was added and
entropy was maximized with four classes. This suggested that a 4-class solution best fit the data.
Although four classes appeared to fit the data better than three based on these indices, inspection
of conditional probabilities for both solutions showed that the 3-class model was more
interpretable and parsimonious. For these reasons, both the 3- and 4-class models were further
evaluated using MLCA to determine the best solution after accounting for clustering.
Table 3-3. Model fit indices
Single-level LCA; n=1,582 days
Model 1 LC 2 LC 3 LC 4 LC 5 LC
BIC 16235 15563 15437 15198 15147
Entropy .71 .70 .76 .64
Multilevel LCA; n=1,582 days; N= 57 people
BIC 10239 9885 ---
Entropy .79 .81 ---
The 3- and 4-class models selected from single-level LCA were extended to include
random means for the probability of day belonging to a particular latent class. We began first
with the 3-class MLCA model, which showed improved BIC and entropy over the single level 3-
79
class model. When the 4-class MLCA model was attempted, the resulting solution did not
converge with replicated log-likelihoods even after increasing random starts to 10,000 and
specifying variance for the random means at 0. Henry and Muthén have noted that MLCA
models with random means are computationally intensive to run, especially as the number of
level 1 (day-level) latent classes are increased
91
. Thus, we report on the 3-class solution as the
final MLCA model.
MLCA results
As can be seen in Table 3-4, the most prevalent class of day-level UE patterns (Class 3;
45%) showed low probability of interaction with almost all PUSH and PULL features (3-14%)
except for EOD EMA (35%). This class represents a pattern that can be characterized as Low
UE. The next most prevalent class (Class 2) also had low probability of interacting with the
PULL features, but is distinguished from the former class in that there was a high probability of
interaction with PUSH features: both EMAs (86-88%) and acknowledging II reminders (63%).
This latent class, which can be described as Passive UE, is expected to represent 35% of the
days. The least common class was characterized by high probability of interaction with all
features, i.e., Active UE, and 20% of the day-level UE patterns were estimated to belong to this
class. This day-level UE pattern had similar probabilities of interaction with PUSH features as
the Passive UE class, but was notably the class with the highest probability of interacting with all
three self-initiated PULL features: 31% and 47% for crave and lapse buttons respectively, and
94% for the progress tracker.
The variances of the random means (arrows in Figure 3-1) for Class 1 (Active UE) and
Class 2 (Passive UE) were 10.5 (se = 3.7) and 6.6 (se = 2.3) respectively, p’s <.01. This indicates
80
that the probability of day-level UE patterns belonging to Active and Passive UE classes varied
significantly across users.
Table 3-4. Multilevel latent class solution for 3-class model with conditional probabilities
for indicators
Class 1 Class 2 Class 3
Indicator for feature
interaction
20%
Active UE
35%
Passive UE
45%
Low UE
PUSH
II Reminders 50% 63% 10%
Momentary EMAs 65% 86% 3%
End of day EMAs 77% 88% 35%
PULL
Crave button 47% 10% 14%
Lapse button 31% 20% 13%
MyProgress 94% 14% 12%
Next, we assessed whether covariates were associated with latent class membership using
multinomial logistic regression. Day in quit attempt and using a loan phone to interact with
MQU were found to be significant predictors of class membership in separate regression models.
Table 3-5 presents the estimated log odds from a final model including both covariates
simultaneously where the Low UE class served as the reference group. As each day passed, the
log odds of a given day belonging to the Active UE class relative to the Low UE class decreased
linearly (est.=-.10, OR=.91, p=.01) adjusting for loan phone use. Interpreted as an odds ratio, this
means that the odds of a day belonging to Active UE (relative to Low UE) decreased by 9% per
day. Day in quit attempt was not associated with membership in the Passive UE class relative to
Low UE class (est.= -.02, p=.24). However, using a loan phone did significantly predict days
belonging to the Low UE class. Adjusting for day in quit attempt, UE patterns among loan phone
81
users had 86% lower odds of being in Passive UE relative to the Low UE class (OR = .14,
p=.02).
Table 3-5. Covariates of day-level latent class membership
Log odds (SE)
(HIGH vs. LOW UE)
Log odds (SE)
(PUSH vs. LOW UE)
Day-level predictor
Day in study -.10 (.04)** -.02 (.02)
User-level predictors
Loan Phone -1.13 (1.4) -1.98 (.83)*
* = p<.05; ** p<.01
Table 3-6. Multilevel latent class solution for 3-class model with conditional probabilities
for indicators with covariates
Class 1 Class 2 Class 3
Indicator for feature
interaction
21%
Active UE
35%
Passive UE
44%
Low UE
PUSH
II Reminders 53% 62% 10%
Momentary EMAs 65% 86% 2%
End of day EMAs 77% 88% 35%
PULL
Crave button 46% 10% 13%
Lapse button 33% 19% 12%
MyProgress 92% 12% 12%
We note here that when introducing covariates to a base latent class model, latent class
formation can be affected, i.e., conditional probabilities for indicators may change
93,94
. To assess
potential changes as a result of including covariates, we present Table 3-6. It displays re-
estimated class prevalence along with conditional probabilities for indicators in a MLCA model
with covariates included simultaneously (i.e., loan phone, day in quit attempt). Notably, the
prevalence of each class, along with conditional probabilities associated with each indicator, are
82
I D
D A Y
1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8
1
5 1
3 7
4 1
1 1
3 6
2 2
4 9
5 2
1 0
2 4
2
4
9
3 9
3
1 2
1 9
2 9
3 5
5 6
8
2 0
3 1
4 6
5 4
6
3 2
4 4
1 4
1 7
3 4
1 6
2 7
1 5
3 3
3 8
4 2
4 7
2 5
2 6
2 8
4 3
4 5
5 3
2 3
7
4 8
5
3 0
4 0
1 3
5 0
5 7
1 8
5 5
2 1
essentially similar to those from the MLCA model with no covariates (Table 3-4). Our
interpretations of the latent classes extracted from the data remain unchanged.
Figure 3-2. Heat map of most likely day-level UE pattern by user and day in quit attempt
Active UE
Passive UE
Low UE
To better visualize the distribution of day-level UE patterns as a function of time, we
generated a heat map of most likely class memberships for days within users (Figure 3-2). Each
row corresponds to an individual user (n=57) and each column corresponds to day in quit attempt
83
(1-28). The colors indicate the most likely class membership for a given day, which is derived
from the MLCA estimates. Green represents Active UE, yellow represents Passive UE, and red
represents Low UE. The white cells indicate missing data for two users: ID 10 and 31. The rows
are ordered from top to bottom based on their distribution of UE patterns; rows on the top have
more Active UE days and rows towards the bottom have more Low UE days.
Descriptively, we note that green cells are less common as day in quit attempt increases,
especially in the last week, confirming our covariate analysis. With regard to within-user
heterogeneity in day-level UE patterns, it appears that a subset of users tend to display only two
UE patterns at most, a frequent pattern interspersed with a less frequent, alternative pattern. For
example, user ID 41’s days were most frequently classified as Active UE except for 2 days,
which were Low UE. Another subset of users in the middle of the heat map, rows between IDs 6-
47, display more balanced distribution of Passive and Low UE patterns.
Discussion
Our findings demonstrate the utility of MLCA for understanding the range of UE patterns
that may exist during an app-supported quit attempt. This analytic approach identifies distinct
subtypes of observations while allowing for the probability of day-level UE class membership to
vary across users. The best fitting MLCA model, which adjusted for day-level data clustered
within users, suggested that there were three distinct subtypes of day-level UE patterns with
MQU. The most prevalent latent class was Low UE, followed by Passive UE and Active UE.
Days in the Low UE class were characterized by poor adherence to EMAs,
acknowledging below average numbers of II reminders, and low probability of PULL feature
interaction. From an implementation perspective, Low UE may represent a concerning pattern
84
because users received the lowest exposure to the primary intervention component, II reminders.
The unexpectedly high prevalence of this class should be explored as a moderator of intervention
outcomes. However, it is important to note that under some circumstances, Low UE may not
necessarily reflect a suboptimal pattern. This could be the case when users no longer perceive
that MQU is useful for support their attempt, especially for those with more successful cessation
outcomes. Intervening to increase UE in this context may result in negative user reaction and
instead lead to either “counter-productive” or “non-productive” engagement
18
. Both states of
engagement have been linked to negative perceptions of app content, stalled progress towards a
behavior change goal, and app abandonment
18
.
In contrast, Active UE days showed relatively greater probability of interaction across all
MQU features, but most notably checking progress. This increased probability of interaction
with the MyProgress feature also coincided with higher probability of reporting cravings and
lapses. Thus Active UE days may represent a UE pattern when users are more attendant to their
smoking behavior and quit progress. This UE pattern might be considered as one with greater
“breadth”, a pattern that has been positively associated with intervention outcomes in the
literature
22
. Because this UE pattern is associated with the greatest exposure to intervention
content, it may reflect a pattern of optimal exposure that should be promoted for all users.
Whether Active UE is associated with positive cessation outcomes should be tested empirically.
The extraction of a Passive day-level UE pattern is notable. This class represents days
that distinguish between different types of feature interaction (i.e., PUSH vs. PULL). In contrast
to Active UE days, Passive UE days showed even greater adherence to the system-initiated
PUSH features and similarly low levels of the self-initiated PULL feature use as Low UE days.
This day-level UE pattern may reflect users’ perceived need to adhere to study related protocols,
85
but not necessarily a need to manage their quitting through PULL features. This interpretation is
corroborated by several interviews in Study 1 in which users reported feeling “obligated” to
respond to reminders and EMAs, even when MQU’s perceived usefulness declined.
Importantly, Passive UE days may have been overlooked if we instead measured usage
with respect to total counts of interactions each day ignoring feature type. Take for example a
day where 5 momentary EMAs were completed and 3 II reminders were acknowledged
compared to a day with 3 momentary EMAs completed, 3 II reminders acknowledged, and
reports of both a craving and lapse. If we were to only count the total counts of interactions, both
days may be interpreted as “high” UE (8 total interactions), However, the subjective experience
of interaction may be qualitatively different between these days despite being comprised of the
same total counts of interactions. Passive UE may have indicated days where an individual was
motivated to remain adherent to study protocols but not mentally processing II reminders. In
contrast, Active UE days could have represented days where the user was actively managing
their quit attempt. The distinction between these two patterns can be assessed as an alternative
measure of usage in research investigating the relationship between usage and intervention
outcomes.
Interestingly, Passive UE days were associated with greater momentary and EOD EMA
adherence and more II reminders acknowledged than Active UE days. Despite low probability of
interaction with PULL features, Passive UE may also represent an optimum level of exposure for
some users
6
. This is because users would be re-engaging with MQU enough so that PULL
features would still be easily accessible to the user when needed. Indeed, the use of PUSH
notification features such as reminders and texts have been described in Study 1 and other
studies as supportive of long-term re-engagement with behavior change tools
95
. To the extent
86
that Passive UE reflects willingness to continue engagement with MQU, strategies should be
devised to prolong this UE pattern. We highlight however that prolonging this UE pattern may
only be beneficial to individuals who are still actively working towards their quit goal and not
among those who have successfully remained abstinent during a quit attempt and do not require
further support.
Our findings also support that class membership is associated with day in quit attempt.
As days went on in quit attempt, the odds of a day being Active UE relative to Low UE
diminished markedly. This is also graphically represented in the heat map (Figure 3-2). That the
probability of Active UE decreases as each day passes is consistent with behavioral UE trends
described in Study 1 and existing literature showing that users interact with quit app features less
frequently and with less diverse features over time
18
. It is important to note that our analysis
assumed a linear association between day in quit attempt and class membership. Although this
assumption is consistent with linear declines in feature interaction observed in Study 1, whether
a non-linear trend better describes this association warrants further inspection. Interestingly, our
findings show that the odds of a day being Passive vs. Low UE was not influenced by day in quit
attempt. Whether a day UE pattern was classified as Passive or Low was consistent across time
and can be observed in Figure 3-2 as well.
Loan phone use was a strong predictor of days belonging to the Low UE class compared
to Passive UE class. Indeed, several users reported difficulty adhering to study assessments when
using a loan phone. From our interviews, common reasons included not charging it, forgetting to
bring it with them, or not hearing alerts when they carried it in their bags. These findings suggest
that interventions should be delivered on primary personal devices that are readily accessible,
87
particularly for push feature interactions that are intended to occur within predefined timeframes
(II reminders).
Although behavior change outcomes are of primary interest to interventionists, our
present analysis did not consider a lapse outcome. Rather, the scope of this study was limited to
elucidating the range of possible daily UE patterns as a function of time. It has been
recommended that researchers determine the number of latent classes before evaluating
additional variables as these auxiliary variables can influence how latent classes are extracted
94
.
A natural extension of our findings is to associate daily UE patterns with cessation outcomes on
the same day. Based on findings from this study and Study 1, we might hypothesize that days
characterized by Passive UE are associated with lower odds of lapsing. Passive UE days show
the greatest number of II reminders acknowledgements and very limited reports of lapse or
craving. We also note that there may be unexpected psychological mechanisms through which
UE influences cessation outcomes. For example, our findings in Study 1 highlighted the
heterogeneous ways that users interpreted the II reminder messages. Therefore, a mixed methods
approach is needed in addition to MLCA to achieve a nuanced understanding of how daily
distinct UE patterns are potentially associated with outcomes.
Implications
Our analytic approach and findings are useful for informing the design of strategies to
promote desired UE patterns that can be deployed dynamically during an intervention period. By
analyzing within-user heterogeneity in day-level UE patterns, we explored the extent to which
users displayed a diverse range of UE patterns across the quit attempt. Once a range of UE
patterns are explored, researchers can subsequently identify the specific patterns they believe
88
provide users optimal exposure to intervention content. Strategies such as making specific
features more prominent or accessible on the app’s home screen or through push notifications
can be deployed dynamically during key time points in an effort to promote optimal UE patterns
96
. By identifying time points within the quit attempt to re-engage users, we can conserve
intervention resources only for days within users that require additional nudging for re-
engagement.
The heat map visualization (Figure 3-2), which depicts most likely membership within a
particular day-level UE pattern, highlights specific time points (days) within the quit attempt
during which we might enact re-engagement strategies. One such day could be when users
transitioned from one day-level UE pattern to another. For example, during days identified as
Low UE following an Active or Passive day, we might deliver a prompt that assesses if the user
is experiencing technical difficulty or other usability issues with the app in order to troubleshoot
potential app issues. Alternatively, we could provide positive reinforcement to users during days
when they transition from Passive or Low UE to Active UE days. This transition may signal a
user’s renewed interest in managing their quit attempt or a day when they need additional
cessation support.
An important limitation of leveraging day-level UE patterns to deliver user-tailored
prompts is that we currently lack information about why a user displays a particular UE pattern,
which is critical for tailoring. Furthermore, the MLCA presented in this study represents post-
hoc classification of UE days. If dynamic or ongoing classification of days is desired, additional
work may be needed to understand how many prior days a user needs to experience before
estimates about a given day’s class membership can be made. Notwithstanding, day-level UE
89
patterns may represent a novel tailoring variable for just-in-time interventions designed to
promote re-engagement with MQU
59
.
Limitations
Our findings should be considered in light of study limitations. First, our sample is not
representative of all smokers using an app to quit smoking. Our users were predominantly light
smokers with low levels of nicotine dependence (FTND Mean score=2.57 / 10) compared to
other populations studied. For instance, individuals with higher levels of nicotine dependence
and greater motivation to quit smoking may use PULL features more frequently
44
. Importantly,
the daily UE patterns identified may be limited to MQU or other apps that include a mixture of
PUSH and PULL features. Furthermore, UE was evaluated in the context of a research study,
where there were set recommendations for interacting with the app, i.e., response to at least 4
momentary EMAs per day.
A second limitation of the present study is its primary emphasis on behavioral UE or
usage data. As has been discussed in Study 1, cognitive and affective dimensions of UE have
been understudied with regard to digital behavior change interventions, despite representing
important UE dimensions. It is possible that the daily UE patterns identified here can be further
characterized by psychological processes such as motivation to continue using the app or
emotional responses to feature interaction. With that said, timely measures of such variables are
more difficult to capture passively and were beyond the scope of the present study. Future work
on daily UE patterns might consider including periodic self-report EMA measures of cognitive
and affective processes such as focused attention and pride to better understand how these
constructs might provide a nuanced picture of engagement.
90
Despite these limitations, our study represents a novel application of a multivariate
statistical method to identify a range of possible daily UE patterns, how prevalent they were
expected to be, and their probability of occurring as a function of time. Up until this point, we
have relied on descriptive and interview data to characterize behavioral UE with MQU. By using
a multivariate approach we were able to conceptualize UE as patterns of feature use that tended
to co-occur together on a given day. A major strength of this work is its emphasis on day in quit
attempt as a predictor of class membership, which allows us to model how daily UE patterns
unfolds over time.
Conclusion
To our knowledge, this is the first study to apply MLCA to user log data in order to
explore day-level UE patterns with a smoking cessation app. Until now, behavioral UE has been
assessed via user-level proxies of exposure, such as total number of log-ins within the
intervention period. By using a multivariate approach instead (i.e., use of multiple indicators), we
represented UE as a pattern of interaction with a diverse set of app features. Our findings helped
to formally uncover heterogeneity in day-level UE patterns over time within users and their
relative probability of occurring on given day in a quit attempt. Monitoring and identifying UE
patterns on the day level during an intervention is important because it provides information
about when to deploy personalized strategies to promote ideal UE patterns. Ideal UE patterns
may be those daily patterns that are effective for supporting lapse avoidance but do not provide a
level of exposure beyond what is needed to achieve outcomes. Additional work is needed to
define which day-level UE patterns are associated with improved cessation outcomes.
91
Chapter 4: Investigation of effective engagement with a quit app and daily lapse during a
quit attempt using mixed methods
Introduction
Effective engagement vs. user engagement
A recent review showed that using text-based smoking cessation interventions was
generally effective for supporting quit attempts compared to usual care
20
. Available evidence
also suggests that more frequent app usage, and therefore greater exposure to intervention
contents, is positively associated with cessation outcomes
38,39,84
. However, emerging research
asserts that more frequent user engagement, conceptualized as greater usage of digital behavior
change interventions, has not been consistently associated with desired behavior change
22,40
.
Donkin and colleagues contend that this inconsistency in findings may be due to a plateau in the
positive effects of usage (i.e., “dosage”) on behavioral outcomes. They suggest that beyond the
amount of usage needed to achieve behavior change, more usage would not result in additional
benefits to the user.
Similarly, Perski and colleagues suggest that there may be an “optimal dose” of
interaction with digital behavior change interventions that can explain intervention outcomes
26
.
The conclusion that more usage is not necessarily better for intervention outcomes is made more
clear in research where declines in usage were observed as individuals progressed towards their
behavior change goals
18,60
. Furthermore, the notion of an optimal or sufficient dose to achieve
outcomes is consistent with available models of user engagement that specify a recursive
relationship between behavioral outcomes and intervention usage over time
5,26
. This means that
92
users’ progress towards their behavior change goal likely influences their subsequent interactions
with the intervention tool.
In contrast to the assumption that greater levels of usage necessarily result in intended
intervention effects, Yardley and colleagues have recently proposed that “effective engagement”
may be more important for achieving intervention outcomes
6
. Effective engagement
encompasses the specific levels of usage and thus, exposure to intervention content, which are
sufficient to mediate intended behavior change. For example, effective engagement may occur
without individuals actively interacting with a system over an extended period of time
6
. This is
the case, for example, when individuals enact behavior regulation strategies outside the scope of
an app (e.g., nicotine replacement patches while using a self-monitoring based quit app) and
therefore do not need additional support from the intervention tool.
Effective engagement is hypothesized to be associated with development of sufficient
mastery over changing one’s behaviors, consequently reducing users’ need for support from the
intervention
6
. Once users have achieved mastery over their behavior change process, usage or
interaction with the intervention may not yield additional benefits to behavioral outcomes. Under
some circumstances, continued engagement may in fact lead to unanticipated adverse outcomes.
For example, Smith et al. describe four distinct states of user engagement an individual might
experience with their quit app, one of which they term “counterproductive engagement”. In
counterproductive engagement, users consistently interacted with the app’s cessation resources
but developed antipathy and other negative emotions towards the app and their quit attempt
18
.
93
Existing research on effective engagement with quit apps
According to Yardley and colleagues, what constitutes effective engagement is specific to
an intervention and the context of its use and thus, can only be defined empirically with respect
to that intervention
6
. In terms of smoking cessation specifically, two studies have examined
interaction with specific app features as they relate to quit outcomes
41,42
. First, Heffner and
colleagues reported that out of the ten features on the SmartQuit app that were examined, only
two were associated with abstinence after 60 days: high (vs. low) usage of (1) viewing quit plan
and (2) tracking practice of letting urges pass
42
. Notably, the authors showed that even though
certain app features were used more than others, use of these popular features did not predict
abstinence. For instance, the strongest predictor, tracking the practice of letting urges pass, was
used by only half of the participants
42
. This study therefore implies that only certain subsets of
features may explain intervention outcomes and that the effect of these features may not
necessarily have to do with how frequently they are used.
In another study, Heminger and colleagues evaluated a text-based cessation program
called Text2Quit, which provided interactive, individually tailored text messages to individuals
encouraging them to track their smoking and report on their cravings. Participants could
additionally request help from the system using text message keywords to view their progress
and report a lapse.
41
. The sum of all these interactions reflected intervention “dose”. Study
findings revealed that overall dose was not associated with outcomes. Instead, participants who
texted the keyword “pledge” or responded to a prompt to report their smoking status were more
likely to be abstinent at 6-month follow-up
41
. The authors suggested that these particular
interactions may have been an indication of a participant’s willingness to quit
41
. This conclusion
94
highlights that the subjective experience of app interactions may be additionally important for
understanding how and why specific features are associated with intervention outcomes.
In combination, the two aforementioned studies highlight that aggregate measures of
usage typically used in behavior change research may be insufficient to identify 1) which
specific sets of app features lead to positive cessation outcomes and 2) the mechanisms by which
interacting with app features facilitates intended behavior change goals. Focusing on the specific
app features with which users are interacting allows us to better understand which usage patterns
might lead to smoking cessation and the specific behavioral and psychological processes that are
initiated during interaction.
UE patterns that might represent effective engagement
While it is possible to test the association between every app feature and intervention
outcome, this process can be cumbersome as the number of app features designed to support
behavior change increases. Furthermore, the specific combinations of app features one interacts
with and their potential synergy may also represent usage patterns that could be considered
effective engagement and would thus be overlooked
37
. For example, greater usage of a self-
monitoring feature may on its own not be associated with quitting, but when coupled with active
use of goal setting features, may be positively associated with cessation outcomes. Researchers
should therefore consider exploring “patterns” of interactions across diverse app features, as an
alternative to individual interactions, to predict behavior change outcomes. Here, we use the term
“UE pattern” to describe combinations of interactions across multiple available app features.
Exploration of these patterns may add to our understanding of what constitutes forms of usage
constitute effective engagement.
95
There is preliminary evidence that investigating UE as patterns of interaction with diverse
app features in relation to intervention outcomes is useful. Donkin and colleagues report that of
four measures of program use, only the average number of features a user interacted with during
each log-in was associated with reduced depression symptoms
40
. Specifically, those who
achieved intended intervention outcomes had interacted with a more diverse set of intervention
features per log-in. The authors suggest that this pattern of highly active interaction within log-in
sessions may reflect optimal usage of the intervention. Similarly, Couper and colleagues
demonstrated that whereas the total amount of time spent interacting with intervention features
was not associated with dietary outcomes, the extent to which a user interacted with all available
intervention features (i.e., breadth) was
22
. For these studies, interaction with a breadth of diverse
app features appears to represent effective engagement.
Mechanisms underlying effective engagement
In Study 2, we identified underlying subsets of day-level UE patterns comprised of
interactions with combinations of MQU features. These UE patterns thus represent candidate
patterns for assessing effective engagement with MQU. Once a particular UE pattern is identified
as effective engagement, clear understanding of the specific processes through which a given UE
pattern mediates lapse avoidance is still needed
5
. Michie and colleagues have recently referred
to these processes as “mechanisms of action” and assert that effective digital behavior change
interventions target relevant mechanisms to drive behavior change
97
. Examples of behavioral
and psychological mechanisms of action include changes to a user’s knowledge, beliefs, ability,
and motivation to modify their behavior
1,5
. As day-level UE patterns are comprised of
combinations of feature interactions, we can begin identifying relevant mechanisms of lapse
96
avoidance by examining how users perceived each of the features supported lapse avoidance, if
at all.
Efforts have been made recently to evaluate the link between 26 commonly studied
mechanisms of action and various behavioral intervention strategies
97
. For example, there is
strong evidence linking motivation for behavior change, one type of mechanism of action, with
interventions that provide feedback on behavior
98
. While one might assume that a given feature
is associated with a particular mechanism of action for lapse avoidance based on theoretical
knowledge
97
, it is possible that the feature could facilitate quitting in unexpected ways. For
example, in Study 1, we described how some users interpreted II reminders as cues for enacting
their IIs (as we had intended), whereas others interpreted them as general reminders that they
were attempting to quit. To address this potential limitation, Yardley and colleagues have
strongly encouraged the use of qualitative data to understand relevant behavioral and
psychological processes that could have helped users achieve their intended behavioral change
goals.
6,18,61
. With respect to technology-based cessation interventions, research on mechanisms
of lapse avoidance is sparse and none to our knowledge have been conducted on quit apps
9
.
Present study
Therefore, the present mixed methods study combines quantitative and qualitative data
sources to investigate specific day-level UE patterns that may reflect effective engagement with
MQU, along with perceived mechanisms of lapse avoidance. Effective engagement is defined for
the purpose of this study as daily patterns of UE associated with reduced probability of lapsing,
relative to other patterns. To achieve this objective, we first extended multilevel latent class
analyses (MLCA) conducted in Study 2 to examine the association between day-level UE
97
patterns and lapse on the same day. Data from semi-structured interviews were then examined to
identify specific cognitive and affective processes that represent mechanisms of lapse avoidance.
Study 3: Specific Aims
The specific aims and hypotheses for this study are as follows:
Aim 1: To examine whether certain latent classes of day-level UE patterns associated with
reduced probability of same-day lapse, i.e., effective engagement patterns.
H1: Days characterized by a highly active pattern of UE with MQU (interaction with all
available MQU features), would be associated with reduced probability of same-day lapse,
compared to days characterized by a less active pattern of UE.
Aim 2: To describe the behavioral, cognitive, and affective mechanisms cued by UE that
facilitated users’ cessation outcome.
H2: Users would describe varied behavioral, cognitive, and affective mechanisms with
respect to interaction with different app features when describing how MQU helped them
avoid lapse.
98
Methods
Quantitative Methods and Analytic Plan: MLCA with lapse outcome
We used quantitative data to investigate effective engagement with MQU via MLCA.
These data were drawn from the same user logs analyzed in Studies 1 and 2, which provided
time stamped records of MQU feature interactions. Each user was instructed to use MQU for 28
days, resulting in a total of 1596 maximum observation days (N=57*28=1596 days). At the end
of each day during the intervention period, participants responded to “How many cigarettes did
you smoke today?” in a summary “end of day” (EOD) EMA. A dichotomous variable indicating
whether the user smoked at least 1 cigarette that day served as our lapse outcome.
As described in detail in Study 2, variables representing the daily interactions with each
MQU feature per day served as multivariate indicators of underlying latent classes of day-level
UE patterns
91
. These indicators were binary variables representing whether each feature
interaction occurred at least once that day. The step-wise approach to MLCA from Study 2 was
repeated to extract latent classes, but modified to exclude EOD assessment as a latent class
indicator. This is because our lapse outcome variable is assessed within the EOD EMA and thus
the two variables would be highly correlated. This correlation would violate the assumption that
within latent classes, the observed indicators are independent from covariates
99
. The remaining
5 feature interactions were 3 PULL feature interactions: 1) reporting lapse, 2) reporting craving,
3) checking progress and 2 PUSH feature interactions: 4) momentary EMA adherence and 5)
acknowledging II reminders.
As described in Study 2, we conducted MLCA by first specifying a series of single level
latent class models that ignored clustering of days within users. After a best fitting model was
selected using model fit indices (BIC and entropy), we adjusted for the clustering by introducing
99
random means to the selected model. Random means allowed for the probability of a day-level
UE pattern belonging to a given class to vary across users. Finally, we introduced the EOD lapse
variable as a day-level covariate to assess its association with day-level UE patterns as has been
previously recommended by Lanza and colleagues
100
.
Figure 4-1 depicts an example 3-class MLCA model with 5 latent class indicators (UE1-
5). This model is similar to the example model described in Study 2 and is described in complete
detail there. An additional observed variable X is included at the day-level in Figure 1 to
represent the lapse outcome. At the user level, two random means for the probability of
belonging to a given latent class are specified as C#1 and C#2. C#1 allows the log odds of
belonging to latent class 1 relative to latent class 3 to vary across users. C#2 allows the log odds
of belonging to latent class 2 relative to latent class 3 to vary between users. The random mean
for latent class 3 is set to 0 and is therefore not included in the figure
91
. All analyses were
conducted using Mplus v.6 with default settings: maximum likelihood estimators and robust
standard errors (Mplus, Los Angeles, CA
92
).
Figure 4-1. Example 3-class multilevel latent class model with lapse outcome
100
Qualitative Methods and Analytic Plan
Qualitative data for this study were drawn from the original sample of N=55 semi-
structured interviews conducted as described in Study 1. Briefly, all users were invited to
participate in 30-60 minute interviews within 1 month after completing the 4-week quit attempt.
The interviews explored specific ways in which participants engaged with MQU, including
cognitive and affective responses related to app interaction, facilitators and barriers to app use,
and temporal changes in UE dimensions.
For the purpose of the present study, a subset of probes were used to explore the extent to
which users perceived interacting with MQU features facilitated quitting, if at all. Of particular
interest were specific cognitions and emotions that users perceived to lead to lapse avoidance but
were not captured in user logs. Codes were only applied to quotes if a user explicitly mentioned a
specific mechanism for avoiding lapse or if the quote was in response to a question about which
particular aspects of MQU helped users to avoid smoking. Additional emphasis was placed on
understanding whether and how non-MQU factors may have also influenced lapse outcomes in
an attempt to isolate lapse avoidance mechanisms directly attributable to the MQU.
An inductive approach was used to generate open codes from the first 7 interviews. These
open codes were then compared to the 26 predefined mechanisms of actions proposed by Michie
and colleagues and refined for coding the remaining interviews
97
. Additional codes were created
as coding continued, which allowed unexpected mechanisms of lapse avoidance to emerge.
Where relevant, each mechanism was linked to a specific MQU feature using preconceived,
deductive codes representing the MQU feature. For those mechanisms that were not associated
with an MQU feature, no additional code was used.
101
A subset of 6 interviews was selected at random after half of the interviews were coded to
establish inter-rater reliability and to refine the coding scheme. CC and SY discussed codes and
emergent themes iteratively to reach consensus on code meaning. All available audio recorded
interviews were transcribed, coded, and analyzed using Atlas.ti V7.5. Thematic analysis was
used to identify and analyze patterns in responses across interviews. Quotes that captured similar
mechanisms for lapse avoidance were grouped together to constitute themes
80
. As part of our
mixed methods approach, lapse avoidance mechanisms identified from the interviews were used
to conjecture about how each indicator used in MLCA (i.e., MQU feature interaction) might
have influenced lapse avoidance
78
.
Results
MCLA results
Table 4-1. Single level LCA and MLCA fit indices; n=1582 days, N=57 users
1-class 2-class 3-class 4-class 5-class
Single level LCA
BIC 9282.98 8941.87 8849.47 8862.11 8899.60
Entropy - .61 .68 .83 .69
VLMR Test - p <.001 p <.001 p <.001 p = .50
MLCA
BIC - 8470.14 8107.28 - -
Entropy - .75 .80 - -
As was the case in Study 2, the 3-class MLCA solution with 5 indicators yielded the best
fitting model (Table 4-1). Although a 4-class solution initially appeared to be more favorable
among single level LCA models based on greater entropy and significant VLMR test, the best
102
log-likelihood failed to replicate for the 4-class solution in MLCA after 10,000 random starts.
Importantly, the 3-class solution provided substantively meaningful interpretation. Thus, we
selected the 3-class solution as best fitting.
The lapse outcome was available from EOD assessments on 62% of days (n=981 of 1582
days). Therefore, the association between day-level UE patterns and lapse was examined only on
this subset of days. On days where lapse information was available, users reported at least one
lapse on 508 days (52%).
As discussed in Study 2, latent class formation may be affected when covariates are
introduced to a base MLCA model without covariates
94
. Tables 4-2 and 4-3 provide a
comparison of the estimated prevalence for each latent class and conditional probabilities for
indicators in a model where lapse was not included as a covariate and one where lapse was
included. Similar to Study 2, we note that class prevalence and conditional probabilities remain
similar to those in the base model without lapse. We therefore examine the association between
day-level UE patterns and same-day lapse using the model with lapse as a covariate (Table 4-3).
As can be seen in Table 4-3, the largest class in the 3-class solution was characterized by
low conditional probability of interaction with all five MQU features (16%-24%). This day-level
UE pattern was identified as “Low UE” and described 42% of the days. In contrast, the smallest
class (22% of the days) showed higher probability of interaction with all MQU features
compared to the Low UE class. The greatest difference in conditional probability between this
class and the Low UE class was for checking ones’ progress (93% vs. 22%). We identified this
class as “Active UE” because compared to the other two classes, it had the highest probability of
PULL feature interactions that required user initiative for re-engagement. The final latent
103
Table 4-2. 3-class MLCA solution with conditional probabilities for indicators
without lapse covariate
Class 1 Class 2 Class 3 Overall
Indicator for MQU feature 43%
Low UE
23%
Active UE
35%
Passive UE
N=981
PUSH feature interactions
Acknowledging II
Reminders (≥3 reminders)
21% 62% 73% 48%
Momentary EMA adherence
(≥4 EMAs)
25% 79% 97% 62%
PULL feature interactions
Reporting craving 19% 40% 11% 21%
Reporting lapse 16% 29% 20% 20%
Checking progress 20% 94% 12% 34%
Table 4-3. 3-class MCLA solution with conditional probabilities for indicators with lapse
covariate
Class 1 Class 2 Class 3 Overall
Indicator for MQU feature 42%
Low UE
22%
Active UE
35%
Passive UE
N=981
PUSH feature interactions
Acknowledging II
Reminders (≥3 reminders)
20% 64% 72% 48%
Momentary EMA adherence
(≥4 EMAs)
24% 79% 96% 62%
PULL feature interactions
Reporting craving 19% 40% 11% 21%
Reporting lapse 16% 31% 19% 20%
Checking progress 22% 93% 11% 34%
104
class showed that days in this UE pattern had high probability of PUSH feature interactions:
acknowledging II reminders (72%) and adherence to momentary EMA assessments (96%). For
PULL feature interactions, however, this pattern showed the lowest probability of reporting a
craving and checking progress compared to the previous two classes (11% for both feature
interactions). Days that belonged to this UE pattern were identified as “Passive UE” and
characterized 35% of days.
Association between day-level UE patterns and lapse outcome
Relative to Passive UE days , Active UE days were associated with greater odds of same-
day lapse (log odds (SE) = 1.88 (.79), p = .02). Days belonging to the Active UE pattern also
showed greater odds of same-day lapse compared to days in the Low UE pattern (log odds (SE)
= 1.98 (.91), p=.03). Interpreted as an OR, the odds of same-day lapse for Active UE days was
6.5 times greater relative to Passive UE days and 7.3 times greater relative to Low UE days.
Relative to Passive UE days, the odds of lapse for Low UE days was slightly lower, but
not significantly so (log odds (SE) = -.12 (.76), p = .88; OR = .89). Because the probability of
same-day lapse was lower for both Passive UE and Low UE days relative to Active UE days, our
findings suggest that these two specific day-level UE patterns represent effective engagement
with MQU, according to the definition provided by Yardley and colleagues
6
.
105
Qualitative Results
Interview data were used to explore the specific mechanisms through which day-level UE
patterns could have influenced same-day lapse outcomes. This was accomplished by probing
users about the specific thoughts, feelings, or behaviors associated with each MQU feature that
they believed helped them to avoid lapse. Findings were then used to conjecture about which
specific lapse avoidance mechanisms likely occurred during each day-level UE pattern identified
from our MLCA model. In addition to lapse avoidance mechanisms that were directly
attributable to MQU, interview questions and follow up probes also elucidated external
mechanisms and non-MQU contextual factors that helped users to avoid lapses.
Five themes were identified from the interviews, each of which described a specific
mechanism for lapse avoidance. The primary mechanisms identified were (1 & 2) behavioral
regulation (two types), (3) feedback processes, (4) motivations to quit smoking, and (5) non-
MQU mechanisms. Behavioral regulation was divided into lapse avoidance strategies that users
planned at baseline to enact during high risk smoking situations and those that were not planned
ahead of time. Each quote coded reflects at least one instance where the mechanism occurred
during users’ quit attempt, as the particular mechanism may have occurred repeatedly during the
intervention period within one user.
Behavioral regulation: II enactment (n=111 quotes)
This theme describes II enactment, pre-planned behavioral regulation skills chosen by
users to perform during high-risk smoking situations (HRSS) in order to avoid lapsing
56
. Users
varied in the range of IIs they were prompted to perform by MQU as each user could choose to
compile their own personalized IIs or select ones from a list of suggested IIs. The most
106
commonly enacted IIs according to the interviews included staying away from other smokers,
leaving cigarettes out of reach (at home or with a friend), engaging in physical activity,
performing morning activities such as brushing teeth or showering, and substituting cigarettes
with a snack or beverage.
Most users (n=47) described at least one instance during their quit attempt where they
performed the specific II associated with their scheduled HRSS. Of these 47 users, 34 described
II enactment at least once in direct response to receiving and acknowledging an II reminder. We
note here that II enactment during HRSS was the primary lapse avoidance mechanism MQU was
designed to facilitate. For some, the II reminder came “just in time”, immediately prior to the
HRSS. The user below would not have enacted his II if not for the reminder.
“It came literally at the right time. They were literally saying let’s go out and smoke and my
phone rang and I’m going to stay inside. Another time, I was actually outside and about to pull
[a cigarette] out and the phone came in and I put it back.” – U6, M
Approximately half of the quotes in this theme did not explicitly describe II enactment in
direct response to receiving the II reminders. One likely explanation is that users had become
accustomed to performing IIs as part their everyday routine. As a result of II enactment
becoming a “habit”, users felt that they no longer needed the reminders.
“I think I was already pretty well aware about my plan. So even without the reminders, if I were
going out to a bar, I would leave my cigarettes at home. Since I made these plans myself I was
107
really aware of what to do in certain situations and I would complete them before the
reminders.” –U22, F
“It started more, started to be like a habit, so when I felt like I needed a cigarette, I would just
use the three [IIs] myself.” – U50, M
To a lesser extent, some users (n=9) described II enactment in response to pressing the "I
want to smoke!" button (report craving feature). This feature was also designed to provide II
reminders and cue II enactment, but during times not identified at baseline as high-risk smoking
situations, e.g., spontaneous instances of craving.
“I clicked the button for “I want to smoke!” right now […] even though I thought it was kind of
weird that the app was going to tell me to get up and go walk right now, I did it even though it
sounded silly, […] I got up and I walked. And I did resist the cigarette.” – U36, M
Behavioral regulation: non-II (n=96 quotes)
In contrast to the previous theme, this theme encompassed behavioral regulation skills
that were not initially planned at baseline, and thus, not IIs. Forty users described enacting such
“non-II” skills in 96 quotes, approximately the same number of quotes related to II enactment.
Interestingly, about a third (35/96) of these non-II quotes were associated with II reminders and
"I want to smoke!" interactions, illustrating how receiving II reminders did not always cue the
intended II plan for users. Non-IIs included self-talk or reflection on motivation to quit
(described by U53), not buying cigarette packs, and consuming food or beverages to substitute
108
smoking. Others simply relied on their own “willpower” to make a conscious decision not to
smoke in the moment.
“I would just come up with random reasons, just to stop myself from hitting it at that moment.
Like oh it’s going to taste so bad, my fingers are going to start stinking like cigarettes, and I
have to go home soon and I don’t want to be smelling like that.” – U53, M
“I’m outside with people and then I get […] “I’ll stay inside where smoking’s not allowed”. You
know we might not actually physically do that where we stay inside. I might go out with them but
maybe step to the side. It wasn’t like always the literal action of what it reminded me to do. But it
was enough of a reminder for me to maybe step to the side or you know, congregate closer to the
people who don’t smoke in the group” –U8, M
Users offered a few reasons for not enacting their specified IIs. Some reasons included
perceiving a given II as not possible or not ideal to enact in the moment and being unwilling to
enact a specified II repeatedly over time.
“I didn’t feel like doing [the II] when it showed up, because it doesn’t change. It’s just that same
one. […] Sometimes I would want to be alone so I would just watch my show.” – U19, F
“Tell your coworkers that you’re staying away from the place they smoke at’ […] It’s okay the
first time the alert shows, but then after 30th time, it becomes meaningless. Like I already did.” –
U27, M
109
Using vape pens was a common behavioral regulation strategy. Several individuals
described using “JUULs” specifically, a novel and highly popular type of vape pen that delivers
nicotine at faster rates than other electronic nicotine delivery systems
101
. Twelve users reported
vaping at least once during the quit attempt with a subset using vape pens as their primary
strategy to “replace” cigarettes. Only 3 users had planned to vape as an II while the rest initiated
vaping just before or after the quit attempt began.
“I started doing this 1 or 2 days before I started this [study] and JUUL changed my cigarette
usage a lot. I don’t really smoke cigarettes anymore.” –U24, M
Feedback processes (n=56 quotes)
This theme described feedback processes as a result of interaction with MQU. Feedback
processes refer to the comparison of current behavior with a standard
97
. For the first feedback
process, awareness of smoking habits, users compared the smoking frequency logged on
MyProgress and EMA assessments with perceptions of their own smoking habits. Nineteen users
reported greater awareness of their smoking frequency and being more “conscious” of social and
emotional antecedents of their smoking events (U43). Many of these users, also commented on
realizing that their smoking frequency was greater than they had previously thought.
“I just feel more […] conscious of the fact that that I was smoking I guess. […] I didn’t realize
how much of a habit it was…the smoking and the vaping and how prevalent it was to me
throughout my schedule.” U43, M
110
“Oh shoot, I resisted like 10 [cigarettes], but I also smoked like 5 today. In total I would have
smoked like 15 cigarettes. It helped me realize that it was a lot of cigarettes I smoked.” – U36, M
A second feedback process was related to awareness of one’s quit progress. Of the 13
users reporting that MQU was associated with increased awareness of their quit progress, the
majority attributed their awareness to checking MyProgress. MyProgress helped users to confirm
that they were on track towards their quit goal and provided positive reinforcement to continue
the quit attempt for some.
“Looking at the [MyProgress] number just made me feel proud…oh that’s how many cigarettes I
resisted today”. – U47, F
Although relatively rare, a few quotes within this theme suggested that feedback
processes may have been associated with decreased lapsing via self-efficacy for quitting. Four
users described how seeing their progress specifically helped them feel more confident about
continuing to avoid smoking.
“To see the actual concrete numbers really helped me solidify that I can do this. […] If I keep
looking at these stats and see how much money I saved and how many cigarettes I didn’t
smoke.” U53, M
“On the fifth day, I went back to the first two days progress, right. I reminded myself, “Oh, I
resisted 8 cigarettes. I can resist again.“ –U1, M
111
Motivations to quit smoking (n=76 quotes)
User’s increased motivation to quit smoking represents another common mechanism
through which MQU facilitated quitting. This theme describes how specific MQU features made
users’ “purpose, ” (i.e., motivation) for their quit attempt more salient
97
. This is in contrast to
more general motivations users reported for participating in the study such as improving their
health. For some users, interacting with MQU instilled or strengthened motivations to take their
quit attempt more seriously than they have in the past. For example, the two users below describe
how MyProgress and II reminders, respectively, fostered the prerequisite motivation and
“mindset” to quit.
“It was shocking to see that my habit had grown in everyday lifestyle. I thought I was smoking 6
cigarettes a day, but I was truly smoking about 10. Therefore I knew I had to cut down.” – U45,
F
“Through the years, I’ve just been telling myself, you should quit, but it hasn’t been written
down or nobody’s been telling me. And since the app was doing that, like written down, “Yo, this
is exactly what you wrote,” kind of like a contract […] I was like, “Yea, I really need to quit.” –
U50, M
Once the quit attempt had started, 39 users described how II reminders, MyProgress, and
EMA assessments motivated them to continue. Some described how MyProgress motivated them
to continue challenging themselves to cut down on smoking. Being able to report non-smoking
events on EMA assessments was also motivating according to the user below.
112
“The achievement really becomes my motivation again, of quitting smoking […] whenever I got
the reminder that you usually smoke at this time and I said “No”. – U25, M
A notable finding was that although II reminders did not always cue II enactment as had
been the intention of MQU, reminders nevertheless cued users’ motivations to quit and fostered
perceptions of accountability. Fifteen users interpreted II reminders as “general” or “constant”
reminders that they had made a commitment to quit, which eventually became psychologically
salient, i.e., “engrained”.
“Just being aware that you were going to be reminded […] definitely kept me in check and
accountable. I definitely smoked a lot less with those reminders. I thought it was a very good
physical tangent [sic] thing for me to have accountability. Not really in a personal way but it’s
good to have those constant reminders that you’re trying to quit. – U22, F
“Even though specifically [II reminders] didn’t directly help, like I should go to the gym so I
won’t smoke, it got me to think more often throughout the day. This is something I want for me;
it’s a personal goal that I want to quit smoking. It got me to think more of who I surround myself
with and the environment I put myself in to smoke and if I even enjoy smoking, or if it’s like
social for me. I think like maybe a week or two after the quit date, that thought really solidified.”
–U46, F
113
Non-MQU mechanisms: Environmental context and decreased craving (n=76 quotes)
Forty-one users described at least one environmental context that facilitated lapse
avoidance but did not involve behavioral regulation or interaction with MQU. These
environmental constraints included momentary availability to smoke and the influence of friends
and family. On some days, users felt they were too busy with work or school and that it wasn’t
“worth it” to smoke. Others described being in social situations where smoking was not possible
(spending time with family) or instances where their friends and family provided encouragement
to avoid smoking or were actively monitoring their smoking.
Several users described additional contextual factors that limited their access to
cigarettes. These non-MQU factors included increases in cost of cigarettes, an increase in age
limit to buy cigarettes, and one university campus adopting a smoke-free policy, all of which
occurred during the course of the active study period.
One of the suggested activities was “You should stay in to avoid, stay in a place where you can’t
smoke anything”, and when they asked you if it’s useful, most of the time it’s “No it’s not useful
because [campus] is just non-smoking”. I can’t do anything here. The number one thing that
prevented me from smoking was just not being able to access any cigarettes.” U15, M
A minority of users (n=14), such as U42, described perceived reduction in craving to
smoke or withdrawal symptoms during the course of the quit attempt. This was the case for users
who spent time with non-smoker friends. Nine users reported one reason they reduced their
smoking was because they no longer enjoyed the smell and taste of cigarettes.
114
“Towards the end I was used to not smoking...after a while I realized that even if I don’t smoke
then it’s not really a big deal. I don’t have withdrawals or anything and I felt good about all of
this progress that I’m making, so towards the end when a survey comes up, I just don’t really
mind or think about it because I’m not going to smoke.” U42, M
Table 4-4 presents a summary of the themes and corresponding lapse avoidance
mechanisms identified from the interviews. In summary, users described a broad range of
behavioral, cognitive, and affective mechanisms through which MQU usage helped them to
avoid lapse. In many cases, the mechanisms that supported lapse avoidance were triggered by
interaction with specific MQU features. For example, acknowledging II reminders and reporting
craving cued thoughts about behavioral regulation strategies personally-tailored to certain HRSS
(i.e., implementation intention enactment) as intended in MQU’s design. Checking one’s
progress was associated with feedback processes about users’ quit progress and pride, which was
important for boosting some users’ motivation to quit.
However, behavioral and psychological mechanisms for lapse avoidance also arose in
unexpected ways. For example, some users reported that II reminders cued unplanned or
alternative behavior regulation strategies or simply made user’s motivation to quit more salient
without cueing a specific action. EMAs, for instance, which were not intentionally designed as
an intervention feature, cued feedback processes about users’ smoking habits and quitting
progress. Finally, a subset of lapse avoidance mechanisms was ostensibly unrelated to MQU,
such as non-II behavior regulation strategies that were not cued by II reminders and social and
environmental contexts that limited users’ ability to smoke. Taken together, although MQU
triggered behavioral and psychological mechanisms that facilitated lapse avoidance, non-MQU
115
mechanisms played a similarly large role in supporting user’s quit attempts. We would not have
been able to systematically uncover these non-MQU mechanisms without interview data.
Table 4-4. Themes representing mechanisms of lapse avoidance
Themes MQU feature Examples from interviews
1. Behavioral Regulation
II enactment
II reminders; report
cravings
Enact pre-planned behavioral
regulation plans (JIT and non-JIT)
2. Behavioral Regulation
non-II enactment
II reminders Alternative lapse avoidance
strategies: napping, vaping, eating
3. Feedback Processes MyProgress; EMAs Awareness of current smoking
frequency or quit progress thus far
4. Increased motivation
to quit
II reminders
MyProgress
Reasons one “should” quit,
strengthened resolve to continue
quitting
5. Non-MQU
mechanisms
NA Too busy to smoke, social influence,
no cigarettes available
Discussion
In this mixed methods study, we investigated the extent to which day-level UE patterns
were associated with reduced probability of same-day lapse, thus representing effective
engagement with MQU. We additionally sought to uncover specific behavioral, cognitive, and
affective processes associated with feature interaction that represented mechanisms for lapse
avoidance. Additionally, whether these mechanisms were associated with any of the 5 MQU
features (i.e., MLCA indicators) was noted where appropriate. Whereas the MCLA approach was
used to identify UE patterns with reduced probability of lapse, the qualitative approach was
intended to explore the mechanisms through which such patterns facilitated lapse avoidance.
116
Results from our MLCA models showed that Active UE pattern days were associated
with greater odds of same-day lapse compared to both the Passive UE and Low UE pattern days.
These results were contrary to the initial hypothesis that days characterized by high probability
of interaction with a diverse set of MQU features, as is the case in the Active UE class, would be
associated with reduced lapse
22
. One possible explanation is that Active UE represents a
prerequisite UE pattern where users are exposed to app features that are useful when they are not
yet abstinent. This interpretation can be evaluated by considering the conditional probabilities of
feature interactions identified in our MLCA. Specifically, we showed that the conditional
probabilities for checking progress, reporting craving, and reporting were highest during Active
UE days compared to other patterns. According to our qualitative data, checking one’s progress
was commonly associated with cognitive mechanisms such as feedback processes and
motivation for quitting smoking. For example, some users described how becoming aware of the
number of cigarettes they actually smoked via MyProgress strengthened their motivation or
“need” to quit. Given these findings, we might infer that Active UE days were associated with
greater probability of lapse because during Active UE days, users were still becoming aware of
their smoking habits and building their motivation to quit, although not yet abstinent.
According to our a priori operationalization of effective engagement, Passive and Low
UE daily patterns represented effective engagement with MQU. Inspection of conditional
probabilities for feature interaction was useful for interpreting how the Passive UE class was
associated with lower odds of lapsing. Passive UE days were associated with the highest
probability across all latent classes of acknowledging II reminders. Consequently, on these days,
users were more frequently exposed to cues of their pre-planned behavioral regulation strategies.
Indeed, interview data corroborates that behavioral regulation was the most commonly reported
117
mechanism of lapse avoidance associated with II reminders. In at least half of the quotes
pertaining to behavioral regulation, II reminders cued users to enact either their pre-planned II or
some other alternative behavioral regulation strategy to avoid smoking. Another cognitive lapse
avoidance mechanism associated with II reminders was increased salience of users’ motivation
to quit. When users were no longer enacting their IIs or any other behavioral regulation
strategies, II reminders boosted user’ commitment to their quitting goals. Thus, it is reasonable to
conclude that on Passive UE days where users tended to acknowledge more II reminders, they
were more likely to be reminded of their IIs or their motivation to quit, subsequently leading to
lapse avoidance.
The final and most prevalent (48%) day-level UE pattern is notable. Days in the Low UE
pattern were characterized by low conditional probabilities (<25%) for interaction with all 5
MQU features analyzed and yet were still associated with lower odds of lapsing relative to the
previous two latent classes. This finding can be explained in either of the two following ways.
The first potential explanation gleaned from interview data is that over time users became
“habituated” to II enactment and thus no longer needed reminders or support from MQU as a
whole. This is consistent with theory underlying implementation intentions, where repeated
enactment of a behavior in specific situations becomes more automatic and less effortful over
time
56
. Furthermore, we note that in Study 2 that Low UE days were more probable relative to
Active UE days as the quit attempt went on. That the probability of Low UE days increased as
function of time, just as habituation is hypothesized (and described by users) to also increase
over time, provides plausible evidence that users exhibited Low UE days because they no longer
needed additional support from the app. Once mastery of behavioral regulation skills has
118
occurred, lack of re-engagement with MQU, as evidenced by the Low UE pattern, should be
considered positive and sustained engagement is no longer needed
6
.
Alternatively, the Low UE pattern could have also represented days among users who, at
no point during the quit attempt, perceived any MQU feature to be useful. Study 1 interview data
revealed that there were several such users who fit this criterion. Despite their limited interaction
with MQU features, these users reported smoking reductions in their interviews but attributed
cessation outcomes to what appeared to be non-MQU related mechanisms. These non-MQU
mechanisms included environmental constraints on smoking such as rise in cigarette costs, being
too busy with school or work to smoke, and monitoring from friends and family. Users also
described enacting unplanned, non-II behavior regulation skills that supplanted support from
MQU, such as vaping to substitute cigarette smoking. Whether we consider Low UE patterns
truly effective engagement may depend on the reason why a user disengaged with MQU features
on that day. Low UE attributed to a user gaining mastery over behavior regulation skills may be
considered effective engagement whereas Low UE attributed to perceptions of poor usability or
low perceived usefulness would not be.
Up to this point, our examination of UE patterns has focused on individual days within
users. With data visualization tools such as heat maps, we can descriptively examine the
distribution of daily UE patterns within users across time. Figure 4-4 below presents daily UE
patterns in relation to user-level cessation outcomes during the entire 4-week intervention period.
Rows represent users, columns represent day in the quit attempt, and colors represent a given
day’s most likely latent class membership based on our MLCA results. The heat map confirms
our covariate analysis suggesting that Active UE days were associated with greater probability of
same-day lapse relative to Passive and Low UE days. Visually, green cells representing Active
119
UE days appear more prevalent among users who lapsed on over 50% of the 28 days in their quit
attempt. Passive or Low UE days appear more prevalent among users from the 0% lapse group
(rows between IDs 36-50), which further supports our MLCA finding that these two daily UE
patterns represent effective engagement with MQU.
The user-level view of daily UE patterns provided by the heat map also offers helpful
information about users who, retrospectively, could have benefitted from additional support for
quitting. As can be seen in Figure 4-2, days where users were using a diverse set of app features
(Active UE days, green cells) are relatively frequent across the 28 days for about a third of the
users who lapsed on more than 50% of their days (rows between IDs 1 and 35). Days where
users exhibited Low UE throughout the intervention period were common for about half of the
users in this group (rows between ID 6 and 55). Given that Low UE days were common among
these users, even as they struggled with lapse avoidance during the study, we might conjecture
that MQU was not useful for them early on in the quit attempt. These users might have benefitted
from strategies to re-engage them after the first week or from a different type of cessation
support. In contrast, the prevalence of Low UE days among the 0% lapse group is not concerning
and we would not allocate additional resources to re-engage these users.
120
Figure 4-2. Heat map of most likely day-level UE pattern by user and day in quit attempt
ordered by lapse outcome
Implications
Our findings have implications for designing more effective future versions of MQU and
other quit apps that combine both PUSH and PULL feature interactions. Given that the Passive
UE day pattern was associated with lower likelihood of lapsing and users had not yet disengaged
with MQU, strategies to optimize feature interactions common within this pattern should be
considered. For example, some users informed us that II reminders were helpful only when the
timing of delivery correctly matched their situation they specified and they were willing to enact
R e a l I D s ( g r o . . I D
D A Y
1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8
> 5 0 % l a p s e
d a y s
1
3 7
1 1
2 2
4 9
5 2
1 0
2 4
2
4
3 9
3 5
5 6
8
4 6
5 4
6
3 2
1 7
3 4
1 6
1 5
3 3
2 5
2 6
2 3
4 8
5
5 7
1 8
5 5
< 5 0 % l a p s e
d a y s
5 1
4 1
9
1 2
2 9
2 0
3 1
4 4
2 7
4 2
2 8
5 3
4 0
1 3
2 1
0 % l a p s e
d a y s
3 6
3
1 9
1 4
3 8
4 7
4 3
4 5
7
3 0
5 0
Active UE
Passive UE
Low UE
121
the associated II. This ideal set of circumstances occurring simultaneously happened less than we
expected it to (38% of HRSS) and may have contributed to perceptions of poor app usability and
led to premature disengagement from MQU for some users
5,23
. To optimize the usability of II
reminders and therefore prolong Passive UE, MQU could periodically prompt users to adjust the
time of day in which HRSS occurred. This suggestion is based on interview data that users were
not experiencing the scheduled HRSS when the II reminder was delivered. While adjusting
HRSS was possible in MQU, several users discussed not editing the timing of HRSS because
they did not know how to or the process was too burdensome. In addition, we might also
consider tailoring the frequency of II reminder delivery in response to user reports of being
abstinent. For example, a future version of MQU might deliver II reminders at reduced
frequencies as users report fewer lapses over time, i.e. progressing towards their cessation goals
60
.
Finally, our findings prompt further thoughts on the time scale for identifying UE
patterns that represent effective engagement with MQU. In this analysis, we defined effective
engagement as specific UE patterns associated with reduced probability of same-day lapse.
However, it is possible that while Active UE may not be associated with reduced probability of
lapse outcomes on that day, it might be related to cognitive and affective processes that
eventually lead to positive cessation outcomes on subsequent days. For instance, users reported
increased awareness of smoking habits and motivation to quit in response to checking their
progress, a feature interaction shown to be common during Active UE days. As it may take
several days before some users develop a level of motivation to quit sufficient enough to
successfully avoid lapses, Active UE may be considered effective engagement at the beginning
of a quit attempt if it is associated with lapse reduction at subsequent time points, such as the
122
next instance of the HRSS or the day. Taking this into consideration, a revised definition of
effective engagement might include specification of the time scale (e.g., the same day or a future
time point) in which usage is hypothesized to lead to intervention outcomes.
Limitations
Study limitations warrant discussion. As with Study 2, our MLCA approach focused
exclusively on behavioral UE during model building. Although psychological processes related
to cognitive and affective UE dimensions e.g., increased self-efficacy for quitting, were not
included in this analysis, they represent candidate indicators for MLCA, and should be examined
in future analysis. Despite this limitation, our use of semi-structured interviews allowed us to
explore a range of lapse avoidance mechanisms associated with feature interaction, such as
feedback processes and increased motivation to quit, which would not have been possible
without interview data.
Another important limitation was our inability to make causal inferences regarding the
association between specific day-level UE patterns and probability of lapsing. This was because
our lapse outcome was assessed on the same day as the day-level UE pattern and we were unable
to determine specifically when it was that the lapse occurred during the day. Consequently, we
are unable to draw strong conclusions about whether Passive UE led to reduced probability of
lapsing that day or whether one’s likelihood of lapsing that day influenced an individual to
exhibit Passive UE. To address this limitation, future studies should consider real-time, passive
assessment of lapsing to help improve precision in evaluating the micro-temporal relationship
between UE patterns and cessation outcomes.
123
A final limitation is that effective engagement has been argued to be dependent on a
given intervention’s behavior change goals, the context it is delivered, and characteristics of the
intended users. Therefore, the subsets of day-level UE patterns that we identified to reflect
effective engagement may be highly specific to apps like MQU that combine PUSH and PULL
elements
6
. Notwithstanding, future studies on user engagement can build upon this work and
investigate whether the three UE patterns we identified exist for other quit apps and whether they
are similarly associated with intervention outcomes.
Conclusion
To our knowledge, this is the first study to combine MLCA with interview data to
identify effective engagement with a quit app. Using a mixed methods study design, we
illustrated the utility of MLCA for identifying subgroups of daily UE patterns associated with
reduced lapse probability and further supplemented our understanding of why a particular day-
level UE pattern might represent effective engagement with interview data. Specifically,
interview themes helped to uncover relevant behavioral, cognitive, and affective mechanisms
through which each day-level UE pattern may have led to lapse avoidance. Until now, limited
attention has been placed on the mechanisms of action associated with quit apps and our study
addresses this research gap by considering users’ subjective experience of effective engagement.
Taken together, findings from this study can be used to inform the design of future versions of
MQU that can dynamically intervene to promote specific daily UE patterns and app features that
are hypothesized to initiate lapse avoidance mechanisms.
124
Chapter 5: Summary and Conclusions
Summary of aims and findings
The studies in this dissertation aimed to address gaps in the behavior change literature
regarding user engagement in the context of an app-supported cessation attempt. These gaps
included 1) limited research on how usage and subjective experience of a quit app across diverse
app features unfolds over time, 2) lack of research about the varied usage patterns with a quit app
that users might display within themselves across a quit attempt and 3) lack of formal
investigation on what specific usage patterns constitute effective engagement with a quit app. To
address these aforementioned gaps, this dissertation examined behavioral, cognitive, affective
and temporal dimensions of UE with MQU using a mixed methods approach. The following
section briefly summarizes key findings from the three studies.
First, in Study 1, temporal trends in behavioral UE (usage) across MQU features were
examined over the quit attempt. Descriptive analysis and visualization of behavioral UE plots
showed that the percentage of users reporting lapse, craving, and checking progress decreased
over time, indicating that some users had disengaged with these PULL features without re-
engaging at a subsequent time point. In contrast, the percentage of users interacting with EOD
EMAs, momentary EMAs, and acknowledging II reminders were relatively consistent across
weeks in the quit attempt. This finding indicates that on average, users continued to re-engage
with these PUSH features over time.
Our qualitative data highlighted users’ varied cognitive and affective responses to
interaction with each MQU feature and provided explanations for disengagement and re-
engagement observed in behavioral UE engagement plots. For example, triangulation of
125
interview data with user logs identified the second week as an important time point during which
users’ thoughts and motivations regarding feature interaction corresponded to notable changes in
their own user-level engagement plots. These cognitions included perceptions about the
usefulness of II reminders and strengthened motivation for making a quit attempt. Furthermore,
our engagement plots and interview data, in combination, suggested that patterns of interaction
could vary day-to-day within users. For example, there were days where users interacted with a
diverse set of MQU features and a less diverse set (or no features) on another day. Interview data
suggested that these within-user patterns were associated with users’ progress towards their quit
goals and users’ availability to interact with MQU, e.g., being too busy to with work or being in
the presence of friends.
Second, in Study 2, we identified 3 distinct day-level UE patterns using MLCA that
represented days within users across the quit attempt: Low UE days, Active UE days, and
Passive UE days. By investigating UE patterns rather than individual feature interactions, we
extended conceptualization of behavioral UE beyond total number of daily interactions and
distinguished between Active UE and Passive UE days. Furthermore, MLCA allowed us to
explore within-user variation in UE patterns that users could display across days, as alluded to in
Study 1. We formally tested covariates of usage identified in Study 1 and showed that relative to
Low UE days, days characterized by interaction with a diverse set of features (Active UE) were
less probable as days passed in the quit attempt. We also showed that relative to Low UE days,
days characterized primarily by PUSH feature use (Passive UE) were less probable for users who
interacted with MQU via a loan phone.
Finally, in Study 3, we illustrated one possible approach for identifying effective
engagement with MQU, defined as the level of intervention usage and exposure sufficient to
126
mediate intended cessation outcomes
6
. According to this a priori definition, we showed that
relative to Active UE days, Passive and Low UE days were associated with lower probability of
lapse. This finding suggests that these two day-level UE patterns represent effective engagement
with MQU. We supplemented this analysis by investigating relevant lapse avoidance
mechanisms specific to each MQU feature. Interview data was used to uncover cognitive,
affective, and behavioral lapse avoidance mechanisms that could have influenced lapse that day.
For example, acknowledging II reminders cued users’ action plans to avoid smoking and led to
subsequent II enactment (i.e., behavioral regulation mechanisms for lapse avoidance.)
In combination, our quantitative and qualitative data sources provide a rich picture of
MQU usage and participants’ subjective experience during an intervention period. Taken
together, this information may be helpful for addressing low rates of usage and re-engagement
commonly observed with digital behavior change interventions
1,4
. Specifically, our findings
regarding the importance of perceived novelty and usefulness of II reminders, along with
effective day-level UE patterns, can inform the development of strategies that can be deployed
dynamically during the quit attempt in order to promote usage (re-engagement) with MQU over
time and additionally enhance app effectiveness
9
. Potential strategies are discussed further in the
following section.
Implications
Refining content and delivery of II reminders to increase usage
Examining users’ subjective experience with MQU was helpful for identifying which
specific factors related to the intervention’s design could benefit from additional refinement and
personalization to promote usage. Specifically, we found that perceived usefulness and novelty
127
were important factors that may have undermined re-engagement with II reminders, the primary
behavior change strategy used in MQU, for some users. Although the engagement plot (Figure 2-
5) showed that the percentage of users acknowledging II reminders was relatively consistent over
the 4 weeks, several users reported that the frequency of delivery was too burdensome and that
the specific message in the reminder became less novel over time, causing them to acknowledge
the reminder without reading the message contents. This was especially the case beginning in the
second week, after users had become habituated to enacting the associated IIs or they felt they no
longer needed the support from II reminders as they reached their cessation goals. In a scenario
where MQU was used outside the context of a research study, where there is no perceived
obligation to interact with II reminders, there may have been a more pronounced decline in the
percentage of users who continued to re-engage with this specific feature over time. That users
perceived the II reminders to be too frequent and lose novelty over time presents an opportunity
for redesign.
The finding that II reminders were perceived to be too frequent points to potential
misalignment between users’ cessation needs over time and our II reminder delivery schedule. In
order to more closely match the needs of the user, the level of support provided should be
dynamically tailored to the user at a particular time point during the quit process
4
. One approach
for achieving dynamic personalization is to allow users opportunities for “periodic reflection on
goals” and to self-report the level of support they need from MQU
4
. For instance, a user might
realize in the middle of the quit attempt that they continue to struggle with lapse during social
situations only. By asking participants to reflect on and report perceived usefulness of II
reminders at different points during the quit attempt, such as the second week of the quit attempt,
we could tailor II reminder delivery to an optimum frequency or timing. For the user in the
128
example provided, we may consider delivering II reminders only during social HRSSs and not
others.
With respect to enhancing the novelty of II reminder messages, Ubhi and colleagues have
encouraged “designing for curiosity” by presenting the user with new information each time the
app is accessed
12
. Increasing II reminder novelty can be accomplished by compiling a list of
similar types of reminders, such as those suggesting distraction strategies, and selecting one at
random for delivery each time. We might also consider prompting users to update the content of
their II reminders to better reflect what behavior regulation strategies they would be willing to
enact as their quit attempt goes on. For example if a user is repeatedly fails to enact their II for a
particular HRSS, they could be prompted to do a different II in future occurrences of that HRSS.
Together these strategies can help to create an interactions that are dynamically tailored to user’s
evolving need for support throughout their quit attempt and to promote continued re-engagement
to the extent that MQU is needed.
Promoting Passive and Low UE pattern days to enhance effectiveness
Our analysis of effective engagement revealed that relative to days where users interact
with a diverse range of app features (Active UE), Passive and Low UE days were associated
reduced probability of same-day lapse. Given this finding, one might conclude that over the
course of a quit attempt, strategies should be designed to promote or prolong these particular
day-level UE patterns. Such an interpretation requires further consideration of the conditions
under which promoting these particular UE patterns would be likely to enhance app
effectiveness.
129
Intervening to promote and prolong Passive UE days, via efforts to refine II reminder
novelty described above, may enhance the effectiveness of MQU for lapse avoidance, but under
certain conditions. Passive UE days are associated with the highest probabilities of
acknowledging 3 or more II reminders on that day. Considering the lapse avoidance mechanisms
we elucidated from interview data, this MQU feature may lead to reduced lapse probability due
to increased exposure to cues for enacting behavioral regulation strategies. However, in order for
II reminders to cue II enactment, users likely need to have strong motivation to quit in the first
place. Some users emphasized in their interviews that acknowledging II reminders would only be
useful for individuals who were highly motivated to quit and thus willing to enact their
behavioral regulation strategies. Therefore, intervening to promote or prolong Passive UE days
should only be done for users who report strong motivations to quit. For users with weak
motivations to quit, Active UE days may be promoted instead, since these days are associated
with high probability of checking progress, which is associated with feedback processes and
increased motivation to quit as lapse avoidance mechanisms.
We note that models of UE and behavioral outcomes hypothesize that a recursive
relationship may exist, whereby Passive UE days lead to reduced lapse via lapse avoidance
mechanisms and lapse outcomes in turn influence the occurrence of subsequent Passive UE days
5,26
. Thus, it is possible that Passive UE days occur because it belongs to a user who was less
likely to lapse that day. Even in these scenarios, however, continuing to promote Passive UE can
still be beneficial to the user, as it would provide additional opportunities for users to interact
with MQU and consequently allow PULL features to be easily accessible when the app is
opened.
130
Although Low UE days similarly reflect effective engagement with MQU, it is likely
through different lapse avoidance mechanisms than Passive UE days. Rather than actively
intervene to promote Low UE days, which represents usage patterns where users are
underexposed to intervention content intended to support lapse avoidance, we might instead
consider the conditions under which strategies to re-engage users should be deployed. Findings
from our qualitative analysis in Study 3 illustrate that Low UE days may not necessarily require
intervention to re-engage a user, especially when a user is not lapsing. Low probability of lapsing
during these days may be because these days belong to users who have already mastered
behavior regulation strategies and attained the required level of motivation to remain quit.
Alternatively, Low UE days may also belong to users who have access to alternative approaches
for quitting (e.g., vaping) and thus do not need to engage actively with MQU for cessation
support. Thus, when deciding whether to intervene during Low UE days, information about
users’ progress with lapse avoidance and access to non-MQU mechanisms for lapse avoidance
should be assessed and incorporated into decision rules.
Limitations
This dissertation includes limitations that should be noted. First, user engagement was
examined in the context of a research study, which is not representative of real world settings in
which smokers would interact with a quit app. Aspects of the research process, such as
compensation for completing EMAs and correspondence with the research staff, may have
influenced some users to re-engage with these features more frequently or consistently than they
would have otherwise. Despite the potential influence of study-related factors, we note that
several users actually decreased their interaction with momentary and EOD EMAs over the
131
course of the quit attempt. This observation highlights the varied motivations that individuals
might have had for interacting with MQU over the quit attempt.
Another study-related limitation was that all users needed an Android device to
participate in the study. MQU was developed to operate on this particular platform due to
technological constraints during the design phase. Only a handful of users ended up using their
personal Android phones to participate in the study while the majority of participants used a
study loan phone. Loan phone users described challenges with keeping their loan phone charged,
getting used to the interface of a different device, and remembering to bring the phone with them
throughout the day. This understandably contributed to observed differences in rates of PUSH
feature interaction between loan phone and personal phone users, which required users to be
attentive to their loan phone. Interestingly, we observed that PULL feature interaction did not
appear to be affected. Our findings regarding the prevalence of specific day-level UE patterns
should be interpreted with this consideration in mind.
Finally, as noted in Studies 2 and 3, our analysis of day-level UE patterns relied only on
behavioral UE data from user interaction logs. This was because the parent MyQuit USC study
was not designed to assess users’ subjective experience with interaction, such as perceived
usefulness of app features or emotional reactions to viewing content on the app, repeatedly over
time. Furthermore, our interest was to examine interactions that could be passively collected and
did not require relying on self-report data, which can often be missing and limit statistical power
for analysis. Nonetheless, we contend that these psychological processes are an important aspect
of understanding UE with quit apps and future work in this area should consider assessing
repeated measures of such variables.
132
Contributions to the literature
User engagement is a prerequisite and key mediator of the effect of digital behavior
change interventions on intended outcomes. The findings described in this dissertation advance
our understanding of user engagement in the context of an app supported quit attempt.
Importantly, we conceptualized UE with MQU across behavioral, cognitive, affective, and
temporal dimensions. In doing so, we characterized UE as both usage and subjective experience,
which is consistent with how UE in described human computer interaction literature. Our
investigation of day-level UE patterns is a key contribution to the literature, both for
demonstrating a novel approach for exploring UE patterns and for modeling day-to-day variation
in UE within users over time. The use of a mixed methods approach to understand the subjective
experience of interaction was a key strength of this dissertation as well, as we were able to
elucidate and formally test hypothesized factors associated with temporal trends in usage
observed in engagement plots. Interview data was additionally critical in identifying aspects of
users’ subjective experience that were relevant for explaining why certain UE patterns reflect
effective engagement via lapse avoidance mechanisms.
Based on our findings, we provide recommendations for strategies to promote re-
engagement with quit apps over time, such as adapting the content and frequency of II reminders
to increase their perceived novelty and usefulness. Furthermore, we describe a strategy to
enhance MQU effectiveness by promoting the occurrence of a day-level UE pattern found to be
associated with reduced probability of lapse (Passive UE days). These strategies may be
applicable to other quit apps that combine PUSH and PULL features and after further testing,
could inform the development of future quit apps that users are motivated to use throughout a
quit attempt and provide effective cessation support.
133
Appendix A: Examples of operationalization of user engagement in digital behavior change interventions
Study Term used Operationalization / indicator Delivery platform and
health outcome
Effect size Temporal
Trend
Donkin et al. 2011 1. Adherence 1. Degree to which user engaged with
website (log-ins, time spent on site,
modules and activities completed,
visits/posts made to forums)
Web-based
Review
Overall positive,
varies by health
behavior; logins
generally
positive for
physical
outcomes
Kelders et al. 2012 1. Adherence 1. Extent to which participant achieves
intended usage
Web-based
Review
NA
Kelders et al. 2013 1. Adherence
2. Usage
1. Starting all available lessons
2. Record of action types (log-ins)
Web-based Depression NA
Merchant et al.
2014
1. Exposure
2. Engagement
1. # of posts delivered by coach
2. # of posts interacted with or initiated
Web-based Weight loss NA Engagement
declined over
24 months
Heffner et al. 2015
1. App utilization 1. # of times each feature was used Mobile-based
Quit smoking
1. ORs=10.5 to
16.4
Heminger et al.
2016
1. Program engagement
1. Aggregate count of keyword texts,
survey responses, web log-ins; Individual
keyword texts
Mobile-based
Quit smoking
1. NS
2. ORs=.24 to
1.43
Glasgow et al.
2011
1. Website usage 1. 5 summary website use variables
including total # of log-ins and # of
website components visited at least twice
Web-based
Diet management
1. r’s ranged
from .20 to .37
for healthy
eating
Decline in
website
usage over 4
months
Helander et al.
2014
1. Adherence 1. Total # of pictures taken + length of
usage period
Mobile-based
Diet management
NA
Duncan et al. 2014 1. Usage 1. Total # of log-ins
2. Number of self-monitoring entries
Mobile- and web-based
Diet management
1. NS
2. NS
Decline in
usage over 9
months
Kirwan et al. 2013 1. Engagement 1. Text messages sent or received
2. Logs entered in app
Mobile-based
Diabetes management
1. NS
2. NS
Dennison et al.
2014
1. Website usage 1. # of sessions completed Web-based
Diet management
NA
Guertler et al. 2015 1. Engagement 1. Duration of program use
2. # of challenges initiated
3. # of days of self-monitoring
Mobile- and web-based
Physical activity
NA Decline in
web and app
for all
measures
134
Ware et al. 2008 1. Engagement 1. Log-in frequency
2. Log-in duration
3. Total interaction time
Web-based
Weight loss
3. Beta=-.082* Decline in all
indicators
over 12
weeks
Maher et al. 2015 1. Engagement 1. # of app visits (dosage)
2. Step-logging patterns
3. # of virtual gifts sent
4. # posts on message wall
Mobile-based
Physical activity
1. F=3.06 (high
dosage
associated with
greater MVPA
increase)
Ingersoll et al.
2015
1. Program engagement 1. Average # of log-ins
2. Average time spent on site
3. Program completion (at least 75%)
Web-based
Autism knowledge
3. Beta=.45
Morris et al. 2015 1. Engagement 1. Subjective user experience
2. Average # of words/log-in
3. Frequency and duration of log-ins
Web-based
Depression
NA
Baltierra et al. 2016 1. Engagement 1. Total intervention exposure
2. Score of participant-initiated use of
features (active vs. passive activities)
Web-based
Sexual health
NA
Owen et al. 2015 1. User engagement 1. # and duration of sessions
2. Retention rate
3. Click streams (series of clicks)
Mobile-based
PTSD
NA Decline in
retention
over 1 year
135
Appendix B: Semi-structured interview guide
Instructions: We would like to know about your experience while using our MyQuit USC app.
Please know that you have provided such valuable information for our project. We hope to
extend this study so we can contribute to becoming a healthier community. To this end, the more
details you can tell us with lots of honesty, the more improvement we can make on the next
generation of our app.
1. Could you please describe what your “relationship with your cigarettes” was like before the
study started/or you began using our app? Probe for how the relationship changed (if it changed)
when the app was used and what happened over time.
2. Please walk me through how you used our app. (If participant has trouble understanding the
request – ask them what they did first...)
-Probe, depending on the responses -For iPhone users, ask to describe issues or problems with
the project mobile phone. -Probe about whether this process is the same now as it was before.
3. What worked well in this app? (Probe features that were used/not used and why) What did not
work well in this app? What would you change?
4. How often do you use a calendar-like app on your own phone (to put down schedules or
reminder)? Probe based on their responses, please tell us about your experience with
MyCalendar.
5. What were some specific plans that you set in the beginning? Did they change over the last 4
weeks? Were there specific plans that became your go-to habit or plans?
6. Could you tell us about times or situations that were more challenging to follow through with
your plans than others? And ask why? Think about the times when being reminded about your
plans helped you to avoid smoking. What were some things that made you more likely to follow
the plan? What were some things aside from the plans that helped you avoid smoking?
7. We would like to know your “wish list” for the app. Please tell us some things that you felt
would have been helpful but were missing in our app. Or, any tips on how we can change
existing features upon which we can improve? What would other Korean American smokers
need in app to quit?
8. You indicated that you consider yourself XX-type smoker (draw from baseline data). How
would you feel about that statement now? Why?
9. In what ways would you use this app for your future quit attempts?
10. What were your thoughts about weekly compensation on ClinCard? What would you change
about that process?
136
Appendix C: Study 1 coding scheme
Category Codes Definition
MQU Features
"I just smoked"
Use of "I just smoked" button
"I want to smoke"
Use of "I want to smoke" button
MyProgress
Use of MyProgress feature
II Reminder
Receiving or acknowledging an II reminder
EMAs
Completing either EOD or momentary surveys
Perceived
usability and
usefulness
App usability
(malfunction)
Loss of data and battery issues, glitches in survey,
accuracy of MyProgress
App perceived usefulness Description of how effective the app or specific
features were at avoiding lapse, e.g. beneficial,
"helpful", needed
App design & aesthetics Reference to design of the app, reminder
alerts/sounds, number of questions, survey
design, comments or suggestions about
scheduling flexibility, accessibility
Loan phone
Any mention of carrying two phones
User
engagement
dimensions
UE: Temporal How app usage changed over time: reasons, rate
of change, timing, including how cognitions and
emotions related to how usage changed over time
UE: Affective Emotions that led up to or occurred as a result of
app use, e.g., guilt, pride, annoyance, frustration
UE: Cognitive Thoughts, beliefs, or attitudes that led up to or
occurred as a result of app use (e.g., perception of
craving or risk for lapse, momentary motivation
to avoid lapse, increased awareness/realization of
smoking, actively reading II reminders, novelty,
reminder of quit or quit motivation, intrinsic
motivation to use app, increased awareness about
smoking habits)
UE: Barriers / facilitators Situations when MQU was used or environmental
constraints that limited app use (e.g., busy,
availability, with friends, not having phone on
them)
UE: Frequency of app use Numeric (once a week, daily, never) indicator of
how often user engaged with MQU at any given
time
UE: Compensation
ClinCard incentive for interacting with MQU
UE: Anthropomorphize Describing the MQU as a person or having
human like qualities
137
Appendix D: Study 3 coding scheme
Theme Mechanism
Codes
Examples Definition
Feedback
Processes
Awareness of quit
progress
Awareness of
smoking reduction,
money saved
Realization or reflection on lapse
avoidance progress or smoking
habit
Awareness of
smoking habit
Awareness of
smoking frequency,
contexts
Motivation
for lapse
avoidance
Motivation for
lapse avoidance
Symptoms, being in a
study, general
reminder of
motivation to quit,
willingness to
continue the quitting
process, cue for "do
not smoke"
Whether and why one should quit,
specific reasons for lapse
avoidance cued during the study,
or general reminder not to smoke
Behavioral
Regulation
II enactment
Behavioral
Regulation II
enactment
Enactment of IIs
planned at baseline
(consult comments
section)
II enactment planned at the
beginning of the study; only code
with “FEATURE: II reminder” if
they explicitly mention “reminder”
Behavioral
Regulation
Unintended
Behavioral
Regulation
Unintended
Napping, stop buying
packs, convincing self
to avoid lapse/holding
self back, self-
reflection, requesting
social support, or
spacing out lapses
Willful strategies or actions used
to avoid smoking that were not
planned at baseline. Note this is
acting on a motivation to quit vs.
being reminded of motivation
which should be coded
“motivation for lapse avoidance”
Vaping Smoking a vape pen
instead of cigarettes
Non-app
processes
Availability to
smoke
Too busy, no time to
buy or smoke, price of
cigarettes
Environmental contexts including
social and individual factors that
facilitated lapse avoidance. These
are not willful, self-initiated
strategies for lapse avoidance. Reduced Urges Reduced desire or
craving
Social Influence
Others encourage LA
but not self-initiated
138
References
1. West R, Michie S. A Guide to Development and Evaluation of Digital Behavior Change
Interventions in Healthcare. 2016.
2. Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral Intervention
Technologies: Evidence review and recommendations for future research in mental health.
Gen Hosp Psychiatry. 2013;35(4):332-338. doi:10.1016/j.genhosppsych.2013.03.008.
3. Klasnja P, Pratt W. Healthcare in the pocket: Mapping the space of mobile-phone health
interventions. J Biomed Inform. 2012;45(1):184-198. doi:10.1016/j.jbi.2011.08.017.
4. Gulotta R, Forlizzi J, Yang R, Newman MW. Fostering Engagement with Personal
Informatics Systems. In: Proceedings of the 2016 ACM Conference on Designing
Interactive Systems - DIS ’16. New York, New York, USA: ACM Press; 2016:286-300.
doi:10.1145/2901790.2901803.
5. Ritterband LM, Thorndike FP, Cox DJ, Kovatchev BP, Gonder-Frederick LA. A behavior
change model for internet interventions. Ann Behav Med. 2009;38(1):18-27.
doi:10.1007/s12160-009-9133-4.
6. Yardley L, Spring BJ, Riper H, et al. Understanding and Promoting Effective Engagement
With Digital Behavior Change Interventions. Am J Prev Med. 2016;51(5):833-842.
doi:10.1016/j.amepre.2016.06.015.
7. Danaher BG, Smolkowski K, Seeley JR, Severson HH. Mediators of a successful web-
based smokeless tobacco cessation program. Addiction. 2008;103(10):1706-1712.
doi:10.1111/j.1360-0443.2008.02295.x.
8. Danaher BG, Boles SM, Akers L, Gordon JS, Severson HH. Defining participant exposure
measures in web-based health behavior change programs. J Med Internet Res.
139
2006;8(3):1-12. doi:10.2196/jmir.8.3.e15.
9. Strecher VJ, McClure J, Alexander G, et al. The role of engagement in a tailored web-
based smoking cessation program: Randomized controlled trial. J Med Internet Res.
2008;10(5). doi:10.2196/jmir.1002.
10. Abroms LC, Lee Westmaas J, Bontemps-Jones J, Ramani R, Mellerson J. A Content
Analysis of Popular Smartphone Apps for Smoking Cessation. Am J Prev Med.
2013;45(6):732-736. doi:10.1016/j.amepre.2013.07.008.
11. BinDhim NF, McGeechan K, Trevena L. Who uses smoking cessation apps? A feasibility
study across three countries via smartphones. J Med Internet Res. 2014;16(2):1-11.
doi:10.2196/mhealth.2841.
12. Ubhi HK, Michie S, Kotz D, van Schayck OCP, Selladurai A, West R. Characterising
smoking cessation smartphone applications in terms of behaviour change techniques,
engagement and ease-of-use features. Transl Behav Med. 2016;6(3):410-417.
doi:10.1007/s13142-015-0352-x.
13. Haug S, Schaub MP, Venzin V, Meyer C, John U. Efficacy of a text message-based
smoking cessation intervention for young people: A cluster randomized controlled trial. J
Med Internet Res. 2013;15(8). doi:10.2196/jmir.2636.
14. Bock B, Heron K, Jennings E, et al. A text message delivered smoking cessation
intervention: The initial trial of TXT-2-quit: Randomized controlled trial. J Med Internet
Res. 2013;15(7). doi:10.2196/mhealth.2522.
15. Douglas N, Free C. “Someone batting in my corner”: Experiences of smoking-cessation
support via text message. Br J Gen Pract. 2013;63(616):768-776.
doi:10.3399/bjgp13X674459.
140
16. Ubhi HK, Michie S, Kotz D, Wong WC, West R. A mobile app to aid smoking cessation:
Preliminary evaluation of smokefree28. J Med Internet Res. 2015;17(1):e17.
doi:10.2196/jmir.3479.
17. Ploderer B, Smith W, Pearce J, Borland R. A mobile app offering distractions and tips to
cope with cigarette craving: a qualitative study. JMIR mHealth uHealth. 2014;2(2):e23.
doi:10.2196/mhealth.3209.
18. Smith W, Ploderer B, Wadley G, Webber S, Borland R. Trajectories of Engagement and
Disengagement with a Story-Based Smoking Cessation App. In: Proceedings of the 2017
CHI Conference on Human Factors in Computing Systems - CHI ’17. New York, New
York, USA: ACM Press; 2017:3045-3056. doi:10.1145/3025453.3026054.
19. Baskerville NB, Struik LL, Hammond D, et al. Effect of a Mobile Phone Intervention on
Quitting Smoking in a Young Adult Population of Smokers: Randomized Controlled Trial
Study Protocol. JMIR Res Protoc. 2015;4(1):e10. doi:10.2196/resprot.3823.
20. Whittaker R, McRobbie H, Bullen C, Rodgers A, Gu Y. Mobile phone-based
interventions for smoking cessation. In: Whittaker R, ed. Cochrane Database of
Systematic Reviews. Vol 4. Chichester, UK: John Wiley & Sons, Ltd; 2016:1-22.
doi:10.1002/14651858.CD006611.pub4.
21. Haskins BL, Lesperance D, Gibbons P, Boudreaux ED. A systematic review of
smartphone applications for smoking cessation. Transl Behav Med. 2017;7(2):292-299.
doi:10.1007/s13142-017-0492-2.
22. Couper MP, Alexander GL, Zhang N, et al. Engagement and retention: Measuring breadth
and depth of participant use of an online intervention. J Med Internet Res. 2010;12(4).
doi:10.2196/jmir.1430.
141
23. O’Brien HL, Toms EG. What is user engagement? A conceptual framework for defining
user engagement with technology. J Am Soc Inf Sci Technol. 2008;59(6):938-955.
doi:10.1002/asi.20801.
24. Lehmann J, Lalmas M, Yom-tov E, Dupret G. Model of User Engagement. Proc 20th
Conf user Model Adapt Pers (UMAP 2012) (pp. 2012:164–175). doi:10.1007/978-3-642-
31454-4_14.
25. Attfield S, Kazai G, Lalmas M. Towards a science of user engagement (Position Paper).
WSDM Work User Model Web Appl. 2011.
26. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital
behaviour change interventions: a systematic review using principles from critical
interpretive synthesis. Transl Behav Med. 2017;7(2):254-267. doi:10.1007/s13142-016-
0453-1.
27. Dennison L, Morrison L, Lloyd S, et al. Does Brief Telephone Support Improve
Engagement With a Web-Based Weight Management Intervention? Randomized
Controlled Trial. J Med Internet Res. 2014;16(3). doi:10.2196/jmir.3199.
28. Peters C, Castellano G, de Freitas S. An exploration of user engagement in HCI. Proc Int
Work Affect Virtual Agents Soc Robot. 2009:9:1--9:3. doi:10.1145/1655260.1655269.
29. Hart J, Sutcliffe A, De Angeli A. Using affect to evaluate user engagement. CHI ’12 Ext
Abstr Hum Factors Comput Syst. 2012:1811-1834. doi:10.1145/2212776.2223714.
30. Bickmore T, Schulman D, Yin L. Maintaining Engagement in Long-term Interventions
with Relational Agents. Appl Artif Intell. 2010;24(6):648-666.
doi:10.1080/08839514.2010.492259.
31. Sama PR, Eapen ZJ, Weinfurt KP, Shah BR, Schulman K a. An evaluation of mobile
142
health application tools. JMIR mHealth uHealth. 2014;2(2):e19.
doi:10.2196/mhealth.3088.
32. Krebs P, Duncan DT. Health App Use Among US Mobile Phone Owners: A National
Survey. JMIR mHealth uHealth. 2015;3(4):e101. doi:10.2196/mhealth.4924.
33. Shaw RJ, Steinberg DM, Bonnet J, et al. Mobile health devices: will patients actually use
them? J Am Med Informatics Assoc. January 2016:ocv186. doi:10.1093/jamia/ocv186.
34. Anderson K, Burford O, Emmerton L. Mobile health apps to facilitate self-care: A
qualitative study of user experiences. PLoS One. 2016;11(5):1-21.
doi:10.1371/journal.pone.0156164.
35. Cho J, Quinlan MM, Park D, Noh GY. Determinants of adoption of smartphone health
apps among college students. Am J Health Behav. 2014;38(6):860-870.
doi:10.5993/AJHB.38.6.8.
36. Dou K, Yu P, Deng N, et al. Patients’ Acceptance of Smartphone Health Technology for
Chronic Disease Management: A Theoretical Model and Empirical Test. JMIR mHealth
uHealth. 2017;5(12):e177. doi:10.2196/mhealth.7886.
37. Murray E, Hekler EB, Andersson G, et al. Evaluating Digital Health Interventions: Key
Questions and Approaches. Am J Prev Med. 2016;51(5):843-851.
doi:10.1016/j.amepre.2016.06.008.
38. Zeng EY, Heffner JL, Copeland WK, Mull KE, Bricker JB. Get with the program:
Adherence to a smartphone app for smoking cessation. Addict Behav. 2016;63(4):120-
124. doi:10.1016/j.addbeh.2016.07.007.
39. Schmidt CA, Romine JK, Bell ML, Armin J, Gordon JS. User Participation and
Engagement With the See Me Smoke-Free mHealth App: Prospective Feasibility Trial.
143
JMIR mHealth uHealth. 2017;5(10):e142. doi:10.2196/mhealth.7900.
40. Donkin L, Hickie IB, Christensen H, et al. Rethinking the dose-response relationship
between usage and outcome in an online intervention for depression: Randomized
controlled trial. J Med Internet Res. 2013;15(10). doi:10.2196/jmir.2771.
41. Heminger CL, Boal AL, Zumer M, Abroms LC. Text2Quit: an analysis of participant
engagement in the mobile smoking cessation program. Am J Drug Alcohol Abuse.
2016;2990(July):1-9. doi:10.3109/00952990.2016.1149591.
42. Heffner JL, Vilardaga R, Mercer LD, Kientz JA, Bricker JB. Feature-level analysis of a
novel smartphone application for smoking cessation. Am J Drug Alcohol Abuse.
2015;41(1):68-73. doi:10.3109/00952990.2014.977486.
43. Ainsworth B, Steele M, Stuart B, et al. Using an analysis of behavior change to inform
effective digital intervention design: How did the PRIMIT website change hand hygiene
behavior across 8993 users? Ann Behav Med. 2016. doi:10.1007/s12160-016-9866-9.
44. Klasnja P, Hekler EB, Shiffman S, et al. Microrandomized trials: An experimental design
for developing just-in-time adaptive interventions. Heal Psychol. 2015;34(Suppl):1220-
1228. doi:10.1037/hea0000305.
45. Shiffman S, Stone A a, Hufford MR. Ecological momentary assessment. Annu Rev Clin
Psychol. 2008;4:1-32. doi:10.1146/annurev.clinpsy.3.022806.091415.
46. Shiffman S. Ecological momentary assessment (EMA) in studies of substance use.
Psychol Assess. 2009;21(4):486-497. doi:10.1037/a0017074.
47. Maxwell AE, Crespi CM, Alano RE, Sudan M, Bastani R. Health risk behaviors among
five Asian American subgroups in California: identifying intervention priorities. J Immigr
Minor Health. 2012;14(5):890-894. doi:10.1007/s10903-011-9552-8.
144
48. An N, Cochran SD, Mays VM, McCarthy WJ. Influence of American acculturation on
cigarette smoking behaviors among Asian American subpopulations in California.
Nicotine Tob Res. 2008;10(4):579-587. doi:10.1080/14622200801979126.
49. Chen MS. Cancer health disparities among Asian Americans. Cancer.
2005;104(S12):2895-2902. doi:10.1002/cncr.21501.
50. Arnett JJ. The Developmental Context of Substance use in Emerging Adulthood. J Drug
Issues. 2005;35(2):235-254. doi:10.1177/002204260503500202.
51. Cerrada CJ, Unger JB, Huh J. Correlates of Perceived Smoking Prevalence Among
Korean American Emerging Adults. J Immigr Minor Heal. 2016;18(5):1183-1189.
doi:10.1007/s10903-015-0264-3.
52. Kim SS, Son H, Nam KA. The Sociocultural Context of Korean American Men’s
Smoking Behavior. West J Nurs Res. 2005;27(5):604-623.
doi:10.1177/0193945905276258.
53. Cerrada CJ, Dzubur E, Blackman KCA, Mays V, Shoptaw S, Huh J. Development of a
Just-in-Time Adaptive Intervention for Smoking Cessation Among Korean American
Emerging Adults. Int J Behav Med. 2017;24(5). doi:10.1007/s12529-016-9628-x.
54. Cerrada CJ, Ra CK, Shin H-S, Dzubur E, Huh J. Using Ecological Momentary
Assessment to Identify Common Smoking Situations Among Korean American Emerging
Adults. Prev Sci. 2016;17(7):892-902. doi:10.1007/s11121-016-0687-7.
55. Cerrada CJ, Ra CK, Shin H-S, Dzubur E, Huh J. Using Ecological Momentary
Assessment to Identify Common Smoking Situations Among Korean American Emerging
Adults. Prev Sci. 2016;17(7). doi:10.1007/s11121-016-0687-7.
56. Gollwitzer PM, Sheeran P. Implementation intentions and goal achievement: A meta-
145
analysis of effects and processes. Adv Exp Psychol. 2006;38(38):69-119.
doi:10.1016/S0065-2601(06)38002-1.
57. van Osch L, Lechner L, Reubsaet A, Wigger S, de Vries H. Relapse prevention in a
national smoking cessation contest: effects of coping planning. Br J Health Psychol.
2008;13(Pt 3):525-535. doi:10.1348/135910707X224504.
58. Armitage CJ. Efficacy of a brief worksite intervention to reduce smoking: the roles of
behavioral and implementation intentions. J Occup Health Psychol. 2007;12(4):376-390.
doi:10.1037/1076-8998.12.4.376.
59. Nahum-shani I, Smith SN, Witkiewitz K, Collins LM, Spring B, Murphy SA. Just-in-Time
Adaptive Interventions ( JITAIs ): An Organizing Framework for Ongoing Health
Behavior Support. Vol 073975.; 2014.
60. Gouveia R, Karapanos E, Hassenzahl M. How do we engage with activity trackers?: a
longitudinal study of Habito. Proc 2015 ACM Int Jt Conf Pervasive Ubiquitous Comput.
2015:1305-1316. doi:10.1145/2750858.2804290.
61. Klasnja P, Consolvo S, Pratt W. How to Evaluate Technologies for Health Behavior
Change in HCI Research. 2011;(May). doi:10.1145/1978942.1979396.
62. Davis FD. Perceived Usefulness , Perceived Ease of Use , and User Acceptance of
lnformation Technology. MIS Q. 1989;13(3):319-340. doi:10.2307/249008.
63. Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic
review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res.
2011;13(3). doi:10.2196/jmir.1772.
64. Kelders SM, Bohlmeijer ET, Van Gemert-Pijnen JE. Participants, Usage, and Use Patterns
of a Web-Based Intervention for the Prevention of Depression Within a Randomized
146
Controlled Trial. J Med Internet Res. 2013;15(8):e172. doi:10.2196/jmir.2258.
65. Merchant G, Weibel N, Patrick K, et al. Click “like” to change your behavior: a mixed
methods study of college students’ exposure to and engagement with Facebook content
designed for weight loss. J Med Internet Res. 2014;16(6):e158. doi:10.2196/jmir.3267.
66. Van Gemert-Pijnen JEWC, Kelders SM, Bohlmeijer ET. Understanding the usage of
content in a mental health intervention for depression: An analysis of log data. J Med
Internet Res. 2014;16(1):1-16. doi:10.2196/jmir.2991.
67. Guertler D, Vandelanotte C, Kirwan M, Duncan MJ. Engagement and nonusage attrition
with a free physical activity promotion program: The case of 10,000 steps Australia. J
Med Internet Res. 2015;17(7). doi:10.2196/jmir.4339.
68. Duncan M, Vandelanotte C, Kolt GS, et al. Effectiveness of a web- and mobile phone-
based intervention to promote physical activity and healthy eating in middle-Aged males:
Randomized controlled trial of the manup study. J Med Internet Res. 2014;16(6).
doi:10.2196/jmir.3107.
69. Dennison L, Morrison L, Conway G, Yardley L. Opportunities and challenges for
smartphone applications in supporting health behavior change: qualitative study. J Med
Internet Res. 2013;15(4):1-12. doi:10.2196/jmir.2583.
70. Swendeman D, Ramanathan N, Baetscher L, et al. Smartphone self-monitoring to support
self-management among people living with HIV: perceived benefits and theory of change
from a mixed-methods randomized pilot study. J Acquir Immune Defic Syndr. 2015;69
Suppl 1(0 1):S80-91. doi:10.1097/QAI.0000000000000570.
71. Baumer EPS, Gay GK, Katz SJ, et al. Prescriptive persuasion and open-ended social
awareness. In: Proceedings of the ACM 2012 Conference on Computer Supported
147
Cooperative Work - CSCW ’12. ; 2012:475. doi:10.1145/2145204.2145279.
72. Buman MP, Epstein DR, Gutierrez M, et al. BeWell24: development and process
evaluation of a smartphone “app” to improve sleep, sedentary, and active behaviors in US
Veterans with increased metabolic risk. Transl Behav Med. 2015;(February 2016).
doi:10.1007/s13142-015-0359-3.
73. Nahum-shani I, Hekler EB, Spruijt-metz D. Building health behavior models to guide the
developmnet of just-in-time adaptive interventions: A pragmatic framework. Heal
Psychol. 2015;34(Supplement):1209-1219. doi:10.1037/hea0000306.
74. Cerrada CJ, Dzubur E, Blackman KCA, Mays V, Shoptaw S, Huh J. Development of a
Just-in-Time Adaptive Intervention for Smoking Cessation Among Korean American
Emerging Adults. Int J Behav Med. 2017. doi:10.1007/s12529-016-9628-x.
75. Liao P, Klasnja P, Tewari A, Murphy S a. Micro-Randomized Trials in mHealth. arXiv.
2015;(November):1-27.
76. Hartzler AL, BlueSpruce J, Catz SL, McClure JB. Prioritizing the mHealth Design Space:
A Mixed-Methods Analysis of Smokers’ Perspectives. JMIR mHealth uHealth.
2016;4(3):e95. doi:10.2196/mhealth.5742.
77. Danaher BG, Brendryen H, Seeley JR, Tyler MS, Woolley T. From black box to toolbox:
Outlining device functionality, engagement activities, and the pervasive information
architecture of mHealth interventions. Internet Interv. 2015;2(1):91-101.
doi:10.1016/j.invent.2015.01.002.
78. Creswell JW, Fetters MD, Ivankova N V. Designing a mixed methods study in primary
care. Ann Fam Med. 2004;2(1):7-12. doi:10.1370/afm.104.
79. Hsieh H-F. Three Approaches to Qualitative Content Analysis. Qual Health Res.
148
2005;15(9):1277-1288. doi:10.1177/1049732305276687.
80. Braun, V. & Clarke V. Using Thematic Analysis in Psychology. Qual Res Pyschology.
2006;3:77-101.
81. Heatherton TF, Kozlowski LT, Frecker RC, Fagerstrom KO. The Fagerstrom Test for
Nicotine Dependence: a revision of the Fagerstrom Tolerance Questionnaire. Br J Addict.
1991;86(9):1119-1127. doi:doi:10.1111/j.1360-0443.1991.tb01879.x.
82. McHugh ML. Interrater reliability: the kappa statistic. Biochem Medica. 2012;22(3):276-
282. doi:10.11613/BM.2012.031.
83. Migneault JP, Farzanfar R, Wright JA, Friedman RH. How to write health dialog for a
talking computer. J Biomed Inform. 2006;39(5):468-481. doi:10.1016/j.jbi.2006.02.009.
84. Abroms LC, Boal AL, Simmens SJ, Mendel JA, Windsor RA. A Randomized Trial of
Text2Quit. Am J Prev Med. 2014;47(3):242-250. doi:10.1016/j.amepre.2014.04.010.
85. Otsuki M, Tinsley BJ, Chao RK, Unger JB. An ecological perspective on smoking among
Asian American college students: the roles of social smoking and smoking motives.
Psychol Addict Behav. 2008;22(4):514-523. doi:10.1037/a0012964.
86. Ybarra ML, Holtrop JS, Prescott TL, Strong D. Process evaluation of a mHealth program:
Lessons learned from Stop My Smoking USA, a text messaging-based smoking cessation
program for young adults. Patient Educ Couns. 2014;97(2):239-243.
doi:10.1016/j.pec.2014.07.009.
87. Lanza ST, Rhoades BL. Latent class analysis: an alternative perspective on subgroup
analysis in prevention and treatment. Prev Sci. 2013;14(2):157-168. doi:10.1007/s11121-
011-0201-1.
88. Sepah SC, Jiang L, Ellis RJ, McDermott K, Peters AL. Engagement and outcomes in a
149
digital Diabetes Prevention Program: 3-year update. BMJ Open Diabetes Res Care.
2017;5(1):e000422. doi:10.1136/bmjdrc-2017-000422.
89. Glasgow RE, Christiansen SM, Kurz D, et al. Engagement in a diabetes self-management
website: Usage patterns and generalizability of program use. J Med Internet Res.
2011;13(1). doi:10.2196/jmir.1391.
90. Finch WH, French BF. Multilevel latent class analysis: Parametric and Nonparametric
Models. J Exp Educ. 2014;82(3):307-333. doi:10.1080/00220973.2013.813361.
91. Henry KL, Muthén B. Multilevel Latent Class Analysis: An Application of Adolescent
Smoking Typologies With Individual and Contextual Predictors. Struct Equ Model A
Multidiscip J. 2010;17(2):193-215. doi:10.1080/10705511003659342.
92. Muthén LK, Muthén BO. Mplus User’ s Guide. Seventh Edition.
93. Vermunt JK. Latent class modeling with covariates: Two improved three-step approaches.
Polit Anal. 2010;18(4):450-469. doi:10.1093/pan/mpq025.
94. Asparouhov T, Muthén B. Auxiliary Variables in Mixture Modeling: Three-Step
Approaches Using M plus. Struct Equ Model A Multidiscip J. 2014;0(June):1-13.
doi:10.1080/10705511.2014.915181.
95. Bidargaddi N, Pituch T, Maaieh H, Short C, Strecher V. Predicting which type of push
notification content motivates users to engage in a self-monitoring app. Prev Med Reports.
2018;11(July):267-273. doi:10.1016/j.pmedr.2018.07.004.
96. Wagner III B, Liu E, Shaw SD, et al. e wrapper: operationalizing engagement strategies in
mHealth. Proc 2017 ACM Int Jt Conf Pervasive Ubiquitous Comput Proc 2017 ACM Int
Symp Wearable Comput. 2017:790-798. doi:10.1145/3123024.3125612.Abstract.
97. Michie S, Carey RN, Johnston M, et al. From Theory-Inspired to Theory-Based
150
Interventions: A Protocol for Developing and Testing a Methodology for Linking
Behaviour Change Techniques to Theoretical Mechanisms of Action. Ann Behav Med.
2018;(May):1-12. doi:10.1007/s12160-016-9816-6.
98. The Theory and Techniques Tool.
https://theoryandtechniquetool.humanbehaviourchange.org/tool.
99. Reboussin BA, Ip EH, Wolfson M. Locally dependent latent class models with covariates:
An application to under-age drinking in the USA. J R Stat Soc Ser A Stat Soc.
2008;171(4):877-897. doi:10.1111/j.1467-985X.2008.00544.x.
100. Lanza ST, Tan X, Bray BC. Latent Class Analysis With Distal Outcomes: A Flexible
Model-Based Approach. Struct Equ Model A Multidiscip J. 2013;20(1):1-26.
doi:10.1080/10705511.2013.742377.
101. Willett JG, Bennett M, Hair EC, et al. Recognition, use and perceptions of JUUL among
youth and young adults. Tob Control. 2018:1-2. doi:10.1136/tobaccocontrol-2018-054273.
Abstract (if available)
Abstract
Mobile smoking cessation apps have the potential to reduce smoking rates on a large scale but app abandonment after minimal use remains a significant challenge. Furthermore, little is known about the diverse ways smokers use quit apps and how usage mediates positive cessation outcomes. Thus, the overarching aim of this dissertation was to better understand user engagement (UE) with a recently developed cessation app, MQU, over the course of a quit attempt. Following a mixed methods approach, user logs of interaction with each of MQU’s features were combined with semi-structured interviews to examine longitudinal trends in usage, factors that influenced usage over time, and relevant mechanisms for lapse avoidance. Multilevel latent class analysis was used to explore underlying day-level UE patterns and to examine their association with covariates and a lapse outcome. ❧ Our findings illustrated differences in usage trends between “PUSH” and “PULL” app features over time and highlighted the second week of the quit attempt as an important time point for changes in usage and perceptions of MQU novelty and usefulness. Multilevel latent class analysis uncovered three day-level UE patterns within users: Active, Passive, and Low UE patterns. Relative to Low UE days, Active UE days were less probable among loan phone users and as days passed in the quit attempt. Contrary to our hypothesis, Active UE days were associated with greater probability of same-day lapse relative to Low and Passive UE days. Finally, we elucidated mechanisms for lapse avoidance attributable to MQU usage, such as behavioral regulation and increased motivation to quit, along with mechanisms not related to MQU, such as momentary lack of access to cigarettes and availability to smoke. ❧ Based on our findings, we provide recommendations for strategies to promote re- engagement with quit apps over time, such as dynamically adapting the content and frequency of II reminders to increase their perceived novelty and usefulness. Furthermore, we describe a strategy to enhance MQU effectiveness by promoting the occurrence of a day-level UE pattern found to be associated with reduced probability of lapse (Passive UE days). These strategies may be applicable to other quit apps that combine PUSH and PULL features and after further testing, could inform the development of future quit apps that users are motivated to use throughout a quit attempt and provide effective cessation support.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
The acute relationship between affective states and physiological stress response, and the moderating role of moderate-to-vigorous physical activity
PDF
Effects of the perceived and objectively assessed environment on physical activity in adults and children
PDF
Effects of sugar and fiber consumption in minority adolescents and self-tracking as a potential dietary intervention tool
Asset Metadata
Creator
Cerrada, Christian Jules
(author)
Core Title
Mixed methods investigation of user engagement with a smoking cessation app
School
Keck School of Medicine
Degree
Doctor of Philosophy
Degree Program
Preventive Medicine (Health Behavior Research)
Publication Date
03/12/2019
Defense Date
11/27/2018
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
mixed methods,mobile app,OAI-PMH Harvest,smoking cessation,user engagement
Format
application/pdf
(imt)
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Huh, Jimi (
committee chair
), Bluthenthal, Ricky (
committee member
), Chou, Chih-Ping (
committee member
), Spruijt-Metz, Donna (
committee member
)
Creator Email
cerrada@usc.edu,christian.j.cerrada@gmail.com
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c89-132862
Unique identifier
UC11675693
Identifier
etd-CerradaChr-7149.pdf (filename),usctheses-c89-132862 (legacy record id)
Legacy Identifier
etd-CerradaChr-7149.pdf
Dmrecord
132862
Document Type
Dissertation
Format
application/pdf (imt)
Rights
Cerrada, Christian Jules
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
mixed methods
mobile app
smoking cessation
user engagement