University of Southern California Dissertations and Theses

Revision of the multisystemic therapy (MST) adherence coding protocol: assessing the reliability and predictive validity of adherence to the nine MST principles

(USC Thesis Other)

Revision of the multisystemic therapy (MST) adherence coding protocol: assessing the reliability and predictive validity of adherence to the nine MST principles

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Running Head: MST CODING PROTOCOL i

Revision of the Multisystemic Therapy (MST) Adherence Coding Protocol:
Assessing the Reliability and Predictive Validity of Adherence to the Nine MST Principles
Marie L. Gillespie, B.S.
University of Southern California

Faculty Advisor: Stan Huey Jr., Ph.D
Masters Thesis
May 2014

MST CODING PROTOCOL ii
Table of Contents
Table of Contents ii
List of Tables iii
List of Figures iv
Abstract 1
Background and Significance 2
Method 10
Results 18
Discussion 20
Conclusion 25
References 27
Appendix A: MST Principles 44
Appendix B: Therapist Adherence Measure 45
Appendix C: MST Adherence Audiotape Coding Form 47
Appendix D: Test of Differences 51
Appendix E: Individual Therapist Adherence 52

MST CODING PROTOCOL iii
List of Tables
Table 1: Demographic Characteristics of Selected Subjects 35
Table 2: Means, Standard Deviations, and Sample Sizes for Outcome Variables 36
Table 3: Means, Standard Deviations, Ranges, and Sample Sizes for 37
Predictor Variables

Table 4: Intraclass Correlation Coefficients (ICCs) for Rater Dyads 38
on the 9 Multisystemic Therapy (MST) Principles
Table 5: Intercorrelations Among Caregiver, Youth, and Therapist TAM 39
Composite Ratings and Correlations Between TAM ratings and
the MST Principle Adherence Composite

Table 6: Generalized Estimation Equation (GEE) Model Estimates Comparing 40
MST Principle Adherence and Outcome Variables Across Time

Table 7: Logistic Regression of Dichotomized Data with MST Principle 41
Adherence for T2 and T3

MST CODING PROTOCOL iv
List of Figures
Figure 1: Generalized Estimation Equation (GEE) Model Estimates 42
Comparing MST Principle Adherence (MPA) and Form 90
Alcohol Use Over Time

Figure 2: Generalized Estimation Equation (GEE) Model Estimates 43
Comparing MST Principle Adherence (MPA) and CBCL
Externalizing T-score (Youth-report) Over Time
MST CODING PROTOCOL 1

Abstract
Treatment adherence to multisystemic therapy (MST) has typically been measured using the
MST Therapist Adherence Measure (TAM; Henggeler & Borduin, 1992); however, no
behavioral observation measure exists to address the limitations of self-report. The current study
revised an existing protocol for assessing adherence to the nine MST principles (Huey, 2001),
and assessed the reliability and validity of the new coding system. Using data from a clinical trial
of MST for juvenile drug offenders, 40 audiotaped sessions were selected from the first month of
treatment and coded by trained undergraduate students. Results showed moderate interrater
reliability for most of the nine principles and good reliability for our composite index of
adherence to all nine principles (MST Principle Adherence: MPA). Although MPA did not
demonstrate concurrent validity with TAM scores, high MPA during the first month of therapy
predicted decreases in externalizing behavior at 4-months post-treatment and decreases in youth
alcohol consumption at 12-month follow-up. The revised coding system is a useful, observer-
rated tool for reliably measuring MST adherence and could be a valuable addition to the MST
quality-assurance model.
Keywords: multisystemic therapy, treatment adherence, coding protocol

MST CODING PROTOCOL 2
Revision of the Multisystemic Therapy (MST) Adherence Coding Protocol:
Assessing the Reliability and Predictive Validity of Adherence to the Nine MST Principles
There has been much support for the effectiveness of youth-focused, multi-component
treatments that intervene across multiple systems, including the home, school, and peer contexts
(Centis, Marzocchi, Di Luzio, Moscatiello, Salardi, Villanova, & Marchesini, 2012; Cong, Feng,
Liu, & Esperat, 2012; Henggeler & Borduin, 1990; Kazdin, Siegel, & Bass, 1992; Prevatt &
Kelly, 2003; Tremblay, Pagani-Kurtz, Mâsse, Vitaro, & Pihl, 1995). Perhaps the best validated
of these interventions is multisystemic therapy (MST). Originally developed by Henggeler and
colleagues (Henggeler, Rodick, Borduin, Hanson, Watson, & Urey, 1986; Henggeler,
Schoenwald, Borduin, Rowland, & Cunningham, 1998) for juvenile offenders, MST is an
evidence-based treatment which intervenes at multiple levels of the youth’s social context. MST
has been shown to be effective for targeting wide-ranging problems in diverse populations,
including youth with diabetes (Naar-King, Ellis, Idalski, Frey, & Cunningham, 2007), juvenile
sex offenders (Henggeler, Letourneau, Chapman, Borduin, Schewe, & McCart, 2009), African
American youth with poor asthma management (Naar-King, Ellis, King, Lam, Cunningham,
Secord, Bruzzese, & Templin, 2014), and substance-abusing youth (Henggeler Clingempeel,
Brondino, & Pickrel, 2002; Pickrel & Henggeler, 1996). To date, MST has been supported by 19
randomized clinical trials (RCTs) and two quasi-experimental studies (Henggeler, 2011).
MST adopts Bronfenbrenner's (1979) social-ecological model which emphasizes the role
of contextual influences within systems in a person’s life. In order to maximize ecological
validity, MST is conducted within the youth’s natural environment and identifies problems in
their broader systemic contexts (Henggeler et al., 1998). This flexible and individualized
treatment utilizes nine principles (see Appendix A for comprehensive list of principles). These
MST CODING PROTOCOL 3
principles guide therapist behavior, and by adhering to these principles, the therapist is better
able to select appropriate treatment strategies and prioritize targets of intervention.
MST Treatment Adherence
Measuring treatment adherence (i.e., integrity, fidelity) is crucial in order to determine
whether an intervention is delivered in line with its theoretical foundations and to facilitate
treatment replication and generalizability (Alvarez-Jimenez et al., 2008; Moncher & Prinz,
1991). Schoenwald, Henggeler, Brondino, and Rowland (2000) state that adherence is the extent
to which delivered treatments include prescribed components and omit proscribed ones. To
assess MST adherence, Henggeler and Borduin (1992) developed the 26-item MST Therapist
Adherence Measure (TAM), which utilizes caregiver, youth, and therapist ratings of items on a
5-point Likert scale (Appendix B). Factor scores were constructed for each rater and include
dimensions such as: family and therapist effort to problem-solve, lack of therapeutic direction
during sessions, family-therapist conflict, and nonproductive sessions (Henggeler et al., 1997).
However, caregiver reports have typically been found to be better predictors of youth outcomes
compared to youth and therapist reports (Schoenwald et al., 2000).
Several studies have found inconsistent associations between TAM-rated MST adherence
and various treatment outcomes. For example Henggeler, Melton, Brondino, Scherer, and Hanley
(1997) found that the caregiver, youth, and therapist TAM factors were intermittently predictive
of drug and recidivism outcomes, and patterns differed across informants. Similarly, outcome
analyses from Henggeler, Pickrel, and Brondino (1999) showed modest and inconsistent TAM
predictions. Moreover, Schoenwald et al. (2000) state in their limitations that their analyses “did
not suggest consistent patterns of association between particular aspects of adherence and
process level outcomes” (p.99).
MST CODING PROTOCOL 4
Recently, a revised version of the TAM (TAM-R; Henggeler, Borduin, Schoenwald,
Huey, & Chapman, 2006) was developed and validated with a large study sample. This 28-item
revised scale also aimed to assess therapist adherence to the nine principles and was developed
through expert consensus of items. Schoenwald, Carter, Chapman, and Sheidow (2008)
measured 1979 youth and 429 therapists involved in MST at pretreatment, post-treatment, 6-
month follow-up, and 12-month follow-up. Therapist adherence was evaluated monthly during
treatment and behavior problems were assessed through caregiver ratings on the Child Behavior
Checklist (CBCL; Achenbach, 1991). The authors found that greater therapist adherence to MST
predicted steeper reductions in juvenile behavior problems one year following treatment
termination. However, contrary to findings from other TAM outcome studies (e.g., Henggeler et
al., 1997), TAM-R-rated adherence was not associated with changes in recidivism.
This research shows some evidence suggesting that therapist adherence and proper MST
implementation can predict various treatment outcomes. The TAM does not appear to predict the
same outcomes across studies and factor structures are unstable across informants and across
studies (Huey et al., 2000). The construction, utilization, and validity of TAM ratings are
explained here as they will be used in assessing the revised protocol’s concurrent validity.
Limitations of the TAM
Although research on the TAM’s validity is fairly extensive, the scale has several
shortcomings that limit its utility. These limitations are discussed below and the case is made for
utilizing a more objective index of MST adherence.
Rater bias. In past studies, individuals completing the TAM were directly involved in
the therapy process. Youth and caregiver ratings of treatment fidelity may simply reflect their
perceptions of how well they are doing in treatment (Bechger, Maris, & Hsiao, 2010; Kozlowski
MST CODING PROTOCOL 5
& Kirsch, 1987; Thorndike, 1920). Therapists are also vulnerable to this bias. For example,
Breitenstein, Fogg, Garvey, Hill, Resnick, and Gross (2010) supplemented therapists’ self-report
of treatment adherence with ratings by independent judges. Eighty-five percent agreement was
found between therapist and independent adherence ratings, with therapists having a tendency to
report higher adherence to the protocol than did independent raters. The finding that therapists
tended to view themselves as more treatment adherent than did independent raters further
emphasizes the need for objective adherence measures. Rater bias may deter therapists from
being aware of and improving on potential areas of weakness (Horenstein, Houston, & Holmes,
1973); using independent judges to rate treatment fidelity should minimize this bias (Bechger et
al., 2010).
Common method variance. In prior TAM validation studies, youth and/or caregivers
completed both the TAM and treatment outcome measures (e.g., delinquent behavior). Common
method variance bias (Campbell & Fiske, 1959) may inflate the correlations among research
variables (i.e., youth rating their own delinquent behavior and their therapist’s performance) due
to the common method used to collect the data in one sitting (i.e., filling out questionnaires). As
is the case with rater bias, common method variance bias can threaten the validity of the
conclusions drawn about the association between measures of different constructs (Podsakoff et
al., 2003; Reio, 2010). An independent rater would have the single task of rating the clinician’s
behavior and would conduct ratings in a less biased manner.
MST principles are unevenly represented. While the TAM was developed to evaluate
how well therapists engage in MST-related behaviors, the items do not fully represent the nine
principles that operationalize MST. This is a significant limitation given that “these principles
serve as the foundation for the model and a template against which all interventions can be
MST CODING PROTOCOL 6
compared to judge fidelity” (p.13; Henggeler, Schoenwald, Borduin, Rowland, & Cunningham,
2009). Further, the literature does not clearly explain how these items were developed. For
example, the item “my family and the therapist had similar ideas about ways to solve problems,”
while arguably important to general treatment alliance, does not clearly stem from one of the
principles. A cursory review of the TAM shows that a large percentage of the items correspond
to Principle 4 (Interventions are present-focused and action-orientated) and Principle 7
(Interventions should require daily/weekly effort), whereas most principles are represented
minimally by this measure. By contrast, the current study’s revised protocol measures fidelity to
each of the nine MST principles equally.
The limitations noted above could be addressed by using an observer-rated measure of
MST fidelity, a method that has been suggested in past studies of MST (Huey et al., 2000).
Moncher and Prinz (1991) recognized the importance of multiple internal raters but argued that
“outside sources are the most objective and independent means of verifying protocol adherence”
(p. 261). In a meta-analysis of treatment fidelity in therapy outcome research, they found that
77% of studies conducting treatment adherence checks primarily used observer ratings (Moncher
& Prinz, 1991).
Current Study Aims
More than a decade ago, Huey (2001) aimed to address the TAM’s limitations and
developed an adherence coding protocol based on the nine MST principles. This system
incorporated quantitative ratings as well as qualitative evaluations of therapist behavior in order
to provide detailed feedback to clinicians. Independent raters coded audiotaped sessions, with 30
items rated on a 6-point Likert scale (1= low degree to 6=high degree) representing the nine
MST principles. To provide better context for the ratings, coders were required to identify the
MST CODING PROTOCOL 7
key target problems, list informants who provided critical information, and diagram how well
clinicians identified the “fit” between the target issues and the youth’s social context.
Although this adherence protocol was comprehensive in scope and was fairly reliable
during the initial development phase, reliability was inconsistent in subsequent iterations.
Anecdotally, reliability was hampered by three primary factors (S. Huey, personal
communication, March 21, 2013). First, raters had a difficult time agreeing on what the key
target problems were (a Principle 1 issue), and this may have led to divergent ratings for other
principles as well. Second, coder “drift” was apparent, with raters appearing to diverge from one
another after the initial training period. Although ongoing monitoring procedures were put in
place to address this potential problem, those efforts were inadequate. Finally, a primary function
of the coding system was to provide qualitative data to use as corrective feedback for MST
clinicians. This focus on qualitative feedback may have impeded efforts to maximize coder
reliability. In order to address these reliability issues, the current study involved several revisions
to the original adherence coding system.
Revising the Huey (2001) Coding System
To refine the Huey (2001) MST coding system, several existing protocols developed for
other treatments were reviewed to determine which procedures appeared to enhance reliability
and validity. In the Motivational Interviewing (MI) context, efforts made to simplify the coding
of treatment fidelity improved reliability and ease of use of the coding system (Miller & Mount,
2001; Moyer, Martin, Manuel, Hendrickson, & Miller, 2005). Ten main elements of MI practice
were derived from coded sessions, thereby collapsing categories and shortening the original
measure by half. Similar to the protocol changes and adaptations made by Moyers et al, the
MST CODING PROTOCOL 8
current study revised the original coding system to reliably assess therapist behavior in a less
complex manner.
An important consideration in designing a coding protocol is the level of complexity of
the rating system and the level of inference required when interpreting items. The Huey (2001)
coding protocol included several qualitative coding criteria that were meant to be evaluative
(e.g., “What is the evidence that therapist understands/does not fully understand ‘fit’?”);
however, the 1 to 6 scaled items are more descriptive in nature (e.g., “To what degree did the
therapist appear to understand “fit”?”) and were retained in the revised protocol. Another coding
system, the Vanderbilt Psychotherapy Process Scale (VPPS; Strupp, Hartley, & Blackwood,
1974; Suh, Strupp, & O'Malley, 1986), uses multiple independent clinical raters to evaluate
video/audiotaped therapy sessions and ensure reliability of adherence ratings. VPPS items were
designed to minimize the level of inference required to rate specific behaviors, and are presented
as descriptive rather than evaluative (e.g., “the therapist showed warmth and friendliness toward
the patient”). Similar to the VPPS, for our revised protocol, a detailed rating manual was
developed wherein each item was clearly defined.
For the current revision efforts, coder training and monitoring was consistent with many
of the common practices described in the therapy adherence literature. For example,
undergraduate students were recruited as coders and trained/monitored extensively by the author,
a clinical science graduate student. Strupp and colleagues (1974) support the use of students with
minimal clinical experience as coders, and undergraduates are often used as coders in therapy
process studies (Dumas et al., 2001; Miller & Mount, 2001). As specified in other coding
protocols, training practices also included the use of graded learning tasks, weekly group
MST CODING PROTOCOL 9
meetings, and ongoing reliability checks of coded tapes (Dumas et al., 2001; Miller & Mount,
2001; Moyers et al., 2005).
The Huey (2001) coding manual also included detailed examples and a coding rubric that
differentiated between low (1-2), moderate (3-4), and high (5-6) adherent therapist behaviors.
This approach is typical of therapy process coding systems, and is believed to facilitate training
of novice coders. For example, Plumb and Vilardaga (2010) developed a similar process for
assessing adherence to Acceptance and Commitment Therapy (ACT), and argued that using
examples to anchor ratings categories helps guide coders during the training process. Because
other coding systems similarly endorse the use of a detailed coding manual, example coding
sheets, and corresponding examples of audiotaped sessions (Breitenstein et al., 2010; Dumas et
al., 2001), we emulated these approaches for present purposes.
While several revisions were made to the original protocol, the current study retained the
method of coding entire treatment sessions. The relevance of particular MST principles varies
significantly over the course of a session. For example, Principle 1 (Understand “fit” between
the problem and context) tends to be most relevant earlier in sessions whereas Principle 9
(Interventions promote long-term generalization) is perhaps most relevant later in sessions.
Given our goal to capture all nine MST principles, entire sessions were coded rather than
segments. Existing coding protocols have used 30-minute segments of taped sessions in their
adherence coding procedures (Dumas et al., 2010), the first 20 minutes of sessions (Moyer et al.,
2005), 15-minute composites (O’Malley et al., 1983), and full sessions (Breitenstein et al., 2010;
VNIS; Suh, Strupp, & O'Malley, 1986). While shorter segments may be more convenient to
code, the MST principles cannot be captured comprehensively in this manner. In conclusion, the
MST CODING PROTOCOL 10
current study focused on shortening and improving the Huey (2001) coding system in order to
enhance its reliability and validity.
Methods
Study Sample
The sample was drawn from an RCT assessing the efficacy of four treatments for juvenile
drug offenders, with two of the conditions incorporating MST (Henggeler, Halliday-Boykins,
Cunningham, Randall, Shapiro, & Chapman, 2006). Participants were 161 youth and their
families recruited from the Department of Juvenile Justice (DJJ) in Charleston County, South
Carolina, and all youth met DSM-IV diagnostic criteria for alcohol or drug abuse or dependence
(APA, 1994). Youth were also included if they carried a formal or informal probationary status
and resided with at least one caregiver (52% lived with a single biological or adoptive parent).
Youth who were already formally involved in substance abuse treatment or had a family member
who had already received MST treatment were excluded from the study. However, youth were
not excluded if they had a preexisting difficulties (e.g., mental health, physical health, or
intellectual), thereby improving the ecological validity of the study.
At the time of enrollment, youth were between 12-17 years of age (M= 15.2, SD= 1.1),
and mostly male (83%). Sixty-seven percent were African American, 31% European-American,
and 2% biracial. Primary caregivers achieved a median of a 12
th
grade education and youth had
been arrested an average of 3.5 times (SD=2.5) prior to participation in the study. The median
family-reported income was in the $10,000-$15,000 range and over a third of the sample was
receiving financial assistance at the time of the study. Consistent with the demographics for the
original study, the majority of the sample selected for this study was comprised of male (77.5%),
African American (65%) youth with female primary caregivers (95%) who were also
MST CODING PROTOCOL 11
predominantly African American (65%). Additional demographic information can be found in
Table 1.
The original study used a 4 (treatment type) X 3 (time) factorial design with random
assignment to conditions. Youth were assigned to usual services through family court (FC), drug
court (DC), drug court plus MST (DC/MST), or drug court plus MST with contingency
management (DC/MST/CM) (Henggeler et al., 2006). Pretreatment (T1) assessments were
conducted within 3 days of the youth’s entry into the study and follow-up assessments were
conducted at 4 months (T2) and 12 months post recruitment (T3). Families who participated in
the study were paid $75 for completing each assessment. For present purposes, the sample
included only those families who were treated in the two MST conditions.
MST was administered by six master’s-level therapists with degrees in social work,
psychology, or education, and with an average of 5 years post-degree clinical experience. Three-
person teams provided treatment in the two MST conditions. Two of the therapists had previous
MST experience ranging from 1 to 3 years; all were female, three were African American, three
were European American, and ages ranged from 25 to 50 years. MST therapists provided in-
home services, were available 24 hours a day, 7 days a week, and interacted with families from 2
to 15 hours a week as needed. Working closely with caregivers, therapists emphasized family
strengths, and provided skills to cope with disruptive or maladaptive behaviors. DC/MST was
provided under the drug court rubric and focused on risk factors across multiple contexts. In
addition to the services provided in DC/MST, the DC/MST/CM condition incorporated
components of contingency management. Specifically, therapists created self-management plans
in collaboration with caregivers and completed functional analyses of youth’s substance use.
MST CODING PROTOCOL 12
Moreover, youth could be rewarded for clean drug screens (i.e., voucher system; Henggeler et
al., 2006).
Procedures
Coding system revisions. As previously stated, the primary goal in revising the Huey
(2001) coding protocol was to simplify the coding process, enhance reliability, and validate the
protocol. In constructing the revised coding system, the protocol was reduced from 46 to 29
pages. Lengthy examples formatted as conversations between an MST therapist and family
members were removed. Shorter and varied examples were retained under the low, medium, and
high adherence categories. These examples were supplemented as needed in order to provide
case illustrations demonstrating low, moderate, and high levels of adherence. Second, the coding
form was reduced from 14 to 4 pages (Appendix C). While the Likert item rating format (ranging
from 1 to 6) was retained, the fit circles and comment boxes were removed, and coders were no
longer required to take detailed, structured notes regarding session content.
The 30 coded items reflect the nine treatment principles, with each Principle represented
by two to four “subprinciple” items and one “overall” item. For instance, Principle 1 (The
primary purpose of assessment is to understand the "fit" between the family-identified problems
and their broader systemic context) is composed of Subprinciple 1 (P1S1; The therapist
understood what key factors contributed to the target problems(s) or issues; i.e., understood
“fit”), Subprinciple 2 (P1S2; The therapist tapped all sources needed to appropriately evaluate
“fit”), and an Overall Principle score (P1S3; Overall, this principle was properly implemented)
that reflects the conceptual average of the two subprinciples. Total session adherence was
calculated by taking the average of all nine “overall” items (i.e., P1S3, P2S3, P3S3, P4S5, P5S3,
P6S3, P7S3, P8S4, P9S3), thus creating an “MST Principle Adherence” (MPA) composite.
MST CODING PROTOCOL 13
Training. Once revisions were made to the coding system, five undergraduate research
assistants with no prior exposure to MST completed 16 hours of training (not including reading
and coding hours) over a period of nine weeks to become familiar with the MST intervention
protocol and the revised coding system. In addition to weekly group meetings, graded learning
tasks covering MST material were administered to trainees in the form of weekly quizzes.
Research assistants were required to pass four quizzes covering ten chapters (Henggeler, 1998)
with a grade of 80% or better. Those who did not pass the quizzes were required to re-take a
different version of the quiz covering the same material.
After meeting the above training criteria, coders were assigned seven training tapes to
code independently using the revised system; ratings were then compared to an expert rating
1

and discussed during meetings that were held twice a week. The training tapes used for the
current study were selected from a prior MST clinical trial focused on substance-abusing youth;
these tapes had previously been used for training purposes. The coding manual was further
revised throughout training when discrepancies in coding became apparent and when
descriptions of principles needed further clarification as per coder feedback. Following training,
two research assistants left the project due to coding inconsistencies and scheduling conflicts.
Three research assistants were retained and achieved acceptable reliability (ICCs=0.83) and
consistency (64-79% agreement) during the training process.
Measures
Coding. Using a random number generator, the three coders were organized into pairs to
independently rate each of the 40 selected sessions. In order to monitor and address rater drift,
weekly meetings were held and discrepant ratings for every session were discussed. Moreover,

1
Expert ratings were developed by the author and Dr. Stanley Huey, Jr., the author of the original MST coding
system. Sessions were individually coded and consensus ratings were determined following a discussion regarding
discrepant ratings.
MST CODING PROTOCOL 14
every third week, an extra identical session was assigned to all three coders and compared to the
expert rating to further control for rater drift; as with the training tapes, discrepant ratings were
then discussed as a group. Although coders were aware that an extra session had been added to
their weekly assignments, they were blind as to which session was used for reliability purposes.
Sessions were randomly selected from the first month of treatment for each of the
families with recordings available. Families had to have at least one session available that met all
of the following criteria: (1) recorded within the first month of treatment, (2) more than 15
minutes and less than 90 minutes in length, (3) reasonably audible (i.e., no severe static,
background noise, or muffled voices), and (4) involved therapy for most of the session (e.g.,
sessions wherein the family mostly completed surveys or learned how to use a breathalyzer were
excluded). While 81 families were included in the MST conditions, only 69 of these had
recordings available, but 29 of these families did not meet criteria and were thus excluded.
Sessions for the remaining 40 families were randomly assigned to coder dyads. When more than
one recorded session met criteria for a particular family, one session was randomly selected and
assigned. Session length ranged from 23 minutes to 78 minutes (M=44 minutes) and sessions
were recorded 5 days to 31 days
2
after the start of treatment.
Therapist adherence measure. This measure was used to assess the concurrent validity
of the revised coding protocol. The TAM (Henggeler & Borduin, 1992) is a 26-item
questionnaire that evaluates the extent to which therapists engage in behaviors consistent with
MST. Participants completed an 80-item version of the TAM, a measure developed for a prior
study, which includes items reflecting the general principles underlying MST interventions. The
original 26 items were used for current purposes. Ratings were collected from youth, caregivers,

2
One of our selected tapes was recorded 35 days after the start of treatment. This was the only available recording
for the family and was determined to be close enough to the first month of treatment to use for current purposes.
MST CODING PROTOCOL 15
and therapists at one month intervals over the first five months of treatment. As the coded
sessions were randomly selected from the first month of treatment, only TAM data from month
one of MST was used to assess concurrent validity.
Using the TAM, adherence constructs have typically been developed through factor
analysis (Henggeler et al., 1997). The TAM factor scores used in the current study were based on
a prior evaluation of MST with substance-abusing youth, a population similar to that of the
current study (Huey et al., 2000). Five caregiver factors (Therapist-Directed Sessions, Family-
Therapist Collaboration, Family-Therapist Consensus, Therapist Encourages Responsibility, and
Nonproductive Sessions), five youth factors (Adherence, Nonproductive Sessions, Therapist
Attempts to Change Interactions, Lack of Direction, and Focus on Progress/Noncompliance), and
four therapist factors (Family-Therapist Collaboration, Productive Sessions, Collaboration to
Change Interactions, and Therapist Adherence) were used. A composite factor score was then
calculated for each informant.
Outcome measures. The following outcome measures were collected at pretreatment, 4
months post-treatment, and 12 months post-treatment, and were used to assess the predictive
validity of the revised coding protocol.
The Child Behavior Checklist (CBCL; Achenbach, 1991) was completed by adolescents
and caregivers. This 113-item questionnaire evaluates youth internalizing problems (i.e.,
depression) and externalizing problems (i.e., aggressive or delinquent behaviors). T scores for the
broadband Externalizing scale were used for the current study.
The Self-Report Delinquency Scale (SRDS; Elliott et al.,1983) is a 24-item questionnaire
that examines a range of delinquent behaviors from minor (“How many times have you run away
from home?”) to major (“How many times have you attacked someone with the idea of seriously
MST CODING PROTOCOL 16
hurting or killing him or her?”) acts of delinquency. Youth were asked to report on behavior
occurring over the last 6 months. The General Delinquency subscale was used for outcome
analyses.
Youth arrests were tracked through computerized records housed at the South Carolina
DJJ. For youth over the age of 16 years, adult criminal records were also collected from the
South Carolina Law Enforcement Division.
The Form 90 (Miller, 1996) is an interview based on the time line follow-back
methodology used to estimate specific amounts of alcohol and other drugs consumed on a daily
basis. Youth were asked to highlight important events and report quantities of alcohol and drug
use on a calendar of the previous 90 days. For the present study, this data was used to calculate
the total number of days of alcohol use and total number of days of marijuana use.
Urine drug screens for cannabis and cocaine were collected using the 3-Test Integrated
Cup supplied by BioTechNostix (Markham, Ohio) before each drug court appearance. For
cannabis, the minimum detectable level is 50 ng/μl and the sensitivity is 50%. In line with the
juvenile drug court protocols, youths with unexcused absences (e.g., did not show, runaway) and
youths that had been recently placed in detention, thereby missing court, were counted as having
positive cannabis urine screens. Further, juveniles with excused absences (e.g., attending class)
were counted as having negative/clean drug screens. For present purposes, drug screens were
dichotomized as positive (1 or more positive drug screens) or negative (no failed drug screens)
for compiled drug screen data at 4 months post-treatment and 12 months post-treatment.
Analyses
Statistical analyses were conducted using SPSS 19. Intraclass correlation coefficients
(ICCs) were calculated for all three pairs of coders to estimate reliability. The convention
MST CODING PROTOCOL 17
developed by Cicchetti (1994)’s for evaluating the usefulness of ICCs was adopted for the
current study: below .40 = poor, .40 to .59 = fair, .60 to .74 = good, and .75 to 1.00 = excellent.
Because several outcomes in the current study represent a type of count variable (e.g.,
number of delinquent acts [SRDS], number drinking days [Form 90]), predictive validity was
tested using negative binomial generalized estimating equations (GEE; Liang & Zeger, 1986).
GEE detaches from the classic assumption of statistical independence in traditional regression
models by estimating a working correlation matrix. Two-way interactions between the MST
Principle Adherence (MPA) composite and time on the outcome variables were separately
evaluated in a GEE model in the presence of covariates. To interpret the negative binomial
results, the coefficients were exponentiated (i.e., Expß) and yielded rate ratios (RRs). Similar to
odds ratios in logistic regression, a value larger than 1 indicates a percentage increase in counts
for each unit increase in the predictor variable, and a value less than 1 indicates a percentage
decreases in the outcome for each unit decrease in the predictor variable. Additionally, a GEE
linear model was used for CBCL data (i.e., normal distribution), and logistic regression analyses
were used for dichotomized data (i.e., arrest/no arrest and dirty/clean drug screens).
Youth and primary caregiver sex and race were associated with attrition patterns of
outcome data (included in Table 2). Further, as these demographic variables may also be
theoretically related to the outcomes (e.g., different baselines of alcohol consumption for males
and females), they were added as covariates in our models. The relationship of MPA to each
outcome was not significantly different between the adjusted and non-adjusted models. The
following results represent data from the non-adjusted models.

MST CODING PROTOCOL 18
Results
Descriptive Statistics
The sample sizes, means, standard deviations, and ranges for all study variables are
presented in Tables 2 and 3. Most outcome variables were positively skewed, indicating that less
severe problem behavior was more common than severe problem behavior. For instance, 48% of
youth at T1, 78% of youth at T2, and 89% of youth at T3 reported zero drinking days on the
Form 90. For analytic purposes, both youth arrest and drug screen data were dichotomized to
either zero (no arrest, all negative drug screens) or one (one or more arrests, one or more positive
drug screens). This method is commonly used in prevention research when variables are not truly
continuous and skewed with infrequent data greater than zero; researchers often focus on
comparing youth who do engage to those who do not engage in certain problem behavior
(Farrington & Loeber, 2000).
Our sample only included youth in the DC/MST (n=14), or DC/MST/CM (n=26)
conditions. No significant group differences in adherence were found, t(38) = -0.10; p=.92, with
both conditions yielding average ratings of 4.6. Further, Chi Square analyses showed no
differences in demographic variables between families that were selected (N=40) and families
not selected (N=41) for the current study (see Appendix D). However, DC/MST/CM families
were significantly more likely than DC/MST families to be selected for inclusion in the final
sample, χ2 (1) = 4.50, p=.03.
Reliability
During training, it was apparent that reliability varied greatly across the nine principles.
For instance, training reliability for Principle 8 was poor (ICC=0.36) while reliability for
Principle 3 was excellent (ICC= 0.88). Moreover, Principle 6 was nearly always rated as “high,”
MST CODING PROTOCOL 19
as the developmental needs of the youth were rarely ignored during the MST sessions; because
this principle showed very little variance, reliability could not be calculated. While measuring
reliability for individual principles was informative in revising the coding manual, the MPA
composite was ultimately used as an overarching index of adherence to all nine principles.
Reliability estimates for each separate principle and for the MPA composite are presented
in Table 4 for the 40 families included in this study. Reliability varied from fair (ICC=.549) to
excellent (ICC=.759) between the rater dyads for MPA and was good when calculated across all
raters (ICC=.642). Principles 1 and 4 were the most consistently reliable indices across dyads
and Principle 5 yielded the poorest reliability.
Concurrent Validity
Concurrent and predictive validity were tested by computing the mean of each dyad’s
MPA for each session. For concurrent validity, correlations were conducted between MPA and
caregiver, youth, and therapist TAM scores. Table 5 presents the Pearson correlations among the
TAM ratings, and between MPA and the TAM ratings. Results show that there were no
relationships between MPA and the TAM ratings, and TAM informant’s ratings were not
correlated with each other; all correlations yielded non-significant p-values ranging from .403 to
.868.
Predictive Validity
A negative binomial GEE with unstructured covariance pattern was fit to each outcome
for count data. Predictors included time and MPA. GEE results are presented in Table 6 and
logistic regression results are presented in Table 7. MPA was not associated with most treatment
outcomes. However, a significant effect of MPA on changes in alcohol use was found
(Expß=.022, 95% CI [.002, .214]); specifically, number of days of alcohol use decreased by 98%
MST CODING PROTOCOL 20
from T1 to T3 for youth involved in sessions that were rated as highly adherent to MST. MPA as
related to alcohol consumption over time is presented in Figure 1. While this is a significant
effect, it should be noted that 70% of the sessions were given MPA ratings higher than 4.5.
Further, a GEE linear model with unstructured covariance pattern was fit to CBCL
outcome data. Predictors included time and MPA. There was a significant effect of MPA on
Externalizing T-scores from T1 to T2 (B= -.8.804, 95% CI [-15.532, -2.077]); sessions with high
MPA ratings predicted a nearly 9-point decrease in youth-reported externalizing symptoms at
T2. CBCL scores over time as related to MPA are represented in Figure 2.
Discussion
In the current study, we attempted to establish the reliability and initial validity of a
revised rater protocol for assessing adherence to MST. Trained undergraduate coders were used
to assess the MST treatment principles, as these principles were developed to guide therapist
behavior throughout treatment (Henggeler et al., 2009). While the TAM is the common index of
MST adherence, several factors may limit its utility. First, the TAM is used by informants
directly involved in the therapy process, which presents various biases when assessing adherence
(Bechger et al., 2010; Breitenstein et al., 2010; Horenstein et al., 1973; Kozlowski & Kirsch,
1987; Podsakoff et al., 2003; Reio, 2010). Second, TAM items measure some principles
extensively whereas other principles are poorly represented (e.g., Principle 1 is represented by
only 1 item). Thus, there may be a need for an observer-rated index of adherence to the nine
MST principles.
Indeed, recent efforts have been made to address this gap in the adherence literature. In
the context of a randomized trial of MST, Weiss and colleagues (2013) had an independent
expert in MST review audiotaped sessions and rate adherence to the nine principles, with the
MST CODING PROTOCOL 21
clinical goal of assuring adequate fidelity during treatment. While the authors acknowledged the
importance of the principles, they did not report on the reliability or validity of those ratings.
Reliability of the Revised System
Ratings based on the original coding protocol (Huey, 2001) frequently showed poor
reliability, perhaps because raters often disagreed on the key target problems presented in
sessions (S. Huey, personal communication, March 21, 2013). Using the revised system,
however, there was 75% agreement on the target problems and Principle 1 yielded the highest
reliability (ICC=.847). While there is considerable variation in the ICCs across the different
dyads and across the principles (see Table 4), reliability was good overall.
There are several possible explanations for variations in reliability across coders.
Ambiguity of session content may have led to disparate interpretations. During our weekly
coding meetings, several topics were discussed regarding discrepant ratings, particularly relating
to Principles 5 and 9. Not surprisingly, these two principles yielded the lowest reliability. A
common disagreement had to do with the level of therapist effort in relation to certain treatment
barriers (e.g., ignoring versus addressing a difficult youth’s stubbornness). Further, poor sound
quality (e.g., static, background noise, inaudible voices) frequently prevented one member of the
coding pair from hearing a crucial statement made in session, thereby leading to divergent
ratings. Undoubtedly, the highly subjective nature of the nine MST principles, among other
complexities, makes the coding process extremely challenging.
Concurrent Validity of the Revised System
Despite variations in the TAM’s validity across published studies, this measure is widely
used as the primary index of MST fidelity. Conceptually, we expected to find a moderate
MST CODING PROTOCOL 22
relationship between the observer-based MPA composite and the client- and therapist-rated
TAM scores. However, no significant associations were found.
There are several possible reasons for the lack of concurrent validity. First, although our
coding protocol and the TAM are conceptually linked, methodological differences may have
obscured potential associations. On the one hand, coding was completed by well-trained,
independent raters who focused on the content of a single session. On the other hand, TAM
informants were directly involved in therapy, and were asked to report their ratings of what
happened in treatment “over the past week.” It may simply be that the two approaches reveal
very different things about adherence in the context of MST.
Second, the unstable nature of TAM factors may have affected our findings. Indeed, other
authors (Henggeler et al., 1997; Huey et al., 2000) have noted that TAM factor structures vary
considerably across informants and across studies. This structural variation may explain why the
three informant TAM scores did not correlate with each other. Further, the factor scores used
presently may not have been the best fit to our study sample, as they were based on a prior
evaluation of MST, albeit with a similar population. Additional TAM scoring methods may
address the limitations in our findings. For instance, not all TAM items correspond to one of the
nine principles; perhaps excluding such extraneous items from the factor structures would
establish concurrent validity with MPA.
Predictive Validity of the Revised System
Given past literature on the positive effects of MST, it was hypothesized that higher
adherence to MST, as measured by our revised coding system, would be associated with
decreases in problem behavior at post-treatment and follow-up. MST adherence significantly
predicted two of the eight outcomes assessed in the present study. Youth who were in high-
MST CODING PROTOCOL 23
adherence sessions in the first month of treatment decreased their alcohol consumption by 98%
from pre-treatment to 12-month follow-up. Youth in high-adherence sessions drank considerably
more than youth in low-adherence sessions at pre-treatment and showed sharper decreases in
drinking at follow-up. The directionality of this effect is unclear. It may be that therapists are
responding to youths’ severe drinking with higher rates of adherence to the treatment model.
Conversely, heavy drinkers may be more responsive to high-adherence therapists compared to
youth who drink less. While no research currently exists on the direct association between
baseline drinking rates and subsequent MST adherence, TAM scores have been found to be
lower when youth displayed more severe pre-treatment antisocial behavior (Schoenwald,
Halliday-Boykins, & Henggeler, 2003). Nonetheless, high MPA predicted steep reductions in
alcohol consumption for our study sample.
MPA was also associated with reductions in youth-rated externalizing symptoms. Youth
who were involved in high-adherence sessions during the first month of treatment reported a
nearly 9-point decrease in externalizing symptoms on the CBCL from pre-treatment to post-
treatment. The association between higher MST adherence and decreases in adolescent
externalizing symptomology has often been found when measuring adherence with the TAM
(Chapman & Schoenwald, 2011; Henggeler et al., 1997; Schoenwald et al., 2000; Schoenwald et
al., 2008).
The broader treatment literature presents mixed results regarding the effect of adherence
on outcomes (Webb, DeRubeis, & Barber, 2010; Perepletchikova & Kazdin, 2005). Some
explanations for discrepancies in research findings are related to the TAM limitations we address
in our study. Perepletchikova and Kazdin (2005) noted that self-reported therapist adherence
may inflate the reported level of integrity implemented within a treatment protocol. Furthermore,
MST CODING PROTOCOL 24
what appears to be a weak link between adherence and outcomes may be due to the methods
used to measure treatment fidelity. Despite the mixed findings presented in Webb et al.’s (2010)
recent meta-analysis, strong associations have been found between observer-rated treatment
adherence and reductions in externalizing behaviors and substance use (Hogue, Henderson,
Dauber, Barajas, Fried, & Liddle, 2008). Our results provide additional support for the moderate
association between treatment fidelity and outcomes, and add to the body of knowledge relating
to measurements of adherence that use independent raters.
Clinical Implications
The MST quality-assurance process aims to implement evidence-based interventions with
the fidelity needed to produce positive treatment outcomes for youth and their families
(Schoenwald et al., 2004). Our revised coding system, once replicated with a larger and more
diverse study sample, has the potential to be used within this quality-assurance framework. The
present study used coders with minimal clinical experience; when compared to treatment experts,
non-experts have been shown to be equally capable of rating complex interventions (Baker,
Haltigan, Brewster, Jaccard, & Messinger, 2010; Waldinger, Schulz, Hauser, Allen, & Crowell,
2004). Providing clinicians with detailed feedback could be beneficial in strengthening the
implementation of MST. As shown in the current study, the difference between an MPA of 4.0
and 4.5 can result in very different impacts on youth alcohol consumption and behavioral
outcomes. Our findings further support the benefits of well-implemented MST for substance-
abusing youth.
Limitations and Future Directions
In addition to the obvious restrictions our small sample size presents, a major limitation
was the homogeneity of MST skill in our therapist sample. The six therapists in this trial showed
MST CODING PROTOCOL 25
moderate to high adherence to the nine MST principles (Appendix E). While high fidelity is ideal
when providing services to high-needs families, adherence diversity across therapists is
preferable when attempting to establish the reliability and validity of therapy process coding
protocols (Breitenstein et al., 2010). As with other abbreviated coding systems (MITI: Moyer et
al., 2005), our protocol may be more useful during a therapist’s first year as an MST clinician
while skills are still being developed. Also, the generalizability of these findings may be limited,
as the “high adherence” clinicians assessed here may not be representative of the larger number
of MST therapists who practice in “real-world” settings (Huey et al., 2000).
Once stronger evidence for reliability is established, future research might investigate
how adherence to specific treatment principles affects treatment outcomes. For example, given
its central role in the MST assessment and treatment process, adherence to Principle 1 (Finding
the Fit) could be more predictive of problem remediation than the MPA composite. Lastly,
future iterations of this coding system could analyze adherence trajectories across the course of
treatment rather than limit its utility to the first month of MST (Schoenwald et al., 2003).
Conclusion
Overall, our revised protocol for evaluating MST adherence was considerably more
reliable than the previous system (Huey, 2001). Additionally, our coding system addresses rater
bias and common method variance bias associated with self-report measures. Although reliability
varied from poor to excellent across coder dyads and principles, overall reliability was
consistently good. Moreover, higher MST adherence during the first month of therapy was
predictive of significant decreases in externalizing behavior and alcohol consumption. While
several limitations were apparent (e.g., small sample size, narrow range of adherence), our
MST CODING PROTOCOL 26
findings show promise for future research examining the effectiveness of observer-rated
measurements of MST adherence.

MST CODING PROTOCOL 27
References
Achenbach, T. M. (1991). Manual for the Child Behavior Checklist and 1991 Profile.
Burlington: University of Vermont, Department of Psychiatry.
American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders
(4th ed.). Washington, DC: Author.
Alvarez-Jimenez, M., Wade, D., Cotton, S., Gee, D., Pearce, T., Crisp, K., & Gleeson, J. F.
(2008). Enhancing treatment fidelity in psychotherapy research: Novel approach to
measure the components of cognitive behavioural therapy for relapse prevention in first-
episode psychosis. Australian and New Zealand Journal of Psychiatry, 42, 1013-1020.
Baker, J. K., Haltigan, J. D., Brewster, R., Jaccard, J., & Messinger, D. (2010). Non-expert
ratings of infant and parent emotion: Concordance with expert coding and relevance to
early autism risk. International Journal of Behavioral Development, 34, 88-95.
Bechger, T. M., Maris, G., & Hsiao, Y. (2010). Detecting halo effects in performance-based
examinations. Applied Psychological Measurement, 34, 607-619.
Breitenstein, S. M., Fogg, L., Garvey, C., Hill, C., Resnick, B., & Gross, D. (2010). Measuring
implementation fidelity in a community-based parenting intervention. Nursing Research,
59, 158-165.
Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and
design. Cambridge, MA: Harvard University Press.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the
multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
Centis, E. E., Marzocchi, R. R., Di Luzio, R. R., Moscatiello, S. S., Salardi, S. S., Villanova, N.
N., & Marchesini, G. G. (2012). A controlled, class ‐based multicomponent intervention
MST CODING PROTOCOL 28
to promote healthy lifestyle and to reduce the burden of childhood obesity. Pediatric
Obesity, 7, 436-445.
Chapman, J. E., & Schoenwald, S. K. (2011). Ethnic similarity, therapist adherence, and long-
term multisystemic therapy outcomes. Journal of Emotional and Behavioral Disorders,
19, 3-16.
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and
standardized assessment instruments in psychology. Psychological Assessment, 6, 284-
290.
Cong, Z., Feng, D., Liu, Y., & Esperat, M. (2012). Sedentary behaviors among Hispanic
children: Influences of parental support in a school intervention program. American
Journal of Health Promotion, 26, 270-280.
Dumas, J. E., Lynch, A. M., Laughlin, J. E., Smith, E., & Prinz, R. J. (2001). Promoting
intervention fidelity: Conceptual issues, methods, and preliminary results from the
EARLY ALLIANCE prevention trial. American Journal of Preventive Medicine, 20, 38-
47.
Farrington, D. P., & Loeber, R. (2000). Some benefits of dichotomization in psychiatric and
criminological literature. Criminal Behavior and Mental Health, 10, 100–122.
Foley, S. H., O'Malley, S., Rounsaville, B., Prusoff, B. A., & Weissman, M. M. (1987). The
relationship of patient difficulty to therapist performance in interpersonal psychotherapy
of depression. Journal of Affective Disorders, 12, 207-217.
Godfrey, E. E., Chalder, T. T., Ridsdale, L. L., Seed, P. P., & Ogden, J. J. (2007). Investigating
the 'active ingredients' of cognitive behaviour therapy and counselling for patients with
MST CODING PROTOCOL 29
chronic fatigue in primary care: Developing a new process measure to assess treatment
fidelity and predict outcome. British Journal of Clinical Psychology, 46, 253-272.
Henggeler, S.W. (2011). Efficacy studies to large-scale transport: The development and
validation of multisystemic therapy programs. Annual Review of Clinical Psychology, 7,
351-381.
Henggeler, S. W., & Borduin, C. M. (1990). Family therapy and beyond: A multisystemic
approach to treating the behavior problems of children and adolescents. Pacific Grove,
CA: Brooks/Cole.
Henggeler S.W., Clingempeel W.G., Brondino M.J., & Pickrel S.G. (2002). Four-year follow-up
of multisystemic therapy with substance-abusing and substance-dependent juvenile
offenders. Journal of the American Academy of Child and Adolescent Psychiatry, 41,
868.
Henggeler, S.W., Halliday-Boykins, C.A., Cunningham, P.B., Randall, J., Shapiro, S.B.,
Chapman J.E. (2006). Juvenile drug court: enhancing outcomes by integrating evidence-
based treatments. Journal of Consulting and Clinical Psychology, 74, 42–54
Henggeler, S.W., Letourneau, E.J., Chapman, J.E., Borduin, C.M., Schewe, P.A., & McCart,
M.R. (2009). Mediators of change for multisystemic therapy with juvenile sexual
offenders. Journal of Consulting and Clinical Psychology, 77, 451-462.
Henggeler, S. W., Borduin, C. M. (1992). Multisystemic therapy adherence scale. Unpublished
instrument, Department of Psychiatry and Behavioral Sciences, Medical University of
South Carolina.
Henggeler, S. W., Borduin, C. M., Schoenwald, S. K., Huey, S. J., & Chapman, J. E. (2006).
Multisystemic therapy adherence scale – revised (TAM-R). Unpublished instrument,
MST CODING PROTOCOL 30
Department of Psychiatry and Behavioral Sciences, Medical University of South
Carolina.
Henggeler, S. W., Melton, G. B., Brondino, M. J., Scherer, D. G., & Hanley, J. H. (1997).
Multisystemic therapy with violent and chronic juvenile offenders and their families: The
role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical
Psychology, 65, 821-833.
Henggeler, S. W., Rodick, J., Borduin, C. M., Hanson, C. L., Watson, S. M., & Urey, J. R.
(1986). Multisystemic treatment of juvenile offenders: Effects on adolescent behavior
and family interaction. Developmental Psychology, 22, 132-141.
Henggeler, S. W., Schoenwald, S. K., Borduin, C. M., Rowland, M. D., & Cunningham, P. B.
(1998). Multisystemic treatment of antisocial behavior in children and adolescents. New
York: Guilford Press.
Henggeler, S. W., Schoenwald, S. K., Borduin, C. M., Rowland, M. D., & Cunningham, P. B.
(2009). Multisystemic therapy for antisocial behavior in children and adolescents (2nd
ed.). New York, NY US: Guilford Press.
Henggeler, S. W., Schoenwald, S. K., Liao, J. G., Letourneau, E. J., & Edwards, D. L. (2002).
Transporting efficacious treatments to field settings: The link between supervisory
practices and therapist fidelity in MST programs. Journal of Clinical Child and
Adolescent Psychology, 31, 155-167.
Henggeler, S. W., Pickrel, S. G., & Brondino, M. J. (1999). Multisystemic treatment of substance
abusing and dependent delinquents: Outcomes, treatment fidelity, and transportability.
Mental Health Services Research, 1, 171-184.
MST CODING PROTOCOL 31
Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008).
Treatment adherence, competence, and outcome in individual and family therapy for
adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544-
555.
Horenstein, D., Houston, B., & Holmes, D. S. (1973). Clients', therapists', and judges'
evaluations of psychotherapy. Journal of Counseling Psychology, 20, 149-153.
Huey, S. J. (2001). Adherence Training Manual for Multisystemic Therapy (MST): Anchors and
Guidelines for Coding Audiotaped Therapy Sessions. Unpublished instrument,
Department of Psychology, University of Southern California.
Huey, S. J. & Henggeler, S. W. (2005). Family-therapist collaboration as a mechanism of
change in multisystemic therapy. Paper presented at the 118
th
Annual Meeting of the
American Psychological Association, San Diego, CA.
Huey, S. J., Henggeler, S. W., Brondino, M. J., Pickrel, S. G. (2000). Mechanisms of change in
multisystemic therapy: reducing delinquent behavior through therapist adherence and
improved family and peer functioning. Journal of Consulting and Clinical Psychology,
68, 451–67.
Kazdin, A. E., Siegel, T. C., & Bass, D. (1992). Cognitive problem-solving skills training and
parent management training in the treatment of antisocial behavior in children. Journal of
Consulting and Clinical Psychology, 60, 733-747.
Kozlowski, S. W., & Kirsch, M. P. (1987). The systematic distortion hypothesis, halo, and
accuracy: An individual-level analysis. Journal of Applied Psychology, 72, 252-261.
Liang, K., & Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models.
Biometrika, 73, 13–22.
MST CODING PROTOCOL 32
Moncher, F. J., & Prinz, R. J. (1991). Treatment fidelity in outcome studies. Clinical Psychology
Review, 11, 247-266.
Moyers, T. B., Martin, T., Manuel, J. K., Hendrickson, S. L., & Miller, W. R. (2005). Assessing
competence in the use of motivational interviewing. Journal of Substance Abuse
Treatment, 28, 19-26.
Naar-King, S., Ellis, D. A., Idalski, A., Frey, M. A., & Cunningham, P. (2007). Multisystemic
therapy decreases parental overestimation of adolescent responsibility for type 1 diabetes
management in urban youth. Families, Systems, & Health, 25, 178-189.
Naar-King, S., Ellis, D., Kolmodin, K., Cunningham, P., & Secord, E. (2009). Feasibility of
adapting multisystemic therapy to improve illness management behaviors and reduce
asthma morbidity in high risk African American youth: A case series. Journal of Child
and Family Studies, 18, 564-573.
Naar-King, S., Ellis, D., King, P. S., Lam, P., Cunningham, P., Secord, E., Bruzzese, J., Templin,
T. (2014). Multisystemic therapy for high-risk African American adolescents with
asthma: A randomized clinical trial. Journal of Consulting and Clinical Psychology.
Advance online publication. http://dx.doi.org/10.1037/a0036092.
Perepletchikova, F., & Kazdin, A. E. (2005). Treatment integrity and therapeutic change: Issues
and research recommendations. Clinical Psychology: Science and Practice, 12, 365-383.
Pickrel, S. G., & Henggeler, S. W. (1996). Multisystemic therapy for adolescent substance abuse
and dependence. Child and Adolescent Psychiatric Clinics of North America, 5, 201-211.
Plumb, J. C., & Vilardaga, R. (2010). Assessing treatment integrity in acceptance and
commitment therapy: Strategies and suggestions. International Journal of Behavioral
Consultation and Therapy, 6, 263-295.
MST CODING PROTOCOL 33
Podsakoff, P. M., MacKenzie, S. B., Lee, J., & Podsakoff, N. P. (2003). Common method biases
in behavioral research: A critical review of the literature and recommended remedies.
Journal of Applied Psychology, 88, 879-903.
Prevatt, F., & Kelly, F. (2003). Dropping out of school: A review of intervention programs.
Journal of School Psychology, 41, 377-395.
Reio, T. R. (2010). The threat of common method variance bias to theory building. Human
Resource Development Review, 9, 405-411.
Schoenwald, S. K., Halliday-Boykins, C., & Henggeler, S. W. (2003). Client-level predictors of
adherence to MST in community service settings. Family Process, 42, 345-359.
Schoenwald, S. K., Sheidow, A. J., Letourneau, E. J. (2004). Toward effective quality assurance
in evidence-based practice: Links between expert consultation, therapist fidelity, and
child outcomes. Journal of Clinical Child & Adolescent Psychology, 33, 94-104.
Schoenwald, S. K., Carter, R. E., Chapman, J. E., & Sheidow, A. J. (2008). Therapist adherence
and organizational effects on change in youth behavior problems one year after
multisystemic therapy. Administration and Policy in Mental Health and Mental Health
Services Research, 35, 379-394.
Schoenwald, S. K., Henggeler, S. W., Brondino, M. J., & Rowland, M. D. (2000). Multisystemic
therapy: Monitoring treatment fidelity. Family Process, 39, 83-103.
Suh, C. S., Strupp, H. H., & O'Malley, S. (1986). The Vanderbilt process measures: The
Psychotherapy Process Scale (VPPS) and the Negative Indicators Scale (VNIS). In L. S.
Greenberg, W. M. Pinsof (Eds.) , The psychotherapeutic process: A research handbook
(pp. 285-323). New York, NY US: Guilford Press.
MST CODING PROTOCOL 34
Thorndike, E. L. (1920). A constant error in psychological ratings. Journal of Applied
Psychology, 4, 25-29.
Tremblay, R. E., Pagani-Kurtz, L., Mâsse, L. C., Vitaro, F., & Pihl, R. O. (1995). A bimodal
preventive intervention for disruptive kindergarten boys: Its impact through mid-
adolescence. Journal of Consulting and Clinical Psychology, 63, 560-568.
Waldinger, R. J., Schulz, M. S., Hauser, S. T., Allen, J. P., & Crowell, J. A. (2004). Reading
others' emotions: The role of intuitive judgments in predicting marital satisfaction,
quality, and stability. Journal of Family Psychology, 18, 58-71.
Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and
treatment outcome: A meta-analytic review. Journal of Consulting and Clinical
Psychology, 78, 200-211
Weisman, A. G., Okazaki, S., Gregory, J., Tompson, M. C., Goldstein, M. J., & Rea, M. (1998).
Evaluating therapist competency and adherence to behavioral family management with
bipolar patients. Family Process, 37, 107-121.
Weiss, B., Han, S., Harris, V., Catron, T., Ngo, V. K., Caron, A., Gallop, R., Guth, C. (2013). An
independent randomized clinical trial of multisystemic therapy with non-court-referred
adolescents with serious conduct problems. Journal of Consulting and Clinical
Psychology, 81, 1027-1039.
Weisz, J. R., Huey, S. J., & Weersing, V. (1998). Psychotherapy outcome research with children
and adolescents: The state of the art. Advances in Clinical Child Psychology, 20, 49-91

MST CODING PROTOCOL 35
Table 1

Demographic Characteristics of Selected Subjects (N=40)
Characteristic M SD Range
Age (years) 15.37 1.13 14-17
n %
Youth
Gender (male) 31 77.5
Ethnicity
Black/African American 26 65
White/Caucasian 13 32.5
Other 1 2.5
Primary Caregiver
Gender (female) 38 95
Ethnicity
Black/African American 26 65
White/Caucasian 14 35

MST CODING PROTOCOL 36
Table 2

Means, Standard Deviations, and Sample Sizes for Outcome Variables
Outcome Variable
T1 T2 T3
Mean SD n Mean SD n % miss Mean SD n % miss
CBCL Externalizing
T-score

Caregiver report 62.75 10.52 40 55.36 12.44 39 2.5 51.94 13.27 35 12.5
Youth report 59.13 12.87 40 52.62 14.79 37 7.5 50.69 13.51 36 10
Form 90 – Alcohol 5.18 12.01 40 1.05 4.62 40 0 1.25 6.50 36 10
Form 90 – Marijuana 28.30 27.27 40 2.75 7.97 40 0 1.67 4.84 36 10
SRDS (General
Delinquency)
30.95 37.78 40 20.74 30.95 38 5 15.67 28.19 36 10
% n % miss % n % miss
Arrests 35 14 0 50 20 0
Urine Screens – Cannabis 65 24 7.5 51 18 12.5
Urine Screens – Cocaine

11 4 7.5 11 4 12.5
Note: CBCL=Child Behavior Checklist; SRDS=Self-Report Delinquency Scale; % miss=percentage of missing data.
a
T1 = pretreatment, T2 = 4-month post-treatment, T3 = 12-month post-treatment
b
Arrests represent all types of arrest (drug and violent).
c
Arrest and Drug data represent dichotomous proportions with “%” representing percentage of sample with one or more arrest and one or more
positive drug screen
MST CODING PROTOCOL 37
Table 3

Means, Standard Deviations, Ranges, and Sample Sizes for Predictor Variables

Note: TAM=Therapist Adherence Measure; MPA=MST Principle Adherence.
a
TAM composite scores based on 5 caregiver, 5 youth, and 4 therapist factor scores.

Predictor Variable
Time 1
Mean SD Range N
TAM Composite
Caregiver 3.59 .45 2.46-4.54 36
Therapist 4.10 .99 1.46-5.24 38
Youth 3.88 .81 1.60-5.28 35
MPA 4.63 .34 3.67-5.06 40
N %
MPA Range
3.5 – 4.0 2 5
4.1 – 4.5 10 25
4.6 + 28 70
MST CODING PROTOCOL 38
Table 4
Intraclass Correlation Coefficients (ICCs) for Rater Dyads on the 9 Multisystemic Therapy
(MST) Principles

Note: Interrater reliability estimates using the revised MST coding protocol.
MPA=MST Principle Adherence Composite.
a
Intraclass correlation coefficients using a consistency definition.
b
Single measures are reported.
c
N/A indicates zero variances in the ratings; coefficients were unable to be calculated
* fair reliability, ** good reliability, *** excellent reliability.

Variable
Raters 1 & 2

(n=14)
Raters 1 & 3

(n=16)
Raters 2 & 3

(n=14)

Principle 1

.442*

.847***

.653**
Principle 2 .507* .335 .672**
Principle 3 .587* .263 .576*
Principle 4 .660** .582* .479*
Principle 5 .386 .347 N/A
Principle 6 N/A N/A N/A
Principle 7 .091 .482* .708**
Principle 8 N/A .469* .552*
Principle 9 .258 .481* .264
MPA .549* .591* .759***
MST CODING PROTOCOL 39
Table 5

Intercorrelations Among Caregiver, Youth, and Therapist TAM Composite Ratings and
Correlations Between TAM ratings and the MST Principle Adherence Composite

Measure 1 2 3 4
1. TAM- Caregiver -- .148 -.039 -.162
2. TAM- Youth -- -.030 -.116
3. TAM- Therapist -- -.122
4. MPA --
Note: TAM=Therapist Adherence Measure; MPA=MST Principle Adherence.
a
Based on N = 33-38.
b
P > .05 for all correlations presented.
c
TAM composite scores based on 5
caregiver, 5 youth, and 4 therapist factor scores.

MST CODING PROTOCOL 40

Table 6

Generalized Estimation Equation (GEE) Model Estimates Comparing MST Principle Adherence and Outcome
Variables Across Time

Outcome Variable
T1 – T2 T1 – T3
B
95% CI
Lower Upper p-value B
95% CI
Lower Upper p-value
Scale – Linear
CBCL Externalizing T-score
Caregiver report -.383 -8.444 7.677 .926 4.954 -3.143 13.050 .230
Youth report -8.804 -15.532 -2.077 .010 -4.017 -12.220 4.186 .337
Counts- Negative Binomial
Exp(ß)
95% CI
Lower Upper

p-value Exp(ß)
95% CI
Lower Upper p-value
Form 90 – Alcohol .405 .011 14.932 .624 .022 .002 .214 .001
Form 90 – Marijuana
.352 .021 5.941 .469 7.614 .678 85.506 .100
SRDS (General
Delinquency)
.731 .207 2.587 .627 .437 0.139 1.375 .157
Note: CBCL=Child Behavior Checklist; SRDS=Self-Report Delinquency Scale.
a
T1 = pretreatment, T2 = 4-month post-treatment, T3 = 12-month post-treatment

MST CODING PROTOCOL 41

Table 7

Logistic Regression of Dichotomized Data with MST Principle Adherence for T2 and T3
Outcome Variable
T2 T3
Exp(ß)
95% CI
Lower Upper p-value Exp(ß)
95% CI
Lower Upper p-value
Arrest
0 arrest / 1+ arrest 1.124 .157 8.036 .908 9.938 .242 908.003 .226
0-1 arrest / 2+ arrests 15.517 .103 2338.847 .284 .754 .092 6.179 .792
Drug Screens (positive/negative)
Cannabis 3.754 .494 28.519 .201 1.40 .176 11.152 .751
Cocaine 10.899 .110 1081.844 .309 1.162 .021 64.963 .942
Cannabis and Cocaine -- -- -- -- 4.292 .512 35.967 .179
Note: Drug screens dichotomized as negative/clean drug screens vs. one or more positive/dirty drug screens
a
T1 = pretreatment, T2 = 4-month post-treatment, T3 = 12-month post-treatment
b
Arrests represent all types of arrest (drug and violent)
MST CODING PROTOCOL 42
Figure 1
Generalized Estimation Equation (GEE) Model Estimates Comparing MST Principle Adherence
(MPA) and Form 90 Alcohol Use Over Time

MST CODING PROTOCOL 43
Figure 2
Generalized Estimation Equation (GEE) Model Estimates Comparing MST Principle Adherence
(MPA) and CBCL Externalizing T-score (Youth-report) Over Time

MST CODING PROTOCOL 44
Appendix A: MST Principles
1. Finding the Fit: The primary purpose of assessment is to understand the ‘fit’ between the
identified problems and their broader systemic context.
2. Positive and Strength Focused: Therapeutic contacts should emphasize the positive and
should use systemic strengths as levers for change.
3. Increasing Responsibility: interventions are designed to promote responsible behavior
and decrease irresponsible behavior among family members
4. Present- focused, Action-orientated and Well-defined: interventions are present-focused
and action-orientated, targeting specific and well-defined problems.
5. Targeting Sequences: interventions target sequences of behavior within and between
multiple systems that maintain identified problems.
6. Developmentally Appropriate: interventions are developmentally appropriate and fit the
developmental needs of the youth.
7. Continuous Effort: interventions are designed to require daily or weekly effort by family
members.
8. Evaluation and Accountability: interventions effectiveness is evaluated continuously
from multiple perspectives, with providers assuming accountability for overcoming
barriers to successful outcomes.
9. Generalization: interventions are designed to promote treatment generalization and long
term maintenance of therapeutic change by empowering caregivers to address family
members’ needs across multiple systemic contexts.

MST CODING PROTOCOL 45
Appendix B: Therapist Adherence Measure

MST CODING PROTOCOL 46

Schoenwald, Henggeler, Brondino, and Rowland (2000)
MST CODING PROTOCOL 47

Appendix C: MST Adherence Audiotape Coding Form

MST CODING PROTOCOL 48

MST CODING PROTOCOL 49

MST CODING PROTOCOL 50

MST CODING PROTOCOL 51
Appendix D: Test of Differences

Pearsons Chi Square and Crosstabs Assessing Differences in Conditions and Demographics for
MST Conditions

Note: DC/MST=Drug Court/Mutlisystemic Therapy condition; DC/MST/CM=Drug Court/Mutlisystemic
Therapy/Contingency Management condition

Characteristic

Selected
for sample
Not selected
for sample
X
2
df Sig. (2-sided)
Sex of youth
Females 9 4 2.44 1 .12
Males 31 37
Race of youth
African American 26 29 1.19 2 .55
White 13 12
Other 1 0
Sex of caregiver
Females 38 39 .001 1 .98
Males 2 2
Race of caregiver
African American 26 29 1.51 2 .47
White 14 11
Asian 0 1
Condition
DC/MST 14 24 4.50 1 .034*
DC/MST/CM 26 17
Total 40 41
MST CODING PROTOCOL 52
Appendix E: Individual Therapist Adherence

Therapist ID Number of Sessions MPA
51 11 4.54
49 6 4.60
48 3 4.61
35 5 4.62
50 6 4.63
23 9 4.77
Note: MPA= MST Principle Adherence

Abstract (if available)

Abstract Treatment adherence to multisystemic therapy (MST) has typically been measured using the MST Therapist Adherence Measure (TAM

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Within session therapist behavior in multisystemic therapy

PDF

Prison of Peace in a jail setting: a randomized trial of a conflict resolution program for incarcerated women

PDF

Cumulative risk as a moderator of multisystemic therapy effects for juvenile offenders

PDF

Perspective taking behavior and social outcomes in late adolescence

PDF

Using integrative data analysis to evaluate gender differences in effects of multisystemic therapy for justice-involved youth

PDF

Towards validating therapists’ in-session behaviors of cultural competence

PDF

Predictors and outcomes across the transition to fatherhood

PDF

Links between discrimination and sleep difficulties: romantic relationships as risk and resilience factors

PDF

The role of primary care in the use of specialty mental health services: an investigation of utilization patterns among African Americans and non-Hispanic Whites

PDF

Direct and indirect predictors of traumatic stress and distress in orphaned survivors of the 1994 Rwandan Tutsi genocide

PDF

Racial and gender exclusion affect novel group identity

PDF

A "second chance" program for vulnerable young adults: a program evaluation and a randomized controlled trial of adjunctive motivational interviewing to improve retention

PDF

Differential associations between same- and cross-ethnicity negative peer attitudes and adjustment outcomes among Vietnamese and Mexican American adolescents

PDF

Risky behaviors, interpersonal conflict, and their relation to fluctuations in adolescents’ diurnal HPA rhythms

PDF

Minority stress and intersectionality: how are identity, discrimination, and biological stress related to depression and suicidality among multiple minority subgroups?

PDF

An investigation of the factors associated with electronic aggression among college students

PDF

Popularity as a predictor of friendship affiliation in adolescence

PDF

Family aggression, prosocial friends, and the risk of dating and friend victimization in late adolescence

PDF

Social support, self-efficacy, and gender in treatment adherence of heart failure patients

PDF

Savoring the service: a mixed-methods study of Latine community health workers’ health promotion efforts

Asset Metadata

Creator Gillespie, Marie Laura (author)

Core Title Revision of the multisystemic therapy (MST) adherence coding protocol: assessing the reliability and predictive validity of adherence to the nine MST principles

School College of Letters, Arts and Sciences

Degree Master of Arts

Degree Program Psychology

Publication Date 08/11/2014

Defense Date 05/27/2014

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag coding protocol,multisystemic therapy,OAI-PMH Harvest,treatment adherence

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Huey, Stanley J., Jr. (committee chair), John, Richard (committee member), Margolin, Gayla (committee member)

Creator Email marie.gillespie@usc.edu,mls.gillespie@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-458867

Unique identifier UC11286945

Identifier etd-GillespieM-2808.pdf (filename),usctheses-c3-458867 (legacy record id)

Legacy Identifier etd-GillespieM-2808.pdf

Dmrecord 458867

Document Type Thesis

Format application/pdf (imt)

Rights Gillespie, Marie Laura

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA