Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
The impact of healthcare interventions using electronic health records: an evaluation within an integrated healthcare system
(USC Thesis Other)
The impact of healthcare interventions using electronic health records: an evaluation within an integrated healthcare system
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
1
The impact of healthcare interventions using electronic health records:
An evaluation within an integrated healthcare system
by
Lewei Duan
A Dissertation Presented to the
FACULTY OF THE USC GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
Health Economics
December 2021
Copyright 2021 Lewei Duan
ii
Dedication
This dissertation is dedicated to Aeneas Shen.
iii
Acknowledgements
I would like to acknowledge everyone who played a role in my academic accomplishments. First
of all, my family, who supported me with love and understanding. Secondly, my committee
members, each of whom has provided patient advice and guidance throughout the research
process. Finally, my friends –without you, I could never have reached the current level of
success.
iv
v
vi
vii
viii
Abbreviations
Abbreviation Meaning Page
EHR electronic health record 1
KPSC Kaiser Permanente Southern California 3
DOAC direct oral anticoagulants 4
AF atrial fibrillation 4
BHV bioprosthetic heart valves 4
CABG coronary artery bypass grafting 5
PCI percutaneous coronary intervention 5
RCT randomized clinical trial 2
CDS clinical decision support 5
* The above table describes abbreviations used throughout the dissertation. The page on which each one
is defined or first used is also given.
ix
Abstract
Electronic health records (EHRs) provide ample opportunities to enhance the quality of patient
care, aid in medical decision-making within clinical practice, and facilitate clinical research.
Kaiser Permanente Southern California (KPSC) implemented a comprehensive health
information system that represents the largest collection of private electronic health record data
in the world. In this dissertation, we utilized the rich information concerning both patients and
clinicians that Kaiser Permanente Southern California’s EHR system offers, and we evaluated
the use of EHR data for clinical and research purposes across three disciplines. First, we
compared the safety and effectiveness of different medications on patient outcomes; and we
found that off-label use of direct oral anticoagulants (DOACs) in patients with bioprosthetic
heart valves can achieve comparable outcomes to warfarin. Second, we addressed the important
issue of unmeasured confounding in EHR data; and we created a simulation-based, prior-
knowledge guided algorithm to support the inference of clinical study results when an
unmeasured confounding issue is an area of concern. Third, we evaluated the effects of a policy-
based, computer-generated clinical decision support tool on physician behavior; and we found
that the extra friction created by the way KPSC integrated clinical decision support into its
processes in response to California State Law had a demonstrable influence on prescribers’
behavior, leading to a reduction in opioid prescription rates.
1
Introduction
From the early 1990s, the Institute of Medicine in the United States endorsed a digital turn in
medical record-keeping in order to compensate for the inadequacies of paper-based health
records
1, 2
. In recent decades, electronic records have continued to develop in response to rapid
technological advancements
3
, and substantial progress has been made to improve digital
healthcare delivery systems. The term “Electronic health record” (EHR) has now been widely
adopted by clinicians and researchers to describe the digital transformation of healthcare data.
A patient’s EHR provides longitudinal information relevant to that individual’s wellness,
health and healthcare in a computer-processable form which is shared across patients and
healthcare providers, and is interoperable across healthcare settings
4
. EHR enables authorized
access to patients’ health records in real-time, allowing healthcare professionals to spend more
time with patients
5
, and facilitating enhanced communication within the healthcare team
6
. In
addition to increasing accessibility to clinical information, EHR reduces the duplication of tests
and work, while also improving the safety and quality of care provided
7
. Furthermore, EHR
expands the capacity to utilize clinical data for monitoring patient outcomes and conducting
audits and medical research.
Medical informatics has experienced major growth, particularly in the field of integrating
medical information found in EHRs into diagnostic tools such as clinical decision support
(CDS)
8, 9
. EHR-based CDS functions enable knowledge sharing and peer learning by providing
access to up-to-date medical information
10
. In its earlier developmental stages, CDS was used to
remind clinicians about allergies to medication, drug-drug interactions and any other abnormality
in a patient’s health record
11
. Gradually, CDS evolved into a tool to aid and influence physicians’
decision-making. For instance, some CDS tools were designed to improve clinicians’ compliance
2
to evidence-based clinical guidelines and preventive care protocols by requiring them to respond
to a computer auto-generated reminder when prescribing certain medications
12
.
EHR systems open new horizons, and let investigators explore potential causal
relationships between treatment and outcome by using observational data, but without the
budgetary constraints and ethical issues associated with randomized clinical trials (RCTs). EHR
amounts to millions of separate records, and thus allows for the estimation of treatment effects
with lower variance in the results when compared to clinical studies using a smaller sample size.
On the other hand, the growing volume and complexity of healthcare datasets, as well as their
dynamic nature, pose diverse challenges in the implementation, application, and analysis of
EHRs. Issues of missing data, selection bias, measurement errors and other in-built limitations of
EHR systems inevitably engender risks of substantial bias in effect estimations.
Although the functionality of EHR has expanded towards goals of enhancing patient care,
sharing evidence-based knowledge among clinicians and patients, facilitating medical research,
and improving healthcare delivery, significant challenges in the implementation and adoption of
EHR nevertheless remain. This dissertation evaluates EHR experiences across different
disciplines within an integrated healthcare system. I am specifically interested in the following
subjects. First, how can EHR be used to assess and evaluate patient outcomes based on the
treatments they receive, taking treatments to refer to either medications or medical procedures?
Can EHR cast light on the comparative effectiveness of different treatment options? Second, how
do we account for problems or gaps encountered during the data collection stage and address the
potential bias in our study results as a consequence? For example, with the issue of unmeasured
confounding, researchers have identified numerous techniques to address these potential
problems, but none are entirely satisfactory; the deployment of statistical tools on EHR datasets
3
can lead to a more accurate range of true estimated effects. Third, how can EHR work in
practical ways to achieve policy-based goals? Of particular interest here is how EHR, when
incorporated with clinical decision supports, can effectively influence clinicians’ medical
decision-making and behavior.
The core dataset for this dissertation comes from the EHR system of Kaiser Permanente.
Founded in 1945, Kaiser Permanente is one of the nation’s largest U.S. not-for-profit health
plans, serving 12.5 million members in eight regions
13
. This integrated healthcare delivery
system provides its members with the whole breadth of healthcare services, including but not
limited to preventive care, hospital and medical services, emergency care, and pharmaceutical
services
14
. Beginning in 2014, Kaiser Permanente started to implement Kaiser Permanente (KP)
HealthConnect
®15
, a comprehensive health information system. KP HealthConnect
®
included
multiple functions, of which a key one was EHR. KP’s EHR system incorporated comprehensive
documentation from across all care settings, such as emergency services, inpatient, and
outpatient settings. Integral to KP’s technology was a clinical decision support system, as well as
the facility for the real-time integration of information from lab, pharmacy, radiology and other
ancillary systems
14
.
The three retrospective cohort studies that make up this dissertation are based on the EHR
data from one of KP’s eight regions: namely, the Kaiser Permanente Southern California (KPSC)
Health System. The KPSC Healthcare system is a regional integrated healthcare system
consisting of 15 medical centers and 235 medical offices across 14 areas in the Southern
California region. More than 7,600 physicians and 26,000 nurses provide health services for 4.6
million members
13
. Through the Kaiser Foundation Health Plan, members enroll for
comprehensive healthcare insurance that also includes pharmaceutical benefits. KPSC provides
4
care to an ethnically and socioeconomically diverse population that is broadly representative of
the racial / ethnic groups in Southern California
16
. Comprehensive information on the medical
care that KPSC members receive is captured electronically through a centralized data warehouse,
with electronic datasets that link information on demographics, administrative data, pharmacies,
laboratories, and healthcare utilization data from both ambulatory visits and hospitalizations. All
of the three study protocols were approved by the KPSC Institutional Review Board. And
waivers of informed consents were obtained for each of the studies because of their observational
nature.
Underpinning this dissertation are three studies (chapters), each of which evaluates the
impact of healthcare interventions by using electronic health records in the context of KPSC’s
integrated healthcare system. Each study utilizes the rich information about either patients and/or
clinicians that the EHR system offers; some of them compare the effectiveness of medications
and medical procedures on patient outcomes, while the other evaluates the effects of policy-
based, computer-generated interventions on physician behavior. Finally, this dissertation also
addresses the important issue of unmeasured confounding in EHR data, for which I have
provided a potential solution that I call “L-Table”.
The first study compares the effectiveness and safety of direct oral anticoagulants
(DOACs) as opposed to warfarin in patients with atrial fibrillation (AF) and bioprosthetic heart
valves (BHV). This study shows that off-label use of DOACs in patients with bioprosthetic heart
valves can achieve comparable outcomes in terms of both effectiveness and safety in comparison
to warfarin. The second study addresses issues of sensitivity with unmeasured confounding. I
created a simulation-based, prior-knowledge guided algorithm to support the inference of clinical
research when an unmeasured confounding issue is suspected. This study then illustrates how
5
this innovative approach can be applied when comparing the effectiveness of coronary artery
bypass grafting (CABG) vs. percutaneous coronary intervention (PCI) on the health outcomes
for patients with stable ischemic heart disease. The third study estimates the effects on opioid
prescription rates of an electronic reminder – or clinical decision support tool – to encourage
compliance with California State law that was designed to address the opioid epidemic. In this
study, we study the extra friction created by the way KPSC integrated clinical decision support
into its processes had a demonstrable influence on prescribers’ behavior, leading to a reduction
in opioid prescription rates. In all three studies, KPSC’s EHR system was used to manage
patients’ health records, assist physicians’ treatment decisions, or influence behavioral changes
on the part of physicians. Combining all three studies, therefore, is a deep methodological
interest in how EHR can contribute to a broad range of study-design and research questions, and
how it might do so in ways that are reliable and efficient.
6
Chapter 1: Comparison of Direct Oral Anticoagulants versus
Warfarin in Patients with Atrial Fibrillation and Bioprosthetic
Heart Valves
Duan L, Doctor JN, Adams JL, Romley JA, Nguyen LA, An J, Lee MS. Comparison of Direct
Oral Anticoagulants Versus Warfarin in Patients With Atrial Fibrillation and Bioprosthetic Heart
Valves. Am J Cardiol. 2021 May 1;146:22-28. doi: 10.1016/j.amjcard.2021.01.016. Epub 2021
Jan 30. PMID: 33529622.
1.1 Abstract
There are limited data regarding direct oral anticoagulants (DOACs) for stroke prevention in
patients with bioprosthetic heart valves (BHVs) and atrial fibrillation (AF). The objectives of
this study were to evaluate the ambulatory utilization of DOACs and to compare the
effectiveness and safety of DOACs versus warfarin in patients with AF and BHVs. We
conducted a retrospective cohort study at a large integrated health care delivery system in
California. Patients with BHVs and AF treated with warfarin, dabigatran, rivaroxaban or
apixaban between September 12, 2011 and June 18, 2020 were identified. Inverse probability of
treatment-weighted comparative effectiveness and safety of DOACs compared with warfarin
were determined. Use of DOACs gradually increased since 2011, with a significant upward in
trend after a stay-at-home order related to COVID-19. Among 2672 adults with BHVs and AF
who met the inclusion criteria, 439 were exposed to a DOAC and 2233 were exposed to
warfarin. For the primary effectiveness outcome of ischemic stroke, systemic embolism and
transient ischemic attack, no significant association was observed between use of DOACs
compared with warfarin (HR 1.19, 95% CI 0.96-1.48, p=0.11). Use of DOACs was associated
with lower risk of the primary safety outcome of intracranial hemorrhage, gastrointestinal
bleeding, and other bleed (HR 0.69, 95% CI 0.56 to 0.85, p <0.001). Results were consistent
7
across multiple subgroups in the sensitivity analyses. These findings support the use of DOACs
for AF in patients with BHVs.
Keywords:
Atrial fibrillation (AF); Bioprosthetic heart valves (BHVs); Anticoagulation; Stroke; Bleeding
1.2 Introduction
Atrial fibrillation (AF) increases the risk of thromboembolic ischemic stroke and systolic
embolism
17
. The use of oral anticoagulation therapy substantially reduces ischemic stroke
risk
18
. Compared to warfarin, direct oral anticoagulants (DOACs) are associated with similar or
better ischemic stroke prevention and lower risk of serious bleeding complications
19-21
. The
ease of dosing with DOACs have led to a steady increase in their use
22, 23
. The coronavirus
disease 2019 (COVID-19) pandemic has brought additional consideration to the care of patients
receiving warfarin therapy
24
. For patients receiving warfarin therapy, frequent blood draws for
INR monitoring may be difficult because of lockdowns. Switching patients from warfarin to
DOACs is a potential strategy to minimize patients’ need to leave their homes. DOACs were
approved for use in nonvalvular AF. The efficacy and safety of DOACs achieved in clinical
trials for nonvalvular AF may not apply to patients with valvular heart disease. For example,
dabigatran used in patients who had undergone mechanical valve replacement led to excess
thromboembolic and bleeding events
25
. One recent study suggests DOAC may be a reasonable
alternative to warfarin in patients with BHVs
26
. Using a large population-based cohort from an
integrated health care delivery system in California, we evaluated the ambulatory utilization
8
pattern of DOACs in patients with AF and BHVs, and compared the effectiveness and safety of
DOACs versus warfarin in this population.
1.3 Methods
This is a retrospective cohort study using data from the Kaiser Permanente Southern California
(KPSC) Health System
16
. The study protocol was approved by the KPSC Institutional Review
Board. A waiver of informed consent was obtained because of the observational nature of the
study.
Adult patients (age ≥ 18 years) with AF and BHVs between September 12, 2011 and June
18, 2020 were initially identified using International Classification of Diseases (ICD) 9/10 codes
(Table 1-1). Presence of BHVs was confirmed by manual review of diagnoses, problem lists and
anticoagulation clinic notes. In this cohort, 1724 had bioprosthetic aortic valves (332 in the
DOAC group and 1392 in the warfarin group) and 943 had bioprosthetic mitral valves (104 in
the DOAC group and 839 in the warfarin group). The type of bioprosthetic valve could not be
determined in 5 cases due to incomplete documentation. The index date was defined as the first
medication dispensed date (DOAC or warfarin) during the study period. Patients who were not
KPSC members, did not have continuous one-year membership prior to the index date (allowing
at 30-day gap), or did not have continuous one-year pharmacy-benefit coverage were excluded.
For the primary analysis, we included patients from September 12, 2011 to March 18, 2020 with
at least 6 months of follow-up to allow adequate follow-up data.
Covariates were identified in the following categories: demographics, medical
comorbidities, cardiac risk factors, and use of cardiac medications. Medical comorbidities and
cardiac risk factors were collected using ICD codes from the year prior to the index date.
Baseline concomitant medications was identified using outpatient pharmacy records. Patients
9
were followed until they reached a study end point, death, disenrolled from the health plan, or
the end of the study (June 18, 2020). Patients were then classified into DOAC-exposed or not
exposed groups based on their receipt of dispensed DOAC prescriptions from a KPSC pharmacy
during the study period. The following DOACs were evaluated: dabigatran, rivaroxaban and
apixaban. Patients were considered exposed to a DOAC if they had at least two dispensed
prescriptions of a DOAC. Patients were considered exposed to warfarin and not exposed to a
DOAC if they were never dispensed a prescription of DOAC, but received at least two
prescriptions of warfarin during the study period. The DOAC group included a subset of patients
who had previous warfarin exposures, while the warfarin cohort did not include any patients
exposed to a DOAC.
The primary effectiveness outcome was a composite of ischemic stroke, transient
ischemic attack (TIA) or systemic embolism. The primary safety outcome was a composite of
major bleeding including gastrointestinal bleeding, intracranial hemorrhage, and bleeding from
other sites. The secondary outcomes were all-cause mortality, and the individual ischemic or
bleeding outcomes. Outcomes were identified using ICD-9 and ICD-10 codes (Table 1-1) in the
primary discharge diagnosis position for inpatient hospitalizations.
Mortality data was extracted from a mortality data mart with integrated death information
derived from multiple sources including California state death master files, Social Security
Administrative death master files, hospital deaths and insurance enrollment records.
Descriptive statistics on covariate include counts and percentages, as well as means and
standard deviations. Time-series plots of utilization were used to depict DOAC versus warfarin
usage during the study period. Plots of the 30-day average proportion of DOAC dispensed at
KPSC outpatient pharmacy indicate dynamic dispensing
27
. Inverse probability of treatment
10
weighting (IPTW) operated to balance baseline characteristics between the groups
28-30
.
Specifically, generalized boosted models estimated the propensity score, and corresponding
weights were applied to estimate the average treatment effect (ATE) of switching the population
from Warfarin to DOAC. The underlying propensity score model included 29 baseline covariates
such as age, sex, race, comorbidities and baseline medication use, with up to 3-way interactions
among these covariates. Monte Carlo methods were used to establish the reference distribution
for Kolmogorov-Smirnov (KS) statistics for each covariate
29
. Maximum of KS statistics across
variables were used as stopping criteria. Absolute standardized bias was used to quantify the
balance between the variables in two comparison groups resulting from inverse probability of
treatment weighting. This value is also referred to as standardized effect size, calculated as the
difference in means or proportions of a variable divided by a pooled estimate of the standard
deviation of the variable
31
. A difference of 0.10 or less was considered as adequate balance
between the two groups
32
.
Reports of both crude and weighted event rates employ a denominator of 1000 person-
years. Weighted rates were based on the IPTW population and expressed as population average
treatment rate per 1000 person-years. A combination of propensity score weighting and
covariate adjustment for unbalanced covariates was applied in a Cox proportional hazard
regression model to ensure a “doubly robust” treatment effect estimator
33, 34
. Incident Rate
Difference (IRD) per 1000 person-year, hazard ratios (HRs) and 95% confidence intervals (CIs)
depict group differences.
Stratified analyses using IPTW Cox proportional hazards models evaluated heterogeneity
of the ATE by potential effect modifiers, including age (18-64, 65-74, ≥ 75 years), sex, race
(white, black, hispanic, asian, other), body mass index (BMI) (<30 vs. ≥ 30), chronic kidney
11
disease (CKD) class, Charlson comorbidity index
35
, CHA2DS2-VASc score
36
, and HAS-BLED
score
37
(Table 1-2). ATEs on 9 endpoints: all-cause mortality, composite stroke, ischemic
stroke, systemic embolism, transient ischemic attack (TIA), composite bleed, gastrointestinal
(GI) bleeding, Intracranial Hemorrhage, and other bleed, were assessed. P value < 0.05 was the
nominal level of significance.
Sensitivity analyses were performed to assess the robustness of the findings based on
primary analyses. Definition of the cohort inclusion and the length of follow-up time were
adjusted for the sensitivity analyses. The effects of DOAC use versus warfarin use on clinical
outcomes were assessed for DOAC patients without previous warfarin exposure, and DOAC
patients using dabigatran only. Measurement of effects were repeated at 3- and 12- month
follow-up. All statistical analyses were conducted in SAS 9.4 (SAS Institute Inc., Cary, NC,
USA) and R version 3.60
38
.
1.4 Results
The temporal utilization pattern of anticoagulation therapy is shown in Figure 1 and Figure 2 in
the Supplement. At the start of the study period, most patients were treated with warfarin. There
was a gradual increase in the use of DOAC over time. The trend of the proportion of patients
who treated with DOAC increased after March 19, 2020, when the State of California issued a
stay-at-home order due to the COVID-19 pandemic.
To compare the effectiveness and safety of direct oral anticoagulants versus warfarin, we
identified patients with AF and BHVs treated with anticoagulant therapy. Between 2011 and
2020, there were 3,351 adults with BHVs and concomitant AF who were treated with
12
anticoagulation (Figure 3). After excluding 197 patients without continuous 1-year membership
information and 491 patients without adequate follow-up time, the final study cohort included
2,672 patients. Of these patients, 439 patients received DOAC, and 2,233 patients took warfarin
alone. Among the DOAC users, 362 took dabigatran, 60 apixaban, and 17 rivaroxaban.
Table 1-3 shows the baseline characteristics of the study population. Demographic
factors including age, sex, and race / ethnicity were similar between the DOACs and the warfarin
group. A higher proportion of patients in the warfarin group had heart failure, coronary artery
disease and end-stage renal failure. A higher proportion of patients in the DOACs group had a
history of ischemic stroke. After propensity score weighting, the groups were well-balanced,
with most covariates having absolute standardized differences below 0.10 except for age and
chronic kidney disease stages.
During follow-up (mean=2.9, SD=2.2) years, there were 180 ischemic strokes, 11
systemic embolisms, and 82 TIAs. The primary effectiveness outcome was a composite of
ischemic stroke, TIA, and systemic embolism. After applying propensity score weighting, no
statistically significant association was observed between use of DOACs vs warfarin and
composite stroke events (HR 1.19, 95% CI 0.96-1.48, p=0.106) (Table 1-4).
A total of 371 composite bleeding events were observed. Patients on DOACs had fewer
bleeding events relative to warfarin (HR 0.69, 95% CI 0.56 to 0.85, p <0.001). The rate of
intracranial hemorrhage was significantly lower in the DOACs group vs warfarin (HR 0.43, 95%
CI 0.25 to 0.73, p<0.001).
There were 40.37 and 52.78 total deaths per 1000 person-years among DOACs users and
warfarin users (weighted IRD -10.92/1000 person-years, 95% CI -31.72 to 9.88). No association
13
was observed between exposure to DOACs and all-cause mortality (HR 0.87, 95% CI 0.72 to
1.05, p=0.16).
Stratified analyses were consistent with the main findings (Figure 4, Figure 5, and Figure
6). No significant interaction was observed with age, sex, race, heart failure, history of ischemic
stroke, or history of bleeding were not significant.
Several sensitivity analyses yielded consistent results. When comparing DOACs users
without previous warfarin exposure to those treated with warfarin, DOAC exposure was
associated with comparable ischemic risk and lower risk of bleeding (Table 1-5). When the
analyses were performed comparing an individual DOAC (dabigatran) with warfarin, we found
that dabigatran was associated with lower risk of composite bleeding and comparable risk of the
composite ischemic endpoint (Table 1-6). The analyses were repeated with different follow-up
time periods (3 months and 12 months), resulting in similar findings (Table 1-7 and Table 1-8).
Similar findings were observed in patients with aortic valves and mitral valves (Supplementary
Table 1-9 and Table 1-10).
1.5 Discussion:
In this cohort of patients with BHVs and AF, we assessed the effectiveness and safety of DOACs
compared with warfarin. The principal findings are as follows: first, DOACs were as effective
as warfarin in preventing ischemic strokes, systemic embolism and transient ischemic attacks;
second, patients treated with DOACs had fewer bleeding events, including fewer intracranial
hemorrhage, when compared to warfarin; third, there was a gradual increase in DOAC use, with
14
a marked increase after March 19, 2020, when stay-at-home orders were issued in California due
to the COVID-19 pandemic.
To our knowledge, this study is one of the largest real-world evaluations comparing
DOACs and warfarin in patients with BHVs and AF. In patients with non-valvular AF, DOACs
have demonstrated at least equivalent efficacy in comparison to warfarin
19-21
. However, patients
with mechanical valves had higher thromboembolic events when treated with dabigatran
25
.
DOACs were also found to be associated with increased risk of major adverse cardiovascular
events in patients with adult congenital heart disease, a population who frequently had cardiac
surgery including valve replacements
39
.
There is limited data on the use of DOACs in the setting of BHVs. A small pilot study
involving 27 patients showed low rates of thromboembolic events associated with the use of
dabigatran in patients with BHVs and AF
40
. A sub-analysis of 104 patients with BHVs enrolled
in the ARISTOTLE trial showed no significant difference in thromboembolic events between
apixaban and warfarin
41
. In patients with BHVs and AF, one study found rivaroxaban to be
noninferior to warfarin
42
. These early results suggest DOAC use may be reasonable in patients
with BHVs. The current study expands on these early findings, showing that use of DOACs in
patients with BHVs can achieve comparable outcomes for both effectiveness and safety in
comparison to warfarin. The results were consistent regardless of age, sex, and race.
Intracranial bleeding is the most serious bleeding complication since mortality and
morbidity associated with intracranial bleed is much higher than other types of bleeding
43
. Our
finding that DOACs were associated with lower rates of intracranial hemorrhage is consistent
with results from the pivotal clinical trials. In RELY (dabigatran), ROCKET-AF (rivaroxaban),
and ARISTOTLE (apixaban), the relative risk for major bleeding and intracranial bleeding with
15
DOACs were significantly lower compared with warfarin
44
. Given the comparable risks of the
primary effectiveness endpoint, and lower risk of intracranial bleeding, DOACs may have higher
net clinical benefits compared with warfarin in this population.
The use of DOACs has steadily increased because of their ease of use, and favorable
efficacy and safety profile
45
. Yet, these drugs have seen even a greater rise in use since the
COVID-19 pandemic. Patients treated with warfarin require ongoing Internationalized
Normalized Ratio (INR) monitoring, which can be difficult during COVID-19 because of social
distancing and quarantine rules. In suitable candidates, switching patients from warfarin to a
DOAC is a potential strategy to minimize patients’ need to travel. In California, the governor
issued a stay-at-home order on March 19, 2020 due to COVID-19. We observed a significant
increase in proportion of patient treated with DOAC since March 19. Given similar safety and
effectiveness of DOACs in this population, switching from warfarin to DOAC may be advisable
during a pandemic.
There are several limitations to this study. First, there may be exposure misclassification
related to non-compliance. Nevertheless, using pharmacy dispense information allows us to
avoid any recall bias. Second, residual confounding may persist despite careful adjustment with
the use of IPTW, and firm conclusions cannot be drawn about causality. Third, use of over-the-
counter medications such as aspirin and non-steroidal anti-inflammatory agents that may
increase bleeding risk could not be accurately assessed in this study. Fourth, the majority of
patients in the DOAC group were treated with dabigatran. Clinicians wrote only a small number
of prescriptions to patients for rivaroxaban and apixaban. Additional studies that specifically
evaluate apixaban and rivaroxaban may be warranted. Fifth, information on prosthetic valve
16
function was not available. Finally, the study population has insurance. As such, the results do
not generalize to patients without insurance.
In conclusion, in this large contemporary study of patients with AF and BHVs, DOACs
were as effective as warfarin in preventing ischemic events, while associated with less
intracranial bleeds. These findings support the use of DOACs for AF in patients with BHVs.
17
18
Table 1- 1. Clinical Risk Scores.
Risk Score Condition
Points if
present
CHA 2DS 2-VASc
Congestive heart failure 1
Hypertension 1
Age ≥ 75 2
Diabetes 1
Stroke/TIA 2
Vascular disease 1
Age 65-74 1
Female 1
HAS-BLED
Uncontrolled Hypertension (hypertension with Systolic BP
≥160 mmHg)
1
Renal disease, renal transplant, Serum Creatinine >2.26 mg/dL) 1
Liver disease 1
Stroke 1
Prior Bleeding 1
Age >65 1
Medication use predisposing to bleeding 1
Alcohol use 1
Labile INR (TTR < 60%) 1
19
Table 1- 2. Baseline Demographics before and after IPTW weighting
Crude
IPTW Weighted
Variable
Warfarin
(N=2233)
DOAC
(N=439)
Standard
ized Bias
P
value
Warfarin
(N=2233)
DOAC
(N=43
9)
Standard
ized Bias
P
value
Age (years)
18-64 13.6% 12.1% 0.045
0.682
13.2% 9.8% 0.101
0.15
65-74 30.2% 31.0% 0.017
30.2% 33.9% 0.08
>=75 56.2% 56.9% 0.015
56.5% 56.3% 0.005
Sex
Women 39.0% 41.2% 0.046
0.374
39.2% 39.9% 0.014
0.82
Men 61.0% 58.8% 0.046
60.8% 60.1% 0.014
Race / Ethnicity
Asian or
Pacific
Islander
6.8% 5.9% 0.034
0.601
6.5% 4.9% 0.067
0.453
Hispanic 18.0% 17.5% 0.013
17.8% 17.9% 0.003
Non-Hispanic
Black
7.3% 5.5% 0.07
7.1% 5.1% 0.079
Non-Hispanic
White
67.4% 70.4% 0.064
68.0% 71.4% 0.073
Others 0.5% 0.7% 0.02
0.5% 0.7% 0.023
BMI (Kg/
)
<18.5 1.9% 1.6% 0.024
0.038
1.9% 1.8% 0.005
0.712
18.5 - 25 30.3% 24.6% 0.126
29.6% 27.8% 0.039
25 - 30 35.8% 35.8% 0.001 35.7% 34.1% 0.034
> 30 31.9% 38.0% 0.13 32.8% 36.3% 0.074
Insurance
Commercial 12.2% 10.5% 0.054
0.238
11.9% 9.4% 0.079
0.375
Medicaid 0.7% 1.4% 0.072
0.7% 0.9% 0.025
Medicare 87.1% 88.2% 0.033
87.4% 89.7% 0.07
20
Comorbidities
Hypertension 54.3% 56.9% 0.053 0.312
54.2% 52.4% 0.037 0.566
Congestive
Heart Failure
76.0% 70.2% 0.136 0.009
75.3% 74.3% 0.022 0.701
Myocardial
Infarction
22.1% 19.8% 0.056 0.285
22.0% 19.1% 0.071 0.252
Peripheral
Vascular
Disease
11.9% 10.7% 0.037 0.459
12.0% 10.3% 0.054 0.329
Alcoholism 10.3% 11.4% 0.036 0.496
10.4% 9.9% 0.016 0.788
Dementia 13.6% 14.4% 0.023 0.663
13.5% 12.4% 0.031 0.585
COPD 28.8% 24.6% 0.092 0.077
28.4% 25.8% 0.058 0.345
Dialysis 3.8% 1.4% 0.133 0.011
3.5% 1.8% 0.095 0.173
Chronic Kidney
Disease stage
1 21.6% 23.9% 0.055
0.017
21.8% 22.1% 0.008
0.097
2 33.7% 37.8% 0.086
34.2% 35.1% 0.019
3 37.8% 35.1% 0.057
37.7% 39.8% 0.044
4 4.7% 2.7% 0.096
4.4% 2.6% 0.088
5 2.1% 0.5% 0.123
1.9% 0.3% 0.116
Prior Events of
Clinical Outcome
Prior
Ischemic
Stroke
12.2% 16.9% 0.139 0.008
12.6% 14.2% 0.046 0.464
Prior
Systemic
Embolism
1.1% 1.4% 0.023 0.659
1.1% 0.9% 0.026 0.613
Prior TIA 9.6% 10.7% 0.038 0.469
9.7% 10.9% 0.039 0.556
Prior GI
Bleeding
12.9% 13.9% 0.028 0.589
12.9% 13.8% 0.028 0.66
Prior IC
Bleeding
2.7% 3.9% 0.071 0.175
2.7% 2.6% 0.009 0.856
Prior Other
Bleeding
6.8% 6.8% 0.003 0.957
6.7% 4.9% 0.071 0.17
21
Baseline
Medications
Anti-
hypertensive
medications
94.5% 94.5% 0 0.998
94.5% 94.9% 0.021 0.72
Diabetes
medications
27.5% 22.1% 0.122 0.019
26.8% 23.8% 0.069 0.295
Antiplatelet 29.3% 39.4% 0.218 <0.001
30.5% 33.3% 0.062 0.326
Statin 83.3% 85.9% 0.069 0.188
83.4% 84.4% 0.027 0.695
Antiarrhythm
ic
21.9% 22.8% 0.022 0.669
21.7% 20.9% 0.019 0.76
Charlson Comorbidity Index
(CCI)
0-1 5.8% 4.3% 0.063
0.4
5.6% 4.0% 0.069
0.352
2-3 23.5% 22.6% 0.023
23.8% 26.5% 0.063
>=4 70.7% 73.1% 0.053
70.6% 69.5% 0.024
CHA2DS2–VASc
Score
0-1 4.9% 3.4% 0.073
0.254
4.8% 4.0% 0.036
0.824
2-3 25.2% 27.6% 0.055
25.3% 26.0% 0.018
>=4 69.9% 69.0% 0.018
70.0% 70.0% 0
Has-Bled Score
0-2 48.7% 42.4% 0.126
0.016
47.8% 47.7% 0.003
0.962
>=3 51.3% 57.6% 0.126 52.2% 52.3% 0.003
* Baseline covariates used for IPTW: age, sex, race, body mass index, insurance type, dialysis status,
Charlson Comorbidity Index, CHA2DS2-VASc Score, HAS-BLED score, history of congestive heart failure,
history of myocardial infarction, history of peripheral vascular disease, history of uncontrolled
hypertension, history of alcoholism, history of dementia, history of chronic obstructive pulmonary
disease, history of gastrointestinal bleed, history of intracranial hemorrhage, history of other bleed,
history of ischemic stroke, history of systemic embolism, history of transient ischemic attack, indication
for warfarin usage prior to index date, baseline antiarrhythmic medication, baseline antiplatelets,
baseline heparin, baseline statin, baseline diabetes drugs, baseline antihypertensive drug. COPD:
chronic obstructive pulmonary disease; TIA: transient ischemic attack; GI: gastrointestinal; IC:
intracranial.
22
Table 1- 3. Association Between DOAC Use and Clinical Outcomes after IPTW Weighting
Crude rate/1000
person-years
IPTW Weighted
DOAC
user
(N=439)
Warfarin
user
(N=2233)
Incidence rate
difference/1000
person-year (95%CI) Hazard Ratio (95%) P value
All-Cause Mortality 40.37 52.78
-10.92 (-31.72 to 9.88) 0.87 (0.72 to 1.05) 0.155
Stroke (Composite) 43.72 31.54
10.26 (-9.89 to 30.4) 1.19 (0.96 to 1.48) 0.106
Ischemic Stroke 28.84 23.98
4.8 (-12.27 to 21.86) 1.05 (0.82 to 1.35) 0.705
Systemic Embolism 4.18 1.15
0.32 (-3.47 to 4.1) 2.1 (0.64 to 6.88) 0.219
Transient Ischemic
Attack (TIA) 15.55 10.44
3.89 (-7.85 to 15.63) 1.36 (0.94 to 1.98) 0.106
Bleed (Composite) 41.32 52.95
-16.05 (-36.01 to 3.9) 0.69 (0.56 to 0.85) <0.001
Gastrointestinal (GI)
bleeding 32.58 29.56
-0.81 (-18 to 16.37) 0.92 (0.72 to 1.17) 0.48
Intracranial
Hemorrhage 5.58 11.80
-7.69 (-15.45 to 0.07) 0.43 (0.25 to 0.73) 0.002
Other Bleed 7.00 13.97 -6.54 (-15.5 to 2.41) 0.5 (0.31 to 0.79) 0.003
* Baseline covariates used for IPTW: age, sex, race, body mass index, insurance type, dialysis status,
Charlson Comorbidity Index, CHA2DS2-VASc Score, HAS-BLED score, history of congestive heart failure,
history of myocardial infarction, history of peripheral vascular disease, history of uncontrolled
hypertension, history of alcoholism, history of dementia, history of chronic obstructive pulmonary
disease, history of gastrointestinal bleed, history of intracranial hemorrhage, history of other bleed,
history of ischemic stroke, history of systemic embolism, history of transient ischemic attack, indication
for warfarin usage prior to index date, baseline antiarrhythmic medication, baseline antiplatelets,
baseline heparin, baseline statin, baseline diabetes drugs, baseline antihypertensive drug. COPD:
chronic obstructive pulmonary disease; TIA: transient ischemic attack; GI: gastrointestinal; IC:
intracranial.
23
24
25
26
27
28
29
30
31
Figure 1- 1. Derivation of the study cohort.
Patients with prosthetic heart valves
between September 12, 2011 and March 18, 2020
(N=15342)
Patients with bioprosthetic heart valves treated
with oral anticoagulation therapy between
September 12, 2011 and March 18, 2020
(N=3951)
Patients age ≥ 18 years old on index date (first
anticoagulant dispense date)
(N=3350)
Patients with at least 6 months of follow-up time
by June 18, 2020
(N=2672)
Patients with bioprosthetic heart valves
(N=9889)
Warfarin Group
(N=2233)
DOAC Group
(N=439)
Patients with mechanical heart valves
(N=5453)
Patients with concomitant atrial fibrillation
(N=3351)
Patients <18 years of age
(N=1)
Patients with less than 1 year continuous
membership prior to the index date
(N=197)
Patients with ≥ 1 year of continuous membership
prior to the index date (allowing a 30-day gap)
(N=3163)
Patients with less than 6 months of
follow-up time by June 18, 2020
(N=491)
32
33
34
35
Chapter 2: Addressing unmeasured confounding bias with a prior
knowledge guided approach: Coronary Artery Bypass Grafting
(CABG) vs. Percutaneous Coronary Intervention (PCI) in patients
with stable ischemic heart disease
2.1 Abstract
BACKGROUND:
Unmeasured confounding may undermine the validity of observational studies. In randomized
clinical trials (RCTs) that compared CABG and PCI in patients with stable ischemic heart
disease, patients with low coronary disease complexity had similar outcomes with CABG and
PCI, while patients with high coronary disease complexity had better clinical outcomes with
CABG. Although RCTs have good internal validity they may have limited generalizability to
other settings. Some exploration of the generalizability to a particular setting may be warranted if
observational data is available. We present a method to compare the outcomes for CABG and
PCI with observational data when coronary disease complexity was unmeasured.
METHODS:
We proposed a simulation-based, prior knowledge-guided approach to present users with an
estimated true effect with observational data when unmeasured confounding is an issue. Using
data from Kaiser Permanente Southern California, we compared the effectiveness of CABG vs.
PCI on various endpoints at 1, 3, 5, and 10 years for patients with stable ischemic heart disease.
We used propensity score-based matching weights to balance the treatment groups. We present a
sensitivity analysis method with parameters extracted from internal and external sources to
achieve an omitted confounder-adjusted estimated true effect.
36
RESULTS:
After adjustment using our proposed method, CABG patients are 57.6% less likely to encounter
MACCE at 1 year (OR[CI%]: 0.424 [0.396,0.517]), 56.4% less likely at 3 year (OR[CI%]: 0.436
[0.369,0.527]), 48.9% less likely at 5 year (OR[CI%]: 0.511 [0.451,0.538]). CABG patients are
also 49.5% less likely to die by the end of 10 years than PCI patients (OR[CI%]: 0.505
[0.446,0.582]). Compared to PCI patients, CABG patients were 54.3% less likely to die, 30.6%
less likely to be hospitalized due to stroke, 64.2% less likely to be hospitalized due to MI, and
69.6% less likely to require repeated revascularization by the end of 5 years.
CONCLUSIONS:
This method can be used for sensitivity analysis when unmeasured confounding is a concern.
After adjustments using our proposed method, we have greater confidence that if unmeasured
coronary disease complexity was included, CABG is associated with better clinical outcomes
than PCI when treating patients with ischemic heart disease.
Key Words
Unmeasured confounding, bias, simulation-based, prior knowledge guided approach,
Percutaneous Coronary Intervention (PCI), Coronary Artery Bypass Grafting (CABG)
2.2 Introduction
It remains a matter of controversy of whether percutaneous coronary intervention (PCI) or
coronary artery bypass surgery (CABG) is the optimal coronary revascularization strategy. Only
a few randomized trials have compared PCI with CABG in patients with coronary artery disease.
37
The SYNTAX and SYNTAX Extended Survival (SYNTAXES) study investigated patients’
health outcomes at 1, 3, 5 and 10 years
46-49
to compare the treatments (PCI vs CABG) received
by patients with left main coronary disease (LM) or multi-vessel disease (MVD). These trials
suggest that revascularization with CABG is beneficial in those patients with complex coronary
anatomy (i.e., a high SYNTAX score). For patients with relatively simple coronary anatomy,
CABG and PCI lead to similar results.
The SYNTAX score is a scoring system calculated on the basis of elements indicating
disease stages, such as coronary lesion location, complexity and characteristics. It is a useful
surrogate indicator for disease severity. However, in clinical practice, the complexity of coronary
lesions and the severity of coronary artery disease are often evaluated by physicians’ visual
inspection, and the SYNTAX score is rarely calculated. In most electronic health records,
SYNTAX score is not an available data element. When using large healthcare datasets to study
stable ischemic heart disease, coronary artery severity (as estimated by the SYNTAX score)
remains an important unmeasured confounder.
Unmeasured confounding may undermine the validity of observational studies
50-56
. With
observational data, unmeasured confounding bias is a central limitation -- uncontrolled
covariates may confound the relationship between exposure and outcome, and thus bias the
estimated treatment effect
57, 58
.
A common problem across healthcare studies that use observational data is how to assess
the effect(s) of a healthcare intervention (or exposure) on a healthcare outcome when the
intervention itself is not randomized. Examples of healthcare interventions include screening
tests, medical procedures, medication, health-related promotions, policies, etc. While every
attempt is made to collect all the available data, when it comes to observational data, it is
38
nevertheless often the case that one or more important covariates will remain unmeasured. These
uncontrolled confounding variables distort the effect estimates of any such intervention, and
make it difficult for observational studies to draw meaningful conclusions.
Healthcare utilization databases represent a major data source for studies seeking to infer
causal relationships between treatments and outcomes. The principal challenge for such studies
is presented by unmeasured confounders (Figure 2-1); this is because these databases, both in
their design and in how data is input, operate in a more general setting, and have not been
designed with specific research aims in mind. Further, because of the complexity and inadequate
understanding of the causal nexus between intervention and outcome, it often remains uncertain
whether the statistical model employed in such studies has been correctly specified with all of
the necessary confounders
59
. It is especially important that observational studies intending to
produce evidence of causality address the issue of unmeasured confounding: simply put, causal
relationships cannot be established with confidence when alternative explanations for the results
are probable.
A comprehensive array of sensitivity analysis techniques has been designed for the
purposes of evaluating evidence of causation in the presence of unmeasured confounding
50, 52, 56,
60-66
. Sensitivity analysis evaluates how strong the unmeasured confounding would have to be in
order to explain away the estimated association between intervention and outcome
56
. The
magnitude of strength can guide decisions about whether the association can be interpreted as a
potential causal relationship. In other words, sensitivity analysis typically assesses the scale of
possible biases, and reports the level of confidence for the study results. Yet, one criticism of
these techniques is that they are ‘subjective’, which means that sensitivity parameters can be
manipulated to make the results appear anywhere from robust to confounding.
39
The E-value was recently introduced as a supplement to existing sensitive analysis to
address unmeasured confounding concerns
56
. The authors defined E-value as “the minimum
strength of association, on the risk ratio scale, that an unmeasured confounder would need to
have with both the treatment and the outcome to fully explain away a specific treatment–
outcome association, conditional on the measured covariates”
56
. In general, a small magnitude of
E-value should raise caution for unmeasured confounding bias. Since its introduction,
researchers have applied E-values to a wide range of applications in observational studies
evaluating causality. E-values have been used to alleviate the over-reliance on the p-value, and
offset inadequate assessment of robustness to bias. Nevertheless, there are limitations in the
implementations, and the interpretations are prone to misconstruction. For example, no general
rules exist to apply a reasonable qualifier (low, moderate, or high) to categorize E-values
67
.
Furthermore, concerns have arisen that some investigators may be biased and use the E-value to
conclude that confounding caused the effect
68
. The concerns over E-values drove us to contribute
to this discussion, and introduce a new tool that can supplement some previously uncovered
areas in sensitivity analysis of unmeasured confounders.
The main purposes of our study are to compare the safety and effectiveness of CABG vs.
PCI for patients with stable ischemic heart disease using observational data, and to introduce the
L-table, a simulation-based, prior knowledge guided approach, which enables investigators to
use external information about the unmeasured confounders (e.g., SYNTAX score) to identify a
plausible range of estimated true effects in observational studies. We call it the L-table, named
after uncovering the ‘Location’ of the graphical frame in the numeric table that contains the
estimated true effect corresponding to assumed correlations. Our framework is flexible enough to
40
adapt to different types of models. The statistical code for L-table is open source, easy to use,
with customizable parameters for the users’ observational data and external information.
Note that the approach we propose uses well known relationships from the literature. We
hope to provide guidance and tools to help researchers understand and implement these ideas.
We do have a somewhat different focus from the bulk of the literature. Much of the literature is
focused on testing and statistical significance. We are more focused on understanding plausible
effect sizes. This is an often underappreciated element in applying confounder analysis to
practical decision-making. Some consideration of the practical significance of the results must
follow statistical significance. This is especially true in comparative effectiveness problems
where an incumbent treatment already exists.
2.3 Methods
2.3.1 Model Framework
The method proposed here aims to provide clinicians and investigators with an estimated true
effect when using observational data where uncontrolled confounding is an issue. We assume
that the effect of a treatment on the outcomes can be estimated by a generalized linear model. In
this study, we use a logistic model to illustrate how we have used the proposed method to
evaluate the effectiveness of cardiovascular procedures. The dependent variable is a binary
outcome of interest (i.e., 5-years survival); the independent variables include treatment, and
observed and unobserved confounders.
It is important to be precise in the way we talk about correlations when applying this
method. The main issue is the difference between an estimator and an estimand. An estimator
41
uses a computation on observed data to calculate an estimate. Here we will use estimand for
parameters, typically unknown, that we wish to estimate.
An easy source of confusion in our setting is that an estimator may be estimating
different things depending on the type of data it is applied to. For example, the Pearson
correlation (estimator) estimates the product-moment correlation (estimand) when applied to
bivariate normal data. The same calculation calculates the Spearman correlation when applied to
binary data.
Our simulation framing uses a trivariate normal distribution to generate the data that we
summarize to estimate the bias from unmeasured confounders. We will refer to these correlations
as “product-moment” correlations. This is similar to the way many econometrics books motivate
limited dependent variables
69
. The challenge is that correlations available in the literature or that
inform clinical intuition may not be straight-forward estimates of the product-moment
correlations. It may be necessary to transform the available correlations into these product-
moment correlations to run the simulations. After the simulations have been run it may be
necessary to transform the results back to the original scale for reporting in the original context.
1) The tetrachoric correlation case
Tetrachoric correlation characterizes the data in a 2 by 2 table by hypothesizing a
latent bivariate normal variable that is dichotomized in both dimensions to generate the 2
by 2 table. The simple Spearman correlation calculated from the 2 by 2 table is not an
estimate of the latent correlation. What is needed is a tetrachoric correlation estimate.
This will be the common case for the relationship between treatment and
outcome. Fortunately, this is data we will have in observational studies. Getting the
needed latent correlation estimate is simply a matter of calculating the tetrachoric
42
correlation with appropriate software. If the original Spearman metric is desired for
reporting results the dichotomization that defines the tetrachoric correlation can be
mimicked in the simulation.
2) The point-biserial case
Point-biserial correlation is the application of the Pearson correlation calculation
to a binary and a continuous variable. Conceptually, our framework is similar to the
tetrachoric correlation situation, but in this case only one of the two latent variables has
been dichotomized. We still want to use the latent correlation in our simulation.
Unfortunately, the point-biserial correlation does not estimate the latent correlation.
There is another estimator, the biserial correlation
70
, that is an estimate of the latent
correlation we need.
This is the case we commonly need to address when we estimate the correlation
between the treatment and a continuous unmeasured confounder. This estimate usually
comes from the literature where we are unlikely to have access to the subject level data.
Fortunately, there is a formula to transform the point-biserial correlation into a biserial
correlation using only summary quantities that are often available in the literature. As in
the tetrachoric correlation case, dichotomization in the simulation can transform the
results to the original point-biserial scale if needed as an aid to interpretation.
Our method simulates from these underlying correlations among the drivers in the causal
relationship to assess the estimated true effect when an important confounder is omitted. The
model framework can be illustrated in a pipeline composed of four major steps (Figure 2-2). We
will describe these procedures and elucidate more details in the following sections.
43
Step 1. Model based on empirical data.
1. (Optional but Optimal) Balance your empirical data, i.e., using
matched propensity score weighting (other propensity score methods can be
substituted)
2. Estimate the effect based on balanced empirical data:
_
3. Calculate rate of treatment of interest and rate of outcome:
treatment (%) and outcome (%)
Step 2. Ascertain correlation coefficients.
1. Calculate correlation between outcome and treatment (
_
)
based on empirical data. In this study, both outcome and treatment are
dichotomized variables, so we prepared tetrachoric correlation (estimate of the
latent correlation, used in simulation).
2. Deduce correlations between unmeasured confounder with
outcome (
_
), and unmeasured confounder with treatment () based on external
information. In this study, the unmeasured confounder (SYNTAX score) reported
from the literature is an ordinal variable, and the outcome and treatment are binary
variables. We calculated polychoric correlations (estimate of the latent correlation,
used in simulation) and derived biserial correlations (used to match the coordinate
labels in L-table).
3. Make a sequence of adequate ranges for each latent correlation
coefficient estimation with small intervals. We created a sequence using +/-0.3
with 0.05 intervals.
Step 3. Simulation.
44
1. Construct a collection of correlation matrices (check for positive
definite) based on all combinations of the latent correlation coefficients estimations
(from Step 2-3).
2. Simulate multivariate normal distributed datasets based on the
collection of correlation matrices. Each dataset is generated based on one
correlation coefficient matrix.
a. Simulate 100 (an adjustable parameter) datasets according
to each correlation matrix. Each dataset contains 1000 (an adjustable
parameter) datapoints.
b. Dichotomize the outcomes and treatments based on the
proportions of treatment and outcome in empirical data (from Step 1-
3).
c. Reassess the correlation coefficients in simulated datasets
based on the recently dichotomized variables, and result in biserial
_
and biserial
_
(estimates of the latent correlations).
d. Perform suitable model for simulated datasets. We used
logistic regression in this study.
e. Output means or medians of your estimates (
_
) based
on each of the 100 datasets without including unmeasured cofounding,
and the estimated true estimates (
_
) when the unmeasured
confounder was included in the model (from Step 3-2d), along with the
corresponding estimates of the latent correlation in simulated data
(from Step 3-2c).
45
Step 4. Obtain estimated true effects
1. Select a small range around your empirical estimate
_
(from
Step 1-2) (We used ln
_
+/-0.1)
2. Subset the outputs (from Step 3-2e) based on the collections of
values around
_
(from Step 4 -1) that match
_
(from Step 3-2e). This
selected data includes
_
,
_
,
_
, and
_
.
3. Construct L-table
a. (optional) Output contour plot
b. Use the subgroup data (from Step 4-2) to tabulate
_
to form L-table, which employs
_
as column label and
_
as row label.
4. Assess the estimated true effects
Based on L-table (from step 4-3b), we can locate the estimated true effect
using
_
and
_
(from Step 2-2) to match the coordinates. Biserial correlations
are used in the output for this study.
Confidence intervals of the estimated true effect can be obtained by repeating step 4 with
corresponding parameters.
2.3.2 Simulation
In a simplified simulation setup, we first constructed a linear relationship between outcome and
treatment adjusted for an unmeasured confounder. The outcome, treatment and unmeasured
variables are generated based on a multivariate normal distribution with an assigned correlation
46
matrix (Figure 4). Each correlation matrix is composed of different combinations of correlation
coefficients, with values falling within an adequate range of each correlation coefficient derived
from either empirical data or an external source. Within this range, possible values of the
correlation coefficient are specified with a gradient level of a selected incrementation. To
establish reasonable computation time, we performed local simulations. Because the correlation
will shift after addressing the proportion of outcome and treatment as the empirical data, the user
needs to assign an arbitrary but adequate range that encompasses the correlation coefficients.
The simulations only made use of a correlation matrix that satisfied positive definiteness.
By working with the correlation matrix, we are implicitly using standardized regression
coefficients. We did not include observed confounding variables in the simulation model because
we assume that the composite effect from the observed confounders will be addressed in study
design (i.e., RCT) or in modified empirical data, such as a propensity score (PS)-weighted
adjustment. We also assume that the unmeasured confounder used in the simulation represents
the composite effect from all uncontrolled confounders.
Once the initial dataset was simulated, we dichotomized outcome and treatment variables
based on their corresponding proportions as found in the empirical data. We made the cut points
adjustable parameters which define the proportion of the outcome event and treatment. We then
reassessed the correlation coefficients of outcome and unmeasured variable, and correlation
coefficients of treatment and unmeasured variable. These updated correlation coefficients will be
used in the coordinates of the contour plot or as column and row names for the L-table. We
simulated each dataset with 1000 observations based on a corresponding correlation matrix. We
made 100 iterations of this data generation procedure. We also generated the mean and the
median of the estimated effects and estimated true effects based on models from these datasets
47
that are generated with the same set of correlation matrix. When outliers are suspected, we would
report the median values. Note that in principle we could define the relationships in terms of
point-biserial correlations or in some cases Spearman correlations. We have stayed with the
multivariate normal approach when generating the data, working on the assumption that it will
be more generally useful for other models (e.g. time-to-event).
2.3.3 Data Source
Our analysis is based on a retrospective cohort study using electronic health record
(EHR) data from the Kaiser Permanente Southern California (KPSC) Health System. KPSC
provides care to an ethnically and socio-economically diverse population that is broadly
representative of the racial-ethnic groups of Southern California
16
. The study protocol was
approved by the KPSC Institutional Review Board (IRB). A waiver of informed consent was
obtained due to the observational nature of the study.
Using procedure codes (Table 2-1), we searched EHR data for adult patients (age ≥ 18
years) whose records indicate that they underwent a revascularization procedure (CABG or PCI)
between January 1
st
, 2006 and March 1
st
, 2015. We chose March 1
st
, 2015 as the end of the
inclusion period to mitigate any impacts on hospitalization, mortality, and the means to facilitate
revascularization procedures caused by the Covid-19 pandemic and to ensure this cohort has at
least 5 years follow up time until March 2020, when healthcare utilization in California was
affected by the state’s stay-at-home order. The index date was defined as that of the first
procedure the patient received at KPSC (either CABG or PCI) during the cohort inclusion
period. We excluded patients who were not KPSC members or who did not hold continuous one-
year membership prior to the index date (allowing for a 45-day gap). Patients who underwent
revascularization procedures prior to the index date were also excluded. To identify patients with
48
ischemic heart disease, two conditions were used: first, a principal diagnosis of Coronary Artery
disease or Angina; and second, no prior history of acute myocardial infarction. To be included in
this cohort, both conditions had to be met at least twice in outpatient visits or at least once in
inpatient admissions within a year prior to the index date (Figure 2-3).
Using the primary cohort, we created sub-cohorts for the 1-year, 3-year, 5-year, and 10-
year endpoints to study the treatment effects in the short- and long-term. To ensure sufficient
follow-up time for each patient, patients were censored if their KPSC membership lapsed prior to
the end of each study period. We moved forward the inclusion date for the 10-year cohort to
March 1
st
, 2010 to allow sufficient follow-up time. Patients’ disease history and outcome
endpoints were identified using the International Classification of Diseases (ICD) 9/10 codes
(Table 2-1).
We identified covariates in the following categories: baseline demographics, medical
comorbidities, cardiac risk factors, and the use of cardiac medication. Medical comorbidities and
cardiac risk factors were collected using ICD codes for one year prior to the index date. Baseline
concomitant medications were identified using outpatient pharmacy records. The study endpoints
include: all-cause mortality, hospitalization from either myocardial infarction or stroke, repeat
revascularization, and a composite major cardiovascular events (MACCE)
48
, defined as one of
above endpoints by the end of 1 year, 3 years, 5 years, or 10 years from the index date.
Mortality data was pulled from a mortality data mart with integrated mortality
information derived from multiple sources, including the state of California’s death master files,
Social Security Administrative death master files, hospital death records, and insurance
enrollment records. The endpoints for myocardial infarction and stroke were identified with a
49
principal diagnosis at inpatient settings. The repeated revascularization procedures were
identified as the event after discharge following the first procedure.
2.3.4 External Information
To locate the estimated true effects using the L-table, we need to identify potential unmeasured
confounders and to extract external information from available sources. The essential
information from outside our data concerns the correlation between the unmeasured confounding
and outcomes, and the correlation between unmeasured confounding and the treatment. These
correlation coefficients can be obtained from a randomly selected sub-cohort, a pilot study, or
other relevant publications either by using simulations or closed form formulas when available.
When the identified unmeasured confounder is attainable in a sub-cohort of study data, one can
calculate the correlations and locate the estimated true effect from the L-table.
We derived the correlation between SYNTAX scores and outcomes from published
results of the SYNTAX trials
46-49
. We chose to gather this information from RCTs because most
sources of measured and unmeasured confounding are mitigated by the study design. We derived
the correlation between the SYNTAX scores and treatment from a high-quality observational
study
71
which accounts for the behavior effects of physicians’ and patients’ treatment choices.
Analysis on the main cohort and subgroup analysis were both performed and adjusted taking into
consideration both correlations, using externally available information. In applications where a
high quality RCT is not available, a plausible range of correlations may be estimated using
clinical judgement.
50
2.3.5 Statistical Analysis
Descriptive statistics on patients’ characteristics, including demographics, comorbidity, and
medication history were reported by treatment groups using frequencies and percentages. P
values from Chi-squared tests and t-tests were reported to assess the differences of the traits
between treatment groups for categorical variables and continuous variables respectively. We
calculated standardized mean difference (SMD) before and after applying matching weights to
our data. A difference of 0.10 or less was considered an adequate balance between the two
groups.
To mimic the enrollment patterns in RCTs and adjust for selection bias in our data, we
prepared data with matching weight
72
, a propensity score weighting method. Matching weight is
a variant of the inverse probability weight, but the matching weight estimator assigns greater
weight to individuals whose PS is close to 0.5 (i.e., the circumstance generated in a two-arm
RCT with clinical equipoise). The underlying propensity score model included 57 baseline
covariates including age, sex, race, comorbidities and baseline medication use. The PS was used
to create matching weights which were applied to balance treatment groups in our study data.
We ran logistic regression to assess the treatment effects on outcomes at 1, 3, 5, and 10
years. We reported crude (unadjusted), and matching weighted Odds Ratios, along with their
95% confidence interval, p values, and E values. A small E value (close to 1) suggested that
unmeasured confounding should be a concern. A p value of <0.05 was the nominal level of
significance.
We then simulated correlation matrices to obtain estimated effects within a small range of
the estimated effects derived from our observational data. In so doing, we also obtain the
estimated true effects as generated with the same simulated correlation matrix, where the
51
unmeasured confounder is taken into account. Using a method of bivariate interpolation for
irregularly distributed data points
73, 74
, we are able to plot the estimated true effects against
biserial correlation of outcome and unmeasured confounding, and of treatment and unmeasured
confounding. The pattern of the estimated true effect in relation to these correlations can be
visualized in a contour plot. We tabulated these estimated true effects into the L-table, similar to
the construction of a Chi-squared table, where the column was marked with the correlation
coefficients between treatment and unmeasured confounders, and the row was marked with the
correlation coefficients between outcome and unmeasured confounders. The user can adopt the
L-table as a reference to locate a plausible range of estimated true effects when prior information
about correlations is incorporated.
The values bounded by the perimeter within the L-table represent the estimated true
values of the causal effect on the scale of the selected model. This causal effect is aligned with
the various correlations between unmeasured confounding and outcome on the one hand, and
between unmeasured confounding and intervention—conditional on the measured covariates—
on the other.
Sometimes published studies reported mixed findings. The polychoric correlation
between treatment selection and the syntax score derived from external publications was 0.625.
To explore how susceptible L-Table is at responding to a variety of inputs, we performed a
sensitive analysis by assigning two different correlation coefficients (0.5 and 0.7) for
relationships between the treatment and the unmeasured confounders in simulation, and ran the
models with the same procedures. All statistical analyses were conducted in SAS 9.4 (SAS
Institute Inc., Cary, NC) and R version 4.1.0
75
.
52
2.4 Results
During the study period, 12,216 adult (age≥18) KPSC members were identified with stable
ischemic heart disease who underwent PCI or CABG between January 1, 2006 and March 1,
2015. Among these patients, 5,513 received CABG, and 6,703 received PCI.
In this group, 11,298 patients maintained membership at the end of 5 years: 5,158
received CABG and 6,140 received PCI (table 2-2). The average age of CABG patients was 2.5
years older than their PCI counterparts (p<0.001). More men than women underwent CABG
(77.5% vs. 73.5% p<0.001). Asians were more likely to undergo CABG (11.6%) compared to
PCI (9.0%); while Black patients had PCI with greater frequency than CABG (8.0% and 5.9%
respectively). In general, CABG patients had more comorbidities and more frequently used
cardiac medications. The baseline characteristics of the study cohort between CABG and PCI
were balanced (SMD < 0.1) after weight adjustment with matching weights, and we used the
weighted cohorts for our analysis.
Table 2-3 presents the crude and adjusted treatment effects on outcomes at the end of 1,
3, 5, and 10 years, comparing CABG versus PCI. Patients treated with CABG were less likely to
experience MACCE, mortality, hospitalization for MI, or repeat revascularization compared to
PCI patients (Table 2-3). CABG patients were also 29% less likely to encounter MACCE at 1
year (OR[CI%]: 0.713 [0.643,0.789]), 30% less likely at 3 years (OR[CI%]: 0.701
[0.646,0.761]), 24% less likely at 5 year (OR[CI%]: 0.759 [0.707,0.814]). Moreover, CABG
patients were 8.3% less likely to die by the end of 10 years (OR[CI%]: 0.917 [0.86,0.977]).
Patients treated with CABG were, however, more likely to be hospitalized due to stroke after the
revascularization procedure in comparison to patients treated with PCI, but these effects were not
53
statistically significant. All E values are small (close to 1), which indicates that unmeasured
confounding is an issue.
We will illustrate the application of L-table using 5-year MACCE as an example. We
assessed the proportion of outcome and the proportion of treatment from adjusted empirical data
(Table 4). Correlation coefficients between each of the end points and SYNTAX score (low,
intermediate, high) based on the published results from SYNTAX trials
46-49
are presented in
Table 2-4. The correlation coefficients between MACCE and SYNTAX score is 0.042 at 5 years.
We derived correlation coefficients between revascularization type and SYNTAX score
( ρ
=0.449) from a published population-based study
71
where patients’ and physicians’ behavior
on treatment decisions were well represented. We assigned these parameters to generate
simulated data.
The estimated Odds Ratio is 0.759 for MACCE at 5 years, and the parameter in the
logistic model is ln(0.759) = -0.276. The contour plot (Figure 2-5) shows the estimated true
effects based on a proposed method where the estimated effects were within a close range of -
0.276. The x-axis represents the correlation between treatment and the unmeasured confounder,
and the y-axis represents the correlation between outcome and the unmeasured confounder. The
lines in the contour plot represent the values of the estimated true effects, which can be identified
based on the correlation coefficients on the horizontal and vertical labels. The set of the
estimated true effects were tabulated to form an L-table (Table 2-5). We presented an example of
a partial L-table (Table 2-6) to illustrate how we identify the estimated true effects based on the
information described above: ρ
=0.586 and ρ
=0.055 for MACCE at 5 years. We took the
mean of the values bordered by the frame and reached the Odds Ratio we are looking for: 0.511.
54
After the adjustments made with the L-table, CABG patients are 57.6% less likely to
encounter MACCE at 1 year (OR[CI%]: 0.424 [0.396,0.517]), 56.4% less likely at 3 years
(OR[CI%]: 0.436 [0.369,0.527]), and 48.9% less likely at 5 years (OR[CI%]: 0.511
[0.451,0.538]). CABG patients are also 49.5% less likely to die by the end of 10 years in
comparison to PCI patients (OR[CI%]: 0.505 [0.446,0.582]). Compared to PCI patients, CABG
patients were 54.3% less likely to die, 30.6% less likely to be hospitalized due to stroke, 64.2%
less likely to be hospitalized due to MI, and 69.6% less likely to require repeated
revascularization by the end of 5 years (Table 2-3).
Different correlation inputs resulted in different L-table adjusted true effects. When we
assign
= 0.5 in simulation, the L-table adjusted true odds ratio was 0.45 (CI%:[0.397,0.516])
for 5-year MACCE, and when we assign
= 0.7 in simulation, the L-table adjusted true odds
ratio was 0.517 (CI%:[0.459, 0.607]) for 5-year MACCE. Overall, the change of scale was small,
and the direction of the adjusted effect was unvaried. However, we found that hospitalization due
to stroke at 5 years was no longer statistically significant (OR[CI%]: 0.824 [0.649,1.035]) when
we assign
= 0.7 in simulation (Table 2-7).
2.5 Discussion
We aim to introduce the L-table and illustrate how it can be used to make potential causal
inferences by using observational data. We have illustrated the application of the L-table in a
real-world example, studying the safety and effectiveness of CABG vs. PCI for patients with
ischemic heart disease at 1, 3, 5 and 10 years. We incorporated external information into the
simulation with customizable parameters in order to locate the plausible range of estimated true
effects. The foundation of the L-table is established on the matrix relation for an ordinary linear
55
square (OLS) estimate in simple or multivariate linear regression; nevertheless, this model
framework can be expanded to the application of all sorts of generalized linear models.
Existing frameworks of sensitivity analysis formulate useful rules of thumb for assessing
the robustness of the estimated effects in light of unmeasured confounding issues. In some cases,
we can obtain the traits of the major confounders that have been omitted at the data collection
stage. This might involve the use of theory, pilot studies, external data analysis, or some
informed speculation. Nevertheless, neither sensitivity analysis nor instrumental variable
methods utilize the above-mentioned techniques to obtain existing information specifically about
the unmeasured confounders. Our tool thus allows researchers to use an L-table report to identify
a credible estimated true effect under various scenarios, thereby addressing the risks to the
reliability of research findings posed by unmeasured confounding.
After adjusting for observed confounders, we found CABG to be associated with better
safety and effectiveness than PCI for patients with stable ischemic heart disease. This finding
applied to MACCE, mortality, hospitalization for MI, and repeated revascularization at 1,3,5,
and 10 years. Interestingly, we found that patients treated with CABG had a higher rate of
hospitalization for stroke, although this difference was not statistically significant. These results
were generated using a model that did not include an important unmeasured confounder, in this
case, the SYNTAX score (a measure of coronary disease severity). The L-table allows us to
apply the attributes of a SYNTAX score derived from external studies to our model to address
unmeasured confounding. We found that the scale of the Odds Ratio shifted to smaller values,
suggesting greater protective effects from CABG when unmeasured confounding is considered
than had been previously estimated.
56
We recommend performing a propensity score adjustment on the user’s empirical data to
balance the treatment and control group and properly address all potential measured
confounders, to allow the simulation to run without observed confounders. In this study, we used
MW-adjusted real-world data to emphasize the causal effects inference on patients whose
characteristics are such that each treatment has an equivalent chance to be chosen (i.e., we have
equipoise).
Because of the nature of random assignment in RCT design, we cannot derive the
correlation between treatment and unmeasured confounder from the same RCT study. We found
that the correlation between treatment selection and SYNTAX score is similar in a US-based
study
71
and in a South Korean based study
76
, suggesting that clinicians from different countries
may share common treatment preferences. It is worth noting that the L-table adjusted true effect
could vary depending on the values of inputs in simulation. Therefore, users are advised to
discuss the source of the external studies they refer to, and to address the generalizability of the
external data, as well as the study limitations, if any.
Although the SYNTAX trials suggest that CABG and PCI are generally similar for
patients with relatively simple coronary anatomy, and that CABG is beneficial for patients with
complex coronary anatomy, a decision between CABG and PCI is more complex in real life.
What is best based on coronary anatomy is only one of many considerations. Our results
confirmed the findings from the RCTs, and validated the generalizability of the inference from
the SYNTAX trials to our study. RCTs have great internal validity for the population they study,
but patients enrolled in RCTs are generally treated at academic medical centers and receive care
that differs from real-world settings. The strategy we employed -- extracting information (i.e.,
correlations) from external sources, and integrating the information into our data, enables us to
57
ask the following question: do we see the same results in our system and population (in the real
world) as found in RCTs?
Our proposed methods are based on several assumptions. First, we define unmeasured
confounding in our model as a composite measurement of omitted confounders. In an ideal
situation, the user is aware of what confounder variables remain unmeasured, hence she can
devote her recruiting efforts towards targeting the traits of these unmeasured variables or their
proxy in the study design. However, our proposed method is not suitable for scenarios where
researchers remain in the dark about the traits of the unmeasured confounders. Second, we
assumed the correlation coefficients retrieved from a different population were commensurate in
our population. To imitate the features in the data from RCTs from which we derived the
correlation, and to mitigate sources of unmeasured confounders in the real-world, we applied
propensity sore analysis with matching weights adjustment on our empirical data. Third, we used
multivariate normal distribution to the simulated dataset to approximate the set of correlated
random variables, each of which clustered around a mean value. We applied all available
external and internal information in the simulation parameters to reproduce a dataset with
comparable properties as the balanced (PS-adjusted) empirical data, but with the unmeasured
confounder.
A few limitations should be mentioned. First, the proposed method assumes that the
effect can be estimated with a generalized linear model (GLM). The R code we provide with our
proposed method is written for a logistic regression model with a binary outcome. In principle,
based on the pipeline of our model structure, a savvy statistical program user should be able to
switch the model with other GLM models or with multiple types of outcomes. Our future work
includes expanding the application to other scales of outcomes and to increasingly complex
58
models. Second, in the analysis of 3-, 5- and 10-year effects, we did not censor mortality from
previous study periods. This may create problems, especially when it comes to capturing the
timely assessment of other endpoints; accordingly, a time-to-event analysis may be the optimal
statistical model. Third, our method is simulation-based, so it comes without a closed-form
solution. Future work is warranted to develop a mathematically supported theory. Fourth, the
scales of correlation between SYNTAX scores and outcomes based on the SYNTAX trials are
small. Although many studies have supported the utility of using SYNTAX scores in the
selection of revascularization strategies, contradictory findings have undermined the validity of
the application of such scores
77
. Finally, the estimated true effects adjusted by the L-table
method is sensitive to the inputted parameters. We advise that users carefully consider the
assumptions and limitations when they select a proxy for unmeasured confounders.
To summarize, we recommend using L-table as a supplement to established tools of
sensitivity analysis. L-table, built with customized parameters, adaptable models, and modifiable
precision, provides investigators with a more plausible value of estimated true effects based on
the influence of unmeasured confounders than is currently possible via other sensitivity
techniques. The L-table enables greater confidence on the part of researchers when explaining
their study results based on observational data, and it can guide clinicians and patients in making
better informed treatment decisions.
59
2.6 Tables and Figures
2.6.1 Tables
Table 2- 1. ICD9, ICD10, and procedure code.
60
Table 2- 2. Characteristics of Study Cohort included at 5 years endpoints
Overall
(N=11298)
CABG
(N=5158)
PCI (N=6140) p value
SMD SMD
Unadjuste
d
MV-
weighted
Demographics
Age (mean (SD))
67.48
(10.70) 68.84 (9.76) 66.33 (11.31) <0.001 0.238 0.002
GENDER: Male (%) 8513 (75.3) 4000 (77.5) 4513 (73.5) <0.001 0.094 0.003
Race/Ethnicity (%) <0.001 0.118 0.006
Asian 1151 (10.2) 596 (11.6) 555 ( 9.0)
Black 795 ( 7.0) 305 ( 5.9) 490 ( 8.0)
Hispanic 2346 (20.8) 1064 (20.6) 1282 (20.9)
Other 351 ( 3.1) 141 ( 2.7) 210 ( 3.4)
White 6655 (58.9) 3052 (59.2) 3603 (58.7)
Education: More than 75%
High School Graduate (%) 8527 (75.5) 3976 (77.1) 4551 (74.1) <0.001 0.069 0.001
Income: Median household
more than $45,000 (%) 8301 (73.5) 3848 (74.6) 4453 (72.5) 0.013 0.047 0.006
Insurance Type: Medicare or
Medicaid (%) 3457 (30.6) 1724 (33.4) 1733 (28.2) <0.001 0.113 0.004
BMI (mean (SD)) 29.03 (5.91) 28.76 (5.66) 29.25 (6.11) <0.001 0.083 0.003
Comorbidity
Pulmonary Embolism (%) 114 ( 1.0) 55 ( 1.1) 59 ( 1.0) 0.643 0.011 <0.001
Deep Venous Thrombosis
(%) 292 ( 2.6) 196 ( 3.8) 96 ( 1.6) <0.001 0.139 0.015
Present Of Prosthetic Heart
Valve (%) 364 ( 3.2) 292 ( 5.7) 72 ( 1.2) <0.001 0.249 0.01
Myocardial Infarction (%) 3812 (33.7) 1281 (24.8) 2531 (41.2) <0.001 0.354 0.002
Heart Failure (%) 3048 (27.0) 1692 (32.8) 1356 (22.1) <0.001 0.242 0.006
Atrial Fibrillation (%) 2438 (21.6) 1630 (31.6) 808 (13.2) <0.001 0.454 0.002
Heart Valve Disease (%) 2743 (24.3) 1738 (33.7) 1005 (16.4) <0.001 0.408 0.005
Congenital Heart Disease
(%) 293 ( 2.6) 197 ( 3.8) 96 ( 1.6) <0.001 0.14 0.002
Disorders Of Lipoid
Metabolism (%) 9932 (87.9) 4629 (89.7) 5303 (86.4) <0.001 0.104 <0.001
Atherosclerosis (%) 3788 (33.5) 2064 (40.0) 1724 (28.1) <0.001 0.254 0.003
Intracerebral Hemorrhage
(%) 70 ( 0.6) 38 ( 0.7) 32 ( 0.5) 0.182 0.027 0.004
Gastrointestinal Bleeding
(%) 541 ( 4.8) 263 ( 5.1) 278 ( 4.5) 0.17 0.027 0.004
Other Bleeding (%) 1360 (12.0) 681 (13.2) 679 (11.1) 0.001 0.066 0.003
Stroke Or Cardiovascular
Accident (%) 2524 (22.3) 1397 (27.1) 1127 (18.4) <0.001 0.209 0.007
High Blood Pressure (%) 9855 (87.2) 4630 (89.8) 5225 (85.1) <0.001 0.141 0.007
Diabetes Mellitus (%) 4986 (44.1) 2498 (48.4) 2488 (40.5) <0.001 0.16 <0.001
Arrhythmias (%) 4689 (41.5) 2501 (48.5) 2188 (35.6) <0.001 0.263 0.003
Chronic Kidney Disease (%) 3452 (30.6) 1830 (35.5) 1622 (26.4) <0.001 0.197 0.002
Dementia (%) 49 ( 0.4) 12 ( 0.2) 37 ( 0.6) 0.005 0.057 0.002
Acute Pulmonary Heart
Disease (%) 683 ( 6.0) 467 ( 9.1) 216 ( 3.5) <0.001 0.23 0.009
61
Chronic Pulmonary Heart
Disease (%) 3027 (26.8) 1531 (29.7) 1496 (24.4) <0.001 0.12 0.003
Rheumatic Disease (%) 287 ( 2.5) 128 ( 2.5) 159 ( 2.6) 0.762 0.007 0.001
Peptic Ulcer Disease (%) 196 ( 1.7) 95 ( 1.8) 101 ( 1.6) 0.468 0.015 0.002
Hemiplegia (%) 103 ( 0.9) 46 ( 0.9) 57 ( 0.9) 0.917 0.004 0.001
Neurologic Disorder (%) 601 ( 5.3) 301 ( 5.8) 300 ( 4.9) 0.028 0.042 <0.001
Hypothyroidism (%) 1439 (12.7) 681 (13.2) 758 (12.3) 0.182 0.026 0.002
Maliganant Cancer (%) 851 ( 7.5) 393 ( 7.6) 458 ( 7.5) 0.776 0.006 0.006
Liver Disease (%) 671 ( 5.9) 329 ( 6.4) 342 ( 5.6) 0.077 0.034 0.005
Human Immunodeficiency
Virus (Hiv) (%) 1263 (11.2) 1042 (20.2) 221 ( 3.6) <0.001 0.53 0.009
Obesity (%) 3309 (29.3) 1639 (31.8) 1670 (27.2) <0.001 0.101 0.003
Weight Loss (%) 577 ( 5.1) 300 ( 5.8) 277 ( 4.5) 0.002 0.059 0.006
Anemia (%) 436 ( 3.9) 226 ( 4.4) 210 ( 3.4) 0.01 0.05 0.001
Drug Abuse (%) 1296 (11.5) 543 (10.5) 753 (12.3) 0.004 0.055 0.006
Psychosis (%) 133 ( 1.2) 64 ( 1.2) 69 ( 1.1) 0.626 0.011 0.003
Depression (%) 1716 (15.2) 757 (14.7) 959 (15.6) 0.172 0.026 0.004
Medication History
Implantable Cardioverter
Defibrillator (%) 155 ( 1.4) 52 ( 1.0) 103 ( 1.7) 0.003 0.058 <0.001
Antiotensin Converting
Enzyme (Ace) Inhibitor And
Angiotensin-Receptor
Blockers (Arbs) (%) 7143 (63.2) 3456 (67.0) 3687 (60.0) <0.001 0.145 0.007
Aldosterone Antagonist (%) 273 ( 2.4) 136 ( 2.6) 137 ( 2.2) 0.181 0.026 0.002
Antiarrhythmic Drugs (%) 262 ( 2.3) 138 ( 2.7) 124 ( 2.0) 0.025 0.043 0.002
Anticoagulants (%) 874 ( 7.7) 457 ( 8.9) 417 ( 6.8) <0.001 0.077 0.005
Antiplatelet Drugs (%) 2162 (19.1) 676 (13.1) 1486 (24.2) <0.001 0.288 0.009
Beta_Blocker (%) 6983 (61.8) 3363 (65.2) 3620 (59.0) <0.001 0.129 0.001
Calcium Channel Blockers
(%) 3051 (27.0) 1530 (29.7) 1521 (24.8) <0.001 0.11 0.003
Colchicine (%) 306 ( 2.7) 138 ( 2.7) 168 ( 2.7) 0.889 0.004 0.005
Digoxin (%) 377 ( 3.3) 189 ( 3.7) 188 ( 3.1) 0.085 0.033 0.003
Diuretics (%) 1849 (16.4) 952 (18.5) 897 (14.6) <0.001 0.104 0.004
Hydralazine (%) 506 ( 4.5) 257 ( 5.0) 249 ( 4.1) 0.02 0.045 <0.001
Nitrate (%) 1658 (14.7) 771 (14.9) 887 (14.4) 0.469 0.014 0.001
Statins (%) 7868 (69.6) 3809 (73.8) 4059 (66.1) <0.001 0.169 0.004
*Myocardial Infarction reported included MI diagnosed for only once at outpatient setting during
baseline or non-primary diagnosis
62
63
64
65
Table 2- 3. Sample of Partial L-table where the true effect is identified
0.537 0.551 0.565 0.579 0.593 0.607 0.621 0.635
0.12 0.528 0.506 0.484 0.459 0.43 0.404 0.38 0.362
0.108 0.54 0.518 0.496 0.471 0.442 0.415 0.391 0.373
0.097 0.553 0.53 0.508 0.483 0.453 0.426 0.404 0.385
0.085 0.56 0.542 0.521 0.495 0.464 0.44 0.417 0.397
0.074 0.566 0.549 0.532 0.507 0.48 0.454 0.43 0.409
0.062 0.576 0.558 0.541 0.52 0.492 0.465 0.441 0.423
0.051 0.589 0.571 0.552 0.53 0.502 0.475 0.452 0.435
0.039 0.602 0.582 0.561 0.541 0.513 0.486 0.463 0.446
0.028 0.609 0.588 0.567 0.549 0.522 0.495 0.474 0.457
0.016 0.607 0.586 0.568 0.553 0.525 0.503 0.488 0.468
0.005 0.605 0.588 0.573 0.556 0.532 0.517 0.502 0.482
* MACCE at 5 year,
=0.759,
=0.586,
=0.055
66
Table 2- 4. Sensitivity Analysis for Variation of Correlation Inputs
Year Endpoint
L-table adjusted
(95% CI)
=0.625
=0.5
=0.7
1 years MACCE 0.424 (0.396, 0.517) 0.342 (0.333, 0.431) 0.456 (0.416, 0.592)
3 years MACCE 0.436 (0.369, 0.527) 0.373 (0.349, 0.458) 0.459 (0.433, 0.575)
5 years
MACCE 0.511 (0.451, 0.538) 0.45 (0.397, 0.516) 0.517 (0.459, 0.607)
Death 0.457 (0.376, 0.468) 0.379 (0.337, 0.416) 0.428 (0.397, 0.485)
Stroke 0.694 (0.551, 0.873) 0.538 (0.45, 0.702) 0.824 (0.649, 1.035)
MI 0.358 (0.264,0.368) 0.258 (0.241, 0.326) 0.411 (0.283, 0.436)
Repeat
revascularization 0.304 (0.235, 0.344) 0.272 (0.213, 0.318) 0.329 (0.256, 0.412)
10 years Death 0.505 (0.446, 0.582) 0.501 (0.421, 0.539) 0.632 (0.534, 0.652)
67
2.6.2 Figures
Figure 2- 1. Causal Relationship between Treatment, Outcome, and Confounders.
68
Figure 2- 2. Pipeline of Model Framework to generate L-table.
69
Figure 2- 3. Study Cohort Flowchart.
70
Figure 2- 4. Simulation Structure
A. Multivariate Normal Distribution
* Y is outcome, X is intervention, and U is unmeasured confounder, ∑ is the covariate matrix, which
shares the same structure as correlation matrix in multivariate normal setting.
B. Correlation Matrix Structure
* Y is outcome, X is intervention, and U is unmeasured confounder, ρ is the correlation coefficient
(product-moment correlation)
71
Figure 2- 5. Contour plot generated by L-table method illustrating estimated true effects corresponding to
specified correlations
72
Chapter 3. Prompts in the electronic health record to encourage
compliance with California State Opioid Law
3.1 Abstract
To prevent deaths caused by opioid overdose, the State of California passed into law AB2760 on
September 10, 2018. This law requires that prescribers educate their high-risk patients and
families about overdose prevention and about the use of naloxone when prescribing opioids.
Kaiser Permanente Southern California (KPSC) created prompts in the electronic health record
to encourage compliance with the law. We hypothesize that this initiative will increase naloxone
prescription rates and decrease the prescription of opioids. We tested this hypothesis using
Kaiser Permanente’s electronic health record data, and we found a reduction in opioid orders and
an increase in naloxone prescriptions after the implementation of the prompts. On a monthly
basis, opioid prescription rates decreased; the total Morphine Milligram Equivalents (MME) per
prescriber-month also decreased in the wake of the prompt. Finally, the pre-post analysis shows
an improvement in all safe opioid prescription measures.
Keywords:
Electronic Health Record, California State Opioid Law, Clinical Decision Support
3.2 Introduction
The US is in the midst of an opioid epidemic, exacerbated by the combination of illicit and
prescription opioid use
78
. Despite ongoing efforts to curb opioid-associated risks, rates of fatal
73
overdose continue to rise
79, 80
While illicit now overshadow licit deaths, most persons who are
addicted or dependent on opioids first were exposed to them through a prescription for pain
management
81
. Though opioid prescriptions are decreasing they are still far higher than they
were before the crisis began. Because of the escalating number of deaths, healthcare practitioners
and investigators have been searching for methods to curb the prescription rates of opioid
medications
82-85
.
The drug naloxone offers one potential method to curtail fatal overdose
86
. Naloxone is the
short-acting, life-saving opioid antagonist that is used to reverse respiratory suppression that
leads to fatal overdose
87
. The Centers for Disease Control and Prevention recommends
distributing naloxone to individuals deemed at a high risk of overdose
88
, thereby enabling them
to respond to and counteract overdose events – either on their own or with the aid of others that
that they witness. In fact, naloxone has regularly been distributed to people who inject drugs at
community-based organizations and through syringe-exchange programs for nearly two
decades
89
. However, several US-based studies have suggested that naloxone awareness remains
low among patients who are prescribed opioid. Expanding naloxone availability represents an
important strategy for overdose prevention.
California law AB2760
90
requires that when receiving prescription opioids, patients at
high-risk of overdose also be offered a prescription for naloxone, or another drug approved by
the United States Food and Drug Administration (FDA) suitable for the complete or partial
reversal of opioid depression. This law also requires that when prescribing opioids, prescribers
educate their high-risk patients and caregivers about overdose prevention and about the use of
naloxone.
74
Over the past two decades, prompts in the electronic health record have been used to
assist and improve healthcare providers’ workflow and practice
91, 92
. Physician leadership at
Kaiser Permanente Southern California have worked closely with the health information
technology staff to build prompts that help satisfy requirements of AB2760. Within the
electronic health record, a prompt appears when certain conditions are met. The prompt
identifies the specific risk and recommends appropriate further actions, letting prescribers adjust
their orders. The prescriber can choose to keep or remove the order or adjust the dose
accordingly. The prompt also reminds prescribers to order Naloxone, and the system documents
the Naloxone prescription. Finally, prescribers can choose either to accept or dismiss the
message. Sixteen categories of alerts were implemented to support AB2760-related clinical
decision-making for outpatient prescription.
The primary objective of our study is to evaluate whether AB2760 effectively achieves
its goals of increasing naloxone possession among opioids users with the assistance of the
prompts, and to ask whether the extra friction created by State policy reduces opioid prescribing
when this prompt is integrated into practitioners’ workflow. We examined whether this
intervention had a positive influence on prescribers’ general prescription behavior in terms of
safely ordering opioids and preventing the overlapping prescription of medications which
interact with opioids. We hypothesize that naloxone orders will increase after the implementation
of the AB2760-based prompts, and, conversely, that opioid prescription rates will decrease.
Specifically, we hypothesize that opioid orders will decrease in connection to visits by the types
of patients directly targeted by the prompts, such as those with an overdose history and
concomitant a benzodiazepine prescription. Furthermore, we hypothesize that clinicians’
75
changing prescription patterns will also influence their behavior when it comes to initial opioid
users and chronically high-dose users, and they will likewise exercise more caution at renewal.
3.3 Method
3.3.1 Setting
The Kaiser Permanente Southern California Healthcare (KPSC) system is a regional integrated
healthcare system consisting of 15 medical centers and 235 medical offices across 14 areas in the
Southern California region. At Kaiser Permanente Southern California, more than 7,600
physicians and 26,000 nurses provide health services for 4.6 million members, comprising an
ethnically and socioeconomically diverse population that is broadly representative of the
racial/ethnic groups of Southern California
16
. A customized electronic health record (EHR) has
been implemented at an institutional level at all clinics across KPSC-covered service areas, but
variables in staffing levels and work culture/environment in different areas may affect the EHR
workflow.
The prompt for outpatient opioid prescriptions is activated when one or more of the
following conditions is met: (1) the prescriber writes an opioid order that will result in the patient
receiving a total opioid Morphine Milligram Equivalents (MME) of 90 or greater per day; (2) the
patient has concurrent active prescriptions of an opioid and benzodiazepine; and (3) the patient
has had an overdose diagnosis within the past 2 years. Once the prompts are triggered, a safety
alert window pops up. The window explains the reasons that triggered the prompt (i.e., the
patient is at high risk for overdose) and lists an array of recommended actions (i.e., remove order
for opioid). The clinicians are given the choice between removing (default) or keeping the order
of opioid or other concomitant medication. The links for opioid dose calculator and opioid laws
76
and regulations are included to aid the decision. Finally, the clinician is advised to choose
between placing naloxone order and documenting patients declining naloxone. In the later
scenario, the clinician is advised to add visit diagnosis for the declination.
3.3.2 Study sample and data sources
We used retrospective data from the Kaiser Permanente Southern California Healthcare System,
and the study protocol was approved by the Kaiser Permanente Southern California Institutional
Review Board. We obtained a waiver of informed consent due to the observational nature of the
study.
Based on the National Drug Code for opioids issued by the Center for Disease Control
and Prevention
93
, we searched the electronic health records for Kaiser Permanente Southern
California members aged 18 years or older who were prescribed an opioid analgesic between
January 1
st
, 2018 and December 31
st
, 2019. The index date was defined as the date of active
opioid prescription, and the baseline was defined as one year prior to the prescription date. The
electronic prompt was designed to automatically exclude terminally ill patients, patients who
have been discharged to a hospice, and patients who had declined naloxone at baseline, and so
they are not included in this study. We have also excluded those patients suffering from
malignant cancer at baseline.
All eligible clinicians (non-resident) selected in this study must have a valid Federal Drug
Enforcement Agency number and must have placed outpatient opioid prescriptions for the
patients thus identified in 2018. Each clinician must also have been employed at Kaiser
Permanente Southern California continuously between January 1
st
, 2018 and December 31
st
,
77
2019; this is to ensure that each clinician enjoyed equal exposure to the prompts and could be
followed for the same amount of time.
3.3.3 Endpoints
The primary endpoints are the changes in overall levels of opioid prescription rates at clinician
level, as well as naloxone monthly prescription and possession rates, both before and after the
prompt was introduced at Kaiser Permanente Southern California to comply with the policy
guidelines as envisaged in AB2760. In this study, we define a prescription rate as the medication
orders per prescriber-month and per prescription-year in different model settings. We assessed
the immediate effect and sustainable effect of the prompts on clinicians’ prescription patterns
measured on a monthly basis. We also assessed the annual changes on average, comparing pre-
and post-intervention at clinicians’ level.
The secondary endpoints are as follows. The first is the impact on the prescribed dosage
levels: i.e., the monthly total Morphine Milligram Equivalents (MMEs) prescribed by each
clinician, and the monthly median MMEs prescribed per prescriber per patient encounter. The
second concerns the prompt-targeted objectives: opioid prescriptions for patients at risk of opioid
overdose, and concomitant prescription of benzodiazepines. The third endpoints center on the
unintended consequences: the prescription of either muscle relaxants or zolpidem concomitant
with all opioid prescriptions in the study cohort. Specifically, we classify a concomitant
prescription as follows: when an opioid and a benzodiazepine, muscle relaxant, or zolpidem
prescription was placed either 30 days before or after the order of the other medication.
Furthermore, we checked whether the prompt influenced opioid prescription rates for
three types of patients at different levels of risk of opioid abuse
94-98
. First, we looked at initial
orders for patients who were prescribed an opioid without having previously had such a
78
prescription within at least 90 days prior to the index date. Second, we analyzed renewal orders,
capturing those patients who received an opioid prescription within 90 days of the index date, but
had not been prescribed opioids more than 90 days prior to the index date. And finally, we
examined patients who received chronically high doses of opioids (MME ≥ 50): this last group
had also received two or more prescriptions with two different start dates within the 90 days
preceding the index date, as well as at least one prescription between 91 and 180 days prior to
the index date.
3.3.4 Statistical Analysis
We reported descriptive statistics of demographic information for patients and clinicians
included in the study. Frequencies and percentages were reported for categorical variables, while
mean and standard deviations were reported for continuous variables. We used descriptive
graphs (histograms and line plots) to illustrate the firing patterns of the prompts and the monthly
frequency of naloxone prescription and possession rates respectively. We used an interrupted
time series (ITS) study design to explore how the prompts might have changed clinicians’ opioid
prescribing patterns. Interrupted time series is used to determine whether the time at which an
intervention was introduced can be associated with changes in an outcome that is measured
continuously over time in the absence of an established control group.
Analysis for the data arising from the interrupted time series design used generalized
linear mixed models to assess the changes in opioid and concomitant medication prescriptions
utilizing a segmented regression framework
99, 100
. The model adjusted for within-clinician
clustering, nested within medical center areas. We used Poisson distribution for count outcome
variables, negative binomial distribution for over-dispersed count outcome variables, and zero-
79
inflated Poisson or negative binomial if the outcome variables contain excessive zeros; finally,
we used the Tweedie distribution for continuous outcomes that had non-ignorable positive mass
at zero
101, 102
. The estimates were reported at a scale of rate ratio.
Models were further adjusted for additional variables: clinicians’ age, gender, race and
ethnicity; types of medical degree; and whether the clinicians are primary care providers. We
define the primary care provider as one who specializes in internal medicine, family medicine, or
general practice. We categorized three types of medical degree: physicians (Medical Doctor
(M.D.) or Osteopathic Doctor (D.O.)), mid-level providers (Physician Assistant (P.A.) or Nurse
Practitioner (N.P.)), and other specialists (Podiatrist (D.P.M), Certified nurse midwife (C.N.M.),
and Doctor of Dental Surgery (D.D.S)). We also adjusted for the number of years the clinician
had been working at Kaiser Permanente Southern California, which serves as a proxy for the
familiarity each clinician would have with the workflow in general and with the organization’s
electronic health record system. The model tests whether the prompts to encourage adherence to
AB2760 led to an immediate change in opioid and naloxone prescription volume (i.e. counts and
MMEs) at the time of the “interruption”, and what its long-term effects were on the clinicians’
prescribing patterns.
Our main model used random intercept only because only very small variance in random
slope, but we also analyzed data allowing individual clinician having different level of trends
(with random slope) to explore whether the estimation of the effect would change. For other
sensitivity test, we analyzed data for pre- versus post-intervention without incorporating the
monthly trend. This analysis tests whether the average monthly prescription per clinician
changed in the year after the intervention comparing to the year before.
80
Furthermore, to address the heterogeneity in our data, we evaluated whether the clinicians
with different features would have responded to the prompts differently by incorporating an
interaction term. These features include age (a proxy for experience of medical practice), gender,
and medical degrees. We subset a subgroup with physicians only and performed the pre-post
analysis with interaction to study the difference of intervention effect on primary care physicians
versus non-primary care physicians. We assume each model intercept as a baseline value while
reporting changes in total MMEs. We used multiple comparisons approach to evaluate whether
the impacts differ for clinicians with different features. We report results with p<0.05 as
statistically significant. Graphs based on aggregated data were made with Microsoft Excel. All
statistical analyses were conducted in SAS 9.4 (SAS Institute Inc., Cary, NC) and R
75
version
4.0.4.
3.4 Study Results
Our study cohort consists of 1,903,289 selected encounters between 2018 and 2019 (with
500,711 unique patients) in which an opioid analgesia was prescribed at the point of the patient’s
discharge. As shown in Table 3-1, the average age of patients who received opioid prescriptions
is 60.4 years (SD = 15.67). Opioids were ordered more frequently for female patients (58.9%).
The majority of encounters involved non-Hispanic White patients (54%), followed by Hispanic
(28.2%) and Black (11.9%). Among all encounters, 21.2% concerned patients who had a history
of overdose within the two years prior to the encounter. We selected 6,515 eligible clinicians
who served these patients in this study (Table 3-1). The average age of clinicians at the
beginning of the study was 45.9 years (SD = 9.43), and they had been employed on average at
Kaiser Permanente for 11.4 years (SD = 9.09). Of our eligible clinicians, 44.7% were female; the
81
majority were Asian (43%) and Non-Hispanic White (39.6); 87.2% had an MD or OD degree;
and 43.7% were primary care providers.
Between December 27
th
, 2018, and December 31
st
, 2019, the prompts were triggered
812,571 times. Among all such instances, a total of 263,484 prompts (32.4%) were triggered for
orders placed by the eligible clinicians of our study cohort, and 132,111 (16.3%) prompts were
triggered for visits involving the patients identified in our study cohort (Figure 3-1). The
frequency of activated prompts reached its peak in January 2019, before gradually decreasing
over the following five months. Since June 2019, the numbers of prompts that each of the three
categories of healthcare provider saw remained stable for the rest of the year.
Outpatient naloxone orders remained minimal throughout the first months of 2018
(Figure 3-2). They experienced a negligible rise in September 2018, when the policy was
approved but had yet to come into legislative effect. Prescription rates experienced a marked
increase from December 2018, likely due to the implementation of the BPA system in the final
week of December. From January 2019, outpatient prescription rates of naloxone sharply
climbed, before steadily dropping off during the next few months; naloxone prescription rates
then stabilized between June and December 2019. This pattern is consistent with the frequency
with which the AB2760-based prompts were activated. The frequency of inpatient naloxone
orders remained relatively stable throughout our study period, in contrast to changes in the levels
of outpatient naloxone prescriptions. Such a finding validates the design parameters of the
prompts, which only targeted outpatient prescriptions.
Naloxone possession rates in the study cohort, shown on a month-by-month basis
covering 2018 and 2019, sharply rose in the wake of the prompts (Figure 3-3). While a rate of
82
less than 1% was observed for each month prior to September 2018, this rate doubled in
December 2018 (2%), and then increased to 13.2% in January 2019. Naloxone possession rates
among patients prescribed with opioids likewise continued to rise to 27.1% by the end of 2019,
with a steadily rising trajectory.
In general, all outcome measures experienced moderate decreasing trends in 2018; and
clinicians prescribed fewer opioids in response to the prompts in 2019 (Table 3-2). Opioid orders
per prescriber per month dropped by 15.1% (p<0.001). Before the intervention, the number of
opioid prescriptions had been decreasing by 1.6% per month (p<0.001). However, in a delicate
scale the post-intervention trend increased by 0.7% (p<0.001), indicating a slight tendency of
regression to the mean. The total Morphine Milligram Equivalents (MME) per prescriber per
month dropped by 7.8% in the wake of the prompt, which is comparable to an expeditious
reduction of 377.8 total MME per prescriber-month. The post trend remained the same as pre-
intervention time at a monthly decreasing rate (RR=0.979, p<0.001). Interestingly, the median
total MMEs per order increased by 11.2% (p<0.001) immediately—although the post-
intervention trend dropped slightly (RR=0.994, p=0.001).
Opioid prescription rates decreased by 11.9% (p<0.001) in response to the prompts at the
visits involving patients with overdose history. The post-intervention trend remained the same as
pre-intervention trend at a range of reduction of 1.7% (p<0.001) per prescriber-month.
Concomitant benzodiazepine orders fell by 20.9% per prescriber-month immediate after the
intervention, followed with a small change in the monthly post-intervention trend (RR=0.997,
p=0.062). We also observed a reduction of concomitant muscle relaxant orders (RR=0.941,
p=0.031), followed with a small change in the monthly trend (RR=0.989, p<0.001).
83
Overall, opioid prescription rates were reduced in efforts to mitigate risks of opioid
abuse. Opioid orders placed for initial opioid users decreased by 14.3% (p<0.001), but the post-
intervention trend shows a slight rise of 1% (p<0.001) per prescriber-month compared to the pre-
intervention trend. Clinicians showed greater caution when renewing an opioid order, with
prescription rates decreasing sharply by 34.8% (p<0.001) after the prompts were implemented;
the post-intervention trend increased by 3.1% (p<0.001) per prescriber-month. Chronically high-
dose orders reduced moderately by 4.2% per prescriber-month shortly after the launch of the
prompts, and minor change in post-intervention trends was observed (RR=0.995, p<0.001). The
simpler, pre-post model shows that on average, all estimable endpoints decreased significantly
per prescriber-month after the wake of the prompts Table 3-2).
At an annual level, opioid prescription rates reduced by an average of 25.2% (p<0.001)
per prescriber-year in 2019 compared to in 2018. A selected clinician is expected to curtail
13941.24 total MMEs (RR=0.724, p<0.001) in 2019. All the other outcome measures for safe
opioid prescription practices show substantial improvement after the intervention (Table 3-3).
There are interesting differences in how various groups of clinicians reacted to the
prompts (Table 3-4). Male clinicians, for instance, prescribed opioids at a rate of 37.7% more
(p<0.001) than their female counterparts prior to the intervention. However, female clinicians
experienced a decline of 24.8% (p<0.001) in their prescribing rates following the intervention,
but no significant changes in opioid prescription rates were observed for male clinicians. The
intervention influenced male and female clinicians differently (p<0.001) for all outcome
measures except for the change in the rate of total MME –although, in an absolute scale, the
annual total MME ordered is expected to fall by 6979.57 units for a female prescriber, while it is
expected to drop by 1541.73 units for a male prescriber on average.
84
Moreover, the volume of opioid prescriptions and safety measures reduced markedly for
physicians after the intervention; no effect on annual opioid prescription was observed for mid-
level providers, and prescription rate increased after intervention for other specialists (RR=1.188,
p<0.001) (Table 3-4). Mid-level providers prescribed more opioids than physicians prior to the
intervention (RR=1.576, p<0.001), and other specialists prescribed fewer opioids than physicians
prior to the intervention (RR=0.683, p<0.001). Significant differences of the intervention effects
were observed between a physician and a mid-level provider, and between a physician and a
specialist.
We did not find any changes in opioid prescribing behaviors for a clinician at average age
(45.9 years) in the study cohort, but for every increased year of age, a clinician prescribed 0.7%
fewer opioids per month (p<0.001) (Table 3-4). At dosage level, the annual total MME is
expected to drop by 3892.98 (p=0.043) for a 45.9-year-old clinician in 2019 compared to the
amount she prescribed in 2018; and as she ages each year, she is expected to prescribe 231.91
fewer total MME annually (p<0.001).
Subgroup analysis on physicians confirmed that all the quantity and safety measures for
opioid prescription reduced for physicians after the intervention (Table 3-5). However, while the
opioid prescription rates (RR=0.747, p<0.001) and other safety prescription measures were
significantly affected by the implementation of the prompt for a primary care physicians, we
observed modest effects for the practices of non-primary physicians in an opposite direction
(RR=1.063, p<0.001). The intervention changed levels of the prescription for PCPs much
different from non-PCP (P<0.001), expect for the dosage per order.
85
3.5 Discussion
In this study, we evaluated changes in naloxone and opioid prescription patterns as influenced by
a prompt in the electronic order entry screen that was designed to improve compliance with
AB2760. We found that naloxone prescription and possession rates were boosted in response to
the intervention. We also found that opioid prescription rates decreased immediately after the
implementation of the prompt, and the median dosage of opioid orders continued to decline. Our
findings thus confirmed our hypothesis. Several insights from behavioral economics and social
psychology may help to explain our findings.
First, people may overlook risks when they lack personal experience, and thus they may
falter in the absence of an injunction from authority
82
. In the context of this study, physicians
may change their prescription behaviors out of regard for an injunctive norm: namely, they
evaluate their own response to the prompt on the basis of a socially determined consensual
standard by which they understand that high-dose opioid prescriptions—or any opioids for
patients with high overdose risk—would be judged negatively by and within the healthcare
community. Second, AB2760 and its associated prompt create psychological friction, meaning
that prescribers may avoid ordering opioids due to the “sludge” effect
103
– the sheer number of
procedural obstacles that need to be cleared in order to successfully prescribe such-and-such a
drug may deter them from so doing. The new law further mandates that prescribers educate their
patients and their designated caregivers about overdose prevention. These additional
requirements may accordingly discourage prescribers from ordering opioids unless absolutely
necessary. Third, we found overall safer opioid prescribing behaviors, regardless of whether the
patients met the criteria that were targeted by the prompt, which suggests that an overriding
observer effect (or Hawthorne effect
104
) may play a role in changing prescription patterns. The
86
prompts were implemented through the HealthConnect
®
system across all regions covered by
Kaiser Permanente. The recording and scrutiny of prescribers’ responses to the prompts may
likewise have induced prescribers to comply more readily with the reminders, thereby facilitating
safer prescription practices. These three insights all provide a strong theoretical basis to account,
at least in part, for why the prompt had a demonstrable effect on prescribing practices. Because
we found that opioid prescription rates were already declining, the low prescribing levels at
baseline mean that the true effects of the intervention may in fact be underestimated because of a
floor effect.
The pattern of the decreasing, then flattening frequency of prompts implies that
clinicians had quickly learned to avoid triggering the alert by not ordering opioids except for
those patients who required them. We also find similar patterns in the frequency of naloxone
prescription, which suggests that it was influenced by the prompts. Of course, not all firings of
the prompt necessarily lead to a naloxone prescription: clinicians might override it for any
number of reasons, and patients likewise might decline naloxone for whatever reason.
We observed that naloxone prescription patterns changed only for outpatient but not
inpatient prescriptions, suggesting that the prompts are effective on their target population.
Among those visits where the prompt was triggered, most eligible patients were discharged
without an opioid prescription. From this, it is possible that the prompt curtailed opioid orders,
though additional factors are likely to be at play.
We found evidence that opioid prescription rates declined after the implementation of the
prompts. Although a small rise in post-intervention trends was observed, the size was negligible,
so the effect was sustained throughout the study period. While the monthly frequency and total
dose of opioid prescription per prescriber decreased after intervention, the monthly median dose
87
per prescriber per order nevertheless increased. A possible explanation for this may rest on a
combination of two factors. First, the monthly median of total Morphine Milligram Equivalents
(MME) per opioid order decreased; second, fewer orders for low-dose prescriptions were made
following the intervention. Accordingly, it seems that prescribers responded to the policy by
taking the low-hanging fruit: that is, they curtailed the low-dose/low-value opioid prescriptions
for patients whose pain is manageable via alternative means, and they kept high-dose opioid
orders for those patients in greater need. One of the advantages of the prompts is that it allows
clinicians to retain their autonomy in practice. A well-designed prompt lets users make informed
clinical decisions while exercising discretion. Overriding the prompts does not necessarily mean
inadequate compliance to the law, nor does it imply good or bad practice. Alert fatigue or lack of
clinical relevance may lie behind some overriding
105
. On the contrary, we ought to guard against
the overly fast tapering of opioids that may cause harm the patients
106
.
The design of the prompts identified key overdose risks, such as overdose history and a
concomitant benzodiazepine and opioid prescription, and we found a significant immediate
reduction in both measures. Although not an objective of the prompt, we found a significant
immediate reduction in the prescription of opioids with concomitant muscle relaxants (monthly
and annually) and concomitant zolpidem prescriptions (annually), suggesting that the
intervention succeeded in enhancing safer prescription practices. The marginally declining trend
continued to fall monthly for concomitant muscle relaxant prescription; Zolpidem, on the other
hand, enjoys a very low concomitant prescription rate to start with, and when combined with
opioids, it has fewer side effects compared to muscle relaxants and is therefore a less harmful
drug.
88
Among the three types of patients at risk of opioid abuse, we found notable immediate
effects on changes in initial or renewal opioid orders, and a modest reduction in chronically high-
dose orders. Their low rates and already abating trends prior to the intervention suggest that
prescribers had been vigilant in practicing safe opioid prescription. The slight growth of the
slope in the post-intervention period may be a result of the floor effect from the low pre-
intervention prescription rate, which left little room for the intervention to take further effect on
reducing the trend. Nevertheless, it is an unalarming degree of rise, and a tapering trend
remained. The rate continued to decline during the post-intervention period for persons receiving
chronic high-dose opioid prescriptions.
Differences in prescribing patterns at clinician-level are more difficult to explain. We had
adjusted for clinician -level variation to address the fact that each clinician started with different
levels of prescription behavior, and each might have developed personal preferences when
responding to the same electronic health record (EHR) system within the same clinical network.
For instance, some clinicians are less likely to be influenced by external recommendations about
treatment selections, whereas others are more flexible and open to suggestion.
We found that an increase in the age of a clinician is positively associated with a
reduction of opioid prescription rates as well as a reduction in the dose ordered, along with other
safe prescription measures. Although the changes are minuscule, it suggests that, in general, a
clinician’s greater experience of medical practice may make her more susceptible to welcome a
useful clinical advisory technology.
We found that female clinician s prescribed opioids less frequently than males prior to the
intervention, suggesting that female clinicians were more conservative than male clinicians with
regard to opioid prescription to begin with. We also found that opioid prescription rates and
89
dosages reduced considerably for female clinicians after the intervention, but not for male
clinicians. The same pattern was also observed for other safe prescription measures. It is possible
that a male clinician places considerable confidence in his own judgement on treatment
selections and is therefore less affected by external influences; it is also possible that he believes
patients’ naloxone possession had protected them from overdose death, hence it is safer to
prescribe opioids. And third, male clinicians may simply be less adaptive to changes. Further
research is needed.
Physicians’ behavior towards opioid prescription changed after the launch of the prompts,
but we did not find a similar change for mid-level providers or other specialists. With
investigation, we identified that 57.46% of mid-level providers in the study cohort worked at the
departments of orthopedics, urgent care, emergency medicine, general or special surgery, and
pain management. Only 20% worked at the department of family medicine, internal medicine
and adult primary care, and the remaining mid-level providers worked at other specialty
departments. These physician assistants and nurse practitioners who obtained authorization to
prescribe opioids are usually the ones with great demand for opioid prescription due to the type
of the patients they serve. Therefore, it is not surprising that these mid-level providers prescribed
higher rate of opioids. Much lower opioid prescription rates and dosage levels were ordered by
certified nurse midwifes or podiatrists when compared to a physician, but these measures
noticeably increased after the intervention. Further scrutiny is warranted to investigate the reason
for this change.
Among physicians, primary care physicians are more susceptible to the changes the
prompts recommended compared to non-primary care physicians. We found that non-primary
care physicians prescribed considerably fewer opioids than their primary care counterparts prior
90
to intervention, so an explanation would be that little room was left to change for non-primary
care physicians. Non-primary care physicians received medical specialty trainings, so it is
possible that they were prepared differently in terms of safely prescribing opioids, for example,
they may be more skilled in practicing alternative pain management method. It is also possible
that on average, the nature of some medical specialties do not require these clinicians to treat
patients with substantive pain management needs. These findings suggested that electronic
reminders are effective to influence primary care clinicians’ prescription behavior. Additionally,
these findings should inspire further investigations on a plausible margin of reduction of opioids
orders expected for a primary care physician.
Clinician-level variability might be caused by numerous factors, including the design of
the EHR system, the culture of the organization, the psychology of individual clinicians and the
characteristics of their patients. We adjusted for variation at clinician level in the model, and the
results should be interpreted as indicative of how prescription patterns changed on average for a
selected clinician. We also adjusted for variation among medical center areas but found that its
contribution to statistical variance was minimal. Therefore, the organization-level
standardization of EHR across clinics may have introduced minimal variation among facilities.
The specific processes that KPSC used to implement its EHR system and how it integrated the
prompts into practitioners’ workflow merits further reflection, since it seems to have successfully
minimized potential variations created by staffing differences and avoided introducing a new
confounding.
There are nevertheless some limitations to our study. First, the increased media attention
on the overdose problem in the wake of the assembly bill being passed has likely affected
prescription rates in ways that make defining clear pre- and post- legislation periods difficult.
91
Second, some of the variability of clinicians’ prescription behaviors may be explained away by
patients’ characteristics and patient-clinician interactions, which we did not adjust for in this
study, and patient-level analyses are not included in the scope of this study. Third, we studied a
specifically designed prompt within a single health-care organization’s EHR system, hence the
result may not generalize to non-Kaiser Permanente prescribers. Fourth, the prompts were not as
accurate as they could be. The data evaluated for triggering the alert was only updated once a
week with patients’ information, rather than in ‘real time’; thus, an alarm of concomitant
prescription may not have been activated if an order for another drug was placed during the same
week. Conversely, the alerts could overfire if an old prescription had indeed run out during the
same week of the new prescription. Fifth, the design of the high-dose prompt responded to a
combination of MMEs assessed with health maintenance modifiers created by the National
Pharmacy Analytics group and selected opioid classes to determine whether the order reached
90+MME/day. Due to the lack of access of such premium data, we were not able to assess the
intervention effect on the opioid prescription dosage at daily level. Finally, we did not adjust for
the volume of patients visits each clinician encountered, which could introduce heterogeneity in
the inference. To address this, we performed subgroup analysis.
Large efforts have been taken to reduce the harm from the overuse of opioids. The
policy-based clinical decision support killed two birds with one stone – it increased naloxone (for
resuscitation after overdose) prescription rates, and it influenced prescribers’ attitudes towards
safer opioid prescription through a behavioral intervention. Prescription practices can be nudged,
but further interventions are necessary for such adjustments to stick. To end the opioid crisis,
implementation scientists should not only consider stopping the injudicious prescription of
opioids, but they should also develop a healthcare pipeline to direct patients to alternative pain
92
management methods, such as counseling and other form of medication-assisted therapy. To
enhance sustainable good practice, we may borrow a few strategies from behavioral
interventions that have successfully influenced antibiotic prescription
104, 107-110
. For example,
providing feedback that directs each participant’s attention to peer comparison can be a powerful
instrument to reinforce desired behaviors
111
. A monthly auto-generated email summarizing the
receiver’s performance compared with her “top performer” peers (low frequency and low dose
opioids prescribers) would enhance the visibility of the aspired-for social norm at no additional
cost
112-114
. The recipients will likely be motivated to improve their performances in order to join
an esteemed social group
108, 110
. Another useful strategy might be to display posters in the
prescriber’s examination room stating her commitment to responsible and safe prescription
practices, and such a poster might include photographs and signatures of the prescriber
109
. This
intervention has proven to be effective because public statements of commitment promote
consistent responses
115
; additionally, the open commitment on display may lower expectations
on the part of patients that they will receive opioids and they may therefore be less likely to
request them. Most importantly, however, we should not ignore our efforts to address the social
determinants underlying and ultimately responsible for opioid abuse.
3.6 Conclusion
The clinical decision support tools embedded in electronic health records (EHR) effectively
enhanced compliance to state law. This intervention, with a thoughtful and positioned design
based on the group of targeted clinicians, should be promoted in more healthcare systems
throughout the US. It is thus feasible to mitigate overdose risks by encouraging safe prescription
habits using behavioral insights. Nudges, when incorporated into EHR, can effectively influence
93
clinicians’ behaviors. The influence of an institutional EHR doesn’t vary across facilities, but
clinician variations were detected. Further investigation of constructive nudge on clinicians with
certain attributes is warranted. A combination of behavioral interventions should be applied to
facilitate the effective implementation of public-health policies.
94
3.7 Tables and Figures
3.7.1 Tables
Table 3- 1. Clinician and patient characteristics.
Characteristic Descriptive Statistics
Clinicians, n 6515
Age at the beginning of the study, mean (SD) 45.9 (9.43)
Years since employed at KPSC, mean (SD) 11.4 (9.09)
Gender, n (%)
Female 2911 (44.7%)
Male 3604 (55.3%)
Race and Ethnicity, n (%)
American Indian or Alaska Native 184 (2.8%)
Asian 2803 (43%)
Hispanic 600 (9.2%)
Non-Hispanic Black 250 (3.8%)
Non-Hispanic White 2583 (39.6%)
Other or Unknown 95 (1.5%)
Clinician Type, n (%)
Physician (MD or OD) 5683 (87.2%)
Mid-level Practitioners (PA or NP) 630 (9.7%)
Other Specialist (DPM, CNM or DDS) 202 (3.1%)
Primary Care Provider, n (%)
Yes 2850 (43.7%)
No 3665 (56.3%)
Patients, n 500711
Encounters (Patients), n 1903289
Age at encounter, mean (SD) 60.4 (15.67)
Gender, n(%)
Female 1121004 (58.9%)
Male 782277 (41.1%)
Other 8 (0%)
Race and Ethnicity, n (%)
American Indian or Alaska Native 6254 (0.3%)
Asian 73817 (3.9%)
Hispanic 537061 (28.2%)
Native Hawaiian or Other Pacific Islander 7282 (0.4%)
Non-Hispanic Black 225920 (11.9%)
Non-Hispanic White 1028457 (54%)
Other or Unknown 24498 (1.3%)
Overdose history, n (%)
Yes 404121 (21.2%)
No 1499168 (78.8%)
* MD, Doctor of Medicine; DO, Doctor of Osteopathic; PA, Physician Assistant; NP, Nurse Practitioner; DPM,
Doctor of Podiatric Medicine; CNM, Certified Nurse-Midwives; DDS, Doctor of Dental Surgery
95
96
Table 3- 2. Sensitivity Analysis: Pre-Post Analysis with monthly data
Intercept p value
Rate Ratio
(pre vs post) p value
Quantity and Dosage of Opioid
Prescription
Opioid Prescription 15.891 <0.001 0.793 <0.001
Total MME 4243.141 <0.001 0.723 <0.001
Median total MME per order 148.116 <0.001 0.895 <0.001
The Prompt Targeted Objective
Overdose history 3.438 <0.001 0.730 <0.001
Concomitant
Benzodiazepines 1.962 0.001 0.520 <0.001
Unintended Consequence
Concomitant Muscle
Relaxants 1.739 0.012 0.667 <0.001
Concomitant Zolpidem NA
Risk of Opioid Abuse
Initial Opioid Order 1.989 0.001 0.819 <0.001
Renewal Opioid Order 1.486 0.019 0.640 <0.001
Chronically High Dose Order 4.147 <0.001 0.757 <0.001
97
98
Table 3- 3. Subgroup analysis: Pre-Post analysis (annually) for physicians only.
All
PCP Non-PCP Difference
Rate
Ratio
p
value
Rate
Ratio
p
value
Rate
Ratio
p
value P value
Quantity and Dosage of Opioid Prescription
Opioid Prescription 0.768 0.000
0.747 0.000 1.063 0.000 0.000
Total MME 0.722 0.000
0.717 0.000 1.021 0.250 0.000
Median total MME per order 0.861 0.000
0.931 0.000 0.845 0.000 0.116
The Prompt Targeted Objective
Overdose history 0.706 0.000
0.696 0.000 1.036 0.061 0.000
Concomitant
Benzodiazepines 0.497 0.000
0.481 0.000 1.091 0.000 0.000
Unintended Consequence
Concomitant Muscle
Relaxants 0.653 0.000
0.618 0.000 1.185 0.000 0.000
Concomitant Zolpidem
0.489 0.000
0.474 0.000 1.110 0.053 0.000
Risk of Opioid Abuse
Initial Opioid Order 0.788 0.000
0.724 0.000 1.198 0.000 0.000
Renewal Opioid Order 0.597 0.000
0.543 0.000 1.217 0.000 0.000
Chronically High Dose Order 0.692 0.000 0.730 0.000 0.842 0.000 0.000
*Interaction term was used for the comparison between primary care physicians and non-primary care
physicians.
99
3.7.2 Figures
Figure 3- 1. Monthly frequency of AB2760-based prompts triggered.
* All, all AB2760 related BPAs fired at KPSC; Provider, the prompts fired for eligible healthcare
providers in this study; Encounter, the prompts fired for encounters included for this study for eligible
patients.
0
50000
100000
150000
200000
Jan-18
Feb-18
Mar-18
Apr-18
May-18
Jun-18
Jul-18
Aug-18
Sep-18
Oct-18
Nov-18
Dec-18
Jan-19
Feb-19
Mar-19
Apr-19
May-19
Jun-19
Jul-19
Aug-19
Sep-19
Oct-19
Nov-19
Dec-19
Frequency of AB2760-based BPAs Triggered
All Provider Encounter
100
Figure 3- 2. Monthly frequency of naloxone prescribed.
* All, all outpatient Naloxone orders at KPSC during study period; Provider, naloxone orders placed by
eligible healthcare providers in this study; Encounter, the naloxone ordered at patient visits eligible for
this study
0
2000
4000
6000
8000
10000
12000
14000
16000
Jan-18
Feb-18
Mar-18
Apr-18
May-18
Jun-18
Jul-18
Aug-18
Sep-18
Oct-18
Nov-18
Dec-18
Jan-19
Feb-19
Mar-19
Apr-19
May-19
Jun-19
Jul-19
Aug-19
Sep-19
Oct-19
Nov-19
Dec-19
Frequency of Naloxone Prescription
All Provider Encounter
101
Figure 3- 3. Baseline Naloxone Possession Rate at the Opioid.
* Baseline Naloxone possession rate is defined as percentage of visits where the patients were prescribed
with Naloxone in the past one year.
0
5
10
15
20
25
30
Jan-18
Feb-18
Mar-18
Apr-18
May-18
Jun-18
Jul-18
Aug-18
Sep-18
Oct-18
Nov-18
Dec-18
Jan-19
Feb-19
Mar-19
Apr-19
May-19
Jun-19
Jul-19
Aug-19
Sep-19
Oct-19
Nov-19
Dec-19
Naloxone Perssession Rate (%)
102
Summary
This dissertation comprises three studies that evaluate the impact of healthcare interventions by
using electronic health records (EHR) from Kaiser Permanente Southern California, an
integrated healthcare system. EHR opens up a whole new horizon, and lets investigators explore
potential causal inferences by using observational data without encountering the budgetary and
ethical issues of an RCT. EHR amounts to millions of records, and enables researchers to
generate low variance estimates; on the other hand, missing data, selection bias, measurement
errors and other limitations of EHR engender risks of substantial bias in parameter estimates.
The studies that make up this dissertation utilize the rich information (both concerning patients
and clinicians) that EHR offers when conducting healthcare studies. They address clinical and
policy questions, as well as the important issue of unmeasured confounding.
The first study assessed the effectiveness and safety of DOACs compared to warfarin in
patients with atrial fibrillation and bioprosthetic heart valves. As far as we know, this study is
one of the largest real-world evaluations comparing DOACs and warfarin in such patients. We
found that DOACs were as effective as warfarin in preventing ischemic strokes, systemic
embolism and transient ischemic attacks. Moreover, patients treated with DOACs had fewer
bleeding events, including fewer instances of intracranial hemorrhage. We also found that
DOAC usage increased during the stay-at-home orders in California that were given in the wake
of the COVID-19 pandemic, not least because the frequent monitoring and lab tests necessary for
patients using warfarin were unavailable. The main findings from this study suggest that the use
of DOACs in the treatment of atrial fibrillation in patients with bioprosthetic heart valves
represents a viable and effective alternative to warfarin.
103
EHR data often makes it possible to adjust for various observed confounders in our
models; the risk of unmeasured confounding nevertheless remains real. The variation in a
clinician’s personal preferences or past experience in using a medication, the patient’s prognosis
or willingness to take medication as prescribed, or even the characteristics of individual
physician-patient interactions all play a role in treatment selections. EHR provides no guarantees
that we can catch the myriad subtle details at play in such variations. For the existing information
that EHR does catch, data may interact in an intricate and poorly understood fashion. A pressing
limitation for cohort studies based on EHR data centers on the uncertainty surrounding a
patient’s fidelity to prescription instructions. Fortunately, our EHR system links patient
encounter data to outpatient pharmacy data, thus meaning that we can identify whether the
prescribed medications were in fact dispensed to the patients. Doubts, however, continue to
linger concerning the precise moment when a patient began a course of medication, and whether
she adhered to the medication as instructed. Variations in patients’ compliance may affect
outcomes and influence the inferences that can be made based on the study results. Another
common source of bias arising from EHR data concerns confounding by indication, which has its
roots in a clinician’s tendency to select treatments for patients who are most likely to benefit
from them. The challenge when addressing issues of confounding by indication centers on the
evaluation of underlying disease severity based on EHR data. For instance, a clinician may
perform a more invasive procedure for patients with a more severe prognosis; yet at the same
time, a patient with a more severe prognosis is more likely to experience a poor outcome in the
first place. Consequently, the unadjusted results may lead to a misconstrued conclusion, namely
that a more invasive procedure causes a poor outcome.
104
The second study in this dissertation, therefore, examined such issues by looking at cases
where disease severity remains an unmeasured variable in the EHR data available to us. We thus
introduced a novel statistical tool that can be used as a supplement to established sensitivity
analyses when unmeasured confounding is a concern. After the adjustments using our proposed
method (L-table), we have greater confidence that had an unmeasured variable of coronary
disease complexity been included, coronary artery bypass surgery (CABG) would be associated
with better clinical outcomes than percutaneous coronary intervention (PCI) when treating
patients with ischemic heart disease. In this study, we used SYNTAX scores as a proxy for
disease severity in our target patients. The SYNTAX score is a system that calculates disease
severity based on disease stages, including coronary lesion location, complexity and
characteristics. Although the SYNTAX score does not directly affect cardiovascular risk, it is a
useful surrogate indicator for disease severity. Both PCI and CABG are designed to improve
coronary blood flow and reduce the symptoms of angina. Of the two, PCI is the less invasive.
The RCTs addressing the issue found that patients with higher SYNTAX scores benefitted more
from CABG. However, SYNTAX scores remain unmeasured in our EHR data. Fortunately, the
prior-knowledge guided, simulation-based L-table method enables researchers to use external
knowledge (information of SYNTAX scores based on RCTs from previous publications) to
derive estimated true treatment effects of CABG versus PCI. The new method we developed in
this study extends the usage of EHR data and empowers researchers to make a meaningful
inference on the study results when they anticipate the problem of unmeasured confounding.
The third chapter of this dissertation presents a study of the effectiveness of a series of
electronic prompts, a type of clinical decision support tool, that was incorporated into the EHR
system at Kaiser Permanente Southern California, to encourage compliance with California’s
105
Opioid Law. Efforts to mitigate the harms of opioid overuse have created a paradox in healthcare
provision. On the one hand, healthcare providers must balance the needs of patients who require
adequate tools for pain management, while on the other they must also comply with regulatory
guidelines that seek to address the opioid crisis head-on. Policy interventions that attempt to
reduce opioid prescription rates, despite the best intentions, risk marginalizing the real needs of
patients. Therefore, we need policy interventions that are patient-centered and evidence-based:
such an approach can minimize the risk of overdose while also protecting those patients whose
pain management strategies necessitate prescription opioids. We found evidence that cogently
demonstrates that the incorporation of a clinical decision support (CDS) tool in the EHR system
effectively improved clinicians’ safe opioid prescription behavior, yet also managed to retain
clinicians’ autonomy in treatment selection. We hope our study will encourage the further
application of thoughtfully designed CDS tools in EHR, and that we successfully convey the
insights to be gained from policy, guidelines, knowledge-sharing and innovation to front-line
medical providers.
We found that the implementation of the prompts in the EHR system was associated with
a reduction in opioid orders and an increase in naloxone prescription rates. On a monthly basis,
opioid prescription rates decreased; and although the median of Morphine Milligram Equivalents
(MME) per order increased immediately, in general, it witnessed a decreasing trend. In addition,
the pre-post analysis shows an improvement in all safe opioid prescription measures. It is worth
noting that behavioral interventions, when implemented in the EHR system along with other
alerts, require careful calibration
107, 116
. The nudges provided by CDS tools offer inductive rules
of thumb for better practice. However, healthcare providers may be overwhelmed by the volume
of alerts, running the risk that they ignore the nudges
117
. Furthermore, we ought to guard against
106
the overly fast tapering of opioids that may cause harm the patients. Although we found that the
prompts had an immediate impact on clinicians’ behavior, leading to a reduction in opioid
orders, further assessment on the appropriateness of opioid prescribing with nudges is warranted.
We hope our study could inspire further research in the design, implementation and evaluation of
effective CDS tools to improve the quality of healthcare.
Underlying all three studies is a common interest in how to provide the most effective
healthcare. A key barrier holding clinicians back from delivering the highest quality healthcare is
the evidence-practice gap, a split between what we know from research and what is implemented
in healthcare settings. Research shows that American medical care suffers from a deficiency in
the quality of care, largely owing to unnecessary patient encounters and uncertainty over whether
clinicians deliver the recommend treatments
118, 119
. On the one hand, American adults do not
receive approximately half of recommended healthcare services
120
, and this deficiency does not
solely depend on socio-demographic disparities
121
; on the other hand, Medicare could save
nearly $500 million per year by curtailing unnecessary and low-value services while still
protecting patients against potential harm
122
. Electronic health records areone of the best cost-
effective tools to close the evidence-practice gap, optimize the healthcare delivery systems and
improve patient care.
Electronic health records introduced a wealth of data that is critical for providing
opportunities for evidence-based care and improving patient-centered outcomes. Although major
challenges still exist in the design of EHR systems, along with their implementation,
dissemination and evaluation, increasing numbers of healthcare providers and users are
embracing the idea of making EHR systems a routine part of healthcare practice. A growing
body of scientists have endorsed the high-quality observation studies that use EHR data, and
107
clinicians have progressively come to accept best practice advisory prompts and alerts to aid in
therapeutic decision-making.
Numerous factors underpin the effective implementation of EHR systems at the personal,
organizational, environmental, and technological levels. Other challenges include large-scale
implementations of EHR, the dissemination of best practices for implementation, and the
application of rigorous evaluative methodologies. Veteran Affairs (VA) health system is one of
the most mature and successful examples of EHR implementations. The EHR at VA has
incorporated decision support tools at the point of care, and includes automated order entry,
routine measurement on quality, and financial incentives based on performance. Researchers
have found that VA patients received recommended care more frequently than the general
population
122
. The lessons learned from the success of VA should be applied to other integrated
healthcare systems. And of course, the EHR data found in KPSC, which has provided the
foundations for three studies in this dissertation, has been equally valuable. From it we have been
able to conduct high-quality research, answering important clinical, statistical, and policy-based
questions.
Some directions for future study exploring the potential for the greater integration of
EHR data and healthcare provision might look in particular at further unmeasured confounders,
such as individual user variability. Why do we see clinician-level differences, whether based on
age, gender, and so on, and how can EHR data help us understand such phenomena? EHR data
might lead us towards a more in-depth understanding of what causes variation, and thereby
might open new ways of improving EHR systems in order to standardize (as much as is
medically justified) how various clinicians operate and choose the most appropriate treatment or
medication plans. A simple way forward might be to construct performance measures that are
108
derived from EHR data and evaluate if such forms of personalized feedback would lead to better
health outcomes in general. As always, the overarching goal is to improve the quality and safety
of patients and the healthcare they receive.
109
References
1. Ornstein SM, Oates RB and Fox GN. The computer-based medical record: current status.
J Fam Pract. 1992;35:556-65.
2. Institute of Medicine—Committee on Improving the Patient Record. The Computer-
Based Patient Record an Essential Technology for Health Care; Revised Edition; National
Academy Press: Washington, DC, USA, 1997.
3. Evans RS. Electronic Health Records: Then, Now, and in the Future. Yearb Med Inform.
2016;Suppl 1:S48-61.
4. International Organization for Standardization. Health Informatics—Capacity-Based
eHealth Architecture Roadmap—Part 1: Overview ofNational eHealth Initiatives; ISO: Geneva,
Switzerland, 2012; Volume ISO/TR 14639-1:2012(en).
5. Warraich HJ, Califf RM and Krumholz HM. The digital transformation of medicine can
revitalize the patient-clinician relationship. NPJ Digit Med. 2018;1:49.
6. King J, Patel V, Jamoom EW and Furukawa MF. Clinical benefits of electronic health
record use: national findings. Health Serv Res. 2014;49:392-404.
7. Nguyen L, Bellucci E and Nguyen LT. Electronic health records implementation: an
evaluation of information system impact and contingency factors. Int J Med Inform.
2014;83:779-96.
8. Prgomet M, Li L, Niazkhani Z, Georgiou A and Westbrook JI. Impact of commercial
computerized provider order entry (CPOE) and clinical decision support systems (CDSSs) on
medication errors, length of stay, and mortality in intensive care units: a systematic review and
meta-analysis. J Am Med Inform Assoc. 2017;24:413-422.
9. Shah K, Lo C, Babich M, Tsao NW and Bansback NJ. Bar Code Medication
Administration Technology: A Systematic Review of Impact on Patient Safety When Used with
Computerized Prescriber Order Entry and Automated Dispensing Devices. Can J Hosp Pharm.
2016;69:394-402.
10. Jha AK, Bates DW, Jenter C, Orav EJ, Zheng J, Cleary P and Simon SR. Electronic
health records: use, barriers and satisfaction among physicians who care for black and Hispanic
patients. J Eval Clin Pract. 2009;15:158-63.
11. Johnston ME, Langton KB, Haynes RB and Mathieu A. Effects of computer-based
clinical decision support systems on clinician performance and patient outcome. A critical
appraisal of research. Ann Intern Med. 1994;120:135-42.
12. Litzelman DK, Dittus RS, Miller ME and Tierney WM. Requiring physicians to respond
to computerized reminders improves their compliance with preventive care protocols. J Gen
Intern Med. 1993;8:311-7.
13. Fast facts. (2021, April 1). Kaiser Permanente. https://about.kaiserpermanente.org/who-
we-are/fast-facts.
14. Chen C, Garrido T, Chock D, Okawa G and Liang L. The Kaiser Permanente Electronic
Health Record: transforming and streamlining modalities of care. Health Aff (Millwood).
2009;28:323-33.
15. HealthConnect® enables care improvement and transformation. (2013, March 5). .
Kaiser Permanente https://aboutkaiserpermanenteorg/community-health/news/kaiser-
permanente-healthconnect-enables-care-improvement-and-tra. (Last Accessed on July 1, 2021).
16. Derose SF, Contreras R, Coleman KJ, Koebnick C and Jacobsen SJ. Race and ethnicity
data quality and imputation using U.S. Census data in an integrated health system: the Kaiser
Permanente Southern California experience. Med Care Res Rev. 2013;70:330-45.
110
17. January CT, Wann LS, Calkins H, Chen LY, Cigarroa JE, Cleveland JC, Jr., Ellinor PT,
Ezekowitz MD, Field ME, Furie KL, Heidenreich PA, Murray KT, Shea JB, Tracy CM and
Yancy CW. 2019 AHA/ACC/HRS Focused Update of the 2014 AHA/ACC/HRS Guideline for
the Management of Patients With Atrial Fibrillation: A Report of the American College of
Cardiology/American Heart Association Task Force on Clinical Practice Guidelines and the
Heart Rhythm Society. J Am Coll Cardiol. 2019;74:104-132.
18. van Walraven C, Hart RG, Singer DE, Laupacis A, Connolly S, Petersen P, Koudstaal PJ,
Chang Y and Hellemons B. Oral anticoagulants vs aspirin in nonvalvular atrial fibrillation: an
individual patient meta-analysis. JAMA. 2002;288:2441-8.
19. Connolly SJ, Ezekowitz MD, Yusuf S, Eikelboom J, Oldgren J, Parekh A, Pogue J,
Reilly PA, Themeles E, Varrone J, Wang S, Alings M, Xavier D, Zhu J, Diaz R, Lewis BS,
Darius H, Diener HC, Joyner CD, Wallentin L, Committee R-LS and Investigators. Dabigatran
versus warfarin in patients with atrial fibrillation. N Engl J Med. 2009;361:1139-51.
20. Granger CB, Alexander JH, McMurray JJ, Lopes RD, Hylek EM, Hanna M, Al-Khalidi
HR, Ansell J, Atar D, Avezum A, Bahit MC, Diaz R, Easton JD, Ezekowitz JA, Flaker G, Garcia
D, Geraldes M, Gersh BJ, Golitsyn S, Goto S, Hermosillo AG, Hohnloser SH, Horowitz J,
Mohan P, Jansky P, Lewis BS, Lopez-Sendon JL, Pais P, Parkhomenko A, Verheugt FW, Zhu J,
Wallentin L, Committees A and Investigators. Apixaban versus warfarin in patients with atrial
fibrillation. N Engl J Med. 2011;365:981-92.
21. Patel MR, Mahaffey KW, Garg J, Pan G, Singer DE, Hacke W, Breithardt G, Halperin
JL, Hankey GJ, Piccini JP, Becker RC, Nessel CC, Paolini JF, Berkowitz SD, Fox KA, Califf
RM and Investigators RA. Rivaroxaban versus warfarin in nonvalvular atrial fibrillation. N Engl
J Med. 2011;365:883-91.
22. Olesen JB, Sorensen R, Hansen ML, Lamberts M, Weeke P, Mikkelsen AP, Kober L,
Gislason GH, Torp-Pedersen C and Fosbol EL. Non-vitamin K antagonist oral anticoagulation
agents in anticoagulant naive atrial fibrillation patients: Danish nationwide descriptive data
2011-2013. Europace. 2015;17:187-93.
23. Vinter N, Linder M, Andersen M, Pedersen AB, Madsen M, Schachterle SE, Ataher Q,
Mo J, Johnsen SP and Ehrenstein V. Classification and characteristics of on-label and off-label
apixaban use in Denmark and Sweden. Pharmacoepidemiol Drug Saf. 2019;28:867-878.
24. Bikdeli B, Madhavan MV, Jimenez D, Chuich T, Dreyfus I, Driggin E, Nigoghossian C,
Ageno W, Madjid M, Guo Y, Tang LV, Hu Y, Giri J, Cushman M, Quere I, Dimakakos EP,
Gibson CM, Lippi G, Favaloro EJ, Fareed J, Caprini JA, Tafur AJ, Burton JR, Francese DP,
Wang EY, Falanga A, McLintock C, Hunt BJ, Spyropoulos AC, Barnes GD, Eikelboom JW,
Weinberg I, Schulman S, Carrier M, Piazza G, Beckman JA, Steg PG, Stone GW, Rosenkranz S,
Goldhaber SZ, Parikh SA, Monreal M, Krumholz HM, Konstantinides SV, Weitz JI, Lip GYH,
Global Covid-19 Thrombosis Collaborative Group EbtINE, the Iua SbtESCWGoPC and Right
Ventricular F. COVID-19 and Thrombotic or Thromboembolic Disease: Implications for
Prevention, Antithrombotic Therapy, and Follow-Up: JACC State-of-the-Art Review. J Am Coll
Cardiol. 2020;75:2950-2973.
25. Eikelboom JW, Connolly SJ, Brueckmann M, Granger CB, Kappetein AP, Mack MJ,
Blatchford J, Devenny K, Friedman J, Guiver K, Harper R, Khder Y, Lobmeyer MT, Maas H,
Voigt JU, Simoons ML, Van de Werf F and Investigators R-A. Dabigatran versus warfarin in
patients with mechanical heart valves. N Engl J Med. 2013;369:1206-14.
26. Guimaraes HP, Lopes RD, de Barros ESPGM, Liporace IL, Sampaio RO, Tarasoutchi F,
Hoffmann-Filho CR, de Lemos Soares Patriota R, Leiria TLL, Lamprea D, Precoma DB, Atik
111
FA, Silveira FS, Farias FR, Barreto DO, Almeida AP, Zilli AC, de Souza Neto JD, Cavalcante
MA, Figueira F, Kojima FCS, Damiani L, Santos RHN, Valeis N, Campos VB, Saraiva JFK,
Fonseca FH, Pinto IM, Magalhaes CC, Ferreira JFM, Alexander JH, Pavanello R, Cavalcanti
AB, Berwanger O and Investigators RT. Rivaroxaban in Patients with Atrial Fibrillation and a
Bioprosthetic Mitral Valve. N Engl J Med. 2020;383:2117-2126.
27. Tidy Quantitative Financial Analysis [computer program]. Version version 1.0.0.; 2020.
28. Robins JM HMaBB. Marginal structural models and causal inference in epidemiology.
2000.
29. Ridgeway G MD, Morral A, Burgette L and Griffin BA. RAND Corporation. Toolkit for
Weighting and Analysis of Nonequivalent Groups: A tutorial for the twang package. 2020.
30. Ridgeway G MD, Morral A, Griffin BA, Burgette L, Burgette ML, Ridgeway M and
Burgette M. Package ‘twang’. 2020.
31. Wooldridge JM. Econometric analysis of cross section and panel data. Cambridge,
Mass.: MIT Press; 2002.
32. Harder VS, Stuart EA and Anthony JC. Propensity score techniques and the assessment
of measured covariate balance to test causal associations in psychological research. Psychol
Methods. 2010;15:234-49.
33. Bang H and Robins JM. Doubly robust estimation in missing data and causal inference
models. Biometrics. 2005;61:962-73.
34. Hullsiek KH and Louis TA. Propensity score modeling strategies for the causal analysis
of observational data. Biostatistics. 2002;3:179-93.
35. Charlson ME, Pompei P, Ales KL and MacKenzie CR. A new method of classifying
prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis.
1987;40:373-83.
36. Lip GY, Nieuwlaat R, Pisters R, Lane DA and Crijns HJ. Refining clinical risk
stratification for predicting stroke and thromboembolism in atrial fibrillation using a novel risk
factor-based approach: the euro heart survey on atrial fibrillation. Chest. 2010;137:263-72.
37. Pisters R, Lane DA, Nieuwlaat R, de Vos CB, Crijns HJ and Lip GY. A novel user-
friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial
fibrillation: the Euro Heart Survey. Chest. 2010;138:1093-100.
38. R: A language and environment for statistical computing. . R Foundation for Statistical
Computing. [computer program]. 2019.
39. Freisinger E, Gerss J, Makowski L, Marschall U, Reinecke H, Baumgartner H, Koeppe J
and Diller GP. Current use and safety of novel oral anticoagulants in adults with congenital heart
disease: results of a nationwide analysis including more than 44 000 patients. Eur Heart J.
2020;41:4168-4177.
40. Sun JC, Davidson MJ, Lamy A and Eikelboom JW. Antithrombotic management of
patients with prosthetic heart valves: current evidence and future trends. Lancet. 2009;374:565-
76.
41. Guimaraes PO, Pokorney SD, Lopes RD, Wojdyla DM, Gersh BJ, Giczewska A,
Carnicelli A, Lewis BS, Hanna M, Wallentin L, Vinereanu D, Alexander JH and Granger CB.
Efficacy and safety of apixaban vs warfarin in patients with atrial fibrillation and prior
bioprosthetic valve replacement or valve repair: Insights from the ARISTOTLE trial. Clin
Cardiol. 2019;42:568-571.
42. Giugliano RP, Ruff CT, Braunwald E, Murphy SA, Wiviott SD, Halperin JL, Waldo AL,
Ezekowitz MD, Weitz JI, Spinar J, Ruzyllo W, Ruda M, Koretsune Y, Betcher J, Shi M, Grip
112
LT, Patel SP, Patel I, Hanyok JJ, Mercuri M, Antman EM and Investigators EA-T. Edoxaban
versus warfarin in patients with atrial fibrillation. N Engl J Med. 2013;369:2093-104.
43. Fang MC, Go AS, Chang Y, Hylek EM, Henault LE, Jensvold NG and Singer DE. Death
and disability from warfarin-associated intracranial and extracranial hemorrhages. Am J Med.
2007;120:700-5.
44. Ruff CT, Giugliano RP, Braunwald E, Hoffman EB, Deenadayalu N, Ezekowitz MD,
Camm AJ, Weitz JI, Lewis BS, Parkhomenko A, Yamashita T and Antman EM. Comparison of
the efficacy and safety of new oral anticoagulants with warfarin in patients with atrial
fibrillation: a meta-analysis of randomised trials. Lancet. 2014;383:955-62.
45. Barnes GD, Lucas E, Alexander GC and Goldberger ZD. National Trends in Ambulatory
Oral Anticoagulant Use. Am J Med. 2015;128:1300-5 e2.
46. Serruys PW, Morice MC, Kappetein AP, Colombo A, Holmes DR, Mack MJ, Stahle E,
Feldman TE, van den Brand M, Bass EJ, Van Dyck N, Leadley K, Dawkins KD, Mohr FW and
Investigators S. Percutaneous coronary intervention versus coronary-artery bypass grafting for
severe coronary artery disease. N Engl J Med. 2009;360:961-72.
47. Kappetein AP, Feldman TE, Mack MJ, Morice MC, Holmes DR, Stahle E, Dawkins KD,
Mohr FW, Serruys PW and Colombo A. Comparison of coronary bypass surgery with drug-
eluting stenting for the treatment of left main and/or three-vessel disease: 3-year follow-up of the
SYNTAX trial. Eur Heart J. 2011;32:2125-34.
48. Mohr FW, Morice MC, Kappetein AP, Feldman TE, Stahle E, Colombo A, Mack MJ,
Holmes DR, Jr., Morel MA, Van Dyck N, Houle VM, Dawkins KD and Serruys PW. Coronary
artery bypass graft surgery versus percutaneous coronary intervention in patients with three-
vessel disease and left main coronary disease: 5-year follow-up of the randomised, clinical
SYNTAX trial. Lancet. 2013;381:629-38.
49. Thuijs D, Kappetein AP, Serruys PW, Mohr FW, Morice MC, Mack MJ, Holmes DR, Jr.,
Curzen N, Davierwala P, Noack T, Milojevic M, Dawkins KD, da Costa BR, Juni P, Head SJ
and Investigators SES. Percutaneous coronary intervention versus coronary artery bypass
grafting in patients with three-vessel or left main coronary artery disease: 10-year follow-up of
the multicentre randomised controlled SYNTAX trial. Lancet. 2019;394:1325-1334.
50. Greenland S. Basic methods for sensitivity analysis of biases. Int J Epidemiol.
1996;25:1107-16.
51. Greenland S. Accounting for uncertainty about investigator bias: disclosure is
informative. J Epidemiol Community Health. 2009;63:593-8.
52. Greenland S. Useful methods for sensitivity analysis of observational studies. Biometrics.
1999;55:990-1.
53. Greenland S. Multiple-bias modeling for analysis of observational data. Journal of Royal
Statistical Society Series A. 2005;168:267-306.
54. Imbens GR, DB. Sensitivity analysis and bounds. . In: Causal Inference for Statistics,
Social, and Biomedical Sciences. 2015;New York: Cambridge University Press:496-512.
55. Rosenbaum P. Design sensitivity and efficiency in observational studies. The Journal of
the American Statistical Association. 2010:692-702.
56. VanderWeele TJ and Ding P. Sensitivity Analysis in Observational Research:
Introducing the E-Value. Ann Intern Med. 2017;167:268-274.
57. Rosenbaum PR. Discussing hidden bias in observational studies. Ann Intern Med.
1991;115:901-5.
113
58. Cornfield J, Haenszel W, Hammond EC, Lilienfeld AM, Shimkin MB and Wynder EL.
Smoking and lung cancer: recent evidence and a discussion of some questions. J Natl Cancer
Inst. 1959;22:173-203.
59. Brookhart MA, Sturmer T, Glynn RJ, Rassen J and Schneeweiss S. Confounding control
in healthcare database research: challenges and potential approaches. Med Care. 2010;48:S114-
20.
60. Rosenbaum PR and Small DS. An adaptive Mantel-Haenszel test for sensitivity analysis
in observational studies. Biometrics. 2017;73:422-430.
61. Greenland S and Mansournia MA. Penalization, bias reduction, and default priors in
logistic and related categorical and survival regressions. Stat Med. 2015;34:3133-43.
62. Chiba Y. Sensitivity analysis for unmeasured confounding of attributable fraction.
Epidemiology. 2012;23:175-6.
63. Vanderweele TJ and Arah OA. Bias formulas for sensitivity analysis of unmeasured
confounding for general outcomes, treatments, and confounders. Epidemiology. 2011;22:42-52.
64. Schneeweiss S. Sensitivity analysis and external adjustment for unmeasured confounders
in epidemiologic database studies of therapeutics. Pharmacoepidemiol Drug Saf. 2006;15:291-
303.
65. Brumback BA, Hernan MA, Haneuse SJ and Robins JM. Sensitivity analyses for
unmeasured confounding assuming a marginal structural model for repeated measures. Stat Med.
2004;23:749-67.
66. B. RPRRD. Assessing sensitivity to an unobserved binary covariate in an observational
study with binary outcome. Journal of the Royal Statistical society. 1983;45:212-218.
67. Hamra GB. Re: "Applying the E Value to Assess the Robustness of Epidemiologic Fields
of Inquiry to Unmeasured Confounding". Am J Epidemiol. 2019;188:1578-1580.
68. Ioannidis JPA, Tan YJ and Blum MR. Limitations and Misinterpretations of E-Values for
Sensitivity Analyses of Observational Studies. Ann Intern Med. 2019;170:108-111.
69. Maddala, G.S. (1983) Limited-Dependent and Qualitative Variables in Econometrics.
Cambridge University Press, Cambridge.
70. Cox NR. Estimation of the correlation between a continuous and a discrete variable.
Biometrics. 1974;30:171-8.
71. Valle JA, Glorioso TJ, Bricker R, Baron AE, Armstrong EJ, Bhatt DL, Rao SV,
Plomondon ME, Serruys PW, Keppetein AP, Sabik JF, Dressler O, Stone GW and Waldo SW.
Association of Coronary Anatomical Complexity With Clinical Outcomes After Percutaneous or
Surgical Revascularization in the Veterans Affairs Clinical Assessment Reporting and Tracking
Program. JAMA Cardiol. 2019;4:727-735.
72. Li L and Greene T. A weighting analogue to pair matching in propensity score analysis.
Int J Biostat. 2013;9:215-34.
73. Akima H. A Method of Bivariate Interpolation and Smooth Surface Fitting for Irregularly
Distributed Data Points. ACM Transactions on Mathematical Software. 1978;4:148-164.
74. Akima H. Algorithm 761: scattered-data surface fitting that has the accuracy of a cubic
polynomial. ACM Transactions on Mathematical Software. 1996;22:362-371.
75. Team RC. R: A language and environment for statistical computing. R Foundation for
Statistical Computing, Vienna, Austria. URL https://www.R-project.org/. 2021.
76. Kim YH, Park DW, Kim WJ, Lee JY, Yun SC, Kang SJ, Lee SW, Lee CW, Park SW and
Park SJ. Validation of SYNTAX (Synergy between PCI with Taxus and Cardiac Surgery) score
114
for prediction of outcomes after unprotected left main coronary revascularization. JACC
Cardiovasc Interv. 2010;3:612-23.
77. He YM, Shen L and Ge JB. Fallacies and Possible Remedies of the SYNTAX Score. J
Interv Cardiol. 2020;2020:8822308.
78. Trecki J. A Perspective Regarding the Current State of the Opioid Epidemic. JAMA Netw
Open. 2019;2:e187104.
79. Seth P, Rudd RA, Noonan RK and Haegerich TM. Quantifying the Epidemic of
Prescription Opioid Overdose Deaths. Am J Public Health. 2018;108:500-502.
80. Seth P, Scholl L, Rudd RA and Bacon S. Overdose Deaths Involving Opioids, Cocaine,
and Psychostimulants - United States, 2015-2016. MMWR Morb Mortal Wkly Rep. 2018;67:349-
358.
81. Kaye AD, Jones MR, Kaye AM, Ripoll JG, Galan V, Beakley BD, Calixto F, Bolden JL,
Urman RD and Manchikanti L. Prescription Opioid Abuse in Chronic Pain: An Updated Review
of Opioid Abuse Predictors and Strategies to Curb Opioid Abuse: Part 1. Pain Physician.
2017;20:S93-S109.
82. Doctor JN, Nguyen A, Lev R, Lucas J, Knight T, Zhao H and Menchine M. Opioid
prescribing decreases after learning of a patient's fatal overdose. Science. 2018;361:588-590.
83. Bohnert ASB, Guy GP, Jr. and Losby JL. Opioid Prescribing in the United States Before
and After the Centers for Disease Control and Prevention's 2016 Opioid Guideline. Ann Intern
Med. 2018;169:367-375.
84. Lin LA, Bohnert ASB, Kerns RD, Clay MA, Ganoczy D and Ilgen MA. Impact of the
Opioid Safety Initiative on opioid-related prescribing in veterans. Pain. 2017;158:833-839.
85. Shoemaker-Hunt S, Sargent W, Jr., Swan H, Mikosz C, Cobb K, McDonald D, Keane N,
von Korff M, Parchman M and Losby J. Developing Clinical Quality Improvement Measures
Aligned With the CDC Guideline for Prescribing Opioids for Chronic Pain: An Important
Strategy to Support Safer Prescribing in Primary Care. Am J Med Qual. 2021;36:125-128.
86. Lin LA, Brummett CM, Waljee JF, Englesbe MJ, Gunaseelan V and Bohnert ASB.
Association of Opioid Overdose Risk Factors and Naloxone Prescribing in US Adults. J Gen
Intern Med. 2020;35:420-427.
87. Fairbairn N, Coffin PO and Walley AY. Naloxone for heroin, prescription opioid, and
illicitly made fentanyl overdoses: Challenges and innovations responding to a dynamic epidemic.
Int J Drug Policy. 2017;46:172-179.
88. Dowell D, Haegerich TM and Chou R. CDC Guideline for Prescribing Opioids for
Chronic Pain - United States, 2016. MMWR Recomm Rep. 2016;65:1-49.
89. Behar E, Bagnulo R and Coffin PO. Acceptability and feasibility of naloxone prescribing
in primary care settings: A systematic review. Prev Med. 2018;114:79-87.
90. Bill Text - AB-2760 Prescription drugs: prescribers: naloxone hydrochloride and other
FDA-approved drugs. (2018) California Legislative Information
https://leginfolegislaturecagov/faces/billTextClientxhtml?bill_id=201720180AB2760. (Last
Accessed on July 1st, 2021).
91. Sinha S, Jensen M, Mullin S and Elkin PL. Safe Opioid Prescription: A SMART on
FHIR Approach to Clinical Decision Support. Online J Public Health Inform. 2017;9:e193.
92. Finley EP, Schneegans S, Tami C, Pugh MJ, McGeary D, Penney L and Sharpe Potter J.
Implementing prescription drug monitoring and other clinical decision support for opioid risk
mitigation in a military health care setting: a qualitative feasibility study. J Am Med Inform
Assoc. 2018;25:515-522.
115
93. Data Resources | CDC’s Response to the Opioid Overdose Epidemic | CDC. (2020). .
Centers for Disease Control and Prevention https://wwwcdcgov/opioids/data-
resources/indexhtml. (Last Accessed on July 1, 2021).
94. Larochelle MR and Bohnert ASB. Opportunities to Address First Opioid Prescriptions to
Reduce Incident Long-Term Opioid Use. Ann Intern Med. 2019;171:129-130.
95. Shah A, Hayes CJ and Martin BC. Characteristics of Initial Prescription Episodes and
Likelihood of Long-Term Opioid Use - United States, 2006-2015. MMWR Morb Mortal Wkly
Rep. 2017;66:265-269.
96. Pollini RA, Banta-Green CJ, Cuevas-Mota J, Metzner M, Teshale E and Garfein RS.
Problematic use of prescription-type opioids prior to heroin use among young heroin injectors.
Subst Abuse Rehabil. 2011;2:173-180.
97. Cicero TJ, Ellis MS, Surratt HL and Kurtz SP. The changing face of heroin use in the
United States: a retrospective analysis of the past 50 years. JAMA Psychiatry. 2014;71:821-6.
98. Barnett ML, Olenski AR and Jena AB. Opioid-Prescribing Patterns of Emergency
Physicians and Risk of Long-Term Use. N Engl J Med. 2017;376:663-673.
99. Linden A and Adams JL. Applying a propensity score-based weighting model to
interrupted time series data: improving causal inference in programme evaluation. J Eval Clin
Pract. 2011;17:1231-8.
100. Linden A, Adams JL and Roberts N. Evaluating disease management program
effectiveness: an introduction to time-series analysis. Dis Manag. 2003;6:243-55.
101. Ma R, Yan G and Hasan MT. Tweedie family of generalized linear models with
distribution-free random effects for skewed longitudinal data. Stat Med. 2018;37:3519-3532.
102. Kurz CF. Tweedie distributions for fitting semicontinuous health care utilization cost
data. BMC Med Res Methodol. 2017;17:171.
103. Thaler RH. Nudge, not sludge. Science. 2018;361:431.
104. Persell SD, Doctor JN, Friedberg MW, Meeker D, Friesema E, Cooper A, Haryani A,
Gregory DL, Fox CR, Goldstein NJ and Linder JA. Behavioral interventions to reduce
inappropriate antibiotic prescribing: a randomized pilot trial. BMC Infect Dis. 2016;16:373.
105. Ancker JS, Kern LM, Edwards A, Nosal S, Stein DM, Hauser D, Kaushal R and
Investigators H. How is the electronic health record being used? Use of EHR data to assess
physician-level variability in technology use. J Am Med Inform Assoc. 2014;21:1001-8.
106. Darnall BD, Juurlink D, Kerns RD, Mackey S, Van Dorsten B, Humphreys K, Gonzalez-
Sotomayor JA, Furlan A, Gordon AJ, Gordon DB, Hoffman DE, Katz J, Kertesz SG, Satel S,
Lawhern RA, Nicholson KM, Polomano RC, Williamson OD, McAnally H, Kao MC, Schug S,
Twillman R, Lewis TA, Stieg RL, Lorig K, Mallick-Searle T, West RW, Gray S, Ariens SR,
Sharpe Potter J, Cowan P, Kollas CD, Laird D, Ingle B, Julian Grove J, Wilson M, Lockman K,
Hodson F, Palackdharry CS, Fillingim RB, Fudin J, Barnhouse J, Manhapra A, Henson SR,
Singer B, Ljosenvoor M, Griffith M, Doctor JN, Hardin K, London C, Mankowski J, Anderson
A, Ellsworth L, Davis Budzinski L, Brandt B, Hartley G, Nickels Heck D, Zobrosky MJ, Cheek
C, Wilson M, Laux CE, Datz G, Dunaway J, Schonfeld E, Cady M, LeDantec-Boswell T,
Craigie M, Sturgeon J, Flood P, Giummarra M, Whelan J, Thorn BE, Martin RL, Schatman ME,
Gregory MD, Kirz J, Robinson P, Marx JG, Stewart JR, Keck PS, Hadland SE, Murphy JL,
Lumley MA, Brown KS, Leong MS, Fillman M, Broatch JW, Perez A, Watford K, Kruska K,
Sophia You D, Ogbeide S, Kukucka A, Lawson S, Ray JB, Wade Martin T, Lakehomer JB,
Burke A, Cohen RI, Grinspoon P, Rubenstein MS, Sutherland S, Walters K and Lovejoy T.
116
International Stakeholder Community of Pain Experts and Leaders Call for an Urgent Action on
Forced Opioid Tapering. Pain Med. 2019;20:429-433.
107. Fox CR, Doctor JN, Goldstein NJ, Meeker D, Persell SD and Linder JA. Details matter:
predicting when nudging clinicians will succeed or fail. BMJ. 2020;370:m3256.
108. Meeker D, Linder JA, Fox CR, Friedberg MW, Persell SD, Goldstein NJ, Knight TK,
Hay JW and Doctor JN. Effect of Behavioral Interventions on Inappropriate Antibiotic
Prescribing Among Primary Care Practices: A Randomized Clinical Trial. JAMA. 2016;315:562-
70.
109. Meeker D, Knight TK, Friedberg MW, Linder JA, Goldstein NJ, Fox CR, Rothfeld A,
Diaz G and Doctor JN. Nudging guideline-concordant antibiotic prescribing: a randomized
clinical trial. JAMA Intern Med. 2014;174:425-31.
110. Linder JA, Meeker D, Fox CR, Friedberg MW, Persell SD, Goldstein NJ and Doctor JN.
Effects of Behavioral Interventions on Inappropriate Antibiotic Prescribing in Primary Care 12
Months After Stopping Interventions. JAMA. 2017;318:1391-1392.
111. Kiefe CI, Allison JJ, Williams OD, Person SD, Weaver MT and Weissman NW.
Improving quality improvement using achievable benchmarks for physician feedback: a
randomized controlled trial. JAMA. 2001;285:2871-9.
112. Lavie N. Distracted and confused?: selective attention under load. Trends Cogn Sci.
2005;9:75-82.
113. Sedikides C and Gregg AP. Self-Enhancement: Food for Thought. Perspect Psychol Sci.
2008;3:102-16.
114. Garcia SM, Tor A and Gonzalez R. Ranks and rivals: a theory of competition. Pers Soc
Psychol Bull. 2006;32:970-82.
115. Deutsch M GH. A study of normative and informational social influences upon
individual judgement. JAbnormPsychol. 1955;51:629-36.
116. Delgado MK, Shofer FS, Patel MS, Halpern S, Edwards C, Meisel ZF and Perrone J.
Association between Electronic Medical Record Implementation of Default Opioid Prescription
Quantities and Prescribing Behavior in Two Emergency Departments. J Gen Intern Med.
2018;33:409-411.
117. Payne TH. EHR-related alert fatigue: minimal progress to date, but much more can be
done. BMJ Qual Saf. 2019;28:1-2.
118. Schuster MA, McGlynn EA, Brook RH. How good is the quality of health care in the
United States? Milbank Q 1998;76: 517-63. .
119. Institute of Medicine. Crossing the quality chasm: a new health system for the 21st
century. Washington, D.C.: National Academy Press, 2001.
120. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A and Kerr EA. The
quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635-45.
121. Asch SM, Kerr EA, Keesey J, Adams JL, Setodji CM, Malik S and McGlynn EA. Who is
at greatest risk for receiving poor-quality health care? N Engl J Med. 2006;354:1147-56.
122. Oronce CIA, Fendrick AM, Ladapo JA, Sarkisian C and Mafi JN. The Utilization and
Costs of Grade D USPSTF Services in Medicare, 2007-2016. J Gen Intern Med. 2021.
Abstract (if available)
Abstract
Electronic health records (EHRs) provide ample opportunities to enhance the quality of patient care, aid in medical decision-making within clinical practice, and facilitate clinical research. Kaiser Permanente Southern California (KPSC) implemented a comprehensive health information system that represents the largest collection of private electronic health record data in the world. In this dissertation, we utilized the rich information concerning both patients and clinicians that Kaiser Permanente Southern California’s EHR system offers, and we evaluated the use of EHR data for clinical and research purposes across three disciplines. First, we compared the safety and effectiveness of different medications on patient outcomes; and we found that off-label use of direct oral anticoagulants (DOACs) in patients with bioprosthetic heart valves can achieve comparable outcomes to warfarin. Second, we addressed the important issue of unmeasured confounding in EHR data; and we created a simulation-based, prior-knowledge guided algorithm to support the inference of clinical study results when an unmeasured confounding issue is an area of concern. Third, we evaluated the effects of a policy-based, computer-generated clinical decision support tool on physician behavior; and we found that the extra friction created by the way KPSC integrated clinical decision support into its processes in response to California State Law had a demonstrable influence on prescribers’ behavior, leading to a reduction in opioid prescription rates.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Value in health in the era of vertical integration
PDF
Use of electronic health record data for generating clinical evidence: a summary of medical device industry views
PDF
A framework for examining relationships among electronic health record (EHR) system design, implementation, physicians’ work impact
PDF
Learning to diagnose from electronic health records data
PDF
Understanding primary nonadherence to medications and its associated healthcare outcomes: a retrospective analysis of electronic medical records in an integrated healthcare setting
PDF
Impact of pharmacy-based transitional care on healthcare utilization and costs
PDF
Essays in pharmaceutical and health economics
PDF
Discriminating changes in health using patient-reported outcomes
PDF
Essays on the equitable distribution of healthcare
PDF
Characterization of health outcomes in patients with hemophilia A and B: Findings from psychometric and health economic analyses
PDF
The role of healthcare volunteers in addressing social determinants of health
PDF
Crohn’s disease: health outcomes and resource utilization in the biologic era
PDF
Impact of medication persistence on clinical and health care cost outcomes in patients with major depressive disorder using retrospective claims data
PDF
Evaluating treatment options for metastatic, castration-resistant prostate cancer: a comprehensive value assessment
PDF
Essays on opioid prescribing and abatement
PDF
The price and clinical impact of specialty drug management in the US healthcare system
PDF
The impact of adherence to guidelines on the health care expenditures of COPD patients
PDF
Essays on the U.S. market for substance use treatment and the impact of Medicaid policy reform
PDF
Examining the adoption of electronic health records system in patient care and students’ education using the GAP analysis approach
PDF
Three essays on estimating the effects of government programs and policies on health care among disadvantaged population
Asset Metadata
Creator
Duan, Lewei
(author)
Core Title
The impact of healthcare interventions using electronic health records: an evaluation within an integrated healthcare system
School
School of Pharmacy
Degree
Doctor of Philosophy
Degree Program
Health Economics
Degree Conferral Date
2021-12
Publication Date
09/13/2023
Defense Date
08/19/2021
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
electronic health records,integrated healthcare system,OAI-PMH Harvest
Format
application/pdf
(imt)
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Doctor, Jason N. (
committee chair
), Adams, John L. (
committee member
), Romley, John (
committee member
)
Creator Email
leweidua@usc.edu,leweiudan@gmail.com
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-oUC15918311
Unique identifier
UC15918311
Legacy Identifier
etd-DuanLewei-10063
Document Type
Dissertation
Format
application/pdf (imt)
Rights
Duan, Lewei
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright. The original signature page accompanying the original submission of the work to the USC Libraries is retained by the USC Libraries and a copy of it may be obtained by authorized requesters contacting the repository e-mail address given.
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Repository Email
cisadmin@lib.usc.edu
Tags
electronic health records
integrated healthcare system