University of Southern California Dissertations and Theses

Use of electronic health record data for generating clinical evidence: a summary of medical device industry views

(USC Thesis Other)

Use of electronic health record data for generating clinical evidence: a summary of medical device industry views

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Use of Electronic Health Record Data for Generating Clinical Evidence:
A Summary of Medical Device Industry Views
by
Cheryl Mariscal Hergert, MPH
A Dissertation Presented to the
FACULTY OF THE SCHOOL OF PHARMACY
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF REGULATORY SCIENCE
December 2023
Copyright 2023 Cheryl Hergert

ii
Dedication
This research is dedicated to the many patients who have freely offered their health
information to the medical product industry with the hope of a better future for themselves and
other patients. May this body of work help improve the approval process for future products
such that they are available to patients faster, cheaper, and better.

iii
Acknowledgments
Getting me to this point took a lifetime filled with amazing people who, at times, had
more faith in me than I had in myself. It is likely I will not remember everyone who influenced
me to pursue this Doctorate along my journey, but with certainty, I am forever grateful to my
teachers from High School through higher education and beyond. Educators work tirelessly to
encourage, motivate, and support their students with little reward in return. I acknowledge all
my teachers for their many acts of kindness and support.
Remembering back to the far reaches of High School, I thank my Science teacher, Mr.
Dozier, who gave me a failing grade and yet gently reminded me I could do better. That small
act of kindness influenced me to believe in myself and pursue my love of science. I am deeply
grateful to my undergraduate professors, Dr. Simek, Dr. Bailey, and Dr. Maxwell, who blazed a
path for my success and, even today, continue to mentor me and support my many crazy ideas
while never questioning my ability to succeed. I thank them for always taking the time to
mentor, listen, and encourage me when I call.
I had presumed I would obtain a higher degree in Chemistry or Biology. Little did I
know my path would be redirected into the world of regulations that impact medical products
and our healthcare. To this, I must thank Dr. Richmond and the numerous Professors at USC.
Interesting how things turn in our lives. I met Dr. Richmond years before applying to the
program. At that time, I was a single mom to four little children focused on day-to-day
responsibilities. She encouraged me to consider the program which, at the time, I declined.
Years later, I was reminded of the opportunity, and she graciously considered my application.
What a wonderful journey it has been participating in the Regulatory Science Program. I have
met amazing individuals and dynamic Professors. To my professors and the administrative staff,

iv
I thank you for your support and encouragement as I struggled through the program and my
research.
No thesis can be secured without responses to the survey and the guidance and support of
an Advisory Committee. I am grateful to those individuals who took time out of their busy
schedules to respond to my survey. Without their input, there would be no results, no analyses,
and no thesis. With deep gratitude, I acknowledge my advisory committee for their time and
thoughtful input to ensure my research was successful and informative. I am most grateful for
my advisor, Dr. Pacifici, who met with me late in the day to assist me with my writing skills, was
kind in her criticism, and was always available for support. Her presence was the calm during
my struggles and successes, the encouragement when I was ready to give up, and the strength
when I lost faith in myself. Dr. Pacifici, Dr. Richmond, Dr. Bain, and Dr. Beringer provided
hours of reviews and discussions by attending my qualifying exam, survey focus group, and the
final defense of my thesis. I also must give a heartwarming thank you to my lifelong friends and
mentors, Dr. Simek, Judy Lang, and Carol Lowell, for their hours of review and input to my
written body of work. Your insights were invaluable and helped guide me in directions I had not
thought of.
My parents have never flinched in supporting my siblings and me to be all that we could
be. I am most appreciative of their forward-thinking parenting approach reminding me, as a girl,
I could be and do anything I set my mind to. When I made choices that did not align with their
expectations, they were there with love and support. I thank them for this as it moved me back
onto a path of growth and onward toward my dream.
And finally, always my first love, to my children, Michele, Michael, Anna, and Angela,
who selflessly sacrificed hours, dinners, movie time, vacations, and important discussions so I

v
could focus on my studies and complete this body of work. You are my heart’s desire and more
valuable to me than this research I’ve undertaken. Yet, you understood the importance of my
work to me and possibly to patients and the medical industry. While I could never have
imagined my life with you when first considering a graduate degree in high school, I cannot
imagine my life without you. It is you who fulfill my dreams and give me hope for the future.
You give my life value and bring me great joy. I give you and our future generations the gift my
parents gave me. Know you can be and do anything you put your soul into. And, when you
have the chance to sit it out or dance, I hope you dance.

vi
TABLE OF CONTENTS
Dedication....................................................................................................................................... ii
Acknowledgments.......................................................................................................................... iii
List of Tables ................................................................................................................................. ix
List of Figures..................................................................................................................................x
Abstract.......................................................................................................................................... xi
Chapter 1. Overview ........................................................................................................................1
1.1 Introduction .................................................................................................................. 1
1.2 Statement of the Problem ............................................................................................. 4
1.3 Purpose of the Study..................................................................................................... 4
1.4 Importance of the Study ............................................................................................... 5
1.5 Limitations, Delimitations, Assumptions..................................................................... 5
Chapter 2. Literature Review.........................................................................................................10
2.1 Literature Review....................................................................................................... 10
2.2 The Evolution and Utilization of Electronic Health Records..................................... 10
2.2.1 The History of Health Records.................................................................... 11
2.2.1.1 Introduction of Digitization ............................................................ 12
2.2.1.2 Adoption by Health Facilities......................................................... 13
2.2.2 The Structure and Function of an Interoperable System ............................. 16
2.2.3 The Current State......................................................................................... 19
2.2.3.1 Uses ................................................................................................ 19
2.2.3.2 Challenges ...................................................................................... 22
2.2.3.3 Opportunities.................................................................................. 25
2.3 FDA’s Consideration of Electronic Health Record Data for Clinical Evidence ........ 27
2.3.1 Legislative Actions...................................................................................... 28
2.3.2 Regulatory Framework................................................................................ 29
2.3.2.1 Evolution of FDA Guidance Documents on the Use of EHR
Data................................................................................................. 31
2.3.2.2 Examples of EHR Use for Regulatory Submissions ...................... 33
2.4 Industry Consideration of Electronic Health Record Use for Clinical Evidence ....... 36
2.4.1 Comparing RCT Data and EHR Data.......................................................... 38
2.4.2 Potential Benefits and Risks of EHR Use.................................................... 39
2.5 Research Framework.................................................................................................. 42
2.5.1 Frameworks of Technology Adoptions....................................................... 42
2.5.2 The Framework of This Research................................................................ 44

vii
Chapter 3. Methodology ................................................................................................................47
3.1 Introduction ................................................................................................................ 47
3.2 Survey Development.................................................................................................. 47
3.2.1 Data Collected ............................................................................................. 48
3.2.2 Survey Verification...................................................................................... 49
3.2.3 Survey Deployment ..................................................................................... 49
3.3 Population Selection................................................................................................... 49
3.3.1 Inclusion Criteria ......................................................................................... 50
3.3.2 Distribution of Survey ................................................................................. 51
3.4 Survey Analysis.......................................................................................................... 51
Chapter 4. Results..........................................................................................................................52
4.1 Survey Participation ................................................................................................... 52
4.2 Demographics............................................................................................................. 54
4.2.1 Respondent Demographics.......................................................................... 54
4.2.2 Company Demographics ............................................................................. 55
4.2.3 Familiarity with EHR .................................................................................. 56
4.3 TOE Analyses ............................................................................................................ 60
4.3.1 Perceived Benefits....................................................................................... 60
4.3.2 Organizational Readiness............................................................................ 65
4.3.2.1 Quality of Data ............................................................................... 66
4.3.2.2 Infrastructure Sufficiency ............................................................... 71
4.3.2.3 Management Commitment ............................................................. 82
4.3.3 Perceived External Pressures....................................................................... 83
4.4 Final Thoughts............................................................................................................ 92
Chapter 5. Discussion ....................................................................................................................97
5.1 Overview.................................................................................................................... 97
5.2 Considerations for Research Methodology ................................................................ 98
5.2.1 Delimitations ............................................................................................... 98
5.2.1.1 Framework...................................................................................... 98
5.2.1.2 Respondent Demographics............................................................. 99
5.2.2 Limitations................................................................................................. 100
5.2.2.1 Response Rate............................................................................... 100
5.2.2.2 Survey Design............................................................................... 101
5.2.2.3 Providence of Results................................................................... 102
5.3 Consideration of Results .......................................................................................... 102
5.3.1 Perceived Benefits..................................................................................... 103
5.3.2 Organizational Readiness.......................................................................... 104
5.3.2.1 Data Access and Quality............................................................... 105
5.3.2.2 New Skills..................................................................................... 107
5.3.2.3 Strategy Development .................................................................. 107

viii
5.3.3 External Pressures...................................................................................... 108
5.4 Final Comments ....................................................................................................... 110
References....................................................................................................................................114
Appendix A. Survey........................................................................................................ 123
Appendix B. TOE Cohort Data Tables (n = 46)............................................................. 142

ix
List of Tables
Table 1: Key Terms ....................................................................................................................7
Table 2: Acronyms ......................................................................................................................8
Table 3: Key Legislative Actions ...............................................................................................9
Table 4: Medical Device Submissions Utilizing Data from Electronic Health Records
for Clinical Evidence ..................................................................................................34
Table 5: Characteristics of EHRs and RCTs .............................................................................39
Table 6: Strengths and Challenges of Data from EHRs Compared to Data from RCTs ..........41
Table 7: TOE Influences of Adoption and Drivers to be Explored ..........................................45
Table 8: Number of Questions by Survey Categories ..............................................................48
Table 9: Respondent Distribution and Final Cohort .................................................................52
Table 10: Weighted Means of Respondents’ Choosing Advantages with Using EHR
Data for Generating Clinical Evidence as Compared to Using Data from
randomized Controlled Trials .....................................................................................62
Table 11: Weighted Means of Experienced Respondents’ Choosing Advantages with
Using EHR Data for Generating Clinical Evidence as Compared to Using
Data from Randomized Controlled Trials ..................................................................65
Table 12: Weighted Means of Ranking of Concerns when Using EHR Data ............................69
Table 13: Weighted Means of Respondent’s Agreement with Statements Regarding
Analyzing EHR Data as Compared to Analyzing Data from Randomized
Clinical Trial ...............................................................................................................76
Table 14: Weighted Means of Respondent’s Agreement with Statements Regarding
Access to EHR Data as Compared to Data from Randomized Clinical Trial ............81

x
List of Figures
Figure 1: Ideal Interoperable EHR System ................................................................................18
Figure 2: Phases of the Gartner Hype Curve..............................................................................37
Figure 3: Iacovou Adoption Framework....................................................................................44
Figure 4: Survey Cohorts and Exclusion....................................................................................53
Figure 5: Distribution of Self-identified Level of Authority (n=45)..........................................54
Figure 6: Distribution of Self-identified Functions (n = 46)......................................................55
Figure 7: Distribution of Organizational Size (n=46)................................................................56
Figure 8: Distribution of Length of Time Engaged in the Use of EHR Data (n=46).................57
Figure 9: Distribution of Organizational Stage of Adoption (n=46)..........................................58
Figure 10: Proportion of FDA Submissions Over the Past 12 Months that Used EHR
Data as the Primary Source of Evidence ....................................................................59
Figure 11: Distribution of Respondents' Involvement in Discussions or Activities about
the Use of EHR Data (n=46)......................................................................................60
Figure 12: Possible Advantages of Using EHR Data for Generating Clinical Evidence
as Compared to Using Data from Randomized controlled Trials (n=46) ..................61
Figure 13: Advantages with Using EHR Data for Generating Clinical Evidence as
Compared to Using Data from Randomized Controlled Trials as Determined
by Respondents from Organizations that Utilized EHR Data to Support a
Submission (n = 24) ...................................................................................................64
Figure 14: Perception of the Sufficiency of EHR Data for Supporting a Primary
Endpoint (n=42) .........................................................................................................66
Figure 15: Ranking of Concerns when Using EHR Data (n=41).................................................68
Figure 16: The Perceived Opportunity to Reduce the Number of Randomized Trials
(n=45) .........................................................................................................................71
Figure 17: Functional Areas Needing New Skillsets to Advance the Use of EHR Data.............72
Figure 18: Respondent’s Agreement with Statements Regarding Analyzing EHR Data
as Compared to Analyzing Data from Randomized Clinical Trials...........................75
Figure 19: Perceived Organization’s Accessibility to EHR Data (n=46) ....................................77
Figure 20: Factors Impacting Access to Data as a Comparison between EHR Data and
Data from Randomized Clinical Trials (n=46)...........................................................80
Figure 21: Prevalence of Organizations with Goals and Objectives Including the Use of
EHR Data (n=42)........................................................................................................82
Figure 22: Distribution of Respondents Hearing Senior Executives Speaking About
Using EHR Data to Generate Clinical Evidence (n=42)............................................83
Figure 23: Distribution of Factors that Influenced the Advancement of using EHR Data
(n=46) .........................................................................................................................85
Figure 24: Perception of CDRH Encouragement for the Use of EHR Data (n = 43)..................87
Figure 25: Helpfulness of FDA Guidance Documents and Summary Report (n=46) .................91

xi
Abstract
The 21st Century Cures Act (The Cures Act) of 2016 was written with the intent ‘to
accelerate the discovery, development, and delivery of 21st-century cures’ (H.R. 34, 2016). To
facilitate this acceleration, a provision is included for fostering the use of existing real-world
data, data that is obtained outside the realm of clinical trials, to generate clinical evidence for
product effectiveness and safety. This mandate has significant implications for the
biopharmaceutical and medical device industries. Interestingly, using real-world data is not
altogether new for medical device manufacturers. The Medical Device Amendments of 1976
allow patient information from medical records to be incorporated into regulatory submissions.
However, this practice has not been used often due the lack of organized data obtained from
patient paper charts. Now, with the recent implementation of electronic health records (EHR)
the ability to incorporate organized digital patient data for regulatory purposes is more
feasible. Given the long-standing allowance of patient data to support effectiveness by the
Center for Devices and Radiological Health (CDRH), implementing this approach would seem to
be a welcomed and easy shift. Even so, the industry continues to rely on data obtained from
costly and time-consuming randomized controlled trials (RCTs).
The goal of this research was to investigate and understand perspectives of the medical
device industry towards the use of EHR data for generating clinical evidence. Responses were
obtained via a survey containing questions based on the Technology, Organization, and
Environment (TOE) framework. Results suggest that the industry perceives using data obtained
from EHRs may reduce costs, shorten timelines, and provide access to generalizable
data. However, there is reluctance by the industry due to a perceived lack of skills, regulatory

xii
uncertainty, and data quality concerns. The findings from this research provide insights into
possible next steps to support the adoption of using EHR data for generating clinical evidence.

1
Chapter 1. Overview
1.1 Introduction
Randomized controlled trials (RCT) have been established over time as the gold standard
for generating clinical evidence of safety and effectiveness. What makes RCTs robust is the
unambiguous determination of treatment effect by minimizing bias through the use of
randomization between the treatment and a comparator, the control of variables like
co-morbidities, and a strict regimen of therapy and tests. Regulators globally are familiar with
the evidence generated in this manner and are readily open to accepting data from RCTs as
evidence of the safety and effectiveness of product performance. Over time, however, RCTs
have become increasingly costly and time-consuming to conduct, often presenting as a
bottleneck in the product development process, thus delaying patient access to novel medical
therapies (Schilsky, 2017). Moreover, concerns have been raised over results obtained from
RCTs and their lack of ability to translate to real-world clinical practice. Some products have
shown to present different safety profiles when administered beyond the RCT setting, such as
within the clinical setting. This difference is primarily due to the strict inclusion criteria often
utilized within RCTs (Niraula et al., 2012).
As product development processes become increasingly burdensome, stakeholders have
struggled to find ways to increase efficiency, reduce cost, and expedite time to market for
innovative treatments. The entrance of electronic health records (EHR) across healthcare
facilities provided an alternative pathway for generating clinical evidence that could be less
resource-intensive and provide more generalizable results. Through the passage of the Health
Information Technology for Economics and Clinical Health (HITECH) Act in 2009, (H.R. 1,
2009), EHRs expanded across the healthcare sector throughout the United States (US) and soon

2
became the primary means of storing, processing, and accessing patient health information (Gold
and McLaughlin, 2016). While the primary purpose of EHR platforms is to manage healthcare
information of patients, these platforms have also become powerful tools for aggregating and
summarizing large sets of clinical data. Moreover, since the data captured within EHR platforms
are obtained in real-world clinical settings, summarized analyses are generalizable across broad
populations (Sherman et al., 2016). These benefits have led regulators, industry representatives,
and patients to perceive this data as having the potential to provide clinical evidence more
efficiently while being more reflective of real-world clinical use. In turn, these perceived
benefits promise faster access to new affordable therapies for patients (Hogle and Das, 2017;
Hwang, Sachs and Kesselheim, 2017). These assumptions were, in part, responsible for the
passage of the 21st Century Cures Act (The Cures Act) which gave added impetus to using realworld data, such as those residing in EHRs, for generating clinical evidence (H.R. 34, 2016).
The Cures Act was enacted after many years of deliberations among regulators, industry
representatives, and patient advocacy organizations. The law’s objective is to accelerate the
development and approval of innovative therapies through the exploration of new ideas of
generating clinical evidence, including the use of real-world data (H.R. 34, 2016). While RCT
data are collected within research settings to generate clinical evidence, EHR data are captured
within clinical settings to deliver patient care (Sherman et al., 2017). Consequently, EHR data
may contain variables and nuances that have not been controlled leading to biases within the
data. For example, a chosen treatment for a patient in the real-world setting is based on disease
progression, the clinician’s access to a specific therapy, and insurance coverage, variables not
found in RCTs. In addition, data housed in EHR systems are often fragmented, unmonitored,
and lacking in quality assurance oversight (Bowrin et al., 2019). Even though new forms of

3
statistical analyses are being investigated to address these challenges, the validity of results
obtained from these data remains in question (Lu et al., 2021). Given the long-term wellestablished trust in RCTs, a shift in the product development process requires organizational
changes for both regulators and industry, including revisions to processes, analyses, and best
practices.
It should be noted that while The Cures Act brought the use of patient health information
to center stage for the purposes of therapeutic product development, this practice is not entirely
new. For example, both the pharmaceutical and the medical device industries have been using
patient health records for obtaining post-market safety data. In addition, since 1976, regulations
allowed for the use of patient data by the medical device industry for product development
through the passage of the Medical Device Amendments. However, there were few guidance
documents that provided strategies for using this data. What The Cures Act introduced, was a
mandate for the Food & Drug Administration (FDA) to explore pathways for accepting realworld data into its regulatory decision-making. While pharmaceuticals are specifically
mentioned, the mandate had major implications across all medical products.
To address this new mandate, the FDA issued guidance documents containing the
agency’s expectations on how real-world data could be used to generate clinical evidence
(H.R. 34, 2016). For example, in 2017, the Center for Devices and Radiological Health (CDRH)
issued its guidance on the Use of Real-World Evidence to Support Regulatory Decision-Making
for Medical Devices. Further, CDRH published a report in 2021 summarizing examples of
product applications which had been approved or cleared using clinical evidence generated from
different types of real-world data, including EHRs (FDA, 2021c). While the report lists 90
submissions, only 15 utilized EHR data as the primary source of clinical evidence. These

4
examples represent only a minor dent in the overall number of submissions CDRH receives
annually (FDA, 2021d). There has been little research about the medical device industry’s views
on using EHR data for generating clinical evidence. The intent of this research was to
investigate and understand perspectives of the medical device industry towards the use of EHR
data for generating clinical evidence. Barriers and influences were investigated along with
industry’s attitude toward the use of EHR data for generating clinical evidence.
1.2 Statement of the Problem
Unless the medical device industry s increases its adoption of EHR data for product
development and commercialization, the intent of The Cures Act may not be fully realized. In
consequence, patient access to new therapeutic devices may continue to be limited due to
expense and time delay. There has been and continues to be an apparent hesitancy by the
medical device industry toward the use of patient data, paper and digital, for regulatory purposes.
Minimal research had been conducted to understand this hesitancy towards the use of EHR data.
Without the research defined in this document, factors that influence the adoption of using EHR
data to generate clinical evidence will remain unclear.
1.3 Purpose of the Study
The purpose of this research was to investigate the perspectives and attitudes towards
EHR data of professionals working in the medical device industry who are familiar with using
EHR data for generating clinical evidence. To guide this research the Technology, Organization,
and Environment (TOE) framework, as revised by Iacovou, Benbasat and Dexter (1995) was
selected. Three determinants of adoption were investigated: 1) perceived benefits of using EHR
data, 2) readiness of the organization to integrate the data, and 3) external pressures from

5
stakeholders to adopt the use of the data to generate clinical evidence. Based on these factors, a
survey was developed and administered to medical device professionals who have investigated
or utilized EHR data to support a CDRH submission. Results from this research provide insight
into possible barriers and influences on adoption and lessons learned from those who have
investigated the use of or successfully used EHR data to support a CDRH submission. The
findings of this research may help guide the adoption process of generating clinical evidence
from EHR data, thus meeting the intent of The Cures Act.
1.4 Importance of the Study
The results obtained from this research provide information based on concepts defined
within the TOE framework that can inform the medical device industry, regulators, and
policymakers for advancing EHR data for the purpose of generating clinical evidence. For
example, the findings afford the industry insights into optimal practices for preparedness,
drawing from the experiences of those adept in using EHR data. Further, the results present
recommendations for regulators, emphasizing best practices to harmonize messaging and ensure
consistency in application. The study highlights key considerations for policymakers as they
contemplate revising well-established processes. Most importantly, the study provides
actionable information to help implement changes toward expediting patient access to innovative
therapies, aligning well with the objectives outlined in The Cures Act.
1.5 Limitations, Delimitations, Assumptions
The scope of this research was delimited to professionals from the medical device
organizations that market Class II and III devices within the US. This research focused only on
the use of EHR data for the generating of clinical evidence to support a submission. Thus, the

6
investigation did not include the use of data outside of EHR systems, such as paper medical
records or observational study registries. Furthermore, the research did not encompass the
utilization of other technologies that capture health data, such as artificial intelligence, social
media, wearable devices, mobile health apps, and software that does not fit the definition of
EHRs. Inclusion criteria of respondents was limited to those with experience investigating or
using EHR data for generating clinical evidence. The study was further delimited to the TOE
framework as defined in Section 1.3 and 2.5.2 which may have further limited the depth of
information on the topic.
Limitations included the scarcity of respondents who are familiar with the concepts,
barriers, and influences, due to the relative novelty of using EHR data for regulatory
submissions. In addition, it is well documented surveys often show a low rate of completion by
respondents. Furthermore, in consideration of respondents’ time availability, the survey was
limited in length, which resulted in a lack of in-depth exploration of the topic. The digital design
of the survey created another limitation due to its potential lack of accessibility.
Certain assumptions were made in this research. For instance, it was assumed
respondents were honest with their responses and are well-informed about the actions being
taken within their company regarding the adoption of EHR data for generating clinical evidence.
Further, it was assumed a sufficient number of respondents would take the survey such that
results would be valid, reliable, and informative.

7
Table 1: Key Terms
Key Term Definition
Electronic Health
Record (EHR)
An electronic record of health-related information on an individual
that (1) includes patient demographic and clinical health
information, such as medical history and problem lists; (2) has the
capacity to provide clinical decision support, support physician
order entry, capture and query information relevant to health care
quality, exchange electronic health information with, and integrate
such information from other sources; and (3) has been certified to
the certification criteria adopted by the Secretary per 45 CFR 170
(FDA, 2021b)
Electronic data
interchange
(EDI)
The intercompany communication of business documents in a
standard format (IBM, 2022)
Health
Information
Technology
(HIT)
The hardware and software used for the electronic creation,
access, and exchange of patient health information (H.R. 1, 2009)
Interoperability Health information technology that (A) enables the secure
exchange of electronic health information with, and use of
electronic health information from, other health information
technology without special effort on the part of the user; (B)
allows for complete access, exchange, and use of all electronically
accessible health information for authorized use under applicable
State or Federal law; and (C) does not constitute information
blocking (H.R. 34, 2016)
Office of
National
Coordinator of
Health IT
Technology
(ONC)
Charged with developing, maintaining, and overseeing a strategic
plan to guide nationwide adoption of health information
technology in both the public and private sectors (Thompson and
Brailer, 2004)
Randomized
Controlled Trial
(RCT)
A study in which participants are randomly (i.e., by chance)
assigned to one of two or more interventions (or diagnostic tests)
of a clinical study (FDA, 2013a)

8
Table 2: Acronyms
Acronym Term
ANSI American National Standards Institute
ARRA American Recovery and Reinvestment Act
CDER The Center for Drug Evaluation and Research
CDRH Center For Devices and Radiological Health
CMS Centers for Medicare and Medicaid Services
CRF Case report form
DOI Diffusion Of Innovation
EMT Emergency Medical Technician
FCC Federal Communication Commission
FDA Food & Drug Administration
FD&C Food, Drug & Cosmetic Act
FHIR Fast Health Interoperability Resources
HHS Health and Human Services
HL7 Health Level Seven International Organization
IOM Institute of Medicine
LOINC Logical Observation Identifiers Names and Codes
MDIC Medical Device Innovation Consortium
MU Meaningful Use
NLP Natural Language Processing
PCORnet Patient-Centered Clinical Research Network
RWE Real world evidence
SNOMED Systematized Nomenclature of Medicine
TAM Technology Acceptance Model
TAUT Theory of Acceptance and Use of Technology
TOE Technology, Organization, Environment
TPB Theory of Planned Behavior
US United States
USC University of Southern California
USD US Dollars

9
Table 3: Key Legislative Actions
Date Legislation Impact to Research
1962 Kefauver -Harrison
Amendment
Established the requirement for substantial
evidence of safety and effectiveness for
medical products.
1976 The Medical Device
Amendments
Established the criteria for classifying medical
devices and the determinants of valid scientific
evidence for safety and effectiveness as
codified in 21 CFR Part 860.
2003
Medicare Prescription Drug
Improvement and
Modernization Act (MMA)
Initiated the development of EDI standards for
sharing health information across facilities.
2004 Executive Order Established the ONC.
2009
Health Information
Technology for Economics
and Clinical Health Act
(HITECH)
Provided funds to healthcare facilities for the
implementation of EHRs.
2012
Food and Drug
Administration Safety and
Innovation Act (FDASIA)
Mandated the FDA, ONC, and FCC to develop
a HIT regulatory framework.
2016 The 21st Century Cures Act
(The Cures Act)
Mandated a program to investigate the use of
real-world data for generating clinical
evidence.

10
Chapter 2. Literature Review
2.1 Literature Review
The primary focus of the literature review included three topics, 1) the evolution of EHR
systems under the Office of National Coordinator of Health IT Technology (ONC), 2) the
evolution of the use of EHR data for generating clinical evidence by the medical product
industry and the FDA, and 3) the laws and regulations impacting the ONC and the FDA and the
use of EHR data for product development. The body of literature included legislative actions
and regulations, books, legal briefs, the Federal Registry, FDA guidance documents, and peerreviewed articles. Public conferences, such as those hosted by the Clinical Trials Transformation
Initiative (CTTI) and the Institute of Medicine (IOM), were also examined. Governmental
documents were retrieved from agency websites, such as the FDA and the ONC. Databases
explored include PubMed, Embase, the University of Southern California (USC) Digital Library,
and Google Scholar. Keywords used for literature searches included electronic health record,
EHR, regulatory submission, real-world data, real-world evidence, randomized controlled trial,
RTC, clinical evidence, and the titles of pertinent laws and regulations. Approximately 400
publications were reviewed, summarized, and assessed to understand the history, development,
and adoption of EHRs within the ONC, the FDA, and the medical product industry. Of these,
109 publications are referenced within this document. An analysis of the findings is summarized
below.
2.2 The Evolution and Utilization of Electronic Health Records
Today, EHRs have become commonplace across healthcare facilities in the US,
encompassing hospitals, community clinics, and doctor’s offices (Gold and McLaughlin, 2016).
Like traditional paper-based medical charts, EHRs contain comprehensive patient information

11
such as medical history, diagnoses, treatment plans, medications, and laboratory test results.
However, the digital nature of EHRs facilitates real-time access and entry of information by
healthcare professionals at different locations allowing for the possibility of a complete and
accurate profile of the patient’s health status (Hoffman, 2016; Miksad and Abernethy, 2018).
The data within the EHR platform can be used in other ways (Gliklich, Leavy and Dreyer, 2019;
Hoffman, 2016). A notable example is the utilization of aggregated EHR data to monitor
COVID-19 diagnoses from global (Bourgeios, Gutiérrez-Sacristán and Keller, 2021) to regional
assessments (Deeds et al., 2020). Further, researchers are exploring the use of EHR data to
generate clinical evidence regarding the safety and effectiveness of medical products (Resnic and
Matheny, 2018; Callahan et al., 2019). As applications and regulations continue to evolve, the
true potential and impact of EHRs are yet to be fully realized.
2.2.1 The History of Health Records
The practice of documenting a patient’s medical care and health status goes far back in
history, with the earliest known example found in Egypt (Rothman, 1991; Lederer, 1995). Until
recently, paper was the medium used to capture health information, from ancient papyrus rolls to
modern patient charts. The content expanded over time from simple descriptions of diseases and
treatments to the inclusion of family history, lifestyle, physical status, and previous illnesses
(Gillum, 2013). The format also evolved from loose sheets of physician’s notes to a more
organized structure that allowed access by members of the clinical care team. But without
adequate quality control measures, the information contained in the paper medical record was
often incomplete, inaccurate, and illegible, resulting in treatment errors (Gillum, 2013). Sharing
of information across the healthcare team in real-time was not feasible because access to paper
records was only available to one user at a time and transmission of information across multiple

12
locations was limited. With the introduction of computers, data residing in paper records were
transitioned to digital, impacting the way health information is stored, accessed, and utilized.
2.2.1.1 Introduction of Digitization
The idea of digitizing patient health records was introduced in 1964 when Akron General
Hospital placed computer terminals in key locations within the hospital as part of a pilot
initiative. These terminals were placed in the admitting area and on individual floors within the
hospital allowing patient information to be transmitted between central computers and
workstations. Patient demographics, medications, test results, health status, and discharge notes
could be entered into ‘electronic’ health charts at different workstations. The information was
stored and maintained within the hospital mainframe computers. In partnership with
International Business Machines Corporation (IBM), the hospital examined the effectiveness of
digital patient health records to reduce hospital costs. The results showed that digital health
records reduced overall facility costs by improving operational efficiencies. In addition, the
results also pointed to a reduction in medical errors (Akron General Hospital, 1966). Despite
these positive outcomes, the Akron study did not lead to a broad adoption of digital health
records across the healthcare sector. At the time of the Akron study, computer technology was
still in its infancy and available only in limited settings where highly trained personnel were
available to utilize and maintain the technology. Few healthcare facilities had the space,
resources, or technical experts to manage complex mainframe computers, considered to be state
of the art at the time. In fact, it would be another four decades before a paradigm shift to digital
records would take place.
Catalyzed by the introduction of desktop computers in the 1970s, and the expansion of
the internet in the 1990s, large hospitals started to adopt digital platforms but took a hybrid

13
approach. For example, paper charts were used to capture clinical notes, while digital
technology was used to transmit information to the hospital’s pharmacy (Shortliffe, 1999; Evans,
2016). Sharing of information with outside institutions still occurred primarily through paperbased means such as facsimiles (Ajami and Bagheri-Tadi, 2013). Ultimately, a comprehensive
approach for creating a national digital health system was prompted by a critical report authored
by the IOM.
2.2.1.2 Adoption by Health Facilities
In 1999, the Institute of Medicine (IOM) published a key report on the state of the U.S.
healthcare infrastructure (Institute of Medicine, 2000). According to the report, 44,000 patients
were dying each year in the U.S. due to medical errors, a figure that far exceeded deaths due to
car accidents during that period. In addition to the cost of human lives, the economic cost of
medical errors was an annual estimate of up to 29 billion U.S. Dollars (USD). The IOM report
identified the nation’s decentralized, hybrid, and fragmented healthcare infrastructure as a major
contributing factor for the medical errors. This served as a call to action to create a standardized
healthcare information system (Institute of Medicine, 2000).
By this time the internet had become the “electronic information highway” across the
nation and around the globe. Paper communication was swiftly being replaced by emails and the
sharing of information between multiple parties became what seemed to be real-time. Electronic
data interchange (EDI), which enabled interoperability for finding, sending, receiving, and
integrating information across different institutions, was becoming a standard across the financial
business sector. The success of this innovative technology for centralizing and defragmenting
data drove the passage of the Medicare Prescription Drug Improvement and Modernization Act
(MMA) in 2003. The Act mandated the Centers for Medicare and Medicaid Services (CMS) to

14
develop EDI standards for sharing health information across facilities as a means to address the
concerns realized by the IOM report (Thompson and Brailer, 2004). To drive this initiative, the
ONC was established in 2004 within the Department of Health and Human Services (HHS)
through Executive Order. The objective of this new office was to advance the development and
use of health information technology (HIT), such as EHRs, and to establish the expectations for
data sharing through EDI and interoperability between different entities (H.R. 1, 2009;
Thompson and Brailer, 2004).
Despite these initial national efforts, the adoption of EHRs was sluggish. According to a
2008 survey, only 13-17% of physicians reported that they were using a basic EHR system
(Fleming et al., 2011). With an average initial cost of 50,000 USD for set up and an annual
maintenance cost of 20,000 USD, most providers considered the cost too prohibitive (Miller and
West, 2007). Many healthcare facilities were not willing to make the upfront investment even
though data indicated a 5-year net benefit of 57,500 USD per provider due to the reduction of
administrative costs and a decrease in medical errors once EHRs were implemented (Wang
et al., 2003; Hoffman, 2016). Moreover, a report by the Congressional Budget Office estimated
that 78 billion USD could be saved annually if EHRs were implemented nationwide (Orszag,
2008). Hence, it became clear that Congress needed to take further action to address the
financial roadblock. The answer came in the form of a 27 billion USD fund through the
HITECH Act of 2009 (H.R. 1, 2009).
The HITECH Act, as part of Congressional Bill H.R.1, more commonly known as the
American Recovery and Reinvestment Act (ARRA), called for HHS to establish programs for
improving patient quality of care through a national digital healthcare system. The Act granted
10 million USD to be distributed to federally qualified healthcare facilities that implemented an

15
EHR certified system that could be used in meaningful ways for the clinician and the patient
(Chaney et al., 2020). The requirements, as defined by the ONC, became formally known as
Meaningful Use (MU) and focused on features that would improve a patient’s quality of care
through accessibility, accuracy, and completeness of EHR data (ONC, 2021). One of the
primary features included the ability for providers to send prescriptions electronically to the
pharmacy and share information with other healthcare entities (Jha, 2010). The success of the
HITECH Act became evident when, in 2014, survey data showed 97% of hospitals and 83% of
office-based practices had implemented an EHR system meeting the Meaningful Use (MU)
requirements (Gold and McLaughlin, 2016).
While the adoption of EHRs was successful, interoperability across different platforms
lagged. Between 2011 and 2012, over 300 different EHR vendors emerged with little effort
toward aligning platforms that would allow sharing of electronic information. Vendors used
different platform languages and form layout for data entry. For example, one facility might
capture date of birth in separate fields for day, month, and year while another facility might
capture this same information in one field. The health information landscape became further
complicated as other sources of data were introduced. Many forms of medical devices like
pacemakers and continuous glucose monitoring systems began integrating software that captured
health data specific to the use of the device. These types of unconventional health data were not
integrated into the patient’s EHR, resulting in health information that became fragmented and
disconnected. As these new products blurred the boundaries between health records and medical
devices, a need for clarity regarding regulatory jurisdiction became apparent. In response, the
2012 Food and Drug Administration Safety and Innovation Act (FDASIA) addressed regulatory
jurisdiction and issues of interoperability.

16
Contained within FDASIA was a mandate for the FDA, the ONC, and the Federal
Communication Commission (FCC) to develop a HIT regulatory framework. The resulting
report, “The FDASIA Health IT Proposed Strategy and Recommendations for a Risk-Based
Framework”, defined a path forward. The framework tasked each federal agency with actions
that would support the adoption of EHRs nationwide. The ONC was mandated to improve
interoperability, patient accessibility, and information sharing. The FDA was tasked with
regulating software in medical devices which captured health data. And the FCC, per its role of
providing oversight of electronic communications, committed to accelerating the authorization of
HIT (FDA, FCC, and ONC, 2014). This framework was eventually enacted through The 21st
Century Cures Act (The Cures Act) located within Legislative Bill H.R. 34. The Cures Act
intent is to ‘accelerate the discovery, development, and delivery of 21st century cures’ (H.R. 34,
2016). The process took two years of deliberation and input from over 1400 stakeholders,
including patient groups, resulting in overwhelming approval (Hogle and Das, 2017). Today, the
basic core elements of a well-functioning EHR application include clear display of information,
patient accessibility, tracking of test results, administrative and clinical decision supports, and
electronic ordering of prescriptions. Additionally, through interoperability, data can be shared
with public health agencies for community health initiatives (Hoffman, 2016).
2.2.2 The Structure and Function of an Interoperable System
An EHR, as defined by the ONC and codified within 45 CFR Part 170, is a certified
digital patient health record that contains complete clinical information from multiple systems
and is available in real-time for the clinician to make evidence-based decisions (FDA, 2021b). A
medical record is primarily a repository of information about a patient’s health, but a fully
interoperable EHR system allows for the expansion of its application (Figure 1). It would have

17
the ability to contain not only information entered by the primary clinician, such as
demographics, vitals, prescriptions, diagnoses, immunizations, and clinician notes, but also
information from other sources (Evans, 2016; Hernandez-Boussard et al., 2019). For instance,
notes and test results from specialists, data transmitted from diabetic sensors and insulin pumps,
as well as information obtained through wearable devices like personal lifestyle tracking apps,
could be uploaded and integrated into a patient’s EHR. In addition, patients would have the
ability to enter information directly into their record such as quality of life information, health
history, treatment history, and disease symptoms, as well as lifestyle choices, like diet and
exercise habits. By providing access through a patient portal, patients could engage with their
own health information as well as track their health status over time using personalized
dashboards (Hoffman, 2016). A patient’s EHR information could be available to the entire
healthcare team, including the primary clinician, specialists, hospitals, pharmacists, and
emergency medical technicians (EMT). Further, information about diagnoses and procedures
would be provided electronically to payers for billing information (Pylypchuk and Johnson,
2020; Hoffman, 2016).

18
Figure 1: Ideal Interoperable EHR System
While a fully interoperable EHR platform is not yet realized, the HIT infrastructure
continues to evolve and improve, driven by the MU requirements. The infrastructure for a fully
interoperable HIT system requires accommodation for multiple interfaces such that data can be
stored, communicated, and exchanged between different functions and sites (Menachemi and
Brooks, 2006) and flexibility to evolve along with health systems and innovative products.
Initially, the MU requirements established a nationwide EHR system with basic capabilities for
data entry of a patient’s vital signs, demographics, prescriptions, current diagnoses, and smoking
status (Blumenthal and Tavenner, 2010). In later stages, additional capabilities were added,
including patient access to their health information and the tracking of health outcomes in
alignment with the newly defined Medicare/Medicaid value-based payments system (Gold and
McLaughlin, 2016; CMS, 2021). Most recently, improvements in data sharing and patient
accessibility have been added (ONC, 2021). These requirements have been codified within 45

19
CFR Part 170 (FDA, 2021b) which also defines the standards needed for certification. Although
the HITECH Act states certification is voluntary, it is required for federally qualified healthcare
facilities and those who exchange data with them (ONC, 2020). As of 2019, 87% of physicians
and 94% of hospitals are using certified EHR systems (Chaney et al., 2020) with recertification
occurring on an annual basis (ONC, 2020). ONC continues to drive the national HIT
infrastructure toward full interoperability through standards and improved technology.
2.2.3 The Current State
Today, EHR systems are established nationwide, from large institutions to small private
clinics. While clinicians use the data in real-time to inform medical decisions to treat individual
patients, the data also has the potential for many other applications. The current uses and
challenges of data from EHRs will be examined below.
2.2.3.1 Uses
It is well understood that a health system that is interconnected nationwide would benefit
patients while lowering costs. And indeed, evidence shows that quality of care for patients
improves as EHR systems are integrated into healthcare. A recent analysis conducted by
Mathematica Policy Research is a case in point (Lammers, McLaughlin, and Barna, 2016). The
study examined Medicare and Medicaid EHR data captured between 2010 to 2013, a period
spanning the early introduction to integration of EHR systems. Results showed an inverse
correlation between hospital admissions and EHR s integration at the healthcare facilities. That
is, as EHR systems were implemented, reductions in hospital admissions were observed. The
authors attributed the positive results to physicians’ access to complete and up-to-date patient
information. Further, the authors suggested that the primary goal of the HITECH Act, improving

20
quality of care through the use of EHRs, had essentially been met (Lammers, McLaughlin, and
Barna, 2016).
Other EHR benefits continue to be realized as familiarity with EHR technology increases.
According to Dr. Sherman from the FDA, real-world data, such as data from EHRs, can be used
in many ways beyond patient care, like informing public health initiatives, driving quality
improvements within a health system, and advancing therapeutic product development (Sherman
et al., 2016). The recent example of using EHRs to track COVID-19 immunizations
demonstrates the potential of this technology to impact public health. Following the introduction
of the COVID-19 vaccine to fight against the global pandemic, it was important to track the
uptake, effectiveness, and safety of the vaccine. EHR systems enabled the communication of the
safety information across local and national public health entities in almost real-time.
Beyond vaccines, data residing in EHR systems have informed evidence-based treatment
practices. An example is the Sentinel Initiative, established by the FDA in collaboration with
private, academic, and governmental entities to evaluate the safety of medical products. Data are
uploaded to the Sentinel System at regular intervals and are used to inform providers and
manufacturers on the use patterns and safety of therapeutics. To date, the aggregated data have
contributed to 200 published papers and 30 drug assessments (Sentinel Coordinating Center,
2021).
The National Patient-Centered Clinical Research Network (PCORnet), a nongovernmental consortium of healthcare facilities and patients, provides another example in which
EHR data are used to inform providers on approaches to patient care. In this case, EHR data are
uploaded to a large, shared database that can be queried by healthcare providers to obtain real-

21
world answers to real-world questions. Over 60 million patients of diverse backgrounds and
ethnicities across the U.S. contribute to this shared database (Waitman, 2021).
The medical product industry has also used EHRs to generate real-world evidence that
can support safety and effectiveness of products from development to commercialization. For
example, EHR data can be used to gain market approval, expand product label, demonstrate costeffectiveness to support reimbursement, and, most recently, support machine learning algorithms
(Ramamoorthy and Huang, 2019; Zou et al., 2020). In fact, Dr. Schneeweiss of the FDA, in
collaboration with Dr. Franklin from Harvard Medical School, suggest the agency has been using
real-world data ‘for decades’ to assess product safety and to support regulatory decisions
(Franklin and Schneeweiss, 2017).
An example of product approval based on EHR data is the 2012 approval of HeartWare®
Ventricular Assist System, a left ventricular assist device (LVAD) to treat heart failure. The
company conducted a prospective, non-randomized, open-label clinical trial comparing the
safety and effectiveness of its device against standard of care, with the primary endpoint being
overall survival at 180 days following implantation of HeartWare. The active control was
aggregated data from the Interagency Registry for Mechanically Assisted Circulatory Support
(INTERMACS) (HeartWare, Inc., 2012), a registry database containing health data from patients
who have advanced heart failure and have received an FDA approved mechanical circulatory
support device (Holman, 2012). Data residing in INTERMACS are obtained primarily from
patients’ EHRs. After reviewing the data from HeartWare compared to data from
INTERMACS, the FDA panel agreed the HeartWare device was beneficial, and approval was
granted.

22
More recent examples include the Medtronic cardiovascular permanent pacemaker lead
and the Zoll Medical Corporation automated external defibrillator. Both devices were originally
cleared per the 510(k) pathway and were on the market until July 6, 2012, at which time FDA
issued a ruling which reclassified certain Class II devices to Class III. In response, a PMA was
submitted for each device using EHR data to support safety and effectiveness. Medtronic
utilized a combination of EHR data, literature, and in-vitro bench testing, while Zoll Medical
utilized a combination of EHR data, literature, animal testing, and in-vitro bench testing as their
substantial clinical evidence to gain market approval. Medtronic obtained approval in 2015
under PMA P120017 and the Zoll AED received approval in 2017 under PMA P160022
(Medtronic, Inc., 2015; ZOLL Medical Corporation, 2017).
Just as EHR data can be used for product approvals, the data can also be used for product
withdrawals. Recently, HeartWare, discussed above, was removed from the market based on
data from EHRs and the FDA's Manufacturer and User Facility Device Experience (MAUDE)
database (Zipp, 2021). This real-world evidence indicated that HeartWare posed a slightly
higher risk of serious adverse events as compared to other devices on the market (Numan et al.,
2021; Schramm et al., 2020). Hence, the FDA and manufacturer determined the product no
longer met the needs of the patients and removed the product from distribution (FDA, 2022).
2.2.3.2 Challenges
In today’s U.S. healthcare system, most healthcare entities utilize an EHR platform that is
not fully interconnected with other entities. As a result, patients visiting different providers,
hospitals, and networks for multiple reasons may find their health information increasingly
fragmented over their lifetime. For example, if a patient is hospitalized and the primary care
physician is not connected to the hospital’s EHR system, the physician will not have immediate

23
access to the discharge information and, instead, may need to rely on the patient’s recollection.
This process may lead to incomplete and inaccurate information (Holmgren and Adler-Milstein,
2017).
Although larger hospitals and medical groups often possess advanced IT networks, the
same cannot be said for many smaller hospitals and physician practices. The statistics indicate
that just one in eight small independent hospitals and physician practices possess systems that
can be categorized as interoperable (Pylypchuk and Johnson, 2020). Instead, most of these
entities rely on a fragmented assortment of communication channels, such as patient recall,
telephone calls, faxes, and emails (Holmgren and Adler-Milstein, 2017). Furthermore, numerous
significant everyday healthcare occurrences are absent from current datasets. A prime example
is the area of mental health services, which frequently displays reluctance towards utilizing
EHRs and sharing data with other healthcare providers (Glasgow et al., 2012). The absence of
transparent data exchange between healthcare entities and patients leads to disjointed and
incomplete health information, making it susceptible to errors.
The lack of interoperability across the U.S. healthcare system was highlighted in a 2020
report by the ONC. The report stated that fifty percent of hospitals in some major cities were
unable to exchange health information with public health agencies, putting the population at risk
during infectious outbreaks such as COVID-19 (Pylypchuk and Johnson, 2020). For the
successful exchange of data between facilities, EHR platforms must have similar software
languages, table formats, and data entry styles, including units of measurements. Inconsistencies
in EHRs between facilities contribute to the lack of interoperability. For example, EHR
platforms are typically maintained by large vendors (Evans, 2016) with the two largest, Epic and
Cerner, using different application languages and database table structures that are incompatible

24
with each other (Holmgren and Adler-Milstein, 2017; Maissenhaelter, Woolmore, and Schlag,
2018). Considering that a typical EHR contains over 100,000 different data elements within a
single record (Hripcsak and Albers, 2013), there is high probability that data across different
platforms would be configured differently. For example, blood pressure and blood sugar
information may be captured inconsistently across different EHRs. Blood pressure might be
entered as one variable, diastolic over systolic, in one system and as two separate variables in
another. Moreover, blood sugar can be recorded as millimoles per liter or milligrams per
deciliter (Shull, 2019).
In addition to challenges related to data configuration, there exists a lack of consistency
between structured data, such as tabulated test results, and unstructured data, such as clinic notes.
A recent white paper authored by Bonafede (2022) forVeradigm (formally Allscripts), a
healthcare technology company specializing in data platforms, highlighted this issue. Veradigm
offers two distinct platforms for tracking cardiovascular data from clinical sites. The Veradigm
Cardiology Registry platform compiles data from 4000 sites encompassing 16.5 million patients.
These data are extracted from sites’ EHRs systems and mapped to specific quality
measurements. The Practice Fusion EHR platform, on the other hand, is cloud-based and
provides secure access to EHR data from sites, including clinic notes and free-text entries. This
platform contains records for 48 million patients across 123 thousand providers. While
Veradigm Cardiology Registry predominantly employs structured data, the Practice Fusion
platform contains a mix of structured and unstructured data. In an effort to assess the
effectiveness of each system in identifying key patient populations for heart failure management
and recruitment for clinical trials, the company discovered left ventricular ejection fraction
(LVEF) results, a critical measurement for evaluating heart health, were more frequently

25
documented in the clinic notes rather than the structured data fields within the Practice Fusion
platform. Among the data set of approximately 650 thousand records, only 5000 LVEF data
points could be directly retrieved from the structured fields. As the paper’s authors noted, this
disparity between structured and unstructured data could lead to a biased understanding of a
patient’s overall health leading to significant implications for research and healthcare
management (Bonafede, 2022). Similarly, Hernandez-Boussard and colleagues presented
evidence that unstructured clinic notes often contain more vital information than structured data.
Consequently, the authors concluded that a comprehensive review of both unstructured and
structured data is imperative to accurately assess a patient’s clinical progress (HernandezBoussard et al., 2019). However, most of the data employed for research purposes remains
confined to the structured format. The laborious nature of reviewing unstructured data was
exemplified by Shull’s research at the University of Barcelona, shared during the 2018 Artificial
Intelligence (AI) Conference for Health (Shull, 2019). In her study, she examined 53,000
admission records spanning 2001 to 2012, with the aim of utilizing AI and machine learning
algorithms to predict patient outcomes. Despite the data being sourced exclusively from a single
hospital, Shull stated it took two years to clean the data before it could be employed for the
intended research purposes.
2.2.3.3 Opportunities
While the challenges previously discussed are monumental, they are being addressed
globally. Collaborations across regions are ongoing to define common standards and best
practices for connecting EHR platforms. (Miani et al., 2014). To build consistency across
structured data, two common data languages, Logical Observation Identifiers Names and Codes
(LOINC) and Systematized Nomenclature of Medicine (SNOMED), are being adopted

26
worldwide for recording test results. Of the two, LOINC has been adopted by more than 78,000
agencies and healthcare institutions for coding health measurements and observations. Building
an EHR system to LOINC standards ensures consistency in how test results are entered. For
example, with LOINC, blood pressure is entered as only two independent variables, thus
eliminating the inconsistency noted above (Shull, 2019). The second of the two global
standards, SNOMED, is being adopted as a common language for a broad coverage of clinical
terms, including anatomy, diseases, and laboratory procedures and tests. Implementing
SNOMED across HIT systems standardizes how data units are recorded in EHRs, such as the
units of blood sugar noted above (SNOMED International, 2021). Technology is also being
developed to tackle the challenge of dealing with unstructured data that is difficult to access. For
example, Natural Language Processing (NLP) reads and interprets unstructured text, allowing
important information to be extracted and integrated with structured data. NLP has already
demonstrated its effectiveness in bringing consistency when documenting adverse events (Wang
et al., 2019). For example, NLP was employed to extract LVEF data from the unstructured data
within the previously mentioned platform, Veradigm Practice Fusion EHR. By utilizing NLP to
extract LVEF data from unstructured notes, the quantity of usable LVEFs values increased
significantly, rising from 5000 to 36,000 (Bonafede, 2022)
Addressing interoperability, the Health Level Seven International Organization (HL7),
founded in 1987 for the purpose of creating standards for the exchange of data across hospital
information systems, initiated Fast Health Interoperability Resources (FHIR). This certified
American National Standards Institute (ANSI) standard is now widely recognized as the gold
standard for EHR EDI. Within the US, 32% of HIT developers follow FHIR (Shull, 2019).

27
Establishing interoperable EHR systems is a global pursuit. Australia launched an
interoperable nationwide EHR system in 2016 that allows patients to move their health record
across different healthcare facilities. Estonia implemented an EHR system that allows patients to
access their data via blockchain technology. Hong Kong initiated a web-based system that
integrates real-time patient information sharing across clinics and hospitals (Evans, 2016; Shull,
2019). Yet, the U.S. continues to struggle with providing a nationwide EHR that is interoperable
and widely accessible. With limited interoperability and high fragmentation, the use of EHR
data for research and medical product development remains challenging. Even so, regulations
have been introduced to encourage the utilization of EHR data for product development. The
history and impact of these regulations are discussed in the next section.
2.3 FDA’s Consideration of Electronic Health Record Data for Clinical Evidence
The earliest FDA regulation addressing digital health information was the 1997 21 CFR
Part 11 (Part 11). As health information became digitized, data used to support product
development, including evidence of safety and effectiveness, moved from paper to digital
format. At the urging of the medical product industry, the FDA issued Part 11, which describes
the regulatory criteria for establishing equivalency between paper and electronic records (FDA,
1997). These rules attempted to standardize electronic data and records storage. However,
discussions continue to this day between regulators and industry on how to interpret these
regulations as new technologies emerge.
As EHR platforms became more commonly used at clinical sites and familiarity with
digital processes increased, stakeholders in the medical product sector saw the potential of using
EHR data for generating clinical evidence for product development. With RCTs becoming
increasingly expensive and time-consuming, stakeholders wondered if health data obtained

28
within a clinical setting, and organized within an EHR platform, had the potential to supplement
or replace the well-established RCT process. Patients and industry stakeholders supported the
use of this non-traditional means for generating evidence, while regulatory agencies appeared to
be ambiguous. In response, The Cures Act mandated the FDA to develop a program for
investigating the use of real-world data in the regulatory environment, resulting in the issuance
of key guidance documents.
2.3.1 Legislative Actions
Since the HITECH Act, the industry's utilization of digital patient information has
evolved to the point that EHR data is being explored to generate clinical evidence as
supplemental to or replacement of RCTs. Initially, sponsors became familiar with the new EHR
technology when clinical trial sites began transcribing data from EHRs to paper case report
forms (CRF). Seeing the benefit of EHR, the medical product industry introduced a similar
platform for CRFs called remote data capture (RDC), which allowed clinical trial sites to enter
study data directly into the sponsor’s database. Sites could now transcribe EHR data directly
into predefined fields in the RDC electronic CRF. Soon, it became evident that the EHR
technology could be harnessed further to transform the process of product development.
Regulators, industry, and other stakeholders came together to advance the use of EHR data
within the medical product clinical research realm. This collaboration catalyzed the issuance of
The Cures Act. The new law mandated the establishment of a program to evaluate the potential
use of real-world data to generate evidence of safety and effectiveness outside of RCTs. To
codify the mandate, a new section, 505F, was created within the Food, Drug & Cosmetic Act
(FD&C), which established the program. While pharmaceuticals are specifically mentioned in

29
505F, the new mandate had major implications across all medical products. This research is
focused on the implications toward medical devices and their approval.
It should be noted that the use of data obtained outside of RCTs was a contentious topic
as the Act was drafted. Patient advocacy organizations promoted the use of real-world data
because of its potential to expedite patient access to new therapies (Hogle and Das, 2017;
Hwang, Sachs, and Kesselheim, 2017). Other stakeholders, however, were not in favor, citing
the possibility of reducing the robustness of evidence and increasing the influence of bias (Avorn
and Kesselheim, 2015). To help mitigate these concerns, The Cures Act explicitly states the
regulatory standard of clinical evidence, as defined in Section 505 of the FD&C, is not to be
altered (H.R. 34, 2016).
2.3.2 Regulatory Framework
The opportunity to use real-world data within the medical device industry did not depend
on the passage of The Cures Act. The allowance had been in place by CDRH since 1976 via the
passage of the Medical Device Amendment and the codified within 21 CFR Part 860.7(c)(2):
Valid scientific evidence is evidence from well-controlled investigations, partially
controlled studies, studies and objective trials without matched controls, welldocumented case histories conducted by qualified experts, and reports of
significant human experience with a marketed device, from which it can fairly
and responsibly be concluded by qualified experts that there is reasonable
assurance of the safety and effectiveness of a device under its conditions of use
(FDA, 2021a).
However, medical device sponsors rarely utilize this type of data for a number of reasons.
When the Medical Device Amendment was first passed, case histories were documented in paper
format, which made data processing resource intensive and inefficient. Further, regulators
questioned the quality of the data and, thus, did not encourage the use for regulatory purposes.

30
However, with the introduction of digital formatting through EHRs, organized data became
readily accessible suggesting an improvement in the quality.
The 2013 Guidance on Design Considerations for Pivotal Clinical Investigations for
Medical Devices, often regarded as a foundational resource for clinical research conducted by
industry, introduced strategies for incorporating patient health data into clinical trials, thereby
elevating this approach. However, the guidance also includes cautionary notes regarding the use
of such data (FDA, 2013a). This wording within the guidance suggests that the agency is
hesitant to consider data that deviate from the RCT framework. Such a stance might dissuade
sponsors from utilizing this approach to generate evidence.
Also in 2013, the FDA issued the Guidance for Industry, Electronic Source Data in
Clinical Investigations to address the shift of patient health information from paper to digital
format. The purpose of the guidance was to assist industry in defining best practices in the
capture, retention, and storage of electronic data. With the influx of information from multiple
places, such as laboratory reports, medical images, and patient diaries contributing to a patient’s
EHR, determining the original source of data became confusing. The guidance defines source as
the initial documentation in which it is attributable, legible, contemporaneous, and accurate. The
guidance also introduces EDI within the regulatory space by identifying it as best practice for
transferring data into the sponsor’s database to ensure reliability, quality, integrity, and
traceability of the data (FDA, 2013b).
To further encourage the use of real-world data, The Cures Act granted funding to the
FDA to create three crucial guidance documents and a white paper. These resources were
intended to provide support for the various initiatives and requirements outlined within the Act
(H.R. 34, 2016).

31
2.3.2.1 Evolution of FDA Guidance Documents on the Use of EHR Data
The first guidance document to be issued by CDRH was the Use of Real-World Evidence
to Support Regulatory Decision-Making for Medical Devices guidance in 2017, nine months
after the passage of The Cures Act. This guidance provides FDA’s expectations regarding the
quality of real-world data when used for clinical evidence in a regulatory submission. In this
document, the term real-world data was officially defined as ‘data relating to patient health status
and/or the delivery of health care routinely collected from a variety of sources’ (FDA, 2017).
The guidance does not mandate the use of real-world data in regulatory submissions yet
describes the criteria which should be considered if a sponsor chooses to do so. The FDA
considers the importance of the relevance and reliability of the data when generating valid
scientific evidence for a given study. Additionally, the FDA recognizes the significance of
incorporating evidence from diverse sources, including data obtained during routine clinical care,
when assessing the safety profile of a medical device (FDA, 2017).
In parallel with the work being conducted by CDRH, CDER also published a document
highlighting the division’s expectations for the use of real-world data for clinical evidence. This
white paper, The Framework for FDA’s Real-World Evidence Program, aligns with CDRH’s
expectations for the use of real-world data for generating clinical evidence (FDA, 2018).
In 2018, CDRH and CDER jointly produced the guidance document Use of Electronic
Health Record Data in Clinical Investigations. This guidance extends the conversation
surrounding the utilization of EHR data for clinical evidence. It acknowledges the integration of
various sources, now seamlessly incorporated in real-time into a patient’s health record through
EDI, as discussed in Section 2.2.2 and depicted in Figure 1. Sponsors are now prompted to
consider utilizing investigative sites that maintain an ONC Health IT Certified EHR system,
increasing the confidence in the reliability and security of the data (FDA, 2018).

32
The final guidance issued to date because of funding provided by The Cures Act was the
revision to the 2002 The Least Burdensome Provisions of the FDA Modernization Act of 1997.
The initial 2002 issued guidance defined least burdensome as ‘a successful means of addressing
a premarket issue that involves the most appropriate investment of time, effort, and resources on
the part of industry and FDA’ (FDA, 2002). This guidance was revised and reissued in 2019 by
CDRH and the Center for Biologics Evaluation and Research (CBER) with added reference to
the acceptable use of real-world data during the review process (FDA, 2019a). At the urging of
the industry, the divisions also agreed to train regulatory reviewers on best practices for
implementing this guidance. The FDA has since initiated tracking of metrics on the use and
acceptance of least burdensome evidence to ensure time to market is optimized (FDA, 2019b).
The work conducted by the FDA to develop the guidance documents described above
illustrates its commitment towards accepting data outside of RCT for generating clinical
evidence. However, gaps remain. For example, how the FDA will determine the reliability of
EHR data was not explicitly stated in the 2017 guidance Use of Real-World Evidence to Support
Regulatory Decision-Making for Medical Devices. Additionally, past guidance documents like
the 2013 Guidance on Design Considerations for Pivotal Clinical Investigations for Medical
Devices still contain the same warnings concerning the use of EHR data for generating evidence.
Likewise, CDER recently issued a draft guidance titled Considerations for the Design and
Conduct of Externally Controlled Trials for Drug and Biological Products. While this guidance
document discusses CDER’s current thinking on clinical study designs and analyses when using
real-world data as evidence of effectiveness, it also includes concerns regarding risk of bias and
lack of validity when utilizing such data (FDA, 2023b; FDA, 2023c). To date, the FDA’s
expectations between pre-existing guidance documents, new guidance documents, and the

33
mandates of The Cures Act have not been sufficiently reconciled. Hence, industry may be
confused as to what data the agency would be willing to accept outside of RCTs.
2.3.2.2 Examples of EHR Use for Regulatory Submissions
To demonstrate the agency’s acceptance of using EHR data within submissions, CDRH
provided a summary of successful examples in a report titled Examples of Real-World Evidence
(RWE) Used in Medical Device Regulatory Decisions in 2021. The report summarizes ninety
submissions between 2012 and 2019 that utilized real-world data for clinical evidence and where
the FDA granted approval or clearance. Fifteen of the ninety examples provided in the report
used EHR data as the primary source of clinical evidence. Eight of the fifteen obtained 510(k)
clearance, four were granted de novo reclassification, and three received pre-market approvals
(PMAs). Half of the fifteen [five 510(k)s and two PMAs] were expansions or modifications to
an already approved indication for use (FDA, 2021c). Table 4 provides a summary of these
fifteen applications along with CDRH’s final decision.

34
Table 4: Medical Device Submissions Utilizing Data from Electronic Health Records
for Clinical Evidence
FDA
File Sponsor Device
Date of
submission
Date of
decision
Type of
submission Study design Decision
K171120
TransEnteri
x, Inc.
Senhance
Surgical
System 4/14/2017 10/13/2017
510(k)
clearance
Retrospective
medical chart
review
comparison to
published
literature
Substantially
equivalent
K172959
PeraHealth,
Inc.
PeraServer
and
PeraTrend
System 9/26/2017 5/1/2018
510(k)
clearance
Retrospective
medical chart
review
Substantially
equivalent
K180986
XableCath,
Inc.
XableCath
Support
Catheter
Product
Family 4/16/2018 5/15/2018
510(k)
clearance
Retrospective
medical chart
review
Substantially
equivalent
K180163
TransEnteri
x, Inc.
Senhance
Surgical
System 1/19/2018 5/25/2018
510(k)
clearance
Expansion
of IFU
Retrospective
medical chart
review
Substantially
equivalent
K181323
C. R. Bard,
Inc.
Atlas Gold
PTA
Dilatation
Catheter 5/18/2018 8/03/2018
510(k)
clearance
Expansion
of IFU
Retrospective
medical chart
review
Substantially
equivalent
K180111
Pursuit
Vascular,
Inc.
ClearGuard
HD
Antimicrobi
al Barrier
Cap 01/16/2018 5/25/2018
510(k)
clearance
Modificatio
n of IFU
Randomized
multi-arm
unblinded study
of routine
collected blood
surveillance data
Substantially
equivalent
K180894
Levita
Magnetics
Internationa
l Corp.
Levita
Magnetic
Surgical
System 4/5/2018 9/19/2018
510(k)
clearance
Modificatio
n of IFU
Retrospective
medical chart
review
Substantially
equivalent
K191173
Abbott
Vascular
Emboshield
NAV6
Embolic
Protection
System 5/1/2019 7/25/2019
510(k)
clearance
Modificatio
n of IFU
Retrospective
medical chart
review
Substantially
equivalent
DEN170
001
Vapotherm,
Inc.
Precision
Flow HVNI 1/3/2017 4/10/2018
De Novo
reclassificat
ion
Retrospective
medical chart
review Granted
DEN170
015
Hemospray
Endoscopic
Hemostat 3/9/2017 5/7/2018
De Novo
reclassificat
ion Registry data Granted
DEN170
064
RapidMedi
cal Ltd
Comaneci
Embolizatio
n Assist
Device 9/28/2017 4/24/2019
De Novo
reclassificat
ion
Retrospective
medical chart
review Granted
DEN170
073 Viz.Al, Inc ContaCT 9/29/2017 2/13/2018
De Novo
reclassificat
ion
Performance
Goal of
secondary use of
RWE Granted

35
FDA
File Sponsor Device
Date of
submission
Date of
decision
Type of
submission Study design Decision
P160022
Zoll
Medical
Corporation
X Series, R
Series, AED
Pro, AED 3
BLS
Professional
Defibrillator
s, ProPadz
Radiotransp
arent
Electrode,
etc. 7/7/2016 12/27/2017 PMA
Device
generated data +
routine clinical
data Approval
P140010/
S037
Medtronic
Vascular,
Inc.
IN.PACT
Admiral
PaclitaxelCoated
Percutaneou
s
Translumina
l
Angioplasty
(PTA)
Balloon
Catheter 12/08/2017 4/19/2018
PMA
Expansion
of IFU
Retrospective
analysis of subcohort Approval
P140017/
S005
Medtronic,
Inc
Melody
TPV 2/24/2017
PMA
Expansion
of IFU
Review of
medical records Approval
(FDA, 2021c, FDA, 2021d)
Most of the examples in Examples of Real-World Evidence (RWE) Used in Medical
Device Regulatory Decisions and those listed above were submitted after The Cures Act yet
before the issuance of the final guidance documents summarizing FDA’s expectations on the use
of real-world data. Given the limited number of examples provided as compared to the
numerous applications submitted to CDRH each year, it seems questionable that the medical
device industry is interested in shifting towards using EHR data on a more sustainable level.
Given the long-standing allowance of patient data for medical device evidence, implementing
this approach would have been a welcomed and simple shift. While there have been surveys
directed at the pharmaceutical industry’s perception of using EHR data, equivalent evaluations of
the medical device industry have not been carried out. Hence, the medical device industry’s
perception regarding the use of EHR data for clinical evidence has yet to be understood.

36
2.4 Industry Consideration of Electronic Health Record Use for Clinical Evidence
As early as 1997, when EHRs were in their infancy, healthcare representatives saw the
potential of utilizing data stored in this new platform for research. The Committee on Improving
the Patient Record within the IOM envisioned an interconnected healthcare system where
patients had their own digital record that would be readily accessible to both the clinician and
patient. This new healthcare information system could also make data accessible to researchers
transforming existing conventions (Institute of Medicine, 1997).
By 2012, the concept of using EHR data for medical product development was being
discussed. During the 2012 IOM workshop, “Envisioning a Transformed Clinical Trials
Enterprise in the US”, the workshop participants described how an EHR platform could be used
to improve the clinical trial process. Participants envisioned streamlining data collection that
would lead to cost reduction of clinical trials and generalized results that were more reflective of
real-world clinical use. While this vision was far from being achieved in 2012, the workshop
participants predicted that the goal of using EHR data for product development was certainly
within reach by 2020 (Institute of Medicine, 2012).
Regrettably, we are now in 2023 and despite some instances of market approvals aided
by EHR data, we have not fulfilled the potential envisioned two decades ago. The use of EHR
data for product development is still hindered by limitations associated with EHR systems,
including data quality, systems harmonization, and interoperability (Richesson et al., 2021).
Nevertheless, there has been consistent progress in these areas. Notably, the recent inclusion of
enhanced interoperability to the Health IT Certification has widened the accessibility of EHRs
for research purposes (US HHS, 2020).
It appears that the medical product industry’s use of EHR data for regulatory purposes
may mirror patterns seen with other innovative technologies. For example, the Hype Cycle

37
model, identified by the research and consulting firm Gartner, Inc., graphically depicts the
adoption pattern of a new technology or process (Figure 2). As diagrammed, an innovation
typically starts with a period of intense interest based on hyped expectations followed by a
period of disillusionment in the face of limitations and challenges. The use of the innovation
increases again as benefits are realized. As an organization experiences consistent real-world
benefits, adoption of the innovation is sustained (Blosch and Fenn, 2018).
Figure 2: Phases of the Gartner Hype Curve
(Blosch and Fenn, 2018)
The introduction of EHR platforms created high expectations among regulators and the
industry, who believed that the technology had the potential to revolutionize the development of
innovative medical products. Soon thereafter, however, it became clear that the data lacked the
robust quality controls expected by the regulators. Moreover, best practices for integrating this

38
new technology into already established processes for generating clinical evidence had not been
defined. Thus, the initial enthusiasm appeared to wane; yet industry and regulators continue to
explore how to overcome the challenges of using EHR data for product development.
2.4.1 Comparing RCT Data and EHR Data
Regulators consider data generated through RCTs as the gold standard for demonstrating
the efficacy of a therapeutic intervention. Through randomization, blinding, and control of
variables, RCTs reduce bias and enhance the ability to determine if a particular intervention
leads to a predetermined study outcome. Essentially, data from RCTs are gathered by
administering a predetermined treatment to a carefully selected group of individuals at a specific
time.
In contrast, data from EHRs are collected as part of routine clinical care to address the
patient’s healthcare needs. Treatment decisions are influenced by factors such as treatment
availability, cost, and insurance coverage. Follow-ups and tests are not strictly predetermined
but rather guided by the established standard of care for a patient’s specific medical condition. It
is common for a patient to undergo multiple treatments, consult with multiple healthcare
providers, and have multiple co-morbidities.
The difference between these two types of data is striking. Overall, RCTs are designed to
obtain data for regulatory use whereas EHRs are designed to obtain data for patient care and
payment (Sherman et al., 2017). Kim from the Catholic University of Korea College of
Medicine summarizes the characteristics of EHR and RCT data, revealing the nuanced
differences between the two (Table 5) (Kim, Lee and Kim (2018).

39
Table 5: Characteristics of EHRs and RCTs
Characteristic EHR RCT
Purpose Effectiveness Efficacy
Setting Real world Experimental
Follow up Current practice Designed
Treatment Variable pattern Fix pattern
Study group Heterogenous Homogenous
Attending physician Multiple healthcare providers Site investigator
Comparator Many routine comparators One comparator
Patient monitoring Per practice Per protocol
(Kim, Lee, and Kim, 2018)
2.4.2 Potential Benefits and Risks of EHR Use
Processes, analyses, and best practices are needed for the industry to use EHR data for
regulatory purposes. Although this shift may require an initial investment of time and resources,
it may also provide much-needed relief from the high cost of RCTs. Conducting RCTs has
become burdensome, often representing the most expensive and time-consuming component of
product development. According to Dr. DiMasi from Tufts University, the cost of a first-inhuman clinical trial increased 8.5% annually between 2000 and 2016, exceeding inflation. In
fact, the total cost of bringing a new drug to the market in 2016 was estimated at 2.6 billion USD
(DiMasi, Grabowski and Hansen, 2016). Thus, the industry has been looking for ways to reengineer its product development paradigm to be more efficient. In fact, the McKinsey Global
Institute suggests that the use of EHR data for clinical research and product development could
generate an estimated savings of 100 billion USD annually across the U.S. healthcare system
(Cattell, Chilukuri and Levy, 2013).
In addition to increasing efficiency, the use of EHR data for product development allows
the findings to be more generalizable. As mentioned previously, RCTs are conducted in very
controlled settings to mitigate bias and increase validity. For example, patients with advanced
age, multiple co-morbidities, and lower socioeconomic status are often underrepresented

40
(Schilsky, 2017; Unger, 2016). Hence, RCTs may limit the full understanding of the safety and
efficacy profile of a treatment. In fact, new cancer therapies have shown an increased incidence
of death from toxicities, treatment discontinuation, and adverse effects when administered within
a clinic setting as compared to a controlled RCT setting (Niraula et al., 2012). The use of EHR
data, therefore, allows the inclusion of a more diverse patient population so that the study
findings would be more translatable to real-world effectiveness (Califf et al., 2016).
Along with potential benefits of using EHR data as described above, there are also
potential challenges (Table 6). These challenges could lead to regulatory uncertainties that the
industry may view as too risky. As previously discussed, data in EHRs are often fragmented,
incomplete, and disorganized due to a lack of quality assurance and interoperability. The
potential for bias is also a concern since treatment is determined by insurance coverage, national
guidelines, and the health status of the patient. Because of these issues, new tools and skills are
required to manage and analyze the data (Bowrin et al., 2019). Sandi Siami, Senior Vice
President at the Medical Device Innovation Consortium (MDIC), recently described the
processes for integrating and validating data from various sources as a great challenge for both
the FDA and the industry (Slabodkin, 2021). Additionally, the industry lacks proven best
practices for working with EHRs in ways that would be akin to Good Clinical Practices.

41
Table 6: Strengths and Challenges of Data from EHRs Compared to Data from RCTs
Attribute Data from EHRs Data from RCTs
Strengths
Generalizable2, 4, 6
Patient centric2
Real-time5
Cost-saving7
Lack of bias1
Controlled2
Standardized2
Internal validity3
Quality assurance5
Challenges Missing data1, 5, 6
Potential bias1, 4, 5, 6
Lack of interoperability2, 6
Quality assurance1, 2
Standardization5
Skillset6
Lack of control8
Under-representative population2
Generalizability4
Cost5
Time5
Administrative burden6
Regulatory burden6
1
(Bowrin et al., 2019); 2
(Dhruva, Ross and Desai, 2018); 3
(Khozin, Blumenthal and Pazdur, 2017); 4
(Maissenhaelter, Woolmore and Schlag,
2018); 5
(Schilsky, 2017), 6
(Spitzer, Cannon and Serruys, 2018), 7
(Cattell, Chilukuri and Levy, 2013), 8
(Slabodkin, 2021)
To better understand the industry’s views on the use of real-world data including EHR for
generating clinical evidence, Tufts University conducted a survey of 200 pharmaceutical and
biotechnology companies. Of the 200 invited to participate, 30 responded. The survey identified
five key challenges perceived by industry: 1) inaccessibility of data, 2) uncertainty of regulators,
3) costs associated with acquiring and managing the data, 4) inability to clearly determine
treatment effect, and 5) quality and reliability of the data (Lamberti et al., 2018). Of the 30
respondents, only four indicated that their use of real-world data to support product labeling had
received regulatory acceptance. However, all respondents, except one, stated that their company
had a functional group dedicated to advancing the use of real-world data.
It is important to note that the study here did not include any medical device companies.
The medical device industry should already be familiar with using EHR data since regulations
governing medical devices have long included the use of patient health records. However,
information to date suggests that the medical device industry is also hesitant to use EHR data for
regulatory submissions as suggested from the fifteen submissions highlighted in the CDRH’s

42
report, Examples of Real-World Evidence (RWE) Used in Medical Device Regulatory Decisions.
These represent a small fraction of the more than 1000 premarket submissions the FDA receives
annually (FDA, 2021d). While the research by Tufts is valuable for advancing the use of EHR
data within the biopharmaceutical industry, comparable research is needed for the medical
device sector. Since the regulatory framework for medical devices is quite different from that of
biopharmaceutical products, the attitude towards the use of EHR data may also be different.
Understanding what factors influence the medical device industry to adopt the use of EHR data
could help advance the goals of the Cures Act and accelerate access to therapeutic innovations.
2.5 Research Framework
The adoption of EHR data for regulatory purposes by the medical device industry could
be considered like the adoption of the EHR technology by healthcare facilities in that both
require laws and regulations to foster adoption. In the case of healthcare facilities, the early
awareness of potential benefits was tempered by concerns about upfront cost and resources.
Benefits of EHRs were realized, however, after significant investment had been made to install
and operationalize the technology. Then, EHRs quickly became an integral part of the national
healthcare infrastructure. Likewise, the medical device industry is now aware of the potential
benefits of using EHR data for regulatory purposes, but is cautious, perhaps because of
uncertainties related to cost, resources, and regulatory acceptance. To better understand these
concerns, a more systematic examination of the industry’s views and experiences would be
helpful.
2.5.1 Frameworks of Technology Adoptions
Various models have been used to study the adoption of innovative information
technology (IT). Oliveira identified the more common models of adoption, which include

43
Technology Acceptance Model (TAM), Theory of Planned Behavior (TPB), Theory of
Acceptance and Use of Technology (TAUT), Diffusion Of Innovation (DOI), and Technology,
Organization, and Environment framework (TOE). While most of the models look at adoption
from the individual user level, TOE looks at adoption from the organizational level. Based on
the TOE model, the organization goes through technological, cultural, and structural changes as
new technology is adopted (Oliveira and Martins, 2011). The TOE model was revised by
Iacovou and colleagues while investigating the drivers of EDI adoption within the financial
business sector. The revised model looked at influencers of adoption which include 1) perceived
benefits from the technology, 2) the readiness of the organization to integrate the technology, and
3) pressures from external stakeholders to adopt the technology (Iacovou, Benbasat, and Dexter,
1995; Oliveira and Martins, 2011). Perceived benefits refer to an increase in competitive
advantage and a reduction in operational costs; organizational readiness refers to the
organization’s personnel maturity of skills as well as IT platform, and the financial commitment
to address gaps in skills and existing technological infrastructure; and external pressures refer to
the attitudes of business partners, customers, and government towards the adoption of the
technology. These attitudes can be in the form of competitive pressures and supportive
regulations (Figure 3) (Oliveira and Martins, 2011).

44
Figure 3: Iacovou Adoption Framework
(Iacovou, Benbasat and Dexter, 1995)
Through his research, Iacovou et al. concluded that perceived benefits of cost savings and
competitive advantage were the most influential drivers of adoption for EDI. In fact, the more
the organization’s management understood the benefits of using the technology, the more likely
financial support would be provided for implementation. Conversely, insufficient financial
resources and staffing capabilities were shown to hinder integration. Interestingly, the quality of
the technology had little influence on adoption and external stakeholders had only modest
influence (Iacovou, Benbasat, and Dexter, 1995).
2.5.2 The Framework of This Research
Based on the observation that the adoption of EHR data by the medical device industry is
influenced by similar factors that appear to have influenced EDI technology within the financial
industry, the TOE framework, as revised by Iacovou, provides the conceptual foundation for this
research. As discussed previously, the framework focuses on three influences of adoption at the

45
organizational level: perceived benefits, organizational readiness, and external pressures. This
research examined multiple drivers associated with each influence (Table 7).
Perceived benefits of EHR data included generalizability of results, reduction in cost, and
shortening of timeline as compared to RCTs as well as other possible benefits less noted in the
literature. Organizational readiness included management’s commitment to financial resources,
integration of new skillsets for managing and analyzing the data, and access to EHR data were
investigated.
The third influence, external pressures from multiple stakeholders included regulators
who provide their expectations through guidance documents and other forums, policy makers
and patients who demand faster, cheaper innovations, and consumer watchdogs who may be
concerned with the validity of the EHR evidence. Finally, pressures from other medical device
companies that might be achieving a competitive foothold due to earlier adoption of EHR data
were considered (Table 7). Further, to determine if additional drivers were evident beyond what
was investigated within the survey, open text responses were solicitated for the three areas.
Table 7: TOE Influences of Adoption and Drivers to be Explored
Influences of
Adoption
Drivers to be explored
Perceived benefits • Generalizability of results
• Reduction in cost
• Shortened of timeline
Organizational
readiness
• Commitment of financial resources
• Integration of new skillsets
• Procedural readiness
• Access to data partners
• Comprehensive interoperability
External pressures • Regulators
• Policy makers
• Patients
• Consumer watchdogs
• Data providers
• Industry competitors

46
The results of this research add to the discussion on how the medical device industry can
move toward adopting EHR data as part of their clinical evidence as well as how regulators may
support this change. Identifying the potential benefits along with preparing for the upfront costs
for establishing tools and processes and being aware of the external pressures experienced by the
industry may assist companies in planning for successful implementation. Overall, the findings
from this research are able to inform the medical device industry and regulators on best practices
for the use of EHR data, translating into accelerated patient access to new therapies as intended
by The Cures Act. Details on the development of the survey used to capture data for this
research based on the above information are discussed in Chapter 3.

47
Chapter 3. Methodology
3.1 Introduction
This research aimed to investigate the perspectives and firsthand experiences of
professionals working in the medical device industry regarding the utilization of EHR data for
generating clinical evidence. To gather the necessary data, a unique online survey, based on the
TOE framework as modified by Iacovou, was administered to leaders in the medical device
industry. The survey consisted of targeted questions pertaining to three crucial factors affecting
adoption: 1) perceived benefits associated with the technology, 2) organizational readiness to
integrate the technology and new processes, and 3) external stakeholder pressures for adopting
the new technology. The collected data underwent various descriptive analytical analyses to
identify patterns, trends, and correlations.
3.2 Survey Development
A survey instrument was developed to assess various aspects related to the use of EHR
data in generating clinical evidence. The survey included questions structured around three key
areas: 1) the perceived benefits of EHR utilization within the organization, 2) the organization’s
readiness to adopt EHR for this purpose, and 3) external pressures influencing the use of EHR
data. Questions were posed in multiple formats, including binary (yes/no), multiple choice,
Likert, and open text, to gather demographic information and contextual details guided by the
TOE framework. Prior to its deployment, a focus group was conducted with subject matter
experts from diverse backgrounds in the medical device industry and academia. This group
reviewed the survey to ensure its functionality, clarity, and conciseness.

48
3.2.1 Data Collected
The survey began by gathering demographic information from the respondents and their
organizations. This was followed by questions that aimed to explore the industry’s perspective
on the utilization of EHR data for generating clinical evidence. The survey was structured into 5
sections: 1) respondent and company demographics, 2) influential external pressures, 3)
perceived benefits for the organization, 4) organizational readiness, and 5) concluding thoughts
in an open text format (Table 8). Questions were designed to capture both structured and
unstructured data to facilitate a deeper understanding of the views and experiences of the
respondents.
Table 8: Number of Questions by Survey Categories
Survey Section Survey Category Number of Questions
1 Demographics 10
2 External Pressures 3
3 Perceived Benefits 2
4 Organizational Readiness 9
5 Open Text 2
Each section included questions that explored numerous drivers of influence such as
those summarized in Table 7, Section 2.5.2 For example, questions regarding perceived benefits
explored the impact of generalizability of results, reduction in cost, and shortening of a timeline
as compared to RCTs. Questions on organizational readiness explored management’s
commitment to financial resources, integration of new skill sets for managing and analyzing the
data, and procedural readiness for critical processes such as statistical controls and data privacy.
Additionally, the organization’s access to EHR data and the ability for the organization to obtain
the data through interoperability were investigated. External pressures from stakeholders such as

49
regulators, policymakers, patients, competitors, and other business sectors were explored. Data
were further analyzed to identify other drivers beyond those already stated.
3.2.2 Survey Verification
A focus group examined and tested the survey content, format, and sentence structures
before launching. The focus group was made up of six individuals, plus my thesis advisor, who
are well-knowledgeable in clinical research and EHR data. Three individuals from academia and
three from industry were petitioned to review the survey and provide input. A survey draft was
provided to everyone a week before the committee of six met with me to discuss their
recommendations. The focus group also provided a synopsis of my thesis to guide their review.
The survey was then revised based on input from the reviewers. This process verified the survey
ensuring sufficient information was captured to address the research topic.
3.2.3 Survey Deployment
The survey was deployed through the Qualtrics XM Platform as provided by University
of Southern California (USC) and offered by Qualtrics International, Inc. Qualtrics Platforms
are known for versatility in developing surveys with questions from a variety of styles, such as
yes/no answers, multiple-choice, matrices, ranked preference, and open text. Further, answers
remain anonymous, ensuring minimal impact on the company’s intellectual property,
confidentiality, and potential bias during analysis.
3.3 Population Selection
Views of the medical device industry professionals were sought from medical device
employees in mid-to-senior positions involved in clinical operations, regulatory strategy,
biostatistics, product development, and clinical quality. Respondents were identified through a
mix of sampling methods, such as targeted, random, and snowball via LinkedIn, professional

50
organizations, and through mutual acquaintances. Individuals associated with companies
included in the FDA’s Examples of Real-World Evidence (RWE) Used in Medical Device
Regulatory Decisions were also targeted to participate. Efforts were made to ensure individuals
were from medical device companies of various sizes, device classifications and indications,
stages of product development, and years of incorporation. Respondents could forward the
survey to other appropriate individuals to participate. The planned number of returned surveys
was 70.
3.3.1 Inclusion Criteria
Based on input from the focus group, inclusion criteria for respondents were as follows:
1. Respondent was an employee of a medical device company, and
2. The company markets or plans to market their product in the US, and
3. The company markets or develops Class II and/or Class III products.
The survey was developed with questions to determine if the respondent met the above
criteria. If the respondent indicated they did not meet one of the criteria, they skipped to the end
of the survey and received a thank you note. In addition to the above criteria, surveys were
further filtered based on the respondents’ and organizations’ participation in and familiarity with
activities or discussions with using EHR data relevant to the research topic. If respondents or
their organization were not familiar with or currently not involved in discussions with using EHR
data, their responses were excluded. Surveys that met the following criteria were included in the
final analyzed cohort:
1. Respondent had to indicate they were involved in discussions or activities relevant to the
research topic, and

51
2. Respondents had to indicate the company they were an employee of was at the stage of
discussing the use of EHR data at a minimum.
3.3.2 Distribution of Survey
The survey was distributed for three and a half months to a wide variety of medical
device professionals at different levels of authority within the medical device industry. An initial
request was sent via LinkedIn® or email asking for their participation. Upon a positive response,
a link to the survey was sent to the participants via their personal email or LinkedIn messenger,
based on their stated preference. Results were maintained within Qualtrics and downloaded in a
CSV format for analysis by the author.
3.4 Survey Analysis
Upon closure of the survey, demographic data were summarized and filtered per the
inclusion criteria to identify the final cohort. The TOE cohort was analyzed using various
descriptive analytical methods to identify patterns, trends, and correlations and displayed in a
variety of ways, including descriptive text, tables, charts, and figures. Open text and comment
fields were qualitatively analyzed to identify themes, patterns, and common elements.

52
Chapter 4. Results
4.1 Survey Participation
The survey was published for use on November 1, 2022, and closed on February 16,
2023. During this time, 436 professionals were contacted and invited to participate in the survey.
Out of the 436, 178 professionals received the survey link; 95 individuals initiated the survey.
Of the 95 who initiated the survey, 20% (19/95) did not meet the inclusion criteria defined in
Section 3.3.1 and were eliminated. Respondents were further filtered from the TOE analysis if
they indicated that they were not involved in discussions or activities using EHR data or if their
organization had not begun discussing using EHR data to generate clinical evidence. This
caused 31% (30/95) of surveys to be filtered out, resulting 46/95 (48%) surveys included in the
TOE analysis cohort. Table 9 and Figure 4 provide summaries of the distribution.
Table 9: Respondent Distribution and Final Cohort
Filtering process Number of respondents Percent
Requested 436 --
Respondents who initiated
the survey
95 --
Surveys not meeting
inclusion criteria:
49/95 52%
Demographic exclusion 19/95 20%
Insufficient data 6/95 6%
Does not manufacture
medical devices,
IVDs, or SaMD
5/95 5%
Does not market in the
US
4/95 4%
Does not market class
II or III
4/95 4%
Familiarity exclusion 30/95 31%

53
Filtering process Number of respondents Percent
Respondent not
involved in
discussions/activities
17/95 18%
Company has not
begun
discussions/activities
13/95 14%
TOE analysis cohort 46/95 48%
Figure 4: Survey Cohorts and Exclusion
Based on the results of the final cohort of 46 surveys, descriptive statistics summarizing the
demographics of respondents and the TOE constructs are presented below.
436 individuals solicited
95 surveys initiated
76 surveys meeting
demographic criteria
46 surveys
meeting
familiarity
criteria

54
4.2 Demographics
The survey obtained demographic information regarding respondents, the organization
the respondents represented, and the familiarity of the respondents and their organizations with
the use of EHR data.
4.2.1 Respondent Demographics
The respondents were asked to indicate their current level of authority within their
organization. One respondent did not provide an answer. Out of the 45 respondents who
provided an answer, the most common level of authority was Director or Senior Director,
accounting for 31% (14/45). The prevalence of other levels varied, with percentages ranging
from 26% (12/46) for Vice President and higher, 22% (10/46) for Manager, and 20% (9/46) for
individual contributor (Figure 5).
Figure 5: Distribution of Self-identified Level of Authority (n=45)
Respondents were also asked about their functional role within the organization.
Respondents were allowed to select “all that apply”. The clinical function (i.e., clinical affairs,
27% (12)
31% (14)
22% (10)
20% (9)
(Q: What is your current position in your organization?)
Vice President or higher
Director/Sr. Director
Manager/Sr. Manager
Individual contributor

55
data management, and biostatistics) was identified most frequently (26/46), followed by
regulatory affairs (11/46), R&D/Engineering (5/46), Quality (4/46), and Medical Affairs (2/46).
Operations/Product Support, Executive Leadership, Publications, and Medical Evidence were
each selected once (Figure 6).
Figure 6: Distribution of Self-identified Functions (n = 46)
4.2.2 Company Demographics
The size of the organization was gauged by the number of employees. Choices included
less than 500 employees, between 501 and 5000 employees, or greater than 5000 employees.
More than half of respondents (59%, 27/46) selected more than 5000 employees, followed by
24% (11/46) of respondents choosing less than 500 employees, and 17% (8/46) of respondents
choosing between 501 and 5000 (Figure 7).
26
11
5
4
2
1
1
1
1
Clinical Team
Regulatory Affairs
R&D/Engineering
Quality
Medical Affairs
Operations/Product Support
Executive Leadership
Publications
Medical Evidence
(Q: With which functional area of your organization do you work?
Select all that apply.)

56
Figure 7: Distribution of Organizational Size (n=46)
4.2.3 Familiarity with EHR
To assess the level of familiarity with EHR, the respondents were asked to provide the
number of years their organizations have been investigating or engaging in the use of EHR data
to generate clinical evidence. Almost half selected three to ten years (48%, 22/46), while 35%
(16/46) selected less than three years. Only 9% (4/46) indicated their organization had been
engaged with EHR data for greater than 10 years. One respondent reported their organization is
not considering the use of EHR data and three did not know how long the organization had been
investigating the use of EHR data (Figure 8).
59% (27) 17% (8)
24% (11)
(Q: What is the size of your organization?)
Greater than 5000
employees
Between 501 and 5000
employees
Less than 500
employees

57
Figure 8: Distribution of Length of Time Engaged in the Use of EHR Data (n=46)
The respondents were asked about their organization’s stage of adoption for the use of
EHR data. Almost half (46%, 21/46) indicated that their organizations were actively using EHR
data as the primary source of clinical evidence in submissions. The rest were equally divided
between experimenting with the use of EHR data (28%, 13/46) and starting discussions or
activities (26%, 12/46) (Figure 9). Cross-tabulation between the length of time the organization
had been investigating or engaged in using EHR data and the stage of adoption was conducted.
Due to the limited sample sizes, results were difficult to interpret.
9%
(4)
48%
(22)
35%
(16)
2%
(1)
6%
(3)
Greater than 10
years
Between 3 and
10 years
Less than 3
years
Not considering
the use of EHR
data
Don’t know
(Q: In your view, how long has your organization been investigating/engaging in
the use of EHR data to generate clinical evidence?)

58
Figure 9: Distribution of Organizational Stage of Adoption (n=46)
To further examine the use of EHR data for clinical evidence, respondents were asked to
what extent their organizations were using EHR data as the primary source of clinical evidence
for 510(k)s and PMAs over the past 12 months. Regarding 510(k) submissions, 11% (5/44)
thought that all their 510(k) submissions used EHR data as the primary source of clinical
evidence, while 7% (3/44) thought more than half and 4% (2/44) thought about half of their
510(k) submissions used EHR data. The remaining 34 respondents were almost equally split
between less than half [25% (11/44)], none [27% (12/44)], and did not know [25% (11/44)]
(Figure 10). Two of the 46 respondents did not provide answers for 510(k) submissions.
Results were similar regarding the use of EHR data for PMA submissions, with 9%
(4/45) of respondents indicating all the organization’s PMA submissions and 2% (1/45)
indicating more than half utilized EHR data. The remaining 40 respondents were equally
divided between less than half [31% (14/45), none [29% (13/45)], and don’t know [29%
(13/45)]. No respondent selected about half (Figure 10). One respondent did not provide an
answer for PMA submissions. In summary, less than 25% of organizations used EHR data to
support half or more of their 510(k) or PMA submissions.
46% (21)
28% (13)
26% (12)
Actively using EHR data
Experimenting with the use of EHR
data
Starting discussions about the use of
EHR data
(Q: In your view, at what stage is your organization with respect to adopting
EHR data for generating clinical evidence?)

59
Figure 10: Proportion of FDA Submissions Over the Past 12 Months that Used EHR
Data as the Primary Source of Evidence
The respondents' proficiency with using EHR data within their organization was
investigated by asking how involved they are in discussions or activities using EHR data. About
half of the respondents indicated they were somewhat involved (46%, 21/46), 30% (14/46)
indicated they were very involved, and 24% (11/46) indicated they were minimally involved.
11% (5)
7% (3)
4% (2)
25% (11)
27% (11)
25% (12)
9% (4)
2% (1)
0%
31% (14)
29% (13)
29% (13)
All
More than half
About half
Less than half
None
Don’t know
(Q: Looking back over the past 12 months, what proportion of 501(k)s
and PMAs submitted to the FDA used EHR data as the primary
endpoint)
510(k) (n = 44) PMA (n = 45)

60
Figure 11: Distribution of Respondents' Involvement in Discussions or Activities about
the Use of EHR Data (n=46)
4.3 TOE Analyses
The survey explored each of the TOE factors discussed in Chapter 3 through quantitative
and qualitative analyses. Structured data responses are analyzed and presented using figures and
tables, while open-text answers are provided in list format. The summary of results is confined
to the 46 respondents who met the inclusion criteria and were familiar with using EHR data to
generate clinical evidence. The discussion of the open-text answers is provided in Chapter 5.
4.3.1 Perceived Benefits
To investigate perceived advantages of using EHR data, respondents were asked if a
possible reduction in cost, time, or workforce, streamlining of analyses, or an improvement of
generalizability were benefits with using EHR data as compared to RCT data for generating
clinical evidence. Respondents indicated they perceived the use of EHR data was more
beneficial than using RCT data in the areas of cost, time, generalizability of data, and workforce.
There was mixed agreement if there was a gained benefit for streamlining analyses with EHR
30% (14)
46% (21)
24% (11)
Very involved
Somewhat
involved
Minimally
involved
(Q: Which of the following statements best describes your level of
involvement in using EHR data for generating clinical evidence
in your organization?)

61
data (Figure 12). Two respondents did not respond. An additional respondent did not provide an
answer to a reduction in the workforce.
Figure 12: Possible Advantages of Using EHR Data for Generating Clinical Evidence as
Compared to Using Data from Randomized controlled Trials (n=46)
24
21
7
8
16
14
15
20
9
24
4
6
8
12
4
1 1
5
9
0
1 1
2
5
0 0 0
1 1
0
Reduction in
cost
Reduction in
time to
submission
Reduction in
workforce
Streamlining
statistical
analyses
Real-world
generalizability
of the data
(Q: In your view, using EHR data for generating clinical evidence provides the
following advantages when compared to using data from randomized controlled
trials.
Strongly agree Somewhat agree
Neither agree nor disagree Somewhat disagree
Strongly disagree Don't know

62
The weighted means of the responses were calculated to fine-tune the ranking of the
benefits. The analysis was ranked accordingly: strongly agree received a value of 1, somewhat
agree received a value of 2, neither agree nor disagree received a value of 3, somewhat disagree
received a value of 4, and strongly disagree received a value of 5. The number of responses
within each advantage type and rank was multiplied by the rank value. Values were added for
each type then divided by the number of responses within that type. The perceived greatest
advantage for using EHR data was in the reduction of cost (1.7), generalizability of the data
(1.7), and the shortening of time to submission (1.8). Reduced workforce (2.4) and streamlining
analyses (2.9) were identified as least beneficial (Table 10).
Table 10: Weighted Means of Respondents’ Choosing Advantages with Using EHR
Data for Generating Clinical Evidence as Compared to Using Data from
randomized Controlled Trials
Strongly
agree
(1)
Somewhat
agree
(2)
Neither
(3)
Somewhat
disagree
(4)
Strongly
disagree
(5)
n Weighted
Mean
Reduction
in cost
24 14 4 1 1 44 1.7
Realworld
generaliza
bility
16 24 4 0 0 44 1.7
Reduction
in time
21 15 6 1 1 44 1.8
Reduction
in
workforce
7 20 8 5 2 42 2.4
Streamlini
ng
statistical
analyses
8 9 12 9 5 43 2.9

63
Respondents with experience utilizing EHR data as clinical evidence within a submission
[i.e., 510(k) or PMA] were further probed to determine if benefits had been realized. Responses
were parsed for only those respondents who indicated their organizations had utilized EHR data
for clinical evidence for 510(k) or PMA submissions. The total number of respondents was 24,
which included 21 respondents using EHR data for 510(k) submissions and 19 using EHR data
for PMA submissions. The results showed that using EHR data for 510(k) or PMA reduced cost,
shortened timelines, improved generalizability of the data (Figure 13).
Weighted means were calculated for the 24 EHR experienced respondents per the same
formula as defined above (Table 11). Results showed shortening of timelines to be the greatest
benefit (1.6) followed by reduction in cost (1.7) and improvement in generalizable data (1.8).
Reduction in workforce and streamlining of statistical analyses were thought to be least
beneficial (2.5 and 2.9, respectively).

64
Figure 13: Advantages with Using EHR Data for Generating Clinical Evidence as
Compared to Using Data from Randomized Controlled Trials as Determined
by Respondents from Organizations that Utilized EHR Data to Support a
Submission (n = 24)
11
13
3
4
7
8
6
10
5
13
2 2
4
5
2
1 1
4 4
0 0 0
1
4
0
Reduction in
cost
Reduction in
time
Reduction in
work force
Streamlining
analyses
Generalizability
of data
Strongly agree Somewhat agree Neither
Somewhat disagree Strongly disagree

65
Table 11: Weighted Means of Experienced Respondents’ Choosing Advantages with
Using EHR Data for Generating Clinical Evidence as Compared to Using
Data from Randomized Controlled Trials
Strongly
agree
(1)
Somewhat
agree
(2)
Neither
(3)
Somewhat
disagree
(4)
Strongly
disagree
(5) n
Weighted
Mean
Reduction in
time 13 6 2 1 0 22 1.6
Reduction in
cost 11 8 2 1 0 22 1.7
Generalizabi
lity of data 7 13 2 0 0 22 1.8
Reduction in
work force 3 10 4 4 1 22 2.5
Streamlining
analyses 4 5 5 4 4 22 2.9
Respondents were asked for additional benefits beyond what was included in the
question. Two respondents provided comments. One suggested there was an “avoidance of
human error due to the data being encrypted”. Another mentioned “most of the
considerations/advantages speak to steady-state use but don't account for startup overhead for
orgs new to using EHR”.
4.3.2 Organizational Readiness
The readiness of the overall system to implement the use of EHR data was explored.
According to the TOE framework, readiness refers to the capability of initiating and integrating
the new process. For this research, organizational readiness was investigated by determining the
perceived quality and accessibility of the data in addition to the adequacy of an organization’s
internal infrastructure and management support. Results of each area are presented below.

66
4.3.2.1 Quality of Data
Respondents were asked if they agreed EHR data were sufficient to support a primary
endpoint for a submission. Four respondents did not provide an answer. Results indicated a lack
of confidence in the robustness of the data. Almost half of the respondents chose Possibly (43%,
18/42), while Probably (14%, 6/42) and Definitely were the least prevalent (12%, 5/42) (Figure
14). A third of the respondents chose Definitely Not (29%, 12/42).
Figure 14: Perception of the Sufficiency of EHR Data for Supporting a Primary
Endpoint (n=42)
Responses from respondents from organizations with experience utilizing EHR data as
clinical evidence within a submission were analyzed separately to determine their views of EHR
data sufficiency. Results were similar to those from the total cohort as displayed in Figure 14.
When respondents were asked to rank reasons for data insufficiencies from most
concerning (ranked as 1) to least concerning (ranked as 7), lack of available data specific to the
12% (5)
14% (6)
43% (18)
29% (12)
2%
(1)
(Q: Do you think EHR data is sufficient, in its current state, to generate
clinical evidence that can support a primary endpoint within a regulatory
submission?
Definitely
Probably
Possibly
Definitely not
Don't know

67
endpoint and missing data were most often ranked as most concerning. Lack of access to
unstructured data, inherent bias, lack of the unique device identification (UDI), and data cleaning
were ranked as least concerning with respondents having mixed reactions to lack of data
consistency. Five respondents did not provide rankings (Figure 15).
Weighted means were determined by assigning values of one through seven to each rank
level, with one being the most concerning and seven being the least concerning. The same
method of calculations was utilized as described above in Section 4.3.1. The results of the
weighted means (Table 12) are associated with the results displayed in Figure 15. The "lack of
available data specific to the end point” was rated as the most concerning (2.4) followed closely
by “missing data” (2.8) while “inherent bias” was rated least concerning (5.0). Other factors
were identified as neither most nor least concerning in the following order: inconsistency of data
across datasets (3.1), the lack of access to unstructured data (4.2), the lack of UDI (4.6), and the
need for data cleaning (4.9).

68
Figure 15: Ranking of Concerns when Using EHR Data (n=41)
14
12
5
1
5
2
2
11
7
10
1
5
3
4
6
9
12
3
1
5
5
5
8
6
9
5
5
3
4
1
5
9
7
9
6
0
2
3
11
11
7
7
1
2
0
7
7
10
14
Lack of available data specific to the end
points
Missing data
Inconsistency of data across datasets
Lack of easy access to unstructured data
Lack of Unique Device Identification
(UDI)
The need for data cleaning prior to
analysis
Inherent bias within the data
(Q: Please rank the following from most concerning (1) to least concerning
(7) when using EHR data for regulatory purposes?)
1 2 3 4 5 6 7 Rank

69
Table 12: Weighted Means of Ranking of Concerns when Using EHR Data
Rank
(1 = most concerning; 7 = least concerning)
1 2 3 4 5 6 7 n
Weighted
Mean
Lack of available
data specific to
the end points 14 11 6 5 4 0 1 41 2.4
Missing data 12 7 9 8 1 2 2 41 2.8
Inconsistency of
data across
datasets 5 10 12 6 5 3 0 41 3.1
Lack of easy
access to
unstructured data 1 1 3 9 9 11 7 41 4.2
Lack of Unique
Device
Identification
(UDI) 5 5 1 5 7 11 7 41 4.6
The need for
data cleaning
prior to analysis 2 3 5 5 9 7 10 41 4.9
Inherent bias
within the data 2 4 5 3 6 7 14 41 5
Respondents were asked to provide additional concerns regarding EHR data. Ten respondents
provided comments, all being consistent with those included in the survey question. Comments
are shown below:
Adequate longitudinal data.
Closed versus open systems--potential to complicate longitudinal follow-up especially in
mixed IP-OP settings and longer term events or surrogate outcomes.
Inability to link different EHR systems to maximize the available pool of data - big data
analysis is currently not possible.

70
My observation of EHRs from medical facilities indicates frequent mistakes and
shortcomings, possibly due to a focus on making sure the record includes everything
needed for insurance/billing as opposed to personalizing the record for a specific
patient.
Data integrity and software verification for EHR software.
Data security, protection, and cybersecurity risks which are to be accounted for when
using electronic data. Data ownership & boundaries of use concerns largely exist
amongst institutions. HIPAA concerns almost always exist, and demonstration of being
SOC2 & HIPAA compliant and potentially certified are issues which normally arise
when considering EHR data use (as well as ownership of that data and if/how it can be
subsequently used).
Technically difficulty for using the data.
Many clinics and hospitals do not possess the personnel dedicated to the extraction of
the needed data.
Privacy challenges.
Provenance.
Data sufficiency was further explored by asking respondents if they thought the use of
EHR data would eventually lead to the reduction of randomized controlled trials. A third of
respondents chose Possibly [35% (16/45)] followed by a quarter of respondents chose Probably
[24% (11/45) and Definitely Not [26% (12/45)]. A small number of respondents, 9% (4/45),
chose Definitely (Figure 16). One respondent did not provide an answer.

71
Figure 16: The Perceived Opportunity to Reduce the Number of Randomized Trials
(n=45)
(Q: Do you think the use of EHR data will reduce the number of randomized controlled trials?)
4.3.2.2 Infrastructure Sufficiency
The new approach of utilizing EHR data to support clinical evidence is likely to require
new skills in the workforce. The respondents were asked which functional areas would need
new skillsets to advance the use of EHR data (Figure 17). The results showed that clinical affairs
received most responses (30), followed closely by data management (29), regulatory affairs (27),
and biostatistics (26). Medical affairs (14) and program management (10) received fewer
responses while legal was selected twice.

72
Figure 17: Functional Areas Needing New Skillsets to Advance the Use of EHR Data
(Q: Which functional areas in your organization need new skillsets and/or processes to advance the use of EHR
data? Select all that apply and elaborate as applicable.)
Respondents were solicited for comments. Of the 46 respondents, 27 provided comments.
I don't think our organization requires new skillsets because we have been using
EHR data for a long time.
All functional areas as this would be something that many are not well versed on
throughout the organization so harmonization with processes and understanding of
the advantages and existing gaps would be beneficial for everyone.
I think most functions are not familiar with the limitations or opportunities of EHR
data. There is a learning curve of what is appropriate use.
People will need to understand the capabilities, and limitations, in order to be
confident using EHR data vs. entirely new data from targeted trials.
The use of EHR to us was totally new, so we have all been learning along the way.
There were many unforeseen challenges that we had to work to overcome. We will
continue to learn through the FDA review which will hopefully better prepare us for
next time.
RWE is limited but is a necessary aspect of submissions when it comes to patient
populations. This information is needed for the regulatory strategy. In streamlining
both clinical and EHR, there are gaps that can be better addressed by both the
healthcare and industry settings.
30
29
27
26
14
10
2
Clinical Affairs
Data Management/Science
Regulatory Affairs
Statistics/Biostatistics
Medical Affairs
Program Management
Legal/privacy

73
There is lot of refinement that is needed in strategy when a RWE project is
undertaken and listing out risks and plans to address them as scientifically as
possible without affecting data integrity is important throughout.
It is important to understand that data generation is always a cross-functional
activity with CA leading and other teams contributing to the data generation. The
entire team needs to understand the inherit limitations to the data and be in
communication with the reg bodies to make sure that data is (1) acceptable and (2)
developed in a way that will be acceptable. Each project will likely require new
ways of analysis and interpretation. SubQ, SubQ, SubQ!!
Current data monitoring, control, and use skills are necessary. SOC2 & HIPAA
compliance and certification requirements are generally needed for liability
exposure reduction. Data control, security controls, threat modeling controls, and
cybersecurity controls are areas which must be duly considered - which many
organizations are ill-equipped to address. Normally, this includes entrepreneurial,
small start-ups, and even mid-cap development organizations. Larger, more
sophisticated, deep-pockets organizations may be in better positions to balance &
address these requirements via multifunctional expertise.
RA - FDA regulations monitoring.
Clinical Affairs - understand the data structure compared to traditional data.
Data management - need to maintain database (HIPPA) and/or creating different
data management system.
RA: Develop understanding on how to position EHR and RWD into more traditional
submissions; how to source especially OUS data.
Stats: Methods development and validation, different processes for generating TLFs
etc.
CA: Understanding how EHR and RWD fit into existing portfolio of studies/evidence
types; how to assess value and use cases compared to traditional studies.
DM/DS: Use of big data, cloud, etc., to better handle (as trad clinical data are
smaller and use different systems).
MedAff: How to speak peer-to-peer and educate about value of the EHR/RWDdriven evidence.
May require additional skills or information to interface with various systems.
Automated processes for data transfer, identification of appropriate datasets,
negotiation with regulators.
Working with data outside of well structured/defined clinical study. Different
view/perspective on managing data.
Data troubleshooting.
Ability to have deep understanding of varying statistical methods to analyze data
using best case/worst case assumptions and the need to explain that to regulators.
There are no SOPs established and stats needs to know how to organize the
data. There are a lot of assumptions required in most cases.

74
More team members with advanced modeling techniques.
Statistical, EU MDR, FDA, PMDA.
Protecting data integrity has only become the focus of regulators over the last
decade. Data analytics is still maturing as an area of resource need. We still
struggle to find resources that have the knowledge and integrity to meet the
demands.
MD leading clinical strategy with strong background in regulatory requirements and
statistical considerations
Respondents were asked which of the following activities were more difficult when
analyzing EHR data compared to RCT data: identifying appropriate analyses, ensuring data were
regulatory grade, integrating the data into the organization’s processes, incorporating
interoperability into the organization’s infrastructure, and cleaning the data. Respondents
strongly agreed or somewhat agreed that all factors were more challenging when using EHR data
(Figure 18). Analysis of data through weighted means revealed meeting regulatory grade quality
(1.9) to be the most challenging, followed by incorporating interoperability (2.2) and cleaning
the data (2.2). Identifying the appropriate analyses (2.4) and incorporating the process into the
organization’s procedures (2.5) were thought to be the least challenging (Table 13).

75
Figure 18: Respondent’s Agreement with Statements Regarding Analyzing EHR Data
as Compared to Analyzing Data from Randomized Clinical Trials
13
7
10
7 7
21
18
16 16
15
3
12
11
10
7
3
2
3
4
6
0 0 0
1
2 2
3
2
4 4
Meeting
regulatory
grade quality is
more difficult
Incorporating
interoperability
into the
organization's
infrastructure is
more
challenging
Reformatting
and cleaning
the data
requires more
time
Identifying
appropriate
analyses takes
more time
Incorporating
into the
organization's
already existing
clinical
procedures is
more
challenging.
(Q: Do you agree with the following statements regarding analyzing EHR data
as compared to analyzing data from randomized clinical trials?)
Strongly agree Somewhat agree Neither
Somewhat disagree Strongly disagree Don't know

76
Table 13: Weighted Means of Respondent’s Agreement with Statements Regarding
Analyzing EHR Data as Compared to Analyzing Data from Randomized
Clinical Trial
Strongly
agree
(1)
Somewhat
agree
(2)
Neither
(3)
Somewhat
disagree
(4)
Strongly
disagree
(5) n
Weighted
mean
Meeting
regulatory grade
quality is more
difficult 13 21 3 3 0 40 1.9
Incorporating
interoperability
into the
organization's
infrastructure is
more
challenging 7 18 12 2 0 39 2.2
Reformatting
and cleaning the
data requires
more time 10 16 11 3 0 40 2.2
Identifying
appropriate
analyses takes
more time 7 16 10 4 1 38 2.4
Incorporating
into the
organization's
already existing
clinical
procedures is
more
challenging. 7 15 7 6 2 37 2.5
Respondents were solicited for additional challenges with analyzing EHR data. Three comments
were provided.
Validation of methods is more challenging.
It is more difficult to identify outliers or gaps in patient outcomes when designing trials
or algorithms in research and development.
Organizational change takes time, processes won’t change overnight so long as there
are human factors involved.

77
Respondents were probed regarding their organization’s accessibility to EHR data. Twothirds of respondents chose Somewhat Accessible (66%, 27/41), while 14% (6/41) chose Very
Accessible, 10% (4/41) chose Not Accessible and Didn’t Know each (Figure 19). Five
respondents did not provide an answer. Further analysis was conducted with the cohort of those
respondents from organizations that have used EHR data to support submissions (n = 24). While
the sample size was small, the cross-tabulated results were similar to those depicted in Figure 19
and did not reveal any new findings.
Figure 19: Perceived Organization’s Accessibility to EHR Data (n=46)
Comments were solicited regarding access to EHR data. Twenty-six respondents
provided comments.
You have to know the right individuals within a given system, for data access.
Institutional clearance for use is required. Also, if using multiple sites for sources of
data, not all institutions use the same data systems, so firewalls can be a logistical
problem. Data ownership, data security, threat modeling protocols, and cybersecurity

78
concerns are all difficult issues (and more) which must be dealt with when using EHR
data.
Our product is integrated into the EMR.
We built a vitals sign device and ECG devices and other diagnostic products
communicating with EMR such as Epic and Allscipts.
If the data is available, we would be able to access it.
We have access to EHR data and work on getting more, but it is challenging to gain
access from the data that didn't originate at our institution.
Because of the partnerships and investment that we have made as an organization
with healthcare institutions.
Appropriate contracts with healthcare institutions can provide access.
We always collaborate with investigators at the owning institution.
Limited use essentially through partnerships for retrospective studies.
EHR data is used for two purposes:
1. Generate RWE for our devices to get approvals of new indications or for
continuing to understand the safety and efficacy profile of the device.
2. For ensuring data integrity during routine monitoring of clinical studies.
Both of these rely on the investigational sites providing access to the EHR
If needed, we can work with partners who possess EHR, either as industry-sponsored
studies or investigator-initiated trials.
While we don't currently have EHR, we do have national chargemaster RWD that has
been digested into some exploratory tools accessible to cross-functional (Clinical,
RnD, Stats, Commercial).
It is accessible through institutions we have partnerships with but a limited amount.
We don't typically have access to other health records for the patient and just
partnerships with specific institutions for data specifically on our products when they
have surgeries with the products from procedure to discharge so we are limited
compared to healthcare companies that also own insurance companies. For example,
CVS or United Health group also own some medical benefit companies so they have
access to data for their patients for research.
There are obstacles in obtaining partnerships and available data can be limiting.
It really depends on the owner. In the US it's much more readily available; in the EU
and other countries due to GDPR, it's under lock and key.
Privacy regulations around the world limit access.
Difficult with new privacy laws and the fact that every EHR is different.
Has been accessible only when as part of data collection in ongoing clinical trials.
Is accessible to those that have the need to know.

79
Depends on needs.
It depends on the consents and how much we can deidentify. Completely privacy and
HIPAA limitations.
It is accessible but there is a lot of work and time that goes into harnessing that data.
So, although it is there, it is not easily obtained for use in a submission.
Some data is available for our indication but not a lot.
We are working with an outside vendor.
We were able to get anonymous patient data from some doctors years ago.
Required detailed process, contracting, privacy review and some data is still not
available.
Diving deeper into accessibility, factors impacting access to EHR were explored.
Respondents were asked if they agreed if the transfer of data between the data vendor and the
organization, the cost of EHR data, data privacy, contract negotiations, and EHR system
compatibilities were more difficult when using EHR data as compared to working with data from
clinical trials. Respondents most often indicated the transfer of EHR data between the data
vendor and the organization, contract negotiations, and system compatibility were more difficult
when using EHR data. Respondents had mixed responses about data privacy and did not indicate
there was a difference in cost between obtaining EHR data or clinical study data (Figure 20).
When looking at weighted means, results indicated the transfer of data between the vendor and
the organization (1.9) as being the most challenging, followed by the compatibility of systems
(2.3) and contract negotiations (2.4). Maintaining data privacy was identified with a weighted
mean of 2.7. The cost of obtaining data was determined to be the least challenging by
respondents (3.5) (Table 14). The calculation for weighted mean did not include the option
Don’t Know.

80
Figure 20: Factors Impacting Access to Data as a Comparison between EHR Data and
Data from Randomized Clinical Trials (n=46)
(Qst: Do you agree with the following statements regarding access to EHR data as compared to access to data from
randomized clinical trials?)
15
8
7
7
20
16
16
13
3
9
9
9
3
5
4
7
0
0
2
4
Transfer of EHR data is more difficult
Compatibility of systems is more
inefficient
Contract negotiation with data vendor
requires more time
Ensuring data privacy is more difficult
Strongly agree Somewhat agree Neither
Somewhat disagree Strongly disagree

81
Table 14: Weighted Means of Respondent’s Agreement with Statements Regarding
Access to EHR Data as Compared to Data from Randomized Clinical Trial
Strongly
agree
(1)
Somewhat
agree
(2)
Neither
(3)
Somewhat
disagree
(4)
Strongly
disagree
(5) n
Weighted
Mean
Transfer of
EHR data is
more difficult 15 20 3 3 0 41 1.9
Compatibility
of systems is
more
inefficient 8 16 9 5 0 38 2.3
Contract
negotiation
with data
vendor
requires more
time 7 16 9 4 2 38 2.4
Ensuring data
privacy is
more difficult 7 13 9 7 4 40 2.7
It is more
costly to
obtain EHR
data 1 4 17 12 7 41 3.5
Comments were solicited for additional factors. Only one respondent stated, “We have only
begun, only time will tell how acceptance in sharing information is taken”.

82
4.3.2.3 Management Commitment
To understand the commitment towards the use of EHR data by the management,
respondents were asked if the goals and objectives of their organizations included using EHR
data for generating clinical evidence. Over two-thirds (69%, 29/42) of respondents indicated
“yes” while 14% (6/42) indicated “no” and 17% (7/42) didn’t know (Figure 21). Four
respondents did not respond.
Figure 21: Prevalence of Organizations with Goals and Objectives Including the Use of
EHR Data (n=42)
Along the same theme, respondents were asked how often they heard senior executives
mention using EHR data for generating clinical evidence. Of the 42 respondents, 40% (17/42)
chose Rarely, 36% (15/42) chose Somewhat, 14% (6/42) chose Very Frequently and 10% (4/42)
chose Never (Figure 22). Four respondents did not provide an answer.
69% (29)
14% (6)
17% (7)
(Q: Your organization's goals and objectives for product development include the
use of EHR data for generating clinical evidence.)
Yes
No
Don’t know

83
Figure 22: Distribution of Respondents Hearing Senior Executives Speaking About
Using EHR Data to Generate Clinical Evidence (n=42)
4.3.3 Perceived External Pressures
As shown from Iacovou’s research, pressure from external stakeholders to use the new
technology or process can influence an organization to adopt the technology or process. Based
on the literature review, several positive influences were identified and included in the
investigation. These factors included the passage of The Cures Act, the interest of the FDA and
the patients in the use of EHR data, the release of FDA guidance documents, business
competition, the potential to reduce cost and time, the confidence in EHR data, and the
accessibility of generalizable data (Iacovou, Benbasat and Dexter_1995).
14% (6)
36% (15)
40% (17)
10% (4)
(Q: To what extent have you heard senior executives of your organization
speaking about using EHR data to generate clinical evidence?)
Very frequently
Somewhat frequently
Rarely
Never

84
The results showed cost reduction and shortening of timelines as most influential,
followed by access to generalizable data, FDA’s interest, and FDA’s guidance documents. The
quality of EHR data, corporate competition, patients’ interest, and the passage of The Cures Act
were less influential (Figure 23).

85
Figure 23: Distribution of Factors that Influenced the Advancement of using EHR Data
(n=46)
22 23
17
14
11
6 6 8 8
20
15
18
19 24
18 17
9 8
3
4
5 3
13
12
15
13
2 2 2 4
4
6
3
1 1 3 3 1 4
3
2
3 5 3 3 3 4 3
4
12
(Q: In your view, the following influenced your organization to advance the use of
EHR data for generating clinical evidence.)
Strongly Somewhat agree
Neither agree nor disagree Somewhat disagree
Strongly Don't know

86
Respondents were asked if there were other concepts that influenced their organization to
investigate the use of EHR data. Seven respondents provided comments, five suggested positive
influences and two suggested negative influences.
Self-identified positive influences:
The ability to make better decisions by using patient centric data
Volume and types of data available relative to other types of clinical evidence
Value of RWE generation
Introduction of pragmatism to regulatory strategy
EU MDR
Self-identified negative influences:
Cybersecurity impacts (HIPPA & privacy protection)
Research and Development interests (Strongly disagree)
To delve deeper into the influence of the FDA, questions specific to CDRH’s attitude
were included. Respondents were asked about their perception of CDRH representatives’
attitudes toward the use of EHR data for clinical evidence. A third of the respondents (35%,
15/43) indicated they experienced a positive attitude, while almost half of respondents neither
agreed nor disagreed (42%, 18/43). Three (7%) indicated CDRH was not supportive of using
EHR data and another three did not provide an answer. (Figure 24).

87
Figure 24: Perception of CDRH Encouragement for the Use of EHR Data (n = 43)
Respondents were asked to provide comments regarding their experience with CDRH.
Twenty-six respondents provided comments.
FDA generally acknowledges data options available, but I’ve not observed a strong
emphasis or encouragement from the Agency to use EHR data. In my experience, the
Agency requires valid scientific evidence in support of submissions and largely leaves
it up to the submitter in deciding the best source. We’ve used RWE in a number of
submissions, with little pushback from FDA, but is generally more readily accepted
when it’s in conformance to protocol controls & represents high level of evidence
(Level 1 or Level 2) of support for 510(k) submissions. This is generally always the
case for Class III PMA products, since safety & effectiveness of the technology must be
proven. I’m only aware of one (1) instance where FDA allowed usage of published
data from medical literature for a DeNovo submission, outside of a protocol controlled
randomized clinical study.
The use of EHR data normally has to be very specific & in conformity to established
requirements for the specific technology, which may/may not exist within a general
EHR.
35% (15)
42% (18)
7% (3)
16% (7)
Agree
Neither agree or disagree
Disagree
Don’t know
(Q: In your view, the FDA's CDRH encourages the use of EHR data to generate
clinical evidence.)

88
I believe it is device-specific but we had mixed experiences so far. EHR data should
have been sufficient for some projects but FDA insisted on RCTs. EHR data seems to
be a nice-have but not sufficient.
The FDA as a regulatory body offers guidance with such information. In doing so they
are trying to convince manufacturers to follow best practices. The hope is that
organizations will respect pharmacovigilance and the integrity of this data as there are
still ways present to manipulate data to one’s advantage. As humans in business, most
are concerned with having an advantage and the bad still creep through sometimes.
This is the risk we balance.
Agency has expectations that health information be documented in a manner that is
reliable, auditable, and computer accessible.
I feel the agency is still struggling with how to use the data for real world applications.
They have approached the theory, but the practice is not consistent.
Agree but this is still developing and requires a lot of discussion and alignment. Also,
may not fulfill all global evidence needs.
CDRH is getting there. While they have guidance documents released on using RWE,
they still struggle to understand and accept the limitations of it. As such, RWE for
device approvals is lagging.
It's been a while since I was directly involved. They were ambivalent to suspect years
ago, but may have changed their attitude
FDA does not openly encourage however it depends on industry appetite to find
capable partners for extraction of EHR data
Implementation of their guidance may fall off deeper in the org at the reviewer level
In my experience it depends a lot on the review group. Some are receptive and some
are not.
With regards to IVDs, the agency has not been as open minded about moving past
traditional comparison studies, which are almost completely absent from EHR data.
It honestly depends on what the scope of the submission has been for. Though it has
been encouraged, the usefulness of the data is often brought to question. RWE is useful
in understanding off label use and potential for label expansions. Additionally, some
claims are better served being evaluated with RWE given there is the potential for less
bias.
Depends on the product
Presenting any data besides clinical study generated is a struggle.

89
We interpret guidance as "use what is fit for purpose" among traditional prospective
and retrospective studies, RWD, EHR, other sources. It is one piece of the puzzle, one
tool available.
Based on the guidance documents released and engagement in informal and formal
discussions I agree that CDRH are encouraging of RWE
Pay more attention to CDER guidelines.
Yes, along with MDIC guidance very useful
Ambiguity still exists in the provided guidance.
Topic presented at peer group conference by regulatory panel.
Some data capture systems lend themselves to data transfer more readily than others
Diversity in clinical trials, better patient representation comes from real world
evidence, and determining impact on patient populations.
Industry working diligently on cloud base shareable data for patient information. Our
Midmark Diagnostic group has been enhancing our vital sign devices, ECG devices for
communicating with hospital EMR and data transfer. It is crucial to ensure these data
transfer well, benefiting clinical investigation.
In the past I have been at companies that supported the use of EHR data; however, it
had to come almost to legal action before it took place.
Respondents were also asked how influential certain FDA documents were in peaking
interest in using EHR data for clinical evidence. The question identified four guidance
documents and one summary report.
• Design Considerations for Pivotal Clinical Investigations for Medical Devices:
Guidance for Industry, Clinical Investigators, Institutional Review Boards and
FDA Staff (2013 Guidance Document)
• Electronic Source Data in Clinical Investigations: Guidance for Industry (2013
Guidance Document)

90
• Use of Real-World Evidence to Support Regulatory Decision-Making for Medical
Devices: Guidance for Industry and Food and Drug Administration Staff (2017
Guidance Document)
• Use of Electronic Health Record Data in Clinical Investigations Guidance for
Industry (2018 Guidance Document)
• Examples of Real-World Evidence (RWE) Used in Medical Device Regulatory
Decisions (2021 Summary Report)
Most respondents indicated that they found the documents to be helpful (50% to 60%) or
very helpful (24% to 38%). A few respondents indicated their organization did not use the
documents (5% to 17%) or found them not helpful (2% to 7%) (Figure 25).

91
Figure 25: Helpfulness of FDA Guidance Documents and Summary Report (n=46)
Respondents were asked to identify other influential documents their organization utilized. Two
respondents suggested the following:
10 11
17 15
18
24
27
24
24
22
4
2
2
2
2 7
5
2
3 3
Guidance:
Design
Considerations
(2013)
Guidance:
Electronic
Source Data
(2013)
Guidance: Use
of Real-World
Evidence (2017)
Guidance: Use
of Electronic
Health Record
Data (2018)
Report:
Examples of
Real-World
Evidence (2021)
Q: In your view, how helpful are the following FDA documents to your
organization when advancing the use of EHR data for clinical evidence?
Very helpful Helpful Not helpful Not used

92
• Submitting Documents Using Real-World Data and Real-World Evidence to FDA
for Drug and Biological Products (issued 2022 by CDER and CBER)
• Benefit/risk factors for determination of substantial equivalence (Interpreted by
this researcher to be Benefit-Risk Factors to Consider When Determining
Substantial Equivalence in Premarket Notifications (510(k)) with Different
Technological Characteristics: Guidance for Industry and Food and Drug
Administration Staff, issued 2018 by CDRH and CBER)
4.4 Final Thoughts
Respondents were asked to share lessons they or their organizations have learned from
investigating the use of EHR data. Of the 46 respondents, 28 provided comments.
(Q: What lessons have you or your organization learned from investigating the use of EHR data for
generating clinical evidence?)
EHR data is generally considered retrospective which impacts the perceived quality
compared to prospectively collected clinical investigation data. We have found that
providing small grants to researchers can greatly enable them to pursue EHR data
which they then publish. Having the EHR based studies published is a key step in
having the data accepted by the FDA as supportive of regulatory filings. The
challenge of doing clinical studies in the field of acute trauma care is a significant
factor in our use of EHR data as there are often few other options. Engineering
studies and usability studies form the core of our regulatory submissions, but EHR
based analyses are important supportive information.
As of now it is more hypothesis generating data, or useful for claims / indication
expansion.
Cleaning data is the most important step. Without proper care, the results may be
misleading.
Device risk may be a factor in how detailed the data may be within the EMR. e.g.
disposable products may not be tracked or have an IDU rendering unusable.
Implantable have more detail in the EMR but then the burden of proof for approval
pathway is more stringent.

93
Expect it to take at least 30% longer than you think and also cost 30% more than you
expect.
Overcoming bias concerns will be challenging.
FDA encourages the use of EHR to generate RWE and support marketing
applications, but the reviewers themselves tend to require RCTs. In my experience the
issue is not the quality of EHR generated evidence but rather a paradigm shift is
needed for the FDA reviewers to deem that kind of evidence sufficient (at least for
class II devices and certain de novo requests).
FDA wants to see more and more clinical data with any classification of the devices.
Therefore, EHR enables the feasibility of multiple studies comparing to traditional
clinical studies or trails.
Institutions and institutional requirements for EHR data vary, and there’s not a
universal protocol of acceptance amongst hospital systems. This is also the case when
international data is desired, with differences of ownership, control, and usage
boundaries between many key global markets.
It is a necessary component but equally difficult to implement at this time.
It is mainly in the discussion phases at the corporate level. We do have it in our
business united but not utilized in the capacity that it can be. I have previously worked
in healthcare in a hospital setting and the EHR are used significantly to draw
outcomes from data. It is used very minimally in the Medical Device setting and could
be used more to capture Real World Data and outcomes more tied to the real life
setting vs the traditional clinical trial. Also each business unit captures their data
differently where it is not harmonized throughout the business with same process
throughout. In addition, the company is so large where that can sometimes be
inhibiting with alignment and agreement of the goal and way to reach the goal
capturing relevant data.
It will not be simple or quick. We have to work closely with the vendor to get to the
right data.
It’s not as straightforward as you would think because much of what’s needed doesn’t
easily come through on the EHR because it wasn’t designed for it.
More accurate representation of the real life use of a device. Best place to explore
other indications and uses of the device. However, inconsistency of data and lack of
details that the FDA expects to review for an approval still lack. This is where systems
like KP that have their own home grown EHR systems could have an advantage. Pre
specifying basic data elements for research purposes, linking EHRs from different
systems, and willing to work with med tech companies on maximizing the
understanding of their devices should be the future. Additionally, professional
societies’ engagement with hospital systems to make the data more useable is also be
considered.
Much easier said than done.

94
My experience has been more the use of EHRs to attempt to obtain clinical data
missing from eCRF collection. From that standpoint, I have not found EHRs to
provide much valuable information. The EHRs I have reviewed are not nearly as
detailed and informative as older records that were written or dictated. It is unlikely
that the specific data items needed to assess a procedure or device would be present
in typical EHRs.
Need to be rigorous in data collection and methodology-prepare upfront to evaluate
multiple populations and demonstrate effectiveness across multiple cohorts; use fit for
purpose data
That it is taking more time than we had anticipated.
The lack of standardization, infrastructure, and openness of regulators to RWE that
does not mimic a sample for sample method comparison study for IVD performance
validation makes its use almost impossible to support regulatory submissions
currently.
The senior management needs to emphasize the importance of the use of EHR data for
generating clinical evidence and improve the awareness of it through training and
putting more resources to this area.
There is potential in EHR (couple of successful examples).
Useful for collection of background information (e.g., medical history), but of limited
use to meet typical prospective data requirements for clinical studies.
Variability in how clinical assessments are made (e.g., different grading scales) can
make data unusable. Often, no dedicated fields are available to store required
product name or endpoints require the extraction of data from text fields. It makes
analysis extremely complicated and tedious, with potentially unreliable and highly
variable data. EMR is great for certain data like blood pressure and IOP, but
horrible for dry eye studies and even some contact lens studies.
Variability of database systems, formats and lack of unified systems creates
bottlenecks (time, cost, workload) in the attempts to incorporate EHR data in the
generation of clinical evidence. Clearer, more transparent regulations/guidance is
needed to ensure alignment of the different regions and healthcare provider systems
towards this.
We are still in evaluation phases for EHR, having prioritized investment in
chargemaster data at a national-sample level and also NIS dataset. EHR value case
will need to be made, and will likely fill gaps of other RWD types/sources (e.g. lab
values, functional assessment outcomes, imaging...)
While it is important to explore the possibility, recognizing the limitations to decision
making is important so that an appropriate and timely pivot could be made.
Not privy to this information.
Proprietary.

95
Respondents were then asked for any final comments regarding their experience with
investigating the use of EHR. Fifteen (15) respondents provided comments.
(Q: Please provide any additional insights regarding the benefits of using EHR data for regulatory
purposes.)
Can be helpful in supporting regulatory approvals.
Consideration of EHR use must begin very early in the development process, with
detailed screening, vetting, and collaboration review between development engineers
and clinical/medical representatives. The data needed to either prove substantial
equivalence (510(k)) or safety & effectiveness (PMA) may not be the customary data
routinely gathered for any given patient. Thus, early Medical Affairs involvement
within the data modeling process is crucial, as well as location specificity, and if a
multi-center approach is to be used for data gathering and what boundary nuances
may exist between participating hospital systems/clinics/investigators.
EHR data can be extremely valuable in the regulatory process - either for submission
or for supplemental evidence. The key is being able to identify discrete groups for
comparison (either pre/post or control/comparison groups) with appropriate control
factors.
EHR data has use as supplementary data and for hypothesis generation which seems to
have better acceptability since typically those have relied on passive reporting in
literature and having deeper level data certainly adds confidence in the study design.
EHR data is real world data that can include more diversity and inclusion patient
populations than randomized controlled trials.
EHR will have more ease-of-use when it comes as structured data (e.g., LOINCS
annotated...) Otherwise, it takes a good deal of collaboration between data science
(for actually working with the data) and clinical science/medical affairs (for
interpreting and guiding what matters). High value in linking EHR to other trials or
other RWD sources to tell a story that is simultaneously both broad and specific/deep.
Expeditious submissions, indications for disease states where running a trial may not
be feasible, reducing the burden and risk of off label use of devices, and a real world
understanding of the device across a larger cross section of patient populations.
I’m also interested to learn more about EHR data benefit as well as cybersecurity
related issues.
If a particular study used only data that are typically recorded in office visits,
especially one using billing data, the EHR data might be useful. If each participating
clinic/hospital EHR system could be modified to collect specific data necessary for a
regulatory submission, that might work too.
Improves the path to market with less time and money and more generalizability but
can lack key elements, distinguishers, endpoints or details.
Large n number in studies. Novel and innovative approach, good FDA engagement.

96
This is not just a limiting factor for regulatory or clinical purposes. EHR will provide
next steps for device strategy and patient needs. All areas can benefit from the
information from EHR, especially in R&D to bridge the gaps, medical affairs to
provide additional insights and for industry trends.
Don't know.
Not privy to this information for my organization.
Proprietary

97
Chapter 5. Discussion
5.1 Overview
The term real-world data has only recently been coined by regulators and industry, driven
by the adoption of digital data repositories and access to real-time data. The term cuts across
industries and disciplines and can fall into the concept of Big Data since many real-world data
repositories can be quite large. These repositories can contain financial data, health data,
location data, and even social determinants obtained from social media. In all cases, these large
bodies of data provide interesting opportunities for access to knowledge. The adoption of EHR
platforms by healthcare entities nationwide allowed access to one such source of real-world
organized digital health data.
In the realm of medical product development, the attraction toward the use of EHR data
is the availability of large, up-to-date datasets that could be harnessed to bring innovative
products to the market faster and cheaper. Moreover, evidence generated from EHR data could
be generalized over a more diverse and expansive population than data from observational or
randomized controlled clinical trials. With the issuance of The Cures Act in 2018, policymakers
aimed to encourage regulators and the medical product industry to embrace EHR data for
generating clinical evidence.
This research sought to understand the medical device industry’s attitude toward EHR
data, and the barriers and influences associated with the use of EHR data to generate clinical
evidence. Lessons learned from those who have been successful with using EHR data for
generating clinical evidence were also captured. The following is a discussion of the
methodology along with the results. Also provided are suggested next steps as interpreted from
the results.

98
5.2 Considerations for Research Methodology
Consideration of the methodology should be given when reviewing the results of this
research. Delimitations and limitations of the framework, the tool used to capture data, and
response pool are discussed below.
A few assumptions were made when designing the study and in obtaining responses, such
as assuming the respondents were truthful when completing the survey. Of most importance is
their identification as a medical device company employee and their familiarity with using EHR
data. Since the survey was anonymous, there is no opportunity to confirm the accuracy of the
responses; thus, this researcher accepted the answers as truthful.
5.2.1 Delimitations
Delimitations refer to the prescribed boundaries of the study, including the research
framework and the respondent demographics (Hassan, 2023). Delimitations imposed on this
research by the author and how they may have impacted the results are discussed below.
5.2.1.1 Framework
The research was delimited to the TOE framework constructs per the revision by Iacovou
(Iacovou, Benbasat and Dexter_1995). While the TOE model is most often utilized to
understand the adoption of new technology, this research used the framework to investigate the
adoption of a new process that utilizes a new technology (i.e., EHR). According to the
framework, the research was focused on three main aspects: 1) the perceived benefits of adopting
the new process, 2) organizations’ readiness to integrate the new process, and 3) the external
pressures urging the adoption of the new process.
While the TOE framework is most often used for understanding the financial sector, this
researcher considered it appropriate for understanding the medical product industry. The cross-

99
utilization of frameworks is common. For example, the Diffusion of Innovation (DOI) has been
utilized to study the adoption of technology, processes, and behaviors across multiple business
sectors (Oliveira and Martins, 2011). For this investigation, the researcher defined possible
elements that made up the three constructs of the framework. These elements were thoughtfully
considered based on the literature review and their alignment within the TOE framework.
However, other drivers beyond those investigated in this study may exist. Thus, to ensure
benefits, pressures, and identifiers of readiness that had not been considered by the researcher or
identified in the literature were captured, respondents were solicited for additional input through
open-ended questions within the survey used for this research. Most of the additional benefits,
pressures, and identifiers of readiness described by the respondents, in fact, align with the
elements of the TOE framework. While there may be other frameworks considered more
suitable for the healthcare sector, it seems that the TOE framework was sufficient for this
research.
5.2.1.2 Respondent Demographics
The scope of this research was delimited to medical device organizations that market
Class II and III devices in the US. Since the use of EHR data by the pharmaceutical industry has
already been investigated by other researchers (Lamberti et al., 2018), it seemed reasonable to
focus on the medical device industry for which research seems to be lacking.
Since the intent of the study was to understand barriers to the adoption of EHR data by
medical device companies, data brokers and clinical organizations that support medical device
companies were excluded. In addition, since the research of interest was focused on U.S. laws,
limiting the scope to those organizations marketing within the U.S. was appropriate. Further,
since the research question pertained to using EHR data for clinical evidence within a regulatory

100
submission, organizations that market Class I devices only were not included within the scope of
this study. Finally, inclusion criteria were confined to individuals engaged in discussions or
activities using EHR data. This restriction was imposed to ensure the validity and clarity of the
results as it mandated the respondent to rely on their own familiarity or experience. Responses
from individuals without such familiarity might have introduced confusion to the results, as these
responses could be influenced by existing literature or personal opinions.
5.2.2 Limitations
Limitations refer to unforeseen weaknesses within the research process that may impact
the results (Ross and Bibler, 2019). These barriers are usually out of the researcher’s control,
such as how many respondents complete the survey sufficiently, the accessibility of the survey
design, and the timeliness of the results. These are discussed below in more detail.
5.2.2.1 Response Rate
As suggested in Chapter 2, there is a limited number of medical device companies that
have investigated and an even smaller number who have successful experience using EHR data
for generating clinical evidence. It should be noted that medical device companies commonly
use EHR data for a variety of purposes, such as economic valuation, patient recruitment, and
identification of medical needs. However, using EHR data for generating clinical evidence
appears to be a less common practice. Thus, identifying respondents meeting the inclusion
criteria was difficult. A vital application that was used for locating and recruiting respondents
was LinkedIn®. However, most companies or individuals do not include identifiers within their
profile that would indicate familiarity with EHR data. Thus, using the search feature in LinkedIn
resulted in only a few connections.

101
Further, no databases nor professional organizations specifically identify companies
using EHR data. The FDA summary report Examples of Real-World Evidence (RWE) Used in
Medical Device Regulatory Decisions proved to be valuable by providing a trove of information
and potential leads. While LinkedIn was used most often, the summary report came to be my
primary source of finding potential respondents with experience using EHR data to support a
submission. Even so, the number of leads remained small.
Even though the participation rate was 22% of those solicited (95/436), the overall
response rate of those who initiated the survey and met the inclusion criteria reached 48%
(46/95). These rates agree with literature that suggests online survey response rates can be as
low as 20% (Nulty, 2008). Further, as suggested by Baruch and Holtom, the representation of
those taking the survey is more important than the number of responses when considering the
validity of the results (Baruch and Holtom, 2008; Holtom et al., 2022). Given the strict inclusion
criteria, the 46 respondents that were included in the final TOE cohort analyses were appropriate
representation of the targeted population, thus supporting the notion that the results are valid.
5.2.2.2 Survey Design
Using a survey as a tool to capture information can cause limitations that may be
compounded with using a digital survey platform compared to a paper survey. A digital survey
can limit accessibility due to the type of device the individual uses to complete the survey.
Further, research shows a lack of engagement when a digital survey is used to capture responses.
To mitigate the lack of engagement, this researcher utilized several techniques, which are known
to improve response rates (Heerwegh and Loosveldt, 2006). These included minimizing the
survey in length, adding a progress indicator within the survey, and ensuring the survey was built
for accessibility.

102
This investigation collected a combination of qualitative and quantitative data.
Qualitative data obtained using open-ended questions can add depth to results since respondents
provide further insight beyond the data obtained from structured guided questions. However,
open-ended questions can be time-consuming and may cause some respondents to skip the
questions or, more detrimental, drop out of the survey altogether before finishing it, introducing
bias into the results.
5.2.2.3 Providence of Results
This research was initiated 13 years after the issuance of the HITECH Act and 6 years
after the passage of The Cures Act. The expectation was that the adoption of EHR data within
the clinical research realm would be nearing the sustainability phase as defined by the Hype
Cycle. Thus, the results from this study might have been considered timely information for
future improvements. However, this study indicates that the adoption of using EHR data to
generate clinical evidence is still in its infancy. As one respondent commented, “Most of the
considerations/advantages speak to steady-state use but don't account for startup overhead for
orgs new to using EHR data.”
It is expected that the viewpoints presented here will evolve over time. Thus, these
results are relevant for the current moment but may become less relevant over time as the use of
EHR data becomes more established. The intent of this research is to offer insights into factors
that could facilitate the transition toward sustainable adoption of EHR data for generating
clinical evidence.
5.3 Consideration of Results
Adopting EHR data for generating clinical evidence represents a great paradigm shift
from a 50-year history of relying on RCTs. From IDE to the final regulatory submission, the

103
processes, analyses, and trial designs have been standardized across the globe for efficient
product development and regulatory oversight. Regulators have become familiar with RCT data
and established standard processes for an efficient and timely regulatory review. A change to the
existing status quo impacts not only those directly involved in product development but also
peripheral activities, resources, and infrastructure. If the benefits obtained from making the
change don’t outweigh barriers for all stakeholders, the new pathway will likely not be adopted
without significant pressure and influence. The following sections will examine one of these
stakeholders, the medical device industry, and captures its perception of the benefits, readiness,
and external pressures associated with the use of EHR data for generating clinical evidence.
5.3.1 Perceived Benefits
The findings from this research indicate a shared agreement among respondents that
using EHR data will elicit benefits of cost reduction, shorter timelines, and access to real-world
data. These findings align with the literature described in Chapter 2. There is much less
consensus associated with reduction in workforce and streamlining analyses.
Comparing the estimated cost of 2.6 billion USD to bringing a new drug to market
(DiMasi, Grabowski and Hansen, 2016) to the estimated cost savings of 100 billion USD
annually when using EHR data for research (Cattell, Chilukuri and Levy, 2013), it becomes
evident that there is substantial value in prioritizing the adoption of EHR data. With the current
scrutiny of corporate profits, it is important to consider the potential allocation of these savings.
The primary goal of the Cures Act was to bring medical innovations to patients faster and
cheaper through the use of the new EHR technology. Thus, it is hoped these savings will find
their way to the patient via cheaper and more affordable products.

104
Further, as the saying goes, "time is money," implying the reduction of timelines may
lead to additional cost savings. However, the perception that using EHR data may reduce
timelines may be short lived as illustrated by Shull’s study summarized in Section 2.2.3. The
process of cleaning EHR data can be both time-consuming and resource-intensive (Shull, 2019),
nullifying anticipated cost reductions. Those who are considering the use of EHR data to
generate clinical evidence should consider this limitation. Further research is needed to
determine how timelines are impacted and if these timelines are still advantageous as compared
to RCTs.
The literature often points to the advantage of using EHR data that would generate
findings that can be applied to a diverse population. This assumption is based on the idea that
assessing product outcomes across a large and diverse population could reduce the risk of
adverse outcomes or unexpected events after the product is marketed and used in the real-world.
However, confirmation is still needed to ascertain whether this genuinely leads to an improved
safety profile for a device once it is on the market.
Considering that most respondents in this study and findings in the literature point to the
potential of EHR to save time and money, one would anticipate a more rapid and broader
adoption by the industry. However, as we will examine in the following sections, factors such as
the organization's readiness, the quality of current data, and unclear communication from
regulators may impede the adoption of a new technology.
5.3.2 Organizational Readiness
Organizational change has a wide-ranging impact, influencing various facets of the
organization's infrastructure. To be ready means embracing changes across multiple elements
that are involved in the new process from beginning to final output. The orchestration of

105
changes must be cohesive and comprehensive across various stakeholders assuming distinct
roles. All stakeholders may support the idea but cannot move forward without cohesively
changing the process together. Thus, the lack of readiness in one area or by one stakeholder can
be a critical barrier to adopting the change. This research identified three primary areas where
readiness is in question.
5.3.2.1 Data Access and Quality
Multiple respondents, both those familiar with and those with successful experience using
EHR data, expressed concern about the accessibility and quality of the data. Accessibility
depends on partnerships with clinical sites as well as data brokers. Accessing EHR data from
clinical sites requires the sponsor to have an IT infrastructure to manage the data, suggesting
upfront costs. Further, sponsors who have partnerships with their clinical sites and can access
the EHR data via interoperable means may have difficulty integrating multiple data sets into one
cohesive data repository for analyses. Data brokers are an alternative resource since they have
already addressed the nuance of combining data from multiple platforms. Unfortunately,
accessing data from data brokers can be an expensive endeavor, thus negatively impacting the
cost-benefit of using EHR data.
Beyond access to the data, the quality of the data was cited as a concern by several
respondents. They noted endpoint data as well as longitudinal data are often missing. For
example, death, a standard mortality endpoint, is most often not captured in EHRs. In fact, the
"lack of available data specific to the end point” and “missing data” were the top concerns
selected by the respondents, pointing to lingering concerns regarding data quality (Figure 15).
Furthermore, as explained in Section 2.2.3, with patients changing healthcare providers many
times throughout their lifetime, it is unlikely records would be complete across time (Holmgren

106
and Adler-Milstein, 2017). Aware of these concerns, the ONC is working to address the
completeness of data through interoperability. As mentioned above, orchestrating a cohesive
change across various stakeholders, such as HIT developers and EHR vendors, is necessary for
success. Even with different EHR vendors becoming aligned regarding system languages,
formatting, and naming convention, achieving interoperability seems daunting.
Further, during a recent ONC symposium on data quality, the ONC acknowledged
improving the quality of EHR data was essential for research. The symposium introduced new
quality standards and frameworks to assist HIT developers in improving their EHR platforms
and the quality of the data captured in the platforms (The Office of the National Coordinator for
Health Information Technology, 2023). This is a positive step forward. However, beyond the
logistics of data completeness, there is the element of how data is captured and interpreted at
different clinics. Cultural differences and health care practices are likely to be different across
geographical regions. As one respondent commented in the open text question ‘What lessons
have you or your organization learned from investigating the use of EHR data for generating
clinical evidence?’:
Variability in how clinical assessments are made (e.g., different grading scales)
can make data unusable. Often, no dedicated fields are available to store
required product name or endpoints require the extraction of data from text
fields. It makes analysis extremely complicated and tedious, with potentially
unreliable and highly variable data.
There was another interesting comment by a respondent who identified shortcomings of
EHR platform compared to the older versions of patient charts:
My experience has been more the use of EHRs to attempt to obtain clinical data
missing from eCRF collection. From that standpoint, I have not found EHRs to
provide much valuable information. The EHRs I have reviewed are not nearly as
detailed and informative as older records that were written or dictated. It is

107
unlikely that the specific data items needed to assess a procedure or device would
be present in typical EHRs.
5.3.2.2 New Skills
The importance of developing skills in managing EHR data was highlighted by multiple
respondents. These insights align with the perspectives of Bowrin and Siami as described in
Section 2.4.2. In 2019, Bowrin recognized new tools and skills would be required to manage and
analyze the data (Bowrin et al., 2019). More recently, Sandi Siami, senior vice president at the
Medical Device Innovation Consortium (MDIC), cautioned that the industry faced challenges in
integrating and validating data from various sources (Slabodkin, 2021). Now, in 2023, a gap in
analytical skills still exists to address these challenges. Yet, respondents emphasized the
importance of understanding EHR data, highlighting the value of developing skills to effectively
analyze EHR data. As suggested by one respondent in the open text question ‘Please provide
any additional insights regarding the benefits of using EHR data for regulatory purposes’:
EHR will have more ease-of-use when it comes as structured data (e.g., LOINCS
annotated...) Otherwise, it takes a good deal of collaboration between data
science (for actually working with the data) and clinical science/medical affairs
(for interpreting and guiding what matters). High value in linking EHR to other
trials or other RWD sources to tell a story that is simultaneously both broad and
specific/deep.
5.3.2.3 Strategy Development
The ability to define and negotiate a strategy for including real-world data early in the
development process was identified as critical to the success of using EHR data to support a
submission. However, there is little guidance from CDRH regarding what an acceptable strategy
might be. It would be helpful if CDRH provided insight into what the division perceives as
acceptable, how reviews will be conducted, and what analyses would provide the greatest
confidence in the data. Most regulatory affairs professionals and FDA reviewers are familiar

108
with submissions containing RCT data. Very few are familiar with utilizing EHR data for
clinical evidence within a submission. Given the lack of direction from CDRH on this matter, it
will be difficult for organizations to clearly define their approach. The following comment by a
respondent in the open text question ‘Which functional areas in your organization need new
skillsets and/or processes to advance the use of EHR data?’ provides a picture of the difficulties
when defining a strategy.
It is important to understand that data generation is always a cross-functional
activity with CA leading and other teams contributing to the data generation. The
entire team needs to understand the inherit limitations to the data and be in
communication with the reg bodies to make sure that data is (1) acceptable and
(2) developed in a way that will be acceptable. Each project will likely require
new ways of analysis and interpretation.
5.3.3 External Pressures
Change, as the saying goes, is difficult. In the realm of physics, it is well-established that
the trajectory of an object remains unaltered unless an external force is exerted upon it. In the
context of the TOE framework, this external force can be conceptualized as external pressures or
influences. Notably, the factors compelling organizations to explore the use of EHR data are
remarkably consistent with the perceived benefits described above. These factors are reduction
in cost, shortening of timelines, and access to generalizable data. Beyond these factors, external
stakeholders played a significant influential role.
The FDA’s expressed interest was pivotal in pushing towards wider adoption. Prominent
FDA representatives lauded the advantages of EHR data, emphasizing its potential to reduce
product costs and improve patient access. This public endorsement of using EHR data for
generating clinical evidence signified the FDA’s willingness to consider real-world data to
support regulatory submissions. As a result of these favorable remarks by the FDA, a few
organizations capitalized on this supportive stance and successfully leveraged EHR data.

109
However, this research uncovered a significant issue: nearly 50% of respondents indicated
confusion due to mixed messages between the administration and reviewers within CDRH.
According to the literature, the FDA expresses verbal support of using EHR data at
conferences and in written communications (Califf et al., 2016; Franklin et al., 2019). Yet,
according to results of this study, reviewers persist in emphasizing the importance of clinical trial
data for submissions. Furthermore, respondents indicated an inconsistency with reviewers across
different divisions, with some reviewers hesitant to embrace real-world data. Part of the
confusion may be due to contradictory and vague language in guidance documents issued by the
FDA. For example, details on how CDRH assesses the validity and quality of data are not
provided within the guidance document Use of Real-World Evidence to Support Regulatory
Decision-Making for Medical Devices. Further, the guidance document Guidance on Design
Considerations for Pivotal Clinical Investigations for Medical Devices continues to include
cautionary notes regarding the use of non-RCT data (FDA, 2013a) as discussed in Section 2.5.2.
Interestingly, it is worth noting that one of the guidance documents, known as “Least
Burdensome” (FDA, 2019a), underwent revisions to facilitate the acceptance of real-world data
within the FDA’s regulatory framework. In fact, the FDA introduced annual training for
reviewers on applying this guidance to support the use of real-world data (FDA, 2019b). Given
the numerous complaints about the reviewers’ lack of support for the use of EHR data, one must
wonder if the training has been effective. It is reasonable to assume regulators are more
comfortable reviewing data from RCTs, and this preference may inadvertently bias a reviewer’s
support of EHR data. As one respondent stated, “I feel the agency is still struggling with how to
use the data for real-world applications. They have approached the theory, but the practice is not
consistent.”

110
Due to the significant time and cost associated with submitting and obtaining regulatory
clearance or approval, organizations are less inclined to use EHR data if they are not certain that
their results will be accepted. Organizations will opt for a more secure approach if the perceived
risk is too high. One potential solution could be implementing a program similar to the
Advancing Real World Data Program which was initiated by CDER/CBER in 2023 (FDA,
2023a). The program allows organizations to collaborate with regulators to define the criteria for
acceptance. While pre-IND meetings with regulators are already available to medical device
organizations, this CDER/CBER program specifically targets those aiming to use real-world data
during the IND stage. A similar program by CDRH would allow for strategic planning from IDE
to PMA, primarily focusing on using real-world data to generate clinical evidence.
Implementing such a program may substantially impact the adoption of EHR data within the
medical device industry.
5.4 Final Comments
This research uncovered potential barriers contributing to the medical device industry’s
reluctance to adopt EHR data for generating clinical evidence. While there are several
interesting findings, four key themes emerged:
• The challenge of gaining access to data repositories, necessitating strong
partnerships between data providers and users,
• The importance of developing skills to comprehend and manage the limitations of
the data while also leveraging its advantages.
• The pivotal role of early strategy development for achieving success, and
• A lack of clarity from the FDA on the use of EHR data for generating clinical
evidence.

111
Not surprisingly, these themes have some similarities with the findings described in the Tufts
survey of pharmaceutical companies discussed in Section 2.4.2 (Lamberti et al., 2018).
Specifically, three themes are aligned between the medical device and pharmaceutical industries:
1) the uncertainty of regulators, 2) the accessibility of data, and 3) the quality and reliability of
the data. This alignment emphasizes the importance of addressing these barriers. Beyond the
Tufts’ survey, this research uncovered further insights. The findings point to discord in
messaging between the FDA administration and reviewers, the need for enhanced skills, and
guiding strategies for managing applications using EHR data.
With the issuance of The Cures Act, there was some hope in revising the costly medical
device development process. The expectation was for the FDA to begin investigating the use of
EHR data to support regulatory submissions soon after issuance, followed by industry quickly
adopting this new pathway with the promise of gained benefits in the form of reduced costs and
time. However, initiating this mandate with limited access to regulatory quality EHR data and
clear thought-out processes, analyses, and study designs beyond those already available for
RCTs would surely cause confusion. Further, development of skills in areas such as analytical
and regulatory strategy are needed to meet this new challenge. The authors of The Cures Act
may not have considered the comprehensive changes needed to cause a shift away from the
established RCT process to the use of real-world data for generating clinical evidence.
The mandate by The Cures Act to investigate the use of EHR data touches many
stakeholders beyond the FDA, such as the ONC, medical product companies, healthcare
providers, and patients. However, only the FDA was mandated to investigate how to utilize
EHR data for product accessibility. Thus, while the ONC acknowledges that the data captured
by EHRs can be used for research (The Office of the National Coordinator for Health

112
Information Technology, 2023), EHRs are not built for the purpose of supporting product
development nor meeting the FDA’s submission requirements. In most instances, platforms do
not include the data required for endpoints that support effectiveness and safety evidence, such
as longitudinal outcomes and mortality. However, during a recent conference, the ONC
expressed commitment toward comprehensive interoperable sharing of data to improve the
quality of that data. Yet there was minimal discussion addressing the need for longitudinal
endpoint data and mortality data (The Office of the National Coordinator for Health Information
Technology, 2023). Without addressing these glaring gaps, improving the quality of EHRs will
have little impact for medical product development.
Further, CDHR defines specific endpoints for specific indications. Since these endpoints
are rarely captured within patient EHRs, it would not be beneficial for the manufacturer to
consider EHRs within their regulatory strategy. Reviewers familiar with these requirements are
expecting to see these endpoints included within the clinical results. If clinical evidence is to
shift to EHR data these endpoints will either need to be realigned with what is already captured
in EHR platforms, or EHR platforms will need to be redesigned with these required endpoints in
mind.
This suggestion is supported by healthcare providers involved in recent discussions
conducted by the Reagan-Udall Foundation. The Reagan-Udall Foundation is an independent
organization created by Congress with its primary purpose being to advance the mission of the
FDA. The Foundation conducted a series of roundtable meetings with healthcare providers for
the purpose of developing a framework for integrating medical product research into clinical
care. While the meetings were not specifically focused on EHR use, the healthcare providers

113
also urged protocol endpoint alignment with clinical standards of care along with adjusting
outcomes to data already available in EHR platforms (reganundall.org, 2023).
Even so, organizations recognize the value in pursuing the use of EHR data for
generating clinical evidence. Interestingly, even with current limitations, almost 50% of
respondents indicated their support in continuing to investigate the use of EHR data, indicating a
possible shift from the trough of disillusionment towards the slope of enlightenment per the
Gartner Hype Cycle (Blosch and Fenn, 2018). Given the benefits of cost reduction, shortening
of timelines, and access to generalized data, organizations would likely be supportive in utilizing
this pathway if access to the right data at the right price occurs. As companies and federal
agencies enhance EHR platforms, it is possible this new process may move to sustainability. For
this shift to occur, however, a collaborative approach between the FDA and the ONC with input
from healthcare providers and patients must be considered.
In summary, results of this research suggest industry perceives using data obtained from
EHRs costs and timelines will be reduced and access to generalizable data will be had. Yet,
there is reluctance by the industry due to a lack of skills, regulatory uncertainty, and data quality
concerns. To be successful with shifting the paradigm, elements still need to be addressed and
investigated, such as investigating the evolution of technology, regulatory clarity, and EHR
platforms. Further research is needed to clearly determine the impact of this new approach on
total cost, timelines, and overall safety and effectiveness of a device once it is on the market.
Understanding patient’s perceptions on the expanded use of EHR data should also be
investigated, given the current emphasis on personal data privacy. Finally, it is essential for
future research to investigate whether the utilization of EHR data can effectively accelerate
patient access to new therapies, aligning with the objectives outlined in The Cures Act.

114
References
Ajami, S. and Bagheri-Tadi, T. (2013) Barriers for adopting electronic health records (EHRs) by
physicians, Acta Informatica Medica, 21(2), 129–134.
Akron General Hospital (1966) 1961 electronic medical records. Available at:
https://www.youtube.com/watch?v=t-aiKlIc6uk (Accessed: 5 December 2020).
Avorn, J. and Kesselheim, A.S. (2015) The 21st Century Cures Act — Will it take us back in
time?, New England Journal of Medicine, 372(26), 2473–2475.
Baruch, Y. and Holtom, B.C. (2008) Survey response rate levels and trends in organizational
research, Human Relations, 61(8), 1139-1160. doi:10.1177/0018726708094863
Blosch, M. and Fenn, J. (2018) Understanding Gartner’s hype cycles, Gartner, Inc. Available at:
https://www.gartner.com/en/documents/3887767/understanding-gartner-s-hype-cycles
(Accessed: 29 December 2021).
Blumenthal, D. and Tavenner, M. (2010) The “Meaningful Use” regulation for electronic health
records, New England Journal of Medicine, 363(6), 501–504.
Bonafede, M. (2022) Comparing registry and electronic health record (EHR) data for real-world
evidence generation: heart failure as a case study. Veradigm Allscripts Healthcare LLC, 1-10.
https://veradigm.com/img/registry-and-electronic-health-record-data-comparison-for-real-worldevidence-generation.pdf (Accessed: 04-Oct-2023)
Bourgeios, F., Gutiérrez-Sacristán, A. and Keller, M. (2021) International analysis of electronic
health records of children and youth hospitalized with COVID-19 infection in 6 countries, JAMA
Network Open, 4(6). doi:10.1001/jamanetworkopen.2021.12596.
Bowrin, K. et al. (2019) Cost-effectiveness analyses using real-world data: An overview of the
literature, Journal of Medical Economics, 22(6), 545–553.
Califf, R.M. et al. (2016) Transforming evidence generation to support health and health care
decisions, New England Journal of Medicine, 375(24), 2395–2400.
Callahan, A. et al. (2019) Medical device surveillance with electronic health records, npj Digital
Medicine, 2(1), p. 94. doi:10.1038/s41746-019-0168-z.
Cattell, J., Chilukuri, S. and Levy, M. (2013) How big data can revolutionize pharmaceutical
R&D. London, UK: McKinsey & Company. Available at:
https://www.mckinsey.com/~/media/McKinsey/Industries/Pharmaceuticals%20and%20Medical
%20Products/Our%20Insights/How%20big%20data%20can%20revolutionize%20pharmaceutica
l%20R%20and%20D/How%20big%20data%20can%20revolutionize%20pharmaceutical%20RD
.pdf (Accessed: 9 November 2021).

115
Chaney, K. et al. (2020) National health IT priorities for research: A policy and development
agenda. Office of the National Coordinator for Health Information Technology (ONC),
Washington, DC. Available at: https://www.healthit.gov/sites/default/files/page/2020-
01/PolicyandDevelopmentAgenda.pdf (Accessed: 10 April 2021).
CMS (2021) Centers for Medicare & Medicaid Services. Available at: https://www.cms.gov/
(Accessed: 19 November 2021).
Deeds, S.A. et al. (2020) Leveraging an electronic health record note template to standardize
screening and testing for COVID-19, Healthcare, 8(3). doi:10.1016/j.hjdsi.2020.100454.
Dhruva, S.S., Ross, J.S. and Desai, N.R. (2018) Real-world evidence: Promise and peril for
medical product evaluation, P&T, 43(8), 464–472.
DiMasi, J.A., Grabowski, H.G. and Hansen, R.W. (2016) Innovation in the pharmaceutical
industry: New estimates of R&D costs, Journal of Health Economics, 47, 20–33.
Evans, R.S. (2016) Electronic health records: Then, now, and in the future, Yearbook of Medical
Informatics, Suppl 1, S48-61.
FDA (1997) 21 CFR Part 11: Electronic records; Electronic signatures, Federal Register, 62(54),
13429–13466.
FDA (2002) The least burdensome provisions of the FDA Modernization Act of 1997: Concept
and principles: Final guidance for FDA and industry, Food and Drug Administration (United
States). Available at: https://www.fda.gov/files/medical%20devices/published/The-LeastBurdensome-Provisions-of-the-FDA-Modernization-Act-of-1997--Concept-and-Principles---
Final-Guidance-for-FDA-and-Industry.pdf (Accessed: 12 December 2021).
FDA (2013a) Design considerations for pivotal clinical investigations for medical devices, Food
and Drug Administration (United States). Available at: https://www.fda.gov/regulatoryinformation/search-fda-guidance-documents/design-considerations-pivotal-clinicalinvestigations-medical-devices (Accessed: 22 September 2020).
FDA (2013b) Electronic source data in clinical investigations, Food and Drug Administration
(United States). Available at: https://www.fda.gov/regulatory-information/search-fda-guidancedocuments/electronic-source-data-clinical-investigations (Accessed: 19 September 2020).
FDA (2017) Use of real-world evidence to support regulatory decision-making for medical
devices - Guidance for industry and Food and Drug Administration staff, Food and Drug
Administration (United States). Available at:
https://www.fda.gov/files/medical%20devices/published/Use-of-Real-World-Evidence-toSupport-Regulatory-Decision-Making-for-Medical-Devices---Guidance-for-Industry-and-Foodand-Drug-Administration-Staff.pdf (Accessed: 10 October 2020).

116
FDA (2018) Use of electronic health record data in clinical investigations, guidance for
industry, Food and Drug Administration (United States). Available at:
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/use-electronichealth-record-data-clinical-investigations-guidance-industry (Accessed: 10 October 2020).
FDA (2019a) The least burdensome provisions: Concept and principles, Food and Drug
Administration (United States). Available at: https://www.fda.gov/regulatoryinformation/search-fda-guidance-documents/least-burdensome-provisions-concept-andprinciples (Accessed: 22 October 2020).
FDA (2019b) The least burdensome provisions: Concept and principles; Guidance for industry
and Food and Drug Administration staff, Federal Register, 84(1747), 1747–1749.
FDA (2021a) 21 CFR 860: Food and Drug Administration, Department of Health and Human
Services, Medical device classification procedures. Available at:
https://www.ecfr.gov/current/title-21/chapter-I/subchapter-H/part-860?toc=1 (Accessed: 7
December 2021).
FDA (2021b) 45 CFR 170: Health information technology standards, implementation
specifications, and certification programs for health information technology. Available at:
https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-D/part-170 (Accessed: 7 December
2021).
FDA (2021c) Examples of real-world evidence (RWE) used in medical device regulatory
decisions, Food and Drug Administration (United States). Available at:
https://www.fda.gov/media/146258/download (Accessed: 18 March 2021).
FDA (2021d) Premarket Approval (PMA). Food and Drug Administration (United States).
Available at: https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMA/pma.cfm (Accessed: 11
November 2021).
FDA (2022) Medtronic Stops Distribution and Sale of HeartWare HVAD System Due to Risk of
Neurological Adverse Events, Mortality, and Potential Failure to Restart. Food and Drug
Administration (United States). Available at: https://www.fda.gov/medical-devices/medicaldevice-recalls/medtronic-stops-distribution-and-sale-heartware-hvad-system-due-riskneurological-adverse-events (Accessed: 17 January 2022).
FDA. (2023a) Advancing real world data program, Food and Drug Administration (United
States). Available at: https://www.fda.gov/drugs/development-resources/advancing-real-worldevidence-program. (Accessed: 30 Oct 2022)
FDA. (2023b) Considerations for the use of real-world data and real-world evidence to support
regulatory decision-making for drug and biological products, Food and Drug Administration
(United States). Available at: https://www.fda.gov/media/171667/download. (Accessed: 05 Sep
2023)
FDA. (2023c) Considerations for the design and conduct of externally controlled trials for drug
and biological products, Federal Registry, 88(21), 6748-6750.

117
FDA, FCC, and ONC (2014) FDASIA health IT report: Proposed strategy and
recommendations for a risk-based framework. United States: Congress. Available at:
https://www.healthit.gov/sites/default/files/fdasia_healthitreport_final.pdf (Accessed: 27 July
2021).
Fleming, N.S. et al. (2011) The financial and nonfinancial costs of implementing electronic
health records in primary care practices, Health Affairs, 30(3), 481–489.
Franklin, J.M. and Schneeweiss, S. (2017) When and how can real world data analyses substitute
for randomized controlled trials?, Clinical Pharmacology and Therapeutic, 102(6), 924–933.
Franklin, J.M., Glynn, R.J., Martin, D. and Schneeweiss, S. (2019) Evaluating the use of
nonrandomized real-world data analyses for regulatory decision making, Clinical Pharmacology
and Therapeutics, Apr;105(4):867-877.
Gillum, R.F. (2013) From papyrus to the electronic tablet: A brief history of the clinical medical
record with lessons for the digital age, The American Journal of Medicine, 126(10), 853–857.
Glasgow, R.E. et al. (2012) Patient-reported measures of psychosocial issues and health behavior
should be added to electronic health records, Health Affairs (Project Hope), 31(3), 497–504.
Gliklich, R. E., Levy, M. B., & Dreyer, N. A. (eds.) (2019) Registries for evaluating patient
outcomes: A user’s guide. 3rd edn., Addendum 2. Washington, DC: L&M Policy Research, LLC,
with partners OMI and IQVIA. Available at:
https://www.ncbi.nlm.nih.gov/books/NBK551879/pdf/Bookshelf_NBK551879.pdf (Accessed:
22 September 2020).
Gold, M. and McLaughlin, C. (2016) Assessing HITECH implementation and lessons: 5 years
later, The Milbank Quarterly, 94(3), 654–687.
Hassan, M. (2023, October 1) Delimitations in research – types, examples and writing guide,
ResearchMethod.net. Available at: https://researchmethod.net/delimitations/ (Accessed: 3 Oct
2023)
Heerwegh, D. and Loosveldt, G. (2006) An experimental study on the effects of personalization,
survey length statements, progress indicators, and survey sponsor logos in web surveys, Journal
of Official Statistics, 22(2), 11-210.
Holtom, B., Baruch, Y., Aguinis, H. and Ballinger, G.A. (2022) Survey response rates: Trands
and a validity assessment framework, Human Relations, 75(8), 1560-1584.
H.R.1 (2003) Medicare Prescription Drug, Improvement, and Modernization Act of 2003. U.S.
108th Congress: Medicare Prescription Drug, Improvement, and Modernization Act of 2003.
Available at: https://www.congress.gov/108/plaws/publ173/PLAW-108publ173.pdf

118
H.R. 1 (2009) The American Recovery and Reinvestment Act, U.S. 111th Congress: The
American Recovery and Reinvestment Act. Available at:
https://www.congress.gov/111/plaws/publ5/PLAW111publ5.pdf#:~:text=Public%20Law%20111%E2%80%935%20111th%20Congress%20An%2
0Act%20,ending%20September%2030%2C%202009%2C%20and%20for%20other%20purpose
s (Accessed: 9 November 2021)
H.R. 34 (2016) 21st Century Cures Act, U.S. 114th Congress: 21st Century Cures Act. Available
at: https://www.congress.gov/114/plaws/publ255/PLAW-114publ255.pdf (Accessed: 2 August
2020)
HeartWare, Inc. (2012) PMA-P100047 Heartware ventricular assist system pre-market approval
summary of safety and effectiveness, Food and Drug Administration (United States). Available
at: https://www.accessdata.fda.gov/cdrh_docs/pdf10/P100047b.pdf (Accessed: 2 August 2020)
Hernandez-Boussard, T. et al. (2019) Real world evidence in cardiovascular medicine: Ensuring
data validity in electronic health record-based studies, Journal of the American Medical
Informatics Association, 26(11), 1189–1194.
Hoffman, S. (2016) Electronic health records and medical big data. New York: Cambridge
University Press.
Hogle, L.F. and Das, A. (2017) The social production of evidence: Regenerative medicine and
the 21st Century Cures Act, Regenerative Medicine, 12(6), 581–586.
Holman, W.L. (2012) Interagency Registry for Mechanically Assisted Circulatory Support
(INTERMACS): What have we learned and what will we learn?, Circulation, 126(11), 1401–
1406.
Holmgren, A.J. and Adler-Milstein, J. (2017) Health information exchange in US hospitals: The
current landscape and a path to improved information sharing, Journal of Hospital Medicine,
12(3), 193–198.
Hripcsak, G. and Albers, D.J. (2013) Next-generation phenotyping of electronic health records,
Journal of the American Medical Informatics Association, 20(1), 117–121.
Hwang, T.J., Sachs, R.E. and Kesselheim, A.S. (2017) Public participation in drafting of the 21st
Century Cures Act, Journal of Law, Medicine & Ethics, 45(2), 212–220.
Iacovou, C.L., Benbasat, I. and Dexter, A.S. (1995) Electronic data interchange and small
organizations: Adoption and impact of technology, Management Information Systems Research
Center, University of Minnesota, 19(4), 465–485.
IBM (2022) What is EDI: Electronic data interchange? Available at:
https://www.ibm.com/topics/edi-electronic-data-interchange (Accessed: 18 January 2022).

119
Institute of Medicine, Committee on Improving the Patient Record. (1997) The computer-based
patient record: Revised edition: An essential technology for health care, in Dick, R.S, Steen E.B.
and Detmer, D.E. (eds.) Washington, DC: National Academies Press.
Institute of Medicine, Committee on Quality of Health Care in America. (2000) To err is human:
Building a safer health system, in Kohn L.T., Corrigan J.M., Donaldson M.S. (eds) Washington,
DC: National Academies Press.
Institute of Medicine (2012) Envisioning a transformed clinical trials enterprise in the United
States, establishing an agenda for 2020: Workshop summary, Washington, DC: The National
Academies Press.
Jha, A.K. (2010) Meaningful use of electronic health records: The road ahead, JAMA, 304(15),
1709–1710.
Khozin, S., Blumenthal, G.M. and Pazdur, R. (2017) Real-world data for clinical evidence
generation in oncology, Journal of the National Cancer Institute, 109(11).
doi:10.1093/jnci/djx187.
Kim, H.S., Lee, S. and Kim, J.H. (2018) Real-world evidence versus randomized controlled trial:
Clinical research based on electronic medical records, Journal of Korean Medical Science,
33(34), e213.
Lamberti, M.J. et al. (2018) The use of real-world evidence and data in clinical research and
postapproval safety studies, Therapeutic Innovation & Regulatory Science, 52(6), 778–783.
Lammers, E.J., McLaughlin, C.G. and Barna, M. (2016) Physician EHR adoption and potentially
preventable hospital admissions among Medicare beneficiaries: Panel data evidence, 2010-2013,
Health Services Research, 51(6), 2056–2075.
Lederer, S. (1995) Subjected to science: Human experimentation in America before the Second
World War. Baltimore/London: The Johns Hopkins University Press.
Lu, N. et al. (2021) Leverage multiple real-world data sources in single-arm medical device
clinical studies, Journal of Biopharmaceutical Statistics, 1–17.
doi:10.1080/10543406.2021.1897994.
Maissenhaelter, B.E., Woolmore, A.L. and Schlag, P.M. (2018) Real-world evidence research
based on big data: Motivation-challenges-success factors, Onkologe (Berl), 24(Suppl 2), 91–98.
Medtronic, Inc. (2015) PMA P120017: Cardiovascular permanent pacemaker electrode Model
5071 pre-market approval summary of safety and effectiveness data. Food and Drug
Administration (United States). Available at:
https://www.accessdata.fda.gov/cdrh_docs/pdf12/P120017b.pdf (Accessed: 19 January 2022).
Menachemi, N. and Brooks, R.G. (2006) Reviewing the benefits and costs of electronic health
records and associated patient safety technologies, Journal of Medical Systems, 30(3), 159–168.

120
Miani, C. et al. (2014) Health and healthcare: Assessing the real-world data policy landscape in
Europe, Rand Health Quarterly, 4(2), 15.
Miksad, R.A. and Abernethy, A.P. (2018) Harnessing the power of real-world evidence (RWE):
A checklist to ensure regulatory-grade data quality, Clinical Pharmacology & Therapeutics,
103(2), 202–205.
Miller, R.H. and West, C.E. (2007) The value of electronic health records in community health
centers: Policy implications, Health Affairs, 26(1), 206–214.
Niraula, S. et al. (2012) The price we pay for progress: A meta-analysis of harms of newly
approved anticancer drugs, Journal of Clinical Oncology, 30(24), 3012–3019.
Nulty, D. (2008) The adequacy of response rates to online and paper surveys: What can be done?
Assessment & Evaluation in Higher Education, 33(3), 301-314.
Numan, L. et al. (2021) Propensity score-based analysis of long-term outcome of patients on
HeartWare and HeartMate 3 left ventricular assist device support, ESC Heart Failure, 8(2),
1596–1603.
Oliveira, T. and Martins, M.F. (2011) Literature review of information technology adoption
models at firm level, The Electronic Journal Information Systems Evaluation, 14(1), 110–121.
ONC (2020) ONC health IT certification program overview. Washington, DC: The Office of the
National Coordinator for Health Information Technology. Available at:
https://www.healthit.gov/sites/default/files/PUBLICHealthITCertificationProgramOverview.pdf
(Accessed: 10 April 2021).
ONC (2021) Official website of The Office of the National Coordinator for Health Information
Technology (ONC). Available at: https://www.healthit.gov/ (Accessed: 31 May 2021).
Orszag, P. (2008) Evidence on the costs and benefits of health information technology,
Washington, DC: Congressional Budget Office.
Pylypchuk, Y. and Johnson, C. (2020) ONC data brief: State of interoperability among major US
cities. 53, Washington, DC: The Office of the National Coordination for Health Information
Technology.
Ramamoorthy, A. and Huang, S.M. (2019) What does it take to transform real-world data into
real-world evidence?, Clinical Pharmacology & Therapeutics, 106(1), 10–18.
Reagan-Udall Foundation for the FDA (2023) Integrating clinical studies into health care
delivery: Post-market evidence generation for medical products. [Virtual Public Meeting].
14th November. Available at: https://reaganudall.org/news-and-events/events/integrating-clinicalstudies-health-care-delivery-post-market-evidence
Resnic F.S. and Matheny M.E. (2018) Medical devices in the real world, New England Journal
of Medicine, 378(7), 595–597.

121
Richesson, R. et al. (2021) Enhancing the use of EHR systems for pragmatic embedded research:
Lessons from the NIH Health Care Systems Research Collaboratory, Journal of the American
Medical Informatics Association, 28(12), 2626–2640.
Ross, P.T., and Bibler, A. (2019) Limited by our limitations, Perspectives on Medical Education,
8(4), 261-264.
Rothman, D. (1991) Strangers at the bedside; A history of how law and bioethics transformed
medical decision making. New York: HarperCollins Publishers, Inc.
S. 3187. (2012) Food and Drug Administration Safety and Innovaton Act. U.S. 112th Congress:
Food and Drug Administration Safety and Innovation Act. Available at:
https://www.congress.gov/112/plaws/publ144/PLAW-112publ144.pdf
Schilsky, R.L. (2017) Finding the evidence in real-world evidence: Moving from data to
information to knowledge, Journal of the American College of Surgeons, 224(1), 1–7.
Schramm, R. et al. (2020) Comparing short‐term outcome after implantation of the HeartWare®
HVAD® and the Abbott® HeartMate 3®, ESC Heart Failure, 7(3), 908–914.
Sentinel Coordinating Center (2021) FDA Sentinel Initiative. Available at:
https://www.sentinelinitiative.org/ (Accessed: 10 November 2021).
Sherman, R.E. et al. (2016) Real-world evidence - what is it and what can it tell us?, New
England Journal of Medicine, 375(23), 2293–2297.
Sherman, R.E. et al. (2017) Accelerating development of scientific evidence for medical
products within the existing US regulatory framework, Nature Reviews Drug Discovery, 16(5),
297–298.
Shortliffe, E.H. (1999) The evolution of electronic medical records, Academic Medicine: Journal
of the Association of American Medical Colleges, 74(4), 414–419.
Slabodkin, G. (2021, October 4) FDA’s real-world evidence push hampered by data challenges,
‘million-dollar question’, Medtech Dive. https://www.medtechdive.com/news/fda-real-worlddata-pandemic-ivd-advamed/607584/ (Accessed: 16-Jan-2023)
Shull, J.G. (2019) Digital health and the state of interoperable electronic health records, JMIR
Medical Informatics, 7(4), e12712.
SNOMED International (2021) SNOMED International. Available at: https://www.snomed.org/
(Accessed: 1 June 2021).
Spitzer, E., Cannon, C.P. and Serruys, P.W. (2018) Should real-world evidence be incorporated
into regulatory approvals?, Expert Opinion on Drug Safety, 17(12), 1155–1159.

122
The Office of the National Coordinator for Health Information Technology. (2023, September 7)
ONC tech forum: Data quality symposium. Available at:
https://www.youtube.com/watch?v=yXXzLo6wwiU
Thompson, T. and Brailer, D. (2004) The decade of health information technology: delivering
consumer-centric and information-rich health care; framework for strategic action. Washington,
DC, United States: Department of Health & Human Services.
Unger, J. (2016) Patient income level and cancer clinical trial participation: A prospective survey
study, JAMA Oncology, 2(1), 137–139.
U.S. Department of Health and Human Services (2020) 21st Century Cures Act:
Interoperability, information blocking, and the ONC Health IT Certification Program, Federal
Register, 85(85), 25642–25959.
Waitman, D.R. (2021) PCORnet® observations: After 13 years, has “Meaningful Use” generated
data that is meaningful for research?, [PowerPoint presentation] ONC Interoperability Standards
Priorities Task Force (ISPTF), 16 April. Available at:
https://www.healthit.gov/sites/default/files/facas/2021-04-16_Presentation_on_PCORNET.pdf
(Accessed: 20 July 2021).
Wang, S.J. et al. (2003) A cost-benefit analysis of electronic medical records in primary care,
The American Journal of Medicine, 114(5), 397–403.
Wang, S.V. et al. (2019) Transparent reporting on research using unstructured electronic health
record data to generate “Real World” evidence of comparative effectiveness and safety, Drug
Safety, 42(11), 1297–1309.
Zipp, R. (2021) Medtronic pulls HeartWare HVAD pump from market amid recalls, patient
deaths, MedTechDive, 03 June. Available at: https://www.medtechdive.com/news/medtronicpulls-heartware-hvad-pump-from-market-amid-recalls-patient-death/601212/ (Accessed: 2
February 2022).
ZOLL Medical Corporation (2017) PMA P160022: Automated external defibrillator summary of
safety and effectiveness data, Washington, DC: Food and Drug Administration.
Zou, K.H. et al. (2020) Harnessing real-world data for regulatory use and applying innovative
applications, Journal of Multidisciplinary Healthcare, 13, 671–679.

123
Appendix A. Survey
The Use of EHR Data for Regulatory Purposes
Start of Block: Default Question Block
Q1.1 Dear Survey Respondent, this survey is designed to capture your views and experiences
with the use of Electronic Health Record (EHR) data for clinical evidence at your
organization. It is part of research associated with the Doctor of Regulatory Science degree at
the University of Southern California. Your responses are important to the success of the study
and are completely anonymous. At the end of the survey, there will be an option to voluntarily
participate in a follow-up phone interview. You can also receive the survey results upon request
once the study is completed. Thank you in advance for your participation. For questions related
to the survey, please contact me at hergert@usc.edu.
End of Block: Default Question Block
Start of Block: Demographics
Q2.1 What is your current position in your organization?
oVice President or higher (1)
o Director/Sr. Director (2)
o Manager/Sr. Manager (3)
oConsultant (4)
o Other (please specify) (5) __________________________________________________

124
Q2.2 With which functional area of your organization do you work?
oRegulatory Affairs (1)
oStatistics/Biostatistics (2)
oResearch/Engineering (3)
o Medical Affairs (4)
o Data Management/Data Science (5)
oClinical Affairs (6)
o Quality (7)
o Other (please specify) (8) __________________________________________________
Q2.3 Which of the following statements best describes your level of involvement in using EHR
data for generating clinical evidence in your organization?
o Minimally involved (1)
oSomewhat involved (2)
oVery involved (3)
oI am not involved (4)
Skip To: End of Survey If Which of the following statements best describes your level of involvement in using EHR
data for... = I am not involved

125
Q2.4 Which category best describes the types of product(s) your organization develops? Please
select all that apply.
▢ Pharmaceuticals (1)
▢ Biologics (2)
▢ Medical Devices (including in vitro diagnostics and Software as Medical Device) (3)
Skip To: End of Survey If Which category best describes the types of product(s) your organization develops? Please
select a... != Medical Devices (including in vitro diagnostics and Software as Medical Device)
Q2.5 Does your organization only develop Class I devices?
oYes, we only develop Class I devices. (1)
o No, we also develop Class II and/or Class III devices. (2)
Skip To: End of Survey If Does your organization only develop Class I devices? = Yes, we only develop Class I devices.
Q2.6 Does your organization market products within the US?
oYes, we market products within the US. (1)
o No, we do not market any products within the US. (2)
Skip To: End of Survey If Does your organization market products within the US? = No, we do not market any
products within the US.

126
Q2.9 In your view, at what stage is your organization with respect to adopting EHR data for
generating clinical evidence?
o We have not started discussions or activities. (1)
o We are just starting discussions. (2)
o We are experimenting with using EHR data. (3)
o We are actively using EHR data as primary source of clinical evidence to support submissions.
(4)
Skip To: End of Survey If In your view, at what stage is your organization with respect to adopting EHR data for
generating... = We have not started discussions or activities.
Q2.8 In your view, how long has your organization been investigating/engaging in the use of
EHR data to generate clinical evidence?
oThe organization is not currently considering the use of such an approach (1)
oLess than 3 years (2)
oBetween 3 to 10 years (3)
o More than 10 years (4)
o Don't know (5)

127
Q2.10 Looking back over the past 12 months, what proportion of 501(k)s and PMAs submitted
to the FDA used EHR data as the primary source of clinical evidence to support the primary
endpoint?
None (1)
Less than
half (a few)
(2)
About a
half (3)
More than
half (most)
(4)
All (5) Don't
know (6)
510(k) (1) o o o o o o
PMA (2) o o o o o o
Q2.7
What is the size of your organization?
oless than 500 employees (1)
o501 - 5000 employees (2)
ogreater than 5000 employees (3)
End of Block: Demographics
Start of Block: External Pressures

128
Q3.1 In your view, the following influenced your organization to advance the use of EHR data
for generating clinical evidence.
Strongly
agree (1)
Somewhat
agree (2)
Neither
agree
nor
disagree
(3)
Somewhat
disagree
(4)
Strongly
disagree
(5)
Don't
know
(6)
The passage
of the 21st
Century
Cures Act (1)
o o o o o o
The FDA's
visible
interest
towards to
use of EHR
data (2)
o o o o o o
The release
of FDA
guidance
documents
(3)
o o o o o o
Competition
with other
medical
device
companies
(4)
o o o o o o
Patients (5) o o o o o o
Confidence
in the quality
of EHR data
(6)
o o o o o o
Potential for
reduction in
cost (7) o o o o o o

129
Access to
reliable
generalizable
data (8)
o o o o o o
Potential for
reduction in
time (9) o o o o o o
Other
(please
elaborate)
(10)
o o o o o o

130
Q3.2 In your view, how helpful are the following FDA documents to your organization when
advancing the use of EHR data for clinical evidence?
Not
helpful
(1)
Helpful
(2)
Very
helpful
(3)
Not
used
(4)
Design Considerations for Pivotal Clinical
Investigations for Medical Devices: Guidance
for Industry, Clinical Investigators, Institutional
Review Boards and FDA Staff
https://www.fda.gov/media/87363/download
(1)
o o o o
Electronic Source Data in Clinical
Investigations: Guidance for Industry
https://www.fda.gov/media/85183/download
(2)
o o o o
Use of Real-World Evidence to Support
Regulatory Decision-Making for Medical
Devices: Guidance for Industry and Food and
Drug Administration Staff
https://www.fda.gov/media/99447/download
(3)
o o o o
Use of Electronic Health Record Data in Clinical
Investigations Guidance for Industry
https://www.fda.gov/media/97567/download
(4)
o o o o
Examples of Real-World Evidence (RWE) Used
in Medical Device Regulatory Decisions
https://www.fda.gov/media/146258/download
(5)
o o o o
Other (please specify) (6) o o o o

131
Q3.3 In your view, the FDA's CDRH encourages the use of EHR data to generate clinical
evidence.
oAgree; please elaborate. (1) __________________________________________________
o Neither agree nor disagree; please elaborate. (2)
__________________________________________________
o Disagree; please elaborate. (3) __________________________________________________
o Don't know (4)
End of Block: External Pressures
Start of Block: Technology Benefits

132
Q4.1 In your view, using EHR data for generating clinical evidence provides the following
advantages when compared to using data from randomized controlled trials.
Strongly
agree (1)
Somewhat
agree (2)
Neither
agree nor
disagree
(3)
Somewhat
disagree
(4)
Strongly
disagree
(5)
Don't
know (6)
Reduction in
cost (1) o o o o o o
Reduction in
time to
submission (2) o o o o o o
Reduction in
workforce (8) o o o o o o
Streamlining
statistical
analyses (3) o o o o o o
Real-world
generalizability
of the data (4) o o o o o o
Other (please
elaborate) (6) o o o o o o
Q4.2 Do you think the use of EHR data will reduce the number of randomized controlled trials?
o Definitely (1)
oPossibly (2)
oProbably (3)
o Definitely not (4)
o Don't know (5)

133
End of Block: Technology Benefits
Start of Block: Organizational Readiness
Q5.1 How accessible is EHR data to your organization?
o Not accessible; please elaborate. (1)
__________________________________________________
oSomewhat accessible; please elaborate. (2)
__________________________________________________
oVery accessible; please elaborate. (3)
__________________________________________________
o Don't know (4)
Q5.2 Do you think EHR data is sufficient, in its current state, to generate clinical evidence that
can support a primary endpoint within a regulatory submission?
o Definitely (1)
oPossibly (2)
oProbably (3)
o Definitely not (4)
o Don't know (5)

134
Q5.3 Please rank the following from most concerning (1) to least concerning (7) when using
EHR data for regulatory purposes?
______ Missing data (1)
______ Lack of available data specific to the end points, including the intended population (2)
______ Inconsistency of data across datasets (3)
______ Lack of easy access to unstructured data (4)
______ Lack of Unique Device Identification (UDI)/inability to link a device to the records (5)
______ Inherent bias within the data (6)
______ The need for data cleaning prior to analysis (7)
Q5.4 Please provide any other concerning factors that are not listed in the previous question:
________________________________________________________________
________________________________________________________________
________________________________________________________________

135
Q5.5 Do you agree with the following statements regarding access to EHR data as compared to
access to data from randomized clinical trials?
Strongly
agree (1)
Somewhat
agree (2)
Neither
agree nor
disagree
(3)
Somewhat
disagree
(4)
Strongly
disagree
(5)
Don't
know
(6)
The transfer
of EHR data
between
vendors and
the
organization
is more
difficult. (1)
o o o o o o
It is more
costly to
obtain EHR
data. (2)
o o o o o o
Ensuring
data privacy
when
transferring
EHR data is
more
difficult. (3)
o o o o o o
Contract
negotiation
with the data
vendor
requires
more time.
(4)
o o o o o o

136
Compatibility
of systems is
more
inefficient.
(5)
o o o o o o
Other
(please
specify) (6) o o o o o o

137
Q5.6 Do you agree with the following statements regarding analyzing EHR data as compared to
analyzing data from randomized clinical trials?
Strongly
agree (1)
Somewhat
agree (2)
Neither
agree
nor
disagree
(3)
Somewhat
disagree
(4)
Strongly
disagree
(5)
Don't
know
(6)
Identifying
appropriate
analyses takes
more time. (1)
o o o o o o
Meeting
regulatory
grade quality is
more difficult.
(2)
o o o o o o
Incorporating
into the
organization's
already
existing clinical
procedures is
more
challenging. (3)
o o o o o o
Incorporating
interoperability
into the
organization's
already
existing
infrastructure
is more
challenging. (4)
o o o o o o

138
Reformatting
and cleaning
the data
requires more
time. (5)
o o o o o o
Other (please
specify) (6) o o o o o o
Q5.7 Which functional areas in your organization need new skillsets and/or processes to advance
the use of EHR data? Select all that apply and elaborate as applicable.
▢ Regulatory Affairs; please elaborate below (1)
▢ Statistics/Biostatistics; please elaborate below (2)
▢ Clinical Affairs; please elaborate below (3)
▢ Data Management/Science; please elaborate below (4)
▢ Program Management; please elaborate below (5)
▢ Medical Affairs; please elaborate below (6)
▢ Other (please specify) (7) __________________________________________________
▢ Please comment on skillsets and/or processes needed. (8)
__________________________________________________

139
Q5.8 Your organization's goals and objectives for product development include the use of EHR
data for generating clinical evidence.
oYes (1)
o No (2)
o Don't know (3)
Q5.9 To what extent have you heard senior executives of your organization speaking about using
EHR data to generate clinical evidence?
o Never (1)
oRarely (2)
oSomewhat frequently (3)
oVery frequently (4)
End of Block: Organizational Readiness
Start of Block: Final
Q6.1 What lessons have you or your organization learned from investigating the use of EHR data
for generating clinical evidence?
________________________________________________________________
________________________________________________________________
________________________________________________________________
________________________________________________________________
________________________________________________________________

140
Q6.2 Please provide any additional insights regarding the benefits of using EHR data for
regulatory purposes.
________________________________________________________________
________________________________________________________________
________________________________________________________________
________________________________________________________________
________________________________________________________________
Q6.3 Thank you for participating in this survey. Are you interested in participating in a followup phone interview? If you choose Yes, personal information will be requested that will allow
me to contact you. Your contact information will only be viewed by me and will not be
shared. If you choose No, the survey will end.
oYes (1)
o No (2)
Display This Question:
If Thank you for participating in this survey. Are you interested in participating in a follow-up p... = Yes
Q6.4 Please provide your name:
________________________________________________________________
Display This Question:
If Thank you for participating in this survey. Are you interested in participating in a follow-up p... = Yes

141
Q6.5 Email address:
________________________________________________________________
Display This Question:
If Thank you for participating in this survey. Are you interested in participating in a follow-up p... = Yes
Q6.6 Phone number:
________________________________________________________________
End of Block: Final

142
Appendix B. TOE Cohort Data Tables (n = 46)
Questions Exploring Demographics
Q2.1 - What is your current position in your organization?
# Answer % Count
1 Vice President or higher 28.26% 13
2 Director/Sr. Director 26.09% 12
3 Manager/Sr. Manager 19.57% 9
4 Consultant 8.70% 4
5 Other (please specify) 17.39% 8
Total 100% 46
Other (please specify) = 0 responses
Q2.2 - With which functional area of your organization do you work?
# Answer % Count
1 Regulatory Affairs 21.28% 10
2 Statistics/Biostatistics 2.13% 1
3 Research/Engineering 8.51% 4
4 Medical Affairs 2.13% 1
5 Data Management/Data Science 8.51% 4
6 Clinical Affairs 40.43% 19
7 Quality 6.38% 3
8 Other (please specify) 10.64% 5
Total 100% 47
Other (please specify) = 0 responses

143
Q2.3 - Which of the following statements best describes your level of involvement in using
EHR data for generating clinical evidence in your organization?
# Answer % Count
1 Minimally involved 25.53% 12
2 Somewhat involved 48.94% 23
3 Very involved 25.53% 12
Total 100% 47
Q2.4 - Which category best describes the types of product(s) your organization develops?
Please select all that apply.
# Answer % Count
1 Pharmaceuticals 8.62% 5
2 Biologics 10.34% 6
3 Medical Devices (including in vitro diagnostics and Software as Medical
Device) 81.03% 47
Total 100% 58
Q2.5 - Does your organization only develop Class I devices?
# Answer % Count
2 No, we also develop Class II and/or Class III devices. 100.00% 47

144
Q2.6 - Does your organization market products within the US?
# Answer % Count
1 Yes, we market products within the US. 100.00% 47
Total 100% 47
Q2.7 - What is the size of your organization?
# Answer % Count
1 less than 500 employees 26.67% 12
2 501 - 5000 employees 15.56% 7
3 greater than 5000 employees 57.78% 26
Total 100% 45
Q2.8 - In your view, how long has your organization been investigating/engaging in the use
of EHR data to generate clinical evidence?
# Answer % Count
1 The organization is not currently considering the use of such an approach 4.44% 2
2 Less than 3 years 31.11% 14
3 Between 3 to 10 years 46.67% 21
4 More than 10 years 8.89% 4
5 Don't know 8.89% 4
Total 100% 45

145
Q2.9 - In your view, at what stage is your organization with respect to adopting EHR data
for generating clinical evidence?
# Answer % Count
2 We are just starting discussions. 26.67% 12
3 We are experimenting with using EHR data. 26.67% 12
4 We are actively using EHR data as primary source of clinical evidence to
support submissions. 46.67% 21
Total 100% 45
Q2.10 - Looking back over the past 12 months, what proportion of 501(k)s and PMAs
submitted to the FDA used EHR data as the primary source of clinical evidence to support
the primary endpoint?
# Questio
n
None
Less
than
half (a
few)
Abou
t a
half
More
than
half
(most
)
All Don't
know
Tota
l
1 510(k) 25.58
%
1
1
25.58
%
1
1
2.33
% 1 9.30
% 4 11.63
% 5 25.58
%
1
1 43
2 PMA 29.55
%
1
3
27.27
%
1
2
2.27
% 1 2.27
% 1 9.09% 4 29.55
%
1
3 44

146
Questions Exploring External Pressures
Q3.1 - In your view, the following influenced your organization to advance the use of EHR
data for generating clinical evidence.
# Field Minimum Maximum Mean Std
Deviation Variance Count
1 The passage of the 21st
Century Cures Act 1.00 6.00 3.49 1.85 3.41 43
2
The FDA's visible
interest towards to use
of EHR data
1.00 6.00 2.40 1.48 2.19 43
3 The release of FDA
guidance documents 1.00 6.00 2.40 1.45 2.10 43
4
Competition with other
medical device
companies
1.00 6.00 2.86 1.41 1.98 43
5 Patients 1.00 6.00 3.05 1.46 2.14 42
6 Confidence in the
quality of EHR data 1.00 6.00 2.77 1.38 1.90 43
7 Potential for reduction
in cost 1.00 6.00 2.00 1.56 2.42 43
8 Access to reliable
generalizable data 1.00 6.00 2.17 1.41 2.00 42
9 Potential for reduction
in time 1.00 6.00 1.86 1.34 1.79 43
10 Other (please elaborate) 1.00 6.00 3.91 2.15 4.63 11
# Question
Stron
gly
agree
Somew
hat
agree
Neith
er
agree
nor
disag
ree
Somew
hat
disagre
e
Stron
gly
disagr
ee
Don't
know
Tot
al
1
The
passage
of the
21st
Century
Cures
Act
18.60
% 8 13.95
% 6 27.91
%
1
2 6.98% 3 4.65
% 2 27.9
1%
1
2 43

147
# Question
Stron
gly
agree
Somew
hat
agree
Neith
er
agree
nor
disag
ree
Somew
hat
disagre e
Stron
gly
disagr
ee
Don't
know
Tot
al
2
The
FDA's
visible
interest
towards
to use of
EHR
data
30.23 %
13
39.53 %
17
11.63 %
5 4.65%
2 6.98 %
3 6.98 %
3 43
3
The
release of
FDA
guidance
documen
ts
25.58 %
11
48.84 %
21
6.98 %
3 4.65%
2 6.98 %
3 6.98 %
3 43
4
Competit
ion with
other
medical
device
compani
es
13.95 %
6 34.88 %
15
25.58 %
11 9.30%
4 9.30 %
4 6.98 %
3 43
5 Patients 16.67 %
7 19.05 %
8 33.33 %
14
14.29 %
6 7.14 %
3 9.52 %
4 42
6
Confiden
ce in the
quality of
EHR
data
13.95 %
6 37.21 %
16
27.91 %
12 9.30%
4 2.33 %
1 9.30 %
4 43
7
Potential
for
reduction
in cost
51.16 %
22
32.56 %
14
4.65 %
2 0.00%
0 0.00 %
0 11.6
3%
5 43
8
Access to
reliable
generaliz
able data
38.10 %
16
38.10 %
16
9.52 %
4 4.76%
2 2.38 %
1 7.14 %
3 42
9
Potential
for
reduction
in time
48.84 %
21
41.86 %
18
0.00 %
0 0.00%
0 2.33 %
1 6.98 %
3 43

148
# Question
Stron
gly
agree
Somew
hat
agree
Neith
er
agree
nor
disag
ree
Somew
hat
disagre
e
Stron
gly
disagr
ee
Don't
know
Tot
al
1
0
Other
(please
elaborate
)
18.18
% 2 27.27
% 3 0.00
% 0 0.00% 0 9.09
% 1 45.4
5% 5 11

149
Q3.2 - In your view, how helpful are the following FDA documents to your organization
when advancing the use of EHR data for clinical evidence?
# Field Minimu
m
Maxim
um
Mea
n
Std
Deviati
on
Varian
ce
Cou
nt
1
Design Considerations for Pivotal
Clinical Investigations for Medical
Devices: Guidance for Industry,
Clinical Investigators, Institutional
Review Boards and FDA Staff
https://www.fda.gov/media/87363/d
ownload
1.00 4.00 2.50 0.85 0.73 42
2
Electronic Source Data in Clinical
Investigations: Guidance for
Industry
https://www.fda.gov/media/85183/d
ownload
1.00 4.00 2.43 0.76 0.58 42
3
Use of Real-World Evidence to
Support Regulatory DecisionMaking for Medical Devices:
Guidance for Industry and Food and
Drug Administration Staff
https://www.fda.gov/media/99447/d
ownload
1.00 4.00 2.43 0.66 0.44 42
4
Use of Electronic Health Record
Data in Clinical Investigations
Guidance for Industry
https://www.fda.gov/media/97567/d
ownload
1.00 4.00 2.46 0.67 0.44 41
5
Examples of Real-World Evidence
(RWE) Used in Medical Device
Regulatory Decisions
https://www.fda.gov/media/146258/
download
1.00 4.00 2.48 0.70 0.49 42
6 Other (please specify) 2.00 4.00 3.43 0.90 0.82 7

150
# Question
Not
helpf
ul
Helpf
ul
Very
helpf
ul
Not
used
Tot
al
1
Design Considerations for Pivotal
Clinical Investigations for Medical
Devices: Guidance for Industry,
Clinical Investigators, Institutional
Review Boards and FDA Staff
https://www.fda.gov/media/87363/d
ownload
7.14
% 3 52.38
%
2
2
23.81
%
1
0
16.67
% 7 42
2
Electronic Source Data in Clinical
Investigations: Guidance for
Industry
https://www.fda.gov/media/85183/d
ownload
4.76
% 2 59.52
%
2
5
23.81
%
1
0
11.90
% 5 42
3
Use of Real-World Evidence to
Support Regulatory DecisionMaking for Medical Devices:
Guidance for Industry and Food and
Drug Administration Staff
https://www.fda.gov/media/99447/d
ownload
4.76
% 2 52.38
%
2
2
38.10
%
1
6
4.76
% 2 42
4
Use of Electronic Health Record
Data in Clinical Investigations
Guidance for Industry
https://www.fda.gov/media/97567/d
ownload
2.44
% 1 56.10
%
2
3
34.15
%
1
4
7.32
% 3 41
5
Examples of Real-World Evidence
(RWE) Used in Medical Device
Regulatory Decisions
https://www.fda.gov/media/146258/
download
4.76
% 2 50.00
%
2
1
38.10
%
1
6
7.14
% 3 42
6 Other (please specify) 0.00
% 0 28.57
% 2 0.00
% 0 71.43
% 5 7

151
Q3.3 - In your view, the FDA's CDRH encourages the use of EHR data to generate clinical
evidence.
# Answer % Count
1 Agree; please elaborate. 32.50% 13
2 Neither agree nor disagree; please elaborate. 42.50% 17
3 Disagree; please elaborate. 7.50% 3
4 Don't know 17.50% 7
Total 100% 40
Q3.3_1_TEXT - Agree; please elaborate.
Agree; please elaborate. - Text
In my experience it depends a lot on the review group. Some are receptive and some are not
Industry working diligently on cloud base shareable data for patient information. Our Midmark
Diagnostic group has been enhancing our vitals sign devices, ecg devices for communicating
with hospital EMR and data transfer. It is crucial to ensure these data transfer well and
benefiting clinical investigation.
Agency has expectations that health information be documented in manner that is reliable,
auditable, and be computer accessible.
Based on the guidance documents released and engagement in informal and formal discussions
I agree that CDRH are encouraging of RWE
agree but this is still developing and requires a lot of discussion and alignment. Also, may not
fulfill all global evidence needs
Topic presented at peer group conference by regulatory panel
Diversity in clinical trials, better patient representation comes from real world evidence, and
determining impact on patient populations

152
Q3.3_2_TEXT - Neither agree nor disagree; please elaborate.
Neither agree nor disagree; please elaborate. - Text
CDRH is getting there. while they have guidance documents released on use of RWE, they
still struggle to understand and accept the limitations of it. As such, RWE for device approvals
is lagging.
It's been a while since I was directly involved. They were ambivalent to suspect years ago, but
may have changed their attitude
Ambiguity still exists in the provided guidance.
FDA generally acknowledges data options available, but I’ve not observed a strong emphasis
or encouragement from the Agency to use EHR data. In my experience, the Agency requires
“valid scientific evidence” in support of submissions and largely leaves it up to the submitter
in deciding the best source. We’ve used RWE in a number of submissions, with little
pushback from FDA - but is generally more readily accepted when it’s in conformance to
protocol controls & represents high level of evidence (Level 1 or Level 2) of support for
510(k) submissions. This is generally always the case for Class III PMA products, since safety
& effectiveness of the technology must be proven. I’m only aware of one (1) instance where
FDA allowed usage of published data from medical literature for a DeNovo submission,
outside of a protocol controlled randomized clinical study. The use of EHR data normally has
to be very specific & in conformity to established requirements for the specific technology,
which may/may not exist within a general EHR.
Implementation of their guidance may fall off deeper in the org at the reviewer leve
FDA does not openly encourage however it depends on industry appetite to find capable
partners for extraction of EHR data
We interpret guidance as "use what is fit for purpose" among traditional prospective and
retrospective studies, RWD, EHR, other sources. It is one piece of the puzzle, one tool
available.
Depends on the product
I believe it is device-specific but we had mixed experiences so far. EHR data should have been
sufficient for some projects but FDA insisted on RCTs. EHR data seems to be a nice-have but
not sufficient.
Pay more attention to CDER guidelines
The FDA as a regulatory body offers guidance with such information. In doing so they are
trying to convince manufacturers to follow best practices. The hope is that organizations will
respect pharmacovigilance and the integrity of this data; as there are still ways present to
manipulate data to one’s advantage. As humans in business, most are concerned with having
an advantage and the bad still creep through sometimes. This is the risk we balance.
It honestly depends on what the scope of the submission has been for. Though it has been
encouraged, the usefulness of the data is often brought to question. RWE is useful in
understanding off label use and potential for label expansions. Additionally some claims are
better served being evaluated with RWE given there is the potential for less bias.
Some data capture systems lend themselves to data transfer more readily than others

153
Q3.3_3_TEXT - Disagree; please elaborate.
Disagree; please elaborate. - Text
With regards to IVDs, the agency has not been as open minded about moving past traditional
comparison studies, which are almost completely absent from EHR data.
I feel the agency is still struggling with how to use the data for real world applications. They
have approached the thory but the practice is not consistant
Presenting any data besides clinical study generated is a struggle.

154
Questions Exploring the Benefits of
Q4.1 - In your view, using EHR data for generating clinical evidence provides the following
advantages when compared to using data from randomized controlled trials.
# Field Minimum Maximum Mean Std
Deviation Variance Count
1 Reduction in cost 1.00 5.00 1.63 0.90 0.82 41
2 Reduction in time to
submission 1.00 5.00 1.73 0.91 0.83 41
3 Reduction in workforce 1.00 6.00 2.42 1.14 1.29 40
4 Streamlining statistical
analyses 1.00 6.00 2.98 1.37 1.88 41
5
Real-world
generalizability of the
data
1.00 3.00 1.71 0.63 0.40 41
6 Other (please elaborate) 1.00 6.00 3.17 2.03 4.14 6
# Question
Stron
gly
agree
Somew
hat
agree
Neith
er
agree
nor
disag
ree
Somew
hat
disagre
e
Stron
gly
disagr
ee
Don't
know
Tot
al
1 Reduction
in cost
56.10
%
2
3
31.71
%
1
3
7.32
% 3 2.44% 1 2.44
% 1 0.00
% 0 41
2
Reduction
in time to
submission
48.78
%
2
0
36.59
%
1
5
9.76
% 4 2.44% 1 2.44
% 1 0.00
% 0 41
3
Reduction
in
workforce
17.50
% 7 47.50
%
1
9
17.50
% 7 12.50
% 5 2.50
% 1 2.50
% 1 40
4
Streamlini
ng
statistical
analyses
19.51
% 8 17.07
% 7 26.83
%
1
1
21.95
% 9 12.20
% 5 2.44
% 1 41
5
Real-world
generaliza
bility of
the data
39.02
%
1
6
51.22
%
2
1
9.76
% 4 0.00% 0 0.00
% 0 0.00
% 0 41
6
Other
(please
elaborate)
16.67
% 1 50.00
% 3 0.00
% 0 0.00% 0 0.00
% 0 33.3
3% 2 6

155
Q4.2 - Do you think the use of EHR data will reduce the number of randomized controlled
trials?
# Field Minimum Maximum Mean Std
Deviation Variance Count
1
Do you think the use of
EHR data will reduce the
number of randomized
controlled trials?
1.00 5.00 2.76 1.02 1.04 42
# Answer % Count
1 Definitely 9.52% 4
2 Possibly 35.71% 15
3 Probably 26.19% 11
4 Definitely not 26.19% 11
5 Don't know 2.38% 1
Total 100% 42

156
Questions Exploring Organizational Readiness
Q5.1 - How accessible is EHR data to your organization?
# Field Minimum Maximum Mean Std
Deviation Variance Count
1
How accessible is EHR
data to your organization?
- Selected Choice
1.00 4.00 2.21 0.73 0.53 38
# Answer % Count
1 Not accessible; please elaborate. 10.53% 4
2 Somewhat accessible; please elaborate. 65.79% 25
3 Very accessible; please elaborate. 15.79% 6
4 Don't know 7.89% 3
Total 100% 38
Q5.1_1_TEXT - Not accessible; please elaborate.
Not accessible; please elaborate. - Text
Limited use essentially through partnerships for retrospective studies
required detailed process, contracting, privacy review and some data is still not available
Q5.1_2_TEXT - Somewhat accessible; please elaborate.
Somewhat accessible; please elaborate. - Text
If needed, we can work with partners who possess EHR, either as industry-sponsored studies
or investigator-initiated trials.
Appropriate contracts with healthcare institutions can provide access.
We always collaborate with investigators at the owning institution
EHR data is used for two purposes: 1. Generate RWE for our devices to get approvals of new
indications or for continuing to understand the safety and efficacy profile of the device. 2. For
ensuring data integrity during routine monitoring of clinical studies. Both of these rely on the
investigational sites providing access to the EHR
we were able to get anonymous patient data from some doctors years ago
You have to know the right individuals within a given system, for data access. Institutional
clearance for use is required. Also, if using multiple sites for sources of data, not all
institutions use the same data systems, so firewalls can be a logistical problem. Data

157
ownership, data security, threat modeling protocols, and cybersecurity concerns are all difficult
issues (and more) which must be dealt with when using EHR data.
We are working with an outside vendor
privacy regulations around the world limit access
While we don't currently have EHR, we do have national chargemaster RWD, that has been
digested into some exploratory tools accessible to crossfunctional (Clinical, RnD, Stats,
Commercial)
It is accessible through institutions we have partnerships with but a limited amount. We don't
typically have access to other health records for the patient and just partnerships with specific
institutions for data specifically on our products when they have surgeries with the products
from procedure to discharge so we are limited compared to healthcare companies that also own
insurance companies. For example CVS or United Healthgroup also own some medical benefit
companies so they have access to data for their patients for research.
Difficult with new privacy laws and the fact that ever EHR is different.
Has been accessible only when as part of data collection in ongoing clinical trials
Some data is available for our indication but not a lot
Is accessible to those that have the need to know.
It depends on the consents and how much we can deidentify. Completely privacy and HIPA
limitations.
It is accessible but there is a lot of work and time that goes into harnessing that data. So,
although it is there, it is not easily obtained for use in a submission.
There are obstacles in obtaining partnerships and available data can be limiting
Q5.1_3_TEXT - Very accessible; please elaborate.
Very accessible; please elaborate. - Text
We (Midmark) built Vitals signs device and ECG devices and other diagnostic products
communicating with EMR such as Epic and Allscipts
Because of the partnerships and investment that we have made as an organization with
healthcare institutions.
Our product is integrated into the EMR
We have access to EHR data and work on getting more, but it is challenging to gain access
from the data that didn't originate at our institution.
If the data is available, we would be able access it

158
Q5.2 - Do you think EHR data is sufficient, in its current state, to generate clinical
evidence that can support a primary endpoint within a regulatory submission?
# Field Minimum Maximum Mean Std
Deviation Variance Count
1
Do you think EHR data is
sufficient, in its current
state, to generate clinical
evidence that can support
a primary endpoint within
a regulatory submission?
1.00 4.00 2.62 1.05 1.11 39
# Answer % Count
1 Definitely 12.82% 5
2 Possibly 43.59% 17
3 Probably 12.82% 5
4 Definitely not 30.77% 12
5 Don't know 0.00% 0
Total 100% 39

159
Q5.3 - Please rank the following from most concerning (1) to least concerning (7) when
using EHR data for regulatory purposes?
# Field Minimum Maximum Mean Std
Deviation Variance Count
1 Missing data 1.00 7.00 2.95 1.69 2.84 38
2
Lack of available data
specific to the end points,
including the intended
population
1.00 7.00 2.37 1.48 2.18 38
3 Inconsistency of data
across datasets 1.00 6.00 3.16 1.46 2.13 38
4 Lack of easy access to
unstructured data 1.00 7.00 5.11 1.41 1.99 38
5
Lack of Unique Device
Identification
(UDI)/inability to link a
device to the records
1.00 7.00 4.45 2.02 4.09 38
6 Inherent bias within the
data 1.00 7.00 5.16 1.90 3.61 38
7 The need for data
cleaning prior to analysis 1.00 7.00 4.82 1.80 3.26 38

160
# Question
1
2
3
4
5
6
7 To
tal
1 Missing
data
26.3
2%
10
15.7
9%
6 23.6
8%
9 21.0
5%
8 2.63 %
1 5.26 %
2 5.26 %
2 38
2
Lack of
available
data
specific
to the end
points,
including
the
intended
population
36.8
4%
14
26.3
2%
10
15.7
9%
6 10.5
3%
4 7.89 %
3 0.00 %
0 2.63 %
1 38
3
Inconsist
ency of
data
across
datasets
13.1
6%
5 23.6
8%
9 26.3
2%
10
15.7
9%
6 13.1
6%
5 7.89 %
3 0.00 %
0 38
4
Lack of
easy
access to
unstructu
red data
2.63 %
1 2.63 %
1 5.26 %
2 21.0
5%
8 23.6
8%
9 28.9
5%
11
15.7
9%
6 38
5
Lack of
Unique
Device
Identifica
tion
(UDI)/ina
bility to
link a
device to
the
records
13.1
6%
5 13.1
6%
5 2.63 %
1 13.1
6%
5 18.4
2%
7 23.6
8%
9 15.7
9%
6 38
6
Inherent
bias
within
the data
2.63 %
1 10.5
3%
4 13.1
6%
5 7.89 %
3 10.5
3%
4 18.4
2%
7 36.8
4%
14 38

161
# Question 1 2 3 4 5 6 7 To
tal
7
The need
for data
cleaning
prior to
analysis
5.26
% 2 7.89
% 3 13.1
6% 5 10.5
3% 4 23.6
8% 9 15.7
9% 6 23.6
8% 9 38

162
Q5.4 - Please provide any other concerning factors that are not listed in the previous
question:
Please provide any other concerning factors that are not listed in the previous question:
Many clinics and hospitals do not possess the personnel dedicated to the extraction of the
needed data
Inability to link different EHR systems to maximize the available pool of data - big data
analysis is currently not possible
Technically difficulty for using the data
Data security, protection, and cybersecurity risks which are to be accounted for when using
electronic data. Data ownership & boundaries of use concerns largely exist amongst
institutions. HIPAA concerns almost always exist, and demonstration of being SOC2 &
HIPAA compliant and potentially certified are issues which normally arise when considering
EHR data use (as well as ownership of that data & if/how it can be subsequently used).
Adequate longitudinal data
Closed versus open systems--potential to complicate longitudinal follow-up especially in
mixed IP-OP settings and longer term events or surrogate outcomes
Rank order from above list (most to least): 2,3,4,1,6,7,5
Provenance
privacy challenges
My observation of EHRs from medical facilities indicates frequent mistakes and shortcomings,
possibly due to a focus on making sure the record includes everything needed for
insurance/billing as opposed to personalizing the record for a specific patient.
Data integrity and software verification for EHR software

163
Q5.5 - Do you agree with the following statements regarding access to EHR data as
compared to access to data from randomized clinical trials?
# Field Minimum Maximum Mean Std
Deviation Variance Count
1
The transfer of EHR data
between vendors and the
organization is more
difficult.
1.00 6.00 2.05 1.22 1.50 40
2 It is more costly to obtain
EHR data. 1.00 6.00 3.55 1.05 1.10 40
3
Ensuring data privacy
when transferring EHR
data is more difficult.
1.00 6.00 2.90 1.41 1.99 40
4
Contract negotiation with
the data vendor requires
more time.
1.00 6.00 2.70 1.45 2.11 40
5 Compatibility of systems
is more inefficient. 1.00 6.00 2.67 1.42 2.02 40
6 Other (please specify) 2.00 6.00 5.20 1.60 2.56 5
# Question
Stron
gly
agree
Somew
hat
agree
Neith
er
agree
nor
disag
ree
Somew
hat
disagre
e
Stron
gly
disagr
ee
Don't
know
Tot
al
1
The
transfer
of EHR
data
between
vendors
and the
organizati
on is
more
difficult.
35.00
%
1
4 47.50% 1
9
5.00
% 2 7.50% 3 0.00
% 0 5.00
% 2 40
2
It is more
costly to
obtain
EHR
data.
2.50
% 1 10.00% 4 40.00
%
1
6 27.50% 1
1
17.50
% 7 2.50
% 1 40

164
# Question
Stron
gly
agree
Somew
hat
agree
Neith
er
agree
nor
disag
ree
Somew
hat
disagre
e
Stron
gly
disagr
ee
Don't
know
Tot
al
3
Ensuring
data
privacy
when
transferri
ng EHR
data is
more
difficult.
17.50
% 7 27.50% 1
1
22.50
% 9 17.50% 7 10.00
% 4 5.00
% 2 40
4
Contract
negotiatio
n with the
data
vendor
requires
more
time.
17.50
% 7 40.00% 1
6
20.00
% 8 10.00% 4 2.50
% 1 10.00
% 4 40
5
Compatib
ility of
systems
is more
inefficien
t.
17.50
% 7 40.00% 1
6
20.00
% 8 12.50% 5 0.00
% 0 10.00
% 4 40
6
Other
(please
specify)
0.00
% 0 20.00% 1 0.00
% 0 0.00% 0 0.00
% 0 80.00
% 4 5

165
Q5.6 - Do you agree with the following statements regarding analyzing EHR data as
compared to analyzing data from randomized clinical trials?
# Field Minimum Maximum Mean Std
Deviation Variance Count
1 Identifying appropriate
analyses takes more time. 1.00 6.00 2.67 1.37 1.86 39
2 Meeting regulatory grade
quality is more difficult. 1.00 6.00 2.03 1.05 1.10 39
3
Incorporating into the
organization's already
existing clinical
procedures is more
challenging.
1.00 6.00 2.72 1.43 2.05 39
4
Incorporating
interoperability into the
organization's already
existing infrastructure is
more challenging.
1.00 6.00 2.44 1.13 1.27 39
5
Reformatting and
cleaning the data requires
more time.
1.00 6.00 2.38 1.21 1.47 39
6 Other (please specify) 2.00 6.00 5.33 1.49 2.22 6
# Question
Stron
gly
agree
Somew
hat
agree
Neith
er
agree
nor
disag
ree
Somew
hat
disagre
e
Stron
gly
disagr
ee
Don't
know
Tot
al
1
Identifying
appropriat
e analyses
takes more
time.
17.95
% 7 35.90
%
1
4
25.64
%
1
0
10.26
% 4 2.56
% 1 7.69
% 3 39
2
Meeting
regulatory
grade
quality is
more
difficult.
30.77
%
1
2
51.28
%
2
0
7.69
% 3 7.69% 3 0.00
% 0 2.56
% 1 39

166
3
Incorporati
ng into the
organizati
on's
already
existing
clinical
procedures
is more
challengin
g.
17.95
% 7 38.46
%
1
5
17.95
% 7 12.82
% 5 5.13
% 2 7.69
% 3 39
4
Incorporati
ng
interopera
bility into
the
organizati
on's
already
existing
infrastruct
ure is
more
challengin
g.
15.38
% 6 46.15
%
1
8
28.21
%
1
1 5.13% 2 0.00
% 0 5.13
% 2 39
5
Reformatti
ng and
cleaning
the data
requires
more time.
23.08
% 9 38.46
%
1
5
25.64
%
1
0 7.69% 3 0.00
% 0 5.13
% 2 39
6
Other
(please
specify)
0.00
% 0 16.67
% 1 0.00
% 0 0.00% 0 0.00
% 0 83.3
3% 5 6
Q5.7 - Which functional areas in your organization need new skillsets and/or processes to
advance the use of EHR data? Select all that apply and elaborate as applicable.
# Answer % Count
1 Regulatory Affairs; please elaborate below 17.61% 25
2 Statistics/Biostatistics; please elaborate below 16.20% 23
3 Clinical Affairs; please elaborate below 19.01% 27
4 Data Management/Science; please elaborate below 19.72% 28

167
5 Program Management; please elaborate below 5.63% 8
6 Medical Affairs; please elaborate below 7.75% 11
7 Other (please specify) 4.23% 6
8 Please comment on skillsets and/or processes needed. 9.86% 14
Total 100% 142
Q5.7_7_TEXT - Other (please specify)
Other (please specify) - Text
I think most functions are not familiar with the limitations or opportunities of EHR data.
There is a learning curve of what is appropriate use.
There is lot of refinement that is needed in strategy when a RWE project is undertaken and
listing out risks and plans to address them as scientifically as possible without affecting data
integrity is important throughout.
all functional areas as this would be something that many are not well versed on throughout
the organization so harmonization with processes and understanding of the advantages and
existing gaps would be beneficial for everyone.
Legal and Privacy. The other functions have easily adapted.
The use of EHR to us was totally new, so we have all been learning along the way. There were
many unforeseen challenges that we had to work to overcome. We will continue to learn
through the FDA review which will hopefully better prepare us for next time.
Q5.7_8_TEXT - Please comment on skillsets and/or processes needed.
Please comment on skillsets and/or processes needed. - Text
automated processes for data transfer, identification of appropriate datasets, negotiation with
regulators
RA - FDA regulations monitoring Clinical Affairs - understand the data structure compared to
traditional data Data management - need to maintain database (HIPPA) and/or creating
different data management system
People will need to understand the capabilities, and limitations, in order to be confident using
EHR data vs. entirely new data from targeted trials
Current data monitoring, control, and use skills are necessary. SOC2 & HIPAA compliance
and certification requirements are generally needed for liability exposure reduction. Data
control, security controls, threat modeling controls, and cybersecurity controls are areas which
must be duly considered - which many organizations are ill-equipped to address. Normally,
this includes entrepreneurial, small start-ups, and even mid-cap development organizations.
Larger, more sophisticated, deep-pockets organizations may be in better positions to balance &
address these requirements via multifunctional expertise.
There are no SOPs established and stats needs to know how to organize the data. There are a
lot of assumptions required in most cases.

168
Working with data outside of well structured/defined clinical study. Different view/perspective
on managing data.
Ability to have deep understanding of varying statistical methods to analyze data using best
case/worst case assumptions and the need to explain that to regulators.
RA: Develop understanding on how to position EHR and RWD into more traditional
submissions; how to source especially OUS data. Stats: Methods development and validation,
different processes for generating TLFs etc CA: Understanding how EHR and RWD fit into
existing portfolio of studies/evidence types; how to assess value and use cases compared to
traditional studies DM/DS: Use of big data, cloud, etc., to better handle (as trad clinical data
are smaller and use different systems) MedAff: How to speak peer-to-peer and educate about
value of the EHR/RWD-driven evidence
More team members with advanced modeling techniques
May require additional skills or information to interface with various systems
Protecting data integrity has only become the focus of regulators over the last decade. Data
analytics is still maturing as an area of resource need. We still struggle to find resources that
have the knowledge and integrity to meet the demands.
data troubleshooting
I don't think our organization requires new skillsets because we have been using EHR data for
a long time.
RWE is limited but is a necessary aspect of submissions when it comes to patient populations.
This information is needed for the regulatory strategy. In streamlining both clinical and EHR,
there are gaps that can be better addressed by both the healthcare and industry settings.

169
Q5.8 - Your organization's goals and objectives for product development include the use of
EHR data for generating clinical evidence.
# Field Minimum Maximum Mean Std
Deviation Variance Count
1
Your organization's goals
and objectives for product
development include the
use of EHR data for
generating clinical
evidence.
1.00 3.00 1.46 0.75 0.56 39
# Answer % Count
1 Yes 69.23% 27
2 No 15.38% 6
3 Don't know 15.38% 6
Total 100% 39

170
Q5.9 - To what extent have you heard senior executives of your organization speaking
about using EHR data to generate clinical evidence?
# Field Minimum Maximum Mean Std
Deviation Variance Count
1
To what extent have you
heard senior executives of
your organization
speaking about using
EHR data to generate
clinical evidence?
1.00 4.00 2.64 0.80 0.64 39
# Answer % Count
1 Never 5.13% 2
2 Rarely 41.03% 16
3 Somewhat frequently 38.46% 15
4 Very frequently 15.38% 6
Total 100% 39

171
Final Comments
Q6.1 - What lessons have you or your organization learned from investigating the use of
EHR data for generating clinical evidence?
What lessons have you or your organization learned from investigating the use of EHR data
for generating clinical evidence?
Variability in how clinical assessments are made (e.g., different grading scales) can make data
unusable. Often, no dedicated fields are available to store required product name or endpoints
require the extraction of data from text fields. It makes analysis extremely complicated and
tedious, with potentially unreliable and highly variable data. EMR is great for certain data like
blood pressure and IOP, but horrible for dry eye studies and even some contact lens studies.
The lack of standardization, infrastructure, and openness of regulators to RWE that does not
mimic a sample for sample method comparison study for IVD performance validation makes
its use almost impossible to support regulatory submissions currently.
Cleaning data is the most important step. Without proper care, the results may be misleading.
EHR data is generally considered retrospective which impacts the perceived quality compared
to prospectively collected clinical investigation data. We have found that providing small
grants to researchers can greatly enable them to pursue EHR data which they then publish.
Having the EHR based studies published is a key step in having the data accepted by the FDA
as supportive of regulatory filings. The challenge of doing clinical studies in the field of acute
trauma care is a significant factor in our use of EHR data as there are often few other options.
Engineering studies and usability studies form the core of our regulatory submissions, but
EHR based analyses are important supportive information
More accurate representation of the real life use of a device. Best place to explore other
indications and uses of the device. However, inconsistency of data and lack of details that the
FDA expects to review for an approval still lack. This is where systems like KP that have their
own home grown EHR systems could have an advantage. Pre specifying basic data elements
for research purposes, linking EHRs from different systems, and willing to work with med
tech companies on maximizing the understanding of their devices should be the future.
Additionally professional societies’ engagement with hospital systems to make the data more
useable is also be considered.
FDA wants to see more and more clinical data with any classification of the devices.
Therefore, EHR enables the feasibility of multiple studies comparing to traditional clinical
studies or trails
Variability of database systems, formats and lack of unified systems creates bottlenecks (time,
cost, workload) in the attempts to incorporate EHR data in the generation of clinical evidence.
Clearer, more transparent regulations/guidance is needed to ensure alignment of the different
regions and healthcare provider systems towards this.
Institutions & institutional requirements for EHR data vary, and there’s not a universal
protocol of acceptance amongst hospital systems. This is also the case when international data
is desired, with differences of ownership, control, and usage boundaries between many key
global markets.
It will not be simple or quick. We have to work closely with the vendor to get to the right
data.

172
As of now it is more hypothesis generating data, or useful for claims /indication expansion.
There is potential in EHR (couple of successful examples)
While it is important to explore the possibility, recognizing the limitations to decision making
is important so that an appropriate and timely pivot could be made.
We are still in evaluation phases for EHR, having prioritized investment in chargemaster data
at a national-sample level and also NIS dataset. EHR value case will need to be made, and will
likely fill gaps of other RWD types/sources (e.g. lab values, functional assessment outcomes,
imaging...)
It is mainly in the discussion phases at the corporate level. We do have it in our business
united but not utilized in the capacity that it can be. I have previously worked in healthcare in a
hospital setting and the EHR are used significantly to draw outcomes from data. It is used very
minimally in the Medical Device Setting and could be used more to capture Real World Data
and outcomes more tied to the real life setting vs the traditional clinical trial. Also each
business unit captures their data differently where it is not harmonized throughout the business
with same process throughout. In addition the company is so large where that can sometimes
be inhibiting with alignment and agreement of the goal and way to reach the goal capturing
relevant data.
Need to be rigorous in data collection and methodology-prepare upfront to evaluate multiple
populations and demonstrate effectiveness across multiple cohorts use fit for purpose data
Much easier said than done.
Useful for collection of background information (e.g., medical history), but of limited use to
meet typical prospective data requirements for clinical studies
FDA encourages the use of EHR to generate RWE and support marketing applications, but the
reviewers themselves tend to require RCTs. In my experience the issue is not the quality of
EHR generated evidence but rather a paradigm shift is needed for the FDA reviewers to deem
that kind of evidence sufficient (at least for class II devices and certain de novo requests).
That it is taking more time than we had anticipated
Not privy to this information
It’s not as straightforward as you would think because much of what’s needed doesn’t easily
come through on the EHR because it wasn’t designed for it.
Expect it to take at least 30% longer than you think and also cost 30% more than you expect.
Over coming bias concerns will be challenging.
device risk may be a factor in how detailed the data may be within the EMR. e.g. disposable
products may not be tracked or have an IDU rendering unusable. implantable have more detail
in the EMR but then the burden of proof for approval pathway is more stringent
My experience has been more the use of EHRs to attempt to obtain clinical data missing from
eCRF collection. From that standpoint, I have not found EHRs to provide much valuable
information. The EHRs I have reviewed are not nearly as detailed and informative as older
records that were written or dictated. It is unlikely that the specific data items needed to assess
a procedure or device would be present in typical EHRs.

173
The senior management needs to emphasize the importance of the use of EHR data for
generating clinical evidence and improve the awareness of it through training and putting more
resources to this area.
Proprietary
It is a necessary component but equally difficult to implement at this time.

174
Q6.2 - Please provide any additional insights regarding the benefits of using EHR data for
regulatory purposes.
Please provide any additional insights regarding the benefits of using EHR data for regulatory
purposes.
Don't know.
Expeditious submissions, indications for disease states where running a trial may not be
feasible, reducing the burden and risk of off label use of devices, and a real world
understanding of the device across a larger cross section of patient populations.
I’m also interested to learn more about EHR data benefit as well as cybersecurity related issues
Consideration of EHR use must begin very early in the development process, with detailed
screening, vetting, and collaboration review between development engineers &
clinical/medical representatives. The data needed to either prove substantial equivalence
(510(k)) or safety & effectiveness (PMA) may not be the customary data routinely gathered for
any given patient. Thus, early Medical Affairs involvement within the data modeling process
is crucial, as well as location specificity, and if a multi-center approach is to be used for data
gathering & what boundary nuances may exist between participating hospital
systems/clinics/investigators.
Can be helpful in supporting regulatory approvals.
EHR data has use as supplementary data and for hypothesis generation which seems to have
better acceptability since typically those have relied on passive reporting in literature and
having deeper level data certainly adds confidence in the study design.
EHR will have more ease-of-use when it comes as structured data (e.g. LOINCS annotated...)
Otherwise, it takes a good deal of collaboration between data science (for actually working
with the data) and clinical science/med affairs (for interpreting and guiding what matters).
High value in linking EHR to other trials or other RWD sources to tell a story that is
simultaneously both broad and specific/deep.
EHR data can be extremely valuable in the regulatory process - either for submission or for
supplemental evidence. The key is being able to identify discrete groups for comparison
(either pre/post or control/comparison groups) with appropriate control factors.
Not privy to this information For my organization
Large n number in studies. Novel and innovative approach, good FDA engagement.
improves the path to market with less time and money and more generalizability but can lack
key elements, distinguishers, endpoints or details
If a particular study used only data that are typically recorded in office visits, especially one
using billing data, the EHR data might be useful. If each participating clinic/hospital EHR
system could be modified to collect specific data necessary for a regulatory submission, that
might work too.
EHR data is real world data that can include more Diversity & Inclusion patient populations
than randomized controlled trials.
Proprietary
This is not a just a limiting factor for regulatory or clinical purposes. EHR will provide next
steps for device strategy and patient needs. All areas can benefit from the information from

175
EHR, especially in R&D to bridge the gaps, medical affairs to provide additional insights and
for industry trends.

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Evaluation of FDA-sponsor formal meetings on the development of cell and gene therapies: a survey of industry views

PDF

Computerized simulation in clinical trials: a survey analysis of industry progress

PDF

Sharing the results of clinical trials: industry views on disclosure of data from industry-sponsored clinical research

PDF

21 CFR Part 11 compliance for digital health technologies in clinical trials

PDF

Contract research organizations: a survey of industry views and outsourcing practices

PDF

Institutional review board capabilities to oversee new technology: social media as a case study

PDF

Examining the regulatory framework for drug compounding: industry views and experiences

PDF

Regulatory team development in post-merger integration: a survey of views from medical product companies

PDF

The impact of FDA Patient-Focused Drug Development (PFDD) on US drug development strategies: a survey of views from pharmaceutical product companies

PDF

FDA influence on advisory committees through documentation: a content analysis and survey of industry views

PDF

A survey of US industry views on implementation of decentralized clinical trials

PDF

Current practices in biocompatibility assessment of medical devices

PDF

Regulatory harmonization in a resource-limited setting: the World Health Organization Collaborative Procedure for Accelerated Registration

PDF

Organizational communication of regulatory intelligence: a survey of the medical device industry

PDF

Using telemetry to ensure safe and reliable medical device operation: experience with defibrillators and infusion pumps

PDF

Regulatory dissonance in the global development of drug therapies: a case study of drug development in postmenopausal osteoporosis

PDF

Regulatory programs to foster medical product development: user experience in the United States and Japan

PDF

Implementation of unique device identification in the medical device industry: a survey of the change management experience

PDF

The impact of healthcare interventions using electronic health records: an evaluation within an integrated healthcare system

PDF

An industry survey of implementation strategies for clinical supply chain management of cell and gene therapies

Asset Metadata

Creator Hergert, Cheryl (author)

Core Title Use of electronic health record data for generating clinical evidence: a summary of medical device industry views

School School of Pharmacy

Degree Doctor of Regulatory Science

Degree Program Regulatory Science

Degree Conferral Date 2023-12

Publication Date 12/21/2023

Defense Date 12/13/2023

Publisher Los Angeles, California (original), University of Southern California (original), University of Southern California. Libraries (digital)

Tag CDRH,Center For Devices and Radiological Health,clinical evidence,EHR,electronic health record,Environment,FDA,Food & Drug Administration,Health Information Technology for Economics and Clinical Health Act,HITECH,OAI-PMH Harvest,Office of National Coordinator of Health IT Technology,ONC,organization,randomized controlled trial,RCT,real world data,real-world data,regulatory submission,RWD,Technology,The 21st Century Cures Act,The Cures Act,TOE

Format theses (aat)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Pacifici, Eunjoo (committee chair), Bain, Susan (committee member), Beringer, Paul (committee member), Richmond, Frances (committee member)

Creator Email cherylmhergert@gmail.com,hergert@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC113800516

Unique identifier UC113800516

Identifier etd-HergertChe-12582.pdf (filename)

Legacy Identifier etd-HergertChe-12582

Document Type Dissertation

Format theses (aat)

Rights Hergert, Cheryl

Internet Media Type application/pdf

Type texts

Source 20240105-usctheses-batch-1118 (batch), University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu