University of Southern California Dissertations and Theses

Design control for software medical devices: an industry survey of views and experiences

(USC Thesis Other)

Design control for software medical devices: an industry survey of views and experiences

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Design Control for Software Medical Devices: An Industry Survey of Views and Experiences
by
Colleen Watson
A Dissertation Presented to the
FACULTY OF THE USC SCHOOL OF PHARMACY
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF REGULATORY SCIENCE
August 2024
Copyright 2024 Colleen Watson

ii
Dedication
For my family, in all its forms. Thank you for your support and the freedom that allowed
me to follow my passion, learn, explore, and conduct this research. Thank you for being present
with me along this journey. Also, thank you for ensuring I looked up from the computer,
laughed, and enjoyed the path.

iii
Acknowledgments
I am beyond grateful to the many individuals who pushed, pulled, and aided me on my
journey to this milestone. First to my mentors on this journey Dr. Susan Bain and Dr. Frances
Richmond: Thank you for the tireless and countless hours guiding me on this adventure. When I
felt I was done, you pushed me to refine my work further. The lessons learned from this will stay
with me as I complete this milestone.
Thank you as well to the University of Southern California and the Mann School of
Pharmacy for the opportunity to continue my education. To my other committee members, Dr.
Jerry Loeb, and Dr. Eunjoo Pacifici, thank you for challenging my assumptions and offering
your expertise in this research. This has enriched not only the research here but also my journey
throughout my doctorate. Thank you as well to Dr. Terry Church for the technical support on the
formatting of this paper and citations.
I am honored to be part of the 2020 cohort, whose time at the University of Southern
California was marked by COVID-19. I have learned a great deal from each of you and treasure
the journeys we made together. To all the staff and educators at this school, thank you for all the
work you do for the students in this program.
I am also grateful to my employers and the many colleagues that I have been fortunate to
know along my educational journey. In continuing to allow me to pursue my dreams you have
opened pathways for me to become a better writer and researcher.
Finally, thank you to my family. I am grateful for my parents, Gary and Pat Watson, who
kindled my love of science and technology. Thank you, Joe Lind, for keeping me going and
allowing me time to focus on my dream. To the kids we have been privileged to watch grow:

iv
Garrett, Grace, Michael, Aidan, Victoria, and Dylan - you make my days brighter. Additionally,
an extra special acknowledgment to my children Grace and Michael O’Keeffe, whose editorial
support and expertise in Qualtrics were helpful in the writing of this dissertation. I am excited to
watch each of you continue your journey and thrilled to complete this milestone.

v
TABLE OF CONTENTS
Dedication....................................................................................................................................... ii
Acknowledgments.......................................................................................................................... iii
List of Tables ................................................................................................................................ vii
List of Figures................................................................................................................................ ix
Abstract.......................................................................................................................................... xi
Chapter 1. Overview ........................................................................................................................1
1.1 Introduction................................................................................................................. 1
1.2 Statement of the Problem............................................................................................ 5
1.3 Purpose of the Study ................................................................................................... 6
1.4 Importance of the Study.............................................................................................. 6
1.5 Limitations, Delimitations, Assumptions.................................................................... 7
1.5.1 Limitations..................................................................................................... 7
1.5.2 Delimitations ................................................................................................. 7
1.5.3 Assumptions.................................................................................................. 8
1.6 Organization of Thesis................................................................................................ 8
1.7 Definitions and Abbreviations.................................................................................... 9
Chapter 2. Literature Review.........................................................................................................12
2.1 Introduction............................................................................................................... 12
2.2 Evolution of Design Control..................................................................................... 13
2.3 Regulation of Digital Health..................................................................................... 17
2.4 Software Development Lifecycle Models (SDLC)................................................... 23
2.5 Applying Software Development Models to Medical Devices................................. 33
2.5.1 Recent regulatory changes to address DHT................................................. 44
2.5.1.1 Quality Reviews: The FDA Precertification Pilot Program .......... 45
2.5.1.2 Alternative Review and Submission approaches............................ 47
2.5.1.3 Submission decisions...................................................................... 49
2.5.1.4 Context for Change......................................................................... 50
2.6 Framing proposed research ....................................................................................... 50
Chapter 3. Methodology ................................................................................................................54
3.1 Introduction............................................................................................................... 54
3.2 Survey Development................................................................................................. 54
3.3 Target Population...................................................................................................... 55
3.4 Survey Validation ..................................................................................................... 55
3.5 Survey Deployment................................................................................................... 57

vi
3.6 Survey Analysis........................................................................................................ 58
Chapter 4. Results..........................................................................................................................60
4.1 Survey Participation.................................................................................................. 60
4.2 Demographics........................................................................................................... 61
4.3 Content...................................................................................................................... 67
4.4 Context...................................................................................................................... 83
4.5 Process ...................................................................................................................... 95
Chapter 5. Discussion ..................................................................................................................104
5.1 Overview................................................................................................................. 104
5.2 Consideration of Study Methodology ..................................................................... 104
5.2.1 Limitations................................................................................................. 104
5.2.2 Delimitations ............................................................................................. 108
5.3 Consideration of the Results................................................................................... 111
5.3.1 Content....................................................................................................... 112
5.3.1.1 Resource Quality for Device Classification.................................. 112
5.3.1.2 Resource Quality for Design Control Implementation................. 114
5.3.1.3 Regulating Cybersecurity ............................................................. 116
5.3.1.4 Implementing Design Controls within the Regulatory
Framework.................................................................................... 117
5.3.2 Context....................................................................................................... 120
5.3.3 Process....................................................................................................... 123
5.4 Conclusions and Recommendations ....................................................................... 126
5.5 Future Research....................................................................................................... 127
References....................................................................................................................................129
Appendices...................................................................................................................................138
Appendix A. Survey Questions............................................................................................ 139
Appendix B. Survey Full Text Comments........................................................................... 162
B.1 Content.................................................................................................................... 162
B.2 Context.................................................................................................................... 168
B.4 Process .................................................................................................................... 170
Appendix C. Cross Tabulation Tables................................................................................. 173

vii
List of Tables
Table 1: Definitions and Abbreviations..........................................................................................9
Table 2: FDA guidance, Software review.....................................................................................36
Table 3: Perceived versus Actual Barriers to implementation of Agile techniques. ....................44
Table 4: Example Questions based on HPT Framework..............................................................53
Table 5: Number of Questions in Final Survey ............................................................................57
Table 6: Weighted Average Value Assignment............................................................................59
Table 7: Comments: Implementing Design Control....................................................................68
Table 8: Cross Tabulation - How easy is it to implement Design Control? .................................70
Table 9: Comments: Other categories for Software Classification .............................................71
Table 10: Statements for Design Control......................................................................................73
Table 11: Review of Policy and Guidance Documents - Group 1................................................77
Table 12: Comments: Improvements - Group 1 ...........................................................................78
Table 13: Review of Policy and Guidance Documents - Group 2................................................80
Table 14: Cross Tabulation - FDA Guidance Documents - Group 2 ...........................................82
Table 15: Comments: Improvement - Group 2.............................................................................83
Table 16: Ranking of Barriers for Development of Medical Device Software ............................84
Table 17: Comments: Other Barriers for Development of Medical Devices ..............................85
Table 18: Agreement with Statements Regarding FDA Submissions..........................................88
Table 19: Comments: Statements regarding FDA Interactions...................................................89
Table 20: Comments: Team View of FDA Feedback .................................................................90
Table 21: Comments: View of FDA Feedback............................................................................92
Table 22: Comments: Quality of FDA Response ........................................................................93
Table 23: Statements for Design Control for Software ................................................................97
Table 24: Comments: Design Control SaMD..............................................................................97
Table 25: Agreement with Statement for Agile and Design Control............................................98
Table 26: Comments: Agile SDLC and Design Control .............................................................99
Table 27: Ranking of Current Challenges ..................................................................................101
Table 28: Final Comments..........................................................................................................103
Table 29: Survey Flow................................................................................................................139
Table 30: Additional Comments - Ease implementing Design Control .....................................162
Table 31: Additional Comments - Determining Risk Classification..........................................164
Table 32: Additional Comments - Useful in determining Software Classification....................164
Table 33: Additional Comments - Updates to Guidance Documents Group 1...........................165
Table 34: Additional Comments - Updates to Guidance Documents - Group 2 ........................166
Table 35: Additional Comments - Business Reasons not listed .................................................168
Table 36: Additional Comments - Method for interaction with FDA ........................................168
Table 37: Additional Comments - Interactions with FDA..........................................................169

viii
Table 38: Additional Comments- FDA Knowledge ...................................................................169
Table 39: Additional Comments – Team View FDA Feedback.................................................170
Table 40: Additional Comments - Add question how the Team felt..........................................170
Table 41: Additional Comments - Statement Design Control Sufficient ...................................170
Table 42: Additional Comments - Design Control and Agile ....................................................171
Table 43: Additional Comments - Challenges............................................................................171
Table 44: Cross Tabulation - FDA Acceptance of Agile............................................................173
Table 45: Cross Tabulation - FDA Guidance Documents - Group 1 .........................................173
Table 46: Cross Tabulation - Modification FDA Guidance - Group 1.......................................175
Table 47: Cross Tabulation - Modification FDA Guidance - Group 2.......................................175
Table 48: Cross Tabulation - Type of recent Submission...........................................................175
Table 49: Cross Tabulation - Agreement Agile meets Design Control......................................176

ix
List of Figures
Figure 1: Waterfall Design Process ..............................................................................................16
Figure 2: Historical Development Regulatory and Software Models...........................................18
Figure 3: Medial Device Software on a Risk Spectrum from 2013-2019 ....................................23
Figure 4: Royce Cascade Method of SDLC .................................................................................25
Figure 5: Vee (V) Model of Development....................................................................................26
Figure 6: Spiral SDLC ..................................................................................................................28
Figure 7: Scrum Agile Method .....................................................................................................30
Figure 8: Test Driven Agile Development....................................................................................31
Figure 9: Extreme Programming ..................................................................................................32
Figure 10: Health Policy Triangle (HPT) Framework..................................................................53
Figure 11: Survey Participation ....................................................................................................60
Figure 12: Primary function in Organization................................................................................61
Figure 13: Experience with Medical Devices or Consumer Software..........................................62
Figure 14: Experience with Specific Medical Devices.................................................................62
Figure 15: Experience with Non-Medical Consumer Products....................................................63
Figure 16: Company developmental focus...................................................................................64
Figure 17: Experience with SDLC Methods.................................................................................65
Figure 18: Experience with Agile Methods..................................................................................66
Figure 19: Company development of Software ............................................................................66
Figure 20: Factors for inclusion of Design Control for Consumer Software ...............................67
Figure 21: Difficulty in implementing Design Control ................................................................68
Figure 22: Determination for Software Classification..................................................................71
Figure 23: Usefulness of Methods for Software Classification....................................................72
Figure 24: Statements for Design Control ....................................................................................74
Figure 25: Word Choice for FDA Guidance.................................................................................75
Figure 26: Review of Policy and Guidance Documents - Group 1 ..............................................77
Figure 27: Review of Policy and Guidance Documents - Group 2 ..............................................81
Figure 28: Ranking of Barriers for Development of Medical Device Software...........................85
Figure 29: Context for Discussions with FDA .............................................................................86
Figure 30: Agreement with Statements regarding FDA Interactions ...........................................89
Figure 31: How did the Team feel about FDA Feedback.............................................................90
Figure 32: How knowledgeable did you find the FDA in Software Discussions.........................91
Figure 33: Length of Time since last Submission with Software.................................................92
Figure 34: Quality of Response from FDA...................................................................................93
Figure 35: Agile Documentation in Submission...........................................................................94
Figure 36: Why submit Agile Documentation..............................................................................95
Figure 37: Statements for Design Control for Software ...............................................................96

x
Figure 38: Agreement with Statement for Agile and Design Control ..........................................98
Figure 39: Participation in the development for Guidance / Regulations.....................................99
Figure 40: Trade Group participation .........................................................................................100
Figure 41: Ranking of Current Challenges.................................................................................102

xi
Abstract
The practical applications of software have exploded in the past twenty years to the point that
software as a medical device (SaMD) is now embedded as standard technology in the medical
health care system. The opportunity to enter the device market has attracted a new population
of companies with less experience in medical product regulation than traditional medical device
companies. Many of these companies have become accustomed to software development
methods, such as the popular Agile methods, whose approaches do not fit comfortably in FDA’s
conception of design controls for medical devices. The structured nature of current design
control methods for regulated medical devices has raised questions about whether the current
regulatory requirements impede rapid device development and innovation without justifiably
improving product safety.
The Health Policy Triangle (HPT) framework was applied to data received from an
online survey soliciting the views and experience of 83 industry professionals involved in
medical device software development regarding opportunities and challenges related to the
implementation of design controls. Results suggested that the industry is broadly satisfied with
the current state of design control for digital health technology (DHT) and is actively involved in
continuing to shape the regulations that oversee the development. The views on the process,
content, and context of developing software as a medical device (SaMD) and software in a
medical device (SiMD) were similar across regulatory and engineering experts. Most felt that
Agile software development lifecycles (SDLC) can be used compliantly within the existing
design control framework but may be best suited to specific situations rather than the sole SDLC

xii
methodology. Many, though, want more specific technical guidance to operate effectively
within the existing framework
Differences in opinions were expressed with respect to specific guidance and policy
documents. Better promotion of new guidance documents, such as those for low-risk software,
may need to be directed to newer entrants to the field, where many identify deficiencies.
Further, the industry respondents indicated the need for more specific and practical examples
related to best practices and technical solutions, especially in clinical decision-support and
cybersecurity guidance on cybersecurity documents.

1
Chapter 1. Overview
1.1 Introduction
The healthcare ecosystem has been unalterably changed over the past fifteen years
through a Digital Health Technology (DHT) revolution. Its wide reach is reflected by the way
that it has been characterized by the Food and Drug Administration (FDA).
Digital health technologies use computing platforms, connectivity, software, and
sensors for health care and related uses. These technologies span a wide range of
uses, from applications in general wellness to applications as medical device.
(FDA, 2020b, Introduction)
It is also reflected in the wide range of applications that fit under this umbrella, from
telemedicine /telehealth and personalized medicine to electronic records and mobile wearable
technology. The needs of these applications drive DHT to respond in turn with further evolution.
It has forced the change of many physical products into virtual software and has permitted the
entry of consumer software developers who previously had little or no medical product
experience. Thus, companies developing products and services using such technologies can be
challenged to understand if, and when, new offerings might be considered medical devices.
When that occurs, they are faced with understanding how to ensure that their products, now
considered medical devices, will have to comply with a complicated set of medical device
regulations with which they were previously unfamiliar.
Healthcare products, including software products, regarded as medical devices have been
regulated in the US for decades. In the United States, the foundational legislation from which
the regulations extend is the 1938 Federal Food, Drug, and Cosmetic Act (FD&C), as amended
by the 1976 Medical Devices Amendments (21USC§§301-392, 1976). These laws established
the boundaries for what would and would not be defined as medical devices.

2
“An instrument, apparatus, implement, machine, contrivance, implant, in vitro
reagent, or other similar or related article, including a component part or
accessory which is: recognized in the official National Formulary, or the United
States Pharmacopoeia, or any supplement to them, intended for use in the
diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or
prevention of disease, in man or other animals, or intended to affect the structure
or any function of the body of man or other animals, and which does not achieve
its primary intended purposes through chemical action within or on the body of
man or other animals and which is not dependent upon being metabolized for the
achievement of any of its primary intended purposes.” (21USC§§301-392, 1976,
Sect. 321(h))
At the time the definition was written into legislation, the types of electronic and digital
medical products commonplace today could not have been anticipated by the legislators. Thus,
in 2016, it became necessary to amend the definition of medical devices with Section 30609a of
the 21st Century Cures Act (H.R. 34, 2016). This codified and clarified the definition of a
medical device to guide what types of digital products would and would not be considered
medical devices. For example, it excluded software used for medical administrative support,
lifestyle applications, transfer of electronic patient records, or transferring clinical test data
without analysis.
If a digitally based technology meets the definition of a medical device, it will be
regulated by the Food and Drug Administration (FDA) according to its risks. Lower-risk
(Class I) devices require only general controls, whereas higher-risk devices (Class II-III)
additionally must satisfy specific controls (called “special” controls) to demonstrate proof of
efficacy and safety. These products are also subject to postmarket oversight and require adverse
event reporting.
It is a regulatory requirement that Class II and III medical devices be designed and tested
under a “design control” framework, introduced in 1997 (FDA, 1997). This systematic method of
product development, illustrated in Figure 1, requires the designer to determine user needs early

3
in the design process. These are translated into “design inputs”, usually captured as engineering
specifications for the new product. The eventual design is then tested through a series of
verification activities to ensure that the output of the design meets its design inputs. Risk
management methods are used to determine where design changes are needed, and the final
medical device is validated through user feedback, clinical testing, and/or human factors
evaluation to ensure that it meets the requirements of the end users. Key steps in this design
cycle are marked by technical review and oversight to assure compliance with appropriate design
practice and to add another opportunity to identify and advise on residual risks.
Software is also regulated in the same risk framework. Software in a medical device
(SiMD) refers to traditional hardware devices that contain software. By contrast, Software as a
medical device (SaMD) consists of a specific subset of medical devices that includes software
that stands alone and does not provide its effect directly through physical hardware. Examples of
SaMD include a software application on a watch to monitor blood pressure, or an app on an
iPhone to monitor and alert patients connected to a ventilator at some distance away from it.
Such devices can be low in risk, in which case they would be placed in Class I with associated
modest controls. In some situations, the software may be much riskier and would fall into Class
II to Class III device risk classifications. Examples in this category include the use of a mobile
application as an audiometer to check for hearing loss or a device that alters radiology images to
allow for clearer diagnostic imagery.
This regulatory framework, however, does not specify how controls should be applied to
software. Manufacturers choose from various design models to create new software. A
“cascade” or “waterfall” is a conventional design strategy for hardware /software development
that is most like the design control “waterfall” defined by the FDA. On the other hand, an

4
“Agile” approach, commonly used by software developers, is based on a more iterative set of
cycles in which the software is built in modules, starting with those that are most central or risky,
and progress is made in “sprints”.
The disconnect between software development methods and design control requirements
presents challenges for a regulatory agency faced with overseeing modern software. In the US,
the FDA began to experiment with alternative quality control methods to find a middle ground.
Notably, it established a precertification pilot program to explore a new regulatory framework
for software in 2017. In January 2018, it held a public workshop entitled “Fostering DigitalHealth Innovation for Developing the Software PreCertification Program”, which looked for a
working model that could foster innovation and provide regulatory oversight.
At the same time, specific approaches to software requirements were being taken in other
jurisdictions. For example, the European Union, in its current transition from the 1993 Medical
Device Directives to the new Medical Device Regulations (EU-MDR), used the change as an
opportunity to address the unique needs of medical device software, including digital
applications. This was further clarified in the MDCG Guidance document on the classification
of software (Council Directive 93/42/EEC, 1993).
Additionally, the EU introduced the General Data Protection Regulation (GDPR) to
govern the ways in which personal data can be collected and managed, a key point of concern for
many digital applications (Regulation (EU) 2016/679, 2016). The EU has grappled with how to
approach DH, through its green papers on mHealth. More globally, the International Medical
Device Regulators Forum (IMDRF) developed voluntary guidance to define software as a
medical device and suggested a risk-based framework for its classification. These have been
used across many jurisdictions to harmonize relevant regulations and have been integrated into

5
both the US and EU regulatory frameworks. Central to all of these rules and guidances,
however, is a common thread of traditional medical device regulation.
1.2 Statement of the Problem
The rapid evolution of software development methods has outpaced the development of
software-sensitive regulations. This comes as new players unfamiliar with regulatory systems
have been drawn to the medical ecosystem. Industry software developers entering the medical
device environment bring with them different approaches and expectations than those usually
held by those who have worked in the highly regulated fields governed by Health Authorities. A
key challenge where these two worlds collide is in the regulatory requirement for design
controls, called out in the 1997 FDA design control and ISO 13485 framework.
Design controls became a regulatory requirement about 25 years ago and have now
become well-established for those producing traditional medical devices based on hardware or in
vitro diagnostic platforms. When those methods were introduced, they were initially not
welcomed by the medical device community but have become accepted standards for the
development of more traditional medical devices. Special needs, though, are now faced by
software development, which takes its philosophies and methods from consumer-driven software
models. Anecdotal reports point to dissatisfaction with the current design control framework
when applied to software as a medical device. The specific areas of the challenge for the
application of design control when applied to SaMD development, however, are not well
documented in the available literature. This presents an opportunity to understand the views of
industry on this topic to define the current views and experience of the development community
that could lead to improvements in the regulatory system.

6
1.3 Purpose of the Study
This study examined design controls for software by using a Health Policy Triangle
(HPT) framework to survey the views of industry professions involved with the design and
development of SaMD. The HPT framework examines a situation by looking at industry views
on three key elements: Context, which refers to the external environment; Process, which
outlines the way to establish a new requirement; and Content, which encompasses the
information stated in the existing regulatory documents.
The research began with a literature search to understand the available literature on the
regulation of digital health, applicable software development lifecycle (SDLC) models, and how
these models are applied within the current regulatory system. Next, a fit-for-purpose survey
was designed and critiqued through a focus group of experts. The finalized survey was then
distributed to engineering, regulatory, and quality professionals involved in the design and
development of software as a medical device through the online software tool, Qualtrics.
1.4 Importance of the Study
Medical device companies increasingly see the value and need for software and mobile
applications to support their existing technology. They can potentially benefit by using evolving
methods for software design. This research may help software developers in the medical product
industry to understand the current challenge, so that they can better prepare for them. Surveys
such as this often help to define best practices from those who have already navigated the system
successfully. Regulators, as they look to propose new regulations, can benefit from an
understanding of what worked well in the past and what is still causing problems. As they look
to solving those problems, this research may help them to identify the greatest areas of concern

7
that might be usefully addressed by modifying their approaches to review and oversight of
SaMD.
1.5 Limitations, Delimitations, Assumptions
1.5.1 Limitations
This research was conducted using a survey of professionals working in the quality,
regulatory, or technical areas of SaMD development. Because these individuals are busy
professionals, the response rate would be relatively low, as the time needed to complete the
survey competes with other pressing professional demands. The length was constrained to limit
survey fatigue, but this may have limited the scope and depth of the research. The survey was
provided in English, only, so was limited to individuals able to respond in this language.
Design control systems were likely implemented in medical device companies using both
hardware and software models, especially in large traditional companies with many SiMD as
well as SaMD products. Thus, it may not be entirely possible to separate the implementation of
design control for SaMD, specifically when speaking with individuals about past
implementations. The survey sought out individuals across the spectrum of software
development from low-risk to high-risk applications in the medical field. This meant, though,
that the number of individuals in certain subgroups was relatively small and that limited my
ability to analyze and draw firm conclusions related to the views of those in certain subgroups.
1.5.2 Delimitations
This research was limited to individuals with direct experience with software
development for medical devices and who have been involved in the integration of regulatory
requirements with software development and testing. The research focused on the approaches of
the US FDA and acknowledges that somewhat different experiences may be faced when

8
attempting to commercialize software in other constituencies, especially those for which
software regulations are less mature. The field of software development is evolving rapidly, and
the results of this study are delimited to the period during which the research was conducted. It
may then, however, set a benchmark from which further evolution can be understood and
studied.
1.5.3 Assumptions
It is assumed that the professionals completing the survey are truthful and accurate in
their responses. It is also assumed that the confidential nature of completing the survey allowed
the respondents to share information that they might not otherwise have been willing to share
using more direct methods of communication.
1.6 Organization of Thesis
This dissertation is organized into five chapters. Chapter 1 provides a summary of digital
health, design control as it pertains to software, and the scope of this research. Chapter 2
contains the literature research related to design control, digital health, and software
development. Chapter 3 defines the study methods, including the specifics of the survey.
Chapter 4 presents the results of the research. Finally, chapter 5 provides a detailed review of
the results obtained during the research, conclusions, and recommendations.

9
1.7 Definitions and Abbreviations
Table 1: Definitions and Abbreviations
Term Abb. Definition
AFib Atrial Fibrillation
Clinical
Decision
Support
Software
CDS A software function is considered CDS if it meets the
following:
· Not intended to acquire, process, or analyze [criterion
(1)];
· Intended for displaying, analyzing, or printing medical
information [criterion (2)];
· Intended for supporting or providing recommendations
[part of criterion(3)].
Device/Non-device determined by criteria 4:
IS a Device: If the intended user is HCP and the user
cannot independently review the basis.
IS NOT a device, if targeted for patient or caregiver OR
if the user cannot independently review the basis. (84
FR 51167, 2019)
Design Control A set of quality practices and procedures to ensure
process control to meet user needs, intended use, and
design requirements
Device 201(h) FD&C “instrument, apparatus, implement,
machine, contrivance, implant, in vitro reagent, or other
similar or related article, including any component, part,
or accessory, which is …intended for use in the
diagnosis of disease or other conditions, or in the cure,
mitigation, treatment, or prevention of disease, in man
… or intended to affect the structure or any function of
the body of man...” and “does not include software
functions excluded pursuant to section 520(o) of the
FD&C Act.” (21USC§§301-392 (1976))
Digital Health
Technologies
DHT Digital health technologies use computing platforms,
connectivity, software, and sensors for health care and
related uses. These technologies span a wide range of
uses, from applications in general wellness to
applications as a medical device. (FDA, 2020b)
DMAP Data Modernization Action Plan
DOD Department of Defense
EU-MDR European Union Medical Device Regulation
FDA Food and Drug Administration

10
FD&C Federal Food, Drug and Cosmetic Act
GDPR European Union General Data Protection Regulation
GMP Good Manufacturing Practices
General
Wellness
Product
Products (1) are intended for only general wellness use,
as defined in this guidance, and (2) present a low risk to
the safety of users and other persons. (FDA, 2019c)
Health Policy
Triangle
HPT Framework developed by Walt and Gilson to analyze
health care, focusing on actors, context, and process
rather than the content of policy. (O’Brien et al., 2020)
IMDRF International Medical Device Regulators Forum
KPI Key Performance Indicators
MDCG Medical Device Coordination Group
Medical Device
Data Systems
MDDS Software solely intended to transfer, store, convert
format, and display medical device data including
results, medical images, waveforms, signals, or other
clinical information are NOT devices (FDA, 2019d)
Mobile
Platform
Mobile platforms are defined as commercial off-theshelf (OTS) computing platforms, with or without
wireless connectivity, that is handheld in nature.
Examples of these mobile platforms include mobile
computers such as smartphones, tablet computers, or
other portable computers. (FDA, 2019d)
Mobile Medical
Application
(Mobile
Medical App)
A “mobile medical app” is a mobile app that
incorporates device software functionality that meets the
definition of a device in section 201(h) of the FD&C
Act11; either is intended: · to be used as an accessory to
a regulated medical device; or · to transform a mobile
platform into a regulated medical device.
Intended use determines if it meets the definition of
device.
As stated in 21 CFR §801.4,12, intended use may be
shown by labeling13 claims, advertising materials, or
oral or written statements by manufacturers or their
representatives. (FDA, 2019d)
ONC Office of the National Coordinator for Health
Information Technology
OSEL Office of Science and Engineering Laboratories
NICE National Institute for Health and Care Excellence
Pre-Cert Software Precertification Pilot Program
PTSD Post-Traumatic Stress Disorder
QMS Quality Management System

11
QSR Quality System Regulation
RAD Rapid Application Development
RUO Research Use Only
RWE Real World Evidence
SDLC Software Development Lifecycle Models
Software as a
Medical Device
SaMD Software intended to be used for one or more medical
purposes that perform these purposes without being part
of a hardware medical device. (IMDRF, 2013)
Software in a
Medical Device
SiMD Embedded, Firmware, Operating System, Middleware,
Applications (IMDRF, 2013)
TMAP Technology Modernization Action Plan
US United States of America
USC University of Southern California
WHO World Health Organization
XP Extreme Programming

12
Chapter 2. Literature Review
2.1 Introduction
Technology changed irrevocably when computerized methods became available to do
what previously could only be done by mechanical devices. Regulations have had to evolve to
deal with the unique challenges presented by medical devices that are partly or fully based on
software. This literature search explores the available literature on this evolution- the history of
design controls, the conflicting methodologies of software development, and the views of
industry and regulators in today’s medical device environment.
A multifaceted literature search was conducted using scholarly search engines, including
Google Scholar (scholar.google.com) and PubMed. Relevant government websites not
commonly associated with scholarly search engines, including FDA.gov, EUR-Lex (eurlex.europa.eu), European Medicines Agency (ema.europa.eu), and the website of the European
Commission (commission.europa.eu) were also searched. The searches attempted to capture
information about the history of regulations, the current state of technology for regulated
software development, and the current social-political environment in which industry is
operating. First, a history of design control was examined utilizing the key terms “history +
design control + medical device.” This returned over 1,100 references. Preference was given to
articles written in the past five years, leading to 360 references. The abstracts and description of
the references were reviewed for articles that most closely covered the history of design control.
Next, a review of the regulations pertaining to digital health was developed using both scholarly
searches and a review of government repositories. Third, a review of software development
lifecycle models (SDLC) was conducted utilizing scholarly searches and the phrase “software
development lifecycle models.” Following the initial readings, subsequent searches were
conducted that included a review of articles related to “SDLC + regulated software”, which

13
yielded 21 English references, and “Agile + regulated software development”, which yielded 60
English references. After assessing the relevance of these papers by reading the abstracts/
descriptions, the full articles and cross- references were reviewed further. Finally, additional
relevant documents were sought by examining references within the reviewed articles. Articles
that did not pertain to medical devices or DHT were excluded. Next, the government website
publications related to these technologies were examined for historical changes.
2.2 Evolution of Design Control
Medical devices first became regulated in the United States under the 1938 Federal Food,
Drug, and Cosmetic Act (FD&C). This key piece of legislation defines a medical device and
establishes a role for the federal government in device regulation and oversight. For the next 25
years, however, devices received scant attention from regulators such as the Food and Drug
Administration (FDA). The need to deepen the oversight became clear from the accelerating
frequencies of adverse events associated with devices such as the Dalkon Shield for pregnancy
prevention and artificial intraocular lenses to improve visual acuity. To examine these public
health threats more systematically, a legislative committee, called the Study Group on Medical
Devices or “Cooper Committee”, was formed (Benson, Eccleston, and Barnett, 1988). Their
findings- that the FDA was fragmented, ineffective, and lacked authority drove the passage of
the 1976 Medical Device Amendments. Those amendments authorized the FDA to develop
more directive regulations - regulations that have become more rigorous over time.
An immediate aim was to ensure the safety of manufactured products. In 1978, the FDA
published a final regulation to govern Good Manufacturing Practices (GMPs) for medical
devices (FDA, 1978). Failures of products such as heart valves, however, led to a review of
medical device recalls that implicated design defects as an important cause of adverse events

14
(AAMI, 2007; Pilot and Waldmann, 1998). These observations highlighted the need for further
attention to device design and recommended an expert review process that would focus on
design in addition to manufacturing. In response, the FDA published a June 1990 notice that
identified its intent to modify GMPs. The greatest changes were directed at strengthening the
design process by requiring that design be organized through a regimented series of steps that
would document the design history of riskier devices. These requirements were codified into
regulations under 21 CFR §820 under the Final Rule, known as the Quality System Regulations
(QSR) (89 FR 7496, 1996). The passage of the 1990 Safe Medical Device Act gave the FDA the
authority that it needed to introduce “preproduction design controls,” defined in 21 CFR §820.
The intent of the FDA to regulate design was contentious. The first version of the
proposed revision was published in 1993. “In response, the industry took a common position
that FDA did not have the authority to add design controls to the regulations.” (AAMI, 2007)
The proposals were met with detailed comments from approximately 280 individuals or groups.
The FDA extended the comment period by six weeks to accommodate the large number of
comments and held a series of public meetings as well as advisory meetings before publishing
the final rule in 1997 (AAMI, 2007). In the preamble to this rule, the FDA explained how it
considered the voluminous feedback from industry. These comments from industry made clear
that that industry was far from ready to implement the new requirements. Consequently, to
facilitate the implementation for design controls, a phased approach was proposed that allowed
industry to understand and comply with the new requirements.
In 1997, the FDA also expanded on its expectations for design controls in a guidance
document for medical device manufacturers, “Design Control Guidance for Medical Device
Manufacturers” (FDA, 1997). It outlined the case for quality and defined design control as a

15
systematic process to identify the needs of the users, translate them into specifications, verify
that the specifications have been met, and finally validate that the device meets the requirements
of the users. This stepwise process is illustrated in the frequently referenced waterfall image
shown in Figure 1. The methodical progression of activities was meant to ensure that safety is
considered from the beginning of device development. Additionally, it required that developers
maintain a documentation trail that could be reviewed and audited by the FDA. The FDA
guidance, though, did not prescribe any one method by which manufacturers must establish their
design control system. It only required that companies implement procedures and processes with
checks and balances, to assure that serious attention would be given to designing with as much
safety and quality as possible.
The requirement for a methodical, documented design process is also a requirement
internationally. The international standard most relevant to the quality management system, ISO
13485: 2016, makes clear that the quality management system for medical devices must extend
to include design and development (ISO, 2016). Although the standard does not use the US term
design control explicitly, it describes the same methodical process as detailed in the American 21
CFR §820.30, in section 7.3. The standard requires documentation of user needs, design inputs,
design outputs, design verification, and design validation. Thus, a modern expectation for
designing safety into the device now exists globally. Predictability for device design, the cost
case for quality, transparent requirements for documentation for premarket approval, as well as
legal protection in torte law (Sunshine, 1995) have all forced organizational changes to assure
that design controls are a de facto requirement for medical devices. This provides predictability
and an established system of staging recognized by manufacturers and external auditors alike.

16
Figure 1: Waterfall Design Process
(FDA, 1997)
Throughout the history of device regulations, a similar pattern of change resistance can
be seen in the reactions of industry to constraints on their development practices, even in the face
of product risks. The Cooper Committee formed in 1970 was directed at reducing risks to
patients from dangerous devices, as experts recognized that technical developments were
outpacing regulatory approaches (Pilot and Waldmann, 1998). The medical device industry,
however, was hesitant to support the 1976 Medical Device Amendments, concerned that the
“regulatory policies that purportedly had a dampening effect on the clinical availability of
promising new technologies.” (Benson, Eccleston, and Barnett, 1988) Similarly, industry
objections were voiced by industry as regulators began to call for design controls in 1997. In the
review that follows, dissonance between the preferences of regulators and industry can be seen
as regulators attempt to extend design controls to devices containing software, a group of
products perhaps unanticipated by the design control regulations of 1997.

17
2.3 Regulation of Digital Health
In 1978, when the FDA introduced its first GMPs for devices, Microsoft had 13
employees and had just developed the programming language FORTRAN-80. Apple had just
introduced the Apple II, an 8-bit home computer. By the time the 1990 Safe Medical Device Act
was passed by the US Congress, IBM had released Windows 3.0 and Apple introduced the
Macintosh Classic home computer. Technology was evolving.
Twenty-five years later, by 2015, the implications of digital transformation became a
global focus for regulatory systems overseeing medical devices. Apple iPhones could now be
found in the pockets of many individuals and the use of mobile apps had crept into daily life.
In response to the dramatic increase in computer applications for medical devices,
regulators sought to establish common definitions and a risk-based framework for these novel
devices. This progression is shown in Figure 2 and further discussed in the following sections.
The International Medical Device Regulators Forum (IMDRF) took the lead in forming a
working committee to review software in medical devices. This working group was composed
of regulators from Australia, Brazil, Canada, China, Europe, Japan, Russia, Singapore, South
Korea, and the United States. Their activities led the IMDRF to publish a series of guidance
documents for Software as a Medical Device (SaMD) between 2013 and 2017, the initial
documents set forth definitions for this technology followed by a risk-based framework for its
regulation. Subsequently, IMDRF produced guidance to support the integration of requirements
into a company’s quality management system (QMS) and to guide clinical evaluation for this
technology. This structure has been largely adopted globally (IMDRF, 2013; IMDRF, 2014;
IMDRF, 2015; IMDRF, 2017).

18
Figure 2: Historical Development Regulatory and Software Models
The United States was an active participant in the development of the IMDRF guidance
documents, including its recommended risk-based approach, which it supplemented by its own
risk-based regulations for SaMD. The FDA faced a challenge, however, as it tried to integrate
the requirements for SaMD into the previously described regulations for design control. The
design control regulations had been originally designed for hardware-based technologies, and
many considered those same design controls to be ill-suited for software-based technologies,
such as mobile apps. In the opinions expressed by both industry and regulators, the traditional
application of the FDA’s long-standing regulatory framework might stifle the development of,
and thus access to, new and improved SaMDs (Shuren, Patel, Gottlieb, 2018).
Over time, the FDA gained experience with computer technology. To clarify its
approaches to software, the FDA introduced a series of guidance documents between 2015 and
2019, illustrated in Figure 3 and described in more detail below. This series of guidance
Background: Historical Development
FD&C Act
1938
Medical
Device
Amendments
1976
Design Control
1997
Safe Medical
Device Act
1990
21st Century
Cures Act
2016
2013 forward - Guidance Documents
SiMD / SaMD: Defini�on, Risk,
Development
Waterfall /
Cascade
1950s
Spiral Method
1986
“V”
1991
RAD
1991
Agile
Manifesto
2001
Regulatory Evolu�on
Development Evolu�on
iPhone
2007
Apple II
1977
Windows 1.0
1985
FDA
Pre-Cert
2017

19
documents introduced new terms to categorize different uses of software clinically and to define
specific areas of concern, at least through the eyes of the FDA. Under the new schema, devices
containing software (either SaMD or SiMD) would receive a more detailed review and would
have more stringent requirements if those devices had higher risk profiles and clinical impact.
Devices would be placed in a low-risk category if they served only to display medical
information but would be promoted to higher-risk categories if they interpreted the collected
data, made diagnostic decisions, or integrated artificial intelligence in their analyses. This riskbased tiering became a common framework for the policy approaches outlined in later guidance
documents.
Subsequent guidance documents went into more detail and gave examples of specific
software concerns. One such area was concerned with the increasing need to communicate, store,
and move data with clinical relevance amongst different users. These aspects were addressed in
2015 when FDA published a guidance document, Medical Device Data Systems, Medical Image
Storage Devices, and Medical Image Communications Devices, that defined the term Medical
Device Data System (MDDS). Within this category, software that is intended to display data
would be placed in a lower risk category (US class I), whereas software that leads to an alarm or
clinician alert would be placed in a higher-risk category (US class II or III). The FDA’s attention
then would be more focused on the review and oversight of the higher risk devices (FDA,
2019d).
Guidance documents alone, though, were insufficient to clarify the regulatory rules
associated with this new technology. The passage of the 21st Century Cures Act (H.R. 34, 2016)
was an important inflection point in the management of software as a medical device. The law
modified the definition of software and codified the exemption of several types of software

20
previously mentioned in guidance documents. (21 USC Section 3060(a)) The following types of
software uses were considered to place its device outside of regulation as a stand outside of the
spectrum of regulated devices: (1) For administrative support of a healthcare facility, (2) for
maintaining or encouraging a healthy lifestyle, (3) to serve as electronic patient records, (4) for
transferring, storing, converting formats, or displaying data, or (5) providing certain types of
clinical decision support to a healthcare provider unless interpreting or analyzing a clinical test or
other device data (FDA, 2019a).
It, therefore, clarified specific types of products for which regulatory oversight was not
required.
The rules were explained further in the 2017 FDA Guidance Document titled “Changes
to Existing Medical Software Policies Resulting from Section 3060 of the 21st Century Cures
Act” issued as a draft in 2017 and then modified in 2019 (FDA, 2019a). It extended the
interpretation of the Act quoted above regarding software that the FDA does not regulate,
including software addressing lifestyle, managing electronic records, and transforming or storing
records. The FDA also revised previously issued guidance to ensure the definitions remained
consistent throughout the guidance portfolios. In short, using both guidance documents and
changes in policy, the US government acted to sculpt the levels of oversight that would be
applied to SaMD devices.
In parallel with the Cures Act and changes to the definition of devices, the FDA
introduced the term “general wellness device” for products such as a watch to count steps or an
app intended to help an individual obtain a state of general wellness. For these devices, FDA
identified its intent to exercise “enforcement discretion”. This option communicated that FDA
could not see value in focusing its review efforts on products that pose a low risk to public

21
health. Nevertheless, it could hold them strictly to membership in the technical classification of a
medical device that could, if necessary, receive regulatory attention. If enforced as medical
devices, general wellness devices would fall into US class I and II medical devices (FDA, 2019c;
FDA, 2019)
In 2019, the FDA published another guidance document, “Clinical Decision Support
Software Draft Guidance. Guidance for Industry and Food and Administration Staff.” Among
other things, it introduced the term “Clinical Decision Support Software” (CDS) (84 FR 51167,
2019). This term covers software that can integrate medical data to inform a clinician about a
medical condition (non-device) or create a diagnosis for a clinician (device). CDS applications
can be found across the full spectrum of risk classifications. Most of the applications, though,
fall into US class I or II devices. This guidance document builds on the previous MDDS
guidance in which data transfer was considered because it explicitly recognizes that software can
now integrate data from different sources.
In 2020, the FDA released another guidance, Multiple Function Device Products: Policy
and Considerations. (FDA, 2020a) This guidance addressed technologies that might have
different functions, some of which may not be medical devices. The guidance defines the term
function as:
“…the term “function” is a distinct purpose of the product, which could be the
intended use or a subset of the intended use of the product. For example, a
product with an intended use to analyze data has one function: analysis. A
product with an intended use to store, transfer, and analyze data has three
functions: (1) storage, (2) transfer, and (3) analysis.” (FDA, 2020a, pg. 4)
The guidance indicates the FDA does not intend to review non-device, low risk device, or
devices subject to enforcement discretion. It does, however, place expectations that the sponsor
must be able to separate and articulate the medical device functions from the non-device

22
functions in the indications for use, system architecture, and verification/ validation of the
device. Additionally, in designing the device, software partitions may be required to clearly
separate the device from the non-device functions. One example in the guidance is for a mobile
app that detects skin cancer. In this case, the software that detects skin cancer would be the
device subject to review. The “other functions”, not subject to review, would be the smartphone
(including computer operating system and electrical-safety testing for the phone) as well as the
phone’s camera. The manufacturer must be able to separate the software to analyze the photo for
skin cancer from the general operating system and software for the camera. Additionally, while
the non-device functions are not reviewed, the manufacturer must understand their limitations
and potential impact on safety and effectiveness. This is a similar expectation for when an
external computer is used with a medical device, however, it is now applied with the SaMD
available through a handheld computer. Through its activities during this time, the FDA sought
to use guidances to carve out areas of low risk devices that would require less oversight, so that
greater focus could be directed toward ensuring the safety of higher clinical risk devices.

23
Figure 3: Medial Device Software on a Risk Spectrum from 2013-2019
Software as a medical device (SaMD)
Software intended to be used for one or more medical purposes that
perform these purposes without being part of a hardware medical device
Medical Device Data Systems(MDDS)
Software solely intended to transfer, store, convert format, and display medical device data
including results, medical images, waveforms, signals, or other clinical information are not devices
General Wellness Devices
Intended only for general wellness and present a low risk
to the safety of users and other persons
Clinical Decision Support Software (CDS)
Software not intended to acquire, process, or analyse
Intended for purpose of displaying, analyzing, or printing medical information.
Intended for purpose of supporting or providing recommendations
Establishes Device / Non Device criteria.
Low RISK CLASS High
alarm
display
Modification
printing
Interpreting
signals
Display vitals
Clinical Artificial
Intelligence Data
2.4 Software Development Lifecycle Models (SDLC)
Not only has technology evolved, but so too have the models to develop digital products.
The first so-called “lifecycle” models of software development (SDLCs) had frameworks of
varying complexity that had been well characterized, applied, and studied for a variety of
consumer and business-based applications (Samra, 2012; Özcan‐Top and McCaffrey, 2018;
Lima, 2020). Like the design control model utilized by the FDA, software design models sought
to ensure that a product performs as intended and meets the needs of the end user. These models,
though, were not directed at the development of medical devices but rather at consumer products
that were not governed by FDA regulations. Thus, they did not have to function under the
constraints posed by the expectations and oversight typically associated with the clinical use of
SaMD. Given this freedom, developers began to experiment and then adopt new software
development models, some of which would have strategies for software design without the same
staging structure as the FDA’s regulatory design control strategies.

24
Perhaps the oldest and most traditional of the software development models has been the
waterfall, also known as the cascade method. This method appears to have gained widespread
recognition in the 1950s when the MIT Lincoln Laboratory used it to help produce the semiautomatic ground environment air defense system (SAGE) (Benington, 1983). In this complex
project, nearly half a million computer instructions had to be programmed and tested, a challenge
that called for a systematic, documented approach between the programmers and the machines.
The waterfall method was further refined by Royce in 1970, who described its construction as a
series of steps with defined activities (Figure 4). Each step in the sequence contains a feedback
loop designed to test and document the relationships between the activities under study with
respect to the requirements or inputs already defined at a previous stage (Royce, 1970;
Rajagopalan, 2014).

25
Figure 4: Royce Cascade Method of SDLC
This figure, adapted from Royce’s publication on the cascade method, shows the progression
from requirements through coding to testing. There is an additional feedback loop during design
that may lead to a change in system requirements. Similarly, during testing, this may lead to a
change in design (Royce, 1970).
A modification to the waterfall method known as the Vee (V) modification was proposed
in 1991 by NASA as part of its Software Management and Assurance Program. The modified
system incorporated specified stages of design review more explicitly to satisfy the requirement
for progress reports to satisfy Department of Defense (DOD) contracts. Like its parent method,
the V method starts with a systematic assessment of user needs that can then be designed and
built into the product; these are illustrated on the left side of Figure 5 by a downward arrow.
Once the design phase has progressed to a certain stage, development progresses through
integration and verification steps on the other side of the V, depicted on the right side of Figure
5. What differentiates the model is the need to identify points at which feedback will be reviewed
as development and implementation progress to connect requirements (inputs) on the left side
with implementations (outputs) on the right side. Thus, the V model emphasizes the importance
of the feedback points to test the detailed design, compare the system integration with the system

26
architecture, and confirm that the final product meets the initial requirements (Forsberg and
Mooz, 1991).
Figure 5: Vee (V) Model of Development
(Ruparelia, 2010)
Development in this model progresses down the left side from requirements, further defining and
dissecting them to development. Next, the development progresses up the arrow through
integration and verification testing. At each stage, there is a tie back to the original requirements
to review and ensure the requirements have been met.
The waterfall/ cascade and V models are all based on a sequential design plan that places
most planning at the beginning and implementation at the end. They are amenable to concurrent
hardware and software development, which is often important when developing hardware-based
medical devices. Because the models were originally designed specifically to meet the
documentation requirements for projects associated with the US Federal Government, they could

27
fit easily into the design control frameworks of regulatory agencies, that specified nodes for
reviews, verification, and acceptance.
More recently, however, another class of SDLC methods has emerged, mostly from
actors outside of the medical device sector. The culture and views of Silicon Valley software
companies differ from other industries and have brought a different perspective to development.
Their new methods emphasize collaborative techniques in which the development process is
structured differently (Cockburn, 2006).
One such model that demonstrates this shift is the Spiral Method shown in Figure 6. In
this model, the first areas on which development is focused are on those that can identify and
resolve risks, working first on areas of greatest difficulty or risk. Design inputs are revisited at
the time that each progressive prototype is created, fostering more frequent cross-functional
conversations to refine the attributes of the product.
One such model, the Spiral Method shown in Figure 6, emerged in 1986. This model
shifted away from pursuing development according to a stepwise plan and focused instead on
maximizing attention to project and product risk. In this model, the first areas on which
development is focused would be those that can identify and resolve risks, so that software
engineers can be directed to work first on areas of greatest difficulty or risk. The effectiveness of
the method is often improved by assuring a close interaction between the developers and the
users, since users can help to illuminate areas of risk that might be unseen by the engineers.
Design inputs are revisited at the time that each progressive prototype is created, fostering more
frequent cross-functional conversations to refine the attributes of the product.
A modification of this approach, the Rapid Application Development (RAD)
methodology proposed in 1991, emphasizes rapid development by expanding the use of a

28
collaborative approach within the technical team. Techniques such as open-source code
development and an increased focus on a partnership with customers are also characteristic of
this collaborative SDLC model. Open-sourced coding is an approach in which existing software
code is made available for anyone to use, modify or share. This allows developers of multiple
products to incorporate or modify existing software material into new products (Ruparelia,
2010).
Figure 6: Spiral SDLC
(Ruparelia, 2010)
In this model, development starts in the center of the spiral and progresses out. There are
multiple passes through the development stages: determining the objectives, identifying risks,
designing, and testing. At each pass, the key risks are addressed refining the product.

29
Experience with the benefits of collaboration during software development led to even
newer models of SDLC, including Agile techniques, that attempted to understand the customers’
needs and develop suitable software more efficiently. These models have been previously
described in detail in other publications (Samra, 2012). They draw on principles of behavioral
and teamwork studies expounded by the “Agile Manifesto.” The Agile Manifesto is a collective
statement laying out twelve principles that define an ideal environment for Agile development
and emphasize interaction and collaboration in staged activities. One technique central to the
Agile method is the use of “scrums.” (Figure 7) In a scrum, a reduced set of prioritized product
requirements is created, and software development teams meet frequently, often even daily, to
address these requirements. Those meetings include individuals from the business side of the
company as well as a core team, usually consisting of a scrum master, the product owner and
development staff. The work is done in rapid, small sprints, with a specific objective toward a
prioritized customer need, to get at least some of the more critical parts of the software into the
hands of users so that they can provide feedback early in the implementation phase. Through
these collaborations, the developers form an ecosystem in which the exchanges provide learning
opportunities and shared expertise to ensure that the software works best for the users (Beck, et
al., 2001; Cockburn, 2006).

30
Figure 7: Scrum Agile Method
In each sprint the scrum master leads focused work between the developers in collaboration with
future consumers. A defined and limited set of features is targeted for development. Through
this series of sprints, a completed product is designed.
Another method employed by Agile is called “test-driven” development. Here the test
for the software defines both the test scenario and acceptance criteria for a specified feature even
before the code is written. (Figure 8) The development team then writes the code, tests it, and
revises it until it meets the predetermined test and acceptance criteria (Dima and Maassen, 2018).
Once one set of features seems in control and is being polished, a new set of prioritized features
are added to be coded into the software next. A key feature of both techniques, in contrast to
traditional methods, is that specific user needs and product requirements can be added, changed,

31
or removed at later stages of the development process based on feedback from both the
developers and the users.
Figure 8: Test Driven Agile Development
Agile methods also have additional sub-models with slightly different approaches and
goals. For example, “extreme programming” (XP), shown in Figure 9, was introduced to deal
with unacceptably long timelines for developmental activities. Extreme programming (XP) pairs
two developers who, together, use firm coding rules to solve an issue rapidly by progressing
through a systematic process of design with frequent iterations. In the working relationship, one
developer of the pair is responsible for driving the main writing. The other member
independently edits and refines the code. Its driving philosophy is to take certain Agile
principles- simplicity, communication, feedback, respect, and courage- to the “extreme” (Lima,
2020).
Another modified method takes its direction from what is called “behavior-driven”
development. In this method, software developers work in tandem with the consumers who will
ultimately use the product to write the preliminary user needs in a plain language coding system

32
known as Gherkin language. This step captures not only the purpose of the software but also the
way in which the user will interact with it (Lima, 2020). This initial stage is only a framework
for the initial development. As more detailed codes and features are added, the developers
continue to work with the consumer of the product to understand how the product will be used in
practice.
Figure 9: Extreme Programming
Agile techniques such as these can incorporate user-driven “stories” in place of
formalized user requirements and foster cross-functional teams with users to determine how best
to write that story (Boehm, and Turner, 2003). In all these methods, the software is tested
traditionally for its functionality and validated by studying the experience of the user. The
methods are particularly helpful for consumer-driven software that could be greatly improved by
attention to real-life feedback gained when the product is placed in the hands of the users. The
focus of development becomes the interactions and experience gained from users rather than the
collection of auditable steps with associated specified documentation. For consumer products,
such as cellular phones, however, the main “user” is the purchaser. For medical devices, there is

33
another stakeholder- the regulator- whose needs must be met before the product can be
purchased.
2.5 Applying Software Development Models to Medical Devices
As explained in the previous sections, the last few decades have been a time of change
for both regulations and technology. Tension exists between the need to integrate novel software
engineering approaches such as those outlined above with the more rigid regulatory expectations
for design controls. Perhaps the best match of design methods to mate with regulatory
expectations is the traditional waterfall method for software development. Like the design
control framework for medical devices, it uses customer requirements to direct product
specifications (“inputs”), refines the design through verification and validation testing, and
documents all activities.
In the book The Inmates are Running the Asylum, the tension between software experts
and users is examined by Cooper, a recognized software expert (Cooper, 2004). He posits that
the engineers who excel at developing software often differ from the average individuals who
use the technology routinely in their daily lives. The precise, detailed, technical knowledge
required to write and develop software is often divorced from an equivalent knowledge of user
needs and aptitudes. This concern is particularly important when applied to the development of
software for medical devices. In his book, Cooper reflects on a startup technology company:
No matter how early in the development process specifications are drafted,
they cannot substitute for interaction design. And no matter how hard they try,
programmers cannot consistently arrive at a successful design. Not only are
their methods, training, and aptitude wrong for the job, but they are caught in
a strong conflict of interest between serving the user’s needs and making their
programming job easier. Yet, in company after company, software engineers
are allowed to control the development process, often from start to finish.
(Cooper, 2004, pg. 81-82.)

34
The evolution of software methods such as Agile methods makes clear that efforts are
being made to work with users more closely. In those more recent development paradigms, a
stronger connection between software performance and user feedback allows the developer to
respond rapidly should deficiencies be detected during development (Gordon and Stern, 2019).
Without the specified staging typically required for design controls, however, FDA inspectors
and reviewers who must evaluate the design process are often challenged to understand whether
software developed with collaborative methods has in place the required design control elements
specified in design control regulations (McHugh, et al., 2013; Kim, 2022).
One of the first steps in developing software for use as a medical device is to decide
whether the proposed product even meets the definition of a medical device, and thus will be
under the purview of SaMD regulatory requirements. A 2018 survey of 130 experts involved in
creating standalone software (“mobile apps”) under the European Union Medical Device
Regulation showed that respondents struggled to understand if they met this first test (Morrison,
et al, 2018).
Medical devices are regulated based on how they are used clinically. SaMD, though, can
frequently be used for both medical and lifestyle applications, and this blurs the lines between
their classification as a regulated versus non-regulated product. The user may not know, or care,
if their product is considered as a medical device. Without the solid anchors of a physical
product, a defined use environment, or limits on the technology, the ability to categorize
software as a device can be confusing and challenging.
DHT are the product of a process of ‘medicalization’ and ‘life stylization’ at
the same time: they enable the entry of lifestyle data (e.g. sleep tracking
information, step counting, and dietary habits) into the medical sphere, while
medicalizing people’s lifestyle choices at home… In attempting to standardize
quality and safety requirements, thus defining the scope of medical DHT, these

35
national regulatory bodies play a crucial role in shaping the lifestyle/medical
boundary for digital health technologies. (Lievevrouw, Marelli, and Van
Hoyweghen, 2022b, page 551)
If the new product is a medical device, then the degree of control for software will be
determined by the device risk classification, a second key step for companies to assess (Figure
3). A risk-based approach allows software with a less serious medical impact to require less
documentation, whereas more critical software applications must undergo more rigorous testing
and will be subjected to a full design review when marketing permission is sought. The
expectations for this documentation are outlined in the Guidance Content of Premarket
Submission for Deice Software Functions from draft guidance released on November 2021 and
final guidance released June 2023, summarized in Table 2 (86 FR 60838, 2021; FDA, 2023). The
final guidance expands considerably on the draft guidance, including more detailed examples
and a larger description on risk management. This document delineates two different review
pathways based on the criticality of the software. In the final guidance, the FDA clarifies that any
SDLC method may be utilized by the company as long as it meets the requirements under the
QSR. Regardless of the path, however, the company must be able to demonstrate documented
evidence to support the design history of the product. The final guidance also mentions the use
of specific Agile techniques for inclusion in premarket submissions, including incremental/
evolutionary development practices, user stories, and use- cases. Whatever its form, the
documentation must show that the device has been designed and tested to meet the intended use
of the device.

36
Table 2: FDA guidance, Software review
(FDA, 2023)
Software
Documentation
Basic
Documentation
(Low Risk)
Enhanced
Documentation
(High Risk)
Commentary
Documentation
Level
Examination
Yes Yes In this section, the
manufacturer determines and
explains the risk and
documentation level for the
FDA. This allows the FDA
reviewer to understand the
logic and what to expect in
the submission.
Software
Description,
Architecture, and
Risk Management
File
Yes Yes This section explains the
software to the reviewer.
Software
Requirements
Specification
(SRS)
Yes Yes This section defines the
needs for the software and
traceability throughout the
documentation.
Software Design
Specifications
(SDS)
None Yes This section allows the FDA
to fully understand the
design of the software.
Software
Configuration
Management and
Maintenance
Practices
Yes Yes In this section, the company
can declare compliance with
IEC 62304 or provide details
to support a controlled
practice.
Software Testing
(Verification,
Validation)
Summary Only Detailed
Protocols and
Results
This section is the main
difference requiring
verification and validation
reports that can be reviewed
in sufficient documentation
to ensure the design inputs
have been met.
Revision History List List This section allows the FDA
to trace software updates.
Unresolved
Anomalies
List List This section allows the FDA
to understand any
unresolved issues or
software bugs.

37
If the SaMD meets the first test for being a device and the second test for requiring
design control, the developer must then choose a SDLC to guide the development of the
software. Many developers find that waterfall or V models are limited for their purposes.
Consequently, they have been trying to understand how to incorporate aspects of Agile
development within the constraints of regulatory expectations that emphasize the need for
structured documentation. Many reports, case studies, and proposals have been made in this
regard (Heeager and Nielsen, 2020). Perhaps leading this investigation was the work of McHugh
and colleagues to modify the V model, previously discussed in Section 2.4, to marry the benefits
of compatibility of the waterfall methods with the benefits of time and cost associated with Agile
methods. In 2013, a cohort of researchers named McHugh, Cawley, McCaffery, Richardson, and
Wang conducted a series of structured interviews with seven companies to determine the SDLC
each was using. They confirmed previously identified views that the waterfall method met the
design control requirements most easily but imposed a high regulatory burden on the companies,
resulting in the need for additional personnel and specialized documentation. At the same time,
most traditional companies reported challenges when they tried to adopt Agile methods despite
the benefits of time and cost savings. Following the structured interviews, the research cohort
responded to the challenges regarding the use of Agile methods by proposing a modification to
the V Model that incorporates aspects of Agile development during each phase. During that
work, they identified 59 possible Agile practices that were broadly in use, 13 of which appeared
to show promise for integration into the V Model at different stages of medical device
development. For example, they proposed that use- cases and user stories could be applied when
specifications were being established. Then during verification/ validation the techniques of pair
programming, continuous integration, and continuous testing could be utilized (McHugh, et al.,

38
2013; McHugh, McCaffrey, and Casey, 2014). The 2013 paper by McHugh et al., however, did
not specify the criteria by which they came to designate the 13 Agile techniques as appropriate to
meet regulatory requirements.
The model framework was further developed and validated through several iterations.
Özcan‐Top and McHugh (2018) then positioned the proposed framework as a product of a spin
out company, MDevSPICE, from the Dundalk Institute of Technology, where the previous
research was conducted. That company then tested its model with the help of four unnamed
companies that had a large presence in Ireland, chosen for their ability to participate in the
validations. One was a small company developing medical device apps, a second was a small to
midsize company developing non-device software, a third was a small company developing web
based medical device software, and a fourth was a multinational company developing medical
software. The study was limited, however, because only one of the companies, the multinational
company listed fourth, had a functioning QMS. Specifically, the following items were suggested
as Agile methods that could be successfully integrated:
Commitment Scale: A practice for Agile stakeholder management
• User Story Mapping: A process of laying out requirements in the order that the
user will experience them.
• Acceptance Criteria: Defining what the software must do to better establish
criteria for verification and validation testing.
• Definition of Ready: Detailing out and prioritizing the exact deliverables for the
definition of done
• Automated testing suites: A step towards continuous integration of the software
code and deployment.

39
While the main point of the research was to validate the future development software for
commercialization, the study provided beginning evidence that Agile techniques could be
incorporated by companies (Özcan‐Top and McCaffery, 2018).
In a separate paper, Zema and colleagues reviewed the regulatory requirements and
mapped a workflow that they felt could incorporate waterfall and Agile techniques while still
meeting regulatory requirements. They found that the essential element was not the specific
engineering techniques that were chosen, but rather how the technique was mapped and
integrated into the QMS and how they were captured by the technical documentation. It was
limited, however, by its theoretical basis, so it did not address how well this could be utilized by
industry (Zema, et al., 2015).
Lima, in her 2020 master’s thesis, sought to design and test an Agile-based SDLC
process that could integrate the scrum practice and user stories while still meeting the regulatory
requirements of the US and EU through compliance with IEC 62304. She noted that previous
academic approaches had explained how Agile could be integrated but did not specifically
address the challenges that a company would experience during evelopment, such as the need for
specific documentation appropriate for regulatory review. In designing her Agile method for
regulated software, she brought an Agile expert and a quality auditor into the process so that the
team could address together how to integrate Agile and, at the same time, document the process.
She then applied this method by developing a software package for the Hospital Pedro Hispano
that included a mobile medical app that allowed a patient to track headaches and a web
application to capture that information for physicians. The complexities that she found with this
process were seen to relate not to the technical method chosen to develop the software, but rather
to helping the team to understand how to use the techniques and how to produce documentation

40
appropriate for a review by the quality team. She additionally interviewed three Agile experts -
Analia Irigoyen, CEO of Promove; Carlos Rodriguez, Scrum Master / Agile consultant at Radix;
and Samira Tavares, Agile expert at Knowledge 21- for their opinions on her proposed process
and more generally on their thoughts regarding the use of Agile SDLC in regulated medical
devices. Their opinions differed. Irigoyen stated that she preferred a hybrid approach for
regulated software. She found that more Agile could be integrated if the development team had
defined budgets and members that were well-trained and established at working together.
Rodriguez also favored a hybrid model. “For Carlos, every project could be agile, but not every
project should be.” He pointed out that issues arise not only from the software process but also
from the management of clients and stakeholders. This would then affect the choice of SDLC.
However, Tavares felt that every project could be managed with Agile as long as developers
were diligent about addressing the risks at each stage.
Exemplar cases also exist as proof that software device development can meet medical
device requirements, even when using Agile techniques. In 2009, Abbott Laboratories described
how they used Agile techniques to develop one of their diagnostic systems, the m2000 PCR
molecular system, which obtained a Class III approval in the United States. The paper describing
their experiences spoke to the challenges of implementing Agile techniques into the company
since the techniques were new to both the software teams and the broader leadership.
Nevertheless, they identified that the shift had cost benefits. When compared to previous
experiences with a similar device, the ARCHITECT i2000, which was developed using a
waterfall methodology, the company found that it needed a smaller staff and had shorter
development timelines, in part because the development teams made fewer errors in the final
testing of the m2000 system (Rasmussen et al., 2009).

41
In that same year, Heeager and Nielson studied the experience of a Danish
pharmaceutical company that was moving to incorporate Agile software development into one of
their regulated high-risk devices. They found that the greatest benefits were associated with the
implementation of some of Agile’s teamwork elements, including self-organizing teams, burndown charts, and stand-up meetings. A burn-down chart is a project management illustration that
graphs the work that needs to be done versus time. At the same time, the path was not without
challenges. The authors stated:
“We find that the agile process suffers from its adaptation to the FDA
standard, the bureaucratic management of the project, and the sequential work
processes of the other project groups, which cause interruptions within the
iterations, changes to the hardware and user interface that the software has to
cooperate with. This implies that the software team experience trouble
implementing a fully tested increment during the iterations and it leaves the
software unstable.” (Heeager and Nielsen, 2009, pg. 213)
Both case studies demonstrate that it is possible to use blended methods to produce medical
device software, but there are challenges to confront with this integration.
Given the benefits of Agile methods for development, the question then arises about the
extent to which these methods are being used in medical device companies. Some information
on this evolution might come from comparing what we know about their adoption from studies
at different points in time. In 2014, McHugh, McCaffery, and Casey performed a literature
search to understand perceived barriers to the adoption of different SDLC approaches. They
defined “perceived barriers” as “barriers that organisations feel exist, but through the literature
review performed are in fact only superficial… Examples of perceived barriers include the
following: Regulatory Control; Traceability Issues and Lower Quality Software.” (see also
Table 3). They then used a survey of twenty companies with branches in Ireland that were
involved in software device development to understand actual compared to perceived barriers to

42
implementation. In this small sample, 50% of the companies were using the Vee (V) Model,
25% were using modified Agile techniques, and 25% were using other methods, including
waterfall development. The companies that they surveyed had, in fact, a different set of actual
implementation barriers than might have been expected from previous perceptions. The actual
challenges were largely based on personnel issues such as lack of expertise or problems with
management support. Half (50%) reported a lack of experience with the techniques, 33% with
needs to change their current practices, 16% with management opposition, and 16% with the
sizes of teams.
Five years later, Dima and Maasen surveyed large IT companies, including Google,
Oracle, Avira Soft SRL, Freescale, Gameloft, Ubisoft, Luxoft, Ixia, and Cargo Partner, by using
the more rigorous Delphi method. The Delphi method progresses through an examination of
previous literature to the development of study objectives, and then structured interviews to
collect data. It then circles back to the participants for a discussion to validate their findings. The
structured interviews with experts in Europe and America focused first on current SDLC
utilization and then moved to considerations of management and future trends. This survey
found that 68% of respondents were using Agile techniques. Additionally, 52% of the
respondents mentioned that the future trend would be to move to Agile or other new techniques.
One of the main disadvantages found with Agile was the stress placed on employees from the
process (Dima and Maassen, 2018).
A 2019 literature survey specifically related to medical device development was
performed by Alsaadi, Lisitsa, Khalaf, and Qasaimeh to compare SDLC techniques with
requirements in the European Union. In addition to a literature review, they also mapped the
requirements from the European Union Medical Device Regulations that could impact the

43
software development process. They concluded that Agile techniques do not meet the
requirements of the European Union Medical Device Regulations, mainly due to challenges with
documentation and traceability. Specifically, they identified that the philosophy in XP, which is
characterized by software with simple documentation, was a challenge because that limited
documentation would not meet regulatory requirements. The results of the study were obtained
from authors who specialize in computer science and were not confirmed by industry or crossfunctional experts (Alsaadi et al., 2019). These studies and experiences expose a complex system
with variable levels of implementation success and satisfaction for the SDLC chosen in device
software development.

44
Table 3: Perceived versus Actual Barriers to implementation of Agile techniques.
(McHugh, McCaffery, and Casey, 2014)
Perceived barriers to the implementation
of Agile techniques from the study
Actual barriers to the implementation of
Agile techniques from the study
25% Lack of documentation
25% Regulatory compliance
16% Lack of upfront planning
50% Lack of experience
33% Need to change existing practice
16% Team size
16% Management opposition
2.5.1 Recent regulatory changes to address DHT
Even ten years ago, the FDA had already heard much about the concerns of industry
regarding barriers when developing software as a medical device. In response, it began a series
of policy shifts that might address some of these concerns, to the extent that they could stay
within the current constraints of the enshrined medical device regulations. This attention was
initially directed at solving problems brought to their attention by established medical device
companies that presumably had a history of working closely with the FDA. Presumably also
would be the expectation that those companies would be well acquainted with requirements for
design controls, such as the need to develop a design history file and to work within the current
constraints in the existing legislative frameworks. However, more recently, these are not the only
developers with an interest in SaMD. An increasing number of such software applications are
coming from new entrants to the medical device field. These entrants are more familiar with
consumer applications that are not subject to FDA oversight, so they may not be very
knowledgeable about FDA processes (Gordon and Stern, 2019). The consumer-based entrants
might not even be aware that regulations exist, and, when informed, might still struggle with
understanding the constraints imposed by those regulations. In response, FDA has had to
readjust its approach, to work with a broader community of industry stakeholders.

45
2.5.1.1 Quality Reviews: The FDA Precertification Pilot Program
In 2017, the FDA launched a pilot program, called the Software Precertification Pilot
Program (“Pre-Cert Program”) (FDA, 2017b) to deal with the special challenges that software
can pose. They selected nine industry participants- Apple, Fitbit, Johnson & Johnson, Pear
Therapeutics, Phosphorus, Roche, Samsung, Tidepool, and Verily- as partners in this initiative.
Only two of these companies, Roche and Johnson & Johnson, were companies that had
traditionally produced drugs or medical devices. The others were consumer-focused technology
companies that had more recently adopted a mission and vested interest directed at health care.
The novel regulatory program explored a different approach to exercising oversight for software,
based on the premise that it was more appropriate to monitor the quality and design control
practices of the company rather than focusing on the individual software products. In this
alternative system, FDA proposed that companies could certify their quality management system
and then monitor their system by inspecting key product indicators (KPI)- metrics that could be
tracked to ensure that the company’s QMS is effective (Shuren, Patel, Gottlieb, 2018).
In 2018, the FDA began to operationalize the program using a pilot group of companies.
Both the companies and FDA would monitor certain KPIs: complaints, data quality, defects,
device activations/ user adherence, regression testing, releases, rollbacks, services, and security
(FDA, 2022c). Once precertified, only high risk devices produced by that company would
require premarket reviews before they were placed into interstate commerce. The new approach
was welcomed by the companies. They expressed the view that such an oversight system would
facilitate the quick responses often needed in the software development cycle and would foster
the company’s ability to adapt and reduce risk in software. Nonetheless, Shuren, Patel, and
Gottlieb, all working for the FDA, wrote an editorial in JAMA the following year that questioned
how easily the program could be integrated into the existing framework for design controls,

46
considering that a framework designed for medical devices based on hardware was not well
suited to regulate software as a medical device (Shuren, Patel, Gottlieb, 2018).
The idea of precertification constituted a dramatic shift in the way that products could be
designed and developed in the US regulatory system. It would redirect FDA’s review efforts
from a traditional focus in which the detailed design and function of the device were scrutinized
to one based on assessing the culture and performance of the parent corporation. This novel
approach also signaled a new role played by software developers in affecting policy design
(Lievevrouw, Marelli, and Van Hoyweghen, 2022b).
The implementation of the FDA’s pilot program over the next five years was described in
a recent summary report, The Software Precertification (Pre-Cert) Pilot Program: Tailored
Total Product Lifecycle Approaches and Key Findings (FDA, 2022c). In it, the FDA described
the challenges and constraints of the pilot program. Of concern was the small number of
companies that had been involved in the program, all of which were developing devices
submitted through the de novo review pathway. The report concluded that precertification pilot
program was able to provide proof of concept for the novel principle of regulating the company
and not the product. However, the report concluded that the system was not workable within the
current regulatory framework, as predicted previously by Shuren and colleagues. In order to be
workable, the regulations would need to be modified to allow for variation from the current legal
requirement that medical devices conform to the tiered model for risk classification and
demonstrate a design control approach with defined stages, as stipulated in the existing
regulations (FDA, 2022c).

47
2.5.1.2 Alternative Review and Submission approaches
The FDA has looked not only at its review processes and regulations, but also at the
suitability of its own structure to address DHT. To this end, it announced the Technology
Modernization Action Plan (TMAP) (FDA, 2019b; FDA, 2022b). TMAP involves investing in
infrastructure and internal resources to strengthen its review processes and engage more
effectively in partnerships with its stakeholders (FDA, 2022a). The plan is part of a larger
investment in modernization that includes a reorganization and formation of the Office of Digital
Transformation as well as a Data Modernization Action Plan. (DMAP) One area of focus within
this plan is “use-cases” - targeted areas in which it might explore solutions to vexing problems
identified in many areas, from premarket support and reviews to larger surveillance programs. A
use case examines a device from the point of view of the user, modeling how the device will
work in a situation and how the user will interact with the device. The knowledge gained from
this research is envisioned to allow FDA to build and test new or modified requirements before
scaling them up more generally.
The value of use-cases has already been explored in a different application. An approach
based on use-cases was piloted in 2006 as a novel submission and review approach to the review
of infusion pumps, which had been plagued with safety issues. These issues included at least
56,000 adverse event reports and 87 recalls reported in the period from 2005-2009 (FDA, 2010).
North Carolina State University and the FDA together designed a new paradigm for developing
regulatory submissions for these troublesome infusion pumps. Among other things, they
considered how best to conduct software reviews with a focus on improving product safety.
Together, the same group built a usage model, which was a formal software
representation focusing on safety and device characteristics. Once the model was built, test case
scenarios could then be developed to test the actual design of the software. For example, the

48
infusion pump works when a healthcare professional enters a value into the keyboard and the
pump then dispenses the medication according to the parameters entered. Models can be built to
test for common errors that have resulted in adverse events, for example missed or duplicated
keystrokes, conflicting value entries, and challenges with alarms. With the usage model, the
software can be tested against these real-world scenarios. The authors state:
“Although these regulatory processes work reasonably well for device
production processes, they’re insufficient for assessing software. From a
premarket perspective, subtle errors and latent bugs might exist in the software
that only formal model-checking techniques or exhaustive white-box testing
can detect….Few device manufacturers use model-based designs. Rather, they
build the software based on their understanding of the product requirements.
This can produce usage inconsistencies among the various devices which can
result in harmful situations.” (Jetley, Iyer, and Jones, 2006, pg. 61)
In other words, the test case focused on reviewing the performance of the device, rather than
focusing on the documentation for the design and development of the product. This presents
another alternative route by which SaMD, more generally, might be regulated in the future.
Changing regulatory approaches are also reflected in the activities of the Office of
Science and Engineering Laboratories (OSEL) at the FDA. This research arm of CDRH was
charged with exploring emerging issues that could challenge the effective review of medical
devices. In 2018, OSEL published a report that discussed the potential advantages of having a
Center of Excellence for software expertise, a central group whose members are trained
specifically in software reviews and who could be utilized by all centers at the FDA. By drawing
on the Center’s experts, individual review teams would not need to develop their own experts
(Morrison et al., 2018). This center was formed and began to track their reviews. By 2018, the
OSEL Center of Excellence had tracked 2500 consultations in which the expert pool aided in
regulatory reviews. Additionally, this Center participated in mock reviews that allowed the

49
industry and the Agency to practice interactions for new technologies using a theoretical,
structured review format. This experiment again illustrates the commitment of the FDA to deal
with challenges associated with emerging technologies – including software.
2.5.1.3 Submission decisions
Perhaps the critical test of whether a system is effective is the ability to use it
successfully. The records now available in the public domain suggest that at least some
software-based products are gaining market authorization under the current review system. For
example, in 2018, the FDA announced that the Apple iWatch had been cleared to store, transmit,
and display ECG waveforms as a method to detect and notify atrial fibrillation (AFib) or sinus
rhythm (DEN180044 and DEN180042) using the de novo pathway (FDA, 2018b;FDA, 2018a).
These clearances, though, came after significant negotiations regarding the nature of clinical
trials required to demonstrate that the software worked as intended (FDA, 2018c). In 2020,
Apple and the FDA announced breakthrough designation and clearance for Nightware, a
prescription device to detect and monitor for nightmares with Post-Traumatic Stress Disorder
(PTSD) through De Novo DEN200033. These cases are interesting because the Apple iWatch
hardware can be used as a non-medical device, lifestyle device, over-the-counter device, and
prescription medical device, all regulated by different rules and pathways and through the
development of individual applications.
The fact that a product has gained regulatory approval, though, does not give much
insight into the challenges that might have been faced in gaining that approval. It is the nature
of regulatory submissions that the detailed conversations and recommendations from FDA are
largely confidential. Thus, any compromises between the FDA and industry related to the
methods for software development, verification and validation cannot be obtained as publicly

50
available information. These clearances point out that while the system of regulation may need
to be reviewed, however, it is still possible to bring this technology to market.
2.5.1.4 Context for Change
All the information that can be gained from the current literature suggests that regulatory
review and oversight of software products is still challenged by the very different nature of
software development. However, the process for a change in policy involves more than industry
and the FDA. The ability to change regulations to address new technology is closely tied to the
legislative and political process. In this domain, opinions have shifted from questioning whether
the FDA has the authority to regulate DHT to questioning how that is done. Many are concerned
that regulations may stifle innovation that could be a benefit to society. Soo, in his dissertation,
states that the social force, including satisfaction with the current state, impacts both the pace and
the text of the regulations that emerge (Soo, 2014). Thus, it is important to understand better how
each of the stakeholders contributes to this complex set of political forces. The work proposed
here looks at the issue through the eyes of one stakeholder, the industry trying to commercialize
SaMD.
2.6 Framing proposed research
To date, the existing literature points to an oversight system for SaMD that is in
transition. In this changing environment, some companies have found ways to navigate the
system in order to have their products approved. It is not clear, however, if companies are
finding this path transparent or satisfactory. Thus, we have relatively little systematic
understanding of how industry is coping as it attempts to implement software design and how it
views some of the current and proposed changes with which the FDA has been experimenting. A
more systematic analysis of the views and frustrations of those who are developing software

51
products has the potential to inform existing and future regulations. It is the aim of the research
presented here to develop and use a novel survey instrument to explore the views and challenges
of industry experts engaged in the development of SaMD.
In order to create a survey instrument, it is important to frame that instrument with
reference to a model that helps to ensure that the topic is explored comprehensively. Various
frameworks with different perspectives, including implementation and satisfaction frameworks,
were considered for this purpose (Fixsen, 2009). I felt that a broader base of questions, however,
might be achieved by using a framework that focuses on policy analysis, since the policies of the
FDA seem to be central to the concerns outlined here concerning software development. Policy
Analysis focuses on how laws and regulations are developed and take effect. As stated by Walt
and Gilson (1994), it “draws on concepts from a number of disciplines: economics, political
science, sociology, public administration and history, and emerged as a subdiscipline in the late
1960s.” (Walt and Gilson, 1994) Walt and Gilson argued that much of health policy has been
focused on what changed in the laws and regulations, and not enough on the actors and process
that they went through in establishing the requirements. They presented the Health Policy
Triangle (HPT) to offer a broader framework for analysis that would address these aspects, as
diagrammed in Figure 10.
In the HPT, three elements represent the points of a triangle. The Context element is
directed at understanding the environment that can include the legal and regulatory environment
currently in place that might place constraints on policy development options. The second
element, Process, refers to the process of establishing, writing, negotiating, passing, and
communicating a new policy. The third element, Content, includes the information and
requirements included, or omitted, from the policy. In this research, Content can include the

52
nature of current regulation and guidance as well as proposed and pilot options under
consideration. These factors interact to some extent. For example, the content of regulatory
policy is heavily influenced by the process used to develop and communicate the policy change
as well as the context in the existing social-political environment. The groups that are active in
creating or using the policy system are called “actors”, which alternatively could be called
stakeholders. They are influenced by the external environment and, in turn, influence the
process, content, and context (Walt and Gilson, 1994).
In a recent literature review, O’Brien and coworkers attempted to gauge the extent to
which the HPT framework has been used by screening articles published over a five-year period
starting in 2015. They were able to find 54 published studies in which the HPT framework had
been utilized. They found that this framework had been generalized to study a wide variety of
policy situations, including those addressing health resources, physical/ mental health issues, and
postnatal care for example. Thus, O’Brien concluded that the HPT framework provides a
flexible base to simplify and structure studies for complex social systems (O’Brien et al., 2020).
Building on the conclusion of this prior analysis, which allows for the broad application of the
framework, the framework can also be applied for the purpose of this study.

53
Figure 10: Health Policy Triangle (HPT) Framework
(O’Brien et al., 2020)
This framework used here was therefore chosen as appropriate to study industry views
and experience by developing questions within each domain to take advantage of their expertise.
Target recipients were mid and senior-level regulatory and quality specialists who have dealt
with the challenges of managing the implementation of design control strategies for SaMD.
Table 4: Example Questions based on HPT Framework
HPT Node Example Questions
CONTENT 1. Is the available guidance related to SDLC clear and able to be
implemented?
2. Based on current guidance documents, is it clear what should be
included in the design history file for SaMD?
3. Is there transparent information from the Agency for when Agile
SDLC methods will / will not be accepted?
CONTEXT 1. What are barriers to implementing design control?
2. Have you had discussions with FDA on software?
PROCESS 1. Is the current regulatory system sufficient for design control for
SaMD?
2. Do you, or your organization, participate in the development of
software guidance and regulation?

54
Chapter 3. Methodology
3.1 Introduction
The goal of this research is to examine industry views regarding the implementation of
design control for SaMD. I began with a literature search, first to describe the current regulatory
framework for SaMD and then to evaluate what is now known about SDLC models and their
application to SaMD. This is summarized in Chapter 2. The research then proceeds through four
additional stages: the development of a novel survey to solicit feedback on the views and
experiences of the industry related to software design control; a critique of the draft survey by a
small focus group of experts; collection of survey responses from industry experts in this
domain; and the analysis of the results.
3.2 Survey Development
A novel survey instrument was developed with reference to the HPT framework
described previously in Chapter 2. First, questions were written characterizing and identifying
demographics for the respondents to allow for a stratified analysis based on the “actors”
described in the framework. Questions were then developed to address each of the main elements
in the HPT framework, relating specifically to the content of current regulations and guidance;
the context on the external environment; and the process for oversight and regulation changes of
design control. The number of questions in the survey was limited to respect the time of the
participants as well as to ensure that the survey would be completed by participants.
Additionally, skip logic was used to ensure that only relevant questions were presented to survey
participants. Together these sections provided the basis for the study.

55
3.3 Target Population
The survey targeted industry professionals and consultants at various professional levels
with direct experience in software design and development. Specifically, this target group
included mid to senior industry professionals in technical / engineering, quality, or regulatory
affairs with prior experience developing or supporting software in medical devices. To identify
participants, I contacted known individuals currently working in or having previously worked
with SiMD and SaMD through the following methods. I searched LinkedIn® for target profiles
with work experience in software medical devices and SaMD based on engagement with or
posting articles related to regulatory requirements for SaMD. I also screened job titles and
searched for companies known to work in development for this type of technology. I posted two
messages on LinkedIn® in October and November 2023 to solicit interest and ask for individuals
willing to receive the survey. I also reached out to alumni contacts through the University of
Southern California (USC) Department of Regulatory and Quality Science and alumni contacts
at the University of Wisconsin-Madison biotechnology program. Additionally, individuals
attending the 2022 US Medtech Summit, Software Symposium were approached to collect their
contact information. Then, utilizing a snowball method, the identified respondents were asked
to share the survey with similar industry experts following completion of the survey.
3.4 Survey Validation
Prior to administering the industry survey, a focus group was convened to solicit
feedback on the draft survey. The use of a focus group to validate the draft survey helped to
ensure that the wording for specific questions would be clear and able to be understood by
industry professionals. It also increased the likelihood that the results would be valid, avoid

56
unintended bias, and perform in a way that is fit-for-purpose (Seale, 2022; Devroe and Wauters,
2019; Ouimet, et al., 2004).
The focus group contained two industry experts, four professors from the Department
Regulatory and Quality Sciences, USC Mann Pharmacy, and one from the Viterbi USC School
of Engineering. The participants in the focus group were given a copy of the abstract, Chapter 1
of this Dissertation, and a copy of the draft survey. The focus group was then assembled through
a video conference meeting where the participants were briefly introduced to the project. To
accommodate the schedules of the participants, two separate video conferences were scheduled
on December 4 and December 6, 2023, each for approximately 90 minutes. In this meeting, each
question was reviewed, and feedback was received on the content, format, and design of the
proposed research survey. Following the focus group, the draft survey was then refined by
incorporating the feedback received from the focus group, ultimately reaching its finalized form.
The final survey contained 48 questions of which 12 questions were only displayed to
respondents indicating they had experience with a particular topic. The number of questions is
shown in Table 5 and the final survey is in Appendix A.

57
Table 5: Number of Questions in Final Survey
Survey Block Total Number of Questions
(Number of questions displayed based on
selected responses)
Demographics
Experience with medical devices
Experience with consumer software
Experience with Agile SDLC
9
(1)
(2)
(1)
Content
Experience with consumer software
Experience with Agile SDLC
Indicate Guidance need updating
14
(1)
(1)
(3)
Context
Experience with Agile SDLC
15
(1)
Process
Experience with Agile SDLC
Participate in Trade Organizations
10
(1)
(1)
3.5 Survey Deployment
The draft survey was developed utilizing Qualtrics. The use of an online, anonymous
survey made it possible to reach industry professionals with diverse geographic locations without
the constraints of needing to be administered at a specific time or place. The finalized survey
was also sent to two test participants to test the user experience. The test participants confirmed
they received the survey link, they were able to open the link, the format of the survey was
viewable on their computer, and that they were able to read the survey. Their feedback also
resulted in updates to the format of the survey to make it more usable.
The survey was then directly sent from Qualtrics open-access mailer to all individuals
willing to supply their email with a simple link requesting participation in the survey in
December 2023. The respondents opened the link and completed the survey on any available
computer. The email included a simple introduction to the purpose of the study, a statement on

58
the anonymity of responses, as well as an option to “opt out” of participation. No financial
compensation was offered to encourage participation.
Two follow-up reminder email was sent to recipients, approximately two and three weeks
after the initial deployment to the individuals who have not yet completed the survey.
Individuals who completed the survey received a thank you email that encouraged them to
forward an anonymous link, allowing for snowballing of the survey to other recipients.
Additionally, an anonymous link was also posted on LinkedIn and send out to the Regulatory
Affairs Professional Society (RAPS) distribution list. This link did not make it possible to
identify how many respondents were reached by the survey and anonymized the identities of
respondents replying by suing the link. However, the links contained embedded source data that
allowed these responses to be differentiated from those of individuals who were targeted directly.
3.6 Survey Analysis
The final results of the survey were analyzed utilizing Qualtrics and Excel after the
closure of the survey on February 11, 2024. In the follow sections the data for the survey is
presented utilizing the totals and percentage calculations. Text responses are analyzed with each
question and full text responses are in Appendix B.
Questions with responses on a preference or rank-ordered scales were analyzed utilizing a
weighted average (WA) to compare the strength of relative preferences or experiences. To
calculate the WA, each answer on the scale was assigned a numerical value, which was
multiplied by the number of selected responses for that choice and divided by the number of
respondents who answered that particular question. Respondents who indicated that they were
unable to respond to that question were excluded from the calculations. For example, Table 6
provides the different numerical values assigned on a 5-point scale.

59
Table 6: Weighted Average Value Assignment
Example Wording Example Wording Weighted Point Value
Strongly Disagree Extremely Difficult 1
Somewhat Disagree Somewhat Difficult 2
Neither Agree or Disagree Neither Easy or Difficult 3
Somewhat Agree Somewhat Easy 4
Strongly Agree Very Easy 5
Data to compare subpopulations for actors for Quality/Regulatory compared to
Engineering, medical device compared to consumer products, and self-reported agile experience
were cross-tabulated. Results are presented in Chapter 4 and Appendix C.

60
Chapter 4. Results
4.1 Survey Participation
The survey was activated on December 26, 2023 and sent via Qualtrics Direct Mailer to
118 individuals who had been selected to receive the survey. Of this population, 33% (39/118)
opened the survey and 30% (35/118) completed the survey, for a response rate of 33% (39/118).
The completion rate for individuals who began the survey was 90% (35/39). The population of
individuals who completed the survey were also invited to forward the survey to other colleagues
with relevant experience through an anonymous link. Additionally, an anonymous link to the
survey was also posted on LinkedIn, Regulatory Affairs Professional Society (RAPS), and
TOPRA. Individuals from the University of Wisconsin-Madison biotechnology alumni email
group were also sent an anonymous link. The anonymous link resulted in 83 additional
participants beginning the survey.
In total for all distribution methods, 122 individuals began the survey and 61 completed it
by answering all questions or by being screened out through skip logic (Figure 11). For
respondents who did not complete the survey, 56% (34/61) self-selected to leave in the first
quarter (0-25%) of the survey, another 33% (20/61) left the survey in the second quarter, and
only a few (3/61, 5%) dropped out at a later point. The survey was closed to responses on
February 11, 2024. Not all participants answered every question, therefore the number of
respondents for each question will be noted in the following presentation of results.
Figure 11: Survey Participation
Email Link 118 Individuals
Invited 39 Began 35 Completed
Anonymous
Link
Publically
Available
(no total)
83 Began 26 Completed

61
4.2 Demographics
Initial survey questions collected demographic information about the survey respondents.
Most had job functions in Quality / Regulatory Affairs (67/104, 64%) (Figure 12). Fewer
identified roles with Product Development / Engineering (23/104, 23%), Sales / Marketing /
Business / Legal (4/104, 4%), and Other (10/104, 10%). “Other” respondents self-identified as
portfolio management / strategy (1), medical / scientific (3), product management (1), operations
(1), manager (1), and consultant (1). Respondents who selected “Sales / Marketing / Business /
Legal” were moved using the skip logic function to the end of the survey.
Figure 12: Primary function in Organization
Q: What is your primary function in your current organization? (Choose the best answer.) n=104
Most respondents had experience with medical devices (77/116, 66%) (Figure 13). Fewer
had experience with consumer software (22/116, 19%) and some respondents (17/116, 15%)
selected “none of these apply”. Respondents who selected “none of these apply” were moved
using the skip logic function to the end of the survey.
Individuals who indicated that they had experience with consumer software were further
asked if they were familiar with design controls for medical devices. Most indicated that they
were familiar (16/19, 84%); only a minority were not familiar (3/19, 16%). Respondents who
were not familiar were moved to the end of the survey.

62
Figure 13: Experience with Medical Devices or Consumer Software
Q: Do you have experience developing software, or supporting a team that develops, any of the
following? (Select all that apply.) n=116
Those with experience in medical devices were asked about the nature of those products.
Most commonly, respondents had experience with Software as a Medical Device (50/69, 72%)
or Software in a Medical Device (51/69, 74%) (Figure 14). Fewer had experience with
lifestyle/low-risk health software (21/69, 30%) or software as an accessory to a pharmaceutical
product (14/69, 20%).
Figure 14: Experience with Specific Medical Devices
Q: Do you have experience developing any of the following products? (Select all that apply.)
n=69
Individuals who indicated that they had experience with consumer products were asked to
clarify the nature of those products. Most commonly respondents had experience with lifestyle /

63
health/ fitness products (9/20, 45%) (Figure 15). Other responses included media / gaming (1/20,
5%), information sharing (4/20, 20%), other regulated software (4/20, 20%), security (2/20,
10%), creative tools (1/20, 5%), productivity / planning (2/20, 10%), and database / data transfer
(7/20, 35%). Several (8/20, 40%) also added other experiences including software related to
pharmaceuticals that did not qualify as a device, Computer System Validation (CSV), the
military, robotics, analytics, and Laboratory Information Systems (LIMS). Those who added
additional comments also included “mobile apps” (fitness, health), “CDS” (Clinical Decision
Support) software, and “Health IT”.
Figure 15: Experience with Non-Medical Consumer Products
Q: Select the type of non-medical consumer products you have experience with.
(Select all that apply.) n=20
Most commonly respondents indicated that their company focused on the development of
medical devices (52/71, 73%); the others focused on both medical device and consumer software

64
(13/71, 18%), consumer software (5/71, 7%), or accessories for pharmaceuticals (1/71, 1%)
(Figure 16).
Figure 16: Company developmental focus
Q: Please select the best answer for your current position. n=71
Most respondents indicated that they had experience with Agile software development
(51/68, 75%) and waterfall / cascade methods (50/68, 74%) (Figure 17). Others indicated
experience with the “V” method (37/68, 54%), spiral method (3/68, 4%), and/or other methods
(6/68, 9%). The “other” category contained free text responses that included concurrent and
iterative SDLC models.

65
Figure 17: Experience with SDLC Methods
Q: Please select the software development lifecycle models (SDLC) with which you have
experience. (Select all that apply.) n=68
Only those with experience in Agile were asked to indicate the specific Agile methods
that they utilized (Figure 18). Most commonly, respondents had experience with scrum (38/46,
83%). Many also indicated test case (29/46, 63%), automated testing (25/46, 54%), user story
(28/46, 61%), XP (5/46, 11%), test driven development (18/46, 39%), automated testing (25/46,
54%), pair programming (10/46, 22%), and other methods (1/46, 2%). Free text comments
included “risk-based scenario” and “machine learning enabled algorithm”.

66
Figure 18: Experience with Agile Methods
Q: With which type(s) of agile method have you worked (select all that apply)?
n=46
About half of the respondents indicated that their company developed software both
internally and externally (30/63, 48%) (Figure 19). Nearly as commonly they developed the
software internally only (27/63, 43%) and a few only developed software through an external
organization (6/63, 10%).
Figure 19: Company development of Software
Q: Do you or your company develop software internally? n=63

67
4.3 Content
Fourteen questions in this survey explored the domain identified as “content”- the
information and requirements related to design control for software.
First, respondents who were involved with the development of consumer software were
given a list of 5 factors that might drive decisions to include design controls for consumer
software and were asked to choose the two most significant factors for their company. Most
respondents identified the role of regulatory requirements (12/15, 80%) (Figure 20). A half
pointed to assuring a future goal of a medical application (8/15, 53%) and a third to having a
common design process for software (5/15, 33%). Factors also to be considered by a few were
cost (1/15, 7%), experience in design control (3/15, 20%), and “other” (1/15, 7%). The “other”
category identified “customer requirements”.
Figure 20: Factors for inclusion of Design Control for Consumer Software
Q: What factors were critical in deciding to incorporate or exclude design control in your
process for consumer software or lifestyle software? Move the top 2 reasons into the box. n=15
When the full group of respondents was asked about the level of difficulty faced when
implementing design control, about half found it to be somewhat difficult (28/57, 49%) (Figure

68
21). More than a third found it neither easy nor difficult (21/57, 37%), and a few found it either
somewhat easy (7/57, 13%) or extremely difficult (1/57, 2%). None found it extremely easy. A
follow up question gave individuals the chance to add additional free text explanations and
received 17 responses. Two comments focused on the company’s capabilities, 8 comments
focused on either individual or teams’ knowledge of requirements, 5 comments discussed SDLC
methods/ standards, and 2 comments discussed other ideas. Comments are provided in Table 7.
Figure 21: Difficulty in implementing Design Control
Q: How easy is it to implement the FDA requirements for design control for software into the
development process? n=57
Table 7: Comments: Implementing Design Control
n=17
Company
Capabilities
• It depends on the Exec Mgmt [sic] understanding of their products and regulatory
obligations. For standard med device orgs, it’s relatively easy to implement. However,
for less traditional med orgs like CLIA Labs, Research Use Only, or LDT, it becomes
more challenging since they are in the “grey” area of FDA discretionary oversight. I’d
argue that design controls maybe viewed as industry best practice, regardless of
regulations compliance as it introduces a comprehensive, and risk based approach to
complex product development
• If the company does not spend the time to ensure the process is developed correctly
and engrained into the development and deployment process, it will be tough. That
has been the approach I have seen. But it doesn’t have to be. [sic]
Knowledge of
Requirements
• Most of the FDA (and IEC 62304) requirements are basics that are learned in first
semester of a software engineering bachelor. And than later get ignored as it’s “not
cool”. [sic] But if you work with best practices for software engineering, the gap to
FDA and IEC 62304 requirements are really small.

69
• The guidance the FDA has provided doesn’t uniformly align with IEC 62304 or IEC
82304, which are both standards that are being commonly used.
• It is not difficult, but time consuming since a lot of documentation is needed. Also,
change control is strict.
• Primarily, IEC 62304 and its tech documents provide the framework for med device
software development. However, the 69orrelation [sic] to the CFR and ISO 13485 is
not 1:1. How post-market surveillance is conducted is dependent on organization’s
maturity in using risk management file fir disposition. Use of RMF [risk management
file] is SaMD is an emerging body of knowledge that is not readily understood by
senior leadership.
• In my opinion, the FDA requirements are a somewhat difficult to apply because we
have to think of the entire system versus just control of SW [software]. The overall
product impact due to SW change is a critical part of design control and it gets
cumbersome when you have multiple SW releases in DVT phase.
• It's fumdamental [sic] and uncomplicated, but we have occasionally hired people who
believed sw should be exempt from control, and had to do remedial training.
• Many guidance documents for software in or as medical devices and general software
so product development team may be overwhelmed or confused by these
requirements.
• It will highly depend on the organizational structure, procedures, and how well the
development team are at capturing the voice of customer (VOC) early and updating it
often, with RA and QA close. The VOC is critical for the development of the
intended use and indications for use statements and will dovetail into the quality and
regulatory requirements that should also be included within the overall design
requirements suite. I’m my experience, these are often an afterthought and cause
rework and delays either due to testing or poor regulatory strategy because the intent
of the product (including how they wish to promote the product) drifts from the
original plan.
SDLC/
Standards
• FDA's requirements ar quite mature now. The V-model is almost de rigeur for active
devices (60601) and software, I think, with waterfall more common for non-active
devices (i.e. those without a source of electrical power). Also IEC 62304 lays out a
well established process which FDA recognises. [sic]
• Although we have many of the requirements to meet ISPE GAMP 5 expectations, we
do not have all the required deliverables for the FDAs design control requirements.
• FDA Design Controls requirements are not aligned with SDLC best practices (e.g.
agile/scrum method, SDLC tools like Atlassian) in the software industry; Although
FDA recognizes IEC 62304 and AAMI TIR 45 etc to fill some of that gap but still
software industry best practices (non-medical) are something else altogether. FDA did
also attempt to bridge the gap thru their Digital Health Software Pre-Cert Pilot
Program (2017-18) but it didn't end of anything meaningful in terms of a clear
regulatory pathway. Hence such disconnect and challenges remain.
• FDA design control requirements are so basic it's not that hard. I wonder if you ask
about 62304 on top of that, later. This would change my answer to Somewhat
difficult.
• The FDA requirements specified in 21CFR820 are too generic and do not provide the
insight needed to meet the current standards for regulatory submissions. More helpful
is IEC 62304, though the implementation of this SDLC standard needs some
additional insight to satisfy the needs of the FDA review team
Other • I don't have direct experience implementing the FDA's requirements so I'm neutral on
the topic.
• [De]pending on the intended use, some of the requirements could be overkilled.

70
A cross-tabulation was performed to compare the responses of Quality/Regulatory (QRA)
versus engineering (ENG) actors and between those with medical device (MD) / consumable
product (CON) experience, Agile experience and no experience. Engineers (ENG) more
commonly viewed design control implementation as extremely or somewhat difficult than
Quality/Regulatory Affairs professionals (QRA). This returns a value of p<0.01 when run
through Chi-Squared test in Qualtrics. The data was also analyzed by Fisher’s exact test
returning a significance of p=0.01 (Table 8). No similar trends appeared to be present between
those with medical device (MD) versus consumer product (CON) experience or Agile
experience.
Table 8: Cross Tabulation - How easy is it to implement Design Control?
Q2.2: How easy is it to implement the FDA requirements for design control for software into the
development process?
Total QA/RA Eng Total Med
Dev
Cons Total Agile Other
Total
Count
54 44 10 70 55 15 95 42 53
Extremely
difficult
1
1.9%
0
0%
1
10%
1
1%
1
2%
0
0%
2
2%
1
2%
1
2%
Somewhat
difficult
26
48%
24
55%
2
20%
25
50%
27
49%
8
53%
46
48%
18
43%
28
53%
Neither
easy nor
difficult
20
37%
13
30%
7
70%
25
36%
20
36%
5
33%
35
37%
17
41%
18
34%
Somewhat
easy
7
13%
7
16%
0
0%
9
13%
7
13%
2
13%
12
13%
6
14%
6
11%
Extremely
easy
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
p-value 0.01
Respondents were asked to select the methods that they used to classify software from 4
choices. Most chose regulatory guidance documents (46/57, 81%) and internal discussion
(40/57, 70%) (Figure 22). Other responses included communication with the FDA (31/57, 54%),

71
external consultants (16/57, 28%), “other” (7/57, 12%), or no comment (4/57, 7%). The “other”
category comments are provided in Table 9 and include information from other regulators,
standards, guidance documents, search of FDA database, voice of customers, and standards.
Figure 22: Determination for Software Classification
Q: Please consider how you determined the classification for your software. Select all items that
you used in determining the software classification. n=57
Table 9: Comments: Other categories for Software Classification
Other • IEC 62304 (2)
• ROW Authorities, FDA, and Notified Bodies
• Voice of Customer to develop the correct intended use, etc.
• Search of existing product codes
• Research into similar products globally and understanding risk profile based off
IMDRF guidance documents
• Your query does not specify if this is software regulatory classification or software
safety classification. For safety classification, used international standard 62304 in
view of 14971 to determine. For EU software regulatory classification, this was done
via a 2.5+ year long adjudication between our Notified Body as well as the Competent
Authority of our Authorized Representative (on account of our being a US Based
manufacturer of devices, per EU MDR) - classification (between I, IIa, IIb, III) has
not been reached since initiation in 2021, may be escalated to EU Council for eventual
adjudication per our NB. [sic]

72
Respondents were also asked to classify the usefulness of the methods that they used to
classify software. Most respondents selected regulatory guidance documents (36/51, 71%)
(Figure 23). Others chose communication with the FDA (24/43, 56%), internal discussions
(20/51, 39%), and “other” (4/51, 8%). The “other” category included “IEC 62304”, “VOC [voice
of customer]”, and “Early engagement with the regulators”.
Figure 23: Usefulness of Methods for Software Classification
Q: For the categories that you used, which did you find most useful? (Please select all that
apply) n=51
Respondents were then asked about the extent to which they agreed with 3 statements
related to the regulatory requirements for design control. For the statement, “The FDA
requirements for design control aligns well with industry standards for software development.”,
most respondents either agreed, somewhat (24/51, 47%) or strongly (5/51, 10%) (Figure 24,
Table 10). Several others selected neither agreed nor disagreed (6/51, 12%) or somewhat
disagreed (12/51, 24%). Their answers yielded a WA of 3.2 on a scale of 1-5, with a score of 1
indicating they strongly agree while a score of 5 would be strongly agree. Respondents who
indicated that they could not respond (4/51, 8%) were excluded from the calculation.

73
Responses for the other two statements were similar. For the statement, “The elements to
be included in a SaMD's design history file are clear”, most either somewhat (26/51, 51%) or
strongly agreed (6/51, 12%). Others selected neither agreed nor disagreed (6/51, 12%),
somewhat disagreed (6/51, 12%) or strongly disagreed (1/51, 2%). The resulting WA was 3.24
on the same scale. Respondents who indicated that they cannot respond (6/51, 12%) were
excluded from the calculation.
The final statement “The FDA's acceptance of Agile software development is well
defined” was only offered to individuals who indicated that they have used Agile. Most selected
somewhat (14/36, 39%) or strongly agree (3/36, 8%). Other respondents selected strongly (1/36,
3%) or somewhat disagree (8/36, 22%). The resulting WA was 3.12 on the same scale.
Respondents who indicated that they could not respond (4/36, 11%) were excluded from the
calculation. Cross-tabulation for this question was performed for the actors in QRA versus ENG
functions, MD versus CON experience, and Agile experience versus no experience. None of
these showed any significant differences (Appendix C).
Table 10: Statements for Design Control
WA: Weighted Average, 1=Strongly Disagree (SD), 2=Somewhat Disagree (D), 3=Neither
Agree or Disagree (NAG), 4=Somewhat Agree (A), 5=Strongly Agree (SA) Cannot respond
(CR) removed from WA calculation. n=51, 36
Statement SD D NAD A SA CR WA
The FDA requirements for
design control aligns well
with industry standards for
software development.
0/51,
0%
12/51,
24%
6/51,
12%
24/51,
47%
5/51,
10%
4/51,
8%
3.20
The elements to be included
in a SaMD's design history
file are clear.
1/51,
2%
6/51,
12%
6/51,
12%
26/51,
51%
6/51,
12%
6/51,
12%
3.24
The FDA's acceptance of
Agile software development
is well defined.
1/36,
3%
8/36,
22%
6/36,
17%
14/36,
39%
3/36,
8%
4/36,
11%
3.12

74
Figure 24: Statements for Design Control
Q: To gain insight on design control for software, please indicate if you agree or disagree with
each of the statements. n=51, Agile n=36
Individuals were then asked to select the word that best reflected how they felt about
current FDA software guidance from 5 offered choices (Figure 25). Most commonly respondents
choose “too high level” (17/48, 35%) or “too general” (15/48, 31%). Somewhat fewer found
them to be confusing (12/48, 25%), easy to apply (12/48, 25%) or comprehensive (4/48, 8%).
0% 10% 20% 30% 40% 50% 60%
The FDA requirements for design control aligns well
with industry standards for software development.
The elements to be included in a SaMD's design
history file are clear.
The FDA's acceptance of Agile software development
is well defined.
Cannot Respond Strongly agree Somewhat agree
Neither agree nor disagree Somewhat disagree Strongly disagree

75
Figure 25: Word Choice for FDA Guidance
Q: Choose the answers that best describe how you feel. I find the existing FDA software
guidances are... n=48
Individuals were asked to indicate their familiarity with and use of specific policy and
guidance documents, with choices of “not familiar” (3, NF), “familiar with the document, but do
not find it useful” (2, FNU), or that they “use[d] the document” (1, U) (Figure 26, Table 11). The
documents were divided into two groups that were placed in two different questions to improve
readability for the respondents. The first were the following documents on which feedback was
sought:
General Wellness: Policy for Low Risk Devices: half knew of the document (U+FNU:
22/45, 49%); if they were familiar most (U: 13/22, 59%) found it useful while many (41%) did
not find it useful. The other half (NF: 23/45, 49%) were not familiar with the document.
For the Policy, Policy for Device Software Function and Mobile Medical Application
Guidance: most (U+FNU: 30/46, 65%) knew of the document; if they were familiar most found
it useful (U: 25/30, 83%) while a few did not find it useful (FNU: 5/30, 17%). Approximately a
third (NF: 16/46, 35%) were not familiar.
For the guidance, Medical Device Data Systems, Medical Image Storage Devices, and
Medical Image Communication Devices: most knew the document (U+FNU: 31/47, 66%); if

76
they were familiar most found it useful (U: 26/31, 84%) while a few did not find it useful (FNU:
5/31, 16%). The remaining third of all respondents (NF: 16/47, 34%) were not familiar.
For the guidance, Clinical Decision Software Guidance: most (U+FNU: 32/47, 68%)
knew of the document; if they were familiar most found it useful (U: 24/32, 75%) found it useful
and some (FNU: 8/32, 25%) did not find it useful. The rest of the total population (NF: 15/47,
32%) were not familiar.
The final support document in group 1, Your Clinical Decision Support Software: Is it a
Medical Device? FDA graphic / Policy Navigator: the responses indicated that half knew of the
tool (U+FNU: 26/46, 58%); if they were familiar most found it useful (U:22/46, 85%) while a
few did not find it useful (FNU: 4/26, 15%). The other half of the total population (20/46, 43%)
were not familiar.
Cross-tabulation for this question was performed for the actors QRA versus ENG, MD /
CON experience, and Agile experience versus no experience. No significant observations were
obtained, and the results are presented in Appendix C.

77
Table 11: Review of Policy and Guidance Documents - Group 1
WA: Weighted Average: 1=I use this document (U), 2=I am familiar, but do not find it useful
(FNU), 3= I am not familiar (NF)
Guidance Document I use this
document.
I am familiar,
but do not
find it useful.
I am not
familiar.
WA
General Wellness: Policy for Low
Risk Devices
13/45,
29%
9/45, 20% 23/45, 45% 2.22
Policy for Device Software
Function and Mobile Medical
Application Guidance
25/46,
54%
5/46, 11% 16/46, 35% 1.80
Medical Device Data Systems,
Medical Image Storage Devices,
and Medical Image
Communication Devices
26/47,
55%
5/47, 11% 16/47, 34% 1.79
Clinical Decision Software
Guidance
24/47,
51%
8/47, 17% 16/47, 32% 1.81
Your Clinical Decision Support
Software: Is it a Medical Device
FDA Graphic / Policy Navigator
22/46,
48%
4/46, 9% 20/46, 43% 1.96
Figure 26: Review of Policy and Guidance Documents - Group 1
Q: Q2.9 - Please consider the guidance documents below and indicate if you find them useful.
n= 45-47

78
Respondents were then asked whether any of the guidance documents in this grouping
required modification. Most respondents (27/37, 73%) indicated that that they did and a quarter
that they did not (10/37, 27%). In a follow up question, individuals were given the option to
explain their views in a free text field. Specific improvements, identified in Table 11, identified
Policies for Device Software function (3), the Mobile Medical Application Guidance (1), the
Clinical Decision Software Guidance (3), and, in two cases, “all of them”. Full text responses
are found in Table 12.
Table 12: Comments: Improvements - Group 1
Policy for Device Software • For example, a Class II medical device requiring 510(k) may have
an app, and the app could include a combination of software
components that are Class I, Class II 510(k) exempt, and/or Class II
requiring 510(k). It would be nice if guidance documents like
"Policy for device software function and mobile medical application
guidance" could offer discussions or examples for similar scenarios.
• The general wellness policy needs to shift to a more patient-focused
claims approach. It’s not clear from guidance but you can have one
product that straddles all three subtypes of claims.
Mobile Medical Application
Guidance
• Policy of device function and mobile and data storage
Clinical Decision Software • The Clinical Decision Software guidance could probably be more
clear. It's OK as is but could be better.
• Clinical decision software guidance
• Clinical Decision Software Guidance could use additional examples
on how software can be used in the clinical space with regards to
sample management.
• A large concern is the joint oversight of CDS from the FDA and the
ONC [versus Office of the National Coordinator for Health
Information Technology].
• ALL need more specific examples now that FDA has more
experience regulating different software in different device
applications. The general wellness policy needs to shift to a more
patient-focused claims approach. Its not clear from guidance but
you CAN have ONE product that straddles ALL three subtypes of
claims described by FDA but only IF you apply these to three
different patient populations re: value proposition a product can
drive in each.
General Comments • Regarding all of the guidance documents, there really needs to be
more strenuous alignment with the IEC standards. The IEC

79
standards have been what organizations have been using in SaMD
development as a baseline. However the guidances provided by the
FDA reflect the agency's current thinking, but are NOT directly
correlated to the QSR and only cursorily align to IEC. An effort
similar to the QSR alignment to ISO 13485 would be beneficial
with regards to IEC 62304/82304.
• I think th[e] policies for all could be improved.
• As expected, most FDA guidance tend to be high level and less
prescriptive by design in order to accommodate a wide variety of
applications, intended uses, etc. Therefore, it would helpful to have
a little more prescriptive guidance, language, examples, similar to
those in industry standard (ex ISO 14971, IEC 62304, etc)
• More Details on the documentation requirements based on the risk
of the SaMD or wellness device.
• The Agency guidance, like regulation, is frequently general. This
results in various interpretations. The Agency could benefit from
providing clearer definition in guidance documents to standardize
the product development requirements. There were several
situations where the Agency provided guidance through interaction
that was much clearer regarding expectation. Unfortunately,
guidance solicited for subsequent devices was inconsistent. A
standardized approach could alleviate this issue.
• These guidance are high-level and serve their intended purpose fine;
the underlying SDLC methodology and deliverables can be more
specific (but that applies to other FDA guidances and recognized
standards more, and to 21 CFR 820.30 itself).
• All of them. They're too high level, very device focused in their
language whereas most developers are completely new to this area.
It confuses common terms in software development with regulatory
terms (same word means different things in each field).
• All mentioned. More details or easy to follow flowcharts would be
helpful. The guidances are very high level and not clear in
[requ]irements.
The second question of this type also asked if they were familiar with certain additional
documents and, if familiar, whether they found the documents useful (Figure 27; Table 13). For
the guidance, Changes to Existing Medical Software Policies Resulting from Section 3060 of the
21st Century Cures Act. 2017: about half (U+FNU: 21/40, 53%) knew the guidance; if they were
familiar most found it useful (U:14/21, 67%) and many (FNU: 7/21, 33%) did not find it useful.
The other half of the total population (NF: 19/40, 48%) were not familiar with the document.
For the guidance, Content of Premarket Submission for Device Software Functions: most
(U+FNU: 32/40, 80%) knew of the guidance; if they were familiar most found it useful (U:

80
31/32, 94%) while only one individual (FNU:1/32, 3%) did not find the document useful. The
remainder of the total population (NF: 8/40, 20%) were not familiar with the document.
For the document, Policy related to Low Risk Wellness Document: about half (U+FNU:
17/39, 44%) knew of the policy; if they were familiar most found it useful (U: 11/17, 65%)
whereas some (FNU: 6/39, 15%) did not find it useful. The other half of the total population
(NF: 22/39, 56%) were not familiar.
In the final document, Cybersecurity in Medical Devices: Quality System considerations
and Content of Premarket Submissions Draft Guidance for Industry and Food and Drug
Administration: most respondents (U+FNU: 34/40, 85%) knew of the guidance; if they were
familiar most found it useful (U: 29/34, 85%) while a few (FNU: 5/34, 15%.) did not find it
useful. The rest of the total population (NF: 6/40, 15%) were not familiar.
Table 13: Review of Policy and Guidance Documents - Group 2
WA: Weighted Average: 1= I am not familiar (NF), 2=I am familiar, but do not find it useful
(FNU), 3=I use this document (U)
Guidance Document I use this
document.
I am familiar, but
do not find it
useful.
I am not familiar. WA
Changes to Existing Medical
Software Policies Resulting
from Section 3060 of the
21st Century Cures Act.
2017
14/40, 35% 7/40, 18% 19/40, 50% 2.13
Content of premarket
submission for device
software functions. Draft
guidance.
31/40, 78% 1/40, 3% 8/40, 20% 1.43
General wellness: policy for
low risk devices.
11/39, 28% 6/39, 15% 22/39, 56% 2.28
Cybersecurity in medical
devices: Quality system
considerations and content
of premarket submissions,
draft guidance
29/40, 73% 5/40, 13% 6/40, 15% 1.43

81
Figure 27: Review of Policy and Guidance Documents - Group 2
Q: Q2.9 - Please consider the guidance documents below and indicate if you find them useful.
n= 39-40
Cross-tabulation using chi-square calculation for this question was performed for the
actors QRA versus ENG, MD / CON experience, and Agile experience versus no experience.
Individuals in QRA functions reported more frequently that they were familiar with for the FDA
guidance for cybersecurity in premarket submissions and found it useful when compared with
those in ENG functions (p <0.04) (Table 15).

82
Table 14: Cross Tabulation - FDA Guidance Documents - Group 2
Q2.12 - Please consider the guidance documents below and indicate if you find them useful. I
use this document (U), I am familiar, but do not find it useful (FNU), I am not familiar (NF)
Total QA/
RA
Eng Total Med
Dev
Cons Total Agile Other
Total
Count
40 29 8 51 40 11 64 27 37
Changes to Existing Medical Software Policies Resulting from Section 3060 of the 21st Century Cures
Act. 2017
U 14
35%
13
45%
1
13%
19
37%
14
35%
5
46%
21
33%
7
26%
14
38%
FNU 7
18%
6
21%
1
13%
10
20%
7
18%
3
27%
13
20%
7
26%
6
16%
NF 19
48%
10
35%
6
75%
22
43%
19
48%
3
27%
30
47%
13
48%
17
46%
p-val 0.097
Content of premarket submission for device software functions. Draft guidance for industry and Food and
Drug Administration.
U 31
78%
25
86%
5
63%
40
78%
31
78%
9
82%
51
89%
21
78%
30
81%
FNU 1
3%
1
3%
0
0%
1
2%
1
3%
0
0%
2
3%
1
4%
1
3%
NF 8
20%
3
10%
3
38%
16
32%
11
28%
5
46%
11
17%
5
18%
6
16%
p-val 0.11
General wellness: policy for low risk devices. Guidance for industry and Food and Drug Administration.
U 11
28%
10
36%
1
13%
16
32%
11
28%
5
46%
19
30%
9
33%
10
28%
FNU 6
15%
5
18%
1
13%
6
12%
6
15%
0
0%
12
19%
6
22%
6
17%
NF 22
56%
13
46%
6
75%
28
56%
22
56%
6
55%
32
51%
12
44%
20
56%
p-val 0.32
Cybersecurity in medical devices: Quality system considerations and content of premarket submissions
draft guidance for industry and Food and Drug Administration staff.
U 29
73%
24
83%
4
50%
39
76%
29
73%
10
91%
48
75%
20
74%
28
76%
FNU 5
13%
3
10%
2
25%
5
10%
5
13%
0
0%
9
14%
4
15%
5
14%
NF 6
15%
2
7%
2
25%
7
14%
6
15%
1
9%
7
11%
3
11%
4
11%
p-val <0.04
Respondents were then asked if any of the documents presented in this second question
required modification. Nearly half felt that they did (yes: 15/34, 44%; no: 19/34, 56%). Ten
comments were also received (Table 15). Nearly half (5/11) felt that cybersecurity guidance
should be modified, and the others were more general to documents as a group.

83
Table 15: Comments: Improvement - Group 2
Cybersecurity • The cybersecurity document should be more specific about what
documents are required for submission, by submission type. I
submitted the same documents for an IDE and a 510(k) (prior to the
eSTAR update for cybersecurity) and [it] was fine for IDE
(probably because that is specifically called out in the guidance) and
got several questions with regards to my 510(k) it was clear that
even a reviewer with cyber expertise was confused by [the
requirements].
• I'll focus on one: Cybersecurity. This is a new area where industry
is struggling on a few areas: What does it mean to be secure, and
how to do it cost effective, and efficiently? What are the regulation
guidances that [we] need to comply with? What are the industry
best practices, in handling the dynamic landscape of SaMD +
security. There are many open questions in this relatively new field
that is making it difficult for OEM [original equipment
manufacturers] to achieve. Hence, it would be helpful to have more
useful examples, frameworks, languages from FDA to understand
what is truly needed from a governance perspective, and let industry
figure out how to comply. [sic]
• Cyber premarket guidance
• Details on the documentation and risk management requirements
specifically for cybersecurity
• [The] cybersecurity guidance can be more specific and aligned to
standards such as NIST/ISO 27001, although the recent Sept'23
version of this guidance is more prescriptive than the previous one.
General Comments • Too general
• The document and processes are evolving. There has to be better
documentation around module and type definition around SaMD
and Class type across therapeutic area and device usage types.
• All but the premarket notification one. Same issue, very high level
and too much device regulatory language that is confusing to
developers.
• Better decision criteria transparency from FDA and clarity on
changes to legacy products.
• Draft guidance documents are drafts for a reason and I doubt this
will be implemented in its current form.
4.4 Context
Another set of 14 questions were posed to explore the “context” in which devices are
developed. They focused on factors related to the business environment and on their experiences
when working with the FDA.
First, all respondents were asked to rank five potential barriers to entry that might be
faced when consumer software companies attempted to develop medical devices, from most (1)
to least significant (5) (Table 16, Figure 28). WA was calculated on a score of 1-5 from most to

84
least impactful. Based on the WAs of their answers, the highest barrier was the regulatory
burden to develop the medical application (WA= 2.29; 1:12/35, 34%; 2:11/35, 31%; 3: 6/35,
17%; 4:2/35, 6%; 5:4/25, 11%),followed by confusion related to the requirements for device
software (WA= 2.86; 1:6/35, 17%; 2: 7/35, 20%; 3: 10/35, 29%; 4:11/35, 31%; 5:9/35, 26%),
and finding qualified people that understand the regulatory requirements (WA= 2.91; 1: 3/35,
9%; 2:10/35, 29%; 3: 12/35, 34%; 4:7/35, 20%; 5:3/35, 9%). Somewhat lower were the choices
related to the match of the product to the business portfolio (WA= 3.54; 1:9/35, 26%; 2:2/35,
6%; 3:2/35, 6%; 4:5/35,14%; 5:17/35, 49%), and the additional cost to develop regulated
software (WA= 3.4; 1:5/35, 14%; 2:5/35, 14%; 3:5/35,14%; 4:11/35, 31%; 5:9/35, 26%). In
addition, individuals were allowed to provide free text responses to this question for other
barriers not listed in the question (Table 17).
Table 16: Ranking of Barriers for Development of Medical Device Software
WA: Weighted Average, Most Impactful (1), Least Impactful (5). n= 35
Barrier 1 2 3 4 5 WA
Regulatory burden to develop 12,
34%
11,
31%
6,
17%
2,
6%
4,
11%
2.29
Confusion on requirements for
device software
6,
17%
7,
20%
10,
29%
11,
31%
9,
26%
2.86
Finding qualified people that
understand the regulatory
requirements
3,
9%
10,
29%
12,
34%
7,
20%
3,
9%
2.91
Additional cost to develop regulated
software
5,
14%
5,
14%
5,
14%
11,
31%
9,
26%
3.4
Business portfolio match 9,
26%
2,
6%
2,
6%
5,
14%
17,
49%
3.54

85
Figure 28: Ranking of Barriers for Development of Medical Device Software
Q: Rank the barriers stopping companies that develop consumer software products from
developing medical device software from the most impactful to the least impactful.
Table 17: Comments: Other Barriers for Development of Medical Devices
Other barriers • Business risk/liability - beyond just GDPR, into healthcare. [sic] It's
different from consumer software, even from other regulated
software like banking - code error can kill someone.
• Wanting to fast track or cut corners. [sic]
• If you know med[ical devices], you know what you need to do.
That's commonly not the case with in-experienced leaders/business
folks (example: Theranos) trying for the first time to develop a med
device. There is an arrogance/disbelief that that regs (FDA, EU,
CLIA) don't apply to them, so that's a leadership issue.
Individuals were then asked if they had held discussions with the FDA regarding
software. Most (25/37, 68%) identified that such discussions had taken place. The rest who
indicated that they had not (12/37, 32%), were not presented with the remainder of questions in
this block of questions relating to context.

86
The respondents who had met with the FDA were asked about the context in which those
discussions occurred. Most commonly, they indicated that the discussions took place using a
Pre-Submission or Q-submission process (21/24, 88%) (Figure 29). Others identified that they
took place during discussions related to premarket notification / approval discussions (19/24,
79%), during an audit (10/24, 42%), in a committee/working group meeting (7/24, 29%), while
drafting/commenting on guidance (6/24, 25%), or “other” (4/24, 17%). In the free text field,
comments included discussions “during an inspection”, “general inquiry to the Agency”,
“collaborative community volunteer work”, and “FDA's spectacular failure that was the software
precert pilot program.”
Figure 29: Context for Discussions with FDA
Q: In what was context did the discussion occur? (Select all that apply.) n=24
Next, individuals asked about the degree to which they agreed with certain statements
regarding their experiences with FDA during submission for these types of devices (Table 16,
Figure 30). For the statement, “The FDA was helpful in determining the risk classification”,

87
individuals most commonly agreed (13/24, 54%). The rest chose either “neither agree or
disagree” (10/24, 42%) or “disagree” (1/24, 4%).
For the statement, “The FDA did not have experience with my type of device”, individuals
most commonly disagreed (11/24, 46%), but many others either agreed (8/24, 33%) or chose to
neither agree nor disagree (5/24, 21%).
For the statement, “The FDA requested additional testing”, individuals most often agreed
(15/23, 65%). The remainder either disagreed (3/23, 13%) or neither agreed nor disagreed (5/23,
22%).
For the statement, “The FDA requested more software documentation than we had
anticipated”, about half agreed (11/23, 48%). The remainder neither agreed nor disagreed (7/23,
30%) or disagreed (5/23, 21%).
For the statement, “The FDA requested additional clinical use documentation”, about
half agreed (12/23, 52%), about a third disagreed (7/23, 30%) and the rest neither agreed nor
disagreed (4/23, 17%).
For the statement, “The FDA requested additional safety use controls be built into the
software”, about half agreed (11/23, 48%), about a third disagreed (7/23, 30%) and the rest
neither agreed nor disagreed (5/23, 22%).
Respondents had the option to provide additional comments clarifying their answers to
the statements in Table 19.

88
Table 18: Agreement with Statements Regarding FDA Submissions
Q: Choose the option that best reflects your feelings about the following statements.
n= 23-24 WA: Weighted Average, 1=Disagree (D), 2=Neither Agree or Disagree (NAG),
3=Agree (A)
Statement D NAG A WA
The FDA did not have experience with
my type of device.
11/24,
46%
5/24,
21%
8/24,
33%
1.88
The FDA requested additional safety
controls be built into the software.
7/23,
30%
5/23,
22%
11/23,
48%
2.17
The FDA requested additional clinical
use documentation.
7/23,
30%
4/23,
17%
12/23,
52%
2.22
The FDA requested more software
documentation than we had anticipated.
5/23,
21%
7/23,
30%
11/23,
48%
2.25
The FDA was helpful in determining
the risk classification.
1/24,
4%
10/24,
42%
13/24,
54%
2.50
The FDA requested additional testing. 3/23,
13%
5/23,
22%
15/23,
65%
2.52

89
Figure 30: Agreement with Statements regarding FDA Interactions
Q: Choose the option that best reflects your feelings about the following statements.
Table 19: Comments: Statements regarding FDA Interactions
Additional Comments • [The FDA asked us to] follow a guidance that specifically indicated
our product was out of scope. We had to follow it anyways which
added time and cost.
• [As] a pilot in FDA Digital Health Software Pre-Cert Pilot Program
(2016-18) at Verily/Google Life Sciences.
• FDA staff were helpful. Pre-submission and interactions with FDA
were always very helpful.
• [The] FDA required additional security controls to be built into the
software. Risk Management to 14971 didn't tell us we needed a
fingerprint lock on the app (app to treat insomnia - incredibly
lightweight borderline bullshit SaMD) to mitigate any safety risk.
But boy did FDA make a fuss about ensuring that feature was
implemented prior to launch as a security control, apparently
because of perceived stigma if someone else were to find out that
the device user is being treated for a mental health disorder because
the app doesn't lock up. [sic]

90
Individuals were then asked how the team felt about the FDA feedback (Figure 31).
Most were satisfied (15/23, 65%), but a few were not (2/23, 9%). About a quarter chose “other”
(6/23, 26%) and were given the chance to comment. Three of these individuals identified that
that the quality of the feedback varied depending on the reviewer and the other three noted
inconsistencies in the feedback (Table 20).
Figure 31: How did the Team feel about FDA Feedback.
Q: When considering the feedback, how did you and/or the team feel about the FDA feedback?
n=23
Table 20: Comments: Team View of FDA Feedback
General Comments • Mostly reasonable feedback.
• Case by case pending on reviewers.
• Depends on the reviewer 100%. The ones that understand software are few and
far between
• Case by case
Feedback
Consistency
• I've had the complete range, from very satisfied, especially in standards working
groups to not really satisfied when they changed their point of view between QSub and 510(k).
• Feedback changed in every Q-sub depending on who review team was assigned.
• Once asked to submit in the 510(k) unit testing for Class B safety/ Class II
regulatory out of an abundance of caution in a Qsub. Burdensome, not reflective
of a true regulatory requirement, and went ignored in the 510(k).
Individuals were asked to rate their impressions of FDA’s knowledge when discussing
software (Figure 32) on a scale of 1 to 4 from low to high; a WA of 1 suggests that the FDA was
not knowledgeable whereas a WA of 4 suggests that it was very knowledgeable. Most

91
commonly, respondents viewed the FDA as either moderately (10/24, 42%) or very
knowledgeable (9/24, 38%). A few rated them as slightly knowledgeable (4/24, 17%) or not
knowledgeable at all (1/24, 4%) resulting in a WA of 3.13. Individuals with no opinion were
excluded from the calculation. Three respondents gave comments suggesting either that the level
of knowledge varied throughout the FDA and two that the prior experience of reviewers was
mostly from academia (Table 21).
Figure 32: How knowledgeable did you find the FDA in Software Discussions.
Q: How knowledgeable did you find the FDA staff when you discussed software with them? n=
24

92
Table 21: Comments: View of FDA Feedback
View of Feedback • It will be case by case and based on role. Many reviewers do not come from
industry and lack real experience but have academic experience. Some reviewers
increase expectation because a company that could afford more testing, etc.,
provided more than required. This is a strategy to increase barriers to the
competition. This is not a specific SW issue, but highly tied to it. [sic]
• Varied. The clinical/stats experts were NOT knowledgeable at all. The younger
reviewers fresh out of biomed masters programs who are Gen Z though, with less
than a year experience at FDA, they were moderately knowledgeable. Also, the
rare chance we got a software expert, they were also moderately knowledgeable.
• Actually it was variable - some very knowledgeable, others not. Hence my
response - moderately!
Respondents were also asked when they had last made a submission that included
software (Figure 33). About half chose less than 2 years (18/35, 51%). For others, the period
was 2-5 years (6/35, 17%) or greater than 5 years (2/35, 6%). About a quarter chose “does not
apply” (9/35, 26%); these individuals were not presented with subsequent questions in this
context block.
Figure 33: Length of Time since last Submission with Software.
Q: How many years has it been since your last submission with the FDA that included software?
n= 35
Respondents were also asked about the type of device addressed in their last submission.
Most indicated that it was for software in a medical device (18/26, 69%). The others that
indicated it was software as a medical device (8/36, 31%). Cross-tabulation for this question was

93
performed for the actors QRA versus ENG, MD / CON experience, and Agile experience versus
no experience. The results did not show clear differences and are presented in Appendix C.
Individuals were then asked how the team felt about the FDA feedback (Figure 34). Most
commonly, they were satisfied (15/23, 65%). Fewer were not satisfied (2/23, 9%) or chose
“other” (6/23, 26%). In an optional comment field, 4 individuals indicated that their experiences
had been variable. Two others remarked that feedback changed between the pre-submission and
the premarket submission (Table 22).
Figure 34: Quality of Response from FDA
Q: For your most recent submission, please indicate the quality of the feedback from the FDA
related to software. n= 26
Table 22: Comments: Quality of FDA Response
Variable quality • Case by case pending on reviewers.
• Depends on the reviewer 100%. The ones that understands software are few and
far between. [sic]
• Case by case
Feedback changed • I've had the complete range, from very satisfied, especially in standards working
groups to not really satisfied when they changed their point of view between QSub and 510(k).
• Feedback changed in every Q-sub depending on who review team was assigned.
Other • Mostly reasonable feedback

94
Individuals who did not submit Agile documentation were further asked to explain why
(Figure 35). Some respondents used development documentation that focused on the
waterfall/cascade SDLC (3/8, 38%), similarly one selected “other” (1/8, 13%) and explained that
the “V” method was used. In other cases, Agile development was not used (3/8, 38%) or the
documentation was not necessary given the risk of the device (1/8, 13%).
Figure 35: Agile Documentation in Submission
Q: With regards to recent submission with the FDA, did you submit documentation related to
Agile software development? n= 20
Individuals who did not submit Agile documentation were further asked to explain their
reasons (Figure 36). Some indicated that the submitted development documentation focused on
the waterfall/cascade SDLC (3/8, 38%) while others indicated Agile development was not used
(3/8, 38%). One indicated that the documentation showing SDLC methods was not necessary for
the risk of the product (1/8, 13%). One selected other (1/8, 13%) stating that the “V” method
was used.

95
Figure 36: Why submit Agile Documentation
Q: Why did you not submit documentation for Agile software development? (Select all that
apply) n= 8
4.5 Process
This final section focused on the domain of “process” with questions that addressed the
individual’s participation in the process of regulation development.
First, respondents were asked to agree or disagree with two statements regarding the
sufficiency of FDA’s approach to design control (Figure 37, Table 21). For the statement
“Design Control is sufficient for Software as a Medical Device (SaMD)”, individuals most
commonly agreed either somewhat (11/36, 31%) or strongly (7/36, 19%). Others disagreed
somewhat (9/36, 25%) or strongly (1/36, 3%). Some neither agreed nor disagreed (6/36, 17%).
Respondents who indicated that they could not answer (2/36, 6%) were not included in the data
analysis.
For the statement “Design control will continue to be sufficient for software in the
future”, individuals most commonly agreed somewhat (12/36, 33%) or strongly (4/36, 11%). A
significant number also disagreed somewhat (7/36, 19%) or strongly (4/36, 11%). Others neither

96
agreed nor disagreed (6/36, 17%). Respondents who indicated that they could not answer (1/36,
3%) were removed from the analysis.
In a follow-up optional question, 10 individuals further explained their answers in free
text comments. Three individuals commented that design control provides the high-level
philosophy but must rely on standards to provide detail with respect to the technical
requirements. Three individuals predicted that the emergence of Artificial Intelligence may
necessitate a change in the future. Additionally, two individuals noted that the interplay between
the SDLC and design control requires flexibility. Full text comments are provided in Table 24.
Figure 37: Statements for Design Control for Software
Q: Please provide your opinion on the statements below. n= 36
0% 5% 10% 15% 20% 25% 30% 35%
Design control is sufficient for Software as a Medical
Device (SaMD).
Design control will continue to be sufficient for
software in the future.
No opinion / Does not apply Strongly agree Somewhat agree
Neither agree nor disagree Somewhat disagree Strongly disagree

97
Table 23: Statements for Design Control for Software
WA: Weighted Average, 1=Strongly Disagree (SD), 2=Somewhat Disagree (D), 3=Neither
Agree or Disagree (NAG), 4=Somewhat Agree (A), 5=Strongly Agree (SA) Cannot respond
(CR) removed from WA calculation. n= 36
Statement SD D NAG A SA CR WA
Design Control is sufficient
for Software as a Medical
Device (SaMD)
1,
3%
9,
25%
6,
17%
11,
31%
7,
19%
2,
6%
3.22
Design control will continue
to be sufficient for software
in the future.
6,
17%
7,
19%
6,
17%
12,
33%
4,
11%
1,
3%
2.94
Table 24: Comments: Design Control SaMD
Standards • FDA provides high level regs for complying to org's own processes, but not how to do
it. Therefore, industry tends to refer to industry standards like IEC 62304, ANSI SW 96
for more prescriptive methods in developing software.
• The IEC 62304 guidance is also needed. Design Control only does not provide enough
detail.
Future • AI will shake things up. DRAFT regulations are coming, but in the EU confusing.
They are regulating IA as a technology rather than as a area focused issue (e.g. medical
devices) - thus requirements which are important for AI for NLP or web-search, for
instance, will be less important than those for medical devices (and vice-versa) and that
may get diluted as things currently stand. [sic]
General • Our SDLC process was subordinate to design control and expanded far beyond the
traditional design control methodologies.
• Design Control is "whats", no "hows". Latter being where the gaps are in terms of
SDLC methodology, tools, activities and deliverables, to the software industry best
practices.
Only individuals indicating that they had experience with Agile SDLC were asked to
agree or disagree with a statement regarding Agile and design control (Figure 38, Table 25). For
the statement “Agile software development can be used in a way that meets the design control
requirements”, individuals most commonly agreed somewhat (11/25, 44%) or strongly (11/25,
44%). Others somewhat disagreed (1/25, 4%) or neither agreed nor disagreed (2/25, 8%)
resulting in a WA of 4.28 on a scale of 1-5, with 5 representing strongly agree. Cross-tabulation
for this question was performed for the actors in QRA versus ENG functions, and MD / CON

98
experience versus Agile experience and no experience. No significant differences were obtained
and the results are presented in Appendix C.
Nine individuals further explained their answers in a text field. One believed that Agile
is easier to use when developing specifications and another that Agile is suitable for research use
only (RUO) software. Two others believed that Agile is difficult to use in a manner that supports
submissions or maintains traceability. One individual indicated that AI would further make this
area challenging. Full text responses are in Table 26.
Figure 38: Agreement with Statement for Agile and Design Control
Q: Choose the option that best reflects your feelings about the following statement. n=25
Table 25: Agreement with Statement for Agile and Design Control
WA: Weighted Average, 1=Strongly Disagree (SD), 2=Somewhat Disagree (D), 3=Neither
Agree or Disagree (NAG), 4=Somewhat Agree (A), 5=Strongly Agree (SA) Cannot respond
(CR) removed from WA calculation. n= 25
Statement SD D NAD A SA CR WA
Agile software development
can be used in a way that
meets the design control
requirements
0,
0%
1,
4%
2, 8% 11,
44%
11,
44%
0,
0%
4.28

99
Table 26: Comments: Agile SDLC and Design Control
Agile in
specific
situations
• Agile development can be used for the implementation itself (coding) very well, where
for the phases before (specification) and after (V&V) it is more difficult. We therefor
use a mixed method usually.
• The agile method seems well suited for development of RUO software. When the
software is a medical device, the agile method seems to lack, sufficient controls,
checks, and balances required to meet the rigor of traditional design control.
Agile and
submissions
• [I have] achieved it before in Class II device clearances, so [I] have [a] working
knowledge.
• Although guidance is now out to support the use of Agile software, it is difficult to get
Agile development to create the right deliverables in a way that can be used in a
submission.
Future • See previous answer [regarding AI].
Respondents were asked if they participated in the development of regulations or
guidance documents (Figure 39). More than half had not (21/36, 58%); these were moved to the
end of the survey. The others had experiences currently (9/36, 25%) or in the past (6/36, 17%).
Figure 39: Participation in the development for Guidance / Regulations
Q: Do you, or your organization, participate in the development of software guidance and
regulation outside your organization (for example with a trade group, standard committee, or
regulatory guidance development)? n= 36
Respondents were then asked to identify all the trade groups in which they participated
(Figure 39). Two thirds of respondents indicated Advamed (10/15, 67%) and direct feedback to
the Agency (10/15, 67%). Fewer participated in the FDA ISO/IEC Working groups (6/15, 40%)
or the FDA Pilot Program (5/15, 33%). A specific committee listed was “HL7”. A few

100
participated in AAMI (4/15, 10%), MDMA (1/15, 7%), or “other” (2/15, 13%). The other
category included “AI-Global Healthcare Initiative (AI-GHI) FDA Collaborative Community” or
“joint FDA/ customer groups.”
Figure 40: Trade Group participation
Q: With what group do you, or your organization, participate? (Select all that apply.) n=15
Individuals were then asked if trade groups can influence policy for software as a medical
device. Most respondents agreed (30/36, 83%) but a small minority disagreed (6/36, 17%).
Individuals were asked whether they believed that technical committees understand the
needs of companies developing SaMD. Most respondents agreed (27/36, 75%) but a quarter
disagreed (9/36, 25%).
Respondents were given a series of statements describing challenges previously identified
in the literature for developing regulated medical device software. They were asked to rank the
choices from most (1) to least impactful (7) (Figure 41, Table 27); the lower the WA selected,
the more impactful the challenge. Based on the WA, the choices selected from most impactful to
least impactful were the pace of regulatory development (WA= 3.03; 1:9/34, 26%; 2: 7/34, 21%;

101
3:5/34, 15%; 4:6/24, 18%; 5:2/34, 6%; 6:2/24, 12%; b:1/34, 3%), the content in current guidance
(WA= 3.56; 1:6/34, 18%; 2:2/34, 6%; 3:11/34, 32%; 4:4/34, 12%; 5:7/34, 21%; 6:1/34, 3%;
7:3/34, 9%), cybersecurity (WA= 3.71; 1:6/34, 18%; 2:7/34, 21%; 3:2/34, 6%; 4:6/34, 18%;
5:4/34, 12%; 6:7/34, 21%; 7:2/34, 6%), finding qualified individuals to meet the regulatory
burden (WA= 3.82; 1:4/34, 12%; 2:6/34, 18%; 3:7/34, 21%; 4:4/34, 12%; 5:5/34, 15%; 6:4/34,
12%; 7:4/34, 12%), technical challenges (WA= 4.32; 1: 3/34, 9:%; 2:4/34, 12%; 3:5/34, 15%;
4:7/34, 21%; 5:3/34, 9%; 6:6/34, 18%; 7:6/34, 18%), design control framework inconsistent with
current development techniques (WA= 4.32; 1:3/34, 9%; 2:5/34, 15%; 3:3/34, 9%; 4:4/34, 12%;
5:8/34, 24%; 6:8/34, 24%; 7:3/34, 9%), and business access to the FDA to clarify questions
(WA= 5.24; 1:3/34, 9%; 2:3/34, 9%; 3:1/34, 3%; 4:3/34, 9%; 5:5/34, 15%; 6:4/34, 12%; 7:15/34,
44%).
Table 27: Ranking of Current Challenges
WA: Weighted Average, Most Impactful (1) to Least Impactful (7). n= 34
Challenge 1 2 3 4 5 6 7 WA
Pace of regulatory
development/change
9,
26%
7,
21%
5,
15%
6,
18%
2,
6%
4,
12%
1, 3% 3.03
Content in the
current guidances
6,
18%
2,
6%
11,
32%
4,
12%
7,
21%
1, 3% 3, 9% 3.56
Cybersecurity 6,
21%
7,
21%
2,
6%
6,
18%
4,
12%
7,
21%
2, 6% 3.71
Finding qualified
individuals to meet
the regulatory
burden
4,
12%
6,
18%
7,
21%
4,
12%
5,
15%
4,
12%
4,
12%
3.82
Technical
challenges
3,
9%
4,
12%
5,
15%
7,
21%
3,
9%
6,
18%
6,
18%
4.32
Design control
framework is not
consistent with
current development
techniques software
3,
9%
5,
15%
3,
9%
4,
12%
8,
24%
8,
24%
3,
9%
4.32
Business access to
the FDA to clarify
questions
3,
9%
3,
9%
1,
3%
3,
9%
5,
15%
4,
12%
15,
44%
5.23

102
Figure 41: Ranking of Current Challenges
Q: Rank order these current challenges in developing regulated software. (Shown from most to
least impactful) n= 34
An optional final free text question allowed individuals to add additional comments or
examples of challenges that they face; five respondents gave comments. Three pointed to certain
business pressures related to management’s understanding, pace of development or resources.
Others highlighted regulatory burdens and one predicted that that cybersecurity and artificial
intelligence would be challenges for the future (Table 28).

103
Table 28: Final Comments
Business
pressure
• My main challenge is to make management and software engineers [understand], that
the burden is not so high as it looks like, if we would follow best practice from what
we've learned in university but do not like to do, as coding is considered more funny
than documentation. [sic]
• Resources and time.
Regulatory
Burden
• [The] pace of SOFTWARE development / change. Regulatory changes at a glacial
pace. I supported 250+ software product releases in under 3 years in my most recent
role, during that time there was only 1 requisite regulatory change (EU MDD to EU
MDR) that I had to manage impact. ALSO - pace of REGULATORY market
authorization, as a separate issue. If you release every week or every 2 weeks, but
regulatory market authorization for those releases can take 3-9 months for FDA? (or
worst case for EU MDR, 2.5+ years?) A design freeze for that length of time AND
continuously ensuring the product meets users needs - can be at great odds. [sic]
• Having been through two recent successful submissions, the regulatory burden seems
to change and gets more intense with each review team and the unique requirements of
each reviewer.
Other • Cybersecurity and infrastructure are huge concerns as well as privacy and access to
data. "AI problem is a data problem." Also, harmonization is the biggest concern.
Harmonization between US Federal Departments AND global harmonization on
SaMD/ SiMD.

104
Chapter 5. Discussion
5.1 Overview
Software is now an essential part of our medical systems and products. In some cases,
such as SaMD and SiMD, it is regulated by the FDA according to its risk. In others, when it is
used tangentially as a consumer product developed for low-risk, lifestyle applications, the
software may start as a consumer product and then migrate into a medical product based on
future business strategies. Companies wishing to transition their software into a SaMD or SiMD
must then determine the regulatory status of their new product and learn to implement the
relevant requirements according to risk and regulatory considerations. One of the central
requirements for most SaMD and SiMD products is the need to implement design controls.
In this study, industry views regarding the implementation of design control for such
products were examined. Within the surveyed population were individuals with backgrounds in
engineering as well as regulatory and quality functions with experience either with SaMD/SiMD
or with consumer lifestyle products that might have clinical applications. In the discussion that
follows, the research approach delimited and potentially limited the results will be discussed.
Then, the results of the survey as they relate to the three domains- context, content, and processthat were specifically explored in the study are addressed.
5.2 Consideration of Study Methodology
5.2.1 Limitations
This research was conducted by using a web-based software application, Qualtrics.com,
so it is affected by the advantages and disadvantages of this research tool. Important advantages
include its ability to reach individuals regardless of time zones and geographics; respondents can
then choose a convenient time and place to complete the survey, which is known to increase the

105
completion rate (Callegaro, Manfreda, and Vehovar, 2015). Internet surveys allow the
researcher to collect information from a relatively large number of individuals more quickly and
at a lower cost (Kelley, et al., 2003; Marshall and Rossman, 2014; Callegaro, Manfreda, and
Vehovar, 2015). Additionally, the value of electronic surveys has been previously demonstrated
to decrease interview bias and increase the value of the responses (Sivo, 2006; Marshall and
Rossman, 2014).
Despite these advantages, participation rates for internet surveys reported in the literature
are often lower than those for in-person, paper-based surveys. Nulty found that online surveys
had a completion rate of 33% compared to 56% for paper-based surveys (Nulty, 2008). Similar
observations were noted in other studies, where the lower response rate was attributed to a higher
degree of ease and accountability when completing a paper survey in person. Baruch and
Holtom (2008) performed a meta-analysis for publications containing internet surveys published
between 2000-2005 and found that the overall response rate has been decreasing over time. The
average response reported in this period was 52.7% with a high standard deviation of 20.4%.
This overall number, though, should be stratified differentiating organizational respondents
compared to general populations. At the organizational level, when managers and executives are
surveyed, the response rate drops to 35.7% with a standard deviation of 18.8% (Baruch and
Holtom, 2008). By comparison, this survey had a response rate of 33% for individuals targeted
with a direct email invitation. This completion rate is within the range of what would be
expected for a survey of this type. It is certainly higher than the minimum 10% response rate
that Nair, Adams, and Mertova (2008) have argued to be sufficient in many cases.
The completion rate of the individuals targeted by direct email contact may also have
depended on the fact that the survey answers were anonymized, a factor previously recognized to

106
affect the response rate (Singer, 1993; Saleh and Bista, 2017). It appears to be particularly
important in the medical products sector where companies may hesitate to criticize a government
agency that ultimately has authority over their ability to remain in business. Surveys such as that
considered here therefore provide a valuable method to collect information on views that may
not be stated publicly. Anecdotally, discussions that I had with the focus group and individual
conversations with potential participants made it clear that many individuals did not want the
results to be attributed to them or their companies.
Theresponse rates are known to be affected by multiple factors related to the target
audience, survey design, survey length, research affiliation, and compensation (Sheehan, 2001).
They are also influenced by the level of interest of participants in the topic being studied (Singer,
1993; Saleh and Bista, 2017). The design controls explored here seem to be of high interest to
the companies operating in the medical device sector as reflected by the participation of so many
unsolicited individuals who responded using a publicly published web link. The overall response
rate was also impacted by the relatively large number of questions, some with complicated
structures. The impact of survey length was studied by Herzog and Bachman, who attributed the
sustained participation in a survey to be heightened if the respondents had an intrinsic interest in
the survey or perceived it to be of high value (Herzog and Bachman, 1981). Nonetheless, in this
48-question survey, 57 people elected to drop out before completing the survey. For this
population of 57, 60% (34/57) dropped off in the first quarter, which mostly clarified
demographic profiles, while another 35% (20/57) dropped out in the second quarter. Thus, a
smaller number of respondents completed the later questions related to the final domain, process.
One of the most challenging aspects of this study was that of finding and sampling
individuals from medical products companies with the appropriate expertise. There is no

107
complete list of individuals working in this area of technology; incomplete lists of suitable
participants could be found, for example, by obtaining attendance lists for participants at relevant
conferences, but no job function or contact information was typically provided by those lists.
Instead, I used sampling methods in which individuals were purposefully chosen for their known
expertise and experience. Others were then identified using a “snowballing” method (MartínezMesa, et al., 2016). The results were supplemented using additional arms-length methods such
as posting an explanation of the survey with an anonymous link on internet sites such as
LinkedIn, which are known to be visited by the target population. These measures all appeared
to reach the desired audience of individuals as judged from demographic profiles reflecting a
high degree of expert knowledge related to design control for SaMD. The fact that some of the
findings of the survey were consistent with anecdotal observations, discussed earlier in chapter 2,
also suggested that a cross-section of the overall population of interest was represented. Still,
care must be taken to recognize that results may not fully represent views across the full
population, especially when more specialized questions are directed at a small subset of
respondents (Banerjee and Chaudhury, 2010). For example, only 3 engineers provided specific
comments regarding the types of improvements needed for certain regulatory documents.
When evaluating if the study has met the research goal, a main consideration for the
response rate is whether saturation was reached by the sampling. Saturation is defined as the
point at which further probing is believed to yield little improvement in the understanding of the
thematic trends of interest. Thus, social science literature recognizes the need to focus not
simply on response rate, but rather on the point where the most salient ideas have been obtained
(Sheehan, 2001; Weller, et al., 2018). Weller et al. studied published research and found that
saturation is reached in as few as 9 responses when there are three or more questions answered

108
by each person related to the specific topic. When less than three responses are received by each
respondent then 16 responses can be needed to reach saturation (Weller, et al., 2018). The larger
the response population the better the estimate, but this must be considered within the context of
completing the study (Kelley, et al., 2003).
In this study, the views of subgroups of participants with different job functions and
employer types appeared broadly similar and strengthened the argument that saturation had been
reached. Overall, in consideration of the limitations, I believe that this research has reached the
desired experts in sufficient numbers and this survey collected enough data to reach saturation
for the key themes related to design control for software. Care, though, will be used when
generalizing results. These limitations are considered in the conclusions drawn in this chapter.
5.2.2 Delimitations
The validity of this survey is not only influenced by the limitations of the research tool
and methodology; it is also subject to the delimitations chosen during the study design. This
survey targeted industry professionals and consultants with direct experience in software design
and development. Specifically, the group included midlevel to senior industry professionals in
technical / engineering, quality, or regulatory affairs with prior experience developing or
supporting software in medical devices. Other professionals, such as those in business, sales,
and marketing, might also have opinions on many of the same topics. Their expertise, however,
concerning the technology and design control approaches is typically limited. Within the
response group, more individuals from quality/ regulatory functions completed the survey than
from engineering. Additionally, nearly three-quarters of the respondents had experience
developing medical devices compared to one-quarter who had experience developing consumer

109
products. Thus, the data set is enriched by people with specific knowledge and experience
regarding regulatory and quality approaches to medical products.
Nonetheless, this survey was conducted on a range of companies involved in the
development of medical device software, including SiMD and SaMD, as well as consumer
products. The results thus provide a broad overview of software design control within this
domain. However, some risk of blurring is inherent when grouping these three types of product
development if the groups were to have different views. The subgroups, though, had similar
answers to most questions, as reflected by the cross-tabulations and comments from the different
subgroups. Nevertheless, a detailed focus on a specific type of software could potentially yield
different results when compared to results apparent from a broader survey such as this. The
impact of using a more focused respondent group on external validity must be considered when
generalizing the observations to a broader population (Price and Murnan, 2004; Ross and Bibler
Zaidi, 2019). This could be important if a specialized subgroup was not identified correctly. For
example, the use of the terms “lifestyle” and “fitness” when recruiting consumer product
developers might have affected the types of individuals who responded. Other developers, such
as gamers, for example, may have seen themselves as outside of the inclusion criteria, even
though such products have been used for medical applications such as those for mental health
and behavior modifications (Pakarinen and Salanterä, 2020).
Regardless, this population has a broad experience in DHT. When it comes to SDLC
methods, only one-third of respondents indicated direct experience with Agile. In the literature,
McHugh, McCaffery, and Casey surveyed a limited number of companies in Ireland that were
developing medical device software. They found that 25% of companies were using Agile and
half of the respondents had experience with Agile (McHugh, McCaffrey, and Casey, 2014).

110
While this is a limited study, the results are consistent with the observations from that earlier
study.
In a separate study of both consumer and regulated software, Dima and Maassen found
that 68% of people reported using Agile and the use was increasing across the industry (Dima
and Maassen, 2018). Their study may suggest that Agile is used more often in software
development outside of medical devices. The predominance of regulatory specialists in medical
device companies may also help to explain why experience with Agile methods was less
common amongst the respondents in the study reported here.
Additionally, the questions in this research centered around three domains suggested by
the HPT framework. However, the topic of software development in medical devices can be
explored from many perspectives, such as the impact of design controls on business strategy or
the efficiency of product development. As previously indicated in chapter 2, I considered other
frameworks, including implementation and satisfaction frameworks for the purpose of this study,
that might have explored some of these other dimensions. After consideration of the literature
and results, though, I believe that the HPT provided a systematic and broad review of the current
state that was more suitable than other frameworks to illuminate how device companies were
handling software methods.
Not only did I delimit the job functions and types of devices associated with the
respondents, but I also restricted the study to developmental activities within US requirements
under the FDA. The requirements within other markets, such as the EU, as well as the
requirements of other Agencies within the US, can impact the broader ecosystem and this in turn
could also impact companies. As noted in the survey comments, companies are frequently
challenged to develop products for global distribution. Thus, some of the views and experiences

111
of the respondents may have been colored by their experiences when trying to satisfy agencies
outside of the US, which often differ and are evolving as well. Therefore, the results must be
interpreted in the context of the time and regulatory environment of the US in the epoch of 2023-
2024.
5.3 Consideration of the Results
As explored in chapter 2, the experiences of the survey population are affected not only
by changes in software technologies but also by the regulations governing software. Central to
their activities is the need to comply with “design controls, " a system introduced in 1997,
designed to ensure that medical care products are safe and effective. Design controls, though,
were initially put in place when most devices were mechanical in nature. They fit uneasily with
the evolving methodologies for software, which have moved from the waterfall/cascade method
introduced in the 1950s to the Agile SDLC circa 2001. Whereas the waterfall or “V” SDLC
aligns closely with design control requirements outlined in regulatory applications, Agile
methods are structured differently and are less amenable to design controls as currently
practiced. In the recent decade, the FDA has issued many guidances and experimental
regulations to clarify the regulatory oversight of these changing methodologies. Relatively little
is known, however, about the industry’s views and experiences. What is available through
literature and public regulatory summaries focuses instead on individual case studies and surveys
rather than detailed documentation of common developmental practices.
The purpose of this research was to explore more systematically the current state of
design control for SaMD/ SiMD in the industry. The HPT framework employed here examines
industry views across three domains: context, content, and process. Content includes the
information and requirements included, or omitted, from the regulations and policy. Context

112
examines the broader environment, including the legal and regulatory environment currently in
place. Process refers to the process of establishing, writing, negotiating, passing, and
communicating a new policy (Walt and Gilson, 1994; O’Brien et al., 2020). Although several
other frameworks were initially explored, this one was predicted to give a relatively broad view
of the topic across different domains. The broad and deep insights that I gained from the survey
make me satisfied that this choice was effective to provide an initial exploration of industry
views.
5.3.1 Content
5.3.1.1 Resource Quality for Device Classification
A first step when dealing with software for a health or medical purpose is to assess
whether FDA regulations for design controls apply to the product in development. Results here
identify that, for most companies, both internal and external communications in conjunction with
existing guidance documents are the primary ways in which companies initially determine what
degree of FDA oversight is required. Further, respondents did not appear to have difficulties
when making this determination and found the available content useful in determining the
classification for software. This is an encouraging finding because consumer products
companies appeared to regard such documentary guidance as important when deciding whether
they would eventually classify their products as medical devices and thus incorporate mandatory
design control into their processes.
Perhaps the positive flavor of these results is surprising, given the large number of critical
comments from industry in response to relevant sections of the 21st Century Cures Act, the FDA
Pre-Certification Pilot Programs, and to the many revised guidance documents and resources
provided by the FDA (H.R. 34, 2016; FDA, 2017), which were summarized in chapter 2.

113
Additionally, this does not appear to align with the study conducted by Alon, Stern, and Torous
in 2020, in which they examined how 120 commercially available mobile applications had been
classified according to whether they required FDA review. In that study, they could not find a
standard differentiator and thus found it difficult to determine if the apps had been correctly or
incorrectly identified as medical devices (Alon, Stern, and Torous 2020). In the survey
described here, though, most individuals had experience developing medical devices, and thus
were more likely familiar with the rules for device classification and less likely to be confused by
the design control framework. A significant number have also engaged with the FDA during the
submission process where they would have been able to gain experience and feedback from the
FDA. On the other hand, individuals with less experience in this area may be more likely to be
confused and then share the more negative views expressed in the public comments provided
earlier, during the legal and rule-making processes.
The ability to learn the rules that govern medical devices as opposed to consumer
software may also be affected by the fact that some companies, especially consumer products
companies, may not know where to find the information that they need to classify their software,
even if that information could be helpful. Indeed, opportunities to hear from the FDA about
these rules are rare. Even in cases where the Agency has given feedback directly to a specific
company, their discussions are not disclosed publicly and thus do not serve as a learning
opportunity for others. Without such learning opportunities, those who develop software might
not be aware that rules and guidances even exist. They then may be overconfident in their ability
to classify their devices and feel generally satisfied that resources are not needed to make their
decisions, however erroneous that belief might turn out to be. This study, though, solicited
feedback from individuals with experience.

114
5.3.1.2 Resource Quality for Design Control Implementation
After a company determines that design control requirements apply to its product, the
company must implement those requirements based on the information available to it. The
respondents gave mixed reviews regarding the usefulness of material related to design controls,
and in particular to the level of technical detail that they contained. Notably, the most common
word association that they gave regarding materials related to design control was “too high
level”. Most individuals stated that they commonly looked instead to standards for detailed
technical procedures to compensate for this lack of detail. This is not completely satisfactory.
Although the FDA recognizes and encourages the use of standards, it can be challenging to
incorporate standards by reference into regulations. Most standards are developed by
independent groups that update the standards frequently to accommodate changes in technology
and the evolution of overarching regulations in different companies. These difficulties were
encountered previously, for example, when regulators tried to incorporate the risk management
standard, ISO 13485, by reference into the 2023 modification to the Quality System Regulations
(89 FR 7496 (2024), Cordie-Bancroft, 2024).
Just as standards change, so too do the guidances developed by regulatory agencies.
Over time, these changing updates to guidances reflect the tangible steps made by the FDA to
encourage innovation, reduce regulatory burden where possible, and meet the required
commitments from the Cures Act (Soo, 2014; Gottlieb, 2018, H.R. 34, 2016; FDA, 2019a).
Nonetheless, leadership in the FDA has noted that challenges arise when working within the
current regulatory framework and interpreting the regulations and guidances (Shureen, 2018). It
appears from the survey results presented here that the industry feels that the current regulatory
system does work in general, but the regulatory framework does not fit easily for the
development of software.

115
Documentation can only be helpful if the users know about it and understand its content.
It was striking, then, that some software developers surveyed here were unfamiliar with some of
the guidance available to them. More specifically, many were not familiar with the two key
documents for low-risk devices: General Wellness: Policy for Low-Risk Devices, Clinical
Decision Support Software, and the FDA publication Changes as a result of 21st Century Cures
Act. These FDA documents describe how the FDA will reduce the regulatory burden for specific
types of software with lower clinical risk and should be particularly relevant for those with
consumer-based software. My research did not, however, specifically explore the reasons why
this guidance is not familiar to the industry participants. These documents are relatively new. It
may be that some software developers are slow to stay current with the increasing number of
guidance documents released by the FDA or they may not find the documents relevant to the
type of products they support. Still, failure to even read the documents now available would not
help to alleviate confusion about when and how to apply the FDA’s policies that reduce the
burden for low-risk software.
When respondents here were asked how FDA’s documentation might be strengthened,
several suggested that detailed examples would help to make guidances more useful. These
comments further suggest that individuals are finding it difficult to extrapolate from the available
high-level content in the guidance to the specific needs of their product. Criticisms here add to
concerns expressed elsewhere with this particular guidance. The Clinical Decision Support
Coalition, for example, has also criticized the guidance which it feels does not go far enough to
establish a risk-based and transparent set of guidelines (Lim, 2018). Without more specific
details, practitioners are more likely to be dissatisfied with the current regulatory requirements.

116
5.3.1.3 Regulating Cybersecurity
Software has an inherent vulnerability- the need for cybersecurity- that is not common to
other types of medical devices. Industry respondents surveyed here appeared to be very familiar
with the FDA guidance on cybersecurity, and many had strongly stated opinions. In particular,
they were critical of what they regarded as superficial technical guidance and wanted more help
to understand what was needed specifically to comply with cybersecurity requirements.
Additionally, when responses of the ENG group were compared to those of the QRA group, the
ENG group more commonly indicated that the guidance was not useful (p<0.04) and more
commonly requested additional technical examples. These differences seem unsurprising, given
that these two groups have different roles when assuring compliance with cybersecurity
requirements. Engineers must implement solutions more directly so depend on having a clear
understanding of technical expectations that would be important in implementing design controls
related to cybersecurity (Gallagher, 2019, Schwartz et al., 2018), whereas regulatory
professionals must organize their design control deliverables into submission to the FDA, so
operate a greater arm’s length. Conclusions should be tempered because the number of
respondents in each group was not large. Nonetheless. results here suggest that clarification of
cybersecurity requirements is an area that the FDA could usefully address in the future. This
parallels some of the findings from a review done by the US Government Accountability Office
which indicated that the FDA should work with the Cybersecurity Infrastructure Agency and
recommended procedural updates to address the needs of cybersecurity in the future (GAO,
2023). I anticipate that this area will be examined more fully in the coming years in a more
systematic analysis.

117
5.3.1.4 Implementing Design Controls within the Regulatory Framework
What did seem intriguing was the finding that ENG versus QRA subgroups differed
when asked how easy it was to implement design controls. As noted above, ENG respondents
are typically responsible for working within the design control system and implementing its
requirements on the ground. Regulatory and quality experts have the luxury of examining the
design control system at a higher level, so may see implementation easier because they are not
privy to the day-to-day challenges involved in its extensive documentation and burdensome
review requirements. This difference in opinion also seems to be recognized by certain
respondents. One comment, “make management and software engineers [understand] the
burden is not so high.”, seems to indicate, at least to me, that engineers see design controls as
more challenging than they should. Interestingly, this difference has not previously been
discussed in the literature to my knowledge. It does, though, align with comments made by
Cooper in his book, Inmates are Running the Asylum, who wrote that the precise, detailed
technical knowledge required to write and develop software is often divorced from an equivalent
knowledge of user needs and aptitudes outlined in the very regulations they are trying to
implement (Cooper, 2004).
No matter how early in the development process specifications are drafted,
they cannot substitute for interaction design. And no matter how hard they try,
programmers cannot consistently arrive at a successful design. Not only are
their methods, training, and aptitude wrong for the job, but they are caught in
a strong conflict of interest between serving the user’s needs and making their
programming job easier. (Cooper, 2004, pg. 81-82)
Respondents in this survey appear to recognize that specialized skills are needed to work
within a design control system. The implementation of design controls, like any other
complicated system, takes time and experience. Thus, satisfaction with the system seems to
increase with familiarity as users understand the system and this then leads to higher quality

118
information and overall satisfaction (Kalankesh, et al., 2020). The shift seems apparent when
one compares the comments from the 1997 preamble to the new design control regulations
(FDA, 1997) to comments accompanying the 2024 modifications of the quality system
regulations (89 FR 7496, 2024); the later comments appear to reflect a better understanding and
more nuanced critique of the requirements for all medical devices, including software. In this
survey, respondents were mostly experienced experts in medical product development. We
might expect that they will most likely be familiar with design control implementation and thus
might express higher levels of satisfaction than those who are introduced to the materials for the
first time.
Given the effect of familiarity on satisfaction and support for the design control
paradigm, whether respondents working with consumer-based rather than medical products
would be less familiar with design controls and less satisfied with them was questioned at the
outset of this research. However, crosstabulations did not yield a significant difference in the
experience and views of respondents working with consumer-based products versus medical
devices. It is premature to suggest why. In part, it may simply be that the number of
respondents in each subgroup is too small to make a meaningful statistical conclusion. However,
it is also quite conceivable that the results represent a pattern more broadly. Software engineers
often move from job to job within a range of companies with different types of products, so some
of the respondents working with consumer-based products at the time of the survey may have
had the occasion to learn about design controls in a previous place of employment. Alternatively,
it is possible that the method of securing respondents working with consumer-based products
was skewed toward those engaged with health-related applications, who would have had more
reason to learn about or even implement design controls.

119
At the outset of this research, the author was particularly interested in the way that
different software methodologies were being used and integrated with design controls. Agile
implementations follow different rules that fit poorly with the structure of the design control
system (McHugh, et al., 2013; McHugh, McCaffery and Casey, 2014; Zema, et al., 2015; Lima,
2020), so one might expect that this might be an area of confusion for Agile practitioners. It is
perhaps unexpected, then, to find that views regarding design controls were not statistically
different when comparing respondents working with Agile methods and those working with
other software platforms. The number of respondents in the consumer group was relatively
small (n=15), so the results could simply be related to the challenges of applying statistics to
small numbers. Nonetheless, the finding suggests that many members in both groups have a
working knowledge of regulatory requirements for software regardless of the software platform
in which they most commonly work.
Overall, results suggest that industry respondents are not looking for a fundamental shift
in design control philosophy or available content regarding requirements for SaMD. The
industry may not always find design control easy, but rather view it as a known requirement for
development. Instead, they are looking for specific and practical technical guidance, detailed
examples, and instructions for how to operate within the framework. Adding this content to
guidance documents or Standards may make them more useful to the industry. Further, more
transparency for specific examples and devices that have gone through the FDA approval
framework may build a database that the industry can reference and draw on to develop their
practice.

120
5.3.2 Context
The second domain of the HPT framework addresses the broader ecosystem in which
companies develop software as/in a medical device. A central theme here was how previous
experiences of industry set the agenda for future planning to obtain regulatory approvals. It
includes the approachability of regulators to encourage software products and to approve
products in a timely and cooperative way. Further, previous literature reviewed in chapter 2
suggested that different stakeholders approached software development from different ideologies
and perspectives. This in turn should impact the environmental context and might be reflected in
their responses to the survey.
Results in the survey suggest that the environment for the development of regulated
software creates several challenges for the industry. Two elements, “regulatory burden” and
“confusion related to regulatory requirements”, were ranked as the most impactful to operations.
These two aspects are unlikely to be such critical concerns in other industry sectors, where
choices such as “additional costs of development” and “match to the company’s business
portfolio” tend to be more important criteria for decision-making (Keathley, et al., 2013). These
are critical concepts that companies need to consider as part of portfolio management and
balancing the economics for innovation. The results, however, align with the perceived barriers
reported in a smaller study conducted with twenty medical device companies in Ireland
(McHugh, McCaffery, and Casey, 2014) in 2014 as well as those reported by Jamot and
Pettersson in interviews of ten industry individuals conducted in 2016 (Jamot and Pettersson,
2016). The consistency in expressed views over time suggests that the regulatory environment
has not greatly improved over a period of ten years, even with the addition of many new
guidance documents and the beta-testing of the FDA Pre-Cert program. It may signal that those

121
steps were unable to significantly impact the overall ecosystem and perceptions of companies
involved in the development of SaMD.
Contributing to regulatory barriers was seen to be the level of documentation, often
unexpectedly in-depth, that the FDA has required prior to market authorization. Over half of the
respondents reported that they had to perform additional testing, provide additional clinical data,
and/or add more safety controls to address FDA questions. These results suggest that the
industry is underestimating the amount of information that the FDA will expect in regulatory
submissions despite their numerous guidances and multiple communication methods. In part,
this may result because limited data is publicly available to help anticipate the more granular
expectations of the FDA during a submission. FDA does publish limited summaries of cleared
or approved submissions, but this is mostly concerned with the clinical profile of the final
product. Conference proceedings and literature could be alternative sources of information, but
these are often in the form of case reports that focus on a specific submission or technology. As
examples, the m2000 software development process (Rasmussen et al., 2009) or Apple iWatch
atrial fibrillation (AFib) submissions outline the negotiations with the FDA for those specific
devices and indications (Gottlieb, 2018). By contrast, my research looks more broadly at the
experiences of individuals who develop a range of software products and who mostly have recent
submission experience (51% reported less than 2 years, only 6% over 5 years). They may give
more insight into the current ecosystem than can be gleaned from previous research literature.
Frustrations were clear when respondents were asked about the inconsistencies in
feedback related to the documentation expected by the Agency or to changes in FDA
expectations over time. For example, one respondent stated, “having been through two recent
successful submissions, the regulatory burden seems to change and gets more intense with each

122
review team and the unique requirements of each reviewer” and another remarked that “feedback
changed in every Q-sub depending on who review team was assigned.” Their concern is
understandable given the investment and planning that goes into the development process for a
commercially viable product. The frustrations remain even after the publication of the detailed
guidance, Content of Premarket Submission for Deice Software Functions (FDA, 2023), despite
feedback suggesting that it was useful to most of the industry respondents. This area of concern
presents an opportunity for the Agency to increase transparency by providing fuller information
on areas of deficiency that they commonly see or by giving more training opportunities to
decrease confusion that is now seen as a barrier by the industry.
FDA’s rules for design control are supposed to be flexible regardless of the types SDLC
development strategies that are used, even though these can have markedly different
documentation practices (FDA, 1997). The differences that have been previously identified for
development practices using Agile (McHugh, McCaffery, and Casey, 2014; Jamot and
Pettersson, 2016; Lima, 2020) can still cause issues, reflected in the fact that most respondents
here rely, at least in part, on using a waterfall or modified waterfall methodology and then use
Agile selectively. It must be acknowledged that the response count was low for this question
(n=20). Nevertheless, the data suggest that Agile is used less commonly than is typical for
software development in general, which has been estimated previously at 68% (Dima and
Maassen, 2018). The results are, however, consistent with published literature and exemplar
cases for the medical device industry. For example, Heeager and Nielsen stated that Agile
methods suffer when adapted to FDA-regulated devices (Heeager and Nielsen, 2009); Lima’s
structured interviews with software developers in the healthcare sector led her to suggest that
Agile methods may not always be the best choice for medical devices (Lima, 2020). Thus,

123
companies are faced with deciding if the positive attributes of Agile methods outweigh the
potential regulatory burden. Indeed, a hybrid approach appears to suit both the regulations and
the industry well. The quote from Lima’s structured interviews appears to summarize the
situation – “For Carlos, every project could be agile, but not every project should be.” (Lima,
2020) Thus, while some companies have successfully found this balance, the sense remains that
information is not transparent or available in a way that others can learn from and apply for
future development opportunities.
Although some aspects of the regulatory context are challenging, this seems not to have
affected the views that respondents hold with respect to their interactions with FDA through
multiple mechanisms. The ability to speak with the FDA before a submission as part of the presubmission process was identified as one positive example of a helpful opportunity for
interactions. At the same time, however, the experience of several respondents suggests that the
software knowledge of many (though not all) FDA reviewers is not technically sophisticated.
This problem is well recognized by FDA leadership and has led to strategic initiatives to
strengthen the review process for software devices, and to develop an Office of Science and
Engineering Laboratories (OSEL) Center of Excellence for software (Morrison et al., 2018;
FDA, 2022a). Despite these concerns, the relatively high level of satisfaction with FDA
interactions was striking. A high level of satisfaction with FDA interactions suggests that FDA
has already built a base of cooperation that will help as it tries to strengthen the regulatory
environment further.
5.3.3 Process
The final domain, “process”, explored how regulations related to software development
are written and modified. Several points exist at which industry can interact and potentially

124
influence those regulatory policies, both directly through review and comment processes
established by law and indirectly through trade groups and standard-setting organizations
(Lievevrouw et al., 2022a; Lievevrouw et al., 2022b).
Trade groups are collective organizations that provide an important avenue by which
companies operating in the same discipline can cooperate and advocate for common interests. In
this survey, AdvaMed ranked highest for membership participation. According to their website,
this advantage reflects one of their key priorities – “A seat at the table alongside industry
stakeholders, peers and competitors to represent your business interests and policy priorities.”
(AdvaMed, 2024, AdvaMed Membership) AdvaMed has been ranked highly as an effective trade
group in 2019 and 2021 by an independent APCO Trademarks study of the influence of trade
groups (AdvaMed, 2021).
In this study, most (83%) of the respondents appeared to believe that trade groups can
successfully influence regulation. This was consistent across subgroups with or without direct
Agile experience and with either MD or CONS development experience. Further, their
importance in the medical device industry is reflected by the finding that half of the respondents
in this survey participate in trade groups and often participate in more than one trade group.
However, the value of the trade groups is not recognized by everyone; about a quarter of the
respondents did not believe that trade groups are effective or that they understand the needs of
companies in this area. This survey did not explore the reasons for the differences in attitude
toward the usefulness of trade groups, and no information on this aspect could be found in the
literature.
In this study, one of the most highly ranked areas in which efforts could be made to
reduce regulatory hurdles was the slow pace of regulatory development/change, including the

125
need to improve the content of current guidance. This priority seems consistent with similar
previous concerns reported in the literature (McHugh, et al., 2013; McHugh, McCaffery, and
Casey, 2014). However, the collated results suggest that industry experts favor a cautious
approach when making changes to the design control system. Few advocated a fundamental
revision, such as that explored by the FDA in its piloted Software Pre-Certification program. In
fact, responses were typically critical of that program. Rather, many seemed to favor finding a
way to introduce more flexibility within the current design control framework. Still, opinions on
the degree of needed intervention appeared to vary amongst the respondents. Some appeared to
feel that the current system of design control was sufficient to meet most needs, but others
expressed stronger concerns that the current system might not be sufficient for the future. For
most, solutions appeared to relate to improving the interplay between the SDLC process and
technical standards within the current design control framework rather than replacing the
framework with something else. Presumably, industry concerns might be reduced if the
technical requirements could be specified in more detail and if design control could offer
flexibility to follow and reference relevant standards.
Individuals with direct Agile experience appeared to think that it should be possible to
use Agile methods in a manner consistent with design controls. Their comments align with
recent literature in which models for such approaches have been developed and with examples in
which previous companies appeared to integrate Agile methods successfully (Rasmussen et al.,
2009; McHugh, McCaffery, and Casey, 2014; Lima, 2020). At present, though, concerns remain
that an Agile approach alone might not be suited for all situations. The key would then be
knowing how to define the applications in which it is – or is not – appropriate.

126
Preparing for the future requires that modified regulations respond to changing needs,
and both industry and trade groups must play a key role in this change. The area of
cybersecurity, discussed above, is a case in point. Another area of increasing focus is the use of
artificial intelligence, which may challenge the basic tenets of design control and may justify the
views by some that design control approaches may not be sufficient in the future. These
technological advances and the regulations that govern them are rich areas for future research.
5.4 Conclusions and Recommendations
Software is an integral part to our healthcare as it is to our society in general. The use of
DHT will only increase in the future; regulations must therefore balance the need to encourage
rapid innovation with the need to assure the safety of the devices.
Based on this study the industry is broadly yet cautiously satisfied with the current state
of design control for DHT and is actively involved in continuing to shape the regulations that
oversee its development. They report positive interactions with the Agency with product
submissions and regulatory advocacy. The differences appear to be tactical, rather than
philosophical; individuals in all actor groups want to ensure that they will be able to meet
regulatory requirements and that their efforts will not be compromised by regulatory changes
that could reset the design control requirements. Engineers in particular appear to need more
specific technical guidance to operate effectively within the existing framework. Increased
transparency about how to structure SDLC and software development strategies, either directly
from the FDA or through experience-sharing between companies, might help to address this
frustration.
Differences were found to exist in the level of awareness and satisfaction with individual
guidance documents. Better promotion of new guidance documents, such as those for low-risk

127
software, may need to be directed at newer entrants to the field, where many seem unaware that
the documents even exist. Further, the guidance for clinical decision-support software could use
additional specific examples and more practical suggestions for how to implement its
recommendations in practice. Additionally, while the industry responded favorably to the newer
version of the FDA guidance on cybersecurity, more content is needed to expand on best
practices and technical solutions in this area. Thus, the observations in this study help to define
approaches to the future development of FDA guidances, including those related to cybersecurity
and artificial intelligence, to make them more useful to the industry. Creating closer
relationships between the technical standards that complement the FDA guidance could help to
bridge the gap between high-level philosophy and technical detail, which seems to be at the
center of some dissatisfaction.
The views on the process, content, and context of developing SaMD and SiMD are
remarkably similar across regulatory and engineering experts whether they deal with FDAapproved medical devices or consumer-based products. They typically feel that Agile SDLC can
be used compliantly within the design control framework but may be best suited to a focused role
supplemented by waterfall methods. rather than a general place across all of development.
5.5 Future Research
This survey was a broad exploration of industry views regarding design controls for
SaMD and SiMD devices. It leaves ample opportunity for future studies that investigate specific
more focused areas of study. For example, it would be useful to explore in more detail how
individuals from consumer products are learning about design control. Additionally, another
interesting area to study would be the ways in which standards-setting bodies are now interacting
with FDA with regard to DHT.

128
The development of novel and important digital health technologies is an area of rapid
technological change, so views will change in concert with that maturation process. The
growing emergence of artificial intelligence and cybersecurity represent areas that will drive
change for DHTs, in ways that will shape the medical system moving forward. The study of
regulatory science in this area will be important to guide best practices that assure the safe use of
DHT, if those products are to contribute most effectively to public health.

129
References
21USC§§301-392 (1976) Chapter 9: Federal Food, Drug, and Cosmetic Act, U.S. 94th
Congress: Federal Food, Drug, and Cosmetic Act, Sections 201-392. Available at:
https://www.loc.gov/item/uscode1976-006021009/. (Accessed: 14 July 2024).
43 FR 47184 (1978) Quality system regulations. Available at:
https://archives.federalregister.gov/issue_slice/1978/10/13/47182-47188.pdf#page=3. (Accessed:
14 July 2024).
84 FR 51167 (2019) Clinical decision support software draft guidance, Federal Register, 84 FR
51167, Docket No. FDA-2017-D-6569, 2019-2100. Available at:
https://www.federalregister.gov/documents/2019/09/27/2019-21000/clinical-decision-supportsoftware-draft-guidance-for-industry-and-food-and-drug-administration. (Accessed: 14 July
2024).
86 FR 60838 (2021) Content of premarket submission for device software functions, Federal
Register, 86 FR 60838, Document No. FDA-2021-D-0775. Available at:
https://www.federalregister.gov/documents/2021/11/04/2021-24061/content-of-premarketsubmissions-for-device-software-functions-draft-guidance-for-industry-and-food. (Accessed: 14
July 2024).
89 FR 7496 (1996) Medical devices; Current good manufacturing practice (C-GMP) Final rule;
Quality system regulation, Federal Register: 61 (195), 52601-52662. Available at:
https://www.govinfo.gov/content/pkg/FR-1996-10-07/pdf/96-25720.pdf. (Accessed: 14 July
2024).
89 FR 7496 (2024) Medical devices; Quality system regulation amendments, Federal Register
89 FR 7496, Document No. FDA-2021-N-0507, 7496-7525.
AAMI (2007) The quality system compendium: GMP requirements & industry practice. 2nd en.
Association for the Advancement of Medical Instrumentation. Arlington, VA: AAMI.
AdvaMed. (2021) AdvaMed named a top industry group in APCO worldwide "Trademarks"
study released today. Available at: https://www.advamed.org/industry-updates/news/advamednamed-a-top-industry-group-in-apco-worldwide-trademarks-study-released-today/ (Accessed
June 2, 2024).
AdvaMed. (2024) Membership & join. Available at: https://www.advamed.org/membership-join/
(Accessed June 2, 2024).
Alon, N., Stern, A. D., and Torous, J. (2020). Assessing the Food and Drug Administration’s
Risk-Based Framework for Software Precertification with Top Health Apps in the United States:
Quality Improvement Study. JMIR mHealth and uHealth, 8(10), e20482.

130
Alsaadi, M., Lisitsa, A., Khalaf, M. and Qasaimeh, M. (2019) Investigating the capability of
agile processes to support medical devices regulations: The case of XP, Scrum, and FDD with
EU MDR regulations, D.-S. Huang et al. (eds.). 15th International Conference on Intelligent
Computing, Springer Nature Switzerland AG, 581-592.
Banerjee, A. and Chaudhury, S. (2010) Statistics without tears: Populations and samples.
Industrial Psychiatry Journal, 19(1), 60-65.
Baruch, Y. and Holtom, B. C. (2008) Survey response rate levels and trends in organizational
research. Human Relations, 61(8), 1139-1160.
Beck, K., et al. (2001) Agile manifesto for agile software development. Available at:
http://agilemanifesto.org (Accessed: 29 October 2021).
Benington, H. D. (1983) Production of large computer programs. Annals of the History of
Computing, 5(4), 350-361.
Benson, M. S., Eccleston, R. C. and Barnett, M P. H. (1988) The FDA's regulation of medical
devices: A decade of change. Food, Drug Cosmetic Law Journal, 43(3), 495-510.
Boehm, B. and Turner, R. (2003) Balancing agility and discipline: A guide for the perplexed.
Boston, MA: Addison-Wesley Professional.
Callegaro, M., Manfreda, K. L. and Vehovar, V. (2015) Web survey methodology, London,
England: SAGE Publications Ltd.
Cockburn, A. (2006) Agile software development: The cooperative game. 2nd edn. Boston, MA.:
Addison-Wesley Longman Publishing Co, Inc.
Cooper, A. (2004) The inmates are running the asylum: Why high tech products drive us crazy
and how to restore the sanity. Indianapolis, IN.: Sams Publishing.
Cordie-Bancroft, L. (2024) FDA and ISO 13485 - Cause for concern or celebration? BSI
Compliance Navigator for Medical Devices. 19 February. Available at:
https://compliancenavigator.bsigroup.com/en/medicaldeviceblog/fda-and-iso-13485--cause-forconcern-or-celebration/ (Accessed: 09 June 2024).
Council Directive 93/42/EEC (1993) Council directive 93/42/EEC of 14 June 1993 concerning
medical devices. Journal of European Union Law. 169, 12 July 1993 0001 - 0043. (European
Union). Available at: https://eur-lex.europa.eu/legalcontent/EN/TXT/?uri=celex%3A31993L0042
Devroe, R. and Wauters, B. (2019) How to enhance the external validity of survey experiments?
A discussion on the basis of an experimental study on political gender stereotypes in Flanders
(Belgium), London, England: SAGE Publications Ltd.

131
Dima, A. M. and Maassen, M. A. (2018) From waterfall to agile software: Development models
in the IT Sector, 2006 to 2018. Impacts on company management. Journal of International
Studies, 11(2), 315-326.
FDA. (1997) Design control guidance for medical device manufacturers, Food and Drug
Administration (United States). Available at: https://www.fda.gov/regulatoryinformation/search-fda-guidance-documents/design-control-guidance-medical-devicemanufacturers (Accessed: 14 July 2024)
FDA. (2010) Infusion pump improvement initiative, Food and Drug Administration (United
States). Available at: https://www.fda.gov/medical-devices/infusion-pumps/white-paperinfusion-pump-improvement-initiative. (Accessed: 14 July 2024).
FDA. (2017b) FDA selects participants for new digital health software precertification pilot
program, Food and Drug Administration (United States). Available at:
https://www.fda.gov/news-events/press-announcements/fda-selects-participants-new-digitalhealth-software-precertification-pilot-program (Accessed: 14 July 2024).
FDA. (2018a) De Novo classification request for irregular rhythm notification feature, De Novo
Summary (DEN180042), Apple Inc. Available at:
https://www.accessdata.fda.gov/cdrh_docs/reviews/DEN180042.pdf. (Accessed: 14 July 2024)
FDA. (2018b) De Novo classification request for ECG App, De Novo Summary (DEN180044),
Apple Inc. Available at: https://www.accessdata.fda.gov/cdrh_docs/reviews/DEN180044.pdf.
(Accessed: 14 July 2024).
FDA (2018c) FDA budget matters: Advancing innovation in digital health. Available at:
https://www.fda.gov/news-events/fda-voices/fda-budget-matters-advancing-innovation-digitalhealth (Accessed: 14 July 2024).
FDA. (2019a) Changes to existing medical software policies resulting from section 3060 of the
21st Century Cures Act, Food and Drug Administration (United States). Available at:
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/changes-existingmedical-software-policies-resulting-section-3060-21st-century-cures-act. (Accessed: 14 July
2024)
FDA. (2019b) FDA Technology Modernization Action Plan (TMAP), Food and Drug
Administration (United States). Available at: https://www.fda.gov/about-fda/reports/fdastechnology-modernization-actionplan#:~:text=The%20FDA%E2%80%99s%20Technology%20Modernization%20Action%20Pla
n%20%28TMAP%29%2C%20described,data%2C%20and%20analytics%E2%80%94to%20adv
ance%20FDA%E2%80%99s%20public%20health%20mission. (Accessed: 14 July 2024).
FDA. (2019c) General wellness: policy for low risk devices, Food and Drug Administration
(United States). Available at: https://www.fda.gov/regulatory-information/search-fda-guidancedocuments/general-wellness-policy-low-risk-devices. (Accessed: 14 July 2024).

132
FDA. (2019d) Medical device data systems, medical image storage devices, and medical image
communications devices, U.S. Department of Health & Human Serivces Guidance Portal.
Available at: https://www.hhs.gov/guidance/document/medical-device-data-systems-medicalimage-storage-devices-and-medical-image-communications. (Accessed: 14 July 2024)
FDA (2019e). Policy for device software functions and mobile medical application: Guidance
for industry and Food and Drug Administration staff, U. S. Department of Health & Human
Services. Available at: https://www.hhs.gov/guidance/document/policy-device-softwarefunctions-and-mobile-medical-applications-guidance-industry-and. (Accessed: 14 July 2024)
FDA. (2020a) Multiple function device products: Policy and considerations: Guidance for
industry and Food and Drug Administration staff, Food and Drug Administration (United
States). Available at: https://www.fda.gov/regulatory-information/search-fda-guidancedocuments/multiple-function-device-products-policy-andconsiderations#:~:text=This%20guidance%20explains%20FDA%27s%20regulatory%20approac
h%20and%20policy,device%20function%20that%20is%20subject%20to%20FDA%20review.
(Accessed: 14 July 2024)
FDA. (2020b) What is digital health? Food and Drug Administration (United States). Available
at: https://www.fda.gov/medical-devices/digital-health-center-excellence/what-digital-health
(Accessed: 14 July 2024).
FDA. (2022a) Cybersecurity in medical devices: Quality system considerations and content of
premarket submissions draft guidance for industry and food and drug administration staff, Food
and Drug Administration (United States). Available at: https://www.fda.gov/regulatoryinformation/search-fda-guidance-documents/cybersecurity-medical-devices-quality-systemconsiderations-and-content-premarket-submissions. (Accessed: 14 July 2024).
FDA. (2022b) Modernization in action 2022. Technology modernization action plan (TMAP)
and data modernization action plan (DMAP) anniversary report, Food and Drug Administration
(United States). Available
at:https://www.fda.gov/files/about%20fda/published/Modernization_in_Action_2022.pdf.
(Accessed: 14 July 2024).
FDA. (2022c) The software precertification (Pre-Cert) pilot program: Tailored total product
lifecycle approaches and key findings, Food and Drug Administration (United States). Available
at: https://www.fda.gov/media/161815/download. (Accessed: 14 July 2024).
FDA. (2022d) Digital health software precertification (pre-cert) pilot program: Software
precertification program 2019 mid-year update. Available at: https://www.fda.gov/medicaldevices/digital-health-center-excellence/digital-health-software-precertification-pre-cert-pilotprogram. (Accessed: 14 July 2024)
FDA. (2023) Content of premarket submissions for device software functions, Food and Drug
Administration (United States). Available at:https://www.fda.gov/regulatory-information/searchfda-guidance-documents/content-premarket-submissions-device-software-functions. (Accessed:
14 July 2024).

133
Fixsen, D. L., Blase, K. A., Naoom, S. F. and Wallace, F. (2009) Core implementation
components. Research on Social Work Practice, 19(5), 531-540.
Forsberg, K. and Mooz, H. (1991) The relationship of system engineering to the project cycle.
INCOSE International Symposium, 1(1), 57-65.
Gallagher. (2019) Cyber security challenges for medical device makers — Managing the
escalating risks. Gallagher, News & Insights, 26 August. Available at:
https://www.ajg.com/us/news-and-insights/2019/08/cyber-security-challenges-for-medicaldevice-makers/. (Accessed: 09 June 2024).
GAO. (2023) Medical device cybersecurity: Agencies need to update agreement to ensure
effective coordination, U.S. Government Accountability Office, GAO-24-106683. Available at:
https://www.gao.gov/products/gao-24-106683 (Accessed: 05 July 2024).
Gordon, W. J. and Stern, A. D. (2019) Challenges and opportunities in software-driven medical
devices. Nature Biomedical Engineering, 3(7), 493-497.
Gottlieb, S. (2018) Statement from FDA Commissioner Scott Gottlieb, M.D., and Center for
Devices and Radiological Health Director Jeff Shuren, M.D., J.D., on Agency efforts to work
with tech industry to spur innovation in digital health. Available at: https://www.fda.gov/newsevents/press-announcements/statement-fda-commissioner-scott-gottlieb-md-and-center-devicesand-radiological-health-director. (Accessed: 14 July 2024).
Heeager, L. T. and Nielsen, P. A. (2009) Agile software development and its compatibility with
a document-driven approach? A case study. ACIS 2009 Proceedings, 84, 204-214.
Heeager, L. T. and Nielsen, P. A. (2020) Meshing agile and plan-driven development in safetycritical software: A case study. Empirical Software Engineering, 25 (2), 1035-1062.
Herzog, A. R. and Bachman, J. G. (1981) Effects of questionnaire length on response quality.
Public Opinion Quarterly, 45(4), 549-559.
H.R. 34 (2016) 21st Century Cures Act. 114th Congress: 21st Century Cures Act of 2016.
Available at: https://www.congress.gov/bill/115th-congress/house-bill/34/text
IMDRF. (2013) Software as a Medical Device (SaMD): Key Definitions, IMDRF -
International Medical Device Regulators Forum. Available at:
https://www.imdrf.org/documents/library?f%5B0%5D=type%3Atechnical_document.
(Accessed: 14 July 2024).
IMDRF. (2014) Software as a Medical Device: Possible Framework for Risk Categorization and
Corresponding Considerations. Available at:
https://www.imdrf.org/documents/library?f%5B0%5D=type%3Atechnical_document.
(Accessed: 14 July 2024).

134
IMDRF. (2015) Software as a Medical Device (SaMD): Application of Quality Management
System. Available at:
https://www.imdrf.org/documents/library?f%5B0%5D=type%3Atechnical_document.
(Accessed: 14 July 2024).
IMDRF. (2017) Software as a Medical Device (SaMD): Clinical Evaluation. Available at:
https://www.imdrf.org/documents/library?f%5B0%5D=type%3Atechnical_document.
(Accessed: 14 July 2024).
ISO. (2016) ISO 13485: Medical devices - quality management systems - requirements for
regulatory purposes. ISO. Available at: https://www.iso.org/standard/59752.html. (Accessed: 14
July 2024).
Jamot, M. and Pettersson, M. (2016) Agile challenges within regulated healthcare environments.
M.S. Thesis in Software Engineering, Chalmers University of Technology and University of
Gothenburg.
Jetley, R., Iyer, S. P. and Jones, P. (2006) A formal methods approach to medical device review.
Computer, 39(4), 61-67.
Kalankesh, L. R., Nasiry, Z., Fein, R. A. and Damanabi, S. (2020) Factors influencing user
satisfaction with information systems: A systematic review. Galen Medical Journal, 9(e1686), 1-
9.
Keathley, J., Merrill, P., Owens, T., Meggarrey, I., and Posey, K. (2013) Leading innovation.
The Journal for Quality and Participation, 36(3), 23-28.
Kelley, K., Clark, B., Brown, V. and Sitzia, J. (2003) Good practice in the conduct and reporting
of survey research, International Journal for Quality in Health Care, 15(3), 261-266.
Kim, S. Y., Moon, J. Y., Shin, J., Sim, J. Y., Kim, M. and Jang, J. (2022) Survey for government
policies regarding strategies for the commercialization and globalization of digital therapeutics.
Yonsei Medical Journal, 63, Suppl:S56-62.
Lievevrouw, E., Marelli, L. and Van Hoyweghen, I. (2022a). The role of US policymaking in the
emergence of a digital health assemblage. Science as Culture, 31(1), 72-91.
Lievevrouw, E., Marelli, L. and Van Hoyweghen, I. (2022b) The FDA’s standard-making
process for medical digital health technologies: Co-producing technological and organizational
innovation. BioSocieties, 17(3), 549-576.
Lim, D. (2018) Industry blasts FDA clinical decision software draft, Healthcare Dive, 07
February. Available at: https://www.healthcaredive.com/news/industry-blasts-fda-clinicaldecision-software-draft/516523/ (Accessed: 09 June 2024).
Lima, E. E. F. (2020) Agile development model for certifiable medical device software, M.S.
Thesis in Software Engineering, Universidade Do Porto, Portugal.

135
Marshall, C. and Rossman, G. B. (2014) Designing qualitative research. 5th edn. London,
England: SAGE Publications Ltd.
Martínez-Mesa, J., González-Chica, D. A., Duquia, R. P., Bonamigo, R. R., and Bastos, J. L.
(2016) Sampling: How to select participants in my research study? Anais Brasileiros de
Dermatologia, 91(3), 326-330.
McHugh, M., Cawley, O., McCaffcry, F., Richardson, I., and Wang, X. (2013) An agile V-model
for medical device software development to overcome the challenges with plan-driven software
development lifecycles. In: 2013 5th International Workshop on Software Engineering in Health
Care (SEHC), IEEE, 12-19.
McHugh, M., McCaffery, F. and Casey, V. (2014) Adopting agile practices when developing
software for use in the medical domain. Journal of Software: Evolution and Process, 26(5), 504-
512.
Morrison, T. M., Pathmanathan, P., Adwan, M. and Margerrison, E. (2018) Advancing
regulatory science with computational modeling for medical devices at the FDA's office of
science and engineering laboratories, Frontiers in Medicine, 5(241), 1-11.
Nair, C. S., Adams, P. and Mertova, P. (2008) Student engagement: The key to improving survey
response rates. Quality in Higher Education, 14(3), 225-232.
Nulty, D. D. (2008) The adequacy of response rates to online and paper surveys: What can be
done? Assessment & Evaluation in Higher Education, 33(3), 301-314.
O'Brien, G. L., Sinnott, S.-J., Walshe, V., Mulcahy, M., and Byrn, S. (2020) Health policy
triangle framework: narrative review of the recent literature. Health Policy Open, 1, 100016.
doi.org/10.1016/j.hpopen.2020.100016.
Ouimet, J. A., Bunnage, J. C., Carini, R. M., Kuh, G. D., and Kennedy, J. (2004) Using focus
groups, expert advice, and cognitive interviews to establish the validity of a college student
survey. Research in Higher Education, 45, 233-250.
Özcan‐Top, Ö. and McCaffery, F. (2018) A hybrid assessment approach for medical device
software development companies. Journal of Software: Evolution and Process, 30(7-e1929), 1-
15.
Pakarinen, A. and Salanterä, S. (2020) The use of gaming in healthcare, in Charalambous, A.
(ed.) Developing and Utilizing Digital Technology in Healthcare for Assessment and
Monitoring. Cham, Switzerland: Springer Nature, 115-125.
Pilot, L. R. and Waldmann, D. R. (1998) Food and Drug Administration Modernization Act of
1997: Medical device provisions. Food and Drug Law Journal, 53(2), 267-295.
Price, J. H. and Murnan, J. (2004) Research limitations and the necessity of reporting them.
American Journal of Health Education, 35(2), 66-67.

136
Rajagopalan, S. (2014) Review of the myths on original software development model. In:
International Journal of Software Engineering & Applications, 5(6), 103-111.
Rasmussen, R., Hughes, T., Jenks, J. and Skach, J. (2009) Adopting agile in an FDA regulated
environment. 2009 Agile Conference, Chicago, IL, USA, 151-155.
Regulation (EU) 2016/679. (2016) Regulation (EU) 2016/679 of the European Parliament and of
the Council of 27 April 2016 on the protection of natural persons with regard to the processing
of personal data and on the free movement of such data, and repealing directive 95/46/EC
(General data protection regulation). Official Journal of European Union. 119, 4 May.
Available at: https://eur-lex.europa.eu/eli/reg/2016/679/oj
Ross, P. T. and Bibler Zaidi, N. L. (2019) Limited by our limitations. Perspectives on Medical
Education, 8(4), 261-264.
Royce, W. W. (1970) Conference Proceedings. Managing the development of large software
systems. Proceedings of IEEE. Wescon, Los Angeles, 328-388.
Ruparelia, N. B. (2010) Software development lifecycle models. ACM SIGSOFT Software
Engineering Notes, 35(3), 8-13.
Saleh, A. and Bista, K. (2017) Examining factors impacting online survey response rates in
educational research: perceptions of graduate students, Journal of MultiDisciplinary Evaluation,
13(29), 63-74.
Samra, T. S. (2012) Software risk management: An exploration of software life cycle
methodologies, best practices and tools for their application to medical device software risk
management. Doctorate of Regulatory Science, University of Southern California, Los Angeles,
CA, USA. Available at: https://doi.org/10.25549/usctheses-oUC11291987. (Accessed: 14 July
2024).
Schwartz, S., Ross, A., Carmody, S., Chase, P., Coley, S. C., Connolly, J., Petrozzino, C, and
Zuk, M. (2018). The evolving state of medical device cybersecurity. Biomedical Instrumentation
& Technology, 52(2), 103-111.
Seale, C., Gabo, G, Silverman, D., and Gubrium, J. F. (2003) Qualitative research practice,
London, England: SAGE Publications Ltd.
Sheehan, K. B. (2001) E-mail survey response rates: A review. Journal of Computer-Mediated
Communication, 6(2).
Shuren, J., Patel, B. and Gottlieb, S. (2018) FDA regulation of mobile medical apps. JAMA,
320(4), 337-338.
Singer, E. (1993) Informed consent and survey response: A summary of the empirical literature.
Journal of Official Statistics-Stockhom, 9(2), 361-361.

137
Sivo, S. A., Saunders, C., Chang, Q. and Jiang, J. J. (2006) How low should you go? Low
response rates and the validity of inference in IS questionnaire research. Journal of the
Association for Information Systems, 7(6), 17.
Soo, C.-W. (2014) Convergence of United States Regulation and reimbursement policies
impacting patient access to humanitarian use devices (HUD). Doctorate of Regulatory Science,
University of Southern California, Los Angeles. Available at: https://doi.org/10.25549/uscthesesc3-363055. (Accessed: 14 July 2024).
Sunshine, P. H. (1995) The preemptive scope of the medical device amendments of 1976. Food
& Drug Law Journal, 50(1), 191-212.
Walt, G. and Gilson, L. (1994) Reforming the health sector in developing countries: The central
role of policy analysis. Health Policy and Planning, 9(4), 353-370.
Weller, S. C., Vickers, B., Bernard, H. R., Blackburn, A. M., Borgatti, S., Gravlee, C. C., and
Johnson, J. C. (2018). Open-ended interview questions and saturation. PloS One, 13(6),
e0198606.
Zema, M., Rosati, S., Gioia, V., Knaflitz, M., and Balestra, G. (2015) Developing medical device
software in compliance with regulations. 37th Annual International Conference of the IEEE
Engineering in Medicine and Biology Society (EMBC), Milan, Italy, IEEE, 1331-1334.

138
Appendices

139
Appendix A. Survey Questions
Table 29: Survey Flow
Block Title
1 Demographics
2 Content
3 Context
4 Process
5 Closing
Start of Block: Demographics
Q1.1 I am asking for your feedback as an industry expert on design control for software. The
survey is being conducted as part of my research in the Doctoral program at the University of
Southern California in Regulatory Science, where I am researching software as a medical device.
All responses to this survey will be held confidentially unless otherwise specified.
If you work as a consultant or have worked in multiple companies, please use your general
experience in the industry when answering the following questions.
This survey should take about 20 minutes to complete. You can stop and return to the link at any
time. Thank you for taking the time to provide your expertise and views on this topic, please
provide thoughtful responses throughout the survey.
Q1.2 What is your primary function in your current organization? (Choose the best answer.)
o Quality / Regulatory Affairs (1)
oProduct Development/ Engineering (4)
oSales / Marketing / Business / Legal (6)
o Other: (8) __________________________________________________
Skip To: End of Survey If What is your primary function in your current organization? (Choose the best answer.) =
Sales / Marketing / Business / Legal

140
Q1.3 Do you have experience developing software, or supporting a team that develops, any of
the following? (Select all that apply.)
▢ Medical Devices or accessories (1)
▢ Consumer Software (not a medical device) (2)
▢ None of these apply. (3)
Skip To: End of Survey If Do you have experience developing software, or supporting a team that develops, any of
the follow... = None of these apply.
Display This Question:
If Do you have experience developing software, or supporting a team that develops, any of the follow... =
Medical Devices or accessories
Q1.4 Do you have experience developing any of the following products? (Select all that apply.)
▢ Software as a medical device (SaMD) (1)
▢ Software in a medical device or IVD (2)
▢ Software that is an accessory to a pharmaceutical product (3)
▢ Lifestyle / low risk health software (5)
Display This Question:
If Do you have experience developing software, or supporting a team that develops, any of the follow... =
Consumer Software (not a medical device)
Or Do you have experience developing software, or supporting a team that develops, any of the follow... =

141
Q1.5 Select the type of non-medical consumer products you have experience with. (Select all
that apply.)
▢ Media / Gaming (1)
▢ Information Sharing / Education (2)
▢ Other Regulated Software (e.g. transportation, financial, banking) (3)
▢ Security (4)
▢ Creative Tool (e.g photo editing) (5)
▢ Productivity / Planning (6)
▢ Lifestyle / Health / Fitness (optional: provide details) (7)
__________________________________________________
▢ Database / data transfer (8)
▢ Other: (optional) (9)
__________________________________________________
Q1.6 Please select the best answer for your current position.
o My company is focused on developing software in or as a medical device. (1)
o My company is focused on developing software in or as a consumer product (non
medical devices). (2)
o My company is focused on developing software as an accessory for drug/ drug products.
(3)
o My company is focused on developing both software with a medical purpose and
consumer software. (4)

142
Q1.7 Please select the software development lifecycle models (SDLC) with which you have
experience. (Select all that apply.)
▢ Waterfall / Cascade (1)
▢ Vee "V" Method (2)
▢ Spiral (3)
▢ Agile (4)
▢ Other (5) __________________________________________________
Display This Question:
If Please select the software development lifecycle models (SDLC) with which you have experience. (S... = Agile
Q1.8 With which type(s) of agile method have you worked (select all that apply)?
▢ Scrum (1)
▢ Test Case (2)
▢ XP (3)
▢ User Story (4)
▢ Automated Testing (5)
▢ Test Driven Development (6)
▢ Pair Programming (7)
▢ Other: (8) __________________________________________________

143
Q1.9 Do you or your company develop software internally?
o We develop software internally. (1)
o We do not develop software internally; we work with an external organization. (2)
o We both develop software internally and externally. (3)
Display This Question:
If Do you have experience developing software, or supporting a team that develops, any of the follow... =
Consumer Software (not a medical device)
Q1.10 Are you familiar with design controls for medical devices?
o Yes (1)
o No (2)
Skip To: End of Survey If Are you familiar with design controls for medical devices? = No
End of Block: Demographics
Start of Block: Content
Display This Question:
If Do you have experience developing software, or supporting a team that develops, any of the follow... =
Consumer Software (not a medical device)
Q2.1 What factors were critical in deciding to incorporate or exclude design control in your
process for consumer software or lifestyle software? Move the top 2 reasons into the box.
Top factors
______ Regulatory requirements (1)
______ Expertise in design control (2)
______ Cost (3)
______ Common development process for all software (4)
______ Future goal of medical application (5)
______ Other (Comment) (6)

144
Q2.2 How easy is it to implement the FDA requirements for design control for software into the
development process?
o Extremely difficult (1)
oSomewhat difficult (2)
o Neither easy nor difficult (3)
oSomewhat easy (4)
o Extremely easy (5)
Q2.3 Optional: Please add any comments to explain why you chose your answer.
________________________________________________________________
Q2.4 Please consider how you determined the classification for your software. Select all items
that you used in determining the software classification.
▢ External consultants (1)
▢ Communication with FDA (2)
▢ Internal discussions (3)
▢ Regulatory guidance documents (4)
▢ Other (5) __________________________________________________
▢ I was not involved / No comment (6)
Skip To: Q2.6 If Please consider how you determined the classification for your software. Select all items that y... = I
was not involved / No comment

145
Carry Forward Selected Choices from "Please consider how you determined the classification for your
software. Select all items that you used in determining the software classification."
Q2.5 For the categories that you used, which did you find most useful? (Please select all that
apply)
▢ External consultants (1)
▢ Communication with FDA (2)
▢ Internal discussions (3)
▢ Regulatory guidance documents (4)
▢ Other (5) __________________________________________________
▢ I was not involved / No comment (6)

146
Q2.6 To gain insight on design control for software, please indicate if you agree or disagree with
each of the statements.
Strongly
disagree
(1)
Somewhat
disagree
(2)
Neither
agree nor
disagree
(3)
Somewhat
agree (4)
Strongly
agree (5)
Cannot
Respond
(6)
The FDA
requirements
for design
control aligns
well with
industry
standards for
software
development.
(1)
o o o o o o
The elements
to be
included in a
SaMD's
design
history file
are clear. (2)
o o o o o o
Display This Question:
If Please select the software development lifecycle models (SDLC) with which you have experience. (S... = Agile

147
Q2.7 Please indicate if you agree or disagree with the following statement.
Strongly
disagree
(1)
Somewhat
disagree
(2)
Neither
agree nor
disagree
(3)
Somewhat
agree (4)
Strongly
agree (5)
Cannot
Respond
(6)
The FDA's
acceptance
of Agile
software
development
is well
defined. (1)
o o o o o o
Q2.8 Choose the answers that best describe how you feel.
I find the existing FDA software guidances are...
▢ Comprehensive (1)
▢ Confusing (2)
▢ Easy to apply (3)
▢ Too high level (4)
▢ Too general (5)

148
Q2.9 Please consider the guidance documents below and indicate if you find them useful.
I use this document
(1)
I am familiar with
this document but do
not find it useful. (2)
I am not familiar /
Uncertain (3)
General Wellness:
Policy for Low Risk
Devices (1) o o o
Policy for device
software function
and mobile medical
application guidance
(2)
o o o
Medical device data
systems, medical
image storage
devices, and medical
image
communication
devices (3)
o o o
Clinical decision
software guidance (4) o o o
Your Clinical Decision
support software: Is
it a medical device
FDA graphic / Policy
Navigator (5)
o o o

149
Display This Question:
If Please consider the guidance documents below and indicate if you find them useful. = I use this document
Q2.10 Do any of the documents listed require modifications to make them more useful?
• General Wellness: Policy for Low Risk Devices
• Policy for device software function and mobile medical application guidance
• Medical device data systems, medical image storage devices, and medical image
communication devices.
• Clinical decision software guidance
• Your Clinical decision support software: Is it a medical device FDA graphic / Policy Navigator
o Yes (1)
o No (2)
Display This Question:
If Do any of the documents listed require modifications to make them more useful? • General Wellness... = Yes
Q2.11 Please elaborate on which guidance documents require updates and why.
________________________________________________________________

150
Q2.12 Are you familiar with the guidance documents below?
I use this document.
(1)
I am familiar with
this document but do
not find it useful. (2)
I am not familiar /
Uncertain (3)
Changes to Existing
Medical Software
Policies Resulting
from Section 3060 of
the 21st Century
Cures Act. 2017 (1)
o o o
Content of premarket
submission for device
software functions.
Draft guidance for
industry and Food
and Drug
Administration. (4)
o o o
General wellness:
policy for low risk
devices. Guidance for
industry and Food
and Drug
Administration. (2)
o o o
Cybersecurity in
medical devices:
Quality system
considerations and
content of premarket
submissions draft
guidance for industry
and Food and Drug
Administration staff.
(5)
o o o
Display This Question:
If Are you familiar with the guidance documents below? = I use this document.
Q2.13 Do any of the documents previously listed require modifications to make them more
useful?

151
• Changes to Existing Medical Software Policies Resulting from Section 3060 of the 21st
Century Cures Act. 2017
• Content of premarket submission for device software functions. Draft guidance for industry and
Food and Drug Administration.
• General wellness: policy for low risk devices. Guidance for industry and Food and Drug
Administration.
• Cybersecurity in medical devices: Quality system considerations and content of premarket
submissions draft guidance for industry and Food and Drug Administration staff.
o Yes (1)
o No (2)
Display This Question:
If Do any of the documents previously listed require modifications to make them more useful? • Chang... = Yes
Q2.14 Please elaborate on which guidance documents require updates and why.
________________________________________________________________
End of Block: Content
Start of Block: Context
Q3.1 Rank the barriers stopping companies that develop consumer software products from
developing medical device software from the most impactful to the least impactful.
______ Business / Portfolio mismatch. (1)
______ Regulatory burden to develop. (2)
______ Finding qualified people that understand the regulatory requirements. (3)
______ Additional cost to develop regulated software. (4)
______ Confusion on requirements for device software. (5)
Q3.2 Optional: Are there other significant reasons not listed above?
________________________________________________________________

152
Q3.3 Have you had formal or informal discussions with the FDA regarding software?
o Yes (1)
o No (2)
Skip To: Q3.11 If Have you had formal or informal discussions with the FDA regarding software? = No
Q3.4 In what was context did the discussion occur? (Select all that apply.)
▢ Committee / Working Group (1)
▢ Drafting / Commenting on guidance document. (2)
▢ Pre-submission / Q-submission (3)
▢ Submission (Premarket Notification or Approval) (4)
▢ Audit (5)
▢ Other: (6) __________________________________________________

153
Q3.5 Choose the option that best reflects your feelings about the following statements.
Agree (1) Disagree (2) Neither Agree or
Disagree (3)
The FDA was helpful
in determining the
risk classification. (1) o o o
The FDA did not have
experience with my
type of device. (2) o o o
The FDA requested
additional software
testing. (3) o o o
The FDA requested
more software
documentation than
we had anticipated.
(4)
o o o
The FDA requested
additional clinical use
information. (5) o o o
The FDA required
additional safety
controls to be built
into the software. (6)
o o o
Q3.6 Optional: Please add any comments or additional details in regard to your experience with
the FDA.
________________________________________________________________

154
Q3.7 How knowledgeable did you find the FDA staff when you discussed software with them?
o Not knowledgeable at all (1)
oSlightly knowledgeable (2)
o Moderately knowledgeable (3)
o Very knowledgeable (4)
o No opinion / I have not met with them. (5)
Q3.8 Optional: Please explain further.
________________________________________________________________

155
Q3.9 When considering the feedback, how did you and/or the team feel about the FDA
feedback?
o We were satisfied with the feedback. (1)
o We were not satisfied with the feedback. (2)
o Other: (3) __________________________________________________
Q3.10 Optional: If any of the feedback pertained to software development or documentation
please explain further.
________________________________________________________________
Q3.11 How many years has it been since your last submission with the FDA that included
software?
o Does not apply / No submission experience. (1)
o Less than ( (2)
o2-5 years (3)
o Greater than (>) 5 years (4)
Skip To: End of Block If How many years has it been since your last submission with the FDA that included software?
= Does not apply / No submission experience.
Q3.12 What type of device did your most recent submission address?
oSoftware As a Medical Device (SaMD) (1)
oSoftware In a Medical Device (SiMD) (2)
o Other (Please Explain): (3)
__________________________________________________

156
Q3.13 For your most recent submission, please indicate the quality of the feedback from the
FDA related to software.
o The comments and questions from the FDA indicated that they did not understand. (1)
o The comments and questions from the FDA were appropriate. (2)
o The FDA had good comments and questions. (3)
o Other, please explain: (4) __________________________________________________
Display This Question:
If Please select the software development lifecycle models (SDLC) with which you have experience. (S... = Agile
Q3.14 With regards to recent submissions with the FDA did you submit documentation related to
Agile software development?
o Yes, documentation related to Agile software development was submitted. (1)
o No, documentation related to Agile software development was not submitted. (2)
o Does not apply / No experience. (3)
Display This Question:
If With regards to recent submissions with the FDA did you submit documentation related to Agile sof... = No,
documentation related to Agile software development was not submitted.
Q3.15 Why did you not submit documentation for Agile software development? (Select all that
apply.)
o This level of documentation was not necessary for the risk classification of the device.
(1)
oSoftware development documentation focused on the waterfall/cascade method. (2)
o Agile development was not used. (3)
o Other/Comment: (4) __________________________________________________
End of Block: Context
Start of Block: Process

157
Q4.1 Please provide your opinion on the statements below.
Strongly
disagree
(1)
Somewhat
disagree
(2)
Neither
agree nor
disagree
(3)
Somewhat
agree (4)
Strongly
agree (5)
No opinion
/ Does not
apply (6)
Design
control is
sufficient
for
Software
as a
Medical
Device
(SaMD). (1)
o o o o o o
Design
control will
continue
to be
sufficient
for
software in
the future.
(2)
o o o o o o
Q4.2 Optional: Please expand on the reasons for your answers above.
________________________________________________________________

158
Display This Question:
If Please select the software development lifecycle models (SDLC) with which you have experience. (S... = Agile
Q4.3 Choose the option that best reflects your feelings about the following statement.
Strongly
disagree
(1)
Somewhat
disagree
(2)
Neither
agree nor
disagree
(3)
Somewhat
agree (4)
Strongly
agree (5)
No
opinion /
Does not
apply (6)
Agile
software
development
can be used
in a way that
meets the
design
control
requirements.
(1)
o o o o o o
Q4.4 Optional: Explain why or why not.
________________________________________________________________

159
Q4.5 Do you, or your organization, participate in the development of software guidance and
regulation outside your organization (for example with a trade group, standard committee, or
regulatory guidance development)?
o Yes (2)
oIn the past, but not currently. (3)
o No (4)
Display This Question:
If Do you, or your organization, participate in the development of software guidance and regulation... = Yes
Or Do you, or your organization, participate in the development of software guidance and regulation... = In the
past, but not currently.
Q4.6 With what group do you, or your organization, participate? (Select all that apply.)
▢ AdvaMed (1)
▢ MDMA (2)
▢ ISO / IEC Working Group (3)
▢ AAMI (4)
▢ Other Trade Group: (5)
__________________________________________________
▢ Direct feedback to FDA from company. (6)
▢ FDA Pilot Program (7)
▢ Other: (8) __________________________________________________

160
Q4.7 Do you believe that trade organizations can influence policy for software as a medical
device (SaMD)?
o Yes (1)
o No (2)
Q4.8 Current technical committees understand the needs of companies developing software as a
medical device (SaMD).
o Agree (1)
o Disagree (6)
Q4.9 Rank order these current challenges in developing regulated software. (Item one is the most
significantly challenging.)
______ Content in current guidance. (1)
______ Pace of regulatory development / change. (2)
______ Design control framework is not consistent with current software development
techniques. (3)
______ Access to FDA to clarify questions. (4)
______ Technical challenges. (5)
______ Cybersecurity. (6)
______ Finding qualified individuals to meet the regulatory burden. (7)
Q4.10 Do you have additional comments or examples of the challenges you face?
________________________________________________________________
________________________________________________________________
________________________________________________________________
________________________________________________________________
________________________________________________________________
End of Block: Process
Start of Block: Closing

161
Q5.1
Would you consider forwarding this survey?
To improve the value of this survey please forward the link to individuals involved in developing
software for medical devices and/or low-risk lifestyle applications. Specifically, consider
forwarding this survey to people in quality, regulatory, engineering, or development teams.
You will receive an email message following the completion of this survey.
Please forward the message, post it on social media, or pass the link to others willing to give
their feedback.
Q5.2 If you would be interested in receiving the results of this research, when complete, please
provide an email contact.
Please note that emails entered here will be kept separate from the responses in this survey.
________________________________________________________________
End of Block: Closing

162
Appendix B. Survey Full Text Comments
Note: Comments are presented in the format they were received, without grammatical or syntax
corrections.
B.1 Content
Table 30: Additional Comments - Ease implementing Design Control
Q2.2 - How easy is it to implement the FDA requirements for design control for software into the
development process? Q2.3 - Optional: Please add any comments to explain why you chose
your answer.
Additional Comments
It depends on the Exec Mgmt understanding of their products and regulatory
obligations. For standard med device orgs, it's relatively easy to implement. However,
for less traditional med orgs like CLIA Labs, Research Use Only, or LDT, it becomes
more challenging since they are in the "grey" area of FDA discretionary oversight. I'd
argue that design controls maybe viewed as industry best practice, regardless of
regulations compliance as it introduces a comprehensive, and riskbased approach to
complex product development
I don't have direct experience implementing the FDA's requirements so I'm neutral on
the topic.
If the company does not spend the time to ensure the process is developed correctly
and engrained into the development and deployment process, it will be tough. That has
been the approach I have seen. But it doesn't have to be.
Most of the FDA (and IEC 62304) requirements are basics that are learned in first
semester of a software engineering bachelor. And than later get ignored as it's "not
cool". But if you work with best practices for software engineering, the gap to FDA
and IEC 62304 requirements are really small.
pending on the intended use, some of the requirements could be overkilled.
The guidance the FDA has provided doesn't uniformly align with IEC 62304 or IEC
82304, which are both standards that are being commonly used.
It is not difficult, but time consuming since a lot of documentation is needed. Also,
change control is strict.
Primarily, IEC 62304 and its tech documents provide the framework for med device
software development. However, the correllation to the CFR and ISO 13485 is not 1:1.
How post-market surveillance is conducted is dependent on organization's maturity in

163
using risk management file fir disposition. Use of RMF is SaMD is an emerging body
of knowledge that is not readily understood by senior leadership.
In my opinion, the FDA requirements are a somewhat difficult to apply because we
have to think of the entire system versus just control of SW. The overall product
impact due to SW change is a critical part of design control and it gets cumbersome
when you have multiple SW releases in DVT phase.
it's fumdamental and uncomplicated, but we have occasionally hired people who
believed sw should be exempt from control, and had to do remedial training
Many guidance documents for software in or as medical devices and general software
so product development team may be overwhelmed or confused by these requirements
FDA's requirements ar quite mature now. The V-model is almost de rigeur for active
devices (60601) and software, I think, with waterfall more common for non-active
devices (i.e. those without a source of electrical power). Also IEC 62304 lays out a
well established process which FDA recognises.
Although we have many of the requirements to meet ISPE GAMP 5 expectations, we
do not have all the required deliverables for the FDAs design control requirements.
FDA Design Controls requirements are not aligned with SDLC best practices (e.g.
agile/scrum method, SDLC tools like Atlassian) in the software industry; Although
FDA recognizes IEC 62304 and AAMI TIR 45 etc to fill some of that gap but still
software industry best practices (non-medical) are something else altogether. FDA did
also attempt to bridge the gap thru their Digital Health Software Pre-Cert Pilot
Program (2017-18) but it didn't end of anything meaningful in terms of a clear
regulatory pathway. Hence such disconnect and challenges remain.
It will highly depend on the organizational structure, procedures, and how well the
development team are at capturing the voice of customer (VOC) early and updating it
often, with RA and QA close. The VOC is critical for the development of the intended
use and indications for use statements and will dovetail into the quality and regulatory
requirements that should also be included within the overall design requirements suite.
I’m my experience, these are often an afterthought and cause rework and delays either
due to testing or poor regulatory strategy because the intent of the product (including
how they wish to promote the product) drifts from the original plan.
FDA design control requirements are so basic it's not that hard. I wonder if you ask
about 62304 on top of that, later. This would change my answer to Somewhat difficult
The FDA requirements specified in 21CFR820 are too generic and do not provide the
insight needed to meet the current standards for regulatory submissions. More helpful
is IEC 62304, though the implementation of this SDLC standard needs some
additional insight to satisfy the needs of the FDA review team

164
Table 31: Additional Comments - Determining Risk Classification
Q2.4 - Please consider how you determined the classification for your software. Select all items
that you used in determining the software classification. Other:
Additional Comments – Other
ROW Authorities, FDA, and Notified Bodies
Voice of Customer to develop the correct intended use, etc.
Your query does not specify if this is software regulatory classification or software
safety classification. For safety classification, used international standard 62304 in
view of 14971 to determine. For EU software regulatory classification, this was done
via a 2.5+ year long adjudication between our Notified Body as well as the Competent
Authority of our Authorized Representative (on account of our being a US Based
manufacturer of devices, per EU MDR) - classification (between I, IIa, IIb, III) has not
been reached since initiation in 2021, may be escalated to EU Council for eventual
adjudication per our NB.
Search of existing product codes
IEC 62304
IEC 62304
Research into similar products globally and understanding risk profile based off
IMDRF guidance documents
Table 32: Additional Comments - Useful in determining Software Classification
Q2.5 - For the categories that you used, which did you find most useful? Other:
Additional Comments – Other
Early engagement with the regulators
VOC
IEC 62304
IEC 62304

165
Table 33: Additional Comments - Updates to Guidance Documents Group 1
Q2.10 - Do any of the documents listed require modifications to make them more useful? •
General Wellness: Policy for Low Risk Devices
• Policy for device software function and mobile medical application guidance
• Medical device data systems, medical image storage devices, and medical image
communication devices.
• Clinical decision software guidance
• Your Clinical decision support software: Is it a medical device FDA graphic / Policy Navigator
Q2.11 - Please elaborate on which guidance documents require updates and why.
Additional Comments
For example, a Class II medical device requiring 510(k) may have an app, and the app
could include a combination of software components that are Class I, Class II 510(k)
exempt, and/or Class II requiring 510(k). It would be nice if guidance documents like
"Policy for device software function and mobile medical application guidance" could
offer discussions or examples for similar scenarios.
Regarding all of the guidance documents, there really needs to be more strenuous
alignment with the IEC standards. The IEC standards have been what organizations
have been using in SaMD development as a baseline. However the guidances provided
by the FDA reflect the agency's current thinking, but are NOT directly correlated to the
QSR and only cursorily align to IEC. An effort similar to the QSR alignment to ISO
13485 would be beneficial with regards to IEC 62304/82304.
The Clinical Decision Software guidance could probably be more clear. It's OK as is
but could be better.
I think th policies for all could be improved.
As expected, most FDA guidance tend to be high level and less prescriptive by design
in order to accommodate a wide variety of applications, intended uses, etc. Therefore,
it would helpful to have a little more prescriptive guidance, language, examples,
similar to those in industry standard (ex ISO 14971, IEC 62304, etc)
Policy of device function and mobile and data storage
More Details on the documentation requirements based on the risk of the SaMD or
wellness device
The Agency guidance, like regulation, is frequently general. This results in various
interpretations. The Agency could benefit from providing clearer definition in guidance
documents to standardize the product development requirements. There were several
situations where the Agency provided guidance through interaction that was much

166
clearer regarding expectation. Unfortunately, guidance solicited for subsequent devices
was inconsistent. A standardized approach could alleviate this issue.
Clinical decision software guidance • Your Clinical decision support software: I
Clinical Decision Software Guidance could use additional examples on how software
can be used in the clinical space with regards to sample management.
These guidance are high-level and serve their intended purpose fine; the underlying
SDLC methodology and deliverables can be more specific (but that applies to other
FDA guidances and recognized standards more, and to 21 CFR 820.30 itself).
All of them. They're too high level, very device focused in their language whereas most
developers are completely new to this area. It confuses common terms in software
development with regulatory terms (same word means different things in each field).
A large concern is the joint oversight of CDS from the FDA and the ONC.
All mentioned. More details or easy to follow flowcharts would be helpful. The
guidances are very high level and not clear in requirements.
ALL need more specific examples now that FDA has more experience regulating
different software in different device applications. The general wellness policy needs to
shift to a more patient-focused claims approach. Its not clear from guidance but you
CAN have ONE product that straddles ALL three subtypes of claims described by
FDA but only IF you apply these to three different patient populations re: value
proposition a product can drive in each.
Table 34: Additional Comments - Updates to Guidance Documents - Group 2
Q2.13 - Do any of the documents previously listed require modifications to make them more
useful?
• Changes to Existing Medical Software Policies Resulting from Section 3060 of the 21st
Century Cures Act. 2017
• Content of premarket submission for device software functions. Draft guidance for industry and
Food and Drug Administration.
• General wellness: policy for low risk devices. Guidance for industry and Food and Drug
Administration.
• Cybersecurity in medical devices: Quality system considerations and content of premarket
submissions draft guidance for industry and Food and Drug Administration staff.
Q2.14 - Please elaborate on which guidance documents require updates and why.

167
Additional Comments
too general
The cybersecurity document should be more specific about what documents are
required for submission by submission type. I submitted the same documents
for an IDE and a 510(k) (prior to the eSTAR update for cybersecurity) and was
fine for IDE (probably because that is specifically called out in the guidance)
and got several questions with regards to my 510(k) it was clear that even a
reviewer with cyber expertise was confused by.
I'll focus on one: Cybersecurity. This is a new area where industry is struggling
on a few areas: What does it mean to be secure, and how to do it cost effective,
and efficiently? What are the regulation guidances that need to comply with?
What are the industry best practices, in handling the dynamic landscape of
SaMD + security. There are many open questions in this relatively new field
that is making it difficult for OEM to achieve. Hence, it would be helpful to
have more useful examples, frameworks, langugages from FDA to udnerstand
what is truly needed from a governance perspective, and let industry figure out
how to comply
Cyber premarket guidance
The document and processes are evolving. There has to be better documentation
around module and type definition around SaMD and Class type across
therapeutic area and device usage types.
Details on the documentation and risk management requirements specifically
for cybersecurity
Cybersecurity guidance can be more specific and aligned to standards such as
NIST/ISO 27001, although the recent Sept'23 version of this guidance is more
prescriptive than the previous one.
All but the premarket notification one. Same issue, very high level and too much
decvice regulatory language that is confusing to developers.
Better decision criteria transparency from FDA and clarity on changes to legacy
products.
Draft guidance documents are drafts for a reason and I doubt this will be
implemented in its current form

168
B.2 Context
Table 35: Additional Comments - Business Reasons not listed
Q3.2 Optional: Are there other significant reasons not listed?
Additional Comments
If you know med, you know what you need to do. That's commonly not the case with
in-experienced leaders/business folks (example: Theranos) trying for the first time to
develop a med device. There is an arrogance/disbelief that that regs (FDA, EU, CLIA)
don't apply to them, so that's a leadership issue.
Wanting to fast track or cut corners
N/A
Business risk/liability - beyond just GDPR, into healthcare. It's different from
consumer software, even from other regulated software like banking - code error can
kill someone
Table 36: Additional Comments - Method for interaction with FDA
Q3.4 In what was context did the discussion occur? (Select all that apply.)
Additional Comments
Inspection
General Agency inquiry
Collaborative community volunteer work
FDA's spectacular failure that was the software precert pilot program

169
Table 37: Additional Comments - Interactions with FDA
Q3.6 Optional: Please add any comments or additional details in regard to your experience with
the FDA.
Additional Comments
FDA staff were helpful. Pre-submission and interactions with FDA were always
very helpful.
N/A
The FDA required us to follow a guidance that specifically indicated our product
was out of scope. We had to follow it anyways which added time and cost.
I was a pilot in FDA Digital Health Software Pre-Cert Pilot Program (2016-18)
at Verily/Google Life Sciences.
FDA required additional security controls to be built into the software. Risk
Management to 14971 didn't tell us we needed a fingerprint lock on the app (app
to treat insomnia - incredibly lightweight borderline bullshit SaMD) to mitigate
any safety risk. But boy did FDA make a fuss about ensuring that feature was
implemented prior to launch as a security control, apparently because of
perceived stigma if someone else were to find out that the device user is being
treated for a mental health disorder because the app doesn't lock up.
Table 38: Additional Comments- FDA Knowledge
Q3.8 - Optional: Please explain further.
Additional Comments
It will be case by case and based on role. Many reviewers do not come from industry
and lack real experience but have academic experience. Some reviewers increase
expectation because a company that could afford more testing, etc., provided more than
required. This is a strategy to increase barriers to the competition. This is not a specific
SW issue, but highly tied to it.
Varied. The clinical/stats experts were NOT knowledgeable at all. The younger
reviewers fresh out of biomed masters programs who are Gen Z though, with less than
a year experience at FDA, they were moderately knowledgeable. Also, the rare chance
we got a software expert, they were also moderately knowledgeable
Actually it was variable - some very knowledgeable, others not. Hence my response -
moderately!

170
Table 39: Additional Comments – Team View FDA Feedback
Q3.9 - When considering the feedback, how did you and/or the team feel about the FDA
feedback?
Additional Comments
Mostly reasonable feedback
I've had the complete range, from very satisfied, especially in standards working groups to not
really satisfied when they changed their point of view between Q-Sub and 510(k)
case by case pending on reviewers
Depends on the reviewer 100%. The ones that understands software are few and far between
Case by case
Feedback changed in every Q-sub depending on who review team was assigned
Table 40: Additional Comments - Add question how the Team felt
Additional Comments
Once asked to submit in the 510(k) unit testing for Class B safety/ Class II regulatory
out of an abundance of caution in a Qsub. Burdensome, not reflective of a true
regulatory requirement, and went ignored in the 510(k)
B.4 Process
Table 41: Additional Comments - Statement Design Control Sufficient
Q4.1 Please provide your opinion on the statements below.
Design control is sufficient for Software as a Medical Device (SaMD).
Design control will continue to be sufficient for software in the future.
Q4.2 Optional: Please expand on the reasons for your answers above.
Additional Comments
FDA provides high level regs for complying to org's own processes, but not how to do
it. Therefore, industry tends to refer to industry standards like IEC 62304, ANSI SW 96
for more prescriptive methods in developing software.
The IEC 62304 guidance is also needed. Design Control only does not provide enough
detail
Our SDLC process was subordinate to design control and expanded far beyond the
traditional design control methodologies.

171
AI will shake things up. DRAFT regulations are coming, but in the EU confusing.
They are regulating IA as a technology rather than as a area focused issue (e.g. medical
devices) - thus requirements fwhich are important for AI for NLP or web-search, for
isntance, will be less important than those for medical devices (and vice-versa) and that
may get diluted as things currently stand.
Design Control is "whats", no "hows". Latter being where the gaps are in terms of
SDLC methodology, tools, activities and deliverables, to the software industry best
practices.
Table 42: Additional Comments - Design Control and Agile
Q4.3 Choose the option that best reflects your feelings about the following statement.
Agile software development can be used in a way that meets the design control requirements.
Q4.4 Optional: Explain why or why not.
Additional Comments
Achieved it before in Class II device clearances, so have working knowledge.
Agile development can be used for the implementation itself (coding) very well, where
for the phases before (specification) and after (V&V) it is more difficult. We therefor
use a mixed method usually.
The agile method seems well suited for development of RUO software. When the
software is a medical device, the agile method seems to lack, sufficient controls,
checks, and balances required to meet the rigor of traditional design control.
See previous answer
Although guidance is now out to support the use of Agile software, it is difficult to get
Agile development to create the right deliverables in a way that can be used in a
submission.
Table 43: Additional Comments - Challenges
Q4.10 Do you have additional comments or examples of the challenges you face?
Additional Comments
My main challenge is to make understand to management and software engineers, that
the burden is not so high as it looks like, if we would follow best practice from what
we've learned in university but do not like to do, as coding is considered more funny
than documentation.

172
Resources and time.
Cybersecurity and infrastructure are huge concerns as well as privacy and access to
data. "AI problem is a data problem." Also, harmonization is the biggest concern.
Harmonization between US Federal Departments AND global harmonization on
SaMD/ SiMD.
Pace of SOFTWARE development / change. Regulatory changes at a glacial pace. I
supported 250+ software product releases in under 3 years in my most recent role,
during that time there was only 1 requisite regulatory change (EU MDD to EU MDR)
that I had to manage impact. ALSO - pace of REGULATORY market authorization, as
a separate issue. If you release every week or every 2 weeks, but regulatory market
authorization for those releases can take 3-9 months for FDA? (or worst case for EU
MDR, 2.5+ years?) A design freeze for that length of time AND continuously ensuring
the product meets users needs - can be at great odds.
Having been through two recent successful submissions, the regulatory burden seems
to change and gets more intense with each review team and the unique requirements of
each reviewer.

173
Appendix C.Cross Tabulation Tables
These tables are discussed with the associated questions in Chapter 4.
Table 44: Cross Tabulation - FDA Acceptance of Agile
Q2.7: Please indicate if you agree or disagree with the following statement.
The FDA's acceptance of Agile software development is well defined.
Total QA/RA Eng Total Med
Dev
Cons Total Agile Other
Total Count 34 27 7 47 35 12 70 36 34
Strongly
disagree
1
3%
1
4%
0
0%
2
4%
1
3%
1
8%
2
3%
1
3%
1
3%
Somewhat
disagree
8
24%
6
22%
2
29%
11
23%
8
23%
3
25%
16
23%
8
22%
8
24%
Neither
agree or
disagree
6
18%
6
22%
0
0%
8
17%
6
17%
2
17%
12
17%
6
17%
6
18%
Somewhat
agree
13
38%
10
37%
3
43%
20
43%
14
40%
6
50%
26
37%
14
39%
12
35%
Strongly
agree
2
6%
2
7%
0
0%
3
6%
3
9%
0
0%
6
9%
3
8%
3
9%
Cannot
respond
4
12%
2
7%
2
29%
3
6%
3
9%
0
0%
8
11%
4
11%
4
12%
p-value 0.47
Table 45: Cross Tabulation - FDA Guidance Documents - Group 1
Q2.9 - Please consider the guidance documents below and indicate if you find them useful. I use
this document (U), I am familiar, but do not find it useful (FNU), I am not familiar (NF)
Total QA/
RA
Eng Total Med
Dev
Cons Total Agile Other
Total
Count
47 36 8 59 46 13 77 33 44
General Wellness: Policy for Low Risk Devices
U 13
29%
13
38%
0
0%
18
32%
13
30%
5
39%
23
31%
11
33%
12
29%
FNU 9
20%
7
21%
2
25%
12
21%
9
21%
3
23%
17
23%
8
24%
9
21%
NF 23
51%
14
41%
6
75%
27
47%
22
50%
5
39%
35
47%
14
42%
21
50%
p-val <0.1
Policy for device software function and mobile medical application guidance

174
U 25
54%
21
60%
3
38%
34
59%
25
56%
9
69%
43
57%
19
58%
24
56%
FNU 5
11%
3
9%
2
25%
7
12%
5
11%
2
15%
9
12%
5
15%
4
9%
NF 16
35%
11
31%
3
38%
17
29%
15
33%
2
15%
24
32%
9
27%
15
35%
p-val 0.4
Medical device data systems, medical image storage devices, and medical image
communication devices.
U 26
55%
23
64%
2
25%
34
58%
26
57%
8
62%
45
58%
20
61%
25
57%
FNU 5
11%
3
8%
2
25%
8
14%
5
11%
3
23%
9
12%
5
15%
4
9%
NF 16
34%
10
28%
3
38%
17
29%
15
33%
2
15%
23
30%
8
24%
15
34%
p-val 0.2
Clinical decision software guidance
U 24
51%
22
61%
1
13%
30
51%
24
52%
6
46%
42
55%
18
55%
24
55%
FNU 8
17%
6
17%
2
25%
11
19%
8
17%
3
23%
15
20%
7
21%
8
18%
NF 15
32%
8
22%
5
63%
18
31%
14
30%
4
31%
20
26%
8
24%
12
27%
p-val <0.1
Your Clinical decision support software: Is it a medical device FDA graphic / Policy
Navigator
U 22
48%
18
51%
2
25%
27
47%
21
47%
6
50%
36
48%
15
47%
21
49%
FNU 4
9%
2
6%
2
25%
5
9%
4
9%
1
8%
8
11%
4
13%
4
9%
NF 20
44%
15
43%
4
50%
25
44%
20
44%
5
42%
31
41%
13
41%
18
42%
p-val 0.4

175
Table 46: Cross Tabulation - Modification FDA Guidance - Group 1
Q2.10: Do any of the documents previously listed require modifications to make them more
useful?
Total QA/RA Eng Total Med
Dev
Cons Total Agile Other
Total Count 37 32 3 46 36 10 61 26 35
Yes 27
73%
24
75%
2
67%
34
74%
27
75%
7
70%
46
75%
21
81%
25
71%
No 10
27%
8
25%
1
33%
12
26%
9
25%
3
30%
15
25%
5
19%
10
29%
p-value 0.7
Table 47: Cross Tabulation - Modification FDA Guidance - Group 2
Q2.13: Do any of the documents previously listed require modifications to make them more
useful?
Total QA/RA Eng Total Med
Dev
Cons Total Agile Other
Total Count 30 27 6 44 34 10 56 23 33
Yes 15
46%
11
41%
4
67%
18
41%
15
44%
3
30%
26
46%
12
52%
14
42%
No 18
55%
16
59%
2
33%
26
60%
19
56%
7
70%
30
53%
11
48%
19
58%
p-value 0.2
Table 48: Cross Tabulation - Type of recent Submission
Q3.12: What type of device did your most recent submission address?
Total QA/RA Eng Total Med
Dev
Cons Total Agile Other
Total Count 25 21 4 34 26 8 45 20 25
SaMD 8
32%
7
33%
1
25%
13
38%
8
31%
5
63%
15
33%
8
40%
7
28%
SiMD 17
68%
14
67%
3
75%
21
62%
18
69%
3
38%
30
67%
12
60%
18
72%
Other 0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
p-value 0.7

176
Table 49: Cross Tabulation - Agreement Agile meets Design Control
Q4.3: Choose the option that best reflects your feelings about the following statement. Agile
software development can be used in a way that meets the design control requirements.
Total QA/RA Eng Total Med
Dev
Cons Total Agile Other
Total Count 23 18 5 34 25 9 49 25 24
Strongly
disagree
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
Somewhat
disagree
1
4%
0
0%
1
20%
1
3%
1
4%
0
0%
2
4%
1
4%
1
4%
Neither
agree nor
disagree
2
9%
2
11%
0
0%
3
9%
2
8%
1
11%
4
8%
2
8%
2
8%
Somewhat
agree
11
48%
8
44%
3
60%
17
50%
11
44%
2
22%
21
43%
11
44%
10
46%
Strongly
agree
9
39%
8
44%
1
20%
13
38%
11
44%
2
22%
22
45%
11
44%
11
46%
No opinion 0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
p-value 0.2

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Use of electronic health record data for generating clinical evidence: a summary of medical device industry views

PDF

Evaluation of FDA-sponsor formal meetings on the development of cell and gene therapies: a survey of industry views

PDF

Organizational communication of regulatory intelligence: a survey of the medical device industry

PDF

FDA influence on advisory committees through documentation: a content analysis and survey of industry views

PDF

Current practices in biocompatibility assessment of medical devices

PDF

Regulatory team development in post-merger integration: a survey of views from medical product companies

PDF

Examining the regulatory framework for drug compounding: industry views and experiences

PDF

The impact of FDA Patient-Focused Drug Development (PFDD) on US drug development strategies: a survey of views from pharmaceutical product companies

PDF

Regulation of pediatric cancer drug development: an industry perspective

PDF

Regulating cosmeceuticals in the United States: a cosmetic industry view

PDF

Using telemetry to ensure safe and reliable medical device operation: experience with defibrillators and infusion pumps

PDF

Use of natural colors: experience and views in pharmaceutical and dietary supplement industries

PDF

Implementation of unique device identification in the medical device industry: a survey of the change management experience

PDF

Regulatory harmonization in a resource-limited setting: the World Health Organization Collaborative Procedure for Accelerated Registration

PDF

Experience with breakthrough therapy designation: an industry survey

PDF

Regulatory agreements for drug development collaborations: practices in the medical products industry

PDF

Regulatory programs to foster medical product development: user experience in the United States and Japan

PDF

Regulatory dissonance in the global development of drug therapies: a case study of drug development in postmenopausal osteoporosis

PDF

Reprocessing of single-use medical devices: a survey investigation comparing the views of three unheard stakeholders

PDF

Promotion of regulated products using social media: an industry view

Asset Metadata

Creator Watson, Colleen (author)

Core Title Design control for software medical devices: an industry survey of views and experiences

School School of Pharmacy

Degree Doctor of Regulatory Science

Degree Program Regulatory Science

Degree Conferral Date 2024-08

Publication Date 09/09/2024

Defense Date 09/09/2024

Publisher Los Angeles, California (original), University of Southern California (original), University of Southern California. Libraries (digital)

Tag digital health technology,FDA,health policy triangle,medical device,OAI-PMH Harvest,regulatory affairs,SaMD,SDLC,SiMD,software

Format theses (aat)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Bain, Susan (committee chair), Loab, Gerald E. (committee member), Pacifici, Eunjoo (committee member), Richmond, Frances J. (committee member)

Creator Email cwokeeffe@gmail.com,watsonc@usc.edu

Unique identifier UC11399AJWU

Identifier etd-WatsonColl-13509.pdf (filename)

Legacy Identifier etd-WatsonColl-13509

Document Type Dissertation

Format theses (aat)

Rights Watson, Colleen

Internet Media Type application/pdf

Type texts

Source 20240910-usctheses-batch-1210 (batch), University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu