University of Southern California Dissertations and Theses

Incentivizing quality in the manufacture of pharmaceuticals: manufacturers' views on quality ratings

(USC Thesis Other)

Incentivizing quality in the manufacture of pharmaceuticals: manufacturers' views on quality ratings

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content

INCENTIVIZING QUALITY IN THE MANUFACTURE OF PHARMACEUTICALS:
MANUFACTURERS’ VIEWS ON QUALITY RATINGS

by

Cesar Medina

A Dissertation Presented to the
FACULTY OF THE USC SCHOOL OF PHARMACY
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF REGULATORY SCIENCE

August 2015

Copyright 2015 Cesar Medina

2
DEDICA TION
This is dedicated to my loving wife and my awesome son.
Thank you for your support and patience that made this all possible.

“If you’re not frightened that you might fail, you’ll never do the job.
If you’re frightened, you’ll work like crazy.”
— César Chavez

3
ACKNOWLEDGEMENTS
I give my sincerest appreciation to my dissertation advisor, mentor, and now
friend, Dr. Frances Richmond. I thank Dr. Richmond for her unwavering support,
patience, motivation, and instruction. I also thank her for expanding my boundaries and
encouraging me to take it to a new level. I honestly could not have had a better advisor
and this work could not have been possible without her.
I would also like to express my gratitude to the rest of my thesis committee,
Dr. Eunjoo Pacifici, Dr. Benson Kuo, Dr. Jerry Loeb, and Mr. Martin VanTrieste for their
guidance and insights that help strengthen this work. I also thank the dedicated staff from
the USC Regulatory Science department for their help throughout my studies.
I extend my thanks to Ms. Deborah Autor for bringing to my attention this
valuable and interesting industry topic and for her support of the study. I also thank my
colleagues and friends, especially Dr. Fernando Alvarez-Nunez, Dr. Sekhar Kanapuram,
Dr. Charles Yang, Mr. Dominick Daurio, Mr. Gary Incorvia, and Dr. Randy Steiner, that
gave me their full support and lifted my energy to complete this doctorate. I would like
to acknowledge the help of the Quality Leadership Network and the many professionals
who took the time to help develop, distribute, and complete this research survey.

4
T ABLE OF CONTENTS
DEDICATION .................................................................................................................... 2
ACKNOWLEDGEMENTS ................................................................................................ 3
LIST OF TABLES .............................................................................................................. 9
LIST OF FIGURES .......................................................................................................... 10
ABSTRACT ...................................................................................................................... 12
CHAPTER 1: OVERVIEW ....................................................................................... 13
1.1 Introduction ................................................................................................... 13
1.2 Statement of the Problem .............................................................................. 18
1.3 Purpose of the Study ..................................................................................... 20
1.4 Importance of the Study ................................................................................ 21
1.5 Limitations, Delimitations, Assumptions ...................................................... 22
1.6 Organization of Thesis .................................................................................. 24
1.7 Definitions ..................................................................................................... 24
CHAPTER 2: LITERATURE REVIEW ................................................................... 27
2.1 Evolution of Current Manufacturing Quality Oversight ............................... 27
2.1.1 Current Regulatory GMP Guidance and Enforcement ..................... 28
2.1.2 FDA’s Efforts to Modernize Quality Standards ............................... 33

5
2.2 Challenges to Quality Oversight by GMP Enforcement ............................... 36
2.2.1 Weaknesses of Agency Audits Alone to Detect
Quality Problems .............................................................................. 37
2.2.2 Increasing Globalization Straining Inspectional Oversight .............. 45
2.2.3 GMP Driven Quality and a Race to the Bottom
(Flawed Incentive Structure) ............................................................ 50
2.3 Current Incentive Framework toward Pharmaceutical Quality ..................... 51
2.3.1 Incentives for GMP Compliance ...................................................... 51
2.3.2 Incentives and Disincentives Linked to Production Efficiency ........ 52
2.3.3 Incentives and Disincentives Associated with Pricing ..................... 55
2.4 Other Mechanisms to Incentivize and Improve Quality ................................ 57
2.4.1 FDA Efforts to Improve Quality ....................................................... 58
2.4.2 Pharmaceutical Industry Views on Improving Quality..................... 60
2.4.3 Changes Advocated by Other Stakeholders ...................................... 63
2.4.4 The Call for a Pharmaceutical Quality Rating System ..................... 64
2.5 Case Studies of Regulated Health Industries with Rating Systems............... 65
2.5.1 Los Angeles County Restaurant Hygiene Ratings ............................ 66
2.5.2 Private and Public Health Plans Ratings ........................................... 67
2.5.3 U.S. News & World Report Hospitals Rankings .............................. 70
2.6 Policy Considerations for Pharmaceutical Quality Ratings .......................... 72
2.6.1 Metric Selection and Assessments .................................................... 72
6
2.6.2 Public Disclosure of Ratings ............................................................. 74
2.6.3 Policy Setting and Evaluative Frameworks ...................................... 75
CHAPTER 3: METHODOLOGY ............................................................................. 79
3.1 Introduction ................................................................................................... 79
3.2 Stakeholder Analysis and Respondent Selection .......................................... 79
3.3 Development of the Survey ........................................................................... 80
3.4 Respondent Descriptions ............................................................................... 84
3.4.1 Expert Panel Respondents ................................................................ 84
3.4.2 Open Access Respondents ................................................................ 85
3.5 Survey Deployment ....................................................................................... 86
3.5.1 Expert Panel Distribution .................................................................. 86
3.5.2 Open-Access Distribution to Professional Organizations ................. 87
3.6 Data Collection and Analysis ........................................................................ 88
CHAPTER 4: RESULTS ........................................................................................... 90
4.1 Survey Demographic Profiles and Response Rates....................................... 90
4.1.1 Expert Panel Survey.......................................................................... 90
4.1.2 Open-Access Survey ......................................................................... 91
4.2 Information Used to Benchmark Quality (Metrics) ...................................... 94
4.2.1 Publicly Available Information ........................................................ 94
4.2.2 Non-Public Proprietary Information ................................................. 98
7
4.3 Views on Quality Ratings ........................................................................... 102
4.3.1 Impact of Ratings on Company Quality Improvements
and Practices ................................................................................... 102
4.3.2 Selection and Presentation of the Quality Rating ........................... 104
4.3.3 Other Policy Views and Concerns Regarding Quality Ratings ...... 110
4.4 Cross-Tabulation Analyses of Demographic Variables .............................. 117
4.4.1 Company Size and Type ................................................................. 117
4.4.2 Regional Location ........................................................................... 118
4.4.3 Occupational Background ............................................................... 119
CHAPTER 5: DISCUSSION ................................................................................... 121
5.1 Considerations of Study Methodology ........................................................ 121
5.2 Considerations of the Results ...................................................................... 127
5.2.1 Manufacturers Already Use Many Metrics to Measure Quality ..... 127
5.2.2 Ratings May Increase Attention on Manufacturing Quality ........... 132
5.2.3 Manufacturers Have Doubts about Quality Ratings for
Public Use ...................................................................................... 134
5.3 Conclusions and Future Considerations ...................................................... 137
REFERENCES ............................................................................................................... 141
APPENDIX A ................................................................................................................. 168
APPENDIX B ................................................................................................................. 170
APPENDIX C ................................................................................................................. 178
8
APPENDIX D ................................................................................................................. 186
APPENDIX E ................................................................................................................. 195
APPENDIX F.................................................................................................................. 199
APPENDIX G ................................................................................................................. 200
9
LIST OF T ABLES
Table 1. Definitions of Key Terms and Acronyms ........................................................... 25
Table 2. US FDA Drug Shortages Root Cause Analysis for Drugs and Biologics .......... 42
Table 3. FDA’s 1998 Proposal of a Tiered Approach to Foreign Manufacturers
GMP Inspection Program ................................................................................... 48
Table 4. Construct of the Exploratory Survey Instrument used in this Study .................. 83
Table 5. Southern Californian Professional Organizations that Distributed
the Open-Access Survey .................................................................................... 86
Table 6. Most Recent Employer Type of Respondents .................................................... 92
Table 7. Primary Occupation of Respondents .................................................................. 94
Table 8. Public Sources other than FDA Used To Obtain Information
on Product and Company Quality ...................................................................... 97
Table 9. Views on Most Effective Way to Communicate a Product Quality Grade ...... 109
Table 10. Views on the Best Organization Suited to Evaluate Quality and
Issue Ratings ................................................................................................... 111
Table 11. Views on Mandated Public Disclosure of Quality Ratings ............................ 112
Table 12. Thematic Views and Concerns on Quality Ratings (Expert Panel) ................ 114
Table 13. Thematic Views and Concerns on Quality Ratings (Open-Access) ............... 116
Table 14. Cross Tabulation of Respondents Occupation and Mandatory Public
Disclosure of Quality Ratings (Expert Panel) ................................................. 120

10
LIST OF FIGURES
Figure 1. Number of FDA Warning Letters Issued to Manufacturers (2008 - 2014) ....... 38
Figure 2. Number of Prescription and Over the Counter Drugs Recalled
(2002 – 2011) ..................................................................................................... 41
Figure 3. Number of New US Drug Shortages Reported by UUHC (2001 – 2014) ........ 43
Figure 4. Overview of the Pharmaceutical Drug Supply Chain from Manufacturer
to Consumer ...................................................................................................... 56
Figure 5. Company Size of Respondents Most Recent Employer .................................... 93
Figure 6. Primary Location of Respondents Most Recent Employer ............................... 93
Figure 7. Mean Level of Importance Placed on FDA Available Information in
Making Quality Evaluations of Companies ....................................................... 95
Figure 8. Mean Level of Access Frequency of Quality Information from the FDA
Website .............................................................................................................. 96
Figure 9. Metrics that Best Described the Types of Quality Performance Analysis
Companies Commonly Use ............................................................................... 99
Figure 10. Metrics that Companies May Be Willing to Anonymously Share for
Industry Quality Improvement Efforts .......................................................... 100
Figure 11. Mean Level of Agreement on Select Consumer and Company Quality
Valuations Influenced by Scorecards ............................................................ 103
Figure 12. Mean Impact Quality Scorecards May Have on Company Decisions .......... 104
Figure 13. Quality Metrics Believed to be Most Useful for Customers to Determine
Product Quality and Supply Reliability ......................................................... 105
11
Figure 14. Perceived Challenges with the Interpretation of Quality Grades .................. 106
Figure 15. Views on the Best Way to Display a Quality Rating .................................... 107
Figure 16. Information Consumers May Need to Know About a Quality Ranking ....... 108

12
ABSTRACT
An increasing number of industry-wide quality breaches and consequential drug
shortages have fueled interest in finding better ways to improve the manufacturing
quality of pharmaceuticals. Quality metrics offer a way of focusing FDA inspectional
resources using risk-based paradigms, and communicating the quality status of different
companies to other stakeholders. This exploratory study investigated industry views
related to the use of quality metrics and ratings to help advance these discussions. A 23-
question survey engaged two separate populations of industry professionals, a panel of
identified US industry experts (n= 110) and a broader population (n = 328) of
professionals at different job levels working in Southern California. Results showed that
industry values FDA metrics, such as warning letters and inspectional observations
documented in FDA 483s, often characterized as “lagging” metrics. Respondents were
generally hesitant to share information that would establish “leading” metrics such as
process performance measures that may warn of incipient problems. Ratings were
recognized to incentivize higher quality by broadening stakeholder influence. However,
concerns were identified related to the equity, misuse, or misunderstanding of the rating
schemes and underlying metrics. Industry is an important stakeholder in the development
of metrics. Their concerns must be recognized and addressed if policies related to
metrics and ratings are to be effective. This study expanded our understanding of
industry views regarding the use of quality metrics and ratings, but it also identified areas
in need of additional study to help find better ways to incentivize quality and build an
industry-wide quality culture.
13
CHAPTER 1: OVERVIEW
1.1 Introduction
For more than 100 years, regulatory agencies have struggled to establish
mechanisms to ensure the quality and safety of drugs. The current drug law in the United
States (US), the Federal Food, Drug, and Cosmetic Act of 1938 (FDCA), was enacted to
address a series of highly-publicized industrial practices that resulted in public
consumption of unsafe products. Some of these incidents included the adulteration of
food exposed by the investigations of Haskell in the 1850s (Pina & Pines, 2008),
accounts of unsanitary food production illustrated in Upton Sinclair’s 1906 novel, The
Jungle (Sinclair, 1906), and the use of toxic ingredients such as the widespread
consumption of sulfanilamide by over 100 individuals, mostly children in 1937
(Ballentine, 1981). The FDCA prohibits the introduction of adulterated or misbranded
foods, drugs, devices, tobacco products, or cosmetics into interstate commerce. Since its
enactment in 1938, the FDCA has been modified throughout the years to help address
numerous public health concerns. One notable addition that was added by the 1962
Kefauver-Harris Amendments was the strengthening of drug adulteration requirements
by requiring manufacturers to comply with current good manufacturing practices (GMP).
The addition of GMP requirements set the minimum quality standards for the production
of all finished pharmaceutical drugs (Pina & Pines, 2008). Under the GMP regulations,
the US Food and Drug Administration (FDA) extended the meaning of adulterated to
pharmaceutical drugs that are manufactured under insanitary conditions, that fail to have
adequate manufacturing controls, that fail to meet appropriate standards, or that are out of
14
conformance with applicable requirements or conditions of market approval (FDCA,
2010a). These changes to the law reflect an important duty of the FDA not only to
enforce the regulations as written to date, but to continually update them and modify the
compliance benchmarks that FDA uses to audit manufacturers’ quality. Such updates are
challenged by the continued evolution of new drug product types and advancing
manufacturing technology that resets the bar year after year.
Typically, the FDA first takes an active role in the enforcement of GMP
regulations at the time when a company applies for drug approval (FDA, 2014c),
although sponsors of clinical trials also have some obligations under GMP regulations. A
required preapproval inspection of the manufacturing site is followed by routine
inspections thereafter, which have been typically carried out every 2 years for domestic
manufacturers and much less frequently for foreign manufacturers (Mathieu, 2008).
Until recently, the FDCA did mandate that FDA conduct GMP inspections biannually for
domestic manufacturers but had no such stipulation for foreign manufacturers (FDA,
2012b). However as the number of new drugs increase and manufacturers expand
overseas, site inspections have become overly burdensome on FDA (GAO, 2011a). To
alleviate FDA’s inspection burden and help place inspectional resources where they are
needed most, regulators and industry have turned to a risk-based inspectional approach so
that sites with a history of noncompliance would have more frequent comprehensive
inspections than those with a proven record of quality compliance (Brookings, 2014). In
2012, the passage of the US Food and Drug Administration Safety and Innovation Act
(FDASIA) included the mandate and resources for FDA to formalize and implement a
risk-based inspection program (FDA, 2012b). To carry out such a policy, a set of criteria
15
is then required to discern the “good players” from the “bad players”. What might then
follow would be a system for rating and comparing drug manufacturers with respect to
manufacturing quality based on an equitable set of quality metrics.
Some argue that FDA may already have a good sense of the “good players” and
“bad players” based on their inspections, and the FDA has indeed made some early
attempts to develop a more formalized risk-ranking model using their repository of
inspection outcomes (FDA, 2004c). Nevertheless, a transparent, equitable, and
consensual quality evaluation framework that determines the frequency of site
inspections using a risk-based approach still does not exist today. Discussions of rating
systems and risk rankings for pharmaceutical quality have also raised additional
questions on the usefulness of GMPs and inspections, ranked by risk or not, as the
primary tools to ensure the quality of drugs. Inarguably GMP regulations have greatly
improved the quality of drugs since their inception, but the use of GMP standards alone
to achieve adequate product quality does not necessarily provide an incentive for
manufacturers to do anything more than meet that minimum benchmark. The
pharmaceutical industry has recognized this challenge for some time now and refers to it
as the difference between targeting “GMP compliance” as opposed to achieving higher
levels of “quality”.
Some companies do elect to go above and beyond GMP standards because they
recognize, as do companies in other sectors, that a sound quality system can reduce
various operational costs incurred due to quality problems (Crosby, 1980; Deming, 1986;
Feigenbaum, 1991; Juran & Gyrna, 1980). However, there currently exists neither a
compelling regulatory nor market incentive for going beyond GMP compliance. FDA’s
16
enforcement of GMP may therefore fail to encourage manufacturers to spend additional
resources so that their products are of higher quality than might be needed to satisfy
minimum GMP standards (Gupta & Huang, 2013; Woodcock & Wosinska, 2013). In
economic terms, enforcement of GMP standards works as a one-sided incentive-based
regulation whereby the manufacturer must comply with the standards in order to sell
their products, but without any reward for exceeding the standards. The incentive
structure for pharmaceutical products is very different from other products where a two-
sided financial incentive structure exists in which companies are increasingly rewarded,
usually by larger profits, for implementing their best quality measures.
When the current GMP enforcement framework is coupled with current
pharmaceutical market dynamics that focus on achieving low drug prices and reduced
overhead costs, a “race to the bottom” can ensue where “GMP compliance” becomes the
only target (IMS Health, 2011; Woodcock & Wosinska, 2013; Yorukoglu, 2012). These
market forces are most prevalent in the generic drug market (FDA, 2011c; Ventola, 2011;
Woodcock & Wosinska, 2013) where manufacturers sell their products in a marketplace
that is often crowded with competitors, and is driven by low pricing, low profit margins,
unpredictable consumption, and no current means to differentiate product quality
(Committee on Oversight and Government Reform, 2012; Fox & Tyler, 2013; IMS
Health, 2011; ISMP, 2014; Jacobson, Alpert, & Duarte, 2012; Yorukoglu, 2012). In this
environment, manufacturers are faced with having disincentives from making needed
investments, such as facility upgrades, to achieve a higher-level of “quality”. Further
some industry analysts have concluded that the poor incentive structure has been
responsible, at least in part, for persistent quality breaches and drug shortages (GAO,
17
2011b; Gupta & Huang, 2013). This can put the public in harm’s way of defective
products or having to delay or replace their treatments with suboptimum substitutions
during periods of drugs shortages (ASCO, 2011; ASH, 2011; Berry, 2011; Cherici et al.,
2011; Fox et al., 2009; Woodcock, 2012). Thus much can be gained if manufacturers can
be incentivized to improve the quality of their products.
There may be several ways to increase incentives for quality in the manufacture
of pharmaceutical drugs. For example, Villax (2014) recently suggested that FDA
establish a “FDA Dean’s List” of companies that go beyond the minimum standards, and
share this information publicly in order to recognize high quality. Others have suggested
mechanisms such as rating scales to make quality more visible to buyers, who then could
decide if the higher quality would be worth a commensurately higher price (Gupta &
Huang, 2013; Woodcock & Wosinska, 2013). Still others have suggested that companies
with a strong record of quality be given shorter regulatory review periods or preferential
regulatory treatments; any such move that brings a company’s product onto the market
more quickly holds the promise of increasing revenues or reducing operational costs for
the firm (Chabner, 2011; FDA, 2013d; Haninger, Jessup, & Koehler, 2011; ISPE, 2013;
Jacques & Stauffer, 2014; Schweitzer, 2013; Van Trieste, 2013; Villax, 2013). Features
shared by all of these alternatives are the perceived need for some kind of pharmaceutical
quality rating system and the motivational value found in publicly sharing more
information on quality.
18
1.2 Statement of the Problem
Any approach to make quality information more visible would be a major
departure from current regulatory policy. At present, information regarding
manufacturing quality is held quite privately between the company under scrutiny and the
FDA unless the behavior of the company is so egregious as to warrant a public warning
letter. However communicating this information, in a manner that is both informative
and understandable to others such as payers, prescribers, and patients (who up to now
have been silent stakeholders), may help broaden their influence in the marketplace by
allowing them to choose higher-quality products over others. By broadening stakeholder
influence, the current quality incentive structure may shift from a limited one-sided
regulatory compliance incentive to one that is more closely aligned to the market-driven
financial incentives seen in other industries. Such a change in regulatory policy and
stakeholder influence might therefore better incentivize companies to improve the quality
and reliability of their products.
Both regulators and industry groups have recently been suggesting that the
differences in the manufacturing quality of individual products and companies be made
more visible to help overcome the many factors that currently limit pharmaceutical
companies from investing in superior quality systems (FDA, 2013d; Fox & Tyler, 2013;
Jacques & Stauffer, 2014; PDA, 2013; Rawson, 2014; Schweitzer, 2013; Villax, 2013;
Woodcock & Wosinska, 2013). Some have used examples where quality has been made
more visible in other industries, such as restaurants and hospitals, through the use of
simple rating systems so their customers can make informed buying decisions (Jin &
Leslie, 2002; Jin & Sorensen, 2005; Pope, 2009; Rosenthal, Chren, Lasek, & Landefeld,
19
1996). In some cases, these rating systems are also used by third-party assessors, such as
employers and collective buying groups, to help determine if a vendor can consistently
deliver high-quality products for their clients (Beaulieu, 2002; Chernew, Gowrisankaran,
& Scanlon, 2001). Similar quality rating systems have thereby been proposed for the
pharmaceutical sector to provide information on quality to their customers, including
payers, group purchasing organizations, pharmacies, hospitals, clinics, doctors, and
patients.
However, we currently know little of the broader views held by industry on the
use of quality metrics and ratings as way to communicate quality information to
stakeholders in the broader community. The information sources available to date are
empirical and internal analyses conducted by a few organizations (ISPE, PDA,
Brookings) as well as the FDA on the metrics that could be measured for the limited
purpose of assisting with risk-based auditing by the FDA (Brookings, 2014; ISPE, 2013;
Johnson, 2013; PDA, 2013). To be credible and effective measures, quality metrics
should depend on not only the views of the regulators regarding suitable quality
attributes, but also on the cooperation of manufacturers industry-wide who can be critical
of overreaching regulatory interventions. The many calls for making quality more visible
up to now have been mostly thought experiments based on what has worked in other
industries that are in many ways quite different than those selling medical products.
Based on the information available to date, there exists little evidence to understand
whether pharmaceutical manufactures would in fact be swayed to increase their
investments in quality if their products were rated publicly on some form of quality
metric.
20
1.3 Purpose of the Study
The primary purpose of this exploratory study is to better our understanding of
industry views regarding the use of quality metrics and quality rating systems. By using
a survey tool directed at industry professionals, I pose questions about the way that those
professionals currently use public and non-public quality information to assess and
benchmark their own product quality and that of others. I further explore how the use of
a rating system is viewed by the respondents to impact the manufacturing quality of
medical products, and whether the quality practices of their own companies would likely
be affected if such ratings were available publicly. Finally, I explore questions that may
help implement a rating system, such as the disclosure policy associated with publicly
visible ratings and how much information stakeholders may need to know about the
rating scale.
The survey engaged two distinct populations. One panel of respondents is
composed of selected quality experts known to me and to my colleagues, and expanded
by the use of snowball methods. This panel is therefore delimited to manufacturing,
quality, regulatory, and supply chain personnel working at middle to senior levels with
substantial expertise in the subject matter and experience working for medical product
manufacturers. The other panel was developed in consultation a California-based
workgroup called the Quality Leadership Network (QLN) and is composed by a broader
group of pharmaceutical professionals who have been furnished a link to an open-access
survey and who are largely drawn from a geographically localized area, Southern
California.
21
1.4 Importance of the Study
To my knowledge, this is the first academic study to explore how manufacturers
currently favor different metrics to benchmark their activities and to explore their views
on the use of visible quality ratings. By better understanding the attributes and methods
that industry favors for using and conveying quality metrics, the study may provide
important information for policy decision-makers on how a visible rating system might
be developed and used as a way to incentivize a firm’s investment in manufacturing
quality. It also identifies concerns that industry has that should be recognized when
considering policy options.
This study also promotes an open exchange of information in the pharmaceutical
industry on how best to progress manufacturing quality into a more proactive system that
goes beyond a retrospective inspection approach by introducing new incentives to drive
quality. This type of information can inform policy setting and improve a dialogue with
FDA and other regulatory agencies on how best to develop and implement regulatory
frameworks that “move the needle” when it comes to quality. As with any evaluative
framework, e.g., educational assessments, process improvements, or business operations,
understanding and early involvement of stakeholders is a prerequisite for success (Kahan,
2008; Milstein & Wetterhall, 2000). More specifically, engagement of pharmaceutical
manufacturers is critical when developing regulatory policy and such an intervention will
add a greater level of credibility to final decisions made by regulators (Almarsdottir &
Traulsen, 2006; Leong, McAuslane, Walker, & Salek, 2013; Liberti et al., 2013; Peay,
Hollin, Fischer, & Bridges, 2014). Although in this study we focus on manufacturers as a
22
critical stakeholder, this study can promote further engagement with other stakeholders,
such as pharmacy benefit managers, prescribers, and patients.
1.5 Limitations, Delimitations, Assumptions
This study is delimited to understanding the practices and views of medical
product manufacturers on quality ratings. Manufacturers are broadly defined as those
involved in the manufacture of the FDA-regulated medical products and can include raw
material suppliers and supply chain managers. The surveyed populations were primarily
drawn from individuals in pharmaceutical drug and medical device companies, and this
restriction may affect the conclusions that are drawn. The study is also delimited to
manufacturing activities pertinent to the US marketplace. Because regulatory, medical
reimbursement, prescribing practices, and other market dynamics differ internationally,
the views of manufacturers operating in other countries may also differ, and should be
captured separately so that differences in the populations do not contribute to scatter in
the final data-set. The study is further delimited to the evaluation of views from two
rather restricted populations of respondents. The restriction to an expert panel of
respondents was deemed necessary to ensure that the participants had both the experience
and expertise in the subject matter. The selection of the participants for the expert panel
was derived from professional networks, professional task-forces, and snowball referrals.
This approach can potentially introduce some bias because the participants may be like-
minded individuals who may have been influenced by attending meetings, workshops, or
quality metrics research conducted by professional associations such as PDA or ISPE.
Further, the small subset of identified experts included personal acquaintances or
23
snowball referrals; this may also increase the likelihood that their views might be
relatively homogenous reflecting their common work experiences and professional
backgrounds. If the expert panel used is not fully representative of all experts in this
field, the external validity of the study may be open to question.
The open-access survey solicited participation from geographically localized
professional organizations and primarily collected the views of manufacturers from
Southern California. As is typical of open-access surveys, the response rate was
presumably low reflecting the fact that many industry professionals are busy in their day
to day work and may see no benefit in participating. Because the survey was described to
the potential respondents as a result of joint work in association with FDA participants,
some individuals may also have been wary about providing information without a deeper
understanding of the study’s intentions. This kind of behavior may reflect concerns about
protecting proprietary company information. The survey was open-access, and thus
furnished no record regarding the level of work experience of the respondents beyond
that probed by the general demographic questions included in the survey. The results
from this population must be interpreted cautiously because the responses were largely
drawn from a California-based subpopulation. However, it does provide some
opportunity to triangulate the data from the two polled groups. Although efforts were
made to prevent individuals from entering more than a single response, also known as
ballot box stuffing, the Qualtrics software cannot completely prevent such an action. No
reason seems to exist for busy professionals to want to repeat the survey, so the
assumption is made that respondents only entered one set of responses and had no
motivation to do otherwise.
24
Finally I made certain number of other assumptions. First I assumed that all
respondents were truthful in their appraisals and expressed their own opinions, and
experience on the subject matter. I also assumed the participants who completed the
survey had a sufficient understanding of the subject matter to provide meaningful
answers to the questions. Likewise, I make the assumption that I was able to exercise fair
balance and objectivity during the conduct of the study and analysis of the results, in part
because of the repeated consultation and oversight by the industry groups with whom I
have been working. However, such bias may persist to some degree.
1.6 Organization of Thesis
The thesis is organized in 5 chapters. Chapter 1 provides an introduction the
study including the problem statement, the purpose and importance of the study, and the
delimitations and limitations of the study. Chapter 2 provides a more detailed overview
of the pharmaceutical quality regulations and oversight by presenting a literature review
representing the current state of knowledge on this topic. Chapter 3 describes the
research methodology used for this study. Chapter 4 reports the results of the study.
Chapter 5 then discusses the results, the researcher’s opinions drawn from the study, and
recommendations for future study.
1.7 Definitions
As used within this study, the following key terms and acronyms are defined as in
Table 1.

25
Table 1. Definitions of Key Terms and Acronyms
Key Term / Acronym Definition/Meaning
ASA American Society of Anesthesiologists

ASCO American Society of Clinical Oncology

ASH American Society of Hematology

ASHP American Society of Health-System Pharmacists

Brookings Engleberg Center for Health Care Reform at Brookings

CBO US Congressional Budget Office

CFR US Code of Federal Regulations

cGMP, cGMPs Current Good Manufacturing Practices

EMA European Medicines Agency

EP Refers to expert panel surveyed in this study

FDA US Food and Drug Administration

FDCA US Federal Food, Drug, and Cosmetic Act of 1938

GAO US Government Accountability Office

GMP Good Manufacturing Practices

GMP Inspection Refers to routine FDA inspections for GMP compliance

Grades An assigned number, letter, or symbol that depicts the
standing of a company or product using a rating system

ICH International Conference on Harmonization of Technical
Requirements for Registration of Pharmaceuticals for
Human Use

IMS Health IMS Institute for Health Care Informatics

ISMP Institute for Safe Medication Practices

ISPE International Society for Pharmaceutical Engineering

ITA US International Trade Administration

26
Key Term / Acronym Definition/Meaning
Manufacturers Broadly used to identify those involved in the manufacture
of FDA-regulated medical products including raw material
suppliers and supply chain managers

OA Refers to the open-access population surveyed in this study

PAI Refers to FDA preapproval GMP inspections

PDA Parenteral Drug Association

QLN Refers to a Southern Californian workgroup called the
Quality Leadership Network

Quality Refers to pharmaceutical products that are free from
defects, deficiencies, or variations that satisfies its intended
need or function. Generally used in this dissertation to mean
the quality of the manufacture

Rating Systems A classification system according to specified performance
criteria or operational capability

Reliability The ability of a system or component to perform its
intended function without failure over time. Generally
applied in this dissertation to product reliability or reliability
of a manufacturer to deliver products

Scorecards A visual graphic that depicts the standing of a company or
product using a rating system

27
CHAPTER 2: LITERATURE REVIEW
2.1 Evolution of Current Manufacturing Quality Oversight
For more than 100 years, regulatory agencies have struggled to establish
mechanisms to ensure the quality and safety of drugs. The current quality standards that
pharmaceutical drugs are required to meet are designated as Good Manufacturing
Practices (GMP). These standards are often described using the terminology of “current
Good Manufacturing Practices” (cGMP, cGMPs) to incorporate the expectation that
practices must meet all current regulatory standards as opposed to outdated or less
stringent expectations that may have been in place when the product was first introduced
(FDA, 2014b). In the US, the FDA mandates that manufacturers must comply with GMP
standards as a one of the conditions to gain and maintain market approval of a new drug.
The legal basis for GMP compliance can be found in the US Food, Drug, and
Cosmetic Act (FDCA). The original Act, passed in 1938, had introduced the term
“adulterated” to describe drugs that did not meet established standards of identity,
strength, quality and purity, that were typically demonstrated though specification testing.
However, the meaning was expanded as part of the Kefauver-Harris Amendments in
1962 to stipulate that a drug would be considered adulterated if it was not manufactured
in adherence to GMP:
(B) if it is a drug and the methods used in, or the facilities or controls used for, its
manufacture, processing, packing, or holding do not conform to or are not
operated or administered in conformity with current good manufacturing practice
to assure that such drug meets the requirements of this chapter as to safety and
has the identity and strength, and meets the quality and purity characteristics,
which it purports or is represented to possess; (FDCA, 2010a)

28
2.1.1 Current Regulatory GMP Guidance and Enforcement
The FDCA provided the FDA with the authority to develop and oversee the
enforcement of regulations and guidance with respect to GMPs. The agency does this in
three ways. First it develops regulations. Second, it communicates expectations through
a range of guidance documents and other forms of interactions to assist manufacturers in
understanding the minimum requirements. Third, it conducts audits by inspecting
manufacturing sites to evaluate their compliance to GMP regulations.
Codified GMP Regulations and Standards
Per FDCA’s legal mandate on GMP adherence, FDA has codified into law GMP
regulations; these regulations now reside in Title 21 of the Code of Federal Regulations
(21 CFR), Parts 210 and 211 (CFR, 2013b, 2013c). 21 CFR 210, titled Current Good
Manufacturing Practice in Manufacturing, Processing, or Holding of Drugs, provides the
definition and applicability of GMP regulations, whereas 21 CFR 211, titled Current
Manufacturing Practice for Finished Pharmaceuticals, contains a voluminous set of
standards that encompass everything from personnel qualifications, building and
equipment design, to investigating quality failures. GMP regulations provide a high-level
description of the minimal requirements to ensure that compliant products can be
manufactured. However, it is left up to the manufacturer to determine how to meet the
GMP requirements. As only one example, 21 CFR 211.113b, Control of Microbial
Contamination, states,
“Appropriate written procedures, designed to prevent microbiological
contamination of drug products purporting to be sterile, shall be established and
followed. Such procedures shall include validation of all aseptic and sterilization
processes.”
29
This standard establishes the expectation regarding microbial control, but it does not
specify how to prevent microbial contamination or how to validate the sterilization
process. Hence, different companies likely have different approaches to meeting this
GMP requirement.
FDA GMP Guidance and “Soft Law”
Without further guidance, manufacturers can implement their controls to meet the
meaning of the GMP standard, but they are often uncertain if FDA will agree with their
chosen approaches. Moreover, many manufacturers working in the global marketplace
aim to implement systems that meet the quality standards of other regulatory agencies as
well. Thus several sources of more detailed guidance have been developed by FDA and
global regulatory agencies. When FDA typically attempts to give insight into its current
thinking on quality topics, the advice is communicated in a FDA Guidance for Industry.
Numerous guidance documents have been either formally approved or promulgated as
drafts to help provide the needed direction for manufacturers as they attempt to
implement GMP systems. This can help to remove some of the uncertainty that
manufacturers may have regarding the acceptability of their chosen strategies. Often
such documents address specific aspects of GMPs but may remain silent on other areas in
which manufacturers might desire some guidance. As an example, FDA Guidance for
Industry, Sterile Drug Products Produced by Aseptic Manufacturing- Current Good
Manufacturing Practice, provides guidance on meeting GMP standards for aseptic
manufacturing as stipulated in 21 CFR 211, but its scope is limited to personnel
qualifications, cleanroom design, process design, environmental monitoring, and review
of production records for the finished drug product. The guidance does not address other
30
GMP expectations such as upstream processing, control of raw materials and
components, or even the selection of the correct sterilization procedure (steam, heat, or
irradiation). Further, the content and applicability of different guidance documents can
vary greatly. Manufacturers can therefore remain uncertain about their approaches to
meet GMP standards, until the regulator provides feedback after an inspection.
The FDA, like other regulatory bodies, relies increasingly on less formal
approaches, often considered as “soft law”, to inform manufacturers about its current
thinking. For manufacturing quality, these approaches include guidance from global
organizations, the use of compendial standards, and collaboration between regulatory
agencies to convey recommended “best practices”. A particularly important organization
is the International Conference on Harmonization of Technical Requirements for
Registration of Pharmaceuticals for Human Use (ICH). ICH is steered by regulators and
industry representatives from US, Europe, and Japan. Its mandate is to develop
guidelines that address a variety of technical issues including (but not restricted to) those
associated with quality. The numerous ICH guidelines relating to quality issues include
not only documents directed at certain product-specific areas of pharmaceutical
development and quality, but also documents that examine more holistically the
organization of quality systems. The ICH guidelines that have been developed to
harmonize quality encompass a broad range of topics, such as analytical test methods,
stability testing, impurity testing, specifications, and more broadly the development of
manufacturing and quality systems. Particularly important is ICH Q10, The
Pharmaceutical Quality System (ICH, 2008). Many companies use ICH Q10 as a
foundational guideline to establish their systems because this guidance explains how a
31
quality framework should be applied across the entire lifecycle of a product, from product
development to commercial manufacturing.
The Pharmaceutical Inspection Convention and Co-operation Scheme (PIC/S) is
another important organization with a role in the standardization of GMP inspection
practices. To date, 44 regulatory agencies and partners including the World Health
Organization (WHO) have joined PIC/S. As a global organization, PIC/S actively
promulgates recommended “best practices” and quality standards that meet the
expectations of regulatory agencies across the globe. PIC/S also develops GMP
inspection guides for regulators on how to conduct and document a GMP inspection.
These guides are especially valuable to manufacturers because they provide more clarity
on global GMP expectations and foster congruence among the numerous regulatory
agencies that inspect their sites. PIC/S has developed a formal member-admittance
process to ensure that participating agencies and partners have the expertise and resources
to abide by the PIC/S guidelines and contribute effectively to the exchange of
information and harmonized GMP inspector training. Once accepted into PIC/S,
regulatory agencies can then identify themselves as being a “PIC/S member” so that their
GMP certifications and inspection activities can be recognized by other PIC/S members.
Although PIC/S has been in existence since the 1970s, the FDA only joined the
organization in 2010, as the challenges of establishing more robust and harmonized
quality standards globally became a priority (Peither, 2010). However, FDA does not
recognize GMP inspections of PIC/S members as a substitute for their own inspections.
Another source of information that companies often use to gain a better
understanding of regulatory expectations are guidance documents issued by other
32
regulatory bodies as well as national compendial standards. As part of regulatory
intelligence, personnel in companies are commonly tasked with identifying
commonalities and differences between regulatory quality standards by comparing
guidance documents from several regulatory agencies. Regulatory agencies recognize
the work that this process entails and frequently make efforts to align or harmonize their
requirements. For example in 2011, FDA updated their guidance document on process
validation, a GMP requirement, to replace a previous historical approach with a one that
is more science-based (FDA, 2011a). Shortly thereafter in 2014, the European Medicines
Agency (EMA) revised their process validation guideline to include much of the advice
given by FDA (EMA, 2014).
However, in most cases congruence between agencies or standards is not
transparent and manufacturers often cite one agency’s guidance to convince inspectors
from other agencies that their approaches should be satisfactory. Ambiguity,
misalignment, or disagreement on quality standards and practices among regulatory
agencies contribute to what has been termed “regulatory dissonance”, such dissonance
challenges companies to find a “common denominator” in their efforts to market their
products internationally (Storm, 2013; Storm & Richmond, 2015).
Regulatory Enforcement of GMP (Inspections)
In the pharmaceutical sector, the primary way in which regulatory agencies assess
the regulatory compliance of companies is by performing on-site inspections.
Preapproval inspections (PAI) are mandated prior to the approval of a new drug, and
routine “GMP” inspections are then required as long as the product is manufactured and
sold. Until very recently, the FDCA mandated that GMP inspections be conducted
33
biannually for domestic manufacturers (FDCA, 2010b). Interestingly, there had been no
similar stipulation required for foreign manufacturers (GAO, 2011a). However in 2012,
the FDA was mandated by the Food and Drug Administration Safety and Innovation Act
(FDASIA) to pursue a risk-based inspection program (FDA, 2012b) that does not
differentiate between domestic or foreign manufacturing sites. An inspection can also
occur if FDA becomes aware of specific quality problems with a manufacturing site,
usually based on previous findings or product complaints and recalls, these are termed
“for cause” inspections. All of these FDA inspections can yield three outcomes, 1) no
action indicated (NAI) meaning there were no objectionable observations noted and no
further action is needed, 2) voluntary action indicated (VAI) meaning there were
potential objectionable conditions found but not to the degree that would warrant
regulatory enforcement, and 3) official action indicated (OAI) meaning that significant
objectionable conditions were found and further action is warranted to establish
compliance with GMP regulations (FDA, 2014d). When FDA inspectors complete an
inspection, they provide the site with their observations and inspection outcome in a
document titled, FDA Form 483 Inspectional Observations (FDA 483). If the
manufacturer fails to correct the identified deficiencies, the FDA will issue a warning
letter that labels the problematic product as adulterated.
2.1.2 FDA’s Efforts to Modernize Quality Standards
The system of inspecting companies to assure compliance with quality regulations
is now more than 50 years old. However, one of the major challenges to a regulatory
agency is to modify its practices to adapt to changing standards and benchmarks. This is
34
particularly important in an industry like the pharmaceutical industry that is so tied to the
advancement of science and engineering. Nonetheless, the last major revision to US
GMPs occurred in 1978 and only a few selected standards have since been updated to
reflect modern manufacturing practices. As a result, many quality practices followed by
industry today appear to be a fragmented assortment of procedures extracted from a
multitude of GMP standards, guidance documents, and recommendations from several
sources. Many manufacturers point out that some of the tests and controls are known to
be outdated. Manufacturers perform these tests only to satisfy a GMP requirement, in
parallel with more modern and effective detection systems that are better able to increase
the assurance of quality; this “redundant testing” can unnecessarily add overhead costs
(Munson, Freeman Stanfield, & Gujral, 2006).
The FDA has been well aware of the need to introduce more holistic and science-
driven quality systems in the pharmaceutical manufacturing arena, but revisions have
been slow in coming (Woodcock, 2012). To help address these concerns, the FDA has
again turned to the “soft law” approach of using guidance documents because they are
easier and faster to release than codified GMP regulations. In 2002, the FDA began to
communicate their current thinking on modernizing the way that pharmaceutical
development and manufacturing is carried out by placing emphasis on “science-based”
and “risk-based” approaches, including the use of new manufacturing and analytical
technologies. They also sought to reconsider regulatory policies relating to GMP
inspections in an effort to improve product quality while reducing cost burdens for both
industry and regulators. FDA termed these efforts, Pharmaceutical Current Good
Manufacturing Practices (cGMP) for the 21
st
Century. In 2004, the FDA issued their
35
final report on how to implement these approaches (FDA, 2004b). In this report, the
guidance given was aspirational with regard using risk-based approaches. Inspectional
practices were recommended to:
“predict where its (FDA) inspections are likely to achieve the greatest public
health impact…(and) assists that the Agency in creating positive incentives that
reduce the frequency or scope of inspectional oversight for firms that FDA
determines have acquired sufficient process understanding and implemented
effective quality systems.” (FDA, 2004b)

FDA also identified that GMP regulations were limited in scope. They focused primarily
on the specific processes and procedures of manufacturing activities, but did not
communicate the more holistic expectations needed to operate an effective quality
system. To address this need, FDA issued another guidance titled Quality Systems
Approach to Pharmaceutical cGMP Regulations (FDA, 2006). This guidance “intended
to serve as bridge between that 1978 (GMP) regulations and our (FDA) current
understanding of quality systems”. A quality systems inspection model was proposed
that called for the assessment of five manufacturing systems: 1) production, 2) facilities
and equipment, 3) laboratory controls, 4) materials, and 5) packaging and labeling. This
model provided the framework to determine whether each system is in a “state of
control” and in compliance with GMPs. However, the guidance noted that new GMP
standards were not being proposed and FDA inspections would continue to focus on
specific GMP regulations rather than the more holistic organization of quality systems.
In this same guidance, emphasis was placed on the tenet that “quality should be
built into the product, and testing alone cannot be relied on to ensure product quality.”
The document therefore opened the door for a number of more modern risk-based and
36
science-based initiatives, such as quality by design (QbD) and advanced process
analytical technology (PAT). These initiatives encouraged the use of modern scientific
methods and critical thinking to improve quality in a way that could be product specific.
For example, a PAT expert workgroup issued a white paper titled Innovation and
Continuous Improvement in Pharmaceutical Manufacturing (FDA, 2004a) that called for
a shift away from a focus on final product testing to a new paradigm in which analytical
instrumentation could be used to monitor the quality of products as they were being
manufactured (called in-line testing). By finding and correcting quality problems during
manufacturing, superior final product quality could be assured. These types of
approaches have been discussed extensively elsewhere, and are beyond the immediate
scope of the present research. The reader is referred to a number of excellent reviews and
applications elsewhere (Daurio, Medina, Saw, Nagapudi, & Alvarez-Núñez, 2011; Vogt
& Kord, 2011; Wu, White, & Khan, 2011; Yu, 2008).
2.2 Challenges to Quality Oversight by GMP Enforcement
The increased attention to quality approaches and assessments in the
pharmaceutical industry came at an important time. The new millennium saw the
emergence of several forces that began to strain the policy of using GMPs alone to ensure
the manufacture of the high-quality products. At least three of those challenges are
particularly germane to this research. The first is the perceived weakness of regulatory
inspections to detect quality problems. Second is the rapidly expanding globalization of
the pharmaceutical industry that increases costs and makes inspectional oversight
37
difficult. Third is the flawed incentive structure in which adherence to just the minimal
requirements appears to contribute to a “race to the bottom”.
2.2.1 Weaknesses of Agency Audits Alone to Detect Quality Problems
The perceived weakness of agency audits as the primary quality enforcement
policy in the industry stems from the persistence of industry-wide quality problems
despite the use of inspections over many years. Quality problems are perceived as
stubbornly high and quality breaches are often not detected until the defective products
are released into the marketplace (FDA, 2013d; GAO, 2011b, 2014; Throckmorton,
2014; Woodcock, 2012). These perceptions are supported by analysis of FDA’s GMP
audits, enforcement actions, medical product recalls, and drug shortages.
Results of FDA Audits and Enforcement Actions
Evidence for quality lapses abound in the recent materials collected about
products and companies by the FDA. A good source of inspectional observations
regarding quality deficiencies can be found, for example in FDA 483s. In a recent FDA
presentation (Silverman, 2014), the reported number of FDA 483 observations were
compiled and sub-sectioned by type of deficiencies found for years 2005 to 2013. Of
particular relevance were the number of deficiencies in the corrective and preventative
actions (CAPA) taken by firms when a quality problem arises. Deficiencies in CAPA
assessments can be used as a good surrogate measure of the effectiveness of a company’s
overall quality system. It has therefore been disappointing that CAPA deficiencies
typically are amongst the most frequently cited observations and have been steadily
increasing. About 450 CAPA observations were made in 2005, and by 2012 the number
38
of observations increased to about 750. In parallel, the number of FDA 483s classified as
OAI has also been on the rise, from 157 in 2009 up to 260 recorded by 2013.
Quality infractions are also documented in warning letters issued to
manufacturers, after companies are found to have serious deficiencies or persistent
quality issues. An increasing trend in the issuance of warning letters serves as evidence
that quality issues remain to be problematic, as shown in Figure 1. It should be noted that
warning letters numbers may go up or down depending on the size of FDA’s inspectional
resources and its criteria for assessing serious deficiencies in any given year. Yet, the rise
in warning letters from 445 in 2008 to 8,690 in 2014 suggests that much work on the
quality front remains to be done.

Figure 1. Number of FDA Warning Letters Issued to Manufacturers (2008 - 2014)

Adapted from: (FDA, 2015a)
445 474
673
1,720
4,882
6,762
8,690
2008 2009 2010 2011 2012 2013 2014
Year
39
Another source of information available to the FDA and extended to the public are Field
Alert Reports (FARs) that manufacturers are required to report when they become aware
of quality defects with their products such as a contamination or chemical deterioration
(FDA, 2013b). A sharp rise in the number of FARs has been reported over the last
decade, from about 250 FARs in 2005 to a peak of about 1,600 FARs in 2011
(Brookings, 2014). It is currently not possible to see the cause of the quality defects
specifically, but the numbers alone suggest that quality problems are relatively common
in the industry.
Medical Product Recalls
Another form of evidence that can be used to indicate the prevalence of quality
issues is that reflected in the performance of products released into the market. When
such products are determined to be defective, both the manufacturer and FDA are
responsible to remove the product as an immediate means to protect the public. There are
different means in which a product can be removed from commercial channels. At one
extreme is a Market Withdrawal ( 21 CFR 7.3(j)), that allow a manufacturer to remove a
product if the deficiency is minor (e.g., a minor cosmetic problem) and does not
necessarily violate regulations (CFR, 2013a). At the other extreme, the FDA can conduct
a Seizure of products that are in clear violation of federal drug law. Because the FDA is
required to obtain a court order for this action, product seizures usually occur only when
products are consistently of poor quality, pose a serious risk to the public, or are illegal or
banned in the US. However, the two above actions are not necessarily good surrogate
indicators of quality problems. For example, a product can be seized because it is not
registered with the FDA, even if that product is of high quality.
40
A third and most common way of dealing with problem products on the market is
the Product Recall. Recalls can be considered a more reasonable and highly visible
surrogate for quality problems. Manufacturers can voluntarily recall their products when
they are defective or known to be in violation of FDCA. FDA can also encourage or
even force a recall under some conditions. Most commonly, FDA would send a formal
request to the manufacturer to initiate a recall event for defective products before forcing
a recall. Product recalls are also commonly reported by the public press (Business
Standard, 2014; Gokhale, 2014; Preidt, 2014; Thomas, 2014) making them a more visible
measure of quality problems at a firm. Product recall statistics can thereby serve another
valuable measure of quality in the industry.
In FDA’s 2014 Annual Summary of Enforcement Statistics, a total of 2,924 recall
events occurred for 8,061 medical products, dietary supplements, and foods (FDA,
2015a) . Further examination showed that the recalls were substantial across all product
sectors, including medical devices (1,283), drug products (345) and biologicals (678). A
more detailed analysis presented by FDA (Brookings, 2014), confirms that the rise in
recalls for pharmaceutical drugs began around the year 2002, as shown in Figure 2. The
rise and total number of products recalled generally indicate that not only do quality
problems persist in the industry today but they are often only detected once the product is
in the marketplace, thus putting the public at harm’s way.

41
Figure 2. Number of Prescription and Over the Counter Drugs Recalled (2002 –
2011)

OTC = Over the Counter Drugs; Rx = Prescription Drugs
Adapted from: (Brookings, 2014)

Drug Shortages
An indirect but potentially important form of evidence reflecting quality issues is
the prevalence of drug shortages, a problem that has recently emerged as a significant
public concern. Drugs can be in short supply for several reasons but one common cause
stems from quality problems that disrupt manufacturing (Table 2). The problem has
therefore become a particular focus for both the FDA and EMA (EMA, 2012; FDA,
2011c, 2013d).

140
214
238
387
577
345
568
468 479
602
52
59
82
119
90
188
134
200
336
652
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
RX OTC
42
Table 2. US FDA Drug Shortages Root Cause Analysis for Drugs and Biologics
Root Cause of Drug Shortages Percent Found
Quality Issue, Facility Remediation

35
Quality Issue, Product Manufacturing

31
Discontinuation of Product

14
API or Raw Material Shortages

8
Increased Product Demand

6
Loss of a Manufacturing Sites

4
Other Component Shortages 2
Adapted from: (FDA, 2013d)

Much data now exists concerning the prevalence of drug shortages, even beyond
mass media reports (Gaffney, 2014; Karlin, 2014b; Stewart, 2014; Tavernise, 2014).
Some of the earliest data to sound the alarm came from the University of Utah Health
Care (UUHC) that partners with the American Society of Health System Pharmacists
(ASHP) to track drug shortages (Figure 3). They define a drug shortage as any supply
disruption that alters pharmacy dispensing, such as searching for different dosage forms
or a different strength (Ventola, 2011). The FDA also has reported an increasing trend of
drug shortages, albeit defined more restrictively as a situation in which no other sources
or alternative drugs are available. FDA’s 2014 Report on Drug Shortages found 61 drug
shortages in 2005 but more recently documents 250 drug shortages in 2011; this is a four-
fold increase over only 6 years (FDA, 2014g). Sterile-injectable generic drugs are found
to be the most prone to experience shortages because of quality problems, accounting for
80% of shortages in 2011 and 75% of shortages in 2014 (FDA, 2011c, 2014g).
43
Figure 3. Number of New US Drug Shortages Reported by UUHC (2001 – 2014)

Adapted from: (ASHP, 2015)

To understand the type of quality and facility problems that lead to shortages of sterile-
injectables, the International Society for Pharmaceutical Engineering (ISPE, 2013)
conducted a survey within its membership to identify areas of quality most likely to
contribute to a supply disruption or a quality problem “near-miss”. Respondents
indicated for example that the facility layout (7.4% for sterile, 25.0% for non-sterile) and
cleaning and maintenance (14.8% for sterile, 15% for non-sterile) were likely due to
facility related quality problems.
The alarming rise in drug shortages has drawn political attention that has put
pressure on regulatory agencies to use their authority and capabilities to help correct the
state of quality in drug manufacturing operations. Both the US Government
120
88
73
58
74
70
129
149
166
211
267
204
140
185
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Year
44
Accountability Office (GAO) and FDA have been providing periodic congressional
updates since 2011 (FDA, 2011c; GAO, 2011b). In 2011, the President of the United
States issued Executive Order 13588 granting the FDA more authority to help prevent
and reduce the number of drug shortages, in which the consequences of drug shortages on
public health were emphasized:
“Interruptions in the supplies of these drugs endanger patient safety and burden
doctors, hospitals, pharmacists, and patients. They also increase health care
costs, particularly because some participants in the market may use shortages as
opportunities to hoard scarce drugs or charge exorbitant prices.” (CFR, 2011)

The challenges of drug shortages are equally recognized by other stakeholders because of
their public health ramifications, further contributing to the pressure felt by regulatory
agencies to correct the problem. Professional organizations have emphasized that
patients are put at unnecessary risk if treatments of patients must be delayed or replaced
with suboptimum drug substitutions (ASCO, 2011; ASH, 2011; Berry, 2011; ISMP,
2014). Healthcare providers are also frustrated because patient health is potentially
compromised, and the reasons for such shortages are not clear to them (Clarke & Berkrot,
2014; Edney, 2014; Tavernise, 2014). Pharmacists, hospitals, and purchasers have
voiced concerns about the difficulties, costs, and added measures that have to be taken to
manage drug shortages (Fox et al., 2009; Haninger et al., 2011; ISMP, 2010; Teagarden
& Epstein, 2013).
In the past few years, the FDA has attempted to reduce the prevalence of drug
shortages by requiring advanced notification from manufacturers that expect or
experience supply disruptions, by increasing the cooperation between FDA and
manufacturers to correct quality problems, and by requesting alternative suppliers to
45
increase production of certain drugs that may be at risk of a shortage (FDA, 2011c,
2013d); in 2012 FDASIA included the legal mandate that companies must inform the
FDA of manufacturing interruptions that may cause a drug shortage (FDA, 2012c).
These steps appear to have had some impact, as FDA reported 117 new shortages in 2012
versus 251 shortages in 2011. These figures show a similar trend to those from the GAO
that reports 195 new shortages in 2012 versus 255 shortages in 2011 (FDA, 2014g; GAO,
2014) . The data suggest that FDA and GAO are not in complete agreement on what
constitutes a new drug shortage (Gingery, 2014; Karlin, 2014c), but nonetheless make it
clear that the problem of drug shortages continues to persist.
2.2.2 Increasing Globalization Straining Inspectional Oversight
The challenges faced with improving quality domestically are multiplied when the
issues related to globalization are also considered. The world is flattening and global
commerce is increasing at a rapid pace. From 2002 to 2010, pharmaceutical imports
more than doubled from about 150,000 imports in 2002 to about 400,000 imports in
2010. Imported pharmaceutical drugs now comprise about 40% of all drugs consumed in
the US (FDA, 2013a). This trend follows an already well-recognized pattern of
importation for many of the critical materials and components used to manufacture
finished drugs; as early as 1998 it was estimated that 80% of the active pharmaceutical
ingredients (API) used to manufacture finished drugs in the US were imported (FDA,
2011b).
Further, many companies are electing to outsource manufacturing operations to
both domestic and foreign contract manufacturing organizations (CMO), further
46
segmenting the supply chain and creating a complex of web of partners for each
manufacturer. In 2001, the total contract manufacturing market was estimated to be $19
billion and mostly made up of domestic CMOs. By 2010, the estimated contract
manufacturing market more than doubled ($46 billion) and significantly shifted with
more CMOs located in Europe and Asia (FDA, 2011b).
This globalization and segmentation of the pharmaceutical supply chain not only
places an increased burden on regulatory agencies to conduct needed quality inspections
and oversight internationally, but also challenges manufacturers that must audit the
quality of their suppliers and build systems that can track and trace the origin of their raw
materials. Firms using materials that can only be obtained from one or two suppliers are
especially vulnerable if those specialized and hard-to-source materials are found to be of
poor quality. If quality problems are detected, firms must not only struggle to find
alternate suppliers, but they must also get them approved by regulatory agencies in time
to avoid any supply disruptions (Chatterjee & Hirschler, 2014; ISMP, 2014).
Frequency and Type of GMP Inspections Conducted
FDA has had trouble responding to the very large increase in global
manufacturing sites that need oversight and inspection. Limits on personnel and budget
allocations have prevented FDA from inspecting every site on a regular basis. As early
as 1998, the GAO identified that most foreign sites were being inspected only prior to the
approval of a new drug application that listed the site as a manufacturer and often were
never inspected again (GAO, 1998). GAO estimated that only 20% of foreign
inspections were conducted as routine GMP inspections and those were typically spaced
by 4 to 5 years. When compliance issues were found, FDA consistently accepted the
47
manufacturer’s written responses to the inspectional findings as sufficient and did not
return to ensure the corrections were actually made.
In 1998 when GAO issued their report, FDA acknowledged that they did not have
the needed resources to inspect all foreign manufacturers on a routine basis. Because the
FDCA never stipulated a biannual inspection frequency for foreign sites (as was the case
for domestic sites), no rules were technically violated. However, the FDA did appreciate
that more oversight was needed at foreign sites. FDA therefore proposed one of the
earliest risk-based approaches, called a “Tiered Approach” to prioritizing inspections
(Table 3).
Since 1998, GAO has conducted additional reports on the state and progress of
FDA’s inspection program. In 2007, it reported that keeping pace with foreign site
inspections remained a challenge for FDA despite efforts to use their tiered approach
(GAO, 2007). GAO estimated that FDA had managed to inspect only 7% of foreign
firms that should have been inspected in a given year, and that it would take FDA 13
years to inspect all of those sites at least once at the current pace. They also confirmed
that most inspections were still conducted to fulfill new drug application reviews and
very few routine post-approval GMP inspections were conducted for foreign firms.

48
Table 3. FDA’s 1998 Proposal of a Tiered Approach to Foreign Manufacturers
GMP Inspection Program
Tier Manufacturer Type Inspection Frequency
1 Foreign pharmaceutical manufacturers whose prior
inspections were classified OAI

Every year
2 Foreign manufacturers producing sterile bulk, finished,
and aerosol pharmaceutical products

Every 3 years
3 Foreign manufacturers producing 10 or more non-
sterile bulk or finished pharmaceutical products; also
foreign manufacturers supply 10 or more U.S.
pharmaceutical manufacturers and foreign
manufacturers producing non-sterile bulk ingredients
used in sterile finished pharmaceuticals

Every 5 years
4 Foreign manufacturers producing fewer than 10 non-
sterile bulk or finished pharmaceutical products
Every 6 years
Adapted from: (GAO, 1998)

Findings regarding poor capabilities to oversee foreign operations were so
alarming that the GAO included the lack of foreign oversight on their recommended
“high risk” areas for the US Federal Government, stating that “the agency (FDA) is
facing significant challenges that compromise its ability to protect Americans from
unsafe and ineffective products” (GAO, 2009). The year following, only a minimal
increase in foreign inspections, 424 in 2009 compared to 324 in 2008, were again
identified (GAO, 2010). Only about 11% of foreign sites that FDA prioritized for
inspection were visited; this percentage would be even lower if all registered sites were
used as the denominator for the calculation. At this current rate of inspections, the GAO
predicted that FDA would need about 9 years in order to inspect all of the registered
foreign manufacturers.
49
By 2011, GAO reiterated that FDA was “challenged” in keeping pace with both
domestic and foreign GMP inspections (GAO, 2011a). It also highlighted the
observation that FDA still only appeared to inspect manufacturing sites listed in drug
applications, thus excluding inspections of raw material and API manufacturers. GAO
stressed the equal importance of holding raw material sites to GMP quality standards
because those materials ultimately would be part of the finished drug product. Reports of
quality problems arising at raw material and API sites are frequently found to be the root
cause of problems encountered for finished drug products (Chatterjee & Hirschler, 2014;
Manocha, 2014). It emphasized how important it was to assure the required regulatory
oversight of these manufacturers in order to protect the public from defective products.
Costs of GMP Inspections
Inspectional oversight is expensive and appears to be a limiting factor in assuring
adequate inspectional oversight. In 2007, GAO estimated that the required number of
inspections of foreign and domestic drug manufacturers would cost $12.7 million and
$52 million, respectively (GAO, 2007). On a per inspection basis, FDA reported that in
2011 it cost on average $23,000 to inspect a domestic site and $52,000 to inspect a
foreign site (FDA, 2011b). In FDA’s 2013 Annual Report on Inspections of
Establishments, the budget allocated for medical product establishment inspections,
including both drugs and devices, was $250.2 million with 46% allocated for domestic
sites and 18% for foreign sites, the remaining of the budget was allocated to other
activities such as enforcement and emergency preparedness (FDA, 2014a).
50
2.2.3 GMP Driven Quality and a Race to the Bottom (Flawed Incentive Structure)
The paradigm of assuring quality by inspections alone is not only expensive; it is
seen by many as flawed at its core. In the current oversight system the onus is placed on
the regulators to find quality issues, so the only incentive for the manufacturer is to pass
the inspection. Improvements in quality thus are often not perceived as useful until
deficiencies are recognized through a regulatory inspection. Thus, the use of GMPs to
establish minimum standards of adequate product quality does not give manufacturers an
incentive to go beyond the minimal standard. Of concern in such a system is the
potential for pharmaceutical production to become a “race to the bottom” as firms
compete to provide low cost drugs by cutting corners and failing to make the needed
investments to consistently produce high-quality products.
Companies that manufacture generic products now account for 86% of US
prescriptions (IMS Health, 2014), and these firms are particularly stressed to run lean.
They operate in a marketplace that is often crowded with competitors and driven by low
pricing, low profit margins, unpredictable consumption (IMS Health, 2011) and no
current means to differentiate quality differences between products and manufacturers
(Woodcock & Wosinska, 2013). Generic drugs are in fact approved and deemed
therapeutically equivalent or comparable and listed as such in formularies, regardless of
the manufacturer. Under the current regulatory structure, it is difficult for consumers to
differentiate products of high versus low quality. As a result, firms that significantly
reduce their operational costs are rewarded by increased profits. In such an environment,
it becomes difficult to justify expensive quality improvements that go beyond regulatory
compliance. The problem is that reducing costs by foregoing costly quality
51
improvements, such as buying new equipment or upgrading a facility, can put these
manufacturers at a higher risk of having quality problems. What then will encourage a
company to implement advanced quality systems and upgrades in manufacturing
technology?
2.3 Current Incentive Framework toward Pharmaceutical Quality
To understand how manufacturers can be encouraged to support quality
initiatives, it is important to appreciate the underlying incentive framework. The
literature points to a current structure that has many disincentives for going beyond
minimum standards and few advantages to having stronger quality systems.
2.3.1 Incentives for GMP Compliance
The use of GMP regulations to establish minimum standards for product quality
does not provide an incentive for manufacturers to do anything more than meet those
standards. This is often referred to as the difference between “GMP compliance” and
“quality”. If viewed in purely economic terms, enforcement of GMP functions as a one-
sided incentive-based regulation. The primary incentive to comply with GMP is to be
able to sell the product. No incentives are obvious to motivate manufacturers to exceed
minimum quality standards. Apart from being able to sell the product, the additional
rewards of complying with GMP requirements are quite easy to understand. FDA 483s
and warning letters are made public and unless corrected can escalate into further
enforcement actions, such as product seizure, court injunctions, or even criminal
prosecutions. Such evidence of deficiency can taint the reputation of the company in the
eyes of the FDA and other regulatory agencies, as well as the public. The publicity
52
surrounding these serious problems could then make investors unsure of the company’s
future (Gokhale, 2014) and they may choose to limit their investments in the company.
The rewards of going beyond GMP compliance are much less clear and may in
fact be detrimental rather than helpful to the financial success of the company. It is
difficult to convince companies about the importance of strong and often costly quality
initiatives in absence of financial incentives that reward such behavior. Any measures
that absorb rather than increase income are particularly difficult to undertake in today’s
environment. Market revenues in the pharmaceutical sector have been leveling off from
historical growth trends of up to 15% to the current projections of about 3% per year
(FDA, 2011b). This makes it difficult for companies, now more than ever, to justify
expensive quality improvements that are not absolute requirements in order to produce or
sell the product. To justify investments in quality, companies must seek out benefits that
accrue from a stronger quality system. These typically fall into two areas, reducing
production costs or increasing product value.
2.3.2 Incentives and Disincentives Linked to Production Efficiency
Of the two areas identified above as important for incentivizing quality, the one
most commonly advanced is that of reducing production costs, reducing waste, increasing
production efficiency, and minimizing deficient product that cannot be sold. This so-
called “cost of poor quality” has been used, for example, to justify changes that would
facilitate more efficient and productive manufacturing even though such changes might
not be required to satisfy GMPs (FDA, 2011c, 2013d; ISPE, 2013). A sound quality
system in any business is widely known to reduce various operational costs (Crosby,
53
1980; Deming, 1986; Feigenbaum, 1991; Juran & Gyrna, 1980). Decreasing costs of
production through systematic quality approaches such as “lean quality” and “six sigma”
are one way to increase the profit of individual companies, but they are limited in the
degree to which they influence profitability in the pharmaceutical sector. Another way
that costs can be contained is through the improvement of facilities and equipment.
However, such upgrades can be even more difficult to justify to executives who insist on
achieving a return on their investment.
To illustrate this problem, we can consider the design of cleanrooms for sterile
manufacturing operations. Cleanrooms are essential to control microbial contamination
when producing sterile products, and must adhere to certain International Standards for
Standardization (ISO) requirements. The proper use of well-maintained cleanrooms,
regardless of age, can be shown to meet ISO standards as well as the GMP requirements,
stated as follows:
“Aseptic processing, which includes as appropriate:(i) Floors, walls, and ceilings
of smooth, hard surfaces that are easily cleanable; (ii) Temperature and humidity
controls; (iii) An air supply filtered through high-efficiency particulate air filters
under positive pressure, regardless of whether flow is laminar or nonlaminar; (iv)
A system for monitoring environmental conditions; (v) A system for cleaning and
disinfecting the room and equipment to produce aseptic conditions; (vi) A system
for maintaining any equipment used to control the aseptic conditions.” (CFR,
2013d)

Recently, isolator equipment called Restrictive Access Barriers (RABS), have become
widely available and are known to provide superior safeguards and controls to prevent
microbial contamination. Superior microbial control can be gained by upgrading
cleanrooms to these new RABS, but renovation comes with a significant expenditure for
the company. Since the cleanroom satisfies the mandated GMP, no regulatory incentive
54
is present that would motivate a company to make an expenditure on RABS even if it
improves their quality control. Justification for such expenditures would have to come
from a difficult argument that money is ultimately saved by decreasing the incidence of
problems that are more costly to remediate.
It is worth noting that maximizing profits and shareholder value are the founding
principles of business management, and the literature is rich in varying analysis of
company behaviors to meet this expectation. Many of the founding principles and
associated behavioral theory can be found in academic works of notable economists, such
as George Stigler (1968), Douglass North (1980), and Oliver Williamson (1985). In
more recent economic studies, the incentives and factors that cause firms to behave in
ways that benefit society have also been examined. In one study, Campbell (2007)
conducted a thorough review of this literature. He concluded that companies behaving in
“socially responsible” manners are driven by: 1) strong financial performance of the
company, 2) competitive factors in the industry, 3) robust regulations that all stakeholder
participated in building, 4) a sense of industry self-regulation, 5) existence of
stakeholders that monitor or care about actions taken, 6) normative standards that are
valued, and 7) if the actions align with employees’ values obtained through education,
training, or labor organizations. The intent of the present research is not to test any of
these hypotheses for applicability in the pharmaceutical industry, but instead to illustrate
that incentives are usually tied to some aspect of financial performance.
55
2.3.3 Incentives and Disincentives Associated with Pricing
The second major way to increase income is to increase product value, by
increasing sales or price. In most retail industries, such as the automobile or consumer
electronics industries, customers typically have the option to pay more for products that
they perceive to be of higher quality. The pharmaceutical industry has a less direct
relationship with its customers, in which decision-making about price remains hidden by
a complex series of transactions between the manufacturer, wholesalers, pharmacy
benefit managers (PBM), group purchasing organizations (GPO), and payers such as
Medicare or private health plans. Manufacturers sell about 64% of their drugs through
wholesalers; the remaining product is sold directly to pharmacies and hospitals, usually
through GPOs (CBO, 2007). Figure 4 illustrates this complex supply chain.
Along this supply chain, manufacturers negotiate pricing with their buyers, who
most commonly are wholesalers. These wholesalers then negotiate with retailers. In
turn, retail pharmacies receive payment for both drugs and services from insurers who
often pay for a significant part of the drug’s price. In this scheme, pricing is negotiable at
each step and is characterized by each party seeking some profit. Manufacturers,
especially those making drugs such as generics, are left to compete on the basis of lowest
price during their negotiations with wholesalers. When large volume wholesalers or
GPOs have several manufacturers whom they can buy from, they hold most of the
bargaining power (Haninger et al., 2011). These negotiations leave little room for
arguing that one product is of higher quality than another. On the contrary, the market
favors companies that can run “lean” and perhaps “cut corners” to deliver the lowest
price (Conti, 2011; Edney, 2014; Woodcock & Wosinska, 2013; Yorukoglu, 2012).
56
Figure 4. Overview of the Pharmaceutical Drug Supply Chain from Manufacturer
to Consumer

Source: (CBO, 2007)

Little or no product differentiation is also predominant in most sectors of the
pharmaceutical industry. Only a small proportion of new drugs can boast of the
therapeutic superiority needed to command premium pricing (Lu & Comanor, 1998). For
the much larger numbers of generic drugs considered “therapeutically equivalent”, little
differentiation is possible. The prescription market is demonstrably different in this
regard from the over-the-counter (OTC) market in which a more direct relationship with
the consumer allows branding and reputation building so they can set their prices higher
than their non-branded competitors (Harrington & Shepherd, 2002; Hollenbeak, 1999).
57
In summary, the current inventive structure toward quality in the pharmaceutical
industry is unlike other sectors that are subjected to the typical free-market forces of
competition and consumer demand. The GMP regulatory framework incentivizes
manufacturers only to the point that meeting a minimum standard will allow them to sell
their products and maintain a reputation with regulators and their investors. However,
little market pull will drive most manufacturers to go beyond the minimum GMP
standards. What must be found are other ways of making efforts to improve quality more
attractive to company executives and their investors.
2.4 Other Mechanisms to Incentivize and Improve Quality
There may be several ways to increase incentives for quality in the manufacture
of pharmaceutical drugs. For example, Villax (2014) recently suggested that FDA
establish a “FDA Dean’s List” of companies that go beyond the minimum standards, and
share this information publicly in order to recognize high-quality. To quote Villax,
“What does the FDA issue when it wants to say ‘Well done!’? There is no FDA form for
‘Well done.”. Others have suggested mechanisms such as rating scales to make quality
more visible to buyers, who then could decide if the higher quality would be worth a
commensurately higher price (Gupta & Huang, 2013; Woodcock & Wosinska, 2013).
Still others have suggested that companies with a strong record of quality be given
shorter regulatory review times or other preferential regulatory treatments. Any move
that brings a company’s product onto the market more quickly holds the promise of
rewarding the firm, by increasing revenue or reducing costs (Chabner, 2011; FDA,
2013d; Haninger et al., 2011; ISPE, 2013; Jacques & Stauffer, 2014; Schweitzer, 2013;
58
Van Trieste, 2013; Villax, 2013). FDA, industry, and other vested stakeholders have all
expressed their views on how to incentivize quality with a central theme of proactive
quality systems, making quality more visible, and explicit calls for a pharmaceutical
quality rating system.
2.4.1 FDA Efforts to Improve Quality
The FDA appears to recognize that it has a problem to be solved, as reflected in
its 2013 Global Engagement Strategic Plan (FDA, 2013a). In that strategic plan, FDA
acknowledged that its two primary mechanisms of oversight, enforcement at the border
and reactive inspections/investigations have not been adequate in addressing the wider
scope of the quality problems seen today. The plan lays out proposals to engage foreign
regulators and manufacturers to help them build more robust quality systems of their
own. Nonetheless, the FDA also clearly continues to identify more traditional routes of
oversight as key to future activities. A central issue that is undoubtedly of concern to the
FDA has been the failure to conduct a sufficient number of inspections. To cope with the
unrealistically high resource demands of these inspections, the FDA has once again
attempted to develop risk-based inspection models, in which manufacturers making many
high-risk or high-volume products or exhibiting a history of noncompliance could be
identified and inspected more frequently than those with a strong record of compliance
(FDA, 2004b). Risk-based inspections would tend to shift time and resources from
domestic to foreign manufacturers in countries where fewer inspections have been
conducted but numerous quality breaches have been reported. Not surprising then is
FDA’s recent budget plan that reduces the number of domestic inspections from 967 in
59
2013 to 591 in 2014 while increasing foreign inspections from 604 in 2013 to 843 in
2014, this amounts to a 40% increase in foreign inspections (FDA, 2014e).
The FDA’s emphasis on risk-based approaches can also be seen in its Strategic
Plan for Prevention and Mitigation of the Drug Shortages (FDA, 2013d). In this report
the FDA expresses its intent to develop risk-based approaches that would identify early
warning signs of manufacturing problems that could lead to supply disruptions. FDA
pointed out that such a system would be “a challenging undertaking” to build, and it
would start by working with stakeholders to understand the quality vulnerabilities at
manufacturing sites. FDA also mentioned areas that they deemed to be out of their direct
control, including the need for the market to incentivize quality through economic
policies:
“Many shortages are caused by manufacturing quality issues. FDA is exploring
ways to use its existing authorities to promote and sustain quality manufacturing.
However, our ability to offer financial or other economic incentives for
innovation and new investments in high-quality manufacturing is limited. Given
the importance of quality and its link to shortages, payers might explore financial
or economic incentives to encourage high-quality manufacturing that could help
reduce the occurrence and severity of shortages.” (FDA, 2013d)

FDA’s more recent communications signal their forward looking aspiration to formalize
a system of risk-based inspections (Brookings, 2014; FDA, 2013c; Throckmorton, 2014).
It includes the aspiration to increase cooperation internationally, in order to assist and
educate global agencies on GMP standards and to share information about the quality of
the manufacturers in their respective regions (FDA, 2011b, 2013a). Additional
collaboration with public-private international partnerships such as the ICH and the
PIC/S (Peither, 2010) is seen to help share the resource burden of oversight and
60
standards-setting activities. If the GMP expectations and methods of regulatory
inspection are harmonized among participating regulatory agencies, the FDA may be able
to use inspectional findings of other regulatory bodies, such as EMA and Japan’s
Pharmaceuticals and Medical Devices Agency (PMDA), to ascertain which sites would
warrant additional FDA inspection.
Risk-based inspection approaches have the potential to improve the quality
oversight system, but they can still be difficult to implement. To prioritize according to
risk, FDA would need a set of criteria to discern the “good players” from the “bad
players”. An important task for FDA is then to develop formalized tools to differentiate
those companies that have proven quality systems with a sound history of GMP
compliance from those that do not. If such approaches are to work, they should be able
to redirect inspectional resources toward firms that are known to be non-compliant, have
recurring quality problems, or have never been audited.
2.4.2 Pharmaceutical Industry Views on Improving Quality
The pharmaceutical industry appears to recognize that the persistent quality
problems are harmful to patients and bad for business. A review of the current literature
points to a common theme of rewarding the “good players” and stronger enforcement of
laws to penalize the “bad players”. This could be done in several ways, including the use
of regulatory and financial incentives.
Regulatory Incentives
Many solutions have been proposed by which regulators might reward companies
that consistently supply high-quality products (especially in the generics sector that are
61
most likely to participate in the “race to the bottom”). If rewards are to be effective, it is
important to understand what is most important to those companies. One attractive
reward might be a reduction in regulatory review times for manufacturers that have
effectively controlled quality. For example, most drug applications for generic products
takes more than two years to be approved (FDA, 2012a). By giving a shorter approval
path to manufacturers with strong quality records, two potentially desirable goals might
be achieved. Not only would the approach encourage better quality efforts in general, but
it might also provide a quicker path for companies with good quality reputations to
provide drugs at risk of shortages (Chabner, 2011; Karlin, 2014a). However, Yorukoglu
(2012) cautions that this solution might encourage more companies to enter an already
crowded market, and it would consequently only intensify competition that is still based
on lowest price.
Other options for regulatory intervention have also been suggested to incentivize
companies. For example, the FDA can reduce the period required to review supplements
that require their approval when manufacturers want introduce new manufacturing lines
or sites (Haninger et al., 2011; ISPE, 2013; Johnson, 2013). Currently, it takes FDA four
months to a year to review such supplements, and sometimes additional GMP inspections
are needed prior to approval (ISMP, 2014). By reducing this review period,
manufacturers could implement changes more rapidly. A similar intervention may have
already been used by FDA in a recent pilot program, the Secure Supply Chain Pilot
(Devaney, 2014). In this program, a selected group of “highly-vetted companies” have
been given expedited reviews so that they can import drugs more easily.

62
Financial Incentives
Not all of the rewards to encourage high quality need to be linked to regulatory
interventions. Proposed financial incentives have also been suggested. For example,
financial incentives might be introduced by creating a tiered formulary system for
Medicare reimbursement in which premium prices are paid to companies that reliably
produce high-quality products (Chabner, 2011). Others have suggested that such rewards
need not be adopted only by Medicare but rather should include a broader scaling of
prescription drug prices across formularies and payers in a manner that would reward
product quality as well as reliability to deliver the needed supplies (Jacobson et al.,
2012). Another proposal has suggested that the government should give tax breaks to
companies with good quality records as an outright reward for attaining high quality
(ISMP, 2014; Ventola, 2011).
The option that has received most traction also appears to be the one most
disruptive of the current status quo. It revolves around the principle that quality should
be more visible to buyers, healthcare providers, and consumers through some form of
rating system, grades, or scorecards. Not surprisingly, some of the groups promoting
such an approach are from companies that have strong quality records (Schniep, 2014;
Van Trieste, 2013; Villax, 2013). However, similar proposals have also been suggested
by individuals or groups without direct links to particular manufacturers (Gupta &
Huang, 2013; Jacques & Stauffer, 2014; Wechsler, 2015b; Woodcock & Wosinska,
2013). These suggestions have not been publicly countered by dissenting views, but
perhaps such views would not be enunciated for fear of looking politically incorrect.
Measures to make quality more visible are anticipated to be effective because they might
63
allow manufacturers to promote their quality in a way that increases their market share or
helps to leverage their negotiations for increased prices (Schweitzer, 2013).
2.4.3 Changes Advocated by Other Stakeholders
As described above, quality differences between different pharmaceutical
companies are not transparent to stakeholders such as buyers, doctors, pharmacies, and
patients. Thus it may not be surprising that little has been written about the views of
these groups. The question must be asked whether and how those groups discriminate
between high-quality and low-quality medical products. A few references have pointed
to ways that drug purchasers might reward quality. One such measure would be to write
stronger “failure to supply” clauses into their purchase contracts (Haninger et al., 2011;
ISMP, 2014; Karlin, 2014c). Most current failure to supply clauses typically require
manufacturers to reimburse the buyer if they fail to deliver an order, usually by paying
the difference between the promised price and that incurred by the buyer when
purchasing an equivalent product elsewhere (Chabner, 2011; FDA, 2013d). A stronger
failure to supply penalty may be needed because supply failures often occur
simultaneously with a market shortage, thus leaving the buyer with no means to find a
replacement at any price. Some have suggested that large pharmacy benefit managers
(PBM) could also play a larger role in the identification of potential problems because
they have the data and electronic systems to signal impending supply disruptions and
alert their clients of predicted shortages. This data by itself could be used to predict the
capabilities of companies to reliably supply product (Teagarden & Epstein, 2013).
Lastly, better controls on the price “spread” charged by PBMs on certain low-margin
64
drugs have also been suggested, in order to reduce the pricing pressures on the
manufacturers (O'Donnell, 2014) which in turn may alleviate the “race to the bottom”.
2.4.4 The Call for a Pharmaceutical Quality Rating System
One of the earliest hints for the need of a pharmaceutical quality rating system
came from FDA in the wake of increasing pressure to help address the rapid increases in
the number of drug shortages. FDA proposed that if quality was made visible to the
public, consumers might be able to select high-quality products preferentially creating a
strong market incentive for quality (FDA, 2011c). But the most direct call for a quality
rating system came from Woodcock and Wosinska (2013), senior officials for FDA’s
Center for Drug Evaluation and Research (CDER). Their analysis assessed the
“economical and technology drivers” of drug shortages, specifically for generic sterile
injectable drugs. The conclusions drawn from this analysis were not surprising given the
previous discussions.
“We argue that the fundamental problem is insufficient market reward for quality
(including reliability of production) stemming from the buyers’ inability to
observe it. This in turn gives manufacturers strong incentives to minimize quality
system investments...FDA could support the buyers and payers in their purchase
and reimbursement decisions by providing them with meaningful manufacturing
quality metrics. This general approach has been successfully used in many other
settings where quality is difficult to observe or quality signals are difficult to
interpret. Restaurant grades, HMO scorecards or even a US Pharmacopeia stamp
on vitamins are just a few among many tools that utilize this concept”
(Woodcock & Wosinska, 2013)

Manufacturers, academia, and health care providers have also expressed support for a
quality rating system as a means to better incentive quality in the pharmaceutical industry
(Fox & Tyler, 2013; Schweitzer, 2013; Van Trieste, 2013; Villax, 2013). For example,
65
Fox and Tyler (2013) agreed “that increased transparency regarding quality will allow
purchasers to make quality-driven decisions and therefore reward manufacturing quality”.
However, a review of the literature indicates that little is known about the value
placed on pharmaceutical quality by buyers, prescribers, and consumers. This is likely an
artifact of the current market structure where such quality information is not visible to the
consumer. At the same time, much has been written in the business and economics
literature regarding product differentiation, pricing models, branding effects, and the
value of information for consumers. It is not the intent of this research to test or apply
the current thinking of these theories. For economic modeling pertaining to health care
markets, the reader is directed to a couple of excellent papers by Gaynor (2006) and
Gaynor and Vogt (1999). Instead, my concerned is with the practicalities of introducing
quality rating systems through the eyes of the manufacturers. Only after such
implementation would it be possible to gauge the effect on the consumer. However by
looking to other sectors in which rating schemes were introduced, we can see some
possible areas to which attention may need to be paid during implementation.
2.5 Case Studies of Regulated Health Industries with Rating Systems
The use of quality rating systems is not new. Businesses such as hospitals,
medical practices, health plans, and restaurants have used simple rating systems to help
their consumers make purchasing decisions. In some cases, the rating systems are also
used by third-party assessors, such as employers and collective buying groups, to identify
vendors capable of consistently delivering high-quality products that meet all
66
standardized criteria for their clients. The structures and depth of these analyses that
constitute metric rating systems do however vary.
2.5.1 Los Angeles County Restaurant Hygiene Ratings
Hygiene grade cards for the restaurants are one of the most visible rating systems
for consumers. In the 1990s, Los Angeles County required restaurants to display hygiene
grades issued by local health inspectors. Prior to this policy, grades were assigned by
inspectors but were not made visible to the consumer. This system, like that of current
GMP systems, created a high degree of information asymmetry where the restaurant and
health inspectors knew about the hygienic quality of the restaurant but their customers
only knew that the restaurant was clean enough to be open for business. The public
display of a hygiene grade was intended to inform the consumer about the more modest
but nevertheless potentially important differences in quality amongst different dining
establishments, and in so doing provide a strong market incentive for restaurants to
improve their hygiene practices. The grades of A, B and C (A= 90-100%, B = 80-89%, C
70-79%) were determined by deducting percentage points for observed deficiencies when
audited by a county health inspector. A restaurant that received two consecutive scores
below 60% or were found to have severe hygiene problems, such as a pest infestation,
would be identified as unsafe and closed.
The immediate outcome of this policy change is well analyzed by Jin and Leslie
(2002) who sought to answer the question, “Does an increase in the provision of
information to consumers about the quality of firms’ products, cause firms to improve the
quality of their products?”. They concluded that, indeed, the visible grade did cause
67
restaurant owners to improve their hygiene practices, as reflected in an aggregate rise
(10 to 15 percentage points) in inspection scores after the grade card was issued. The
increases in scores were causally linked to structural improvements in restaurants,
suggesting that the scores had a real effect and were not an artifact of other potential
confounding factors during the assessment period (1995 to 1998). Further evidence of
the scoring policy’s effectiveness came from the number of hospitalizations due to food
borne illness that decreased about 13% in the county. These findings were attributed to
the better abilities of consumers to sort restaurants based on grade, as well to the
increased efforts of restaurant owners in improving their facilities and practices to get a
better hygiene grade.
2.5.2 Private and Public Health Plans Ratings
Health plan ratings have been introduced over the last decade to help consumers
navigate the large numbers of competing health plans whose details are difficult to
compare. Many evaluations have been subsequently conducted that evaluate the effect of
this information on consumer decision-making (Barringer & Mitchell, 1994; Buchmueller
& Feldstein, 1997; K. Davis, Collins, Schoen, & Morris, 1995; Edgman-Levitan &
Cleary, 1996; Mukamel & Mushlin, 1998). Overall, the effects on consumers’ choices,
cost savings, health outcomes, and overall patient welfare appear to be mixed. In
different studies, the results appear affected by the study’s limitations, constraints,
assumptions, and confounding factors. A few case studies illustrate how different results
are obtained from such evaluations.
68
In 2001, Chernew and colleagues (2001) studied the impact of a simple scorecard
containing eight measures of health care quality. The scorecard was disseminated by the
General Motors Corporation (GM) to their employees during their 1997 health plan
enrollment period. The scorecard included six measures (operational performance,
preventative health care services, medical and surgical care, women’s health, access to
care, patient satisfaction) that were derived from a Health Plan Employer Data and
Information Set (HEDIS) and the National Committee for Quality Assurance (NCQA).
They were simplified and scored as:  = below expectations;   = average
performance;   = superior performance; or ND= no data available. They found that
12.4% of employees switched health plans during the health enrollment period, but only a
modest 3.89% of the switches could be attributed to the scorecard. The few employees
who did switch were willing to pay on average $488 more in order to avoid a plan that
was rated as low in quality. However, at least two factors may have weakened the
effectiveness of the scorecards. First, the scorecard failed to be “fully informative” since
it did not capture information, such as the quality of covered physicians and hospitals,
that was previously determined to be important for GM employees. Second, charges
were levied if plans were changed during the period of open enrollment, thus decreasing
the attractiveness of switching plans altogether.
Beaulieu (2002) analyzed a similar case where health plan ratings were issued by
Harvard University during its 1996 health-plan enrollment period. Harvard developed a
scorecard for their employees that included measures related to patient satisfaction and
quality of care for all of their health plans. As in the GM study, the quality of care data
was derived from the HEDIS database; the additional patient satisfaction data was
69
derived from a survey conducted for Harvard health-plan participants. The scores were
presented using a number of stars ( ) ranging from 1 to 4, with 4 stars indicating the
highest rating. Beaulieu examined the effects of the scorecard had on health plan
switching by analyzing the two years before and the two years after the use of the
scorecard (1994-1997). Like the GM case, relatively few patients were found to have
switched health plans during the 2 years after the scorecards were issued, but most of
these switched to health plans with higher quality ratings. It was estimated that for every
unit increase in scorecard rank a health plan would see a corresponding 10% increase in
enrollment. Confounding the results, however, was the fact that the employer changed its
health plan subsidy policy in 1995 in a way that resulted in higher price increases for
some plans than others. Given the modest numbers of switches it proved difficult to
attribute the switches to the issuance of the scorecards alone.
Other examples on the effect of health-plan ratings can be found in the literature.
Dafny and Dranove (2005) studied the shift in Medicare HMO market share from 1994 to
2002 and modeled the influence that quality information had on market-share shifts.
They found that the public report cards likely had a modest effect on market share as
compared with other unrelated market influences. The researchers also pointed out that
increased attention to public scorecards have the potential to cause firms to “teach to the
test” and fail to improve other valid quality measures. Jin and Sorensen (2005) also
analyzed the influence of public HMO rating information on plan selection but also
included the rankings in U.S. News & World Report to ascertain the influence of a more
comprehensive set of quality information that were available to the consumer. They
concluded that quality information is valued by the consumers and estimated this
70
valuation to be about $160 annually in savings or increase in service to each person in a
health plan. They also tried to determine if HMOs who scored highly would want to
publicize their results while those who scored on the lower end would want to hide or
withhold their scores. In this scenario, it was hypothesized that an “unraveling effect”
would be seen in the market whereby firms that gain benefit from advertising their high
scores would cause those firms with lower scores to increase their quality in order to
remain competitive. However, this study identified that considerable variability existed
with regard to whether or not the scores were disclosed, especially for firms that scored in
the middle of the scale. In part this seemed to occur because of a number of other market
factors that also influences a company’s decisions to disclose or not disclosure its scores.
2.5.3 U.S. News & World Report Hospitals Rankings
One of the best known rating systems in the health care area is that for hospitals,
whose annual rankings have been published in U.S. News & World Report since 1990
under the headline, “America’s Best Hospitals”. The information used to compile these
rankings is provided by an independent third-party, the National Opinion Research
Center (NORC), that over time has evolved a scorecard based on a sophisticated model
that combines physician nominations, survey information, and health outcome data such
as mortality rates (Hill, Winfrey, & Rudolph, 1997). The grades issued by this model are
presumed to reflect the level of investment by a hospital to improve its facility,
technology, personnel, and health outcomes. Researchers have therefore attempted to
understand the extent to which rankings by U.S. News & World Report assist patients
with their choices and motivate hospitals to improve their quality of care. In the early
71
1990s when the report cards were first issued, many hospital and business leaders viewed
being highly ranked as one of America’s Best Hospitals as a “gold mine” because they
used this recognition for advertising (Rosenthal et al., 1996). In 2009, it was estimated
that hospitals that improve their ranks in U.S. News and World Report, experience a 5%
increase in new patients every year (Pope, 2009).
However, care must be taken when using a rating system to be sure that positive
correlations with the rating system are proven and not confounded by other causative
factors. For example, Chen and colleagues (1999) performed one of the first studies to
ascertain if these hospital rankings accurately represented the quality of the institution by
examining outcomes for Medicare beneficiaries who underwent surgical procedures for
heart attacks. They found that the 30-day mortality rate was lower in highly ranked
hospitals versus hospitals that ranked lower in the 1994 edition of U.S News & World
Report. However, the differences in mortality rate were better correlated with use of
aspirin and other medications at the higher ranked hospitals than with the level of quality,
facility, or physician training at the hospitals. It seemed plausible from these
observations that the hospitals with lower rankings may have done just as well as those
with higher rankings if they used the same medication regimes. More recently Dranove
and colleagues (2002) conducted an instructive analysis on the effects of public score
cards by examining outcomes after coronary bypass graft surgeries in New York State.
Although the intent of providing grade cards was to inform the consumer, their results
suggested that hospitals took numerous measures to improve their scores in ways
unrelated to quality, for example by biasing patient selection and delaying treatments.
72
These results should serve as a caution that rankings without metrics that encourage
quality improvements can lead to unwanted behaviors to just get a good mark.
2.6 Policy Considerations for Pharmaceutical Quality Ratings
Any approach to make quality information more visible would be a major
departure from current regulatory policy. The use of rating systems in the pharmaceutical
industry is also a new idea with little precedence in other countries. Thus any new policy
in this direction will need to be taken carefully with attention to least three factors that
will likely require scrutiny: 1) the selection of the assessor and the metrics that they will
use to rate quality, 2) limitations on public disclosure of company proprietary quality
information, and 3) the management of the policy setting exercise to assure wide-spread
adoption and evaluation of its effectiveness.
2.6.1 Metric Selection and Assessments
A critical component in any rating system is the selection of equitable and
informative metrics that will be accepted by the ranked firms and by the stakeholders
who will use the ratings to make decisions. FDA and industry groups have already begun
to critique possible quality metrics (Brookings, 2014; ISPE, 2013; PDA, 2013). At a
recent meeting organized by the Engelberg Center for Health Care Reform at Brookings
in May 2014, industry leaders, academics, and FDA discussed four metrics (lot
acceptance rate, quality complaint rate, out-of-specification rate, and recall rate) as
potential candidates whose usefulness achieved some level of consensus among the
invited participants (Brookings, 2014). They also examined additional metrics such as
manufacturing lead times, number of corrective actions, process capability, and audit
73
history that had not yet achieved the same level of consensus. Although decisions on
metrics are yet to be made, such stakeholder interaction is important to secure the support
of industry.
The challenge of choosing a set of agreeable metrics is complicated by the
difficulty to find universal criteria that can be applied to various types of products and
manufacturers. Different manufacturers produce a variety of products ranging from
simple over-the-counter lotions to complex biological products. Finding quality metrics
that fairly represent the state of quality across all companies whose manufacturing
methods and product specifications vary widely has so far proven to be difficult
(Wechsler, 2015b). Some have recently suggested that quality metrics, such as those
mentioned above, only tell part of the story and what really matters to determine the state
of quality in any organization is their “quality culture” (PDA, 2014; Schniep, 2014,
2015), an important aspect of quality that is even harder to measure (Axelrod, 2013;
Schniep, 2015) than the metrics that have been discussed to date. Any measurement
based on “culture” may also be vague in the minds of other stakeholders, such as buyers
and patients, who often look for information that is more directly relevant and
understandable to them.
Additional considerations to ensure that rating systems are valued include
assuring that the selection of metrics will encourage quality improvement rather than
behaviors that “game the system”. Further, it is important to consider resource
limitations that might affect how often ratings are issued. Frequent assessments are
important so that ratings do not become a “lagging indicator”. Rather they must be able
to provide timely recognition when manufacturers have invested time and money to
74
improve their products’ quality. The assumption seems most prevalent to date that FDA
will develop the ratings in a manner tied to their inspections. However, it is not clear that
this is the only or best organization to issue appropriate ratings for the use of external
stakeholders. In many other applications the ratings are done by third-party assessors.
2.6.2 Public Disclosure of Ratings
If ratings were in fact developed, a second question arises of whether the
disclosure of the “grade” to the public should be voluntary or mandatory. In many other
sectors where rankings are typical, such as automobiles and electronics, a voluntary
disclosure policy can be effective since consumers quickly learn to become wary of those
companies that do not advertise their ranks; this is known as the “unraveling effect”.
However other industries differ and the “unraveling effect” can be incomplete. That is
largely because many customers can remain unaware that a ranking system even exists.
In two of the case studies discussed so far there were mixed results on the degree to
which stakeholders were aware of the ratings and the extent to which companies that got
acceptable ratings would choose to disclose them. In the case of restaurant ratings, the
policy quickly moved from voluntary to mandated disclosure leaving little room for an
analysis of the long-term effects of voluntary disclosure (Jin & Leslie, 2002), i.e. whether
potential customers would eventually come to expect to see the grade before they decide
to eat at a restaurant. In the case of hospital ratings, researchers found that other factors
such as the cost and effort to disclose the rating can influence companies not to disclose
the rating regardless of whether they got a respectable rank (Jin & Sorensen, 2005).
Studies have also shown that such behavior (called incomplete “unraveling effect”) can
75
be more common when the information is difficult to understand, there is no standard
means of presenting the information, and where there is no incentive to do so (Fishman &
Hagerty, 2003; Mathios, 2000).
It is unknown how the pharmaceutical market will react to the proposed quality
ratings if companies were allowed to disclose their ratings voluntarily rather than being
mandated to disclose them. However, the FDA has recently pointed out their legal
obligations with respect to disclosing confidential information necessarily limits the
extent to which quality metrics and ratings collected for their internal use could be
disclosed to the public (Brookings, 2014). Thus a potential first step would be a rating
system that is open for voluntary disclosure, such disclosures would occur without the
need for broader policy reform.
2.6.3 Policy Setting and Evaluative Frameworks
It is an important tenet of policy development that stakeholders should be
involved early in the development of evaluative methods and frameworks to assure that
the approaches are credible and acceptable to them [see Kahan (2008) for an in-depth
analysis]. This imperative has been stressed by the Centers of Disease Control and
Prevention (CDC) that has emphasized the importance of gathering credible evidence
related to policies and issues in a collaborative manner with their stakeholders (Milstein
& Wetterhall, 2000). Such an approach helps them to justify any changes to their
programs. The FDA has also placed much emphasis on stakeholder collaboration and
engagement that they now have an office called Professional Affairs and Stakeholder
Engagement that is charged with conducting research and supporting external
76
collaborations “to ensure that CDER has a thorough understanding of partner,
stakeholder and public opinion about issues of interest to CDER” (FDA, 2015b). One of
the most important of the stakeholders in the development of risk-related evaluations,
whether they are used to prioritize audits, develop rankings for FDA, or produce publicly
disclosed scorecards, are the industries that would be subjected to these assessments.
Thus, their early engagement is likely to be critical when setting this type of policy. In
such policy development efforts, information gathering about industry views and the
subsequent exchange of opinions would add a greater level of credibility to final
decisions made by regulators (Almarsdottir & Traulsen, 2006; Leong et al., 2013; Liberti
et al., 2013; Peay et al., 2014). Yet, it can be difficult to understand the views of industry
stakeholders because they can be difficult to reach. The voices that are often heard
loudly are from professional organizations that tend to represent bigger companies
disproportionately, or from entities with a strong set of views that motivate them to be
vocal about a particular policy.
To my knowledge, formal regulatory research on how pharmaceutical manufacturers
would be influenced by risk-related rankings and quality rating systems has yet to be
conducted. There exists little evidence to suggest that manufacturers would or would not
be swayed to increase investment in quality if rating systems were introduced under
different types of conditions. There also exists little evidence to suggest that all types of
pharmaceutical manufactures would be equally influenced by a quality rating system.
Kitson and colleagues (2008) suggested that the ideal situation for implementation of new
policy relies not only on a high context (willingness or level of preparedness) to
77
“embrace and sustain implementation”, but also on a high level of evidence to support
the change.
To help bridge the evidence gap that exists regarding quality and risk-related
evaluations, I here present the results of a survey study of US medical product
manufacturers. The survey explores the appetite that might be present for policy change
and the options that might be preferred if rating systems were to be developed based on
quality metrics. This study extends from the activities of a coalition of regulatory,
industry, and academic professionals who formed a workgroup to consider the adequacy
of present policies related to the assessments of medical product quality. Coalitions of
this type are often a good way for policy change to be influenced more effectively as
discussed by Sabatier and Jenkins-Smith in their proposed Advocacy Coalition
Framework (ACF). They advance the concept that “individuals have core beliefs about
policy areas, including a problem’s seriousness, its causes, society’s ability to solve the
problem, and promising solutions for addressing it” and a coalition of these individuals
then have a better chance to change policy (Stachowiak, 2009). Coalitions of
stakeholders from both public and private organizations, rather than single parties or
special interest groups, also have a particular better chance to achieve more long-term or
far-reaching policy changes (Jenkins-Smith & Sabatier, 1994; Weible et al., 2011).
However, such coalitions have a potential disadvantage of not being able to speak for all
subgroups within the involved population that they might be thought to represent. As
part of exploring the degree of homogeneity and the strength of beliefs of particular
groups of stakeholders, it is then important to use a systematic research approach that
78
solicits diverse input beyond the small number of interested parties represented by the
stronger coalitions.
The presented research grew out of a local coalition called the Quality Leadership
Network (QLN), further described in Chapter 3. The research was informed by the
review of documents and opinion pieces collected in the literature from other groups
working in this space, such as PDA, ISPE, and Brookings, as has just been described in
the literature review of Chapter 2. A survey instrument was then developed and
disseminated in consultation with the QLN as well as the USC International Center for
Regulatory Science to increase our understanding of manufacturers’ needs, current
practices, and views with respect to the current state of quality evaluations and options
for quality ratings.
79
CHAPTER 3: METHODOLOGY
3.1 Introduction
The research had three parts, a detailed literature review, presented in Chapter 2
and not discussed further here, a stakeholder analysis to define the survey population of
interest for this research, and an exploratory research survey.
3.2 Stakeholder Analysis and Respondent Selection
As a preliminary step before selecting a target population for the research, a
stakeholder analysis was conducted to identify target populations that would have an
interest or expertise either in pharmaceutical quality metrics, quality ratings, or
manufacturing investment decisions. Also considered were groups who might be
impacted by quality failures such as supply chain interruptions and drug shortages.
Stakeholders were then empirically appraised for three factors, 1) their role or potential
interest in quality ratings, 2) their degree of influence in implementing quality ratings or
incentivizing quality, and 3) additional survey research considerations. These
assessments were summarized in a stakeholder analysis table provided in Appendix A.
All of these stakeholders are ultimately important when assessing policy options, but for
the purpose of this exploratory investigation a decision was made to begin the exploration
with pharmaceutical manufacturers.
Pharmaceutical manufacturers were chosen for this research because they are
active participants in the development and collection of manufacturing metrics, they hold
expertise in manufacturing operations and quality management, and ultimately they make
decisions regarding their firm’s quality direction. Their views on quality metrics and
80
rating systems would therefore provide valuable information about what would be
acceptable and not acceptable to this critical group. I also believed that manufacturers
might be open to completing a survey in a timely way because the issues of quality
metrics have been highlighted recently. This set of stakeholders might then be highly
motivated to make their views known. Industry groups, professional associations, and
industry meetings also provide good forums to identify potential survey participants, and
this might increase the external validity of the study.
The target group of this survey was refined to include pharmaceutical
professionals working at middle to senior levels (e.g., Sr. Managers, Directors, Vice
Presidents, and Chief Officers) from pharmaceutical manufacturing firms regardless of
size. Appropriate pharmaceutical professionals were considered to include those persons
who would have insight or be impacted by quality metrics, typically with job functions in
product development, quality management, regulatory affairs, manufacturing operations,
and supply chain oversight. To obtain the broadest set of responses as possible, the
respondent pool was not restricted to firms producing finished products but also included
companies that supplied raw materials or who managed the manufacturing supply chain.
3.3 Development of the Survey
Questions for the survey instrument were developed based on the information
obtained from the literature review, discussions with the research thesis committee from
the USC International Center for Regulatory Science, and collaboration with a California-
based workgroup called the Quality Leadership Network- Metrics Team (QLN). The
QLN was a working group composed of approximately 30 FDA and industry
81
professionals from a range of FDA-regulated manufacturers including pharmaceutical,
nutritional/dietary, raw materials, and medical devices that was supported and organized
by the US FDA Los Angeles District Office (FDA LOS-DO) located in Irvine,
California, USA.
The survey was constructed and managed using Qualtrics web-based survey
software (www.qualtrics.com). A draft survey was constructed that included 27
questions in various formats, such as multiple-choice, matrix, scaled, rank-order, yes/no,
and open text-entry formats. The clarity, application, and content of these questions were
reviewed and discussed at QLN workgroup meetings, held weekly in April - May, 2014.
The QLN workgroup meetings were typically 60 minutes in duration and used on-line
web conferencing. Suggestions from the QLN included improving the use of industry
terms for clarity, adding additional granularity to demographics and scaled questions, and
reducing the number of open-text questions in an effort to reduce the survey length. The
workgroup also recommended offering a summary of survey results as an incentive to all
respondents who answered all the questions in the survey; this was meant to improve the
survey response rate.
An additional measure was taken to pilot a near-final version of the survey to test
the content validity, obtain measures of the time needed to complete the survey and the
response rate per question, and assess the overall functionality and ease-of-use of the
software platform. The pilot survey (Appendix B) was introduced at the Fifth Workshop
meeting of the Quality Executive Leadership Series, titled “FDA and Industry Executives
Working Together to Improve Quality”, on April 30, 2014 at the FDA LOS-DO
conference center. The presentation included an introduction regarding the purpose of
82
the survey, the survey outline, and discussions on survey distribution techniques.
Additional comments were captured from the meeting attendees (~80 people) during a
question and answer exchange. In general, recommendations were provided to
supplement the questions so that they would capture more “best-practices” regarding
quality management to balance the perceived focus on metrics that measure failures, and
to better distinguish “quality” from “GMP compliance”. Meeting participants also
emphasized the importance of ensuring anonymity of the survey responses. After the
meeting, an open-access anonymous link to the survey was distributed by email to all
registered meeting attendees with a request to respond to the pilot. The survey remained
open from April 30 – May 31, 2014 and collected 31 responses, 26 of which contained
answers to all questions. The overall mean survey duration was 22 minutes.
Based on the feedback obtained from the QLN workgroup meetings and the pilot
survey respondents, the survey was finalized and structured as shown in Table 4. One of
the main modifications made for the final survey was to add skip-logic functionality in
the open-text question asking for any other publicly available metrics and proprietary
metrics used at firms. This was done to obtain more specific information from those who
answer “supply chain management” as their primary occupation; these respondents would
skip the more general open text-box question and be presented with a more specific
question asking for information that would be important for supply chain managers.
Minor changes were made to increase the clarity of the questions. The final survey was
also changed to restrict the ability of survey takers from returning to and reviewing
questions that they had already answered. This was done in order to reduce the survey
duration to about 15 minutes and improve the survey completion rate. Finally, two
83
opening introductions were crafted to meet the needs of the two groups of survey
participants and two different survey distribution methods. One group of participants
included an expert panel of preselected participants who were placed in the panel by
invitation only, so that the qualifications of survey participants could be matched to the
criteria established for the expert panel. The second group was less restrictive and was
recruited using an open-access web link that could be distributed through professional
organizations. A copy of the final expert panel survey and the open-access panel survey
is provided in Appendix C and Appendix D, respectively.

Table 4. Construct of the Exploratory Survey Instrument used in this Study
Block Question Content No. of questions
in block
1 Survey introduction

Not applicable
2 Demographics

4
3 Use of currently available external/public quality metrics
information

4
1

4 Use of internal/non-public quality metrics information

5
1

5 Views on the use and application of a quality rating
systems
10
2

6 Request for summary results

Not applicable
7 Thank you message

Not applicable
Total Number of Questions 23
1
Incorporates skip-logic question for supply chain respondents
2
Incorporates one additional open text box follow-up question for respondents answering
“No” to question regarding if scorecards should be mandatorily disclosed.

84
3.4 Respondent Descriptions
3.4.1 Expert Panel Respondents
The expert panel was composed of individuals who had roles related to
pharmaceutical product development, manufacturing, quality, regulatory affairs, and
supply chain management in US-based pharmaceutical companies. Individuals were
identified by the professional networks of the researcher and the thesis advisory
committee, referrals from QLN members, and identified individuals who had given
relevant presentations at industry meetings. Individuals included in the expert panel were
confirmed to be qualified by their respective titles (e.g. Director of QA, VP of
Manufacturing), their presentations and publications on related subject matter, or by
reliable referrals.
An initial list of 254 individuals was compiled. The individuals were pre-notified
by email with a short introduction regarding the purpose of the survey and a request to
participate. In this same email, an additional request was made to recommend others who
were well-positioned to take the survey, using a snowball technique to increase the
diversity of the sample population. This identified 20 individuals that were subsequently
included in the expert panel. In addition a subset of two organizations, the Parenteral
Drug Association (PDA) Quality Metrics Task Force and the International Consortium
for Innovation and Quality in Pharmaceutical Development (IQ Consortium) GMP/QA
working group agreed to participate in the expert-panel survey. The PDA Quality
Metrics Task Force is a subset of 14 individuals who are members of the PDA, a global
organization that develop scientifically sound and practical technical information to
advance science and regulations (www.pda.org). The IQ Consortium GMP/QA
85
workgroup is a subset of 27 individuals who are members of the IQ Consortium, a not-
for-profit organization composed of pharmaceutical and biotechnology companies that
seek to advance science-based global standards and regulations (www.iqconsortium.org).
Some individuals declined to take the survey because their company policies prohibited
them from participating in industry surveys; they were removed from the expert panel.
3.4.2 Open Access Respondents
In collaboration with QLN workgroup, professional organizations within
Southern California were surveyed to identify those with membership from the local
manufacturing industry that had members who dealt with product quality and regulatory
affairs. Individual contacts within these organizations were approached by the QLN team
to ask if their organizations would be willing to distribute the survey amongst their
membership. Four organizations, including the QLN, agreed to distribute the survey,
their descriptions are provided in Table 5. Other organizations that were identified either
did not respond to the QLN’s request or stated they could not participate because any
research studies would have to be administered or approved by the organization; no
additional efforts were made to obtain their participation.

86
Table 5. Southern Californian Professional Organizations that Distributed the
Open-Access Survey
Name Description Distribution
Method
OCRA Orange County Regulatory Affair Discussion Group is a
nonprofit volunteer organization in Southern California.
OCRA mission is to provide educational programs and
information exchange among its members. Members are
primarily regulatory affairs professional. Survey mailed to
~2500 email addresses. (www.ocra-dg.org/index.php/en)

Email Link
SDRAN San Diego Regulatory Affairs Network is a nonprofit
organization in San Diego, California. It serves to provide
professional development and information exchange by
way of events and seminars. Survey mailed to ~970 email
addresses including QLN members. (www.sdran.org)

Email Link
QLN FDA/ Industry Quality Leadership Series participants. This
includes the QLN quality metrics workgroup as well as
three additional QLN workgroups.

Email Link
ASQ-
Orange
Empire
Section
American Society for Quality is a global organization that
champions quality programs, education, and certification
for quality professional. Per its mission, they seek to
“increase the use and impact of quality in response to the
diverse needs of the world.” The Orange Empire Section,
is a regional group of the organization that serves the
Orange County, CA area. Survey mailed to ~3000 email
addresses. (www.asqorangeempire.org)
Email Link
and Posted
on Website

3.5 Survey Deployment
3.5.1 Expert Panel Distribution
On June 02, 2014, a survey web-based link was emailed to the 257 selected
individuals. The survey was then sent to the PDA Metrics Taskforce on June 14
th
, to the
IQ Consortium on June 16
th
, and to individuals obtained from snowball referrals while
the survey remained open. The survey was distributed using the Qualtrics Mailer
function that manages email distribution by sending unique survey links to each
87
participant in a panel. Included in the email was a short message to introduce the survey.
An opt-out link was also provided in the email, a typical practice so that email security
software would not disregard the email as SPAM. The survey was set as invitation-only
to ensure that the survey links could not be passed to others. Altogether, the majority of
the emails sent were confirmed by Qualtrics to be delivered successfully with only 12
returned as undeliverable. Emails returned as undeliverable were checked to make sure
the email address was correctly written. Only three emails were able to be corrected and
resent. The other targeted respondents were presumed to have moved to a new place of
employment making their email addresses invalid.
Qualtrics was set to send an automatic email reminder to individuals who had not
completed the survey after 3 weeks (mid-point of the survey open period). Where
possible, additional follow-up emails and phone calls were made to participants who had
not yet responded to ensure they received the survey link and were still able to
participate. A final reminder was sent during the last week of the survey to all who had
not yet responded.
3.5.2 Open-Access Distribution to Professional Organizations
As of June 06, 2014, the open-access web survey link was made available for
distribution to the professional organizations listed in Table 5. The survey link was an
anonymous link that could be either sent by email or posted on a website. Although most
organizations could not provide email membership lists because of privacy policies, some
organizations agreed to send the email link from their email servers on behalf of the
88
QLN. To reduce the possibility of multiple entries from a single respondent (ballot box
stuffing), Qualtrics was set to restrict more than one response from the same IP address.
3.6 Data Collection and Analysis
The expert panel (EP) and open-access (OA) surveys remained open for a period
of 6 weeks (June 2 – July 20, 2014). All answers to questions were accepted for analysis
regardless of whether every question was answered. However, four individuals selected
for the expert panel communicated that they were not well suited to answer the questions
and wished to abandon the survey; their responses were removed from the results. The
survey response rate, completion rate, duration (time taken to complete the survey), and
the response number (n) for each question are reported in the results.
Responses from each survey were analyzed independent of one another to
preserve the internal validity of the two surveys. Most data was subject only to simple
descriptive statistics using percentages (%), means ( 𝑥 ̅) and standard deviations (SD).
Information from open text questions was examined for new information and reoccurring
themes or suggestions. Cross-tabulations were performed for several questions using the
Qualtrics software, to find common trends and areas of divergence related to differences
in the respondents’ demographic profiles within each survey. Chi-square statistics were
used to measure the significance of these comparisons between discrete datasets, but
generally the numbers were too low to justify confidence in the outcomes of those
evaluations. The Chi-square statistics (x
2
) are reported with the degrees of freedom
(DF), sample size (N), and p-value using the notation, x
2
(DF, N = X) = XX, p = XX.
Comparisons from one survey to the other was analyzed using t-tests for the continuous
89
datasets, but the relatively small numbers in the study precluded the demonstration of all
but strong differences between the two surveys.
90
CHAPTER 4: RESULTS
Results are presented in four sections as follows: 1) demographic profiles and
response rates, 2) information used to benchmark quality, 3) views on quality ratings, and
4) cross-tabulation analyses.
4.1 Survey Demographic Profiles and Response Rates
4.1.1 Expert Panel Survey
The expert panel survey was sent to 309 recipients and yielded 110 responses,
equating to a 36% response rate. Ninety-three answered all of the questions, resulting in
a survey completion rate of 85%. The average time to complete the survey was 40
minutes. Most respondents worked at companies described as manufacturers of
pharmaceutical drugs and biologics (72%), manufacturers of generic drugs (12%),
contract manufacturing services (12%), and API and raw material suppliers (11%)
(Table 6). As shown in Figure 5 and Figure 6, over half (57%) of the respondents from
the expert panel worked for large companies (> 5000 employees) whose primary location
was in either the United States (66%) or Europe (19%). Research and development
scientists (12%) and manufacturing operations managers (6%) were also represented.
Respondents mostly held positions in quality management (39%) and regulatory affairs
(28%). Respondents who selected “other” self-identified as business/program managers,
executives (CEO/president), operational excellence professionals, engineers, and industry
consultants (Table 7).

91
4.1.2 Open-Access Survey
An open-access survey does not permit an accurate estimate of response rate.
What could be verified was that 328 respondents opened the survey, 246 (72% of total)
answered at least the first two-thirds of the survey, and 178 (54% of total) answered all of
the questions. About half of the respondents worked at medical device companies (51%)
and only a minority worked for manufacturers of pharmaceutical drugs and biologics
(21%) (Table 6). On average, respondents took 21 minutes to complete the survey.
Company sizes were distributed quite evenly between small (1–250 employees; 44%) and
large companies (>1000 employees; 38%) (Figure 5). Most (73%) respondents had
employers located in “West Coast, USA” (Figure 6). As in the expert panel, respondents
mostly held positions in quality management (39%) and regulatory affairs (34%)
(Table 7).

92
Table 6. Most Recent Employer Type of Respondents
Expert Panel
Open Access
Survey
Description No. %

No. %
Manufacturer of Pharmaceutical
Drugs/Biologics (Innovator/Branded)

78 72 61 21
Manufacturer of Pharmaceutical
Drugs/Biologics (Generic)

13 12 13 4
Manufacturer of Pharmaceutical Raw
Materials (API, Excipients, Reagents,
Containers)

12 11 4 1
Manufacturer of Cosmetic Products

1 1 5 2
Manufacturer of Dietary Supplements

1 1 6 2
Manufacturer of Food / Nutritional
Products

1 1 6 2
Manufacturer of Medical Devices
(including IVDs)

7 6 150 51
Contract Manufacturer / Services

13 12 26 9
Regulatory Agency / Government Body

0 0 6 2
Analytical Laboratory or Other Testing
Services

1 2 4 1
Distributor or Wholesaler of Medical
Products

2 2 11 4
Other
1

9 8 54 18
Total Responses
2
108 100 296 100
1
Respondents who entered a response of other primarily indicated they were
consultants (regulatory, quality, engineering) or worked for various kinds of medical
suppliers.
2
Total responses identify sample size per question. Respondents were allowed to select
up to two choices resulting in increased counts per choice selection.

93
Figure 5. Company Size of Respondents Most Recent Employer

Figure 6. Primary Location of Respondents Most Recent Employer

12%
13%
3%
5%
10%
57%
23%
21%
9%
8%
13%
25%
1 to 49 50 to 249 250 to 499 500 to 999 1000 to 5000 Over 5000
Company Size per Number of Employees
Expert Panel (n=108)
Open-Access Survey (n=289)
47%
2%
17%
6%
0%
19%
3%
7%
73%
3%
5%
13%
0%
3%
1%
1%
West Coast, USA
Central/South, USA
East Coast, USA
North America
South America
Europe
Asia
Other
Expert Panel (n=107)
Open-Access Survey (n=292)
94
Table 7. Primary Occupation of Respondents
Expert Panel
Open-Access
Survey
Description of Primary Occupation No. %

No. %
Manufacturing Engineer or Scientist

3 3 3 1
Research and Development Scientist

13 12 11 4
Manufacturing Operations Management

6 6 17 6
Quality Management

41 39 113 39
Supply Chain Management

0 0 4 1
Regulatory or Legal Affairs

30 28 99 34
Business Consultant

3 3 9 3
Other
1

10 9 33 11
Total Responses 106 100 289 100
1
Respondents who selected other self-identified as business/program managers,
executives (CEO/president), operational excellence professionals, engineers, and industry
consultants.

4.2 Information Used to Benchmark Quality (Metrics)
4.2.1 Publicly Available Information
A series of questions in the survey were asked to better understand manufacturers’
current use of publicly available information pertaining to product quality. Despite the
significant differences in demographic profiles between the two surveys, responses were
broadly similar. When asked about the importance of different types of information
provided by FDA from a list of eight choices, respondents from both groups most
commonly chose FDA warning letters and 483s and most commonly accessed them
monthly or quarterly from the FDA website; the drug shortage index was least commonly
chosen and least commonly accessed (Figure 7; Figure 8). FDA field corrective actions,
95
company injunctions, drug recalls/withdrawals/safety-alerts, company response letters,
and FDA trend analysis were mostly described to be “important” by both groups but FDA
field corrective actions seemed to have a somewhat higher importance to respondents in
the open-access group. The importance placed FDA field corrective action was
significantly higher (p < 0.001) and accessed more frequently (p = 0.01) by the open-
access group than the expert panel.

Figure 7. Mean Level of Importance Placed on FDA Available Information in
Making Quality Evaluations of Companies

The difference between the two surveys for FDA field corrective actions was determined
to be statistically significant, p < 0.001.

3.7, SD 0.6
3.8, SD 0.4
3.1, SD 0.9
3.2, SD 0.9
3.4, SD 0.8
2.5, SD 1
3.3, SD 0.8
3.3, SD 0.8
3.6, SD 0.7
3.7, SD 0.6
3.5, SD 0.7
3.3, SD 0.8
3.3, SD 0.9
2.4, SD 1.1
3.2, SD 0.8
3.3, SD 0.8
1.0 2.0 3.0 4.0
FDA Form 483 Observations
FDA Warning Letters
FDA Field Corrective Actions
Company Injunctions Information
FDA Drug Recall, Withdrawals, Safety-…
FDA Drug Shortage Index
Company Response Letters
FDA Data and Trend Analysis (e.g.,…
Expert Panel (n=105) Open-Access Survey (n=262)
(1) Not (2) Somewhat
Important
(3) Important
(4) Very
Important
96
Figure 8. Mean Level of Access Frequency of Quality Information from the FDA
Website

The difference between the two surveys determined to be statistically significant for FDA
field corrective actions (p = 0.01), FDA drug shortage index (p = 0.01), and FDA quality
training programs or educational material (p = 0.008).

Other sources of publicly available information beyond those on the FDA website
were also consulted frequently as indicated by selections from a list of eight choices
(Table 8). A preponderance of respondents from the expert panel consulted information
compiled by professional organizations (84%) and trade and industry groups (79%),
educational resources like journals and white papers (69%), and information from other
government and health agency websites (68%). The open-access group appeared to favor
information from trade and industry groups (71%) and more commonly used information
from company websites (61%) as well as “word of mouth” (44%) than the expert group
did, but tended to use the other resources less commonly than the expert group did.
3.5, SD 1.2
3.6, SD 1.1
2.7, SD 1.2
2.5, SD 1.3
3.2, SD 1.4
2.4, SD 1.4
2.8, SD 1.4
2.9, SD 1.2
2.8, SD 1.3
3.3, SD 1.3
3.6, SD 1.2
3.1, SD 1.4
2.7, SD 1.4
3.0, SD 1.5
2.0, SD 1.4
2.8, SD 1.3
2.8, SD 1.2
3.2, SD 1.3
1.0 2.0 3.0 4.0
FDA Form 483 Observations
FDA Warning Letters
FDA Field Corrective Actions
Company Injunctions Information
FDA Drug Recall, Withdrawals, Safety-…
FDA Drug Shortage Index
Company Response Letters
FDA Data and Trend Analysis (e.g.,…
FDA Quality Training Programs or…
Expert Panel (n=104) Open-Access Survey (n=242)
(1 = Never, 2 = Yearly, 3 = Quaterly, 4 = Monthly, 5 = Weekly, 6 = Daily)
97
Information sources indicated as “other” included various professional discussion groups,
newsfeed services, conferences and meetings, and other internet based groups and blogs.

Table 8. Public Sources other than FDA Used To Obtain Information on Product
and Company Quality
Expert Panel
Open-Access
Survey
Description of Source No. %

No. %
Trade or Industry Group
(e.g. PhRMA, BIO, EFPIA , AdvaMed )

84 79 174 71
Pharmaceutical Professional Organizations
(e.g. ASQ, ISPE, PDA, MDDI)

89 84 127 52
Educational Resources
(e.g. journals, white papers, universities)

73 69 148 60
Other Governmental Websites
(e.g. WHO, EMA)

72 68 105 43
Drug Index, Formulary, Supplier Listings
(e.g. orange book, drugs.com, QMED)

30 28 44 18
Company Websites

49 46 150 61
Public Reputation (i.e. word of mouth)

38 36 108 44
Other
1

22 21 31 13
Total Responses
2
106 100 246 100
1
Information sources indicated as other included various professional discussion groups,
newsfeed services, conferences and meetings, and other internet based groups and blogs.
2
Total responses identify sample size per question. Respondents were allowed to select
all that applied resulting in increased counts per choice selection.

Other information options were solicited through open-text responses. The 57 EP
and 56 OA respondents to this option suggested that they also have used legal actions
(e.g., records of debarments, law suits, and fines), new drug application outcomes (e.g.,
98
refusal to file notifications, rejections, complete response letters), medical device
databases (e.g., MDR, MAUDE, MedSun databases), and drug safety databases (e.g.,
AERS) (see Appendix E for a complete listing of the responses).
4.2.2 Non-Public Proprietary Information
Manufacturers commonly use internal company or industry metrics when trending
their own data and systems, as demonstrated in Figure 9. Especially popular metrics for
both groups were noncompliance trend reports and customer complaints, but most of the
seven suggested options were identified as metrics collected by at least half of the
respondents. The least commonly selected option was that of correlating the 483 reports
with warning letters, identified by less than half of the expert panel and by only about
one-quarter of the open-access group. Metrics that were indicated as “other” included
process performance measures (e.g., number of rejects, out of specification events,
invalidated results, lot rejections, lot acceptance rates), internal quality audits, product
risk assessments, and health agency outcomes (e.g., first cycle approvals, inspection
observations, post-approval commitments).

99
Figure 9. Metrics that Best Described the Types of Quality Performance Analysis
Companies Commonly Use

When asked what metrics their company might hypothetically be willing to share
to help improve industry-wide quality under conditions of anonymity, relatively little
enthusiasm was apparent for sharing any of the nine metrics suggested in the question.
The metrics selected most commonly by the expert panel were related to be those
associated with raw material suppliers (38%) and complaint rates (37%), but even these
were chosen by about one-third of respondents. The open-access group expressed more
willingness to share metrics associated with the number of corrective actions taken
(43%), but also suggested complaint rates (39%), and raw material supplier metrics
(38%) (Figure 10). The metrics that companies would be least willing to share publicly
were rates on product rejection (26%) and returns (18%) by the expert panel, and rates on
75%
42%
55%
59%
64%
67%
58%
20%
68%
27%
46%
70%
65%
80%
68%
12%
Non-Compliance Trend
Reports
Correlation Between 483s
and Warning Letters
Number of Recalls or Market
Withdrawals
Number of Corrective
Actions Taken
Product Defect Rates
Customer Complaint Rates
Adverse Event Rates
Other(s)
Expert Panel (n=96) Open-Access Survey (n=212)
100
product rejection (20%) and manufacturing errors (17%) by the open-access group. An
open text box labeled as “other” was meant to capture additional metrics that respondents
may be willing to share, but the most common answers for both groups (EP 32%, OA
16%) were either that they were not in a position to answer this question or that no
information could be currently shared by the company.

Figure 10. Metrics that Companies May Be Willing to Anonymously Share for
Industry Quality Improvement Efforts

When the respondents were queried about the degree to which they had
experienced quality problems such as manufacturing stoppages, failed purchase orders,
product shortages, product recalls, or resulting financial losses, most indicated that all of
37%
31%
31%
28%
32%
28%
18%
26%
38%
32%
39%
33%
30%
43%
23%
17%
23%
20%
38%
16%
Complaint Rates
Adverse Event Rates
Non-Conformance Rates
Number of Corrective Actions…
Process Performance Indicators…
Manufacturing Error Rates and…
Products Return Rates
Product Rejection Rates
Raw Material Supplier Metrics
Other(s)
Expert Panel (n=65) Open-Access Survey (n=114)
101
these outcomes had been experienced but frequency of occurrence varied from quarterly
to up to 5 or more years apart. However, many respondents also indicated that they had
"never” experienced these outcomes (see Appendix F for tabulated data). Additional
quality metrics that expert panel (n= 34) and open-access (n=40) respondents reported as
used by their employer or important to consider in an associated comment field ranged
from specific process performance measures and product attributes to more general
statements on what focus regulators should have to better control quality (see Appendix
G full the listing of text responses). However, no apparent commonalities or themes
could be identified. Selected responses among the comments provided for both public
and internal quality metrics from Appendices E and G are shown below.
“Opportunities for Improvement, Continuous improvement progress/effectiveness,
CAPA metrics; essentially any data that a company can share they indicates they
are aware and on a path of continuous improvement”

“Quality root cause analysis outcomes are a good measure that the quality system
is working, knowing the reasons behind failures or NCs are valuable to see if they
were within control of the company.”

“None, FDA pushing quality metrics moves companies to focus on the regulations
more than the quality of the product, finding root cause of production issues is the
problem not Quality Assurance. Need to go back to the basics ….”

“There are lot of other quality metrics that if public will be an added value:
quality culture index of the company; company performance in terms of leading
and lagging indicators, such as right first time rate; complaints rate, confirmed
OOS rate, invalidated OOS rate, GMP training hours, Stability failures rate,
unplanned down time equipment, external audits per year supply chain index,
process capabilities, number of batches per year, staff rotation, ...”.

“I would like to see the FDA assign letter grades to plants and products or to
issue a report similar to what Consumer Reports does for each product.”
102
4.3 Views on Quality Ratings
The personal views of respondents were probed with regard to ways in which a
quality rating system might impact company and consumer decision-making. The
questions were prefaced by stating,
“Many stakeholders, including regulatory agencies and pharmaceutical trade
groups, have suggested that quality differences between products and firms
should be made more visible to the public. One way that quality has been made
visible in other industries is by the use of scorecards, rankings or grading
schemes based on metrics. The idea of issuing a quality grade or scorecard is to
provide a simplified measure that can be easily communicated and interpreted by
stakeholders…”

Results in this section are organized as follows: 1) the impact that quality ratings may
have on quality improvements and company practices, 2) views on the selection and
presentation of a quality rating, and 3) other policy views and concerns regarding quality
ratings.
4.3.1 Impact of Ratings on Company Quality Improvements and Practices
Respondents in both surveys indicated a high level of agreement that quality
scorecards would encourage manufacturers and consumers to place a higher value on
quality (Figure 11). Given a series of three statements regarding the usefulness of
publicly available scorecards, the greatest agreement was found on the statement that
scorecards would “encourage manufacturers to place a larger focus on improving quality”
(EP 𝑥 ̅ 4.2, OA 𝑥 ̅ 4.1). They also typically agreed that quality scores would “encourage
customers to prefer products of higher-quality according to the assigned grade”
(EP 𝑥 ̅ 3.8, OA 𝑥 ̅ 3.7) and “help consumers make more educated decisions on product
purchases” (EP 𝑥 ̅ 3.6, OA 𝑥 ̅ 3.7).
103
Figure 11. Mean Level of Agreement on Select Consumer and Company Quality
Valuations Influenced by Scorecards

Respondents were asked to rate the level of “impact” that a quality scorecards
might be predicted to have on their company’s decisions to undertake different types of
quality improvement activities. As shown in Figure 12, for both groups, the strongest
effects were predicted to occur at the level of internal and supplier quality audits (EP 𝑥 ̅
3.4, OA 𝑥 ̅ 3.1), employee training (EP 𝑥 ̅ 3.1, OA 𝑥 ̅ 3.1), and increasing the centrality of
quality as a strategic goal of the company (EP 𝑥 ̅ 3.4, OA 𝑥 ̅ 3.2). The impact of a
scorecard on decisions to increase investments in analytical (EP 𝑥 ̅ 2.8, OA 𝑥 ̅ 2.7) and
manufacturing technology (EP 𝑥 ̅ 3.0, OA 𝑥 ̅ 2.9) was rated as potentially having a slightly
smaller impact. Least impact was predicted with regard to decisions to open additional
manufacturing sites (EP 𝑥 ̅ 2.2, OA 𝑥 ̅ 2.3).

3.6, SD 1.2
3.8, SD 1.1
4.2, SD 0.9
3.7, SD 1.1
3.7, SD 1
4.1, SD 1
1.0 2.0 3.0 4.0 5.0
Help consumers make more educated
decisions on product purchases
Encourage customers to prefer products
of higher-quality according to the
assigned grade
Encourage manufacturers to place a
larger focus on improving quality
Expert Panel (n=93) Open-Access Survey (n=182)
(1)
Strongly
(2)
Disagree
(3)
Neither
Agree or
Disagree
(4)
Agree
(5)
Strongly
Agree
104
Figure 12. Mean Impact Quality Scorecards May Have on Company Decisions

The difference between the two surveys for enhancing internal and supplier quality audits
was determined to be statistically significant, p = 0.007.

4.3.2 Selection and Presentation of the Quality Rating
Respondents were asked to suggest the quality metrics that they felt to be most
useful for their customers to identify product quality and reliability from a selection of
eleven options. The top metrics selected by the expert panel were the number of FDA
actions (74%), complaint rates (70%), and a company’s health agency track-record (66%)
(Figure 13). The open-access group typically had fewer responses in each category; thus
the most commonly identified metric, complaint rates (62%), was chosen less often than
the third-ranked selection of the expert panel. Other more popular metrics selected by the
open-access group were recall rates (60%) and the number of FDA actions taken (54%).
2.2, SD 0.9
2.8, SD 0.9
3.0, SD 0.7
3.4, SD 0.8
3.4, SD 0.8
3.1, SD 0.8
3.4, SD 0.8
2.3, SD 0.9
2.7, SD 0.8
2.9, SD 0.8
3.1, SD 0.9
3.1, SD 0.9
3.1, SD 0.8
3.2, SD 0.8
1.0 2.0 3.0 4.0
To open additional manufacturing sites
To increase investment in analytical
technology
To increase investment in manufacturing
technology
To enhance internal quality audits
To enhance supplier quality audits
To increase employee training
To make quality a more strategic
company goal
Expert Panel (n=94) Open-Access Survey (n=177)
(4)
Strongly
Impact
(3)
Some
Impact
(2)
Little
Impact
(1)
No
Impact
105
The “number of back-up manufacturing sites” (EP 30%, OA 18%) and amount of
“emergency inventory held” (EP 26%, OA 12%) were chosen least often by both groups.
Those responses entered as “other” primarily stated “none” or indicated that it would be
difficult to make such comparisons by any means.

Figure 13. Quality Metrics Believed to be Most Useful for Customers to Determine
Product Quality and Supply Reliability

Respondents were also asked to identify what they “perceived to pose the most
challenges in interpretation of a manufacturing quality grade” (Figure 14). Responses by
both sets of respondents were similar. Most commonly they feared that consumers might
misinterpret a grade as an assessment of overall product safety (EP 76%, OA 68%) or a
measure of clinical benefit (EP 54%, OA 55%). Less frequently chosen were options that
70%
60%
48%
51%
26%
43%
74%
66%
30%
47%
36%
11%
62%
53%
50%
43%
12%
31%
54%
41%
18%
40%
36%
6%
Complaint Rates
Recall Rates
Manufacturing Error and Failure Rates
Measure of Manufacturing "Right…
Emergency Inventory Held
Number of Shortages Experienced
Number of FDA Actions (483s,…
Health Agency Inspection Track-…
Number of Back-up Manufacturing…
Measure of Product Robustness
Adverse Event Rates
Other(s)
Expert Panel (n=91) Open-Access Survey (n=177)
106
an otherwise safe product would not be used because of its lower grade or that the grade
would be ignored by consumers. The selection of “other” provided a few responses
suggesting that consumers would not understand or know how to use the grades, and that
expense and legal liabilities would rise as a result of the use of grades. One respondent
however, stated strongly the view that all of the concerns are only challenges to
overcome and serve as “excuses” for not issuing grades.

Figure 14. Perceived Challenges with the Interpretation of Quality Grades

If quality ratings were to be issued, respondents on the expert panel most
commonly chose an alphabetical grade (32%) from a selection of five options, the rest of
which were chosen with similar frequency (Figure 15). Open-access respondents
appeared not to favor one method over another so clearly; they chose the options of a
76%
31%
54%
27%
4%
68%
41%
55%
19%
6%
A manufacturing quality grade will be
misinterpreted as an assessment of overall
product safety
An otherwise safe product would not be used if
assigned a lower grade
The manufacturing quality grade would be
mistaken as a measure of clinical benefit of the
product
Consumers would generally not understand the
grade and would just ignore it
Other
Expert Panel (n=93) Open Access Survey (n=170)
107
graded numerical value (23%), a definitive numerical value (22%), or an alphabetical
grade (21%) with approximately equal frequency. Both groups appeared least
enthusiastic about a color card, chosen by only 10% of respondents in each group. As a
response to the category of “other”, some respondents also suggested that a simple
“Pass/Fail” should be the only criterion.

Figure 15. Views on the Best Way to Display a Quality Rating

When asked about the amount of information that consumers would need to interpret a
quality grade, most respondents believed that the quality grade alone (EP 77%, OA 70%)
and a description of the key metrics associated with that grade (EP 72%, OA 54%) would
be most informative (Figure 16). However the other choices, including the reason why a
32%
16%
22%
10%
14%
6%
21%
22%
23%
10%
18%
6%
Alphabetical
Grade
(A,B,C)
Definitive
Numerical Value
(1-100%)
Graded
Numerical Value
(like a credit
rating)
Color Card
(Green, Yellow,
Red)
Numerical
Ranking Among
Industry
(like hospital
rankings)
Other
Expert Panel (n=93)
Open-Access Survey (n=175)
108
lower grade was issued, a grade for each metric in the rating system, and the date of when
the grade was given, were also selected by at least one-third of respondents in both
groups. Responses suggested in the text box for “other” mostly included “all of the
above”. One respondent suggested that the grade should have a specified term of
validity. A few respondents voiced disagreement with any policy to display quality
ratings publicly; the reader is referred to the results shown in Table 11 for a more detailed
account on views of this kind.

Figure 16. Information Consumers May Need to Know About a Quality Ranking

When asked how the quality rating might be communicated, respondents more
frequently selected the option of posting the grade on a regulatory agency website as their
77%
54%
46%
34%
48%
8%
70%
72%
38%
46%
45%
4%
Quality Grade Description of
Key Metrics
Associated with
the Grade
Description on
Why a Lower
Grade Was
Issued
Grade
Determined for
Each Key Metric
Date of When
Grade was
Issued
Other
Expert Panel (n=92)
Open-Access Survey (n=175)
109
first choice and the option of incorporating the grade into a drug formulary or index as
their last (Table 9).
Table 9. Views on Most Effective Way to Communicate a Product Quality Grade
Method of Communication
Rank Order Selected
1 2 3 4 5 Mean
(SD)

Expert Panel

Display on the Product Label
(similar to a CE mark)

22 10 15 26 8 2.85
(1.39)
Display on Product Instructions
for Use

3 9 15 26 28 3.83
(1.14)
Disclose on Regulatory Agency
Website

40 16 10 14 1 2.01
(1.20)
Post on Company Website

11 27 20 11 12 2.83
(1.26)
Incorporate into Drug Index or
Formulary Listings

5 19 21 4 32 3.48
(1.38)
Open-Access Survey

Display on the Product Label
(similar to a CE mark)

45 19 28 28 28 2.83
(1.51)
Display on Product Instructions
for Use

2 35 32 52 27 3.45
(1.08)
Disclose on Regulatory Agency
Website

60 29 22 24 13 2.33
(1.38)
Post on Company Website

33 55 28 19 13 2.49
(1.22)
Incorporate into Drug Index or
Formulary Listings
8 10 38 25 67 3.90
(1.21)
Shaded cells indicate overall rank order based on the mean

110
4.3.3 Other Policy Views and Concerns Regarding Quality Ratings
Respondents were asked about the organization(s) best suited to issue quality
ratings. Most identified that a regulatory agency would be best suited to issue the grade
(Table 10). However, respondents in the open-access group chose the alternative option
of a private organization (e.g., JD Edwards) with frequencies that gave an average score
equal to that of the regulatory agency ( 𝑥 ̅ 2.81 and 2.82, respectively), despite its second
place when viewed by modal scores. The majority of both sets of respondents agreed that
public disclosure should be mandatory if grades were issued (Table 11). However
dissenting views were also expressed. These included: the potential for increased costs
for manufacturers; unwarranted increase in government intervention; concerns of driving
bad behavior to “game” the system; conflicts of interest for a regulatory agency; misuse
or misinterpretation of the grade; and worries regarding inequitable grades and metrics
across product and company types that may unduly and irreversibly tarnish the
reputations of some organizations over others. Table 11 provides a selection of responses
to illustrate their concerns in their own words.
111
Table 10. Views on the Best Organization Suited to Evaluate Quality and Issue
Ratings
Organization
Rank Order Selected
1 2 3 4 5 6 Mean
(SD)

Expert Panel

Regulatory Agency
(e.g. FDA)

46 14 15 3 2 4 1.96
(1.37)
Independent Gov’t Agency

6 38 12 21 4 3 2.86
(1.22)
Private Organization
(e.g. JD Edwards)

16 12 18 14 14 10 3.33
(1.65)
Industry Trade Group

10 12 20 18 16 8 3.50
(1.49)
Other Non-Profit Organizations

3 15 13 17 34 12 4.31
(1.27)
Consumer Groups

3 3 6 11 14 47 5.04
(1.37)
Open-Access Survey

Regulatory Agency
(e.g. FDA)

62 14 22 20 24 14 2.82
(1.79)
Independent Gov’t Agency

7 54 31 25 23 16 3.33
(1.45)
Private Organization
(e.g. JD Edwards)

45 27 33 22 21 8 2.81
(1.56)
Industry Trade Group

21 21 31 39 26 18 3.53
(1.54)
Other Non-Profit Organizations

12 21 21 27 43 32 4.05
(1.57)
Consumer Groups

9 19 18 23 19 68 4.46
(1.67)
Shaded cells indicate overall rank order based on the sample mean.
112
Table 11. Views on Mandated Public Disclosure of Quality Ratings
Expert-Panel Open-Access Survey

If grades were issued to firms, do you believe it should be mandatory to disclose them
publically?

Yes- 67%, No- 33% (n=91)

Yes- 68%, No- 32% (n=177)

If possible, briefly explain your concerns with mandatory disclosure?
1

“Slippery slope, possibly too much
government intervention.”

“Company should decide what to make
public.”
“In the case of an atypical scoring, a
company should be allowed to correct the
actions/deficiencies without public
disclosure as the reputation of the
company could be compromised
permanently.”

“Mandatory disclosure can have
considerable consequences and should
not be implemented until the system has
most of the bugs worked out within
industry.”
“The consumers are not trained to
interpret and understand quality
grades/rankings.”

“A general public expectation is that all
products are manufactured with quality.
The general public does not understand
cGMPs and what quality standards exist.”

“I would not want the grades to be
immediately made public, I think there
would need to be time to review grades
and meanings …to ensure grading is
meaningful before making public.”

“Correcting errors in the process my take
time for a company to challenge any
errors in the process when a rating
number is incorrect.”
“Very easy for the public to misinterpret a
quality rating.”

“The public is not versed and understand
the meaning behind regulatory findings
and what that impact is to product
quality.”

“conflict of interest”

“Depends on the independence of the
grading system and frequency with which
it is updated”

“Mandatory disclosure would result in
ineffective/ incomplete details influencing
market decisions… would indirectly serve
as a marketing tool.”
“Mandatory disclosure may be used for
marketing purposes and will defeat the
purpose of product quality improvement”
1
Selected comments from respondents that choose “No” to the prior question.
113
An open text-box question was placed at the end of the survey so that respondents
could express any other challenges or concerns related to the potential issuance of quality
ratings generated a large number of responses from both the expert panel (n = 58) and the
open-access group (n = 71). These comments ranged from single-word answers such as
“none” and “fairness” to narrative responses of up to 200 words. Responses could be
grouped into general themes as follows:
• Concerns that quality metrics would not be equitable, fair, or representative of
quality
• Views that quality ratings will not improve quality but only increase costs or lead
to unwanted consequences and behaviors by some manufacturers
• Views that the need for quality improvement or metrics is not required or justified
• Concerns that consumers are not in a position to make decisions regarding quality
Selected responses categorized into these themes are shown in Table 12 and Table 13, for
the expert-panel and open-access groups respectively.

114
Table 12. Thematic Views and Concerns on Quality Ratings (Expert Panel)
Thematic Views (Selected Open Text Responses)

Concerns that quality metrics would not be equitable, fair, or representative of quality

“Inability of the scorecard to recognize differing benchmarks or metrics for complex
versus simple products”

“A grade for a manufacturing facility might be isolated to one area or product, so a
CMO manufacturing hundreds of products could have an isolated incident that impacts
every product. You would need a product specific score to overcome this issue.”

“There are many different size and product variety of pharm mfg companies. The
scorecard would have to be either really simple to capture acceptable quality from
questionable quality or would need to be very complex formula if more refined grades
are needed.”

“It will be harder for smaller company to get a better rating than a larger company.
Smaller companies have lesser finances to invest into quality. Larger companies have
more finances for investment. They manufacture multiple products using multiple site.
So an issue for one product may be diluted out in the overall rating for a larger
company. Overall this gives larger companies advantage over smaller companies.
This would create an even larger hurdle for smaller companies to get something to
market.”

“Focus becomes on the grade (teach to the test) and not on true quality manufacturing
of product. We see cultures around the world that do not have same understanding of
quality product plus we live in a tribal industry hence we do not share to assist all in
providing the very best product to the patient.”

Views that quality ratings will not improve quality but only increase costs or lead to
unwanted consequences

“Misinterpretation of quality metric and amount of money companies need to spend on
advertisement for education of consumers and health care providers”

“Cheating of bad actors and spinning their numbers, the focus on certain metric would
likely lead to unintended consequences”

“I think it would be difficult to make positive changes once a grade had been assigned
to a product. The first grade assigned would be stuck with the product, and possibly
the company forever.”

“A safe drug product may be more efficacious than other products with same API, but
due to lower quality rating physicians/patients would not select.”

115
Thematic Views (Selected Open Text Responses)

Views that the need for quality improvement or metrics is not required

“I actually think a score is not necessary. The existing audit requirements by major
health authorities is sufficient (PAI, every 2 years, etc.). I think the major health
authorities should begin to mutually recognize inspections.”

“Undermines the "approval" of a product or the GMP status of a facility”

“Quality is defined by specific product specifications agreed with the applicable
regulatory agency. Therefore the impurity / purity of a product will vary from one
manufacturer to another and hence the need for a unique specification. There are
indeed some common attributes for the same but there are also differences. All are
approved by the applicable regulatory agency and are therefore considered suitable
for use in the market place.”

“More paperwork. Aren't customer audits enough if they are worried about individual
quality standards? There are also audit reports available from individual auditing
bodies. Why do we need another, especially if it is a simple grade with no detail?”

Concerns that consumers are not in a position to make decisions regarding quality

“Biggest challenge is that third party payers focus on price alone. Consumers are not
often given the choice of supplier. As long as the supplier is not under import ban,
then they will be used. Knowing that one is receiving products from a less than A
grade supplier may increase customer concern but not necessarily influence
decisions.”

“Driver for selection of manufacturer is cost. Insurance reimbursement plans are only
cost driven and patient choice for quality of manufacturing will not override the cost
negotiations with the insurers. Unless the insurance providers are included in the
'quality' value project, this effort is unlikely to have significant success when cost
control is the most important parameter.”

“Consumers generally do not understand quality, only price.”

“I think a key issue would be how a consumer would use this information. Defining
quality for a pharmaceutical product can be challenging because mostly consumers
will want to know if the product works. For example, if I have a headache, will this
product make it go away and how quickly will it work. Unfortunately, the answer to
this question will differ from patient to patient and may be independent of the "quality"
of the product. Unless this is done in a way that the consumer gets a clear and fair
understanding of a product's quality and how this relates to the efficacy of the product,
companies are unlikely to embrace this concept.”

116
Table 13. Thematic Views and Concerns on Quality Ratings (Open-Access)
Thematic Views (Selected Open Text Responses)

Concerns that quality metrics would not be equitable, fair, or representative of quality

“Grades are a good idea but a lot of time quality systems do not actually indicate the
quality of the product. Quality systems can bog down work so that a quality product is
not achieved or there is no room for improvement. Also, financial situations of a
company should be assessed as a small company will not have the resources of a large
company”

“Determining and implementing a uniform and unbiased grading system”

“There needs to be careful thought as to what is measured and how scores are
calculated. The measurements have to be quality indicating and independent of
company size and worth. The little guys have to have as much worth as the big guys.
Consumers and the industry has to know that the score is really indicative of the
quality”

“It is difficult to manage public perception when improvements have been made and
prior grade is still published, or when issue is found but has no effect on product
quality”

Views that quality ratings will not improve quality but only increase costs or lead to
unwanted consequences

“Employees may feel pressured when the company doesn’t score high”

“Any product not given the highest rating would not be purchased. Therefore, every
product would have an "A"... and the rating would have no value, since they all had
"A's". There would be no differentiation. A numerical score wouldn’t really solve this
issue because customers wouldn’t understand the difference between a 94 and a 97,
and would likely tie it to product effectiveness.”

“Understanding how our "quality" compares to other companies that have more or less
complex processes. Since we are the only source of what we make, how will it affect
what customers think? It just adds to overhead.”

“Companies may tend to focus too much on their quality score, at the expense of true
quality (kind of like "teaching to the test" in K-12 schools).”

I have seen metrics drive bad behavior - be careful how you design this program
because you don't want to cause folks to be dishonest in order to obtain a good score.
They should not operate that way, but there are people out there who do, especially
when there is a potential for them to lose or gain business as a result.

117
4.4 Cross-Tabulation Analyses of Demographic Variables
Cross-tabulations were performed within each survey group to find potential
relationships between the respondents’ demographic profiles and their views on quality
metrics and ratings. Categorical responses provided in the demographic questions that
represented at least 10% of the sample were considered amenable for cross-tabulation.
Chi-square statistics were used to measure the significance of any observed relationships;
those with a p < 0.2% are identified here, but the limitations of using such statistics on
relatively small groups were kept in mind.
4.4.1 Company Size and Type
Company Size
For the expert panel, the largest potential difference was observed for the
question, “Please indicate the impact you believe a quality scorecard will have on the
following company decisions”. Those working at smaller companies (< 1000 employees)
rated the scorecard to have slightly more impact (means approximated 3.6) on increasing
employee training than did those working at larger companies (> 1000 employees) who
rated it lower (means approximated 3.1) [x
2
(18, N = 93) = 23.17, p = 0.18]. For the
open-access survey, the largest potential difference was observed for the question, “What
would be the best way to reflect a quality rating”. Those working at small to medium
sized companies (< 5000 employees) chose a “numerical ranking among industry” less
frequently (5% – 20%) than those working at large companies (> 5000 employees; 44%)
[x
2
(30, N = 175) = 38.09, p = 0.15].

118
Company Type
For the open-access survey, a significant difference was seen for the question
asking respondents to rate the level of importance of eight types of information available
through the FDA website. Those working for medical device companies placed a higher
mean value of importance on FDA field corrective actions than did those working for
innovator pharmaceutical companies ( 𝑥 ̅ 3.6 versus 3.4, respectively) [x
2
(33, N = 261) =
62.89, p < 0.001]. Conversely, those working for innovator pharmaceutical companies
place a higher mean value of importance on the FDA drug shortage index than did those
working for medical device companies ( 𝑥 ̅ 2.5 versus 2.2, respectively) [x
2
(33, N = 251)
= 49.26, p = 0.03], but overall both of these groups rated the drug shortage index of lower
importance than all the other selections.
4.4.2 Regional Location
For the expert panel, the largest potentially meaningful differences were observed
for the question, “Please indicate the impact you believe a quality scorecard will have on
the following company decision”. First was the impact that scorecards were believed to
have on decisions to open additional manufacturing sites; those working at companies
primarily located in West and East, USA rated, on average, the impact to be higher ( 𝑥 ̅ 2.4
and 2.0 respectively) while those whose primary location was Europe provided, on
average, a slightly lower rating ( 𝑥 ̅ 1.7) [x
2
(21, N = 94) = 28.44, p = 0.13], but overall
both of these groups predicted a lower impact on decisions to open new manufacturing
sites than all the other selections. The second observation concerned the impact that
scorecards might have on decisions to enhance internal quality audits, where those
119
working at companies located on West Coast, USA rated, on average, the impact to be
higher ( 𝑥 ̅ 3.6) while those in Europe provided a slightly lower rating ( 𝑥 ̅ 3.0) [ x
2
(21, N =
94) = 26.36, p = 0.19].
4.4.3 Occupational Background
For the expert panel, a potentially meaningful difference was observed for the
question, “If quality grades were issued to firms, do you believe that it should be
mandatory to disclose them?, where the respondents could only answer “Yes” or “No”.
Respondents working as quality managers were evenly split on whether or not disclosure
should be mandatory, however the majority of research and development scientists (67%)
indicated that disclosure should be mandatory, whereas those working in regulatory or
legal affairs felt strongly (86%) that disclosure should be mandatory (see Table 14).

120
Table 14. Cross Tabulation of Respondents Occupation and Mandatory Public
Disclosure of Quality Ratings (Expert Panel)
If quality grades were issued to firms, do you believe it should be mandatory to
disclose them publicly?
Primary Occupation Yes No Row
Total
Manufacturing Engineer or Scientist

2 (100%) 0 2
Research and Development Scientist

8 (67%) 4 (33%) 12
Manufacturing Operations Management

3 (50%) 3 (50%) 6
Quality Management

18 (50%) 18 (50%) 36
Supply Chain Management

0 0 0
Regulatory or Legal Affairs

21 (86%) 3 (13%) 24
Business Consultant

3 (100%) 0 3
Other

6 (75%) 2 (25%) 8
Total Responses 61 (67%) 30 (33%) 91
Chi-square

= 12.75, df = 7, p = 0.08. Shaded rows indicate categories were the sample
sizes that comprised at least 10 percent of population surveyed for statistical analysis.
121
CHAPTER 5: DISCUSSION
In the literature, little research has been directed at understanding the views of
industry with regard to the use of quality metrics to benchmark their activities. Further,
little was written on the way that manufacturers might use quality ratings in decisions to
improve their practices. Thus, in this work an exploratory research methodology was
used to gain insight into industry views on the use of different quality metrics and ratings.
It further explored the potential impact that quality ratings might be predicted to have as
an incentive for investments in quality. However, because this study was a first probe of
the topic, certain delimitations and limitations must be borne in mind when interpreting
these data.
5.1 Considerations of Study Methodology
One of the delimitations placed on this study was to confine the respondent
population to individuals employed by medical product manufacturers, primarily
pharmaceutical drug and medical device companies. This was done because
manufacturers are active in developing metrics and have expertise in medical product
manufacturing and quality management. However, sampling from such a population is
not simple, so two different methods were used to gain feedback. For the expert survey,
the use of nonprobability sampling techniques, by recruiting individuals known through
professional networks, conferences, professional associations and snowball referrals, has
the potential limitation that views might be more homogeneous than might be expected of
a truly random sampling (Atkinson & Flint, 2001). The responses of this panel could
represent a biased point of view because the participants might be like-minded
122
individuals who may have been influenced by their attendance at common meetings or
workshops, or by their professional associations. Nevertheless, this approach was seen as
important in order to assure the engagement of busy professionals who otherwise might
have been reluctant to participate in the survey. Snowball sampling techniques are
reputed to be particularly effective in encouraging hard-to-reach “closed populations” to
participate in surveys and therefore seemed fitting to use for this study (Atkinson & Flint,
2001). When reviewing the findings it is therefore important to remember that the
sample has been restricted to attain an “expert opinion” that has the potential to be
unbalanced (Sue & Ritter, 2012). Further, given the global nature of today’s work
environment, one may postulate that most respondents at the level of the identified
respondents will have experience with global operations and therefore may express views
similar to those expressed by quality experts more globally. However, such
interpretations should be made cautiously because this study was delimited to focus on
US regulatory and commercial environments and these can differ in certain respects from
the environments of companies abroad.
Survey methods also appeared to offer the greatest opportunity to scan the views
of many individuals in a short period of time. Although other exploratory methodologies
such as brainstorming sessions or face-to-face interviews could have potentially been
used, online surveys are known to be more effective at collecting a large number of
responses from a geographically dispersed population (Sue & Ritter, 2012). Within the
United States, pharmaceutical firms can be found clustered in parts of the West Coast,
Central/South Coast, and East Coast. Further, the medical product industry is segmented
with respect to product types. Boston and San Francisco, for example, are known as
123
“biotechnology hubs” whereas Pennsylvania, New York and New Jersey are home to the
larger traditional pharmaceutical companies (Diehl, 2015; ITA, 2010). Manufacturers
headquartered overseas also often have operations within the United States, extending the
geographical footprint of available experts in this field. Thus, brainstorming sessions and
interviews were seen to be problematic for this study because such methodologies would
have likely reached only a small pool of individuals in restricted geographical areas who
would likely be even further from representing the parent population than the survey.
A survey was also seen as important to provide the respondents with a forum
where they could express potential problems or unpopular views openly and honestly
without repercussions to their companies and their employment status. Some of the
questions posed in this survey probed issues that might be considered as sensitive or
proprietary, and that are typically not discussed unless requested by FDA. This is
important to consider when evaluating the results because some participants may have
felt apprehensive about the extent to which they could divulge information that is not
already in the public domain. Particular caution may be exercised by respondents who
were told in the survey introduction that representatives of the FDA had been involved
with the development and distribution of the survey. Further, in the survey of the expert
panel, it is theoretically possible to identify the origin of different survey responses,
although care was taken not to exercise this option. The online survey format used for
the open access group presented the advantage that anonymity could be assured
completely for respondents who might need to feel safe before they openly express their
views (Sue & Ritter, 2012). The importance of the anonymity, and by consequence
privacy, provided by online surveys is supported by Loosveldt and Storms (2008) who
124
finds that privacy reduces survey measurement error because without it respondents tend
to give more “socially desirable answers” or may limit the information given since they
“do not know what will happen with their responses.”
As with any methodology, surveys have disadvantages and care must be taken
when conducting the research to remedy these potential shortcomings to the greatest
extent possible. Two principal areas of concern that are often scrutinized are the internal
validity of the survey itself and the subsequent external validity (representativeness) of
the results (Check & Schutt, 2011; Sue & Ritter, 2012). Developing survey questions
that are appropriate and that can obtain the sought after information is fundamental to
assuring a satisfactory level of internal validity. One method used to challenge the
internal validity of the survey instrument in this study was to collaborate with the USC
International Center for Regulatory Science and the QLN who provided feedback on the
content, flow, and clarity of the questions. An additional measure was taken to pilot a
near-final survey with a subset of members in the QLN as a final assessment of its
content validity. This practice, using experts and focus groups to develop the survey
instrument prior to its administration, is becoming a recommended practice in order to
improve content validity of a survey (Sue & Ritter, 2012). It has been found to be
particularly advantageous for regulatory science research since it is effective in not only
increasing the face validity of the survey instrument but also in guiding sample selection
and minimizing personal bias that may be inadvertently introduced by the researcher
(Jamieson, 2011; Storm, 2013).
One way to gain insight into the external validity of the results in a study such as
this is to compare the responses obtained in one survey to those of a matching survey
125
administered to a different subgroup within the same parent population. Thus an
open-access survey was also administered so that the results could be compared against
the expert panel. The open-access group was recruited from associates and contacts of
the QLN, a diverse population of FDA-regulated industry professionals working in
different sectors and at different job levels. Because most members of the QLN were
located in Southern California, the respondents recruited by them were more strongly
associated with the local industry that is enriched with small to medium size companies
and medical device manufacturers (BIOCOM, 2012). It is possible that industry practices
in California differ from that elsewhere in the United States, and this might be reflected
in dissonance found in the responses of the two subgroups. Like the expert panel, the
open-access group may have also been influenced by the professional organizations with
which they were identified. Perhaps surprising, then, was the high degree of congruence
seen between most responses provided by these two rather different groups. Where
differences seemed apparent, for example in the preference of open-access respondents to
view the number of FDA or company-initiated field corrective actions as a metric of
quality, these differences might be related to the nature of manufacturing and support
practices associated with device rather than pharmaceutical operations. An exploratory
survey of this type is only able to point to areas in which further exploration might be
warranted. However, the relatively similar overall response patterns from one survey to
the other suggest that views amongst subpopulations are not greatly different in most
respects and further suggest a certain amount of external validity at least across US
industries.
126
Obtaining a healthy response rate to a survey is often another challenge since the
representativeness of the results can be questioned if the survey response is too low.
Although recent studies have indicated that response rates of online surveys may
approach those of mailed surveys for respondents who are computer literate, the ability to
assure an adequate response rate is still seen to be more challenging with online surveys
(Cook, Heath, & Thompson, 2000; Kaplowitz, Hadlock, & Levine, 2004). In this study,
the expert panel survey obtained a 36% response rate with a survey completion rate of
85%. This response rate is within the mean range of 24% – 40% of online surveys that
have been thought to be adequate to deduce reasonable data and are not dissimilar to
those that have been published in research journals (Cook et al., 2000; Sheehan, 2001).
Estimates of response rates are not equivalently available for the open access survey,
because the survey was distributed by anonymous web links that did not allow a good
estimate of who received the survey. What could be calculated was a 54% completion
rate for those who opened the survey and a higher completion rate of 72% for individuals
who completed at least the first two-thirds of the survey. It is nonetheless likely, given
the broad reach of the organizations that helped to distribute the open-access survey, that
the response rate for this group was lower than that of the expert panel. It is also likely
that at least two factors could be implicated in the differing rates of participation. First
was the degree to which respondents were primed to take the survey. As described in
Chapter 3, individuals in the expert-panel, unlike those in the open-access group, were
pre-notified by email, in-person, or by telephone, to encourage participation. The second
factor that may have influenced the response rates might be the importance of the topic,
that is, the “issue salience”, to the respondents (Cook et al., 2000; Sheehan, 2001).
127
Because quality metrics have been a focus of recent FDA and industry meetings
(Brookings, 2014; ISMP, 2014; PDA, 2014), the importance of quality metrics could
have already been seen as a “hot topic” by experts in companies who are engaged in
strategic quality planning and resource allocation. This might have increased their
willingness to complete this survey. The overall survey completion rates observed for
both panels in this study are encouraging because they suggest that the survey presented
issues viewed as important to the polled respondents, who then appeared to have been
motivated to provide accurate answers to all the questions in the survey.
5.2 Considerations of the Results
5.2.1 Manufacturers Already Use Many Metrics to Measure Quality
A clear message from this study is that manufacturers already use many metrics to
measure quality. A high value is placed on metrics obtained from FDA audits and
analysis, and particularly on published materials associated with FDA’s inspection
actions such as warning letters and inspectional observations documented in FDA 483s.
The importance of such information for companies to evaluate the quality performance of
their company, competitors, partners, and suppliers is not lost on the FDA. They have
already expressed their intent to make such information, both for individual companies
and in the form of industry-aggregated data, more transparent, to “assist the public and
Agency (FDA) stakeholders in making more informed decisions about the products FDA
regulates” (FDA, 2014f). Each company may use these data in different ways depending
on their needs, resources, and “know-how”. However the reliability and value of these
types of analysis by themselves may be limited because the information on FDA
128
inspections and other types of evaluation may be outdated or just unavailable, especially
with regard to foreign manufacturers or suppliers that have seldom been inspected by
FDA (GAO, 2011a).
Interestingly, the metric that was identified to be the least important by the
respondents was that of drug shortages. This finding was somewhat unexpected because
the causes of most drug shortages have been tied to quality breaches that can disrupt
manufacturing and incur significant expense to remediate. Such quality problems have
been particularly problematic for companies making generic sterile-injectable drugs
where 75% of their drug shortages have been attributed to issues of inadequate
manufacturing quality (FDA, 2013d, 2014g). The lower rating of this metric by both
polled groups suggests that drug shortages are not a key metric that manufacturers are
finding useful. Perhaps they have not made a causal connection between quality
problems and drug shortages, or they view it as such a distant metric from the
manufacturing environment that they cannot use it effectively to control the activities of
their company.
The low rating of drug shortage as a quality metric diverges from the emphasis
placed on it by the FDA and probably by other stakeholders as well. For FDA, the recent
rise in drug shortages has been an important “metric” that they associate with quality
problems. Reducing this trend has been a driver for policy change; under FDASIA,
companies must now inform the FDA of manufacturing interruptions that may cause a
drug shortage (FDA, 2012c). FDA has also written much about the need for improving
the quality culture that currently exists in industry in order to minimize shortage
situations (Schniep, 2015; Throckmorton, 2014; Wechsler, 2015a; Wesdyk, 2014). It is
129
less clear if other stakeholders share a view similar to that of the FDA, although much
indirect evidence points to increasing concern amongst user and prescriber groups. For
patients and doctors, for example, shortages put patients at risk if treatments have to be
delayed or replaced with suboptimum substitutions. These individuals are clearly
concerned about the drug shortages and understand the value of tracking and intervening
to manage them (Sullivan, 2015; Tavernise, 2014; Yukhananov, 2012). What may not be
apparent is the value placed by pharmacies, hospitals, and purchasers who may be
responsible for decisions with regard to the selection of a manufacturer on their drug list
or formulary. These stakeholders are often inconvenienced and forced to expend
additional resources to manage the difficulties that arise from drug shortage events. In
2011, a Premier Health Care Alliance analysis reported that these expenses are significant
and estimated that hospitals can incur costs of up to $200 million annually in just buying
more expensive “substitute” drugs (Cherici et al., 2011). Thus associations such as the
American Society of Health-System Pharmacists (ASHP), the American Hospital
Association (AHA), the American Society of Anesthesiologists (ASA), and the Institute
for Safe Medication Practices (ISMP) have been holding meetings and writing position
papers regarding the need to solve the drug shortage problem (ASHP, 2013; ISMP,
2014). These stakeholders may place more value on a quality metric or rating that
reflects the likelihood that a needed drug will fail to be supplied, and thus incur
unanticipated costs and workarounds, than would manufacturers.
On the other hand, one metric that appears to show little disagreement amongst
stakeholders is that of customer complaints (Wechsler, 2015b). In 2013, PDA
recommended complaint rate as a valuable quality metric for FDA because it can
130
“indicate deficiencies in the robustness of the manufacturing process and control
strategy” (Johnson, 2013; PDA, 2013). In 2014, product quality complaint rate was one
of the four metrics, along with lot acceptance rate, confirmed out-of-specification rate,
and recall rate, that was found to have “broad support” by FDA as well as industry
(Brookings, 2014). In this study, respondents also generally agreed that customer
complaint rates and adverse event reporting were important quality metrics. The open-
access group, whose respondents worked more commonly for medical device companies,
indicated a greater use of customer complaint rates (80%) compared to the expert panel
(67%) whose respondents were predominately from pharmaceutical companies. This
finding was paralleled by the significantly higher (p < 0.001) use of FDA field corrective
actions reported by respondents in the device dominated open-access group; these types
of actions may be more closely aligned with medical device practices of iterative design
control and use customer feedback loops as a type of “complaint” to help them develop,
design, and improve their products. Overall, the importance placed on customer
complaint rates observed here and identified elsewhere may reflect the high value that
industry places on its reputation amongst customers and its sensitivity to legal liability for
poor-quality products. In this regard, the “customer” may be broadly defined by
manufacturers to include others apart from the patient or end-consumer, including for
example, group purchasing organizations, wholesalers, payers, and pharmacies.
Customer complaints are a good example of what might be considered a relatively
robust metric that can be measured by every company regardless of size or product type.
Nevertheless even such seemingly robust metrics should be used with caution. First, the
numbers of complaints will vary depending on the type of product under consideration.
131
For example, companies producing over-the-counter products are likely to receive more
complaints simply because they sell more products to a wider segment of the population
than do companies selling specialty prescription drugs. Thus a complaint rate with a
denominator as well as a numerator would be important to use when comparing
companies of different sizes. In addition, some types of products are known to be more
difficult to make or maintain in the field so that higher complaint rates might be
anticipated and accepted when risks are weighed against the benefits. Finally, even
robust metrics can be “gamed” because the company can use its own judgement in
determining which complaints are to be categorized as reflecting poor quality rather than
customer preference. Making complaint rates product specific and differentiating serious
versus non-serious complaints are other challenges that must be overcome if complaint
rates are to be seen as an equitable metric across the medical product industry (Johnson,
2013; PDA, 2013). It is not surprising then that sorting different types of complaints was
viewed to be “challenging” at a recent quality metrics meeting between FDA and
industry (Brookings, 2014).
Customer complaint rates, like warning letters or product recalls, are “lagging”
indicators of quality. Some analysts have suggested that a greater value might be gained
through the collection of more “leading” metrics such as process performance measures
and nonconformance rates that may provide an early signal of incipient quality problems
(Brookings, 2014; PDA, 2013). However, this study suggests that manufacturers are not
enthusiastic about sharing these types of “leading” metrics. A reluctance to share quality
information more generally appeared to be reflected by the fact that respondents least
commonly answered the question probing what quality metrics might be shared and
132
instead provided many open-text responses noting that the company was not prepared to
share any information.
In a recent presentation by Russell Wesdyk (2014) of the FDA, the use of quality
metrics was forecasted as a method that would achieve a “common language to gauge
progress around quality, potentially useful to reduce shortages, objective measures to
provide clarity to all [including patients], a path to achieve regulatory flexibility and
reduced post market change control burdens, and to support risk-based inspections
required under FDASIA.” If these objectives are to be met, industry is a critical
stakeholder whose positions on the use and sharing of certain kinds of information will
need to be further explored and taken into account as FDA attempts to develop new
metrics derived from company supplied data. Further, results in this study may hint at
the possibility that device manufacturers use metrics differently than do pharmaceutical
manufacturers, and this difference should be explored in more detail before quality
metrics determined to be appropriate by the pharmaceutical industry are generalized to
measure quality in other subsectors governed by FDA.
5.2.2 Ratings May Increase Attention on Manufacturing Quality
This study also sought to understand industry views on the impact that visible
quality ratings might have if they were to be implemented for medical product
manufacturers. In a highly competitive industry where company reputation matters, it
was not surprising that visible quality ratings were predicted to encourage “a larger focus
on improving quality”. However, it seems likely from the survey that these investments
would be targeted to specific aspects of quality improvement. Respondents in both
133
surveys predicted higher investments in quality audit programs and employee training.
Conversely, most respondents indicated that capital expenditures such as buying new
equipment or opening new manufacturing sites would likely not be influenced as much
by quality ratings. This runs contrary to the consequences observed after restaurant
hygiene ratings were introduced in Los Angeles County. Then, restaurant owners did
make structural improvements to get a better hygiene grade (Jin & Leslie, 2002). If
obtaining a higher quality rating is not seen by companies to be worth the expenditures of
buying new equipment or expanding manufacturing capacity, this may be a serious
shortcoming to the use of quality metrics as an incentive to achieve certain desirable
outcomes.
Many of the quality problems that occur in pharmaceutical operations have been
tied to deficiencies with the facility and the use of outdated equipment (ISMP, 2014;
ISPE, 2013; Shelly, 2015; Throckmorton, 2014; Woodcock & Wosinska, 2013). FDA is
therefore particularly interested in driving companies to make improvements to their
facilities. Wechsler (2015b) describes investments in modern manufacturing systems as
“key” to FDA’s quality initiatives and needed if industry is to advance manufacturing
flexibility and reduce costs. If the use of ratings is not an effective way to encourage
such actions, perhaps then interviews with the individuals directly responsible for making
decisions regarding equipment and facilities upgrades may provide insight into other
ways to incentivize companies to buy new equipment and open new sites.
It is of course not easy to predict what modifications in thinking and behavior
would in fact occur if a ratings system was put into practice. The responses in this study
suggest that a visible quality scorecard in the pharmaceutical sector may incentivize
134
investment in quality by making the company, industry, and customers more aware of the
quality differences between products and manufacturers, in much the same way as was
seen when hospital and insurance plan ratings were introduced (Beaulieu, 2002; Chernew
et al., 2001; Pope, 2009). As with any evaluative framework, early engagement and
attaining “buy-in” from critical stakeholders is recognized to be a prerequisite for success
(Kahan, 2008). Thus, ongoing inquiry within industry as well as other stakeholders such
as buyers and group purchasing organizations is needed if ratings, either internal or
external to the Agency, are implemented, to affirm whether these investments achieve the
objective of improving quality. This is particularly important because each element has a
potential to be “gamed”, a serious problem that was reported with the introduction of
hospital ratings (Dranove et al., 2002).
5.2.3 Manufacturers Have Doubts about Quality Ratings for Public Use
In this study some doubt is expressed by respondents that quality metrics and
ratings would equip consumers and buyers with the appropriate information on which to
make buying decisions. Although the majority of respondents agreed that firms should
be mandated to disclose any grades that might be issued, they also expressed concerns
that consumers may not fully understand what the grade reflects, and instead may view it
as a sign of overall product safety or clinical benefit. These concerns are not trivial. A
decade ago, the Institute of Medicine (IOM) noted that almost half of US adults have
difficulty understanding health information (Kindig, Panzer, & Nielsen-Bohlman, 2004).
Patients often misinterpret drug information even when experts believe that information
to be helpful and clear (T. C. Davis et al., 2006; Shrank & Avorn, 2007). These
135
observations may temper a common view that ratings would help consumers to make
more educated decisions and choose products from companies rated as higher in quality
(FDA, 2011c; Fox & Tyler, 2013; Schweitzer, 2013; Woodcock & Wosinska, 2013).
Perhaps studies with consumer and patient representatives would be helpful to provide an
understanding of what kinds of quality information that they would find useful and
comprehensible. This is important because misunderstanding of quality information and
ratings may degrade the effectiveness of any rating system at achieving its objectives.
This study also captured the thoughts of industry professionals on the types of
quality information that would be most informative for their customers and how best to
display and communicate that information. The respondents in both surveys thought that
complaint rates and a record of FDA actions taken would be most informative for their
customers, but it is unknown if this type of information would be as informative to
consumers as it appears to be for manufacturers and FDA. The respondents in this survey
also did not seem to agree about the way in which quality information should be
communicated, whether as an alphabetical grade, a numerical value, or a company rank
order. They did show more agreement about where the quality information should be
posted, favoring its announcement on a regulatory agency or manufacturer website rather
than the product label or printed material in pharmacies drug listings or formularies.
These views would be important to validate with the stakeholders more affected by the
ratings before the company views are seen as an accurate proxy for the views of the
groups actually using the information.
Another industry concern was that quality metrics can be misleading or
inequitable across the many types and sizes of companies that comprise the medical
136
product industry; this sentiment is echoed in other reports of industry views (Brookings,
2014) and is an issue that will be difficult to resolve. One of these concerns is developing
metrics that will be independent of company size. When the data in this study is cross-
tabulated, those working at smaller companies seem to be more persuaded to invest in
employee training and preferred to be given a “numerical rank among industry” if ratings
were provided. This may point to a concern that larger, well-resourced, companies would
get all of the “A” grades while smaller companies may rank lower but may otherwise
produce high-quality products within their respective subsector. These types of concerns
should be better understood if equitable metrics are to be developed by FDA. One way to
address some of these concerns may be by applying statistical treatments to metrics and
ratings. Statistical normalization might help to equalize metrics amongst product types,
but statistical data treatments may still be limited in addressing varied company sizes and
types in the medical product industry that range from large multinational finished
pharmaceutical manufacturers, contract manufacturing firms, to small raw material and
component suppliers. Pipino, Lee, and Wang (2002) warned that “one size fits all”
statistical treatments usually do not work. Further, they conclude that the perception of
quality in the minds of consumers comes from a blend of experiential factors that may be
subjective, as well as the objective metrics. The FDA and industry have also recognized
this issue. Some have suggested that FDA would have to “supplement these initial
metrics with additional contextual data, which can help provide a fuller picture of quality
within an organization” (Brookings, 2014). Thus, a more sophisticated ranking model
that includes both subjective and objective inputs will likely be needed if quality metrics
and ratings are to be used to inform the public of the quality differences between
137
pharmaceutical products and manufacturers, as was done for hospital rankings in U.S.
News & World Report (Hill et al., 1997). Policy decision makers should find ways of
addressing these concerns and developing ranking models with industry in order to gain
the cooperation of the broader industry as well as to ensure that quality ratings are seen as
fair and useful.
5.3 Conclusions and Future Considerations
Inarguably GMP regulations have greatly improved the quality of drugs since
their inception, but there currently exists neither a compelling regulatory nor market
incentive for going beyond GMP compliance. In an industry with little tolerance for
product defects or shortages, efforts on part of regulators and industry to “move the
needle” when it comes to quality is imperative. However any approach to make quality
information more visible to the public or within industry would be a major departure
from current regulatory policy. FDA’s legal obligation to avoid the disclosure of
proprietary information limits the extent to which quality metrics can be shared with the
public (Brookings, 2014). Thus voluntary disclosure would be much simpler to achieve
because mandated disclosure would require broader policy reform. In a voluntary
disclosure paradigm, what then must be considered is whether an “incomplete unraveling
effect” would exist where some stakeholders would remain unaware that the ratings even
exist (Fishman & Hagerty, 2003; Mathios, 2000), or if firms would instead see quality
ratings as an advertising “gold mine” and openly boast of their high mark, as was seen
when hospital ratings were first introduced (Rosenthal et al., 1996). This study also
pointed to areas of potential differences in the views between subsectors, defined product
138
types (in this case between device and pharmaceutical manufacturers), company sizes,
and even the professional disciplines found in the medical product industry. For
example, respondents who identified their focus to be in regulatory affairs were found to
be more open to disclosure versus those in quality management and product development
(p = 0.08). Early broad engagement of manufacturers that includes the many subsectors
in the medical product industry is therefore critical to add credibility to any policy
decisions, as has been discussed elsewhere (Almarsdottir & Traulsen, 2006; Liberti et al.,
2013). While in many respects the respondents had similar views in this study, the
divergence in their views identified here may be worth exploring further to ascertain if
this will present roadblocks to the implementation of quality metrics making any quality
ratings policy problematic. Indeed, “the devil is in the details”.
The findings in this study support the premise that visible quality ratings may be
one way to encourage and incentivize quality in the manufacture of pharmaceuticals.
However the first likely step is for the use of rankings by FDA to sort companies for risk-
based inspections. In such a system, manufacturers might be incentivized both by a
reduction of inspections and potentially by the ability to communicate their quality status
to others (Brookings, 2014; FDA, 2013d; Wechsler, 2015a, 2015b). Additional
regulatory incentives such as reduced review periods are also foreseeable since these
have been shown to be highly valued by industry and used elsewhere by FDA. For
example, FDA has recently started to reward companies that develop drugs for rare
diseases, an otherwise unprofitable venture, with expedited review vouchers that can be
sold to other companies. Recently, one of these vouchers sold for $67.5 million
(Langreth & Armstrong, 2015).
139
Although one of main challenges seen with making ratings public is that
consumers would not understand the rating, communicating this information in an
understandable manner may help broaden their influence in the marketplace by allowing
customers to choose or demand higher-quality products. The value seen in educating and
informing patients so they play a greater role in their health care decisions is widely
explored in other areas of medical care (Sowden, Forbes, Entwistle, & Watt, 2001;
Vermeire, Hearnshaw, Van Royen, & Denekens, 2001). Similar research is needed to
help conceptualize how patients may use quality information to decide what medical
products are best for them. Gaining an understanding of how wholesalers and group
purchasing organizations can use quality information is also recommended since they are
largely responsible for supplier selection and price negotiation in the pharmaceutical
supply chain. Similarly, understanding how retail pharmacies may use quality ratings
would be valuable. Would it be advantageous, for example, for retailers such as
Walgreens and CVS to announce that they only sell drugs certified to be of “high-
quality” by FDA? Unlike other industries where a more direct relationship exists
between the manufacturer and the end-consumer, the pharmaceutical industry remains
largely hidden to the consumer with little recognition of the manufacturer let alone their
quality differences. In this environment, typical free market corrections that rely on
competition, branding, as well as free entry and exit is constrained. This is supported by
Gaynor (2006) who concludes that the free market dynamics are “confounded” with the
regulated pricing and reimbursement policies seen in the drug market. If market
incentives are sought as a means to increase investment in quality, empowering more
stakeholders with quality information will be needed to shift from a limited one sided
140
regulatory-compliance incentive structure to one that is more closely aligned to the
market-driven incentives found in other industries.
The purpose of this study, to expand our understanding of industry views
regarding the use of quality metrics and quality rating systems, has been largely met.
This study also identified areas in need of additional study to help find better ways to
incentivize quality and build an industry-wide quality culture. I also hope that this study
will promote a more open exchange of information and ideas within the medical product
industry, regulators, and academia on how best to progress manufacturing quality into a
more proactive system that ensures a reliable supply of high-quality medications to
patients.
141
REFERENCES
Almarsdottir, A. B., & Traulsen, J. M. (2006). Studying and evaluating pharmaceutical
policy- Becoming a part of the policy and consultative process. Pharmacy World
& Science, 28(1), 6-12. doi: 10.1007/s11096-006-9011-0
ASCO. (2011, September 23). Testimony of W. Charles Penley. Paper presented at the
Subcommitte on Health of the Commitee on Energy and Commerce: Hearing
Examining the Increase in Drug Shortages, Washington, DC.
ASH. (2011, December 15). Testimony, Prescription drug shortages: Examining a public
health concern and potential solutions. Paper presented at the Committee on
Health, Education, Labor, and Pensions, Washington, DC.
ASHP. (2013). ASHP, Partners explore solutions for ongoing drug shortage crisis.
Retrieved May 22, 2015, from
http://www.ashp.org/menu/News/NewsCapsules/Article.aspx?id=484
ASHP. (2015). National drug shortage statistics. Retrieved May 23, 2015, from
http://www.ashp.org/DocLibrary/Policy/DrugShortages/Drug-Shortages-
Statistics.pdf
Atkinson, R., & Flint, J. (2001). Accessing hidden and hard-to-reach populations:
Snowball research strategies. Social Research Update, 33(1), 1-4.
Axelrod, J. (2013). Self-enforcement of regulatory compliance through a culture of
credibility. Update: Food and Drug Law Institute, November/December, 12-14.

142
Ballentine, C. (1981). Taste of raspberries, taste of death. The 1937 exlixir sulfanilamide
incident. FDA Consumer Magazine. Retrieved April 20, 2015, from
http://www.fda.gov/aboutfda/whatwedo/history/productregulation/sulfanilamidedi
saster/default.htm
Barringer, M. W., & Mitchell, O. S. (1994). Workers' preferences among company-
provided health insurance plans. Industrial and Labor Relations Review, 48(1),
141-152. doi: 10.1177/001979399404800110
Beaulieu, N. D. (2002). Quality information and consumer health plan choices. Journal
of Health Economics, 21(1), 43-63. doi: 10.1016/S0167-6296(01)00123-6
Berry, A. J. (2011). The anesthesia perspective: The impact of drug shortages on patients
and practitioners. Retrieved April 12, 2015, from
http://www.fda.gov/downloads/Drugs/NewsEvents/UCM274578.pdf
BIOCOM. (2012). 2012 Southern California Economic Impact Report. Retrieved
May 22, 2015, from
https://biocom.org/?m=sp_view_doc&file=Shared%20Documents/Images/Home
%20page/BIOCOM_EconomicImpactReport.pdf
Brookings. (2014). Measuring pharmaceutical quality through manufacturing metrics and
risk-based assessment: Meeting summary. Retrieved April 12, 2015, from
http://www.brookings.edu/~/media/research/files/papers/2014/05/01-quality-
metrics-fda/quality-metrics--meeting-summary.pdf
Buchmueller, T. C., & Feldstein, P. J. (1997). The effect of price on switching among
health plans. Journal of Health Economics, 16(2), 231-247. doi: 10.1016/S0167-
6296(96)00531-0
143
Business Standard. (2014, May 02). Ranbaxy recalls nearly 30,000 packs of anti-allergy
drug in US, Business Standard. Retrieved from http://www.business-
standard.com/article/companies/ranbaxy-recalls-nearly-30-000-packs-of-anti-
allergy-drug-in-us-114050200974_1.html
Campbell, J. L. (2007). Why would corporations behave in socially responsible ways? An
institutional theory of corporate social responsibility. Academy of Management
Review, 32(3), 946-967. doi: 10.5465/AMR.2007.25275684
CBO. (2007). Prescription drug pricing in the private sector (CBO Publication No. 2703).
Retrieved April 12, 2015, from http://www.cbo.gov/sites/default/files/01-03-
prescriptiondrug.pdf
CFR. (2011). 3 CFR 13588, Executive order 13588 of October 31, 2011. Reducing
prescription drug shortages. Retrieved April 11, 2015, from
http://www.gpo.gov/fdsys/granule/CFR-2012-title3-vol1/CFR-2012-title3-vol1-
eo13588/content-detail.html
CFR. (2013a). 21 CFR 7.3, Definitions. Retrieved April 11, 2015, from
http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSearch.cfm?fr=7.3
CFR. (2013b). 21 CFR 210, Current good manufacturing practice in manufacture,
processing, packing, or holding of drugs. Retrieved April 11, 2015, from
http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?fr=210.1
CFR. (2013c). 21 CFR 211, Current good manufacturing practice for finished
pharmaceuticals. Retrieved April 11, 2015, from
http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart
=211&showFR=1&subpartNode=21:4.0.1.1.11.1
144
CFR. (2013d). 21 CFR 211.42, Design and construction features. Retrieved April 11,
2015, from
http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?fr=211.4
2
Chabner, B. A. (2011). Drug shortages- A critical challenge for the generic-drug market.
The New England Journal of Medicine, 365(23), 2147-2149. doi:
10.1056/NEJMp1112633
Chatterjee, S., & Hirschler, B. (2014, March 11). Ranbaxy scrambles for ingredients to
make generic Nexium, Reuters. Retrieved from
http://www.reuters.com/article/2014/03/11/us-astrazeneca-ranbaxy-
idUSBREA2A0N820140311
Check, J., & Schutt, R. K. (2011). Research methods in education. Thousand Oaks, CA:
Sage.
Chen, J., Radford, M. J., Wang, Y., Marciniak, T. A., & Krumholz, H. M. (1999). Do
"America's Best Hospitals" perform better for acute myocardial infarction? The
New England Journal of Medicine, 340(4), 286-292. doi:
10.1056/NEJM199901283400407
Cherici, C., Frazier, J., Feldman, M., Gordon, B., Petrykiw, C. A., Russell, W. L., &
Souza, J. (2011). Navigating drug shortage in American healthcare- A Premier
Healthcare Alliance analysis. Retrieved April 12, 2015, from
https://legacy.premierinc.com/about/news/11-mar/drug-shortage-white-paper-3-
28-11.pdf
145
Chernew, M., Gowrisankaran, G., & Scanlon, D. (2001). Learning and the value of
information: Evidence from health plan report cards (NBER Working Paper
8589). National Bureau of Economic Research. Retrieved April 12, 2015, from
http://www.nber.org/papers/w8589
Clarke, T., & Berkrot, B. (2014, March 18). Unease grows among U.S. doctors over
Indian drug quality, Reuters. Retrieved from
http://www.reuters.com/article/2014/03/18/us-usa-india-genericdrugs-insight-
idUSBREA2H06Y20140318
Committee on Oversight and Government Reform. (2012). FDA's contribution to the
drug shortage crisis. Retrieved April 11, 2015, from
http://oversight.house.gov/wp-content/uploads/2012/06/6-15-2012-report-fdas-
contribution-to-the-drug-shortage-crisis.pdf
Conti, R. M. (2011). An economic assessment of the causes and policy implications of
current specialty drug shortages. Senate Finance Committee. Retrieved April 22,
2015, from
http://www.finance.senate.gov/imo/media/doc/Conti%20Testimony.pdf
Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in
web-or internet-based surveys. Educational and Psychological Measurement,
60(6), 821-836. doi: 10.1177/00131640021970934
Crosby, P. B. (1980). Quality is free. New York, NY: Mentor.

146
Dafny, L., & Dranove, D. (2005). Do report cards tell consumers anything they don't
already know? The case of Medicare HMOs (NBER Working Paper 11420).
National Bureau of Economic Research. Retrieved April 12, 2015, from
http://www.nber.org/papers/w11420
Daurio, D., Medina, C., Saw, R., Nagapudi, K., & Alvarez-Núñez, F. (2011). Application
of twin screw extrusion in the manufacture of cocrystals, Part I: Four case studies.
Pharmaceutics, 3(3), 582-600. doi: 10.3390/pharmaceutics3030582
Davis, K., Collins, K. S., Schoen, C., & Morris, C. (1995). Choice matters: Enrollees'
views of their health plans. Health Affairs, 14(2), 99-112. doi:
10.1377/hlthaff.14.2.99
Davis, T. C., Wolf, M. S., Bass, P. F., Middlebrooks, M., Kennen, E., Baker, D. W., . . .
Savory, S. (2006). Low literacy impairs comprehension of prescription drug
warning labels. Journal of General Internal Medicine, 21(8), 847-851. doi:
10.1111/j.1525-1497.2006.00529.x
Deming, W. E. (1986). Out of crisis. Cambridge, MA: MIT Press.
Devaney, T. (2014, February 18). FDA loosens pharmaceutical regs, The Hill. Retrieved
from http://thehill.com/blogs/regwatch/healthcare/198641-fda-loosens-
pharmaceutical-regs
Diehl, D. (2015). World's largest biotech hubs: Boston and the San Francisco Bay.
Retrieved April 7, 2015, from
http://biotech.about.com/od/investinginbiotech/a/Worlds-Largest-Biotech-Hubs-
Boston-And-The-San-Francisco-Bay.htm
147
Dranove, D., Kessler, D., McClellan, M., & Satterthwaite, M. (2002). Is more
information better? The effects of report cards on health care providers (NBER
Working Paper 8697). National Bureau of Economic Research. Retrieved April
12, 2015, from http://www.nber.org/papers/w8697
Edgman-Levitan, S., & Cleary, P. D. (1996). What information do consumers want and
need? Health Affairs, 15(4), 42-56. doi: 10.1377/hlthaff.15.4.42
Edney, A. (2014, February 27). FDA failed to track substandard generics, Congress told,
The Washington Post. Retrieved from
http://washpost.bloomberg.com/Story?docId=1376-N1M3GT6K50YF01-
5046GFM2F9IP1L3U9D6MRLF90V
EMA. (2012). Reflection paper on medicinal product supply shortages caused by
manufacturing/Good Manufacturing Practice compliance problems
(EMA/590745/2012). Retrieved April 12, 2015, from
http://www.ema.europa.eu/docs/en_GB/document_library/Other/2012/11/WC500
135113.pdf
EMA. (2014). Guideline on process validation for finished products- information and
data to be provided in regulatory submissions
(EMA/CHMP/CVMP/QWP/BWP/70278/2012-Rev1). Retrieved April 12, 2015,
from
http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/20
14/02/WC500162136.pdf

148
FDA. (2004a). Innovation and continous improvement in pharmaceutical manufacturing.
Retrieved April 12, 2015, from
http://www.fda.gov/ohrms/dockets/ac/04/briefing/2004-
4080b1_01_manufSciWP.pdf
FDA. (2004b). Pharmaceutical CGMPs for the 21st century- A risk-based approach.
Retrieved April 12, 2015, from
http://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/Manufacturi
ng/QuestionsandAnswersonCurrentGoodManufacturingPracticescGMPforDrugs/
UCM176374.pdf
FDA. (2004c). Risk-based method for prioritizing CGMP inspections of pharmaceutical
manufacturing sites- A pilot risk ranking model. Retrieved April 12, 2015, from
http://www.fda.gov/ohrms/dockets/ac/04/briefing/2004-4080b1_04_risk-
based.pdf
FDA. (2006). Guidance for industry: Quality systems approach to pharmaceutical CGMP
regulations. Retrieved April 12, 2015, from
http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformatio
n/Guidances/UCM070337.pdf
FDA. (2011a). Guidance for industry: Process validation: general practices and
principles. Retrieved April 12, 2015, from
http://www.fda.gov/downloads/Drugs/Guidances/UCM070336.pdf
FDA. (2011b). Pathway to global product safety and quality. Retrieved April 12, 2015,
from
149
http://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofGlobalRegulat
oryOperationsandPolicy/GlobalProductPathway/UCM262528.pdf
FDA. (2011c). A review of FDA's approach to medical product shortages. Retrieved
April 12, 2015, from
http://www.fda.gov/AboutFDA/ReportsManualsForms/Reports/ucm277744.htm
FDA. (2012a). Fact sheet: New user fees for generic drugs will enhance Americans'
access to less expensive drugs and generate more cost savings. Retrieved
April 12, 2015, from
http://www.fda.gov/RegulatoryInformation/Legislation/FederalFoodDrugandCos
meticActFDCAct/SignificantAmendmentstotheFDCAct/FDASIA/ucm310992.ht
m
FDA. (2012b). Food and Drug Administration Safety and Innovation Act. Sec 705 Risk-
based inspection frequency. Retrieved April 12, 2015, from
http://www.gpo.gov/fdsys/pkg/PLAW-112publ144/pdf/PLAW-112publ144.pdf
FDA. (2012c). Food and Drug Administration Safety and Innovation Act. Sec 1001
Discontinuance or interruption in the production of life-saving drugs. Retrieved
May 22, 2015, from http://www.gpo.gov/fdsys/pkg/PLAW-
112publ144/pdf/PLAW-112publ144.pdf
FDA. (2013a). Global engagement. Retrieved April 12, 2015, from
http://www.fda.gov/downloads/AboutFDA/ReportsManualsForms/Reports/UCM
298578.pdf

150
FDA. (2013b). New drug application (NDA) and abbreviated new drug application
(ANDA) field alert reports. Retrieved April 12, 2015, from
http://www.fda.gov/AboutFDA/CentersOffices/OfficeofMedicalProductsandToba
cco/CDER/ucm082083.htm
FDA. (2013c). Report to Congress: Strategic integrated management plan, FDASIA
2012. Retrieved April 12, 2015, from
http://www.fda.gov/downloads/RegulatoryInformation/Legislation/FederalFoodD
rugandCosmeticActFDCAct/SignificantAmendmentstotheFDCAct/FDASIA/UC
M360047.pdf
FDA. (2013d). Strategic plan for preventing and mitigating drug shortages. Retrieved
April 12, 2015, from
http://www.fda.gov/downloads/Drugs/DrugSafety/DrugShortages/UCM372566.p
df
FDA. (2014a). Annual report on inspections of establishments. Retrieved April 12, 2015,
from
http://www.fda.gov/downloads/regulatoryinformation/legislation/federalfooddrug
andcosmeticactfdcact/significantamendmentstothefdcact/fdasia/ucm384063.pdf
FDA. (2014b). Drug applications and current good manuafacturing practice (cGMP)
regulations. Retrieved April 12, 2015, from
http://www.fda.gov/drugs/developmentapprovalprocess/manufacturing/ucm09001
6.htm
FDA. (2014c). Drug applications and current good manufacturing practices. Retrieved
April 12, 2015, from
151
http://www.fda.gov/Drugs/DevelopmentApprovalProcess/Manufacturing/ucm090
016.htm
FDA. (2014d). Establishment inspection report conclusions and decisions (Document No.
FMD 86). Retrieved April 12, 2015, from
http://www.fda.gov/downloads/iceci/inspections/fieldmanagementdirectives/ucm3
82035.pdf
FDA. (2014e). Fiscal year 2015: FDA justification of estimates for appropriations
committees. Retrieved April 12, 2015, from
http://www.fda.gov/downloads/AboutFDA/ReportsManualsForms/Reports/Budge
tReports/UCM388309.pdf
FDA. (2014f). Food and Drug Administration transparency initiative: Increasing public
access to FDA's compliance and enforcement data. Retrieved May 11, 2015, from
http://www.fda.gov/downloads/aboutfda/transparency/transparencyreports/ucm39
4213.pdf
FDA. (2014g). Report to Congress: First annual report on drug shortages for calender
year 2013. Retrieved April 12, 2015, from
http://www.fda.gov/downloads/drugs/drugsafety/drugshortages/ucm384892.pdf
FDA. (2015a). Fiscal year 2014: FDA enforcement statistics summary. Retrieved
May 23, 2015, from
http://www.fda.gov/downloads/ICECI/EnforcementActions/UCM443198.pdf

152
FDA. (2015b). Professional affairs and stakeholder engagement. Retrieved April 12,
2015, from
http://www.fda.gov/AboutFDA/CentersOffices/OfficeofMedicalProductsandToba
cco/CDER/ucm385522.htm
FDCA. (2010a). Sec. 351 Adulterated drugs and devices. Retrieved April 11, 2015, from
http://www.gpo.gov/fdsys/pkg/USCODE-2010-title21/html/USCODE-2010-
title21-chap9-subchapV-partA-sec351.htm
FDCA. (2010b). Sec. 360 Registration of producers of drugs or devices. Retrieved April
11, 2015, from http://www.gpo.gov/fdsys/pkg/USCODE-2010-
title21/html/USCODE-2010-title21-chap9-subchapV-partA-sec360.htm
Feigenbaum, A. V. (1991). Total quality control. New York, NY: McGraw-Hill.
Fishman, M. J., & Hagerty, K. M. (2003). Mandatory versus voluntary disclosure in
markets with informed and uninformed customers. Journal of Law, Economics, &
Organization, 19(1), 45-63. doi: 10.1093/jleo/19.1.45
Fox, E. R., Birt, A., James, K. B., Kokko, H., Salverson, S., & Soflin, D. L. (2009).
ASHP guidelines on managing drug product shortages in hospitals and health
systems. American Journal of Health-System Pharmacy, 66(15), 1399-1406. doi:
10.2146/ajhp090026
Fox, E. R., & Tyler, L. S. (2013). Call to action: Finding solutions for the drug shortage
crisis in the United States. Clinical Pharmacology & Therapeutics, 93(2), 145-
147. doi: 10.1038/clpt.2012.225
153
Gaffney, A. (2014, April 30). Facing dire shortage of IV saline, FDA again turns to
enforcement discretion approach, Regulatory Focus. Retrieved from
http://www.raps.org/regulatoryDetail.aspx?id=18564
GAO. (1998). Food and Drug Administration: Improvements needed in the foreign drug
inspection program (GAO/HEHS-98-21). Retrieved April 12, 2015, from
http://www.gao.gov/assets/230/225564.pdf
GAO. (2007). Drug safety: Preliminary findings suggest weaknesses in FDA's program
for inspecting foreign drug manufacturers (GAO-08-244T). Retrieved April 12,
2015, from http://www.gao.gov/new.items/d08224t.pdf
GAO. (2009). High risk series: An update (GAO-09-271). Retrieved April 12, 2015, from
http://www.gao.gov/new.items/d09271.pdf
GAO. (2010). Drug safety: FDA has conducted more foreign inpsections and begun to
improve its information on foreign establishements, but more progress is needed
(GAO-10-961). Retrieved April 12, 2015, from
http://www.gao.gov/new.items/d10961.pdf
GAO. (2011a). Drug safety: FDA faces challenges overseeing the foreign drug
manufacturing supply chain (GAO-11-936T). Retrieved April 12, 2015, from
http://www.gao.gov/assets/130/126943.pdf
GAO. (2011b). Drug shortages: FDA's ability to respond should be strengthened (GAO-
12-315T). Retrieved April 12, 2015, from
http://www.gao.gov/assets/590/586979.pdf
154
GAO. (2014). Drug shortages: Threat to public health persists, despite actions to help
maintain product availability (GAO-14-339T). Retrieved April 12, 2015, from
http://www.gao.gov/assets/670/660771.pdf
Gaynor, M. (2006). What do we know about competition and quality in health care
markets? (NBER Working Paper 12301). National Bureau of Economic Research.
Retrieved April 12, 2015, from http://www.nber.org/papers/w12301
Gaynor, M., & Vogt, W. B. (1999). Antitrust and competition in health care markets
(NBER Working Paper 7112). National Bureau of Economic Research. Retrieved
April 12, 2015, from http://www.nber.org/papers/w7112
Gingery, D. (2014, February 10). Drug shortage analysis puts FDA at loggerheads with
GAO, The Pink Sheet Daily. Retrieved from
https://www.pharmamedtechbi.com/publications/the-pink-sheet-
daily/2014/2/10/drug-shortage-analysis-puts-fda-at-loggerheads-with-gao
Gokhale, K. (2014, March 10). Sun Pharma, Ranbaxy recall some generic batches in
U.S., Bloomberg. Retrieved from http://www.bloomberg.com/news/2014-03-
10/sun-pharma-ranbaxy-recall-some-generic-drug-batches-in-u-s-.html
Gupta, D. K., & Huang, S. M. (2013). Drug shortages in the United States: A critical
evaluation of root causes and the need for action. Clinical Pharmacology &
Therapeutics, 93(2), 133-135. doi: 10.1038/clpt.2012.229
Haninger, K., Jessup, A., & Koehler, K. (2011). Economic analysis of the causes of drug
shortages. ASPE Issue Brief. Retrieved April 22, 2015, from
http://aspe.hhs.gov/sp/reports/2011/drugshortages/ib.shtml
155
Harrington, P., & Shepherd, M. D. (2002). Analysis of the movement of prescription
drugs to over-the-counter status. Journal of Managed Care Pharmacy, 8(6), 499-
508.
Hill, C. A., Winfrey, K. L., & Rudolph, B. A. (1997). "Best hospitals": A description of
the methodology for the index of hospital quality. Inquiry, 34(1), 80-90.
Hollenbeak, C. S. (1999). The effect of generic competition on prescription to over-the-
counter switching. Pharmacoeconomics, 16(6), 661-668. doi: 10.2165/00019053-
199916060-00005
ICH. (2008). ICH harmonised tripartite guideline: Pharmaceutical quaility system Q10.
Retrieved April 12, 2015, from
http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality
/Q10/Step4/Q10_Guideline.pdf
IMS Health. (2011). Drug shortages: A closer look at products, suppliers, and volume
volatility. Retrieved April 11, 2015, from
http://www.imshealth.com/deployedfiles/ims/Global/Content/Insights/IMS%20In
stitute%20for%20Healthcare%20Informatics/Static%20Files/IHII_Drug_Shortage
_Report.pdf
IMS Health. (2014). Medicine use and shifiting costs of healthcare: A review of the use
of medicines in the United States in 2013. Retrieved April 11, 2015, from
http://www.imshealth.com/deployedfiles/imshealth/Global/Content/Corporate/IM
S%20Health%20Institute/Reports/Secure/IIHI_US_Use_of_Meds_for_2013.pdf
ISMP. (2010). National survey on drug shortages reveals serious impact on patient safety.
Retrieved April 11, 2015, from http://www.ismp.org/pressroom/PR20100923.pdf
156
ISMP. (2014). Drug shortage summit. Retrieved May 22, 2015, from
http://www.ismp.org/pressroom/2014-Drug-Shortages-Summit.pdf
ISPE. (2013). Report on the ISPE drug shortages survey. Retrieved April 11, 2015, from
http://www.ispe.org/drug-shortages-initiative/about-the-survey
ITA. (2010). Pharmaceutical industry profile. Retrieved April 12, 2015, from
http://ita.doc.gov/td/health/PharmaceuticalIndustryProfile2010.pdf
Jacobson, M., Alpert, A., & Duarte, F. (2012, May 29). Presecription drug shortages:
Reconsidering the role of medicare payment policies [Health Affairs Blog].
Retrieved from http://healthaffairs.org/blog/2012/05/29/prescription-drug-
shortages-reconsidering-the-role-of-medicare-payment-policies/
Jacques, A., & Stauffer, R. (2014). Drug shortage issue, solutions highlighted. PDA
Letter, L(2), 39-41.
Jamieson, M. W. (2011). The role of universities in the commercialization of medical
products: A survey of industry views. (Doctoral dissertation), University of
Southern California. Retrieved May 25, 2015, from
http://digitallibrary.usc.edu/cdm/ref/collection/p15799coll127/id/616319
Jenkins-Smith, H. C., & Sabatier, P. A. (1994). Evaluating the advocacy coalition
framework. Journal of Public Policy, 14, 175-203. doi:
10.1017/S0143814X00007431.
Jin, G. Z., & Leslie, P. (2002). The effect of information on product quality: Evidence
from restaurant hygiene grade cards. Retrieved April 12, 2015, from
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=322883
157
Jin, G. Z., & Sorensen, A. T. (2005). Information and consumer choice: The value of
publicized health plan ratings (NBER Working Paper 11514). National Bureau of
Economic Research. Retrieved April 12, 2015, from
http://www.nber.org/papers/w11514
Johnson, R. (2013). PDA responses to FDA drug shortages task force strategic plan
(Docket No. FDA-2013-N-0124). Retrieved April 12, 2015, from
http://www.pda.org/Science-and-Regulatory-Affairs/RAQAB/Regulatory-
Comments-Resources/2013-PDA-Regulatory-Comments/Final-Submission-of-
PDA-Comments-on-FDAs-Draft-Drug-Shortages-Strategic-Plan-March-13-
2013.aspx
Juran, J. M., & Gyrna, F. M. (1980). Quality planning and analysis. New York, NY:
McGraw-Hill.
Kahan, B. (2008). Excerpts from review of evaluation frameworks prepared for
Saskstchewan Ministry of Education. Kael Consulting. Retrieved April 12, 2015,
from http://www.idmbestpractices.ca/pdf/evaluation-frameworks-review.pdf
Kaplowitz, M. D., Hadlock, T. D., & Levine, R. (2004). A comparison of web and mail
survey response rates. Public Opinion Quarterly, 68(1), 94-101. doi:
10.1093/poq/nfh006

158
Karlin, S. (2014a, March 10). Drug shortages: Purchasers requiring paybacks; FDA's
review policy questioned, The Pink Sheet. Retrieved from
https://www.pharmamedtechbi.com/Publications/The-Pink-Sheet/76/10/Drug-
Shortages-Purchasers-Requiring-Paybacks-FDAs-Review-Policy-
Questioned?result=4&total=5&searchquery=%253fq%253ddrug%252520shortag
es%252520purchasers%252520requiring%252520paybacks
Karlin, S. (2014b, April 9). Hospira's shortage solution draws FDA Warning Letter for
product release, The Gray Sheet. Retrieved from
https://www.pharmamedtechbi.com/Publications/The-Gray-Sheet/40/15/Hospiras-
Shortage-Solution-Draws-FDA-Warning-Letter-For-Product-
Release?result=1&total=3&searchquery=%253fq%253dhospira's%252520shortag
e%252520solution%252520draws%252520FDA%252520warning%252520letter
Karlin, S. (2014c, February 27). Purchaser survey supports FDA's claim that drug
shortages decreasing, The Pink Sheet Daily. Retrieved from
https://www.pharmamedtechbi.com/Publications/The-Pink-Sheet-
Daily/2014/2/27/Purchaser-Survey-Supports-FDAs-Claim-That-Drug-Shortages-
Decreasing?result=2&total=2&searchquery=%253fq%253dpurchaser%252520sur
vey%252520supports%252520FDA's%252520claim%252520that%252520drug%
252520shortages
Kindig, D. A, Panzer, A. M., & Nielsen-Bohlman, L. (2004). Health literacy: A
prescription to end confusion. Washington, DC: National Academies Press.

159
Kitson, A. L, Rycroft-Malone, J., Harvey, G., McCormack, B., Seers, K., & Titchen, A.
(2008). Evaluating the successful implementation of evidence into practice using
the PARiHS framework: Theoretical and practical challenges. Implementation
Science, 3(1), 1. doi: 10.1186/1748-5908-3-1
Langreth, R., & Armstrong, D. (2015, January 25). Sanofi and Regeneron gain an edge in
cholesterol drug race, Bloomberg. Retrieved from
http://www.bloomberg.com/news/articles/2015-01-26/sanofi-and-regeneron-gain-
edge-on-amgen-in-cholesterol-drug-race
Leong, J., McAuslane, N., Walker, S., & Salek, S. (2013). Is there a need for a universal
benefit-risk assessment framework for medicines? Regulatory and industry
perspectives. Pharmacoepidemiology and Drug Safety, 22(9), 1004-1012. doi:
10.1002/pds.3464
Liberti, L., McAuslane, N., Patel, P., Breckenridge, A., Eichler, H. G., & Peterson, R.
(2013). Regulatory review: How do agencies ensure the quality of decision
making? Clinical Pharmacology & Therapeutics, 94(3), 305-308. doi:
10.1038/clpt.2013.127
Loosveldt, G., & Storms, V. (2008). Measuring public opinions about surveys.
International Journal of Public Opinion Research, 20(1), 74-89. doi:
10.1093/ijpor/edn006
Lu, J. Z., & Comanor, W. S. (1998). Strategic pricing of new pharmaceuticals. Review of
Economics and Statistics, 80(1), 108-118. doi: 10.1162/003465398557212
160
Manocha, V. (2014, April 1). FDA finds contaiminated drug ingredient at GSK Ireland
plant, Reuters. Retrieved from http://www.reuters.com/article/2014/04/01/us-gsk-
fda-idUSBREA301OT20140401
Mathieu, M. (2008). New drug development: A regulatory overview (8th ed.). Waltham,
MA: PAREXEL.
Mathios, A. D. (2000). The impact of mandatory disclosure laws on product choices: An
analysis of the salad dressing market. Journal of Law & Economics,
XLIII(October 2000), 651-677.
Milstein, B., & Wetterhall, S. (2000). A framework featuring steps and standards for
program evaulation. Health Promotion Practice, 1(3), 221-228.
Mukamel, D. B., & Mushlin, A. I. (1998). Quality of care information makes a
difference: An analysis of market share and price changes after publication of the
New York State Cardiac Surgery Mortality Reports. Medical Care, 36(7), 945-
954.
Munson, J., Freeman Stanfield, C, & Gujral, B. (2006). A review of process analytical
technology (PAT) in the US pharmaceutical industry. Current Pharmaceutical
Analysis, 2(4), 405-414. doi: 10.2174/157341206778699582
North, D. C. (1980). Institutions, institutional change and economic performance. New
York, NY: Cambridge University Press.
O'Donnell, J. (2014, March 3). Do drug benefit managers reduce health costs?, USA
Today. Retrieved from
http://www.usatoday.com/story/money/personalfinance/2014/03/03/pharmacy-
benefit-managers-healthcare-costs-savings/5495317/
161
PDA. (2013). Parenteral Drug Association points to consider: Pharmaceutical quality
metrics. Retrieved April 12, 2015, from http://www.pda.org/pdf-1/PDA-
Pharmaceutical-Quality-Metrics.aspx
PDA. (2014, December 2-3). Exploring quality culture and quality systems maturity.
Paper presented at the Pharmaceutical Quality Metrics Conference, Washington,
DC.
Peay, H. L., Hollin, I., Fischer, R., & Bridges, J. F. (2014). A community-engaged
approach to quantifying caregiver preferences for the benefits and risks of
emerging therapies for Duchenne muscular dystrophy. Clinical Therapeutics,
36(5), 624-637. doi: 10.1016/j.clinthera.2014.04.011
Peither, T. (2010). FDA will obtain full PIC/S membership in 2010. Maas & Peither AG-
GMP Publishing. Retrieved April 12, 2015, from
http://www.picscheme.org/bo/commun/upload/document/article-on-us-fda-
pdf.pdf
Pina, K. R., & Pines, W. L. (2008). A practical guide to food and drug law and
regulation (3rd ed.). Washington, DC: Food and Drug Law Institute.
Pipino, Leo L, Lee, Yang W, & Wang, Richard Y. (2002). Data quality assessment.
Communications of the ACM, 45(4), 211-218. doi: 10.1145/505248.506010
Pope, D. G. (2009). Reacting to rankings: Evidence from "America's Best Hospitals".
Journal of Health Economics, 28(6), 1154-1165. doi:
10.1016/j.jhealeco.2009.08.006

162
Preidt, R. (2014, May 7). Pfizer recalls some antidepressants after drug mix-up,
HealthDay. Retrieved from http://consumer.healthday.com/public-health-
information-30/food-and-drug-administration-news-315/pfizer-recalls-some-
antidepressants-after-drug-mixup-685603.html
Rawson, K. (2014, January 16). FDA's call for quality: Revamped GMP initiative
stresses cost savings, The Pink Sheet Daily. Retrived from
https://www.pharmamedtechbi.com/Publications/The-Pink-Sheet-
Daily/2014/1/16/FDAs-Call-for-Quality-Revamped-GMP-Initiative-Stresses-
Cost-Savings
Rosenthal, G. E., Chren, M. M., Lasek, R. J., & Landefeld, C. S. (1996). The annual
guide to "America's best hospitals"- Evidence of influence among health care
leaders. Journal of General Internal Medicine, 11(6), 366-369. doi:
10.1007/BF02600049
Schniep, J. S. (2014). Linking drug shortages and quality metrics. Pharmaceutical
Technology, (May), 18-20.
Schniep, J. S. (2015). An update on the quality metrics initiative. BioPharm
International, 28(5), 41.
Schweitzer, S. O. (2013). How the US Food and Drug Administration can solve the
prescription drug shortage problem. American Journal of Public Health, 103(5),
e10-14. doi: 10.2105/AJPH.2013.301239
Sheehan, K. B. (2001). E‐ mail survey response rates: A review. Journal of Computer‐ Mediated Communication, 6(2), 0. doi: 10.1111/j.1083-6101.2001.tb00117.x
163
Shelly, S. (2015). The quality improvement drive to address drug shortages.
Pharmaceutical Commerce. Retrieved May 22, 2015, from
http://www.pharmaceuticalcommerce.com/index.php?pg=manufacturing_and_pac
kaging&articleid=27458&keyword=drug%20shortages-
manufacturing%20quality%20problems
Shrank, W. H., & Avorn, J. (2007). Educating patients about their medications: the
potential and limitations of written drug information. Health Affairs, 26(3), 731-
740. doi: 10.1377/hlthaff.26.3.731
Silverman, S. (2014, April 30). Update on FDA medical device quality initiatives. Paper
presented at the Quality Leadership Executive Leadership Series, Fifth Workshop,
FDA Los Angeles District Office, Irvine, CA.
Sinclair, U. (1906). The jungle. New York, NY: Doubleday.
Sowden, A. J., Forbes, C., Entwistle, V., & Watt, I. (2001). Informing, communicating
and sharing decisions with people who have cancer. Quality in Health Care,
10(3), 193-196. doi: 10.1136/qhc.0100193
Stachowiak, S. (2009). Pathways for change: six theories about how policy change
happens. Organizational Research Services. Retrieved April 12, 2015, from
http://www.organizationalresearch.com/publicationsandresources/pathways_for_c
hange_6_theories_about_how_policy_change_happens.pdf
Stewart, K. (2014, April 9). Utah tracking shows persistent drug shortages nationwide,
The Salt Lake City Tribune. Retrieved from
http://www.sltrib.com/sltrib/news/57790583-78/shortages-drug-saline-
drugs.html.csp
164
Stigler, G. J. (1968). The organization of the industry. Homewood, IL: Irwin.
Storm, N. E. (2013). Regulatory dissonance in the global development of drug therapies:
A case study of drug development in postmenopausal osteoporosis. (Doctoral
dissertation), University of Southern California. Retrieved May 25, 2015, from
http://digitallibrary.usc.edu/cdm/ref/collection/p15799coll3/id/345661
Storm, N. E., & Richmond, F. J. (2015). Global regulatory dissonance: A case study of
industry views on the development of drugs for postmenopausal osteoporosis.
Therapeutic Innovation and Regulatory Science, 49(2), 269-278. doi:
10.1177/2168479014558276
Sue, V. M., & Ritter, L. A. (2012). Conducting online surveys (2nd ed.). Thousand Oaks,
CA: Sage.
Sullivan, T. (2015). Drug shortages policy update. Policy and Medicine. Retrieved May
25, 2015, from http://www.policymed.com/drug-shortages/
Tavernise, S. (2014, February 10). Drug shortages continue to vex doctors, New York
Times. Retrieved from http://www.nytimes.com/2014/02/11/health/shortages-of-
critical-drugs-continue-to-vex-doctors-study-finds.html?_r=0
Teagarden, J. R., & Epstein, R. S. (2013). Pharmacy benefit managers and their
obligations during serious prescription drug shortages. Clinical Pharmacology &
Therapeutics, 93(2), 143-145. doi: 10.1038/clpt.2012.224
Thomas, K. (2014, June 23). Generic versions of Toprol XL, a heart drug, are recalled,
The New York Times. Retrieved from
http://www.nytimes.com/2014/06/24/health/warning-unheeded-heart-drugs-are-
recalled.html?_r=0
165
Throckmorton, D. C. (2014). FDA response to drug shortages. Retrieved April 12, 2015,
from
http://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProdu
ctsandTobacco/CDER/UCM400503.pdf
Van Trieste, M. (2013, June 4). We owe it to patients to provide a reliable supply of
medicines, Xconomy. Retrived from
http://www.xconomy.com/national/2013/06/04/we-owe-it-to-patients-to-provide-
a-reliable-supply-of-medicines/
Ventola, C. L. (2011). The drug shortage crisis in the United States: Causes, impact, and
management strategies. Pharmacy and Therapeutics, 36(11), 740-757.
Vermeire, E., Hearnshaw, H., Van Royen, P., & Denekens, J. (2001). Patient adherence
to treatment: three decades of research: A comprehensive review. Journal of
Clinical Pharmacy and Therapeutics, 26(5), 331-342. doi: 10.1046/j.1365-
2710.2001.00363.x
Villax, G. (2013). FDA needs to step it up. Chemical and Engineering News, 91(16), 3.
Villax, G. (2014). A case for an FDA Dean's List. Pharmaceutical Technology.
Retrieved April 22, 2015, from
http://www.pharmtech.com/pharmtech/Feature+Articles/A-Case-for-an-FDA-
Deans-List/ArticleStandard/Article/detail/853868
Vogt, F. G., & Kord, A. S. (2011). Development of quality‐ by‐ design analytical methods.
Journal of Pharmaceutical Sciences, 100(3), 797-812. doi: 10.1002/jps.22325
Wechsler, J. (2015a). FDA revises field inspections to reflect global market and quality
initiatives. BioPharm International, 28(5), 8-10.
166
Wechsler, J. (2015b). Modern manufacturing systems key to FDA quality initiative.
Pharmaceutical Technology, 39(4), 16-18.
Weible, C. M., Sabatier, P. A., Jenkins‐ Smith, H. C., Nohrstedt, D., Henry, A. D., &
DeLeon, P. (2011). A quarter century of the advocacy coalition framework.
Policy Studies Journal, 39(3), 349-360. doi: 10.1111/j.1541-0072.2011.00412.x
Wesdyk, R. (2014). Quality metrics, why are we going, where are we going. Retrieved
April 12, 2015, from
http://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProdu
ctsandTobacco/CDER/UCM374192.pdf
Williamson, O. E. (1985). The economic institutions of capitilism. New York, NY: Free
Press.
Woodcock, J. (2012). Reliable drug quality: An unresolved problem. PDA Journal of
Pharmaceutical Science and Technology, 66(3), 270-272.
Woodcock, J., & Wosinska, M. (2013). Economic and technological drivers of generic
sterile injectable drug shortages. Clinical Pharmacology & Therapeutics, 93(2),
170-176. doi: 10.1038/clpt.2012.220
Wu, H., White, M., & Khan, M. A. (2011). Quality-by-Design (QbD): An integrated
process analytical technology (PAT) approach for a dynamic pharmaceutical co-
precipitation process characterization and process design space development.
International Journal of Pharmaceutics, 405(1), 63-78. doi:
10.1016/j.jpharm.2010.11.045
167
Yorukoglu, A. (2012). Medicare reimbursements and shortages of sterile injectable
pharmaceuticals (NBER Working Paper 17987). National Bureau of Economic
Research. Retrieved April 12, 2015, from http://www.nber.org/papers/w17987
Yu, L. X. (2008). Pharmaceutical quality by design: Product and process development,
understanding, and control. Pharmaceutical Research, 25(4), 781-791. doi:
10.1007/s11095-007-9511-1
Yukhananov, A. (2012, June 2). Hospitals fight drug scarcity, fear patients harmed,
Reuters. Retrieved from http://www.reuters.com/article/2012/06/02/us-drugs-
shortages-idUSBRE8500K220120602

168
APPENDIX A
STAKEHOLDER ANALYSIS
Stakeholder Role/Interest Influence on Subject Matter Survey Considerations
Innovator Drug
Manufacturers
Manufactures branded drugs and directly
impacted by quality ratings
Significant- Directly makes investment, production,
inventory, and pricing decisions
Industry trade-groups and expert opinions
can be leveraged with some ease to reach
a diverse sample of adequate size for
analysis
Generic Drug
Manufactures
Manufactures of generic drugs and
directly impacted by quality ratings
Significant- Directly makes investment, production,
inventory, and pricing decisions
Same as above.
Patients / Consumers Directly impacted by drug shortages and
potentially sensitive to preferred high-
quality drug product
Unknown- Usually not in position to make buying
(price) decision and primarily guided by physician on
selection and use of drug
Difficult to assess by single survey
without confounding factors and limited
scope. Requires informed consent.
Physicians / Medical
Providers
Works for the best interest of patient to
ensure high-quality care. Strong/respected
opinion in advising patient on drug
Influential in prescribing drug and designation of
brand/type. Somewhat influential in selection of
company to purchase.
Difficult to assess by single survey
without confounding factors and limited
scope
Pharmacy Benefit
Managers (PBMs)
Administrator of drug plans including
processing payments, formulary decision
making, and negotiating pricing/discounts
with wholesale manufactures
Significant- Directly drives market demand by
selection of drug preference and negotiating drug
pricing/discounts/rebates from wholesale market.
Holds supply chain data and incurred costs to manage
shortages within network
Very few large PBMs exists any may be
difficult to get agreement to survey their
group. May not provide need information
on drug quality rating as they don’t have
the expertise to do so.
Group Purchasing
Organizations
(GPOs)
Collective healthcare bargaining
organization negotiates pricing, and
discounts of drugs with wholesale
manufactures
Significant- Directly drives market demand by
negotiation pricing and discounts with wholesale
manufacturer. Often impacted by shortages and
compete to keep cost low per member agreements
Similar challenges as PBMs and also
indirect measurement of client
demands/needs/negotiated contracts
Payers (Private and
Public Insurers)
Ultimate payer that influences drug
formulary and pricing structures, can be
done through PBMs or self-administered
Somewhat Significant- Although sets reimbursement
pricing, often provide structured based reimbursement
for drugs and follow formulary and sets contracts with
PBM
Difficult to survey and likely narrowed to
a single small private insurer willing to be
surveyed. Difficult to conclude who in the
organization will be surveyed

169
Stakeholder Role/Interest Influence on Subject Matter Survey Considerations
Hospital / Clinics /
Formularies
Provider that influences drug selection
through formulary, often uses PBMs, and
also incurs labor costs to manage drug
shortages. May be more cognizant of
legal obligations of institution in
delivering high-level medical care
Little/Somewhat Significant- Influence is often
through formulary and PBMs. Legal considerations of
providing high-level care may be of more importance
as it may require exclusive demand for high-quality
products that meet all expected standards for protect
from negligence lawsuits
Survey must be narrowed to a single
institution medical directors/pharmacy
managers. Captures niche area and/or
limited geographical area. May mostly
provide view on decision making of
graded products but not on metrics.
Regulatory Agency Focus on promoting and protecting public
health. Current quality inspector and may
be provide quality rating information.
Tasked with mitigation drug shortage
within current legal reach
No Influence on manufacturer’s investment and
production/inventory decisions, and pricing.
Strong influence on potential quality rating of a firm
as organization holds expertise in evaluating quality
and quality systems per regulations
Cannot be surveyed

170
APPENDIX B
NEAR-FINAL SURVEY USED FOR QLN ASSESSMENT

171

172

173

174

175

176

177

178
APPENDIX C
FINAL SURVEY USED FOR EXPERT PANEL

179

180

181

182

183

184

185

186
APPENDIX D
FINAL SURVEY USED FOR OPEN-ACCESS POPULATION

187

188

189

190

191

192

193

194

195
APPENDIX E
OTHER PUBLIC METRICS THAT RESPONDENTS FELT WERE IMPORTANT
Expert Panel Responses

196

197
Open-Access Survey Responses

198

199
APPENDIX F
FREQUENCY OF QUALITY PROBLEMS EXPERIENCED BY MANUFACTURERS
Expert Panel Responses

Open-Access Survey Responses

200
APPENDIX G
OTHER INTERNAL METRICS THAT RESPONDENTS FELT WERE IMPORTANT
Expert Panel Responses

201
Open-Access Survey Responses

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

FDA influence on advisory committees through documentation: a content analysis and survey of industry views

PDF

Examining the regulatory framework for drug compounding: industry views and experiences

PDF

An integrated framework to evaluate customer service delivery: a study of electronic systems at FDA's Los Angeles Import Operations Branch

PDF

Challenges in the implementation of Risk Evaluation Mitigation Strategies (REMS): a survey of industry views

PDF

Use of natural colors: experience and views in pharmaceutical and dietary supplement industries

PDF

Evaluation of FDA-sponsor formal meetings on the development of cell and gene therapies: a survey of industry views

PDF

Regulating cosmeceuticals in the United States: a cosmetic industry view

PDF

Risk management and recalls: a survey of medical device manufacturers

PDF

Effect of GDUFA legislation on the development and approval of generic drugs: a survey of industry views and experiences

PDF

The impact of FDA Patient-Focused Drug Development (PFDD) on US drug development strategies: a survey of views from pharmaceutical product companies

PDF

Regulatory dissonance in the global development of drug therapies: a case study of drug development in postmenopausal osteoporosis

PDF

Regulatory harmonization in a resource-limited setting: the World Health Organization Collaborative Procedure for Accelerated Registration

PDF

Clinical trials driven by investigator-sponsors: GCP compliance with or without previous industry sponsorship

PDF

Implementation of tobacco regulatory science competencies in the tobacco centers of regulatory science (TCORS): stakeholder views

PDF

Views on global harmonization of pharmacopeial standards: a survey of key stakeholders

PDF

Risk approaches and standards used in hospitals: a survey of industry views

PDF

Examining the cord blood industry views on the biologic license application regulatory framework

PDF

Reprocessing of single-use medical devices: a survey investigation comparing the views of three unheard stakeholders

PDF

The impact of incomplete monographs on the OTC drug industry: a survey investigation of industry views

PDF

Convergence of United States regulatory and reimbursement policies impacting patient access to humanitarian use devices (HUD)

Asset Metadata

Creator Medina, Cesar (author)

Core Title Incentivizing quality in the manufacture of pharmaceuticals: manufacturers' views on quality ratings

School School of Pharmacy

Degree Doctor of Regulatory Science

Degree Program Regulatory Science

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag drug shortages,GMP,incentives,metrics,OAI-PMH Harvest,scorecards

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Richmond, Frances J. (committee chair), Kuo, Benson (committee member), Loeb, Jerry (committee member), Pacifici, Eunjoo (committee member), VanTrieste, Martin (committee member)

Creator Email cesar70md@msn.com,cesarmed@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-589100

Unique identifier UC11299129

Identifier etd-MedinaCesa-3569.pdf (filename),usctheses-c3-589100 (legacy record id)

Legacy Identifier etd-MedinaCesa-3569.pdf

Dmrecord 589100

Document Type Dissertation

Format application/pdf (imt)

Rights Medina, Cesar

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA