University of Southern California Dissertations and Theses

Benefits-risk frameworks: implementation by industry

(USC Thesis Other)

Benefits-risk frameworks: implementation by industry

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content

BENEFITS-RISK FRAMEWORKS:

IMPLEMENTATION BY INDUSTRY

by

Nancy Pire-Smerkanich

A Dissertation Presented to the
FACULTY OF THE USC SCHOOL OF PHARMACY
UNIVERSITY OF SOUTHERN CALIFORNIA
In Fulfillment of the
Requirements for the Degree
DOCTOR OF REGULATORY SCIENCE

May 2016

Copyright 2016 Nancy Pire-Smerkanich

2
DEDICATION
I dedicate this dissertation to my husband Mark for his patience, understanding, love and
support and to my daughters Kyra and Natalie, of whom I am exceedingly proud and who
have taught me more than any academic exercise ever could. I would also like to
dedicate this to my late father, Walter, who instilled in me the value of education and a
love of learning (Walter P. Pire 1926-2015).

3
ACKNOWLEDGEMENTS
I would like to thank all of my colleagues throughout the pharmaceutical industry whose
support and inspiration guided me throughout my career and whose participation in my
survey made this dissertation possible. It has been my honor to work with all of you over
these last 30 years.
I would like to thank my thesis supervisor, Dr. Frances Richmond PhD, for all of her
guidance and encouragement and for her patience in reviewing the many iterations of this
dissertation. Her support has helped me turn the page on this next chapter of my life. I
would also like to thank my thesis committee members, Eunjoo Pacifici, Pharm D, PhD,
Michael Jamieson, DRSc and Elliott Berger, PhD for the invaluable feedback that they
provided me throughout the entire process. Elliott encouraged me many years ago to
pursue an advanced degree and has been my “sustaining mentor”. In addition I would
like to acknowledge the help that Randa Issa, PhD, has given me during both my master's
and doctoral programs while I tried to navigate through all of the administrative hurdles.
In closing, I would like to thank the other students in the 2012 Doctoral Cohort and the
staff of the Regulatory Science program for their support, both personally and
professionally, over the last three years.

4
TABLE OF CONTENTS
DEDICATION................................................................................................................... 2
ACKNOWLEDGEMENTS ............................................................................................. 3
LIST OF TABLES ............................................................................................................ 9
LIST OF FIGURES ........................................................................................................ 11
ABSTRACT ..................................................................................................................... 13
OVERVIEW ..................................................................................... 14
1.1 Introduction ................................................................................................ 14
1.2 Statement of the Problem .......................................................................... 20
1.3 Purpose of the Study .................................................................................. 22
1.4 Importance of the Study ............................................................................ 23
1.5 Limitation, Delimitations, Assumptions ................................................... 24
1.6 Organization of Thesis ............................................................................... 26
1.7 Definitions of Key Terms ........................................................................... 27
LITERATURE REVIEW ............................................................... 28
2.1 Introduction ................................................................................................ 28
2.2 Background to the Literature Review ...................................................... 30
2.3 Benefit-Risk Assessment: History and Current State ............................. 32
2.3.1 The Influence of Transparency Initiatives ................................. 32
2.3.2 Regulators as Communicators of Risk and Benefit ................... 37
2.4 Industry as a Communicator of Benefits and Risks ............................... 41

5
2.4.1 External Communications from Sponsors ................................. 41
2.4.1.1 Preapproval ...................................................................... 41
2.4.1.2 Post-approval ................................................................... 43
2.5 Methods for Benefits Risk Assessment ..................................................... 47
2.5.1 Qualitative Approaches ................................................................ 47
2.5.1.1 FDA Benefit Risk Framework (BRF) ............................ 48
2.5.1.2 EMA PrOACT-URL ....................................................... 52
2.5.1.3 UMBRA ............................................................................ 56
2.5.1.4 PhRMA BRAT ................................................................. 59
2.5.1.5 IMI PROTECT ................................................................ 64
2.5.2 Quantitative Methodologies ......................................................... 65
2.5.2.1 Multi-Criteria Decision Analysis (MCDA) .................... 68
2.5.2.2 Stochastic Multi-criteria Acceptability Analysis
(SMAA) ............................................................................. 71
2.6 Benefit-Risk Assessment: A Changing Landscape .................................. 74
2.7 Approach to the Research ......................................................................... 76
METHODOLOGY .......................................................................... 79
3.1 Introduction ................................................................................................ 79
3.2 Development of Initial Survey .................................................................. 79
3.3 Focus Group and Survey Finalization ..................................................... 80
3.4 Survey Deployment and Analysis ............................................................. 82

6
RESULTS ......................................................................................... 84
4.1 Analysis of Survey Results ........................................................................ 84
4.2 Profiles of Respondents ............................................................................. 85
4.3 Current Status of Benefit-Risk Assessment ............................................. 89
4.3.1 Benefit-Risk Assessment during Development .......................... 89
4.3.2 Benefit-Risk Assessment as Part of Regulatory Submissions ... 98
4.4 Benefit-Risk Framework Knowledge and Perceptions ......................... 106
4.4.1 Utilization of BR Framework Elements ................................... 109
4.5 Implementation Issues ............................................................................. 111
4.5.1 Location of BR Assessment Elements ....................................... 117
4.5.2 Feedback and Influence of BR Assessment Frameworks ....... 118
4.6 Advantages and Barriers to Implementation of BR Frameworks ....... 121
4.6.1 BR Framework Attributes Important to Overcome
Barriers........................................................................................ 126
DISCUSSION ................................................................................. 138
5.1 Benefit-Risk Frameworks and their Implementation by Industry ...... 138
5.2 Methodological Considerations .............................................................. 139
5.2.1 Limitations .................................................................................. 139
5.2.1.1 Sampling of Respondents .............................................. 139
5.2.1.2 Survey Methodology ...................................................... 140
5.2.1.3 Survey Instrument ......................................................... 142

7
5.2.2 Delimitations ............................................................................... 143
5.3 Consideration of Results .......................................................................... 146
5.3.1 BR Maturity and Implementation ............................................ 146
5.3.1.1 During Development ...................................................... 146
5.3.1.2 During Submissions ....................................................... 149
5.3.2 BR Frameworks as a Communication Tool ............................. 151
5.4 Maturity of Framework Implementation .............................................. 151
5.4.1 Challenges Related to Knowledge of Frameworks .................. 152
5.5 Comparison with Prior Research ........................................................... 154
5.5.1 Use of Frameworks ..................................................................... 155
5.5.2 Utility/Advantages and Barriers to Implementation of BR
Frameworks ................................................................................ 158
5.5.3 Harmonizing Frameworks Globally ......................................... 158
5.6 Conclusions and Future Directions ........................................................ 160
REFERENCES .............................................................................................................. 163
APPENDIX A. NUMERICAL REPRESENTATIONS ........................................ 169
A.1 Basic Terms .............................................................................................. 169
A.2 Indices ....................................................................................................... 170
A.2.1 Computational Indices ............................................................... 170
A.2.2 Health Utility Indices .................................................................. 171
A.2.3 Other Useful Indices ................................................................... 172

8
A.3 Visual Modeling ....................................................................................... 172
A.4 Organizing Approaches ........................................................................... 174
APPENDIX B. FINAL VERSION OF BR FRAMEWORK SURVEY .............. 175
APPENDIX C. RESULTS REPORTS COMMENT SECTION ......................... 186
C.1 Results Report – Expert Panel ................................................................ 186
C.2 Results Report – TOPRA Panel .............................................................. 238
APPENDIX D. ADDITIONAL CROSS TABULATIONS ................................... 278

9
LIST OF TABLES
Table 1: PrOACT-URL Framework ..........................................................................54
Table 2: Multi-Decision Criteria Analysis Model .....................................................69
Table 3: Focus Group Participants ............................................................................81
Table 4: Comments
a
on Internal Approaches Used During Development
(EP Group) ...............................................................................................90
Table 5: Additional Comments on Semi-Quantitative Approaches Used
Internally During Development (EP Group) .........................................92
Table 6: Cross Tabulation of Size of Company and BR Approach Utilized
During Development ................................................................................94
Table 7: Other Reasons for EP Dissatisfaction with Current BR Approach .........97
Table 8: Cross Tabulation of Size of Company and BR Approach Utilized
for Regulatory Submissions ..................................................................101
Table 9: BR Assessment Alignment - By Importance (EP Group) .......................106
Table 10: BR Framework Level of Knowledge of the Expert Panel .......................108
Table 11: Implementation of BR Frameworks (EP Group) ....................................112
Table 12: Implementation of BR Frameworks (TOPRA Group) ...........................114
Table 13: Implementation of Models/Indices (EP Group) .......................................116
Table 14: Comments on BR Assessment Influence on Regulators (EP
Group) .....................................................................................................121
Table 15: Advantages of BR Utilization (EP Group) ...............................................122
Table 16: Additional Advantages to Utilizing BR Frameworks (EP Group) .........123
Table 17: Barriers to Implementation of BR Frameworks ......................................125

10
Table 18: BR Framework Implementation Attributes (EP Group) ........................128
Table 19: BR Framework Utility (EP Group)...........................................................131
Table 20: Stakeholder Engagement Comments ........................................................135
Table 21: Additional Comments from the Expert Panel ..........................................137
Table 22: Comparison of Usage Patterns of Six Models/Indices between
2011 and 2015 .........................................................................................157
Table D.1: Cross Tabulation of Company Size and Models/Indices
Utilization................................................................................................278
Table D.2: Company Size and Reason for Choosing Method ...................................282
Table D.3: Company Size and Framework Implementation .....................................283
Table D.4: Company Size and Models/Indices Utilization.........................................287
Table D.5: Company Size and Perceived Barriers .....................................................289
Table D.6: Company Size and ICH M4E (R2) ............................................................291
Table D.7: Size of Company and Stakeholder Involvement ......................................292
Table D.8: Size of Company and Feedback.................................................................293
Table D.9: Size of Company and Where BR Assessment is Used .............................294
Table D.10: Size of Company and Most Important Barrier ...................................295

11
LIST OF FIGURES
Figure 1: Balancing Benefits and Risks .......................................................................28
Figure 2: ICH Q9 Quality Risk Management.............................................................34
Figure 3: Communication Framework ........................................................................35
Figure 4: US FDA Benefits Risk Framework (BRF) ..................................................49
Figure 5: Sample Effects Table ....................................................................................55
Figure 6: UMBRA Framework ....................................................................................57
Figure 7: BRAT Framework ........................................................................................59
Figure 8: Value Tree(s) .................................................................................................61
Figure 9: Key Benefit Risk Summary Table ...............................................................63
Figure 10: IMI PROTECT .............................................................................................65
Figure 11: Sample Visualization - Forest Plot ..............................................................70
Figure 12: Sample Visualization - Bar Graph ..............................................................71
Figure 13: Roles Played by Respondents to the Expert Panel ....................................86
Figure 14: Geographic Areas of Responsibility of Respondents to Two
Surveys ......................................................................................................87
Figure 15: Company Size of EP Survey Participants ..................................................88
Figure 16: BR Framework Approaches Identified by EP Respondents for
Use During Development .........................................................................91
Figure 17: BR Approaches .............................................................................................93
Figure 18: Satisfaction with BR Approach During Development ..............................95
Figure 19: Reasons for EP Dissatisfaction with Current BR Approach
During Development ................................................................................96

12
Figure 20: BR Assessment for New Medicines .............................................................98
Figure 21: BR Framework Approaches for Regulatory Submissions ........................99
Figure 22: BR Framework Approaches for Regulatory Submissions ......................102
Figure 23: Reasons for EP Dissatisfaction with Current BR Approach for
Submissions ............................................................................................103
Figure 24: Reasons for not using BR Frameworks ....................................................104
Figure 25: Expert Panel BR Framework Familiarity – Number of Responses .......107
Figure 26: BR Approaches in Regulatory Submissions .............................................110
Figure 27: Location of BR Information in Marketing Applications ........................118
Figure 28: Feedback from Regulatory Authorities on BR Frameworks .................119
Figure 29: Barriers to Implementation of BR Frameworks by Importance ...........126
Figure 30: Factors to Consider for ICH M4 (R2) Revision .......................................133
Figure 31: Stakeholder Engagement in BR Frameworks ..........................................134
Figure A.1: Sample of Net Clinical Benefit Display ................................................173

13
ABSTRACT
Numerous benefit-risk (BR) methodologies of varying complexity have been developed
to evaluate drug products but the degree to which these different frameworks are used
across industry is not yet well understood. The research presented here extends our
understanding of the views of industry, a key stakeholder in BR assessment, with regards
to the implementation of tools and methodologies. Survey methods were used to
examine the views of two target groups of respondents, 1) senior regulatory professionals
and consultants who participate in BR activities during drug development and regulatory
submissions, approached personally and 2) members of a European professional
organization, approached through a web-based interface and used to provide some
measure of external validity. The primary data analysis was conducted on responses of
the first group that included 87 participants from large and small companies with a
response rate of 74%. Results showed that formalized BR frameworks are perceived as
having value to enhance transparency and support decision making, yet they are not being
used by many companies. Factors that appear to inhibit uptake of formalized B-R
approaches appear to include poor awareness and expertise with the frameworks
themselves, perceived immaturity of the methodologies and the lack of a globally
harmonized framework that could be applied in all jurisdictions.

14
OVERVIEW
1.1 Introduction
Since the 1930s, US drug laws have given the task of weighing risk and benefit of a new
product to regulators as a precondition of marketing approval. Similar laws with similar
regulatory requirements exist in most of the major developed countries of the world. In
the US, the authority to review the balance between risk and benefit has been assigned to
the Food and Drug Administration (FDA). Unsurprisingly, most attention in the past has
focused on the risk part of the benefit-risk equation, to avoid the repetition of public
health calamities that have occurred over the last century. The original Food, Drug and
Cosmetic Act (FD&C Act) of 1938 was enacted in large part as a reaction to a tragedy
involving over 100 deaths, mostly of children, who ingested an adulterated elixir. This
law required manufacturers to submit proof of safety to the Food and Drug
Administration (FDA) prior to marketing and selling a new drug (Woosley, 2013).
Typically, the risk of a product is now evaluated through a series of safety studies in
animals and humans during drug/biologic product development and simply monitored in
the post-marketing setting after approval.
The benefit of a product is often evaluated in terms of its efficacy or effectiveness, proof
of which became a requirement when the Kefauver-Harris Amendment to the FD&C Act
was enacted in 1962. This legislative measure also came as a consequence of a tragedy
in which the drug thalidomide, given to pregnant women for morning sickness, caused
fetal malformations. Only if the FDA could weigh both efficacy/effectiveness (Mt-Isa,

15
Tzoulaki, Callréus, Micaleff, & Ashby, 2011) and safety (risk or harm) of a new medical
product was it felt that the FDA could make an appropriate and balanced decision that
was in the interest of patients.
The original rules that governed benefit-risk assessment were largely directed at
demonstrating and evaluating the balance of risk and benefit in the premarket or approval
phase of product development. In the last decade, however, there have been some highly
publicized cases of post-marketing statistical evidence coming to light after drugs have
been approved, the most notorious example being the cardiovascular problems with
Vioxx, a COX2 inhibitor, which were associated with long-term use and led to its
withdrawal from the market. Safety issues such as these are typically detected only after
the product had been marketed for several years (Avorn, 2012). These findings were in
large part responsible for a new wave of concern about benefit-risk evaluation. They
spurred regulators in the US and Europe to put in place new rules to increase the
effectiveness of post marketing safety surveillance and continued assessment of benefit
and risk in later phases of the product life cycle. The focus has been skewed to the risk
side of the relationship; most systematic studies dealt with risk identification and
estimation (Eichler, Abadie, Raine, & Salmonson, 2009). Harmonization efforts in the
areas of Risk Management and Pharmacovigilance, led by Working Groups from the
Council for International Organization of Medical Sciences (CIOMS) and the
International Conference on Harmonization (ICH), have been successful at providing
better guidance for industry for these risk assessments, by harmonizing tools such as the
use of standardized coding for adverse drug reactions, [e.g. Medical Dictionary for

16
Regulatory Activities (MEDRA)] and the development of ICH E2E “Pharmacovigilance
Planning” as well as the ICH E2C “Periodic Benefit Risk Evaluation Report (PBRER)”.
These latter formats are now slowly replacing the more traditional Periodic Safety Update
Reports (PSUR) with modified approaches that assess risk in the context of benefit
estimates. They exemplify the more recent trend to identify and estimate benefit in a
more methodical way. However, in order to estimate benefits, a more systematic
approach appeared to be needed. Thus, over the last decade, a series of initiatives have
laid the groundwork for the characterization of benefits using better tools to add value
and transparency to the estimation of benefit. Thus, increasingly the language of risk-
benefit has changes to the language of benefit-risk.
The United States has been one of the first countries to rethink the ways that they deal
with benefit-risk assessment. In 2006, the Institute of Medicine (IoM) published a report
recommending that researchers from both academia and industry investigate new
approaches to conceptualizing, measuring and applying benefit-risk analysis. They
further recommended that the Center for Drug Evaluation and Research (CDER) within
FDA should “develop and continually improve a systematic approach to benefit-risk
analysis for use throughout the FDA in the pre- and post- approval settings” (IoM, 2006).
This recommendation was followed, in 2009, by an initiative led by CDER to develop a
structured approach for the benefit-risk assessment of drugs that could serve as a template
for new product reviews and help to clarify the basis for FDA’s regulatory decisions.
This new approach was seen to align with another requirement placed on the organization
by the President of the United States in a Memorandum to the Heads of Agencies on

17
Transparency and Open Government. This memorandum called for all government
agencies to “take appropriate action, consistent with law and policy to disclose
information rapidly in forms that the public can readily find and use” (Obama, 2009) .
Some of the goals to which the FDA Transparency Initiative had committed were issues
dealing with better communication and a more transparent review process that would
provide the public with an understanding of FDA decision-making.
The two separate but related imperatives to spur more open communication with
stakeholders were reinforced when the FDA Safety and Innovation Act (FDASIA) was
passed in 2012. It reauthorized the Prescription Drug User Fee Act (now-called
PDUFA V) in which the FDA was allowed to collect fees from industry for the review of
new drug approvals, but only by agreeing to specific performance goals. As part of the
most recent performance goals accepted as part of the PDUFA V Implementation Plan,
the FDA committed to using a structured approach to assess the benefits and risks in the
drug regulatory decision making process and to communicate this aspect of the
assessment process more effectively (IoM, 2014).
At the same time, European activities related to benefit-risk were also taking place. The
European Medicines Agency’s (EMA) Committee for Medicinal Products for Human
Use (CHMP) recommended that they too utilize a structured approach for evaluating the
relative benefits and risks associated with new medicines as part of their public
assessment reports (PARs). They formed a working group to assess various methods for
risk-benefit assessment that were under development by several academic and
government-industry partnerships and established working relationships of their own

18
with cooperative groups such as the Innovative Medicine’s Institute (IMI) (Mt-Isa et al.,
2011).
The fact that the regulatory agencies of the two dominant players in the drug
development arena, the US and Europe, were investing substantial resources in the study
of risk-benefit frameworks suggested that change could be anticipated in the way that
drugs would be reviewed. However, in the past, when separate initiatives have taken
place in different constituencies, the resulting approaches have been dissonant and have
created different standards. For this reason, in the latter part of 2014 the International
Conference on Harmonization (ICH) formed an Expert Working Group (EWG) to
evaluate the current multidisciplinary guideline for the Common Technical Document
(CTD) Clinical/Efficacy sections – M4E (R2), a section most directly related to benefit
assessment. As part of their November 2014 meeting they also published the final
concept paper, business plan and work plan to address the revisions that may be needed
in order to “to include greater specificity on the format and structure of benefit-risk
information with the goal of harmonizing the presentation of this information in
regulatory submissions”(ICH, 2014). They also noted:
…this proposal aims to level-set all key stakeholders in drug regulation with
respect to what is important in regulatory decision-making for
pharmaceutical products. Greater structure in the presentation of this
information should aid regulators in understanding an applicant’s
perspective on the Benefit-risk assessment. Similarly, applicants will have a
clear understanding of what is important to a regulator’s Benefit-risk
decision.
The need for harmonization of the methodologies and/or the presentation of BR
Assessment has been identified in multiple forums by industry, including work within

19
their trade organizations. As noted by the Biotechnology Industry Organization (BIO), in
its Comments on the FDA Draft Structured Benefit-Risk 5-Year Plan (BIO, 2013),
We encourage FDA to continue engaging in dialogue with other regulatory
agencies on methodologies of assessing Benefit-risk that acknowledge and
reflect the global endeavor of drug development and regulation in which
sponsors operate. This discourse can serve to advance the goal of developing
a globally harmonized framework for Benefit-risk evaluation to support
decision making.
The implementation of a consistent approach has been challenging. Key to the
implementation of a structured benefit-risk assessment is the adoption of a singular
framework to structure the information and activities. As noted by Lawrence Liberti and
his colleagues, “the use of decision frameworks with an agency should encourage a
quality review leading to well-informed, quality decision making with meaningful,
well-communicated outcomes” (Liberti et al., 2013). Liberti also went on to note that
these same frameworks could be used by companies to structure discussions around
internal assessment of study findings and to help inform development decisions as well as
guide the preparation of marketing applications. However, many frameworks of differing
complexity exist and continue to be developed. In addition to the activities of the health
authorities, trade organizations such as BIO, referenced above, the Pharmaceutical
Research and Manufacturers Association in the US (PhRMA), the European
Pharmaceutical Industry Association (EfPIA), the non-profit Center for Innovation in
Regulatory Science (CIRS) and other public-private partnerships have all been active in
the development of their own frameworks and work products relevant to benefits-risk
assessments for drugs. Some of these approaches are very simple, whereas others rely on
complex quantitative methods.

20
Two key resources that summarize these varying approaches include the seminal work of
Filip Mussen, who with Sam Salek and Stuart Walker, published a comprehensive
textbook, “Benefit Risk Appraisal of Medicines: A Systematic Approach to Decision
making”(Mussen, Salek, & Walker, 2009) and a more recent publication (Stuart Walker,
McAuslane, Liberti, Leong, & Salek, 2015) entitled “A Universal Framework for the
Benefit-Risk Assessment of Medicines: Is This the Way Forward?”. Both works note
the way in which the multiplicity of qualitative and quantitative tools that have become
available and are now used by various stakeholders in the drug development and
commercialization sector. Such information can help stakeholders to shape their research
and regulatory submissions in ways that streamline and support the benefit-risk
assessment process.
1.2 Statement of the Problem
We know much about the types of benefit-risk methodologies that are available for use to
evaluate drug products (Mussen et al., 2009). Further we know from many documents
and presentations by authoritative figures from the FDA and European regulatory
agencies that they are taking this exercise seriously and are using specific but differing
frameworks for their analysis of new products. We also know much about the preferences
of regulatory agencies with respect to the use of specific frameworks for their own
benefit-risk assessments. These choices have been discussed extensively at professional
seminars and trade meetings, and are captured in a number of guidance documents and
other materials. However, we know much less about the views of other stakeholders
regarding the use of benefit-risk analyses.

21
Perhaps the most important alternative stakeholder in the benefit-risk discussion is the
industry sponsor of a new product who seeks a positive decision from the regulatory
authority that is evaluating its drug so that it can go to market. Relatively little
systematic research has been directed at the views and activities of this stakeholder. The
best insight into these views has come from survey conducted by James Leong and
colleagues in 2011 which explored the status of and need for a universal benefit-risk
framework in regulatory agencies and pharmaceutical companies (Leong, McAuslane,
Walker, & Salek, 2013). This survey showed that both regulators and pharmaceutical
companies prefer to have a semi-quantitative, overarching framework that incorporates a
variety of different methodologies. They also express the view that the approaches are not
mature and more work needs to be done with stakeholders to guide the development and
implementation of these tools.
The research presented here is intended to extend our understanding of the views of
stakeholders with regard to the implementation of benefits-risk tools and methodologies.
It would seem important for industry to recognize that it has a role to play in supporting
the activities of the regulators, by aligning the materials they provide in their regulatory
submissions and to facilitate the types of benefit-risk analyses that will be undertaken on
the data that they present. Not only do these considerations affect the way that they
submit their materials to the regulator, they may also affect the way that they design their
communications with consumers. As part of benefit-risk activities, regulatory agencies
increasingly regard the communication of risk as an important undertaking that
compliments their transparency initiatives, and have developed several programs to

22
motivate companies to provide better information to consumers across the life-cycle of a
product rather than just at its premarket stage. However, it is not clear if comprehensive
strategies to evaluate and communicate benefit and risk are being developed by different
pharmaceutical companies. If such strategies are in place, it is not clear what might
constitute best practices in this regard.
1.3 Purpose of the Study
The establishment of many benefits-risk methodologies and frameworks has been well
documented, but we do not know if and how companies are using them. The goal of this
study is to explore the approaches and methodologies used to evaluate benefit-risk within
companies making medical products, and the extent to which they are used to support the
regulators’ evaluations of benefit-risk when submissions are made to those agencies. It
will also be important to determine how the outcomes of benefits-risk analysis are used
for internal or external communication. The research further seeks to identify whether
companies of different sizes during different phases of product development use different
approaches and tools. Previous studies by Leong about 5 years ago concentrated their
evaluations on 24 companies, all but two of which are in the “Top 20” pharmaceutical
companies based on sales volume and R&D spending. In the last 5 years, thinking on
benefits-risk assessment has matured significantly. This study will therefore use a novel
survey tool to gather and compare data from pharmaceutical and biotechnology
companies of different sizes and with products in different stages of development in order
to gain a better understanding of the industry as a whole. Respondents to the survey will
include regulatory professionals at various levels and positions, defined as individuals at

23
the manager level and above, as well as consultants to companies who are participating in
submission related activities.
1.4 Importance of the Study
Policies are most effective when they are developed with a clear understanding of the
views of all stakeholders. This research is intended to give better insight into the views
of one particularly important stakeholder, the biopharmaceutical industry that develops
drugs to improve patient health. A better knowledge of these views might help to
identify specific areas of concern and dissonance related to the implementation of these
methodologies and their use to communicate benefits and risks. The results may also
help companies to benchmark their activities related to peer organizations. Since much
of current innovation and drug development is taking place in companies smaller than
those whose views have been previously examined, or who do not participate in trade
associations or international harmonization efforts, research is particularly needed within
this essential segment of industry. If the survey shows a lack of understanding about the
availability, utility and/or regulatory adoption of benefit-risk methods, regulators might
find this information useful in determining how they can best reach their stakeholders.
The findings may illuminate gaps in the way that the existing body of knowledge is being
disseminated within the regulated medical product community. As the editors of
“Benefit-Risk Assessment in Pharmaceutical R&D” (Sashegyi, Felli, & Noel, 2014)
concluded, “For the future of benefits-risk analysis to evolve, there will be a need for
education”. Regulatory scientists are in a unique position with respect to “ground level
implementation of benefits-risk assessments, particularly drug development experts with

24
keen facilitation skills, who are at ease communicating across disciplines”. This study
will enable those involved in benefits-risk assessment to provide education and training
as the first steps to filling that gap.
1.5 Limitation, Delimitations, Assumptions
There are several limitations that must be recognized as part of this thesis. First, this
study will be limited by the availability of the respondents and their knowledge of the
benefit-risk assessment process as well as their ability to report on it. The substantial
level of experience of regulatory professionals expected of survey participants will be a
factor which may limit the study. Identifying individuals to be surveyed and obtaining
sufficient numbers to insure a balanced and fair analysis may be difficult, given that busy
individuals can be reluctant to participate in the study. Also, the intention to obtain a
cross section of biopharmaceutical companies based on size may result in relatively small
numbers in each group, so caution will be needed when trying to use the small sample as
a fair representation of that industry segment. In addition, depending on where in the
development cycle of medical products a company is positioned may influence their
knowledge and ability to respond to the questions in the survey.
A second potential limitation is related to the nature of the tool and its ability to probe the
research issues fully. Using a focus group to critique the survey will help to ensure better
validity, but limitations still exist related to the capabilities and biases of the researcher
and the maturity of the field.

25
Finally, we will be making the assumption that the individuals participating in the study
will answer the questions honestly based on their current level of understanding.
Sensitivity to this issue will require anonymity to be promised, and this limits the ability
to relate specific data to identifiers of the respondent and his/her employer.
This research is delimited to the examination of the use of benefit-risk frameworks;
primarily in the area of drug development, not devices or biologics. It does not extend its
detailed evaluations to other aspect of risk or benefit activities that might take place for
example, as part of quality assurance functions. Further, it will focus primarily on the use
of “well known” frameworks which are included in recent literature and public
presentations. The goal of the research is find out if and how companies are utilizing
these frameworks rather than trying to invent new options.
Second, the research is delimited to industry sponsors, and thus will not solicit the views
of regulators, since they have already presented their methodologies in public forums.
Further they cannot add information about the use of benefits-risk frameworks in
individual companies due to confidentiality constraints.
The study was further delimited by focusing on those frameworks and methods used by
three regulatory authorities/groups – FDA, EMA and the consortia of Canada, Australia,
Singapore and Switzerland (CASS, formerly COBRA) and the methodologies that have
been published or presented by them. Respondents will have the opportunity to describe
other frameworks as well in the comments sections. In addition, this research is delimited
by the selection of literature that was reviewed. There are tens of thousands of
publications that deal with benefits and risks of biopharmaceutical products, many of

26
which look at specific products and classes of drugs. Only those dealing with the
frameworks and methodologies were reviewed.
Another delimitation will be the number of questions that are presented to the survey
recipient’s bases on the likelihood of assuring that they will complete the survey. In
limiting the number of questions, certain topics may not be fully explored and this in turn
may constrain the analysis.
1.6 Organization of Thesis
Chapter 1 introduces the problem to be studied in this thesis and summarizes briefly the
rationale for the approach that will be taken.
Chapter 2 reviews relevant literature. It includes examines the nature of transparency, the
methods available to communicate benefits and risks, and the manner in which they are
currently used, and the framework that will structure the research to follow.
Chapter 3 describes the methodology used in conducting the survey and includes all
questions as they were developed and used to gather the data.
Chapter 4 will describe the results of the study in the form of a narrative, tables and
graphs.
Chapter 5 will discuss the implications of the research. It interprets the results in the
context of other literature and includes conclusions and recommendations for further
implementation and research.

27
Appendices to this thesis include A) a list of terms and acronyms used throughout this
thesis including detailed descriptions of the numerical representations presented as part of
benefits-risk analyses; B) the survey that was distributed; C) Results reports for both the
Expert Panel (EP) and TOPRA panel; and D) Cross tabulation tables.
1.7 Definitions of Key Terms
A full list of all acronyms used in this thesis as well as additional descriptions of the
indices and statistical representations is provided in Appendix A to this thesis. However,
certain key terms essential to any discussion of this topic are defined here to assure
consistency of interpretation:
Benefit – Merriam-Webster dictionary provides the simplest yet most relevant definition
for benefit as a “good or helpful result or effect”. Effect in biopharmaceutical research
and development is measured as “efficacy” or “effectiveness”.
Risk – the same dictionary also defines risk most appropriately as “the possibility that
something bad or unpleasant (such as an injury or a loss) will happen”.

28
LITERATURE REVIEW
2.1 Introduction
It is a well-known fact that drugs can be dangerous products. Their appropriate use
depends on the ability to balance carefully their benefits and risks at several levels.
Patients and health-care providers require good information to make choices appropriate
to the individual, and regulators must assure that benefits and risks are balanced for a
particular population (Figure 1).
Figure 1: Balancing Benefits and Risks
(FDA, 1999)

If benefits and risks are to be compared and even quantified, it is important that they first
be defined. However, no standard or widely acknowledged definitions of these terms
exist (Luteijn et al., 2012). A typical dictionary derived definition of benefit, such as that

29
from Merriam Webster, defines benefit as a “good or helpful result or effect”
(http://www.merriam-webster.com). In biopharmaceutical research and development
benefit is typically measured as “efficacy”, the power to produce a desired result or effect
or “effectiveness”, producing a decided, decisive, claimed, or desired effect
(http://www.merriam-webster.com). In this context, efficacy is the probability of
expected benefit to individuals in a defined population (Brook & Lohr, 1985). In the
literature, benefit is often considered as something that is quantified in absolute terms.
For example, a drug may have the ability to reduce the incidence of miscarriage by 30%
or to lower blood pressure by 10±2 mm Hg. The term, risk, however, is often quantified
as a statement of probability. Because the benefit and risk are estimated differently, they
can be difficult to compare. Thus, some experts, including Veatch (Veatch, 1993)
recommend the use of the term “harm” which as defined by the International Standards
Organization (ISO) in their Application of Risk Management to Medical Devices
Standard number 14971 as “physical injury or damage of the health of the people” (ISO,
2007). Considerable challenges are associated with estimating the overall benefits or
risks (or harms) of a particular drug or treatment. Thus evaluating a product in terms of
its benefits and risks can be a difficult undertaking and without specialized tools to
communicate assessments unambiguously, can be seen differently by different
stakeholders. A patient may, for example, weigh quality-of-life benefits more highly
than her oncologist, whose primary preoccupation might be to extend her life.
A key point at which the needs of all of the stakeholders come together is at the time
when a new drug is being introduced commercially. Extensive dossiers of safety and

30
efficacy information must be provided to regulatory agencies, whose reviewers then must
weigh the evidence of benefit and risk before allowing a new product on the market.
Because the reviewers must be careful to protect the confidentiality of the proprietary
information that they are given, their decisions have often been criticized as puzzling or
opaque to those outside of the regulatory agency. An increasingly vocal and influential
call for transparency in decision-making has come from critics of the drug-approval
system. As a result, regulators are implementing more transparent approaches to
benefit-risk assessments using consistent frameworks to improve the consistency and
communication of their decisions. However, different regulatory agencies globally
approach the exercise of benefit-risk assessment in different ways as elaborated below.
The fact that no standard or harmonized approach has been adopted to characterize
benefit-risk assessment at various developmental stages from one country or constituency
to another further complicates the activities of companies that operate globally. Thus,
industry will be challenged to institute parallel methods to facilitate their own decision
making and to provide well-framed materials that can guide the review of the drug by
regulatory agencies to a successful approval.
2.2 Background to the Literature Review
A first step in establishing the state of the literature concerning benefit-risk assessment is
to review the current state of publicly available material. Using the Wiley Online Library
as a starting point a literature search was conducted with the goal of obtaining scientific
publications, books, internet references and conference presentations on the benefits-risk
Assessment (benefits-risk or benefits-risk assessment) of drugs. Key words were

31
benefits, risks, assessments, frameworks, drugs and drug development. Any publications
dealing with quantitative or qualitative approaches to benefits-risk assessments and
documents dealing with “decision making” and “patient perspective” were also included.
Publications that were not in English were omitted from review. For the purposes of this
initial literature review, citations that dealt with benefits or risks (Adverse Drug
Reactions/ADR) specific to a particular drug, class of drugs or disease were retained but
not summarized in the review that follows. Also excluded were conference agendas,
educational catalogs, abstracts without specific information on benefits-risk assessment,
blog entries, and general articles related to animal/veterinary medicine. In addition,
internet-based search engines (Google, Bing and Firefox) were employed to look for
presentations that have been made on this topic; selected material is included in the
review below. It is noteworthy to mention that the topic of formalized benefits-risk
assessments is relatively new in the US, so that much of FDA’s current thinking with
regard to benefits-risk assessment is captured only in slides and abstracts from
presentations. FDA and EMA websites provided additional content and guidance
documents on the projects that they are undertaking to develop benefits-risk assessment
methodologies and these are a rich source of insight and secondary reference.
Based on the key words in any combination, over 400,000 citations were identified. By
narrowing the search terms and adopting the inclusion/exclusion criteria discussed above,
the search still yielded over 10,000 citations. For the purposes of this chapter that will
focus on the current state of the literature with respect to the qualitative and quantitative
methods used to determine benefit-risk assessments of drugs, approximately 180 articles,

32
presentations and papers appeared to be most relevant and these are the basis for the
descriptions below. In addition, two textbooks, “Benefit-risk Appraisal of Medicines: A
Systematic Approach to Decision-making” by Mussen, Salek and Walker (Mussen et al.,
2009), and “Benefit-risk Assessment in Pharmaceutical Research and Development” by
Sashegyi, Felli and Noel (Sashegyi et al., 2014) were found to be valuable to support this
research.
2.3 Benefit-Risk Assessment: History and Current State
2.3.1 The Influence of Transparency Initiatives
The assessment of risks and benefits is clearly not new to the pharmaceutical sector. In
the US, the major law that governs drug safety, the Food, Drug and Cosmetic Act of 1938
and its amendments, centers on the protection of public health by assuring that the safety
and efficacy of a drug is assessed before that drug is allowed on the market. What is new
is the insertion of a systematized framework to capture and communicate those
assessments by using a consistent rubric. A driving force to develop better approaches
has been the need for transparency with regard to the important decision-making
processes carried out by regulatory agencies.
Events of the last two decades involving corporate and government scandals have
demonstrated the need for transparency at the nexus where important decisions regarding
drug commercialization come together.

33
Transparency has been defined by Transparency International as:
A principle that allows those affected by administrative decisions, business
transactions or charitable work to know not only the basic facts and figures
but also the mechanisms and processes. It is the duty of civil servants,
managers and trustees to act visibly, predictably and understandably.
Activities related to improving transparency have gone from being a reactive opportunity
to a proactive requirement for both businesses and governments (Oliver, 2004). One key
component to such activities is effective communication (Coplan, Noel, Levitan,
Ferguson, & Mussen, 2011b). A primary reason why regulatory agencies have engaged
in their exercises to implement benefit-risk frameworks has been to communicate better
with each other and with the public about the key considerations basing decisions
regarding approvability of a medical product. Regardless of their granular structure, all
benefit-risk frameworks attempt to explain to those outside of the decision-making group
how the benefits and risks are weighted in order to come up with a decision to approve or
deny market access. Because new knowledge often emerges after products have been on
the market, these benefit-risk assessments made at the time of approval can then become
a tool to map the evolution of the benefit-risk profile of a particular product over time and
with increased experience on the market.
The evolution of thinking with regard to benefit-risk assessment in the pharmaceutical
arena appears to follow a path presaged by the evolution of risk management frameworks
in the earlier part of the decade. What started in the late 1990s as an exercise in risk
assessment rapidly evolved into a larger and more widely used framework of risk
management, as captured in the “FDA Guidance for Industry: Development and Use of

34
Risk Minimization Action Plans”, published in 2005. In this guidance, risk
communication was emphasized as a key element. For drugs a largely similar framework
was later introduced at the global level by the International Council on Harmonization of
Technical Requirements for the Registration of Pharmaceuticals for Human Use in the
guidance document ICH Q9: Quality Risk Management (2005) (see Figure 2). This
globally accepted and widely used guidance on Risk management defines “risk
communication” as
“…the sharing of information about risk between the decision makers and
others. Parties can communicate at any stage of the risk management process
and the output/result of the risk management process should be appropriately
communicated and documented” (ICH, 2005)
Figure 2: ICH Q9 Quality Risk Management

35
The ways that communications are carried out are made complex by the number of
stakeholders and the levels and types of information that is needed by them.
Communications about benefit-risk assessment therefore involve several concerned
parties that exchange information using different communication pathways. For
example, regulators must communicate with industry, health-care professionals and the
public; industry similarly must communicate to the FDA, physician or patient; patients
and physicians can also share their thoughts widely, particularly as more pathways are
introduced by social media. Although the identities of the interacting players may
change, all of the communication pathways will have at their base a similar construct,
that was characterized the seminal model of Shannon and Weaver on Linear
Communication (Weaver, 1963) (Figure 3).
Figure 3: Communication Framework

(Weaver, 1963)

In this model, every communication is assumed to have certain common features- a
source/sender, a destination/listener and a message. As communication theory has
evolved over time, additional elements such as transmission methods, receiving systems,
channels or methods of communication, noise sources and feedback or response loops
have been added to better capture the roles of different participants, channels or

36
technologies in the communication process. These add specificity and complexity to
better characterize the nature of the communication, the ways that such communications
can be degraded, and, the characteristics and needs of the receiver.
A challenge when designing a communication strategy with different types of
stakeholders is to tailor the communication channel or tool to the recipient or audience.
This consideration is particularly important when considering benefit- risk from a
historical perspective because the communication channels have been restricted and
asymmetric. In the past, industry was typically the primary “sender” of messages related
to benefit-risk. These messages were communicated to the regulatory agency through
detailed scientific dossiers and to doctors and patients in more simplified documents such
as product inserts, advertisements or website postings. The regulatory agency
traditionally had a less direct role in communication with the patients and physicians,
instead serving as a “silent partner” overseeing the veracity of the communications
between industry and customers. However, it would appear that the role played by the
regulator is under transition, from a player that indirectly assures that information is
passed on by working through the voice of the industry, to one in which the regulator
exercises a direct voice of its own. How this has happened will be described below, and
how industry views and works with these changes is the subject of this research. In the
sections that follow, the traditional role and approaches of the regulatory agencies and
industry are first identified. The more direct role that has been adopted by the regulatory
agencies is then summarized as part of a more detailed review of the development and
use of benefit–risk frameworks.

37
2.3.2 Regulators as Communicators of Risk and Benefit
As recently as a decade ago, regulators were not seen as the primary source of
information about drug risks and benefits. What the reviewers and scientists did
internally to approve and monitor drugs was largely opaque to the public and to
companies interested in their decision-making processes. The secretive nature of these
activities was justified as essential to protect the proprietary interests of the company
whose information was presented to them. Those documents which could be obtained
were accessed through a slow, cumbersome and unpredictable freedom of information
process. To help improve access to certain types of commonly requested information, the
Freedom of Information Act (FOIA) was amended in 1996 (FOIA Update Vol. XVII, No.
4, 1996) to mandate publicly accessible "electronic reading rooms", where government
agencies would post certain types of information, such as responses to previous FOIA
requests and other information thought to be of general use. However, such information
was still limited in scope by confidentiality restrictions and was often incomplete because
the staffing and tools to organize the data were in short supply. Any information not
posted on this site would still have to be requested by letter; responses to those letters
continued to be notoriously slow and often unhelpful.
Instead, most information on drug risks and benefits was communicated indirectly,
through negotiations with the industry over key documents such as drug product labels
that then would be provided to the consumer by the manufacturer. For prescription drugs,
the label included not only the printed text and graphics on the primary and secondary
product packaging, but also a more detailed description of the drug that accompanies the

38
product, called, for example, a “Package Insert” (PI) or “Prescribing Information” in the
US or a “Summary of Product Characteristics” (SmPC) in the EU. This relatively short
document assumed particular importance because it was the source of information that
brought together all of the product risks and indications for use. The PI was primarily
designed as a communication tool to be read by physicians, so was not crafted in lay
terms and thus was not always an effective communication tool for consumers. The focus
during its composition was instead to create a quasi-legal document that would define the
boundaries within which drug promotion could be confined. Although the package insert
was distributed by companies with each package of product, it became in some ways a
regulatory communication tool that was negotiated, often intensively, before approval by
the agency (Schwartz, Woloshin, & Welch, 2009). Package inserts also play a critical
role in legal proceedings in which misbranding is charged ("Misbranded drugs and
devices," 2006).
As stated by the US FDA, the primary purpose of the package insert is “to give
healthcare professionals the information they need to prescribe the drugs appropriately”
(FDA, 2013a). Thus it appears to be the place where historically one would expect to
find the information related to product benefit and risk. Benefits were typically captured
in the text outlining the indication(s) for use, and were to a modest degree amplified in
the presentation of clinical testing and outcomes in the Clinical Studies section. Risks
were amplified under the headings, Warning, Precautions, Interactions, Adverse Events
and any/all other applicable sections containing safety information.

39
However, the package insert and prescribing information provided little insight into the
decision-making process on which approval of that product had been based and
increasingly over the last decade, the package insert has been viewed as insufficient from
a risk communication perspective (Schwartz & Woloshin, 2009). It has been
supplemented by the use of more formalized tools for risk management and more
recently tools for benefit-risk management. Initially these tools were directed at
improving the neglected communication with patients and consumers, particularly for
products whose effectiveness and safety depended upon their appropriate behaviors
(Fischhoff, 2011). For example the safe use of a teratogenic drug such as thalidomide
would depend upon heightened assurance that the drug would not be taken by pregnant
women (Zeldis, Williams, Thomas, & Elsayed, 1999). The use of risk assessment and
management seems first to have taken hold in the medical products sector in the early
2000s (Woosley, 2013) but were gradually adopted by the pharmaceutical sector. By the
beginning of this last decade, both the US and EU regulatory authorities had adopted a
requirement that companies implement a risk mapping process in which communications
tools figured prominently as a way to assure that users such as health care professionals
and patients were more aware of the particular risks posed by a drug. The US approach
prior to 2007, called a Risk Management Plan (Risk Map), assigned to industry the
responsibility for developing a strategic program designed to minimize known risks of a
product while preserving its benefits (FDA, 2009). Risk Maps were developed for
particular drug products by the company to identify and act on risks that could be
mitigated by additional risk management methods beyond the already available labeling
and safety reporting. These included the implementation of communication and education

40
efforts to educate physicians and consumers about product risks that could be avoided
with additional information or behavior change (Nicholson, Peterson, & Yektashenas,
2012). Soon after, the use of Risk Management Plans was replaced by a more formalized
program that required Risk Evaluation and Mitigation Strategies, or REMS, for products
with the potential to pose risks that could be mitigated or avoided by proper education
and management of the behavior of users (Balian, Wherry, Malhotra, & Perentesis,
2010). These approaches were not needed for all products, and again were
communication strategies in which the FDA participated as a silent partner, by reviewing,
critiquing and finally approving the REMS strategy, then monitoring its implementation
and effectiveness; however, it remained the responsibility of industry to communicate
directly with stakeholders, including physicians and patients.
The aforementioned ways of communicating risk and benefit to other stakeholders did
not completely supplant the communication of benefit and risk information by FDA. For
example, some amount of product-specific technical information was published on
occasion by FDA through review documents generated during the approval process by
those individuals performing the technical assessments of submitted dossiers. Further,
once a drug was approved, a complete set of review documents as well as advisory
committee briefing materials are made public on the FDA website (www.fda.gov) .
Nevertheless, these types of documentation are typically written in the language of the
regulatory professional, are not advertised widely, and do not define in any detail how the
information was weighted to make a marketing decision. Thus this information is
unlikely to be used by the non-expert.

41
2.4 Industry as a Communicator of Benefits and Risks
2.4.1 External Communications from Sponsors
As noted above, industry has held the primary role and responsibility for risk and benefit
communication. Already described has been its role in the distribution of a product
package insert and in the execution of often-costly risk mitigation strategies contained
within the Risk Evaluations and Management programs required by the regulatory
agencies. However, there are other ways in which industry communicates to ensure that
its stakeholder parties receive better information about their therapeutic products during
development (preapproval) and after (post) approval to potential subjects/patients and the
public, investigators/prescribers and regulators.
2.4.1.1 Preapproval
Perhaps the first opportunity at which the potential benefits and/or risks are
communicated is at the time when a drug is first used in humans. During the clinical
development of a new therapy, sponsors are required to disclose to potential study
subjects the benefits and risks which can “be reasonably foreseen” as part of the informed
consent procedure ("Code of Federal Regulations," 2012). Based on this information,
potential subjects would determine whether they wished to participate in the trial.
However, the information that is supplied does not typically go much beyond the
individuals involved directly in the clinical trials, and for novel products the amount of
benefit-risk information is still embryonic. Sponsors also communicate the limited
benefit-risk information that they have at this point to investigators through the
Investigator’s Brochure (IB). The IB should include all previous animal and human

42
experience, and focuses heavily on safety/risk (ICH, 1996). This information has the
goal of assuring that the potential investigator in charge of the trial can make their own
unbiased risk-benefit assessment of the use of the drug in the proposed trial.
At the time when clinical trials are being conducted, much less regulatory oversight or
prescribed language is provided concerning the way in which benefits and risk should be
assessed and certainly no standards are applied across the industry for this purpose.
Further, the communications are directed narrowly at the parties involved in the clinical
trials. The medical community, public and investors will typically only be exposed to
highly scripted and condensed communications in the form of materials on company
sponsored websites, press releases and clinical trial listings in databases such as the
US-sponsored clinicaltrials.gov or the EU sponsored EUDRA CT
(https://eudract.ema.europa.eu/results-web). These databases require only that skeleton
features of the trials are posted to assist potential trial subjects when they consider
participating in a clinical study. They are not reviewed and updated with sufficient rigor
to gather much systematic information about benefits and risks at this stage.
Nevertheless, much work has been directed at gradually increasing the content not only
of trial protocols but also of providing the study results at the time that the trials are
completed, and even of linking publications to those trial results (Hudson & Collins,
2015). The goal of all of these efforts is to increase access to research study outcomes, to
prevent unnecessary duplication of research effort, and to give IRBs considering approval
of new studies a view of similar work and data relevant to the research proposed to them.

43
It also contributes to the overall profile of what is known about the benefits and risks of
compounds in development and those that are approved but still being researched.
Sponsors also must communicate externally to the regulatory authorities during
development to obtain approval to perform human studies and, via their investigational
sites, to Institutional Review Boards/Ethics Committees (IRB/EC). Regulatory
authorities and IRB/ECs will typically receive the same level of information as
investigators, unless they have more comprehensive investigational requirements, such as
would be produced in the US if the trial required FDA oversight ("Code of Federal
Regulations," 1987). In this case, the committees would be able to examine the text of
the investigational drug submission that typically contains more information than the
documentation for simple IRB approval. However, there is no formal benefit-risk
assessment as part of the IND; the only requirement is that risk information be included
in the Informed Consent Form (ICF), the IB, the clinical protocol(s) and its general
investigational plan ("Code of Federal Regulations," 1987).
2.4.1.2 Post-approval
Once a drug is approved, the systematic information gathered during clinical trials can be
gradually supplemented by field experience or additional clinical trials carried out after
the drug is on the market. In addition to the prescribing information that is included in the
package insert and risk management information discussed earlier, external
communication of this information can be provided to different stakeholders in different
ways that are tailored to the nature of the audience. Traditionally, physicians and to a
lesser extent pharmacists and allied healthcare professionals have been the main

44
stakeholders to whom most information has been directed and this occurs through a
variety of routes, especially if the drug is provided by prescription. In such a case, the
legal system recognizes the prescriber as a “learned intermediary” charged with
providing patients with information about their treatments, including drug treatments
(Giliberti, 2003). Thus it is imperative for the industry to be sure that the prescriber is
fully informed about benefits and risks so that this information can be passed on to
patients. Companies accomplish this responsibility in a variety of ways – through
participation in professional meetings, sponsorship of educational activities and through
the use of sales representatives. Perhaps the most reliable source of information
considered by the scientific community is that which appears in peer reviewed journals.
Companies actively support the promulgation of published research either directly
through their medical information specialists or by helping to fund activities related to the
publication of research results by clinicians (Ross, Gross, & Krumholz, 2012).
For much of the history of drug commercialization, the primacy of the relationship
between industry and physician was unchallenged; the physician was presumed to be the
main source of information by which patients learned about drugs. However, the way in
which pharmaceutical companies interact with prescribers has changed considerably in
recent years. Interactions with the physician have been reduced substantially by the
recent implementation of the Physician Payments Sunshine Act ("Physician Payments
Sunshine Act of 2009," 2009), which requires manufacturers of drugs, medical devices
and biologics that participate in US federal health care programs (e.g. Medicare/Medicaid
or VA) to report payments and items of value given to physicians and teaching hospitals.

45
It requires that any interactions or gifts costing more than $10 must be reported on a
publicly available website (http://www.cms.gov/openpayments/). Further, such
exchanges would have to be justified as educational; gifts are restricted to items such as
reprints or anatomical models, and any printed materials must be based on what was
either published in the literature or contained in the product labeling. For example, the
provision of a reprint valued by its copyright holder for a price of $20 would have to be
documented and would appear on the website as income to the physician. This does not
mean that companies would not continue to provide product information, but it would
limit the flexibility and means by which such information might be communicated.
These new transparency requirements have had a markedly dampening effect on the
exchanges with physicians (Agrawal, Brennan, & Budetti, 2013). This dampening effect
has also been reinforced by local requirements by academic centers to reduce interactions
between health care providers and industry, particularly when those interactions involve
research, in efforts to control conflicts of interest (Schmidt, 2014).
Another important factor in changing the dynamics of communications between
physicians and patients has been the growing influence of other types of communication
channels, such as those accessed through print media, internet and social media sources
that now give patients direct access to much more information about medical treatments
(Juhn, Phillips, & Buto, 2007). Further, in the United States and New Zealand, consumers
now receive considerable information about some drugs via direct-to-consumer
advertising (DTCA). This trend to communicate more directly with the consumer may be
further enhanced in future by changes in the way that the package insert is developed so

46
that it is no longer directed at the physician. For example, a number of individuals and
groups have criticized the current prescribing information that is provided in the package
insert for its limited ability to provide meaningful information to consumer (Schwartz &
Woloshin, 2009). This is particularly important because it is the package insert
information that is often used in drug advertisements as a way to communicate detailed
information about the product as required by DTC regulations, yet most consumers do
not understand the technical information included in the insert (Aikin, O'Donoghue,
Swasy, & Sullivan, 2011). The FDA has recently revised their draft guidance which
provides recommendations on the disclosure of risk information in prescription drug product
advertisements and promotional labeling in print media which is directed toward consumers.
They are urging drug makers to provide "consumer-friendly" summaries in promotional
materials and other direct-to-consumer print advertisement. They cite the research done
by Aikin and others from the social sciences and have gone so far as to recommend
specific formats for conveying risk information either as a “Drug Facts Box” or
“Question and Answer” dialogue (FDA, 2015).
Communication to consumers is still in flux and is an area that has been of much concern.
For example, Schwartz and Woloshin (Schwartz & Woloshin, 2009; Woloshin &
Schwartz, 2011) have for many years advocated the use of a “drug facts box” that
simplifies and organizes information in a manner similar to the well-accepted drug box
used for over-the counter drug products. They have demonstrated that this approach can
improve knowledge about the benefits and risks of prescription drugs in tested consumer
(Schwartz & Woloshin, 2013). Their proposal calls for including a specific data table that

47
compares the risks of various outcomes for patients treated with and without the drug, in
a manner similar to that to be used by FDA in its benefit-risk analyses, described in more
detail below. Schwartz and Woloshin went further, to suggest that FDA reviewers take
responsibility for creating the labeling in this format. It is hoped that these changes will
result in “better choices between treatments for current symptoms and corrected the
overestimation of benefits in the setting of prevention” (Schwartz et al., 2009). Such a
step would further transfer control of information more firmly into the hands of the
ultimate consumer.
2.5 Methods for Benefits Risk Assessment
2.5.1 Qualitative Approaches
For much of the history of risk-benefit assessment, regulators have used their experience
and judgment to evaluate the chemical and clinical testing on which market approvability
was based. The goals of these assessments were directed primarily at assuring patient
safety, and thus considerations were focused on the risk profile of the product. However,
concerns that such decision-making could be opaque and inconsistent have led to the
recent development of more systematized and documented methods, formalized as
frameworks to guide decision-making. At its simplest, a framework is a skeleton or
heading structure that captures the thinking about certain elements important to the
decision in a consistent way that can then be shared with others. Several such
frameworks of varying complexity have been developed by different regulatory agencies
and organizations. When these frameworks capture the thinking of the decision-maker in
words, they are generally considered as qualitative frameworks. When they capture the

48
thinking in metrics of some type, they are designated here as quantitative frameworks.
This review will focus on four alternative qualitative approaches that seem most favored
by regulators and industry: 1) the Benefits Risk Framework used by the US FDA (often
differentiated by the acronym, BRF), 2) the PROACT-URL method used by the EMA,
3) the Universal (formerly call Unified) Methodology for Benefits Risk Assessment
(UMBRA) developed by the Center for Innovation in Regulatory Science (CIRS) and
being used by a group of regulators in Canada, Australia, Switzerland and Singapore,
collectively known as the COBRA Consortium, and 4) the Benefits Risk Assessment
Team methodology, BRAT, developed by the pharmaceutical trade organization,
PhRMA.
2.5.1.1 FDA Benefit Risk Framework (BRF)
The benefit-risk framework recently adopted by the FDA is perhaps the simplest and
most user-friendly of the alternatives. The process of developing their framework began
in 2009, when FDA examined six case studies of past regulatory decisions where benefit-
risk information was challenging and complex. They then interviewed individuals with
decision-making roles from different disciplines on the review teams in their own
organization and consulted with individuals in the Office of New Drugs and the Office of
Drug Evaluation (ODE) who were responsible for reviewing and approving the
recommendations of the review teams (Frey, 2012). The importance of these activities
grew when, in 2010, FDA was directed by Congress through the Prescription Drug Use
Fee Act (PDUFA) to develop a more formalized description of the way that it conducted
benefit-risk activities. FDA had several frameworks from which to choose, some of

49
which are described below. The qualitative framework that it released in 2013 is one of
the simplest options. CDER expressed the view that a structured qualitative approach
would be the best solution to formalize the way that decisions were reached in the past. It
could be used as a stand-alone framework, but also could accommodate more complex
quantitative analyses in a flexible way, if these were seen to later add value (Mullin,
2014).
The BRF framework is a matrix in which 5 key decision factors are arranged in a column
as shown in Figure 4. (Frey, 2012) These factors include: 1) an analysis of the condition,
2) an evaluation of current treatment options, 3) an assessment of benefit, and 4) an
evaluation of risk and 5) a consideration of methods to be use to manage risk. For each
of these factors, Evidence and Uncertainties are required to be described and Conclusions
and Reasons are required to be enumerated explicitly in separate cells of the matrix for
each of the decision factor (FDA, 2013b).
Figure 4: US FDA Benefits Risk Framework (BRF)

The five decision factors that form the backbone of the framework can be grouped into
two areas of focus, those that address the intended condition (Analysis of Condition and

50
Current Treatment Options; dark green shaded area in Figure 4) and those that address
the risk and benefits associated with the specific product under review (light green
shaded area, Figure 4). The “Condition” section provides a summative description and
assessment of the severity of the condition for which the product is intended and
discusses other therapies available for the indication. It is in this section that a reviewer
will document the medical considerations and the current state of knowledge about the
condition or disease for which the product would be used and that played a role in their
decision making.
The second three factors comprising the “Product” section is where the reviewer will
summarize the relevant evidence concerning the drug product under review. To assess
benefits, most reliance is placed on the strength of results from clinical trials to
demonstrate improvement in the medical condition. To assess risk(s), the severity and
reversibility of adverse events in animals and humans are documented. Also included
may be other signals of potential concern related to the pharmacological actions, mode of
administration or manufacturing methods for the product. The Risk Management portion
of the framework then allows the reviewer to identify how specific safety concerns will
be addressed.
It is notable that the BRF matrix has two columns in which the reviewer can enter
comments. The “Evidence and Uncertainties” block allows the reviewer to interpret the
strength of the evidence and to identify gaps in knowledge. The “Conclusions and
Reasons” block permits the reviewer to draw and explain conclusions. Taken together
the reviewer can then prepare the Benefit-Risk Assessment Summary (BRAS). One

51
additional element that is required in the summary is the applicable statutory and
regulatory standards that were applied in arriving at the conclusion reached.
The BRF was piloted by CDER in 2012 using information contained in the New Drug
Applications (Mudd, Groenendaal, Bush, & Schmith) of six new molecular entities
(NME), selected by reviewers for their diversity of content, differing in therapeutic area,
indication, size of application and review division. The feedback from this pilot program
gave insight into the timeframes and other considerations needed to plan and implement
the framework for subsequent product reviews by FDA commencing in 2013. As part of
this initiative it plans to evaluate how uncertainty is characterized during the assessment
of the clinical trial data available to it at the time of submission, and whether that
characterization is capable of predicting the risk and benefits later observed when the
product is marketed commercially (FDA, 2013b).
The implementation status of the BRF is also closely tied to a number of PDUFA V
commitments: publication of the five year plan, revision of the templates and manuals,
conducting public workshops on BR, and conducting 20 public meetings to obtain patient
input on specific disease areas (Mullin, 2014). To facilitate the implementation of this
newly introduced approach, revised clinical review templates and decision memoranda
were circulated. These revised word processing templates are currently under review but
have not been made available to the general public as of this writing. The word
processing software presently employed by the FDA will be the only software designated
for the assisting reviewers in completing the Benefits-Risk Framework.

52
A focus of the FDA over the past few years has been the need to incorporate not only the
industry’s information, given to the Agency as part of the submission process, but also
the patient view on the benefit-risk attributes (Mullin, 2014). This has proven to be a
challenging task that has not yet been incorporated well into the BRA process. In some
areas of concern, such as disease severity, impact on daily life and views of available
treatment options, the patient perspective could be valuable but difficult to obtain. As
noted in the presentation of Mullin referenced above, FDA staff can consider this input to
be a valuable resource that can be collected during and outside of review but it is unclear
at this point how that information will be curated and used beyond the summaries
provided in “The Voice of the Patient” which are available on the FDA website.
2.5.1.2 EMA PrOACT-URL
The European regulators in the European Medicine Agency (EMA) have taken a different
approach to the development of their benefit-risk framework. Its work was initially
spurred by an audit in 2004 of the Committee for Medicinal Products for Human Use
(CHMP), its peer group responsible for the review and marketing approval of novel
pharmaceutical and biological products in Europe (EMA, 2007). The EMA recognized
the need to improve the methodology for benefit-risk analysis and issued a “Reflection
Paper on Benefit-Risk Assessment Methods in the Context of the Evaluation of
Marketing Authorization Applications Medical Products for Human Use” (EMA, 2008)
that launched the Benefit-risk Methodology Project (EMA, 2009). The new project was
tasked with a similar project to that underway concurrently at FDA- to develop tools and
processes for balancing the multiple benefits and risks associated with a single product.

53
It was divided into five consecutive work packages (WP). Work Package #1 documented
the current practices of benefit-risk assessment for marketing applications reviewed by
CHMP. Work package #2 examined the applicability of three frameworks and
18 quantitative approaches; from this review it selected one, the PrOACT-URL, to
advance further. Work package #3 tested this model for five drugs that were under
review and surveyed the teams of assessors to evaluate their views on the usefulness of
the approach. Work package #4 synthesized the information from the field test and
fine-tuned the benefit-risk tool and accompanying process in order to facilitate the
implementation of PrOACT-URL. The same working group also reported on risk
perception, a task that will be discussed separately below. Work Package #5 was the
initial implementation phase of this project, which included training for assessors. It also
specifically piloted the use of the Effects Table (ET) that was described in Work
Package #4.
The currently available Work Package #2 report describes 8 steps in the PrOACT-URL
framework. Its first five steps appear to be shared with most risk-benefit decision making
processes. However, three additional steps have been added to extend the framework in a
way that would better capture uncertainties and trade-offs, as shown in Table 1.

54
Table 1: PrOACT-URL Framework
Step 1: Problem The first step is to evaluate the nature of the problem to be solved and it
context using a series of imperatives:
Determine the nature of the problem and its context: What is the medicinal
product? What sort of decision or recommendation is required? Who are the
stakeholders and key players? What factors should be considered in solving
the problem?
Then frame the problem (e.g., as mainly a problem of uncertainty, or of
multiple conflicting objectives, or as some combination of the two).

Step 2: Objectives Identify objectives that indicate the overall purposes to be achieved (e.g.,
maximize favorable effects, minimize unfavorable effects), and develop
criteria against which the alternatives can be evaluated (i.e., what are the
favorable and unfavorable effects?).

Step 3: Alternatives Identify the options (actions about a medicinal product or the products
themselves) to be evaluated against the criteria (e.g., pre-approval: new
treatment, placebo, active comparator; post-approval: no treatment/action,
limits duration, restrict indication, suspend).

Step 4: Consequences Based on available data, describe how the alternative would perform on the
criteria (e.g., describe the magnitude of possible favorable and unfavorable
effects). It may be helpful to consider intermediate outcomes, such as safety
and efficacy effects. Consequences describe clinically relevant effects.

Step 5: Trade-Offs Assess the balance between favorable and unfavorable effects.

Step 6: Uncertainty Consider how the balance between favorable and unfavorable effects would
change by taking account of the uncertainty associated with the
consequences.

Step 7: Risks Judge the relative importance of the Agency’s risk attitude for this medicinal
product (by considering, e.g., the therapeutic area, the unmet medical need
and patients’ concerns) and adjust the uncertainty-adjusted balance between
favorable and unfavorable effects accordingly.

Step 8: Linked
Decisions
Consider the consistency of this decision with similar past decisions, and
assess whether taking this decision could impact future decisions either
favorably or unfavorably (e.g., would it set a precedent or make similar
decisions in the future easier or more difficult).

The PRoACT-URL framework can utilize an Excel platform and is somewhat more
complicated than the singular BRF matrix of FDA. Notably, at Step 6 of the

55
PrOACT-URL, the reviewer is directed to create a secondary Effects Table (Figure 5)
that displays all the favorable and unfavorable effects that the assessors have determined
to influence the BR balance. This secondary table can also be used to capture other key
information such as definitions, units of measurement, plausible ranges of effect and
other measured data associated with the medicinal product.
Figure 5: Sample Effects Table

(EMA Work Package 5, Feb 2014) (EMA, 2014)
In addition to the use of PrOACT-URL and an Effects Table (ET), the working group
suggested that the quantitative benefits-risk tool called the Multi-Criteria Decision

56
Analysis (MCDA) methodology (described below in 2.5.2.1), either in its simple or full
form, might also be a useful adjunct to assist decisions that have multiple variables.
2.5.1.3 UMBRA
In the two preceding examples, the framework proposed for use was developed by a
single regulatory agency. A third, more collaborative approach has been taken by a
grouping of four similarly sized and resourced health authorities from Canada, Australia,
Switzerland and Singapore, who formed a Consortium on Benefit Risk Assessment
(COBRA) as part of their efforts to implement work sharing and joint reviews of product
submissions. The Consortium sought the assistance of the non-profit group, the Center
for Innovation in Regulatory Science (CIRS), and together developed a standardized
method called Unified Methodologies for Benefit-risk Assessment (UMBRA) (later
renamed the Universal Methodology for Benefit-Risk Assessment), to be used
communally when they assessed the benefits and risks of new drugs. This eight step
framework is illustrated in Figure 6 and is supported by a benefit-risk assessment
summary template, which is a WORD based template (BR Template), as well as a
detailed User Manual that are provided free of charge.

57
Figure 6: UMBRA Framework
(Used with permission from the Center for Innovation in Regulatory
Science)

The association with CIRS is interesting and different from approaches elsewhere. CIRS
was seen as an expert group because they had been involved with benefit-risk assessment
methodology for many years and had been conducting workshops on the topic since 2002
(http://cirsci.org/). Numerous publications including that of James Leong had resulted
from research of the organization. Of particular relevance to the initiatives undertaken by
COBRA in this collaboration was the work that explored the status and need for a
universal benefit-risk framework for medicines in regulatory agencies and

58
pharmaceutical companies (Leong et al., 2013). In that study, thirteen questions were
sent to participants in large pharmaceutical companies and regulatory authorities involved
in benefit-risk assessment and decision-making. Results of the survey led Leong to
conclude that an overarching framework for benefit-risk assessment was needed. Further
that framework should incorporate a “toolbox” of different methodologies with the goal
of creating a universal and scientifically valid framework. In a recent (2014) review of
the criteria influencing the quality of a benefit-risk framework (as defined by the EMA
Benefits-Risk Methodology Project), Walker and his colleagues at CIRS demonstrated
that all the criteria presented by EMA, including logical soundness, comprehensiveness,
acceptability of results, practicality, specificity, sensitivity, presentation (visualization)
and scope were captured by the UMBRA framework (Walker et al., 2015).
To study the validity of the UMBRA framework and the BR Template in particular two
studies were conducted by Leong and his colleagues. In one study they compared the
assessment report format for four major regulatory agencies (FDA, EMA, Health Canada
and Australia’s TGA) to the BR Template and found both deficiencies and differences in
these formats that could possibly be overcome with the use of a “universal” template
(Leong Wai Yeen, Salek, & Walker, 2014). An additional retrospective study of
regulatory reviewers from the Health Sciences Authority in Singapore found the BR
Template to be a useful tool in communicating benefit-risk decisions to a variety of
stakeholders (Walker et al., 2015). This study also highlighted the importance of
providing detailed instructions to the reviewers to guide them in completing the template
such as those found in a User Manual or Implementation Guide.

59
2.5.1.4 PhRMA BRAT
As noted previously, not all benefit-risk frameworks have come from regulatory
agencies. The development of the Benefit Risk Assessment Team (BRAT) framework
began in 2005, when the Pharmaceutical Research and Manufacturers of America
(PhRMA) formed a working group to develop a structured framework that they hoped
might provide a more rational, transparent and defensible decision-making process than
was available at that time. The six–step model, developed in Excel, was structured so
that it could integrate both qualitative and quantitative analyses of outcomes but was not
in itself quantitative; it did not explicitly calculate an overall benefits risk score for the
assessment (Coplan, Noel, Levitan, Ferguson, & Mussen, 2011a). The six steps are
described in more detail below in Figure 7).
Figure 7: BRAT Framework
(Used with permission of PhRMA)

BRAT Step 1: Defining the Decision Context
In this first step, the practitioner specifies the pharmaceutical product to be assessed, and
the population and indication for which it is intended. If more than one indication for a
single product is to be studied, a separate framework is needed for each indication.

60
Other factors such as formulation, dosage and contraindications must also be considered.
If the BR decision is being compared to another treatment, information may be needed
about the alternative treatments to which the product might be compared. In this section,
stakeholder perspectives might also be captured but at this time it is unclear how this
would be done.
BRAT Step 2: Identifying Outcomes
In the second step, a value tree is constructed to identify the outcomes of the assessments.
Value trees are generally used to define the benefit and depend primarily on the endpoints
of clinical studies. This is also where outcomes relevant to patients could be called out
and added to the hierarchy (Levitan et al., 2011). The tree then gets “pruned” by
including or excluding outcomes of value. Criteria for these selections or omissions are
documented as part of the narrative Figure 8).

61
Figure 8: Value Tree(s)

(B. Levitan, 2011)
BRAT Step 3: Identifying Data Sources
In step 3, all relevant data obtained from multiple sources are catalogued into a “data
source table” on which decision-makers can draw for the benefits-risk assessment.

62
BRAT Step 4: Customizing the Framework
In this fourth step, the value tree is reorganized or updated based on the quantity and the
quality of the data obtained in step 3. Benefits are presented by endpoint measurement
and efficacy calculations and risks are presented by body system, much as they are in
adverse event tables. Safety outcomes are then further documented as being adverse
events that were calculated from clinical trials (numbers or percentages) or categorized as
potential risks (per EMA Pharmacovigilance Guidelines), such as those related to
mechanism of action. The patient’s perspective can also be used to influence the
framework at this step.
BRAT Step 5: Assessing Outcome Importance
At step 5, quantitative methods for scoring and weighing outcomes may be introduced
into the framework (See Appendix A Numerical Representations). No single model is
promoted for use to support the BRAT framework, and could include any of the methods
that are detailed below (see 2.5.2).
BRAT Step 6: Display and Interpret Key Benefit-risk Metrics
In the final step, a synthesis is developed by constructing a “Key Benefit-risk Summary”
(KBRS) table, that quantifies the outcomes of the value tree. The KBRS table was
considered to be useful because it presents in summative form all major issues underlying
the Benefit-risk assessment. An example of this outcome is shown in Figure 9.

63
Figure 9: Key Benefit Risk Summary Table

After the BRAT tool had been developed, the views of pharmaceutical professionals with
regard to its value and usefulness were explored in order to inform PhRMA’s regulatory
advocacy (Noel, Hermann, Levitan, Watson, & Van Goor, 2012). An online survey
consisting of 106 questions in 6 categories was provided by invitation to 34 experienced
pharmaceutical professionals, many in senior management positions. Of the
27 respondents who completed the survey (79%), nearly two-thirds were from companies
that had used the BRAT Framework to evaluate a pharmaceutical product and most had
experience in BR Assessment and the BRAT framework (67% and 63% respectively).
Responses gave useful insights into the strengths and limitations of the framework. Over
80% found that use of the framework either “moderately” or “very much” affected their
understanding of benefit-risk assessments. However, over half indicated that it was
moderately resource intensive to use. The software visualization (e.g., value tree) and
KBRS table were rated as the two most appreciated components of the framework; over

64
80% of respondents scored them as “extremely” or “very” useful. The two concerns
most emphasized by respondents in open-ended text boxes were 1) that benefit-risk
assessments could be oversimplified or misused, and 2) that regulators were not using a
similar, standardized framework of this type. The need to have a harmonized
methodology among global regulators was rated as “very” or “extremely” important by
93% of participants. Respondents also saw the potential to incorporate more patient
perspectives when they attempted to weight disease severity or unmet medical needs.
They identified that a number of barriers hindered the adoption of this approach- the
initial learning curve for personnel; lack of subject matter expertise amongst clinical
teams; time/resource needs to carry out the assessment; and resistance to change. The
authors acknowledged the exploratory nature of the study, whose sample size was small
and included participation by members of the BRAT development team.
2.5.1.5 IMI PROTECT
In addition to the governmental and industry organizations described above, other groups
have also been working to advance the use of benefit-risk assessment methods (e.g.,
CIRS, IMI). Their studies of alternative methodologies have contributed significantly to
the body of knowledge available for review (Mt-Isa et al., 2014). The most noteworthy
of these efforts may be the Innovative Medicines Initiative (IMI), that has been
conducting a collaborative project, called Pharmacoepidemiological Research on
Outcomes of Therapeutics or PROTECT. This initiative has been coordinated by EMA
as a public-private partnership that includes the European Federation of Pharmaceutical
Industries (EfPIA). The IMI PROTECT initiative has surveyed a number of

65
methodologies (Figure 10) and is a rich source of information on their structure and
content (http://www.imi-protect.eu/benefit-risk.html). Their activities extend from
qualitative to quantitative methodologies, with a work stream (EMA Work Package #5)
that aims to develop quantitative methods for use in Benefit-risk assessment, with a
particular focus on the graphical presentation of this information (Mt-Isa, 2013).
Figure 10: IMI PROTECT

(Mt-Isa et al., 2011)
2.5.2 Quantitative Methodologies
The frameworks presented in 2.5.1 all attempt to summarize key elements in
decision-making processes in text alone. However, words alone cannot capture easily the
relative value and weight assigned to each of the constituent elements. Thus some
experts in benefit-risk assessment have criticized the non-quantitative nature of those

66
assessments, and instead have suggested that effective benefit-risks assessments would
profit from some level of quantitation.
Methods to quantify attributes using some form of arithmetic or mathematical approach
have a long history. One of the first of these methods specific to making decisions on the
basis of drug studies was the Global Risk-Benefit Method or GRBM (Chuang-Stein,
Mohberg, & Sinkula, 1991). It described risk categories in quantitative terms, such as
linear score (e.g., positive or negative effect), and risk ratios, also referred to as relative
risk, or the measure of the risk of a certain event happening in one group compared to the
risk of the same event happening in another group. It then collated the safety information
to reflect the risk experienced by each individual in a clinical trial setting and compared it
with the benefit measured for that same individual (Chuang-Stein, 1994).
At about the same time, the Council for International Organizations of Medical Sciences
(CIOMS) established a group of experts (Working Group IV) dedicated to the topic of
benefit risk assessment using a method to quantify risk according to the “Principles of
Threes”(CIOMS, 1998). The “Principle of Threes” used the three elements of
seriousness, duration and incidence as a framework to evaluate both treatment effects and
adverse events (Mt-Isa et al.) across any indication. However, these approaches were not
received well by regulators and industry, in part because they found it difficult to assign a
numerical value to some of the elements on which decisions were based. Without such a
capability, it would be difficult to carry out the rigorous statistical calculations required
as part of GBRM, for example.

67
A number of comprehensive reviews have since surveyed a variety of quantitative
methods for assessing Benefits/Risks. The reader is referred for more detailed
information to the explanations of several methods detailed in Mussen’s Benefit-risk
Appraisal of Medicines in 2009 (F Mussen et al., 2009) and the review of quantitative
methods performed by Guo and collaborators at the International Society for
Pharmacoeconomic and Outcomes Research or ISPOR (Guo et al., 2010). As noted
above, the IMI PROTECT initiative published a lengthy review of quantitative as well as
qualitative methods in February 2010. Also of great value is the recently published
framework for the organization and selection of methods from sixteen different choices
by Puhan and colleagues (Puhan, Singh, Weiss, Varadhan, & Boyd, 2012). The purpose
of this thesis is not to repeat the excellent work done by these groups, since most of the
techniques discussed in those papers are not currently being considered for use in drug
benefit-risk assessments. For the purposes of this discussion, I will focus on two
particular methods – the Multi-criteria Decision Analysis (MCDA) and the Stochastic
Multi-criteria Acceptability Analysis (SMAA) approach – that are often discussed by the
medical products industry as examples of potentially useful quantitative benefit-risk
assessment frameworks. In addition, the work of Ashby and Smith must also be
mentioned as it describes the use of Bayesian or probability based decision making and
its use in evidence based medicine and medical product development.(Ashby & Smith,
2000)

68
2.5.2.1 Multi-Criteria Decision Analysis (MCDA)
Multi-Criteria Decision Analysis has been used in various industries since it was first
described by its “father”, Howard Raiffa, in the 1960s (Raiffa, 1961), as follows:
“The spirit of decision analysis is divide and conquer: decompose a complex
problem into simpler problems, get one’s thinking straight on these simpler
problems, paste these analyses together with logical glue, and come out with
a program of action for the complex problem”
The use of MCDA in the context of benefit-risk assessment was first proposed by Mussen
and coworkers in 2007 (F. Mussen, Salek, & Walker, 2007), and has perhaps been
championed most persuasively by Larry Phillips, working with collaborators in Cardiff
University and the Center for Innovation in Regulatory Science (CIRS). In addition,
through his own work at the London School of Economics and with interactions at IMI
(Mt-Isa et al.) and the EMA (Benefits-Risk Methodology Project), Phillips was able to
apply MCDA to sixteen new drugs across various indications to examine its proof-of-
principle (Phillips, 2013).
The MCDA model has 7 highly structured steps, listed in the table below (Table 2). Its
initial stages look much like those of qualitative frameworks described previously, and
particularly of the PrOACT-URL framework described previously. However, it’s more
quantitative nature becomes evident in step 4, when scales are constructed for each
criterion on which the decision is based and each criterion is scored numerically. In
step 5, the quantitation is extended by adding the option to apply a weighting function to
each criterion. For example, for a drug that treats obesity one would not only assign a
value to the favorable effect of weight loss versus the unfavorable side effects such as

69
anxiety, sleep disorders and nervousness, but would have the option to weigh those
features differently, so for example, a serious side effect would be weighted more heavily
than an insignificant one. Finally, the assigned values are normalized and used to
calculate an overall weighted score to encapsulate both the risks and benefits.
Table 2: Multi-Decision Criteria Analysis Model
(F Mussen et al., 2009)
Step 1: Establishment of the decision
context
Identification of the indication, dosage for that
indication, contraindications and other critical safety
wording from the proposed label of the subject medicine
Step 2: Identification of the options to be
appraised
Include the medicine being assessed and any
comparators from the pivotal clinical trials
Step 3: Identification of the benefit and
risk criteria and organization in a
value tree
Identification of the pivotal clinical trials and their
primary endpoints and construction of a value tree (see
below).
Step 4: Assessment of the performance of
each option against the criteria
Identification of the relevant data including non-pivotal
trials, extension studies and the safety dataset.
Construction of the scale for each criterion and scoring
of each option on each criterion. Include consistency
check of the scores on each criterion.
Step 5: Assignment of a weight to each
criterion using swing-weighting
Assignment of relative weights to benefits and risks,
using both top-down and bottom-up approaches.
Step 6: Calculation of the weighted scores
at each level and calculation of the
overall weighted scores
Normalization of the weights and criterion at all levels
so that the total sum =100 but insuring that the ratios
between the weights are preserved. Calculation of the
overall preference score for each option is the sum of
the weighted scores.
Step 7: Conduction of a sensitivity
analysis
Variation of the weight and/or score on any criterion
and assessment of its impact on the overall benefit-risk
preference score.

In a system such at the MCDA, the validity of the results will depend on the way that
different criteria used for the decision are scored and weighted. Both favorable and
unfavorable effects must be expressed in comparable units if the method is to capture the

70
trade-offs among the effects. MCDA can be combined with other methods (supporting,
for example, the T in PrOACT), and can be adjusted in their level of complexity.
Because the raw observations are often hidden inside the analysis, it is important with
such methods to evaluate the sensitivity of the outcomes. Insights into sensitivity can be
gained using simulations that vary weights or probabilities, the latter using Probabilistic
Simulation Methods (PSM), that rely most commonly on “Monte Carlo” methods. The
outputs of quantitative methods can be difficult to communicate, so the visualization of
the data can be enhanced with graphical methods such as Forest plots and bar graphs as
illustrated below Figure 11 and Figure 12) and recommended by IMI PROTECT (Mt-Isa,
2013).
Figure 11: Sample Visualization - Forest Plot

71
Figure 12: Sample Visualization - Bar Graph

Phillips contends that MCDA is emerging as the “Gold Standard” for quantitative
benefits-risk assessment because it is comprehensive and flexible. It can be combined
with other statistical methods and utilizes elements already common in qualitative
methods, such as value trees. Nevertheless, MCDA is not intuitive to inexperienced
users, and an experienced facilitator is recommended to those using the model for the
first time (Phillips, 2014).
2.5.2.2 Stochastic Multi-criteria Acceptability Analysis (SMAA)
A newer, more complex approach to B/R assessment, Stochastic Multi-criteria
Acceptability Analysis (SMAA), has been described in a series of publications by
Tervonen and colleagues (Tervonen & Figueira, 2008; Tervonen & Lahdelma, 2007; T.
Tervonen, G. van Valkenhoef, E. Buskens, H. L. Hillege, & D. Postmus, 2011). SMAA

72
is a “family of methods for aiding in multi-criteria group decision making in problems
with inaccurate, uncertain, or missing information” (Tervonen & Lahdelma, 2007).
SMAA attempts to improve benefit-risk assessment by allowing its users not only to
score and weight values for different elements of importance to the decision, but also to
quantify uncertainty. Tervonen defines uncertainty as “a form of ignorance appearing
when the observer is taken into account” (Tervonen & Figueira, 2008). Uncertainties
about the values assigned to different variables are considered as important limitations to
the validity of the decision-making outcome and so are considered by the authors to be an
essential part of a comprehensive benefit-risk assessment (Tommi Tervonen, Gert van
Valkenhoef, Erik Buskens, Hans L. Hillege, & Douwe Postmus, 2011). Thus, SMAA
attempts to overcome the limitations of MCDA, by addressing more explicitly how
uncertainties associated with different elements are gauged and incorporated. It further
considers the risk of losing information when measurements are mapped from one scale
to another (e.g., ordinal to continuous scale). An example of this might be number of
adverse events severity where mild/moderate/severe can be represented by numbers
(ordinal) to changes from baseline, where ratios are used to represent value over time
(continuous scale). In this highly technical approach users would need extensive
mathematical and computational knowledge together with sophisticated software, to
manage the data estimations and modifications necessary for a full benefit-risk
assessment. Like the MCDA framework, SMAA relies on graphical representations of
the data, such as bar graphs and other similar methods, to display the data more clearly to
others.

73
Compared to the MCDA methods described by Mussen et al (F Mussen et al., 2009) and
Felli et al (Felli, Noel, & Cavazzoni, 2009), SMAA has the additional advantage of
incorporating uncertainty and the sampling variation that is part of most clinical trial data.
In addition, SMAA allows construction of the benefit-risk profile without knowing the
exact numerical preferences beforehand (Tommi Tervonen et al., 2011). The SMAA is
one of the approaches that appears in the IMI review of methods, and was considered by
them to be worthy of further study and a “contender” to the MCDA approach as a
quantitative method of choice for the reasons stated above.
It is important before closing on this topic to recognize that these quantitative methods
have been criticized. Detractors of these highly technical methodologies argue that they
do not eliminate the subjective element of benefit-risk assessment; they simply displace it
to the steps of scoring and weighting from different perspectives. Often this concern is
cited as one reason why these methodologies are not being fully implemented (Mt-Isa et
al., 2011). The assignment of numbers is itself subjective especially for elements such as
risk assessment at the time of approval when risks are not always well-characterized.
The veneer of quantitation may hide inappropriate assumptions from those who are
unwilling to look “behind the curtain”. On the other hand, the fact that variables are
assigned numbers allows a variety of stakeholders to consider whether such numbers are
appropriate, and may enhance useful discussion and communication of different views on
scoring and weighting.

74
2.6 Benefit-Risk Assessment: A Changing Landscape
The shift in thinking that has occurred over the last 10 years with regard to benefit-risk
assessment appears to have changed the regulatory landscape in at least two major ways.
First, it has long-term implications for the way that communications take place between
the regulators, industry and other stakeholders. Instead of remaining as silent partners in
the communication of benefit and risk, the regulators have developed approaches that
make its views quite transparent. This type of initiative may allow the regulators to
communicate their views on benefits and risks more directly and thus diminish the
traditional primacy of industry as a singular communication channel to patients and
healthcare professionals. Second, it signals a new era in the way that regulators will
organize and process data, in order to structure the assessments that are now required of
reviewers. However, regulators in different constituencies have adopted different
benefit-risk assessment frameworks, and this has the potential to greatly complicate the
way that benefit-risk assessment methodologies might be implemented by industry. If a
company must gain marketing approval for its drug in many countries, no one
benefit-risk assessment approach will be appropriate for all. A wise company may try to
organize its data and arguments for each marketing submission in a way that assists the
reviewer in that constituency to use the benefit-risk assessment framework of its parent
Agency. In such a situation, the company would have to be familiar with a number of
frameworks and to some extent customize its multiple marketing submissions
appropriately. Such customization is not in keeping with the current efforts of industry
and regulators at a higher policy level, to have a one-size-fits-all, harmonized approach to

75
marketing submissions using the ICH Common Technical Document (CTD) format.
However, it seems to be a place where they are starting, and industry must respond.
In this research project, we are interested in the way that industry is responding to these
challenges, both in the way it communicates with the regulators and other stakeholders,
such as physicians and patients, and also in the way it modifies its work internally to
deliver materials that can feed the decision-making processes of its own executive teams.
It is not clear how industry is responding to the changes that the benefit-risk approaches
bring. As Leong and his colleagues noted in his 2011 survey, the 20, mostly large,
pharma companies appeared to agree that a universal benefit-risk assessment framework
would be best for registration purposes, as well as health technology assessment and
lifecycle management. However, they saw major barriers to the adoption of either
qualitative or quantitative methods, which included resource limitations, lack of
knowledge/experience and a scientifically valid acceptable framework.
In this current research, we wish to explore how industry is adapting to this new set of
initiatives, that they are seeing five years beyond the first survey by Leong. First, how
are companies implementing benefit-risk assessment frameworks? Do they have singular
or multiple frameworks, and at what stage are they in their implementation? Second, are
the benefit-risk assessment approaches that have been adopted by regulators driving
changes in the way that companies are developing their own communications? Do they
feel that these initiatives will affect the way that industry works with regulators to
develop better benefit-risk communication strategies and do they have concerns about the
way that benefit-risk assessment assessments will be communicated to other

76
stakeholders? Do they see a shift in the way that they communicate benefit-risk to
stakeholders through their education and publications programs and their development of
marketing and other materials as a result of these initiatives?
2.7 Approach to the Research
If we are to explore the implementation of benefit-risk assessment frameworks in
industry, it is helpful to have a systematic approach that relies on a sturdy research
platform. Thus to develop a survey of industry practices, we surveyed the literature to
find a research framework that would help to guide this work. Though many options
exist, an approach that seemed well suited to the work to be undertaken here is that
published by the National Implementation Research Network (NIRN). Its publication of
“Improving programs and outcomes: Implementation Frameworks 2013” (R. Bertram,
2013) along with the Active Implementation Hub (aiHUB:
http://implementation.fpg.unc.edu) has provided a practical guide that will be used to
assess the implementation efforts of industry with respect to the adoption and
communication of benefit-risk assessment.
Key to the research framework of the NIRN is the concept that implementation is not an
event but a process of carefully considered organizational adjustments that unfold over
the course of years. NIRN has described the stages of implementation as follows:
1. Exploration
During this phase, organizations should assess their needs, examine the key
elements of the models under consideration and the corresponding activities

77
and resources that will be needed to proceed. In addition, companies will
need to evaluate how they deal with change and what potential drivers as
well as barriers there are to implementation.
2. Installation
After the decision has been made to change or refine current practices,
companies will need to prepare the organization for implementation. This
will involve assessing staff competencies, organizational readiness and
leadership, all of which are considered “implementation drivers”. These may
then require acquisition of additional resources and training.
3. Initial Implementation
The main activity during initial implementation will be deploying new
processes and systems and managing the change. This is also where
improvement cycles can be initiated and adjustments made to the
implementation drivers noted above.
4. Full Implementation
Full implementation is achieved when the groups are providing the new
model in a sustainable manner with high fidelity/reliability.
This study will illustrate where on this continuum various companies are positioned. It
will gather data concerning the activities and challenges of implementing benefit-risk
assessment methodologies as analytic and communication tools at these sequential stages.
Implementation as well as communication will be the focus of this research and survey
questions designed to address these two key components of benefit-risk assessment. Of
particular interest for benchmarking purposes will be the stage at which most companies

78
currently sit, and whether that degree of implementation maturity depends on the
company size, structure or product specialization.

79
METHODOLOGY
3.1 Introduction
The purpose of this study is to evaluate the biopharmaceutical industries with respect to
the ways that frameworks and methodologies are being used to assess benefit-risk of new
medicines. Gap analysis and barriers to implementation will be identified as a way to
inform the stakeholder group(s).
3.2 Development of Initial Survey
This exploratory study used a self-administered survey and unsolicited personal
communications in order to evaluate current practices with respect to the implementation
of benefit-risk frameworks by industry. The survey tool took as its starting point a
previously described survey by the James Leong and colleagues (Leong et al., 2013),
whose approach will be extended and modified to focus on the particular goals of this
study. The initial questionnaire consisted of 25 questions and an open text field at the
end for additional comments.
The questions explored three main areas of inquiry: profiles of the respondents, the
current status of how benefit-risk assessment is performed at the companies represented
and then questions on the methodologies and the knowledge level and general
perceptions of how they are used by industry.

80
3.3 Focus Group and Survey Finalization
The focus group included eight participants from industry and academia. Participants
were selected on the basis of their knowledge of benefit-risk assessment of medical
products and/or their experience with survey development.
The focus group members are described in the table below, Dr. Frances Richmond served
as Co-chair of the focus group (Table 3). Focus group members were asked to consider
the purpose of the thesis research when commenting on the survey instrument and to
provide input on the overall organization of the survey and the relevance and clarity of
each question. In addition, they were asked to validate the survey from a user perspective
and to identify any gaps in topics. Because three of the participants attended from remote
sites, video conferencing with participant consent was provided to facilitate the
interactive experience of remote participants and the proceedings were video captured.
Written feedback from the focus group was also collected at the end of the Focus Group
session which lasted approximately 1 ½ hours.

81
Table 3: Focus Group Participants
NAME TITLE
Frances J. Richmond, BNSc, MSc, PhD
(Focus Group Co-chair)
Director USC Regulatory Science
Program, and DIA Board of Directors
Member
Michael Jamieson, DRSc Assistant Professor, USC Regulatory
Science Program
Eunjoo Pacifici, PharmD, PhD Assistant Professor of Clinical Pharmacy,
USC Regulatory Science Program
Dana Bell, PhD Faculty, USC Regulatory Science
Program
Elliott Berger, PhD Senior VP, Regulatory Affairs,
Biocryst Pharmaceuticals, Inc.
Michelle McGuinness Vice President, Global Regulatory Affairs
and Quality Assurance,
Veloxis Pharmaceuticals, Inc.
1

Neal Storm, DRSc, MS, MBA, RAC Director, Global Regulatory Affairs,
Amgen Inc.
Alicia Baker Director, Global Regulatory Strategy,
Covance
1
Title at time of focus group

The group was provided with an electronic copy of the survey prior to the meeting and
with a paper copy at the time of the focus group. The proceedings began with a short
presentation regarding the purpose of the research and the survey, and a short general
discussion regarding the appropriateness of the targeted respondent pool and methods to
distribute the survey distribution. In sequence each of the proposed questions were
reviewed and a discussion of the value and improvements to each of the topics took
place. The survey questions were modified according to the advice received from the

82
focus group participants. These modifications specifically addressed 1) how company
size could be determined – the number of employees’ ranges was expanded and 2) where
BR assessments were being used in dossiers and where/when feedback on BRA
influenced decision making by regulators. In addition, a number of open text fields were
added to solicit additional details on the questions with binary answers.
The final survey was then reconfigured on the web-based survey platform, Qualtrics
(http://www.qualtrics.com/). The final questionnaire consisted of 31 questions and an
open comments field at the end for participants to provide additional information they felt
would contribute to the research (See Appendix B). The effective operation of the
system was then validated by sending the survey to four individuals from the focus group
to ensure that the emails arrive properly and the answers could be entered and analyzed
appropriately.
3.4 Survey Deployment and Analysis
The initial survey was deployed from 20 May 2015 to 30 July 2015. The target
population was intended to include > 50 participants, all of whom were
biopharmaceutical industry professionals, and upon completion included 87 participants.
Regulatory personnel holding positions at the manager level and above, as well as
Regulatory consultants, were contacted to solicit their participation in the survey.
Recipients also had the option of providing the contact information of other personnel in
their organization or network who may also meet these selection criteria and who may
have been better qualified to address the topic. These individuals were then provided the
survey. Potential respondents were identified from my personal contact database and

83
LinkedIn network. They were contacted by email to identify their willingness to receive
the survey, and to clarify that they met the criteria for inclusion into the surveyed
population. Unless the recipient of the email indicated otherwise they were provided
with the survey. Only one person asked to not be included in the research. The survey
was then distributed to the entire distribution list. No remuneration was provided to
encourage participation, but the respondents who requested it were promised a summary
of the results obtained after the survey has been analyzed.
A second survey was distributed by The Organization of Professionals in Regulatory
Affairs (TOPRA) in order to extend the reach of research to more companies in Europe.
These recipients were not prequalified in any way with regards to their participation.
Sixteen (16) responses were obtained. The data from these surveys will be presented
separately but alongside the basic results in order to provide confirmatory support for the
results.
Results of both of the surveys were collected anonymously and saved electronically.
Data from Yes/No, Choose One and Please Rank type questions was graphed and
analyzed statistically. Descriptive text, tables and figures are used to display the data.
Results in the figures will be based on percentages in most cases where the questions had
singular choice responses and the percentages are true values, in these instances the total
responses are noted in the caption. For questions with multiple responses the total
response for that option are noted but these values will be variable. Open text and
comments fields were examined for their information content and analyzed to see if any
trends or common elements appear.

84
RESULTS
4.1 Analysis of Survey Results
The survey was first disseminated to a specifically developed expert panel (EP) of
approximately 200 potential participants by sending personal email messages with links
to the survey on 20 May 2015. The survey period was kept open until 30 July 2015. One
hundred and twenty-one (121) of these emails were opened (60%) and 90 participants
initiated the survey (74%), 87 completed at least 40% of it. Three individuals opened the
survey but did not answer any of the questions. Through unsolicited email
correspondence three additional participants who did not complete the survey noted that
while taking the survey they felt they were not “qualified” to respond. These individuals
were not included in the 90 person respondent pool identified above, but further insight
into their reasons could not be gained by contacting the respondents because the survey
was anonymous. Notably, most of the respondents who completed a partial survey
(10/87) stopped answering after the question that rated their level of satisfaction with
their existing framework or method for use either for submissions or development, as
described below.
Opportunity was also taken to distribute the survey as a link embedded in a standardized
email invitation to members of The Organization of Professionals in Regulatory Affairs
(TOPRA) on 17 June 2015. The survey was kept open for the same number of days that
were offered by the first mailing to the EP. Fifty-three respondents in this TOPRA group
initiated the survey and 16 completed it (30%). Of the remaining 37 who did not

85
complete the survey were 21 respondents with incomplete surveys (57%) who stopped at
the same Yes/No questions regarding satisfaction with their existing framework for
submissions or development as did the participants in the first EP group. The data from
TOPRA participants was used to provide some measure of insight into the validity of
results from the EP, but was not collected for the purpose of detailed analysis. Thus, data
from the TOPRA panel are reviewed in an abbreviated fashion in this chapter but
reported in more detail in Appendix C Results Report – TOPRA Panel.
4.2 Profiles of Respondents
Most individuals who completed the EP survey were in Regulatory departments at the
level of either Vice President (30%, 26/87) or Director/Senior Director (38%, 33/87).
Seven (7) were Manager/Senior Managers and 10 were consultants. The remaining
11 respondents described “other” roles including a Vice-President of Pharmacovigilance
and other titles that were specific to Benefit Risk departments as shown in Figure 13.
A similar profile of professional roles was apparent in the TOPRA sample. Of the
15 respondents who provided their job titles; 33% were either Vice-Presidents or
Directors/Senior Directors (10/15), 13% were Specialists (2/15), and 7% were Managers
(1/15), 7% Consultants (1/15) and 7% “Other” (1/15) with the other title not provided.

86
Figure 13: Roles Played by Respondents to the Expert Panel
(n=87)

The majority of respondents indicated that they have global responsibilities (62%, 54/87).
Of the others, over one quarter self-identified to have only US responsibilities (29%,
25/87), a single respondent (1/87) had responsibilities for EU and another individual
indicated responsibilities for “Rest of World” (RoW) (1/87). The remaining six (7%)
participants had responsibilities for the US and EU only (2/87), North America (2/87)
EMA (2/87), and for Latin America and Asia Pacific regions (1/87). In the TOPRA
group a similarly large number of respondents had global responsibilities (10/14, 72%);
two respondents had responsibilities for EU only, one had responsibilities for US only
and one had responsibilities for RoW (Figure 14).
0% 5% 10% 15% 20% 25% 30% 35% 40%
Manager/Sr. Manager - Regulatory
Consultant/Sr. Consultant
Other - please specify
Vice President- Regulatory
Director/Sr. Director - Regulatory

87
Figure 14: Geographic Areas of Responsibility of Respondents to Two Surveys
A Expert Panel (n-87)

B TOPRA (n=15)

US only
29%
EU only
1%
Global
62%
Rest of World
1%
Other - please
specify
7%
US only
7%
EU only
14%
Global
72%
Rest of World
7%

88
The majority of EP respondents worked for companies that had headquarters in the US
(75%, 65/87), followed by EU (17%, 15/87), Asia Pacific (5%, 4/87), Israel (2%, 2/87)
and Canada (1%, 1/87). The majority of TOPRA respondents worked for companies
headquartered in Europe (60%, 9/15) but others worked in companies headquartered in
the US (27%, 4/15) or Asia Pacific (13%, 2/15).
Most EP respondents reported that they worked for either small companies of less than
250 employees (48%, 42/87) or large companies with more than 2000 employees
respectively (30%, 26/87). Relatively fewer (22%, 19/87) worked for medium sized
companies with between 250-2000 employees (Figure 15).
Figure 15: Company Size of EP Survey Participants
(n=87)

0%
10%
20%
30%
40%
50%
60%
Less than 250 employees 250-2000 employees More than 2000 employees

89
In contrast, the TOPRA sample had a slightly higher proportion of respondents from
larger companies (53%, 8/15); 5 (33%) worked in small companies and 2 (13%) in
medium-sized companies. Most respondents in both the EP and TOPRA samples
(EP 76%, 65/86; TOPRA 80%, 12/15) supported the commercialization of products that
were both on the market and in development. A minority (EP 23%, 20/86; TOPRA 13%
2/15) worked only with innovator prescription products in development and a very few
(EP 1%, 1/86; TOPRA 7%, 1/15) with only marketed prescription products.
Most EP and TOPRA respondents (EP 77%, 66/86; TOPRA 77%, 10/13 respectively)
had previously participated in benefit-risk assessment of new or existing products. The
two groups had similar amounts of regulatory experience of 11.4 and 11.1 years
respectively.
4.3 Current Status of Benefit-Risk Assessment
4.3.1 Benefit-Risk Assessment during Development
Respondents were queried about the way that their companies assessed risk-benefit for a
new medicine during development, by offering a choice of BR frameworks or the option
to provide an open-text description. In the EP, approximately one-fifth of the
respondents (20%, 17/87) identified that their companies used a purely qualitative
approach whereas about one-third (33%, 29/87) used a semi-quantitative approach that
incorporated both qualitative and quantitative elements, and a few (6%, 5/87) used a
purely quantitative approach. A final third (32%, 28/87) reported that their companies
have no formal approach (Figure 16).

90
In addition, eight (8) chose to provide a text description (9%, 8/87), most of which
identified that no internal formal approach was used. They were not included in these
calculations but are provided below in Table 4 and in Appendix C Results Report –
Expert Panel.
Table 4: Comments
a
on Internal Approaches Used During Development (EP
Group)
Although I have provided presentations and strategic advice we have not participated
in assessing risk benefit of products
Benefit risk is assessed on a case by case basis
Based on my understandings, our internal approach is purely qualitative.
I cannot answer
My company provides services to other companies, therefore BR is not directly
performed
The process is not formal but uses qualitative and semi-quantitative
There are no formal procedures as yet however this is being developed
a
All comments have been displayed in this table with the exception of the “Not
Applicable” comment.

91
Figure 16: BR Framework Approaches Identified by EP Respondents for Use
During Development
(n=87
b
)

b. Additional responses (8/87) account for the remaining 9%
EP respondents whose companies used a semi-quantitative approach typically identified
that approach as based on either the BRAT methodology developed by PhRMA, and the
preclinical assessment work that enables clinical trials and bases risk on the safety signals
that are seen in animal models. Some representative comments of the latter are provided
in Table 5 below. The full text of these comments is included in Appendix C Results
Report – Expert Panel.
0% 5% 10% 15% 20% 25% 30% 35%
Our internal approach is a purely
qualitative
Our internal approach is semi
quantitative and uses both qualitative
and quantitative elements. Please…
Our internal approach is a fully
quantitative model
Our company has no internal formal
approach for assessing benefit-risk of a
new medicine during development.

92
Table 5: Additional Comments on Semi-Quantitative Approaches Used
Internally During Development (EP Group)
Regular team review of clinical and nonclinical data
If there is a safety signal in a study, the signal is thoroughly evaluated to
understand if it is a true risk to patients, and what if anything can be done to
minimize signal if it is a risk to patients. Does benefit outweigh the risk?
We review the data from each study conducted which allows us the
opportunity to examine the safety and efficacy of the investigational product
over time.
The main risk-benefit activities that we conduct internally involve the safety
assessment of drugs prior to entering the clinic, as part of the IND-enabling
program. This assessment is conducted by toxicologists and is partly
quantitative (safety margins etc.) and qualitative (assessment and interpretation
of nonclinical outcomes)
We summarize preclinical data including pharmacology/tox/pk to evaluate a
drug’s potential early for product candidate. Post IND the clinical safety data
would of course be continuously monitored for overall assessment.

In the TOPRA group, similar results were seen (Figure 17). Nearly one-third of
companies reporting they use a purely qualitative approach (31%, 5/16), and a further
25% used a semi-qualitative approach (4/16). None used fully quantitative approaches
and 31% identified that no formal approach was used (5/16). Two additional comments
noted that respondents were not sure what approach was used if any.

93
Figure 17: BR Approaches
(TOPRA Group, n=16)

In order to establish whether a correlation exists between company size and
BR implementation, the responses identifying the choice of BR Approach were
cross-tabulated with the size of the company for the EP Group (Table 6). When analyzed
in this way, small companies (less than 250 employees) used either a qualitative
(21%, 9/42) or semi-quantitative approach to BR (26%, 11/42). Only one had a fully
quantitative approach and 40% (17/42) had no formal approach. The use of BR
frameworks was higher in large companies. Half of large companies used a
semi-quantitative approach (50%, 12/26), and a further quarter used a fully qualitative
approach (19%, 5/26) or fully quantitative approach (8% 2/26). Nineteen percent (5/26)
had no formal approach for assessing BR during development.
0% 5% 10% 15% 20% 25% 30% 35%
Our internal approach is a purely
qualitative
Our internal approach is semi
quantitative and uses both qualitative
and quantitative elements. Please…
Our internal approach is a fully
quantitative model
Our company has no internal formal
approach for assessing benefit-risk of a
new medicine during development.

94
Table 6: Cross Tabulation of Size of Company and BR Approach Utilized
During Development
(n=87)

What is the size of the company for which you are
currently working based on number of
employees?
Total
Less than 250
employees
250-2000
employees
More than
2000 employees
Our internal approach is a
purely qualitative 9 3 5 17
Our internal approach is semi
quantitative and uses both
qualitative and quantitative
elements.
11 5 13 29
Our internal approach is a fully
quantitative model
1 2 2 5
Our company has no internal
formal approach for assessing
benefit-risk of a new medicine
during development. 17 6 5 28
Other/Comment 4 3 1 8
Total 42 19 26 87

When asked if they were satisfied with the existing benefit-risk assessment approach used
by their company during development the opinions of EP respondents were split
(YES 54%, 47/87 versus NO 46%, 40/87). In the TOPRA group respondents were
slightly more biased toward satisfaction (60%, 9/15) than dissatisfaction (40%, 6/15) with
their approaches (Figure 18).

95
Figure 18: Satisfaction with BR Approach During Development
(EP Group, n=87 vs. TOPRA Group, n=16)

Respondents who expressed dissatisfaction with the existing approach were asked to
identify the reason from a list of seven (7) choices, and were allowed to choose all that
applied to their situation. They were also asked to explain their view if the offered
choices were not sufficient. Of the 36 respondents from the EP who answered the
question, 13 identified that they had no approach. Each of the remaining choices was
selected by a small number of participants (Figure 19). These included:
“approach/method is not reliable” (6/36); “poor acceptance by staff” (4/36), “requires
additional training” (4/36), and “benefits of approach are not apparent” (4/36). Notably
many did not select any one of these choices but rather selected “other” (14/36) and
provided additional comments, extracts of which are shown in Table 7. A full listing of
these comments is provided in Appendix C Results Report – Expert Panel.
0%
10%
20%
30%
40%
50%
60%
70%
EP% TOPRA
Yes No

96
Figure 19: Reasons for EP Dissatisfaction with Current BR Approach During
Development
(n=36)

0% 5% 10% 15% 20% 25% 30% 35% 40%
Too time consuming to utilize the approach
Requires additional training to understand
and learn to use the approach
Poor acceptance by staff
Benefits of the approach are not apparent
Approach/method is not reliable
We have no approach
Approach/method is not generally accepted
within the industry

97
Table 7: Other Reasons for EP Dissatisfaction with Current BR Approach
Although quantitative methodologies are not universally accepted, we ought to
at least experiment with these.
The issue is that previously we left this assessment to each team. This must be
formalized to ensure that there is a consistent approach used
The interpretation of results from any method is subjective, so it is unclear what
value the method provides
Methods are still in infancy and not well accepted by regulators. This makes it
hard to get traction in the company
Improvements on quantitative methods needed
There is complete reliance on FDA and EMA acceptability
Do not use frameworks – focus on risk management
In attempting to “standardize” practice guidelines, it is necessary to try to be
"systematic" in adjudicating the existing knowledge, but this good intention is
often unjustifiably taken to the level of religious devotion.
Our approach to benefit-risk assessment is not standardized or formalized
Not a standard template that all areas rely on and understand
Approach/methods used by different regulatory agencies differ substantially
Not sure if the quantitative methodology is understood more broadly across the
organization. Also concerned that the quantitative methodology may not be
well understood by reviewers in the health authorities

Six (6) respondents in the TOPRA group were dissatisfied with the approach used by
their company. Two (2) indicated that they had no approach, 2 that the approach required
additional training to understand and use, 1 that the benefits to the approach were not
apparent, and 1 that the methods were not reliable.
Most, EP participants identified that they perform Benefit-Risk Assessment for all
products (77%, 57/74) but a minority (23%, 17/74) assessed benefit and risk only for
certain products (Figure 20). When asked to describe how these “certain” products are

98
selected as candidates for assessment, the 15 survey participants who added text
comments most frequently stated that such evaluations are directed at high risk and
innovative products that require most review or at products whose benefit-risk parameters
and assessments are not obvious (Appendix C Results Report – Expert Panel).
Figure 20: BR Assessment for New Medicines
(EP Group, n=74)

Eighty-three percent (10/12) of the TOPRA group who answered this question indicated
they perform benefit-risk assessment during development for all products; the small
remaining number (17%, 2/10) both identified that they only assessed it for certain
products when warranted by their safety signals.
4.3.2 Benefit-Risk Assessment as Part of Regulatory Submissions
Respondents also were questioned about their approaches to benefit-risk assessment for
submission to regulatory agencies as part of a marketing application. Almost equal
0% 10% 20% 30% 40% 50% 60% 70% 80%
Only marketed prescription products
Only innovator prescription products in
development
A combination of products that are on
the market and in development

99
numbers of EP respondents indicated they use either a semi-quantitative approach that
incorporates both qualitative and quantitative elements (37%, 29/79) or that they have no
framework apart from the required text in the Clinical Overview (38%, 30/79). Five
(5, 6%)) responded that they use a fully quantitative approach and 15 (19%) reported that
they use qualitative methods only (Figure 21).
Figure 21: BR Framework Approaches for Regulatory Submissions
(EP Group, n=79)

Respondents whose companies used a semi-quantitative approach were also asked to
provide a more specific description of those approaches. Most used either the BRAT
methodology or rely on data driven methods (e.g., outcomes/endpoints that are analyzed
for individual studies and then pooled statistically). The full text of these comments is
included in Appendix C Results Report – Expert Panel.
0% 5% 10% 15% 20% 25% 30% 35% 40%
Our framework is a purely qualitative .
Our framework is semi quantitative and
uses both qualitative and quantitative
elements. Please describe briefly below
Our framework is a fully quantitative
Our company has no framework for
assessing benefit-risk of a new medicine
for submission. We provide the required
text in the clinical overview only

100
In order to explore whether a correlation similar to that examined previously to evaluate
the relationship between company size and BR implementation during development,
choice of BR Approach for submissions was cross-tabulated with the size of the company
for the EP Group. Small companies (less than 250 employees) used either a qualitative
(23%, 9/39) or semi-quantitative (28%, 11/39) approach to BR. Only two (2) reported
that they had fully quantitative approaches and a substantial number (44%, 17/39) had no
formal approach for assessing BR. More than half of large companies (more than 2000
employees) used a semi-quantitative approach (57%, 13/23). The use of a fully
qualitative approach (4%, 1/23) or fully quantitative approach (4% 1/23) appeared to be
rare. Twenty-two percent (5/23) of those from large companies reported that they had no
formal approach for assessing BR for regulatory submissions (Table 8).

101
Table 8: Cross Tabulation of Size of Company and BR Approach Utilized for
Regulatory Submissions
(EP Group, n=79)

What is the size of the company for
which you are currently working based
on number of employees?

Total
Less than
250
employees
250-2000
employees
More than
2000
employees
Our framework is a purely qualitative
9 2 4 15
Our framework is semi quantitative
and uses both qualitative and
quantitative elements. Please describe
briefly below 11 5 13 29
Our framework is a fully quantitative
2 2 1 5
Our company has no framework for
assessing benefit-risk of a new medicine
for submission. We provide the
required text in the clinical overview
only 17 8 5 30
Total 39 17 23 79

In the TOPRA group (Figure 22), many (46%, 6/13) indicated that they use
semi-quantitative frameworks, but 4 identified that they use purely qualitative
frameworks (31%), 2 used no framework (15%) and one used a fully quantitative
framework (8%).

102
Figure 22: BR Framework Approaches for Regulatory Submissions
(TOPRA Group, n=13)

When asked whether they were satisfied with their company’s approach to Benefit-Risk
assessments included in regulatory submissions, the respondents expressed similar levels
of satisfaction to those expressed when they were asked about their satisfaction with
approaches used during development (EP: YES 54%, 42/77; NO 45%, 35/77; TOPRA:
YES 60%, 9/15; NO 40%, 6/15).
The respondents who were dissatisfied were asked to provide the reasons for that
dissatisfaction from the same list of seven (7) reasons provided in the previous question
regarding BR frameworks in development, and were allowed to choose all that applied to
their situation. Most commonly, the respondents (43%, 13/30) expressed dissatisfaction
with the fact that they had no framework. Six respondents (20%, 6/30) indicated that
additional training was required to understand and use the framework, four respondents
0% 10% 20% 30% 40% 50%
Our framework is a purely qualitative .
Our framework is semi quantitative and
uses both qualitative and quantitative
elements. Please describe briefly below
Our framework is a fully quantitative
Our company has no framework for
assessing benefit-risk of a new medicine
for submission. We provide the…

103
(13%, 4/30) felt that the framework was not reliable and another four felt that the benefits
of the framework are not apparent (Figure 23).
Figure 23: Reasons for EP Dissatisfaction with Current BR Approach for
Submissions
(EP Group, n=30)

Eleven respondents added comments to amplify “Other” concerns. Some of these
comments referenced the prior answers provided with respect to Benefit-Risk during
development (Table 7) or highlighted the immaturity of the frameworks and their
acceptance and utilization by regulators (Appendix C Results Report – Expert Panel).
The TOPRA group of six respondents who were dissatisfied had varied responses as well.
Three of the six (50%) felt that additional training was needed, two (33%) were
dissatisfied because they had no framework and single respondents indicating either poor
acceptance by internal staff or by industry in general (17%, 1/6).
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%
Too time consuming to utilize the
framework
Requires additional training to understand
and learn to use the framework
Poor acceptance by staff
Framework is not reliable
Benefits of the framework are not
apparent
We have no framework
Framework is not generally accepted
within the industry
Other - please describe below.

104
In the EP group, the 30 respondents who had no formal system were given a drop down
menu of five choices and an open text field to “choose the reason(s) that best describe
why a formal system is not used”. Survey participants could make multiple selections.
The most common reason was “lack of knowledge of BR frameworks in general” (53%,
16/30) and “lack of a common BR framework among stakeholders” (40%, 12/30).
However, a few individuals selected other options (not required for current product; lack
of a scientifically validated framework; benefits of BR framework not apparent) as
illustrated in Figure 24. There were four respondents in the TOPRA group who
identified no formal BR approach; all chose “lack of knowledge of BR frameworks in
general” as the primary reason why a B-F framework was not used.
Figure 24: Reasons for not using BR Frameworks
(EP Group, n=30)

0% 10% 20% 30% 40% 50% 60%
Other (Please specify)
Not required for current product
developmental processes
Lack of scientifically validated BR
framework
Lack of knowledge of BR frameworks
in general
Lack of a common BR framework
among peers and/or stakeholders
Benefits of a BR framework not
apparent

105
Both the EP and TOPRA groups were provided with a text box to elaborate on other
reasons why their companies do not use BR frameworks. Only EP respondents provided
additional comments (See Appendix C). Reasons included: the small size of the
company; the lack of time available to develop and use a framework; added “costs”
required to maintain a formal approach; or the immaturity of the framework(s) of
frameworks that were still in development. One additional comment stated: “BR
assessment outcomes are generally driven by opinions of influential regulatory agency
staff than any specific process”.
Respondents whose companies had a formalized approach to benefit-risk assessment
were asked to indicate why they selected a particular approach. The survey did not force
respondents to choose only one explanation, so that the number and nature of responses
was varied (Table 9). The reason most selected as “extremely important” was alignment
with company expertise (16/58), followed by alignment with regulator templates (14/56).
Those reasons most frequently identified as “very important” included alignment with
product development strategy (40/57), alignment with regulator templates (28/56),
alignment with company expertise (27/58) and alignment with reimbursement strategy
(21/57). A neutral position (indicated by the response of neither important nor
unimportant) was most common with regard to alignment with reimbursement strategy
(21/57), which also received the most responses for “not at all important’ (6/57).
A similar distribution of responses was given by the 12 respondents who used BR
assessment frameworks in the TOPRA group (not shown).

106
Table 9: BR Assessment Alignment - By Importance (EP Group)

Not at all
Important
Very
Unimportant
Neither
Important
nor
Unimportant
Very
Important
Extremely
Important
Total
Align with
company expertise
1 1 12 27 16 57
Align with
regulator
templates
1 4 9 28 13 55
Align with
reimbursement
strategy
6 3 21 20 6 56
Align with product
development
strategy
1 1 7 39 8 56

4.4 Benefit-Risk Framework Knowledge and Perceptions
When presented with a list of frameworks and methods that have been proposed as ways
to perform benefit-risk assessments, respondents were asked to indicate those with which
they were personally familiar, and could select multiple answers and check all that
applied. There were 59 total responses as shown in Figure 25. The framework with the
most responses (39) was the US FDA Benefit Risk Framework (BRF). Fewer
respondents were familiar with Value/Decision trees (23), BRAT from PhRMA (21) and
the EMA PrOACT-URL (18). Fourteen (14) were not familiar with any of these
frameworks or methods and three indicated they were familiar with other methods,
including the risk management framework of the International Standards Organization
(ISO 14971) and numerical indices including Number Needed to Treat/Harm

107
(NNT/NNH) and Net Clinical Benefit (NCB). One individual also cited the EU Good
Pharmacovigilance Practices (GVP).
Figure 25: Expert Panel BR Framework Familiarity – Number of Responses

In the TOPRA group, Value/Decision trees had the highest degree of familiarity (6) and
only a slightly smaller number were familiar with the US FDA BRF (5) and PhRMA
BRAT (5). An additional four (4) respondents were also familiar with both the EMA
PrOACT-URL and MCDA.
Respondents were then asked to rate their level of knowledge with respect to those
frameworks with which they were familiar using a scale of 1 (not familiar) to
10 (extremely familiar). As shown in Table 10, the US FDA Benefit Risk Framework
(BRF) received the most responses (43) and had a mean self-assessment score of 5.28
(out of 10). The other frameworks with relatively frequent responses were
0 5 10 15 20 25 30 35 40 45
US FDA BR Framework
Value or Decision Tree
Benefit Risk Assessment Team (BRAT)
Framework
EMA PrOACT-URL
Not familiar with any of these frameworks
Multi-Criteria Decision Analysis (MCDA)
Unified/Universal Method for Benefit Risk
Assessment (UMBRA)
Stochastic Multi-criteria Acceptability
Analysis (SMAA)
Other - please specify
Ashby and Smith Framework

108
Value/Decision Tree (33 responses with 5.36 average knowledge value), the BRAT
Framework (24 responses with average knowledge value of 5.38) and the EMA
PrOACT-URL (22 responses with a knowledge value of 4.55). The UMBRA and
MCDA methodologies had fewer responses (15 and 16 respectively), but those who were
familiar had a level of knowledge in the range of those for other methods (5.47 and 4.19).
The number of responses (6) and level of knowledge (1.00) was the least for the Ashby
and Smith framework.
Table 10: BR Framework Level of Knowledge of the Expert Panel
FRAMEWORK Average Value Responses
US FDA BR Framework 5.28 43
EMA PrOACT-URL 4.55 22
Unified/Universal Method for Benefit Risk
Assessment (UMBRA)
5.47 15
BRAT Framework 5.38 24
Stochastic Multi-criteria Acceptability
Analysis(SMAA)
3.33 9
Multi-Criteria Decision Analysis(MCDA) 4.19 16
Ashby and Smith Framework 1.00 6
Value or Decision Tree 5.36 33
[Scale 1 (not knowledgeable) -10 (extremely knowledgeable)]

The TOPRA group had lower levels of familiarity in general than the EP group. The
most frequent responses were given for Value/Decision Tree (10), FDA BRF (8), BRAT
(8), MCDA (8) and EMA PrOACT-URL (7) frameworks. The mean levels of familiarity
that were associated with these were relatively low compared to those indicated by the
EP group (2.70, 3.50, 2.25, 2.00 and 3.86, respectively). The Ashby and Smith

109
Framework received only four responses and was also rated the lowest level of
familiarity (1.00).
4.4.1 Utilization of BR Framework Elements
Participants were asked to “indicate how benefit-risk information is currently
characterized in the company’s regulatory submission”. Most commonly, respondents
(44%, 27/61) indicated they use “a semi-quantitative approach using qualitative text and
some quantitative elements”. A smaller number of companies (26%, 16/61) provide only
qualitative information in the form of text. Nine respondents identified that their
companies utilize quantitative displays of cumulative outcomes in the form of text, tables
and other visualization methods (15%, 9/61). Several respondents (10%, 6/61) did not
know how BR information was characterized in their regulatory submissions (Figure 26).

110
Figure 26: BR Approaches in Regulatory Submissions
(EP Group, n=61)

Respondents were also allowed to choose “other”; three respondents characterized their
approaches as 1) in planning stage, with no decision taken yet on content or how
benefit-risk will be described, 2) as needed for specific areas, and this comment from a
consultant, 3) customized according to the needs of different sponsors/clients.
In the TOPRA group, half of the respondents (50%, 5/10) reported the use of a
quantitative process displaying cumulative outcomes in the form of text, tables and other
visualization methods. Two reported the use of a semi-quantitative approach (20%, 2/10),
and none used text only. Three of the 10 respondents to this question indicated that they
had no knowledge of how benefit-risk information is characterized in regulatory
submissions (30%, 3/10).
0% 10% 20% 30% 40% 50%
Qualitative approach using text only
Semi-quantitative approach that uses
qualitative text and some quantitative
elements (e.g. Effects Table)
Quantitative approach displaying a
cumulative outcome in the form of text,
tables and other visualization methods
(eg forest plots, bar graphs, etc)
I do not know how benefit-risk
information is characterized in the
regulatory submission(s)

111
4.5 Implementation Issues
In order to assess the state of implementation of BR frameworks in the industry, a list of
the proposed benefit-risk assessment frameworks was provided, and respondents were
asked to identify the extent to which each was being considered or used from 4 choices
reflecting various degrees of implementation: evaluating framework; utilizing
framework for internal use; submitting framework with registration dossiers; using to
communicate with patients and prescribers. Each statement started with “My
company…” Respondents could also identify either that they were not using a
framework or not aware of the use of a framework (Table 11).

112
Table 11: Implementation of BR Frameworks (EP Group)
FRAMEWORK
Does not
use this
framework
Has/is
evaluated
this for
use
within
our
company
Utilizes
this
method
for
internal
use
only
Utilizes this
method
and
submits it
with
registration
dossiers
Utilizes this
method to
communicate
externally to
prescribers
and patients
Not aware
of this
framework
Total
US FDA BR
Framework
20 8 8 7 0 9 52
EMA PrOACT-
URL
18 7 2 3 0 21 51
Unified/Universal
Method for
Benefit Risk
Assessment
(UMBRA)
22 3 2 1 0 23 51
BRAT
Framework
18 6 2 3 0 21 50
Stochastic Multi-
criteria
Acceptability
Analysis(SMAA)
21 2 1 0 0 25 49
Multi-Criteria
Decision
Analysis(MCDA)
23 0 4 1 0 22 50
Ashby and Smith
Framework
22 0 0 0 0 27 49
Value or Decision
Tree
20 10 8 2 0 12 52

The US FDA BRF appeared to be the method that was implemented most commonly.
Only 9 respondents were not aware of this framework. Nevertheless, even this
framework was not used extensively. Eight (8) respondents of 51 identified its use for
internal assessments and 7 as part of submissions for registration. Two other methods
evaluated by a small number of companies were Value or Decision Trees (10/52), EMA
PrOACT-URL (7/51) and the BRAT framework (6/50). The remaining frameworks were
evaluated for use or were implemented for internal or registration purposes by less than

113
5 respondents. Perhaps surprisingly most methods were not known to a substantial
minority or even a majority of respondents. It is also notable was that no framework was
currently being used as part of efforts to communicate externally with payers or
providers.
In the TOPRA group, there were only three responses indicating that any of the BR
assessment methods were being used for submission of registration dossiers (Table 12).
One indicated that UMBRA, one that the BRAT was used and another one value trees
were being employed. Single responses were also posted for the use of the FDA and
EMA frameworks to communicate to prescribers and patients (1/13 each). The highest
responses (4-6 each) indicated a general lack of awareness and use of most of the
frameworks.

114
Table 12: Implementation of BR Frameworks (TOPRA Group)
FRAMEWORK
Does not
use this
framework
Has/is
evaluated
this for
use
within
our
company
Utilizes
this
method
for
internal
use only
Utilizes this
method
and
submits it
with
registration
dossiers
Utilizes this
method to
communicate
externally to
prescribers
and patients
Not aware
of this
framework
Total
US FDA BR
Framework
2 2 2 0 1 6 13
EMA PrOACT-
URL
3 1 2 0 1 6 13
Unified/Univers
al Method for
Benefit Risk
Assessment
(UMBRA)
3 1 0 1 0 5 10
BRAT
Framework
3 2 2 1 0 5 13
Stochastic
Multi-criteria
Acceptability
Analysis-SMAA
4 0 0 0 0 6 10
Multi-Criteria
Decision
Analysis-MCDA
4 2 1 0 0 5 12
Ashby and
Smith
Framework
4 0 0 0 0 6 10
Value or
Decision Tree
5 1 1 1 0 4 12

115
Respondents were also asked about certain quantitative tools and metrics that were used
by the companies with which they worked. Using the same scale as in the prior question,
respondents were presented with a list of models and indices that can be used to calculate
benefit-risk. Numerical indices were not used commonly for registration dossiers but the
two most often employed were the Benefit Risk Ratio (10/56) and the Number Needed to
Treat/Number Needed to Harm (8/56). Quality Adjusted Life Years (QALY) was the
index most commonly used for internal BR assessments (10/57). These same indices
were also used occasionally to communicate externally to prescribers and patients.
However, most of the indices were either not being used or were unfamiliar to a
substantial proportion of respondents (Table 13). An additional cross tabulation of these
methodologies matched to company size showed that the lack of knowledge and
familiarity was similar regardless of the size of the enterprise (Appendix D Table D.1).

116
Table 13: Implementation of Models/Indices (EP Group)
MODEL/
INDICES
Does not
use this
framework
Has/is
evaluated
this for
use
within
our
company
Utilizes
this
method for
BR
Assessment
for
internal
use only
Utilizes this
method
and
submits it
with
registration
dossiers
Utilizes this
method to
communicate
externally to
prescribers
and patients
Not aware
of this
framework
Total
Number
Needed to
Treat (NNT)/
Number
Needed to
Harm (NNH)
17 2 7 8 2 20 56
Benefit Risk
Ratio (BRR)
20 3 8 10 2 13 56
Incremental
Net Benefit
(INB)
20 2 5 2 0 24 53
Quality
Adjusted Life
Years
(QALY)
20 2 10 5 3 17 57
Stated
Preference
Method
(Patient)
23 1 3 1 0 24 52
Quality
adjusted Time
Without
Symptoms
and Toxicity
(Q-TWiST)
23 2 3 0 0 25 53
Minimum
Clinical
Efficacy
(MCE)
24 2 6 5 1 18 56
Minimal
Acceptable
Risk (MAR)
25 1 9 3 1 17 56

117

In the TOPRA panel, a single respondent (1/11) identified a single tool, QALY, for
registration purposes. QALY and BBR were also used by 3 respondents each for external
communication to prescribers and patients. BBR and QALY were also used most
commonly for internal purposes (4/11 and 3/11 respectively).
4.5.1 Location of BR Assessment Elements
When questioned about where benefit-risk information is currently utilized as part of the
regulatory submissions for marketing purposes, most (89%, 50/56) indicated that it
appeared either in the Common Technical Document Clinical Overview (Module 2.5.6),
as part of labeling development, including the Target Product Profile (61%, 34/56), or as
part of the development of the Package Insert and Prescribing Information (59%, 33/56)
(Figure 27). Less commonly the material appeared in “other” places, including the
Investigator’s Brochure (IB), Periodic Safety Update Reports (PSUR) and Benefit –Risk
Management Plan (BRMP) (Appendix C Results Report – Expert Panel).

118

Figure 27: Location of BR Information in Marketing Applications
(EP Group, n=56)

The results from the TOPRA group mirrored those of the EP group. The CTD Clinical
Overview (M2.5.6) was the most common location of BR information (91%, 10/11) but
some identified that BR information was placed as part of the TPP and PI development
(55%, 6/11 for each) and four responses for “other” identified placement in the
Investigational Medicinal Product Dossier (IMPD), Periodic Benefit Risk Evaluation
Report (PBRER), Clinical Development Plan (CDP) and Risk Management Plan (RMP).
4.5.2 Feedback and Influence of BR Assessment Frameworks
Registration dossiers are reviewed by regulatory agencies who routinely ask questions
about various aspects of the submission. Thus respondents were asked about the
feedback received regarding the format of their benefit-risk materials. About half of
those respondents (48%, 30/62) identified that they received no such feedback from the
0% 20% 40% 60% 80% 100%
Other - please specifiy
To develop Package Insert/Prescribing
Information (PI)
As part of labeling development (e.g.
Target Product Profile/TPP)
As part of the Clinical Overview (CTD
M2.5.6)

119

regulators (Figure 28). Those with feedback identified , that the feedback came at
different points in the submission process, including discussions with regulators during
the content review of a marketing application (21%, 13/62), as part of discussions
regarding the development plan (19%, 12/62), as part of labeling negotiations (19%,
12/62) or at pre-submission meetings (18%, 11/62). Feedback was also provided as part
of the filing review for the marketing application (16%, 10/61) or at the end of the review
period (13%, 8/62).
Figure 28: Feedback from Regulatory Authorities on BR Frameworks
(EP Group, n=62)

When asked to provide additional details or other information, survey participants either
indicated that they were not sure or not involved or they stated that no formal benefit-risk
0% 10% 20% 30% 40% 50% 60%
Additional details (optional)
Other - please specify
As part of formal agency
communication at the end of review
As part of marketing application filing
review
At pre-submission meeting (e.g.
preNDA/MAA)
As part of discussions with regulators
regarding development plan (e.g.…
As part of labeling negotiations
As part of a question during marketing
application content review
No, no feedback provided

120

comments were discussed. One respondent did note that the US FDA indicated that they
would like a “fuller discussion around BR”.
In the TOPRA group four responses indicated that feedback was provided at/in either the
pre-submission meeting (4/13), and an equal number reported the feedback as part of the
marketing filing review (4/13), as part of labeling negotiations (4/13) or as part of formal
agency communication at the end of the dossier review (4/13). Another avenue for
feedback was as a question during the application content review (23%, 3/13). About a
third of the participants indicated they received no feedback (31%, 4/13).
Respondents from both the EP and TOPRA groups appeared to have mixed views about
the degree of influence of the benefit-risk assessment exercise on decisions regarding the
approvability of product. Many respondents in the EP (65%, 36/55) viewed it as a factor,
specifically for the US and EU regulators, while others (35%, 19/55) did not believe that
it influenced the decision. Survey participants were given the opportunity to elaborate on
this influence if they answered in the affirmative by using an open text comments box.
Several provided additional details about the importance of the BR analysis and these are
included in Table 14. They were also asked if there were a region or country where the
BR information was influential. As detailed in Appendix C Results Report – Expert
Panel, the answers to this question (6/6) mentioned only the US and Europe (EMA or EU
indicated).

121

Table 14: Comments on BR Assessment Influence on Regulators (EP Group)
Yes - comments (optional)
For FDA perspective I believe it helps guide the FDA as they consider their own
decision. It does play a larger role in advisory committee meetings.
Absolutely this is important
Yes, in the sense that the regulatory makes a decision based on benefit-risk, but not in
the sense of a specific methodology as noted in earlier questions
US market only
In past, have seen it discussed in conjunction with product approvals. I feel it was a
key element of the agency decision.
Of course it does
But I can't quantify
Very subjective
I believe our prose description of the benefit-risk considerations was reviewed and
generally accepted by the regulatory reviewers.

Similar responses were provided by the TOPRA group; 73% (8/11) of respondents felt
that it influenced the decision made by regulators and 27% felt that it did not. This group
also commented that the influence was most apparent in the European Union, North
America, Latin America, and Japan. Of those constituencies, the EU was the region
where this influence was viewed to be most apparent.
4.6 Advantages and Barriers to Implementation of BR Frameworks
To examine the perceived advantages to implementing benefit-risk frameworks, a list of
pre-specified possible advantages were provided and survey recipients were asked to rate
them in importance from “Not at all important” to “Very important”. No opinion was
also an option and respondents were not forced to rate each possible advantage, so that

122

the number of responses varied (Table 15). The response that was rated as “Very
important” most often by the EP group (30/45) was that “these frameworks ensure
consistency in quality of BR assessment within the organization”. Also rated as very
important (29/45) were the advantages accrued from acting as a tool for communication
among peers within the organization, to enhancing transparency within the organization
(27/46) and to aligning internal decision-making (27/38). The response that was rated as
“Not at all important” by the most respondents was its use as a tool for communication
with prescribers and patients (4/44).
Table 15: Advantages of BR Utilization (EP Group)
Possible Advantages
Not at all
Important
Somewhat
important
Very
Important
No
Opinion
Total
Acts as a tool for communication
among peers within organization
1 12 29 3 45
Acts as a tool for communication
between organization and external
stakeholders (including regulatory
agencies)
0 6 24 2 32
Provides archival documentation for
structured decision-making
2 20 23 3 48
Ensures consistency in quality of BR
assessment within the organization
2 12 30 1 45
Enhances transparency within the
organization
3 14 27 2 46
Ensures consistency in quality of BR
assessment by regulators
3 12 20 4 39
Aligns internal decision-making 1 9 27 1 38
Facilitates external communications
with prescribers and patients
4 14 18 8 44

123

In addition, survey participants were given the option to contribute “other” potential
advantages in a provided text box, some of which are provided in Table 16.
Table 16: Additional Advantages to Utilizing BR Frameworks (EP Group)
It provides a mechanism to help small companies make development decisions (e.g.,
kill products)
Aligns internal and external communication about the product by organization
Although our company is adopting B:R approaches and submitting in applications,
regulatory agencies do not comment or engage our company in its B:R proposals

In the TOPRA group the factors considered “very important” were the use of a BR
framework as a communication tool among peers within the organization (3/7), as a tool
for communication with stakeholders (3/4), and as a tool to improve transparency within
the organization (3/6).
Questions were also posed about barriers to implementation of BR frameworks. A list of
barriers was provided and participants were asked to rank the level of importance that
these barriers represent to their company when implementing a structured approach to
benefit-risk assessment, using the same scale as that in the previous question from “Not
at all important” to “Extremely important”. Three significant barriers identified by
60 members of the EP panel as “extremely important” included “lack of
knowledge/expertise to execute the framework” (24/60), “lack of a regulatory
harmonized framework” (24/60) and “lack of support from senior management” (24/60).
The same three barriers were ranked as extremely important by the TOPRA group
(3/10 for each). Additional significant barriers that were cited as extremely important by

124

at least one-quarter of respondents in the EP group were resource limitations (18/60),
resistance to change (18/60) and lack of an industry accepted framework (18/60). In
addition to these barriers, those that were considered “very important” also included
significant change to work process (24/60), significant effort to retrain staff (23/60), and
lack of knowledge/expertise to execute the framework (23/60). Nevertheless, all of the
provided selections were ranked as important to extremely important by the large
majority of respondents (Table 17).

125

Table 17: Barriers to Implementation of BR Frameworks
(EP Group, n=60)
Possible Barriers
Not at all
Important
Somewhat
important
Very
Important
Extremely
Important
Cannot
Answer
Total
Responses
Resource limitation
e.g. manpower,
finances
5 12 20 18 5 60
Lack of
knowledge/expertise
to execute
framework
2 6 23 24 5 60
Resistance to change
(culture
documentation or
methodology)
5 14 18 18 5 60
Significant change
to work processes
7 16 24 9 4 60
Significant
retraining of staff
required
7 17 23 9 4 60
Support from senior
management
required
8 4 20 24 4 60
Lack of an industry
accepted framework
2 11 22 18 7 60
Lack of a regulatory
harmonized
framework
1 9 19 24 7 60

In the TOPRA group, the provided selections were ranked as important to extremely
important by the large majority of the 10 respondents.
Survey participants were then asked to rank the barriers presented in the previous
question in order of importance. The EP rankings suggested that the lack of a regulatory
harmonized framework was the most important (15/55, 27%), followed by lack of
knowledge/expertise to execute the framework (12/55, 22%) (Figure 29). The TOPRA

126

rankings were similar; of eight respondents to this question, two identified the lack of a
regulatory harmonized framework and 3 noted the lack of knowledge and expertise to
execute the frameworks. The TOPRA group also identified support from senior
management (2/8) as a barrier to implementation (Appendix C Results Report – TOPRA
Panel).
Figure 29: Barriers to Implementation of BR Frameworks by Importance
(EP Group, n=55)

4.6.1 BR Framework Attributes Important to Overcome Barriers
To gain insight into the ways in which respondents felt that BR frameworks should be
developed for best results, participants were given a series of attributes that a BR
Framework could have and were asked about the importance of each on a scale between
strongly agree and strongly disagree (Table 18).
0% 5% 10% 15% 20% 25% 30%
Lack of a regulatory harmonized framework
Lack of knowledge/expertise to execute
framework
Resource limitation e.g. manpower, finances
Support from senior management required
Lack of a industry accepted framework
Significant change to work processes
Resistance to change (culture documentation
or methodology)
Significant retraining of staff required

127

The statements with which most EP respondents “strongly agreed” (17/60) was “an ICH
harmonized guidance on BR frameworks is needed”, followed by “It is important that any
BR framework, if developed for registration purposes, is utilized across product classes”
(14/58) and, “It is important that all stakeholders are part of the development and
validation” (14/59). Statements with which the respondents simply “agreed” were
observed more frequently. A majority agreed that an appropriate BR framework for
registration should “enable assessment of benefit-risk management plans (BRMP)”
(37/58) and “should …apply to all stages of drug development…” (33/59). For this
reason Table 18 combines the various degrees of agreement into a single combined
number. Using this aggregate number over 75% of respondents agreed that an ICH
harmonized framework is an important factor in implementation (46/60).

128

Table 18: BR Framework Implementation Attributes (EP Group)
Attributes
Strongly
Agree
Agree
Combined
Agree
Score
Neither Agree
nor Disagree
Disagree
Strongly
Disagree
Total
For the registration of new
medicinal products it will be
necessary to develop therapeutic
area specific BR frameworks and
methods
10 29 39 12 7 1 59
It is important that any BR
framework, if developed for
registration purposes, is utilized
across product classes (e.g. drugs,
biologics and devices)
14 19 33 17 8 0 58
It is important that any BR
framework, if developed for
registration purposes, is utilized
worldwide
11 18 29 20 9 1 59

129

Table 18: BR Framework Implementation Attributes (EP Group) (Cont'd)
Attributes
Strongly
Agree
Agree
Combined
Agree
Score
Neither Agree
nor Disagree
Disagree
Strongly
Disagree
Total
An appropriate BR framework for
registration should also enable
assessment of benefit-risk
management plans (BRMP).
7 37 44 12 1 1 58
An appropriate BR framework for
registration should also apply to
all stages of drug development
from drug development to post-
approval changes
9 33 42 9 8 0 59
It is important that all
stakeholders (agencies, companies,
prescribers and patients) are part
of the development and validation
of an appropriate Benefit Risk
framework
14 29 43 11 5 0 59
BR Frameworks should be utilized
more in communications with
prescribers and patients
6 25 31 21 7 0 59
An ICH harmonized guidance on
the use of BR frameworks is
needed
17 29 46 12 2 0 60

130

In the TOPRA group, overall there were far fewer responses in the “strongly agree”
category. Highest (3/9) were the selections of “it is important that any BR framework, if
developed for registration purposes, is utilized worldwide” and “it is important that all
stakeholders (agencies, companies, prescribers and patients) are part of the development
and validation of an appropriate Benefit Risk framework” (Appendix C Results Report –
TOPRA Panel). More responses were recorded in the “agree” column. For example, six
(of 9) listed “an ICH harmonized guidance on the use of BR frameworks is needed” and
“an appropriate BR framework for registration should also enable assessment of benefit-
risk management plans (BRMP)”.
The participants were then asked to “provide their opinion about usefulness and utility of
structured benefit-risk frameworks”. The survey asked participants to complete the
sentence “Having a BR Framework that…” and were then provided with a list of ways in
which BR Frameworks could be utilized. The scale that was provided rated allowed
respondents to choose responses from “extremely useful” to “not useful”. The survey
tool did not force respondents to categorize each statement, so that the number of
responses varied.
A framework that “is standardized and harmonized” was most commonly rated as
extremely useful and received the most responses from the EP panel (28/61). This was
followed closely by the choice of having a BR Framework which “allows for quality in
decision making for sponsors/companies” (25/60) and which “allows for consistency in
decision making for sponsors/companies” (24/60) (Table 19).

131

All of the provided selections were ranked as useful to extremely useful by the large
majority of respondents. There were only single digit responses for those BR
assessments that were considered “not useful”. The statement with the lowest values for
usefulness was that which indicated it as a communication tool for patients (5/60).
Table 19: BR Framework Utility (EP Group)
Utility Extremely useful
Very
Useful
Somewhat
useful
Not
useful
Total
Is standardized and
harmonized
28 27 4 1 61
Can be repurposed for health
technology assessment (e.g.
value
dossier/reimbursement)
20 26 12 2 60
Can be used to communicate
with prescribers
12 31 14 3 60
Can be used to communicate
with patients
11 19 25 5 60
Allows for transparency in
decision making for
sponsors/companies
22 27 9 2 60
Allows for consistency in
decision making for
sponsors/companies
24 24 11 1 60
Allows for quality in decision
making for
sponsors/companies
25 26 8 1 60
Accelerates decision making
for sponsors/companies
21 25 12 2 60
Documents decision making
for sponsors/companies
18 32 10 0 60

Similar to the Expert Panel the TOPRA group found all of the listed utilities of BR
Frameworks to be useful to extremely useful. In particular, a framework that “allows for
transparency in decision making for sponsors and companies” as well as one that “allows

132

for consistency in decision making for sponsors/companies” was rated as “extremely
useful” by half of the TOPRA group (5/10).
It was announced by ICH in 2014 that the M4 Expert Working Group would be issuing a
revision (R2) to the multidisciplinary guidelines for the Common Technical Document.
The survey participants were asked to complete the following statement: “An ICH
sponsored harmonized guidance that is part of M4 (R2) addressing the format for
benefit-risk data presentation would be most useful if it was based on...” They were
provided the same list of frameworks as in prior questions with one addition, which
proposed “an independent harmonized approach”.
Survey recipients had mixed views on which benefit-risk framework would be the most
useful to incorporate into an ICH harmonized guidance [M4(R2)]. Chosen most
commonly by the EP were the US FDA BRF (41%, 24/58) and the EMA PrOACT-URL
(31%, 18/58). An independent harmonized approach was also chosen by a quarter of the
respondents (25%, 14/48) (Figure 30). The opportunity to add “other” options was
provided by supplying respondents with a text box; a quarter of the respondents took the
opportunity to add comments (25%, 14/48), but most of the comments (11/14) identified
that the respondent did not feel qualified to answer (Appendix C Results Report – Expert
Panel).

133

Figure 30: Factors to Consider for ICH M4 (R2) Revision
(EP Group, n=59)

In the TOPRA group there were eight (8) total responses which equally indicated that the
FDA BRF, EM PrOACT-URL or UMBRA would be useful as a basis for an ICH
harmonized guidance, with 4/8 for each of those frameworks.
Finally, respondents were asked if they had engaged external stakeholders, either
prescribers, payers or patients, in their BR assessment (Figure 31). Most of the EP group
(39%, 24/61) said they had not involved such external stakeholders. A minority had
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
FDA BR Framework
EMA PrOACT-URL
An independent harmonized approach
Other
Unified/Universal Method for Benefit
Risk Assessment (UMBRA)
BRAT Framework
Value/Decision Tree(s)
Multi-Criteria Decision Analysis
None of the above
Stochastic Multi-criteria Acceptability
Analysis
Ashby and Smith Framework

134

involved prescribers/physicians (31%, 19/61), but only a few had involved payers (10%,
6/61) or patients (8%, 5/61). Almost one quarter of the respondents did not know
whether stakeholders had been engaged (23%, 14/61).
Figure 31: Stakeholder Engagement in BR Frameworks
(EP Group, n=61)

The survey participants also had the opportunity in provide text box to describe in more
depth how external stakeholders had been engaged in their processes. Nine respondents
provided a range of answers as shown in Table 20.
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
Patients
Payers (e.g. HTA or Reimbursement
Providers)
If yes, please briefly describe how you
have engaged them
I do not know
Prescribers/Physicians
No we have not engaged external
stakeholder in our benefit-risk
assessment process

135

Table 20: Stakeholder Engagement Comments
1. We continuously work with outside clinical and regulatory
experts as part of our ongoing development and regulatory
processes. Their feedback contributes to our BR assessment
2. Oncologist for sample preparation
3. For some products we have conducted stated-preference surveys
of patients or physicians
4. Open discussions of potential and realized risks.
5. Qualitative - discussion
6. Multi-criteria Decision Analysis (MCDA), Patient Reported
Outcomes (PROs), Patient Advisory Groups (PAG)
7. Sought guidance and input on risk-benefit assessment and
interpretations
8. As beta-testers
9. Advisory board review and discussion

Similar results were seen from the eleven (11) responses received from the TOPRA
group. Almost half (45%, 5/11) did not engage stakeholders at all in their BR assessment
process, while 36% (4/11) involved prescribers and 18% (2/11 responses each) indicated
that they involved patients or payers or did not know. For those that chose to describe
stakeholder engagement further, their comments included use of Key Opinion Leaders
(KOLs), HTA scientific advice and guidance and a note that experience in stakeholder
engagement is limited and should be increased.

136

An additional text box was provided at the end of the survey for participants to provide
any additional information which they felt might amplify their views on BR assessment
and the utilization of BR frameworks in decision-making, regulatory filings and
communications. Eighteen (18) comments were provided by the Expert Panel and
another two from the TOPRA group. These comments highlighted concerns about the
resources (e.g. experience, training, time and money) needed to implement a formal BR
Framework. Also, the Expert Panel identified issues with the maturity, acceptability and
harmonization of the BR Frameworks themselves, with participants stating they would
only perform them if requested by regulators. They also reinforced the need for a
harmonized approach across regions and the guidance that is needed. In addition,
respondents were concerned about the use of a standard “one size fits all” approach to BR
assessment and the varying stakeholders involved, noting that BR assessment has
contextual relevance that must be recognized. The comments are provided in their
entirety in Appendix C Results Report – Expert Panel and are excerpted and grouped
below in Table 21.

137

Table 21: Additional Comments from the Expert Panel
Regulatory Consensus
Only speaking for one company, we are unlikely to implement a formal BR tool unless requested to do
so by a regulatory agency
Guidance clarifications (needed)
A single ICH guidance in this area may not fit all - for example, our product is approved in Japan for 7
years, but studies accepted there not accepted by US or EMA.
Even though we have the expertise, the company is waiting for a regulatory agency to require BRA and
provide the framework.
A harmonized "ICH" process for assessment of benefit risk in terms of a quantitative process would be
very useful to standardize across regions
Framework Knowledge and Use
Quantitating a benefit-risk seems to be an exercise in forcing a required standard format onto an
individualized and highly variable assessment. Benefit can be quantified but it means something
different to every party involved. Risk is also valued differently depending on the party involved.
This is not a tool I have a lot of experience with!
I have used structured products for BR while in large companies however I am currently in a start-up
and my experience is there is neither time nor money to spend exploring BR tools that are available
BR needs to be better understood and propagated throughout the industry. It can help all involved with
uniform and sound decisions.
Validation and reproducibility will be key to broad acceptance of a structured framework for B-R
assessment. A core framework can certainly be applied as an overarching structure; however, it will be
important to nest therapeutic-area-specific elements for use, as appropriate. One-size-fits-all will not
suffice.
The amount of investment required to investigate different frameworks will be a massive task requiring
widespread system change

138

DISCUSSION
5.1 Benefit-Risk Frameworks and their Implementation by Industry
It has been clear from the literature reviewed in Chapter 2 that many benefits-risk
methodologies and frameworks exist and that considerable effort has been expended to
consider a role for these frameworks in improving decisions about drug development and
approval. Prior work by Leong and the Center for Innovation in Regulatory Science
(CIRS) surveyed mainly large sized companies on the need for and attributes of a
“universal” benefit-risk assessment framework (Leong et al., 2013). The goal of the
present study was to expand and deepen that understanding, by exploring 1) the
approaches and methodologies used to evaluate benefit-risk by companies of small as
well as large size during the development of medical products, and 2) the extent to which
those methodologies are used to support the companies evaluations of benefit-risk both
during development and when submissions are made to regulatory agencies for marketing
approval. From this exploration, I wanted to estimate the maturity of BR implementation
across the industry as a whole and to identify how the outcomes of benefits-risk analysis
are used across the lifecycle of drug development including use for internal or external
communications.

139

5.2 Methodological Considerations
5.2.1 Limitations
The investigations here relied on survey methods to gain information about views and
practices of industry from respondents in particular job categories. This type of
methodology has several potential limitations that were recognized even before the
surveys were launched. These included issues concerned with: 1) representativeness and
sampling of respondents, 2) validity and reliability of the survey methodology and
3) appropriateness of this specific survey instrument.
5.2.1.1 Sampling of Respondents
It is well known that surveys can introduce selection bias if care is not taken in the
methods used to select respondents (Sadler, Lee, Seung-Hwan Lim, & Fullerton, 2010).
In this survey respondents were chosen to participate because their roles and
responsibilities would be likely to experience with BR assessments and decision-making.
The number of such experienced individuals is relatively small, but I felt that this defined
sample was critical to provide as much insight as possible in this specialized topic. As
noted by Hesse-Bieber and Leavy (Hesse-Biber & Leavy, 2010):
The logic of qualitative research is concerned with in-depth understanding,
usually working with small samples. Unlike quantitative research, in which
the goal is to make generalizations about the degree or extent of a problem or
a set of patterns, qualitative research aims to look at a “process” or the
“meanings” individuals attribute to their given situation.
However small samples of this type do not yield numbers large enough to analyze results
statistically with any confidence, especially when trying to compare subpopulations.

140

No methods for statistical inference exist to quantitate evaluations in the same way as
might be used in randomized, interventional trials so that more judgment is needed to
decide when the collected data is saturating to the point that little additional insight would
be gathered by adding more respondents (Sue & Ritter, 2007).
The sample was also shaped by the need to identify respondents from companies of
different sizes. The Expert Panel (EP) thus would be characterized as a “purposive
sample” of participants who were then approached deliberately and who “opted in” and
agreed to participate. The goals of using such a method were two-fold. First, the
sampling method assured diversity in the population of respondents based on company
size, an objective that appeared to be achieved by the range of company sizes shown in
Figure 15. Second it assured that individuals were more likely to answer the survey
(Hesse-Biber & Leavy, 2010).
5.2.1.2 Survey Methodology
A serious problem related to surveys in general is the relatively low response rate,
especially to surveys that are distributed electronically and are easily deleted. This
challenge was clearly in evidence when the TOPRA members were surveyed by sending
an anonymous link via email to all 3500 registered members and only receiving
16 complete responses. A much better response rate in the EP survey appeared related to
the fact that respondents had “bought into” the importance of responding to the survey.
Such a difference is to be expected. “Nonresponse” is generally lower in pre-recruited
panels than in other samples because the members have agreed to participate. Further, if
the identity of the panel members is known, even if their responses are anonymized, the

141

nonresponse rate can be computed and non-respondents evaluated (Sue & Ritter, 2007).
For these reasons, most attention was paid in the data analysis and the discussion to be
presented below to observations from the EP rather than TOPRA panel, which was
primarily used as a first look at the consistency of responses between a largely American
and a largely European respondent group.
What seemed significant from a comparison of both the EP and TOPRA group was the
finding that both groups provided similar answers to many questions. Validity of results
is generally strengthened by triangulation, in which the research question is approached
from different angles to evaluate areas of convergence in research findings (Jick, 1979).
When considered in a broader context including previous literature, triangulation is
apparent not only in the ability to compare two subpopulations (EP vs TOPRA) with the
same survey, but in the ability to compare the findings of different researchers (CIRS
2011 survey vs Current 2015 survey).
Perhaps the greatest concern in a survey like this is deciding whether sufficient
respondents have been recruited to ensure a balanced and fair analysis. For example,
cross-tabulations of responses based on company size were based on relatively small
numbers in each group, and questions must be asked about whether this small sample can
provide a fair representation of the industry as a whole. However, the generally long
(>11 year) experience of survey participants, 80% of whom had direct involvement in BR
assessments, suggests that the results are likely to be as relevant as possible given the
exploratory nature of the study. As Orcher notes, “As a general rule, obtaining a
representative sample is more important than using a large sample.” (Orcher, 2007).

142

In any survey, some respondents will complete only a portion of the questions before
“losing their breath” or “dropping out” (Galesic & Bosnjak, 2009). However, in this
survey, most respondents who stopped in the middle of a survey appeared to do so by
dropping out at a specific place. Specifically, 10 of the original 87 respondents, a group
constituting 11% of the individuals taking the survey, did so after answering the Yes/No
question that queried their satisfaction with their company’s BR approach for regulatory
submissions. This point preceded a series of questions that asked about the knowledge of
the respondent with regard to existing B-R frameworks, models and indices. Why those
respondents decided to stop the survey at this particular point was not clear but might be
interpreted in one of two ways. On one hand, the respondents may have felt that they had
provided “enough” information after expressing their level of satisfaction with the
system. On the other, respondents may have felt that the questions regarding their
knowledge of frameworks was an area of sensitivity. It might, for example, cause some
respondents to believe that they were not as knowledgeable as necessary to be a qualified
participant, and that continued participation would not be useful. Alternatively, the
respondents may not have felt comfortable to proceed in case it exposed their lack of
knowledge about the BR frameworks, even anonymously.
As noted above this study may have subjected its participants to “survey fatigue”, and it
is also possible that some of the stakeholders are suffering from “risk fatigue”.
5.2.1.3 Survey Instrument
A third potential limitation was related to the ability of the survey instrument to probe the
research issues fully. Two different approaches were used to assist in developing a

143

survey that we felt to have adequate internal validity to capture the information that we
were seeking. First, an implementation framework developed by the National
Implementation Research Network (NIRN) was adopted to guide the development of the
questions, so that the survey would respect the fact that implementation has different
phases and key drivers that should be captured (Bertram, Blase, & Fixsen, 2015).
Second, a focus group was used to critique the survey questions. The use of such a group
has been previously seen to add great value to the survey by rephrasing questions that are
ambiguous, for example in this survey questions about company size and agency
feedback. Nonetheless, it is always possible that the survey will fail to identify important
areas that would have added more clarity to the issues related to BR framework adoption.
In the future, follow-up interviewing methods may be useful to understand certain areas
more deeply and to gain insight into why some respondents did not complete the survey.
5.2.2 Delimitations
This research was delimited to the examination of the use of benefit-risk frameworks
primarily in the area of drug development not medical devices. Such a delimitation was
felt to be important because the frameworks used for device submissions are known to
differ from those for drugs. If the survey were to incorporate many answers from many
individuals outside of the pharmaceutical arena, results could present a blurred picture of
implementation. In this study, two individuals with expert backgrounds self-identified as
having experience in the devices sector. Thus it is unlikely that the results are blurred to
any large extent by heterogeneity in the respondent pool. As a consequence, however,

144

caution must be used in trying to generalize these results to other sectors of the medical
product industry, such as devices.
This survey did not extend its evaluations to other aspects of risk or benefit activities that
might take place beyond development, for example, in the post-market setting. And it
focused primarily on the use of “well known” frameworks including those used by the
US and EU regulatory authorities/groups (e.g., FDA and EMA) as well as those which
were the subject of recent literature and public presentations addressing regulatory
approvals, but these do not represent all of the methodologies available. Nonetheless such
a restriction seems reasonable, particularly because the level of knowledge and level of
implementation of even the most well-established BR frameworks presented here was
surprisingly low.
Second, the research was delimited to industry sponsors, specifically to regulatory
professionals. It did not solicit the views of regulators or clinical statisticians. This may
have affected the analysis since many of the frameworks and indices used in BR
Assessment are derived from clinical data and statistical methodologies. Regulatory
professionals were selected as recipients in part because of the cross functional nature of
their positions and the similarly cross functional nature of constructing a Benefits-Risk
Assessment for registration purposes. However, a follow-up survey could be conducted
to assess whether implementation of these frameworks and models is viewed similarly by
professionals in other disciplines.

145

Finally, this research was delimited by the number of questions that were presented. The
decision to restrict the survey to 30 questions was based on feedback from the focus
group that busy professionals would be reluctant to complete a long survey. Their views
are echoed by repeated observations in methodological literature that longer surveys tend
to reduce participant retention. Even with only 30 questions, the fact that several
respondents chose to terminate their participation in the middle of the survey suggested
the importance of keeping the survey short (but see other reasons why they may have
terminated, above). However, by limiting the number of questions, certain topics could
not explored fully and this in turn constrained the breadth of topics that could be
surveyed systematically here.
One additional item to note with regards to delimitation was the diversity of size
companies that were represented by the Expert Panel (Figure 15). Since one of the
objectives of this study – to assess implementation of BR Frameworks in the industry as
represented by all size companies, it was important to not limit the study to just “big
pharma” who are represented more frequently by trade organizations (i.e., PhRMA and
EfPIA) and at ICH. In fact, small companies participated in this survey more than large
pharma (48% vs 30%). In addition some mid-size companies participated which is
remarkable considering they are disappearing from the landscape due to mergers and
acquisitions. The mix of company sizes may or may not approximate the distribution of
company sizes in the pharmaceutical sector, but it does provide a contrast to prior work
done in this area which focused almost solely on the experience of large companies.

146

5.3 Consideration of Results
5.3.1 BR Maturity and Implementation
A central goal of the present research was to understand the state of implementation of
BR frameworks and to answer the question “Are frameworks used differently for
development versus registration?” However, it appears insufficient to evaluate this
question as though BR assessments have a singular purpose. In fact, BR frameworks
have been recommended as useful not only for registration purposes, to assist regulatory
agencies in structuring their decisions (Eichler, 2008), but also for the purposes of
supporting drug development (Breckenridge, 2012). These two uses may have different
levels of maturity. The differentiation was also felt to be important because many
products do not reach the stage at which a marketing application is submitted, especially
at smaller companies.
5.3.1.1 During Development
Central to drug development is the assessment of BR in order to justify the continued
expenditure of resources on particular products in a product portfolio. Liberti
recommended that benefit-risk criteria should be evaluated at all stage-gate decisions in
the pharmaceutical development lifecycle starting at the time of the first human trials
where benefit-risk is assessed based on preclinical data and mechanism of action (Liberti
L, 2010). The Biotechnology Industry Organization (BIO), in its 2013 comments on the
FDA five-year plan, also suggested that “the framework could be used as a tool in the
same way that the Target Product Profile is used…and would serve as a commonly

147

accepted decision-support tool” (BIO, 2013). Its later white paper reinforced this view on
the usefulness of such an approach “since the (BR) framework establishes the content
(and format) for discussing benefit and risk and it could be helpful for sponsors to adopt
them early in the development process and if used consistently align with regulators
(specifically the US FDA) over the course of a program”(BIO, 2015)).
Results regarding the use of BR frameworks of this study were, however, mixed. The fact
that one third of respondents did not appear to adopt a formalized approach to BR during
development may suggest that a large segment of the industry has not yet embraced the
use of BR frameworks for this purpose. That nearly 40% of small companies lack such a
framework is perhaps not surprising considering that implementation of these programs
will cost time and money that are in very short supply in small companies. More
surprising perhaps was the finding that almost 20% of large companies, which
presumably have more experience, more resources and more products reported that they
did not use a formalized framework. In part this may reflect constraints on resources even
in large companies. However, resistance to use BR frameworks also appear to be
associated with views that available frameworks are relatively immature and have not yet
proven their value. As pointed out by Sirkin and others, there is a “hard side” to change
management and it can be much more challenging to introduce and manage new
processes, so larger companies may be tempted to wait for the regulators to decide on a
specific requirement for the use of a particular format or content (Sirkin, Keenan, &
Jackson, 2005). This latter point is consistent with the research done previously (Leong et

148

al., 2013) that showed over half of the companies in that survey cited “not being required
for current processes” as a reason for not using a more formal and quantitative approach.
When companies, big or small, did use a BR framework as part of development activities,
they most commonly adopted a semi-quantitative approach. It is not clear whether this
choice was made because multiple options were weighed systematically and a purely
qualitative or quantitative approach was then rejected, or whether a semi quantitative
framework was chosen simply because it seemed simpler to understand and implement.
Certainly, amongst both the EP and TOPRA group, most individuals who were familiar
with frameworks seemed to have a relatively limited knowledge of more than the
framework recommended by the regulatory agency in their primary jurisdiction. This
result may suggest that companies tended to study and to adopt a framework used by the
regulators with whom they most commonly did business.
What seemed clear, however, was that most respondents working with BR frameworks
were satisfied with them, and those who expressed dissatisfaction with the approach of
their particular company often noted that their BR approaches lacked formality. The
further fact that more formalized approaches were used in different types of development
activities as diverse as preclinical risk threshold assessments and promotional materials
for approved products suggests that value is being derived from the use of the
frameworks. This reinforces the views expressed by others that BR methods can have
value across the lifecycle of a drug product (Breckenridge, 2011).

149

Another consideration related to the adoption of one or another framework is the
appropriateness of that same framework at other stages of the product lifecycle. Different
methods may need to be employed for products in early development where the focus is
on collecting data to inform the internal development team, than those most valuable
when seeking market approval or managing later commercial distribution, for example to
assess risks and benefits that become apparent as part of pharmacovigilance or
post-marketing trials (Liberti, 2010). Any framework chosen as an “industry standard”
would need to accommodate the lifecycle of a product, including its use in the
post-marketing setting.
5.3.1.2 During Submissions
Diversity was seen in the extent to which different frameworks were used when preparing
regulatory submissions as in assessing products in development. When questioned about
their approaches respondents were not given the option of selecting “no approach”
because BR assessments of some type are required in both US and EU marketing
applications (21CFR314.50, 1988, EMA Notice to Applicants, 2001). This requirement
is further underlined by the pre-specified content fields of the Common Technical
Document (CTD), the format that is normally used by regulatory specialists to organize
the marketing application in both constituencies, which suggests the placement of BR
assessments in the Clinical Overview section, Module 2.5.6. However the instructions
given for preparing a submission in CTD format does not call out a particular type of
framework for that assessment. Instead it gives the unstructured recommendation that
“(T) he purpose of this section is to integrate all of the conclusions reached in the

150

previous sections…” (ICH, 2002). Given this lack of guidance it seems predictable that
many companies will choose to employ the simplest approach possible, in which the
benefits of the product are reported as conclusions regarding efficacy extracted from the
more detailed summaries from other modules of the dossier, and the risks as safety
conclusions from that same set of summary material.
When more structured frameworks were reported, semi-qualitative approaches appeared
to be favored, and often took the form of a qualitative framework supported by
quantitative/statistical data. This finding may be of some concern to those who might
advocate more systematic and analytical approaches such as those that have been
promoted extensively by some academicians and have been presented repeatedly at
professional meetings (Frey, 2012; Mullin, 2014). Nevertheless, the fact that most of the
respondents who use simpler approaches appeared to be satisfied with them suggests that
such approaches may continue to be used, barring new regulatory requirements or strong
incentives of some kind. Further, the text responses of respondents also identified
specific reasons why some conservative companies adopt conservative approaches,
including concerns about the suitability and reliability of the methods that have variously
been proposed and about the way that different BR assessment methods will be accepted
and used by regulators. These concerns are unsurprising. Companies often need
regulatory input to determine the appropriateness of the methods that they adopt; when
the regulators disclose their process and provide specific guidance industry will generally
comply with those suggestions. Thus, it may not be helpful that the two primary agencies

151

to which most drugs will be presented for registration, the US FDA and the EMA, have
not yet harmonized those approaches and expectations. As stated by Huckle:
Despite across-the-board agreement of the advantages of an appropriate,
standard method for assessing B-R, currently, there is no widely agreed
methodology to document how evidence, uncertainty and judgement result in
a specific regulatory decision. None of the main regulatory authorities (EU,
US and Japan) have issued definitive lists, “score-card systems”, or other
forms of calibrated B-R criteria. As a result, methodology is being developed
to facilitate a shared understanding, consistency and communication around
the decision-making process for the benefit of patients, healthcare
professionals and pharmaceutical companies. (Huckle, 2015)
5.3.2 BR Frameworks as a Communication Tool
When presented with some of the possible advantages of using benefit-risk frameworks
respondents saw the potential value of such frameworks for both internal and external
communications. Participants acknowledged that the frameworks do confer certain
advantages with respect to internal consistency and transparency but less with external
communication to patients and prescribers or as part of reimbursement strategy. This
may help to explain why these groups have not been engaged in any benefit-risk process
and it will be interesting to see if this changes now that both the US FDA and EMA have
patient-centric drug development as part of their goals for the future.
5.4 Maturity of Framework Implementation
The results presented here draw a picture of a Benefit Risk system that is still in evolution
when measured against the NIRN’s description of elements that characterize different
stages of implementation maturity. Before even initiating implementation, NIRN
identifies a preliminary state of implementation readiness in which proposed

152

“interventions” should be usable, i.e., well defined and functional. This initial state of
readiness seems already to be present as judged from the volume of literature on BR
frameworks. However, the process of implementation must then proceed, and in this set
of activities industry appears to be less engaged. From present results many (33%)
companies appear to be in the “pre-exploration” phase of implementation, in that they are
still unfamiliar with the available BR approaches being used by regulators and other
companies. In order to advance to the exploration stage of implementation these
companies would have to show evidence of gathering requirements, identifying barriers
and involving stakeholders. For the majority of those surveyed, this has not occurred to
any great extent.
Once a decision to implement has been made, the installation phase can commence. Its
activities include resource allocation (human and financial) and the development of
competency in the selected program. Only a modest proportion of companies, mostly
large ones, seem to be this far along in the process. Even fewer organizations are in
active implementation, either at the initial or full stages, where decision support systems
are in place or sustainable outcomes are being achieved. Such a level of maturity would
be apparent from the answers to such questions on framework and indices knowledge.
5.4.1 Challenges Related to Knowledge of Frameworks
Imperfect implementation might also be related to issues at the level of “implementation
drivers”. The National Implementation Research Network (NIRN) has identified key
drivers to include competency, leadership and organization. One of the main drivers of
great interest in this study was that associated with competencies, that seem still to be

153

underdeveloped, both with respect to the level of knowledge regarding BR frameworks
and the expertise to execute the methods appropriate for implementation. Results
suggested that the respondents in this study, who mostly were at relatively senior levels
of their companies were familiar with the need for BR assessment and this observation
was not surprising given that BR assessment is a strategic underpinning of drug
development and regulatory approval (Walker, McAuslane, Liberti, & Salek, 2009).
However, the more detailed understanding of frameworks by those responsible for
making decisions may still be a factor limiting the effective implementation of BR
evaluations. A particularly telling question was that in which participants responded to
questions about the implementation of specific frameworks by choosing the option “not
aware of this framework” (Table 10). Two possible explanations may account at least in
part for this lack of knowledge. First, a lack of awareness might be anticipated if the
company has only products at early stages of development, when formal BR assessment
is difficult to conduct. Alternatively, this evaluation may have been carried out by
another technical discipline within the company with minimal regulatory involvement.
This latter situation seems unlikely given the centrality of BR assessments for regulatory
activities.
More likely, the low level of BR “literacy” may reflect the relative newness of BR
approaches, and the modest number of educational resources for individuals who are not
part of the relatively small group of BR advocates and researchers amongst whom most
of the discussions and activities related to BR frameworks has taken place. If the wider
set of professionals, such as the respondents sampled here, are to move beyond a level

154

somewhere between “not knowledgeable” and “extremely knowledgeable”, even for the
most familiar framework (the FDA BRF), more effective educational efforts may be
needed. Similarly, a relatively low level of knowledge and usage of the many available
numerical models and indices, indicated in Table 13, would reinforce the need for better
education directed at non-experts. Up until now, BR frameworks have been used by
regulators as an organizing tool for their own internal reviews (IoM, 2007). Thus it is
perhaps not surprising that half of the respondents have never been given feedback from
regulators regarding their approaches to BR. If regulatory agencies included discussions
of BR approaches during their meetings with industry, such feedback might motivate
more companies to explore the use of BR approaches more systematically.
5.5 Comparison with Prior Research
One of the objectives of this research was to extend the work initiated by CIRS (Leong et
al., 2013), in order to determine if progress has been made within the industry from the
time of their initial 2011 survey regarding implementation of BR Frameworks to the
present, nearly 5 years after that was conducted. Areas of particular interest to compare
between the two surveys were 1) the use of frameworks for BR assessment during
development and submission review, and 2) barriers seen to full implementation of BR
assessment frameworks. However, the present study reports on a larger number and
diversity of companies (80+ versus 20) than the initial exploratory survey of 2011 and
this may have an impact on the findings. In the 2011 survey, participating companies all
employed 2000 employees or more and would therefore be classed as “large” companies
by the standards used in the present survey. The differences in makeup of the respondent

155

pools from the previous to present survey warranted a subpopulation analysis by
company size in order to see if differences between the two surveys related primarily to
company size. Cross tabulated results to support this discussion are included as source
information in Appendix D.
5.5.1 Use of Frameworks
Perhaps the strongest difference between the 2011 and 2015 survey was the larger
number of companies that reported using no formalized framework. In this study, the
finding that nearly one-third of respondents appeared to use no formalized BR approach
was inconsistent with the findings in the 2011 survey that only one company of 19 used
no framework. It would be tempting to suggest that the difference might be due to
participation of small companies, 40% of whom had no formal approach. However,
when only large companies were sub stratified so that a more direct comparison could be
made, a grouping of respondents continued to report that their companies had no formal
approach for assessing BR for submissions (19%, 5/26). When companies had a
formalized approach, however, the frameworks that they used were similar from one
survey to the other. Similarities were present in the types of frameworks selected for use
by those companies that used a formalized approach. In the original CIRS survey,
companies were divided quite evenly between using qualitative approaches (53%, 10/19)
and semi-quantitative approaches (47%, 9/19) when making milestone decisions to take a
product further along a registration path. Preferences of respondents who used BR
frameworks in the present study were also split, though they were skewed somewhat
more toward the use of semi-quantitative rather than qualitative approaches both during

156

development and for submission purposes (Figure 16 and Figure 21). In addition, a few
companies reporting used a fully quantitative approach (6%, 5/87), a result not seen in
CIRS survey responses. These differences are relatively modest and would deserve more
in-depth examination, but may suggest a trend toward the wider adoption of
methodologies with numbers as well as text.
At the time of the 2011 CIRS survey the conceptual thinking around BR frameworks was
at a relatively early stage. The US FDA had not published its BRF, the PhRMA BRAT
was only at the pilot stage, the UMBRA was being consolidated and the EMA
PrOACT-URL was being studied as part of IMI Work Package #2 but was not yet in use
by EMA.
The 2011 survey also confined its evaluation to the models identified in the IMI Work
Package #2 which included a group of 17 tools. The present study utilized the subset of
these tools that were advanced to the next level of study by the IMI project, and included
them in the 2015 study so that a comparison could be made with the original questions
regarding usage patterns, as shown in Table 22.

157

Table 22: Comparison of Usage Patterns of Six Models/Indices between 2011
and 2015
Data from 2011 from Leong and colleagues (Leong et al., 2013)
Model/Indices Companies Using
in 2011 (%)
ALL EP
companies using
in 2015 (%)
LARGE EP
companies using
in 2015 (%)
NNT/NNH 44 13 39
Decision Tree(s) 53 15 25
Incremental Net Benefit (INB) 28 9 18
Quality Adjusted Life Years
(QALY)
33 18 39
Stated Preferences 22 6 24
Multi-criteria Decision Analysis
(MCDA)
12 8 25

Inspection of overall data seems to suggest indices are used less rather than more
frequently in 2015 compared to 2011. This difference is, however, moderated when only
large companies are compared. Notably the reported use of Decision/Value Tree(s) and
INB, were still much lower and QALY and MCDA were somewhat higher. With small
numbers it is difficult to be sure that the differences are meaningful. It is also important
to note that the CIRS survey was sent to participants in senior positions who were
involved in benefit-risk assessment and decision making, but the primary working roles
were not constrained to individuals in regulatory affairs but could have included clinical
and statistical experts who might be more familiar with the use of numerical indices.

158

5.5.2 Utility/Advantages and Barriers to Implementation of BR Frameworks
Despite differences in the types of frameworks used by respondents in the two surveys,
the views of respondents seemed aligned regarding the utility of BR frameworks. The
main advantages of a benefit-risk framework to which participants in both studies
“strongly agreed” were to enhance transparency (2011:65%; 2015:59%) and consistency
(2011:65%; 2015:67%) of decision making, as well as to enhance the communication
between stakeholders (2011:55%; 2015:75%). These results suggest that respondents
appreciate the value of these frameworks. Nevertheless, full implementation of BR
frameworks appears to have stalled in many companies, probably because impediments
that they identified in 2011 continue to exist today. Identified factors then and now
include: a lack of consensus about which framework to use even at the regulatory level;
resource limitations in areas that included skills, training and change management; and
concerns about the validity and ease of use of the frameworks.
5.5.3 Harmonizing Frameworks Globally
A recurring theme in this survey, like the 2011 survey before it, was the absence of a
“universal” BR framework. Leong et al (Leong et al., 2013) proposed the formation of a
committee or working group to oversee the development, implementation and promotion
of a “universal” framework. One organization in a strong position to take on this
challenge would be the International Conference on Harmonization (ICH). ICH could
issue more specific guidance regarding the way that frameworks could be used in
submission packages utilizing the Common Technical Document (CTD) format. This

159

objective was added to the business plan of the Expert Working group formed in 2014 to
revise the M4E/CTD Efficacy Guidance (ICH, 2014).
However, its work was only reported shortly after the present study closed. The revision
of the M4E Guidance on CTD attempted to enhance the format and structure of
benefit-risk information by suggesting the use of an FDA-like qualitative approach (with
inclusion of optional quantitative components) in Module 2.5.6 of a marketing
application.
The newly recommended format of the ICH M4E(R2) should not be difficult for industry
to implement, particularly because the FDA BRF approach is already the framework best
understood by respondents in the EP survey (Figure 25). Its adoption may be a little
more challenging for European companies, because the regulators in that constituency
use a more quantitative framework for their European Public Assessment Reports
(EPARs) and it was the leading factor they wanted ICH to consider (Figure 29).
If better implementation is a desirable goal, an important first step would appear to be
achieved if the ICH framework as defined in M4E (R2) were to be adopted in all
constituencies and then templates and implementation guides were to be made available
to both regulators and industry. In its current draft format, the ICH guidance gives only a
general framework without recommendations about specific BR assessment methods.
While this allows each jurisdiction the flexibility to meet its own legislative requirements
and historically derived preferences, it does not provide a harmonized approach amenable
to a multi-purpose template and implementation guide. When such documents are

160

created utilizing such resources they often can be found “fit for use” (Leong Wai Yeen et
al., 2014).
5.6 Conclusions and Future Directions
If formal BR assessments are neither required as part of a registration package, nor
encouraged or shaped by feedback from the regulators, industry may see little incentive
to implement them. Further, if untrained personnel cannot apply such frameworks
effectively, the benefits will not be apparent and senior management will not be
motivated to support the change or allocate the required resources. The ability to prepare
regulatory personnel adequately is not helped by confusion over which framework to use
in the absence of a harmonized framework adopted universally by regulators. Confusion
may be exacerbated by the proliferation of frameworks about which companies are
challenged to stay current and knowledgeable. Some frameworks and models are
sufficiently complex and statistically driven that they are beyond the easy use of
non-experts. It is clear from this survey that those types of frameworks are peripheral to
the approaches of most companies and are unlikely to be favored as a recommended best
practice.
Efforts to encourage the better uptake of BR frameworks could take lessons from the way
in which the Common Technical Document was promoted in the early 2000s, and more
recently the way in which the US Risk Evaluation and Mitigation Strategies (REMS) and
the EU Risk Management Plans (RMP) have been introduced. These programs involved
strong involvement of regulatory agencies to champion the program at professional

161

meetings and through formally stated requirements. Not surprisingly, these governmental
efforts were then followed by numerous educational programs and offerings that can now
be recognized in the core curricula of most academic institutions (USC, Rutgers and
Temple Universities) and professional societies (DIA, RAPS, and TOPRA). As part of
this outreach, it would be important to leverage the experience that already exists within
the industry in order to provide small companies with low cost opportunities for practical
training in venues that would allow professionals across the spectrum of drug
development and post-market activities to participate.
In addition, those who create and maintain CTD templates must update the content based
on the ICH BR guidance M4E (R2) when it reaches the phase of implementation by
individual countries and regions. These modifications could include a structured BR
Assessment for use in the briefing books and background packages that are currently
provided when companies have different forms of advisory meetings with regulators.
This would encourage dialogue with regulators at earlier stages of development, a
position that is endorsed by BIO and others (BIO, 2015; Stuart Walker et al., 2015 and
2011, Colopy, 2015). The companies creating these templates can be important adjuncts
to train company personnel on the use and value of BR frameworks.
The genesis of BR frameworks and methodologies is rooted in the transparency
initiatives of the early 2000s. As efforts to use these frameworks go forward, it would
seem important to evaluate how these efforts are being received by the audiences for
which they were intended– namely the healthcare sector at large. As shown in this study,
few, if any, companies are utilizing BR assessments for communication with either

162

prescribers or patients. Transparency does not necessarily translate to understanding.
One area of future research maybe to examine whether and how these frameworks can
aid in the education of the wider stakeholder audience about drug risks and benefits. This
effort is particularly important if those individuals are typically exposed to more risk than
benefit information, so that a balanced picture can be presented which is a shared
responsibility of both regulators and industry.

163

REFERENCES

Agrawal, S., Brennan, N., & Budetti, P. (2013). The Sunshine Act — Effects on
Physicians. New England Journal of Medicine, 368(22), 2054-2057.
Aikin, K. J., O'Donoghue, A. C., Swasy, J. L., & Sullivan, H. W. (2011). Randomized
Trial of Risk Information Formats in Direct-to-Consumer Prescription Drug
Advertisements. Medical Decision Making.
Ashby, D., & Smith, A. F. M. (2000). Evidence-based medicine as Bayesian decision-
making. Stat Med, 19(23), 3291-3305.
Avorn, J. M. D. (2012). Two Centuries of Assessing Drug Risks. The New England
journal of medicine, 367(3), 193-197.
(BIO), B. I. O. (2013). Comments on the FDA Draft Structured Benefit-Risk 5-Year Plan.
(BIO), B. I. O. (2015). A Lifecycle Approach to FDA's Structured Benefit-Risk
Assessment Framework.
Baglin, T. (2009). Communicating benefit and risk. British Journal of Haematology,
146(1), 31-33.
Balian, J. D., Wherry, J. C., Malhotra, R., & Perentesis, V. (2010). Roadmap to risk
evaluation and mitigation strategies (REMS) success. Therapeutic Advances in
Drug Safety, 1(1), 21-38.
Bertram, R. (2013). Improving Programs and Outcomes: Implementation Frameworks.
Paper presented at the Bridging the Research and Practice Gap, Houston TX.
Bertram, R. M., Blase, K. A., & Fixsen, D. L. (2015). Improving Programs and
Outcomes: Implementation Frameworks and Organization Change. Research on
Social Work Practice, 25(4), 477-487.
Brook, R. H., & Lohr, K. N. (1985). Efficacy, Effectiveness, Variations, and Quality:
Boundary-Crossing Research. Medical Care, 23(5), 710-722.
Chuang-Stein, C. (1994). A new proposal for benefit-less-risk analysis in clinical trials.
Control Clin Trials, 15(1), 30-43.
Chuang-Stein, C., Mohberg, N. R., & Sinkula, M. S. (1991). Three measures for
simultaneously evaluating benefits and risks using categorical data from clinical
trials. Stat Med, 10(9), 1349-1359.
CIOMS. (1998). Benefit-Risk Balance for Marketed Drugs: Evaluating Safety Signals.
Code of Federal Regulations, 21CFR312.23 (1987).
Code of Federal Regulations, 21CFR50.25 (2012).
Coplan, P. M., Noel, R. A., Levitan, B. S., Ferguson, J., & Mussen, F. (2011a).
Development of a framework for enhancing the transparency, reproducibility and
communication of the benefit-risk balance of medicines. Clinical Pharmacology
& Therapeutics, 89(2), 312-315.
Coplan, P. M., Noel, R. A., Levitan, B. S., Ferguson, J., & Mussen, F. (2011b).
Development of a Framework for Enhancing the Transparency, Reproducibility
and Communication of the Benefit–Risk Balance of Medicines. Clinical
Pharmacology & Therapeutics, 89(2), 312-315.
Eichler, H.-G., Abadie, E., Raine, J. M., & Salmonson, T. (2009). Safe drugs and the cost
of good intentions. The New England journal of medicine, 360(14), 1378-1380.

164

EMA. (2007). Report of the CHMP Working Group on Benefit-Risk Assessment Models
and Methods.
EMA. (2008). Reflection paper on benefit-risk assessment methods in the context of the
evaluation of marketing authorization applications of medical products for human
use. London.
EMA. (2009). Benefit-Risk Methodology Project. London.
EMA. (2014). Update on work package 5: Effect Table pilot (Phase I).
FDA. (1999). Managing the Risks from Medical Product Use.
FDA. (2009). Draft Guidance for Industry: Format and Content of Proposed Risk
Evaluation and Mitigation Strategies (REMS), REMS Assessments, and Proposed
REMS Modifications.
FDA. (2013a). Guidance for Industry; Labeling for Human Prescription Drug and
Biological Products – Implementing the PLR Content and Format Requirements.
FDA. (2013b). Structured Approach to Benefit-Risk Assessment in Drug Regulatory
Decision-Making.
FDA. (2015). Draft Guidance for Industry: Brief Summary and Adequate Directions for
Use: Disclosing Risk Information in Consumer-Directed Print Advertisements
and Promotional Labeling for Human Prescription Drugs.
Felli, J. C., Noel, R. A., & Cavazzoni, P. A. (2009). A multiattribute model for evaluating
the benefit-risk profiles of treatment alternatives. Med Decis Making, 29(1), 104-
115.
Fischhoff, B. (2011). Communicating Risks and Benefits: An Evidence-Based User's
Guide.
FOIA Update Vol. XVII, No. 4. (1996).
Frey, P. (2012). Benefit Risk Considerations in CDER: Development of a Qualitative
Framework. Paper presented at the DIA Annual Meeting.
Galesic, M., & Bosnjak, M. (2009). Effects of Questionnaire Length on Participation and
Indicators of Response Quality in a Web Survey. Public Opinion Quarterly,
73(2), 349-360.
Giliberti, F. J. (2003). The Learned Intermediary Doctrine. Marketing Management,
12(3), 53-55.
Guo, J. J., Pandey, S., Doyle, J., Bian, B., Lis, Y., & Raisch, D. W. (2010). A Review of
Quantitative Risk–Benefit Methodologies for Assessing Drug Safety and
Efficacy—Report of the ISPOR Risk–Benefit Management Working Group.
Value in Health, 13(5), 657-666.
Hesse-Biber, S. N., & Leavy, P. (2010). The practice of qualitative research: Sage.
Huckle, R. (2015). Challenges in benefit-risk assessment of orphan drugs. Regulatory
Rapporteur, 12(2), 6.
Hudson, K. L., & Collins, F. S. (2015). SHaring and reporting the results of clinical trials.
JAMA, 313(4), 355-356.
ICH. (1996). GUIDELINE FOR GOOD CLINICAL PRACTICE. INTERNATIONAL
CONFERENCE ON HARMONISATION OF TECHNICAL REQUIREMENTS
FOR REGISTRATION OF PHARMACEUTICALS FOR HUMAN USE.
ICH. (2002). THE COMMON TECHNICAL DOCUMENT FOR THE REGISTRATION
OF PHARMACEUTICALS FOR HUMAN USE: EFFICACY – M4E(R1).

165

ICH. (2005). Quality Risk Management Q9.
ICH. (2014). M4E(R2) Final Concept Paper.
Institute of Medicine (U.S.). Committee on Ethical and Scientific Issues in Studying the
Safety of Approved Drugs. (2012). Ethical and scientific issues in studying the
safety of approved drugs. Washington, D.C.: National Academies Press.
IoM. (2006). The Future of Drug Safety: Promoting and Protecting the Health of the
Public.
IoM. (2007). Understanding the Benefits and Risks of Pharmaceuticals: Workshop
Summary. (9780309107389). The National Academies Press.
IoM. (2014). Characterizing and Communicating Uncertainty in the Assessment of
Benefits and Risks of Pharmaceutical Products.
ISO. (2007). Application of Risk Management to Medical Devices Standard 14971.
Jick, T. D. (1979). Mixing Qualitative and Quantitative Methods: Triangulation in
Action. Administrative Science Quarterly, 24(4), 602-611.
Juhn, P., Phillips, A., & Buto, K. (2007). Balancing Modern Medical Benefits And Risks.
Health Affairs, 26(3), 647-652.
Leong, J., McAuslane, N., Walker, S., & Salek, S. (2013). Is there a need for a universal
benefit-risk assessment framework for medicines? Regulatory and industry
perspectives. Pharmacoepidemiol Drug Saf.
Leong Wai Yeen, J., Salek, S., & Walker, S. (2014). Strategy for communicating benefit-
risk decisions: a comparison of regulatory agencies' publicly available documents.
Frontiers in Pharmacology, 5, 269.
Levitan, B. (2011). A concise display of multiple end points for benefit-risk assessment.
Clinical Pharmacology & Therapeutics, 89(1), 56-59.
Levitan, B. S., Andrews, E. B., Gilsenan, A., Ferguson, J., Noel, R. A., Coplan, P. M., &
Mussen, F. (2011). Application of the BRAT Framework to Case Studies:
Observations and Insights. Clinical Pharmacology & Therapeutics, 89(2), 217-
224.
Liberti, L., McAuslane, N., Patel, P., Breckenridge, A., Eichler, H. G., & Peterson, R.
(2013). Regulatory Review: How Do Agencies Ensure the Quality of Decision
Making? Clinical Pharmacology & Therapeutics, 94(3), 305-308.
Liberti L, M. N., Walker S. (2010). Progress on the Development of a Benefit/Risk
Framework for Evaluating Medicines. Reg Focus (RAPS).
Luteijn, J. M., White, B. C., Gunnlaugsdottir, H., Holm, F., Kalogeras, N., Leino, O., . . .
Verhagen, H. (2012). State of the art in benefit-risk analysis: medicines. Food
Chem Toxicol, 50(1), 26-32.
Misbranded drugs and devices, (2006).
Mt-Isa, S. (2013). Review of visualisation methods for the representation of benefit-risk
assessment of medication: Stage 2 of 2.
Mt-Isa, S., Hallgreen, C. E., Wang, N., Callréus, T., Genov, G., Hirsch, I., . . . On behalf
of the, I. M. I. P. b. r. p. (2014). Balancing benefit and risk of medicines: a
systematic review and classification of available methodologies.
Pharmacoepidemiol Drug Saf, 23(7), 667-678.

166

Mt-Isa, S., Tzoulaki, I., Callréus, T., Micaleff, A., & Ashby, D. (2011). Weighing
benefit–risk of medicines: concepts and approaches. Drug Discovery Today:
Technologies, 8(1), e29-e35.
Mudd, P. N., Groenendaal, H., Bush, M. A., & Schmith, V. D. (2010). Probabilistic Risk
Analysis: Improving Early Drug Development Decision Making. Clin Pharmacol
Ther, 88(6), 871-875.
Mullin, T. (2014). Managing Risk and Uncertainty Through the Drug Life cycle - Recent
FDA Initiatives.
Mussen, F., Salek, S., & Walker, S. (2007). A quantitative approach to benefit-risk
assessment of medicines--part 1: the development of a new model using multi-
criteria decision analysis; part 2: the practical application of a new model.
Pharmacoepidemiol Drug Saf, 16 Suppl 1, S42-46.
Mussen, F., Salek, S., & Walker, S. (2009). Benefit-risk appraisal of medicines : a
systematic approach to decision-making. Chichester, UK ; Hoboken, NJ: Wiley-
Blackwell.
Nicholson, S. C., Peterson, J., & Yektashenas, B. (2012). Risk Evaluation and Mitigation
Strategies (REMS). Drug Safety, 35(2), 91-104.
Noel, R. M. D., Hermann, R. M. D. M. P. H., Levitan, B. M. D. P., Watson, D. J. M. P.,
& Van Goor, K. P. R. A. C. (2012). Application of the Benefit-Risk Action Team
(BRAT) Framework in Pharmaceutical R&D: Results From a Pilot Program.
Drug Information Journal, 46(6), 736-743.
Obama, P. B. (2009). Memorandum to the Heads of Agencies on Transparency and Open
Government.
Oliver, R. W. (2004). What is Transparency? New York, N.Y. : The McGraw-Hill
Companies, Inc.
Orcher, L. T. (2007). Conducting A Survey: Techniques for a Term Project: Pyrczak
Publishing.
Phillips, L. (2013). Quantitative benefit-risk assessment: where are we now, why the
resistance and where are we going? Paper presented at the DIA Annual Meeting.
Phillips, L. (2014). What simple methods could be used to elicit patients views on
benefits, harms and relative importance? Paper presented at the The Assessment
of Benefits and Harms and Their Relative Importance for Patients, Industry and
Agnecies: How should they be captured?, Surrey UK.
Physician Payments Sunshine Act of 2009, Library of Congress (2009).
Puhan, M., Singh, S., Weiss, C., Varadhan, R., & Boyd, C. (2012). A framework for
organizing and selecting quantitative approaches for benefit-harm assessment.
BMC Medical Research Methodology, 12(1), 173.
R2, I. E. W. G. M. E. (2015). Final Concept Paper M4E(R2): Enhancing the Format and
Structure of Benefit-Risk Information in ICH.
Raiffa, H. (1961). Applied Statistical Decision Theory. Harvard University.
Ross, J. S., Gross, C. P., & Krumholz, H. M. (2012). Promoting Transparency in
Pharmaceutical Industry–Sponsored Research. American Journal of Public
Health, 102(1), 72-80.

167

Sadler, G. R., Lee, H.-C., Seung-Hwan Lim, R., & Fullerton, J. (2010). Recruiting hard-
to-reach United States population sub-groups via adaptations of snowball
sampling strategy. Nursing & health sciences, 12(3), 369-374.
Sashegyi, A., Felli, J., & Noel, R. (2014). Benefit-risk assessment in pharmaceutical
research and development: edited by Andreas Sashegyi, James Felli, Rebecca
Noel. Boca Raton, Florida: CRC Press/Taylor & Francis Group.
Schmidt, C. (2014). Stepping into the sunshine. Nat Biotech, 32(9), 859-862.
Schwartz, L., & Woloshin, S. (2009). Lost in Transmission -- FDA Drug Information
That Never Reaches Clinicians. The New England journal of medicine, 361(18),
1717-1720.
Schwartz, L., & Woloshin, S. (2013). The Drug Facts Box: Improving the
communication of prescription drug information. Proceedings of the National
Academy of Sciences, 110(Supplement 3), 14069-14074.
Schwartz, L., Woloshin, S., & Welch, H. G. (2009). Using a Drug Facts Box to
Communicate Drug Benefits and HarmsTwo Randomized Trials. Annals of
Internal Medicine, 150(8), 516-527.
Sirkin, H. L., Keenan, P., & Jackson, A. (2005). The hard side of change management
(Vol. 83, pp. 108). United States: Harvard Business School Press.
Sue, V. M., & Ritter, L. A. (2007). Conducting Online Surveys.
Tervonen, T., & Figueira, J. R. (2008). A survey on stochastic multicriteria acceptability
analysis methods. Journal of Multi-Criteria Decision Analysis, 15(1-2), 1-14.
Tervonen, T., & Lahdelma, R. (2007). Implementing stochastic multicriteria acceptability
analysis. European Journal of Operational Research, 178(2), 500-513.
Tervonen, T., van Valkenhoef, G., Buskens, E., Hillege, H. L., & Postmus, D. (2011). A
stochastic multicriteria model for evidence-based decision making in drug
benefit-risk analysis. Stat Med, 30(12), 1419-1428.
Tervonen, T., van Valkenhoef, G., Buskens, E., Hillege, H. L., & Postmus, D. (2011). A
stochastic multicriteria model for evidence-based decision making in drug
benefit-risk analysis. Stat Med, 30(12), 1419-1428
Veatch, R. M. (1993). Benefit/Risk Assessment: What Patients Can Know That Scientists
Cannot. Drug Information Journal, 27(4), 1021-1029.
Walker, S., McAuslane, N., Liberti, L., & Salek, S. (2009). Measuring benefit and
balancing risk: strategies for the benefit-risk assessment of new medicines in a
risk-averse environment. Clinical Pharmacology & Therapeutics, 85(3), 241-246.
Walker, S., McAuslane, N. M. P., Liberti, L. M. R. R. A. C., Leong, J. M. P., & Salek, S.
R. P. (2015). A Universal Framework for the Benefit-Risk Assessment of
Medicines: Is This the Way Forward? Therapeutic Innovation & Regulatory
Science, 49(1), 17-25.
Weaver, S. a. (1963). The Mathematical Theory of Communication: Univ. of Illinois
Press.
Woloshin, S., & Schwartz, L. M. (2011). Communicating Data About the Benefits and
Harms of TreatmentA Randomized Trial. Annals of Internal Medicine, 155(2),
Woosley, R. L. (2013). One Hundred Years of Drug Regulation: Where Do We Go from
Here? Annual Review of Pharmacology and Toxicology, 53(1), 255-273.

168

Yuan, Z., Levitan, B., & Berlin, J. A. (2011). Benefit-risk assessment: to quantify or not
to quantify, that is the question. Pharmacoepidemiol Drug Saf, 20(6), 653-656.
Zeldis, J. B., Williams, B. A., Thomas, S. D., & Elsayed, M. E. (1999). S.T.E.P.S.™: A
comprehensive program for controlling and monitoring access to thalidomide.
Clinical Therapeutics, 21(2), 319-330.

169

APPENDIX A. NUMERICAL REPRESENTATIONS
The terms metrics and indices are commonly found in the description of B/R as are the
basic terms which are included below and which are (still) suitable to decision making
and which make a veritable alphabet soup of acronyms.
A.1 Basic Terms
The following terms are part of a basic quantitative lexicon that can be used individually
or applied to the computational methods that follow. They are commonly used when
looking at single study results but are limited by their ability to be combined into single
values that can be used to compare results or alternative therapies.
Relative Risk (RR) – the ratio between the proportions of individuals
exposed to a drug who experience an adverse drug reaction (Ashby &
Smith) divided by the proportion of individuals who experienced the same
ADR but were not exposed. This value takes into account differences in
exposure that can be useful in extrapolating data.
Attributable Risk (AR) – the risk difference which is calculated using the
RR divided by the difference in the size of the two groups.
Population Attributable Risk (Mt-Isa et al.) – same as above but using
total population data.

170

Relative Risk Reduction (RRR) – the ratio between the proportions of
exposed individuals who experience a decline in ADRs divided by the
proportion of unexposed individuals who experience a benefit.
Absolute Risk Reduction (ARR) – the decline in ADRs between both
exposed and unexposed group
A.2 Indices
A.2.1 Computational Indices
NNT – Number needed to treat or the number of patients who need to be
treated to prevent one additional unfavorable event (e.g., disease). Also,
the reciprocal of ARR (Baglin, 2009)
NNH – Number needed to harm or the number of patients who need to be
treated before a patient experiences an ADR.(Baglin, 2009)
NNH/NNT – the simplest form of a B/R ratio. If NNH/NNT is >1 then
fewer patients will need to be treated to achieve benefits than will be
treated to have one (additional) occurrence of an ADR. These values do
not value either the benefit or the harm and cannot account for multiple
benefits and harms simultaneously. When this is done using patient
preferences or Relative Utility Values (RV) this deficiency can be
overcome.

171

Utility – this is the term that is borrowed from economics to mean a
measure that is that is to be maximized in any situation involving choice.
Relative Value – this metric is calculated from a numerical representation
of patients’ preferences for specific outcomes. For example: RV=1 –
utility of an AE/utility of improvement using Treatment X.
MCE – Minimum Clinical Efficacy. This is a numerical representation of
a treatment in comparison to a standard treatment, which takes into
account both the efficacy measurements of each and the adverse event
profiles of each. Relative values can be applied to MCE.
A.2.2 Health Utility Indices
QALY – Quality Adjusted Life Years. This is a numerical representation
that adjusts for the length of life and the quality of life experienced during
that time. Quality of life in this context is measured on a scale of 1, full
health to 0, death. The point on this scale is referred to as the utility. As
an example, if you experience as additional 10 years of life expectancy at
a utility value of 0.5 you would have 5 years of full health. QALY is
extremely important in pharmacoeconomic evaluations.
TWiST and Q-TWiST – Time without Symptoms and Toxicity and
Quality adjusted version of TWiST. TWiST takes time lost due to adverse
drug events (or toxicities) and subtracts it from time gained through
treatment. In order to adjust this for “quality” the relative values of the

172

therapies based on the patient’s experience are applied within the context
of clinical outcomes. This methodology is commonly used for cancer
treatments.
A.2.3 Other Useful Indices
INHB - Incremental Net Health Benefit. This differential calculation is
performed using QALYs and is a comparison of treatment against a
standard of care of both effectiveness and risk. It is easiest to understand
when written as a formula:
INHB = (E2 – E1) - (R2-R1)
MAR and SPM – The Maximum Acceptable Risk and Stated Preference
Method are two additional techniques that collect patient preferences on
probabilities of adverse events occurring and allows them to make
trade-offs between treatment options as part of discrete choice
experiments. It has been used in several high profile regulatory decisions
in therapeutic areas as varied as HIV, IBD/Crohn’s disease, Type 2
Diabetes and Multiple Sclerosis.
A.3 Visual Modeling
Some other quantitative methods that focus on the visual modeling of benefit risk
assessment and which may be very helpful in communicating results include the
Risk-Benefit Plane (RBP)/ Risk-Benefit Acceptability Threshold (RBAT), Probabilistic

173

Simulation Methods (PSM) and Monte Carlo Simulation(MCS). All of these rely on
complex statistical calculations but can be translated into basic four square diagrams with
risk on the x-axis and benefit on the y-axis that is easier to interpret.
One additional visualization method that calculates Net Clinical Benefit using Excess
Number of Events (ENE) is termed “simple” by its originators (Yuan, Levitan, & Berlin,
2011) and results in a variation of a forest plot. This graphical takes information from all
relevant end points for both safety and efficacy and integrates and orders it on one axis.
More than one treatment group can be displayed, as well as subsets of subjects, and the
use of color and mean values add to the display.
Figure A.1: Sample of Net Clinical Benefit Display

174

A.4 Organizing Approaches
Other researchers studying this topic have also noted the numerous methodologies
available to industry and regulators and have published the results of their findings in an
additional reference that was extremely useful and is noted here. Milo Puhan and a team
at John Hopkins published a research article “A framework for organizing and selecting
quantitative approaches for benefit-harm assessment” (Puhan et al., 2012). Similar to the
work done by IMI, Puhan identified and categorized 16 methods based on key
characteristics and outcomes. This framework has the potential to be used by companies
seeking to explore and implement frameworks if they have not already begun.

175

APPENDIX B. FINAL VERSION OF BR FRAMEWORK SURVEY

Appendix B

The first portion of this survey is intended to tell me about you and your organization/company.

What title best describes your current role within your organization?

Vice President- Regulatory
Director/Sr. Director - Regulatory
Manager/Sr. Manager - Regulatory
Consultant/Sr. Consultant
Other - please specify

What geographical area are you responsible for?

US only
EU only
Global
Rest of World

Other - please specify

Please indicate where your company headquarters is located.

United States/North America
Europe
Asia Pacific

Other - please specifiy

In this role do you participate in benefit-risk assessment of new or existing products?

Yes, if so please indicate how many years in the space below
No

What is the size of the company for which you are currently working based on number of employees?

Less than 250 employees
250-2000 employees
More than 2000 employees

176

My company has or works with the following prescription drug products (includes therapeutic biologics):

Only innovator prescription products in development
Only marketed prescription products
A combination of products that are on the market and in development

The next portion of the survey will focus on Benefit-Risk Assessments and how they are performed and utilized.
Both Qualitative methods, that rely on the use of expert judgement, and Quantitative measurements, based on
numerical indices, can be used. Many of these frameworks and methods are new and have not been widely
disseminated. Please answer to the best of your ability.

How would you best describe the internal approach that your company uses to assess benefit-risk(B-R) of a new
medicine during development?

Our internal approach is a purely qualitative

Our internal approach is semi quantitative and uses both qualitative and quantitative elements. Please describe briefly
below if you can

Our internal approach is a fully quantitative model

Our company has no internal formal approach for assessing benefit-risk of a new medicine during development.
Comments

Are you satisfied with your existing benefit-risk assessment approach during development?

Yes
No

If NO, please provide the reason(s) that best describe your situation (choose all that apply):

Too time consuming to utilize the approach

Requires additional training to understand and learn to use the approach
Poor acceptance by staff
Benefits of the approach are not apparent
Approach/method is not reliable
Approach/method is not generally accepted within the industry
We have no approach
Other - please describe below

177

Our company uses benefit-risk assessments for new medicines during drug development for:

All products - feel free to describe below

Only certain products - please describe how these products are selected as candidates for assessment (eg high risk or
innovative)

How would you best describe the internal framework that your company uses for assessing benefit-risk (B-R) of a
new medicine for submission to regulatory agencies as part of a marketing application?

Our framework is a purely qualitative .

Our framework is semi quantitative and uses both qualitative and quantitative elements. Please describe briefly below

Our framework is a fully quantitative
Our company has no framework for assessing benefit-risk of a new medicine for submission. We provide the required text
in the clinical overview only

Are you satisfied with your existing benefit-risk assessment framework or method for submission purposes?

Yes
No

If NO, please provide the reason(s) that best describe your situation(choose all that apply):

Too time consuming to utilize the framework

Requires additional training to understand and learn to use the framework
Poor acceptance by staff
Framework is not reliable

Benefits of the framework are not apparent
We have no framework
Framework is not generally accepted within the industry
Other - please describe below.

From the list below, choose reason(s) that best describe why a formal system is not used:

Not required for current product developmental processes
Lack of knowledge of BR frameworks in general
Lack of scientifically validated BR framework

Lack of a common BR framework among peers and/or stakeholders
Benefits of a BR framework not apparent
Other (Please specify)

178

Please indicate below why you have chosen to assess benefits-risk in this way by level of importance.

The following frameworks and methods have been proposed as ways to perform benefit-risk assessments -
please indicate those with which you personally are familiar (check all that apply).

US FDA BR
Framework EMA
PrOACT-URL
Unified/Universal Method for Benefit Risk Assessment (UMBRA)
Benefit Risk Assessment Team (BRAT) Framework
Stochastic Multi-criteria Acceptability Analysis (SMAA)
Multi-Criteria Decision Analysis (MCDA)
Ashby and Smith Framework
Value or Decision Tree
Not familiar with any of these frameworks
Other - please specify
Not at all
Important
Neither Important
Very Unimportant nor Unimportant Very Important
Extremely
Important
Align with company expertise

Align with regulator templates
Align with reimbursement
strategy
Align with product development
strategy
Other

179

If you personally are familiar with any of these frameworks or methods, please indicate your level of knowledge
on the scale below.

Not Familiar Extremely Familiar

0 1 2 3 4 5 6 7 8 9 10

US FDA BR Framework
EMA PrOACT-URL
Unified/Universal
Method for Benefit Risk
Assessment (UMBRA)
BRAT Framework
Stochastic Multi-criteria
Acceptability
Analysis(SMAA)
Multi-Criteria Decision
Analysis(MCDA)
Ashby and Smith
Framework
Value or Decision Tree

180

The following frameworks or methods have been proposed as ways to perform benefit-risk assessments - please
indicate those that your company/organization uses

The following models and indices are often used to calculate benefit-risk. Please indicate those that your
company/organization uses.

My company My company
utilizes this utilizes this My
company model/index method for BR
My company utilizes this for BR Assessment
is evaluating model/index Assessment and uses it to
My company this for BR and submits it communicate
does not use model/index Assessment with externally to
this for use within for internal use registration prescribers Not aware of
model/index our company only dossiers and patients this framework
Number Needed to Treat
(NNT)/Number Needed to Harm
(NNH)
Benefit Risk Ratio (BRR)

Incremental Net Benefit (INB)
Quality Adjusted Life Years
(QALY)
Stated Preference Method
(Patient)
Quality adjusted Time Without
Symptoms and Toxicity
(Q-TWiST)
Minimum Clinical Efficacy (MCE)
Minimal Acceptable Risk (MAR)
My company
My company utilizes this
utilizes this method for BR
My company method for BR Assessment
My company utilizes this Assessment and uses it to
has/is method for BR and submits it communicate
My company evaluated this Assessment with externally to
does not use for use within for internal use registration prescribers Not aware of
this framework our company only dossiers and patients this framework
US FDA BR Framework

EMA PrOACT-URL
Unified/Universal Method for
Benefit Risk Assessment
(UMBRA)
BRAT Framework
Stochastic Multi-criteria
Acceptability Analysis(SMAA)
Multi-Criteria Decision
Analysis(MCDA)
Ashby and Smith Framework
Value or Decision Tree

181

Please indicate how benefit-risk information is currently characterized in your company's regulatory submissions.

Qualitative approach using text only

Semi-quantitative approach that uses qualitative text and some quantitative elements (e.g. Effects Table)

Quantitative approach displaying a cumulative outcome in the form of text, tables and other visualization methods (eg forest
plots, bar graphs, etc)
I do not know how benefit-risk information is characterized in the regulatory submission(s)
Other - please specifiy

Please indicate where benefit-risk information is currently used (choose all that apply):

As part of labeling development (e.g. Target Product Profile/TPP)
To develop Package Insert/Prescribing Information (PI)
As part of the Clinical Overview (CTD M2.5.6)
Other - please specifiy

I do not know where benefit-risk information is used.
None of the above

Have you received feedback from Regulatory Agencies regarding your benefit-risk assessment format (choose all
that apply)? Please provide additional details in the space at the bottom.

As part of discussions with regulators regarding development plan (e.g. Scientific Advice)
At pre-submission meeting (e.g. preNDA/MAA)
As part of marketing application filing review

As part of a question during marketing application content review
As part of labeling negotiations
As part of formal agency communication at the end of review
No, no feedback provided
Other - please specify

Additional details (optional)

Do you feel the benefit-risk assessment exercise influenced the decision made by the regulators to approve or
not approve your product(s)?

Yes - comments (optional)

No

If you answered yes, was there a region or country where this was apparent? Please specify.

182

It is believed that the implementation of a benefit-risk framework should confer certain advantages to the
organization.
From the list below, please rate the following advantages as high, medium, low, or not applicable.

It is believed that the successful implementation of a benefit-risk framework may be impeded by certain barriers.
From the list below, please indicate the level of importance these barriers represent to your company when
implementing a structured approach to benefit-risk assessment

From the dropdown menu provided please indicate which of the barriers above do you consider the most
important?

Not at all
Important
Somewhat
important Very Important
Extremely
Important Cannot Answer
Resource limitation e.g.
manpower, finances
Lack of knowledge/expertise to
execute framework
Resistance to change (culture
documentation or methodology)
Significant change to work
processes
Significant retraining of staff
required
Support from senior
management required
Lack of a industry accepted
framework
Lack of a regulatory harmonized
framework
Others: Please specify
Not at all Somewhat
Important important
Very
Important
Extremely
Important No Opinion
Acts as a tool for communication among peers within
organization
Acts as a tool for communication between organization
and external stakeholders (including regulatory agencies)
Provides archival documentation for structured decision-
making
Ensures consistency in quality of BR assessment within
the organization
Enhances transparency within the organization
Ensures consistency in quality of BR assessment by
regulators
Aligns internal decision-making
Facilitates external communications with prescribers and
patients
Other: Please specify

183

Regarding the need for an appropriate BR framework, please read the following statements and mark one of the
given options. If you would like to modify the statement or have additional comments, please add these in the
text field provided at the end of the survey.

Neither
Strongly Agree nor Strongly
Agree Agree Disagree Disagree Disagree
For the registration of new medicinal products it will be necessary to
develop therapeutic area specific BR frameworks and methods
It is important that any BR framework, if developed for registration
purposes, is utilized across product classes (e.g. drugs, biologics and
devices)
It is important that any BR framework, if developed for registration
purposes, is utilized worldwide
An appropriate BR framework for registration should also enable
assessment of benefit-risk management plans (BRMP).
An appropriate BR framework for registration should also apply to all
stages of drug development from drug development to post-approval
changes
It is important that all stakeholders (agencies, companies,prescribers and
patients) are part of the development and validation of an appropriate
Benefit Risk framework
BR Frameworks should be utilized more in communications with
prescribers and patients
An ICH harmonized guidance on the use of BR frameworks is needed
Please provide your opinion about the usefulness and utility of structured benefit-risk assessment frameworks.
Having a BR Framework that....
Extremely useful Very Useful Somewhat useful Not useful
Is standardized and harmonized
Can be repurposed for health technology assessment (eg
value dossier/reimbursement)
Can be used to communicate with prescribers

Can be used to communicate with patients
Allows for transparency in decision making for
sponsors/companies
Allows for consistency in decision making for
sponsors/companies
Allows for quality in decision making for
sponsors/companies
Accelerates decision making for sponsors/companies
Documents decision making for sponsors/companies

184

185

An ICH sponsored harmonized guidance that is part of M4 (R2) addressing the format for benefit-risk data
presentation would be most useful if it was based on... (check all that apply):

FDA BR Framework
EMA PrOACT-URL
Unified/Universal Method for Benefit Risk Assessment (UMBRA)
BRAT Framework
Multi-Criteria Decision Analysis

Stochastic Multi-criteria Acceptability Analysis
Ashby and Smith Framework
Value/Decision Tree(s)

An independent harmonized approach
None of the above
Other

Have you engaged external stakeholders, either prescribers, payers or patients, in your benefit-risk assessment
process?

Prescribers/Physicians
Patients
Payers (e.g. HTA or Reimbursement Providers)

If yes, please briefly describe how you have engaged them in the space below

No we have not engaged external stakeholder in our benefit-risk assessment process
I do not know

In the space provided below, please provide any additional information or comments that you feel will contribute
my understanding of your views on benefit-risk assessment and the utilization of frameworks in decision-making,
regulatory filings and communications. Thank you!

Comments

186

APPENDIX C. RESULTS REPORTS COMMENT SECTION
C.1 Results Report – Expert Panel

187

Results Report – Expert Panel
Last Modified: 08/30/2015
1. What title best describes your current role within your
organization?
Answer

Response %
Vice President-
Regulatory

26 30%
Director/Sr.
Director -
Regulatory

33 38%
Manager/Sr.
Manager -
Regulatory

7 8%
Consultant/Sr.
Consultant

10 11%
Other - please
specify

11 13%
Total 87 100%

Other - please specify
VP - PV
Sr. Regulatory Specialist- Regulatory
Biostatistician - benefit-risk
Vice President - Global Pharmacovigilance
Sr Dir Benefit Risk Eval
Vice President, Regulatory Strategy and Content Development (Consultant)
Executive Director, Regulatory Affairs
President and Principal Consultant - Strategic Regulatory & Medical Writing & Sr.
Research Fellow -Centre for Innovation in Regulatory Science
Assoc. Dir. - Regulatory
retired

188

2. What geographical area are you responsible for?
# Answer

Response %
1 US only

25 29%
2 EU only

1 1%
3 Global

54 62%
4
Rest of
World

1 1%
5
Other -
please
specify

6 7%
Total 87 100%

Other - please specify
EMEA
North America
U.S. And EU
EMEA, AP, LA
U.S. And EU only
US & Canada

3. Please indicate where your company headquarters is located.
# Answer

Response %
1
United
States/North
America

65 75%
2 Europe

15 17%
3 Asia Pacific

4 5%
4
Other - please
specify

3 3%
Total 87 100%

Other - please specify
Israel
Canada
Israel

189

4. In this role do you participate in benefit-risk assessment of
new or existing products?
# Answer

Response %
1
Yes, if so
please
indicate
how many
years in the
space
below

66 77%
2 No

20 23%
Total 86 100%

190

Yes, if so please indicate how many years in the space below
8
4
20
15
5
20
2
1 year
1 year
20+
3
5
15
4
8
12
25
20
10+
10
25 overall
10
7 years in VP role
2 months
3
20
24
Participate in the monthly meetings assessment
18
7
5
4
15
20
15
5 years
20
>10 years
13
10
Only in developing methodology -

191

7 years
6
7
20
10
18
25+
17
5
20
>6 years
Yes, 11 years
20
10
20
1
30
10
7

5. What is the size of the company for which you are currently
working based on number of employees?
# Answer

Response %
1
Less than
250
employees

42 48%
2
250-2000
employees

19 22%
3
More than
2000
employees

26 30%
Total 87 100%

192

6. My company has or works with the following prescription
drug products (includes therapeutic biologics):
# Answer

Response %
1
Only
innovator
prescription
products in
development

20 23%
2
Only
marketed
prescription
products

1 1%
3
A combination
of products
that are on
the market
and in
development

65 76%
Total 86 100%

193

7. How would you best describe the internal approach that your
company uses to assess benefit-risk (B-R) of a new medicine
during development?
# Answer

Response %
1
Our internal
approach is a
purely
qualitative

17 20%
2
Our internal
approach is
semi
quantitative
and uses both
qualitative and
quantitative
elements.
Please
describe
briefly below if
you can

29 33%
3
Our internal
approach is a
fully
quantitative
model

5 6%
4
Our company
has no internal
formal
approach for
assessing
benefit-risk of
a new
medicine
during
development.

28 32%
5 Comments

8 9%
Total 87 100%

194

Our internal approach is semi quantitative
and uses both qualitative and quantitative
elements. Please describe briefly below if you
can
Comments
Regular team review of clinical and
nonclinical data
Although I have provided presentations
and strategic advice we have not
participated in assessing risk benefit of
products
Team generally utilizes FDA's qualitative
benefit-risk framework implemented in
2013, but assesses specific safety issues
quantitatively.
benefit risk is assessed on a case by case
basis
We summarize preclinical data including
pharmacology/tox/pk to evaluate a drug’s
potential early for product candidate.
Post IND the clinical safety data would of
course be continuously monitored for
overall assessment.
I am in CMC Regulatory Affairs, so don’t
get involved much with this. However,
based on my understandings, our
internal approach is purely qualitative.
We have an internal work stream that is
looking at the various quantitative models
and there are pilots in place. We also
have a staff member on the ICH working
group around B-R assessment.
I cannot answer
assessment for marketing in USA based
on stated regulatory expectations
My company provides services to other
companies, therefore RA is not directly
performed
Risks (i.e., safety) is quantitative. Efficacy
tends to be more qualitative during
development since our products focus on
diagnostics/medical imaging
N/A
percentage of each is a moving target
depending upon the project
The process is not formal but uses
qualitative (e.g.qual market research,
KoL discussion, internal team expertise)
and semi-quantitative (e.g. quant market
research, market assessment, internal
assessment of data)
Initially, ranking and valuing are
intrinsically qualitative, and subsequently
converted to quantitative elements
There are no formal procedures as yet
however this is being developed as we
speak
we use a toolbox of qualitative,
quantitative, and visualization methods

Generally, it is a qualitative approach.

195

If there is a safety signal in a study, the
signal is thoroughly evaluated to
understand if it is a true risk to patients,
and what if anything can be done to
minimize signal if it is a risk to patients.
Does benefit outweigh the risk?

The main risk-benefit activities that we
conduct internally involve the safety
assessment of drugs prior to entering the
clinic, as part of the IND-enabling
program. This assessment is conducted by
toxicologists and is partly quantitative
(safety margins etc.) and qualitative
(assessment and interpretation of
nonclinical outcomes)

Currently BRAT
Design controls, financial assessment and
fit to current products/customer base

We review the data from each study
conducted which allows us the
opportunity to examine the safety and
efficacy of the investigational product
over time. We also conduct market
research using key data from studies to
acquire competitive information.

BRAT + statistics
Both elements are part of the assessment
process

196

8. Are you satisfied with your existing benefit-risk assessment
approach during development?
# Answer

Response %
1 Yes

47 54%
2 No

40 46%
Total 87 100%

9. If NO, please provide the reason(s) that best describe your
situation (choose all that apply):
# Answer

Response %
1
Too time
consuming to
utilize the
approach

1 3%
2
Requires
additional training
to understand and
learn to use the
approach

4 11%
3
Poor acceptance
by staff

4 11%
4
Benefits of the
approach are not
apparent

4 11%
5
Approach/method
is not reliable

6 17%
6
We have no
approach

13 36%
7
Approach/method
is not generally
accepted within
the industry

2 6%
8
Other - please
describe below

14 39%

197

Other - please describe below
Although quantitative methodologies are not universally accepted, we ought to at
least experiment with these. We are quite conservative as, being in the service sector,
we generally follow what our clients use and they do not outsource quantitative BR
work, at least not yet.
The issue is that previously we left this assessment to each team. This must be
formalized to ensure that there is a consistent approach used
The interpretation of results from any method is subjective, so it is unclear what value
the method provides
descriptive only-following OPDP guidelines
Methods are still in infancy and not well accepted by regulators. This makes it hard to
get traction in the company
improvements on quantitative methods needed
There is complete reliance on FDA and EMA acceptability - product is for high unmet
need
Do not use frameworks - focus is on risk management.
I agree that in attempting to “standardize” practice guidelines, it is necessary to try to
be "systematic" in adjudicating the existing knowledge, but this good intention is
often unjustifiably taken to the level of religious devotion. Thus, these methodologies
do not--in my view--take adequate account of what I would call the
"particularizability" problem, i.e., they tend to treat measures of central tendency
from population data as if they were automatically, and unproblematic ally,
"particularizable" to individual patients or different clinical or policy contexts, without
sufficient justification of these inferences. This is sort of a “hegemony of central
tendency", which is at odds with the growing trend toward "personalized medicine".
We really have personalized medicine with impersonal evidence. I certainly have a
strong appreciation of the need for some non-arbitrary approaches, a la structured B-
R analyses; however, “regression toward the mean” modeling does not account for
the outliers and, as we move toward more personalized medicine in which sub-
cohorts of sub-cohorts are identified, we are going to fail at the tails of the Bell Curve.
not a standard template that all areas rely on and understand
Approach/methods used by different regulatory agencies differ substantially
Not sure if the quantitative methodology is understood more broadly across the
organization. Also concerned that the quantitative methodology may not be well
understood by reviewers in the health authorities.
Our approach to benefit-risk assessment is not standardized or formalized.
Working at a CRO, the level of support varies from client to client

198

10. Our company uses benefit-risk assessments for new
medicines during drug development for:
# Answer

Response %
1
All products -
feel free to
describe
below

57 77%
2
Only certain
products -
please
describe how
these
products are
selected as
candidates
for
assessment
(e.g. high risk
or
innovative)

17 23%
Total 74 100%

199

All products - feel free to describe below
Only certain products - please describe how
these products are selected as candidates for
assessment (e.g. high risk or innovative)
same as above, also monitoring post-
marketing AEs for our marketed product
My company develops IVDs and
companion Dx
we have one gene therapy product
Basically those that our clients outsource
to us.
We do not have a unified approach across
all project teams; however, each team is
expected to do some assessment of
benefit-risk. Because the approaches
taken by the various teams differ, it does
not allow for easily comparing across
projects.
high risk and innovative
Starting at onset of Phase 2
innovative revenue and timing of
regulatory approval expected
All products are
High risk or innovative products are
selected for benefit-risk assessment.
Some investigational products are only
for proof-of-concept.
milestone reviews + other Focus is more on safety than efficacy
As a consulting firm, this is a core part of
our business, and as noted earlier we
conduct such assessments on all products
that we involved with prior to FIH studies
being conducted.
when requested by our clients
Align company position on benefit: risk
prior to filing and to inform proposed
labelling, align company key messages in
core global CTD, understand product
valuation
We reserve the formal analyses for those
products which are equivocal in terms of
intuitive decisions (e.g., products of a
well-known class and well-defined
therapeutic index)
products which support the medical
countermeasures for the US government
We provide services, including RA, if
contracted
assessed on a case by case basis no
formal approach
as promising early candidates progress in
development
to some degree all products are assessed;
however depending upon the product
greater or lesser assessment is done
when the benefit-risk trade-off is not
obvious
variety of clients, indications, and
products
Innovative

agreement on parameters to assess
benefit-risk

200

Depends on the project
innovative

201

11. How would you best describe the internal framework that
your company uses for assessing benefit-risk (B-R) of a new

202

medicine for submission to regulatory agencies as part of a
marketing application?
# Answer

Response %
1
Our
framework is
a purely
qualitative .

15 19%
2
Our
framework is
semi
quantitative
and uses
both
qualitative
and
quantitative
elements.
Please
describe
briefly below

29 37%
3
Our
framework is
a fully
quantitative

5 6%
4
Our company
has no
framework
for assessing
benefit-risk
of a new
medicine for
submission.
We provide
the required
text in the
clinical
overview
only

30 38%
Total 79 100%

203

Our framework is semi quantitative and uses both qualitative and quantitative elements.
Please describe briefly below
Based on BRAT, but does not use formal decision trees
Data: outcomes, safety
use the BRAT framework
One product in phase 3, has statistical efficacy endpoints, if reach will submit.
We take a very methodical approach to benefit-risk assessment for submissions. We
analyze study data individually and as part of a pool with other study data.
Additionally, we review other products in the same class and provide analysis in a
comparative manner.
our pre-clinical folks assess the B/R within their framework but the clinical team is
less qualitative or quantitative
BRAT + statistics - may not show BR assessment in submission
Relies on qualitative assessment at outset and attempts to quantify these.
We have a diverse team who provides clients/projects tailored services, as directed
by the customer
we use a toolbox of qualitative, quantitative and visualization methods
see previous
difficult to agree on risks (i.e. real or imagined)
as described in regulations and guidance

12. Are you satisfied with your existing benefit-risk assessment
framework or method for submission purposes?
# Answer

Response %
1 Yes

42 55%
2 No

35 45%
Total 77 100%

204

13. If NO, please provide the reason(s) that best describe your
situation (choose all that apply):
# Answer

Response %
1
Too time
consuming to
utilize the
framework

1 3%
2
Requires
additional
training to
understand
and learn to
use the
framework

6 20%
3
Poor
acceptance
by staff

2 7%
4
Framework
is not reliable

4 13%
5
Benefits of
the
framework
are not
apparent

4 13%
6
We have no
framework

13 43%
7
Framework is
not generally
accepted
within the
industry

3 10%
8
Other -
please
describe
below.

11 37%

205

Other - please describe below.
See previous answer.
being developed
Same as previous: unclear what the formal methods provide since the results of any
method are subjectively interpreted.
Not generally accepted by regulators
Further refinement needed to the framework
A DMC that has access to unblinded data determines if study should proceed. Phase 3
study will not be unblinded until end. Therefore a B/R assessment is not possible until
then.
Agencies are looking for more quantitative assessments
See previous response
Lack of globally accepted approach diminishes value of exercise.
We have no SOP or well defined approach
Inconsistency between projects

206

14. From the list below, choose reason(s) that best describe why
a formal system is not used:
# Answer

Response %
1
Not required
for current
product
developmental
processes

9 30%
2
Lack of
knowledge of
BR frameworks
in general

16 53%
3
Lack of
scientifically
validated BR
framework

10 33%
4
Lack of a
common BR
framework
among peers
and/or
stakeholders

12 40%
5
Benefits of a BR
framework not
apparent

5 17%
6
Other (Please
specify)

7 23%

Other (Please specify)
small company, private, biotech firm
Clients do not provide this.
it is being developed
lack of time to develop one and use it
B:R assessment outcomes generally driven by opinions of influential regulatory
agency staff than any specific process
Cannot comment
In addition to benefits from using a B/R framework, there are also 'costs' or
downsides.

207

15. Please indicate below why you have chosen to assess
benefits-risk in this way by level of importance.
Question
Not at all
Importan
t
Very
Unimporta
nt
Neither
Important
nor
Unimporta
nt
Very
Importan
t
Extremel
y
Importan
t
Total
Response
s
Other 0 0 2 4 1 7
Align with
regulator
templates
1 4 9 28 14 56
Align with
product
development
strategy
1 1 7 40 8 57
Align with
reimburseme
nt strategy
6 3 21 21 6 57
Align with
company
expertise
1 1 13 27 16 58

Other
Each product and indication provides its own priorities, many of which depend on
products currently available on the market.
Requirements
We do not conduct clinical benefit-risk assessments - this is usually done by our client
prior to our involvement in preparation of the regulatory dossier.
company goals, objectives, and funding position
N/A
Include a global plan/approach
Helps define company B:R position: adds rigor to internal benefit: risk assessment,
and raises disconfirming information that challenge company B:R assumptions
senior managers
Experience

16. The following frameworks and methods have been
proposed as ways to perform benefit-risk assessments - please

208

indicate those with which you personally are familiar (check all
that apply).
# Answer

Response %
1
US FDA BR
Framework

39 66%
2
EMA PrOACT-
URL

18 31%
3
Unified/Universal
Method for
Benefit Risk
Assessment
(UMBRA)

10 17%
4
Benefit Risk
Assessment
Team (BRAT)
Framework

21 36%
5
Stochastic Multi-
criteria
Acceptability
Analysis (SMAA)

4 7%
6
Multi-Criteria
Decision Analysis
(MCDA)

11 19%
7
Ashby and Smith
Framework

0 0%
8
Value or Decision
Tree

23 39%
9
Not familiar with
any of these
frameworks

14 24%
10
Other - please
specify

3 5%

Other - please specify
ISO 14971
NNT/NNH, Net Clinical Benefit, Incorporating uncertainty
As in EU GVP modules

209

17. If you personally are familiar with any of these frameworks
or methods, please indicate your level of knowledge on the scale
below.
Answer Average Value Responses
US FDA BR Framework 5.28 43
EMA PrOACT-URL 4.55 22
Unified/Universal Method
for Benefit Risk Assessment
(UMBRA)
5.47 15
BRAT Framework 5.38 24
Stochastic Multi-criteria
Acceptability
Analysis(SMAA)
3.33 9
Multi-Criteria Decision
Analysis(MCDA)
4.19 16
Ashby and Smith
Framework
1.00 6
Value or Decision Tree 5.36 33

210

18. The following frameworks or methods have been proposed
as ways to perform benefit-risk assessments - please indicate
those that your company/organization uses

211

Question
My
compan
y does
not use
this
framew
ork
My
compa
ny
has/is
evaluat
ed this
for use
within
our
compa
ny
My
compan
y utilizes
this
method
for BR
Assessm
ent for
internal
use only
My
compan
y utilizes
this
method
for BR
Assessm
ent and
submits
it with
registrati
on
dossiers
My
company
utilizes
this
method
for BR
Assessme
nt and
uses it to
communic
ate
externally
to
prescriber
s and
patients
Not
aware
of this
framew
ork
Total
Respon
ses
US FDA BR
Framework
20 8 8 7 0 9 52
EMA
PrOACT-URL
18 7 2 3 0 21 51
Unified/Univ
ersal Method
for Benefit
Risk
Assessment
(UMBRA)
22 3 2 1 0 23 51
BRAT
Framework
18 6 2 3 0 21 50
Stochastic
Multi-criteria
Acceptability
Analysis(SMA
A)
21 2 1 0 0 25 49
Multi-Criteria
Decision
Analysis(MC
DA)
23 0 4 1 0 22 50
Ashby and
Smith
Framework
22 0 0 0 0 27 49
Value or
Decision
Tree
20 10 8 2 0 12 52

212

19. The following models and indices are often used to calculate
benefit-risk. Please indicate those that your
company/organization uses.

213

Question
My
company
does not
use this
model/in
dex
My
company
is
evaluatin
g this
model/in
dex for
use
within
our
company
My
company
utilizes
this
model/in
dex for
BR
Assessm
ent for
internal
use only
My
company
utilizes
this
model/in
dex for
BR
Assessm
ent and
submits
it with
registrati
on
dossiers
My
company
utilizes
this
method
for BR
Assessme
nt and
uses it to
communi
cate
externally
to
prescribe
rs and
patients
Not
aware
of this
framew
ork
Total
Respon
ses
Number
Needed to
Treat
(NNT)/Nu
mber
Needed to
Harm
(NNH)
17 2 7 8 2 20 56
Benefit
Risk Ratio
(BRR)
20 3 8 10 2 13 56
Increment
al Net
Benefit
(INB)
20 2 5 2 0 24 53
Quality
Adjusted
Life Years
(QALY)
20 2 10 5 3 17 57
Stated
Preference
Method
(Patient)
23 1 3 1 0 24 52

214

Quality
adjusted
Time
Without
Symptoms
and
Toxicity
(Q-TWiST)
23 2 3 0 0 25 53
Minimum
Clinical
Efficacy
(MCE)
24 2 6 5 1 18 56
Minimal
Acceptable
Risk (MAR)
25 1 9 3 1 17 56

215

20. Please indicate how benefit-risk information is currently
characterized in your company's regulatory submissions.
# Answer

Response %
1
Qualitative
approach
using text only

16 26%
2
Semi-
quantitative
approach that
uses
qualitative
text and some
quantitative
elements (e.g.
Effects Table)

27 44%
3
Quantitative
approach
displaying a
cumulative
outcome in
the form of
text, tables
and other
visualization
methods (e.g.
forest plots,
bar graphs,
etc.)

9 15%
4
I do not know
how benefit-
risk
information is
characterized
in the
regulatory
submission(s)

6 10%
5
Other - please
specify

3 5%
Total 61 100%

216

Other - please specify
Submission in planning stage. No decisions taken yet on content or how benefit-risk
will be described.
as needed for specific areas as needed
Each client is different; not relevant to my business so I do not know how each client
responds to RB

21. Please indicate where benefit-risk information is currently
used (choose all that apply):
# Answer

Response %
1
As part of
labeling
development
(e.g. Target
Product
Profile/TPP)

34 61%
2
To develop
Package
Insert/Prescribing
Information (PI)

33 59%
3
As part of the
Clinical Overview
(CTD M2.5.6)

50 89%
5
Other - please
specify

12 21%

Other - please specify
for CE marking
IB
Periodic Safety Update Report (PSUR)
Describe outcomes from earlier studies
When considering new or additional indications or a product
BRMP
internal milestone reviews and other safety reviews
clinical pharmacology
some CMC areas as needed
pre-meeting briefing packages, labeling negotiations
DSURS, PSURS, PEBERS PADERS
determining whether funding should be applied to specific projects

217

22. Have you received feedback from Regulatory Agencies
regarding your benefit-risk assessment format (choose all that
apply)? Please provide additional details in the space at the
bottom.
Answer

Response %
Additional details
(optional)

4 6%
As part of a
question during
marketing
application
content review

13 21%
As part of
discussions with
regulators
regarding
development plan
(e.g. Scientific
Advice)

12 19%
As part of formal
agency
communication at
the end of review

8 13%
As part of labeling
negotiations

12 19%
As part of
marketing
application filing
review

10 16%
At pre-submission
meeting (e.g.
preNDA/MAA)

11 18%
No, no feedback
provided

30 48%
Other - please
specify

5 8%

218

Other - please specify Additional details (optional)

Discussed content of ISE and ISS at Pre-
NDA meeting and reached agreement
with Division. Formal BR Not discussed
Not applicable - yet
Not involved

FDA has indicated they would like a fuller
discussion around B-R
notified body feedback
not sure
Do Not Know
Generally, regulatory agencies do not
comment directly on sponsor B:R
assessments, limiting their utility.

No information on this subject provided
to me

23. Do you feel the benefit-risk assessment exercise influenced
the decision made by the regulators to approve or not approve
your product(s)?
# Answer

Response %
1
Yes -
comments
(optional)

29 53%
2 No

19 35%
3
If you
answered
yes, was
there a
region or
country
where this
was
apparent?
Please
specify.

7 13%
Total 55 100%

219

Yes - comments (optional)
If you answered yes, was there a region or
country where this was apparent? Please
specify.
For FDA perspective I believe it helps
guide the FDA as they consider their own
decision. It does play a larger role in
advisory committee meetings.
in the US less so EU and Asia
NA no product approved yet US
absolutely this is important EU, US
Yes, in the sense that the regulatory
makes a decision based on benefit-risk,
but not in the sense of a specific
methodology as noted in earlier questions
US and EU
US market only EMA
No- but really N/A since we have not yet
had an approval decision
US, EU
In past, have seen it discussed in
conjunction with product approvals. I feel
it was a key element of the agency
decision.

Of course it does
But I can't quantify
Very subjective
I believe our prose description of the
benefit-risk considerations was reviewed
and generally accepted by the regulatory
reviewers.

220

24. It is believed that the implementation of a benefit-risk
framework should confer certain advantages to the

221

organization. From the list below, please rate the following
advantages as high, medium, low, or not applicable.

222

Question
Not at all
Important
Somewhat
important
Very
Important
No Opinion
Total
Responses
Acts as a tool
for
communication
among peers
within
organization
1 12 29 3 45
Acts as a tool
for
communication
between
organization
and external
stakeholders
(including
regulatory
agencies)
0 6 24 2 32
Provides
archival
documentation
for structured
decision-making
2 20 23 3 48
Ensures
consistency in
quality of BR
assessment
within the
organization
2 12 30 1 45
Enhances
transparency
within the
organization
3 14 27 2 46
Ensures
consistency in
quality of BR
assessment by
regulators
3 12 20 4 39
Aligns internal
decision-making
1 9 27 1 38

223

Facilitates
external
communications
with prescribers
and patients
4 14 18 8 44
Other: Please
specify
1 1 0 4 6

Other: Please specify
This survey is really for innovative companies; as a service provider many of these
questions are not applicable.
it provides a mechanism to help small companies make development decisions (e.g.,
kill products)
Although company is adopting B:R approaches and submitting in applications,
regulatory agencies do not comment or engage company on its B:R proposals
Aligns internal and external communication about the product by organization

25. It is believed that the successful implementation of a
benefit-risk framework may be impeded by certain barriers.
From the list below, please indicate the level of importance

224

these barriers represent to your company when implementing a
structured approach to benefit-risk assessment
Question
Not at all
Importan
t
Somewha
t
important
Very
Importan
t
Extremel
y
Importan
t
Canno
t
Answe
r
Total
Response
s
Resource limitation
e.g. manpower,
finances
5 12 20 18 5 60
Lack of
knowledge/expertis
e to execute
framework
2 6 23 24 5 60
Resistance to
change (culture
documentation or
methodology)
5 14 18 18 5 60
Significant change
to work processes
7 16 24 9 4 60
Significant
retraining of staff
required
7 17 23 9 4 60
Support from senior
management
required
8 4 20 24 4 60
Lack of an industry
accepted
framework
2 11 22 18 7 60
Lack of a regulatory
harmonized
framework
1 9 19 24 7 60

Others: Please specify
Lack of uptake and usage by regulatory agencies
If it is required, company resources and staff will figure out what to do.
Severe lack of competency in these techniques; major resource will be needed to
implement

225

26. From the dropdown menu provided please indicate which of
the barriers above do you consider the most important?
# Answer

Response %
1
Resource limitation
e.g. manpower,
finances

9 16%
2
Lack of
knowledge/expertise
to execute
framework

12 22%
3
Resistance to change
(culture
documentation or
methodology)

2 4%
4
Significant change to
work processes

3 5%
5
Significant retraining
of staff required

1 2%
6
Support from senior
management
required

8 15%
7
Lack of an industry
accepted framework

4 7%
8
Lack of a regulatory
harmonized
framework

15 27%
9
Other as indicated
above

1 2%
Total 55 100%

226

27. Regarding the need for an appropriate BR framework,
please read the following statements and mark one of the given
options. If you would like to modify the statement or have

227

additional comments, please add these in the text field provided
at the end of the survey.

228

# Question
Strongl
y Agree
Agre
e
Neither
Agree
nor
Disagre
e
Disagre
e
Strongly
Disagre
e
Total
Response
s
Mea
n
1
For the
registration of
new medicinal
products it will
be necessary to
develop
therapeutic
area specific BR
frameworks
and methods
10 29 12 7 1 59 2.32
2
It is important
that any BR
framework, if
developed for
registration
purposes, is
utilized across
product classes
(e.g. drugs,
biologics and
devices)
14 19 17 8 0 58 2.33
3
It is important
that any BR
framework, if
developed for
registration
purposes, is
utilized
worldwide
11 18 20 9 1 59 2.51

229

4
An appropriate
BR framework
for registration
should also
enable
assessment of
benefit-risk
management
plans (BRMP).
7 37 12 1 1 58 2.17
5
An appropriate
BR framework
for registration
should also
apply to all
stages of drug
development
from drug
development to
post-approval
changes
9 33 9 8 0 59 2.27
6
It is important
that all
stakeholders
(agencies,
companies,
prescribers and
patients) are
part of the
development
and validation
of an
appropriate
Benefit Risk
framework
14 29 11 5 0 59 2.12
7
BR Frameworks
should be
utilized more in
communication
s with
prescribers and
patients
6 25 21 7 0 59 2.49

230

8
An ICH
harmonized
guidance on
the use of BR
frameworks is
needed
17 29 12 2 0 60 1.98

231

28. Please provide your opinion about the usefulness and utility
of structured benefit-risk assessment frameworks. Having a BR
Framework that....
# Question
Extremel
y useful
Very
Usefu
l
Somewha
t useful
Not
usefu
l
Total
Response
s
Mea
n
1
Is standardized and
harmonized
28 28 4 1 61 1.64
2
Can be repurposed for
health technology
assessment (e.g. value
dossier/reimbursement
)
20 26 12 2 60 1.93
3
Can be used to
communicate with
prescribers
12 31 14 3 60 2.13
4
Can be used to
communicate with
patients
11 19 25 5 60 2.40
5
Allows for transparency
in decision making for
sponsors/companies
22 27 9 2 60 1.85
6
Allows for consistency
in decision making for
sponsors/companies
24 24 11 1 60 1.82
7
Allows for quality in
decision making for
sponsors/companies
25 26 8 1 60 1.75
8
Accelerates decision
making for
sponsors/companies
21 25 12 2 60 1.92
9
Documents decision
making for
sponsors/companies
18 32 10 0 60 1.87

232

29. An ICH sponsored harmonized guidance that is part of M4
(R2) addressing the format for benefit-risk data presentation
would be most useful if it was based on... (Check all that apply):
# Answer

Response %
1
FDA BR
Framework

24 41%
2
EMA PrOACT-
URL

18 31%
3
Unified/Universal
Method for
Benefit Risk
Assessment
(UMBRA)

10 17%
4 BRAT Framework

8 14%
5
Multi-Criteria
Decision Analysis

5 8%
6
Stochastic Multi-
criteria
Acceptability
Analysis

1 2%
7
Ashby and Smith
Framework

1 2%
8
Value/Decision
Tree(s)

7 12%
9
An independent
harmonized
approach

15 25%
10 Other

14 24%
11
None of the
above

3 5%

233

Other
Not really sure which is best - would leave that up to ICH
I do not feel competent yet to state
I'm not familiar with all of the above, so I can't choose a most useful method.
Can't say because I am not familiar with any of them.
Clear guidance on which of the above approaches is best under what
circumstance/condition
unable to comment - see earlier responses
providing Sponsors with the ability to justify which RB model works for their
particular product
I cannot answer intelligently because I'm not familiar with the advantages and
disadvantages of each method. It seems reasonable that the best elements of each
approach could be combined in the ultimate Guidance.
N/A
Consider all frameworks, if that would be possible. This would appease all
stakeholders.
Cannot comment on best std since I am not familiar w all
Can't comment
I don't know
I have no idea what the comparative strengths and weaknesses of these techniques
are

234

30. Have you engaged external stakeholders, either prescribers,
payers or patients, in your benefit-risk assessment process?
# Answer

Response %
1 Prescribers/Physicians

19 31%
2 Patients

5 8%
3
Payers (e.g. HTA or
Reimbursement
Providers)

6 10%
4
If yes, please briefly
describe how you
have engaged them
in the space below

9 15%
5
No we have not
engaged external
stakeholder in our
benefit-risk
assessment process

24 39%
6 I do not know

14 23%

If yes, please briefly describe how you have engaged them in the space below
We continuously work with outside clinical and regulatory experts as part of our
ongoing development and regulatory processes. Their feedback contributes to our BR
assessment
oncologist for sample preparation
for some products we have conducted stated-preference surveys of patients or
physicians
Open discussions of potential and realized risks.
Qualitative - discussion
MCDA, PROs, PAG boards
sought guidance and input on risk-benefit assessment and interpretations
As beta-testers
advisory board review and discussion

31. In the space provided below, please provide any additional
information or comments that you feel will contribute my
understanding of your views on benefit-risk assessment and the

235

utilization of frameworks in decision-making, regulatory filings
and communications. Thank you!
# Answer

Response %
1 Comments

18 100%
Total 18 100%

236

Comments
Only speaking for one company, we are unlikely to implement a formal BR tool unless
requested to do so by a regulatory agency
So sorry I didn't realize this was so focused on Drugs, my current experience reflected
here is with IVD classified as medical devices
I have used structured products for BR while in large companies however I am
currently in a startup and my experience is there is neither time nor money to spend
exploring BR tools that are available
In Europe there is risk management, as yet not benefit risk management. Risk
minimization which is the function of risk management, is specifically reducing the
frequency of occurrence of a specific ADR. If there was a concept of B/R management
then that would be more a product level, whereas risk management is more at an
individual safety issue level.
A single ICH guidance in this area may not fit all - for example, our product is
approved in Japan for 7 yrs., but studies accepted there not accepted by US or EMA.
Quantitating a benefit-risk seems to be an exercise in forcing a required standard
format onto an individualized and highly variable assessment. Benefit can be
quantified but it means something different to every party involved – profit and
exclusivity for those with financial interests in the sponsor company; high compliance
and fewer complications that could lead to increased office visits for prescribers; high
efficacy rate and low cost/inconvenience for patients; ease of manufacture/favorable
formulation for generic companies; low reimbursement cost for insurance companies;
consistent response rates across patient subgroups for regulators looking for clear
statistical differences from comparator products and an easy review/approval. Risk
is also valued differently depending on the party involved. An identifiable clinical risk
is most often viewed as the most likely adverse event. From a clinical perspective,
value is also given to a product with AEs that are not only less serious, but are
identifiable, predictable and/or easily treated. Other factors such as drug interactions,
abuse potential, misuse potential, high cost, poor stability, look-alike/sound-alike,
ease of counterfeit are key factors in the overall quality of a product. Also, what
about the aspects of off-label use – is this a risk? As the BR evaluation will differ
between disease states, between countries, and will depend on related therapies
already available, I believe a less-formalized but openly transparent assessment of
those and other factors should be considered in a marketing application. I believe
that a formalized and quantitated method will most likely provide a false sense of
security in understanding a product’s BR, as there will be more of a focus on satisfying
a method, than in discussing aspects that are less open to quantification.
This is not a tool I have a lot of experience with!
Even though we have the expertise, the company is waiting for a regulatory agency to
require BRA and provide the framework. In our industry, the Drug Safety Dept.
typically executes the benefit-risk assessments, though that might not be where the
expertise sits. There is a prevalent opinion that quantitative BRA is too subjective.

237

We are
Guidance clarifications
It was difficult to answer the question on barriers. Several were important to
overcome, but we overcame them, so they were no longer barriers, e.g., sr.
management support was crucial but was there from the beginning of our group's
formation, so was not a barrier.
It's not an area where I have given a lot of thought as of now. Most of the questions
are of more relevance to larger companies with a number of regular NDA
submissions. A smaller organization is likely to use a less formal and structured
approach where internal communication is not an important factor due to the small
number of staff.
Validation and reproducibility will be key to broad acceptance of a structured
framework for B-R assessment. A core framework can certainly be applied as an
overarching structure; however, it will be important to nest therapeutic-area-specific
elements for use, as appropriate. One-size-fits-all will not suffice.
N/A
BR needs to be better understood and propagated throughout the industry. It can
help all involved with uniform and sound decisions.
As noted above, I have been asked to provide strategic advice on occasion but have
declined to provide specific advice because it is not my area of expertise. I understand
the fundamentals of BR and have answer some strategic questions but cannot
evaluate specifics of any system.
A harmonized "ICH" process for assessment of benefit risk in terms of a quantitative
process would be very useful to standardize across regions
The amount of investment required to investigate different frameworks will be a
massive task requiring widespread system change

238

C.2 Results Report – TOPRA Panel

239

Results Report – TOPRA Panel
Last Modified: 11/06/2015
1. What title best describes your current role within your
organization? If none of these titles align with your organization
please utilize the "Other" text option.
# Answer

Response %
1
Vice
President/Head-
Regulatory

5 33%
2
Director/Sr.
Director -
Regulatory

5 33%
3
Manager/Sr.
Manager -
Regulatory

1 7%
4
Consultant/Sr.
Consultant

1 7%
5
Other - please
specify

1 7%
6
Regulatory
Specialist

2 13%
Total 15 100%

Other - please specify
N/A

240

2. What geographical area are you responsible for?
# Answer

Response %
1 US only

1 7%
2 EU only

2 14%
3 Global

10 71%
4
Rest of
World

1 7%
5
Other -
please
specify

0 0%
6
Country,
please
specify

0 0%
Total 14 100%

Other - please specify Country, please specify

3. Please indicate where your company headquarters is located.
# Answer

Response %
1
United
States/North
America

4 27%
2 Europe

9 60%
3 Asia Pacific

2 13%
4
Other - please
specify

0 0%
Total 15 100%

Other - please specify

241

4. In this role do you participate in benefit-risk assessment of
new or existing products?
# Answer

Response %
1
Yes, if so
please
indicate
how many
years in the
space
below

10 77%
2 No

3 23%
Total 13 100%

Yes, if so please indicate how many years in the space below
11
5
7 years
8-10
15
10
13
15
7
18

5. What is the size of the company for which you are currently
working based on number of employees?
# Answer

Response %
1
Less than
250
employees

5 33%
2
250-2000
employees

2 13%
3
More than
2000
employees

8 53%
Total 15 100%

242

6. My company has or works with the following prescription
drug products (includes therapeutic biologics):
# Answer

Response %
1
Only
innovator
prescription
products in
development

2 13%
2
Only
marketed
prescription
products

1 7%
3
A combination
of products
that are on
the market
and in
development

12 80%
Total 15 100%

243

7. How would you best describe the internal approach that your
company uses to assess benefit-risk (B-R) of a new medicine
during development?
# Answer

Response %
1
Our internal
approach is a
purely
qualitative

5 31%
2
Our internal
approach is
semi
quantitative
and uses both
qualitative and
quantitative
elements.
Please
describe
briefly below if
you can

4 25%
3
Our internal
approach is a
fully
quantitative
model

0 0%
4
Our company
has no internal
formal
approach for
assessing
benefit-risk of
a new
medicine
during
development.

5 31%
5 Comments

2 13%
Total 16 100%

244

Our internal approach is semi quantitative
and uses both qualitative and quantitative
elements. Please describe briefly below if you
can
Comments

No B/R evaluation used in current
company (here for 1 year)
not sure

8. Are you satisfied with your existing benefit-risk assessment
approach during development?
# Answer

Response %
1 Yes

9 60%
2 No

6 40%
Total 15 100%

245

9. If NO, please provide the reason(s) that best describe your
situation (choose all that apply):
# Answer

Response %
1
Too time
consuming to
utilize the
approach

0 0%
2
Requires
additional training
to understand and
learn to use the
approach

2 40%
3
Poor acceptance
by staff

0 0%
4
Benefits of the
approach are not
apparent

2 40%
5
Approach/method
is not reliable

1 20%
6
We have no
approach

2 40%
7
Approach/method
is not generally
accepted within
the industry

0 0%
8
Other - please
describe below

1 20%

Other - please describe below
Not mandated and expected by the regulators yet and some parts of the company are
reticent to take a position, preferring the regulators to do so

246

10. Our company uses benefit-risk assessments for new
medicines during drug development for:
# Answer

Response %
1
All products -
feel free to
describe
below

11 85%
2
Only certain
products -
please
describe how
these
products are
selected as
candidates
for
assessment
(e.g. high risk
or
innovative)

2 15%
Total 13 100%

All products - feel free to describe below
Only certain products - please describe how
these products are selected as candidates for
assessment (e.g. high risk or innovative)
B/R ratio is always reassessed at each
new safety information, and at least
when annual safety report is issued and
Investigator's Brochure is revised. On-
going protocol may be subsequently
amended.
when safety signals are detected

247

11. How would you best describe the internal framework that
your company uses for assessing benefit-risk (B-R) of a new

248

medicine for submission to regulatory agencies as part of a
marketing application?
# Answer

Response %
1
Our
framework is
a purely
qualitative.

4 31%
2
Our
framework is
semi
quantitative
and uses
both
qualitative
and
quantitative
elements.
Please
describe
briefly below

6 46%
3
Our
framework is
a fully
quantitative

1 8%
4
Our company
has no
framework
for assessing
benefit-risk
of a new
medicine for
submission.
We provide
the required
text in the
clinical
overview
only

2 15%
Total 13 100%

249

Our framework is semi quantitative and uses both qualitative and quantitative elements.
Please describe briefly below

12. Are you satisfied with your existing benefit-risk assessment
framework or method for submission purposes?
# Answer

Response %
1 Yes

9 60%
2 No

6 40%
Total 15 100%

250

13. If NO, please provide the reason(s) that best describe your
situation (choose all that apply):
# Answer

Response %
1
Too time
consuming to
utilize the
framework

0 0%
2
Requires
additional
training to
understand
and learn to
use the
framework

3 50%
3
Poor
acceptance
by staff

1 17%
4
Framework
is not reliable

0 0%
5
Benefits of
the
framework
are not
apparent

0 0%
6
We have no
framework

2 33%
7
Framework is
not generally
accepted
within the
industry

1 17%
8
Other -
please
describe
below.

0 0%

Other - please describe below.

251

14. From the list below, choose reason(s) that best describe why
a formal system is not used:
# Answer

Response %
1
Not required
for current
product
developmental
processes

2 50%
2
Lack of
knowledge of
BR frameworks
in general

4 100%
3
Lack of
scientifically
validated BR
framework

2 50%
4
Lack of a
common BR
framework
among peers
and/or
stakeholders

2 50%
5
Benefits of a
BR framework
not apparent

2 50%
6
Other (Please
specify)

0 0%

Other (Please specify)

252

15. Please indicate below why you have chosen to assess
benefits-risk in this way by level of importance.
# Question
Not at
all
Importa
nt
Very
Unimport
ant
Neither
Important
nor
Unimport
ant
Very
Importa
nt
Extrem
ely
Importa
nt
Total
Respons
es
Mea
n
1
Align with
company
expertise
2 0 2 4 4 12
3.6
7
2
Align with
regulator
templates
1 0 2 6 3 12
3.8
3
3
Align with
reimbursem
ent strategy
1 0 5 5 1 12
3.4
2
4
Align with
product
developmen
t strategy
1 1 1 6 3 12
3.7
5
5 Other 0 1 1 0 0 2
2.5
0

Other

253

16. The following frameworks and methods have been
proposed as ways to perform benefit-risk assessments - please
indicate those with which you personally are familiar (check all
that apply).
# Answer

Response %
1
US FDA BR
Framework

5 38%
2
EMA PrOACT-
URL

4 31%
3
Unified/Universal
Method for
Benefit Risk
Assessment
(UMBRA)

2 15%
4
Benefit Risk
Assessment
Team (BRAT)
Framework

5 38%
5
Stochastic Multi-
criteria
Acceptability
Analysis (SMAA)

0 0%
6
Multi-Criteria
Decision Analysis
(MCDA)

4 31%
7
Ashby and Smith
Framework

0 0%
8
Value or Decision
Tree

6 46%
9
Not familiar with
any of these
frameworks

3 23%
10
Other - please
specify

0 0%

Other - please specify

254

17. If you personally are familiar with any of these frameworks
or methods, please indicate your level of knowledge on the scale
below.
# Answer
Min
Value
Max
Value
Average
Value
Standard
Deviation
Responses
1
US FDA BR
Framework
0.00 10.00 3.50 3.66 8
2
EMA PrOACT-
URL
0.00 10.00 3.86 4.06 7
3
Unified/Universal
Method for
Benefit Risk
Assessment
(UMBRA)
0.00 5.00 1.40 2.07 5
4 BRAT Framework 0.00 7.00 2.25 2.38 8
5
Stochastic Multi-
criteria
Acceptability
Analysis(SMAA)
0.00 4.00 1.25 1.89 4
6
Multi-Criteria
Decision
Analysis(MCDA)
0.00 7.00 2.00 2.27 8
7
Ashby and Smith
Framework
0.00 3.00 1.00 1.41 4
8
Value or Decision
Tree
0.00 8.00 2.70 2.83 10

255

18. The following frameworks or methods have been proposed
as ways to perform benefit-risk assessment please indicate
those that your company/organization uses

256

# Question
My
compa
ny
does
not
use
this
frame
work
My
comp
any
has/is
evalua
ted
this
for
use
within
our
comp
any
My
compa
ny
utilizes
this
metho
d for
BR
Assess
ment
for
internal
use
only
My
compa
ny
utilizes
this
metho
d for
BR
Assess
ment
and
submits
it with
registra
tion
dossier
s
My
compan
y utilizes
this
method
for BR
Assessm
ent and
uses it to
commun
icate
externall
y to
prescrib
ers and
patients
Not
aware
of this
frame
work
Total
Respo
nses
Me
an
1
US FDA BR
Framework
2 2 2 0 1 6 13
4.0
8
2
EMA
PrOACT-
URL
3 1 2 0 1 6 13
4.0
0
3
Unified/Uni
versal
Method for
Benefit
Risk
Assessmen
t (UMBRA)
3 1 0 1 0 5 10
3.9
0
4
BRAT
Framework
3 2 2 1 0 5 13
3.6
2
5
Stochastic
Multi-
criteria
Acceptabili
ty
Analysis(S
MAA)
4 0 0 0 0 6 10
4.0
0
6
Multi-
Criteria
Decision
Analysis(M
CDA)
4 2 1 0 0 5 12
3.4
2

257

7
Ashby and
Smith
Framework
4 0 0 0 0 6 10
4.0
0
8
Value or
Decision
Tree
5 1 1 1 0 4 12
3.1
7

258

19. The following models and indices are often used to calculate
benefit-risk. Please indicate those that your
company/organization uses.

259

# Question
My
compan
y does
not use
this
model/i
ndex
My
compan
y is
evaluati
ng this
model/i
ndex
for use
within
our
compan
y
My
compan
y
utilizes
this
model/i
ndex
for BR
Assess
ment
for
internal
use
only
My
compan
y
utilizes
this
model/i
ndex
for BR
Assess
ment
and
submits
it with
registra
tion
dossiers
My
compan
y utilizes
this
method
for BR
Assessm
ent and
uses it
to
commu
nicate
externall
y to
prescrib
ers and
patients
Not
aware
of this
frame
work
Total
Respo
nses
Me
an
1
Number
Needed
to Treat
(NNT)/N
umber
Needed
to Harm
(NNH)
3 1 2 0 1 4 11
3.6
4
2
Benefit
Risk
Ratio
(BRR)
2 1 4 0 3 1 11
3.3
6
3
Incremen
tal Net
Benefit
(INB)
2 1 1 0 1 4 9
4.0
0
4
Quality
Adjusted
Life Years
(QALY)
1 1 3 1 3 2 11
3.9
1
5
Stated
Preferen
ce
Method
(Patient)
3 0 0 0 1 6 10
4.4
0

260

6
Quality
adjusted
Time
Without
Symptom
s and
Toxicity
(Q-
TWiST)
2 0 0 0 2 5 9
4.6
7
7
Minimu
m Clinical
Efficacy
(MCE)
2 1 1 0 2 4 10
4.1
0
8
Minimal
Acceptab
le Risk
(MAR)
2 1 0 0 2 4 9
4.2
2

261

20. Please indicate how benefit-risk information is currently
characterized in your company's regulatory submissions.
# Answer

Response %
1
Qualitative
approach
using text only

0 0%
2
Semi-
quantitative
approach that
uses
qualitative
text and some
quantitative
elements (e.g.
Effects Table)

2 20%
3
Quantitative
approach
displaying a
cumulative
outcome in
the form of
text, tables
and other
visualization
methods (e.g.
forest plots,
bar graphs,
etc.)

5 50%
4
I do not know
how benefit-
risk
information is
characterized
in the
regulatory
submission(s)

3 30%
5
Other - please
specify

0 0%
Total 10 100%

262

Other - please specify

21. Please indicate where benefit-risk information is currently
used (choose all that apply):
# Answer

Response %
1
As part of
labeling
development
(e.g. Target
Product
Profile/TPP)

6 55%
2
To develop
Package
Insert/Prescribing
Information (PI)

6 55%
3
As part of the
Clinical Overview
(CTD M2.5.6)

10 91%
5
Other - please
specify

4 36%

Other - please specify
PBRERs, PSURs, ASRs, DSURs
during clinical development
IMPD
RMP

22. Have you received feedback from Regulatory Agencies
regarding your benefit-risk assessment format (choose all that

263

apply)? Please provide additional details in the space at the
bottom.
# Answer

Response %
2
At pre-
submission
meeting (e.g.
preNDA/MAA)

4 31%
3
As part of
marketing
application
filing review

4 31%
4
As part of a
question during
marketing
application
content review

3 23%
5
As part of
labeling
negotiations

4 31%
6
As part of
formal agency
communication
at the end of
review

4 31%
7
No, no
feedback
provided

4 31%
8
Other - please
specify

1 8%
9
Additional
details
(optional)

2 15%
10
As part of
discussions
with regulators
regarding
development
plan (e.g.
Scientific
Advice)

3 23%

264

Other - please specify Additional details (optional)
not recently, but in previous projects
don't know

23. Do you feel the benefit-risk assessment exercise influenced
the decision made by the regulators to approve or not approve
your product(s)?
# Answer

Response %
1
Yes -
comments
(optional)

6 55%
2 No

3 27%
3
If you
answered
yes, was
there a
region or
country
where this
was
apparent?
Please
specify.

2 18%
Total 11 100%

Yes - comments (optional)
If you answered yes, was there a region or
country where this was apparent? Please
specify.
Apparent in European Union, North
America, Latin America, Japan
EU
EU

24. It is believed that the implementation of a benefit-risk
framework should confer certain advantages to the

265

organization. From the list below, please rate the following
advantages as high, medium, low, or not applicable.

266

# Question
Not at all
Important
Somewhat
important
Very
Important
No
Opinion
Total
Responses
Mean
1
Acts as a tool
for
communication
among peers
within
organization
1 2 3 1 7 2.71
2
Acts as a tool
for
communication
between
organization
and external
stakeholders
(including
regulatory
agencies)
0 0 3 1 4 3.50
3
Provides
archival
documentation
for structured
decision-making
0 2 1 2 5 3.40
4
Ensures
consistency in
quality of BR
assessment
within the
organization
0 0 2 2 4 4.00
5
Enhances
transparency
within the
organization
0 2 3 1 6 3.00
6
Ensures
consistency in
quality of BR
assessment by
regulators
0 1 1 2 4 3.75
7
Aligns internal
decision-making
1 2 1 1 5 2.60

267

8
Facilitates
external
communications
with prescribers
and patients
0 4 1 1 6 2.67
9
Other: Please
specify
0 0 0 1 1 5.00

25. It is believed that the successful implementation of a
benefit-risk framework may be impeded by certain barriers.
From the list below, please indicate the level of importance

268

these barriers represent to your company when implementing a
structured approach to benefit-risk assessment
# Question
Not at
all
Importa
nt
Somewh
at
importa
nt
Very
Importa
nt
Extrem
ely
Importa
nt
Cann
ot
Answ
er
Total
Respons
es
Mea
n
1
Resource
limitation e.g.
manpower,
finances
1 3 3 0 3 10 3.10
2
Lack of
knowledge/expe
rtise to execute
framework
1 1 2 3 3 10 3.60
3
Resistance to
change (culture
documentation
or methodology)
1 1 5 0 3 10 3.30
4
Significant
change to work
processes
1 2 3 1 3 10 3.30
5
Significant
retraining of
staff required
1 1 2 3 3 10 3.60
6
Support from
senior
management
required
1 1 1 3 4 10 3.80
7
Lack of an
industry
accepted
framework
0 2 2 3 3 10 3.70
8
Lack of a
regulatory
harmonized
framework
0 1 3 3 2 9 3.67
9
Others: Please
specify
1 0 0 0 3 4 4.00

269

Others: Please specify
The above barriers listed as 'not important at all' are so because in the
pharmaceutical industry no excuses are acceptable. Not demonstrating initial and
continued benefit-risk of new compounds/biologics and continued for marketed
products would mean revoking of the registration license.

26. From the dropdown menu provided please indicate which of
the barriers above do you consider the most important?
# Answer

Response %
1
Resource limitation
e.g. manpower,
finances

0 0%
2
Lack of
knowledge/expertise
to execute
framework

3 38%
3
Resistance to change
(culture
documentation or
methodology)

0 0%
4
Significant change to
work processes

0 0%
5
Significant retraining
of staff required

0 0%
6
Support from senior
management
required

2 25%
7
Lack of an industry
accepted framework

1 13%
8
Lack of a regulatory
harmonized
framework

2 25%
9
Other as indicated
above

0 0%
Total 8 100%

27. Regarding the need for an appropriate BR framework,
please read the following statements and mark one of the given

270

options. If you would like to modify the statement or have
additional comments, please add these in the text field provided
at the end of the survey.

271

Question
Strongl
y Agree
Agre
e
Neither
Agree
nor
Disagre
e
Disagre
e
Strongly
Disagre
e
Total
Response
s
Mea
n
1
For the
registration of
new medicinal
products it will
be necessary to
develop
therapeutic
area specific BR
frameworks
and methods
2 3 2 1 1 9 2.56
2
It is important
that any BR
framework, if
developed for
registration
purposes, is
utilized across
product classes
(e.g. drugs,
biologics and
devices)
2 5 2 0 0 9 2.00
3
It is important
that any BR
framework, if
developed for
registration
purposes, is
utilized
worldwide
3 2 4 0 0 9 2.11

272

4
An appropriate
BR framework
for registration
should also
enable
assessment of
benefit-risk
management
plans (BRMP).
2 6 1 0 0 9 1.89
5
An appropriate
BR framework
for registration
should also
apply to all
stages of drug
development
from drug
development to
post-approval
changes
2 4 2 1 0 9 2.22
6
It is important
that all
stakeholders
(agencies,
companies,
prescribers and
patients) are
part of the
development
and validation
of an
appropriate
Benefit Risk
framework
3 5 1 0 0 9 1.78
7
BR Frameworks
should be
utilized more in
communication
s with
prescribers and
patients
1 5 3 0 0 9 2.22

273

8
An ICH
harmonized
guidance on
the use of BR
frameworks is
needed
2 6 1 0 0 9 1.89

274

28. Please provide your opinion about the usefulness and utility
of structured benefit-risk assessment frameworks. Having a BR
Framework that....
# Question
Extremel
y useful
Very
Usefu
l
Somewha
t useful
Not
usefu
l
Total
Response
s
Mea
n
1
Is standardized and
harmonized
3 4 3 0 10 2.00
2
Can be repurposed for
health technology
assessment (e.g. value
dossier/reimbursement
)
4 4 1 1 10 1.90
3
Can be used to
communicate with
prescribers
2 6 2 0 10 2.00
4
Can be used to
communicate with
patients
2 4 3 1 10 2.30
5
Allows for transparency
in decision making for
sponsors/companies
5 3 2 0 10 1.70
6
Allows for consistency
in decision making for
sponsors/companies
5 2 2 0 9 1.67
7
Allows for quality in
decision making for
sponsors/companies
4 5 1 0 10 1.70
8
Accelerates decision
making for
sponsors/companies
1 5 4 0 10 2.30
9
Documents decision
making for
sponsors/companies
2 6 2 0 10 2.00

275

29. An ICH sponsored harmonized guidance that is part of M4
(R2) addressing the format for benefit-risk data presentation
would be most useful if it was based on... (Check all that apply):
# Answer

Response %
1
FDA BR
Framework

4 50%
2
EMA PrOACT-
URL

4 50%
3
Unified/Universal
Method for
Benefit Risk
Assessment
(UMBRA)

4 50%
4 BRAT Framework

2 25%
5
Multi-Criteria
Decision Analysis

2 25%
6
Stochastic Multi-
criteria
Acceptability
Analysis

0 0%
7
Ashby and Smith
Framework

0 0%
8
Value/Decision
Tree(s)

0 0%
9
An independent
harmonized
approach

1 13%
10 Other

3 38%
11
None of the
above

0 0%

Other
not able to answer
provided the principles required
don't know

276

30. Have you engaged external stakeholders, either prescribers,
payers or patients, in your benefit-risk assessment process?
# Answer

Response %
1 Prescribers/Physicians

4 36%
2 Patients

2 18%
3
Payers (e.g. HTA or
Reimbursement
Providers)

2 18%
4
If yes, please briefly
describe how you
have engaged them
in the space below

2 18%
5
No we have not
engaged external
stakeholder in our
benefit-risk
assessment process

5 45%
6 I do not know

2 18%

If yes, please briefly describe how you have engaged them in the space below
Ongoing pharmacovigilance monitoring of marketed products, KOL opinions and
clinical study results for compounds in development; HTA scientific advice/guidance
Experience is limited and should be increased

277

31. In the space provided below, please provide any additional
information or comments that you feel will contribute my
understanding of your views on benefit-risk assessment and the
utilization of frameworks in decision-making, regulatory filings
and communications. Thank you!
# Answer

Response %
1 Comments

2 100%
Total 2 100%

Comments
It would be helpful to have a comprehensive information on the different
approaches. Seems to me a complex topic so far, although I have the feeling that it is
a critical assessment to do.
This will be of increasing importance in the next 5 years and we must agree and then
speak the same language as regulators, cross company and find robust technical
methodologies that allow consistent articulation of benefit risk, that are then
interpretable by the general public in simple diagrammatic tools. The Benefit risk
assessment is different for different stakeholders as the context varies, but we also
need to establish a simple way to explain this to the lay person who routinely are
confused by acceptance by one stakeholder and rejection by another resulting in no
or limited access.

278

APPENDIX D. ADDITIONAL CROSS TABULATIONS
Table D.1: Cross Tabulation of Company Size and Models/Indices Utilization

What is the size of the company for
which you are currently working
based on number of employees?
<250
employees
250-2000
employees
> 2000
employees
Number Needed to
Treat
(NNT)/Number
Needed to Harm
(NNH)
My company does not use this
model/index
12 3 2

My company is evaluating this
model/index for use within our company
0 0 2
My company utilizes this model/index
for BR Assessment for internal use only
3 0 4
My company utilizes this model/index
for BR Assessment and submits it with
registration dossiers
6 0 2
My company utilizes this method for BR
Assessment and uses it to communicate
externally to prescribers and patients
1 0 1
Not aware of this framework
6 7 7
Benefit Risk Ratio
(BRR)
My company does not use this
model/index
13 1 6

My company is evaluating this
model/index for use within our company
0 0 3
My company utilizes this model/index
for BR Assessment for internal use only
7 1 0
My company utilizes this model/index
for BR Assessment and submits it with
registration dossiers
5 1 4
My company utilizes this method for BR
Assessment and uses it to communicate
externally to prescribers and patients
0 0 2
Not aware of this framework
4 7 2

279

Table D.1: Cross Tabulation of Company Size and Models/Indices Utilization
(Cont'd)

What is the size of the company for
which you are currently working
based on number of employees?
<250
employees
250-2000
employees
> 2000
employees
Incremental Net
Benefit (INB)
My company does not use this
model/index
14 1 5

My company is evaluating this
model/index for use within our company
0 0 2
My company utilizes this model/index
for BR Assessment for internal use only
1 2 2
My company utilizes this model/index
for BR Assessment and submits it with
registration dossiers
1 0 1
My company utilizes this method for BR
Assessment and uses it to communicate
externally to prescribers and patients
0 0 0
Not aware of this framework
11 6 7
Quality Adjusted
Life Years (QALY)
My company does not use this
model/index
14 2 4

My company is evaluating this
model/index for use within our company
0 0 2
My company utilizes this model/index
for BR Assessment for internal use only
4 1 5
My company utilizes this model/index
for BR Assessment and submits it with
registration dossiers
5 0 0
My company utilizes this method for BR
Assessment and uses it to communicate
externally to prescribers and patients
1 0 2
Not aware of this framework
5 7 5
Stated Preference
Method (Patient)
My company does not use this
model/index
16 3 4

My company is evaluating this
model/index for use within our company
0 0 1
My company utilizes this model/index
for BR Assessment for internal use only
0 0 3
My company utilizes this model/index
for BR Assessment and submits it with
registration dossiers
0 0 1
My company utilizes this method for BR
Assessment and uses it to communicate
externally to prescribers and patients
0 0 0
Not aware of this framework
9 7 8

280

Table D.1: Cross Tabulation of Company Size and Models/Indices Utilization
(Cont'd)

What is the size of the company for
which you are currently working
based on number of employees?
<250
employees
250-2000
employee
s
> 2000
employees
Quality Adjusted
Time Without
Symptoms and
Toxicity (Q-TWiST)
My company does not use this
model/index
15 3 5

My company is evaluating this
model/index for use within our company
0 0 2
My company utilizes this model/index for
BR Assessment for internal use only
2 0 1
My company utilizes this model/index for
BR Assessment and submits it with
registration dossiers
0 0 0
My company utilizes this method for BR
Assessment and uses it to communicate
externally to prescribers and patients
0 0 0
Not aware of this framework
10 7 8
Minimum Clinical
Efficacy (MCE)
My company does not use this
model/index
16 3 5

My company is evaluating this
model/index for use within our company
0 0 2
My company utilizes this model/index for
BR Assessment for internal use only
2 1 3
My company utilizes this model/index for
BR Assessment and submits it with
registration dossiers
4 0 1
My company utilizes this method for BR
Assessment and uses it to communicate
externally to prescribers and patients
0 0 1
Not aware of this framework
6 6 6

281

Table D.1: Cross Tabulation of Company Size and Models/Indices Utilization
(Cont'd)

What is the size of the company for
which you are currently working
based on number of employees?
<250
employees
250-2000
employee
s
> 2000
employees
Minimal Acceptable
Risk (MAR)
My company does not use this
model/index
17 2 6

My company is evaluating this
model/index for use within our company
0 0 1
My company utilizes this model/index for
BR Assessment for internal use only
3 2 4
My company utilizes this model/index for
BR Assessment and submits it with
registration dossiers
2 0 1
My company utilizes this method for BR
Assessment and uses it to communicate
externally to prescribers and patients
0 0 1
Not aware of this framework
6 6 5

282

Table D.2: Company Size and Reason for Choosing Method

What is the size of the company for
which you are currently working
based on number of employees?

Total
<250
employees
250-2000
employees
> 2000
employees
Align with
company expertise
Not at all Important
0 1 0 1

Very Unimportant
0 0 1 1
Neither Important nor
Unimportant
5 2 6 13
Very Important
14 4 9 27
Extremely Important
10 4 2 16
Total 29 11 18 58
Align with
regulator templates
Not at all Important
1 0 0 1

Very Unimportant
1 3 0 4
Neither Important nor
Unimportant
5 0 4 9
Very Important
13 7 8 28
Extremely Important
7 1 6 14
Total 27 11 18 56
Align with
reimbursement
strategy
Not at all Important
5 0 1 6

Very Unimportant
2 0 1 3
Neither Important nor
Unimportant
7 6 8 21
Very Important
9 5 7 21
Extremely Important
5 0 1 6
Total 28 11 18 57
Align with product
development
strategy
Not at all Important
1 0 0 1

Very Unimportant
0 0 1 1
Neither Important nor
Unimportant
2 1 4 7
Very Important
19 9 12 40
Extremely Important
6 1 1 8
Total 28 11 18 57

283

Table D.3: Company Size and Framework Implementation

What is the size of the company for
which you are currently working
based on number of employees?

Total
<250
employees
250-2000
employees
> 2000
employees
US FDA BR
Framework
My company does not use
this framework
15 0 5 20

My company has/is
evaluated this for use within
our company 3 2 3 8
My company utilizes this
method for BR Assessment
for internal use only 2 1 5 8
My company utilizes this
method for BR Assessment
and submits it with
registration dossiers 4 1 2 7
My company utilizes this
method for BR Assessment
and uses it to communicate
externally to prescribers and
patients 0 0 0 0
Not aware of this framework
2 5 2 9
Total 26 9 17 52
EMA PrOACT-
URL
My company does not use
this framework
13 1 4 18

My company has/is
evaluated this for use within
our company 3 0 4 7
My company utilizes this
method for BR Assessment
for internal use only 0 0 2 2
My company utilizes this
method for BR Assessment
and submits it with
registration dossiers 1 1 1 3
My company utilizes this
method for BR Assessment
and uses it to communicate
externally to prescribers and
patients
0 0 0 0
Not aware of this framework
9 7 5 21
Total 26 9 16 51

284

Table D.3: Company Size and Framework Implementation (Cont’d)

What is the size of the company for
which you are currently working
based on number of employees?

Total
<250
employees
250-2000
employees
> 2000
employees
Unified/Universal
Method for
Benefit Risk
Assessment
(UMBRA)
My company does not use
this framework
14 1 7 22

My company has/is
evaluated this for use within
our company 1 0 2 3
My company utilizes this
method for BR Assessment
for internal use only 0 0 2 2
My company utilizes this
method for BR Assessment
and submits it with
registration dossiers 1 0 0 1
My company utilizes this
method for BR Assessment
and uses it to communicate
externally to prescribers and
patients 0 0 0 0
Not aware of this framework
10 8 5 23
Total 26 9 16 51
BRAT
Framework
My company does not use
this framework
13 1 4 18

My company has/is
evaluated this for use within
our company 2 0 4 6
My company utilizes this
method for BR Assessment
for internal use only 0 0 2 2
My company utilizes this
method for BR Assessment
and submits it with
registration dossiers 0 0 3 3
My company utilizes this
method for BR Assessment
and uses it to communicate
externally to prescribers and
patients
0 0 0 0
Not aware of this framework
10 8 3 21
Total 25 9 16 50

285

Table D.3: Company Size and Framework Implementation (Cont’d)

What is the size of the company for
which you are currently working
based on number of employees?

Total
<250
employees
250-2000
employees
> 2000
employees
Stochastic Multi-
criteria
Acceptability
Analysis(SMAA)
My company does not use
this framework
14 1 6 21

My company has/is
evaluated this for use within
our company 1 0 1 2
My company utilizes this
method for BR Assessment
for internal use only 0 0 1 1
My company utilizes this
method for BR Assessment
and submits it with
registration dossiers 0 0 0 0
My company utilizes this
method for BR Assessment
and uses it to communicate
externally to prescribers and
patients 0 0 0 0
Not aware of this framework
10 8 7 25
Total 25 9 15 49
Multi-Criteria
Decision
Analysis(MCDA)
My company does not use
this framework
16 1 6 23

My company has/is
evaluated this for use within
our company 0 0 0 0
My company utilizes this
method for BR Assessment
for internal use only 1 0 3 4
My company utilizes this
method for BR Assessment
and submits it with
registration dossiers 0 0 1 1
My company utilizes this
method for BR Assessment
and uses it to communicate
externally to prescribers and
patients 0 0 0 0
Not aware of this framework
8 8 6 22
Total 25 9 16 50

286

Table D.3: Company Size and Framework Implementation (Cont’d)

What is the size of the company for
which you are currently working
based on number of employees?

Total
<250
employees
250-2000
employees
> 2000
employees
Ashby and Smith
Framework
My company does not use
this framework
14 1 7 22

My company has/is
evaluated this for use within
our company 0 0 0 0
My company utilizes this
method for BR Assessment
for internal use only 0 0 0 0
My company utilizes this
method for BR Assessment
and submits it with
registration dossiers 0 0 0 0
My company utilizes this
method for BR Assessment
and uses it to communicate
externally to prescribers and
patients 0 0 0 0
Not aware of this framework
10 8 9 27
Total 24 9 16 49
Value or Decision
Tree
My company does not use
this framework
15 1 4 20

My company has/is
evaluated this for use within
our company 4 1 5 10
My company utilizes this
method for BR Assessment
for internal use only 4 2 2 8
My company utilizes this
method for BR Assessment
and submits it with
registration dossiers 0 0 2 2
My company utilizes this
method for BR Assessment
and uses it to communicate
externally to prescribers and
patients
0 0 0 0
Not aware of this framework
3 6 3 12
Total 26 10 16 52

287

Table D.4: Company Size and Models/Indices Utilization

What is the size of the company for
which you are currently working
based on number of employees?

Total
<250
employees
250-2000
employees
> 2000
employees
Acts as a Tool for
Communication Among
Peers within
Organization
Not at all
Important 1 0 0 1
Somewhat
important 6 3 3 12
Very Important
13 4 12 29
No Opinion
2 1 0 3

Total 22 8 15 45
Acts as a Tool for
Communication between
Organization and
External Stakeholders
(including Regulatory
Agencies)
Not at all
Important 0 0 0 0
Somewhat
important 2 3 1 6
Very Important
12 5 7 24
No Opinion
2 0 0 2
Total 16 8 8 32
Provides Archival
Documentation for
Structured Decision-
making
Not at all
Important 2 0 0 2
Somewhat
important 9 2 9 20
Very Important
12 7 4 23
No Opinion
2 0 1 3
Total 25 9 14 48
Ensures Consistency in
Quality of BR Assessment
within the Organization
Not at all
Important 2 0 0 2
Somewhat
important 7 1 4 12
Very Important
13 8 9 30
No Opinion
1 0 0 1
Total 23 9 13 45

288

Table D.4: Company Size and Models/Indices Utilization (Cont’d)

What is the size of the company for
which you are currently working
based on number of employees?

Total
<250
employees
250-2000
employees
> 2000
employees
Enhances Transparency
within the Organization
Not at all
Important 2 0 1 3
Somewhat
important 11 0 3 14
Very Important
9 8 10 27
No Opinion
2 0 0 2
Total 24 8 14 46
Ensures Consistency in
Quality of BR Assessment
by Regulators
Not at all
Important 2 0 1 3
Somewhat
important 4 4 4 12
Very Important
11 3 6 20
No Opinion
3 0 1 4
Total 20 7 12 39
Aligns Internal Decision-
making
Not at all
Important 1 0 0 1
Somewhat
important 5 0 4 9
Very Important
13 6 8 27
No Opinion
1 0 0 1
Total 20 6 12 38
Facilitates External
Communications with
Prescribers and Patients
Not at all
Important 1 2 1 4
Somewhat
important 7 0 7 14
Very Important
8 5 5 18
No Opinion
5 2 1 8
Total 21 9 14 44

289

Table D.5: Company Size and Perceived Barriers

What is the size of the company for
which you are currently working
based on number of employees?

Total
<250
employees
250-2000
employees
> 2000
employees
Resource Limitation
e.g. Manpower,
Finances
Not at all Important
2 1 2 5
Somewhat important
6 0 6 12
Very Important
5 8 7 20
Extremely Important
12 3 3 18
Cannot Answer
3 0 2 5
Total 28 12 20 60
Lack of
Knowledge/Expertise to
Execute Framework
Not at all Important
0 0 2 2
Somewhat important
2 1 3 6
Very Important
8 5 10 23
Extremely Important
15 6 3 24
Cannot Answer
3 0 2 5
Total 28 12 20 60
Resistance to Change
(Culture
Documentation or
Methodology)
Not at all Important
4 0 1 5
Somewhat important
6 3 5 14
Very Important
8 5 5 18
Extremely Important
8 4 6 18
Cannot Answer
2 0 3 5
Total 28 12 20 60
Significant Change to
Work Processes
Not at all Important
5 0 2 7
Somewhat important
5 4 7 16
Very Important
9 6 9 24
Extremely Important
7 2 0 9
Cannot Answer
2 0 2 4
Total 28 12 20 60
Significant Retraining
of Staff Required
Not at all Important
4 0 3 7
Somewhat important
7 4 6 17
Very Important
9 6 8 23
Extremely Important
6 2 1 9
Cannot Answer
2 0 2 4
Total 28 12 20 60

290

Table D.5: Company Size and Perceived Barriers (Cont’d)

What is the size of the company for
which you are currently working
based on number of employees?

Total
<250
employees
250-2000
employees
> 2000
employees
Support from Senior
Management Required
Not at all Important
5 0 3 8
Somewhat important
0 1 3 4
Very Important
10 5 5 20
Extremely Important
11 6 7 24
Cannot Answer
2 0 2 4
Total 28 12 20 60
Lack of an Industry
Accepted Framework
Not at all Important
1 0 1 2
Somewhat important
4 3 4 11
Very Important
8 6 8 22
Extremely Important
11 3 4 18
Cannot Answer
4 0 3 7
Total 28 12 20 60
Lack of a Regulatory
Harmonized
Framework
Not at all Important
0 0 1 1
Somewhat important
3 3 3 9
Very Important
8 4 7 19
Extremely Important
13 5 6 24
Cannot Answer
4 0 3 7
Total 28 12 20 60

291

Table D.6: Company Size and ICH M4E (R2)

What is the size of the company for which you are
currently working based on number of employees?

Total
< 250 employees
250-2000
employees
>2000
employees
FDA BR Framework
11 4 9 24
EMA PrOACT-URL
8 3 7 18
Unified/Universal Method
for Benefit Risk
Assessment (UMBRA) 4 3 3 10
BRAT Framework
1 1 6 8
Multi-Criteria Decision
Analysis 1 2 2 5
Stochastic Multi-criteria
Acceptability Analysis 0 1 0 1
Ashby and Smith
Framework 0 1 0 1
Value/Decision Tree(s)
1 2 4 7
An Independent
Harmonized Approach 8 3 4 15
Other
8 3 3 14
None of the Above
2 1 0 3
Total 29 10 20 59

292

Table D.7: Size of Company and Stakeholder Involvement

What is the size of the company for
which you are currently working based
on number of employees?

Total
< 250
employees
250-2000
employees
> 2000
employees
Prescribers/Physicians
8 6 5 19
Patients
0 1 4 5
Payers (e.g. HTA or Reimbursement
Providers) 2 2 2 6
If yes, please briefly describe how you
have engaged them in the space below 5 2 2 9
No we have not engaged external
stakeholder in our benefit-risk
assessment process 14 4 6 24
I do not know
5 2 7 14
Total 29 12 20 61

293

Table D.8: Size of Company and Feedback

What is the size of the company for which
you are currently working based on
number of employees?

Total
< 250
employees
250-2000
employees
>2000
employees
At Pre-submission Meeting (e.g.
preNDA/MAA) 6 1 4 11
As Part of Marketing Application Filing
Review 5 1 4 10
As Part of a Question during Marketing
Application Content Review
8 2 3 13
As Part of Labeling Negotiations
6 1 5 12
As Part of Formal Agency Communication
at the End of Review
3 1 4 8
No, no Feedback Provided
15 8 7 30
Other - Please specify
1 1 3 5
Additional Details (optional)
2 0 2 4
As Part of Discussions with Regulators
Regarding Development Plan (e.g. Scientific
Advice)
6 0 6 12
Total 30 12 20 62

294

Table D.9: Size of Company and Where BR Assessment is Used

What is the size of the company for
which you are currently working
based on number of employees?

Total
<250
employees
250-2000
employees
> 2000
employees
As Part of Labeling Development (e.g.
Target Product Profile/TPP) 16 7 11 34
To Develop Package Insert/Prescribing
Information (PI) 17 5 11 33
As Part of the Clinical Overview (CTD
M2.5.6)
25 8 17 50
Other - Please Specify
4 4 4 12
Total 28 10 18 56

295

Table D.10: Size of Company and Most Important Barrier

What is the size of the company for which
you are currently working based on
number of employees?

<250
employees
250-2000
employees
>2000
employees
Total
Resource Limitation e.g. Manpower,
Finances 6 1 2 9
Lack of Knowledge/Expertise to Execute
Framework 4 6 2 12
Resistance to Change (Culture
Documentation or Methodology) 0 0 2 2
Significant Change to Work Processes
1 1 1 3
Significant Retraining of Staff Required
1 0 0 1
Support from Senior Management
Required 2 2 4 8
Lack of an Industry Accepted
Framework 0 0 4 4
Lack of a Regulatory Harmonized
Framework 10 1 4 15
Total 24 11 20 55

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Experience with breakthrough therapy designation: an industry survey

PDF

Challenges in the implementation of Risk Evaluation Mitigation Strategies (REMS): a survey of industry views

PDF

Contract research organizations: a survey of industry views and outsourcing practices

PDF

Examining the regulatory framework for drug compounding: industry views and experiences

PDF

Regulatory dissonance in the global development of drug therapies: a case study of drug development in postmenopausal osteoporosis

PDF

The impact of FDA Patient-Focused Drug Development (PFDD) on US drug development strategies: a survey of views from pharmaceutical product companies

PDF

Risk approaches and standards used in hospitals: a survey of industry views

PDF

An integrated framework to evaluate customer service delivery: a study of electronic systems at FDA's Los Angeles Import Operations Branch

PDF

Examining the cord blood industry views on the biologic license application regulatory framework

PDF

Value based purchasing: decision-making processes underlying hospital acquisitions of orthopedic devices

PDF

Risk management and recalls: a survey of medical device manufacturers

PDF

Regulating cosmeceuticals in the United States: a cosmetic industry view

PDF

Sharing the results of clinical trials: industry views on disclosure of data from industry-sponsored clinical research

PDF

Validation master plans: progress of implementation within the pharmaceutical industry

PDF

A survey of US industry views on implementation of decentralized clinical trials

PDF

Regulation of pediatric cancer drug development: an industry perspective

PDF

Regulatory agreements for drug development collaborations: practices in the medical products industry

PDF

An industry survey of implementation strategies for clinical supply chain management of cell and gene therapies

PDF

Design control for software medical devices: an industry survey of views and experiences

PDF

Evaluation of FDA-sponsor formal meetings on the development of cell and gene therapies: a survey of industry views

Asset Metadata

Creator Pire-Smerkanich, Nancy J. (author)

Core Title Benefits-risk frameworks: implementation by industry

School School of Pharmacy

Degree Doctor of Regulatory Science

Degree Program Regulatory Science

Publication Date 02/10/2016

Defense Date 11/20/2015

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag benefits-risk frameworks,drug development,OAI-PMH Harvest,pharmaceutical industry

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Richmond, Frances J. (committee chair), Berger, Elliott (committee member), Jamieson, Michael (committee member), Pacifici, Eunjoo (committee member)

Creator Email nancysmerkanich@gmail.com,piresmer@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c40-207725

Unique identifier UC11279048

Identifier etd-PireSmerka-4098.pdf (filename),usctheses-c40-207725 (legacy record id)

Legacy Identifier etd-PireSmerka-4098.pdf

Dmrecord 207725

Document Type Dissertation

Format application/pdf (imt)

Rights Pire-Smerkanich, Nancy J.

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA