Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
An integrated framework to evaluate customer service delivery: a study of electronic systems at FDA's Los Angeles Import Operations Branch
(USC Thesis Other)
An integrated framework to evaluate customer service delivery: a study of electronic systems at FDA's Los Angeles Import Operations Branch
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
AN INTEGRATED FRAMEWORK TO EVALUATE CUSTOMER SERVICE DELIVERY:
A STUDY OF ELECTRONIC SYSTEMS AT
FDA'S LOS ANGELES IMPORT OPERATIONS BRANCH
by
Patrick Raul Dimapindan
A Dissertation Presented to the
FACULTY OF THE USC SCHOOL OF PHARMACY
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF REGULATORY SCIENCE
December 2014
Copyright 2014 Patrick Raul Dimapindan
3
DISCLAIMER
This student dissertation has been developed as part of the requirements of the
Doctor of Regulatory Science program at the University of Southern California. The
views expressed are solely those of the student, and do not necessarily represent the
views of the US Food and Drug Administration, other agencies or the United States.
4
DEDICATION
I dedicate this to my parents,
who have sacrificed relentlessly to give me a fighting chance in this world.
FIGHT ON!
5
ACKNOWLEDGEMENTS
“The hardest thing about writing is writing.” – Nora Ephron
I acknowledge and thank every person who has helped this endeavor reach fruition.
• My dissertation committee: Michael Hamrell, PhD; Benson Kuo, PhD; Eunjoo
Pacifici, PharmD, PhD; and Frances Richmond, PhD
• Regulatory Science Program Faculty and Staff (past and present), particularly:
Erin Chow; Randa Issa, PhD; Michael Jamieson, DRSc; and Gerald Loeb, MD
• Mentors and academic proponents: Therese Fitzgerald; Kellie Ann Moore, Esq.;
and Matteo Villain, PhD
• FDA Import Operations Leadership (Headquarters and Los Angeles District)
• The Los Angeles Customs Brokers and Freight Forwarders Association
• The 2010 DRSc Cohort
• My parents, grandmothers, and siblings
• My friends
6
TABLE OF CONTENTS
DISCLAIMER .................................................................................................................... 3
DEDICATION .................................................................................................................... 4
ACKNOWLEDGEMENTS ................................................................................................ 5
LIST OF TABLES ............................................................................................................ 10
LIST OF FIGURES .......................................................................................................... 12
ABSTRACT ...................................................................................................................... 16
CHAPTER 1:
OVERVIEW ........................................................................................... 18
1.1
Introduction ....................................................................................................... 18
1.2
Statement of the Problem .................................................................................. 24
1.3
Purpose of the Study ......................................................................................... 24
1.4
Importance of the Study .................................................................................... 26
1.5
Limitations, Delimitations and Assumptions .................................................... 28
1.6
Organization of Thesis ...................................................................................... 29
CHAPTER 2:
LITERATURE REVIEW ....................................................................... 30
2.1
Introduction ....................................................................................................... 30
2.2
Policy Initiatives in Government to Promote Customer Service ...................... 31
2.3
Regulation of US Imports ................................................................................. 33
2.3.1
Regulatory Overview of US Imports ............................................................. 33
2.3.2
FDA’s Regulation of US Imports .................................................................. 35
7
2.4
Regulation of US Imports ................................................................................. 37
2.4.1
History of Electronic Processing of Import Entries at FDA .......................... 37
2.4.2
Electronic Processing of Import Entries at FDA Headquarters ..................... 39
2.4.3
Electronic Processing of Entries at FDA LOS IOB ....................................... 41
2.5
Review of Electronic Systems at FDA LOS IOB ............................................. 46
2.5.1
PAR LOS LAImports (LAImports) ............................................................... 46
2.5.2
Import Trade Auxiliary Communication System (ITACS) ........................... 46
2.6
Policy Assessment for Customer Service ......................................................... 50
2.6.1
Defining and Measuring Satisfaction ............................................................. 50
2.6.2
Frameworks for Assessing Customer Satisfaction ........................................ 51
2.7
Choosing a Framework to Evaluate Electronic Systems at FDA LOS IOB ..... 54
2.7.1
Exploring Available Tools to Evaluate Customer Satisfaction ..................... 54
2.7.2
Loyalty Factors that Contribute to Customer Preference ............................... 62
2.8
Customer Service Evaluation of FDA Imports Systems ................................... 64
2.8.1
Current Methods for LAImports and ITACS ................................................. 64
2.8.2
Proposed Research ......................................................................................... 64
CHAPTER 3:
METHODOLOGY ................................................................................. 66
3.1
Introduction ....................................................................................................... 66
3.2
Greater Los Angeles Import/Export Trade Associations .................................. 67
3.2.1
Los Angeles Customs Brokers and Freight Forwarders Association
(LACBFFA) .................................................................................................. 67
8
3.2.2
Foreign Trade Association of Southern California (FTASC) ........................ 68
3.3
Development of Initial Survey .......................................................................... 68
3.4
Survey Deployment and Analysis ..................................................................... 71
CHAPTER 4:
RESULTS ............................................................................................... 73
4.1
Results of the Focus Group ............................................................................... 73
4.2
Survey Results .................................................................................................. 74
4.2.1
General Information About Respondents ...................................................... 77
4.2.2
Study Framework Principles and Dimensions ............................................... 80
4.2.3
Executive Order (EO) 13571 Metrics ............................................................ 91
4.2.4
HowTo.gov Metrics ....................................................................................... 93
4.2.5
Preference and Loyalty Factors ..................................................................... 97
4.2.6
Cross Tabulations ......................................................................................... 100
CHAPTER 5:
DISCUSSION ....................................................................................... 105
5.1
Introduction ..................................................................................................... 105
5.2
Considerations of Limitations, Delimitations, and Assumptions ................... 105
5.3
Consideration of Results ................................................................................. 111
5.3.1
Characterization and Comparison of Customer Service for LAImports and
ITACS ......................................................................................................... 112
5.3.1.1
Areas of Positive Response ......................................................................... 112
5.3.1.2
Areas of Negative Response ....................................................................... 113
5.3.1.3
Areas of Intermediate Response .................................................................. 116
9
5.3.1.4
Other Interesting Findings ........................................................................... 118
5.3.2
System Preferences and Loyalty for LAImports and ITACS ...................... 119
5.3.3
Adequacy of Study Framework as a Customer Service Tool ...................... 120
5.4
Conclusions and Considerations for the Future .............................................. 123
5.4.1
Justification for LAImports and ITACS ...................................................... 124
5.4.2
Recommendations to Improve LAImports and ITACS ............................... 124
5.4.3
Future Research ........................................................................................... 126
REFERENCES ............................................................................................................... 128
APPENDIX A ................................................................................................................. 136
APPENDIX B ................................................................................................................. 138
APPENDIX C ................................................................................................................. 141
APPENDIX D ................................................................................................................. 145
APPENDIX E ................................................................................................................. 153
APPENDIX F ................................................................................................................. 157
10
LIST OF T ABLES
Table 1:
Performance Criteria for Analysis of LAImports and ITACS ......................... 47
Table 2:
Dimensions of Customer Satisfaction for Human-to-Human and Human-to-
Machine Service Interactions (Lin & Hsieh, 2011; Parasuraman et al.,
1991; Parasuraman et al., 1988) ................................................................ 56
Table 3:
Customer Satisfaction Dimensions for Core and Auxiliary Principles (Croley,
2011; Lin & Hsieh, 2011; Obama, 2009; Parasuraman et al., 1991;
Parasuraman et al., 1988) .......................................................................... 60
Table 4:
HowTo.gov and EO 13571 Metrics for Customer Service Evaluation ........... 62
Table 5:
Survey Question Mapping to Dimensions, Factors, and Metrics .................... 70
Table 6:
“If you could customize the systems, what would you change?” (Q21: n=7) . 83
Table 7:
“How satisfied are you with each system’s ability to meet your expectations?
LAImports? ITACS? Please enter any additional comments for either
system below.” (Q25: n=2) ....................................................................... 86
Table 8:
“Please select your level of agreement with the following statements: FDA has
developed a stronger partnership with industry through LAImports; FDA
has developed a stronger partnership with industry through ITACS.
Please enter any additional comments for either system below.” (Q15:
n=2) ........................................................................................................... 90
Table 9:
“How likely are you to recommend the following systems using positive terms
to colleagues: LAImports? ITACS? Please enter any additional comments
for either system below.” (Q9: n=1) ......................................................... 96
Table 10:
“Do you have any additional comments on your overall customer
experience?” (Q29: n=6) .......................................................................... 97
Table 11:
“Which system do you prefer to use for the following transactions with the
FDA Los Angeles Import Operations Branch: Submitting Entry
Documents? Submitting Location/Availability Information? Requesting
Entry Status? Please enter any additional comments for either system
below.” (Q6: n=4) ..................................................................................... 98
Table 12:
“Do you have any additional comments, suggestions, or recommendations?”
(Q33: n=6) ................................................................................................. 99
11
Table 13:
Cross Tabulation of Q6 on Continue Use of LAImports with Q8 Operation-
specific Preferences (L=LAImports; I=ITACS; NE=Non-electronic;
NP=No Preference; Σ=Total) .................................................................. 101
Table 14:
Cross Tabulation of Q6 on Continue Use of ITACS with Q8 on Operation-
specific Preferences (L=LAImports; I=ITACS; NE=Non-electronic;
NP=No Preference; Σ=Total) .................................................................. 101
Table 15:
Cross Tabulation of Q6 on Turnaround Times with Q28 on Overall Customer
Experience for LAImports ...................................................................... 102
Table 16:
Cross Tabulation of Q6 on Turnaround Times with Q28 on Overall Customer
Experience for ITACS ............................................................................ 103
Table 17:
Cross Tabulation of Q28 on Overall Customer Experience with Q25 on
Meeting Expectations for LAImports ..................................................... 104
Table 18:
Cross Tabulation of Q28 on Overall Customer Experience with Q25 on
Meeting Expectations for ITACS ........................................................... 104
12
LIST OF FIGURES
Figure 1:
Timeline of Public Policy Initiatives in Customer Service ............................ 33
Figure 2:
Diagram of Importation Process for Fresh Produce ....................................... 36
Figure 3:
Import Entry Lines Electronically Screened by FDA Since FY2003 (FDA,
2014b) ....................................................................................................... 37
Figure 4:
Electronic Transaction of Import Entries from Filing through PREDICT
Screening. Modified from FDA (FDA, 2011d) ........................................ 40
Figure 5:
Processes associated with LOS IOB .............................................................. 42
Figure 6:
Framework for Customer Service Evaluation ................................................ 59
Figure 7:
Number of Surveys Collected per Date .......................................................... 75
Figure 8:
Survey Question Response Rate for 57 Total Respondents ........................... 76
Figure 9:
“What is the primary classification of your firm’s business operation?” (Q1:
n=50) ......................................................................................................... 77
Figure 10:
“When did you begin working on FDA-regulated entries?” (Q2: n=50) .... 77
Figure 11:
“How many total transactions do you estimate making with LAImports and
ITACS in an average work week?” (Q5: LAImports n=45; ITACS n=43)
................................................................................................................... 78
Figure 12:
“Which entry commodity do you handle most?” (Q3: n=50) and “Which
three (3) entry commodities do you handle most?” (Q31: n=37) ............. 79
Figure 13:
“Which import geographical region of origin do you handle most?” (Q4:
n=50) and “Which three (3) import geographical regions of origin do you
handle most?” (Q32: n=37) ...................................................................... 79
Figure 14:
“How often is each system available when you need it?” (Q17: LAImports
n=27, n/a=4; ITACS n=30, n/a=2) ............................................................ 81
Figure 15:
“How satisfied are you with the level of computer security when you
transmit information through: LAImports? ITACS?” (Q18: LAImports
n=28, n/a=4; ITACS n=29, n/a=3) ............................................................ 81
13
Figure 16:
“How satisfied are you with the user interface of: LAImports? ITACS?”
(Q19: LAImports n=29, n/a=4; ITACS n=29, n/a=3) ............................... 82
Figure 17:
“How satisfied are you with the level of system customization for:
LAImports? ITACS?” (Q20: LAImports n=27, n/a=5; ITACS n=29, n/a=4)
................................................................................................................... 82
Figure 18:
“How satisfied are you with the completeness of information provided by:
LAImports? ITACS?” (Q22: LAImports n=28, n/a=5; ITACS n=29, n/a=3)
................................................................................................................... 84
Figure 19:
“How satisfied are you with the relevance of information provided by:
LAImports? ITACS?” (Q22: LAImports n=28, n/a=5; ITACS n=30, n/a=3)
................................................................................................................... 85
Figure 20:
“How often do you get turnaround time you need from each system?” (Q24:
LAImports n=26, n/a=5; ITACS n=30, n/a=3) ......................................... 85
Figure 21:
“How satisfied are you with each system’s ability to meet your
expectations?” (Q25: LAImports n=27, n/a=5; ITACS n=29, n/a=3) ..... 86
Figure 22:
“How satisfied are you with the instructions for use of: LAImports?
ITACS?” (Q10: LAImports n=31, n/a=2; ITACS n=31, n/a=3) ............... 87
Figure 23:
“How satisfied are you with FDA’s explanation of services provided by:
LAImports? ITACS?” (Q11: LAImports n=29, n/a=4; ITACS n=30, n/a=3)
................................................................................................................... 88
Figure 24:
“How satisfied are you with FDA’s explanation of processing times for:
LAImports? ITACS?” (Q12: LAImports n=29, n/a=4; ITACS n=31, n/a=3)
................................................................................................................... 88
Figure 25:
“How satisfied are you with the communication of FDA entry decisions
through: LAImports? ITACS?” (Q13: LAImports n=30, n/a=4; ITACS
n=30, n/a=4) .............................................................................................. 89
Figure 26:
“How satisfied are you interacting with the FDA through: LAImports?
ITACS?” (Q14: LAImports n=29, n/a=3; ITACS n=31, n/a=2) ............... 89
Figure 27:
“Please select your level of agreement with the following statements: FDA
has developed a stronger partnership with industry through LAImports;
FDA has developed a stronger partnership with industry through ITACS.
Please enter any additional comments for either system below.” (Q15:
LAImports n=31, n/a=4; ITACS n=31, n/a=3) ......................................... 90
14
Figure 28:
“How satisfied are you with the ability to provide the FDA with feedback
on: LAImports? ITACS?” (Q16: LAImports n=27, n/a=5; ITACS n=30,
n/a=4) ........................................................................................................ 91
Figure 29:
“If you use LAImports: 1) Have you reduced telephone calls to FDA LOS
IOB? Have you reduced faxes to the FDA LOS IOB? Have you reduced
postal mail to the FDA LOS IOB? Have you reduced courier delivery to
FDA LOS IOB? Have you reduced paper consumption? Have you reduced
company spending? Have fewer documents been lost since using the
system? Has total entry processing time decreased since using the
system?” (Q26: n=30) ............................................................................... 92
Figure 30:
“If you use ITACS: Have you reduced telephone calls to FDA LOS IOB?
Have you reduced faxes to the FDA LOS IOB? Have you reduced postal
mail to the FDA LOS IOB? Have you reduced courier delivery to FDA
LOS IOB? Have you reduced paper consumption? Have you reduced
company spending? Have fewer documents been lost since using the
system? Has total entry processing time decreased since using the
system?” (Q27: n=30) ............................................................................... 93
Figure 31:
“How often are you able to complete your intended transactions with:
LAImports? ITACS?” (Q7: LAImports n=34, n/a=3; ITACS n=36, n/a=3)
................................................................................................................... 94
Figure 32:
“How satisfied are you with continuing to use: LAImports? ITACS?” (Q8:
LAImports n=35, n/a=3; ITACS n=36, n/a=3) ......................................... 95
Figure 33:
“How likely are you to recommend the following systems using positive
terms to colleagues: LAImports? ITACS? Please enter any additional
comments for either system below.” (Q9: LAImports n=36, n/a=3; ITACS
n=36, n/a=4) .............................................................................................. 95
Figure 34:
“How satisfied are you with your overall customer experiences with:
LAImports? ITACS?” (Q28: LAImports n=28, n/a=3; ITACS n=30, n/a=1)
................................................................................................................... 96
Figure 35:
“Which system do you prefer to use for the following transactions with the
FDA Los Angeles Import Operations Branch: Submitting Entry
Documents? Submitting Location/ Availability Information? Requesting
Entry Status? Please enter any additional comments for either system
below.” (Q6: n=50) ................................................................................... 98
Figure 36:
“Please select the three (3) service areas below that are most important to
you when using electronic systems at the FDA Los Angeles Import
Operations Branch.” (Q30: n=34) ............................................................ 99
15
Figure 37:
Positive Response-Importance Priority Plot for Service Areas .................. 122
16
ABSTRACT
FDA plays a critical role in imports management to ensure the regulatory
compliance of foreign health products entering United States commerce. Entry of these
products, including foods, tobacco, veterinary, medicinal and medical products, has
tripled in volume in the last decade, placing considerable strain on imports management
infrastructure. According to the FDA, more than 99% of entries, or import shipments
offered for admission into the US, are filed and processed electronically. Therefore,
implementation of a functional, reliable, and user-friendly IT infrastructure to manage
import service transactions may be an important strategy for the FDA to perform its
regulatory function effectively while providing the public with timely access to imported
products.
This study examines the level of customer service provided by the electronic
transaction systems in place at FDA’s Los Angeles Import Operations Branch (LOS
IOB), specifically, the PAR LOS LAImports system (LAImports) and Import Trade
Auxiliary Communication System (ITACS), via mixed methods approach of document
analysis and survey distribution. Public policy documents for the two systems were
examined to identify functional attributes associated with customer service to be
incorporated into the study. Methodologies used previously to evaluate customer service
in a range of industries were assessed for their relevance for the proposed research and
were blended to compile a comprehensive framework tailored to assess and compare user
satisfaction with the LAImports and ITACS systems. Key dimensions in this framework
include not only customer service elements typical for any service organization but also
17
the government mandated dimensions of transparency, collaboration and public
participation. This framework was used to develop a survey for the key industry
stakeholders—users of LAImports and ITACS—in order to obtain feedback on the
customer service delivery provided by the two systems.
The framework and the subsequent survey effectively provide a snapshot of
customer service delivered by FDA LOS IOB through LAImports and ITACS. The
results show that while users were most positive toward areas related to the IT
infrastructure of the two systems, they were much less positive toward areas related to
timeliness of service delivery and associated communications with the agency. These
low scoring areas were also identified as those that are most important to the respondents
and, therefore, point to future opportunities for improvement.
18
CHAPTER 1: OVERVIEW
1.1 Introduction
The economy of the United States relies increasingly on international trade.
According to the Fiscal Year 2012 Year-End Report from US Customs and Border
Protection (CBP), 126 million entry lines were imported into the US in 2012, with an
estimated total value of $2.38 trillion USD (CBP, 2013). To handle this volume of
goods, import management has become a sophisticated and concerted effort amongst
several key agencies. Of these, CBP bears the primary administrative responsibility for
processing imports and duties, and shares the responsibility for ensuring the safety of
imported goods with other agencies. For example, the US Food and Drug Administration
(FDA) works with CBP by reviewing entries that fall into the product classes critical to
health including foods, housewares and food-related items, medical devices and
radiation-emitting electronics, drugs and biologics, animal drugs and feeds, cosmetics,
and tobacco products. Altogether, FDA regulates imports from over 300,000
manufacturers in more than 150 countries who supply their products through 300 US
ports to approximately 130,000 stateside importers (FDA, 2011f). In 2012, the oversight
of 26.7 million entry lines, representing nearly 21% of all imports and more than triple
the volume seen ten years prior, was primarily vested in FDA (FDA, 2012e).
When millions of imported products are subject to rules and review by multiple
departments within multiple agencies, internal and external documentation and
communication processes can become burdensome and bureaucratic. Those ports that
can efficiently and effectively perform the entry, review, and release functions will be
19
successful in attracting importers by expediting delivery of goods to their final
destination. According to the Journal of Commerce, the world’s top container ports are:
1) Shanghai, China, 2) Singapore, Singapore, 3) Hong Kong, China, 4) Shenzhen, China,
and 5) Busan, South Korea (JOC, 2012). The rise of Chinese and Korean ports may be
credited to their superior port service, which emphasizes “prompt response,” “zero
waiting time service,” and most importantly, “24 hours/seven days a week service” (Yeo,
Roe, & Dinwoodie, 2008). The port of Singapore, which has been described as the
“world’s busiest transshipment hub,” has relied heavily on the use of information
technology since the 1960s to improve efficiency and provide high levels of customer
service (JOC, 2012; Wan, Wah, & Meng, 1992). The current approaches of leading
world ports, which highlight IT infrastructure and customer service, are also seen in the
evolving policy and operations affecting FDA regulation at US ports.
It is estimated that more than 99% of the entries reviewed by FDA are transmitted
electronically via CBP’s Automated Broker Interface (ABI) (Solis, 2010). Prior to 2010,
however, the electronic handling of information was limited to the initial entry step and
the subsequent exchanges of information involved lengthy and cumbersome paper
transactions. Importers had to submit their documents to FDA via postal mail, parcel
service, courier delivery, or facsimile, and FDA in turn sent correspondence via postal
mail. The inherent time, resource, and ecological issues associated with each of these
delivery methods made the FDA entry process slow and inefficient. The FDA Office of
Global Regulatory Operations and Policy, Office of Regulatory Affairs, Pacific Region,
Los Angeles District, Import Operations Branch, also referred to as OGROP, ORA, PAR,
20
LOS, IOB, or FDA LOS IOB, therefore established a system called PAR LOS
LAImports, or LAImports, during fiscal year 2010 in an attempt to reduce these
problems, streamline service delivery and increase transparency. This was followed in
2012 by the launch of another electronic interface called Import Trade Auxiliary
Communications System, or ITACS. These two systems, PAR LOS LAImports and
ITACS, will be discussed below in further detail.
PAR LOS LAImports is an email account, LAImports@fda.hhs.gov, designed to
serve as an alternate means for industry to submit documentation and communicate with
FDA (FDA, 2012d). This email account accepts portable document format (pdf) versions
of import documentation and records, and responds to industry requests for FDA Notices
of Action and import entry status (FDA, 2012d). LAImports is monitored by a trained
cadre of FDA Investigators and is used to route all incoming emails and documents to
appropriate FDA personnel. The system targets to provide industry with FDA Notices of
Action and entry status updates within one business day of email receipt (FDA, 2012d).
This turnaround response time is a great improvement over previous systems and is one
of the reasons that LAImports has become the industry’s primary means of
communication with FDA LOS IOB. In fact, less than a year after the launch of
LAImports, public comments from the industry have recommended that similar email
accounts be established at other FDA district import operations. Since then, status email
accounts have been rolled out nationally and the email addresses of all the local import
operations are available on the FDA webpage (FDA, 2013a).
In March 2012, FDA’s Division of Import Operations and Policy, located in
Rockville, MD, launched the Import Trade Auxiliary Communications System, or ITACS
21
to further enhance FDA’s import management capabilities. ITACS is an internet portal,
www.itacs.fda.gov, which allows industry to check entry status, upload entry
documentation, and submit location information for FDA inspections of imported goods
(FDA, 2012c). With LAImports and ITACS, FDA is working to shift importation
management activities toward a truly electronic paradigm.
The development of electronic systems by FDA and its Los Angeles import
operations reflects a growing shift toward electronic government, or e-government, seen
over the last several decades not only in the US but also around the world. E-government
essentially uses “information technology to enable and improve the efficiency with which
government services are provided to citizens, employees, business and agencies” (Carter
& Bélanger, 2005). This definition has evolved to include both information and
communication technologies (Carter & Bélanger, 2005; Chun, Shulman, Sandoval, &
Hovy, 2010).
The US is not the only, or even the first, country to implement on-line public
management systems. The Chinese government launched its Government Online Project
in 1999, and within seven years, 80% of local and national agencies of China had
established websites (Lollar, 2006). The Canadian government launched its Government
Online initiative, seeking to transform Canada into “the most connected nation on earth”
(Marche & McNiven, 2003). According to the Cyber Policy Research Group, there are
168 national governments with their own websites (Wong & Welch, 2004). The United
States government enacted the E-Government Act of 2002 “to enhance the management
and promotion of electronic government services and processes” ("E-Government Act of
2002," 2002). This follows president George W. Bush’s implementation of the e-
22
government strategy with three goals: 1) “Increase the ease of access for citizens;” 2)
“Increase efficiency/effectiveness of government;” and 3) “Increase government
responsiveness to citizens” (Evans & Yen, 2005). Since then, federal, state, and local
governmental agencies have all moved to transform traditional paper based activities to
electronic processes.
Efforts to improve government services continued beyond the Bush
administration. On his first full day in office, January 21, 2009, President Barack Obama
issued a Memorandum on Transparency and Open Government outlining his
administration’s commitment to create an “unprecedented level of openness in
Government” through establishment of a system of transparency, public participation,
and collaboration (Obama, 2009). These three principles aim to 1) promote and improve
accountability and effectiveness in decision-making, 2) strengthen Americans’ role in the
Government, and 3) bolster democracy (Obama, 2009). Therefore, all governmental
processes established to interact with the public should be assessed for their alignment
within the above framework.
As a follow up to the above memorandum, the White House issued the Open
Government Directive requiring federal agencies to: 1) publish government information
online; 2) improve the quality of government information; 3) create and institutionalize a
culture of open government; and 4) create an enabling policy framework for a more open
government (Orszag, 2009b). All government programs are statutorily subject to analysis
and evaluation annually under the Government Performance and Results Act of 1993
(GPRA) and quarterly under the GPRA Modernization Act of 2010. These are in
addition to various other memoranda, directives, and executive orders relating to program
23
evaluation, effectiveness, efficiency, government accountability, and customer service
(Jacob J. Lew, 2011; Jacob J Lew & Zients, 2011; Metzenbaum, 2010; Obama, 2011a,
2011b; Orszag, 2009a, 2009b; Zients, 2010).
The concept of viewing users of government services as customers is a relatively
recent development. The Executive Order 13571, issued on April 27, 2011, builds on
activities previously initiated by President Bill Clinton to call on federal agencies to
develop a Customer Service Plan directly linked to a technology initiative (Clinton, 1993,
1995, 1998; Obama, 2011a). The executive order provides the following six streamlining
and improvement activities:
a) Establishing one major initiative (signature initiative) that will use
technology to improve the customer experience
b) Establishing mechanisms to solicit customer feedback on Government
services and using such feedback regularly to make service improvements
c) Setting clear customer service standards and expectations, including,
where appropriate, performance goals for customer service required by the
GPRA (Government Performance and Results) Modernization Act of 2010
(Public Law 111-352)
d) Improving the customer experience by adopting proven customer service
best practices and coordinating across service channels (such as online,
phone, in-person, and mail services)
e) Streamlining agency processes to reduce costs and accelerate delivery,
while reducing the need for customer calls and inquiries; and
24
f) Identifying ways to use innovative technologies to accomplish the
customer service activities above, thereby lowering costs, decreasing
service delivery times, and improving the customer experience. (Obama,
2011a).
1.2 Statement of the Problem
According to the above executive order, LAImports and ITACS can be looked
upon as FDA LOS IOB’s signature initiative where technology is used to improve the
customer experience. However, neither of these systems has undergone formal study,
making it difficult to evaluate and assess processes that collectively serve one of the
largest FDA import operations in the US.
1.3 Purpose of the Study
The goal of this study was to evaluate the customer service delivered by FDA
LOS IOB by using a framework developed based on the two key guiding documents, the
“Memorandum on Transparency and Open Government” and the “Executive Order
13571 on Streamlining Service Delivery and Improving Customer Service” in the context
of customer service elements typical for private sectors that incorporate customer service
delivery in their business models. In these documents, customer service is anticipated to
be at the core of satisfaction in three key areas: transparency, public participation, and
collaboration. This approach is consistent with the working definition of a framework as
a results-driven structure comprised of contributing factors like goals and strategies,
policies, organization and culture, relationship contracts and arrangements, business
processes, roles, tools, systems, objectives, and measures and incentives (Sadri, 2012).
25
An integrated framework, hence, was developed and used to evaluate the satisfaction
with customer service experienced by the users of the two main electronic systems
employed at FDA LOS IOB, specifically the LAImports and ITACS systems, which can
be viewed as FDA LOS IOB’s signature initiative, as per Executive Order 13571.
The research carried out was based on a mixed methods approach of document
analysis and survey distribution. The study began with the examination of available
public documents to examine the two electronic import initiatives in detail. For
LAImports, FDA LOS IOB has developed and disseminated its policy document, PAR
LOS LAImports Email Guidelines whereas, for ITACS, the FDA headquarters has
developed and posted online programmatic documentation. This document analysis
served as a primer for identifying customer service elements that are specified in the
documents and also those that are not included but should be further examined.
Methodologies available to evaluate these initiatives in terms of customer service
standards and expectations were then reviewed.
A survey, developed and administered to industry participants, was used to
furnish data regarding effectiveness of LAImports and ITACS in achieving their explicit
goals of streamlining communication and increasing transparency between FDA and
industry. Of particular focus was the level of customer satisfaction where the “customer”
is the importer and the FDA, to a certain extent, is the “service provider/partner”. The
survey sought feedback on FDA’s level of customer service through LAImports and
ITACS within the framework developed around transparency, public participation, and
collaboration in the service delivery process.
26
1.4 Importance of the Study
This study is significant to the field of public management in that it scales
programs and performance appraisals down to a more granular level of governmental
activities and service delivery. Building granularity in program analysis and performance
appraisal is important to federal agencies, especially for evaluating pilot programs that
are often established at one location and then subsequently rolled out nationally, as was
the case with LAImports. Less than a year after the launch of LAImports, public
comments from the industry recommended expanding the system by establishing an
email address for each of the other FDA Districts (FDA, 2011a). Within two years
following the launch of LAImports, the FDA headquarters launched ITACS to enhance
its import management capabilities. ITACS is an Internet portal with some of the
functionalities that overlap those of LAImports. If there had been mechanisms in place to
capture the lessons learned from LAImports, those findings could have been used in the
development of ITACS to enhance its functionality and minimize potential dissonance
between the two systems.
Program analysis and evaluation are, by law, required elements of public
management, but there is high degree of flexibility built into the rules. For example,
statutes such as GPRA and the GPRA Modernization Act make provisions for agencies to
cluster activities, rather than to assess each individual program. Executive memorandum
M-11-24, Memorandum on Implementing Executive Order 13571, makes provisions for
high level Cabinet Departments, but has no specific assessment requirements for agencies
within these departments. Therefore, the Department of Health and Human Services
(HHS) has developed and posted its Customer Service Plan online, but no such plan
27
exists for the FDA. Four years after implementation of LAImports and two years after
implementation of ITACS, FDA has no apparent obligation to gather data in order to
justify continued support of the two systems.
According to the 2012 FDA California Fact Sheet, “California receives an
estimated 25-30 percent of all FDA regulated commodities imported into the US and
contains the largest harbor complex in the country” (FDA, 2011b). The referenced
harbor complex is the Los Angeles/Long Beach seaport, monitored by FDA LOS IOB.
Together, the Los Angeles and Long Beach harbor complex ranks first in the country and
eighth among the top global ports (JOC, 2012). FDA LOS IOB has led many initiatives
in the field of import operations and is often the proponent of developing new technology
and establishing systems that are later adopted by other districts. Therefore, the methods
and results of this study may have the potential to impact the efficiency and effectiveness
of FDA’s broader national import operations. Furthermore, the framework presented in
this study could be modified for use in other government operations that involve
customer service delivery.
Most importantly, this study bears significance to public health and trade in that
the two systems affect up to about one-third of all FDA-regulated imports into the US.
The findings from this study should provide insights into the agency’s ability to prevent
adulterated and misbranded products from entering while expediting the release of safe
and effective products into the domestic commerce. As the global trade continues to
expand and shift toward electronic systems, assessing and optimizing service delivery
mechanisms will be critical to FDA’s ability to protect the nation’s public health as well
as its economy.
28
1.5 Limitations, Delimitations and Assumptions
Geographical and recruitment delimitations were required to define the scope of
this study. The majority of survey participants were from the Greater Los Angeles
metropolitan area, though LAImports and ITACS are globally accessible. Survey
participants were primarily recruited through the local trade association for brokers,
filers, freight forwarders and importers. The study focuses on the main stakeholders of
the two systems, who are the industry users of LAImports and ITACS. The electronic
survey required respondents to have email and Internet access, which are already
requirements for use of the two systems being studied. Government employees who
utilize LAImports and ITACS were not actively recruited as participants.
Studies such as this are likely to have several limitations whose impact must be
borne in mind when interpreting the results. There may be limited access to some helpful
information. For example, certain information regarding LAImports and ITACS may not
have been readily available to the general public and would need to be requested via the
Freedom of Information Act (FOIA) for which response times are difficult to estimate.
Furthermore, certain information may be privileged and thus remain undisclosed. The
quality of information available for examination also limited policy analysis and
subsequent survey development. Finally, the number of available respondents who were
willing to participate in the survey limited the size of the study and thereby impacted the
robustness of the conclusions that can be drawn.
In the earlier stage of research design, it was assumed that the local import/export
trade association, the Los Angeles Customs Brokers and Freight Forwarders Association,
would assist in identifying potential survey respondents to participate in the study. The
29
trade association did assist in identifying and contacting survey respondents, and as all
respondents were filers, brokers, and freight forwarders, it is assumed that their views
concur with the importers they represent. These delimitations have been designed with
consideration that this population represent the majority of the most active users of
LAImports and ITACS operating within the jurisdiction of FDA LOS IOB. It is also
assumed that study delimitations do not affect the representativeness of collected data.
1.6 Organization of Thesis
This thesis is divided into five chapters. Chapter 1 begins with a high level
overview of the global shift toward electronic service delivery of import processes,
leading to the introduction of FDA’s LAImports and ITACS systems. Chapter 1 then
identifies a research hypothesis, and the purpose, significance, and constraints of a study
of customer service delivery at FDA LOS IOB via LAImports and ITACS. Chapter 2
discusses the regulation of imports and electronic processing of imports in the US,
analyzes LAImports and ITACS policies, and surveys literature pertaining to the two
systems, customer service in public and private sectors, and the current state of analysis
and evaluation of the research problem. A framework is then developed to evaluate
customer service delivery of LAImports and ITACS. Chapter 3 explores survey
methodologies and techniques using the above framework to examine the research
problem. Chapter 4 reports and analyzes the data output from the survey methodology in
Chapter 3. Chapter 5 discusses the strengths, weaknesses, and validity of the employed
survey methodology and yielded results, and provides research-based conclusions and
future considerations for LAImports and ITACS.
30
CHAPTER 2: LITERATURE REVIEW
2.1 Introduction
According to the American Customer Satisfaction Index (ACSI), government
services score significantly below private sector services in terms of customer satisfaction
(ACSI, 2013). A possible explanation for this gap is the lack of competition for services
within the government sector resulting in development of delivery processes that lack a
customer-centric perspective (ACSI, 2013). For private enterprises, there is plentiful
competition that provides both customers with choices and companies with motivation to
measure and improve their customer service delivery. Nonetheless, the shift toward e-
government has also shifted the government’s role from a service provider to a service
partner with an increased emphasis on performance and accountability that could
potentially narrow the disparity between the level of customer satisfaction in public and
private sectors. To this end, the US government has implemented measures to increase
transparency, participation, and collaboration, and modernize performance management
in an effort to streamline customer service.
It is important to note that certain governmental sectors may have higher
motivation to improve customer service delivery. FDA’s Los Angeles District Import
Operations Branch (FDA LOS IOB) may be a good example of a public sector that
interacts closely with industry in a global commerce environment and has growing
expectations with respect to customer service. Exploring specific federal policies with a
focus on customer service provides context from which to evaluate FDA regulation of
imports. The evolution of the two electronic systems within FDA LOS IOB, namely
31
PAR LOS LAImports email account and ITACS web portal, is examined and analyzed
with the goal of developing an appropriate customer service survey instrument. Because
there are currently no formalized methods to evaluate customer service delivery of e-
government import services, documents associated with government initiatives and
literature on customer service from private sectors (marketing, retail, banking, health care
delivery, e-commerce, and self-service technologies) were reviewed to initially identify
methodologies that might be applicable to the study; these were then assessed for their
relevance to the proposed research and were blended to compile a comprehensive
framework tailored to assess and compare user satisfaction with the LAImports and
ITACS systems.
2.2 Policy Initiatives in Government to Promote Customer Service
A number of recent activities have contributed to the enhanced sensitivity toward
the government as an effective service provider (Figure 1). President Barack Obama’s
first policy initiative in office, the Memorandum on Transparency and Open Government,
emphasizes the progression of government’s role as a service partner. This initiative
introduces a framework of transparency, public participation, and collaboration to
advance efficiency and effectiveness, and strengthen democracy (Obama, 2009). The
subsequently issued Open Government Directive, Memorandum M-10-06, better defines
the three principles of open government, as follows.
Transparency promotes accountability by providing the public with
information about what the Government is doing. Participation allows
members of the public to contribute ideas and expertise so that their
government can make policies with the benefit of information that is
32
widely dispersed in society. Collaboration improves the effectiveness of
Government by encouraging partnerships and cooperation within the
Federal Government, across levels of government, and between the
Government and private institutions (Orszag, 2009b).
The Government Performance and Results Act (GPRA) of 1993 and the GPRA
Modernization Act of 2010 represent major legislative policy initiatives in improving
customer service delivery. The GPRA Modernization Act requires federal agencies to
develop an annual performance plan that includes specific indicators and measures for
customer service, priority goals and quarterly progress reviews ("GPRA Modernization
Act of 2010," 2010).
Executive Order 13571 followed the above initiatives, calling for federal agencies
to develop a Customer Service Plan detailing the six streamlining and improvement
activities discussed in Section 1.2. Briefly, the executive order requires that agencies
establish a technology-based initiative to improve customer service delivery (Obama,
2011a). M-11-24 Memorandum on Implementing Executive Order 13571 on
Streamlining Service Delivery and Improving Customer Service provides further
guidance on: 1) Technology-based customer service initiatives; 2) Customer feedback
collection pursuant to the Paperwork Reduction Act; 3) Improving online services; and 4)
Customer service task forces and performance factors for customer service plans. FDA’s
regulatory management of imports falls squarely within the parameter of this executive
order to streamline service delivery and improve customer service in a function critical to
protecting public health and stimulating domestic economy.
33
Figure 1: Timeline of Public Policy Initiatives in Customer Service
2.3 Regulation of US Imports
Global production of FDA-regulated goods has exploded over the past ten
years. In addition to an increase in imported finished products,
manufacturers increasingly use imported materials and ingredients in
their U.S. production facilities, making the distinction between domestic
and imported products obsolete…There has been a perfect storm - more
products, more manufacturers, more countries and more access. A
dramatic change in strategy must be implemented.
Dr. Margaret Hamburg, Commissioner of Food and Drugs
2.3.1 Regulatory Overview of US Imports
It is challenging to manage imports in an increasingly globalized world. In the
US, a number of different federal agencies share statutory responsibilities for import
safety (CBP, 2011). These federal agencies include Alcohol, Tobacco, Firearms and
Explosives (ATF), Animal and Plant Health Inspection Service (APHIS), Consumer
Product Safety Commission (CPSC), Customs and Border Protection (CBP),
Environmental Protection Agency (EPA), Food and Drug Administration (FDA), Food
Safety Inspection Service (FSIS), Immigrations and Customs Enforcement (ICE),
National Highway Traffic Safety Administration (NHTSA), National Marine Fisheries
34
Service (NMFS), and Pipeline and Hazardous Materials Safety Administration
(PHMSA). However, overseeing the regulatory management of goods imported into the
US falls primarily on CBP that administers laws “relating to import, export and the
collection of duties,” by facilitating communications with other agencies, the
international trade community, and foreign governments (FDA, 2011e).
CBP, formed by the merger of the US Customs Service and various border
enforcement agencies after September 11, 2001, has the primary mission of homeland
security. It staffs all the US ports of entry and conducts daily port operations, including
cargo clearance, collection of duties and other import monies, and processing of travelers
arriving from foreign nations (CBP, 2006). CBP port officers will initially examine a
shipment to determine its value and duty schedule, accuracy of invoicing, compliance
with CBP labeling requirements, absence of prohibited articles or illegal narcotics, and
compliance with applicable legal and regulatory requirements. They also perform
agricultural inspections to prevent the entry of pests and diseases that could cause
ecological and environmental harm in the US. If the shipment is subject to examination
by other agencies prior to release, the importer must contact those agencies and follow
the appropriate steps for review and release. Within 15 calendar days of a shipment’s
arrival at a US port, the importer or designated agent submits to the port director entry
documents needed to establish admissibility, to assess duties, and to pay owed money.
These submission processes can be performed electronically via the Automated Broker
Interface (ABI) program of the Automated Commercial System (ACS).
35
2.3.2 FDA’s Regulation of US Imports
Within the network of agencies responsible for import safety, FDA plays a pivotal
role because of the critical nature of the products that it regulates. FDA “is responsible
for determining whether or not an article offered for importation is in compliance with or
in violation of the acts enforced by FDA,” thereby directly impacting public health and
wellness (FDA, 2011e). Of the approximately 29.4 million entry lines imported into the
United States during fiscal year 2013, devices and foods accounted for most of the FDA’s
regulatory efforts, representing 48% and 33% of the total volume, respectively
(Veneziano, 2014).
An example of a seemingly simple FDA-regulated product class with a complex
importation process is fresh produce, as illustrated in Figure 2. Its entry is initially filed
administratively with CBP’s Automated Commercial System (ACS); the “Prior Notice of
Imported Foods” is filed prior to arrival at the port with FDA headquarters as per the
Bioterrorism Act of 2002. The arriving produce is screened by local CBP Agriculture
Specialists, may subsequently be examined by local Animal and Plant Health Inspection
Service (APHIS) inspectors from the US Department of Agriculture (USDA), and may
then undergo sampling and testing by FDA field components. Importers may also be
required to submit other evidence supporting compliance and admissibility to FDA’s
field import operation branches.
36
Figure 2: Diagram of Importation Process for Fresh Produce
This process appears to be straightforward and linear, but can be complicated by
multiple convoluted and iterative sub-processes. Complex processes have higher
potential to fail when faced with increasing pressures to perform. For example, FDA has
experienced, over the last decade, a three-fold expansion of imports (Figure 3) consisting
of a myriad of products that represent a substantial proportion of domestic consumption.
These products represent 20% of fresh vegetables, 50% of fresh fruits, 80% of seafood,
40% finished drug products and 80% of active pharmaceutical ingredients for US
domestic market (FDA, 2012a). Given the breadth and complexity of this increasing
workload, the agency must find ways to manage imports effectively and efficiently if it is
to ensure that the products meet the needs of the consumer while safeguarding the health
of the US public.
37
Figure 3: Import Entry Lines Electronically Screened by FDA Since FY2003
(FDA, 2014b)
As previously discussed, a strong IT infrastructure to improve service delivery at the
world’s largest port operations is an important part of an approach consistent with the
global shift toward e-government. Effective electronic delivery of FDA import services
may be a key strategy to manage the increasing volume of imported goods that fall under
FDA’s jurisdiction and the need for timely review and release of these products.
2.4 Regulation of US Imports
2.4.1 History of Electronic Processing of Import Entries at FDA
The FDA began its work on electronic processing of imports in 1987, when the
Commissioner created a task force to develop a new automated IT system, currently
known as the Operational and Administrative System for Import Support (OASIS), to
manage the oversight of imported products (Reilly et al., 1995). This system was
38
designed to: 1) increase productivity through automated interfaces with laboratories,
brokers and Customs; 2) suggest actions likely to result in discovery of violations and
thus improve screening; 3) increase turnaround time for processing of entries, as well as
consistency in agency responses; 4) provide agency-wide uniformity in entry processes;
and 5) maintain an information base for reporting (Reilly et al., 1995). After eight years
and nearly $14 million dollars of development, the initial program was judged to be a
failure by the US General Accounting Office (GAO) and underwent extensive
reengineering “to achieve dramatic changes in overall performance and customer
satisfaction,” as seen at leading global port operations (Reilly et al., 1995). The
reengineered OASIS was fully implemented in 1998 with an integrated historical data
library that screens imports continually to assist FDA personnel with admissibility
decisions (FDA, 2011d).
FDA is currently developing another system to replace OASIS, called Mission
Accomplishment and Regulatory Compliance Services (MARCS) Imports, in order to
more comprehensively address FDA’s import management needs. MARCS Imports has
two components, Predictive Risk-based Evaluation for Dynamic Import Compliance
Targeting (PREDICT) and MARCS Entry Review (MARCS ER). PREDICT is already
in operation, replacing the electronic legacy screening function of OASIS (Figure 4), and
MARCS ER has replaced the entry review functionality of OASIS (FDA, 2011d). The
FDA’s OASIS and MARCS systems proved inadequate to meet all the evolving needs of
the local field offices working directly with the ports and the importers. Therefore, FDA
LOS IOB implemented LAImports locally in 2010 and FDA headquarters implemented
ITACS nationally in 2012, as reviewed and discussed below.
39
2.4.2 Electronic Processing of Import Entries at FDA Headquarters
FDA defines an entry as a “delivery or offer for delivery of merchandise into the
Customs Territory of the U.S. from an outside point” (FDA, 2011e). The initial entry
information may be filed and processed either manually in paper format, or electronically
using the Automated Broker Interface (ABI). Because more than 99% of the entries
reviewed by FDA is now transmitted electronically, the entry of a product into the US
follows a well defined path that incorporates multiple steps, and several different
specialized electronic systems (Solis, 2010). Figure 4, derived from FDA’s briefing
material for the import industry, provides a schematic overview of the electronic entry
process of an FDA-regulated product from the initial filing to the first decision point
where it may proceed into domestic commerce or is referred to the local district office for
further review (FDA, 2011d).
40
Figure 4: Electronic Transaction of Import Entries from Filing through
PREDICT Screening. Modified from FDA (FDA, 2011d)
For an FDA-regulated product, the entry is initially filed with the Automated
Broker Interface (ABI) by the importer or the customs broker. This information is
transmitted to the FDA for initial screening via Automated Commercial System (ACS).
If the shipment contains food, entry information is transmitted to OASIS for Prior Notice
Screening by the FDA Division of Food Defense and Targeting. Prior Notice is the
advanced notice required prior to the shipment of foods as mandated by the Bioterrorism
Act of 2002 (FDA, 2011c). If the shipment does not contain food or has undergone Prior
Notice Screening, the entry will electronically flow through to PREDICT, the FDA
program that screens all entry information for compliance with Section 381 “Imports and
Exports” part 801(a) of the Food, Drug and Cosmetic Act (Verbeten, 2011). The entry
41
will then receive either a notification that it may proceed into domestic commerce or that
it requires further FDA review by the district office.
2.4.3 Electronic Processing of Entries at FDA LOS IOB
This study examines the local import operations in the FDA Los Angeles District
(FDA LOS IOB) following notification of further review as described above. Entries
requiring further FDA review at the end of the process depicted in Figure 4 are
transmitted electronically from FDA headquarters to FDA districts in both OASIS and
MARCS Imports systems. Figure 5A is a schematic representing transaction of an entry
through the electronic processes in place at FDA LOS IOB, including the associated
electronic systems. Figure 5B is a schematic representation of the import detention
process at FDA LOS IOB. These are derived from FDA’s briefing material for industry,
the FDA Regulatory Procedures Manual and the FDA Basics Webinar on Import
Operations.
42
Figure 5: Processes associated with LOS IOB
A. Electronic Transaction of Import Entries at the FDA LOS IOB. Modified from (FDA,
2011d, 2011e; Verbeten, 2011)
43
B. Import Detention Process at the FDA LOS IOB. Modified from (FDA, 2011d, 2011e;
Verbeten, 2011)
Figure 5A shows the process starting from a local FDA district’s request for
additional documents and further review to the decision point where the shipment is
either released or detained for further action. Figure 5B provides an overview of the
FDA import detention process. Prior to the 2010 launch of LAImports, the FDA LOS
IOB could only accept hardcopies of entry documents. After 2010, LAImports provided
importers and broker with the option of submitting pdf versions of required documents
via email. Soon thereafter, ITACS was developed with electronic submission
capabilities. FDA LOS IOB now prefers that documents be submitted via ITACS instead
of LAImports whenever possible, as indicated in Figure 5A. Following examination of
the entry documents, the entry reviewer enters the results into the MARCS ER, and
workflow continues in OASIS.
44
The entry reviewer may issue a “May Proceed” notice, which does not require
physical examination of the product, or may submit a request for detention. The field
investigator uses OASIS to enter results from physical inspection and/or sample
collection. After the product has been physically inspected in the field, the investigator
has the option to issue a “Release”. Otherwise, the information is passed on to the
compliance officer for further review and action. If the compliance officer detains the
entry, the Detention and Hearing Process begins. FDA sends the “Notices of FDA
Action” to the importer and broker via US postal service.
Figure 5B provides an overview of the FDA import detention process, which does
not employ any specialized electronic systems other than OASIS to generate Notices of
FDA Action (FDA, 2011e). The FDA import detention process is administrative in
nature, and does not cause the product to be held physically, but stored within the
jurisdiction of FDA LOS IOB (Verbeten, 2011). The FDA LOS IOB jurisdiction is
limited to Los Angeles, Orange, Riverside, San Bernardino, and Ventura counties (CBP,
2012). After the notice of detention is issued, a Detention and Hearing Process is
available for the importer to protest the decision by providing evidence that the product is
not violative (Verbeten, 2011). Traditionally submitted as paper documents, this
evidence can include petitions to recondition the product into compliance, results from
private laboratory analyses, process documents and other manufacturing information, or a
request to extend the hearing process (FDA, 2012d). Evidentiary documents are
submitted electronically via LAImports and forwarded to the Compliance Officer for
review. There may be several iterative cycles of submission followed by review of
testimony and evidence before admissibility is finally decided. Following this process,
45
the detained product may be released, or the detention may remain in effect and the
product is refused admission into the US. Refused entries will either be exported out of
the US or destroyed by the importer. Otherwise, civil liquidation penalties are incurred
(Verbeten, 2011).
The processing of imported shipments described above shows that information
and documents are exchanged at several different transaction points including initial
entry document requests, information to refute detention, reconditioning petitions, refusal
documentation, and requests for other supporting evidence. Each of these transaction
points may be further optimized to streamline customer service delivery. Although FDA
estimates that more than 99% of the entries reviewed by FDA are initially transmitted
electronically via CBP’s ABI, the subsequent exchanges of information take place
through several lengthy paper transactions that leave a large environmental footprint and
greater potential for mistakes. Historically, importers submitted documents to the FDA
via postal mail, parcel service, courier delivery, or facsimile, and the FDA routinely used
postal mail for correspondence. Postal mail is slow and may not be appropriate for
perishable goods. Parcel and courier delivery services are costly and incur large fuel
consumption. Facsimiles can be lost or backlogged because of limits on paper and ink
capacity of the equipment, and may result in hundreds of loose pages needing to be sorted
manually. As the volume of imports continues to increase, the strains of these
inefficiencies would intensify. The electronic transactions through LAImports and
ITACS should help streamline the import management process to be more efficient with
a higher level of customer service.
46
2.5 Review of Electronic Systems at FDA LOS IOB
2.5.1 PAR LOS LAImports (LAImports)
LAImports is an email account, LAImports@fda.hhs.gov, that was established as
an alternate means for industry to submit documentation and communicate with FDA
regarding regulated imports into Greater Los Angeles Ocean Ports and the UPS Ontario
Airport (Dimapindan, 2012; FDA, 2012d). Since its inception, LAImports has quickly
become the primary means of communication, receiving approximately 130 document
submissions and 75 inquiries each day (Dimapindan, 2012). To improve the speed with
which emails are processed, FDA LOS IOB developed “PAR LOS LAImports Email
Guidelines,” which can be seen in Appendix C. This document refers to itself as policy
and, thus, will be used to identify criteria for policy analysis (Munger, 2000). These
criteria include goals, services, timeframes, and feedback mechanisms, and are listed in
Table 1.
2.5.2 Import Trade Auxiliary Communication System (ITACS)
ITACS made its first appearance in FDA literature in 2009 in the published
questions and answers from the a webinar given by FDA to the National Customs
Brokers and Forwarders of America of Association (FDA, 2009). Its development has
been linked to FDA’s major agency transparency initiative (FDA, 2011a). ITACS,
launched in March 2012 as an Internet portal, https://itacs.fda.gov, allows the importing
community to: 1) check the status of an entry or line; 2) submit entry documents which
will be linked to a specific entry or line; and 3) provide location and availability
information for targeted shipments (FDA, 2012c).
47
Table 1: Performance Criteria for Analysis of LAImports and ITACS
LAImports
ITACS
Goals
• Streamlined
communication
• Timely,
concise,
useful
information
• Reduction
in
phone
calls
to
FDA
• Reduction
in
paper,
time,
and
money
waste
• Reduction
in
calls
for
status
and
submission
of
location
• Reduction
in
faxes
of
entry
documents
and
location
• Reduction
in
courier
delivery
of
entry
documents
• Reduction
in
loss
of
records
Services
• Entry
status
• General
inquiries
• Emailed
pdf
Notices
of
Action
• Submission
of
pdf
records
for
entry
reviewers
• Submission
of
pdf
records
for
compliance
officers
• Submission
of
other
documents
• Time-‐stamped
archive
of
emails
• Entry
status
• Submission
of
entry
documents
and
linkage
to
a
specific
entry
• Submission
location
and
availability
information
and
linkage
to
a
specific
entry
Timeframes
• 1
business
day
for
receipt
confirmations
• Automatic
routing
of
emails
with
proper
formatting
per
guidelines
• 1-‐2
business
days
to
process
emails
requiring
reformatting
• 2
business
days
to
review
entry
documents,
location
letters,
and
other
documents
• 24
hours/day,
7
days/week
(24/7)
access
to
the
web-‐
based
system
• 24/7
submission
and
linkage
of
entry
documents
in
• Immediate
notification
in
FDA
entry
review
system
that
new
documents
are
available
for
review
• 24/7
submission
and
linkage
of
location
information
(no
need
to
manually
log)
Feedback
Mechanisms
• Submission
of
feedback
to
LAImports@fda.hhs.gov
for
review
by
FDA
LOS
IOB
• Submission
of
feedback
to
itacssupport@fda.hhs.gov
for
review
by
FDA
HQ
staff
Flowcharts (Figures 5A and 5B) assist with the examination of LAImports and
ITACS as alternative courses of action. There are three transaction points between
industry and FDA LOS IOB in Figure 5A: submission of entry documents, submission of
location information for sampling, and submission of location information for
48
examination. During this process, information can be submitted at the indicated
transaction points via LAImports or ITACS based on the submitter’s preference. Within
the detention process depicted in Figure 5B, there is one transaction point between
industry and the FDA—submission of testimony to compliance—that is submitted via
LAImports instead of ITACS, since documents submitted via ITACS are not readily
accessible by FDA LOS IOB Compliance Officers. There is another transaction point,
status request, not specifically charted in Figures 5A and 5B, but can be achieved via
LAImports or ITACS. Even with these electronic services in place, industry can still
request and submit information via traditional methods of telephone, fax, mail and
courier delivery.
LAImports and ITACS offer different timelines to the importer. ITACS appears
to provide immediate system-driven responses regarding admission status, submission of
entry documents, and submission of location information. On the other hand, LAImports
requires that the documents are processed first and then acted upon by an FDA
investigator. ITACS provides an immediate confirmation message whereas LAImports
requires personnel to provide a response, thereby creating a lag time. Feedback
mechanisms were thought likely to be more effective for LAImports, where a local team
is already in place processing and responding to incoming emails. ITACS feedback, on
the other hand, must be routed through the FDA headquarters outside of the automated
process. Lag times with ITACS feedback mechanisms are likely to exist because the
majority of FDA LOS IOB’s customers operate on Pacific Time and headquarters
operates on Eastern Time. Further, ITACS feedback is submitted to an email address,
which is separate than its primary automated Internet portal. Although not discredited as
49
alternatives, telephone, fax, mail and courier delivery will not be considered in the
present analysis, or in the present study of electronic service delivery.
Review of the systems led to the following initial impressions. ITACS appears to
be a more transparent system than LAImports, since it is accessible at any time for
checking status or to conduct transactions. However, LAImports has the ability to
provide projected timeframes while ITACS does not. Adherence to provided timeframes,
ability to query into the processing timeframes of documents submitted, as well as a
comparison of the usefulness of status information provided by the two systems should
be further assessed. LAImports appears to provide better opportunities for participation
through direct communication with the local FDA personnel, where feedback
mechanisms are integrated with the normal operating system. ITACS feedback
mechanisms are submitted to an email address independent of the main one-way
transaction system, and are managed across the country. LAImports also appears to
provide better opportunities for collaboration because FDA LOS IOB works with local
trade organizations, particularly the Participating Government Agencies committee of the
Los Angeles Customs Brokers and Freight Forwarders Association, which facilitates
knowledge transfer on LAImports. Because ITACS is managed at headquarters, there is
less opportunity for collaboration with industry users within the FDA LOS IOB
jurisdiction.
50
2.6 Policy Assessment for Customer Service
2.6.1 Defining and Measuring Satisfaction
In any system of policy implementation, it is important that assessments are made
to establish that the initiatives are meeting objectives. A first step toward such a goal is
to establish criteria deemed important in assuring that the initiative is effective. The FDA
has already provided some general guidance about the metrics that it might use for such
an evaluation. The performance criteria for analysis of one of the systems, ITACS, for
example, are reflected in the publications of FDA that are posted on the FDA webpage
and those for LAImports are listed in the “PAR LOS LAImports Email Guidelines,”
developed by FDA LOS IOB (FDA, 2012b, 2012c). The key elements of this set of
metrics are shown in Table 1. It is interesting to note that most of its performance criteria
are directed at transactional elements of interactions with users. These are useful in
determining whether at minimum, the basic logistical goals of the program are being met.
However, they do not by themselves give information on the level of user satisfaction
with the programs or provide mechanisms for evaluating customer satisfaction.
The concept of “customer satisfaction” is central to this thesis but is one that
extends beyond the scope of the thesis. Any study of this area depends first on
characterizing what is meant by the concept of customer satisfaction. The pivotal work
of Churchill and Surprenant suggested that customer satisfaction could be captured in a
model that looked at what they described as a paradigm of confirmation-disconfirmation
(Churchill & Surprenant, 1982). In this paradigm, customer expectations are: 1)
confirmed when a product or service performs as expected; 2) negatively disconfirmed,
51
when a product or service performs more poorly than expected; and 3) positively
disconfirmed when the product or service performs better than expected (Churchill &
Surprenant, 1982). The disconfirmation paradigm was built upon the following series of
customer percepts: 1) Expectations- the customer makes an intuitive decision about what
would constitute the anticipated benchmark performance of a product or service; 2)
Performance- the customer observes or experiences the actual performance of the service
or product; 3) Disconfirmation- the customer makes a judgment about the extent, if any,
of discrepancies between expectations and performance; and 4) Satisfaction- the
customer decides on the level of satisfaction, by comparing the rewards and costs
associated with the product or service use with performance (Churchill & Surprenant,
1982).
2.6.2 Frameworks for Assessing Customer Satisfaction
Since the concept of customer satisfaction enunciated by Churchill and
Surprenant (1982) relies on a comparison between observed and desired performance, it
leads logically to the question of how to assess performance in a customer service
scenario. More than 20 years ago, Parasuraman, Berry and Zeithaml developed the
service quality instrument SERVQUAL for service and retailing organizations, which is
heavily referenced in literature. SERVQUAL is a framework of 22 scale items clustered
into the following five dimensions: 1) Tangibles- physical facilities and equipment, and
personnel appearance; 2) Reliability- dependability and accuracy in service execution; 3)
Responsiveness- willing to help customers and providing prompt service; 4) Assurance-
provider knowledge and courtesy, and ability to inspire customer trust and confidence;
52
and 5) Empathy- individualized customer attention (Parasuraman, Berry, & Zeithaml,
1991; Parasuraman, Zeithaml, & Berry, 1988).
The work of Parasuraman and coworkers, that was initiated to provide a quality
instrument for service and retail organizations, has been emulated in whole or part by
other groups interested in assessing customer service in other sectors. For example,
Andaleeb examined determinants of customer satisfaction in the healthcare delivery
sector, where satisfaction was assessed relative to the overall patient experience within
hospitals (Andaleeb, 1998). Andaleeb developed a customer service framework by using
variables and language that patients employed to explain satisfaction and variables that
could be actionable for hospital managers. The framework had the following constructs:
1) Better perceived quality of communication results in greater levels of satisfaction; 2)
Greater perceived provider competence results in greater levels of satisfaction; 3) Better
perceived quality of facilities results in greater levels of satisfaction; 4) More positive
staff demeanor results in greater levels of satisfaction; and 5) Greater perceptions of
excessive service costs results in lower levels of satisfaction (Andaleeb, 1998). Four of
these customer service constructs can be mapped directly onto the dimensions of
satisfaction identified in the Parasuraman model. For example, good communication
relates to “responsiveness”, competence relates to “assurance” and “reliability”, facilities
relate to “tangibles”, and demeanor relates to “tangibles”, “assurance” and “empathy”.
The last item, service costs, is also an important dimension of satisfaction that might be
added to the existing dimensions of the Parasuraman model in situations where cost is a
deciding factor in the selection of a product or service.
53
Recently, the evaluation of customer service has been revisited in large part
because the nature of customer interactions is changing from one in which humans
interact with humans to one in which humans interact with technology. Steyaert adapted
and applied a set of principles related to electronic commerce developed by Watson and
colleagues to study e-government performance, particularly identifying factors that
promote the public’s use of e-government transactional services (Steyaert, 2004). Their
framework used five indicators of electronic service efficiency, that could be related to
quantifiable metrics: 1) Customer awareness – reflected in number of site visitors; 2)
Customer popularity – related to the rank of the agency rank in terms of its monthly
visitors; 3) Contact efficiency – related to evaluations of overall site content and
usability; 4) Conversion – related to customer satisfaction with electronic transactions,
and visitor time; and 5) Retention – related to the numbers of repeat visits and
transactions (Steyaert, 2004). The framework was successful in evaluating, analyzing
and comparing multi-level e-government services for a study of web traffic reports and
customer surveys (Steyaert, 2004).
An expansion of approaches to evaluate customer satisfaction in self-service
technologies (SSTs) most recently led Lin and Hsieh to develop an alternative
instrument, called SSTQUAL, to examine satisfaction in regard to the interactions
between customer and technology (Lin & Hsieh, 2011). This framework, like that
described earlier by Parasuraman and coworkers, identified a series of dimensions: 1)
Functionality- including responsiveness, reliability and ease of use; 2) Enjoyment-
perception of enjoyment with delivery and outcome; 3) Security- perceived safety of data
and information; 4) Assurance- confidence in the reputation and competence of the
54
provider; 5) Design- overall design of the system; 6) Convenience- service accessibility;
and 7) Customization- degree of system configuration to fit consumer preference (Lin &
Hsieh, 2011). This framework has some elements that were not captured in the work of
Parasuraman because they are quite specific for services that have computers or
technology at their core.
2.7 Choosing a Framework to Evaluate Electronic Systems at FDA LOS IOB
The aforementioned customer satisfaction frameworks and their associated
dimensions must be considered as potential options for structuring the research proposed
here. The new electronic systems at FDA are in fact hybrid systems that have elements
of human-to-human interaction combined with human-to-computer interaction. Further,
the offerings of FDA LOS IOB are unique in that customers of the import process have
their choice of two different systems that are essentially substitutable for a select number
of transactions depending on preference. Two types of questions can therefore be
explored. First, can customer satisfaction with regard to the two systems be characterized
and compared? Second, what causes one system to be chosen as the method of
preference over the other?
2.7.1 Exploring Available Tools to Evaluate Customer Satisfaction
The previous review of customer satisfaction frameworks suggests that no
singular framework can be applied to evaluate customer satisfaction in all situations
where customers interact with services or products. There are, however, commonalities
amongst the most widely used tools that vary depending on the specific nature of the
anticipated interactions. We can consider categorizing the different frameworks into two
55
groups. First is the group of approaches most suited to human-to-human interactions.
The well-respected framework of Parasuraman and coworkers, with its five dimensions,
can easily be applied to such situations. Another framework that can be placed in this
group is that of Andaleeb, since it covers essentially the same elements as those of
Parasuraman, but with an added dimension related to costs and value of the service. This
is interesting to consider in the context of government services that are free to the user,
yet have an explicit goal of developing cost saving measures as in the case of the
LAImports system.
Second is the group of approaches more suited for human-to-machine interactions.
Both the Steyaert and Lin and Hsieh frameworks would fall into this group (Lin & Hsieh,
2011; Steyaert, 2004). The newer model of Lin and Hsieh covers all of the dimensions
that were highlighted in the earlier work of Steyaert, but also includes additional
dimensions that might be useful for the analysis here. For example, the identification of
convenience and enjoyment as explicit dimensions important to customers were not so
clearly captured by Steyaert as by Lin and Hsieh.
From a comparison of commonalities and differences amongst the frameworks
(Table 2), it may be useful to integrate the dimensions of Parasuraman with those of Lin
and Hsieh, in order to construct a net grouping of dimensions to support the work
proposed here. The resulting list of dimensions appears to be appropriate to assess
services and products in the private sector that involve both human-to-human and human-
to-machine interactions. However, this approach may not be adequate for the evaluation
of government services.
56
Table 2: Dimensions of Customer Satisfaction for Human-to-Human and
Human-to-Machine Service Interactions (Lin & Hsieh, 2011;
Parasuraman et al., 1991; Parasuraman et al., 1988)
Dimensions appearing on the same row are considered to be broadly
related.
Principle
Human-‐to-‐Human
Human-‐to-‐Machine
Parasuraman
et
al.
Lin
and
Hsieh
Customer
Service
Tangibles
Design
Reliability
Functionality
Responsiveness
Assurance
Assurance
Empathy
Security
Enjoyment
Convenience
Customization
A government service has special needs compared to that in the private sector,
including a higher level of accountability to the taxpayer. As identified in the
Memorandum on Transparency and Open Government, government services should be
particularly sensitive to issues of transparency, public participation, and collaboration.
The government meaning of each term in this triad of sensitive issues must be examined
closely to determine whether these areas of focus are part of or equivalent to dimensions
in the customer satisfaction frameworks described above or additional dimensions need
to be considered. According to the White House’s Obama Open Government Status
Report, transparency, participation and collaboration are defined as follows.
Transparency. Transparency means providing the public with information
about their government’s activities. It contemplates disclosure about, for
example, what federal agencies have done or will do. Transparency’s
57
premise is that citizens are entitled to know what, how, and why
government does what it does.
Participation. Citizens are entitled to more, however, than a transparent
view of their government from the outside looking in. Participation
emphasizes citizens’ voice in public affairs, recognizing that public
officials stand to benefit from the perspective of expert and non-expert
knowledge that resides outside of government. Participation is fostered by
expanding citizens’ opportunities to express their views about policy
alternatives, and in ways beyond voting in elections.
Collaboration. Collaboration further erodes the us-versus-them divide
between citizens and government by taking participation to another level.
Citizens are capable, after all, of more than simply registering their views
about policy alternatives defined in advance. They can usefully help
shape the government’s agenda. They can also help determine even the
tools and methods by which public policy goals are pursued. Where
government is collaborative, citizens become true partners with
government, in both the identification and pursuit of public goals (Croley,
2011).
Although several White House documents order the three principles as
transparency, public participation, and collaboration, working definitions for this study
establish a more natural progression from transparency (expectations) to collaboration
(transactional performance) to public participation (feedback). These definitions are, of
course, open to interpretation and expansion from literature outside of government. A
more specific definition of transparency for example, is that by Solberg and Richmond,
who identified three dimensions to transparency as clarity, accessibility and
accountability (Solberg & Richmond, 2012). In this study of transparency of drug
58
submission processes to regulatory agencies, they defined clarity as knowing procedures,
accessibility as level and ease in obtaining information, and accountability as regulators’
roles in making and communicating decisions (Solberg & Richmond, 2012). Similarly
outside of government, the Eysenbach model for Medicine 2.0 provides a more succinct
view of collaboration as activities connecting “groups of people with each other who
have not, or have not sufficiently, interacted with each other” (Eysenbach, 2008). In this
model, “participation” was defined to also encompass privacy issues (Eysenbach, 2008).
Such privacy issues are analogous to the computer and data security dimension of Lin
and Hsieh.
Are transparency, participation and collaboration dimensions already captured in
the core concepts of Customer Service as described by social science theorists and
discussed above? Transparency might be captured in some measure as part of the
dimensions relating to “responsiveness” and “assurance” in the language of Parasuraman.
However, it is not clear whether all of the dimensions of transparency could be fit under
these other dimensions of customer satisfaction. Given the emphasis placed on this
principle by government, it was considered important to assure that it was identified and
explored in this research by adding it explicitly as an auxiliary principle alongside the
core principle of traditionally viewed “ Customer Service”. Similarly, participation and
collaboration do not seem to be dimensions called out in previous models to describe
dimensions of “Customer Service”. This is probably due to the fact that the customer
service appeared in those analyses as “something provided”; in such an approach, the
customer might be viewed as a largely passive receiver, rather than a partner or
contributor to the service. Thus, the only previously identified dimension in which
59
participation or collaboration would seem to fit easily would be that of “customization”.
However, capturing principles of public participation and collaboration under this rather
narrow dimension risks the problem that they would be mostly ignored or explored
incompletely. In order to avoid the possibility that any of the elements considered
particularly important to the federal administration might be overlooked in the analysis
proposed here, they have been added as auxiliary principles to that of Customer Service
as illustrated in Figure 6 and have been added to Table 3 as separate dimensions as well.
Figure 6: Framework for Customer Service Evaluation
60
Table 3: Customer Satisfaction Dimensions for Core and Auxiliary Principles
(Croley, 2011; Lin & Hsieh, 2011; Obama, 2009; Parasuraman et al.,
1991; Parasuraman et al., 1988)
Status
of
Principle
Principle
Parasuraman
et
al.
Lin
and
Hsieh
Open
Government
Summary
of
Dimensions
Core
Customer
Service
Tangibles
Reliability
Responsiveness
Assurance
Empathy
Design
Functionality
Security
Assurance
Enjoyment
Convenience
Customization
Design
(Tangibles)
Functionality
(Reliability
and
Responsiveness)
Security
Assurance
Empathy
Enjoyment
Convenience
Customization
Auxiliary
Transparency
Clarity
Accountability
Accessibility
Clarity
Accountability
Accessibility
Auxiliary
Public
Participation
Feedback
Feedback
Auxiliary
Collaboration
Collaboration
Partnership
Collaboration
Partnership
Government Tools for Customer Service Evaluation: EO 13571 and HowTo.gov
Before accepting a particular framework, it is important to assess whether
government has developed methods already in place for assessing its programs. The
White House’s HowTo.gov website and Executive Order (EO) 13571 are two of the
governmental sources for customer service factors that are relevant to LAImports and
ITACS. The White House maintains the website HowTo.gov in order to provide
resources and tools for collecting data to improve customer service provided by federal
services (Sustein & Zients, 2011). The website provides customer service metrics based
on extensive research, but does not identify a central methodology for measuring
customer satisfaction, citing lack of an “industry standard” (Fox & Piazza, 2013).
Amongst the approaches favored by government is an existing tool called the American
Consumer Satisfaction Index, or ACSI. This tool is used by some of the federal agencies,
61
including the Federal Consulting Group of the US Department of the Interior (FCG,
2011). However, ACSI is a fee-for-service, patented, statistically intensive methodology
that is not well suited for the current study. If such a labor-intensive tool is not used or
appropriate, a much less systematic approach is recommended, in which agencies are
requested to collect the following four common customer service metrics from industry:
1) Overall customer experience; 2) Completion rate of intended task; 3) % of visitors
likely to return; and 4) % of visitors likely to recommend the service (Fox & Piazza,
2013).
From the streamlining activities in Executive Order 13571 as described in Section
1.2, the following customer service factors for innovative technologies are extracted to be
used to improve customer service delivery: 1) Soliciting customer feedback; 2) Utilizing
feedback to improve service delivery; 3) Setting customer service standards and
expectations; 4) Adopting proven customer service best practice; 5) Providing service
channel coordination; 6) Reducing cost, delivery times, calls and inquiries. With the
federal government scoring below the lowest performing private sector in customer
satisfaction, it could be questioned whether the tools provided by the government are
used effectively (ACSI, 2013). Factors 1, 5 and 6, which parallel performance criteria in
Table 1, are sufficiently granular that they could easily be considered metrics similar to
those of HowTo.gov, and examined alongside the dimensions of Customer Service,
Transparency, Participation and Collaboration that are identified in Table 3 to provide
insight on service delivery of LAImports and ITACS. The EO 13571 metric on feedback
will be captured under the government auxiliary principle of public participation.
62
Table 4: HowTo.gov and EO 13571 Metrics for Customer Service Evaluation
HowTo.gov
Metrics
EO
13571
Metrics
• Overall
customer
experience
• Completion
rate
of
intended
task
• %
of
visitors
likely
to
return
• %
of
visitors
likely
to
recommend
the
service
• Soliciting
customer
feedback
(Public
Participation)
• Providing
service
channel
coordination
• Reducing
cost,
delivery
times,
calls
and
inquiries
2.7.2 Loyalty Factors that Contribute to Customer Preference
The fact that customer satisfaction has a multiplicity of dimensions raises a
second interesting question when more than one service is available, and that is why one
service is preferred over another. It may be that not all dimensions are equally important
to the customer, and special attention should be given to those that the customer uses as
key determinants in making a decision regarding the system to be used. A theoretical
construct that is associated with this question is that of “loyalty”. Rust and Zahorik, for
example, related customer satisfaction and loyalty, by differentiating service delivery
elements from loyalty factors. Service elements are specific areas that can be directly
influenced by service management, and loyalty factors are general impressions that
propel loyalty amongst existing customers. Their study examined the following three
constructs: 1) Satisfaction with specific service delivery elements results in satisfaction
with fewer, yet broader loyalty factors; 2) Satisfaction with loyalty factors results in
propensity for loyalty with the service; and 3) Loyalty is probabilistic based on
propensity for loyalty (Rust & Zahorik, 1993). Rust and Zahorik argue that loyalty
involves a choice, and the importance of various loyalty factors in the choice process is
measurable. In the current research, the intention is not to explore systematically the
63
“loyalty factors” responsible for the explicit choice of one electronic import transaction
system over another. However, the research will explore why customers choose to be
loyal to one system over another. The dimensions that seem most important to making
this choice may furnish insights into those aspects of service that are key determinants of
overall customer satisfaction with one or another of the systems.
It is also possible that users of the LAImports or ITACS systems may like to
blend the two systems. Given the choice, for example, industry may prefer to use ITACS
for initial document submission and status inquiry, because it provides immediate
information accessible at any time, but prefer to use LAImports to track subsequent
submissions and engage in direct communications with FDA personnel. For submission
of testimony to the compliance officer following notice of detention, LAImports is
currently the only electronic option available to industry.
This document analysis served as a primer for customer service evaluation of
LAImports and ITACS. The analysis recommends a high-level framework that will
allow many aspects of service and interaction to be evaluated that can be used to group
more granular questions regarding issues relating to both the perceived and measurable
functions of the system. In an analysis such as that proposed here, a multi-dimensional
customer service framework with a portfolio of dimensions and indices should provide a
more stable and balanced dataset just as in the financial sector it is recognized that a
diversified portfolio provides greater stability and balance (Ryan, Buzas, & Ramaswamy,
1995). Empirical data supports utilization of multiple-item scales for tracking
improvements, setting priorities for quality initiatives, and predicting retention (Ryan et
al., 1995).
64
2.8 Customer Service Evaluation of FDA Imports Systems
2.8.1 Current Methods for LAImports and ITACS
As discussed in Section 2.2, the White House has identified various techniques for
program and customer service evaluation. These can be added to one or more of the
different frameworks used in the academic and private sectors. However, currently no
specific methods are in place to evaluate LAImports and ITACS with regard to customer
service, transparency, collaboration, and public participation that would meet the needs of
these two electronic systems.
2.8.2 Proposed Research
This study proposed to use a multi-dimensional framework around the dimensions
of customer service, transparency, collaboration, and public participation to
systematically evaluate the two service delivery systems, LAImports and ITACS. This
proposed research incorporated elements from both the public and private sectors in order
to provide a broader scope of assessment and evaluation. Taking into consideration that
the primary client base for the FDA’s import interface is the trade industry, the survey
instrument was developed in consultation with the import/export trade association in the
FDA LOS IOB jurisdiction.
The goal of this research is to identify actionable items to plan improvements and
justify resource allocations and expenditures. To collect data for this analysis, I
developed, validated, and administered a survey instrument exploring industry’s views on
the ability of LAImports and ITACS to deliver transparent, collaborative, and
participative services. Currently, there does not appear to be literature that gives a single
65
tool appropriate to evaluate customer service. By incorporating dimensions of customer
service from both private and public sector literature, I hope to assure a sufficiently broad
scope of the study instrument to capture the important variables affecting satisfaction and
preference. Data will be made available to the FDA for its use in understanding the
industry’s experience, identifying areas of improvement, and planning future resource
allocations. This systematic approach produced quantitative and qualitative responses
that could be analyzed and used in evidenced-based decision-making processes to
improve the delivery of services by LAImports and ITACS. In addition to filling the
gaps in the government’s own service delivery evaluation, this comprehensive and
systematic approach can be used for future data collections.
66
CHAPTER 3: METHODOLOGY
3.1 Introduction
This exploratory study of the ability of PAR LOS LAImports and ITACS to
deliver transparent, collaborative, and participative service was designed around focus
group and survey techniques. After initial development of the multi-dimensional survey,
it was discussed in a focus group to test and enhance content validity. The focus group
consisted of employees from the International Center for Regulatory Science at the
University of Southern California, with experience as customers of government services
and global regulatory affairs. After the focus group, the survey was modified
accordingly, and forwarded to members of the Participating Government Agencies
(PGA) committee of the Los Angeles Customs Brokers and Freight Forwarders
Association (LACBFFA) for review of suitability for industry. The PGA committee,
which collaborates regularly with FDA on electronic systems in place at FDA LOS IOB,
consists of senior employees at customs brokerages familiar with LAImports and ITACS.
After receiving PGA feedback, the survey was modified accordingly, and translated into
electronic format via Qualtrics. Members of the University of Southern California
Regulatory Science programs familiar with Qualtrics functionality participated in pilot
testing of the electronic survey. The PGA committee assisted in identifying potential
respondents to the survey. Potential respondents received information, instructions, and a
link to the survey via email.
67
3.2 Greater Los Angeles Import/Export Trade Associations
Collaboration with greater the largest Los Angeles area import/export trade
association was established to identify, obtain contact information for, and recruit
possible survey respondents. These types of trade associations are most likely to enroll
industry members utilizing LAImports and ITACS within FDA LOS IOB. Also, FDA
LOS IOB maintains strong working relationships with one such organization, as
evidenced by joint meetings, conferences, and partnerships in disseminating new
guidance documents and updates.
3.2.1 Los Angeles Customs Brokers and Freight Forwarders Association (LACBFFA)
LACBFFA was established in 1949 to “promote the Southern California customs
brokerage, freight forwarding and NVOCC [non-vessel operating common carrier]
industry as a whole” (LACBFFA, 2012). The industry is affected by the FDA regulation
of imports, and as such, the association has a Participating Government Agencies (PGA)
committee, which is “responsible for liaison with all other government agencies”
(LACBFFA, 2012). The PGA committee has one chair and 15 members, all of whom are
from the industry. LACBFFA’s searchable online directory lists 576 members.
LACBFFA regularly disseminates emails to its members to “keep them apprised of
important notices and issues affecting their day-to-day business practices” (LACBFFA,
2012). LACBFFA also provides opportunities for more personal interactions through
“monthly luncheons or dinners with guest speakers who address a full range of issues of
concern to members” (LACBFFA, 2012).
68
3.2.2 Foreign Trade Association of Southern California (FTASC)
Established in 1919 as the World Traders Club of Los Angeles and renamed as
FTASC on May 3, 1926, FTA is the “oldest organization promoting the growth of
international trade in Southern California.” It acts “as an informative resource and
networking center” while monitoring and advocating “legislative issues on a state and
federal level” (FTASC, 2012). FTASC has also been a pioneer in international trade,
collaborating with the Los Angeles Chamber of Commerce “to set aside a week every
May to promote international trade and educate the community on the value of global
commerce” (FTASC, 2012).
Today, FTASC is a “private, non-profit trade association” representing more than
"200 members of the international trade community” (FTASC, 2012). The association’s
membership is comprised of “major exporters, importers, manufacturers, customs
brokers, freight forwarders, international bankers, attorneys and other prominent service
industries” (FTASC, 2012).
The FTASC was not used to identify potential survey respondents. In the time
leading up to the survey, FTASC and LACBFFA both listed the same business contact
information. According to the LACBFFA PGA committee chair, both associations were
colocated as they were under the same management, and PGA colleagues are members of
both.
3.3 Development of Initial Survey
The survey began with questions collecting demographic and administrative
information regarding the respondent and the importers they may represent. Survey
69
questions for dimensions related to the framework were designed using definitions found
in the literature reviewed in Chapter 2. Human-to-human and human-to-machine
dimensions were paired as seen in Table 2, such that one integrated framework and
survey could be used to explore customer service, transparency, collaboration, and public
participation for both LAImports and ITACS. Questions were also designed using
HowTo.gov and EO 13571 metrics (Tables 1 and 4). This was to compare the
effectiveness of these available government tools against that of the study framework. A
question was designed to explicitly collect information on transactional preferences
between the two systems in order to probe loyalty. Table 5 provides mapping of survey
questions to their most closely associated dimensions, factors, or metrics.
70
Table 5: Survey Question Mapping to Dimensions, Factors, and Metrics
Q
Measure
General
Question
Topic
Associated
Dimensions,
Factors
and
Metrics
6
Preference
Preferred
system
for
particular
transactions
Loyalty
Factors
7
Frequency
Ability
to
complete
intended
transaction
HowTo.gov
Metric
8
Satisfaction
Continued
system
use
9
Likelihood
Recommendations
using
positive
terms
10
Satisfaction
Instructions
for
use
Transparency
Clarity
11
Satisfaction
Explanation
of
services
Accessibility
12
Satisfaction
Explanation
of
processing
times
Accessibility
13
Satisfaction
Communication
of
entry
decisions
Accountability
14
Satisfaction
Interaction
with
FDA
Collaboration
Interaction
15
Agreement
Partnership
with
FDA
Partnership
16
Satisfaction
Feedback
mechanisms
Public
Participation
Feedback
17
Frequency
System
availability
Customer
Service
Reliability
Functionality
Convenience
18
Satisfaction
Computer
security
Customer
Service
Security
Public
Participation
Privacy
19
Satisfaction
User
interface
Customer
Service
Tangibles
Design
20
Satisfaction
System
customization
Customer
Service
Customization
22
Satisfaction
Completeness
of
Information
provided
Customer
Service
Assurance
23
Satisfaction
Relevance
of
information
provided
Customer
Service
Assurance
24
Frequency
Turnaround
times
Customer
Service
Responsiveness
Functionality
25
Satisfaction
Ability
to
meet
expectations
Customer
Service
Satisfaction
Enjoyment
26
Reductions
(Operational)
Changes
in
firm
due
to
LAImports
EO
13571
Metric
27
Reductions
(Operational)
Changes
in
firm
due
to
ITACS
EO
13571
Metric
28
Satisfaction
Overall
customer
experience
HowTo.gov
Metric
71
3.4 Survey Deployment and Analysis
The survey was deployed electronically via Qualtrics. Collaborating with the
LACBFFA trade association, possible respondents were identified. The LACBFFA
emailed a link to the survey to members of the trade association. Clicking the link in the
email directed potential respondents to the online survey, completed surveys were
recorded in Qualtrics, and I monitored Qualtrics to track the number of completed
surveys. The LACBFFA PGA committee chair specifically identified the 16 firms of the
members of the PGA committee as meeting inclusion criteria, and sent a follow-up
message to the committee requesting their participation in the survey. Email addresses of
LACBFFA members were obtained from the association’s online directory, to which a
reminder was sent to members after two full business weeks of survey launch. Follow-up
phone conversations were conducted with PGA committee members on the fifth business
day after the email reminder. Although links to the survey were sent to personal email
addresses, email addresses were not linked to names such that data would not linked to
persons, thus maintaining anonymity of survey participants. Remuneration was not
offered to encourage participation in the survey. After the close of the survey, responses
were analyzed in Qualtrics, which has statistical and analytical functions.
Cross tabulations of responses to particular questions were performed to observe
relationships between certain variables. The three sets of cross tabulations are:
(Continuing to Use Systems & Recommending Systems Using Positive Terms) * (System
Preferences); (Turnaround Times, Communication of FDA Entry Decisions & Ability to
Interact with FDA) * (Overall Customer Experience); and (Ability to Meet Expectations)
* (Overall Customer Experience).
72
Survey results are presented in the Chapter 4 as bar graphs capturing total
response data for LAImports and ITACS users. “Not Applicable/I do not use the system”
responses have not been included in figures and related data analysis because they do not
contribute to the measure or understanding of customer service delivery of the two
systems. However, this information is included in the figure and table legends as “n/a”
for number of “Not Applicable/I do not use the system” responses. Total sample size for
each question is shown as “n”.
73
CHAPTER 4: RESULTS
4.1 Results of the Focus Group
The focus group consisted of five individuals from the International Center for
Regulatory Science at the University of Southern California. These individuals held
senior positions in the organization and had academic and industry experiences in
international regulatory affairs, government service delivery, and/or survey development
techniques. During the 90-minute session, the focus group reviewed and provided
feedback on the content and construction of each question of the initial survey
instrument. A suggestion was made to add a question on the experience of respondents
related to their interactions with FDA that could differentiate respondents who were
working on FDA-regulated entries 1) before the launch of LAImports, 2) after the launch
of LAImports but prior to that of ITACS, and 3) after the launch of ITACS. Potential
areas of ambiguity and confusion related to academic terms, such as clarity, accessibility
and reliability, were addressed by either eliminating the terms or rephrasing the question.
Certain questions were thought to be too cumbersome due to the high level of granularity
in the solicited information and, therefore, suggestions were made to modify them to be
less dependent on the detailed recollections of the respondents. Another suggestion was
to redesign and simplify questions that specified rankings. Because not all respondents
are familiar with all functionalities of LAImports and ITACS, a suggestion was made to
include additional response choices of “Not Applicable” and “I do not use the system” for
certain questions. Other suggestions included grouping and ordering questions in order
to shorten the length of the survey and optimize its flow.
74
The revised survey instrument was forwarded to the chair of the LACBFFA PGA
committee for review. The chair and another member of the LACBFFA PGA committee
affirmed readability and suitability of the questions for industry users of LAImports and
ITACS. The LACBFFA PGA committee members expressed concern with the length of
the survey. In response, several questions were further combined to shorten the survey to
33 questions. The final survey was then reviewed by top management within FDA LOS
IOB, who provided positive feedback regarding the potential opportunities for
improvements to entry processing within the branch.
4.2 Survey Results
The LACBFFA PGA committee chair sent a general link to the electronic survey
to its members via email on February 26
th
, and by the tenth day, the number of surveys
collected had fallen to zero (Figure 7). At the start of the third business week after survey
launch, I sent a reminder email to the 525 of 576 LACBFFA members, whose email
addresses were listed in the online directory, through the survey distribution function in
Qualtrics, which provided for response rate tracking. Qualtrics recorded 36 email
bounce-back failures; because LACBFFA has the same email addresses on file, these
incorrect email addresses reduce the total sample size of potential respondents to 489.
From my email reminder, 15 surveys were collected. In total, 57 individuals responded
to the survey during the 31-day period spanning from February 26, 2014 to March 28,
2014, corresponding to a 12% (n=489) response rate. Figure 8 details response rates to
each survey question from the 57 surveys collected.
75
Figure 7: Number of Surveys Collected per Date
The LACBFFA sent the initial request to participate in the survey to its
members on February 26, 2014. I sent a reminder message on March 17,
2014. Collection peaks are observed on these two dates.
24
8
3
2
1
1
13
2
2
1
0
5
10
15
20
25
#
Surveys
Collcted
Date
76
Figure 8: Survey Question Response Rate for 57 Total Respondents
As Figure 8 shows, a majority of respondents answered all of the multiple-choice
questions. Questions 21, 29, and 33 provided space for optional open-ended text entries
and had relatively low response rates of 12.3%, 10.5%, and 10.5%, respectively.
Questions 6, 9, 15, and 25 were multiple-choice questions that also provided respondents
with a text box to enter additional comments, and resulted in 4, 1, 2, and 2 text responses,
respectively. The raw survey responses can be viewed in Attachment D.
0
10
20
30
40
50
60
70
80
90
100
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
Response
Rate
(%)
Question
Number
77
4.2.1 General Information About Respondents
A majority (48 out of 50) of respondents identified themselves to be affiliated
with brokers/filers and the remainder identified themselves as freight forwarders (Figure
9). Most began working on FDA-regulated entries prior to the launch of LAImports, two
respondents began after the launch of LAImports but before the launch of ITACS, and
one began after the launch of ITACS (Figure 10).
Figure 9: “What is the primary classification of your firm’s business operation?”
(Q1: n=50)
Figure 10: “When did you begin working on FDA-regulated entries?” (Q2: n=50)
0
10
20
30
40
50
60
Broker/Filer
Self-‐filing
Importer
Importer
Freight
Forwarder
Other
Number
of
Respondents
Primary
ClassificaVon
of
Firm's
Business
OperaVon
0
10
20
30
40
50
Before
Jan
1,
2010
Jan
1,
2010
-‐
Mar
14,
2012
AZer
Mar
14,
2012
Number
of
Respondents
Experience
Working
on
FDA-‐regulated
Entries
78
With regard to system usage, a majority of LAImports and ITACS users reported
executing between 1 and 15 transactions in an average work week with very few
executing greater than 50 transactions per week.
Figure 11: “How many total transactions do you estimate making with LAImports
and ITACS in an average work week?” (Q5: LAImports n=45; ITACS
n=43)
According to Figure 12, human foods (defined per FDA LOS IOB as human
foods minus seafood and dietary supplements) were reported as the commodity handled
most by the respondents. The second and third categories were medical
devices/diagnostics (22%) and seafood (12%), respectively. When asked to rank their
three most-handled commodities, human foods and medical devices/diagnostics
continued to rank first and second, whereas seafood and dietary supplements tied for third
(Figure 12). These respondents reported handling products that predominantly originate
from Asia with the top three geographical regions identified in the following in order: 1)
China, including Hong Kong and Macau; 2) Southeast Asia; and 3) Japan, Korea and
Taiwan (Figure 13).
0
2
4
6
8
10
12
14
16
0
1-‐5
6-‐15
16-‐30
31-‐50
51-‐100
>100
Number
of
Respondents
#
TransacVons
Number
of
TransacVons
per
Average
Work
Week
ITACS
LAImports
79
Figure 12: “Which entry commodity do you handle most?” (Q3: n=50) and “Which
three (3) entry commodities do you handle most?” (Q31: n=37)
Figure 13: “Which import geographical region of origin do you handle most?”
(Q4: n=50) and “Which three (3) import geographical regions of origin
do you handle most?” (Q32: n=37)
0
5
10
15
20
25
Animal
Drugs
Animal
Feeds/Pet
Foods
Biologics
Dietary
Supplements
Drugs
Electronics/Lasers
Housewares
Human
Foods
Medical
Devices/DiagnosVcs
Seafood
Tobacco
Number
of
Responses
CommodiVes
Handled
Q31
Responses
Q3
Responses
0
5
10
15
20
25
30
Canada
&
Mexico
Central
America
South
America
Caribbean
Northern
Europe
Western
Europe
Eastern
Europe
Southern
Europe
Sub-‐Saharan
Africa
Middle
East,
N.
Africa
&
Greater
India
&
Asian
SubconVnent
China,
Hong
Kong
&
Macau
Japan,
Korea
&
Taiwan
Southeast
Asia
Oceania
Number
of
Responses
Regions
of
Origin
Handled
Q32
Responses
Q4
Responses
80
4.2.2 Study Framework Principles and Dimensions
The study framework depicted in Figure 6 consists of four principles that are each
associated with a set of dimensions (Table 3). The survey questions were designed to
address these dimensions as they pertain to LAImports and ITACS, and the results are
presented below according to the four framework principles: Customer Service,
Transparency, Collaboration, and Public Participation.
Customer Service Dimensions
Overall, respondents were satisfied with the Information Technology (IT)
infrastructure of LAImports and ITACS, as demonstrated by their responses to
dimensions related to human-to-machine service interactions. Most respondents reported
the two systems to be available “always” or “most of the time”, with ITACS performing
slightly better than LAImports. All ITACS users except one were satisfied with the
system availability whereas 1 in 5 LAImports users were dissatisfied, reporting that the
system was available only half of the time or less when needed (Figure 14). Users of
ITACS and LAImports were equally and predominantly satisfied with the computer
security provided by the two systems with over 80% of each group reporting that they
were satisfied (Figure 15).
81
Figure 14: “How often is each system available when you need it?” (Q17:
LAImports n=27, n/a=4; ITACS n=30, n/a=2)
Figure 15: “How satisfied are you with the level of computer security when you
transmit information through: LAImports? ITACS?” (Q18: LAImports
n=28, n/a=4; ITACS n=29, n/a=3)
More respondents appeared satisfied with the ITACS user interface (79%) than
with the LAImports user interface (66%) (Figure 16), with approximately 1 in 3
LAImports users expressing neither satisfaction nor dissatisfaction. More respondents
were satisfied with the level of system customization provided by ITACS (62%) than by
LAImports (52%), but neither system performed well and about 1 in 3 users expressed
that they were neither satisfied nor dissatisfied (Figure 17). The text responses in Table 6
13
12
9
17
3
1
2
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
System
Availability
Always
Most
of
the
Time
Half
of
the
Time
Some
of
the
Time
Never
10
10
13
14
5
5
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
Computer
Security
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
82
add further insight into areas of concern and suggested improvements, including the lack
of access to contact information for FDA personnel working entries, the suggestion for
notification of document submissions, and the need to be able to access multiple entry
lines rather than a single entry line at a time.
Figure 16: “How satisfied are you with the user interface of: LAImports? ITACS?”
(Q19: LAImports n=29, n/a=4; ITACS n=29, n/a=3)
Figure 17: “How satisfied are you with the level of system customization for:
LAImports? ITACS?” (Q20: LAImports n=27, n/a=5; ITACS n=29,
n/a=4)
7
9
12
14
9
5
1
1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
User
Interface
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
5
4
9
14
10
9
1
2
2
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
System
CustomizaVon
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
83
Table 6: “If you could customize the systems, what would you change?” (Q21:
n=7)
#
Response
1
Providing
the
compliance
Officer(s)
contact
informtion
on
ITACS
for
shipments.
Providing
up
to
date
status
information
on
ITACS.
Days
can
go
by
with
no
update
and
which
makes
you
have
to
call
the
FDA
LA
Status
line...only
to
leave
a
voicemail.
2
There
should
be
a
way
to
find
out
the
specific
person
and
contact
information
on
who
is
handling
the
shipment/documents
at
FDA.
In
case
of
no
responses
from
either
ITACS
or
LAImports,
you
have
the
option
to
follow
up
through
email
or
telephone
directly.
3
More
options
with
Itacs.
4
ITACS:
auto
send
report
reference
entry#
to
FDA
personnel
whenever
document
or
information
is
submitted
to
ITACS.
auto
send
report
to
FDA
personnel
if
doc
and
info
submitted
to
ITACS
has
not
been
reviewed
by
FDA
within
2
working
days.
to
keep
information
on
ITACS
for
at
least
15
days
after
release
(currently,
status
immediately
removed
from
ITACS
after
FDA
released)
5
ITACS
-‐
only
allows
you
to
input
1
line.
I
have
several
lines
but
I
can
only
input
001
to
001.
Sometimes
in
order
to
follow
up
on
a
shipment,
it
is
faster
to
go
through
LAImports.
I
will
get
a
result.
Also
I
uploaded
some
documents
but
got
a
call
from
the
Officer
stating
that
she
never
got
all
the
documents.
So
I
had
to
email
her
documents.
I
want
the
system
to
be
able
to
tell
me
whether
all
the
documents
were
received
properly.
6
If
I
could
customize
ITACS:
-‐I
would
create
a
user
account
that
would
store
my
name,
company
name,
contact
information.
-‐
I
would
make
it
so
that
ITACS
could
store/list
my
most
recent
entries
that
have
been
monitored
or
updated
for
quick
reference
when
returning
to
the
website
throughout
the
day.
-‐I
would
arrange
it
to
store
warehouse
locations
(when
uploading
line
availability)
so
that
we
could
select
from
a
list
of
locations
that
we
frequently
use
(since
most
of
our
customers
utilize
the
same
locations
consistently.)
Instead
of
having
to
manually
having
to
enter
these
details
line
by
line.
-‐
I
would
MOST
DEFINITELY
add
the
function
of
INCLUDING
a
history
of
all
status
changes/updates
that
occur
to
each
entry.
Presently,
ITACS
operates
in
a
'override'
function
whereas
the
new
status
and
effective
date,
override
the
previous
status
and
date
and
it
makes
it
difficult
to
revert
back
and
present
a
documented
timeline
to
the
importer
when
questions
arise
of
(for
ex:
when
a
shipment
is
placed
on
document
review
and
when
the
documents
are
successfully
uploaded
to
ITACS).
-‐Lastly,
I
am
not
sure
this
Is
applicable
to
this
particular
survey/study,
but
I
have
found
that
different
ports
have
varied
messages
that
they
are
using
in
ITACS
and
it
can
be
difficult
to
discern
what
specific
steps
some
of
the
messages
signify,
and
it
would
be
my
recommendation
to
streamline
this
internally
in
the
system
when
the
FDA
officer
updates
the
status
OR
to
have
a
Key
that
further
explains
the
comment/update.
7
just
need
FDA
to
continue
to
make
information
available
via
ITACS
very
quickly
within
minutes.
84
Overall, respondents were less satisfied with post-transactional dimensions, such
as information provided and FDA’s turnaround times, than with pre-transactional
dimensions related to IT infrastructure. However, more users were satisfied with the
completeness of information provided by ITACS (72%) than by LAImports (57%) and,
similarly, more were satisfied with the relevance of information provided by ITACS
(77%) than by LAImports (61%) (Figures 18 and 19).
Figure 18: “How satisfied are you with the completeness of information provided
by: LAImports? ITACS?” (Q22: LAImports n=28, n/a=5; ITACS n=29,
n/a=3)
4
5
12
16
7
6
4
2
1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
Completeness
of
InformaVon
Provided
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
85
Figure 19: “How satisfied are you with the relevance of information provided by:
LAImports? ITACS?” (Q22: LAImports n=28, n/a=5; ITACS n=30,
n/a=3)
Concerns with the responsiveness of the two systems were reflected in the
relatively low frequency ratings associated with turnaround times. Only about half of
ITACS users reported receiving needed turnaround times “always” or “ most of the time”
(Figure 20). LAImports users expressed even lower frequency in this regard, with none
reporting that they “always” received their needed turnaround times.
Figure 20: “How often do you get turnaround time you need from each system?”
(Q24: LAImports n=26, n/a=5; ITACS n=30, n/a=3)
There was a noticeable difference between each system’s ability to meet user’s
expectations. More ITACS users (69%) than LAImports users (48%) reported higher
3
6
14
17
7
6
3
1
1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
Relevance
of
InformaVon
Provided
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
5
12
10
7
7
3
5
4
3
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
Turnaround
Times
Always
Most
of
the
Time
Half
of
the
Time
Some
of
the
Time
Never
86
levels of satisfaction, and more LAImports users (40%) reported that they were neither
satisfied nor dissatisfied (Figure 21). Table 7 captures the two text responses regarding
each system’s ability to meet their expectations. The second response details the user’s
higher expectations for LAImports than for ITACS because of the human engagement
presumed with email conversation.
Figure 21: “How satisfied are you with each system’s ability to meet your
expectations?” (Q25: LAImports n=27, n/a=5; ITACS n=29, n/a=3)
Table 7: “How satisfied are you with each system’s ability to meet your
expectations? LAImports? ITACS? Please enter any additional
comments for either system below.” (Q25: n=2)
#
Response
1
There
is
limited
option
in
case
there
is
inaction
on
an
entry,
for
some
reason.
2
while
I
am
satisfied
with
ITACS
meeting
my
expectations,
it's
more
because
I
know
what
I
can
expect
from
ITACS.
LAImports
is
at
a
disadvantage
because
I
only
utilize
them
when
I
need
assistance
with
a
specific
issue,
usually
an
entry
has
sat
too
long
without
any
activity
from
FDA
and
I
have
to
reach
out
for
follow
up,
and
while
I
understand
that
the
technicians
at
LAImports
are
limited
in
their
authority,
many
of
them
to
not
seem
to
be
as
inherently
helpful
in
nature
as
a
select
few
that
are
there.
also,
I
don't
know
if
this
was
a
result
of
a
policy
change,
but
LAImports
used
to
forward
our
emails
to
the
compliance
office
who
could
assist
and
they
no
longer
extend
this
courtesy.
I
suppose
my
expectations
for
LAImports
are
higher
because
we
are
dealing
with
actual
people
who
should
be
able
to
engage
in
conversation
with
regard
to
our
inquiries,
and
they
so
frequently
fail
to
do
so.
2
6
11
14
11
7
3
2
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
Ability
to
Meet
ExpectaVons
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
87
Transparency Dimensions
Although a majority of respondents reported that they were satisfied or very
satisfied with the instructions for use for both systems, ITACS performed slighter better
than LAImports (Figure 22).
Figure 22: “How satisfied are you with the instructions for use of: LAImports?
ITACS?” (Q10: LAImports n=31, n/a=2; ITACS n=31, n/a=3)
Respondents were much less satisfied with the explanation of services and
processing times (Figures 23 and 24). Only 45% of LAImports users were satisfied or
very satisfied with explanation of services and that value decreased further to 34% for
explanation of processing times. ITACS fared slightly better than LAImports in both
areas. However, users were distinctly dissatisfied with both systems for explanation of
processing times with about 30% reporting dissatisfied or very dissatisfied.
9
13
11
10
9
7
1
1
1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
InstrucVons
for
Use
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
88
Figure 23: “How satisfied are you with FDA’s explanation of services provided by:
LAImports? ITACS?” (Q11: LAImports n=29, n/a=4; ITACS n=30,
n/a=3)
Figure 24: “How satisfied are you with FDA’s explanation of processing times for:
LAImports? ITACS?” (Q12: LAImports n=29, n/a=4; ITACS n=31,
n/a=3)
More than half of all respondents appeared to be satisfied with the communication
of FDA entry decisions through either the ITACS or LAImports systems although ITACS
rated slightly better (Figure 25).
7
8
6
10
12
11
4
1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
ExplanaVon
of
Services
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
3
4
7
10
10
8
8
8
1
1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
ExplanaVon
of
Processing
Times
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
89
Figure 25: “How satisfied are you with the communication of FDA entry decisions
through: LAImports? ITACS?” (Q13: LAImports n=30, n/a=4; ITACS
n=30, n/a=4)
Collaboration Dimensions
Most users were satisfied or very satisfied with the use of the two import systems
to interact with the FDA (Figure 26). However, ITACS again fared better. The
difference between the two systems, in this case, was primarily seen at the two extreme
views, very satisfied and very dissatisfied, where ITACS users were more positive and
LAImports more negative.
Figure 26: “How satisfied are you interacting with the FDA through: LAImports?
ITACS?” (Q14: LAImports n=29, n/a=3; ITACS n=31, n/a=2)
2
4
15
15
11
8
2
3
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
CommunicaVon
of
FDA
Entry
Decisions
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
4
7
13
14
6
7
3
3
3
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
InteracVon
with
FDA
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
90
Most users agreed that FDA has developed stronger partnerships with industry
through the two systems (Figure 27). Additional text comments describe inadequate
communication by both systems and point to potential areas of improvement (Table 8).
Figure 27: “Please select your level of agreement with the following statements:
FDA has developed a stronger partnership with industry through
LAImports; FDA has developed a stronger partnership with industry
through ITACS. Please enter any additional comments for either system
below.” (Q15: LAImports n=31, n/a=4; ITACS n=31, n/a=3)
Table 8: “Please select your level of agreement with the following statements:
FDA has developed a stronger partnership with industry through
LAImports; FDA has developed a stronger partnership with industry
through ITACS. Please enter any additional comments for either system
below.” (Q15: n=2)
#
Response
1
LAImports
allows
us
to
inquire
information
and
status
with
FDA,
but
seldom
receive
communication
back
from
them.
2
ITACS
is
limited
in
serving
the
industry
by
the
lack
of
details
that
are
given
with
regard
to
status
updates
while
LAImports
sometimes
lacks
a
sense
of
service
to
the
importer/brokers
with
regard
to
answering
questions
or
requests.
Most
of
my
inquiries
to
LAImports
are
only
answered
with
a
contact
name
and
number
for
me
to
call,
and
the
personnel
are
rarely
available
to
take
the
call,
which
puts
us
through
to
a
voicemail,
with
no
call
back.
This
leaves
the
importers
and
customers
with
a
feeling
of
FDA
NOT
being
in
tuned
with,
or
connected
with
and
inconsequetial
to
the
needs
of
the
industry.
3
5
17
18
9
7
2
1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
Partnership
with
FDA
Stongly
Agree
Agree
Neither
Agree
nor
Disagree
Disagree
Strongly
Disagree
91
Public Participation Dimensions
Only about half of users expressed satisfaction with the ability to provide the
FDA feedback on LAImports and ITACS (Figure 28). Contrary to other findings in the
study, more ITACS users (23%) expressed dissatisfaction with the feedback mechanism
than LAImports users (15%).
Figure 28: “How satisfied are you with the ability to provide the FDA with feedback
on: LAImports? ITACS?” (Q16: LAImports n=27, n/a=5; ITACS
n=30, n/a=4)
4.2.3 Executive Order (EO) 13571 Metrics
Data for performance metrics from Executive Order 13571 were collected to
determine if LAImports and ITACS are meeting operational goals listed in Tables 1 and
4. Figures 29 and 30 show that most respondents viewed both systems favorably with
regard to all the metrics. Respondents identified that the greatest reduction observed with
the use of electronic communication channels was associated with facsimile
transmissions, which were the predominant mode of communication prior to the
establishment of LAImports and ITACS. However, the use of all non-electronic forms,
including telephone, courier and surface mail appeared also to be reduced. Less change
3
1
10
14
10
8
2
6
2
1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
Feedback
Mechanisms
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
92
in performance was associated with spending and processing time of entry activities.
Nevertheless, most users acknowledged reductions in all areas of efficiency using both
systems.
Figure 29: “If you use LAImports: 1) Have you reduced telephone calls to FDA
LOS IOB? Have you reduced faxes to the FDA LOS IOB? Have you
reduced postal mail to the FDA LOS IOB? Have you reduced courier
delivery to FDA LOS IOB? Have you reduced paper consumption? Have
you reduced company spending? Have fewer documents been lost since
using the system? Has total entry processing time decreased since using
the system?” (Q26: n=30)
24
27
24
25
22
19
23
20
4
1
3
4
7
6
3
7
2
2
3
1
1
5
4
3
0%
20%
40%
60%
80%
100%
Phone
Calls
Faxes
Mail
Courier
Delivery
Paper
ConsumpVon
Spending
Document
Loss
Entry
Processing
Time
Number
of
Respondents
(Totaling
100%
n)
LAImports
Performance
Metrics
Yes
No
Unsure/Do
Not
Know
93
Figure 30: “If you use ITACS: Have you reduced telephone calls to FDA LOS
IOB? Have you reduced faxes to the FDA LOS IOB? Have you reduced
postal mail to the FDA LOS IOB? Have you reduced courier delivery to
FDA LOS IOB? Have you reduced paper consumption? Have you
reduced company spending? Have fewer documents been lost since
using the system? Has total entry processing time decreased since using
the system?” (Q27: n=30)
4.2.4 HowTo.gov Metrics
As discussed in Section 2.7.1, HowTo.gov instructs federal agencies to collect
data using the following four baseline customer satisfaction metrics: Completion rate of
intended task; Percent of visitors likely to return; Percent of visitors likely to recommend
the service; and Overall customer experience.
The results of HowTo.gov metrics were consistent with those seen in the
customer service, transparency, collaboration, and public participation dimensions; the
majority of respondents are satisfied with both systems, but the performance of ITACS
was markedly superior to that of LAImports. The most striking difference was seen in
the ability of users to complete their intended transactions (Figure 31). While 94% of
ITACS users were able to complete their intended transactions most of the time to
24
27
24
24
23
19
23
21
4
1
3
3
5
5
2
7
2
2
3
3
2
6
5
2
0%
20%
40%
60%
80%
100%
Phone
Calls
Faxes
Mail
Courier
Delivery
Paper
ConsumpVon
Spending
Document
Loss
Entry
Processing
Time
Number
of
Respondents
(Totaling
100%
n)
ITACS
Performance
Metrics
Yes
No
Unsure/Do
Not
Know
94
always, only 68% of LAImports users were able to do as often. In fact, over 20% of
LAImports users reported that they were able to complete their intended tasks only some
of the time or, or in case of one user, never.
Figure 31: “How often are you able to complete your intended transactions with:
LAImports? ITACS?” (Q7: LAImports n=34, n/a=3; ITACS n=36,
n/a=3)
A few respondents expressed clear negative response with LAImports on all four
metrics and reported that they were less likely to recommend the system to other possible
users or continue to use it themselves (Figures 32 and 33). The negative and intermediate
views toward LAImports are most effectively captured in Figure 34, which shows that
only 57% LAImports users were satisfied or very satisfied with overall customer service,
compared to 87% of ITACS users. Table 9 lists one text response describing lack of
timely response provided by ITACS and Table 10 lists additional comments on the areas
of dissatisfaction and overall customer experiences with the systems. These comments,
generally more favorable for ITACS than LAImports, point to problems with
communication and timeliness. The comments also include differential expectations for
the two systems.
14
16
9
18
4
2
6
1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
CompleVng
Intended
TransacVons
Always
Most
of
the
Time
Half
of
the
Time
Some
of
the
Time
Never
95
Figure 32: “How satisfied are you with continuing to use: LAImports? ITACS?”
(Q8: LAImports n=35, n/a=3; ITACS n=36, n/a=3)
Figure 33: “How likely are you to recommend the following systems using positive
terms to colleagues: LAImports? ITACS? Please enter any additional
comments for either system below.” (Q9: LAImports n=36, n/a=3;
ITACS n=36, n/a=4)
11
17
11
14
10
5
3
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondents
Users
(Totalin
100%
n)
ConVnued
Use
of
Systems
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
11
20
13
13
8
3
4
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
Recommending
Use
of
Systems
Very
Likely
Likely
Neither
Likely
nor
Unlikely
Unlikely
Very
Unlikely
96
Figure 34: “How satisfied are you with your overall customer experiences with:
LAImports? ITACS?” (Q28: LAImports n=28, n/a=3; ITACS n=30,
n/a=1)
Table 9: “How likely are you to recommend the following systems using positive
terms to colleagues: LAImports? ITACS? Please enter any additional
comments for either system below.” (Q9: n=1)
#
Response
1
ITACS
provides
visibility,
but
officers
do
not
always
respond
to
ITACS
submissions.
there
is
no
automatic
notification
to
FDA
officers
when
document
or
information
is
submitted
to
ITACS,
this
cause
delay
in
response
time
3
10
13
16
9
4
2
1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LAImports
ITACS
Number
of
Respondent
Users
(Totaling
100%
n)
Overall
Customer
Experience
Very
SaVsfied
SaVsfied
Neither
SaVsfied
nor
DissaVsfied
DissaVsfied
Very
DissaVsfied
97
Table 10: “Do you have any additional comments on your overall customer
experience?” (Q29: n=6)
#
Response
1
Since
we
started
using
ITACS
we
only
email
LAIMPORTS
when
we
haven't
seen
any
change
in
ABI
for
over
2
days
or
to
check
status
inquiry.
Other
than
that,
ITACS
is
FANTASTIC!
2
ITACS
Reviewers
seem
to
be
proactive
and
will
make
contact,
immediately,
should
additional
information
is
required.
LAIMP
can
take
days
to
respond
and,
often,
we
are
given
a
number
to
the
Review
but
will
not
receive
a
call
back
once
a
message
or
multiple
messages
are
left.
3
as
I
mentioned
before,
and
I
hope
I
am
not
being
too
long
winded
here,
but
the
expectations
for
LAImports
are
higher
than
those
of
ITACS.
with
ITACS,
it's
very
clear
what
I
can
and
cannot
have
access
to,
and
so
I
am
not
often
let
down
by
it.
I
feel,
and
maybe
I
am
incorrect
here,
that
LAImports
is
more
service
based
in
regards
to
responding
and
assisting
with
inquiries,
etc.
with
that
expectation,
LAImports
is
often
a
let
down
because
often,
when
I
request
for
an
update
on
an
entry
(I
will
include
a
screen
shot
of
the
current
status
on
ITACS)
and
LAImports
will
merely
respond
to
me
with
the
same
information,
which
I
have
obviously
already
gathered
from
ITACS.
there
are
some
personnel
at
LAImports
who
seem
to
take
more
initiative
to
explain
and
strive
towards
a
bit
of
customer
service,
but
those
one
or
two
people
are
clearly
outnumbered
and
it
leaves
my
importers
dissatisfied
with
FDA's
communication.
4
Processing
times
and
decision
making
timeliness
needs
to
be
improved.
5
As
a
customer
of
FDA,
it
is
very
unsatisfactory
when
there
are
shipments
which
become
problems.
For
routine
shipments,
both
systems
work
equally
well.
For
problem
shipments,
it
is
nearly
impossible
to
contact
anyone
at
FDA
for
any
reason
as
we
are
unable
to
speak
with
anyone
by
phone
and
e-‐mail
queries
are
usually
not
answered.
6
i
do
not
use
ITAC,
it
only
communicated
one
way,
what
I
sent,
but
it
did
not
have
the
capacity
to
come
back
and
tell
us
what
additional
information
was
being
requested
by
the
FDA.
4.2.5 Preference and Loyalty Factors
Most respondents preferred ITACS for submitting entry documents and
submitting location/availability information but preferred LAImports for requesting entry
status (Figure 35). Comments suggested that the preference for LAImports might be
related to higher expectations for a timely response from an email system (Table 10, #3).
98
Figure 35: “Which system do you prefer to use for the following transactions with
the FDA Los Angeles Import Operations Branch: Submitting Entry
Documents? Submitting Location/ Availability Information? Requesting
Entry Status? Please enter any additional comments for either system
below.” (Q6: n=50)
Table 11: “Which system do you prefer to use for the following transactions with
the FDA Los Angeles Import Operations Branch: Submitting Entry
Documents? Submitting Location/Availability Information? Requesting
Entry Status? Please enter any additional comments for either system
below.” (Q6: n=4)
#
Response
1
Cannot
always
get
entry
status
from
ITACS
2
WE
USE
ITACS
EXCLUSIVELY
NOW
3
We
are
a
little
wary
about
using
ITACS
because
sometimes
the
information
does
not
go
through.
We
are
dealing
with
Perishable
Produce
that
must
be
expedited
and
we
cannot
afford
any
delays.
4
i
start
with
ITACs
and
call
if
no
progress
When asked to rank the three most important service areas while using electronic
systems at FDA LOS IOB, respondents identified turnaround times to be most important,
followed by communication of FDA entry decisions, and ability to interact with FDA
(Figure 36). Additional comments, suggestions, and recommendations from survey
respondents are captured in Table 12 and point to further need to shorten response times,
12
15
33
35
30
14
2
2
2
1
3
1
0%
20%
40%
60%
80%
100%
Submijng
Entry
Documents
Submijng
LocaVon/
Availability
RequesVng
Entry
Status
Number
of
Respondents
(Totaling
100%
n)
Preferred
System
for
Given
TransacVons
LAImports
ITACS
Non-‐electronic
No
Preference
99
build feedback mechanisms, provide the ability to reach an FDA agent, and be able to
send emails from within ITACS.
Figure 36: “Please select the three (3) service areas below that are most important
to you when using electronic systems at the FDA Los Angeles Import
Operations Branch.” (Q30: n=34)
Table 12: “Do you have any additional comments, suggestions, or
recommendations?” (Q33: n=6)
#
Response
1
Thank
you
for
this
study.
I
hope
you
could
come
up
with
a
better
system
than
current
than
may
facilitate
and
expedite
feedback
and
turn-‐around
times
and
accuracy
of
FDA
handling
of
imports.
Henceforth,
imports
personnel
like
me
will
have
easier
than
the
current
in
dealing
with
FDA
and
hopefully
this
study
can
transcend
to
other
government
agencies,
as
well.
2
It
would
be
great
if
we
could
contact
FDA
right
from
ITACS
instead
of
having
to
email
from
our
company
email
system.
3
FDA
needs
to
speed
up
their
response
time.
4
There
must
be
a
HUMAN
available
to
answer
urgent
questions.
5
none
6
We
do
not
handle
much
FDA
shipment.
0
0
1
1
2
3
4
5
7
8
11
17
20
23
0
5
10
15
20
25
User
Interface
System
CustomizaVon
FDA-‐Industry
Partnership
Computer
Security
When
Transmijng
InformaVon
Ability
to
Provide
Feedback
Relevance
of
InformaVon
Provided
by
the
FDA
ExplanaVon
of
Services
Provided
System
Availability
When
Needed
ExplanaVon
of
Processing
Timeframes
Completeness
of
InformaVon
Provided
by
the
FDA
Clear
InstrucVons
for
Use
Ability
to
Interact
with
the
FDA
CommunicaVon
of
FDA
Entry
Decisions
Turnaround
Times
Total
Number
of
Responses
Service
Areas
Ranked
by
Respondents
100
4.2.6 Cross Tabulations
A representative number of cross tabulations are shown below; full cross
tabulation data are available in Appendix F. Green cells indicate responses contributing
to satisfaction or likelihood or preference with either LAImports or ITACS, and red cells
indicate noncontributing responses.
[Continuing to Use Systems & Recommending Systems Using Positive Terms]*[System
Preferences]
According to HowTo.gov, metrics assessing “continued use” and “recommending
use” might be related to preference and loyalty factors. Thus, questions on HowTo.gov
metrics (Questions 8 and 9) were cross tabulated with questions on system preference
(Question 6). Tables 13 and 14 (Q8 vs. Q6) demonstrate that: 1) users most satisfied with
continuing to use LAImports preferred LAImports for requesting status, but preferred
ITACS for submitting entry documents and location/availability information; and 2) users
most satisfied with continuing to use ITACS preferred ITACS for submitting entry
documentation and location/ availability information, but also preferred LAImports for
requesting status. Similar response patterns are seen when Q9 data on recommending use
of the two systems are cross tabulated with Q6 data on system preferences.
101
Table 13: Cross Tabulation of Q6 on Continue Use of LAImports with Q8
Operation-specific Preferences (L=LAImports; I=ITACS; NE=Non-
electronic; NP=No Preference; Σ=Total)
LAIMPORTS
Q6.
“Which
system
do
you
prefer
to
use
for
the
following
transactions
with
the
FDA
Los
Angeles
Import
Operations
Branch:
Submitting
Entry
Documents?
Q6.
“Which
system
do
you
prefer
to
use
for
the
following
transactions
with
the
FDA
Los
Angeles
Import
Operations
Branch:
Submitting
Location/Availability
Information?
Q6.
“Which
system
do
you
prefer
to
use
for
the
following
transactions
with
the
FDA
Los
Angeles
Import
Operations
Branch:
Requesting
Entry
Status?
Σ=5
Σ=17
Σ=7
Σ=15
Σ=14
Σ=8
Q8.
“How
satisfied
are
you
with
continuing
to
use:
LAImports?”
L
I
NE
NP
Σ
L
I
NE
NP
Σ
L
I
NE
NP
Σ
Very
Satisfied
3
7
1
0
11
5
5
1
0
11
9
1
1
0
11
Satisfied
2
8
0
1
11
2
7
0
2
11
5
6
0
0
11
Neither
Dis/Satisfied
1
9
0
0
10
1
8
1
0
10
5
4
1
0
10
Dissatisfied
1
2
0
0
3
2
1
0
0
3
3
0
0
0
3
Very
Dissatisfied
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Σ
7
26
1
1
35
10
21
2
2
35
22
11
2
0
35
Table 14: Cross Tabulation of Q6 on Continue Use of ITACS with Q8 on
Operation-specific Preferences (L=LAImports; I=ITACS; NE=Non-
electronic; NP=No Preference; Σ=Total)
ITACS
Q6.
“Which
system
do
you
prefer
to
use
for
the
following
transactions
with
the
FDA
Los
Angeles
Import
Operations
Branch:
Submitting
Entry
Documents?
Q6.
“Which
system
do
you
prefer
to
use
for
the
following
transactions
with
the
FDA
Los
Angeles
Import
Operations
Branch:
Submitting
Location/Availability
Information?
Q6.
“Which
system
do
you
prefer
to
use
for
the
following
transactions
with
the
FDA
Los
Angeles
Import
Operations
Branch:
Requesting
Entry
Status?
Σ=29
Σ=2
Σ=24
Σ=7
Σ=11
Σ=20
Q8.
“How
satisfied
are
you
with
continuing
to
use:
ITACS?”
L
I
NE
NP
Σ
L
I
NE
NP
Σ
L
I
NE
NP
Σ
Very
Satisfied
0
17
0
0
17
2
14
0
1
17
12
5
0
0
17
Satisfied
1
12
0
1
14
3
10
0
1
14
8
6
0
0
14
Neither
Dis/Satisfied
2
2
1
0
5
2
2
0
1
5
3
1
0
1
5
Dissatisfied
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Very
Dissatisfied
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Σ
3
31
1
1
36
7
26
0
3
36
23
12
0
1
36
102
[Turnaround Times, Communication of FDA Entry Decisions & Ability to Interact with
FDA]*[Overall Customer Experience]
In Question 30, respondents ranked the top three most important service areas
when working with FDA LOS IOB to be turnaround times, communication of FDA entry
decisions, and ability to interact with FDA. To provide insight into these three areas as
they relate to the overall customer experience, Questions 24, 13, and 14 on the user
satisfaction with the top three service areas were cross tabulated with Question 28 on
overall customer experience (Q24 vs. Q28 are shown in Tables 15-16). Users most
satisfied with their overall experiences with LAImports and ITACS are those who receive
the turnaround time they need most of the time to always. Similar response patterns are
seen when satisfaction with communication of FDA entry decisions (Q13) and
satisfaction with interacting with FDA (Q14) through LAImports and ITACS are cross
tabulated with satisfaction with overall experiences with the systems (Q28).
Table 15: Cross Tabulation of Q6 on Turnaround Times with Q28 on Overall
Customer Experience for LAImports
LAIMPORTS
Q28.
“How
satisfied
are
you
with
your
overall
customer
experiences
with:
LAImports?”
Σ=10
Σ
=8
Very
Satisfied
Satisfied
Neither
Dis/Satisfied
Dissatisfied
Very
Dissatisfied
Total
Q24.
“How
often
do
you
get
turnaround
time
you
need
from
each
system?”
Always
0
0
0
0
0
0
Most
of
the
Time
3
7
2
0
0
12
Half
of
the
Time
0
4
3
0
0
7
Some
of
the
Time
0
1
0
2
0
3
Never
0
1
2
0
1
4
Total
3
13
7
2
1
26
103
Table 16: Cross Tabulation of Q6 on Turnaround Times with Q28 on Overall
Customer Experience for ITACS
ITACS
Q28.
“How
satisfied
are
you
with
your
overall
customer
experiences
with:
ITACS?”
Σ=15
Σ
=4
Very
Satisfied
Satisfied
Neither
Dis/Satisfied
Dissatisfied
Very
Dissatisfied
Total
Q24.
“How
often
do
you
get
turnaround
time
you
need
from
each
system?”
Always
4
1
0
0
0
5
Most
of
the
Time
5
5
0
0
0
10
Half
of
the
Time
1
5
1
0
0
6
Some
of
the
Time
0
2
3
0
0
5
Never
0
3
0
0
0
3
Total
10
16
4
0
0
30
[Ability to Meet Expectations]*[Overall Customer Experience]
Another variable related to a customer’s overall experience with a system might
be its ability to meet preconceived expectations. Cross tabulation of Question 25 on the
ability to meet expectations with Question 28 on overall customer experience (Tables 17
and 18) demonstrates that users most satisfied with their overall experiences with
LAImports and ITACS are those who were most satisfied with each system’s ability to
meet their expectations.
104
Table 17: Cross Tabulation of Q28 on Overall Customer Experience with Q25
on Meeting Expectations for LAImports
LAIMPORTS
Q25.
“How
satisfied
are
you
with
each
system’s
ability
to
meet
your
expectations:
LAImports?”
Σ=11
Σ
=9
Very
Satisfied
Satisfied
Neither
Dis/Satisfied
Dissatisfied
Very
Dissatisfied
Total
Q28.
“How
satisfied
are
you
with
your
overall
customer
experiences
with:
LAImports?”
Very
Satisfied
2
1
0
0
0
3
Satisfied
0
8
4
1
0
13
Neither
Dis/Satisfied
0
2
6
0
0
8
Dissatisfied
0
0
1
1
0
2
Very
Dissatisfied
0
0
0
1
0
1
Total
2
11
11
3
0
27
Table 18: Cross Tabulation of Q28 on Overall Customer Experience with Q25
on Meeting Expectations for ITACS
ITACS
Q25.
“How
satisfied
are
you
with
each
system’s
ability
to
meet
your
expectations:
ITACS?”
Σ
=19
Σ
=2
Very
Satisfied
Satisfied
Neither
Dis/Satisfied
Dissatisfied
Very
Dissatisfied
Total
Q28.
“How
satisfied
are
you
with
your
overall
customer
experiences
with:
ITACS?”
Very
Satisfied
5
5
0
0
0
10
Satisfied
1
8
5
2
0
16
Neither
Dis/Satisfied
0
1
2
0
0
3
Dissatisfied
0
0
0
0
0
0
Very
Dissatisfied
0
0
0
0
0
0
Total
6
14
7
2
0
29
105
CHAPTER 5: DISCUSSION
5.1 Introduction
Customer service has long been an area of focus for private industries because
superior service delivery and higher customer satisfaction could help an organization to
succeed in a competitive landscape. Government services, however, are not typically
confronted with market competition, so that customer service techniques are often less
advanced or emphasized (ACSI, 2013; Fountain, 2001). With the movement toward e-
government and through relatively recent executive initiatives in customer service and
accountability, there has been an attempt to bridge this gap. PAR LOS LAImports and
ITACS are two electronic systems that were implemented to streamline import processes
of FDA-regulated products entering through greater Los Angeles seaports. These
systems have been operating for several years without a formal evaluation for customer
service delivery, perhaps due to the lack of an appropriate framework for such an
assessment. The current framework and the corresponding survey instruments were
developed to address this area of need to evaluate the customer service delivery of
LAImports and ITACS within FDA LOS IOB. The results of this survey show that
although the majority of respondents expressed satisfaction with their overall user
experiences with LAImports and, even more so, with ITACS, considerable opportunities
exist to improve both systems.
5.2 Considerations of Limitations, Delimitations, and Assumptions
This study employed a mixed methods approach of document analysis and survey
distribution. These methods appeared well suited for gaining insight into customer
106
satisfaction, but both also have limitations, delimitations, and/or assumptions that must be
considered when interpreting results.
Analysis of LAImports and ITACS policy documents served as a primer for
identifying FDA-specified customer service elements, as well as unspecified elements
that should be examined further. Publicly available policy documents appeared to be
sufficient for the current study. They stated goals, services delivered, service timeframes,
and feedback mechanisms for each system, and these elements could be examined using
the study framework and available government metrics. Thus, it was not necessary to
request further documentation via the Freedom of Information Act as per Section 1.5.
Document analyses led to the development of a high-level framework for
customer service evaluation, and the subsequent survey was intended to study customer
service delivery, which was constructed from the principles of customer service,
transparency, public participation, and collaboration. Each of these four principles was
divided further into multiple dimensions to provide further granularity to the assessments.
This approach helped to ensure that several well accepted concepts relating to consumer
satisfaction were considered. It further helped to build construct validity into the study
framework and survey instrument (Orcher, 2006). Tiered reviews of the survey were also
found useful to improve the validity of the tool. First, the use of a focus group helped to
improve validity by assuring that questions were critiqued by knowledgeable individuals
about the appropriateness of the questions for the purposes of the study (Orcher, 2006).
Further enhancement of validity was thought to come from the reviews by the LACBFFA
PGA committee chair and another senior member of her organization, who were both
experienced in managing FDA-regulated entries, and by the LACBFFA Board of
107
Directors, who are senior members of the international trade community (Orcher, 2006).
Finally, the internal review of the survey by top management within FDA LOS IOB
provided useful input. Positive feedback on the survey received from multiple user
groups helped to build confidence in the internal validity of the survey instrument.
Furthermore, this process was helpful in gaining support from experts who could assist
with the identification of recipients of the survey, thereby contributing to additional
confidence in the external validity of the sampled population.
Other ways of sampling potential end-users were considered but rejected for
different reasons. For example, HowTo.gov provides guidance for pop-up or online
surveys for websites, where respondents are sampled among website visitors (Flagg,
2013). While this would be an alternative sampling and survey technique for the ITACS
web portal, it was not possible in this study because it would require access by an
authorized administrator, unavailable to us, in order to reprogram the ITACS webpage.
HowTo.gov also suggests that internal customer lists could be used for email surveys
(Flagg, 2013). Such an approach was not feasible because I did not have access to the
full list of industry users of LAImports and ITACS. HowTo.gov, however, does advise
that without such list, one can be purchased from “list brokers”, who maintain databases
of volunteers for surveys (Flagg, 2013). By recruiting participants through an
import/export trade association, I was essentially working with a type of list broker. This
approach may have been limited by the fact that only some of the 525 members of
LACBFFA handle FDA-regulated entries. However, the use of email survey methods in
cooperation with a third party such as LACBFFA has the great advantage of ensuring that
the survey is propagated widely within the specific group targeted for this study. The
108
recipients can subsequently forward the email to their colleagues who may also use the
systems. Two limitations of this approach are difficulty in knowing the number of
potential respondents who actually received the survey, and whether respondents were
primary recipients of the email directly from the LACBFFA PGA committee chair, or
were secondary and subsequent respondents who were forwarded the email by primary
recipients.
Working with a list broker, a government-recognized alternative to internal lists, I
was also able to launch one survey instrument for both systems. In fact, a separate survey
on the ITACS webpage could have been disadvantageous because pop-up surveys tend to
have “significantly lower” response rates than email surveys, and thus might have
skewed the responses to one analysis preferentially (Flagg, 2013). Further, the
LACBFFA has a strong working relationship with FDA LOS IOB as discussed in section
3.1, and the strength of the customer-agency relationship is the largest influencer of
response rate (Flagg, 2013). Thus, while potential survey respondents who were initially
identified through the LACBFFA may have composed only a subset of the total number
of industry users of LAImports and ITACS with respect to FDA’s Los Angeles import
operations, the selected sampling techniques were felt to be appropriate for the study, in
that they maximized opportunities to increase response rate to a single survey
incorporating questions on both LAImports and ITACS.
One other tool to increase survey response rates is a follow-up email (Flagg,
2013). From my email reminder, 15 additional surveys were collected. This second
stage of reminders elicited a further survey response rate of 3% (n=489). Subtracting
these 15 from the total 57 surveys collected, the first stage of LACBFFA solicitation had
109
an estimated survey response rate of 9% (n=489). Together, the 57 responses from the
addressed pool of 489, resulting in a 12% response rate, compares favorably to the
expected 10-15% response rate identified by HowTo.gov as typical for “a carefully
targeted email survey” (Flagg, 2013). A number of factors may have accounted for the
failure of many recipients to respond. Some may have dismissed survey requests because
they do not handle FDA-regulated entries. Thus, the calculated response rate is probably
too conservative if one were to be able to remove the inappropriate recipients from the
overall pool. If only 21% of entry lines fall under FDA jurisdiction, as previously
calculated from executive reports (CBP, 2013; FDA, 2012e), the appropriately addressed
recipient pool would be reduced from 489 to 103. This assumes that the FDA and non-
FDA lines are transmitted under different entries and are handled by different subgroups.
In such a case, the response rate would be recalculated to be 55%, which is much higher
than the 12% value calculated earlier. The underestimated and overestimated 12-55%
response rate is in line with the varied response rates reported elsewhere in the literature.
A study of a sample of journal articles published in 2005 found response rates for surveys
collected from individuals to range from 3.0 to 91.2%, with a mean of 52.7% and a
standard deviation (SD) of 21.2 (Baruch & Holtom, 2008). The same study found a
response rate range of 23.7-89.0% with a mean 54.7 and SD 23.9% for emailed surveys,
and 27.0-82.8% with a mean 54.5 and SD 16.7% for surveys in public/state sectors
(Baruch & Holtom, 2008). Another article reports response rates for emailed surveys
without follow-up to approximate 25-30%, and approximate 60% for those with
“multimode contacts” (Finchman, 2008). Another review found a 15-year downward
trend in email survey responses, with a mean response rate of 61.5% in 1986 and a mean
110
of 24.0% in 2000 (Sheehan, 2001). Response rates may still be on the decline, as
evidenced by one article reporting a 1% response rate for a survey of pharmaceutical
industry professionals who are also members of professional organizations (Hoffman,
2010).
Survey results raised questions of whether the findings are reliable and could be
generalized to broader populations. Internal consistency within the survey results was
evaluated to assess reliability of the study findings (Orcher, 2006). Although the levels
of satisfaction varied, users of both LAImports and ITACS expressed satisfaction with
most dimensions, and there was considerable agreement within each principal metric.
This is consistent with the findings in Figure 35 where majority of users expressed
satisfaction with their overall customer experience. Reliability of the study was further
supported by the results of cross tabulation of system preference and HowTo.gov metrics
(Tables 13-16), where positive relationships were observed between the two.
It would appear from internally consistent and reliable results that the framework
and survey instrument measured what they were intended to measure, but can the data
collected be generalized to all users of LAImports and ITACS within FDA LOS IOB? As
noted in Section 1.5, the limited number of respondents willing to participate in a survey
such as this limits the size of studies, and thereby impacts external validity and the
robustness of the conclusions that can be drawn (Shadish, Cook, & Campbell, 2002).
Although not all LACBFFA members responded to the survey, members of the PGA
committee, which liaises with FDA and authoritatively represents LACBFFA on FDA-
regulated entries, did respond. LACBFFA has 245 member firms, which would represent
69% of the 357 licensed customs brokerage firms registered with CBP for the seaports of
111
Los Angeles and Long Beach (CBP, 2014). Thus, views of survey respondents could
represent the majority of brokers and filers working with FDA LOS IOB, and their views
mirror those of the importers—the ultimate customers of LAImports and ITACS—they
represent under power of attorney (CBP, 2006). Survey respondents mostly handle foods
and devices from the Far East, which are the typical commodities and import region of
origin seen by FDA LOS IOB at “The Gateway to the Orient” (FDA, 2013c). Further,
most (92%, Figure 9) respondents are experienced and knowledgeable in working with
FDA-regulated entries and, thus, should provide good insight on the use of both non-
electronic and electronic modes of communication with FDA LOS IOB. They also fit the
archetypal FDA-regulated industry profile for Los Angeles ports and, as such, data from
this survey should be broadly applicable to other LAImports and ITACS users.
5.3 Consideration of Results
The primary questions raised in Chapter 2 with regard to customer service were
simple: 1) can customer satisfaction with regard to the two systems be characterized and
compared? And 2) what causes one system to be chosen as the method of preference
over the other? From the results of these questions comes a third, implicit question: 3) is
the study’s integrated framework adequate in evaluating customer service delivery?
Findings related to these questions are discussed below according to the following three
areas: 1) characterization and comparison of customer service for LAImports and ITACS;
2) system preferences and loyalty for LAImports and ITACS; and 3) adequacy of the
study framework as a customer service tool.
112
5.3.1 Characterization and Comparison of Customer Service for LAImports and ITACS
Study findings related to customer service for the two electronic systems can be
broadly divided into areas of positive, negative, and intermediate responses.
5.3.1.1 Areas of Positive Response
The areas of performance that seemed most reliable and/or satisfactory for users
of both systems appeared to be: 1) system availability and 2) computer security. A
common feature of both areas is their relationship with IT infrastructure. The fact that
97% of ITACS users and 81% of LAImports users report that the systems are available
most of the time when they are needed, and that they feel protected against IT threats
suggests that system availability and security are not at issue. While the question on
system availability did not ask respondents about their satisfaction explicitly, it is likely
that users whose needs are met with respect to system availability as with the
confirmation-disconfirmation paradigm discussed in section 2.6.1, would lead us to
predict (according to the confirmation-disconfirmation paradigm discussed in section
2.6.1) that the ability of the system to meet expectations in this domain would be
satisfying.
When survey responses related to the two different systems, ITACS and
LAImports, were compared, it appeared that ITACS outscored LAImports in positive
responses for all dimensions and metrics studied, with the largest score differential
associated with completing transactions. Nearly all ITACS users could complete
intended transactions most of the time or always, but only about two-thirds of LAImports
users could make the same claim (Figure 31). One possible explanation for this
113
difference is the fact that LAImports has been operational for two years longer than
ITACS. If users are reflecting on their experiences with failed transactions since system
launch, they would have two more years of negative experiences to report for LAImports.
Alternatively, differences may relate to the requirements of the user interface.
LAImports users are required to satisfy formatting requirements for submissions, but
ITACS does not require such formatting. Formatting requirements may be burdensome
or difficult to remember, and could lead to failed transactions.
5.3.1.2 Areas of Negative Response
Three top areas of negative experience identified by the users of both systems are:
1) explanation of FDA processing times; 2) turnaround times for transactions; and 3)
ability to provide feedback to FDA, features respectively related to transparency,
customer service, and public participation. These three areas appear to be related.
Dissatisfaction of most users with explanations of processing times (Figure 24) is perhaps
not surprising considering that many users did not receive their needed turnaround times
more than half the time (Figure 20); we might presume that many of these would initiate
a request for explanation of the delay. Their responses are consistent with the findings of
a recent American Consumer Satisfaction Index (ACSI) report that dealt with concerns
with respect to the services across the broader federal government. According to that
report, users of government services believe that information should be delivered in a
more timely fashion (ACSI, 2014). When turnaround times are not met, industry needs a
means of communicating with FDA to overcome roadblocks. Of concern are results
suggesting that only about half of LAImports and ITACS users were satisfied with their
114
ability to provide feedback to FDA (Figure 28). Together, these areas represent
impediments to the entry of products into domestic commerce.
Explanation of Processing Times
ITACS provides automatic linkage of documents to an entry at the time of
document submission, but its policy documents do not inform users about the subsequent
processing and review times that they should expect (FDA, 2012c). On the other hand,
LAImports policy documents identify explicit expectations of 1-2 business days to
process emails and 2 business days to review documents submitted with an email (FDA,
2012d). The relatively low satisfaction score for LAImports may be due to the policy of
specifying (and then failing to deliver on) the defined 2-4 business days to complete
review of a document. Inability to meet expected timeframes is likely to result in
dissatisfaction, as predicted by the confirmation-disconfirmation paradigm (Churchill &
Surprenant, 1982). Such failures to meet expectations have been identified as “process
failures” by others. (Meuter, Ostrom, Roundtree, & Bitner, 2000). No matter how
failures are explained in relationship to an underlying concept, they are likely to be a
source of frustration or dissatisfaction. Perhaps the 2-4 business days window is not
feasible for FDA LOS IOB, as seems to be the case also for other websites, including
private websites, that process email inquiries (Wilcox, 1999; Yang & Jun, 2008). If
private industries cannot respond to emails in timeframes shorter than five days, perhaps
FDA LOS IOB should rethink its own timeframes.
Turnaround Times
In the present survey, only a minority of ITACS users reported that their dossiers
were handled in the needed timeframe. This challenge was illustrated emphatically by
115
the text response of a user who was wary of ITACS because transactions related to
perishable imports were sometimes unsuccessful, and this could hamper their ability to
sell the contents of the shipment. Notably, for LAImports, none of the users reported that
transactions are “always” processed within the needed turnaround time. A difference
between the responsiveness of the two systems was underlined by the comment of one
user that ITACS reviewers seem more “proactive” and were willing to contact industry
users immediately when additional information is needed, whereas LAImports reviewers
could take days to respond (Table 10, #2). Turnaround, by definition, requires FDA to
reciprocate an action. The fact that respondents identified instances when LAImports
failed to respond entirely would suggest clear breakdowns in FDA turnaround procedure
(Table 8, #2; Table 10, #5). Such breakdowns, or process failures, would be expected to
result in customer dissatisfaction (Meuter et al., 2000).
Feedback Mechanisms
The inability to provide feedback when a process failure is encountered could
further exacerbate feelings of dissatisfaction. Both ITACS and LAImports employ email
communications as feedback mechanisms. It is relatively easy to send email via
LAImports, because the system is already based on email interactions. In contrast,
ITACS has a more cumbersome system for feedback that requires the user to exit the
ITACS webpage and email the support team using their firm’s email system.
Accordingly, this was the one dimension in which ITACS appeared less helpful than
LAImports. The inconvenience of the ITACS process was reflected in the user response
that stated: “It would be great if we could contact FDA right from ITACS instead of
having to email from our company email system” (Table 12, #2).
116
The two systems also differ with respect to the location of the FDA staff members
who handle each system. It may be possible that industry perceives that communications
with ITACS, managed at FDA headquarters in Washington, DC, are less personal and
more constrained than communications with the local FDA office on the west coast,
whose staff members come to be known personally to the users and they can then put a
“face to the name”.
5.3.1.3 Areas of Intermediate Response
Not all dimensions generate clear results of satisfaction or dissatisfaction,
frequency or infrequency, agreement or disagreement, or likelihood or unlikelihood. In
fact, intermediate responses on the Likert scales were found to occur frequently in this
study. Such midpoint responses are difficult to interpret. An intermediate response may
be due to ambivalence, neutrality, uncertainty, or indecision on the part of the respondent
(Klopfer & Madden, 1980). The top three areas in which intermediate scores were most
common were: 1) explanation of services; 2) customization; and 3) each system’s ability
to meet customer expectations.
Intermediate responses on explanation of services may be related to ambiguity of
the survey question: “How satisfied are you with FDA’s explanation of services provided
by LAImports? ITACS?” While the question was intended to solicit level of satisfaction
with individual transactions such as document submission and status inquiry, survey
respondents may have interpreted the question to mean a more comprehensive
explanation of service delivery through document review and final entry release or
refusal. If the respondents were considering a palette of elements, some positive and
117
some negative, they might have to combine them into one response with mixed
assessments related to different functional aspects. For example, users may be aware of
different transaction types, but are unable to predict processing and turnaround times and,
hence, decide to choose the “neither satisfied nor dissatisfied” response.
System customization, perhaps another ambiguous term, is the dimension that
received the second highest level of intermediate response for both LAImports and
ITACS. While the question on customization was intended to examine satisfaction with
respect to options for system personalization, users may have interpreted customization
as the choices between the two systems for particular transactions that would allow for
customized workflow. Answering the question in this manner would require concurrent
examination of LAImports and ITACS, which could make evaluations and results less
straightforward. Interestingly, and also perhaps related to question ambiguity, users for
both systems expressed satisfaction with customization, even though neither system
offers customization options to the users. It may be that users of LAImports are satisfied
with capabilities to customize font, color, signatures, and other tools available in their in-
house email clients, and ITACS users are satisfied with their web browser settings.
Notably, when asked about customizing the systems, respondents only provided text
responses on customizing ITACS, and not LAImports (Table 6). It may be that the users
were comparing ITACS system attributes with those provided by other transactional
websites that they encounter online.
The dimension “ability of systems to meet expectations” is another area in which
the LAImports users expressed higher numbers of intermediate responses. The reason for
this is not clear, but the text responses do provide some insight. For example, an ITACS
118
user reported that one knows what can be expected from the system (Table 7, #2), which
is consistent with previous work that found customers to have “clear expectations” for
SSTs compared to human service channels (Johnson, Bardhi, & Dunn, 2008).
Surprisingly, clear expectations do not necessarily lead to better expectations. One text
respondent for example, had higher expectations for LAImports because he/she expected
human interactions to be more engaging, but unfortunately, found this not to be the case
(Table 7, #2).
5.3.1.4 Other Interesting Findings
The study results show that ITACS performed better than LAImports in almost
every dimension of the customer service framework. An exception to these findings was
in the area of feedback mechanisms as was discussed above in section 5.3.1.2. Another
area where ITACS seem to fall short was in its ability to communicate entry decisions.
This may be due to the fact that the ITACS system generates standardized boilerplate
responses, making it difficult to communicate decisions in specific terms. Such
boilerplate responses can decrease the perceptions that interactions are accountable and
transparent (Solberg & Richmond, 2012). For example, ITACS provides the
standardized text of “Detained - Refer to Notice of FDA Action for Violation Charges”,
whereas an LAImports operator may provide the specific details of the detention charges,
and may even attach the Notice of FDA Action to the email response. Users also
criticized the fact that communication with ITACS flowed only one way and that status
of a released entry is removed too quickly instead of remaining available online (Table 6
#4 and #6; and Table 11 #1). Interchanges between the user and FDA in ITACS can
119
reach a point where FDA stops communicating entry decisions altogether. This may be
one other reason for users to prefer LAImports for status inquiries.
5.3.2 System Preferences and Loyalty for LAImports and ITACS
Results of this survey suggest that respondents in industry do not wholly prefer
one system to another, but rather have transactional preferences. Respondents preferred
ITACS when submitting entry documents and location information and LAImports when
requesting status (Figure 36). Why the split? In a banking study, convenience (system
availability in the current study) was the key factor driving SST usage (Rugimbana &
Iversen, 1994). However, frustration and uncertainty appear to negate conveniences of
SST usage when problems arise (Johnson et al., 2008). In this study, respondents
preferred ITACS for activities that are straightforward and require only one-way
communication, such as the submission of documents. However, for those activities that
require feedback, such as requests for status, respondents preferred LAImports over
ITACS. These results thus are consistent with the important role of system availability
and feedback mechanisms as determinants of preference.
The preference of LAImports over ITACS for status inquiries may also touch
upon the differences between human-to-human and human-to-computer interactions.
Even for those who prefer SSTs, “human encounter satisfaction is still important in
determining their overall satisfaction” (Haytko & Simmers, 2009). Given the stronger
emphasis on governmental transparency, collaboration and public participation, as well as
the ACSI report that users of federal services most often interact with federal agencies via
120
websites, agencies must be careful not to distance customers when trying to improve
service delivery through e-government (ACSI, 2014).
5.3.3 Adequacy of Study Framework as a Customer Service Tool
Although Executive Order (EO) 13571 and HowTo.gov metrics offer federal
agencies tools to evaluate customer service, none are in place at FDA LOS IOB to
evaluate LAImports and ITACS. Even if such tools were used, they would likely be
limited to capturing data related to operational outcomes and not customer service
delivery. Therefore, the current study framework provides additional dimensions
associated with customer service delivery and effectively complements the operational
metrics.
EO 13571 metrics demonstrate that both LAImports and ITACS are achieving
operational goals (Figures 30 and 31). Both systems have reduced phone calls, faxes,
mail, and courier service channels, and thus have reduced paper consumption, spending,
lost documents, and total entry processing time at FDA LOS IOB. The HowTo.gov
metrics queried in this survey demonstrate that most users are able to complete intended
transactions through LAImports and ITACS, and are satisfied with continuing to use
them. They are also likely to recommend use of the systems to others. Overall, most
users seemed to be satisfied with their customer experiences for both systems when
viewed according to these operational metrics alone. However, when dimensions of the
study framework are applied to explore customer service delivery of the two systems, a
different landscape of user satisfaction emerges.
121
It is interesting to note that the top three service areas deemed most important to
the users (Figure 37) are those associated with principles of customer service,
transparency, and collaboration in the current study framework. For example, the
element of “turnaround time”, associated with customer service, is ranked to be the most
important service area. Disappointingly, this is an area in which the needs of neither
LAImports nor ITACS users appear to be met (Figure 20). The second most important
service area, “communication of FDA entry decisions”, associated with the transparency
principle, is also an area in which satisfaction is not compelling. In fact, respondents
were also relatively dissatisfied with other dimensions related to the transparency
principle in the current framework. Therefore, although the government tools may
suggest that the two systems are meeting operational metrics, they are relatively deficient
in assuring satisfactory customer service, transparency, public participation, and
collaboration in the eyes of its users. By using a study framework that encompasses a
broader range of critical domains, a more comprehensive view of customer service
emerges. The ability to delineate areas in which users are positive, negative and mixed in
their feelings provides insights into customer preference that may be important to drive
future improvements of the two systems.
Results from the study framework were used to develop a tool to prioritize areas
of improvement (Figure 38). In the plot, the y-axis represents the importance assigned by
respondents to a particular dimension and the x-axis represents degree of positivity in the
responses associated with satisfaction/frequency/agreement/likelihood for that same
dimension. Presented in this manner, the service areas can be categorized into four
groups that have either 1) low importance and low positive response, 2) low importance
122
and high positive response, 3) high importance and high positive response, and 4) high
importance and low positive response. In terms of risk to customer service delivery, the
latter area, designated 4, is the one that presumably should be given the highest priority
(reflected by the red shading) while categories 1 and 3 (shown in yellow) should be given
medium priority, and elements in category 2 (shown in green) should be given the lowest
priority.
Figure 37: Positive Response-Importance Priority Plot for Service Areas
The percentages used in calculating means were positive responses of very
satisfied/satisfied, always/most of the time, or strongly agree/agree for
each dimension. Axis midpoints were used to delineate quadrants, which
maintains consistency with interpretation of results throughout this study.
(e.g. The customer satisfaction value for communication of a FDA entry
decision is the mean of positive ratings for the systems in Figure 23—
[57%+63%]/2=60%).
123
5.4 Conclusions and Considerations for the Future
The survey captured, through the eyes of its users, a “snapshot in time” of
customer service delivered by FDA LOS IOB through LAImports and ITACS. The
survey results show that users were most positive toward areas related to the IT
infrastructure of the two systems, particularly with the computer security and system
availability, and less commonly with user interface and customization options. Users
were much less positive with areas related to timeliness of service delivery and associated
communications with the agency. Specifically, turnaround timeframe needs are rarely
met and explanation of processing times stands out as one area for which the users
express lowest satisfaction. Communication of entry decisions and the ability to interact
with FDA are also areas that users view as very important, but less satisfactory.
The study was successful in constructing and applying the integrated framework
to evaluate FDA LOS IOB’s electronic systems for customer service delivery against
dimensions of transparency, collaboration, and public participation, as well as available
government metrics. As governmental systems move toward addressing customer
service in a manner akin to that typical for private industry, it may be helpful to compare
the performance of these systems to what would be considered by industry as measures of
success. The systems are meeting performance goals as outlined in policy documents.
However, only two out of fifteen dimensions of the study framework received a positive
score of 80 or above, which, according to the ACSI e-government satisfaction index, is
considered to be the threshold of excellence (Freed, Lewan, Feinberg, & Anderson,
2014). Clearly, neither system operates optimally, and ample opportunities to improve
the two systems exist.
124
5.4.1 Justification for LAImports and ITACS
The majority of users of both systems do express satisfaction, and each system
does achieve its performance objectives, particularly, reductions to total import
processing times and industry spending. While the delivered services may not be judged
as excellent at this time, FDA is rightly offering its customers options across service
channels to meet different needs, which has been deemed useful for customer satisfaction
in other markets (Thornton & White, 2001). In allowing industry to make decisions
between the two systems, FDA affords itself the opportunity to deliver industry’s desired
services and improve overall satisfaction by providing service choices (Haytko &
Simmers, 2009). Thus, FDA’s expenditures on LAImports and ITACS appear justified.
Improvements to the systems will provide for more efficient use of and further
justification for appropriations.
5.4.2 Recommendations to Improve LAImports and ITACS
As seen in the SERVQUAL literature reviewed in Chapter 2, customer
satisfaction is a transactional assessment and service quality is a global assessment
(Parasuraman et al., 1988). Linking these definitions to the current study, an increase in
the level of satisfaction within each individual framework dimension should lead to an
increase in the ability of LAImports and ITACS to deliver transparent, collaborative, and
participative customer service. Thus, in order to most effectively improve overall
customer service delivery of its electronic import systems, FDA should give priority to
improving the three service areas deemed to be most important to its customers. Of
these, improving operations related to turnaround times may be at the top of the list.
125
While industry may want to expedite processing times, it is important to keep in mind
that FDA needs adequate time to make admissibility decisions. FDA should, however,
give industry realistic expectations for processing times and adhere to them to the extent
that is reasonably possible. Doing so would also reduce the need to explain long
processing times and reduce the perception that the system is not transparent.
It should be quite easy to improve communication of entry decisions for both
systems. Currently, once an entry is released, the status is no longer available online in
ITACS. This information is available to brokers/filers in ABI, and available to
brokers/filers and importers as mailed notices of action. According to LAImports policy
documents (Appendix C), reprints of notices of action may be requested after five days.
Thus, it should be possible to have the information available in ITACS for five days after
entry release so that the importers could query entry status on their own while notices of
action are en route via postal mail; five additional days of availability of statuses in
ITACS should not place undue strain on FDA’s data storage. ITACS could also increase
transparency by leaving status information online for five days longer, and by adding
more specifics to its boilerplate responses. LAImports should work toward providing
consistent communications from its responders. Most importantly, LAImports should
have ways to assure that responses are always given to emails. Both systems should
improve feedback mechanisms to prevent or minimize process failures that can cause
great dissatisfaction among the users.
The third most important service area is interaction with FDA. By giving industry
greater opportunities to feel vested in policies and operations that affect them, FDA
promotes collaboration, as per executive mandate. Incorporating public comments into
126
the development of ITACS and tailoring the system to customer specifications would
increase satisfaction by meeting user expectations. Further, tailoring the system to
increase usage turns the users into “productive resources involved in the service delivery
process”, which would lead to cost-savings at FDA (Weijters, Rangarajan, Falk, &
Schillewaert, 2007).
After the top three most important service areas, FDA should focus on dimensions
yielding mixed results because ambivalent perceptions can be more easily edged toward
satisfaction (Johnson et al., 2008; Olsen et al., 2005). Such an approach would allow
FDA to achieve more immediate improvements. And because currently available
government tools to evaluate customer service appeared to be insufficient in evaluating
LAImports and ITACS, the government should consider developing a broader set of
customer satisfaction metrics to be added to their customer service toolbox.
5.4.3 Future Research
Because this study was the first of its kind, the usefulness of the study framework
should be assessed further through follow up data collection at other time points in the
future. This is consistent with the statutory obligation of federal agencies to conduct
quarterly progress reviews of customer service. The current survey results should then
provide baseline information that can be a benchmark against which subsequent data can
be compared. In addition, the usefulness of the framework could be tested in other
sectors, both public and private, as a lower cost alternative to existing customer
satisfaction models (Fornell, Johnson, Anderson, Cha, & Bryant, 1996; Lewan &
Anderson, 2014).
127
While the proposals above could take advantage of the already established survey
utilizing the extended framework developed by reference to broad literature, there
nonetheless may exist opportunities to improve data. Should FDA decide to conduct
future data collections of this sort, including a survey link in the body of LAImports
emails and on ITACS screens may allow FDA to better sample among system users. The
survey questions should be uniformly phrased to measure customer service delivery and
satisfaction and improve cross-survey comparisons.
128
REFERENCES
ACSI. (2013). Commentary on U.S. Federal Government. Retrieved June 17, 2013, from
http://www.theacsi.org/images/stories/images/commentaries/13febgov_comm.pdf
ACSI. (2014). ACSI Federal Government Report 2013. Retrieved April 6, 2014, from
http://www.theacsi.org/images/stories/images/commentaries/14jan-report-
final.pdf
Andaleeb, S. S. (1998). Determinants of customer satisfaction with hospitals: a
managerial model. International Journal of Health Care Quality Assurance,
11(6), 181-187.
Baruch, Y., & Holtom, B. C. (2008). Survey response rate levels and trends in
organizational research. Human Relations, 61(8), 1139-1160.
Carter, L., & Bélanger, F. (2005). The utilization of e-government services: citizen trust,
innovation and acceptance factors. Information Systems Journal, 15(1), 5-25.
CBP. (2006). Importing into the United States: A Guide for Commerial Importers. (CBP
Publication No. 0000-0504). Washington, DC. Retrieved from
http://www.cbp.gov/linkhandler/cgov/newsroom/publications/trade/iius.ctt/iius.pd
f
CBP. (2011). Import Trade Trends: Fiscal Year 2011 Mid Year Report. Washington, DC:
Retrieved from
http://www.cbp.gov/linkhandler/cgov/trade/trade_programs/trade_trends/trade_tre
nd_2011.ctt/trade_trend_2011.pdf
CBP. (2012). Public Bulletin LA13-011: LA/LB Seaport Procedures for CBP/FDA
Refused Merchandise. Los Angeles, CA.
CBP. (2013). Import Trade Trends: Fiscal Year 2012, Year-End Report. (CBP
Publication 0173-0113). Washington, DC: Office of International Trade.
Retrieved from http://www.cbp.gov/xp/cgov/trade/trade_programs/trade_trends/
CBP. (2014). U.S. Customs and Border Protection Brokers. Retrieved April 18, 2014,
from http://apps.cbp.gov/brokers/index.asp?portCode=2704
Chun, S. A., Shulman, S., Sandoval, R., & Hovy, E. (2010). Government 2.0: Making
connections between citizens, data and government. Information Polity, 15(1), 1-
9.
129
Churchill, G. A., Jr., & Surprenant, C. (1982). An investigation into the determinants of
customer satisfaction. Journal of Marketing Research, 19(4), 491-504.
Clinton, W. J. (1993). Executive Order 12862: Setting Customer Service Standards.
Washington, DC. Retrieved from http://www.archives.gov/federal-
register/executive-orders/pdf/12862.pdf
Clinton, W. J. (1995). Memorandum on Customer Service. Washington, DC: The White
House, Office of the Press Secretary. Retrieved from
http://www.gpo.gov/fdsys/pkg/WCPD-1995-03-27/pdf/WCPD-1995-03-27-
Pg456.pdf
Clinton, W. J. (1998). Memorandum on Conducting "Conversations with America" to
Further Improve Customer Service. Washington, DC. Retrieved from
http://www.gpo.gov/fdsys/pkg/PPP-1998-book1/pdf/PPP-1998-book1-doc-
pg319-2.pdf
Croley, S. (2011). The Obama Administration's Commitment to Open Government: A
Status Reports. Washington, DC. Retrieved from
http://www.whitehouse.gov/blog/2011/09/16/status-report-administration-s-
commitment-open-government
Dimapindan, P. R. (2012). PAR LOS LAImports & ITACS: Electronic Communications
Updates at the US Food & Drug Administration Los Angeles District, Import
Operations Branch. Torrance, CA.
E-Government Act of 2002, Pub. L. No. 107-347, 116 Stat. 2899 (2002).
Evans, D., & Yen, D. C. (2005). E-government: An analysis for implementation:
Framework for understanding cultural and social impact. Government Information
Quarterly, 22(3), 354-373.
Eysenbach, G. (2008). Medicine 2.0: Social Networking, Collaboration, Participation,
Apomediation, and Openness. Journal of Medical Internet Research, 10(3).
FCG. (2011, October 20). ACSI: Measuring customer satisfaction and benchmarking
your agency scores across government and industry. Retrieved July 28, 2012
from http://www.fcg.gov/measuringcustomersatisfaction.shtml
FDA. (2009). Questions and Answers from the November 19, 2009 FDA-NCBFAA
Webinar. Retrieved September 23, 2012, from
http://www.fda.gov/downloads/ForIndustry/ImportProgram/UCM198644.pdf
130
FDA. (2011a). FDA Transparency Initiative: Improving Transparency to Regulated
Industry. Retrieved July 3, 2012, from
http://www.fda.gov/downloads/AboutFDA/Transparency/TransparencytoRegulate
dIndustry/PhaseIIITransparencyReport/UCM239088.pdf
FDA. (2011b). Fiscal Year 2012 Food and Drug Administration Congressional
Justification - State Fact Sheets. Retrieved January 30, 2012, from
http://www.fda.gov/downloads/AboutFDA/ReportsManualsForms/Reports/Budge
tReports/UCM283037.pdf
FDA. (2011c). Investigations Operations Manual. Retrieved from
http://www.fda.gov/ICECI/Inspections/IOM/default.htm
FDA. (2011d). PREDICT Briefing Slides for Importers and Entry Filers. Retrieved from
http://www.fda.gov/downloads/ForIndustry/ImportProgram/UCM176726.pdf
FDA. (2011e). Regulatory Procedures Manual. Retrieved from
http://www.fda.gov/ICECI/ComplianceManuals/RegulatoryProceduresManual/de
fault.htm
FDA. (2011f). Strategic Priorities 2011-2015: Responding to the Public Health
Challenges of the 21st Century. Retrieved from
http://www.fda.gov/downloads/AboutFDA/ReportsManualsForms/Reports/UCM
252092.pdf
FDA. (2012a). Global Engagement: US Food and Drug Administration. Silver Spring,
MD. Retrieved from
http://www.fda.gov/downloads/AboutFDA/ReportsManualsForms/Reports/UCM
298578.pdf
FDA. (2012b). ITACS Facts Sheet #2. Retrieved from
http://www.fda.gov/downloads/ForIndustry/ImportProgram/UCM316470.pdf.
FDA. (2012c). ITACS Import Trade Auxiliary Communications System Overview and
Walkthrough of Functionality. Retrieved from
http://www.fda.gov/downloads/ForIndustry/ImportProgram/UCM296352.pdf.
FDA. (2012d). PAR LOS LAImports Email Guidelines. San Pedro, CA.
FDA. (2012e). Predictive Risk-base Evaluation for Dynamic Import Compliance
Targeting (PREDICT). Retrieved from
http://www.fda.gov/downloads/ForIndustry/ImportProgram/UCM310772.pdf.
131
FDA. (2013, July 25). FDA Import Office Locations and Contact Information. Retrieved
August 8, 2013, from
http://www.fda.gov/ForIndustry/ImportProgram/ucm319216.htm
FDA. (2013b). FDA ITACS Public Statuses. Retrieved April 6, 2014, from
http://www.fda.gov/downloads/ForIndustry/ImportProgram/UCM348679.pdf
FDA. (2013c, May 7). Fiscal Year 2014 Food and Drug Administration Congressional
Justification - State Fact Sheets. Retrieved May 5, 2014, from
http://www.fda.gov/downloads/AboutFDA/ReportsManualsForms/Reports/Budge
tReports/UCM349814.pdf
FDA. (2014a). ITACS Facts Sheet. Retrieved April 5, 2014, from
http://www.fda.gov/downloads/ForIndustry/ImportProgram/UCM316470.pdf
FDA. (2014b). Predictive Risk-based Evaluation for Dynamic Import Compliance
Targeting (PREDICT). Retrieved April 2, 2014, from
http://www.fda.gov/downloads/ForIndustry/ImportProgram/UCM310772.pdf
Finchman, J. E. (2008). Response Rates and Responsiveness for Surveys, Standards, and
the Journal. American Journal of Pharmaceutical Education, 72(2).
Flagg, R. (2013, May 16). Online Surveys. Retrieved April 8, 2014, from
http://www.howto.gov/customer-experience/collecting-feedback/online-surveys-
fact-sheet
Fornell, C., Johnson, M. D., Anderson, E. W., Cha, J., & Bryant, B. E. (1996). The
American Customer Satisfaction Index: Nature, purpose, and findings. Journal of
Marketing, 60(4), 7-18.
Fountain, J. E. (2001). Paradoxes of Public Sector Customer Service. Governance, 14(1),
55-73.
Fox, M., & Piazza, A. (2013). Digital Metrics for Federal Agencies. Retrieved from
http://www.howto.gov/web-content/digital-metrics
Freed, L., Lewan, D., Feinberg, E., & Anderson, J. (2014). ACSI E-government
Satisfaction Index (Q3 2013): Mobile Naturalization in Digital Government.
Retrieved April 7, 2014, from http://www.foreseeresults.com/research-white-
papers/acsi-egov-q3-2013-form-foresee.shtml
FTASC. (2012). Foreign Trade Association of Southern California. Retrieved January
27, 2012, from http://www.ftasc.org/
GPRA Modernization Act of 2010, Pub. L. No. 111-352, 124 Stat. 3866 (2010).
132
Haytko, D. L., & Simmers, C. S. (2009). What's your preference? Management Research
News, 32(4), 337-353.
Hoffman, M. (2010). Survey Says. Pharmaceutical Technology, 34(9), 12.
JOC. (2012). Special Report: The JOC Top 50 World Container Ports. Journal of
Commerce (August 20-27, 2012), 27-30.
Johnson, D. S., Bardhi, F., & Dunn, D. T. (2008). Understanding how technology
paradoxes affect customer satisfaction with self-service technology: The role of
performance ambiguity and trust in technology. Psychology and Marketing, 25(5),
416-443.
Klopfer, F. J., & Madden, T. M. (1980). The Middlemost Choice on Attitude Items:
Ambivalence, Neutrality, or Uncertainty? Personality and Social Psychology
Bulletin, 6(1), 97-101.
LACBFFA. (2012). Los Angeles Customs Brokers and Freight Forwarders Association,
Inc. Retrieved January 27, 2012, from http://www.lacbffa.org/
Lew, J. J. (2011). M-11-17: Delivering on the Acountable Government Initiative and
Implementing the GPRA Modernization Act of 2010. Washington, DC. Retrieved
from http://www.whitehouse.gov/sites/default/files/omb/memoranda/2011/m11-
17.pdf
Lew, J. J., & Zients, J. D. (2011). M-11-31: Delivering an Efficient, Effective, and
Accountable Government. Washington, DC. Retrieved from
http://www.whitehouse.gov/sites/default/files/omb/memoranda/2011/m11-31.pdf
Lewan, D., & Anderson, J. (2014). The Foresee E-Government Satisfaction Index (Q4
2013). Retrieved April 7, 2014, from http://www.foresee.com/research-white-
papers/_downloads/e-gov-q4-2013-foresee.pdf
Lin, J. C., & Hsieh, P. (2011). Assessing the self-service technology encounters:
development and validation of SSTQUAL scale. Journal of Retailing, 87(2), 194-
206.
Lollar, X. L. (2006). Assessing China's E-Government: information, service,
transparency and citizen outreach of government websites. Journal of
Contemporary China, 15(46), 31-41.
Marche, S., & McNiven, J. D. (2003). E-Government and E-Governance: The Future Isn't
What It Used To Be. Canadian Journal of Administrative Sciences / Revue
Canadienne des Sciences de l'Administration, 20(1), 74-86.
133
Metzenbaum, S. (2010). M-10-24: Performance Improvement Guidance: Management
Responsibilities and Government Performance and Results Act Documents.
Washington, DC. Retrieved from
http://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10-
24.pdf
Meuter, M. L., Ostrom, A. L., Roundtree, R. I., & Bitner, M. J. (2000). Self-Service
Technologies: Understanding Customer Satisfaction with Technology-Based
Service Encounters. Journal of Marketing, 64(3), 50-64.
Munger, M. C. (2000). Analyzing Policy: Choices, Conflicts, and Practices. New York,
NY: W.W. Norton & Company, Inc.
Obama, B. (2009). Memorandum of January 21, 2009: Transparency and Open
Government. (E9-1777). Washington, DC: Federal Register. Retrieved from
http://www.gpo.gov/fdsys/pkg/FR-2009-01-26/pdf/E9-1777.pdf
Obama, B. (2011a). Executive Order 13571: Streamlining Service Delivery and
Improving Customer Service. Washington, DC: The White House, Office of the
Press Secretary. Retrieved from http://www.whitehouse.gov/the-press-
office/2011/04/27/executive-order-streamlining-service-delivery-and-improving-
customer-ser
Obama, B. (2011b). Executive Order 13576: Delivering an Efficient, Effective, and
Accountable Government. Washintgon, DC: The White House, Office of the Press
Secretary. Retrieved from http://www.whitehouse.gov/the-press-
office/2011/06/13/executive-order-delivering-efficient-effective-and-accountable-
governmen
Olsen, S. O., Wilcox, J., & Olsson, U. (2005). Consequences of ambivalence on
satisfaction and loyalty. Psychology and Marketing, 22(3), 247-269.
Orcher, L. T. (2006). Conducting a Survey: Techniques for a Term Project. Glendale:
Pyrczak Publishing.
Orszag, P. R. (2009a). M-10-01: Increased Emphasis on Program Evaluation.
Washington, DC. Retrieved from
http://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10-
01.pdf
Orszag, P. R. (2009b). M-10-06: Open Government Directive. Washington, DC.
Retrieved from
http://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10-
06.pdf.
134
Parasuraman, A., Berry, L. L., & Zeithaml, V. A. (1991). Refinement and Reassessment
of the SERVQUAL Scale. Journal of Retailing, 67(4), 420-450.
Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1988). Servqual: A Multiple-Item
Scale For Measuring Consumer Perc. Journal of Retailing, 64(1), 12-40.
Reilly, F. W., Lew, H., Hadesty, W. D., Bell, K. D., DelVecchio, F. M., & Chin, S. T.
(1995). FDA Import Automation: Serious Management and Systems Development
Problems Persist. (GAO/AIMD-95-188). Washington, DC: GAO. Retrieved from
http://www.gao.gov/assets/230/221876.pdf
Rugimbana, R., & Iversen, P. (1994). Perceived attributes of ATMs and their marketing
implications. The International Journal of Bank Marketing, 12(2), 30.
Rust, R. T., & Zahorik, A. J. (1993). Customer satisfaction, customer retention, and
market share. Journal of Retailing, 69(2), 193-215.
Ryan, M. J., Buzas, T., & Ramaswamy, V. (1995). Making CSM a power tool. Marketing
Research, 7(3), 10-16.
Sadri, H. (2012, October 9). The difference between the business model, framework and
architecture. Business Model Institute. Retrieved July 20, 2013, from
http://businessmodelinstitute.com/the-difference-between-the-business-model-
framework-and-architecture/?goback=%2Egde_4171511_member_173423260
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-
Experimental Designs for Generalized Causal Inference. Belmont: Wadsworth
Cengage Learning.
Sheehan, K. B. (2001). E-mail Survey Response Rates: A Review. Journal of Computer-
Mediated Communication, 6(2), 0-0.
Solberg, M., & Richmond, F. J. (2012). Transparency in Drug Submission Processes of 3
Asian Countries: A Survey of Industry Views. Drug Information Journal, 46(2),
216-225.
Solis, D. R. (2010). FDA Presentation on Import Process and Other Current News.
Retrieved February 7, 2012, from
http://www.lachamber.com/clientuploads/Global_Programs/FDAImportProcessE
verything.pdf
Steyaert, J. C. (2004). Measuring the performance of electronic government services.
Information & Management, 41(3), 369-375.
135
Sustein, C. R., & Zients, J. D. (2011). M-11-26: New Fast-Track Process for Collecting
Service Delivery Feedback Under the Paperwork Reduction Act. Washington,
DC. Retrieved from
http://www.whitehouse.gov/sites/default/files/omb/memoranda/2011/m11-26.pdf
Thornton, J., & White, L. (2001). Customer orientations and usage of financial
distribution channels. The Journal of Services Marketing, 15(3), 168-185.
Veneziano, D. J. (2014, Februar 20). What You Really Need to Know About Importing
into the US. Food and Agriculture Border Summit. Retrieved April 2, 2014, from
http://www.michigan.gov/documents/mdard/Veneziano_FDA_February_20_2014
_1_448707_7.pdf
Verbeten, J. E. (2011). FDA’s Import Operations: How FDA Regulates Imported
Products. FDA Basics Webinar Series. Retrieved January 27, 2012, from
http://www.fda.gov/downloads/AboutFDA/Transparency/Basics/UCM281623.pdf
Wan, T. B., Wah, E. L. C., & Meng, L. C. (1992). The use of information technology by
the port of Singapore authority. World Development, 20(12), 1785-1795.
Weijters, B., Rangarajan, D., Falk, T., & Schillewaert, N. (2007). Determinants and
Outcomes of Customers' Use of Self-Service Technology in a Retail Setting.
Journal of Service Research : JSR, 10(1), 3-21.
Wilcox, M. D. (1999). E-commerce: not your grandfather's five-and-ten. [Article].
Kiplinger's Personal Finance Magazine, 53(1), 19-20.
Wong, W., & Welch, E. (2004). Does E-Government Promote Accountability? A
Comparative Analysis of Website Openness and Government Accountability.
Governance, 17(2), 275-297.
Yang, Z., & Jun, M. (2008). Consumer Perception of E-Service Quality: From Internet
Purchaser and Non-purchaser Perspectives. Journal of Business Strategies, 25(2),
59-84.
Yeo, G., Roe, M., & Dinwoodie, J. (2008). Evaluating the competitiveness of container
ports in Korea and China. Transportation Research Part A: Policy and Practice,
42(6), 910-921.
Zients, J. D. (2010). Memorandum for the Senior Executive Service: The Accountable
Government Initiative - An Update on Our Performance Management Agenda.
Washington, DC. Retrieved from
http://www.whitehouse.gov/sites/default/files/omb/memoranda/2010/Accountable
GovernmentInitiative_09142010.pdf
136
APPENDIX A
ABBREVIATIONS
ABI: Automated Broker Interface
ACS: Automated Commercial System
APHIS: Animal Plant Health Inspection Service
ATF: Alcohol, Tobacco, Firearms and Explosives
CBP: Customs and Border Protection
CO: Compliance Officer
CPSC: Consumer Product Safety Commission
CSI: Consumer Safety Inspector
CSO: Consumer Safety Officer
DHHS: Department of Health and Human Services
DIOB: Director, Import Operations Branch
DIOP: Division of Import Operations and Policy
DWPE: Detention Without Physical Examination
EPA: Environmental Protection Agency
ER: Entry Review
FDA: Food and Drug Administration
FDA LOS IOB: Los Angeles District Import Operations Branch
FSIS: Food Safety Inspection Service
FTASC: Foreign Trade Association of Southern California
FY: Fiscal Year
137
GAO: Government Accountability Office
GPRA: Government Performance and Results Act
HQ: Headquarters
ICE: Immigrations and Customs Enforcement
IOB: Import Operations Branch
IT: Information Technology
ITACS: Import Trade Auxiliary Communication System
LACBFFA: Los Angeles Customs Brokers and Freight Forwarders Association
LOS/LOS-DO: Los Angeles District
MARCS: Mission Accomplishments and Regulatory Compliance Services
NHTSA: National Highway Traffic Safety Administration
NMFS: National Marine Fisheries Services
OASIS: Operational and Administrative System for Import Support
OMB: Office of Management and Budget
ORA: Office of Regulatory Affairs
PAR: Pacific Region
PHMSA: Pipeline and Hazardous Materials Safety Administration
PNC: Prior Notice Center
PREDICT: Predictive Risk-based Evaluation for Dynamic Import Compliance Targeting
RPM: Regulatory Procedures Manual
SCSO: Supervisory Consumer Safety Officer
WH: The White House
138
APPENDIX B
ADDITIONAL DEFINITIONS
Consignee: Person named in a bill of lading to whom or to whose order the bill promises
delivery (FDA, 2011e).
Consumer Safety Inspector/Consumer Safety Officer: FDA personnel performing
inspections, investigations, sample collections, and examinations of imported articles
(FDA, 2011e).
Customs Broker: Private individual or firm licensed by Customs to represent importers
concerning import matters. See Filer (FDA, 2011e).
Customs Entry Number: The number assigned to an entry document and used by
Customs for future references to the entry. The CEN is an 11-character entry number in
the format XXX-NNNNNNN-Y. XXX represents an entry filer code assigned by
Customs, NNNNNNN is a unique number, which is assigned by the broker or importer,
and Y is a check digit computed from the first 10 characters based on a formula provided
by Customs. Also referred to as Entry Number (FDA, 2011e).
Detention (Imports): Administrative act where FDA requires that imported articles that
appear violative under the laws FDA administers be held intact. Detained articles are
released if brought into compliance with or rendered not subject to the FD&C Act (the
Act), or are refused entry if not brought into compliance. Other types of detention -
section 304(g) of the Act-Administrative detention of devices; section 304(h) of the Act-
Administrative detention of food (FDA, 2011e).
Detention Without Physical Examination (DWPE): An action directed against specific
products manufactured or shipped by specific foreign firms. "Import Alerts" list products
which may be detained without physical examination due to their violative history or
potential (FDA, 2011c).
Entry: Delivery or offer for delivery of merchandise into the Customs Territory of the
U.S. from an outside point (FDA, 2011e).
Entry Documents (Entry Package): Documents describing the articles offered for entry
which may contain a consumption entry form, commercial invoice, manifest, or other
FDA notification (FDA, 2011e).
Filer: A Customs term used to identify the individual or firm responsible for filing an
entry, which is usually the broker but may be the importer (FDA, 2011e).
139
Importer of Record: The individual responsible for assuring that imported goods are in
compliance with all laws affecting the importation. While the importer may authorize
others to carry out certain tasks such as filing, the importer of record holds the bond and
is ultimately responsible for the entry (FDA, 2011e).
Line Item: Each portion of an entry that is listed as a separate item on an entry document.
An importer may identify merchandise in an entry in multiple portions; however, an item
in the entry having a different tariff description must be listed separately (FDA, 2011e).
May Proceed: "Product may proceed without FDA examination. FDA has made no
determination the product complies with all provisions of the Food, Drug, and Cosmetic
Act, or other related acts. This message does not preclude action should the products later
be found violative." No compliance decision has been made (FDA, 2011c).
Notice of Action: Generated in OASIS, and provides more specific information on the
actions taken broken down by each entry line (e.g., "sample collected" or "intended for
sampling", "detained", "released", or "refused"). As the status changes for a particular
line, a new "Notice of FDA Action" will be issued to advise the appropriate individuals
of the changes (FDA, 2011e).
Private Laboratories: Independent laboratories providing analytical services to importers,
customshouse brokers, and others (FDA, 2011e).
Reconditioning: A process by which the importer of record, owner, or consignee may
submit to FDA a written application requesting permission to bring into compliance any
article, adulterated, misbranded, or in violation of Section 505 by relabeling or other
action, or by rendering it other than a food, drug, device, or cosmetic (FDA, 2011e).
Refusal: Section 801(a) of the Federal Food, Drug, and Cosmetic Act directs the
Secretary of the Treasury to issue a Notice of Refusal when it appears from examination
of samples, or otherwise, that an imported shipment is in violation. This Section also
orders the destruction of any such shipment refused admission, unless it is exported
within 90 days of the date of the notice, or within such additional time as may be
permitted pursuant to such regulations (FDA, 2011e).
Release: "The product is released after FDA examination. This message does not
constitute assurance the product complies with all provisions of the Food, Drug and
Cosmetic Act, or other related Acts, and does not preclude action should the product later
be found violative." A compliance decision has been made (FDA, 2011c)
Release with Comment: A Release Notice advising the importer the goods offered for
entry are released but appear to have minor violations which the agency, in the exercise
of its enforcement discretion, has refrained from enforcing at this time, and that future
entries violating the Act may be detained (FDA, 2011e).
140
Response to “Notice of Detention and Hearing”: May be made personally, by
representative or by mail. The importer may present evidence supporting the
admissibility of the article, request refusal of admission, propose an effective manner of
reconditioning, or a method to remove the product from the authority of the FD& C Act
(FDA, 2011c).
141
APPENDIX C
PAR LOS LAIMPORTS EMAIL GUIDELINES
142
143
144
145
APPENDIX D
RAW SURVEY DATA
1. What is the primary classification of your firm’s business operation?
2. When did you begin working on FDA-regulated entries?
3. Which entry commodity do you handle most?
146
4. Which import geographical region of origin do you handle most?
5. How many total transactions do you estimate making with LAImports and ITACS in
an average work week?
6. Which system do you prefer to use for the following transactions with the FDA Los
Angeles District Import Operations Branch:
7. How often are you able to complete your intended transactions with:
147
8. How satisfied are you with continuing to use:
9. How likely are you to recommend the following systems using positive terms to
colleagues?
10. How satisfied are you with the instructions for use of:
11. How satisfied are you with FDA’s explanation of services provided by:
12. How satisfied are you with FDA’s explanation of processing times for:
13. How satisfied are you with the communication of FDA entry decisions through:
148
14. How satisfied are you interacting with the FDA through:
15. Please select your level of agreement with the following statements:
16. How satisfied are you with the ability to provide FDA with feedback on:
17. How often is each system available when you need it?
18. How satisfied are you with the level of computer security when you transmit
information through:
19. How satisfied are you with the user interface of:
149
20. How satisfied are you with the level of system customization for:
21. If you could customize the systems, what would you change?
22. How satisfied are you with the completeness of information provided by:
23. How satisfied are you with the relevance of information provided by:
24. How often do you get the turnaround time you need from each system?
150
25. How satisfied are you with each system’s ability to meet your expectations?
26. If you use LAImports: (FDA LOS IOB = FDA Los Angeles District Import
Operations Branch)
27. If you use ITACS: (FDA LOS IOB = FDA Los Angeles District Import Operations
Branch)
28. How satisfied are you with your overall customer experience with:
151
29. Do you have any comments on your overall customer experience?
30. Please select the three (3) service areas below that are most important to you when
using electronic systems at the FDA Los Angeles Import Operations Branch.
31. Which three (3) entry commodities do you handle most?
152
32. Which three (3) import geographical regions of origin do you handle most?
33. Do you have any additional comments, suggestions, or recommendations?
153
APPENDIX E
FDA ITACS PUBLIC STATUS
Below is a list of the public statuses FDA provides to industry via ITACS (FDA, 2013b).
Audit Sample Received by FDA Lab
Compliance Response to Trade Communication - Please Refer to Notice of FDA Action
Detained - Refer to Notice of FDA Action for Violation Charges
Detained Without Physical Exam - Refer to Notice of FDA Action for Violation Charges
Detention Withdrawn - Pending Further FDA Compliance Review
Document Accepted
Document Failed Virus Scan
Document Rejected - Attempt to Resubmit
Document Submitted
Document Upload Failed
Documents Required
Entry Deficient - Please Refer to Notice of FDA Action for Details
Entry Documents Required - Notify FDA of Location for FDA Examination
Extension Granted - Refer to Notice of FDA Action for Response Deadline
Extension Request Denied
Extension Request Received
FDA Audit Lab Analysis Completed
FDA Audit Lab Analysis in Process
FDA Audit Sample Collected
154
FDA Examination Completed
FDA Examination Pending
FDA Lab Analysis Completed
FDA Lab Analysis in Process
FDA Reconditioning Lab Analysis Completed
FDA Reconditioning Lab Analysis in Process
FDA Reconditioning Sample Collected
FDA Sample Collected
Field Exam Completed
Further Reconditioning Approved
Goods Were Not Available for FDA Examination - Notify FDA When Available
Hold All Lines - Do Not Devan
Hold All Lines - Do Not Devan - Documents Required
Hold All Lines - Do Not Devan - Documents Required - Notify FDA of Availability for
Examination
Hold All Lines - Do Not Devan - Notify FDA of Availability for Examination
Hold All Lines - Documents Required
Hold All Lines - Documents Required - Notify FDA of Availability for Examination
Hold All Lines - Notify FDA of Location for FDA Examination
Hold Designated Lines - Notify FDA of Location for FDA Examination
Hold Pending Further Written Notice from FDA
Interface Error Contact FDA
Line Availability Received, FDA Examination Pending
155
Line Split Due Multiple Products Identified - See FDA Notice of Action for Details
May Proceed Rescinded - Hold for Further Information from FDA
May Proceed Without FDA Examination
Non FDA Lab Rejected - Additional Private Lab Submission Not Allowed
Non-FDA Analysis Rejected following Audit Sample - No Resubmission Allowed
Non-FDA Analysis Rejected following Audit Sample – Resubmission Allowed
Non-FDA Lab Rejected
Non-FDA Lab Rejected - Resubmission Allowed
Non-FDA Lab Report Received
Non-FDA Lab Under Review
Notify FDA of Location for FDA Examination
Partial Refusal - Inform FDA After Export or Destruction of Refused Portion
Partial Refusal - Inform FDA Before Export or Destruction of Refused Portion
Pending Review By FDA Compliance Staff
Private Laboratory Analysis Confirms Product is Violative
Proof of Export or Destruction Received
Reconditioned Materials Released
Reconditioning Completion Notification Received by FDA
Reconditioning Request (FDA-766) Denied
Reconditioning Request (FDA-766) Under Review
Reconditioning Request Approved - FDA Supervision Not Required
Reconditioning Request Approved - Contact FDA to Arrange Supervision
Reconditioning Request Conditionally Approved
156
Reconditioning Request Conditionally Approved - Contact FDA to Arrange Supervision
Reconditioning Request Conditionally Approved - Refer to Notice of FDA Action for
Conditions
Reconditioning Request Denied - Resubmission Allowed
Reconditioning Request Incomplete - Resubmission Allowed
Reconditioning Supervision Waived - Contact FDA when Reconditioning is Complete
Reconditioning Unsatisfactory
Reconditioning Unsuccessful - Resubmission Allowed
Refusal Follow-up Completed
Refusal Rescinded - Product Still Detained
Refused - No Further FDA Action
Refused Inform FDA After Export
Refused Inform FDA After Export or Destruction
Refused Inform FDA Before Export
Refused Inform FDA Before Export or Destruction
Release Rescinded - Hold for Further Information from FDA
Released
Released With Comment - Future Violative Shipments May Be Detained
Sample Received by FDA Lab
Submission Under Technical Review
Submit Entry Documents to FDA (Invoice, B/L, CBP Entry Document)
Submit Proof of Export or Destruction to FDA (CBP Form and On-Board shipping
records)
157
APPENDIX F
CROSS TABULATIONS
(Continuing to Use Systems) * (System Preferences)
(Recommending Systems Using Positive Terms) * (System Preferences)
158
(Turnaround Times) * (Overall Customer Experience)
(Communication of FDA Entry Decisions) * (Overall Customer Experience)
159
(Ability to Interact with FDA) * (Overall Customer Experience)
(Ability to Meet Expectations) * (Overall Customer Experience)
Abstract (if available)
Abstract
FDA plays a critical role in imports management to ensure the regulatory compliance of foreign health products entering United States commerce. Entry of these products, including foods, tobacco, veterinary, medicinal and medical products, has tripled in volume in the last decade, placing considerable strain on imports management infrastructure. According to the FDA, more than 99% of entries, or import shipments offered for admission into the US, are filed and processed electronically. Therefore, implementation of a functional, reliable, and user-friendly IT infrastructure to manage import service transactions may be an important strategy for the FDA to perform its regulatory function effectively while providing the public with timely access to imported products. ❧ This study examines the level of customer service provided by the electronic transaction systems in place at FDA’s Los Angeles Import Operations Branch (LOS IOB), specifically, the PAR LOS LAImports system (LAImports) and Import Trade Auxiliary Communication System (ITACS), via mixed methods approach of document analysis and survey distribution. Public policy documents for the two systems were examined to identify functional attributes associated with customer service to be incorporated into the study. Methodologies used previously to evaluate customer service in a range of industries were assessed for their relevance for the proposed research and were blended to compile a comprehensive framework tailored to assess and compare user satisfaction with the LAImports and ITACS systems. Key dimensions in this framework include not only customer service elements typical for any service organization but also the government mandated dimensions of transparency, collaboration and public participation. This framework was used to develop a survey for the key industry stakeholders—users of LAImports and ITACS—in order to obtain feedback on the customer service delivery provided by the two systems. ❧ The framework and the subsequent survey effectively provide a snapshot of customer service delivered by FDA LOS IOB through LAImports and ITACS. The results show that while users were most positive toward areas related to the IT infrastructure of the two systems, they were much less positive toward areas related to timeliness of service delivery and associated communications with the agency. These low scoring areas were also identified as those that are most important to the respondents and, therefore, point to future opportunities for improvement.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Risk approaches and standards used in hospitals: a survey of industry views
PDF
Benefits-risk frameworks: implementation by industry
PDF
Incentivizing quality in the manufacture of pharmaceuticals: manufacturers' views on quality ratings
PDF
Effect of GDUFA legislation on the development and approval of generic drugs: a survey of industry views and experiences
PDF
Evaluation of FDA-sponsor formal meetings on the development of cell and gene therapies: a survey of industry views
PDF
FDA influence on advisory committees through documentation: a content analysis and survey of industry views
PDF
Examining the regulatory framework for drug compounding: industry views and experiences
PDF
Regulatory team development in post-merger integration: a survey of views from medical product companies
PDF
Design control for software medical devices: an industry survey of views and experiences
PDF
Regulatory dissonance in the global development of drug therapies: a case study of drug development in postmenopausal osteoporosis
PDF
Challenges in the implementation of Risk Evaluation Mitigation Strategies (REMS): a survey of industry views
PDF
Contract research organizations: a survey of industry views and outsourcing practices
PDF
The impact of FDA Patient-Focused Drug Development (PFDD) on US drug development strategies: a survey of views from pharmaceutical product companies
PDF
Regulatory programs to foster medical product development: user experience in the United States and Japan
PDF
Promotion of regulated products using social media: an industry view
PDF
Regulatory CMC strategies for gene and cell therapies during mergers and acquisitions: a survey of industry views
PDF
Value based purchasing: decision-making processes underlying hospital acquisitions of orthopedic devices
PDF
Implementation of risk management in medical device companies: a survey analysis of current practices
PDF
Institutional review board capabilities to oversee new technology: social media as a case study
PDF
Using telemetry to ensure safe and reliable medical device operation: experience with defibrillators and infusion pumps
Asset Metadata
Creator
Dimapindan, Patrick Raul
(author)
Core Title
An integrated framework to evaluate customer service delivery: a study of electronic systems at FDA's Los Angeles Import Operations Branch
School
School of Pharmacy
Degree
Doctor of Regulatory Science
Degree Program
Regulatory Science
Publication Date
09/02/2014
Defense Date
06/22/2014
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
collaboration,customer service,framework,OAI-PMH Harvest,public participation,transparency
Format
application/pdf
(imt)
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Richmond, Frances J. (
committee chair
), Hamrell, Michael (
committee member
), Kuo, C. Benson (
committee member
), Pacifici, Eunjoo (
committee member
)
Creator Email
dimapind@usc.edu
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c3-467465
Unique identifier
UC11286728
Identifier
etd-Dimapindan-2870.pdf (filename),usctheses-c3-467465 (legacy record id)
Legacy Identifier
etd-Dimapindan-2870.pdf
Dmrecord
467465
Document Type
Dissertation
Format
application/pdf (imt)
Rights
Dimapindan, Patrick Raul
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
collaboration
customer service
framework
public participation
transparency