University of Southern California Dissertations and Theses

A search for theory: Performance management to improve transportation safety

(USC Thesis Other)

A search for theory: Performance management to improve transportation safety

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content A SEARCH FOR THEORY: PERFORMANCE MANAGEMENT
TO IMPROVE TRANSPORTATION SAFETY
by
Beverly A. Daniel
A Dissertation Presented to the
FACULTY OF THE SCHOOL OF POLICY, PLANNING,
AND DEVELOPMENT
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PUBLIC ADMINISTRATION
May 2001
Copyright 2001 Beverly A. Daniel
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
UMI Number: 3 0 2 7 7 1 0
Copyright 2001 by
Daniel, Beverly Ann
All rights reserved.
___ ®
UMI
UMI Microform 3027710
Copyright 2001 by Bell & Howell Information and Learning Company.
All rights reserved. This microform edition is protected against
unauthorized copying under Title 17, United States Code.
Bell & Howell Information and Learning Company
300 North Zeeb Road
P.O. Box 1346
Ann Arbor, Ml 48106-1346
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
UNIVERSITY OF SOUTHERN CALIFORNIA
SCHOOL OF POLICY, PLANNING, AND DEVELOPMENT
UNIVERSITY PARK
LOS ANGELES, CALIFORNIA 90089
This dissertation, written by
... .....................................
under the direction of her.... Dissertation
Committee, and approved by all its
members, has been presented to and
accepted by the Faculty of the School of
Policy, Planning, and Development, in
partial fulfillment of requirements for the
degree of
DOCTOR OF PUBLIC ADMINISTRATION
Dean
DISSERTATION COMMITTEE
F rperson
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
ACKNOWLEDGEMENTS
This document is the product of unselfish support from an extraordinary
dissertation committee and the most thoughtful husband anyone ever had. Joseph
Wholey chaired a committee that knew how to ask constructive, brilliant questions.
With Joe's leadership, my committee made sure that I would be proud of the final
product. Catherine Burke remained unselfishly committed to my success. Kathryn
Newcomer treated my imperfect efforts with the respect and honesty of a trusted
colleague. Maria Aristigueta led the way for me with confidence and friendship.
Robert Biller stepped in at the eleventh hour with clear and challenging questions
that I wished I had thought of on my own. Every step of the way, Ron Daniel
guided me out of every thicket I wandered into and kept me on the path to success.
Mr. Mortimer Downey, the Deputy Secretary of Transportation, took time to
set me out on the right path. He and countless other members of the Department of
Transportation were generous with both their time and knowledge. They were all
shining examples of the public service ethic that reveals itself in competent,
principled governance. That is what this entire project was really about—
discovering the value of leadership and expertise in our national government.
ii
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
TABLE OF CONTENTS
ACKNOWLEDGEMENTS ................................ ii
LIST OF TABLES ................... vii
LIST OF FIGURES .................. ix
ABSTRACT ........................................... x
Chapter
1. MANAGEMENT TO IMPROVE FEDERAL GOVERNMENT
OUTCOMES ............. 1
Introduction .................. 1
A Public Administration Research Issue ....................... 2
Managing To Improve National Transportation Safety
Outcomes .................. 7
Research Questions, Design, and Methodology 9
Conclusions .......................................... 10
Dissertation Chapters .............. 12
2. THE PROBLEM IN THE CONTEXT OF THE LITERATURE .... 14
Using Measures of Government Results . . . . . . . . . . . . . . . . . . . 14
Federal Government Contexts for Using Measures of Results .. 19
Relevance of the Literature to the Problem ........... 36
3. RESEARCH DESIGN AND METHODOLOGY.......................... 38
Research Questions and Design 39
Basis for Selecting Cases ...................... 42
Methodology ............................ 44
Definitions .......... 46
Data Collection Framework .............................. 48
iii
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Criteria for Judging Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Limitations of the Study 59
4. MANAGEMENT TO IMPROVE TRANSPORTATION SYSTEM
SAFETY OUTCOMES ............. 62
Background ........... 63
DOT's Organization and Role ............... 65
Evolution of Transportation Safety Goals and Indicators ...... 70
A Transportation System of Technical and Social
Components ........................................................................... 82
Summary: DOT Contextual Conditions and Strategies ........ 87
Institutional Contextual Conditions . . . . . . . . . . . .. . . . .. . 88
Organizational Contextual Conditions................................... 89
Technical Contextual Conditions . . .. . . . . . . . . . . . . . . . . . 89
Resource Contextual Conditions.......................................... 89
Uses of Performance Information . . . . . . . . . . . . . . . . . . . . . 90
Conclusions and Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
5. MANAGEMENT TO IMPROVE HIGHWAY SAFETY
OUTCOMES ............ 92
Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Part I: The Federal Highway Administration (FHWA) ........ 101
Institutional Conditions in FHWA ............... 103
Organizational Conditions in FHWA . . .. . . . . .. . . . . . . . . 106
Technical Conditions in FH W A .......................................... 108
Resource Conditions in FHWA . . . . . . . . . . . . . . . . . . . . . . 110
Performance Information Uses in FHWA . ...... . . .. . . . . Ill
Part II: The National Highway Traffic Safety
Administration(NHTSA) . . . . . . . . . . . . . . . . . . . . . . . . . . . Ill
Institutional Conditions in NHTSA .................. 113
Organizational Conditions in NHTSA . . . . . . . . .. . . . .. . . 115
Technical Conditions in NHTSA . . . . . . . . . . . . . . . . . . . . . 118
Resource Conditions in NHTSA ........................................ 119
Measurement Information Uses in NHTSA . ............ 120
Part III: The Federal Motor Carrier Safety
Administration (FMCSA) . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Institutional Conditions in FMCSA .. . . . . . . . . . . . . . . . . . 125
Organizational Conditions in FMCSA . . . . . . . . . . . .. . . . . 127
Technical Conditions in FMCSA .......................... 128
iv
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Resource Conditions in FMCSA............... 129
Measurement Information Uses in FMCSA ........ 130
Summary and Conclusions on Highway Safety Contextual
Conditions ................................................. 130
Properties of Highway Safety Contextual Conditions .......... 134
Institutional Contextual Conditions . . . . . . . . .. . . . .. . . . . 134
Organizational Contextual Conditions .. .. . . . .. . . . . .. . . 135
Technical Contextual Conditions........................................ 135
Resource Contextual Conditions . . .. . . . . . . . . . . . . . . . . . . 136
Uses of Performance Information................. 136
6. MANAGEMENT TO IMPROVE AVIATION SAFETY
OUTCOMES ............................ 138
Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Institutional Conditions ................................ 149
Organizational Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
Technical Conditions ............. 158
Resource Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Performance Information Uses . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Summary: Properties of Contextual Conditions for Use of
Aviation Safety Performance Information To Improve
Outcomes ...................... 164
Aviation Safety Contextual Conditions...................................... 167
Institutional Contextual Conditions . . . . . . . . . . . . . . . . . . . 167
Organizational Contextual Conditions ............. 167
Technical Contextual Conditions . . . . . . . . . . . . . . . . . . . . . 168
Resource Contextual Conditions . . . . . . . . . . . . . . . . . . . . . . 168
Uses of Performance Information.................... 168
7. MANAGEMENT TO IMPROVE WATERWAY SAFETY
OUTCOMES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Background .......... 176
Institutional Conditions ..................... 179
Organizational Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Technical Conditions .............................................................. 186
Resource Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Performance Information Uses . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
Summary: Properties of Contextual Conditions for Use of
Waterway Safety Performance Information To Improve
Outcomes.................................................. 189
v
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Waterway Safety Contextual Conditions ............. 193
Institutional Contextual Conditions ................................ 193
Organizational Contextual Conditions ....... . .. . . . . .. . 193
Technical Contextual Conditions........................ 194
Resource Contextual Conditions . . . .. . . . . . . . . . . . . . . . . . 194
Uses of Performance Information . . . .. . . . . . . . . . . . . . . . . 194
8. SUMMARY AND CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . 196
DOT's Approach to Performance Based Management................ 198
Contexts and Strategies for DOT's Approach ............................ 203
Institutional Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Organizational Complexity ................................................ 206
Technical Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
Resources for Performance Based Management ....................... 213
Toward a Theory for Performance Management of
Transportation Safety .................... 215
Areas for Further Research ............................ 219
SELECTED BIBLIOGRAPHY .......... 223
APPENDICES
A. Introductory Letter to Deputy Secretary of Transportation ..... 236
B. Discussion Guide for Interviews .......... 238
C. Model Letter to Transportation Administrators .............. 242
D. List of Officials Interviewed.......................... 244
v i
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
LIST OF TABLES
1. Data Collection Framework .................................................. 52
2. DOT Strategic Goals and Indicators, 1997-2002 . . . . . . . . . . . .. . . . .. . 72
3. Results Reported in DOT Fiscal Year 1999 Performance Report and
Goals for Fiscal Year 2001 ................ 78
4. Direct Transportation Safety Program Budget Request,
Fiscal Year 2001 ........ 86
5. Highway Safety Performance Goals and Preliminary Results
for 1999 .......... 93
6. Estimated Obligations for Highway Safety Program Activities,
Fiscal Year 2001 ............ 95
7. Estimate Obligations for FHWA Safety Program Activities,
Fiscal Year 2001 .................... 103
8. Estimated Obligations for NHTSA Safety Program Activities,
Fiscal Year 2001 .................................. 112
9. DOT Aviation Safety Performance Goals and Preliminary
Results, 1999 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
10. Actual Obligations for Direct Aviation Safety Programs,
Fiscal Year 1999 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
11. Estimated Obligations for Aviation Safety Program Activities,
Fiscal Year 2001 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
12. DOT Waterway Safety Performance Goals and Actual Results,
Fiscal Year 1999 ......... 173
vii
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
13. Actual Obligations for Direct Waterway Safety,
Fiscal Year 1999 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
14. Estimated Obligations for Waterway Safety Program Activities,
Fiscal Year 2001 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
15. Performance Progress Report for Transportation System Safety ...... 200
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
LIST OF FIGURES
1. GAO Model of Effective GPRA Implementation ....................... 4
2. Cross-Case Analysis Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3. Office of the Secretary of Transportation . . . . . . . . . . . . . . . . . . . . . . . . . . 67
ix
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
ABSTRACT
This dissertation responded to a big, two-part question in public
administration. First, can public managers use performance information to improve
performance? Second, to what extent do societies know that public institutions are
producing desirable results?
Constant comparative analysis identified high level concepts and themes of
a theory of performance based management. Department of Transportation’s
(DOT) widely praised approach to implementing the Government Performance and
Results Act (GPRA) to improve transportation system safety outcomes was the
source of the theory. Different contextual conditions presented different challenges
to five operating administrations using performance information to achieve a
common outcome. The elements of the underlying theory of DOT’s approach to
using performance information to improve safety outcomes were:
1. Systems thinking focused inquiry on both social and technical
dimensions of problem solving.
2. Top-down direction supported by bottom-up goal setting, strategy
development, and implementation facilitated organizational
commitment.
x
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
3. Reflection on why implementation strategies did not achieve goals
facilitated learning as the principal use of performance information.
4. Collaborative processes that included stakeholders and government
managers facilitated learning from the individual level to the
institutional level.
5. Partnerships among government and non-government organizations
facilitated diffusion of knowledge to improve outcomes at federal, state,
and local levels.
Performance information was also used to improve accountability, to improve
internal management, to increase public confidence in the capability of the federal
government, and to improve congressional decision making.
Application of the theory of DOT’s approach suggested a less linear process
model than recommended by the U.S. General Accounting Office. Also, uses for
performance information would be continuous and reflective rather than proceeding
through the sequential action steps in the current model. Leadership and facilitation
activities at the departmental level would be coupled with collaborative and
systematic learning in organizations. Institutional as well as organizational
measures of performance would reflect the complex constitutional, societal, and
managerial conditions for performance based management at the federal level.
Additional research is needed to more fully explain how each of the theory’s
elements influences performance outcomes over the long term.
xi
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
CHAPTER 1
MANAGEMENT TO IMPROVE FEDERAL GOVERNMENT
OUTCOMES
Introduction
Scholars and practitioners alike have asked how government performance can
be measured and why government organizations have difficulty using performance
information to improve outcomes. There is little information in the literature about
how successful government organizations use performance information to improve
outcomes when policies and programs are implemented.
Knowledge about the conditions affecting performance based management
can improve guidance as well as inform judgments about implementation of relevant
laws. One source of such knowledge is to compare the experiences of organizational
units that contribute jointly to a single federal strategic objective in diverse contexts.
Beneath the surface of rational, coherent national strategic and performance plans,
numerous obstacles may stand in the way of using performance information to
achieve the long-term outcomes described in plans. For government managers who
implement performance measurement systems, information about how obstacles
have been overcome can be used for problem solving. For both executive and
1
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
legislative decisionmakers, examples of conditions that constrain implementation of
well-conceived plans can moderate judgments about the results. For scholars,
exploration of an executive department’s efforts to use performance information to
improve outcomes is a step toward building a model of how the federal government
manages its performance.
A Public Administration Research Issue
Recognizing inadequate knowledge about how to measure government
performance, public administration scholars have posed questions about both the
organizational and societal aspects of the problem. From a public management
perspective, how can public managers use performance information to improve
performance? From a societal perspective, to what extent do societies know that
public institutions are producing desirable results?
In a Public Administration Review article, Robert D. Behn (1995) posed the
question, “How do we know if a public agency is doing a good job?” He then
reformulated the question from a managerial perspective: “How can public managers
use measures of the achievements of public agencies to produce even greater
results?” (1995, p. 320)
In a response to Behn’s article, John J. Kirlin (1996) presented alternatives to
Behn’s questions. For example, “Measurement: How can society measure its overall
progress and the contribution or hindrance contributed by major institutions,
including business, civic infrastructures and government, and various policies,
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
toward desired goals and use that knowledge to learn regarding future choices of
goals and strategies of action?” (1996, p. 422)
The Government Performance and Results Act of 1993 (GPRA), 107
U.S.C.A. § 285, responds to both Behn’s organizational question and Kirlin’s
societal question about improving the results of government actions. Federal
agencies are expending resources to establish and reconfigure measurement systems
to provide information about performance outcomes. Initial judgments about the
ability of agency managers to establish such systems are being formed now, based on
the perceived quality of strategic objectives, strategies to achieve objectives, systems
for measuring results, and reports of performance achieved.
Section 2 of GPRA lays out six purposes for strategic planning and results
management in the federal government. Four of the purposes are managerial:
Help Federal managers improve service delivery;
Improve Federal program effectiveness and public accountability
by promoting a new focus on results, service quality, and
customer satisfaction;
Initiate program performance reform with a series of pilot
projects; and
Improve internal management of the Federal Government.
The other two purposes are policy-related:
“Improve the confidence of the American people in the capability of
the Federal Government” and “Improve congressional
decisionmaking.”
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
GAO has provided leadership in supporting implementation of GPRA. In
1996, GAO published the Executive Guide: Effectively Implementing the
Government Performance and Results Act, (GAO/GGD-96-118). Figure 1 illustrates
the GAO model of effective implementation of the law. This model suggests an
organizational process that moves from goal setting to performance measurement to
use of performance information. These process phases revolve around
implementation through leadership practices to reinforce organizational changes.
FIGURE 1
GAO MODEL OF EFFECTIVE GPRA IMPLEMENTATION
Step 3 ;
Itea Performame*
Information
Practicaas:
6 . idaniljc p®>fbiroar«M»
7 .K & , ^form ation
© . Use inform ation
Step 1:
- /.i
- G ' * "
: ■ 1 ■ ■
’ ■ .*• '• 8 »
. I V
.
c :. v .. r.f ...
9. 8ev*riwed«sfefe>ii3Vi*i’e
titty
tO .C n aats faoankftvn*
1 1 . 8 . 4 * 1 « « » « * * * «
12.1 m o H a Q te n r n n i
* X 2 r
Maacur® Performance
PrattHe**:
4 . P ro d u c e m a a s u r - f c s &
esoh organizationji
le v * ! m a t
- d e m o n stra te re su lts,
• are limited to the vital
- respond to muittpte
priorities.
■ "
p?o$r*m?
U : ! ■ ■ ■ ■ c .
Source: U.S. General Accounting Office (1996, June). Executive Guide: Effectively Implementing the
Government Performance and Results Act. Washington, DC: U.S. General Accounting Office, p. 10.
4
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The following year, GAO published two evaluation reports—The
Government Performance and Results Act: 1997 Governmentwide Implementation
Will Be Uneven (GAO-GGD-97-109) and Managing For Results: Analytic
Challenges in Measuring Performance (GAO-GGD-138). Among the conclusions in
these reports was that the pilot tests conducted early in GPRA implementation
demonstrated that state and local government activities as well as federal activities
influence many federal program outcomes. In other words, many programs were
authorized to influence complex systems with outcomes outside direct federal
government control.
Another conclusion from GAO’s reports on GPRA implementation was that
performance measurement was difficult for most federal agencies. Fewer than one-
third of the federal managers surveyed in 1997 reported the existence of performance
measures for their programs. Among the problems reported by respondents were
lack of technical resources, lack of multiple iterations and tests of performance
indicators, and the difficulty of separating the effects of federal program actions from
the actions of others involved in the networks implementing programs.
In Government Executive, Anne Laurent (1998) added to the list of
constraints facing GPRA implementation. She pointed to lack of coherent,
measurement-oriented statutory policy structures and the presence of political
controversy as conditions that underlie some federal program performance
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
measurement difficulties. When programs had conflicting goals or ambiguous
purposes, defining outcomes was difficult.
Derek Bok (1997) pointed out that public trust may have eroded more from
dissatisfaction with political rather than with administrative actions. He pointed out
that blaming public officials for undesired outcomes is insufficient. “Since 1960,
America’s policies and programs do not appear to have been conceived or executed
as well as those of other leading democracies in addressing a wide range of concerns
common to large majorities of citizens” (p. 65).
He pointed to the complexity of government programs based on contracts,
grants, and regulation—“a realm where different participants-federal, state, and local,
as well as public and private-interact” (Kettl, 1988, p. 16). According to Kettl, this
produced problems of goal setting, because the goals of all the participants must
mesh. At the same time, government monitoring of results was dependent on
information provided by external participants in government actions. “Managing this
public-private interdependence means finding a way of aligning goals and of
collecting feedback” (p. 155).
Donald Kettl also described how this search for the public interest inherent in
programs based on grants, loans, contracts, or regulations required a new type of
managerial leadership. The focus of this type of management was long-term problem
solving rather than short-term certainty of outputs. “These leadership skills call for
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
more sophisticated managers and more advanced training to prepare them for the
challenges they face” (p. 160).
Managing To Improve National Transportation Safety Outcomes
The U.S. Department of Transportation (DOT) is a federal leader in
implementing GPRA’s framework for results management. The national outcome
goals and indicators for transportation safety were singled out by Congressional
leaders for praise as part of a review of federal strategic plans.
Contrast the above examples [of outcome goals from other
department’s strategic plans] with the following goals from
Transportation’s plan, each of which is accompanied by specific
performance measures and data sources:
(1) Reduce the number of transportation-related deaths.
(2) Reduce the number and severity of transportation-related
injuries.
(3) Reduce the rate of transportation-related fatalities per
passenger-mil-traveled and per ton-mile (or vehicle miles
traveled).
(4) Reduce the rate and severity of transportation-related injuries
per passenger-mile traveled and per ton-mil (or vehicle miles
traveled).
These are real, measurable goals. (Armey et al., 1997, p. 5)
The first two DOT transportation performance plans have received similar
praise. (GAO, May 1998; GAO, May 1999; GAO, July 1999; Thompson, 1999).
Despite such praise at the departmental level, the major operating
administrations with safety missions are not consistently successful in attaining their
portions of the overall transportation goal. As GPRA pilot programs, the National
Highway Safety Administration (NHTSA) and the U.S. Coast Guard (USCG) have
7
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
already reported measurable success in using performance information. In 1994, the
Office of Management and Budget (OMB) assessed the performance plans of the
GPRA pilot projects and named both USCG and NHTSA as federal exemplars.
USCG was cited for its outcome-related goals and having the most historic trend data
of all pilot plans. NHTSA was cited for having “a large number of measurable goals,
with 1993 baselines. A higher proportion of the goals are national outcome goals
than in any other pilot project” (Office of Management and Budget, 1994, p. 7).
Other transportation operating administrations with safety missions, however,
have not received such praise. External evaluations have pointed to data system and
analytical weaknesses that constrain their ability to manage for improving safety
results (GAO, March 1999; GAO, February 1998; GAO, December 1997; GAO,
October 1996).
Under the umbrella of DOT’s leadership, these operating administrations
offer a source of comparison to learn about what kinds of conditions influence using
information about past performance to improve future outcomes. They also provide
a source for comparing how each of these organizations has overcome or is working
to overcome obstacles to managing for results. Differences between the exemplars
and the others could be helpful in identifying what kinds of institutional,
organizational, technical, and resource conditions constrain managing for results and
what kinds of corrective actions are being used to overcome these obstacles. The
patterns of constraints and actions to overcome constraints among these diverse
8
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
federal organizations with a common strategic goal is a step toward a model of how
managing national outcomes occurs. In Fiscal Year 1999, the National Highway
Traffic Safety Administration, the U.S. Coast Guard, the Federal Highway
Administration, the Federal Motor Carrier Administration, and the Federal Aviation
Administration were pursuing a common strategic goal and comparable performance
goals. They functioned in different policy, managerial, and social environments, and
they achieved different levels of success in attaining performance goals.
Research Questions, Design, and Methodology
The research questions for this inquiry are: What were the conditions
associated with using information about past performance to improve future
transportation safety outcomes? What strategies were used to facilitate use of
performance information to improve transportation safety outcomes?
The overall design is an illustrative case study at the departmental level, with
embedded, comparative case studies of five operating administrations supporting the
national transportation safety strategic goal. These organizations faced different
contextual conditions in pursuing a common goal in DOT’s strategic plan.
Methods that were used included content analysis of a wide range of
documents to identify institutional, organizational, technical, and resource conditions
affecting management to improve transportation safety results. Documents reviewed
included previous research about DOT organizations and transportation safety
management, published plans and analyses of performance of transportation safety
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
activities, internal evaluations and studies of safety program outcomes and
organizational factors that affect performance, as well as testimony and proceedings
related to Congressional hearings on transportation policy and safety program
performance outcomes. For areas where the documentation raised unanswered
questions or was contradictory, interviews were used to collect additional
information. All analyses were also reviewed and commented on by personnel in the
staff offices responsible for implementing GPRA to help ensure that no important
information was missing and everything that was included was accurate.
The design, methodology, and quality criteria for this research are described
in detail in Chapter 3. It includes definitions of key terms and a description of the
method used for an exploratory inquiry into contextual factors and coping strategies
used by multiple federal organizations in jointly managing to achieve a national
outcome.
Conclusions
Performance information can be used in many ways to improve the results of
government activities. Public administration scholars interpret the need to improve
government results as both a public management problem and a societal learning
problem. The purposes of the Government Performance and Results Act are
responsive to both these conceptions of the problem in stating purposes to improve
management, policy decisions, and accountability.
10
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The implied theory of the GAO guidance for effective implementation of
GPRA, illustrated in Figure 1, suggests a set of hypotheses about organizational
practices that lead to use of performance information to improve results. The GAO
guidance implies an hypothesis that the four-step process from: (1) defining mission
and desired outcomes, (2) to measuring performance, (3) using the performance
information, (4) and reinforcing implementation can be followed at all organizational
levels, from cabinet level departments to operating administrations and their sub
units.
Also, the twelve supporting practices and the examples of successful
application of the practices discussed in the GAO guidance are not described in
relationship to organizational and policy contexts that influenced implementation.
Illustrations are provided of how the practices are used in federal cabinet level
departments, major operating administrations of federal departments, and
independent federal agencies without addressing the contextual differences.
Federal agencies are proceeding with implementation of GPRA, and some are
reporting progress in managing for results. Other organizations have had difficulty
each step of the way through implementation. To what factors do leaders, managers,
and staff in federal organizations attribute their progress or lack of progress? What
have been the major barriers they have encountered, and how have they worked their
way around these barriers? How do organizations reporting success at using
11
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
performance information to improve results differ from organizations that are not
reporting such success?
Dissertation Chapters
The next two chapters lay the groundwork for answering the research
questions. Chapter 2 examines the research problem in the context of the literature.
Chapter 3 presents the research design and method, defines key terms, provides the
analytical template applied to each case, and explains the basis for comparisons
across cases to draw overall conclusion.
Chapters 4 provides the context for answering the research question. It
describes the U.S. Department of Transportation's framework for results oriented
management of safety in the national transportation system.
Chapters 5 through 7 then explore the institutional, organizational, technical,
and resource conditions that affect managing for safety results in each component of
the transportation system—highways, aviation, and waterways. Chapter 5 discusses
the framework for national highway safety policy and how the Federal Highway
Administration, the National Highway Safety Administration, and the Federal Motor
Carrier Safety Administration manage national policies and programs to improve
safety results on the highway component of the transportation system. Chapter 6
discusses how the Federal Aviation Administration manages national policies and
programs to improve safety in the aviation component of the transportation system.
12
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Chapter 7 discusses how the U.S. Coast Guard manages national policies and
programs to improve safety in the waterway component of the transportation system.
Chapter 8 provides a cross-case analysis. The cases are composed of the
components of the national transportation system safety strategic goal. This means
that similarities and differences are explored from the system oriented strategic
planning perspective—highways, aviation, and waterways. Three operating
administrations contribute to performance in the highway component of the
transportation system. Individual operating administrations contribute to the
performance of the aviation and waterways components of the transportation system.
Common problems as well as the common strategies used to overcome
problems are identified. Based on this analysis, conclusions are drawn about what
was learned about contextual obstacles to managing for transportation safety results
and management strategies used to overcome these obstacles. Areas for further
research are identified. Finally, the relationship of conclusions in this research to
GAO’s model of effective GPRA implementation is addressed.
13
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
CHAPTER 2
THE PROBLEM IN THE CONTEXT OF THE LITERATURE
How do government organizations use measurements of past results to
improve future results? There are multiple threads in scholarly literature about how
government organizations use measurements of results, how contextual factors affect
the ability of government managers to use such information, and the relationships
among organizational performance feedback, learning, and action to improve
outcomes. The following sections summarize this literature about government use of
measurement information, constraints on use, and theories about the social role of
measurement information in organizational plans and actions to change results.
Using Measures of Government Results
In theory and practice, uses of performance measurement range from
instrumental applications that change organizational actions to process applications
that affect participants’ attitudes and behaviors. For example, The Government
Performance and Results Act (GPRA) explicitly includes government reform as an
important use of performance measurement information, but such use is subject to
multiple interpretations.
14
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Paul C. Light (1997) discussed performance measurement as a means to
rationalize oscillating reform efforts by making judgments about prior reforms.
The only way to alter the gravity of reform is to introduce much
greater knowledge about what succeeds and what does not in
actually making government work. And the best route to that
knowledge is through better measures of the performance of
government, a greater commitment to experimentation to reform,
and clean settings in which to conduct the needed tests. The
measures provide the essential baselines against which to measure
reform, the commitment supplies the needed will to declare a
given reform a failure, and the settings provide a place to generate
valid data. (1997, p. 232)
He believed that measurement can be a rational tool for initiating and judging the
impact of policy change.
John W. Kingdon (1993), on the other hand, saw measurement as part of a
nonrational policy process. Problems, solutions, and decision opportunities “floated”
through streams surrounding the national political agenda. Quantified information
became indicators that a problem really existed and could be linked to proposed
solutions. “The countable problem sometimes acquires a power of its own that is
unmatched by problems that are less countable.. . . The indicator is very powerful”
(1993, p. 93). He concluded that a fragmented policy community made
transportation policies more susceptible to change, particularly in reaction to crises
(Kingdon, 1994, p. 120).
James March made a similar observation about organizations and quantified
information. In a 1996 Administrative Science Quarterly, he observed that such
“magic numbers” as performance measures and summary statistics often guide
15
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
organizational action—suggesting that politics and the technology of numbers are
part of social construction. “Decision-making processes are seen as signals and
symbols of legitimacy and, thus, valuable in their own right, regardless of any
consequences for decision outcomes” (March, 1996, p. 287).
Furthermore, Nils Brunsson and Johan P. Olsen cited research by March and
Cyert that suggested reform stems from growth in the gaps between an organization’s
performance and the expectations and ambitions of the people attached to the
organization. Reforms became attempts to change public opinion about the
organization as well as to change its structures and processes. Reforms also often
had educational objectives that were communicated through ideologies in an attempt
to strengthen the legitimacy of the organization. Finally, reforms were a form of
novelty for bureaucratic participants and of symbolic action for observers.
“Administrative reform is determined to a large extent by culturally conditioned rules
and institutional values; the reformer’s prison walls are mental rather than physical”
(Brunsson & Olsen, 1993, p. 202).
Brunsson and Olsen went on to observe that reforms influenced opinions and
were best understood as ideas that were rational, clear, and simple. Strong belief in
the rationality embodied in a reform could, however, prevent reformers from
achieving changes in practice when reforms strengthened stability rather than
initiating change. This happened when the reforms were controversial and the
reformers were not strong enough.
16
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
If a reform is meant to affect the opinion of external parties, then
it is essential to its success that it be made acceptable to these
parties. This is a further reason for adapting reform contents to
general external norms rather than to the specific situation of the
organization to be reformed. (Brunsson & Olsen, 1993, p. 199)
These authors also pointed out that such a reform is not well suited to organizational
implementation.
At the organizational level, for the U.S. federal government, Joseph S.
Wholey (September 1997) specified four uses of performance measurement in
organization and program management. First, performance measurement
information was used as the foundation of performance management systems that
tied improved results to delegation of managerial flexibilities in contracting and
personnel systems. Second, performance information was used to communicate
accountability regarding the value of agency and program activities to important
stakeholders. Third, performance information demonstrated effective or improved
performance. Fourth, it was used to support such policy decisions as resource
allocation, development of legislative proposals, regulations, or guidelines.
Paul Light (1993) described three types of accountability—compliance,
performance, and capacity building—that complement Joseph Wholey’s concept of
accountability as communication of the value of organizational and programmatic
activities to important stakeholders. According to Paul Light, each of these types
implied a different role for government managers, but all three roles could be
pursued concurrently. Seen as compliance, accountability made managers
17
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
responsible for supervision and discipline, and managers accountable for
performance also established and reinforced goal attainment Capacity building
accountability focused on the advocacy and stewardship roles of managers, which is
conceptually close to Joseph Wholey’s concept of accountability as communication
of value added by government organizations.
Accountability is also linked to the policy tools available for implementing
government programs. In examining the workforce effects of grants, regulations, and
contracts as the means to deliver government services, Paul Light concluded that
performance planning in transportation programs created the need for “a new bridge
between those who hire labor through civil service appointments, those who
purchase labor through contracts and grants, and those who create labor through
mandates” (Light, 1999, p. 173).
Joseph Wholey (1996) also compared differences in use of performance
measurement to the differences between formative and summative evaluation.
Formative efforts were described as useful for improving operational performance,
and summative efforts were useful for overall accountability and resource allocation.
In fact, performance measurement was described as “a bridge to more sophisticated
formative and summative evaluation studies . . . by facilitating consensus . . .
providing (time series) data on program outcomes,” and by laying the groundwork
for more exploratory qualitative evaluations (Wholey, 1996, p. 147).
18
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Federal Government Contexts for Using Measures of Results
Within federal organizations, the uses discussed above do not occur in a
consistent set of contextual factors. In cabinet level departments, for example,
diverse organizations are loosely coupled into systems with common outcome goals
but different statutory authorities and different stakeholder groups as well as different
organizational resource profiles. What follows is an overview of the contextual
factors that are evident in the literature as relevant to an organization’s adoption of
performance based management.
Since measurement and evaluation are interdependent activities (each being a
component of the other), the trends in use of evaluation information suggest how
performance measurement may be used. In addition, literature about the experiences
of other countries in using performance information to improve outcomes suggests
many ways in which such information is actually used.
In “Evaluation Use: Theory, Research, and Practices Since 1986,” Lyn M.
Shulha and J. Bradley Cousins (1997) reviewed and synthesized the literature on
how evaluation use had changed in a decade. They observed that recent
developments included:
The rise of considerations of context as critical to understanding
and explaining use; identification of process use as a significant
consequence of evaluation activity; expansion of conceptions of
use from the individual to the organization level; and
diversification of the role of the evaluator to facilitator, planner
and educator/trainer. (1997, p. 195)
19
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Learning was central to process use and was “integral to the context in which it is
produced” (Shulha & Cousins, 1997, p. 198). In government organizations, “the
complexity of bureaucratic levels . . . the lines of communication . . . and the
dominant mechanisms for framing the meaning of evaluation information all
contributed to the potential utility of evaluation findings” (Shulha & Cousins, 1997,
p. 198).
Stakeholders who used evaluation information were immersed in “deliberate
political action concerning their programs . . . coupled with engagement, self-
determination, and ownership” (Shulha & Cousins, 1997, p. 199). Therefore, within
an organization, “the likelihood of inculcating a penchant for systematic inquiry”
depended on changes in organizational culture, particularly the willingness of
managers to support participation as well as “to accept, appreciate, and use
evaluative information” (Shulha & Cousins, 1997, p. 102). Moving an organization
in the direction of self-evaluation also required paying attention to the technical,
analytic, and reporting skills required for systematic inquiry and reflective activity.
Similarly, the experience of the United Kingdom in use of performance
measurement information suggested that results vary among government
organizations, depending on management styles. Tony Bovaird and David Gregory
(1996) summarized the results of efforts to use performance information at the end of
a 20-year period. During that time, performance measurement requirements for
government organizations moved from cost-benefit analysis and program planning
20
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
analysis to analysis of process quality and customer satisfaction. Currently,
performance information is required to support analysis for strategic planning in the
British government.
The actual role played by the measures used in British agencies, however,
reflected the styles of management in each organization. Organizations emphasizing
strategic planning had resource allocation, control, and learning as the purposes of
measurement. Organizations emphasizing financial control used measures for
resource allocation and managerial control. Organizations with both top-down and
bottom-up strategic controls used measures to give direction and to encourage
learning, with control becoming more important as the functions were decentralized
and lower tiers were delegated significant authority. The complex concept of
accountability was never explicitly defined in British performance measures, but the
term was used in public documents that essentially covered some or all of the above
roles of performance indicators.
Bovaird and Gregory found that the degree of use for learning was related to
the degree of stability and predictability in the organization. Practices that appeared
to encourage learning included use of demonstration projects, inclusion of
stakeholders in processes, linking greater autonomy with justifications based on
measurements, and requirements for self-evaluation by managers as the path to
advancement. Very little progress, however, had been made in achieving learning as
an organizational use of performance measurement in British government offices.
21
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Common reasons cited included emphasis on institutional concepts of performance,
top-down implementation that failed to engage middle management commitment,
and lack of resources to train managers in how to design and monitor the
performance measurement systems.
Two decades earlier, James Q. Wilson (1984) concluded that U.S. federal
agency executives are the important factors affecting organizational ability to change
internal processes. He concluded that, in order for executives to lead change, they
must understand the organizational culture, beginning with what the operators
believe the core tasks are. Also, political leaders had to be included in negotiations
for change; results had to be measured in terms of explicit and consistent standards;
and these standards had to be meaningful to the agency stakeholders. Government
agencies with particularly ambiguous missions of interest to complex and competing
constituencies faced challenges that could not be solved internally.
Wilson described such problems as governmental and not merely
organizational, He noted that all organizations had greater difficulty adopting
innovations that would change the relationships among the groups affected by the
change and the way core tasks were defined. Public organizations with undefined or
vague core tasks and outcomes represented a particular challenge. Executive level
orientation to change was critical, but government executives worked in a highly
politicized environment that increased the difficulty in adopting changes that affected
relationships among the interests affected.
22
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Beryl Radin (1999) cited the types of complexity discussed by Wilson as
obstacles facing implementation of GPRA. The mix of a wide variety of budget
functions, organizational structures, and decentralized decision authorities promoted
by recent reforms had produced a fragmented decision environment. Radin
concluded that, within federal agencies, changes associated with GPRA had emerged
when leaders used the GPRA process to further their own agendas. She suggested
that:
Instead of focusing on problems with implementation, it may be
time to step back from a government-wide approach and, instead,
highlight the changes that can take place at a departmental,
agency, or program management level. . . in the context of
specific program areas. (Radin, 1999, p. 134)
Similarly, Rosabeth Kanter (1981) emphasized the significance of differences
among organizations in measuring goal attainment. “Relatively autonomous subunits
are likely to pursue multiple and sometimes inconsistent objectives. Such ‘loosely
coupled systems’ are especially prevalent in nonprofit, governmental, and service
organizations” (Kanter, 1981, p. 327). She discussed a variety of organizational
characteristics that complicated measurement of goal attainment and effectiveness.
She concluded that, rather than “universal dimensions for organizational
comparison,” organization researchers were focusing on problems of how
measurement systems arose, which interests the systems served, and how such
systems brought about change (Kanter, 1981, p. 344).
23
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Citing a history of uncertain success of state and local governments in
establishing performance measurement systems, John M. Greiner (1996) called for
research on the subject. He suggested the need for identification of “appropriate
‘readiness’ criteria and the infrastructure necessary for effective performance
measurement systems” through examination of “successful and unsuccessful efforts
to introduce performance measurement” (Greiner, 1996, p. 41).
Greiner identified institutional, pragmatic, financial, and technical obstacles
that he believed hampered the spread of performance measurement efforts.
Institutional problems included public sector management styles—lack of confidence
coupled with the desire to avoid possible criticism, saturation with the innovations-
of-the-day, and lack of sustained high level leadership. Pragmatic problems “arise
from human rather than technical factors” and included reaching agreement on
service goals and objectives (Greiner, 1996, p. 18). Pragmatic obstacles also
included lack of managerial competence in using performance data in decisions.
Financial obstacles arose from the fact that designing new measurement systems
increased costs while the value of the benefits were unknown, making it difficult to
justify the effort.
According to Greiner, technical obstacles were numerous and included
conflicts about what constitutes government performance (process and quality or
outcome and efficiency) as well as the difficulty of collecting, analyzing, and
reporting timely data for decisions. Also, measurement challenges arose when there
24
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
were multiple performance objectives of concern to multiple stakeholder groups with
conflicting goals at the same time there were few accepted procedures for combining
and interpreting such numbers. Unavoidable “statistical noise” and lack of a
“deterministic causal chain” further complicated this situation (Greiner, 1996, pp. 24-
25).
Regarding the experiences of state governments in managing for results,
Maria Aristigueta found that “managing-for-results efforts remain an evolutionary
process” (1999, p. 158). The impact of performance information on policy decisions
was unclear, and the concept of accountability to the public was too narrowly focused
on oversight to support public scrutiny of multiple, related programs. “Benchmarks,
as found in Florida, Minnesota, and Oregon, provide the opportunity for state
government to operate toward common goals requiring the joint action of multiple
agencies” (Aristigueta, 1999, p. 160). The outcome of state efforts to plan, measure,
and improve public programs strategically was also not a uniform success, but there
was progress and continuing attention to these efforts,
Joseph S. Wholey (May 1997) specified two prerequisites for useful
performance measurement. First, a reasonable level of agreement on goals and
strategies for achieving the goals was needed. This condition could be met through
planning processes that included both the internal and external stakeholders. The
role of the Congress was particularly important Second, the measurement system
had to report information at each organization level and had to be of adequate quality
25
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
that the information it provided was reliable and in a form to be used. Wholey cited
a framework that included goal attainment, production, culture and values
maintenance, and environmental adaptation as aspects of performance. Each of these
concepts implied a different type of measurement information use.
In a series of case studies about processes, practices, and behavior that
“inhibit the detection and correction of error” in development of national policies,
Chris Argyris explored managerial behavior at the organization level (Argyris, 1993,
p. 19). He concluded that the context in each case included managerial attention to a
wide variety of issues that were complex and non-trivial. Each case also included
potential embarrassment or threat to important organizational members, groups, and
the organization as a whole, leading to defensive behaviors.
Argyris’ theory was that organizational defensive routines were learned,
skillful actions that worked without conscious attention and were, therefore, taken for
granted. Organizational cultures were created by individuals and included strategies
to avoid dealing with reality. “These strategies persist because organizational norms
sanction and protect them” (Argyris, 1993, p. 20). Argyris believed that these
counterproductive processes existed in all groups and were dormant until
embarrassment or threat activated them. Situations that entailed embarrassment or
threat for the organization led to behavior that avoided confronting reality through
excuses and evasiveness.
26
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Joseph Wholey and Kathryn E. Newcomer (1997) also put the focus on
management by pointing out the need for early involvement of managerial users in
developing performance measurement systems. They recognized that it may be
“impossible to measure programmatic outcomes in the sort of real time that the Act
[GPRA] envisions” (Wholey & Newcomer, 1997, p. 96).
Kathryn Newcomer (1997) pointed out that “decisions about what to measure
reflected two key factors: the intended use of the performance data and the value
priorities of those stakeholders who choose what to measure” (1997, p. 7). The
relative priority of the various values reflected in performance measures—efficiency,
equity, and service quality—also affected design of the measurement system.
In addition, Kathryn Newcomer (1997) listed five factors that could present
obstacles to use of performance measurement systems. First, the time and resources
required to develop an effective system may not be allowed if “political will from the
top” was lacking. Second, defining performance was inherently political and
“concepts of performance and the most useful measures of it will evolve,” which
required “flexibility to change how performance is measured.” Third, “authority and
resources must accompany responsibility for performance measurement,” or line
managers may not participate. Fourth, “even rumors of punitive use of performance
measures can wreak quick damage.” Finally, “performance measurement is but one
facet of program evaluation, and it can be well served by the evaluation profession’s
27
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
institutional memory about enhancing utilization of strong performance data”
(Newcomer, 1997, pp. 11-13).
The role of performance measurement in organizational learning was found in
research about how humans make sense of coordinated social action. Such research
focused on the managerial role of interpreting the results of organizational actions to
influence beliefs and behavior regarding those actions. Numbers were often the
vehicles for such interpretations and for communications of meaning.
Karl Weick (1995) relied heavily on research by Chris Argyris and Donald
Schon, among many others, to explain how individuals and groups made sense of
organizational life. He stated that most models of organization were espoused in the
vocabulary of argumentation despite the fact that thought was experienced as
narrative; the search for the sense of a situation relied on stories about related
experiences. Stories that made unexpected circumstances understandable guided
conduct by interpreting cues for behavior in unfamiliar circumstances.
Actions are fleeting; stories about action are not. If organizations
are social forms distinguished by their capability for coordinated
action, and if the distinguishing character of those forms
disappears the moment it occurs, then we must be concerned with
what persists when actions keep vanishing. (Weick, 1995, p. 127)
Exploring organizational action required attention to two key points. First, theories
of action about routine use of performance measurements for improving outcomes
included two modes of use—the reflective (learning) and decision modes. Second,
action to improve performance outcomes in government agencies took place through
28
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
institutional, political processes as well as organizational, operational processes
(Weick, 1995, p. 122).
Regarding action as reflection on performance, Barbara Czamiawska-Joerges
(1992) pointed out that, in studying organizational action, processes of reflection as
well as action must be addressed.
Theories, it is claimed, are simplifications of reality. This is true,
if they are to serve as grounds for action; or false, if they are to
serve as grounds for reflection. Interpretive theories attempt to
simulate the complexity of social reality in order to make people
aware of this very complexity. Provided that we strive for both
action and reflection, we need both theories: simplifications, that
is correspondence theories that stand for reality; and
problematizations, that is, representative theories, that discard the
taken-for-grantedness of everyday action, (pp. 204-205)
According to Chris Argyris, a valid theory of action met three requirements.
It was useful for understanding reality by describing actual behavior in context. It is
useful for inventing new solutions to problems. It provided a comprehensive road
map for planned action, implementation, and evaluation of effectiveness. “Theories
to produce actionable knowledge are, therefore, descriptive, normative, and
prescriptive” (1993, p. 250).
Learning occurs when we detect and correct error. Error is any
mismatch between what we intend an action to produce and what
actually happens when we implement that action. It is a mismatch
between intentions and results. Learning also occurs when we
produce a match between intentions and results for the first time.
Effective actions are not only stored as rules in actors’ heads;. . .
their requirements are known publicly, usually in the form of
formal and informal policies and routines that are rewarded by
organizational cultures. Building policies, routines, and culture
requires learning. (Argyris, 1993, pp. 3-4)
29
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
He concluded that, for knowledge to result in the desired action, the specific skills
required and contextual conditions necessary to maintain that knowledge must be
articulated.
In this regard, Donald A. Schon’s (1983) research identified a common set of
skills and conditions conducive to reflection-in-action, which is the overarching
management skill required. He pointed out that “as we try to understand the nature
of reflection-in-action and the conditions that encourage or inhibit it, we study a
cognitive process greatly influenced by ‘cognitive emotions’ and by the social
context of inquiry” (Schon, 1983, p. 322). This observation led him to link
organizational learning to individual knowledge, action strategy, and experiences.
According to Karl Weick (1995), experience was absorbed into theories of
action and changed behavior at the group level as well as the individual level (Weick,
1995, p. 180). Managers, in particular, behaved in ways to affect the behavior of
their subordinates, and organizational knowledge was bound by the orientation of its
managers to self-reflective learning and openness to changes in the environment
(Weick, 1995, p. 183).
Chris Argyris (1976) noted that theories-in-use tended to be resistant to
change, because they provided a stable picture of the world. They were central to a
basic human need for a sense of meaning and control of interactions with the
environment. At the individual level, unilateral control of the environment was a
30
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
basic human behavior strategy and was expressed as interpretation of information as
a means to shape the behavior of others.
One of the most powerful ways to control others is to control the
meaning of valid information.. . . [W]hatever learning
[organizational] actors develop will tend to be within the confines
of what is acceptable. This is called single-loop learning because,
like a thermostat, the individuals leam only about those subjects
within the confines of their theories. They will find out how well
they are hitting their goal.. . . However, few people will confront
the validity of the goal, or the values implicit in the situation, just
as a thermostat never questions its temperature setting. [Such
confrontation would be double-loop learning.] (Argyris, 1976,
pp. 19-20)
Also, “theories-in-use specify which variables we are interested in (as
opposed to the constants in our environment about which we can do nothing) and
thereby set boundaries of action” (Argyris, 1976, p. 9). Organizational members,
however, do not consciously adopt these theories-in-use, which function outside the
actors’ awareness.
Theories of action are theories of governance; they explain how
individuals or groups put their arms around reality in order to
manage it effectively. As such, they are normative theories, not
theories claiming some objective truth. They specify the action
strategies required, the consequences that follow, and the
underlying governing values that are satisficed. (Argyris, 1993,
pp. 249-250)
Regarding public institutions, Donald Shon (1983) observed that “policy
analysis is not rational choice but a rationalization of political interests” (p. 341). In
the public arena, the organizational defenses against learning through inquiry and
discourse are magnified as different interests enter a technical, rational debate with
31
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
the purpose of controlling the agenda rather than building consensus. He concluded:
“In order for society’s conversation with its situation to become reflective,
individuals involved in adversarial processes must undertake reflective inquiry”
(p. 350).
These views complemented conclusions by Jeffrey L. Pressman and Aaron
Wildavsky (1984) that policy implementation in government was exploration and not
amenable to prediction or control. Implementation processes were best evaluated as
learning activities and judged by the discoveries made as well as the accomplishment
of stated objectives. “A case can be made for the reconceptualization of
implementation as an exploratory rather than an unquestioning, instrumental, and
even subservient type of behavior” (Wildavsky & Pressman, 1984, p. 256). They
tied implementation and evaluation of implementation to institutional learning.
James March and Johan Olsen (1989) pointed out that “outcomes can be less
significant—both behaviorally and ethically—than process” (March & Olsen, p. 51).
Processes that lead to change entail learning.
It is possible to see an institution as the intermeshing of three
systems: the individual, the institution, and the collection of
institutions that can be called the environment. Many of the
complications in the study of change are related to the way those
three systems intermesh.
Consider signals related to success and failure. Changes
through problem solving or learning are particularly sensitive to
information that indicates a gap between aspirations and
performance. Institutions devote more attention to activities that
are failing to meet targets than to activities that are meeting them.
Success and failure are normally seen as subject to intentional
control primarily through effects on performance. So, efforts to
32
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
control change through signals of success and failure involve
trying to affect performance.
Although modifying performance or aspirations affects the
subjective sense of success, and thereby affects institutional
change, the precise nature of the change cannot be controlled.
When failure induces search, the results of that search are only
partly predictable or controllable. (March & Olsen, 1989, p. 58)
The need for institutional learning can be particularly relevant to government
pursuit of public safety. In Searching for Safety, Aaron Wildavsky (1988) concluded
that risk aversion commonly distorted governmental discussions of risk and safety.
[Improving] safety. . . is a quality of institutions that encourage a
search so vigorous that no one can be said either to have designed
or controlled it.. . . Attempting to short-circuit this competitive,
evolutionary, trial and error process by wishing the end—safety—
without providing the means—decentralized search—is bound to
be self-defeating. (Wildavsky, 1988, p. 228)
More recently, Julianne Mahler (1997) defined learning in organizations as
“concerned with how organizations monitor their operations, their results, their
environments and their clients for clues to the adequacy of their performance”
(Mahler, 1997, p. 519). She stressed that learning in public organizations “depends
not only on the collection and retrieval of output data and other kinds of information,
it also depends on the culture of beliefs, norms, and professional identities that
provides the context of meaning for the information” (Mahler, 1997, p. 521).
Furthermore, dialogue about alternatives and interpretation of evidence were the key
to policy learning, which manifested itself as cognitive change in positions taken by
members of policy coalitions.
33
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Edgar Schein (1997) also pointed out that the role of measurement in
introducing organizational culture change was gaining consensus among change
leaders about the criteria for measuring results. “How an organization decides to
measure its own activities and accomplishments—the criteria it uses and the
information system it develops to measure itself—become central elements of its
culture as consensus develops around these issues” (Schein, 1997, p. 65). Learning
about the organizational culture was the first challenge faced by leaders who wanted
to establish learning as a dominant organizational value.
Regarding organizational reform, Brunsson and Olsen (1993) observed that
organizations are embedded in social institutions that are the source of rales for
behavior in groups. “A well-developed institution generates a capacity for action. It
facilitates effective co-ordination. But it also creates inertia or friction in face of
attempted reforms” (Brunsson & Olsen, 1993, p. 5). Both organizations and
institutions needed continuity and predictability. The greater the gap between
proposed reforms and the basic values, interests, and opinions of organizational
members, the greater threat it posed for organizational identity.
Within organizations, managers played a distinct role. Czamiawska-Joerges
called managers the “thick waist” and “the middle class” of complex organizations
(1992, p. 187). These organizational actors formed one of the important “nets of
collective action,” and these nets were the phenomena through which organizational
reality was socially constructed (p. 186).
34
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
When engineers confronted management problems, according to Peter
Checkland, an engineering scholar, there was an underlying engineering “assumption
that the world can be taken to be a set of interacting systems, some of which do not
work very well and can be engineered to work better” (1999, p. A10). An engineer
hoping to solve management problems faced a very complex world where improving
the results of systems was a process of inquiry. For systems of human behavior,
however, inquiry was constrained by the role of history in human affairs.
It is their history which determines, for a given group of people,
both what will be noticed as significant and how what is noticed
will be judged. It reminds us that in working in real situations we
are dealing with something which is both perceived differently by
different people and is continually changing.
A problem relating to real-world manifestations of human
activity systems is a condition characterized by a sense of
mismatch, which eludes precise definition, between what is
perceived to be actuality and what is perceived might become
actuality. (Checkland, 1999, pp. A15-155)
Peter Checkland pointed out that systems engineering provided a useful
cybernetic model of goal-seeking-with-feedback to enable machines to achieve the
goal. Personal, institutional, and cultural activity consisted, however, in maintaining
desired relationships as a means for attaining goals (p. 260). Therefore, Peter
Checkland’s research indicated that the purpose of human activity systems was to
change the results of a situation while maintaining desired relationships (p. 262).
35
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Relevance of the Literature to the Problem
Four themes in scholarly literature are relevant to the problem of what
contextual conditions influence a federal executive department’s uses of performance
information to improve results.
1. Government performance information can be used for many different
purposes;
2. The structure of federal government organizations and policies can
complicate adoption of innovative management processes;
3. Social processes underlie how performance measurement information is
used in an organization; and
4. Managerial reflection on the meaning of performance feedback
information is linked to learning, to organizational change, and to maintenance of
social relationships.
The next chapter explains the design and methodology used to explore
problematic contextual factors and solution strategies in implementing GPRA. It
uses four categories of contextual factors suggested by the literature—institutional,
organizational, technical, and resources. Institutional factors link federal executive
agencies and their operating administrations to their stakeholders by policies and
participation processes. Organizational factors are apparent in the size and variety of
core tasks of an organization, indicating the degree of managerial complexity and
types of skills required. Technical factors are the characteristics of systems of
36
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
phenomena that are associated with achieving the desired results of organizational
action. Resources are the financial and human assets available for planning,
monitoring, reporting on, and achieving performance results.
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
CHAPTER 3
RESEARCH DESIGN AND METHODOLOGY
This chapter describes the research design and methodology used to explore
the conditions associated with performance information use and management
strategies for overcoming barriers to performance based management to achieve a
national outcome. The purpose of this research is to contribute to theory about
performance based management in the federal government. It describes how a
federal cabinet level department and five of its major operating administrations
identified and worked around contextual factors that represented barriers to use of
performance information to improve a transportation system safety.
The overall design is multiple, embedded case studies of the contexts,
barriers, and management strategies for implementation of performance based
management to achieve national transportation safety outcomes. The overall case
under study is performance based management of national transportation safety at the
departmental level. Embedded within this larger case are specific case studies of
how obstacles to using performance information to improve highway, aviation, and
waterway safety were overcome by principal operating administrations within the
U.S. Department of Transportation (DOT). These specific cases provided
38
R eproduced with perm ission o f the copyright owner. Further reproduction prohibited without perm ission.
information for constant comparative analysis to identify consistencies among
contextual factors and management strategies using information about past
performance to improve future performance in the three components of the
transportation system. Conclusions were then drawn about the underlying theory of
DOT’s approach to implementing performance based management of transportation
system safety.
Research Questions and Design
The research questions for this inquiry are: What were the conditions
associated with using information about past performance to improve future
transportation safety outcomes? What strategies were used to facilitate the use of
performance information to improve transportation safety outcomes?
As was discussed in Chapter 2, scholarly and practitioner literature contains
many points of view about problems and solution strategies for using government
performance information to improve outcomes. Various barriers and prerequisites
for systematic, goal-oriented use of performance information have been discussed,
and some discussions of strategies to overcome practical limitations are available.
Taken as a whole, however, this knowledge does not yield a model of how federal
government organizations identify and moderate the effects of barriers to using
performance information to implement performance based management.
The literature does suggest the dimensions of the contextual factors
associated with performance information use. The connections between government
39
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
organizations and the outside world are institutional factors. They include processes
and policies that link federal executive agencies and their operating administrations
to their stakeholders. For example, stakeholder participation in performance
assessment can include Congressional, organized interest groups, and individual
citizens.
Organizational factors are apparent in the mission and the types of core tasks
performed. These factors indicate managerial complexity posed by organizational
size and dominant workforce groups. Another type of organizational factor is the
commitment of organizational leaders to interpret the meaning of performance
reports and promote internal change to improve outcomes.
Technical factors are the characteristics of systems of phenomena that are
associated with achieving the desired results of governmental action. Causes of
safety risks, public perceptions of risks, and strategies chosen to reduce risks are
examples of technical factors. Data collection and reporting processes are also
aspects of technical factors.
Resources are the financial and human assets available for planning goals,
implementing interventions, monitoring change, and reporting on the results of
interventions to improve performance. Budgets and staff skills are types of
resources.
According to Robert Yin, the case study method is appropriate when “the
phenomenon under study is not readily distinguishable from its context” (1993, p. 3).
40
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The comparative cases examined in this research represent embedded components of
federal efforts to achieve national transportation safety goals. The phenomena of
managing for improving transportation safety outcomes blend into the contexts
within the institutions and organizations engaged in that enterprise.
Yin also criticized many logic models used in case studies because they “only
identify different effects or stages but do not give an actual explanation of how
events move from one state to another” (1997, p. 74). Therefore, this research
focused on how diverse organizations faced different conditions in implementing
performance based management to achieve a shared outcome.
The result of this inquiry was knowledge for building a theory grounded in
the reality of practice in a federal cabinet level department. Such an approach leads
to a greater likelihood of practical use.
A grounded substantive theory that corresponds closely to the
realities of an area will make sense and be understandable to the
people working in the substantive area.. . . Their understanding
the theory tends to engender a readiness to use it, for it sharpens
their sensitivity to the problems that they face and gives them an
image of how they can potentially make matters better. (Glaser &
Strauss, 1967, p. 239)
The emergent quality of such theory is reflected in the “ joint collection, coding, and
analysis of data” because the “generation of theory, coupled with the notion of theory
as process, requires that all three operations be done together as much as possible”
(Glaser & Strauss, 1967, p. 43).
41
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Basis for Selecting Cases
DOT's national strategic goal for transportation safety was selected for two
reasons. First, the DOT strategic plan received the highest grade of all the federal
agencies' plans reviewed by the Congress in 1997. Second, as discussed in Chapter
1, the transportation safety strategic goal and outcome indicators were singled out by
Congressional leaders for praise. This suggested that DOT might provide an
exemplary case of preparation for performance based management to achieve
national goals. The strategies employed by DOT’s senior leadership illustrate how
constraints were handled in developing performance measures and tracking progress
in transportation safety. This provided a framework for comparative case studies of
how performance based management was pursued in operating agencies that shared
DOT’s national outcome goal.
Five DOT operating administrations were chosen as embedded, comparative
cases in that each organization managed different components of the transportation
system to approve aggregate safety outcomes. Two of these operating
administrations, the National Highway Transportation Safety Administration
(NHTSA) and the U.S. Coast Guard (USCG) were chosen because they had been
cited by OMB as federal exemplars in the pilot phase of GPRA implementation and
have subsequently been praised by external evaluators for the contents of their
performance plans. They have also been praised by external evaluators for the
strategies they have followed to cope with constraints to improve safety outcomes.
42
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Also, they both met or were on a positive trend toward achieving most of the annual
performance goals they had developed for Fiscal Year 1999.
The other operating administrations chosen are the Federal Highway
Administration (FHWA), the Federal Motor Carrier Safety Administration
(FMCSA), and the Federal Aviation Administration (FAA). FHWA and FAA have
been cited by external evaluators for a variety of weaknesses that inhibit their ability
to manage safety results. FAA did not report goal attainment or a positive
performance trend for the aviation safety goals in DOT’s first annual performance
report.
FMCSA, having been established in December 1999, had no goals in the
DOT performance plan for Fiscal Year 1999. It did, however, add a goal for future
years that its predecessor in the Federal Highway Safety Administration had not
attained in 1999. FMCSA, the former motor carrier safety office of FHWA, was
separated from FHWA in late 1999 and subsequently became an independent
operating administration. Concerns about the performance of the office, which has
regulation of motor carrier operations as its primary mission, led the Congress to
remove it from FHWA, which has a primary mission of managing Highway Trust
Fund grants to states for construction and maintenance of the National Highway
System. FHWA retained responsibility for administering the safety aspects of the
highway construction and maintenance grants, which provide DOT a tool for
influencing state traffic laws and regulations.
43
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Within these operating administrations, the program activities that were
covered were those identified as major contributors to the national safety outcomes in
DOT’s plans. Appendix II of DOT’s Strategic Plan for Fiscal Years 1997-2002
distributes the total departmental budget among the five strategic goals. The
operating administrations chosen for this research accounted for 95 percent ($3,634
billion) of the safety goal’s portion of the Fiscal Year 1999 transportation budget
(DOT, 1997, p. II-5).
Methodology
The overall approach to exploring relevant contextual factors and strategies
was constant comparative analysis—contemporary data collection and analysis.
According to Barbara Czamiawska-Joerges, this approach is important in studies of
complex organizations with diverse occupational subcultures.
The main difference between constant comparative analysis and
conventional qualitative analysis is the fact that it is constant.
Instead of collecting innumerable events (a year in the field) and
then trying to acquire some understanding, the researcher starts
comparative analysis at the second interview, and each continuing
step in “gathering insights” is guided by previous analyses . . . to
deal with diversity by trying to understand its roots. (1992,
pp. 205-206)
This method had five stages. Data collection was initiated by contacting the
Deputy Secretary of the Department of Transportation to request an interview. The
letter and the interview guide are in Appendix A and B, respectively. The same
interview guide was submitted as an attachment to introductory letters for the five
44
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
administrators of the case study organizations. The model for these introductory
letters is in Appendix C.
The offices of the administrators of the operating administration generally
provided the names of individuals who could speak for the administrator on the
questions in the interview guide. The names and current positions of these
individuals, some of whom changed jobs during the course of this research, are listed
in Appendix D. Each individual contacted also suggested using documentation that
they provided or that was published on DOT’s Web page to answer many of the
questions.
As data was collected, it was coded into categories of analysis that were
relevant to the research question. The categories and their properties were integrated
through memoranda in margins of the data as it was analyzed. These marginal
annotations were analyzed to identify underlying uniformities in the original set of
categories and properties that produced a more limited number of higher level
concepts—the underlying theory. Finally, these theories were organized into major
themes to provide “a reasonably accurate statement of the matters studied. . .
couched in a form that others going into the same field could use” (Glaser & Strauss,
1967, pp. 107-113).
In addition, recognizing that history matters in shaping organizational
responses to change (Weick, 1995; Schein, 1997), each case study began with an
analysis of organizational histories. Critical historical events that influenced the
45
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
safety policies and programs of each operating administration were identified.
Background information also included a brief organizational history and a
description of each organization’s structure, mission and functions, resources, and
known implementation problems for performance based management of safety. As
Barney Glaser and Anselm Strauss suggested, multiple sources of qualitative
information were used, including interviews, documents, articles, and books as
categories emerged and properties were identified (Glaser & Strauss, 1967, p. 104).
As concluded in Chapter 2, the numerous contextual factors and performance
Information uses suggested in the literature dealing with government performance
measurement provided initial, broad categories for analysis of the data collected.
Contextual categories were institutional, organizational, technical, and resource
factors. Categories of uses of performance information included initiating and
monitoring policy reforms, guiding organizational action, influencing public opinion
about the value of government, changing organizational structures, improving
program management, and facilitating consensus on program goals and
organizational direction.
Definitions
Contextual conditions were characteristics of the environment in which
implementation of performance based management occurred. In organization studies
and public administration literature, contextual conditions were generally identified
46
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
as either necessary for or obstructive to performance information use. These
conditions fell into four broad categories identifiable in the literature.
Institutional conditions link federal executive agencies and their operating
admini strations to their stakeholders by policies and participation processes. Public
laws, communication media, and data exchange processes are examples of links
among various participants. Collaborative participation of representatives of the
networks of governmental, industrial, voluntary, professional, and interest groups is
another example.
Organizational conditions are apparent in the size and dispersion of the
operating structure. These conditions plus such characteristics as labor-management
relationships, processes for inter-group collaboration, and core tasks of the
organization can indicate the degree of managerial complexity and the difficulty of
implementing change.
Technical conditions are the characteristics of systems of phenomena that are
associated with the desired results of organizational action. One example is the
degree of difficulty for learning how to reduce causes of transportation accidents.
Knowledge of the cause-effect relationships and availability of data and valid
analytical approaches to monitor the results of interventions are also examples of
technical conditions.
Resource conditions relate to the financial and human assets available for
planning, monitoring, and reporting on the results of organizational action. The
47
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
degree to which managers have skills in technical and social aspects of performance
based management is an example. Other resource conditions include the availability
of funds to establish, maintain, or up-grade information systems.
As with contextual conditions, uses of performance information discussed in
the literature were diverse. Uses of performance information discussed in the
literature did not fall neatly into categories. For this research, the relevance of
determining the uses of performance information was principally to confirm that use
did occur with the intent of improving performance outcomes.
Data Collection Framework
The type of required data was information about the historical backgrounds of
the safety related organizations and policies in the Department of Transportation.
With the historical context established, data about the institutional conditions,
organizational conditions, technical conditions, and resource conditions described
DOT’s efforts to use and to guide the operating administrations’ implementation of
performance based management.
Historical information was collected from the DOT Web site and its links to
the Web sites of the operating administrations. For both the Department and its
operating administrations, organizational histories and statutory authorities were
extracted from descriptions of missions and functions. The operating administrations
each had organizational histories that traced the current organization back to its
earliest predecessors. These descriptions included changes in both statutory and
48
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
organizational arrangements dating from the initial establishment of the safety related
function to the current statutory and organizational arrangements, as of March 2000.
Published histories of transportation safety issues and transportation
operating administrations expanded on what was provided in the official histories
available on the Internet. Published histories of national policy changes that
accompanied expansion of the U.S. transportation infrastructure explained the role of
safety in policy and organizational structure decisions. These histories also described
the development of stakeholder groups and institutional relationships for
participation in these decisions.
Institutional conditions were identified in both the contemporary and
historical documents. Descriptions of DOT and operating administration activities to
elicit stakeholder participation in efforts to improve transportation safety were well
documented on the Internet. All of the departmental and operating administrations’
pilot project analyses, strategic and performance plans, and budgets were available
electronically. In many cases, transcripts of preparatory discussions and requests for
public input were also available electronically. Transcripts of testimonies and press
releases were also available electronically. Interviews with key individuals verified
and expanded on this information.
Organizational conditions were explored through information and data
published by the Office of Personnel Management, the Office of Management and
Budget, the Congressional Research Service, the U.S. General Accounting Office,
49
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
and the DOT Inspector General. These sources were augmented with information
published in special studies by White House and Congressional task forces as well as
from hearing transcripts of Congressional proceedings. Interviews with key
individuals verified and expanded on this information.
Technical conditions were explored through interviews to identify and
interpret scholarly and technical sources of information on transportation safety,
accidents, and risk. Additional information was gathered in published documents of
DOT and its operating administrations, particularly organizational units that did
research and analysis or managed data collection systems.
Resource conditions were initially identified in interviews. Then analysis of
organizational documents identified support specifically relating resource availability
to technical support for managing performance planning, monitoring, and reporting
on performance results.
Multiple types of uses of performance information to improve government
result were identified in the literature.
■ To involve stakeholders in decision making and in implementing
strategies to improve results of government programs;
■ As the basis for allocating resources and for management controls of
resource consumption;
■ To justify requests for policy changes and as a tool for improving internal
management; and
50
R eproduced with perm ission o f the copyright owner. Further reproduction prohibited without perm ission.
■ To support learning about the causes and how to reduce transportation
accidents through systematic inquiry coupled with reflective action.
These types of uses were identified through document analysis and interviews
to establish that there was a reasonable basis to conclude that performance
information was actually used in efforts to improve safety results. Table 1 is a
summary of the data collection questions and approach.
The Deputy Secretary of Transportation provided his support and ideas at the
outset of the data collection process. He was the winner of the National Capital Area
Chapter of the American Society for Public Administration and Government
Executive magazine 1998 Leadership Award, in part for his active support of GPRA
implementation. Also, USC alumni and current doctoral students in the case
agencies provided advice about how best to proceed and who to contact. In addition,
individuals who prepared or updated the NHTSA and USCG case studies for the
American Society for Public Administration offered suggestions.
51
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
TABLE 1
DATA COLLECTION FRAMEWORK
Research Question: What are the conditions associated with using measurements
of results to improve transportation safety outcomes?
Category Being
Explored
Data Collection I Data Collection Approach
Question |
Institutional
Conditions
History of policy
uses of
performance
information
How has performance
information been used
historically?
Summary of policy and
organizational histories from
published documents and
literature
Degree safety
results can be
controlled by
policy tools and
organizational
functions
To what degree do
policies and functions
for reducing safety risk
limit ability to manage
safety results?
Types of policy and
organizational functions
Summary of policies and
functions for reducing safety risk
from performance plans and
reports, budget documents, and
published studies
Interviews to corroborate
documents as needed
Degree of
consensus on safety
goals and
indicators
To what degree is there a
reasonable level of
agreement about safety
goals and indicators
among stakeholders?
Summary from performance
plans and reports, press releases,
and testimony
Interviews to corroborate
documents as needed
Organisational
Conditions
Structure and size
of organization
To what degree does the
structure of the
organization make it
difficult to manage
safety results?
How many
organizational units and
hierarchical layers
contribute to safety
results?
What is the geographic
area covered by the
organization?
Summary from such documents
as plans, workforce assessments,
performance planning, and
reporting publications
Studies and analyses of
organizational issues
Interviews to corroborate
documents as needed
52
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
TABLE 1 (continued)
Research Question: What are the conditions associated with using measurements
of results to improve transportation safety outcomes?
Category Being
Explored
Data Collection
Question
Data Collection Approach
Core tasks for
safety mission
What are the core tasks
that contribute to safety
results?
Summary from such documents
as plans, workforce assessments,
performance planning, and
reporting publications
Studies and analyses of
organizational issues
Interviews to corroborate
documents as needed
Technical
Conditions
Knowledge about
causes of accidents
To what degree are the
causes of accidents
understood?
Summary from such documents
as plans, workforce assessments,
performance planning, and
reporting publications
Studies and analyses of technical
issues
Interviews to corroborate
documents as needed
Knowledge about
how to monitor
results of
interventions
To what degree can the
results of interventions
be monitored?
Summary from such documents
as plans, workforce assessments,
performance planning, and
reporting publications
Studies and analyses of technical
issues
Interviews to corroborate
documents as needed
Availability and
quality of data to
study causes of
accidents and
monitor
performance
Is adequate, high quality
data available for
studying causes of
accidents and results of
interventions?
Summary from such documents
as plans, workforce assessments,
performance planning, and
reporting publications
Studies and analyses of technical
issues
Interviews to corroborate
documents as needed
53
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
TABLE 1 (continued)
Research Question: What are the conditions associated with using measurements
of results to improve transportation safely outcomes?
Category Being
Explored
Data Collection
.Question .
Data Collection Approach
Resource
Conditions
Availability of
funds to support
planning,
monitoring, and
reporting on
performance
To what degree are
funds available for
planning, monitoring,
and reporting on
performance?
Interviews to identify potential
problem areas
Corroboration of problem areas
through review of documents,
including studies and analyses of
resource conditions
Availability of
workforce and
managerial skills to
plan monitor, and
report on
performance
To what degree are skills
for planning, monitoring,
and reporting on
performance available?
Interviews to identify potential
problem areas
Corroboration of problem areas
through review of documents,
including studies and analyses of
resource conditions
Performance
Information Uses
How is performance
information used?
Summary from such documents
as plans, workforce assessments,
performance planning, and
reporting publications
Studies and analyses, hearing
transcripts, testimony, and
speeches
Interviews to corroborate
documents as needed
54
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
TABLE 1 (continued)
Research Question: What are the conditions associated with using measurements
of results to improve transportation safety outcomes?
Category Being
Explored
Data Collection
Question
Data Collection Approach
Initiate and
monitor reform
Guide
organizational
action
Influence public
opinion about the
value of
government
Justify
organizational
restructuring
Improve program
management
Facilitate
consensus on
program goals
Research Question: What types of strategies are
using information about results to improve trams
used to overcome obstacles to
portation safety outcomes?
Category Being
Explored
Data Collection
Question
Data Collection Approach
Strategies to cope
with obstructive
institutional,
organizational,
technical, and
resource conditions
What types of strategies
are used to cope with
institutional conditions?
Summary from such documents
as plans, workforce assessments,
performance planning, and
reporting publications
Studies and analyses, hearing
transcripts, testimony, and
speeches
Interviews to corroborate
documents as needed
55
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Cross-case analysis was performed through constant comparative analysis of
the data as it was collected. (See Figure 2.) The sources of the items to be compared
were the specific characteristics or properties of the categories identified in the single
cases. The diversity of histories, policy foundations, institutional relationships,
organizational structures, technical challenges, and resource constraints faced by
these organizations made it impossible to predict the common characteristics or
properties below the broad categories. Collecting and comparing the details at the
single case level suggested similarities in the properties of these categories. These
comparable properties then provided the basis for organizing the information into
underlying themes that represent a theory of performance based management.
56
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
FIGURE 2
CROSS-CASE ANALYSIS TEMPLATE
Research Question: What are the.'Conditions associated with using measurements of results
to improve transportation safety outcomes?
Variable Category Being Highway Safety Aviation Waterway
Explored Safety Safety
FHWA MBTSA i FMCSA FAA USCG
Institutional Conditions
Organizational Conditions
Technical Conditions
Resource Conditions
Research Question: What strategies are used to overcome barriers to achieve transportation
safety outcomes?
Institutional Barriers
Organizational Barriers
Technical Barriers
Resource Barriers
Performance Information
Uses
Initiate and monitor reform
Guide organizational action
Influence public opinion about
the value of government
Justify organizational
restructuring
Improve program management
Facilitate consensus on
program goals
57
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Criteria for Judging Quality
Validity was established in three ways. Construct validity was safeguarded
through a rigorous audit trail, multiple sources of evidence, and key informant review
of interview records and draft study reports. Reliability was established by
extensively documenting procedures.
Potential researcher biases were minimal, since the researcher has not
participated in work at NHTSA, FMCSA, FHWA, or USCG and was never directly
involved in developing or evaluating performance monitoring or results management
systems at FAA. To the degree that bias could be perceived as an issue at FAA,
continuous review and sign-off by agency representatives were used as safeguards.
Regarding generalizability, in Qualitative Methods in Management Research,
Evert Gummesson (1991) points out that the advantage of case study research is the
holistic view it provides. He says that case studies allow generalizations about how
actions occur in ways that quantitative approaches do not. If the social context is
critical to the phenomena under study, “theory becomes local theory; knowledge in a
social context arises when one is able to deal with a specific situation” (p. 84).
Furthermore, the purpose of such studies is to improve knowledge rather than to
identify an enduring truth. So long as an adequate number of cases are included, so
that “the marginal utility of an additional case approaches zero,” inability to
generalize is not a barrier to drawing conclusions useful to others studying similar
phenomena (p. 85). For this research, multiple cases, including a federal department
58
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
plus five operating agencies supporting it, provided a reasonable basis for
generalization at the level of government addressed in laws and guidance published
for performance measurement implementation.
As recommended by Caudle (1994), findings from qualitative data were
corroborated through informant feedback. Also, as recommended by Czamiawska-
Joerges (1992), representation of differing views of reality was recognized through
constant comparative analysis to limit researcher bias in interpreting information.
Procedurally, reviews of published documents, including electronic
publications available on the DOT Web home page, preceded interviews. Interviews
were conducted as needed to corroborate or expand on documentary information.
Limitations of the Study
The sample size is a limitation of this approach. The uniqueness of each
organization constrains generalizations. On the other hand, use of initial categories
of variables previously reported in relevant research literature linked the local theory
developed in these cases to a growing body of knowledge about adoption and use of
performance measurement information to improve government performance. Also,
constant comparative analysis helped to differentiate multiple perceptions of reality
and avoid researcher biases regarding the information collected by interviews.
Finally, triangulation strengthened qualitative data by providing multiple sources of
confirmation.
59
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The choice of a strategic goal in a federal cabinet level organization and five
operating agencies delimits the coverage of this research. This was necessary to meet
the purpose for the research: to contribute to improved understanding of how
obstacles are alleviated to facilitate use of measurement information to improve
transportation safety outcomes. Also, the choice of the national strategic goal and
indicators for transportation safety was justified, because they had been singled out
for praise by external reviewers as among the highest quality. This provided a
positive or best case source in which to identify useful practices. Cases were selected
to provide contrast to the positive cases in that weaknesses in the ability to measure
outcomes had been identified by external reviewers.
In addition, the choice of operating agencies was limited to five to
accommodate practical needs and because there was no clear reason to include one
more. The major components of the national transportation system were covered,
and they represented adequate diversity to provide rich contexts for comparisons.
As discussed in Chapter 1, the GAO model of effective GPRA
implementation offers potential hypotheses about strategies to be considered in this
research. The practices described by GAO are types of strategies that can be
employed by government organizations to implement performance based
management, but the Influence of organizational contexts on these strategies is not
discussed. GAO’s model was used as a source of a hypothesis about management
strategies for implementing results management as the model of implementation of
60
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
transportation safety results management emerged from this research. It suggests that
the four-step process from (1) defining mission and desired outcomes, (2) to
measuring performance, (3) using the performance information as input to mission
and outcome analysis, and (4) reinforcing implementation can be followed
independently for all organizational levels, from cabinet level departments to
operating administrations and their sub-units.
61
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
CHAPTER 4
MANAGEMENT TO IMPROVE TRANSPORTATION SYSTEM
SAFETY OUTCOMES
The Department of Transportation (DOT) provides an illustrative case of
progress and challenges in the transition to using performance information to plan
improved safety outcomes. This chapter describes how DOT led its operating
administrations in implementing performance based management of national
transportation safety outcomes. The departmental approach to improving
transportation safety outcomes was evident in DOT plans and reports. The following
sections explore the components of this approach.
The historical context description will be followed by a description of how
DOT was organized and the role the Department plays in accomplishing national
goals. Then, the ways in which safety system performance planning and reporting
evolved during implementation of the Government Performance and Results Act of
1993 (GPRA) will be summarized. Next, the components of the transportation
system are described. Finally, a summary and conclusion section will explore what
conditions and strategies were associated with using performance information to
improve safety outcomes in the transportation system.
62
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Background
The Department of Transportation was established in 1967 to improve
coordination, cooperation, technology research and development, and problem
solving for national transportation policies and programs (49 USC § 101). As will be
discussed in more detail in chapters 5, 6, and 7, the organizations that became part of
DOT—the Federal Highway Administration (FHWA), the Federal Aviation
Administration (FAA), and the U.S. Coast Guard (USCG)—brought with them long
histories of collecting data about transportation safety.
Two more transportation safety-management organizations were added after
DOT had been established. In 1970, the National Highway Safety Bureau of FHWA
formed the foundation of the National Highway Traffic Safety Administration
(NHTSA), which, as its name implies, was established exclusively to improve
national highway traffic safety (Mashaw & Goddard, 1990, p. 6). In December 1999,
the Congress also separated the Office of Motor Carrier Safety from FHWA to create
the Federal Motor Carrier Safety Administration (FMCSA) and strengthen safety
management (Public Law 106-159 § 1).
DOT’s efforts to plan performance strategically began before GPRA became
law. The National Transportation Policy (NTP) initiative began in 1989, and it
included development of a national strategic planning process for the transportation
system (Transportation Research Board, 1991, p. I). The NTP stated that DOT
would ensure that legislative, budgetary, and regulatory planning and decision
63
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
making would be guided by a strategic plan. Strategic planning would integrate
mode-specific planning with overall national policies for the future of the national
transportation system.
DOT published the first strategic plan for the national transportation system
in January 1994, but two of the four largest modal administrations had already
published separate strategic plans for the aviation system and part of the highway
system. FAA had undertaken strategic planning in late 1987 and published the first
strategic plan for the national airspace system in August 1990 (Faigin, 1996, p. 2).
FHWA developed the first strategic plan for part of the national highway system
between December 1990 and May 1992 (Faigin, 1996, p. 2). NHTSA and USCG
strategic plans were completed in 1994 (Faigin, 1996, p. 3).
After passage of GPRA, the first draft strategic plans submitted to the
Congress received an unfavorable review (Congressional Institute, 1997, p. 1).
DOT’s plan had been criticized, along with all other federal agency plans, for
focusing on process outputs rather than organizational and programmatic outcomes
(Congressional Institute, 1997, p. 1). To overcome this shortcoming, the lessons
learned from the GPRA pilot projects in DOT operating administration were used.
Pilot projects in NHTSA and USCG were particularly useful. They provided a
technical framework for defining safety outcomes as reducing fatalities (Mortimer
Downey, personal communication, July 1,1999).
64
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
DOT’s Organization and Role
Administrators of thirteen subordinate organizations report to the Secretary of
Transportation. Nine of these organizations are operating administrations with
responsibility for implementing policies dealing with the specific parts of the
transportation system. They include:
e Federal Highway Administration (FHWA),
• Federal Motor Carrier Safety Administration (FMCSA),
• U.S. Coast Guard (USCG),
e Federal Aviation Administration (FAA),
• National Highway Traffic Safety Administration (NHTSA),
® Maritime Administration (MARAD),
• Federal Railroad Administration (FRA),
■ • St. Lawrence Seaway Development Corporation, and
• Federai Transit Administration (FTA).
The four remaining organizations provide common support to the operating
administrations listed above and to the Secretary:
• Bureau of Transportation Statistics,
• Research and Special Programs Administration,
• Surface Transportation Board, and
• Transportation Administrative Service Center.
65
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
All thirteen of DOT s organizations also support the National Transportation
Safety Board, which is independent of DOT but served by personnel from within the
Department when conducting investigations into the causes and recommended
preventions of major transportation accidents.
DOT’s strategic plan recognizes that the modal administrations “lead highly
discreet [sic], autonomous lives” (DOT, DOT Strategic Plan for Fiscal Years 1997-
2002, p. 4). At the same time, the plan envisions that “the future Department will
become ONE DOT . . . to optimize transportation efficiency and effectiveness”
(DOT, DOT Strategic Plan for Fiscal Years 1997-2002, p. 4).
The Administrators of the modal administrations report directly to the
Secretary of Transportation. In addition, the Office of the Secretary of
Transportation has 16 organizational units as displayed in Figure 3.
Within this structure, the office of the Assistant Secretary For Transportation
Policy leads the strategic planning process. The Office of the Assistant Secretary for
Budget, Programs/Chief Financial Officer organization is the departmental focal
point for performance planning and reporting. Specifically, the Office of Budget and
Program Performance provides day-to-day direction for planning and reporting
departmental performance.
6 6
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
FIGURE 3
OFFICE OF THE SECRETARY OF TRANSPORTATION
Secretary and Deputy Secretary
• Associate Deputy Secretary, Office of Intermodalism
• Office of Drug and Alcohol Policy and Compliance
• Executive Secretariat
• Office of Civil Rights
• Board of Contract Appeals
• Office of Small/Disadvantaged Business
• Office of Intelligence and Security
• Office of Public Affairs
• Office of the Chief Information Officer
• General Counsel
• Asst. Sec. For Transportation Policy
• Asst. Sec. For Aviation and International Affairs
• Asst. Sec. For Budget, Programs/Chief Financial Officer
• Asst. Sec. for Governmental Affairs
• Asst. Sec. For Administration
9 Office of Inspector General
Source: U.S. Department of Transportation (2000, March) Telephone Book, p. 62.
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
DOT’s Strategic Plan for Fiscal Years 1997-2002, the departmental role is
described as “coordinated activities” that include the following:
Emphasize research in human performance and behavior such as
biomechanics and lifetime driver learning.
Promote public-private partnerships to demonstrate cost-
effective, safety technologies, such as intelligent vehicles, air
traffic management, and enhanced weather services.
Develop, deploy, and promote cost-effective IT and ensure that
DOT’s IT systems are Year 2000 compliant to prevent
mission/service performance disruptions. For example, the
FAA’s Display System Replacement (DSR) will modernize
control display systems in air traffic control centers in the
coterminous United States and Alaska. This project will replace
aging and unsupportable display equipment with functionally
equivalent, expandable hardware and software. DSR will provide
air traffic controllers with a modem digital display system capable
of processing and providing information in a fast reliable manner.
Build both domestic and international partnerships to integrate
fully safety as a basic business principle.
With our partners, reinforce the importance of individual
responsibility for achieving improvements in safety, and
encourage industry to take the lead in partnering activities.
Improve the delivery of services through better
communications with our customers.
Use a common sense approach in our focus on the highest
safety risks through risk-based management and incentives, and
performance-based regulations to optimize use of resources.
Strengthen enforcement to promote maximum compliance.
Emphasize child safety.
Maximize the use of education and advocacy to promote safe
behavior and practices. For example, NEXTEA includes a
flexible safety program that would provide over $2 billion for
safety projects through 2001. The program would allow projects
targeted to safety problems and risks and allow for funding on
non-infrastructure highway safety projects.
Emphasize drug and alcohol-free workplace programs and
operating environments, and work with industrial partners to
assist them in adopting appropriate testing regimes in non
regulated workplaces.
6 8
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Advance transportation research exploring causes of, and
countermeasures for, transportation incidents in all modes of
transportation.
Encourage the reporting of near-miss incidents and
communicate lessons learned following transportation incidents.
Promote transportation safety excellence through more
effective recognition programs that acknowledge positive safety
achievements, (pp. 13-14)
This role is slightly redefined in the December 1999 draft of the strategic plan
for Fiscal Years 2000 through 2005. DOT redefined its role more succinctly and
emphasized leadership:
We aim to achieve our goals through innovation in technology, in
institutional management, and in operational and administrative
processes that will link the power of new technologies and their
corollary management changes to realize their economic potential
and to bring real improvements to people’s lives. The
Department is dedicated to fostering a climate that accelerates
innovation, by providing leadership and investment guidance;
supporting an educated and motivated transportation workforce;
investing in long-term strategic research; collaborating with
industry, government, and universities; creating a supportive legal
and regulatory framework, and ensuring investment support
(DOT, DOT Strategic Plan Fiscal Year 2000-2005, p. 1, available
at http://www.dot.gov)
This is a more focused statement of the departmental role than the list of activities in
the previous strategic plan. It casts the departmental role as facilitator of innovation
and integrator of processes for technology development, investment, and
organizational management to allow positive change to occur.
69
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Evolution of Transportation Safety Goals and Indicators
This section describes DOT’s progress in refining strategic and performance
planning goals and indicators and in reporting performance results. It will describe
how DOT changed its performance plans slightly with each iteration and attributed
these changes to learning from stakeholders and from program operations. DOT also
attributed missed goals to the need for learning more about relevant aspects of the
transportation system.
The Congress criticized DOT’s, along with all other executive agencies’,
draft strategic plan for reliance on output measures of programs and functions rather
than measures of outcomes (Armey, 1997). The final strategic plan for Fiscal Years
1997-2002 was singled out for Congressional praise because of its outcome goals and
indicators of goal attainment (Army, et al., 1997).
Improving safety in the transportation system is the first of DOT’s five
strategic goals in both the current and draft strategic plan. In addition to safety, DOT
goals address mobility, economic growth and trade, human and natural environment,
and national security. The national strategic goal for transportation safety in the final
Fiscal Years 1997-2002 plan is: “Promote the public health and safety by working
toward the elimination of transportation-related deaths, injuries, and property
damage” (DOT, DOT Strategic Plan for Fiscal Years 1997-2002, p. 12). The
outcome goals for the strategic goal are stated as:
70
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Success in achieving the safety strategic goal will be measured
by realizing an improvement in each year of the Plan over the
previous year, for the following outcome goals....
The first two outcome goals capture the progress toward
elimination of transportation-related deaths and injuries measured
directly in whole numbers. The third and fourth outcome goals
are normalized data to account for risk exposure due to
noncontrollable changes in use of the transportation system. The
fifth outcome goal. . . provides an indirect measure of property
loss, health care, and productivity costs and is a leading indicator
of potential problems that could lead to deaths and injuries.
These outcome goals cover not only movement of people and
goods but safety at terminals, ports and interchange and transfer
points, as well as at the intersection of transportation modes such
as highway rail grade crossings. (DOT, DOT Strategic Plan 1997-
2002, pp. 12-13)
These goals and indicators are summarized in Table 2.
The Deputy Secretary emphasized the importance of the decision to define
“safety” as elimination of risks of death or injury to overcome the methodological
problem of finding common indicators of performance. It took time to reach
consensus among the modal administrators that fatalities would be the bottom line
measure. Injuries and property damage were then selected as additional measures of
safety. Even given “fatalities” as a measure, there was extensive discussion about
whether the fatality from a transportation crash would be counted only if it occurred
immediately or if injuries that led to subsequent death would be counted as well.
NHTSA supported including the latter definition though it requires a more complex
data collection system (Mortimer Downey, personal communication, July 1, 1999).
71
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
TABLE 2
DOT STRATEGIC GOALS AND INDICATORS, 1997-2002
Strategic Goals Indicators
Reduce the number of transportation-
related deaths
Aggregate number of
transportation-related fatalities
Reduce the number and severity of
transportation-related injuries
Aggregate number of injuries
attributable to transportation
Reduce the rate of transportation-
related fatalities per passenger-mile-
traveled and per-ton-mile-of-freight-
shipped (or vehicle miles traveled)
Aggregate number of
transportation-related fatalities
divided by 100 million
passenger-miles, and
Aggregate number of fatalities
attributable to transportation
divided by 100 million ton-miles
Reduce the rate and severity of
transportation-related injuries per
passenger-mile-traveled and per-ton-
mile (or vehicle miles traveled)
Aggregate number of
transportation-related injuries
divided by 100 million passenger
miles, and
Aggregate number of
transportation-related injuries
divided by 100 million ton-
miles.
Reduce the dollar loss from high-
consequence, reportable transportation
incidents
Property damage from high-
consequence, reportable
transportation incidents
Reduce the number of reportable
transportation incidents and their
related economic costs
Aggregate number of
transportation incidents
Source: U.S. Department of Transportation (1997). Strategic plan for fiscal years
1997-2002, p. 13, available at http://www.dot.gov.hot/dotplan/html.
DOT’s first performance plan, for Fiscal Year 1999, translated the outcome
goals and general strategies into annual goals and specific strategies for each
operating administration in the Department. In evaluating this plan, GAO found that
the plan “generally provides a clear picture of intended performance . . . includes
72
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
performance goals and measures that cover the program activities in the
Department’s budget, and the goals and measures are objective, quantified, and
useful for assessing progress” (GAO, May 1998, p. 2). Weaknesses in the plan
included the need to explain links between the performance and strategic goals and
the program activities. Also, GAO found that more information was needed about
proposed regulatory efforts and external factors that could affect meeting planned
goals. It also needed more complete information about the quality and uses of
performance data.
To implement the plan, DOT used extensive public involvement and outreach
to involve the diverse interests associated with national transportation safety. The
National Transportation Safety Conference in March 1999 included panel
discussions to review “roles of the federal government, industry and the public in
making safe transportation choices” (DOT, Press Release DOT 28-99, February 23,
1999). At the conclusion of the safety conference, industry, trade, labor, law
enforcement and community leaders signed a memorandum of understanding with
DOT “to make safety a priority in organization activities, and to be partners in the
conference’s safety action plan” (DOT, Press Release DOT 35-99, March, 3, 1999).
Individual participation was elicited with a call to “Sign on for Safety” pledge.
In August 1999, Rodney Slater, Secretary of Transportation, announced that
“the Senate Governmental Affairs Committee, Chairman Fred Thompson, and the
General Accounting Office have concluded that the U.S. Department of
73
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Transportation’s Fiscal Year 2000 Performance Plan is the best in the federal
government,. . . [for] the second year in a row” (p. 1). He expressed his appreciation
to the executives, managers and staff whose leadership “brought about institutional
change and successfully implemented the Government Performance and Results Act”
(p. 1).
DOT’s Fiscal Year 2000 Performance Plan integrated budget requests with
the strategic goals. A total of $3.4 billion for direct safety programs, a 4.3 percent
increase over the preceding year, was requested and justified in terms of performance
goals (DOT, DOT Performance Plan FY 2000, p. 7, available at
http://www.dot.gov.hot/pplanr/html). Performance goals were explained as
reductions in rates of accidents and injuries and were specified with more detail than
provided the previous year. For example, rather than reductions in highway fatality
and injury rates alone as stated in the previous year, goals for alcohol related highway
fatalities, seat belt use, large truck fatality and injury rates were described. Air
carrier and general aviation fatal accident rates as well as numbers of runway
incursions, aviation operational errors and deviations, recreational boating fatalities,
maritime searches and rescues, and passenger vessel safety goals were also
described.
DOT’s Fiscal Year 2000 performance plan also named direct operations,
infrastructure investment, capital investment in operations technology, financial tools
(such as loan guarantees for shipbuilding), rulemaking, enforcement, technology
74
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
development and application, and education as the interventions and actions
available for managing the transportation system.
Some of these interventions and actions reside entirely within the
Federal government, but most involve significant partnering with
state and local authorities and with the transportation industry.
Tax expenditures are also a significant tool by which the Federal
government encourages transportation investment, but [they] do
not represent a key tool of intervention by DOT. (DOT, DOT
Performance Plan for Fiscal Year 2000, p. 3)
The DOT Performance Plan FY 2000 also contained a discussion of data
limitations in performance measures and remedial measures that were being put in
place. It noted that among the problems were the wide variety of data sources and
the challenge of maintaining statistical quality standards. In particular, it noted:
Several measures (particularly in safety) require aggregation
across modes. This can be problematic in some cases because of
the use of different definitions (an injury may be defined
differently in each industry and mode). Also, data from outside
the Department may have unknown error properties, (p. 91)
To address this and other data quality issues, a statistical policy framework led by the
Bureau of Transportation Statistics was described as a remedial measure that was
under development.
Integration of efforts for the strategic safety goal was reinforced when the
DOT Safety Council was established in 1998 to coordinate interagency strategies for
common transportation safety problems. This council included executive
representatives from all of the modal administrations. As of November 1999, Safety
Council strategies included:
75
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
I. Promote public information, education, safety awareness and
community involvement across all modes. Permanently expand
and institutionalize NHTSA's Safe Communities Program into a
DOT-wide safety outreach development and delivery system for
safety program messages directed towards the general public and
commercial operators.
II. Identify safety-related data needs and improve the data
presently collected for all modes. Near-term focus will be on
intermodal data needed to measure DOT's General Performance
goals; four conferences will be held in FY1999 to identify longer-
term data needs in all modes and cost estimates for data
acquisition.
III. Hours-of-service rulemaking—provide assistance to
FHWA/Office of Motor Carrier and Highway Safety.
IV. Promote corporate management/labor partnerships in
human factors, with particular emphasis on fatigue management
programs and automated instructional training systems
development, (available at http://www.dot.gov.safetycouncil)
The role of learning was cited in DOT’s combined Fiscal Year 1999
performance report and Fiscal Year 2001 performance plan. In a reader’s guide to
the Fiscal Year 2001 performance plan, it was noted that “this year’s product builds
on the suggestions of our stakeholders and what we have learned within our own
programs” (DOT, March 2000, p. 5). The Secretary explained that the prior year's
performance report was being published with the Fiscal Year 2001 performance plan
for the following reason:
In order to make our goals and results more useful to our
stakeholders, we have chosen to combine our report on 1999
results with our plan for 2001 performance. At DOT we have
come to understand that using measurement to manage will only
succeed when we understand historical trends, study recent
results, and then integrate this information into strategies and
resource decisions. By putting this information in one document,
we hope we have created a tool that is more relevant and useful to
you, the reader. (DOT, March 2000, p. 1)
76
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The Secretary then summarized overall performance:
In 1999 we met or saw good trends in 77 percent of our goals....
While we did not meet 15 goals, in most cases the goal was
missed by a small margin or the results was effected by an
unexpected event such as unusually bad weather. We believe that
the results for only five of these goals require consideration of
new strategies and we are taking the necessary steps to make that
determination. (DOT, March 2000, p. 1)
After an introduction to what the report contained, the document suggested that
readers “view 1999 results with an eye both to the target and the long-term trend”
(DOT, March 2000, p. 5). Table 3 summarizes the actual results for Fiscal Years
1997 through 1999 and goals for 2000 and 2001.
The performance goals, strategies, and results for each mode-specific goal
were explained in the report. Of these goals, only the goal for automobile front seat
belt use and all four aviation goals were reported as neither met nor on a good trend
toward the goal. All of DOT’s safety goals and reported performance results are
discussed in Chapters 5 through 7, which explore conditions associated with
performance measurement use in achieving highway, aviation, and waterway safety.
Chapter 8 summarizes the analysis and discusses the underlying theory.
77
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Table3
Results Reported in DOT Fiscal Year 1999 Performance Report and
Goals for Fiscal Year 2001
Strategic Aggregate
Measures
1997 1999 2000 2001
Highway fatalities/100 million
vehicle miles traveled
1.6
Actual
1.6
Actual
1.6 Goal
1.5 Actual
1.5 Goal 1.5
Goal*
maybe
lowered
Highway injuries/100 million
vehicle miles traveled
133
Actual
122
Actual
127 Goal
119 Actual
116 Goal 113 Goal
% Highway fatalities alcohol
related
38.6
Actual
38
Actual
38 Goal
36 Actual
35 Goal 34 Goal
% Front occupants using seat
belts
69
Actual
70
Actual
80 Goal
67 Actual
85 Goal 86 Goal
# Fatalities involving large trucks No Goal
5,203
Actual
4,934
Goal
4,830
Goal
# Injuries involving large trucks No Goal
127,000
Actual
125,000
Goal
122,000
Goal
U.S. commercial fatal aviation
accidents/100,000 flight hours
.055
Actual
.006
Actual
.037 Goal
.040 Actual
.033
Goal
.031
Goal
Runway incursions 318
Actual
325
Actual
270 Goal
322 Actual
248 Goal 241 Goal
Operational errors/100,000
activities
.48
Actual
.55
Actual
.49 Goal
.57 Actual
.486
Goal
.5 Goal
Deviations/100,000 activities .12
Actual
.18
Actual
.099 Goal
.18 Actual
.097
Goal
Discon
tinued
Recreational boating fatalities 857
Actual
864
Actual
773 Goal
763 Actual
763 Goal 749 Goal
% Mariners rescued that are
reported in life-threatening
danger
93
Actual
94
Actual
95 Goal
93 Actual
93 Goal Discon
tinued
% All mariners in imminent
danger rescued [new measure]
88 Actual No 2000
Goal
85 Goal
Fatalities/100,000 workers aboard
commercial vessels
34 Goal
28 Actual
Goal
Discon
tinued
# High risk passenger vessel
casualties per 1,000 vessels
53 Goal 52 Goal
*A 11 Fiscal Year 1999 actual figures are preliminary and may be revised.
Source: U.S. Department of Transportation (2000, March). 1999performance report 2001
performance plan U.S. Department of Transportation, pp. 10-34, available at
http://www.dot.gov.
78
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The Deputy Secretary believed that the real test of maintaining program
manager decision authority based on performance management depended on how the
feedback loop, including the Congress, worked. He had not seen a lot of evidence
that appropriators, for example, want to change the traditional way of justifying
appropriations. He pointed out that, in Fiscal Year 1999 and through Fiscal Year
2000 to-date, the Office of Management and Budget (OMB) had supported the move
to performance management, but the future was uncertain. “The whole experiment
in performance management will fail if the Congress or OMB use the information to
punish or demean the career managers” (Mortimer Downey, personal
communication, July 1,1999). Also, he believed that there must be big rewards for
good performance and that focusing only on bad news will “pervert the process” and
lead to evasive reporting about performance (Mortimer Downey, personal
communication, July 1,1999).
To make the programmatic relationship to goals more explicit, preparation of
the Fiscal Year 2001 budget required that all program change justifications to be
based on performance information. The Deputy Secretary believed that making
performance management the way of doing business would take a few more cycles of
testing the goals, measurements, and analysis of results. Enough of the right
resources—financial, physical, and knowledge—were present, but they were not
necessarily in the right place (Mortimer Downey, personal communication, July 1,
1999).
79
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Performance agreements were another means used to keep the focus on
results. The Secretary of Transportation had performance agreements with his direct
reports to maintain the focus of attention on outcomes. These agreements were based
on the individual organizations’ performance plans and include milestones for
completion of actions to measure performance and use the results to achieve
improvements. These agreements covered the modal administrations, the
administrative office executives, and the Inspector General. Performance is
measured by accomplishment of milestones and development of alternative strategies
in response to delays. Also, the feedback given to the modal administrators was not
punitive, and the performance agreement process manager did not perceive
reluctance by the political leaders to report negative information (Marylou Batt,
personal communication, February 25,2000).
DOT’s draft strategic plan for Fiscal Years 2000 through 2005 contained
incremental improvements just as the performance plans did. The draft DOT
Strategic Plan FY 2000-2005 improved on its predecessor with a succinct, coherent
discussion of the policy tools and strategies planned to accomplish these goals. In
addition, the Department changed its planning approach by “developing scenarios set
30 years in the future and fleshing out the context for transportation in 2028.”
The Department is dedicated to fostering a climate that
accelerates innovation, by providing leadership and investment
guidance; supporting an educated and motivated transportation
workforce; investing in long-term strategic research; collaborating
with industry, government, and universities; creating a supportive
legal and regulatory framework; and ensuring investment support.
80
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The Department has many tools to bring to hear in support of
innovation. They include research and development, extensions
programs, information dissemination, and the diffusion of new
technologies into the transportation system. They also include the
ability to leverage private and other non-federal funds for
innovative projects through seed money and through innovative
finance, (p. 2, available at http://www.dot.gov)
In addition, the draft DOT Strategic Plan FY 2000-2005 slightly modified
how the national outcome goals were stated. As was illustrated in Table 1, the DOT
Strategic Plan FY 1997-2002 expressed the outcome goals as numerical targets as
well as rates to be achieved in annual increments by the end of Fiscal Year 2002. In
the new draft plan, however, outcome goals were expressed as percentage reductions
in numbers and rates of fatalities, injuries, and incidents to be achieved by dates that
were yet to be determined.
When it was first made publicly available, in December 1999, the draft
strategic plan for Fiscal Years 2000-2005 was a featured item on the DOT Web home
page. Beneath a banner quoting the Secretary of Transportation saying, “Safety is
our number one priority,” the first link was the Draft Strategic Plan FY 2000-2005,
with another link to allow viewers to comment on the draft plan (available at
http://www.dot.gov).
DOT developed an integrated national plan and consolidated goals, but the
departmental performance management process recognized that different systems of
physical and human factors determine the outcomes of federal programs. The policy
tools and strategies for improving transportation safety on highways, aviation, and
81
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
waterways were different, and the stakeholders in each component of the
transportation system were different.
According to the Deputy Secretary, a remaining obstacle facing DOT’s
leadership was to explain links between safety performance indicators and safety
programs (Mortimer Downey, personal communication, July 1, 1999). For example,
public education on behavioral risks in highway and waterway travel was sponsored
by NHTSA and USCG, because changes in human behavior were necessary to reduce
safety risks. However, the link between changes in aggregate measures with the
results of educational activities at the state and local level were not direct. External
factors played a large role in highway safety, and that had to be recognized in
planning and reporting performance.
A Transportation System of Technical and Social Components
As stated in DOT’s early strategic and performance plans, national
transportation safety outcome goals were to be achieved by taking action to influence
the transportation system. This system was described as follows in the Fiscal Year
1997-2002 strategic plan:
The U.S. transportation system carries over 4 trillion passenger
miles of travel and 3.7 trillion ton miles of domestic freight
generated by more than 260 million people, 6 million business
establishments, and 87 thousand units of government. The system
includes 3.9 million miles of public roads, 1.5 million miles of oil
and natural gas pipelines, 123 thousand miles of major railroads,
over 25 thousand miles of commercially navigable waterways,
over 5 thousand public coasts and inland waterways. In 1994, the
system carried 2.3 trillion miles of travel by cars and trucks, 7.4
82
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
billion trips on public transit, 500 million passenger boardings on
airplanes, and 22.1 million trips on Amtrak, and 28.5 million rail
freight car miles. School buses alone carry more than 9 percent of
the U.S. population during a typical school day. The
transportation sector of the United States’ economy has remained
near eleven percent of gross domestic product (GDP) for several
years at a level of $775 billion annually. (1997, p. 3)
This description encompassed the physical and economic scope of the
transportation system, but it did not address the social system associated with
improving transportation program outcomes. This omission was important because
state and local government organizations retain much of the regulatory and
enforcement authority and service delivery related to transportation safety. In
addition to other governmental jurisdictions, organized interest groups and regulated
industries participated in policy tests, results measurement, program implementation
oversight, and evaluation of transportation safety results. These groups included
transport manufacturers, vehicle operators, passengers, labor unions, and local
communities, to name just a few of an extremely large, diverse set of organizations.
In fact, the Department and the modal administrations have numerous partnership
agreements with participants in these social systems.
Analysis of the social aspect of the transportation system required different
methods of inquiry than were required by the physical and economic aspects of the
system. Peter Checkland, an engineering scholar, described systems thinking as
thinking holistically, using both technical viewpoints and social viewpoints. A
technical view perceived the “hard system,” which was appropriate to well-defined
83
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
technical situations. A social view perceived a “soft system,” which was appropriate
to fuzzy, ill-defined situations requiring purposeful human activity. In examining
soft systems “the word ‘system’ was no longer applied to the world, it is instead
applied to the process of dealing with the world”; it is “the process of inquiry into the
world” (Checkland, 1999, p. A10). His research indicated that measures to signal
progress in pursuing objectives were essential in managing both technical and social
systems The only difference was that the objectives for the social system “may never
be finally achieved” (p. 173).
A need for learning to improve transportation system safety performance
arises from problems with measuring and controlling the phenomena that influence
achieving transportation safety goals. As Donald J. Wheeler (1993) stated in
Understanding Variation: The Key to Managing Chaos,
Accidents are a result, not a cause. They cannot be managed by
goal setting. The data may be distorted by pressure to meet goals,
but the system that gives rise to accidents will not be affected by
any arbitrary numerical target, (pp. 74-75)
Therefore, anyone wanting to change processes leading to accidents must “study the
system that gives rise to the accidents” (p. 75). The purpose of policy and program
management in DOT’s modal administrations can, therefore, can be viewed as
increasing knowledge about the circumstances that result in transportation accidents
and finding alternative actions that can contribute to changing those circumstances.
The Draft DOT Strategic Plan for Fiscal Year 2000-2005 directly responded
to the social nature of some transportation safety problems and articulated how the
84
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
transportation system would be studied to improve knowledge. Using Checkland’s
definition of systems thinking as a holistic approach that addresses both technical and
social aspects of a problem, this change indicated that systems thinking was
increasingly adopted as a tool for managing transportation safety.
Strategies to achieve the outcome goals were grouped into five policy or
service delivery areas. These groups of strategies included actions in both the
technical (hard system) and social (soft system) aspects of the national transportation
system. Infrastructure strategies were directed to investments and other actions to
improve the physical transportation system. Research and development strategies
included increasing knowledge about both physical and human factors associated
with transportation safety. Strategies for standards, regulation, and legislation
included rulemaking, partnerships with industry and other organizations, and
innovative approaches to enforcement and certification. Education and promotion
strategies were directed at public outreach and improving public knowledge about
transportation system safety. Finally, analysis and information strategies were
directed at improving the quality and usefulness of transportation safety data, with a
focus on risk-based management and attention to identifying best practices.
The transportation system budget program activities that fund these strategies
were displayed in the Fiscal Year 2001 performance plan. Table 4 summarizes the
Direct Safety Programs and budget cited in that plan.
85
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Table 4
Direct Transportation Safety Programs Budget Request, Fiscal Year 2001
Direct Safety Programs FY 2001 Budget Request Smilliop
Total $3,983
U.S. Coast Guard $1,008
Search and Rescue Operations 383
Acquisition 64
Research 2
Marine Safety Operations 440
Acquisition 48
Research 7
Boating Safety Grants 64
Federal Aviation Administration $1,101
Operations Regulation & Certification &
Commercial Space 705
Facilities & Equipment Safety-related Projects 255
Research Aircraft Safety Technology 49
Human Factors & Aviation
Medicine 25
Airport Grants Safety-related Work 67
Federal Highway Administration
Federal-aid Highways 10% Safety Set-aside 719
Seat Belt Grants 92
Safety Incentive Grants 8 1
Federal Motor Carrier Safety Administration $279
Administration 92
Grants 177
Revenue Aligned Budget Authority 10
National Highway Traffic Safety Administration $499
Operations and Research 286
Highway Traffic Safety Grants 213
Federal Railroad Administration $117
Federal Transit Administration ' $6
Research & Special' Programs Administration ■ $ 8 1
Source: U.S. Department of Transportation (2000, March). 1999performance report 2001
performance plan U .S. Department o f Transportation, p. 46, available at http://www.dot.gov.
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Summary: DOT Contextual Conditions and Strategies
The evolution of DOT’s strategic and performance plans and its first
performance report revealed a three-part approach. First, the departmental role was
leadership and facilitation. The Deputy Secretary played a direct role through
performance feedback to the administrators of the operating administrations. This
feedback was directed at learning about why planned strategies were not achieving
targeted results. Strategic direction was given from the top down, and
implementation was accomplished through performance planning, strategy
development, and performance monitoring from the bottom up.
Second, transportation plans and reports focused at the system level—with
both technological and social components acknowledged. Budget programs included
research for highways, aviation, and waterway safety. This research encompassed
both technological and human factors associated with transportation safety.
Third, goals and strategies for transportation safety included learning
components. As plans were changed, the changes were attributed to learning from
performance. Action strategies included research and collaboration among
stakeholder groups to learn about the sources of safety risk and how to reduce risks in
the transportation system. The networks of federal, state, local, industry, non-profit
organizations, and interested individuals were recognized as participants in learning
how to improve safety results.
87
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Performance information about the safety results in the transportation system
included information about the context for producing the results. External factors
that affected policy and program actions were analyzed along with the evidence
about how government policies and actions influenced results.
The performance report recognized that achievement of national goals for
safety depended upon very complex and large networks of state and local
governments, the transportation industry, other groups, and individuals who take an
interest in research and advocacy for improving safety. While goals for system
change were used to guide this action, it was recognized that these goals were more
difficult to attain in the social aspects of the system than in the technical aspects of
the system. Looking backward into what actions did not result in goal attainment
was used as the starting point for looking forward into modifying the goals or
modifying the action strategies for attaining them.
A summary of the contextual conditions for DOT in using performance
information to improve results is outlined below:
Institutional Contextual Conditions
► The transportation system was composed of multiple, diverse
components with safety risk as a common characteristic.
► Department-led forums and events were used to facilitate public
involvement at the national and the community level.
88
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
► Partnerships for learning how to reduce safety risks were established
among networks of government organizations and stakeholders that
shared the common goal of reducing safety risks.
► A Safety Council facilitated cross-organizational and multiple discipline
collaboration.
Organizational Contextual Conditions
► Transportation safety policies and programs were managed by five semi-
autonomous operating administrations.
► Departmental leadership interpreted performance information as the
means for learning how to improve results over time.
Technical Contextual Conditions
► Solutions to methodological and data quality problems were led from the
departmental level.
► Strategies for improving transportation system safety results included
both technical and social interventions.
Resource Contextual Conditions
► Performance planning and monitoring were linked to budget formulation.
► Finding the right organizational location for resources to implement
performance based management.
89
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Uses of Performance Information
► Social learning through public information and education.
► Building consensus among highly diverse stakeholders.
► Systems thinking about both technical and social strategies to reduce risk.
► Justifying resource requests.
► Monitoring changes in safety risks over time.
► Organizational learning about causes and interventions to control safety
risk.
Conclusions and Hypotheses
Transportation system safety goals for 1999 were met at the aggregate level,
but the goals for the different components of the system were not consistently met.
The operating administrations moved at different paces toward improving safety
performance. This fact suggests two hypotheses.
First: Different contextual circumstances presented different challenges to
operating administrations using performance information to achieve a
common goal.
Second: DOT’s management approach for implementing performance based
management facilitated use of performance information to improve results in
diverse contextual conditions.
The substantiation or rejection of these hypotheses will be addressed in Chapter 8.
90
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The next three chapters answer the question: What were the contextual
conditions associated with using performance information to improve highway,
aviation, and waterway safety outcomes? What strategies were used to facilitate
using performance information to improve highway, aviation, and waterway system
safety outcomes?
91
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
CHAPTER 5
MANAGEMENT TO IMPROVE HIGHWAY SAFETY
OUTCOMES
Most fatalities in the transportation system occur on highways. More than
41,345 people died in highway crashes in Fiscal Year 1999, and 3 million people
were injured. Highway crashes contribute over 90 percent of all fatalities in the
transportation system and are the leading cause of accident-related deaths for youth
in the U.S. (Martinez, 1996,1997,1998, 1999).
A host of governments, industries, and stakeholder groups play roles in
planning, monitoring, and taking action to reduce the risks of death and injury on
highways. DOT’s role includes administrative leadership and facilitation of actions
at state levels plus regulation and enforcement of standards for vehicle
manufacturing and commercial trucking operations.
As Table 5 illustrates, DOT’s 1999 Performance Report described success at
attaining or being on a good trend to attain all but one of its Fiscal Year 1999
highway safety goals, based on preliminary data. A time lag in data reporting from
state and local highway safety officials and the fact that data are normally reported on
an annual rather than a fiscal year basis necessitated using preliminary data. The
92
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
DOT Performance Report noted that shifting data reporting to a fiscal year basis
would make past-year comparisons difficult, add confusion, and provide little benefit
to program management.
TABLES
HIGHWAY SAFETY PERFORMANCE GOALS AND PRELIMINARY
RESULTS FOR 1999*
Highway Safety Performance Goals 1998 1999
Goal
1999
Actual
Highway fatalities/100 million vehicle miles
traveled
1.6 1.6 1.5
Highway injuries/100 million vehicle miles
traveled
122 127 119
% highway fatalities alcohol-related 38% 36% 38%
% front occupants using seat belts 70% 80% 67%
*Calendar year data
Source: U.S. Department of Transportation (2000, March). 1999performance report 2001
performance plan U.S. Department o f Transportation, available at http://www.dot.gov.
Successes were reported for reductions in highway fatalities and injuries.
Highway fatalities in Fiscal Year 1999 were 1.5 for each 100 million vehicle miles
traveled (VMT), a better outcome than the goal of 1.6 for each 100 million VMT.
Highway injuries were 119 for each 100 million VMT, a better outcome than the goal
of 127 for each 100 VMT (DOT, March 2000, p. 15).
There were two highway safety goals that were not met. The percent of
highway fatalities in alcohol-related accidents was 38 percent, which does not reach
the goal of 36 percent but was statistically on a favorable trend. The use of seat belts
93
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
by front seat occupants, however, did not increase from 1998 to 1999, while there
was a goal to increase use from 70 to 80 percent (DOT, March 2000, p. 15).
Table 6 summarizes DOT’s budget program activities that had primary or
secondary influence on highway safety. Primary influences were achieved through a
variety of grants, research, and regulation activities that will be discussed in more
detail later. Secondary influences came from activities primarily undertaken to
achieve another national transportation strategic goal—mobility, economic growth
and trade, the environment, or national security.
It shows that highway budget programs are under the administrative
leadership of three Department of Transportation (DOT) modal administrations. The
Federal Highway Administration (FHWA) manages grants for the construction,
maintenance, and improvement of the National Highway System; for improved data
collection on highway conditions and driver safety records; and for incentives to
states for improving bicycle and pedestrian safety. The National Highway
Transportation Safety Administration (NHTSA) sets passenger vehicle
manufacturing standards, monitors vehicle accident data for safety assurance, and
manages grants to demonstrate ways to reduce drunk driving, to increase seat belt
and child seat restraint use, and to expand use of locally developed practices that are
found to be effective in reducing driving and pedestrian risks.
94
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
TABLE 6
ESTIMATED OBLIGATIONS FOR HIGHWAY SAFETY PROGRAM
ACTIVITIES, FISCAL YEAR 2001
FHWA Federal Aid Highways Total $892 Million
Estimated
0 Surface Transportation Program 0 $719 million
0 National Highway Program 0 Secondary influence
0 Interstate Maintenance o Secondary influence
0 Bridge Program 0 Secondary influence
0 Minimum guarantee 0 Secondary influence
0 Intelligent Trans. System (ITS) Standards, research, tests, dev. 0 Secondary influence
0 ITS Deployment 0 Secondary influence
0 Transportation Research o Secondary influence
0 Federal Lands Highways 0 Secondary influence
0 Administration 0 Secondary influence
o Safety Incentive Grants (seat belts) 0 $ 92 million
0 Safety Incentive G rants (alcohol) o $ 81 Million
0 Internal Revenue Initiative 0 Secondary influence
0 Minimum allocation/guarantee (exempt) 0 Secondary influence
FMCSA Total $279 million
0 Administration o $ 92 million
0 G rants 0 $177 million
0 Revenue Aligned Budget Authority 0 $ 92 million
NHTSA $ 500 million
Operations & Research
0 Safety Performance Standards 0 $ 20 million
0 Safety Assurance o $ 26 million
o Highway Safety Programs 0 $ 95 million
0 Research & Analysis 0 $121 million
0 Office of the Administrator 0 $ 5 million
0 General Administration 0 $12 million
Highway Traffic Safety G rants
0 Section 402 Formula G rants 0 $ 158 million
0 Section 405 Occupant Protection G rants 0 $ 13 million
o Section 410 Alcohol Incentive G rants 0 $ 36 million
0 Section 511 Safety Data G rants 0 $ 9 million
0 Section 2003(b) Child Passenger G rants 0 $ 8 million
Source: U.S. Department of Transportation (2000, March). 1999performance report 2001
performance plan U.S. Department o f Transportation, pp. H-3 -II-4, available at
http://www.dot.gov.
95
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The Federal Motor Carrier Safety Administration (FMCSA) was created from
part of the Office of Motor Carrier and Highway Safety in FHWA in January 2000.
It publishes and enforces regulatory standards to improve the safety of large truck
and bus interstate commercial operations. FMCSA also supports research and
development activities to improve commercial driver and vehicle safety.
This chapter discusses the contextual conditions associated with using
performance information to improve highway safety outcomes in FHWA, NHTSA
and FMCSA as they implemented the activities previously summarized in Table 4. It
also describes the management strategies these organizations used to overcome
obstacles to using information about past performance to improve future
performance. First, a background section briefly summarizes the historical evolution
of federal policies and organizational arrangements for highway safety. The
background section also describes the categories of conditions that potentially affect
use of such information. The background section is followed by descriptions of the
obstacles and related management strategies used in each of the highway operating
administrations as they implemented the Government Performance and Results Act
(GPRA).
Background
From their beginnings, federal activities for highway safety have been
intergovernmental and have included performance monitoring. Building roads was
considered the responsibility of private citizens and states until World Wars I and II.
96
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The earliest predecessor of the Federal Highway Administration was the Office of
Road Inquiry, which collected information on highway conditions. It was established
in the Department of Agriculture in 1894 as a temporary agency with two employees
(Weingroff, 1994, p. 2; Goddard, 1994, p. 59).
Within two decades, the Office of Road Inquiry had been made permanent. It
was renamed the Office of Public Road Inquiry, with a mission of performing
research and development, continuing to collect information on highway conditions,
and building roads in National Parks, National Forests, and other federal lands
(Weingraff, 1994, p. 4).
The engineers who managed these early federal highway activities allied their
efforts with the emerging public interest groups that advocated highway construction
and automobile travel, including the Automobile Association of America (AAA),
established in 1903, and the American Association of State Highway Officials
(AASHO), established in 1914 (Goddard, 1994, pp. 50-59). Also, the Federal Office
of Public Road Inquiry became a training ground for highway engineers and for
technology development to improve the durability and safety of roads (Goddard,
1994, p. 55). Goddard notes that expansion of federal highway policies were linked
to data collection and analysis:
The Progressive era had demanded that what you know rather
than whom you know should govern public decision making....
Within two years of arriving in Washington in 1919, [Thomas
MacDonald, Director of the Bureau of Public Roads] had come to
dominate the fledgling highway community simply by gathering,
using and manipulating the largest database in town. (p. 103)
97
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Moving to the forefront of the lobbying effort [in 1921] were
some unlikely power brokers—the engineers from the American
Association of State Highway Officials (AASHO). Their quiet
marshaling of fact convinced Washington to split the cost of a
system of national routes, yet to let the states themselves choose
where it would go. (p. 91)
This orientation to performance-related support for policy development
extended to safety practices as well. For example, the construction standards for
interstate highways included safety standards to decrease the numbers of accidents
and fatalities (Goddard, 1994, p. 194).
During World War I, the Federal-aid Highway Program was established to
allocate funds for highway construction in aid of the war effort. It also established
state highway departments (Goddard, 1994, p. 63).
Further expansion of the federal role in highways was justified by its
supporters, in part, by the need to improve road safety. As an interstate commerce
concern, safe operation of commercial trucking companies came under federal
regulation in 1925 (Goddard, 1994, p. 99). Thirty years later, the Interstate Highway
Act and the Highway Revenue Act of 1956 established a program to build a federally
funded interstate highway system, financed through the Highway Trust Fund by
gasoline taxes (Goddard, 1994, p. 196). Vice-President Richard Nixon stated to a
national governor’s conference in 1954 that highway crashes resulted in “an annual
death toll comparable to the causalities of a bloody war” (Goddard, 1994, p. 184).
By the time the Department of Transportation was established in 1967, the
Federal-aid Highway program had come under attack as a source of financial
98
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
corruption, environmental damage, and irresponsible land use (Goddard, pp. 216-
220; Weingraff, 1994, pp. 10-12). The Bureau of Public Roads joined the newly
established Bureaus of Motor Carrier Safety and National Highway Safety wi thin
DOT’s Federal Highway Administration (FHWA) (Weingraff, 1994, p. 12).
Then, in 1970, the National Highway Safety Bureau was separated from
FHWA to become the National Highway Traffic Safety Administration (NHTSA).
NHTSA’s mission was focused on highway safety through safety standards for
vehicles and promotion of state and local highway safety efforts.
In 1996, after the Interstate Commerce Commission was abolished, FHWA
inherited the mission of regulating commercial trucking safety. These functions were
added to the Office of Motor Carrier Safety’s data collection and analysis functions
(Public Law 104-88 § 14123). In December 1999, the Office of Motor Carrier Safety
was once again reorganized. It was separated from FHWA to become the Federal
Motor Carrier Safety Administration (FMCSA) (Public Law 106-159 § 1) to increase
attention to improving safety standards for commercial truck and bus operations.
These organizational changes for highway safety program management in the
last decade of the 20th Century occurred as federal highway policies underwent a
series of changes to improve integrated planning for all modes of surface
transportation. The Interniodal Surface Transportation Efficiency Act of 1991
(ISTEA) created the National Intermodal Transportation System to emphasize that all
surface transportation was interconnected in ways that required integrated planning.
99
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Subsequently, the National Highway System Designation Act of 1995 created
the National Highway System. It included approximately 155,000 miles of roads—
the Interstate System (42,795 miles) and other principal highways critical for defense
and for the movement of people and goods. Finally, when ISTEA expired in 1997,
the Transportation Equity Act for the 21s t Century (TEA-21) extended authorizations
for programs to improve surface transportation planning and outcomes through Fiscal
Year 2003 (Federal Highway Administration History, available at
http://www.fhwa.dot.gov/ctdiv/history.htm).
The remainder of this chapter will discuss DOT’s modal administration
activities to use information on the results of past policies and actions to manage for
future highway safety results. Part I will discuss the oldest of these organizations,
FHWA; Part II will discuss NHTSA; and Part III will discuss FMCSA.
Four categories of contextual conditions that can affect a government
organization’s ability to manage for results will be presented. The emphasis will be
on problematic conditions that inhibit using performance feedback for results
management and the management strategies used to overcome the problems. First,
institutional conditions that link the federal organizations to the outside world
through policies, programs, and networks of stakeholders will be described. Then,
organizational conditions that focus on the internal operations of the organization
will be covered. Next, technical conditions that affect organizational ability to
develop goals, indicators, high quality data bases, analytical methods, or useful
100
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
reports will be described. Finally, resource factors that affect organizational capacity
for financing and staffing functions necessary for planning, analyzing, monitoring,
and reporting safety results will be described.
Part I: The Federal Highway Administration (FHWA)
In 1998, the Transportation Equity Act for the 21s t Century (TEA-21)
designated “the safety and security of the transportation system for motorized and
non-motorized users” as one of the seven areas to be considered in the highway
planning process (FHWA, July 1998, available at
wysiwyg://21/http:www.fhwa.dot.gov/tea21/sumtoc.htm). TEA-21 also authorized
including bicycle and pedestrian safety activities in the formula grants process.
FHWA manages the Federal-aid Highway budget program authorized by
TEA-21, and it will distribute a total of $11.5 billion to states in Fiscal Year 2000
(GAO, June 2000). This program is funded through fuel taxes and accounted for in
the Highway Trust Fund. Funding comes from fuel taxes deposited in the Highway
Trust Fund and is allocated to states by complex formulas developed and
implemented by the Internal Revenue Service and by FHWA (GAO, June 2000,
p. 4).
The policy tools available to FHWA are grants, public information, and
promotional activities to bring attention to road conditions. The Surface
Transportation Program, within the Federal-aid Highway budget program, provides
formula grants to states for correction of safety hazards. Separate budget activities
101
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
fund incentive grants for increasing seat belt use and for reducing intoxicated
driving.
Successes were reported for reductions in highway fatalities and injuries.
Highway fatalities in Fiscal Year 1999 were 1.5 for each 100 million vehicle miles
traveled (VMT), a better outcome than the goal of 1.6 for each 100 million VMT.
Highway injuries were 119 for each 100 million VMT, a better outcome than the goal
of 127 for each 100 VMT (DOT, March 2000, p. 15).
There were two highway safety goals that were not met. The percent of
highway fatalities in alcohol-related accidents was 38 percent, which does not reach
the goal of 36 percent but was statistically on a favorable trend. The use of seat belts
by front seat occupants did not increase from 1998 to 1999, while the goal was to
increase use from 70 to 80 percent (DOT, March 2000, p. 15).
Table 7 summarizes FHWA’s budget program activities that had primary or
secondary influence on highway safety. Primary influences were achieved through a
variety of grants, research, and regulation activities that will be discussed in more
detail later. Secondary influences came from activities primarily aimed at
implementing the other national transportation strategic goals—mobility, economic
growth and trade, the environment, or national security.
The following sections discuss the institutional, organizational, technical, and
resource issues FHWA faced during full implementation of GPRA.
102
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
TABLE 7
ESTIMATED OBLIGATIONS FOR FHWA SAFETY PROGRAM
ACTIVITIES, FISCAL YEAR 2001
Program Safety Goal Mobility Goal
Surface Transportation
Program
$ 719 million
National Highway Program Secondary influence $ 5,487 million
Interstate Maintenance Secondary influence $ 4,561 million
Bridge Program Secondary influence $3,902 million
Minimum guarantee Secondary influence $ 1,905 million
Intelligent Trans. System
(ITS) Standards, research,
tests, dev.
Secondary influence $ 100 million
ITS Deployment Secondary influence $ 238 million
Transportation Research Secondary influence $ 294 million
Federal Lands Highways Secondary influence $ 728 million
Administration Secondary influence $316 million
Safety Incentive Grants (seat
belts)
$ 92 million
Safety Incentive Grants
(alcohol)
$ 81 Million
Internal Revenue Initiative Secondary influence $ 20 million
Minimum allocation/guarantee
(exempt)
Secondary influence $ 664 million
FHWA Total $ 892 million $18,215 million
Source: U.S. Department of Transportation (2000, March). 1999performance report 2001
performance plan U.S. Department of Transportation, pp. 11-3-11-4, available at
http://www.dot.gov.
Institutional Conditions in FHWA
Policies assigning FHWA highway safety responsibilities were primarily
directed at financing improvements in physical infrastructure safety factors, but two
incentive grant activities were directed at behavioral safety factors. For these
behavior efforts, FHWA collaborated with NHTSA, which has departmental
103
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
leadership responsibilities for activities funded through grants for “work on
educational and enforcement activities designed to change human behavior while
using the road way environment” (FHWA Strategic Plan 1998, p. 38).
FHWA’s Office of Highway Safety
. . . is responsible for administering highway safety activities
related to the roadway and the road user. These safety programs
remove, relocate, or shield roadside obstacles, identify and correct
hazardous locations, eliminate or reduce hazards at railroad
crossing and improve signing, pavement, markings, and
signalization (available at http://www.ohs.fhwa.dot.gov].
Safety standards and guidelines were prepared to help state authorities identify and
correct conditions that contribute to highway accidents. FHWA's highway safety
program also provides guidance and funding for “traffic engineering services, speed
management, pedestrian and bicycle safety, and safety management systems”
(available at http://www.ohs.fhwa.dot.gov).
Authorized in TEA-21, within the Surface Transportation Program, the 10
percent Safety Set Aside activity was the source of funding for highway safety
infrastructure improvements administered by states. Obligations for this activity
were estimated to increase by 9 percent in Fiscal Year 2000 and to increase by
another 5 percent increase in the Fiscal Year 2001 budget request (DOT, March
2000, p. 46).
These funds were to be used by states to “remove, relocate, or shield roadside
obstacles, identify and correct hazardous locations, eliminate or reduce hazards at
railroad crossings, and improve signing, pavement markings, and signalization”
104
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
(FHWA Web page, available at http:www.fhwa.dot.gov/). States, in turn, use
contracts to implement approved projects.
In addition, FHWA administered incentive grants to states that meet specified
criteria and that meet specified performance criteria. Incentives were provided to
states that enact laws: (1) to allow drivers license revocation for repeated drunk
driving offenses and (2) to issue graduated licenses for young drivers. Performance
grants provide incentives based on results attained to reduce alcohol-related fatality
rates and were intended to reinforce actions to continue the trend.
In 1999, approximately $57.4 million went to states with 0.08
BAG laws and another $55 million in FY 2000 to states for good
seat belt use rates.. . . Forty-seven states, the District of
Columbia, Puerto Rico, and four territories (Guam, American
Samoa, Northern Mariana Islands, and the U.S. Virgin Islands)
applied for and received grants [to improve traffic record systems]
in FY 1999. (Wykle et al., 2000, p. 6)
Within the federal administrative structure, agreement on performance
planning goals and indicators were sought at two levels. The Administrator of
FHWA pointed out that the DOT Office of the Secretary led the effort to get
agreement on performance planning goals with Congressional stakeholders. For the
modal administrations,
. . . there is an advantageous process set up between the Office of
Management and Budget (OMB) and the federal agencies to allow
frequent interaction with OMB to explain information that may be
unclear, and to reach an agreement on questionable issues.
(Kenneth Wykle, personal communication, June 19,1999).
105
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Non-governmental groups were also involved as stakeholders in FHWA’s
planning and implementation process. As a means to collaborate on how to
accomplish shared outcomes, FHWA had partnership agreements with the
Associated General Contractors of America (AGC), the American Public Works
Association (APWA), the American Society of Civil Engineers (ASCE), the National
Society of Professional Engineers (NSPE), and the National Association of County
Engineers (NACE). Furthermore, “FHWA will support initiatives developed by the
Office of the Assistant Secretary for Transportation Policy to expand partnership
opportunities” (Kenneth Wykle, personal communication, June 19,1999).
Organizational Conditions in FHWA
In Fiscal Year 1999, FHWA obligations were over $26 billion (DOT, March
2000, pp. 46-47), and its workforce was about 3,390 full-time equivalent staff (OPM,
available at http://www.opm.gov/feddata/table 15). This workforce is decentralized
in ten regional offices and liaison offices in each state.
The Associate Administrator for Safety and System Applications led
FHWA’s safety activities. Four administrative offices—Office of Highway Safety,
Office of Traffic Management and Intelligent Transportation Systems Applications,
Office of Technology Applications, and the National Highway Institute—provide
headquarters leadership and oversee operations in the field.
Direct safety activities included the Safety Set Aside Program, seat belt
grants, safety incentive grants, and highway safety grants. The Safety Set Aside
106
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Program, in the Office of Highway Safety, obligated about 86 percent of FHWA’s
funds allocated to safety in Fiscal Year 1999 (DOT, March 2000, p. 46).
The other offices under the Associate Administrator for Safety and System
Applications led implementation of activities for technological improvements and for
providing training, guidance, and other technical assistance for practitioners in the
highway safety field. As Table 7 indicated, these budget activities had secondary
rather than direct influence on safety objectives.
FHWA’s performance plans mirrored the DOT performance plans in format
and approach, but they described strategies unique to the activities FHWA leads. For
example, among the “Special Challenges” discussed in FHWA’s Fiscal Year 2000
Performance Plan was: “Overcoming a traditional highway development and
operations resistance to addressing human behavior” (FHWA, 1999, pp. 7-8). The
FHWA strategy for addressing this challenge was to follow NHTSA leadership in
implementing activities for behavioral change aimed at increasing use of seat belts
and reducing the number of alcohol-related crashes. This strategy also included a
leadership role for FHWA in implementing activities to reduce road crashes from red
light running, unsafe work zone practices, and unsafe rail grade crossing conditions.
Many of the activities and initiatives planned for Fiscal Year 2000 were
collaborative. For example, one initiative for improving the safety management
processes linked FHWA action with the Strategic Highway Safety Plan of the
American Association of State Highway Transportation Officials (AASHTO).
107
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Another initiative was to “achieve 1,000 Safe Communities” through public and
private partnerships. Others were promotional—to do such things as increase use of
rumble strips near intersections, market self-assessment of safety practices in states
and communities, and increase the number of state and local agencies adopting Road
Safety Audits. (FHWA, 1999, pp. 9-10).
Planned implementation steps for these initiatives and activities are described
in a performance agreement between the Administrator of FHWA and the Secretary
of Transportation (Marylou Batt, personal communication, February 25,2000). In
Fiscal Year 1999, DOT tracked 20 corporate management strategies for achieving
safety performance goals in FHWA, all of which were completed at year end (DOT,
January 2000, p. 1).
Organizational feedback on performance is directed at looking at any need for
changes in strategy to improve in areas that appear to be lagging. The Administrator
of FHWA indicated that “we are trying to assure program managers that the data will
not be used to punish individuals or programs for poor performance” (Kenneth
Wykle, personal communication, June 19, 1999).
Technical Conditions in FHWA
Most of the “Special Challenges,” noted in FHWA’s Fiscal Year 2000
performance plan are technical factors that relate to the organization’s limited control
of factors leading to outcomes. External factors and the complexity of the
implementation process complicate measuring FHWA’s contribution to improving
108
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
safety results. The factors include demographic changes that increase the proportion
of drivers in higher risk age groups (FHWA, 1999, pp. 7-8).
Strategies laid out to address these challenges begin with “facilitate
implementation of comprehensive safety management processes with federal, state,
city and local governments, and the commercial transportation industry” (FHWA,
1999, p. 8). This overall effort is aimed at improving information and analysis
systems “to better identify the causes of crashes and implement improvements to
reduce the number and severity of crashes” (FHWA, 1999, p. 8).
These high level strategies are then broken down into activities and initiatives
to be implemented during Fiscal Year 2000. Many of these activities and initiatives
are collaborative and specify numerical targets. For example, one initiative for
improving the safety management processes qualitatively links FHWA action with
the Strategic Highway Safety Plan of the American Association of State Highway
Transportation Officials (AASHTO). Another initiative was to “achieve 1,000 Safe
Communities” through public and private partnerships and to increase the number of
state and local agencies adopting Road Safety Audits to a total of 14. Others
strategies were to promote incremental change—for example, greater use of rumble
strips to reduce run-off-the-road crashes and safety self-assessments to improve
planning in sates and communities.
The Administrator of FHWA said that “finding a meaningful indicator [for
new activities] that can be measured and the cost and length of time required to
109
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
establish a data system for a new indicator” present problems (Kenneth Wykle,
personal communication, June 17,1999). For example, in 1999, GAO evaluations
found data problems that posed obstacles to studying causes of crashes involving
large trucks (GAO, March 1999). These problems contributed to the Congressional
decision to separate the Office of Motor Carrier Safety from FHWA and will be
discussed in greater detail in Part III of this chapter.
Resource Conditions in FHWA
The FHWA Administrator indicated that he believed that, to a moderate
degree, FHWA had the right resources for planning, analyzing, and managing the
organization’s performance. “The obstacles we have had to overcome include
moving the concept of using performance information to manage down into the
organization (Kenneth Wykle, personal communication, June 17,1999). Among the
strategies used to confront this problem were training and contractor support for
helping to develop performance targets. FHWA managers attended an agency-wide
training class “that instructs the participants in the development of performance
measures and aligning these measures to the overall Agency goals” (Kenneth Wykle,
personal communication, June 17,1999). Some aspects of the problem stemmed
from technical challenges as much as from workforce competency challenges.
One obstacle the contractor is addressing is our challenge in
setting targets. An annual target is required for each performance
goal In the performance plan. This target would reflect the
agency’s analysis of how the resources being used to accomplish
the goal will move the agency toward achieving the 10 targets in
110
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
the 1998 Strategic Plan. Setting targets is complicated by the fact
that although the agency can influence accomplishment of all of
the performance goals, its degree of control varies. Many of the
goals are significantly impacted by the decisions made at the state
level on their use of federal funding and the level of funding
provided by the state.
The fact that resources used in a given year may not impact the
performance goal for several years. This time lag needs to be
specifically identified and a method developed to make it easier
for different audiences to understand this issue when using
FHWA performance plans and reports.. . . Similarly,. . . the data
is frequently not available within the time frame required for
reporting on implementation of annual performance plans.
(Kenneth Wykle, personal communication, June 17,1999)
Performance Information Uses in FHWA
The Administrator of FHWA said that he uses performance information for a
variety of purposes that are limited only by the limitations of the existing systems.
He cited the measurement of fatalities and injuries as among the easiest indicators to
use because “we have established system for collecting and reporting data” (Kenneth
Wykle, personal communication, June 17,1999). He envisioned that further
development of the data systems would increase the usefulness of such information
for indicators of program outcomes.
PART II: THE NATIONAL HIGHWAY TRAFFIC SAFETY
ADMINISTRATION (NHTSA)
The National Highway Traffic and Safety Administration’s (NHTSA)
mission is “to save lives, prevent injuries, and reduce traffic-related health care and
other economic costs” (NHTSA, 1998, p. 3). Three categories of goals described the
organization’s strategic approach to accomplishing these goals—“provide leadership
111
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
and set an agenda, support research and apply the results to education, engineering,
and enforcement to reduce road casualties and costs, and transform NHTSA through
continuous improvement” (NHTSA, Budget in Brief, 1999, p. 3).
NHTSA’s Fiscal Year 2001 performance plans identified 11 budget programs
for highway safety, and these programs were categorized into two areas. Operations
and research-accounted for about 55 percent of the budget request. Highway Traffic
Safety Grants accounted for the remaining 45 percent. Table 8 summarizes
NHTSA’s budget program activities that had influence on highway safety results.
TABLES
•ESTIMATED. OBLIGATIONS FOR NHTSA SAFETY PROGRAM
ACTIVITIES, FISCAL YEAR 2001
Programs and Activities Safety Goal
Operations and Research $173 million
Safety Performance Standards $17.million
Safety Assurance $21 million
Highway Safety Program $€2 million
Research and Analysis $66 million
Office of the Administrator $4 million
General Administration $3 million
Highway Traffic Safety Grants $236 million
Section 402 formula grants $167 million
Section 410 Incentive Grants $39 million
National Driver Register $2 million
Occupant protection Incentive program $20 million
Drugged Driving Incentive Grants $ 8 million
NHTSA Total $500 million
Source: U.S. Department of Transportation (2000, March). 1999 performance report 2001
performance plan U.S. Department o f Transportation, pp. II-3-H-4, available at
http://www.dot.gov.
112
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
In 1994, the Office of Management and Budget named NHTSA as one of ten
exemplars for how to implement the Government Performance and Results Act
(GPRA). NHTSA participated as a pilot project in the early stages of implementing
the law. Problems, solution strategies, and observations about the challenges the
organization faced in implementing GPRA requirements for planning, monitoring,
and reporting results were thoroughly documented in pilot program performance
plans and reports. This record was published in 1996 as a case study for the
American Society for Public Administration, The National Highway Traffic Safety
Administration Case Study : Strategic Planning and Performance Measurement.
This record is amplified by program evaluations performed in the Office of
the Associate Administrator for Plans and Policy. These evaluations indicated how
performance feedback supported judgments about results and development of new
strategies to improve results.
The following sections discuss the institutional, organizational, technical, and
resource issues NHTSA faced both in the pilot phase and during full implementation
of GPRA.
Institutional Conditions in NHTSA
The National Highway Traffic Safety Administration “develops, promotes,
and implements effective educational, engineering, and enforcement programs
directed toward ending preventable tragedies and reducing safety-related economic
costs associated with vehicle use and highway travel” (NHTSA, 1998, p. 3).
113
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
NHTSA’s Strategic Plan noted that “the majority ofNHTSA’s efforts ... are
implemented through the agency’s partners, all of whom have unique organizational
characteristics and strengths” that require different types of delivery systems
(NHTSA, 1998, p. 34).
In 1998, the Transportation Equity Act for the 21s t Century (TEA-21)
reauthorized NHTSA’s regulatory role and enacted four new, performance based
grant programs for NHTSA to administer. A formula grant program funds
performance based safety programs to
... support planning to identify highway safety problems, set
goals and performance measures for highway safety
improvements, provide start-up money for new programs, give
new direction and support for existing safety programs, and fund
analyses to determine progress in improving safety. (DOT, M y
14,1998, available at
http://nhtsa.dot.gov/nhtsa/whatsup/tea21progrms/onepage.grt.html)
TEA-21 also authorized incentive grants activities. These included an
incentive grants program “to encourage States to increase seat belt use rates,” which
based qualification on the applicant’s seat belt use rate. Also, a grant program for
occupant protection incentive grants provided grants for primary safety belt use laws
and special traffic enforcement programs and for specific child passenger protection
and education activities (DOT, M y 14,1998, available at
http ://nhtsa.dot. gov/nhtsa/whatsup/tea21 progrms/onepage.grthtml).
Other grants were authorized to encourage state level adoption of specific
safety initiatives. Incentives to prevent operation of motor vehicles by intoxicated
114
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
persons were authorized based on adoption and demonstration of alcohol-impaired
driving countermeasures at the local level. Similarly, penalties for states that failed
to enact laws to prohibit open alcohol beverage containers in the passenger area of a
motor vehicle were authorized to require transfer of funds from federal highway
construction funds to the state’s highway safety program.
In evaluating the lessons learned from the GPRA pilot projects, GAO cited
the usefulness ofNational Highway Traffic Safety Administration’s strategies in
planning for “situations where they have limited control over outcomes” (GAO,
December, 1998, p. 8). GAO found that useful strategies NHTSA employed
included setting a mix of goals, narrowing the goal’s scope to the program’s span of
influence, and statistically adjusting for effects of external factors.
Organizational Conditions in NHTSA
NHTSA’s had a workforce of approximately 610 full-time equivalent (OPM,
available at http://www.opm.gov/feddata/tablel 5), and annual obligations of $360
million at the end of Fiscal Year 1999 (DOT, March 2000, pp. 46-147. The
organization is located at headquarters with representatives in ten field offices, who
report to the Associate Administrator for State and Community Services. It also has
two field offices with test facilities.
NHTSA’s Performance Plan for Fiscal Year 2000 described the organization
as
115
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
... fundamentally a public health, injury control agency. It is
multi-disciplinary in that it uses techniques of such diverse fields
as epidemiology, engineering, biomechanics, the social sciences,
human factors, economics, education, law enforcement, and
communication to address some of the most complex and
challenging problems in the field of public health. (NHTSA,
1999, p .1)
The Administrator of NHTSA noted why it was recognized in 1994 that
changes were needed to expand the network of organizations allied with NHTSA.
With our fatality rate, seat belt use and drunk driving statistics
stagnant, we realized that the easy gains in traffic safety had been
made and that we had three big options to achieve further success:
big government, big budget, and big change. We knew that big
change was the best answer.. . . We took a look at our partners
and realized we were limiting our involvement to a small segment
of those that could help. Safety is not just an industry problem
and it is not just a government problem—it is everyone’s
problem.. . . When sharp divisions occur between disciplines,
knowledge falls through the cracks and is lost or missed. When
boundaries overlap, we create a safety net for people and
stimulate innovation and creativity. (Martinez, 1998, p. 1)
He described an earlier organizational focus on the crashworthiness of vehicles as an
obstacle to achieving improved highway safety outcomes.
The solution was to use multi-disciplinary analysis and dialogue “to focus
technologies on crash avoidance and post-crash injury control” (Martinez, 1998,
p. 1). In addition, because “it is an unfortunate truth that we know more about the
vehicle than the human in it,” the strategy included research into human factors
associated with highway crashes. It also included public education and outreach
efforts to improve public knowledge about the causes of, and preventative behaviors
to avoid, death and injury on highways.
116
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
NHTSA’s planning approach was a top-down-bottom-up melding of
priorities based on feedback. The case study prepared for ASPA noted that
NHTSA’s 1994 Strategic Plan had “structure and content primarily determined by
top management of the agency, with input from stakeholders and NHTSA staff.”
Two years later, the Strategic Execution Plan content was determined “by agency
goal teams, agency program office staff, and Strategic Planning Division staff;”
“each program office within NHTSA develops a list of their respective priority
activities” (Faigin, Dion, & Tanham, 1996, pp. 9-10).
During the pilot phase, bi-monthly meetings were conducted to discuss
progress. When goals were missed, attention was directed at revising strategies. For
example, the Fiscal Year 1996 Performance Plan states:
The recent poor performance of some agency outcome measures,
has led the National Center for Statistics and Analysis to initiate a
study of the statistical correlations between external factors and
safety to determine on a more detailed basis, the external sources
of short-term increase in crashes and fatalities. (Faigin, Dion, &
Tanham, 1996, pp. 9-10)
In January 1995, NHTSA also used an electronic employee survey “to
determine how staff view their work climate and the organization’s management
style” (Office of Strategic Planning and Evaluation, 1996, p. 5). It found that “53%
of NHTSA employees felt that they were now being delegated more authority in
performing their jobs” (NHTSA, 1996, p. 5). Training and Continuous Improvement
Toolkits were developed specifically to enable employees to participate in GPRA-
related efforts.
117
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Technical Conditions in NHTSA
NHTSA’s lack of direct control over the factors that influence highway safety
outcomes is a theme in all of the agency’s planning and reporting documents.
Discussions of demographic, economic, technological, and social factors that
influence traffic safety are provided from the first GPRA pilot project performance
plan in 1994 through the Fiscal Year 1999 performance report to the Congress. For
example, the performance plan for 1995 stated:
Although clear linkages have been established between program
activities and highway safety outcomes, the agency recognizes
that, while it influences these measures, it does not have complete
control over them. This plan identifies come of the external
factors that will influence overall highway safety performance,
such as the economy and demographics. Barriers to achieving
targets for specific program performance measures also are
highlighted. (NHTSA, 1996A, p. 1)
The strategy followed to overcome the “barriers to achieving targets” cited
above was to report results in the context of those barriers. This is illustrated in the
GPRA pilot project performance report for 1995. The second sentence in the 1995
performance report executive summary states: “Seven out of eleven outcome and
intermediate outcome measures moved in the wrong direction” (NATSA, 1996B,
p. 1). This statement is followed by an explanation of the relationships between
periods of economic expansion, as experienced in 1995, and increased highway
deaths.
There are a variety of external factors that affect the occurrence of
crashes, fatalities, and injuries on the road each year. The most
significant external factors are: the economy, the population,
118
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
exposure factors such as miles driven, licensed drivers and
registered vehicles, and lifestyle factors such as levels of alcohol
consumption.. . . Historically, the number of fatalities has risen
during periods of economic expansions and fallen during
recessions. (NHTSA, 1996B, p. 2)
The subsequent plans and reports identified intermediate outcome results and
program outputs that affect achievement of strategic outcomes to provide
measurement bridges or linkages that span long passages of time in highly complex
causal chains. This approach also showed the alignment of programs to outcomes
through budget crosswalks.
All of NHTSA’s plans refer to a model for learning about highway traffic
crashes. The Haddon Matrix “is composed of three time phases of the crash event
plus three areas (human, vehicle, and environment) influencing each of the phases”
(NHTSA, 1998A, p. 11). Haddon, the first NHTSA administrator, was an
epidemiologist and introduced an alternative professional approach to the highway
engineering solutions previously applied to highway safety by the engineering
profession (Mashaw & Harfst, 1990, p. 3).
Resource Conditions in NHTSA
In 1999, the Associate Administrator for Plans and Policy spoke as the
Administrator's representative to say that NHTSA has the right resources for
planning, analyzing, and managing for results. Funding, hiring authority, and access
to experts have not been issues. “Given the history of performance management in
119
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
NYTSA programs, the staff and associated experts are generally already available”
(William H. Walsh, personal communication, June 15,1999).
Measurement Information Uses in NHTSA
What uses are now made of performance information? What evidence is
there that NHTSA uses performance information to improve outcomes?
NHTSA has demonstrated accountability in each performance and results
plan since initiation of GPRA implementation with a pilot project beginning in 1994.
In its first performance plan for Fiscal Year 1995, it was noted that the historical
cost-benefit of highway safety programs had been studied:
In 1991, in response to a request from the Office of Management
and Budget, NHTSA and FHWA assessed the costs and benefits
of the Federal Government’s highway safety programs since their
inception in 1966. The results of the analysis found that the
economic benefits of these programs in reduced crashes, fatalities
and injuries have well exceeded their costs in government
expenditures and increased consumer costs. In fact, the benefits
exceeded the costs by a factor of about three to one. (NHTSA,
1993, p. 2)
In the subsequent performance report, performance targets that were not met were
reported as well as those that were met. Though formats and types of explanations
changed, the same results reporting occurred in each of the following pilot project
reports and in the first Department of Transportation performance report in 2000.
Performance information stimulated organizational change and was central to
experimentation when NHTSA was a GPRA pilot project. Speaking about the
120
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
organizational and programmatic changes NHTSA initiated in 1994, the
Administrator of NHTSA stated:
We recognized that information is a perishable commodity and
worked to develop an international research agenda that would
leverage all of our resources and speed the creation of knowledge
and development of solutions.
The traditional approach has been to focus exclusively on the
crashworthiness of the vehicle. This has led to tremendous
advances in safety. . . many of the easy gains have been made.
What will the next ESV [Enhanced Safety of Vehicles]
conference and the next century bring forward as safety advances?
. . . With 90 percent of crashes caused by human actions, crash
avoidance clearly has an enormous payoff, yet it is unclear if
emerging systems will decrease or increase the task of driving.
NADS [National Advanced Driving Simulator] helps us
understand that human/vehicle interface and will help us better
understand how human respond at the limits of performance.. . .
(Martinez, 1998, pp. 1-2)
A combination of the GPRA requirement that strategic plans and goals focus
on outcomes and NHTSA’s well-developed data systems provided the foundation for
the agency to develop a conceptual model to link program activities with societal
outcomes. NHTSA’s Case Study prepared for the American Society for Public
Administration discussed a three-tiered structure for linking program output
measures with overall outcome goals. It consisted of numerous program output
measures ranging from improved Auto Safety Hotline service to crashworthiness
rulemakings and changes in state seat belt laws. Intermediate outcome measures
were developed in three areas: reduce the occurrence and consequences of crashes
while serving customers. The intermediate outcome measures included such
information as rates of alcohol involvement in crashes, rates of seat belt use, and
121
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
timeliness of responses on the hotline. Finally, the highest level outcome measures
were total national rates of fatalities, injuries, and crashes per 100 million vehicle
miles traveled and rates of fatalities and injuries in the population (Faigin, Dion, &
Tanham, 1996).
This use of performance information to link program activities and outcomes
occurred along with expansion of program activities for behavioral approaches.
By putting a human face on the safety issue, we are showing
America that everyone has a responsibility to find solutions to this
problem.. . . It is not just a regulatory issue, or a cost issue, or a
legal issue. It is, and always has been, about people saving
people. (Martinez, 1998, p. 1)
At the same time, NHTSA’s research and development activities relied on data
collection and analysis to improve knowledge about the causes and preventions for
highway crashes.
Injury control relies on data resources describing the crash event,
the human, environmental and vehicle-related parameters that
make each crash unique as well as details on the injury outcome
and associated health and other economic costs.. . . Data bases
serve as the basis for planning and implementing successful
vehicle and behavioral safety programs by federal and state
agencies as well as the private sector. Analysis of the data
supports problem identification, program planning, consumer
education, and program evaluation. (Martinez, 1997, p. 6)
PART II: THE FEDERAL MOTOR CARRIER SAFETY
ADMINISTRATION (FMCSA)
The motor carrier functions of the Office of Motor Carrier and Highway
Safety (OMCHS) in the Federal Highway Administration became the Federal Motor
122
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Carrier Safety Administration (FMCSA) in January 2000 (Slater, December 1999,
p. 1). The DOT Office of Inspector General (OIG) and the U.S. General Accounting
Office (GAO) had published a number of reports on weaknesses in enforcement and
lack of data for analyzing causes and potential interventions for highway crashes
involving large trucks. Both of these oversight organizations testified in
Congressional hearings about weaknesses in regulation of safety for commercial
interstate trucking operations.
The OIG testimony on a Senate bill to reform OMCHS pointed out the
relatively long history of proposed reform of motor carrier safety activities.
Over three decades ago the debate was whether the Bureau of
Motor Carrier Safety should remain in the Interstate Commerce
Commission or be placed in DOT. The debate focused on
addressing accident prevention and, ultimately, the Bureau was
placed in FHWA. Twenty years later, a bill to establish a Motor
Carrier Administration was introduced in the Senate Commerce
Committee. Again, the argument was to reduce the number of
accidents by improving the effectiveness of the motor carrier
safety program. That bill was not enacted. (Mead, 1999, p. 1)
This testimony also stated that the OIG favored separating the motor carrier
safety mission from FHWA’s development and financing mission. He cited outcome
and output indicators of weak enforcement of motor carrier safety regulations in
FHWA.
Over 90 percent of transportation deaths involve motor vehicles.
. . . One out of every eight traffic fatalities involved large trucks.
. . . Truck-related fatalities have increased 20 percent since 1992.
Between 1995 and 1998, the number of motor carriers increased
by over 35 percent. OMC’s review of them, however, declined by
30 percent. During this same time frame, 846 carriers were
123
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
subject to multiple enforcement actions. Of these, 127 carriers
had 3 or more enforcement actions, and 117 carriers had multiple
violations of the same, significant safety regulations. Only 17 of
those carriers were issued out-of-service orders, with penalties
averaging $2,500. In FY 1998, enforcement actions were
processed on only 11 percent of the 29 most violated safety
regulations identified by OMC’s safety investigators.. . . In FY
1995, OMC enforced only 11 percent of driver log violations it
identified. In 1998, that number fell to 8 percent. (Mead, 1999,
pp. 2-4)
The first of the five findings in Section 3 of law was that “the current rate,
number, and severity of crashes involving motor carriers in the United States are
unacceptable.” The seventh finding in Section 3 was that “meaningful measures to
improve safety must be implemented expeditiously to prevent increases in motor
carrier crashes, injuries, and fatalities.” Also, the two purposes of the Motor Carrier
Safety Improvement Act were to establish FMCSA and
. . . (2) to reduce the number and severity of large-truck involved
crashes through more commercial motor vehicle and operator
inspections and motor carrier compliance reviews, stronger
enforcement measures against violators, expedited completion of
rulemaking proceedings, scientifically sound research, and
effective commercial driver’s license testing, record keeping and
sanctions. (Public Law 106-159 § 4)
To this end, the Act required a motor carrier safety strategy and specified the
contents of the strategy, beginning with measurable goals and submission of the
strategy and annual plan with the President’s budget submission. It also prescribed
details of what was to be included in four performance agreements and how the
Secretary was to monitor performance. The performance agreements were to be
between the Secretary and the FMCSA Administrator, the Administrator and the
124
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Deputy Administrator, the Administrator and the Chief Safety Officer, and the
Administrator and the regulatory ombudsman.
The following sections discuss the institutional, organizational, technical, and
resource issues that FMCSA faces as it implements the Motor Carrier Safety
Improvement Act.
Institutional Conditions in FMCSA
FMCSA’s performance plan for Fiscal Year 2001 described its service
delivery mechanisms as regulation and enforcement supported by research and
development and data collection and analysis (Federal Motor Carrier Safety
Administration, February 2000, p. 5.3).
FMCSA is dependent on states to implement the Motor Carrier Safety
Improvement Act. When the Secretary of Transportation issued a press release to
announce the President’s signing of the act, he included “the safety groups, organized
labor and trucking industry who worked so hard to enact one of the most
comprehensive measures ever taken to improve highway safety” (Slater, December
1999. p. 1). The two provisions that he singled out for mention were more funding
for states to conduct more inspections of vehicles, drivers and carriers and funding
for a major study of causes of crashes as well as a new system for collecting crash
data from states.
FMCSA’s Fiscal Year 2001 Performance Plan states that, to accomplish its
mission, it “will work closely in partnership with federal, state, and local
125
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
enforcement agencies, the motor carrier industry, highway safety organizations, and
individual citizens” (Federal Motor Carrier Safety Administration, February 2000,
p. 5.3). For example, one of the key programs cited as directly contributing to the
agency’s safety goal is the Performance Registration Information and Systems
Management (PRISM) program.
Funds will be used as grants to recruit new states into the program
and to assist States that are already participating. States will use
the funds for data processing and programming services,
equipment purchases, and personnel and training costs associated
with full deployment of the program. Funds will also support
continued development, implementation, database programming
and maintenance, staff support, training, and improvements to
FMCSA information systems to support the program. (Federal
Motor Carrier Safety Administration, February 2000, available at
http//www.fincsa.gov/perfplan)
The Intermodal Surface Transportation Efficiency Act of 1991 required development
of PRISM as a pilot program to explore the potential of using the commercial vehicle
registration process as a safety tool to address the problems of a growing and diverse
motor carrier industry (GAO, March 2,2000, p. 5).
This dependence on state actions to accomplish its safety goals is made
problematic by lack of state responsiveness. GAO found weaknesses in state
reporting of crashes involving large trucks due to lack of a legal requirement to do
so, dependence on local jurisdictions to provide the data, and insufficient training or
incentives for states to report complete, timely data (GAO, June 1999, p. 11).
126
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Partnerships between the motor carrier industry and FMCSA’s predecessor
organization have also proven problematic, DOT’s Office of Inspector General
testified that:
The decline in compliance reviews and in strong enforcement
actions can be explained, in part, by the fact that OMC [Office of
Motor Carrier Safety] shifted its emphasis from enforcement and
compliance to a more collaborative partnership approach with the
motor carrier industry. This is a good approach for carriers that
have safety as a top priority, but it has gone too far. It does not
work with firms that persist in violating safety rules and that do
not promptly take sustained corrective action. In replying to our
report, FHWA acknowledge “the pendulum has sung too far
towards education/outreach and now must move toward stronger
enforcement, particularly for repeat offenders.” (Mead, 1999, p. 4)
Organizational Conditions in FMCSA
Approximately 700 employees were transferred from FHWA to FMCSA
when it was established. The total budget request for Fiscal Year 2001 is $279
million, 63 percent of which is for grants to states.
The director of FMCSA’s predecessor office was removed from his position
after the OIG found that he had improperly solicited lobbying efforts from the
regulated industry to block legislative consideration of motor carrier safety reform.
Earlier OIG evaluations had found that, of the 73 percent of the Office of Motor
Carrier safety inspectors who responded to an OIG survey, almost half rated the
enforcement program as “poor to fair” and 86 percent of the respondents called for
stronger enforcement actions (Mead, 1999, p. 1).
127
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
After the Motor Carrier Safety Improvement Act became law, Congressman
Frank Wolf was asked who might be appointed to lead the new organization. He was
quoted as saying: “I hope they appoint someone from the enforcement area who is
honest with no economic ties to the trucking industry” (Skrzycki, 1999, p. 1).
One of the findings of the Motor Carrier Safety Improvement Act was that
“additional safety inspectors and inspection facilities are needed in international
border areas to ensure that commercial motor vehicles, drivers, and carriers comply
with United States safety standards” (Public Law 106-159 § 3, paragraph 5). In
Section 304, the law required a safety strategy from the newly created Motor Carrier
Safety Administration to contain measurable goals, including sufficient federal and
state safety inspectors at international border areas.
Technical Conditions in FMCSA
The limited quantity and questionable quality of data available to FMCSA
has been the subject of a number of GAO evaluations. A GAO spokesperson
testified in oversight hearings that:
While many actions outside OMCHS’ authority influence truck
safety, OMCHS has undertaken a number of activities to improve
truck safety, such as identifying high-risk carriers for safety
improvement and educating car drivers about how to share the
road with large trucks. However, the effectiveness of these
activities is limited by (1) data that are incomplete, inaccurate, or
untimely; (2) the length of time it will take to complete several
activities; and (3) the unknown effect of OMCHS’ campaign to
educate car drivers about the limitation of large trucks. For
example, OMCHS’ effort to identify high-risk carriers for safety
improvements depends in part on having complete data on the
128
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
number of crashes experienced by carriers. However, OMCHS
estimated that about 38 percent of all crashes and 30 percent of
the fatal crashes involving large trucks were not reported to
OMCHS in 1997. (June 1999, p. 13)
GAO also found that: “In a pilot program to implement performance based plans, 5
of 13 pilot states reported that they lack sufficient or timely data to accurately
identify areas that need improvement” (March 1999, pp. 8-9).
This lack of adequate data had inhibited studies of causal factors in traffic
accidents involving large trucks. In response to this problem, the Motor Carrier
Safety Improvement Act authorized additional funding for information systems and
limited some funding to specific activities, notably a study of motor vehicle crash
causation (Public Law 106-159 § 225).
Resource Conditions in FMCSA
Given the newness of FMCSA and the detailed statutory guidance on how to
implement reformed motor carrier safety activities, the question of resources for
implementing GPRA cannot be answered at present. Its predecessor apparently
lacked many of the right resources to manage safety results, and it remains to be seen
if FMCSA will justify and acquire the needed resources. While the new law
expanded authorizations for funds, the necessary justifications for the appropriation
process remain to be developed. GAO found that the draft Safety Action Plan
FMCSA prepared in response to the law did not “determine whether it can
129
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
reasonably expect to accomplish all the actions in its draft plan with its expected
budgetary and human resources” (March 2000, p. 1).
Measurement Informations Uses in FMCSA
As it implements the Motor Carrier Safety Improvement Act, the Motor
Carrier Safety Administration will be required to use safety performance information
for many purposes. These include:
■ Identifying gaps between performance goals and results;
■ Determining the public safety risk factors associated with commercial
motor operations;
* Demonstrating accountability for the results of its activities; and
■ Providing public information to influence social behavior among the
types of uses implied in the law’s provisions.
Summary and Conclusions on Highway Safety Contextual Conditions
In slightly more than 100 years, the federal role in promoting highway safety
retained data collection and analysis as it expanded to include research and
development, regulation, and sponsorship of community-level demonstration projects
for changing behavior. With enactment of the Intermodal Surface Transportation
Act, the Transportation Equity Act for the 21s t Century, and the Motor Carrier Safety
Improvement Act, federal policies for highway safety expanded to include
intermodal planning, collaborative approaches to improve social aspects of the
130
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
transportation system, and performance based management of government actions
both at the federal and state level.
The evolution of federal highway policy and the changing roles of the
operating administrations were products of changes in how to think about surface
transportation. In his research about policy changes, John W. Kingdon said:
The emergence of a new category is a signal public policy event.
When people start thinking of transportation or energy, for
instance, instead of their separate components classified into other
categories, entirely new definitions of problems and
conceptualizations of solution come into play. In important
respects, the categories define our ways of looking at the
problems. (1995, p. 113).
This institutional responsiveness to continued learning about highway safety
problems was coupled with organizational leadership to integrate planning and to use
performance based management. The Safe Communities initiative and NHTSA’s
lessons learned as a GPRA pilot project organization helped to frame the goals and
strategies for continued highway safety improvements. NHTSA’s epidemiological
approach also reinforced a system level view of both the social and technological
aspects of highway safety by coupling technical and social analysis of the problem.
When the Congress responded to problems with unacceptable safety results
involving large trucks, the organizational location of the Motor Carrier Safety Office
as a unit in FHWA was identified as a barrier to improved performance. Repeated
inattention to data shortcomings and to behavioral aspects of truck safety risks led to
the separation of the Motor Carrier Safety Office from FHWA. In the face of debate
131
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
about whether it should become part of NHTSA, the decision was to strengthen the
focus on safety by making it a modal administration, parallel to FHWA and NHTSA.
An evolving policy framework that continues to change and makes safety
performance feedback a priority links these three separate organizations. They are
also linked by data collection and analysis systems as well as overlapping networks
of stakeholders with historical ties to surface transportation programs.
There were three separate operating administrations implementing
performance based management of highway safety outcomes. They were all
dependent on data bases and analysis managed by NHTSA’s National Center for
Statistics. This mutual dependence linked them in collaborations to gain technical
knowledge about causal factors associated with risks in the highway system.
All three of these highway operating administrations used grants. These
grants covered activities including highway construction and maintenance standards,
to demonstration of promising approaches for reducing such risky behaviors as not
wearing seat belts, and to improve data collection to support analysis of truck related
accidents. NHTSA and FMCSA also used regulations to improve highway safety
performance, but both were dependent on states to implement controls and provide
data for analyzing risks and results.
In its pilot project for GPRA implementation, NHTSA developed the strategy
of using intermediate performance measures to identify the actions that contribute to
132
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
outcomes for which the organization has no direct control. All three highway safety
organizations subsequently used that strategy.
NHTSA also led the other two organizations in outreach to and education of
stakeholder groups, particularly for grant programs aimed at influencing human
behavior. FHWA, with its expertise in design and construction of roads, depended
on NHTSA for direction of behaviorally-related grants.
FMCSA was established to improve the organizational structure that was an
obstacle to improving motor carrier safety regulation in FHWA. Regulation of safety
in motor carrier and bus operations was dependent on data that was not complete,
and the statute that established FMCSA provided both financial resources and action
priorities for improving the data. The newness of the FMCSA organizational
structure and performance based direction from the Congress makes judgments
tentative about obstacles and strategies for improving the safety of commercial
vehicle operations.
Internally, DOT executives and staff used information about highway safety
results in many ways and at multiple levels. The Secretary and Deputy Secretary
used performance feedback information to focus stakeholder attention, to monitor
intermediate progress toward goals, to facilitate public discussions and education on
highway safety practices, and to interpret results reports. FHWA and NHTSA
coordinated efforts to use information about results from past highway safety
strategies to target areas for improvement in future performance.
133
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Externally, performance information was used to support policy reform and to
expand accountability for safety performance results to states. TEA-21 established
performance based grants to states, and information about motor carrier safety results
influenced the Congress’s decision to reorganize those responsibilities within DOT.
The Congress specified a performance based approach to achieving safety goals in
regulation of commercial truck and bus operations.
These properties of the contextual conditions associated with using
performance information about highway safety performance to improve outcomes
and the uses of performance information are summarized below.
Properties of Highway Safety Contextual Conditions
Institutional Contextual Conditions
► Constitutional limitations on the federal role in transportation created
federal dependence on states for implementation of grants as policy tools.
► The federal role was based on a long history of collecting data to monitor
safety on roads.
► Diverse stakeholder groups with competing value sets participated in
highway safety activities.
► Partnerships were used to build networks of individuals and organizations
with common interests in improving highway safety outcomes.
► Federal organizations had diverse policy tools for both technical and
behavioral sources of safety risk.
134
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
► Limited legal authority for internal reorganization of safety functions
required Congressionally directed reorganization to increase visibility of
commercial truck and bus operational safety.
► Public participation was used to facilitate consensus building on goals
and implementation strategies.
Organizational Contextual Conditions
► Three semi-autonomous operating administrations had j oint
responsibility for safety outcomes.
► Infrastructure and regulatory responsibilities were divided among
separate organizations.
► Manufacturing and commercial transport regulatory responsibilities were
divided among separate organizations.
► Grant programs were the responsibilities in all three organizations.
► Collaboration among operating administrations took advantage of
complementary areas of expertise.
► All three organizations shared central data systems and analysis
capabilities.
Technical Contextual Conditions
► Highway transportation posed the highest level of safety risk in the
transportation system.
135
with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
► An epidemiological framework guided analysis and learning over time.
► Programs for highway construction and technology improvements for
mobility goal provided secondary influence on safety outcomes.
Resource Contextual Conditions
► Multi-discipline collaboration among organizations shared data and
analytical resources necessary for producing and using performance
information.
► There were limited sources of data about causes of truck and bus
accidents.
Uses of Performance Information
► Learn about risk factors for highway accidents.
► Evaluate relationships of programs in highway safety outcomes.
► Monitor changes over time.
► Justify budget requests and allocations.
► Initiate internal management improvements.
► Influence external policy reforms.
► Public information and education.
► Communicate the value of highway safety programs to the public.
The next chapter explores the institutional, organizational, technical, and
resource conditions for the Federal Aviation Administration’s efforts to improve
136
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
aviation safety outcomes. What were the contextual conditions associated with using
performance information to improve aviation safety outcomes? What strategies were
used to facilitate using performance information to improve aviation system safety
outcomes?
137
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
CHAPTER 6
MANAGEMENT TO IMPROVE AVIATION SAFETY OUTCOMES
The Department of Transportation's (DOT) performance goals include four
aviation safety indicators that were developed and monitored by the Federal Aviation
Administration. These indicators are: air carrier fatal accident rates (accidents with
fatalities per 100,000 flight hours), numbers of general aviation accidents with
fatalities, numbers of runway incursions, and rates of air traffic operational errors.
Air carriers are scheduled commercial passenger service providers, and general
aviation includes all other types of aircraft and pilots, including air taxis. The air
carrier and general aviation indicators are directly related to accidents and fatalities,
but the runway incursions and air traffic operational error rates are preventative and
track risk indicators for situations with the potential to result in accidents.
DOTS’s Fiscal Year 1999 Performance Plan noted that “aviation accidents
overall have caused about 1,000 deaths a year in recent years, with the majority of
these in General Aviation” (DOT, March 2000, p. 21). General aviation includes
“public, private and corporate aircraft [that] provide a wide range of services [such
as] crop dusting, fire fighting, law enforcement, news coverage, sight seeing,
138
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
industrial work, and corporate transportation in addition to personal and recreational
flying” (DOT, March 2000, p. 21).
In 1999, there were 24 fatalities from seven air carrier accidents. This
resulted in an accident rate of .040 accidents per 100,000 flight hours; it missed the
DOT performance goal of .034 accidents per 100,000 flight hours. The long-range
goal, set by the White House Commission on Aviation Safety and Security in 1997,
was for FAA to reduce this fatal accident rate by 80 percent by the year 2007. In
reporting the preliminary results for 1999, the DOT report noted that:
FAA and its partners need to focus their efforts on those causal
factors that contribute to the majority of fatal accidents.. . . While
having little immediate impact on fatal accidents rates year-to-
year, identifying and implementing corrective actions in these
areas will positively im pact the fatal accident rate in the future.
(DOT, March 2000, p. 19)
In 1999, FAA had not set a goal for the number of fatal general aviation
accidents but noted in the DOT Performance Report that 354 such accidents had
occurred. In 2000, the goal for 379 accidents was then justified as appropriate
despite the fact that the target was a larger number than the previous year’s actual
result:
Based on preliminary data, in 1999, there were 354 general
aviation fatal accidents, 29 fewer than in 1998. Since 1988, there
has been a gradual trend downward in the number of general
aviation accidents although, on a year-to-year basis, progress has
not been smooth. (DOT, March 2000, p. 21)
The goal for runway incursions in 1999 was 270 incursions, but preliminary
data indicated that the actual was 322, slightly less than the 325 that occurred in
139
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
1998. FAA reported: “This was the first year runway incursions have not increased
since 1993” (DOT, March 2000, p. 23). Runway incursions occur when aircraft,
ground vehicles or people occupy or cross a runway in active use for takeoffs and
landings.
FAA explained the missed goal as a product of increases in airport activity:
Growth in airport operations has increased an average of 3.5
percent per year from 1996 through 1999. With increased
operations, the risk of incursions increases. Runway incursions
are most likely to occur at complex, high volume airports. These
airports typically have multiple parallel or intersecting runways;
multiple taxiway and runway intersections; complex traffic
patterns; and the need for vehicular and aircraft traffic to cross
active runways. (DOT, March 2000, p. 23)
FAA also pointed out that the air crash with the largest loss of life in history occurred
on the island of Tenerife in the Atlantic Ocean. It occurred when a passenger plane
that was taking off on a runway collided with another passenger plane that was using
the same runway.
Air traffic operational errors result from aircraft passing closer to together
than is allowed by separation standards to keep aircraft a safe distance apart. The
performance goal for 1999 was .49 per 100,000 activities. The preliminary data
indicated that it was not met. Actual operational errors totaled .57 per 100,000
activities, missing the goal.
FAA held a Runway Safety Summit in June 2000 in response to missing the
operational error and incursion goals. “We are still working with the industry to
generate new ideas—training, procedures, technologies—until we find some things
140
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
that bring runway incursions down” (Dr. Charles Dennis, personal communication,
October 5,2000). Some of the people working on improving this performance
question whether these are the right indicators.
The goal for deviations (sudden action to avoid near collision) was .099 per
100,000 activities, but the actual result was .18 per 100,000 activities. FAA
developed a Quality Assurance Review process to identify and correct controller
performance deficiencies linked to corrective training. Also, FAA noted that
corrections in data reporting since 1997 may be causing an apparent increase. Data
interpretation procedures were improved at that time.
Table 9 presents DOT’s Fiscal Year 1999 Performance Report results for
aviation safety. Performance results are reported with annual rather than fiscal year
data to avoid disrupting well-established data reporting systems. The DOT
Performance report noted that changing data reporting to a fiscal year basis would
make past-year comparisons difficult, add confusion, and provide little benefit to
program management.
141
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
TABLE 9
DOT AVIATION SAFETY PERFORMANCE GOALS AND
PRELIMINARY RESULTS, 1999
DOT Aviation Safety Performance Goals 1998
Actual
1999
Goal
1999
Actual
Fatal aviation accidents of U.S. commercial air
carriers per 100,000 flight hours
0 .034 .040
Number of fatal general aviation accidents 383 No Goal 354
Number of runway incursions 325 270 322
Operational errors per 100,000 activities Unstated .496 .57
Deviations per 100,000 activities Unstated .099 .18
Source: U.S. Department of Transportation (2000, March). 1999 performance report 2001
performance plan U .S. Department of Transportation, p. 10, available at http://www.dot.gov.
Table 10 summarizes the actual obligations for direct safety programs, and
Table 11 summarizes the FAA budget request for Fiscal Year 2001 estimated
obligations for the program activities that contribute to the aviation safety goals. The
budget request data shows that the aviation safety and mobility goals are
interdependent in that all of the operations activities, airports activities, facilities and
equipment activities, and most of the research, development, and engineering
activities are either split between mobility and safety goals or noted as making
significant safety goal contributions.
142
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
TABLE 10
ACTUAL OBLIGATIONS FOR DIRECT AVIATION SAFETY
PROGRAMS, FISCAL YEAR 1999
Programs Activities FY 1999 Obligations
Operations Regulations &
Certification
Commercial Space
$625 million
Facilities & Equipment Safety-related Activities $193 million
Research Aircraft Safety
Technology
$35 million
Human Factors &
Aviation Medicine
$25 million
Airport Grants Safety-related Activities $76 million
FAA Total $954 million
Source: U.S. Department of Transportation (2000, March). 1999performance report 2001
performance plan U .S. Department of Transportation, p. 46, available at http://www.dot.gov.
143
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
TABLE 11
ESTIMATED OBLIGATIONS FOR AVIATION SAFETY PROGRAM
ACTIVITIES, FISCAL YEAR 2001
Activities Safety Goal Mobility Goal
Operations $812 m illion $5,697 m illion
o Air traffic services o Secondary influence o $5,210 million
o Regulation & Certification o $692 million o Secondary influence
o Airports o Secondary influence
o Research & acquisitions o Secondary influence o $197 million
o Commercial space
transportation
o $13 million o Secondary influence
o Staff offices o $38 million o $290 million
Grants-in-aid Airports $ 69 million $1,606 million
Facilities & Equipment $255 million $1,987 million
o Engineering, development,
test and evaluation
o $38 million o $571 million
o Procurement &
modernization of Air
Traffic Control
o $154 million o $865 million
o Procurement &
modernization non-Air
Traffic Control
o $30 million o $21 million
o Personnel & related
expenses
o $33 million o $264 million
Research, Engineering, and
Development
$74 million $53 million
o System development and
infrastructure
o Secondary influence o $25 million
o Capacity and air traffic
mgmt technology
o Secondary influence
o Weather o Secondary influence o $28 million
o Aircraft safety technology o $49 million
o System security
technology
o Secondary influence
o Human factors and
aviation medicine
o $25 million
TOTAL $1,141 million $10,484 million
Source: U.S. Department of Transportation (2000, March). 1999 performance report 2001
performance plan U .S. Department of Transportation, pp. 11-3-11-4, available at
http://www.dot.gov.
144
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
This chapter explores contextual conditions that influence the Federal
Aviation Administration’s ability to use performance information to improve
aviation safety outcomes. First, a background section describes the evolution of
federal aviation safety policies to provide the context for contemporary events.
Then, four categories of conditions that affect the use of performance information are
discussed in relation to implementation of GPRA. These categories are institutional
conditions that link FAA to its external environment, organizational conditions
associated with FAA’s structure and internal environment, technical conditions
associated with the four goals and indicators used to monitor results, and resource
conditions associated with funding and workforce capacity to implement results
oriented management for aviation safety. The actions taken to overcome obstacles to
using performance information to reduce aviation safety risk are also described in
relation to these four categories of conditions.
Background
National aviation safety policies have evolved through changes related to two
phenomena in the last century—technological advances and expansion of the
economy.
Commercial aviation sprang to life just after World War I, when
airmen flew rickety biplanes and navigated by following railroad
tracks.. . . Commercial aviation has benefited enormously from
policies of government. Government actions brought forth the
first air carriers in both Europe and the United States. Major air
forces, including Germany’s Luftwaffe, played pathbreaking roles
in midwifing the jet engine. The U.S. Air Force brought forth the
145
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
jet airliner, and lay the groundwork for today’s wide-bodies.. . .
Governments also promoted air safety by taking responsibility for
air traffic control. (Heppenheimer, 1995, p. 1)
By 1926, the emerging aviation industry helped establish the regulatory foundation
for assuring the safety of commercial aviation.
The Air Commerce Act of May 20,1926 was passed at the urging
of the aviation industry, whose leaders believed the airplane could
not reach its full commercial potential without Federal action to
improve and maintain safety standards. The Act charged the
Secretary of Commerce with fostering air commerce, issuing and
enforcing air traffic rules, licensing pilots, certificating aircraft,
establishing airways, and operating and maintaining aids to air
navigation. (Federal Aviation Administration, available at
http://www.faa.gov.history.htm)
The Bureau of Air Commerce in the Department of Commerce became the Civil
Aeronautics Board (CAB) in 1938. The CAB was an independent agency with
authorities to regulate airline fares and to determine routes that air carriers would
serve in addition to the safety regulation functions that already existed. Two years
later, the Civil Aeronautics Administration (CAA) was established to perform
certification, safety enforcement, and airway development functions. Rulemaking,
accident investigation, and economic regulation remained with the CAB (Federal
Aviation Administration, available at http://www.faa.gov.history.htm).
By 1958, commercial flights were common, and the limitations of air traffic
control procedures became a matter of concern. The technology supporting air traffic
control consisted of blackboards, telephones and teletypes, and radio transmissions
from pilots about their courses and airspeeds as they passed waypoints. As the skies
146
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
became more crowded with high speed jets, the Civil Aeronautics Administration
proposed development of radar technology to allow positive ground control—air
controllers would follow every move of airplanes and issue directives to pilots. The
military and the commercial air carrier industry did not support the change at first
(Heppenheimer, 1995, p. 180).
A crash of two airliners over the Grand Canyon in 1955 and the crash of a
military plane and airliner near Las Vegas in 1956 led the CAA to formally sponsor
development of technology for positive ground control using radar to track air traffic.
“The advent of positive control meant that aircraft would fly on instruments even in
clear weather and would be barred from certain airways unless they had the necessary
equipment in the cockpit” (Heppenheimer, 1995, p. 183). In 1958, the Federal
Aviation Act transferred rulemaking functions of the CAA to a new Federal Aviation
Agency that reported directly to the President (Heppenheimer, 1995, p. 183).
The CAB retained its accident investigation and economic regulation
functions, and it remained in the Department of Commerce until the creation of the
Department of Transportation in 1967. When the Department of Transportation was
established, the CAB became the National Transportation Safety Board, and the
economic regulation functions of CAB ceased to exist when the Airline Deregulation
Act of 1978 was enacted (Federal Aviation Administration, available at
http://www.faa.gov.history.htm).
147
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The Federal Aviation Agency became the Federal Aviation Administration
within the Department of Transportation in 1967. During the 1960’s, FAA’s
functions had been expanded. In response to hijacking incidents, FAA was assigned
aviation security functions. FAA was also authorized to prescribe aircraft noise
standards (Federal Aviation Administration, available at
http://www.faa.gov.history.htm).
In 1970, FAA was assigned responsibility to implement a new airport aid
program and reduce infrastructure problems in the airport and airway system. This
program was to be funded by an Aviation Trust Fund “made up of aviation-related
excise taxes [including] increases in the gasoline tax on general aviation, and
increase in the passenger ticket tax for domestic flights, a new tax on international
commercial passengers, a new tax on air freight waybills, and a new annual aircraft
registration tax” (GAO, June 1999, p. 19). “The Trust Fund was created to finance
aviation infrastructure, some administrative expenses, research and development, and
the maintenance and operation of the National Airspace System” (GAO, June 1999,
p. 3).
Both the Administration and the Congress established study commissions to
recommend reforms in the federal role in aviation in 1995 and 1996. Section 274 of
FAA’s Appropriation Act for Fiscal Year 1996 established the National Civil
Aviation Review Commission to perform an independent assessment of FAA
148
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
funding and safety issues. Executive Order 13015 commissioned the White House
Commission on Aviation Safety and Security in August 1996.
Both commissions published reports with extensive recommendations to
reform organizational structure, institutional relationships, technical approaches to
aviation safety, and financing and accounting procedures for safety related activities.
The findings and recommendations of these commissions will be discussed in each
of the following sections about the contextual conditions associated with use of
performance information to manage for aviation safety results. Four categories of
contextual conditions will be explored.
1. Institutional conditions that connect the organization to its external
environment;
2. Organizational conditions that exist internally;
3. Technical conditions that affect analytical ability to learn about aviation
accident causal factors and potential interventions to reduce risks; and
4. Resource conditions that include financial and human resources to
implement safety results oriented management.
Institutional Conditions
The federal aviation safety policy structure is highly complex. A vast
network of industries, governmental activities at every level from international to
local, professional and trade associations, labor unions, for research and development
are involved. As the history of national aviation policy indicated, development of the
149
with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
aviation industry was directly related to development of the federal government’s
role in assuring the safety of operations. Also, the history of federal aviation policy
suggests that air disasters have typically resulted in statutory and administrative
changes to strengthen policy and to demonstrate responsiveness to public concerns.
Historically, airway crashes have resulted in national attention to air safety
policies. For example, 1996 was a bad year for air safety—110 people died in a
Valujet passenger jet crash and 230 in the TWA Flight 800 crash. Subsequently, two
national commissions were established to review various aspects of FAA activities in
aviation safety. The President convened the White House Commission on Aviation
Safety and Security under the Vice President’s leadership. The Congress established
a national Civil Aviation Review Commission with two task forces—The Aviation
Funding Task Force and The Aviation Safety Task Force.
The White House Commission included 20 members, a staff of 38, and was
further supported by 56 individuals who were named in the conclusion of the report.
The backgrounds and expertise of these individuals ranged from research in all
aspects of aviation-related technology, to law enforcement, military air operations,
aviation technology manufacture, law, relatives of victims of fatal aviation accidents,
and politicians. The staff came primarily from DOT and its modal administrations;
the Federal Bureau of Investigation and the Bureau of Firearms, Tobacco, and
Alcohol; the Customs Service; and the National Performance Review. External
support came from such organizations as the U.S. Representative to the International
150
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Civil Aviation Organization, the National Transportation Safety Board, and a variety
of business organizations and university researchers (White House Commission,
1997, pp. 42-53).
The first of the White House Commission recommendations was:
“Government and industry should establish a national goal to reduce the aviation
fatal accident rate by a factor of five within ten years and conduct safety research to
support that goal” (1997, p. 8). FAA adopted this goal (a reduction of fatal
commercial aviation accidents in the ten-year period from 1996 to 2007) into
strategic and performance planning for safety; there had previously been no target for
safety. It is the source of the Fiscal Year 1999 DOT goal of .034 fatal aviation
accidents per 100,000 flight hours for commercial aviation. It is also the source of a
goal in NASA’s strategic plan for research for reducing the fatal accident rate by a
factor of 10 within 20 years (DOT, February 1998, p. i).
The National Civil Aviation Review Commission (NCARC) had 21 members
from organizations as diverse as the American Association of Airport Executives, the
Boeing Company, United Parcel Service, Northwest Airlines, the Brookings
Institution, and an aviation trade journal. The 19 members of the commission staff
came from Congressional offices, FAA, NASA, the MITRE Corporation, and DOT.
Written testimony was received from numerous associations, including state aviation
officials, aircraft owners, pilots, business aircraft operators; labor unions; airlines;
151
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
airports; port authorities; and business travel consultants (National Civil Aviation
Review Commission, May 1997, p. 1).
The recommendations resulting from NCARC included re-engineering
regulation and certification functions in the aviation industry by using partnerships
for meeting goals.
The premise behind these partnerships is that government can set
goals, and then work with industry in the most effective way to
achieve them. Partnership does not mean that government gives
up its authorities or responsibilities. Not all industry members are
willing to be partners. In those cases, government must use it full
authority to enforce the law. But, through partnerships,
government works with industry to find better ways to achieve its
goals, seeking to replace confrontation with cooperation. Such
partnerships hold tremendous promise for improving aviation
safety and security. A shift away from prescriptive regulations
will allow companies to take advantage of incentives and reach
goals more quickly. (White House Commission, 1997, p. 4)
This recommendation led to FAA’s Safer Skies project that was announced in April
1998.
In partnership with industry, Safer Skies will use the latest
technology to help analyze U.S. and global data to find the root
causes of accidents and determine the best actions to break the
chain of events that lead to accidents. [It is] a five-year plan to
focus FAA resources on the accident prevention steps that hold
the most potential. (Federal Aviation Administration, 1998, p. 1)
This program established joint efforts between the industry and FAA to further
analyze data about six historically significant causes of fatal accidents. They were
controlled flight into terrain, loss of control, uncontained engine failures, runway
152
R eproduced with perm ission o f the copyright owner. Further reproduction prohibited without perm ission.
incursions, approach and landing, and weather (Federal Aviation Administration,
1998, p. 2).
FAA has two strategies in addition to regulation for improving the level of
aviation safety. It works with the industry to identify recurrent causes of accidents
and implement interventions, and it encourages voluntary information sharing.
“Regulation to establish a baseline, then partnership to go beyond it and be even
safer” (Dr. Charles Dennis, personal communication, October 5, 2000).
At the request of the Subcommittee on Aviation of the House Committee on
Transportation and Infrastructure, GAO evaluated actions taken by FAA in the Safer
Skies initiative. The GAO report (GAO/RCED-OO-111) on this evaluation found that
the initiative held promise for improving aviation safety and that FAA had made
progress in selecting and implementing interventions to improve aviation safety. The
GAO report also discussed the lack of ways to evaluate the effectiveness of the
interventions underway and weaknesses in coordination, including assuring partners
of a commitment to fund and to complete the initiatives and lack of prioritization to
deal with funding limitations that could occur. The report also discusses the lack of
agreement between DOT and FAA on how to measure the progress in achieving the
accident reduction goal for commercial aviation.
In addition to initiatives growing out of special commission reports, FAA’s
Assistant Administrator for System Safety also plays a role in facilitating information
exchange among the aviation industry and governmental analysts researching ways to
153
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
avoid aviation accidents. FAA led an international initiative for data exchange
within the industry through what became the Global Aviation Information Network
(GAIN). One GAIN working group is developing ways for reducing impediments to
data sharing that result from fear of organizational recriminations, regulatory actions,
civil litigation, criminal sanctions, or risk of public disclosure.
This working group will identify and evaluate barriers that
prevent the collection and sharing of aviation safety information
among various organizations and propose solutions that are
reasonable and effective. They will pursue changes in ICAO
[International Civil Aviation Organization] Annexes to
appropriately protect information on accident/incident prevention
programs. They will propose means to obtain legislation to
protect reports and providers of safety information. They will
promote “ jeopardy-free” reporting procedures and create methods
to obtain organizational commitment to sharing safety
information. (GAIN, available at http:www.gainweb.org)
The senior director of safety at the Air Transport Association is also program
director for GAIN and said that: “Almost all situations [that would be reported
voluntarily reported] are inadvertent human error” (Baumgamer, 1999, p. 1). He
cited the policy statement issued by FAA in December 1998. It stated that:
“Voluntarily shared safety information would not be used against an airline unless it
constituted ‘criminal wrong-doing'” as an example of progress facilitating the type of
exchange needed.
FAA’s Office of System Safety has explored a body of knowledge dealing
with public perceptions of risk as helpful in explaining how public perceptions of
risk affect acceptance of proposed policy reforms (Fischoff, 1994; Gowda, 1997).
154
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
These perceptions appear to arise from heuristics and biases in decision making.
Behavioral research suggests that media and political exploitation of public heuristics
and biases have contributed to public acceptance of very different risk regulation in
aviation from what is acceptable in surface transportation. For example, the
President’s and the Congress' commissions on aviation safety demanded a far more
demanding reduction of safety risk for aviation than are contained in DOT’s
performance plans for highway and maritime transportation.
Organizational Conditions
FAA’s total annual budget obligations at the end of Fiscal Year 1999 were
approximately $9.5 billion, and its workforce was 49,548 full-time equivalents
(OPM, available at http://www.opm.gov/feddata/tablel 5). Of the total obligations by
FAA, $954 million were obligated for direct safety programs (DOT, DOT
Performance Report 1999 Performance Plan 2001, March 1999, p. 46). As indicated,
over $1 billion dollars were being requested to perform direct safety activities for
Fiscal Year 2001. (DOT, March 2000,1999 Performance Report 2001 Performance
Plan, pp. II-3 & II-4). The combined total Fiscal Year 2001 budget requested for
safety and mobility activities that were required to accomplish safety performance
goals was approximately $11.5 billion. Most of the Fiscal Year 2001 budget request
was for operations and procurement of facilities and equipment; the only grant
program is for aid to airports.
155
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Air traffic services include air traffic control and operation and maintenance
of the navigation and control infrastructure—the most publicly apparent FAA
functions.
The FAA is a 24 hours/seven days a week service delivery
organization. The FAA controls approximately 20,000 takeoffs
and landings per day and moves over 600 million passengers per
year. This number is expected to reach 1 billion within a decade.
(Garvey, 2000, p. 1)
FAA organizations perform many functions that grew out of national aviation
policy. For example, in addition to air traffic services, FAA regulates and enforces
standards for maintenance and operations of the aircraft fleets and flight crews of
commercial air transportation companies, administers grants for improving airport
safety, provides public information on the safety of the aviation system, and supports
investigations of accidents involving aircraft of all types. FAA also plans and
modernizes the national aviation system infrastructure. In short, the complexity of
aviation policy structure is reflected in FAA’s organizational structure.
At headquarters, FAA has 9 Assistant Administrators and 7 Associate
Administrators who report to the Administrator and lead the activities of 25
headquarters offices. One Assistant Administrator for Region/Center Operations
leads FAA’s 9 field offices and a Support Center; these organizations are headed by
Regional Administrators. Most Associate Administrators lead organizations with
representatives in the nine field offices and with other sub-offices in air traffic
service facilities.
156
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Operations in the field receive policy direction from headquarters offices
headed by Associate Administrators, rather than by regions or Regional
Administrators. This “straight-lining” was done to increase uniformity in the
application of regulations and other programs (Dr. Charles Dennis, personal
communication, October 5,2000). Routine operations for service delivery, however,
are managed in a variety of operating field structures that mirror the variety of
functions that must be performed. For example, Air Traffic Services operations are
managed in such field facilities as Air Route Traffic Control Centers and airport
control towers. Such regulatory functions as Flight Standards operations are
managed in different types of facilities that include Flight Standards District Offices.
At the headquarters level, the Assistant Administrator for System Safety is a
central figure in facilitating and supporting risk management among the numerous
and dissimilar organizations that play roles in aviation safety.
The Office of System Safety reports directly to the Administrator.
Its primary function is to develop and implement improved tools
and processes, including hazard identification, risk assessment,
and risk management tools and processes, to facilitate more
effective use of safety data, both inside and outside the agency, to
help improve aviation safety. It also functions as a coordinator
within the agency for safety issues. (Federal Aviation
Administration, available at http://nasdac.faa.gov/asy)
Planned implementation steps for the initiatives and activities are included in
performance agreement between the Administrator of FAA and the Secretary of
Transportation (Marylou Batt, personal communication, September 9,1999). In
157
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Fiscal Year 1999, DOT tracked 20 strategies for achieving safety performance goals
in FAA, all of which were completed at year-end (DOT, January 2000, p. 1).
GAO attributed FAA’s problems in acquiring new technology to improve the
safety and the mobility of national aviation system, in part, to “the employees’
attitudes and behaviors in the areas of mission focus, accountability, coordination,
and adaptability” as an underlying cause of the organization’s problems (GAO, May
1996, p. 4).
Technical Conditions
An important technical barrier to using performance information to manage
for safety results is the accident rate plateau. FAA’s office of System Safety initiated
the establishment of the Global Aviation Information Network to provide a process
for analysis of normal flight conditions as a means of improving knowledge about
what “safe” aviation means. It also provides feedback about anomalous factors in
aircraft design and air crew practices that did not, but could have, led to accidents
during routine flights.
After declining significantly for about 30 years to a
commendable low rate, the worldwide commercial aviation
fatality accident rate has been stubbornly constant since 1980-85.
Given the projected increase in volume in international aviation
traffic, studies by Boeing forecast that unless the aviation
community resumes its decline from the accident rate “plateau,”
there will be a major hull loss every week to ten days, somewhere
in the world, by the year 2015. The FAA proposed GAIN because
that is an unacceptable result.
The question is how to get off the accident plateau.. . . The
leveling of the accident rate curve, however, suggests that the
158
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
marginal safety returns from these previous ways of improving
safety are diminishing, and that it is necessary to find new ways of
preventing accidents and incidents (Federal Aviation
Administration, May 2000, p. 1)
Another effort to increase data availability about safety risk was managed by
the Associate Administrator for Regulation and Certification. The Aviation Safety
Action Partnership (ASAP) is a voluntary, public-private partnership between FAA,
employee unions, and airlines to protect employees from reprisal when they report
problems. “ASAP’s primary purpose is the capture of important information that
previously went unreported, to bolster flight safety by eliminating human error and
improving man-machine interaction” (Mann, 2000, p. 50).
Along with increasing knowledge about the sources of risk in aviation, FAA’s
Office of System Safety has pursued knowledge about another technical challenge—
how to communicate information about aviation risk. FAA’s Office of System
Safety has explored research results in risk communication to develop appropriate
methods for making data available and for interpreting analyses to provide public
information (R. Davis Balderston, personal communication, July 26,1999). The
technical challenge is to use the extensive and highly technical data collected about
air traffic control and regulatory operations for public communication about aviation
risk. Data formats, informative explanations, and choice of presentation media was
being developed to provide public information about the relative risk of aviation to
highway and waterway safety risks.
159
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
One scholarly explanation is that technical information about aviation safety
risks may not be communicated as effectively as technical information about
highway safety risks.
One of the earliest results in risk perception research was the
discovery that experts and laypeople might agree about the
fatalities that a technology produces in an average year but still
disagree about its degree of “risk.” These disagreements seem to
reflect differences in how “risk” is defined. One of the
differences is that laypeople place greater weight on catastrophic
potential. The size of that potential is a topic for scientific
research (although one where hard estimates are particularly hard
to come by). However, the weight to be given to that potential is
a matter of public policy. (Fischhoff, 1994, p. 8)
Behavioral decision theory’s findings show that people deviate
from the rational choice standard because they use shortcuts when
processing information to arrive a judgments and choices. These
shortcuts, termed heuristics, are sometime efficient in that they
facilitate judgments and decisions without tremendous
information processing costs. However, some heuristics can also
lead to inefficient and suboptimal outcomes when measured
against a standard of expected utility theory, and when these
occur, they are termed biases. (Camerer, 1995; Gowda, 1997,
p. 2)
The availability heuristic states that people “assess the
frequency of a class of probability of and even by the ease with
such instances or occurrences can be brought to mind” (Tversky
& Kahneman, 1974, p. 1127). If people can readily think of
examples of events, they believe that such events are more likely
to occur.. . . The availability heuristic can lead people to demand
the allocation of more regulatory resources toward the prevention
of vivid and available events such as airline crashes, rather than
toward highway safety, even though the latter cost substantially
lives. (Zeckhauser & Viscusi, 1996; Gowda, 1997, p. 4)
Behavioral decision theoretic analysis may thus serve to
illuminate why and how the cycles of policy change identified by
Schlesinger (1986) and Hirschman (1982) occur, and offer an
alternative explanation, grounded in cognitive psychology, to the
sociological explanation offered by Hilgartner and Bosk (1988).
The key to cyclicality may be that policies which exploit
160
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
heuristics and biases get enacted and these inevitably lead to some
level of policy failure. This may be because of the limitations of
the paradigms or ideologies in the face of the complexity or
difficulty of the task in the real world. It is to implement these
paradigms or ideologies that political entrepreneurs exploit
heuristics and biases and to sell their paradigm-based solutions as
guaranteed cures for society’s risk-related ills. To the extent that
members of Congress feel constrained by monitoring [of their
voting records] and reflect the interest groups’ agendas in their
voting, biases will be accentuated in risk regulation. (Gowda,
1997, pp. 17 & 21)
Also, weaknesses in knowledge about the human factors associated with the
numerous occupations that influence aviation safety has inhibited further reduction of
aviation safety risks. To acquire such knowledge, FAA has developed a Human
Factors Plan for research to learn more about human-centered automation, selection
and training, human performance assessment, information management and display,
and bioaerunautics or physiological factors; for example, FAA-sponsored research to
develop criteria forjudging the individual performance of air traffic controllers with
the purpose of developing a computer-based performance measure.
The job of air traffic controller is very complex and potentially
difficult to capture in a criterion development effort. Yet, the goal
here was to develop criterion measures that would provide a
comprehensive picture of controller job performance.. . . More so
than with many jobs, maximum “can-do” performance is very
important in controlling air traffic. There are times on this job
when the most important consideration is maximum
performance—does the controller have the technical skill to keep
aircraft separated under very difficult conditions? Nonetheless,
typical performance over time is also important for this job.
(Borman et al., 1999, p. 1)
161
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Finally, a problem pointed out by FAA’s Assistant Administrator for System
Safety was the fact that measuring the output of the aviation system was easy, but
measuring outcomes was more difficult. Numerous programs, including such
National Airspace Modernization acquisitions as the Air Traffic Operations Safety
System, Free Flight, plus some runway incursion programs in Air Traffic Operations
contributed to meeting FAA’s goals in DOT’s plans. The problem was knowing
which programs contributed what degree of the results that are achieved (Christopher
Hart, personal communication, July 26,1999).
Resource Conditions
While funding for FAA in general is a perennial issue, funding for data
collection, analysis of performance, and reporting results does not appear to be a
problem. Given the extensive partnerships with industry, including the fact that the
GAIN program is entirely industry operated, there is no evidence of financial
resource constraints for performance based management.
There are perceptions of possible funding constraints. GAO’s report on the
Safer Skies initiative discussed the coordination problem FAA had in assuring its
partners that planned interventions would be funded.
Although FAA officials have repeatedly committed to funding
interventions agreed upon by all parties working on the initiative,
skepticism still exists among some participants as to whether this
commitment can or will be honored. This is particularly true in
general aviation. Furthermore, if funding is limited, it remains
unclear what process will be used to reprioritize available
resources to ensure funding for interventions that emerge later but
162
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
have greater potential for reducing the fatal accident rate. (GAO,
GAO/RCED-OG-111, p. 12)
The manager of FAA’s Strategic Planning Branch pointed to a resource
weakness he foresees, although “FAA has a high degree of the right resources for
planning, analyzing, and reporting the performance of the aviation system. The
biggest weakness is evaluation; FAA has virtually no evaluation capability, because
it chooses to devote its resources to other purposes” (Dr. Charles Dennis, personal
communication, July 12, 1999).
Performance Information Uses
Within FAA, measurement of safety related activities is used for every
function from routinely controlling the separation of aircraft to providing public
information about the risks involved in flying. At the operations level, information
about individual air traffic controllers is used to assess performance. Regulatory
planning has been recognized to be based on statistical analysis of risk targets and a
formal evaluation of the benefits and costs of proposed regulation. An abundant
amount of information about the safety of the aviation system is available on line.
Both FAA and the National Transportation Safety Board make extensive information
about accidents and accident investigations publicly available.
163
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Summary: Properties of Contextual Conditions for Use of Aviation Safety
Performance Information To Improve Outcomes
DOT’s performance goals for aviation safety in Fiscal Year 1999 were not
met. Strategies in the Safer Skies initiative (the central source of interventions to
achieve the goal) were changed in an effort to increase the likelihood that future
performance goals would be met. An independent evaluation by GAO found that the
Safer Skies imitative should help improve aviation safety and has made progress in
selecting and implementing interventions. GAO pointed out, however, that FAA had
not developed ways to evaluate the effectiveness of most of the interventions
underway and that coordination could be improved among the joint industry-FAA
participants.
In the period covered by this research, aviation safety was a product of
actions in the industry, in FAA, and among individuals engaged in general aviation.
This fact, plus the span of federal aviation policies and activities into every aspect of
aviation in U.S. airspace made both the institutional and organizational contexts for
FAA’s performance extremely complex.
Stakeholder relationships were complicated by the variety of interests
involved in aviation safety activities. Groups who dealt routinely with FAA included
families of victims of air crashes, passengers who were dissatisfied with airline
services, professional associations for every workforce involved in aviation, and
aviation enthusiasts. Similarly, conflicts arose in priorities among stakeholders with
interests in system efficiency, environmental concerns, and safety.
164
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Policy direction about goals and strategies to improve aviation safety came
from external commissions. For safety regulation, in particular, the industry rather
than FAA was charged with developing solution strategies.
A technical barrier to improving knowledge about safety risks in passenger
transportation led FAA to deal with a stakeholder relationship problem. In order to
improve knowledge about risks in routine, uneventful air carrier operations, FAA had
to work around its regulatory role to facilitate data exchange among air carriers by
assuring them protection from punitive uses of the data.
The inseparability of actions to achieve safety from actions to achieve
mobility goals was one sign of the complexity of laying out discrete performance
strategies for either goal. Mobility in aviation depended on air traffic control, safe
airport runway conditions, safe aircraft, and safe operations on the ground and in the
air. Technology improvements to change the spacing of aircraft, both on the ground
and in flight were under development to increase system efficiency without
increasing safety risks. Similarly, improvements to technical and behavioral factors
to increase mobility were linked to improved safety.
The size and diversity of FAA’s workforce, its geographic dispersion, and its
highly technical functions were also signs of organizational complexity. With
twenty-four-hour-a-day operations that were highly decentralized and a highly
unionized workforce, changes in either technology or procedures were
implementation challenges.
165
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The high level of safety performance already achieved in the aviation system
was another obstacle to continued improvement. The performance plateau was at a
very high level of safety. Improving safety outcomes on the margin of such success
required looking for precursors to accidents that could but did not occur. FAA’s
facilitation of the data exchange discussed above was one strategy to continue
improving a high level of performance. Improvements in technology was another.
FAA received a great deal of political attention and has a large mix of
activities underway and planned to continue reducing safety risks in the aviation
system. The aviation industry (from manufacturers through airlines, pilots, flight
dispatchers and mechanics) was directly involved in many of these activities. While
successful implementation of these activities was aimed reduced aviation risk, it
seemed difficult, at best, to determine which activities contributed to such reductions.
It also may not change political and public reactions to the risks that remain.
Finally, communicating about aviation safety risk was complicated by human
behavior when trying to understand catastrophes. Airplane crashes are catastrophic
events that affect public judgments and political decisions about aviation safety of air
transportation. The overall safety of the transportation system was overshadowed by
public and political reaction to air crashes despite the fact that highway crashes killed
tens of thousands more people annually.
A summary of properties of contextual conditions for FAA in using
performance information to improve aviation safety outcomes is outlined below.
166
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Aviation Safety Contextual Conditions
Institutional Contextual Conditions
► Aviation policy history began with interdependent mobility and safety
responsibilities.
► Aviation safety policies provided a high degree of control to FAA.
► Responsibilities included management of a large, technologically
advanced infrastructure.
► Interdependence of FAA and industry were the basis of partnerships.
► Industry and FAA share responsibility for aviation safety performance
planning and monitoring.
► Public participation was used to facilitate consensus building on goals
and implementation strategies.
Organizational Contextual Conditions
► A single, very large, complex organization had responsibility for
transportation safety.
► The organization had a diverse, geographically dispersed workforce.
► The organization had strong labor unions for the safety workforces.
► The organization had a history of labor-management problems.
► The organization had a history of cultural resistance to change and
innovation.
167
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Technical Contextual Conditions
► Aviation posed a low level of safety risk compared to other components
of the transportation system.
► Risk perception biases influenced public and political interpretation of
safety issues.
► An aviation safety performance plateau made increased the expense and
time required for incremental improvements of safety outcomes.
► Safety and mobility goals were interdependent in the technological
infrastructure.
► Interdependence of safety and mobility goals were recognized in
implementation strategies.
Resource Contextual Conditions
► Aviation safety was pursued in a general environment of constrained
resources and uncertain funding linked to high cost infrastructure
investments.
► Industry sponsored safety information exchange and analysis augmented
government efforts.
Uses of Performance Information
► Air traffic operational control.
► Regulatory agenda setting.
168
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
► Research and acquisition justifications.
► Learning about human factors that affect safety risk.
► Learning about technology applications that affect safety risk.
► Public information and education about aviation safety risks and
practices.
► Budget justification.
► Communicate the value of aviation safety programs to the public.
The next chapter explores the institutional, organizational, technical, and
resource issues that affect the U.S. Coast Guard’s ability to influence safety on the
waterways. What kinds of obstacles were encountered and what strategies were
employed to using performance information to reduce risks in the waterway
component of the transportation system?
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
CHAPTER 7
MANAGEMENT TO IMPROVE WATERWAY SAFETY OUTCOMES
The Department of Transportation's (DOT) performance goals for Fiscal Year
1999 included three safety indicators that were developed and monitored by the
United States Coast Guard (USCG). These included reductions in recreational
boating fatalities, increased percentages of mariners rescued that were reported in
life-threatening danger, and fatalities per 100,000 workers aboard commercial
vessels. Two of these goals were met, and performance for the third was on a good
trend.
USCG has helped reduce fatalities on waterways through three different types
of activities. Recreational boating safety was pursued through education, regulation,
and enforcement activities in cooperation with the states. USCG search and rescue
activities prevented fatalities by rescuing mariners in distress. Marine safety
regulation and enforcement activities prevented fatalities principally by inspecting
maritime vessels and by working with the industry to improve maritime workers’
safety awareness and behavior (Sam Neill, personal communication, October 3,
2000).
170
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Recreational boating was the largest source of waterway fatalities, and over
773 recreational boaters lost their lives in 1999. The goal was to reduce recreational
boating fatalities to 720 or fewer, but the downward trend from the previous year’s
total of 864 fatalities was good.
Maritime worker fatalities have been on a downward trend from 40 to about
30 per 100,000 workers annually since the mid 1990s, and the rate of these fatalities
continued the downward trend, reaching 28 per 100,000 workers in 1999 (DOT,
March 2000, p. 10).
DOT’s Performance Report for 1999 contained a discussion of improvements
in USCG data analysis that led to a revised measure for recreational boating safety
results and search and rescue results. USCG reviewed its Boating Accident Report
Database and revised the measure used in the Fiscal Year 1999 performance plan to
reflect under-reported boating accidents. The original goal had been 720 recreational
boating fatalities; the revised goal was 763; and the preliminary estimate of actual the
number of recreational boating fatalities was 773 (DOT, March 2000, p. 28). USCG
revised this goal based on its estimate that about 6 percent of recreational fatalities
were not reported by state boating law administrators. They found that this revision
would not affect the overall trend in the data but would make it more technically
accurate (Coast Guard, 1999 Report, p. 13).
Also, re-prioritization of which USCG internal goals and measures to report
at the departmental level led to substitution of a passenger vessel safety indicator in
171
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
place of the maritime worker safety indicator reported for Fiscal Year 1999. The
measure of maritime worker safety was retained within USCG’s performance based
management system and will continue to be reported in USCG performance reports
(DOT, March 2000, p. 32). Based on improved data analysis, USCG revised the
indicator and goal for maritime workers’ fatality rate and discontinued reporting it as
a departmental indicator. The rationale for substituting the passenger vessel safety
goal was that: “The new Passenger Vessel Safety goal reflects an area of safety
performance with more broad public impact.” In 1999, there were approximately 39
fatalities per 100,000 vessels. Increasing numbers of such vessels and the increasing
size of their passenger capacity suggested to USCG’s analysts that there could be an
underlying risk, however, that should be reflected in DOT’s performance plans as
well as the USCG’s (DOT, March 2000, p. 34).
The goal for Fiscal Year 1999 was to rescue 93 percent of mariners reported
in imminent danger, and the actual result was that 95 percent of those reported in
danger were rescued. In the future, this indicator will be expanded to better reflect
performance. The percent of all mariners in imminent danger who are rescued will
be reported at the departmental level. The goal for Fiscal Year 2001 will be to rescue
85 percent of all mariners in imminent danger. Though the actual percentage of all
mariner rescues was 88 percent in 1999, the direction of the trend caused the USCG
to retain the previously established goal (DOT, March 2000, p. 30). In the USCG
performance report for 1999, the number of people that were not successfully rescued
172
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
was also reported. “We rescue about 3,800 people in distress each year, but the
number of people we don’t save remains above 500 each year” (Coast Guard, 1999
Report, p. 5).
Table 12 summarizes DOT’s success in attaining waterway safety goals for
Fiscal Year 1999. Table 13 summarizes actual obligations for budget programs and
activities supporting those goals.
TABLE 12
DOT WATERWAY SAFETY PERFORMANCE GOALS AND ACTUAL
RESULTS, FISCAL YEAR 1999
DOT Waterway Safety Measures 1998 Actual 1999 Goal 1999
Actual
Percent of mariners reported in
imminent danger who are rescued
94% 93% 95%
Number high-risk passenger vessel
causalities per 1,000 vessels
New measure
for 2001
New measure
for 2001
39
Fatalities per 100,000 workers
aboard commercial vessels
35 42 original
34 revised
28
Number of recreational boating
fatalities
864 720 original
763 revised
773
estimated
Source: U.S. Department of Transportation (2000, March). 1999performance report 2001
performance plan U.S. Department o f Transportation, pp. 10 & 28-34, available at
http ://www.dot. gov.
173
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
TABLE 13
ACTUAL OBLIGATIONS FOR DIRECT WATERWAY SAFETY,
FISCAL YEAR 1999
Programs Activities
Operations $786 million
Search and Rescue $394 million
Marine Safety $392 million
Acquisition $110 million
Search & Rescue $65 million
Marine Safety $46 million
Research $7 million
Search & Rescue $3 million
Marine Safety $4 million
Boating Safety Grants $59 million
USCG Total $962 million
Source: U.S. Department of Transportation (2000, March). 1999performance report 2001
performance plan U.S. Department o f Transportation, p. 46, available at http://www.dot.gov.
Table 14 summarizes the estimated obligations for waterway safety for Fiscal
Year 2001. It shows that such activities as operation, acquisition, and research about
aids to navigation have secondary influences on waterway safety. While aids to
navigation are primarily established to support mobility, they are also important for
promoting waterway safety. Similarly, the marine safety program primarily supports
the waterway safety goals but has a secondary influence on mobility goals.
This interdependence of waterway safety and mobility programs is similar to
the interdependence of highway safety and mobility programs as well as aviation
safety and mobility programs. Therefore, it must be noted that, given the DOT
commitment to safety as well as mobility in the transportation system, obligation of
funds to programs and activities that promote one goal necessarily influence
174
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
outcomes for the other goal. It is not so apparent, however, to what degree and how
a particular program or activity influences individual goal outcomes. In Chapter 4, it
was noted that the Deputy Secretary observed that a remaining challenge for DOT to
explain relationships among programs and outcomes given the complicated external
influences on transportation safety results.
TABLE 14
ESTIMATED OBLIGATIONS FOR WATERWAY SAFETY PROGRAM
ACTIVITIES, FISCAL YEAR 2001
Programs and Activities Total Safety Goal Mobility Goal
Operating Expenses $ 823 million $ 483 million
Search & Rescue $ 383 million
Aids to Navigation Secondary influence $ 483 million
Marine Safety $ 440 million Secondary influence
Acquisition, Construction &
Improvements
$112 million $ 148 million
Search and Rescue $ 64 million
Aids to navigation Secondary influence $ 148 million
Marine Safety $ 48 million Secondary influence
Retired Pay $194 million $ 121 million
Research, Development, Test &
Evaluation
$ 9 million $ 3 million
Search & Rescue $ 2 million
Aids to navigation Secondary influence $ 3 million
Marine Safety $ 7 million Secondary influence
State Recreational Boating Safety
Programs
$ 64 million
USCG Total $ 1,202 million $ 782 million
Source: U.S. Department of Transportation (2000, March). 1999performance report 2001
performance plan U.S. Department o f Transportation, pp. IE -1-13-2, available at
http://www.dot.gov.
175
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
This chapter discusses the obstacles to using performance information to
improve waterway safety outcomes influenced by USCG as it implemented the
Government Performance and Results Act (GPRA). It also describes the
management strategies this organization used to overcome these obstacles. First, a
background section briefly summarizes the historical evolution of federal policies
and organizational arrangements for waterway safety. The background section also
describes the categories of conditions that potentially affect use of performance
information. The background section is followed by descriptions of the categories of
obstacles and related management strategies used in USCG as the Government
Performance and Results Act (GPRA) was implemented.
Background
The USCG is the oldest of DOT’s modal administrations. It traces its roots to
five formerly distinct federal services that emerged in the early formation of the
federal administrative structure from 1789 through 1884. In 1789, the earliest
predecessor of the contemporary organization was established in the Department of
Treasury and came to be known later as the U.S. Lighthouse Service. In 1790, an
organization known as the Revenue Service or Revenue Marine was established, and
It was officially named the Revenue Cutter Service in 1863 (Coast Guard, available
at http://www.uscg.mil/hq/g-cp/history.html).
In 1852, a function that became known as the Steamboat Inspection Service
was established in Treasury Department. It was an early regulatory organization and
176
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
was established in response to the explosion of a steamboat carrying passengers
(Coast Guard, available at
http://www.uscg.mil/hq/g-cp/history/Policy_Changes.html).
Two more USCG predecessor organizations were established in the late
1880’s. In 1878, the Life-Saving Service was established in the Department of
Treasury. In 1884, the Bureau of Navigation was also established in the Department
of Treasury (Coast Guard, available at http://www.uscg.mil/hq/g-cp/history.html).
The Department of Commerce and Labor was created in 1903. The Bureau
of Navigation and the Steamship Inspection Service were transferred from the
Department of Treasury to the new department (Coast Guard, available at
http://www.uscg.mil/hq/g-cp/histoiy.html).
In 1915, the Life-Saving Service and the Revenue Cutter Service were
combined and renamed the Coast Guard. It remained in the Department of Treasury
(Coast Guard, available at http://www.uscg.mil/hq/g-cp/history.html).
During World War I, the Coast Guard temporarily changed organizational
location. When the U.S. declared war against Germany, the Coast Guard was
transferred by executive order to the Navy Department. It was returned to the
Department of Treasury in 1919 (Coast Guard, available at
http://www.uscg.mil/hq/g-cp/history.html).
In 1932, the Steamboat Inspection Service and the Bureau ofNavigation
combined to form the Bureau ofNavigation and Steamboat Inspection. The new
177
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
organization remained in the Department of Commerce (Coast Guard, available at
http://www.uscg.mil/hq/ g-cp/history.html).
In 1939, the Coast Guard absorbed the Lighthouse Service. The enlarged
organization remained in the Department of Treasury until the beginning of World
War II, when the Coast Guard was once again transferred to the Department Navy. A
year later, the Bureau of Marine Inspection was temporarily transferred to the Coast
Guard. The Coast Guard remained in the Department of Navy until it was returned
to the Department of Treasury in 1946. The Bureau of Marine Inspection was
abolished, and its mission was transferred to the Coast Guard at that time (Coast
Guard, available at http://www.uscg.mil/hq/g-cp/history.html).
Finally, in 1967, the Coast Guard was transferred to the new Department of
Transportation by executive order. USCG retained multiple functions of coastal
protection, regulation of commercial vessels, life saving, national defense, and
management of aids to navigation. During the 1920’s through the 1970’s, the Coast
Guard continued to acquire new functions, including drug interdiction (Coast Guard,
available at http://www.uscg.mil/hq/g-cp/history/Policy_Changes.html).
Performance based management of USCG’s maritime safety (regulatory)
program was a pilot test in early implementation of GPRA. The lessons learned in
conducting that test of using performance information as a basis for planning and
reporting results are recorded in a case study document prepared for the American
Society for Public Administration. Subsequently, the Office of Management and
178
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Budget named USCG as one of ten federal exemplars in 1994. USCG was singled
out for its outcome-related goals and having the most historic trend data of all pilot
plans (Office of Management and Budget, 1994, p. 7). Continuing progress in
implementing performance based management within USCG was documented in a
subsequent, unpublished case study prepared by a USCG student for a university
course.
The following sections discuss findings about four categories of contextual
conditions that can affect a government organization’s ability to manage for results.
First, institutional conditions that link the federal organizations to the outside world
through policies, programs, and networks of stakeholders will be described. Then,
organizational conditions that focus on the internal operations of the organization
will be covered. Next, technical conditions that affect organizational ability to
develop goals, indicators, high quality data bases, analytical methods, or useful
reports will be described. Finally, resource factors that affect organizational capacity
for financing and staffing functions necessary for planning, analyzing, monitoring,
and reporting safety results will be described.
Institutional Conditions
The policy structure for the USCG safety mission was relatively
straightforward, though it has evolved for over 200 years. The Coast Guard’s role in
waterway safety included three different types of program activities. Search and
rescue was the oldest of these program activities and provided the most direct means
179
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
available for USCG to prevent fatalities on the waterways. Less direct means of
preventing fatalities included regulation and grants to coordinate state boating safety
activities. Maritime safety programs stemmed from regulation of safety conditions
on such commercial vessels as freight and passenger carriers.
Recreational boating safety was the most recently established program
activity. The USCG role was administration and coordination of a formula grant
process. Since approximately 70 percent of the USCG search and rescue case load
resulted from recreational boating, efforts to reduce recreational boating accidents
would influence the USCG workload at the same time it improved national
transportation safety outcomes.
The Lighthouse Service established a cornerstone of the USCG mission.
Making the coast safe for marine transportation is still a waterway safety function,
even though the navigational devices and lifesaving techniques employed by the
USCG have greatly increased in sophistication. Search and rescue operations are
probably the most visible function that the public recognizes as the Coast Guard’s
job. In testimony on the Coast Guard’s maritime safety mission, the Commandant
illustrated the reach of the search and rescue mission.
The citizens of Grand Forks, North Dakota may never have
considered how the Coast Guard touched their lives, but when the
swollen Red River flooded their city, the Coast Guard was there
to assist. And very recently, Coast Guard air and boat crews
tirelessly rescued over 500 people, from new Jersey to North
Carolina, stranded by devastating floods left by Hurricane Floyd.
The international maritime community looks to the Coast
Guard as a world leader in the international field of search and
180
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
rescue. Maritime commerce and travel are global activities that
require a global safety system, and a vital aspect of that safety
system is an effective global search and rescue response
capability. (Card, 2000, p. 5)
Over the past several years the number of search and rescue cases
has decreased because of better safety awareness and the maturing
of the commercial assistance industry, which now handles many
non-emergency cases. Consequently the resource hours needed
for search and rescue have dropped. However, the number of
severe cases where lives are most likely to be lost has remained
relatively constant. DOT, March 2000, p. 30).
Acquisition of improved radio and direction-finding technology is planned to
continue improving USCG response time.
In addition to recreational boating, another source of demand for search and
rescue services is the maritime industry. Since 1994, the USCG, in cooperation with
the industry, has been working on a long-term strategic plan to refocus accident
prevention efforts towards changing human behavior. Input from the marine industry
and the findings of an internal quality action team formed the foundation of a
strategic plan for these activities. The Coast Guard has used this approach to involve
the maritime industry in improving safety practices without regulatory intervention.
The initiative is called Prevention Through People and is directed at involving the
maritime industry in self assessment and improvement of unsafe practices. (Card,
1996, p. 1).
The Coast Guard’s study group developed a systematic and
balanced approach to address human factors. The approach looks
to create a new safety culture which include 4 key pillars, all
based on a solid foundation of standards. The pillars are:
(1) management, (2) work environment, (3) behavior, and
181
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
(4) technology. They are important elements that each of these
pillars bring to the safety culture, and they sit on a sound
foundation of safety regulations/guidelines.
The foundation also includes non-governmental organizations
which play an important role in supporting the pillars. Examples
include the direct contributions of class societies, standards
bodies, and port authorities as well as indirect contributions from
insurers. When all of us who make up the foundation act in
concert, we provide operators with an even base upon which
better operators can compete evenly and marginal operators are
forced to either improve or are driven out of business. (Card,
1996, p. 3)
A Port State Control initiative was directed at improving compliance with
and improving the consistency of controls at ports, thereby, increasing accountability
for unsafe vessels.
Of the 7,880 foreign flag ships that arrived in the U.S. in 1998,
373 were detained because of their substandard condition. In
1999, only 260 foreign flag ships were detained,. . . representing
an increased level of safety in the foreign fleet visiting our ports.
.. . Our risk-based matrix appears to be successful in screening all
arriving vessels to ensure that the highest risk vessels are boarded.
(Card, 2000, p. 4)
While passenger vessels account for few of the fatalities on waterways, the
USCG took a “proactive, systematic, risk-based approach” that contributed to
passage of the Passenger Vessel Safety Act of 1993. It focused on passenger vessels
evading inspection through charter arrangements.
Passenger vessel traffic is rapidly growing on cruise ships,
gambling ships, and passenger ferries, and the number of high
speed, high-capacity passenger vessels is booming. From
historical data we know that collisions, allision, and vessels
running aground make up a majority of passenger vessel
accidents—and most of these are caused by human error. But we
also know that the highest danger to passenger safety generally
182
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
occurs in the lower frequency incidents, such as fire, capsizings
and sinkings. These, too, are incidents frequently caused by
human error, as well as by equipment and vessel condition. The
Coast Guard’s Prevention Through People program and
implementation of International Safety Management Code target
these causes. (DOT, March 2000, p. 32)
The Coast Guard’s role in recreational boating safety was established in the
Federal Boat Safety Act of 1971 and included two grant programs—a state grant
program and a grant program for nonprofit public service organizations that support
boating safety activities. The Recreational Boating Safety and Facilities
Improvement Act of 1980 required that the portion of the Highway Trust Fund
attributable to motor boat fuel use would be earmarked for a Recreational Boating
Safety fund to support the grant programs (Coast Guard Web page, available at
http://www.uscgboating.org).
There are about 78 million boaters in the U.S., and about 800 people lose
their lives on boats annually (DOT, March 2000, p. 28). Most of these deaths are by
drowning, so public service announcements and promotional campaigns are directed
at getting people on boats to wear life jackets. The theme for 1999 and 2000 is “Boat
Smart from the Start. Wear Your Life Jacket.” Also, the USCG has formal
partnerships with the Coast Guard Auxiliary and the Association of State Boating
Law Administrators to get involvement at the local level and to expand the resources
applied to safety inspections and other enforcement activities (Coast Guard Web
page, available at http://www.uscgboating.org).
183
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The Recreational Boating Safety State Grant Program provides formula
grants to states and territories for administration, improving law enforcement and
search and rescue capability, boater education, vessel numbering and titling systems,
aids to navigation, and public boating access sites. Boating fatalities have been
reduced from a high of 1,754 in 1973 to about 800 per year in recent years; and,
during the same period, boat ownership nearly doubled (Coast Guard Web page,
available at http://www.uscgboating.org).
Organizational Conditions
In 1999, the USCG had actual, total obligations of slightly over $3.5 billion
(DOT, March 2000, pp. 46-47) and a workforce of about 5,900 civilians and 30,000
military personnel (OPM, available at http://www.opm.gov/feddata/table 15). About
26 percent of the obligations were for safety program activities (DOT, March 2000,
pp. 46-47).
Organizationally, the USCG is decentralized and has eight headquarters
directorates that report to a chief of staff. The chief of staff reports to the
Commandant and Vice Commandant, whose office has support staffs, including the
strategic planning staff. Most of the workforce is in one of the two area
commands—Atlantic and Pacific—which are organized into nine districts.
Following the USCG pilot program, in 1996, a case study report about the
pilot was prepared for the American Society for Public Administration. A number of
184
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
organizational challenges were discussed in the case study and in an un-published
follow-up assessment two years later.
One organizational challenge noted in the GPRA pilot case study report
prepared for the American Society for Public Administration was the importance of
local circumstances in determining risks. This challenge was linked to
organizational data that indicated employee dissatisfaction with centralized
management. This combination of awareness that local conditions influenced
waterway safety risks and that employees wanted to participate in management
decisions led to a cross-organizational approach and managerial flexibility.
Outcome-oriented goals inherently cut across organizational lines,
and therefore their development is facilitated by use of cross-
organizational group (versus delegation to smaller components of
the organization to develop their own goals).. . . Organizations
must have the flexibility (from higher levels with the
Administration and from Congress) to reinvest their own
resources toward higher payback activities. Managerial flexibility
can be increased dramatically by simply reducing the
organization’s own internal rules and standards for activity
performance. (Henn, 1995, pp. 9-10)
The field managers were involved in performance planning and goal setting activities
(Kowaleski, 1996, p. 25).
In the early phases of the pilot program, formulating goals was a challenge,
because the people involved felt that they did not control the results of maritime
safety program activities. When the question was asked as “Why do we do this
activity,” it became easier to work the question down to how strategies being
followed contributed to the outcomes. This was reinforced by developing the goals
185
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
from the bottom up and involving safety officers from all ten field districts. The
threat to centralized control, however, produced organizational inertia (Kowalewski,
1996, p. 24).
The positive influence of the bottom-up approach to developing goals was
reconfirmed in a follow-up to the earlier case study report. “Field personnel now see
a direct return on their investment in accurate and timely data entry, because it now
determines the work they perform and how well their unit achieves its goals”
(McCrimmon, 1998, p. 17).
An additional organizational condition that inhibited using performance
information to manage for results was the external focus of the plans and the lack of
an internal focus. Training and development activities, for example, were not goal-
oriented and tied to performance plans. A Human Resource M anagem ent Plan for
1998 through 2008 was published in 1998 to fill in this gap (McCrimmon, 1998,
p. 20).
Technical Conditions
An early lesson in the pilot phase of GPRA implementation was the need to
disaggregate data into groupings that would account for local conditions. While
numbers of deaths of maritime workers was chosen as the most important outcome, a
meaningful denominator was needed to normalize the data for changes in exposure.
The problem with the best way to measure changes in exposure was that it required
finding a source of data for the total number of maritime workers (Kowalewski,
186
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
1996, p. 10). This problem was still evident when the follow-up case study was done
in 1998 (McCrimmon, 1998, p. 20).
A difficult question that remained to be answered after maritime safety
outcomes were measured was “what is the program contribution to those outcomes”
(Kowalewski, 1996, p. 22). External conditions and the use of multiple interventions
complicate estimating the contributions of separate interventions (McCrimmon,
1998, p. 25).
The recreational boating program was not included in the pilot phase of
GPRA implementation. As a formula grant program, this program activity limits the
USCG role to process administration and coordination.
Resource Conditions
The USCG demonstrated that it had most of the resources required to plan,
monitor, and report on safety performance during the pilot program phase and in the
handling of revisions to previously established measurement methods and goals. In
the pilot program case study and the follow-up, the organization described itself as
“data rich.”
One resource shortage mentioned in both the pilot program case study and the
follow-up was the shortage of program evaluation expertise. The case study
indicated that, faced with data analysis problems, “we created an ad hoc program
evaluation group of experts with academic backgrounds and experience in
economics, operations research, and policy analysis” (Kowalewski, 1996, p. 8). Two
187
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
years later, the follow-up report indicated that “the Coast Guard has neither program
evaluators, nor staff with training or experience in program evaluation”
(McCrimmon, 1998, p. 24).
The follow-up report also noted that, “for most Coast Guard managers, it is a
real challenge to begin managing for results, because their education and training
does not include learning the skills required to measure program outcomes, to
benchmark, and to make strategic plans” (McCrimmon, 1998, p. 8). This lack of
skills is being addressed with “some career training in managing for results”
(McCrimmon, 1998, p. 8).
Performance Information Uses
Waterway safety performance information was used in many ways, beginning
with monitoring change in types of accidents and outcomes over time. Such
information was further used to set regulatory priorities and to identify non-
regulatory solutions to problems. Among the non-regulatory solutions was to
facilitate industry learning about safety risks and to educate and motivate behavior
change in workers at risk on commercial vessels.
Safety performance information is also used to provide public information
and education about recreational boating risks and safety practices. Printed reports,
coordinated state-level safety campaigns, and the USCG Web page feature such
information prominently.
188
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Similarly, performance information is used to communicate the value of
waterway safety programs to the public. Search and rescue functions that take place
in flood waters as well as normal waterways illustrate the human drama behind
quantitative reports of lives saved.
Summary: Properties of Contextual Conditions for Use of Waterway Safety
Performance Information To Improve Outcomes
The USCG had three sets of policy tools to improve waterway safety
outcomes. First, the USCG role in waterway safety included providing search and
rescue services to mariners in distress. Second, USCG regulated safety conditions on
commercial vessels that used U.S. waterways, including safety practices for maritime
workers on these vessels. Finally, USCG administered formula grants to states to
promote safety recreational boating practices.
When performance information was used to identify the sources of demand
for search and rescue services, most of the demand was found to be from recreational
boating. USCG used performance information to identify high risk areas and found
that most recreational fatalities were from drowning that could have been avoided, if
the victim had worn a lifejacket. Partnerships with the Coast Guard Auxiliary and
state boating regulation officials were also established to improve enforcement of
safety practices on recreational boats.
Marine safety on regulated commercial vessels was the mission area
addressed by a pilot program for GPRA implementation. The pilot program resulted
189
Reproduced with perm ission o f the copyright owner. Further reproduction prohibited without perm ission.
in a risk management approach to planning and measuring the safety results of the
maritime safety program. Performance information was used to set priorities for
inspections and enforcement actions. This performance based management approach
has become a way of doing business within USCG.
The GPRA pilot also led to leadership of international efforts to improve
enforcement and consistency of safety codes for merchant vessels involved in
international trade with the U.S. past performance information supported
prioritization for frequency and types of inspections for domestic commercial vessels
and passenger vessels. This led to partnerships with the industry, a Prevention
Through People Initiative to focus on behavioral sources of risk, and a trend of
reduced maritime worker fatalities.
While the fatality rates for passenger vessels was found to be relatively low,
the growth in the numbers of such vessels and the size of the vessels indicated a high
level of risk in the case of an accident. The indicator for USCG performance in
reducing risk of passenger vessel fatalities was elevated to the departmental
performance monitoring system. It replaced the indicator for maritime worker
fatalities as a departmental indicator.
Maritime safety information reported at the departmental level was revised
both in content and coverage in DOT’s Fiscal Year 1999 Performance Report Fiscal
Year 2001 Performance Plan. One revision reflected adjustments for known under
reporting of recreational boating fatality data. The other revision replaced the
190
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
maritime worker indicator with a passenger vessel indicator to reflect a higher
priority in terms of risk to the public. Both indicators continue to be monitored by
USCG for performance based management at the operating administration level.
Institutionally, barriers to using performance information to improve efforts
to manage for results stemmed from the fact that the USCG had no direct control of
the major cause of waterway fatalities—human behavior. A Coast-Guard-wide
program called Prevention Through People was established to provide a framework
for improving the safety of behavior on ships and boats. The objectives and methods
of this program were included in training courses for safety inspectors and promoted
by senior leaders in conferences with maritime industiy representatives.
One strategy for implementing this change to a people-oriented approach to
regulation was to change stakeholder roles. The USCG used leadership and
promotion of awareness of worker safety to modify the previous authority based
regulation and to make the maritime industry partners in improving worker safety.
For the largest source of behavioral risk on waterways—recreational
boating—the USCG used federal level coordination of public outreach efforts and
partnerships with voluntary organizations and state officials to influence waterway
safety outcomes. The strategies implemented included a national campaign to
increase awareness of the risk of not wearing life jackets and partnerships with
voluntary organizations and state boating safety officials.
191
R eproduced with perm ission o f the copyright owner. Further reproduction prohibited without perm ission.
Technically, lack of reliable sources for information about the exposure level
to risks in waterway safety posed a barrier. To allow measures to be normalized for
the maritime safety program, finding a reliable source of information about the
numbers of maritime workers posed a problem that was overcome, in part, by
partnering with the industry.
Another technical challenge was sorting out the contributions of different
activities to the results that were attained. When multiple types of interventions to
risk areas that were identified by disaggregating fatality data, and interventions
included both technology changes and behavioral change initiatives, identifying
whether or how much each intervention contributed was seen as problematic.
This technical challenge was related to two resource challenges. Lack of
skills for program managers to plan and monitor results and the absence of program
evaluation resources made full implementation of GPRA problematic.
USCG’s pilot project under GPRA laid the groundwork for a continuing
internal management reform. Performance based management was implemented
through changes in institutional relationships to involve key stakeholders in problem
solving. Organizationally, managers at each level were involved in developing the
performance system, and USCG leadership facilitated both the external and internal
changes that were necessary—technically, challenges in evaluating the relationships
of multiple interventions to the results achieved.
192
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
A summary of properties of contextual conditions for USCG in using
performance information to improve results is outlined below.
Waterway Safety Contextual Conditions
Institutional Contextual Conditions
► Waterway safety functions were among the oldest federal administrative
responsibilities.
► Waterway safety responsibilities included management of a large,
technologically advanced infrastructure.
► Constitutional limitations on the federal role created dependence on states
for implementation of recreational boating safety policies.
► Partnerships were used to build networks of individuals and organizations
with common interests in improving waterway safety outcomes.
► Public participation was used to facilitate consensus building on goals
and implementation strategies.
Organizational Contextual Conditions
► Waterway safety programs were implemented by a uniformed military
workforce,
► Waterway safety’s GPRA pilot program led to combining internal
management reform with performance based management and quality
improvement.
193
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Technical Contextual Conditions
► There were continuous improvement efforts to refine indicators and
improve data used for analysis.
► Revisions between 1999 and 2000 in priorities of indicators elevated
internal performance planning goals to departmental planning goals.
Resource Contextual Conditions
► Waterway safety programs were implemented in a general environment
of funding constraints linked to high cost infrastructure investments.
Uses of Performance Information
► Monitor change in outcomes over time.
► Set regulatory priorities.
► Identify non-regulatory solutions to problems.
► Facilitate industry learning about safety risks.
► Educate and motivate behavior change in workers at risk on commercial
vessels.
► Public information and education about recreational boating risks and
safety practices.
► Communicate the value of waterway safety programs to the public.
The next chapter explores the high level concepts underlying the diverse sets
of properties associated with highway, aviation, and waterway safety performance
194
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
management. These high level concepts are then organized into the major themes of
the underlying theory for using performance information to improve transportation
safety outcomes. This theory responds to two hypotheses posed in Chapter 4 about
the departmental approach for implementing performance based management as
required by the Government Performance and Results Act.
195
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
CHAPTER 8
SUMMARY AND CONCLUSIONS
This dissertation responds to a big question in public administration that has
been asked at two different levels. First, can public managers use performance
information to improve performance? Second, to what extent do societies know that
public institutions are producing desirable results?
This research explored the contexts associated with using information about
past performance in the national transportation system to improve safety outcomes.
Multiple organizations used performance information to contribute to a common
strategic goal of the U.S. Department of Transportation (DOT). This goal was to
reduce fatalities and injuries in the highway, aviation, and waterway components of
the transportation system.
The Department of Transportation’s approach to implementing the
Government Performance and Results Act (GPRA) was explored as an illustrative
case study of a highly praised approach to planning to improve national outcomes.
The illustrative case was followed by comparative, embedded case studies of the
three components of the transportation system—highways, aviation, waterways.
These embedded case studies explored contextual conditions and strategies that
196
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
facilitated use of performance information in five federal operating administrations
with programs to improve transportation safety outcomes.
Constant comparative analysis was used to identify high level concepts and
themes of a theory for using transportation safety performance information to
improve safety outcomes in the transportation system. The research questions were:
1. What were the conditions associated with using information about past
performance to improve future transportation safety outcomes?
2. What strategies were used to facilitate the use of performance information
to improve transportation safety outcomes?
DOT was selected for study for four reasons. First, the Congress and external
oversight organizations have recognized it as the federal government’s leader in
performance based management. Second, the Office of Management and Budget
named two of its operating administrations with safety missions as federal exemplars
after completing GPRA pilot projects. Third, performance plans by three other
operating administrations with safety missions were criticized by external evaluators
for weaknesses in implementing GPRA. Finally, these five operating administrations
represented a diverse set of organizations with different policy tools, institutional
relationships, technical problems in collecting and analyzing performance data, and
resource environments for assuring funds and expertise to do performance based
management.
197
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
The following section summarizes the high level concepts of DOT’s
approach to performance based management and restates two hypotheses from
Chapter 4. A summary of the contextual conditions in the operating administrations
with transportation safety missions provides the information for testing these
hypotheses. This analysis identifies the major themes of the underlying theory of
DOT’s approach
DOT’s Approach to Performance Based Management
Chapter 4 discussed how DOT’s senior leadership implemented performance
based management to improve safety outcomes. Departmental leaders interpreted
performance information as the means for learning how to improve results over time.
This learning process was facilitated through promoting partnerships among
networks of government organizations and stakeholders that shared the common goal
of reducing safety risks and saving lives. Also, senior DOT executives established
processes for performance monitoring that linked strategic planning and budget
formulation. The Department also used technical support services to coordinate
improvements to overcome methodological and data quality problems from the
departmental level.
The strategic goal for transportation safety was to “promote the public health
and safety by working toward the elimination of transportation-related deaths,
injuries, and property damage” over a five-year period (DOT, DOT Strategic Plan for
Fiscal Years 1997-2002, 1997, p. 12). Progress in attaining this goal was tracked
198
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
through a series of indicators that included rates of deaths to provide a basis for
judging the results among different components of the transportation system. Using
rates also provided indicators that reflected changes in environmental factors for
performance over time. The relative differences in the numbers and related causal
systems associated with deaths in the transportation system necessitated this
approach. In particular, the Department’s performance report for Fiscal Year 1999
noted the dominant effect of highway fatalities on safety results:
A reduction in highway fatalities in 1999, which account for
approximately 94 percent of all transportation fatalities, accounts
for the direction of the overall fatalities [trend]. While relatively
small, the decrease is more substantial in the context of rising
miles of travel and meets the strategic outcome goal of reducing
the total number of transportation-related deaths. (DOT, 1999
Performance Report 2001 Performance Plan, 2000, p. 8)
Transportation system safety performance outcomes are illustrated in Table
15. It summarizes the results for DOT’s transportation safety goals performance,
beginning with actual results in the two years preceding the 1999 results and
including the changes in goals for 2000 and 2001. DOT noted that many
transportation safety programs have reported information on a calendar year basis for
many years rather than on a fiscal year basis as required by GPRA. To allow
comparisons of performance in past years to current year performance and future
goals, DOT reported some areas of performance in calendar years and others in fiscal
years. “Either is a satisfactory indicator for 1999 performance” (DOT, 1999
Performance Report 2001 Performance Plan, p. 5). The highway and aviation fatality
199
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
TABLE 15
PERFORMANCE PROGRESS REPORT FOR TRANSPORTATION SYSTEM SAFETY
Measure 1997
Actual
1998
Actual
1999
Actual
1999
Goal
G oal
M et?
G ood
Trend?
2000
G oal
2001
G oal
Highways
■ > ./
Highway fatalities/100 million vehicle miles traveled 1.6 1.6 1.5 1.6 Yes 1.5 1.5
Highway injuries/100 million vehicle miles traveled 133 122 119 127 Yes 116 113
Per cent highway fatalities alcohol related 38.6 38 38 36 No Yes 35 34%
Per cent front occupants using seat belts 69 70 67 80 No No 85 86%
Number of fatalities involving large trucks 5,203 No 4,934 4,830
Number of injured persons involving large trucks 127,000 No 125,000 122,000
Aviation
U.S. commercial fatal aviation accidents/100,000 flight .055 .006 .040 .034 No No .033 .031
Number of fatal general aviation accidents 354 No 379 379
Number of runway incursions 318 325 322 270 No No 248 241
Operational errors/100,000 activities .48 .55 .57 .49 No No .486 .5
Deviations/100,000 activities .12 .18 .18 .099 No No .097 No
Waterways
Number of recreational boating fatalities 857 864 773 763 No Yes 763 749
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
TABLE 15 (continued)
Measure 1997
Actual
1998
Actual
1999
Actual
1999
Goal
G oal
M et?
G ood
Trend?
2000
Goal
2001
G oal
Percent of reported m ariners in danger rescued 93% 94% 95% 93% Yes No No
Percent of all mariners in danger rescued 85% No No 88%
Number fatalities/100,000 workers on commercial vessels 39 35 28 34 Yes No No
Number of high risk casualties per 1,000 vessels 39 No 53 52
Source: U. S. Department of Transportation, (2000, March). 1999performance report 2001 performance plan U.S. Department of Transportation,
pp. 11-33, available at http://www.dot.gov.
t o
o
data were reported for calendar years, and the waterway data were reported for fiscal
years (DOT, 1999 Performance Report 2001 Performance Plan, 2000, pp. 11,19 &
28).
DOT reported a modest but positive aggregate safety outcome. The results
reported in Table 15 provide data about goal attainment for the individual
components of the transportation system. It is evident which goals were attained and
which were not, but it is not evident from the table how the information was
interpreted. The narrative for each component of the transportation system provided
this interpretation by assessing strategies for goals that were attained as well as goals
that were not attained.
These assessments discussed the meaning of the results for each strategy for
improving safety performance outcomes. For example, for highways, adjustments
were planned in strategies for reducing alcohol-related fatalities and increasing seat
belt use to improve the likelihood of attaining both of those goals in 2000.
The aggregate aviation safety goal of reducing the rate of fatal accidents per
100,000 flight hours was not attained. It was noted that the goal was on the path to
being achieved in 2000 and was not changed. The high annual variances in air
carrier accidents from year to year called for more than a single year to reach a
judgment. An additional goal for general aviation accidents was added, however, to
improve aviation safety coverage in performance plans and reports.
2 0 2
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
For waterways, a substitution of the commercial vessel safety goal with
another was explained as a similar improvement in coverage rather than a change of
strategies for improving waterway safety. Also, continuing work to improve data
quality led to revisions in waterway safety baseline indicators.
Contexts and Strategies for DOT’s Approach
Five of DOT’s operating administrations implemented federal policies and
managed programs to improve safety outcomes in the transportation system. Each of
these operating administrations developed separate organizational and budget
program performance plans with stakeholder participation. The indicators selected
from these plans for inclusion in the DOT plan were negotiated between the
operating administrations and DOT’s senior leaders based on priority and usefulness
to public understanding of the safety risks in the overall transportation system.
Performance plans also conveyed the value of governmental efforts by linking results
to program budgets.
Highway safety performance was the only component of the transportation
system for which more than one DOT operating administration directly contributed
to performance results. Within DOT, highway safety was the joint responsibility of
three operating administrations—the Federal Highway Administration (FHWA), the
National Highway Traffic Safety Administration (NHTSA), and the Federal Motor
Carrier Safety Administration (FMCSA).
203
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Two other operating administrations managed major transportation safety
policies and programs. Aviation safety was the responsibility of the Federal Aviation
Administration (FAA). Waterway safety was the responsibility of the U.S. Coast
Guard (USCG).
Chapters 5 through 7 described the contextual conditions for the components
of the transportation system. Improving safety outcomes for highways, aviation, and
waterways involved five operating administrations. Each organization had policies
and stakeholder relationships that evolved historically as federal policies and
programs changed in response to different circumstances. The organizational
structures and workforces of these organizations also evolved with changes in the
policies and programs. Different types of technical problems affected the use of
performance information to improve safety outcomes on highways, aviation, and
waterways. Each component of the transportation system also had different sets of
financial and human resources to use past performance information to improve future
results.
At the conclusion of Chapter 4, two hypotheses about the Department of
Transportation's approach to implementing performance management were stated.
The first hypothesis was that different contextual circumstances presented different
challenges among the operating administrations using performance information to
achieve a common goal. The second hypothesis was that DOT’s management
204
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
approach for implementing performance based management facilitated use of
performance information to improve results in these diverse contextual conditions.
The following section discusses the elements of DOT’s approach to
facilitating pursuit of a common strategic goal in diverse, highly complex contextual
conditions.
Institutional Complexity
Institutional complexity arose from diverse sets of policy tools, stakeholder
groups, and constitutional limitations of federal authority in states. The federal role
in transportation safety grew with the technology that began on waterways, expanded
to highways, and subsequently to aviation activities. Safety performance policy tools
evolved differently in each component of the transportation system. The institutional
history of how each operating administration evolved organizationally was tied to
changes in policy tools.
Regulatory powers were expanded to provide more direct control of transport
vehicle manufacturing and to operational control of all aspects of air transportation.
Operational responsibilities for direct delivery of aviation and waterway safety
functions grew with new technology for airway and waterway navigation, air traffic
control, and lifesaving functions for search and rescue of aviators and mariners in
distress.
As policies and programs changed, the variety of stakeholders came to
encompass many different industries, interest groups, and citizens working to
205
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
improve transportation safety. For highways and waterways, constitutional limits on
federal authority limited direct control in states. For aviation, however, the national
airways were not subject to such limitations.
Given the span and number of stakeholders for transportation safety, DOT
sponsored public involvement in development of transportation safety plans through
the National Transportation Safety Conference. This conference produced a safety
action plan and a memorandum of understanding between DOT and a wide variety of
industries, law enforcement organizations, and community leaders.
Organizational Complexity
DOT also established a Safety Council to involve executives and managers
from throughout the Department in planning and delivering safety outreach activities.
The Safety Council agenda also included promotion of labor-management
partnerships and gave priority attention to human factors research.
Reforms to improve organizational structure and performance planning for
transportation safety were initiated internally as well as at the direction of the
Congress. USCG continued its implementation of performance based management
improvements that began as part of a GPRA pilot project. The Congress directed
establishment of a partnership between FAA and the airline industry to improve
aviation safety, and the Congress also changed the organization of highway safety
functions in DOT to increase attention to the safety outcomes for interstate truck and
bus operations.
206
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Both the White House and the Congress established commissions io
recommend aviation safety reforms. The White House commission recommended
and the Federal Aviation Administration (FAA) established a partnership between
FAA and the aviation industry “to find better ways to achieve its goals, seeking to
replace confrontation with cooperation” (White House Commission, 1997, p. 4).
The Congressional commission that was also reviewing aviation safety made similar
recommendations for a partnership to re-engineer regulation and certification
functions in aviation regulation by using partnerships for meeting goals. Internally,
FAA established a Safer Skies initiative to implement this recommendation. FAA
also independently promoted data exchange among commercial air carriers to
improve knowledge about aviation system safety.
The Congress restructured the highway safety component of DOT by
separating the safety functions from the Federal Highway Administration to create
the Federal Motor Carrier Safety Administration, a new, independent operating
administration. Performance information was a stimulus to this reform, and this new
organization was established to improve regulation of safety in commercial,
interstate truck and bus operations.
The National Highway Traffic Safety Administration (NHTSA) used a wide
array of partnerships to improve knowledge about highway accidents, injuries, and
emergency medical responses. It was also the only operating administration to track
injuries from transportation accidents. NHTSA continued to develop a network of
207
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
partnerships for coordinating data collection and analysis of injuries as well as
fatalities resulting from highway accidents.
The U.S. Coast Guard also used partnerships in reforming its regulatory
relationships with maritime commercial operators as part of its GPRA pilot program.
In addition to reforming its regulatory functions, USCG continued internal
management reform to folly integrate use of performance information to plan
improved performance in all aspects of its operations.
In summary, there was a clear history of change and continuing reform of
federal transportation safety relationships and organizational arrangements that
continued through implementation of GPRA. This propensity for change to improve
safety reflected two researcher’s views about how institutions and organizations
adopt change. Peter Checkland (1999) found in his research that history determined
what was seen as significant in organizations, and John Kingdon (1993) found that
the “countable problem” was more easily adopted into the reform agenda. Kingdon
also found that “transportation, with its greater [policy agenda] fragmentation, fewer
agreed-upon paradigms, and greater susceptibility to crisis is simply less completely
structured. That relative lack of structure leaves the agenda free to shift from one
time to another” (p. 121).
Throughout DOT, partnerships were used to build networks of individuals
and organizations with common interests in improving transportation safety
outcomes. In addition, public participation was used to facilitate consensus building
208
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
on goals and implementation strategies. Grant programs, particularly for highway
safety and recreational boating safety, involved states and communities in safety
education and promotion of safety practices. Regulatory organizations also used
partnerships and shared performance goals to recognize the essential role of regulated
industries in improving safety practices, particularly changing human behaviors that
increase safety risks.
DOT’s success in widespread networking through partnerships harmonized
with knowledge from research findings about continuity and change during
government reforms. Nils Brunsson and Johan Olsen (1993) found in their research
that maintaining continuity between external, institutional values and organizational
values facilitated commitment to reforms. They described such reforms as learning
processes. DOT’s transportation safety improvement strategies included partnerships
that integrated institutional values and resources into federal operating administration
performance planning and monitoring activities. This approach met the need to learn
more about what causes transportation accidents and how to change technologies and
behaviors to improve outcomes. It also blurred boundaries between institutional and
organizational conditions by facilitating both societal and managerial learning.
Organizational complexity was addressed through internal collaboration that
included top-down direction and bottom-up strategy development and goal setting.
The strategic goal for transportation safety was the responsibility of five semi-
autonomous organizations operating in different contextual circumstances.
209
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Collaborative processes were used to elicit multi-discipline and cross-organizational
communication and problem solving.
DOT’s approach was in harmony with research on how learning occurs in
organizations and why it must be decentralized in the search for safety. Anthony
Bovaird and David Gregory (1996) had found that learning in British government
organizations was encouraged by a combination of strategic controls. Top-down
controls guided and encouraged learning when bottom-up strategic controls were also
used. Similarly, Julianne Mahler (1997) found that policy learning within an
organization required dialogue to interpret evidence and facilitate change among
organizational participants. Aaron Wildavsky (1988) found that institutions that
engaged in a “decentralized search” avoided the pitfalls of believing that safety could
be improved through control and design. Karl Weick (1995) also found that actions
to improve outcomes in government agencies took place through institutional,
political processes as well as organizational, operational processes.
The formulation of the aggregate, strategic goal for transportation safety
provided fatality and risk reduction as common outcomes for all three components of
the transportation system. The roles of the operating administrations, in
collaboration with their stakeholders and partners, were to plan strategies to improve
performance in each separate organization and program, to justify their plans in
budget requests, and to nominate the key goals and indicators to be included in the
DOT plans.
210
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
DOT’s leaders avoided punitive uses of performance information that would
only identify errors and missed goals. This was done to avoid the possibility of
distorted organizational performance reporting to avoid punitive reactions. The
principal interpretation of performance information was for learning about the
sources of error and ways to improve future outcomes. Goals that were attained were
assessed as well as those that were missed. The approach reflected the role of
reflective learning as well as action in implementing change, as research by Chris
Argyris (1977), Aaron Wildavsky (1988), and James March and Johan Olsen (1989)
would have suggested.
Technical Complexity
DOT’s open, collaborative approach was also useful in dealing with technical
complexity. One source of this complexity was the dependence of the operating
administrations on others to learn how to reduce risks in the transportation system.
For example, learning about how to achieve marginal improvements in an aviation
system that was already very safe was advanced by promotion of industry efforts for
data exchange and by an FAA partnership with the regulated industry. Also, learning
about how to change behavior regarding seat belt use and avoidance of alcoholic
beverages when driving was advanced by incentive grants and information exchange
among states and local communities. Similarly, learning about how to prevent
recreational boating accidents and fatalities relied on incentive grants and
information exchange about successful local practices.
211
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Another technical problem stemmed from the fact that safety risks in the
transportation system arose from both technological and behavioral problems.
DOT’s safety organizations grew out of engineering functions with expertise in
technology. Systems thinking and multi-discipline analysis were used to develop
both technical and social strategies to achieve safety outcomes. Social solutions
required inquiry rather than engineering, as Peter Checkland (1999) found in his
research. Also, as Donald Kettl (1988) pointed out, dependence on third parties
(through grants and partnerships) to improve performance required long-term
problem solving rather than a short term focus on outputs.
Another technical problem that DOT’s approach addressed was biased
perception of aviation safety risks. Such bias was known to influence public opinion
and the political agenda about relative risks of highway, aviation, and waterway
transportation. Therefore, aggregate information as well as mode specific
information about transportation system safety was reported to reveal the relative
risks among the different components. Rates as well as numbers of fatalities and
accidents provided a clearer understanding of safety across the entire system.
Reports of the aggregate reductions in numbers of fatalities were also clarified to
reflect relative risks.
One technical problem for performance analysis stemmed from the fact that
funding for some programs supported one strategic goal and had secondary
influences on another strategic goal. Contributions of programs to multiple strategic
212
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
goals were recognized in budget justifications to facilitate discussions about how
outcomes were achieved. Presentation of performance information in budget
program formats made clear that safety and mobility goals were interdependent,
because technological infrastructure programs in highways, aviation, and waterways
contributed to both safety and mobility outcomes.
A key example was the interdependence of aviation safety and mobility goals.
Navigational equipment was essential for both mobility and safety functions inherent
in air traffic control. Such equipment allowed air crews and air traffic controllers to
maintain separation among aircraft moving on the ground and in the air. Maintaining
adequate separation was an essential principle of safe operations, and two of the
aviation performance goals were precisely to monitor the frequency of incursions on
runways and operational errors in airways. Justifications for research and
acquisitions to improve the technology that would allow reduced separation
standards were based on the need for improved procedures and instrumentation to
achieve an even lower level of safety risk while increasing the capacity of the
aviation system.
Resources for Performance Based Management
The definition of resource conditions to be explored in this research was
taken from the literature. Potential lack of resources to establish or improve the data
collection, data management, and analysis needed for performance based
management was an issue discussed by Kathryn Newcomer (1997) as a potential
213
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
obstacle for organizations that lacked “political will from the top.” Given the strong
leadership exerted by DOT’s senior leaders, resources are generally not seen as an
obstacle to implementation performance based management of transportation safety.
The Deputy Secretary viewed the resource issue more as a challenge in placing the
right resources in the right places than as lack of resources.
DOT’s operating administrations had long histories of collecting and using
safety data. Most of them had organizational units dedicated to safety data
management and analysis. Central coordination for solving data system and
performance analysis problems took advantage of expertise in short supply.
This research did find an unanticipated resource issue regarding performance
based management in operating administrations with responsibility for large
infrastructures that are essential for safety. When strategies for improving safety
outcomes depend on research, maintenance, or acquisition of expensive technologies,
resources may become an issue for achieving improved safety outcomes. Limitations
on the long-term funding to develop and acquire new technology may be an obstacle
to accomplishing safety improvement strategies over time. It is an issue worthy of
further research.
The strategies discussed above identify the high level concepts of DOT’s
approach to implementing performance based management in a complex
environment. They can be further reduced to a more limited number of major themes
that delineate the underlying theory that guided DOT’s implementation of
214
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
performance management. Major themes of the DOT theory are discussed in the
next section.
Toward a Theory for Performance Management of Transportation Safety
Essentially, DOT’s approach to performance based management was to
facilitate a focus on the shared safety outcome pursued in highly complex contextual
conditions. This approach emphasized senior leadership’ s role in directing and
modeling managerial use of performance feedback to improve results. Constructive
use of performance information meant using the information to learn at both the
organizational and institutional levels.
A systems level strategic goal provided a framework for learning about both
technical and social factors that affected safety risk. The aggregate strategic goal
provided a framework that accommodated the complexity of institutional and
organizational arrangements for government’s role in improving safety outcomes.
Departmental negotiation and adoption of performance goals, indicators, and
strategies developed by operating administrations assured bottom-up involvement to
achieve a shared strategic goal. It also accommodated the complexity of conditions
faced by the operating administrations.
Collaborative processes expanded individual and group access to information
relevant to solving problems. Inclusion of stakeholders in planning and
implementing strategies to improve safety further decentralized the search. It also
215
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
connected reforms in federal transportation safety organizations to institutionalized
value systems and resources.
These themes suggest five elements of the underlying theory of DOT’s
approach to using performance information to improve future safety outcomes.
These elements are:
1. Systems thinking focused inquiry on both social and technical dimensions
of problem solving.
2. Top-down direction supported by bottom-up goal setting, strategy
development, and implementation facilitated organizational commitment.
3. Reflection on why implementation strategies did not achieve goals
facilitated learning as the principal use of performance information.
4. Collaborative processes that included stakeholders and government
managers facilitated learning from the individual level to the institutional level.
5. Partnerships among government and non-government organizations
facilitated diffusion of knowledge to improve outcomes at federal, state, and local
levels.
Goal attainment assessment was merely the starting point for using
performance information for learning. Given the great complexity of each
component of the transportation system, not to mention the system as a whole,
learning how to improve safety outcomes had to be decentralized. Such an open-
ended search also addressed the requirement to accomplish multiple purposes spelled
216
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
out in GPRA. Performance information was used as a means to improve
accountability, to improve internal management, to increase public confidence in the
capability of the federal government, and to improve Congressional decisionmaking.
As hypothesized in Chapter 4, different contextual conditions presented
different challenges to operating administrations using performance information to
achieve a common goal. Policy tools for improving highway safety were split among
three separate operating administrations with a mix of regulation and grants to states.
Extensive federal control for aviation safety had achieved a very low level of risk,
and a performance plateau complicated additional improvement. Improved waterway
safety outcomes were pursued with a mixture of life-saving services, regulation, and
grants to states. This mix of policy tools was managed by a single operating
administration.
Also, as hypothesized in Chapter 4, DOT’s management approach for
implementing performance based management facilitated use of performance
information to improve results in diverse contextual circumstances. Top-down
strategic planning and coordination was supported by bottom-up performance
planning and strategy development. This top-down/bottom-up approach employed
systems thinking, collaboration, partnerships with stakeholders, and use of
performance information for learning.
217
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Finally, what can be learned from contrasting NHTSA and USCG as federal
exemplars for internal reform to FHWA and FAA as the recipients of external
criticisms and reforms reveal? Three contrasts are notable.
First, FAA and FHWA completed narrowly focused GPRA pilot projects
within single units of their organizations. By contrast, both NHTSA and USCG used
experience from GPRA pilot projects to initiate widespread internal reforms.
NHTSA expanded its data collection and analysis into new program areas for
emergency medicine and tracking the results of injuries in highway accidents. USCG
expanded its original GPRA pilot project for marine safety regulatory reform to
change the way performance information was used in all budget programs.
Second, FAA and FHWA represented two ends of the spectrum for federal
control of outcomes. FAA had a great deal of direct control of outcomes within the
national airways, and FHWA had little or no direct control of outcomes on national
highways. FAA’s air traffic control and extensive regulatory powers contrasted with
NHTSA’s outreach and social learning role and its regulation of manufacturing
standards. FHWA collaborated with NHTSA to incorporate highway safety
incentives into formula grants for highway construction and maintenance.
Similarly, USCG’s waterway safety mission included more limited regulatory
powers than FAA’s mission included. USCG used the GPRA pilot project to
develop a partnership approach to safety regulation, and FAA was directed by
commissions from the both the White House and the Congress to do so.
218
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Finally, highway safety performance had far more room for improvement
than aviation safety performance. FAA’s performance plateau coupled, with the
problem of risk perception bias resulting from the catastrophic nature of airplane
crashes, made change more challenging. No other transportation safety organization
shared FAA’s large, decentralized organization with technically diverse tasks. The
other organizations also did not share the technical problem of trying to accomplish
small incremental improvements in a low risk aviation system.
Areas for Further Research
This descriptive search for the underlying theory of DOT’s highly praised
implementation of performance based management suggests a number of follow-on
research areas. These research areas begin with the key elements of the theory
discussed above and with the General Accounting Office’s (GAO) guidance for
implementing GPRA.
In Chapter 1, the GAO guidance for effective implementation of GPRA and
the examples of successful implementation were discussed. These examples
included all organizational levels, from cabinet level departments to operating
administrations and their sub-units. GAO guidance did not recognize the fact that,
for such departments as DOT, multiple organizations that were different in many
fundamental ways contributed to a common strategic outcome goal. Also, in GAO’s
examples, the role of the senior leaders of cabinet level departments was not
219
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
discussed. In particular, an approach such as DOT’s top-down direction and bottom-
up goal setting and strategy development was not discussed in the guidance.
Changes In guidance to incorporate new knowledge about DOT’s approach
would result in a less linear process model. Uses for performance information would
be continuous and reflective rather than proceeding through sequential action steps.
The leadership and facilitation activities at the departmental level would be coupled
with collaborative and systematic learning in the organizations contributing to a
shared outcome. Institutional as well as organizational measures of performance
would be used.
Such a change in guidance would suggest the need for further research on the
relationship of institutional and organizational actions for societal learning. This
relationship is related to the questions asked by Robert Behn (1995) and John Kirlin
(1995). A managerial focus looks for outcome improvement at the organizational
level, and a societal focus looks for improvement at the institutional level. Research
questions to address this split focus include: Is departmental leadership necessary for
effective implementation of performance based management of federal government
outcomes? What are the respective roles of departmental executives and operating
administration program managers in improving national outcomes? How do federal
government partnerships influence performance outcomes?
This question is related to the link between a top-down, bottom-up approach
and learning that Tony Bovaird and David Gregory (1996) found in the British
220
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
government’s administrative organizations. It also raises another question: How do
other countries with performance monitoring systems manage transportation safety at
the national level? Addressing this question would also respond to Beryl Radin’s
(1999) discussion of the effects of institutional setting on implementation of
government performance monitoring systems. In particular, she pointed out that
parliamentary systems provide an “institutional actor with authority to look at the
government as a whole” that is not present in the presidential system (p. 120).
Another aspect of the DOT approach that is not evident in the GAO guidance
is the institutional nature of performance based management at the federal level to
which Beryl Radin refers. This suggests the following research question: How are
goals that require changes in many parts of an institutional network monitored? For
example, how are state and local goals for highway safety and recreational boating
safety performance coordinated with federal goals? How are the contributions of
state and local governments to national performance monitored?
In her dissertation on managing-for-results in the states, Maria Aristigueta
(1997) suggested a question for further research: “What opportunities and examples
are available to encourage community partnerships as a result of performance
information?” It can be restated as a federal level question: How do federal
organizations use community partnerships to improve performance outcomes? This
would address the roles of organizations like NHTSA and USCG in establishing
2 2 1
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
partnerships to promote learning about highway and recreational boating safety as a
means to reduce accidents.
Finally, it would be a valuable contribution to knowledge to explore the
partnerships among federal, state, local, and non-governmental organizations to
improve national outcomes. What forms do these partnerships take, and what are the
pros and cons of different forms? For example, when are memoranda of
understanding more appropriate than contracts? How are performance goals
addressed in such agreements, and how are contributions to performance outcomes
assessed? When such agreements are established, who is accountable and for what?
2 2 2
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
SELECTED BIBLIOGRAPHY
Agranoff, R. & Radin, B. A. (1991). The comparative case study approach in public
administration. In James L. Perry (Ed.), Research in public administration
(pp. 203-231). Greenwich, CT: JAI Press.
Argyris, C. (1976). Increasing leadership effectiveness. New York: Wiley and Sons.
________ (1993). Knowledge for action: A guide to overcoming barriers to
organizational change. San Francisco: Jossey-Bass.
Aristigueta, M. P. (1999). Managing for results in state government. Westport, CT:
Quorum Books.
Armey, D. (1997, August 7). Letter to Franklin D. Raines, Director, Office of
Management and Budget, Subject: Results Act Implementation.
Armey, D., Craig, L., Burton, D., Livingston, B., & Kasich, J. (1997, November).
Towards a smaller, smarter, common sense government: The results act: It’s
the law. The November 1997 Report, available at
http://freedom.house.gov.results.fmalreport.
Baumgamer, James (1999, July 20). FAA to issue rule on voluntary disclosure of
safety data. Aviation Daily. New York: McGraw-Hill Companies.
Behn, R. D. (1995). The big questions of public management. Public
Administration Review, 55 (4), 313-324.
Bok, D. (1997). Measuring the performance of government. In Joseph S. Nye, Jr.,
Philip D. Zelikow, & David C. King (Eds.), Why people don’ t trust
government (pp. 55-74). Cambridge, MA: Harvard University Press.
Borman, Walter C., Hedge, Jerry W., Hanson, Mary Ann, Bruskiewicz, Kenneth T.,
Mogilka, Henry, Manning, Carol, Bunch, Laura B., & Horgen, Kristen E.
(1999). Unpublished paper prepared for the Federal Aviation Administration.
223
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Bovaird, T., & Gregory, D. (1996). Performance indicators: the British experience.
In Arie Halachmi and Geert Bouckaert (Eds.), Organizational performance
and measurement in the public sector (pp. 239-273). Westport, CT: Quorum
Books.
Brunsson, N., & Olsen, J. P. (1993). The reforming organization. New York:
Routledge.
Bureau of Transportation Statistics. About the Bureau o f Transportation Statistics,
available at http://www.bts.gov/aboutbts.htm.
Card, James C. (2000, February 24). Department of Transportation United States
Coast Guard statement on transportation safety initiatives before the
Subcommittee on Transportation, Senate Committee on Appropriations.
Washington, DC, available at http://uscg.mil/hq/.
_______ (1996, March). Speech to shipping ’ 96 conference: New era— new
realities, available at http://uscg.mil/hq.
________(1996, January). Speech at the Chua Chor Teck Conference, Singapore.
Human element in maritime safety and marine pollutions prevention,
available at http://uscg.mil/hq.
Caudle, S. L. (1994). Using qualitative approaches. In Joseph S. Wholey, Harry P.
Hatry & Kathryn E. Newcomer (Eds.), Handbook o f practical program
evaluation (pp. 63-95). San Francisco: Jossey-Bass.
Checkland, P. (1999). Systems thinking, systems practice. Chichester, UK: John
Wiley and Sons, Ltd.
Congressional Institute (1997). Implementing the Government Performance and
Results Act: General findings from a review o f draft agency strategic plans,
available at http://www.Change_Leader@CongInst.org.
Congressional Research Service (1998). Memorandum o f provisions in public laws
and their associated reports from the 105th Congress relevant to
implementation o f the Government Performance and Results Act.
Washington, DC: Library of Congress.
Czamiawska-Joerges, B. (1992). Exploring complex organizations: A cultural
perspective. Newbury Park, NJ: Sage.
224
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Damanpour, F., & Evan, W. M, (1984, September). Organizational innovation and
performance: The problem of “organizational lag.” Administrative Science
Quarterly, 392-409.
Downey, M. (1998, June 30). Remarks prepared for delivery at the National
Academy of Public Administration government performance and results act
conference, Washington, DC.
Faigin, B., Dion, J., & Tanham, R. (1996). The National Highway Traffic Safety
Administration case study: Strategic planning and performance
measurement, Washington, DC: American Society for Public
Administration.
Federal Aviation Administration. (1998, April 14). APA 42-98 FAA news.
Federal Motor Carrier Safety Administration (2000, February). Fiscal year 2001
performance plan, available at http//www.fmcsa.gov/perfplan.
FHWA (1998, July). TEA-21—Transportation Equity Act for the 21s t Century: A
summary, Available at
wysiwyg://21/http:www.fhwa.dot.gov/tea21/sumtoc.htm.
Fischhoff, B. (1994, June 6). Risk perception and communication unplugged:
Twenty years o f process. Paper presented for Symposium to discuss next
steps: Addressing agencies’ risk communication needs, Annapolis, MD.
Garvey, Jane F. (2000, February 3). Statement of the Honorable Jane F. Garvey,
Federal Aviation Administrator, Before a joint hearing of the Senate
Committee on Budget and the Senate Appropriations Subcommittee on
Transportation and Modernization Challenges and Solutions. Washington,
DC.
Glaser, B. G., & Strauss, Anselm L. (1967). The discovery o f grounded theory.
Hawthorne, NY: Aldine de Gruyter.
Goddard, S. B. (1994). Getting there: The epic struggle between road and rail in the
American century. Chicago: University of Chicago Press.
Gowda, M. V. R. (1997, November 6). Heuristics, biases, and risk regulation.
Paper presented at the annual meeting of the Association for Public Policy
Analysis and Management, San Antonio, TX.
225
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Greiner, J. M. (1996). Positioning performance measurement for the twenty-first
century. In Arie Halacfaml & Geert Bouckaert (Eds.), Organizational
performance and measurement in the public sector (pp. 11 -50). Westport,
CT: Quorum Books.
Gummesson, E. (1991). Qualitative methods in management research (revised
edition). Newbury Park, CA: Sage.
Halachmi, A., & Bouckaert, G. (1996). Performance appraisal and Rubik’s cube. In
Arie Halachmi & Geert Bouckaert (Eds.), Organizational performance and
measurement in the public sector (pp. 1-9). Westport, CT: Quorum Books.
Hedley, T. P. (1998). Measuring public sector effectiveness. Public Productivity &
Management Review, 21 (3), 251-258.
Henn, Arthur E. (1995, June 27). Testimony before the U.S. House of
Representatives Subcommittee on Government Management, Information,
and Technology of the Committee on Government Reform and Oversight. In
Intergovernment Accountability, U.S. Advisory Commission on
Intergovernmental Relations (SR-21, May 1996, p. 26). Washington, DC:
U.S. Government Printing Office.
Intermodal Surface Transportation Efficiency Act of 1991. Pub. L. No. 102-240, §
6006.
Kanter, R. M. (1981). Organizational performance: recent developments in
measurement. Annual Review o f Sociology, 7, 321-349.
Kettl, D. F. (1988). Government by Proxy: [Mis?] managing federal programs.
Washington, DC: Congressional Quarterly Press.
Kingdon, J. W. (1995). Agendas, alternatives, and public policies. New York:
Harper Collins College Publishers.
Kirlin, J. J. (1996). The big questions of public administration in a democracy.
Public Administration Review, 56 (5), 416-423.
Kowalewski, R. (1996). A case study o f the Coast Guard’ s pilot project under the
Government Performance and Results Act. Washington, DC: American
Society for Public Administration.
2 2 6
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Laurent, A. (1998, March). Performance anxiety: Can Results Act performance
plans hold up under the spotlight? Government Executive, 20-26.
Legge, K. (1984). Evaluating planned organizational change. London: Academic
Press.
Lewin, K. (1952). Field theory in social science: Selected theoretical papers.
London: Tavistock Publications, Ltd.
Light, P. C. (1999). The true size o f government. Washington, DC: Brookings
Institution Press.
_______ (1992). Monitoring government: Inspector Generals and the search for
accountability. Washington, DC: The Brookings Institution.
(1997). The tides o f reform. New Haven, CT: Yale University Press.
Mahler, J. (1997). Influences of organizational culture on learning in public
agencies. Journal o f Public Administration Research and Theory, 7 (4), 519-
540.
Mann, Paul (2000). Wider safety reporting gets White House nod. Aviation Week &
Space Technology, 40 (4), 50.
March, J. G. (1996). Continuity and change theories of organizational action
Administrative Science Quarterly, 41, 278-287.
Martinez, Ricardo (1998). Address to the Enhanced Safety of Vehicles Conference.
Windsor, Ontario Canada, available at
http://www.nhtsa.gov/nhtsa/announce/nhtsanow/v3.13.
________(1997, May 22). Statement of the Honorable Ricardo Martinez, M.D.,
Administrator National Highway Traffic Safety Administration before the
Subcommittee on Telecommunications, Trade and Consumer Protection,
House Committee on Commerce. Washington, DC: available at
http://www.nhtsa.gov/nhtsa/announce/testimony/96istea.html.
_ _ _ _ _ _ (1996, September 19), Statement of the Honorable Ricardo Martinez,
M.D., Administrator of the National Highway Traffic Safety Administration
before the Committee on Transportation and Infrastructure, Subcommittee on
Surface Transportation. Washington, DC: available at
http://www.nhtsa.gov/nhtsa/announce/testimony/96istea.html.
227
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
McCrimmon, D. R. (1998). The results are good: A follow-up report on the Coast
Guard’ s pilot project under the Government Performance and Results Act.
Unpublished research paper.
Mead, Kenneth M. (1999, September 29). Statement the Inspector General, U.S.
Department of Transportation, before the Subcommittee on Surface
Transportation and Merchant Marine. Washington, DC.
National Civil Aviation Review Commission (1997, May 28). Written testimony,
available at http://www.faa.gov/ncarc/testimony/index.htm.
National Highway Traffic Safety Administration (1996, April). Office o f Strategic
Planning and Evaluation: Transformation activities 1992-1996. Washington,
DC: National Highway Traffic Safety Administration.
Newcomer, K. E. (1997, Fall). Using performance measurement to improve
programs. New Directions for Evaluation, 75,5-14.
Office of Management and Budget 1994, August 1). Ten GPRA Pilot Project FY
1994 Performance Plans. Transmittal letter.
Pressman, J. L., & Wildavsky, A. (1984). Implementation: How great expectations
in Washington are dashed in Oakland (3rd ed.). Berkeley, CA: University of
California Press.
Radin, B. A. (1999). The Government Performance and Results Act and the tradition
of federal management reform: Square pegs in round holes? Journal of
Public Administration Research and Theory, 10 (1), 111-136.
________(1998). The Government Performance and Results Act (GPRA): Hydra
headed monster or flexible management tool? Public Administration Review,
58(4), 307-316.
Rudestam, K., & Newton, R. (1992). Surviving your dissertation: A comprehensive
guide to content and process. Newbury Park, CA: Sage.
Schein, E. (1997). Organizational culture and leadership. San Francisco: Jossey-
Bass.
________(1984). Coming to a new awareness of organizational culture. In L.
Boone & D. Bowen, The great writings in management and organizational
behavior (2nd ed.). New York: McGraw Hill.
228
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Schon, D. A. (1983). The reflective practitioner: How professionals think in action.
New York: Basic Books.
Shulha, L. M., & Cousins, J. B. (1997). Evaluation use: Theory, research, and
practices since 1986. Evaluation Practice, 18 0 ), 195-208.
Skrzycki, Cindy. (1999, November 26). The regulators: Trying to move truck safety
forward. Washington Post.
Slater, R. E. (1998, April 14). Remarks prepared for delivery regarding aviation
safety agenda, available at http://faa.gov.apa/Safer_Skies/dotremarks.htm.
Slovic, P., Fishchhoff, B., & Lichtenstein, S. (1984). Modeling the societal impact
of fatal accidents. Management Science, 30,464-474.
Thompson, F. (1999, August 17). Letter. Subject: Performance plan assessment.
Transportation Research Board (1991, May). Special Report 234: Data for
decisions: requirements for national transportation policy making.
Washington: U.S. Government Printing Office,
U.S. Department of Transportation (1994). National Highway Traffic and Safety
Administration fiscal year 1994performance plan, available at
http://www.dot.gov.
________(1995a). National Highway Traffic and Safety Administration fiscal year
1994 performance report, available at http://www.dot.gov.
_______ (1995b). National Highway Traffic and Safety Administration fiscal year
1995 performance plan, available at http://www.dot.gov.
________(1996a). National Highway Traffic and Safety Administration fiscal year
1995performance report, available at http://www.dot.gov.
______ (1996b). National Highway Traffic and Safety Administration fiscal year
1996performance plan, available at http://www.dot.gov.
________(1997a). National Highway Traffic and Safety Administration fiscal year
1996performance report, available at http://www.dot.gov.
________(1997b). National Highway Traffic and Safety Administration fiscal year
1997performance plan, available at http://www.dot.gov.
229
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
(1998a). National Highway Traffic and Safety Administration Strategic
Plan, available at http://www.dot.gov.
(1998b). National Highway Traffic and Safety Administration fiscal year
1998performance plan, available at http://www.dot.gov.
(1998, July 14). TEA-21 grants information, available at
http://nhtsa.dot.gov/nhtsa/whatsup/tea21progrms/onepage.grt.html).
(1999, December 17). DOT strategic plan FY 2000-2005: Draft core
elements: introduction, vision, mission, strategic goals, outcomes and
strategies, available at http://www.dot.gov.
(1999, November 19), Press release statement upon Senate passage o f
motor carrier safety improvement bill, available at
http://www.dot.gov/affairs/dotl9299.htm.
(1999, December 10). Press release statement upon signing of the Motor
Carrier Safety Improvement Act by President Clinton, available at
http://www.dot.gov/affairs/dotl9299.htm.
(1999, August 18). Statement by U.S. Transportation Secretary Slater
concerning Best Performance Plan. Press release DOT 128-99.
(1999, March 2). U.S. Transportation Secretary Slater calls on nation to
“ sign on for safety. ” Press release DOT 33-99, available at
http ://www.dot.gov.
(1999, February 23). Transportation Secretary Slater announces national
transportation safety conference. Press release DOT 28-99, available at
http://www.dot.gov.
(1999). Performance plan for F Y 2000, available at
http://www.dot.gov.hot/pplanr/html,
(1998). Performance plan for FY 1999, available at
http://www.dot.gov.hot/allper 1 /html#table 1.
(1997). Strategic plan for fiscal years 1997-2002, available at
http://www.dot.gov.hot/dotplan/html.
230
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
________(1999). Safety Council News, available at
http://www.dot.gov/safetycouncil.
________(2000, March). 1999performance report 2001 performance plan, U.S.
Department o f Transportation, available at http://www.dot.gov.
U.S. Federal Aviation Administration (2000, May). Using information proactively to
improve aviation safety. Paper presented by the Office of System Safety at
the Global Aviation Information Network annual conference, available at
http ://wwwasy .faa. gov/gain
U.S. General Accounting Office (2000, June). Highway Funding: Problems with
highway trust fund information can affect state highway funds,
GAO/RCED/AIMD-OO-148. Washington, DC: U.S. General Accounting
Office.
_ _ _ _ _ _ (2000, March 2). Commercial motor vehicles: Significant actions remain
to improve truck safety, GAO/T-RCED-00-102. Washington, DC: U.S.
General Accounting Office.
________(2000, March 17). Truck safety: Effectiveness o f motor carriers office
hampered by data problems and slow progress on implementing safety
initiatives, GAO/T-RCED-00-122. Washington, DC: U.S. General
Accounting Office.
________(July 1999). Agencies ’ fiscal year 2000performance plans,
GAO/GGD/AJMD-99-215. Washington, DC: U.S. General Accounting
Office.
________(1999, June 29). Truck safety: Motor carriers office hampered by limited
information on causes o f crashes and other data problems, GAO/RCED-99-
182. Washington, DC: U.S. General Accounting Office.
_______ (1999, June 18). National airspace system: Review o f FAA’ s spending for
operations, GAO/RCED/OGC/AIMD-99-222R. Washington, DC: U.S.
General Accounting Office.
________(1999, May). DOT’ s fiscal year 2000performance plan, GAO/RCED-99-
151. Washington, DC: U.S. General Accounting Office.
231
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
(1999, March). Truck safety: Effectiveness o f motor carrier office
hampered by data problems and slow progress implementing safety
initiatives, GAO/RCED-99-122. Washington, DC: U.S. General Accounting
Office.
(1998, September). Managing for results: An agenda to improve the
usefulness o f agencies ’ annual performance plans, GAO/GGD/AIMD-98-
228. Washington, DC: U.S. General Accounting Office.
(1998, May). Observations on DOT’ s annual performance plan,
GAO/RCED-98-180R. Washington, DC: U.S. General Accounting Office.
(1998, February). Aviation safety: Weaknesses in inspection and
enforcement limit FAA in identifying and responding to risks, GAO/RCED-
98-6. Washington, DC: U.S. General Accounting Office.
(1997, December). Aviation safety: Efforts to implement flight
operational quality assurance programs, GAO/RCED-98-10. Washington,
DC: U.S. General Accounting Office.
(1997, July). Results Act: Observations on the Department of
Transportation’ s draft strategic plan, GAO/RCED-97-208R. Washington,
DC: U.S. General Accounting Office.
___ (1997, June). The government performance and results act: 1997
govemmentwide implementation will be uneven, GAO/GGD-97-109.
Washington, DC: U.S. General Accounting Office.
(1997, May). Managing for results: Analytic challenges in measuring
performance, GAO/GGD-97-138. Washington, DC: U.S. General
Accounting Office.
(1996, August). Aviation acquisition: A comprehensive strategy is need
for cultural change at FAA, GAQ/RCED-96-159. Washington, DC: U.S.
General Accounting Office.
(1996, June). Executive guide: Effectively implementing the Government
Performance and Results Act, GAO/GGD-96-118 . Washington, DC: U.S.
General Accounting Office.
232
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
________(1996, June). Human factors: Status o f efforts to integrate research on
human factors into FAA’ s activities, GAO/RCED-96-151. Washington, DC:
U.S. General Accounting Office.
________(1996, October). Aviation safety: New airlines illustrate long-standing
problems in FAA’ s inspection program, GAO/RCED-97-2. Washington, DC:
U.S. General Accounting Office.
________(1995). Budget account structure, GAO/AIMD-95-179. Washington,
DC: U.S. General Accounting Office.
________(1989, March). Content analysis: A methodology for structuring and
analyzing written material, transfer paper 10.1.3. Washington, DC: U.S.
General Accounting Office.
Weick, K. E. (1999). That’s moving: Theories that matter. Journal o f Management
Inquiry, 8 (2), 134-142.
________(1995). Sensemaking in organizations. Thousand Oaks, CA: Sage.
Weingroff, R. F. (1994). Highway existence: 100 years and beyond, available at
http://www.tfhrc.gov/pubrds.
Wheeler, D. J. (1993). Understanding variation: The key to managing chaos.
Knoxville, TN: SPC Press.
White House Commission on Aviation Safety and Security (1997, February 12).
Final report to President Clinton.
Wholey, J. S. (1997, September). Managing for results: Performance-based
management in theory and practice. Paper for Symposium on Performance-
Based Management and Its Training Implications, Caserta, Italy.
_____ (1997, May). Quality control: Assessing the quality and usefulness o f
performance measurement systems. Paper presented at the Midwestern
Intergovernmental Audit Forum, Milwaukee, Wisconsin.
________(1996). Formative and summative evaluation: related issues in
performance measurement. Evaluation Practice, 17 (2), 145-149.
Wholey, J. S., & Newcomer, K. E. (1997, Fall). Clarifying goals, reporting results.
New Directions for Evaluation, 75, 91-98.
233
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Wildavsky, Aaron (1988). Searching for safety. New Brunswick, NJ: Transaction
Publishers.
Wilson, J. Q. (1989). Bureaucracy: What government agencies do and why they do
it. New York: Basic Books.
Wykle, Kenneth R., Molitoris, Jolene M., Coyner Kelley, S, Fernandez, Nuria,
Millman, Rosalyn G., & Cirillo, Julie A. (2000, March 8). Statement before
the Subcommittee on Ground Transportation, Committee on Transportation
and Infrastructure, U.S. House of Representatives: Implementation of the
Transportation Equity Act for the 21s t Century, available at
http://www.dot.gov.
Yin, R. K. (1997, Winter). Case study evaluations: A decade of progress? New
Directions for Evaluation, 76,69-78.
________(1993). Applications o f case study research. Thousand Oaks, CA: Sage.
(1989). Case study research: Design and methods. Thousand Oaks, CA:
Sage.
234
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
APPENDICES
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
APPENDIX A
INTRODUCTORY LETTER TO
DEPUTY SECRETARY OF TRANSPORTATION
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Mr. Mortimer L. Downey
Deputy Secretary of Transportation
Department of Transportation
Washington, DC 20590
Sir:
As a doctoral candidate at the University of California’s Washington Center, I am
writing to request your advice and support for completing my dissertation. The
subject is “Performance Measurement In Pursuit Of A National Strategic Objective:
The Case Of Transportation Safety.” I have discussed my research with the DOT
Inspector General, Mr. Meade, who is a former colleague of mine at the U.S. General
Accounting Office. He concluded that you would find this research interesting and
potentially very useful. Your recent commentary in The Public Manager confirms
Mr. Meade’s judgment.
I want to improve knowledge about why federal agencies differ in their ability to
measure performance as a step toward improving outcomes. In particular, what
makes it easier or more difficult for agencies to support a measurement system for a
national strategic objective?
The attached summary outlines my dissertation approach. It concludes with
interview guides and data summary formats to give you an idea of why I need brief
access to the measurement/evaluation staff, executives, modal administrators, and
senior managers in the department. The interviews that I plan will take from fifteen
minutes to half-an-hour. In order to expedite the process, I would be happy to
receive written responses with an opportunity for face-to-face follow-up on critical
points if necessary. As a former program evaluation manager at the Federal Aviation
Administration, I know how limited time is for the department’s senior leaders.
With your support, I can complete my work expeditiously. I also hope that the
knowledge produced can contribute to the department’s continuing leadership in
implementation of the Government Performance and Results Act. Finally, I believe
that it could help GAO, OMB, and congressional members better understand the
challenges facing implementation.
May I discuss my research with you or your staff in the near future? My telephone
number is (703) 243-2518 and my E-mail address is danielb@ids2.idsonline.com. I
look forward to hearing from you.
Beverly A. Daniel
237
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
APPENDIX B
DISCUSSION GUIDE FOR INTERVIEWS
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
DISCUSSION GUIDE FOR INTERVIEWS
1. How do you use performance information about transportation safety?
Specifically,
which of these terms best describes how often you use performance information?
Frequently Occasionally Infrequently Never
For what purposes do you use performance information?
• To identify gaps between goals and accomplishments?
• To report results?
• To support policy decisions, including allocations of resources?
• To support managerial decisions that change operational processes?
• To learn about what causes accidents and how to control the causes?
• For management accountability?
• To enhance my organization’s image?
• To facilitate agreement among conflicting groups involved in
• Other uses not listed above?
2. Would you like to have performance measurement information that you do not
now receive? For what purposes?
3. What are the aspects of program outcomes that your organization finds it easiest
to measure and to provide reports about? Why do you think these things are easier
measure?
4. What are the aspects of program outcomes that your organization finds it most
difficult to measure and to provide reports about? What are the obstacles that have
to be overcome?
5. To what degree do you believe that managers in your organization feel a need to
avoid reporting embarrassing performance information?
High Need Moderate Need Little Need No Need
6. What strategies has your organization used to reorganize and to realign central
management systems to support performance management?
7. Have you received more authority to make decisions about how to manage your
program and its budget based on performance plans or results? Would you like to
have authority based on performance?
239
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
8. Have you received awards or other recognition for developing useful performance
goals and indicators? For achieving results?
9. Does your organization have effective administrative systems to support
collection and analysis of performance information? Are any improvements planned
for the future?
10. To what degree do such competing management reforms as TQM or NPR
activities distract you from planning program goals and using feedback to manage
performance?
High Degree Moderate Degree Very Little Not at All
11. What has been your experience in developing, testing, and changing
performance indicators as your organization gains experience? How near are you to
having the right indicators and the data and analysis support required?
12. How has your organization established linkages with the political hierarchy to
gain a reasonable level of agreement on outcome expectations for you program?
13. Has your organization used partnerships with states, citizens, or industry groups
to collaborate on how to accomplish shared outcomes?
14. What roles do program managers play in designing performance measurement
systems?
15. How does your organization’s performance measurement system support a
variety of uses by different levels of the organization? For example, is work group or
individual employee performance management linked to organizational performance
goals?
16. How has your organization dealt with conflicts over the meaning of performance
or the purpose of measuring performance? Are your stakeholder groups in agreement
on what should be measured and how the information should be used?
17. How well are the causal relationships of accidents understood, and what kinds of
studies are performed on this subject?
18 Are there aspects of performance that you cannot find either a method for
measuring or an indicator that is useful for decisions about how to change
performance?
240
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
19. To what degree do you have the right resources for planning, analyzing, and
managing your organization’s performance?
High Degree Moderate Degree Very Little Not at All
What obstacles have you had to overcome or are you facing? For example have
funding, hiring authority, or access to experts been issues?
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
APPENDIX C
MODEL LETTER TO TRANSPORTATION ADMINISTRATORS
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
Mr. Kenneth R. Wykle
Administrator, Federal Highway Administration
400 T Street, S.W.
Washington, DC 20590
Sir:
I am a doctoral candidate at the University of California’s Washington Center and
would like your advice and support for completing my dissertation. The subject of
my research is how federal agencies use performance information to support a
national strategic objective. I want to improve knowledge about why federal
agencies differ in their ability to measure performance as a step toward improving
outcomes. In particular, how have DOT’s modal administrations overcome obstacles
to using measurement information to improve transportation safety outcomes?
As an initial step in my research, I am requesting interviews with Mr. Downey, the
Deputy Secretary of the Department, the Commandant of the Coast Guard, and the
Administrators ofNHTSA and FAA as well as you. The Federal Highway
Administration’s important role in transportation safety makes your opinions
especially valuable.
The interview will take from fifteen to thirty minutes, and the attached interview
guide contains the relevant questions. In order to expedite the process, I would be
happy to receive written responses with an opportunity for follow-up on critical
points if necessary. With your support, I can complete my work expeditiously. I also
hope that the knowledge produced can contribute to the department’s continuing
leadership in implementation of the Government Performance and Results Act.
I will be contacting your office the week of June 1 to request an appointment at your
earliest convenience. If you have any questions, my telephone number is (703) 243-
2518 and my E-mail address is danielb@early.com. Thank you.
Beverly A. Daniel
Attachment
243
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
APPENDIX D
LIST OF OFFICIALS INTERVIEWED
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.
LIST OF OFFICIALS INTERVIEWED
Davis Balderston, Acting Division Manager, System Safety Engineering and
Analysis, Federal Aviation Administration
Mary Lou Bott, Special Assistant to the Deputy Secretary of Transportation
Robert Browning, Historian, U.S. Coast Guard
Charles Dennis, Ph.D., Manager, Strategic Planning Branch, Federal Aviation
Administration
Jane Dion, Special Assistant to Associate Administrator for Plans and Policy,
National Highway Traffic Safety Administration
Mortimer Downey, Deputy Secretary of Transportation
Christopher Hart, Assistant Administrator for System Safety, Federal Aviation
Administration
Richard Kowalewski, Deputy Director, Bureau of Transportation Statistics,
Department of Transportation
Kenneth Mead, Inspector General, Department of Transportation
Sam Neill, Office of Systems Analysis, Office of Aids to Navigation, U.S. Coast
Guard
William Walsh, Associate Administrator for Plans and Policy, National Highway
Traffic Safety Administration
Gerry Williams, Acting Deputy Director, Office of Budget and Program
Performance, Department of Transportation
Laura Ziff, Program Analyst, Office of Plans, Policy and Evaluation, U.S. Coast
Guard
245
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

An analysis of third -party regulatory audit systems for medical device safety

PDF

Defending against bioterrorism: Lessons from the 2001 anthrax attacks

PDF

Interoperability among complex defense computer -based systems: The Navy case

PDF

Do uses of human resource information technology (HRInT) tools in federal organizations improve their human resource management productivity?

PDF

Internal venturing in public agencies

PDF

Educational reform for private state -approved schools in California

PDF

Care management for the uninsured: A force field analysis of the business case

PDF

Evaluation of professional services consultants in rural government

PDF

A financial audit model for entrepreneurial governments

PDF

An organizational history of the National Aeronautics and Space Administration: A critical comparison of administrative decision making in two pivotal eras

PDF

A configuration study of multiagency partnerships as practiced in Taipei City government

PDF

Getting 'how' and 'why' straight: A critical discourse analysis of the National Partnership for Reinventing Government's ideological discourse on information and communication technologies

PDF

An analysis of health risk selection and quality of care under Medicare fee -for -service and Medicare managed care health care systems

PDF

Information overload: Exploring management of electronic mail

PDF

Federal mentor -protege programs: A pilot study on protege survival and growth

PDF

Bridled boldness: Reengineering the Federal Bureau of Prisons. Lessons learned

PDF

Drug treatment providers' organizational responses to implementation of California's Proposition 36

PDF

A preliminary assessment of the influence of certain variables on the implementation of recommendations by scientific -technical advisory committees

PDF

Assessment of prognostic comorbidity in hospital outcomes research: Is there a role for outpatient pharmacy data?

PDF

Adapting and applying a mission-focused strategic framework for emergency management

Asset Metadata

Creator Daniel, Beverly Ann (author)

Core Title A search for theory: Performance management to improve transportation safety

School School of Policy, Planning and Development

Degree Doctor of Public Administration

Degree Program Public Administration

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag business administration, management,health sciences, public health,OAI-PMH Harvest,Political Science, public administration,Transportation

Language English

Contributor Digitized by ProQuest (provenance)

Advisor Wholey, Joseph (committee chair), Biller, Robert (committee member), Burke, Catherine (committee member), Newcomer, Kathryn (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c16-95783

Unique identifier UC11338178

Identifier 3027710.pdf (filename),usctheses-c16-95783 (legacy record id)

Legacy Identifier 3027710.pdf

Dmrecord 95783

Document Type Dissertation

Rights Daniel, Beverly Ann

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA