University of Southern California Dissertations and Theses

A preliminary assessment of the influence of certain variables on the implementation of recommendations by scientific -technical advisory committees

(USC Thesis Other)

A preliminary assessment of the influence of certain variables on the implementation of recommendations by scientific -technical advisory committees

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content INFORMATION TO USERS
This manuscript has been reproduced from the microfilm master. UM I films
the text directly from the original or copy submitted. Thus, some thesis and
dissertation copies are in typewriter face, while others may be from any type of
computer printer.
The quality of this reproduction is dependent upon the quality of the
copy submitted. Broken or indistinct print, colored or poor quality illustrations
and photographs, print bleedthrough, substandard margins, and improper
alignment can adversely affect reproduction.
In the unlikely event that the author did not send U M I a complete manuscript
and there are missing pages, these will be noted. Also, if unauthorized
copyright material had to be removed, a note w ill indicate the deletion.
Oversize materials (e.g., maps, drawings, charts) are reproduced by
sectioning the original, beginning at the upper left-hand comer and continuing
from left to right in equal sections with small overlaps.
ProQuest Information and Learning
300 North Zeeb Road, Ann Arbor, M l 48106-1346 USA
800-521-0600
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
A PRELIMINARY ASSESSMENT OF THE INFLUENCE
OF CERTAIN VARIABLES ON THE IMPLEMENTATION OF
RECOMMENDATIONS BY SCIENTIFIC-TECHNICAL
ADVISORY COMMITTEES
by
Steven Lee Krahn
A Dissertation Presented to the
FACULTY OF THE SCHOOL OF POLICY, PLANNING,
AND DEVELOPMENT
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PUBLIC ADMINISTRATION
December 2001
Copyright 2001 Steven Lee Krahn
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UMI Number: 3065810
Copyright 2001 by
Krahn, Steven Lee
All rights reserved.
UMI
UMI Microform 3065810
Copyright 2002 by ProQuest Information and Learning Company.
All rights reserved. This microform edition is protected against
unauthorized copying under Title 17, United States Code.
ProQuest Information and Learning Company
300 North Zeeb Road
P.O. Box 1346
Ann Arbor, Ml 48106-1346
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UNIVERSITY OF SOUTHERN CALIFORNIA
SCHOOL OF POLICY, PLANNING. AND DEVELOPMENT
UNIVERSITY PARK
LOS ANGELES, CALIFORNIA 90089
This dissertation, written by
. .S t e v e n . .L e e . .R ealm .................................................
under the direction o f h.i s . . . Dissertation
Committee, and approved by all its
members, has been presented to and
accepted by the Faculty o f the School of
Policy, Planning, and Development, in
partial fulfillment o f requirements for the
degree o f
DOCTOR OF PUBLIC ADMINISTRA TION
Dean
Date
DISSERTATION COMMITTEE
Chairperson
i . f
A . a
/
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
DEDICATION
This dissertation is dedicated to my wife, Sally Ann K. Krahn. Without her
support and encouragement and towards the end, polite prodding, this project
would never have been completed. She helped me to carve out the time to work on
the research and was unfailing in her support during rough moments in the process.
In addition, as my intellectual soul mate, she has always endeavored to “keep me
sharp.” Sally, you have my sincere gratitude and the dedication of this dissertation.
Thank you.
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ACKNOWLEDGEMENTS
First and foremost, I would like to acknowledge the contribution of my
family: Sally, to whom this volume is dedicated, and my children Dan and
Elizabeth, who suffered through ten years with a Dad who was also a student
(between my Masters degree and Doctoral work). When I started this adventure,
Dan and Elizabeth had just begun their education, at its conclusion Dan is in
college and Elizabeth will join him next year. The support that Sally, Dan and
Elizabeth provided was the foundation upon which this success was built.
The members of my peer committee, John Gray, Penny Kingsbury and Jake
Barkdoll were invaluable in helping me frame my thoughts during the development
of my proposal. In addition, work performed during several classes with Joe
Wholey, Cameron Gordon and Vic Kimm laid the foundation for this research; I
highly valued their comments during those formative courses. Special thanks are
due to my Chair, Cameron Gordon who stuck with me through thick and thin and to
Vic Kimm who was consistent in his moral support. Last but not least, 1 wish to
acknowledge the important role played by Joe Wholey and Jake Barkdoll in firming
up the research approach and analytical framework for this dissertation. I owe a
huge debt of gratitude to all of the people mentioned above.
iii
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1 also wish to acknowledge the outstanding support that was provided to me
by the General Manager of the Defense Nuclear Facilities Safety Board (DNFSB),
Mr. Ken Pusateri, and his staff. Their pleasant manner and efficient work made
wading through the almost five thousand pages of documentation reviewed during
this research much less onerous than it would otherwise have been. Thanks are also
due. for their contributions, to Mark Whitaker of the Department of Energy (DOE);
Captain John W. Crawford (USN, Ret.), former Member of the DNFSB; and Glenn
George, a former DNFSB staff member. I also wish to acknowledge the time that
was spent in interviews by the anonymous staff of the DNFSB and DOE
interviewed during this dissertation. Notwithstanding the above support, the results
and conclusions are those of the author; they do not represent the view of either the
DNFSB or the DOE.
Finally, I wish to acknowledge the influence of Captain John W. Crawford,
mentioned above, on my career and this dissertation. He has been a boss, mentor,
co-author, critic and fan for more than a decade. It was his friendship and
challenging nature that launched me on this ten-year voyage through graduate
education—thank you very much Jack.
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
TABLE OF CONTENTS
DEDICATION ................................................................................................... ii
ACKNOWLEDGEMENTS.................................................................................. iii
LIST OF TABLES................................................................................................ vii
ABSTRACT ...................................................................................................... ix
Chapter
I. INTRODUCTION AND SUMMARY ................................................ 1
Introduction .................................................................................... 1
Summary of Prior Research........................................................... 4
Public Administration Context...................................................... 5
Abbreviated Results....................................................................... 7
II. FINDINGS FROM PREVIOUS RESEARCH ................................... 9
Overview ........................................................................................ 9
A Short History of Advisory G ro u p s............................................ 9
An Introduction to Scientific-Technical Advisory
Committees............................................................................. 12
Structural and Environmental Variables Pertinent to
Scientific-Technical Advisory Committees ......................... 18
Empirical Research................................................................. 19
Insights from Related Research.............................................. 21
Conclusion...................................................................................... 23
ffl. METHODOLOGY ................................................................................ 24
Overview of Previous Research and this Study........................... 24
Sampling Procedure....................................................................... 30
Hypotheses...................................................................................... 31
Further Discussion of Research V ariables................................... 35
Multiple-Case Study M ethod......................................................... 39
Case Study Protocol....................................................................... 42
v
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data Collection and Description ................................................... 44
Detailed Content A nalysis............................................................. 46
Qualitative Analysis........................................................................ 49
Quantitative Analysis...................................................................... 50
IV. RESEARCH SETTING AND C A S E S .................................................. 52
The Defense Nuclear Facilities Safety Board (DNFSB) ............. 52
Selection of Recommendations (Cases) To Be Reviewed 54
Four Problematic Recommendations............................................ 56
Recommendation 90-2—Standards ...................................... 56
Recommendation 90-3 and 90-7—Characterization of the
Hanford High-Level Waste T anks...................................... 62
Recommendation 91-6— Radiation Protection..................... 69
Recommendation 93-6— Maintaining Access to Nuclear
Weapons Expertise............................................................... 76
Four Non-Problematic Recommendations.................................... 83
Recommendation 91-5— Power Limits for K-Reactor
Operations at the Savannah River S ite................................ 83
Recommendation 92-2— DOE Facility Representative
Program ................................................................................ 88
Recommendation 92-6—Operational Readiness Reviews .. 94
Recommendation 93-4— Environmental Restoration
Management Contracts......................................................... 99
Overall Summary............................................................................ 104
V. ANALYSIS OF RESULTS..................................................................... 107
Overview ........................................................................................ 107
Hypothesis Testing Results ........................................................... 108
Another Look at the D ata............................................................... 133
Recommendations for Further Research ...................................... 136
SELECTED BIBLIOGRAPHY............................................................................ 138
APPENDICES
A. Interview Outline for Interview with Dr. G. W. Cunningham.
Technical Director, DNFSB .................................................. 147
B. Example Recommendation Data T a b le ........................................ 149
C. Interview Form for Recommendation L ead s................................ 151
D. Statistical Analysis.......................................................................... 156
vi
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
LIST OF TABLES
1. Estimates of the Number of Federal Advisory Groups by Y ear................. 10
2. Federal Advisory Committee Statistics During Fiscal Years
1988-1997 ................................................................................................. 1 1
3. Summary of Interviewee Variable Ratings for Recommendation 90-2 . . . 61
4. Summary of Interviewee Variable Ratings for Recommendation 90-3/7 .. 68
5. Summary of Interviewee Variable Ratings for Recommendation 91-6 . . . 75
6. Summary of Interviewee Variable Ratings for Recommendation 93-6 . . . 82
7. Summary of Interviewee Variable Ratings for Recommendation 91-5 . . . 87
8. Summary of Interviewee Variable Ratings for Recommendation 92-2 . . . 92
9. Summary of Interviewee Variable Ratings for Recommendation 92-6 . . . 98
10. Summary of Interviewee Variable Ratings for Recommendation 93-4 . . . 103
11. Interviewee Rating Results for All C ases..................................................... 106
12. Number of Interest Groups Involved in Non-Problematic and
Problematic C ases.................................................................................... 110
13. Measures of the Degree of Involvement of Interest Groups in
Non-Problematic and Problematic C ases............................................... I ll
14. Quantitative Information from Interviews Concerning Interest
Group Involvement.................................................................................. 112
15. Two Measures of the Influence of Level of Technical Uncertainty
Associated with Non-problematic and Problematic C ases................... 116
vii
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
16. Quantitative Information from Interviews Concerning the Degree
of Uncertainty........................................................................................... 117
17. Measures of the Influence of the Perception of Cost on the
Implementation of Non-Problematic and Problematic C ases............... 120
18. Quantitative Information from Interviews Concerning the Influence of
the Perception of Cost on Non-Problematic and Problematic Cases .. 121
19. Measures of the Influence of Communications on Non-Problematic
and Problematic Cases ............................................................................ 125
20. Quantitative Information from Interviews Concerning the Influence of
Communications on Non-Problematic and Problematic Cases ........... 126
21. Measures of the Influence of Organizational Arrangements on
Non-Problematic and Problematic C ases............................................... 130
22. Quantitative Information from Interviews Concerning the Influence of
Organizational Arrangements on Non-Problematic and Problematic
C ases.......................................................................................................... 131
23. Another Look at the Effect of Variables on the Implementation of
Scientific-Technical Advisory Committee Recommendations—
Statistical Data Concerning Perceived Effect (All Cases) .................... 133
24. Another Look at the Effect of Variables on the Implementation of
Scientific-Technical Committee Recommendations—Aggregate
Number of Citations and Statements (All Cases).................................. 134
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ABSTRACT
This dissertation addresses whether certain variables are associated with the
non-problematic implementation of recommendations from scientific-technical
advisory committees. The research examines the relationship between such a
committee, the Defense Nuclear Facilities Safety Board (DNFSB), and the agency
that it advises, the Department of Energy (DOE). It determines whether the
variables being studied are present and explores the nature and degree of
association between these variables and the non-problematic DOE implementation
of recommendations made by the DNFSB.
The variables chosen for study address environmental and structural factors.
Environmental variables include interest group participation, uncertainty of the
subject matter, and the perception of the cost required to implement the
recommendation. The structural variables are the frequency and nature/tone of
communications between the DOE and DNFSB, and the clarity of organizational
arrangements devised by DOE. The variables were drawn from a review of the
literature on advisory committees. Since the available literature on advisory
committees is sparse, literature from the related fields of interorganizational policy
ix
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
implementation, organizational communications and decision-making theory were
consulted; they provided additional support for the variables that were chosen.
Case studies were developed using eight recommendations made by the
DNFSB; four of which were deemed to have experienced non-problematic
implementation and four having experienced problematic implementation.
Determination of whether a recommendation experienced problematic
implementation was based on criteria developed from the DNFSB's enabling
legislation. After selection of the sample cases, interviews were conducted with
personnel who had been involved in the implementation of the recommendations
and the public record was reviewed to determine the presence of the variables and
the effect that they had on the implementation of the recommendation.
This study determined that the environmental variables evaluated had an
effect on DOE’s implementation of DNFSB recommendations. It also provided
support for the effect of one of the structural variables, organizational
arrangements; communications, however, was only partially supported. Since some
support was found for each of the independent variables, recommendations for
further research are provided.
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER I
INTRODUCTION AND SUMMARY
Introduction
More than forty years ago, in a letter to the Mayor of New York. Luther
Gulick framed a particularly nettlesome issue in modem public management, “how
should a layman in a position of authority weigh scientific evidence and determine
what is sound policy?’* Gulick (1957) provided a partial answer to his own question
by noting that scientific and technical questions needed to be addressed by those
recognized in their field as “top men” (pp. 106-110). Writing at about the same time,
researcher David Brown (1954) addressed the overall subject of public advisory
groups, first in his doctoral dissertation and then in an article in Public
Administration Review. He noted that “Few institutions of government had
multiplied so rapidly . . . ” and, further, that “the advisory board may be the twentieth
century way of coping” with the ever more complex problems of government (1955.
p. 196).
The scientific-technical advisory committee has been developed as a solution
to the problem posed by Gulick. Writing approximately twenty years after his initial
article, Brown (1972) described scientific-technical advisory committees as one of
1
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
several major categories of advisory groups. He noted that: “The scientific and
technical advisory committee has developed in response to the demands of an
increasingly complex society . . . [it] makes available to the organization talents it
would otherwise be unable to acquire in any other way” (p. 336). However. Brown
and the limited number of other researchers in this area of public management have
consistently lamented the paucity of research that has been done regarding scientific-
technical advisory committees (or even advisory groups, in general). For example.
Fremont Lyden (1972) noted that although there was extensive use of such groups,
“in point of fact, we know little about what advisory committees do and how they
relate to governmental machinery” (p. 256).
The use of advisory groups (or committees, as they are often referred to) has
burgeoned since the Second World War. Their use has become so prevalent that, in
1972, Congress enacted legislation to regulate their use. This bill was entitled the
Federal Advisory Committee Act (FACA): it is discussed further in Chapter Two
(U.S. Congress. 1972). The most recent data reported concerning FACA indicates
that just under one thousand such advisory groups are presently functioning in the
federal government and they consumed approximately $178 million in fiscal year
1997 (GSA, 1999, p. 3).
The research presented herein investigates several variables that have been
proposed, in the available literature, as influencing the effectiveness of advisory
groups. Advisory groups are put in place to provide recommendations to various
2
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
agencies and departments in government; one measure of whether these groups are
effective is if their recommendations are practicable and, if so, if they are
implemented in a timely manner. At this point it is important to provide two
definitions. For clarity, as discussed further in Chapter Two, the group to whom
advice is being provided will be referred to as “the advised agency.” Advisory
groups are “effective,” therefore, when their recommendations are understood and
accepted by the advised agency and implemented by the advised agency in a timely
manner.
Using literature review, this research has identified five variables that are
believed to influence advisory group effectiveness. The variables can be divided,
conceptually, into two groups: those that are within the control of the advisory group
and/or the advised agency, known as internal or “structural factors,” and those that
are outside of the direct control of the advisory group and advised agency, known as
external or “environmental variables.” Once these factors were identified, a variable-
based, multiple case study approach was used to assess the influence that these
variables had on the implementation of recommendations. The unit of analysis for
the multiple case study was the individual advisory group recommendation. The
advisory group studied is a scientific-technical advisory committee, the Defense
Nuclear Facilities Safety Board (DNFSB). Eight recommendations were assessed
using analysis of the public record and interviews with principal participants. The
recommendations ended up evenly split between those which were implemented in a
3
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
timely fashion (“non-problematic”) and those which were not implemented in a
timely fashion (“problematic”).
Summary of Prior Research
Sheila Jasanoff noted in The Fifth Branch, which reviewed the work of
scientific-technical advisory committees at the Environmental Protection Agency
(EPA) and Food and Drug Administration (FDA), that the recent explosion of
scientific-technical advice-giving has taken place “against a backdrop of growing
public concern about technological hazards, accompanied by diminished trust in
government” (Jasanoff, 1990, p. 2). One might have expected that a good deal of
research had been done to better understand this phenomenon; however, this has not
been the case. In fact, the amount of research is somewhat sparse, although growing
slowly.
Eight articles or books were identified that addressed the subject of advisory
groups and three dissertations were found on the subject; the research presented
therein was largely empirical in nature. To provide additional theoretical insight,
several related areas of research were investigated—these included the areas of
interorganizational communications and policy implementation; intragovemmenta'
regulation; and decision-making.
The literature directly related to advisory groups was analyzed for variables
that were believed to enhance or impede the effectiveness of these groups. The
variables developed were then compared to the theoretical literature on
4
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
interorganizational relationships, intra governmental regulation and decision-making
to “vet” the variables, so to speak. Based on this literature review, it is hypothesized
that the following five variables will influence the effectiveness of advisory groups,
in general, and scientific-technical advisor}' committees, specifically:
1. The degree of interest group involvement:
2. The degree of technical uncertainty associated with the subject matter of
the recommendation;
3. The perception of the cost associated with implementing the
recommendation:
4. Whether frequent and non-adversarial communications exist between the
scientific-technical advisory committee and the advised agency during the
recommendation; and
5. Whether well-defined roles and responsibilities (or organizational
arrangements) exist within the advised agency during the implementation
of the recommendation.
Public Administration Context
Scientific-technical advisory committees provide policy input to the
government on a wide variety of issues. The committee to be reviewed during the
proposed research, the Defense Nuclear Facilities Safety Board (DNFSB). has a
direct impact on the health and safety of millions of citizens of the United States who
live and work in and near the plants and laboratories of the nuclear weapons
5
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
complex. These plants and laboratories are administered by the Department of
Energy (DOE). On a broader front, scientific-technical advisory committees, through
the advice they provide regarding nuclear safety, environmental protection and the
safety of food and drugs (to name just a few topics), impact the health and safety of
the entire populace. In addition, such committees can provide an alternative to
formal regulation (as discussed further in Chapter Two). Further, they are taking up
a consistently growing portion of the federal budget; their price tag is now
approaching two-hundred million dollars per year. There is clear advantage to be
gained in resolving Lyden's concern, voiced almost thirty years ago. that “we know
little about what advisory committees do and how they relate to governmental
machinery” (Lyden, 1972, p. 256).
Along the lines of the latter half of Lyden’s quote, the importance of
scientific-technical advisory committees can also be understood by placing them
within the “big picture” of public administration. Robert Behn. in a challenging
article in 1995, attempted to delineate what he believed were the “big questions” of
public management. In Behn’s paradigm the importance of advisory group influence
is felt in the “micromanagement” theme and advisory groups have ofien been put in
place to reduce the friction between the Executive and Legislative branches (pp. 315-
318). In the Fall of 1996, John Kirlin responded to Behn’s challenge; he developed a
more elaborate set of “big questions,” ones which transcended public management to
encompass public administration in a broader manner. In Kirlin’s (1996) recast
6
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
structure, the functioning of advisory groups fits most closely under his seventh and
final question; which addresses how the process of societal learning can be
improved, to include “the knowledge of choices available, of consequences of
alternatives, and of how to achieve desired goals” (p. 421). Kirlin describes therein
three of the major roles that scientific-technical advisory committees can and do play.
In this broader context, the topic is not as new; in fact, the overall issue
significantly predates the Behn and Kirlin articles. Probably the most comprehensive
treatment of the overall issue, which has been called “neutral competence.” was
provided by Herbert Kaufman in 1956. He noted that three basic, but competing,
values co-existed within public administration: representativeness, neutral
competence and executive leadership. Each of these values has had their day,
Kaufman notes. Kaufman wrote at a time when it appeared that a change from
neutral competence to executive leadership was taking place (pp. 1057-1059). More
recently initiatives such as the National Performance Review have a decidedly
“representative” flavor. From this viewpoint, it is probably not surprising that the
amount of literature addressing a neutral competence topic, such as scientific-
technical advisory committees, is somewhat thin.
Abbreviated Results
This dissertation showed that the data available from case studies at the
DNFSB supported an association between four of the independent variables and the
non-problematic implementation of recommendations from scientific-technical
7
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
advisory committees. The independent variables for which support was
demonstrated were: the degree o f interest group involvement, degree of technical
uncertainty, the perception o f cost and the clarity of roles and responsibilities in the
organizational arrangements developed in the advised agency. Only partial support
was demonstrated for an association between non-problematic recommendation
implementation and frequent, non-adversarial communications between the
scientific-technical advisory committee and the advised agency.
This study is intended to contribute some initial insights to further the
understanding of the relationship between scientific-technical advisory committees
and advised agencies. By assessing the influence of several variables, some
environmental and some structural, it is hoped that scientific-technical advisory
committees and advised agencies can appreciate when issues may arise, develop
strategies to mitigate any potential issues and, therefore, enhance the chances for
effective implementation of the recommendations that have been made and accepted.
8
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER II
FINDINGS FROM PREVIOUS RESEARCH
Overview
This section will begin with an abbreviated history of advisory groups, in
general, to provide context. It will be shown that such groups have grown from
relative obscurity fifty years ago to a stage where their budgets are approaching the
two-hundred million dollar level and they are the subject of federal regulation. The
subject will then be narrowed to a particular type of advisory group, the scientific-
technical advisory committee, of which it has been said that they have become "an
indispensable aid to policymakers.” Finally, the variables that have been proposed in
the literature as important to advisory group effectiveness will be summarized and
applied to the particular case of scientific-technical advisory committees.
A Short History of Advisory Groups
The use of advisory groups (or committees as they are often referred to) has
burgeoned since the Second World War (see Table 1), so much so that Congress
enacted the Federal Advisory Committee Act (FACA) in 1974 to regulate such
bodies. The Act established standards and uniform procedures for the establishment,
operations, administration, and duration of advisory committees; it also established
9
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the requirement that the President issue and annual report summarizing the number,
purpose, membership, activities, and cost of advisory committees (U.S. Congress,
1974). An abbreviated history of the growth of such bodies can be seen in their
numbers, as shown in Table 1 (pre-1974 estimates are by Brown, subsequent
numbers are from the General Services Administration, GSA. tasked by Congress
with FACA oversight).
TABLE 1
ESTIMATES OF THE NUMBER OF FEDERAL ADVISORY GROUPS
BY YEAR
Year Estimated Number of Advisory' Groups
1938 Less than 100
1955 Several hundred
1972 More than 1,000
1993 1,305
1997 963
The most recent estimate of the number of federal advisory groups has dropped
somewhat, to less than 1,000: this has been due to initiatives started under the
National Performance Review that resulted in Executive Order 12838, which
directed that agencies reduce the number of discretionary advisory committees by
one-third by Fiscal Year 1997. However, expenditures on such groups have not
dropped, as recent spending figures (GSA, 1999), reproduced in Table 2, show.
10
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
TABLE 2
FEDERAL ADVISORY COMMITTEE STATISTICS DURING
FISCAL YEARS 1988-1997
Fiscal
Year
Total Naaiber
of Committees
Number of
Discretionary
Committees
Number of
Congressionally
Mandated
Committees
Numbers of
Committee
Members
Total Costs
(millions)
1989 1042 636 416 22,960 $98.4
1990 1128 721 407 22,391 112.3
1991 1212 783 419 27.580 132.6
1992 1230 767 463 29.020 146.3
1993 1305 833 462 28,317 143.9
1994 1195 739 456 30.446 137.3
1995 1110 643 467 34,823 160.9
1996 1000 530 470 35,329 152.4
1997 963 491 472 36,586 178.0
So it can be seen that, even though actions in accordance with Executive Order
12838 have been able to reduce the number of committees, membership and
expenditures have continued to grow. Even this, however, does not paint the full
picture. Numerous scientific-technical advisory groups have been established as
small, independent agencies by Congress: therefore these advisory groups fall outside
the purview of FACA—agencies such as the National Transportation Safety Board
(NTSB), the Nuclear Waste Technology Review Board (NWTRB), and Defense
Nuclear Facilities Safety Board (DNFSB) fall into this category, with a combined
budget o f more than $80 million in Fiscal Year 1997. These small agencies are
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
similar in structure and intent to advisory committees. Their combined budgets add
to the amount of the federal budget utilized for “advice.”
An Introduction to Scientific-Technical Advisory Committees
Scientific-technical advisory committees play a particularly pivotal role, as
advisory groups go. Jasanoff (1990) has noted, in discussing the scientific-technical
advisory committees of the EPA and FDA, that: “Advisory committees are generally
perceived as an indispensable aid to policymakers across a wide range of technical
decisions.. . . They inject a much-needed strain of competence” (p. 1). Further,
Diane Vaughan (1990), who chronicled the failures of safety committees at the
National Aeronautics and Space Administration (NASA) that led up to the
Challenger disaster, described the crucial role played by scientific-technical advisory
committees:
[The tragedy]. . . sent the nation into mourning and forced a
citizenry ordinarily preoccupied with other matters to confront
again the risks of living in a technically sophisticated age.. . .
Technology is not the only culprit, however. The organizations
that run these risky enterprises often contribute to their own
technological failures, (p. 225)
As will be discussed more fully below, scientific-technical advisory committees are
often put in place, either by Congress or senior Executive Department officials, to
provide a needed external review to agency actions.
In his study of the conflict between the Tennessee Valley Authority (TVA)
and EPA and Nuclear Regulatory Commission (NRC) during the 1970's and 1980’ s.
12
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Robert Durant coined the term “intragovemmental regulation’' to describe this
overall phenomenon. However, his study looked only at formal regulatory schemes
(Durant, 1985, pp. 1-14). More recently, Glenn George (1995) has expanded the
typology of intragovemmental regulation to include the type of health and safety
oversight provided by scientific-technical advisory committees; in discussing the
DNFSB, he noted:
Under most circumstances. Congress exercises [its] oversight
function by holding hearings at which executive branch officials
testify, by setting and reviewing budgets, by commissioning
studies.. . . If legislators . . . continue to believe that agency
actions are significantly out of line with its [Congress’]
preferences, then Congress can take legislative action to “rein in”
the agency. Such action can include subjecting the agency to . . .
intragovemmental risk regulation, (p. 11)
Durant noted that, in essence, intragovemmental risk regulation was an extension of
the notion of checks and balances or as James Madison put it, "Ambition must be
made to counteract ambition” (Rossiter, 1961, p. 322). It was the tyranny of a
Congressional majority that Madison worried about; in the sense illustrated by
intragovemmental risk regulation, one branch of the Executive Branch brings the
actions of another under greater scrutiny, to deter various forms o f malpractice.
Kathleen Bawn (1995) described the environment in which intragovemmental
regulation is bom. She noted that “Congress employs a variety o f strategies to
mitigate the principle-agent problem between itself and the bureaucracy. Delegation
allows better information to be obtained about the consequences o f alternative
policies,. . . ” and she goes on to use safety as a particular example. Bawn then noted
13
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
that Congress will delegate decisions when the gains from agency expertise outweigh
the potential for the agency to diverge from Congressional intent (pp. 62-63).
Matthew McCubbins further refined this concept when he noted that Congressional
delegation is not an ‘'all or nothing decision” (quoted in Bawn, 1995, p. 62). Bawn
continued that the degree of agency independence ‘"depends on the technical
uncertainty about policy consequences as well as uncertainty about the political
environment.” Thus. McCubbins and Bawn focused on procedural controls that
Congress could place on agencies to make delegation less of an “all of nothing
decision.”
Durant (1985) introduced the concept of “intragovemmental regulation.” after
his research on the relationship between TVA and the EPA. as a distinct and
particularly strong procedural control that Congress places on some agencies when
they are concerned about the potential consequences of agency decisions:
It is not uncommon to find the activities of federal agencies can
have the effect of placing them within the regulatory
responsibilities of another federal agency And in extreme
cases, Congress has even deliberately charged “adversarial
bureaucracies” with holding sister agencies accountable to the law
of the land. (p. 4)
George (1995) investigated the phenomena of intragovemmental regulation by
scientific-technical advisory groups. He looked at the National Transportation Safety
Board (NTSB) at the summary level and at the Defense Nuclear Facilities Safety
Board (DNFSB) in detail. Both of these agencies had been “deliberately charged
with holding sister agencies accountable” (pp. 11-12).
14
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Congress is not the only body that discerns the need for the injection of
scientific and technical expertise; frequently, heads of departments and agencies will
deem that additional expertise (from outside the entity) is needed to address
particular problems or classes of problems. The FDA has noted three major
advantages in the formation of scientific-technical advisory committees: (1) it
provided access to the “highest levels of scholarship” in the scientific and technical
communities; (2) in areas where the agency lacked intramural competence, selected
scientific-technical committees permitted “a prompt and responsive effort'*; and
(3) participation by the scientific and technical community improved the credibility
and acceptability of agency decisions, ‘'because the public recognizes that
professional competence and balanced considerations are paramount in the formation
of public policy” (Friedman, 1978, p. 214).
In discussing scientific-technical advisory committees, authors have noted
that they do not all operate the same. However, some generalizations are possible.
George developed a three-tiered model of intragovemmental risk regulation that aids
in understanding some of the differences between scientific-technical advisory
committees. He focused on the tools that such an advisory committee had to force
the advised agency to implement its recommendations. At the top of the model are
formal regulatory groups, such as the Nuclear Regulatory Commission (NRC), whose
recommendations are often issued as rules, with the force of law. In the middle are
scientific-technical advisory' committees where the advised agency is required to
15
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
respond to a recommendation, but is not required to accept them or implement them;
the relationship between the NTSB and FAA and that which exists between the
DNFSB and DOE are consistent with this description. At the lowest tier in George's
model are advisory committees whose advice is sought by agencies, but the advised
agencies are required to neither to accept nor act on the recommendations (George.
1995, p. 15). The majority of advisory groups fall into this last group; however, the
middle group continues to grow; the operational start up of the Chemical Safety
Board to advise Congress, the EPA and Occupational Safety and Health
Administration (OSHA), among others, is the most recent example (U.S. Congress,
1990). Such “action-forcing” advisory committees can provide a viable option to
formal regulation.
The high level processes that scientific-technical advisory committees use to
perform their function are fairly similar. They review scientific and technical
information with respect to a given subject and provide recommendations to the
advised agency (as it has been defined above). It is the advised agency’s
responsibility to either accept or reject the recommendations of the committee and to
implement them. For example, in the specific instance of the DNFSB, the process is
a multi-step one (described in more detail in Chapter Four). The DNFSB issues a
recommendation to DOE and if the DOE accepts the recommendation (and all 40
issued by the DNFSB, to date, have been accepted) the Department develops an
implementation plan which is reviewed and approved by the DNFSB. After
16
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
completion of the actions described in the implementation plan the DOE
recommends closure of the recommendation and at that point the DNFSB can either
agree to close the recommendation or request that DOE take additional actions (U.S.
Congress, 1988). This model of policy-making has been analyzed by Laurence
O’Toole and Robert Montjoy (1986); they describe it as the sequential-reciprocal
model and note, in part, that it requires substantial coordination between the entities
(pp. 493-495). Their results are discussed further below.
Jasanoff (1990) is quick to note that scientific advice is not a panacea: it is.
though, a way to .. harness the collective expertise of the scientific community so
as to advance the public interest” (p. 250). The researchers who have specifically
reviewed the role played by advisory groups, in general, and scientific-technical
advisory committees in particular, have pointed to deficiencies in the basic
understanding of how such advisory groups function. Brown (1955) has noted that
“no two advisory groups are alike in their purpose, organization or effectiveness"
(p. 196). Lyden (1972) bemoaned the “apparent waste of valuable resources"
engendered when advisory groups were not provided sufficient support (p. 257).
Finally, Jasanoff (1990) stated that the “... need for better advisory mechanisms has
frequently been noted in works dealing with policy-making for science” (p. 5).
17
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
Structural and Environmental Variables Pertinent
to Scientific-Technical Advisory Committees
Two inter-related streams of literature were evaluated for information
pertaining to scientific-technical advisory committees. The existing empirical
research regarding such committees, and advisory groups in general, was reviewed
for variables that have been associated with apparently effective or ineffective
performance. Little theory building has been done, to date, in either the field of
scientific-technical advisory committees or in the area of advisory groups, more
generally. Therefore, guided by discussion with my peers and professors, and by the
references quoted by the empirical research, I expanded my literature review into
several related fields of study.
Three areas, in addition to intragovemmental regulation, appeared closely
related. Since these committees provide input on policy implementation, this field of
literature was reviewed and a particularly useful article by O'Toole and Montjoy
(1986) was identified that addressed the topic of policy implementation involving
multiple organizations. Organizational communications were mentioned by seven of
nine empirical research authors; several pertinent books and articles were identified
that addressed the theoretical aspects of this subject. Finally, the recommendation
process closely resembled the decision-making process, as investigated by Michael
Cohen, James March and Johan Olsen (below); therefore, applicable literature in this
field was reviewed.
18
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
Empirical Research
Ten authors were identified to have addressed the either the subject of
advisory groups in general, or scientific-technical committees specifically. These
references included three dissertations and eight journal articles and/or books. In
reviewing this literature, the variables mentioned appeared to logically divide
themselves into two categories: variables that were largely within the control of the
advisory committee and the advised agency and those which were apparently external
to the relationship between the two groups. In their research on multi-agency policy
implementation, Provan and Milward (1995) also identified this type of dichotomy,
the internal variables they referred to as “structural” and the external variables,
“environmental.” This nomenclature will also be adopted herein (p. 27).
Environmental variables are those issues that pertain to the relationship
between a scientific-technical advisory committee and an advised agency, but which
are not directly controlled by either. Two such variables were mentioned by the
authors on a consistent basis: “political” input and uncertainty (which was sometimes
described as complexity). Three of the authors addressed the subject of political
input. They described the impact various powerfully situated interest groups can
have on the deliberations and recommendations of advisory groups. Robert
Friedman (1978) noted that advisory group recommendations that were rejected by
the FDA were most often those that were “politically charged” (p. 210). Durant
(1985) described the impact that an interested President, Congress and public could
19
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
have on the intragovemmental regulatory scheme (p. 134). Finally, Vaughan (1990)
noted the impact that a scientific-technical advisory committee might have had if it
had resorted to its special relationship with Congress (p. 250). Four of the authors
(Turner, 1976 p. 378; Vaughan, 1990, pp. 228-229; Jasanoff, 1990, p. 8; and Arthur
Wilson, 1983, pp. 302-303) described the role that uncertainty can play in the
relationship between an scientific-technical advisory committee and an advised
agency; in summary, they noted that significant uncertainty, where it existed,
complicated advisory proceedings.
Structural variables were mentioned more frequently by the empirical
researchers. In fact, all but one of the authors commented on organizational
attributes such as roles and responsibilities and operating processes. In general, they
discussed the need for: a well-defined relationship between the advisory group and
regulated agency, and operating processes that were mutually agreed upon. Seven
authors discussed the importance of communications between an advisory group and
the advised agency. Attributes that were deemed to be important included the
frequency with which interactions took place and the tone of the interactions
(adversarial or non-adversarial).
A final variable was identified that was initially difficult to place in either the
environmental or structural category. This variable was the perceived cost of
implementing recommendations. A recommendation could be costly to implement
(or perceived to be so) for any number of reasons. For example, the approach that
20
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the advisory committee recommended or the advised agency decided upon could be
an unnecessarily costly one, or the problem could be so pervasive that addressing it
would be costly simply due to the extent of the problem (it should be noted that this
list of possibilities is not exhaustive). Durant (1985) adds a note of caution when he
observes that costly regulatory decisions (i.e.. recommendations in advisory group
model) were not uniformly opposed by the advised agency (pp. 116-117). However,
interviews with several principals at DOE and the DNFSB led to the conclusion that
this variable should be grouped with the environmental variables for the purposes of
this study.
Insights from Related Research
To provide additional insight, the research from the areas of decision-making,
interorganizational communications and interorganizational policy implementation
was investigated for support for each of the variables identified. As will be discussed
below, fairly broad support was found for each of the factors identified by the
empirical research. Three or more authors (out of the five reviewed) addressed each
of the five factors. The emphasis of the five authors was as follows.
Structural Issues: OToole and Montjoy (1986) discuss the various
relationships that can exist between two organizations involved in interorganizational
policy implementation. They discuss environmental factors to a degree (mentioning
both political/interest group input and uncertainty); however they focus most of their
attention on the structure of the relationship between the two organizations. They
21
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
discuss the critical role of communications as they explain the role o f high
coordination costs in policy implementation; they also note, regarding organization,
that important factors include the nature of mandated interaction and the number of
decision points in the interaction process established. Cost is also alluded to when
O’Toole and Montjoy note: “Resources will play a key role in interorganizational
implementation” (p. 499). Cohen, March and Olsen (1972), in describing their
famous “Garbage Can” model o f decision-making, explicated an environment that
closely matches that faced by scientific-technical advisory committees; in so doing
they noted that organizational structure was one of the four factors determined to
have a substantial effect on decision-making in such environments (p. 16).
Environment: Jeffrey Pfeffer and Gerald Salancik (1978), in their book on
the impact of environment on organizations, noted that an organization’s
effectiveness depended, in a material way, on its ability to manage the demands of
important interest groups; summarizing as follows: “Establishing a coalition large
enough to ensure survival is an organization's most critical activity” (pp. 25-26).
Richard Osborne and James Hunt (1974), who specifically looked at the impact of
environment on organizational effectiveness, commented on the uncertainty or
complexity of the organization’s environment; they noted that uncertainty “is
considered an important, if not the most important, variable in the environment
surrounding an organization” (p. 245). Provan and Milward (1995), who looked at
22
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the interorganizational implementation environment surrounding the mental health
community, commented on the majority of the variables used in this study
(pp. 26-27).
Three of the articles also addressed the subject of cost. O'Toole and Montjoy
(1986) probably stated the impact most succinctly: “Resources will play a key role in
interorganzational implementation” (p. 499). Provan and Milward (1995) further
note that the adequate funding is “critical for maintaining an effective system”:
however, they caution, “high funding alone is insufficient” (p. 26).
Conclusion
The history of advisory groups, in general, and scientific-technical advisory
committees, specifically has been reviewed. They have been shown to be a
significant part of policy-making in the United States. However, the effectiveness of
such groups, which appear to be taking an ever-larger share of the federal budget, has
not been exhaustively researched. The research reported herein provides a basis for
research design. Both environmental and structural variables have been discussed
that are believed to be important to the effectiveness of advisory groups such as
scientific-technical advisory committees.
23
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER III
METHODOLOGY
Overview of Previous Research and this Study
Some research on scientific-technical advisory groups has concluded that the
recommendations of these groups are taken seriously by the agencies receiving them
(numbers in the 75-99 percent acceptance rate have been quoted (Friedman, 1978,
p. 210). On the other hand, several detailed case studies have shown that specific
scientific-technical advisory groups displayed limited effectiveness. Also, some
potential factors that can make recommendations problematic have already been
mentioned; these have included, for example: political sensitivity, high
implementation costs, and confrontational deliberations. The limited research that
has attempted to focus on the workings of scientific-technical advisory committees
(and advisor)' groups in general) from the standpoint of effectiveness, is discussed
below.
Two dissertations, three articles and one book have been identified which
were written with the goal (at least partially) of looking into advisory groups and
their effectiveness. Three of these efforts used survey methods to assess
effectiveness; the others used detailed review of the public record along with some
24
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
interviews to draw inferences concerning issues associated with the environment and
decision-making structure that these groups work in and the influence that they have
on advisory group recommendations and policy decisions.
Two of these studies dealt with advisory groups in the education field; so I
will discuss them first before turning to the work that is more directly applicable to
scientific-technical advisory committees. In a short article in 1972, Lyden addressed
advisory groups as a potential vehicle for public participation in government; in that
article, however, he quoted survey results that indicated that ‘"few of the respondents
believed their committee contributed significantly to the policy process.” His
database was survey information from 26 advisory groups to the (then) Department
of Health, Education and Welfare (HEW) in the area of education. The survey data
spanned the time period 1966-1969; it showed that less than 25 percent of
respondents thought that the committees they worked on were effective (p. 257).
Joan Givens (1987) looked into one specific advisory group, the National Advisory
Committee on Accreditation and Institutional Eligibility, during the period 1981 to
1986. Her research tool was a survey instrument that asked whether three groups of
people (committee members, Department of Education staff and accreditation
officials) believed that this advisory committee was effective in reaching its goals
and what factors contributed to the effectiveness (or lack thereof). Her results
showed mixed effectiveness in achieving the goals of the committee, with some
goals being generally accepted as achieved and others, as not achieved. Her
25
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
interview results developed a short list of factors important to advisory group success
which are subsumed in the factors to be used for this study. In a few cases, success
factors were directly associated with survey results; however, most were not
(pp. 105-106).
The research directly into scientific-technical advisory committees will be
handled chronologically in what follows. Friedman (1978) looked into the FDA's
use o f scientific-technical advisory committees using a survey instrument
administered to committee members, FDA staff and interest group officials. These
results showed that the general reaction to FDA’s system of advisory committees was
“quite favorable"; 89 percent of all respondents rated the system satisfactory or
higher. As regards effectiveness:
Staff respondents were unanimous in stating that F.D.A. took
advisory committee advice seriously. When figures were given,
the frequency with which committee recommendations were
accepted and adopted ranged from 75 percent to 99 percent.. . .
When experts speak . . . they speak with authority and their views
cannot be neglected . . . the agency would have to be well armed
to overrule such recommendations, (pp. 210-211)
The staff did note (as discussed in the empirical results section, above) that political
and cost considerations could lead to the rejection of advisory committee
recommendations. Wilson (1983) looked at one committee in detail and assessed the
policy implementation o f its recommendations (as indicated by direct quotation or
authoritative reference). He found that the effectiveness of the National Advisory
Committee on Oceans and the Atmosphere was limited, but did not draw direct
26
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
inferences as to factors that caused this lack of success (pp. 304-307). Some
hypotheses were forwarded, however, which have been taken into consideration in
framing the factors to be assessed in this study. The results o f these two studies, as
regards effectiveness, are therefore, equivocal; however, they did develop some
hypotheses that are helpful in framing independent variables for further research.
The final two studies, those of Vaughan and Jasanoff, involved in-depth
ethnographic descriptions of the decision-making processes at EPA, FDA and
NASA. Scientific-technical advisory committees play significant roles in the
decision-making processes at all of these agencies. However, effectiveness of the
scientific-technical advisory committees was not the primary focus of either of these
studies. In Jasanoffs (1990) case, she sought to better explicate the role of science
in policymaking, in general (pp. 1-5). Vaughan (1990) was attempting to provide an
alternate, or additional, explanation for the decisions leading up to the loss of the
space shuttle Challenger (p. 225). While both o f these studies provide substantial
background information for the present study, neither had, as a focus, how scientific-
technical advisory committees can be more effective in their primary role as makers
of recommendations to advised agencies.
As can be seen from the above, it is believed that some amount of
background information has been developed to begin to study the effectiveness in
scientific-technical advisory committees. However, significant gaps exist and they
can be summarized as follows. The two studies which specifically looked for
27
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
advisory group effectiveness did not find it. In general, they were left groping for
reasons that effectiveness was absent. In fact, failures have been analyzed in detail
(e.g., Vaughan and most of Jasanoffs cases), but specific successes are less well-
researched (e.g., Friedman provided only very general survey results and
effectiveness was not Jasanoff s primary focus). Finally, a number of factors that are
believed to increase advisory group effectiveness have been advanced, but their
explanatory value, as regards effectiveness, has not been assessed.
To address the deficiencies in the literature mentioned above, and to extend
the boundaries of the review of scientific-technical committees to a new agency, the
following overall approach has been taken in this study. The overall framework of
this study is a variable-based, multiple-case study (as described by Huberman &
Miles, 1998, pp. 179-210; Yin, 1993, pp. 5-12), one that specifically evaluates the
presence or absence of a set of five variables derived from the available literature.
The unit of analysis is a single scientific-technical advisory committee
recommendation from the Defense Nuclear Facilities Safety Board (DNFSB). The
assessment includes analysis of the public record for the recommendation for the
presence of the variables of interest and interviews with the DNFSB and DOE (the
advised agency) personnel who led, or were closely associated with, the
implementation efforts for the recommendation, to assess the influence that these
variables had on the implementation of the recommendation. A total of eight case
studies are involved. The cases are evenly split between recommendations that were
28
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
determined to be problematic and those that were evaluated as non-problematic,
based on the criteria described below.
It is important to note, however, at this point, that the author was employed at
the DNFSB from 1991 to 1998; and further, was the Deputy Technical Director and
Chief Operating Officer for the final 18 months of that time period. However, the
author was only directly involved in one of the eight recommendations that make up
the sample studied. It is intended that several independent reviews, discussed below,
will address any potential biases that may have been introduced based on the above-
mentioned familiarity. On the positive side, having what Patton (1990) describes as
a “skilled observer,” is also an advantage (p. 72). The work of scientific-technical
advisory committees is often technical in nature. Therefore, having a researcher with
specific technical competence (and the author has more than 22 years of experience
in the field of engineering and nuclear safety) is important to understanding the
communications between a scientific-technical advisory committee and the agency
that it advises.
In the next chapter, the case studies are summarized individually to provide
the context in which the variables will be analyzed. In Chapter Five, the detailed
analysis of the variables is performed. This approach is consistent with the
recommendations of Michael Patton (1990):
Description must be carefully separated from interpretation.. . .
Each case study in a report stands alone, allowing the reader to
understand the case as a holistic entity. At a later point in analysis
it is possible to compare and contrast cases, (p. 375)
29
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
With the general flow of the remainder of this study established, the data collection,
management and analysis methods used in the study will be more fully described.
Sampling Procedure
The scientific-technical advisory committee that will be studied is the
DNFSB. This committee presents several advantages for use in this analysis. The
recommendations that they make are required, by law, to made public. Similarly, the
actions taken by the DOE, by way o f acceptance and implementation of
recommendations are also required to be placed on the public record. Therefore,
there is oflen a rich public record (running in some cases to the thousands of pages)
in existence regarding actions taken in regard to DNFSB recommendations. Further,
the subject matter of the recommendations, nuclear safety, is one that the author is
very familiar with—having worked in and around it most of his professional career.
Since the subject matter dealt with by scientific-technical advisory committees is
often very technical in nature, this will permit the research to more easily focus on
effectiveness issues.
Limiting research to a single advisory committee also has potential
drawbacks, especially from the standpoint of generalizability. Evaluating a single
agency has the potential to limit the utility of the results due to, for example,
legislative or organizational peculiarities (to name just a few reasons). Two methods
will be employed to address this potential weakness; first, the structural and legal
context of the DNFSB will be explicated, so that differences between it and any
30
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
future research in the area can be easily understood; and second the structure of the
review, which will be looking at both “successes” and “failures,” as it were, will
partially alleviate some of the agency-specific nature of the study. The agency-
specific nature of the study will be further ameliorated by the general definition of
the factors being reviewed. In short, it is anticipated that factors, which have been
gleaned from the available literature, will be assessed as to their potential influence,
based on this particular instance, and that a framework will have been suggested for
further research into the effectiveness of scientific-technical advisory committees.
The sample of recommendations that has been reviewed is a purposive one.
Purposive sampling is common in multiple case studies, where one is trying to
discern trends from a limited number of instances. In this application, DNFSB
recommendations will be evaluated against two criteria to present “extreme cases,”
that is, cases that were clearly problematic and those where implementation
progressed more smoothly (GAO, 1990, pp. 40-49).
Hypotheses
The hypotheses listed below were developed through systematic review of
pertinent literature. The following steps were utilized: (1) reviewing the available
literature on scientific-technical advisory committees and advisory groups (in
general), using a grounded research mode to determine the attributes deemed
important to advisory group success; (2) these attributes were then grouped into
higher-level, more general factors; (3) since the theoretical literature on advisory
31
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
groups is somewhat limited, these factors were then vetted by comparison to insights
from theoretical literature in the related fields of interorganizational
communications, decision-making and interorganizational policy implementation;
and, finally (4) some iteration among the above steps, along with peer review, was
required to reach the final set. Dividing the factors into environmental and structural
groups is consistent with the approach taken by Provan and Milward (1995) when
they analyzed the complex implementation environment associated with regional
mental health care delivery (pp. 27-28). The hypotheses to be tested by this
dissertation are discussed below.
1. Environment
a. Interest Group Involvement
Discussion; This factor addresses the number of outside interest groups
involved in the deliberation and implementation process and the degree or
nature of their involvement. Major outside interest groups could include:
government organizations other than the scientific-technical advisory
committee and the advised agency, trade associations, citizen action groups
and the courts. These groups provide input that must be addressed and that
can often significantly impact the deliberation and/or implementation process.
Hypothesis: The greater the degree o f interest group involvement, the
more problematic will be the implementation o f that recommendation.
32
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
b. Uncertainty
Discussion: The theoretical authors often discuss “complexity,” as
opposed to uncertainty; however, it is believed that the meaning of the term,
in context, is similar. Complexity makes the collection and interpretation of
information for decision-making purposes difficult. Uncertainty describes a
situation where there is, for whatever reason, insufficient information or
information of questionable quality for decision-making purposes. For the
development of this factor, the two are considered sufficiently similar in their
impact on the decision-making processes of scientific-technical advisory
committees and their advised agencies to be addressed together.
Hypothesis: The greater the degree o f technical uncertainty associated
with the subject o f the recommendation, the more problematic will be the
implementation o f the recommendation.
c. Cost
Discussion: Much of theoretical discussion revolved around the need to
adequately fund interorganizational policy implementation; the focus was on
the advised agencies having the wherewithal to execute mandates. Overall it
was noted that “resources play a key role in interorganizational
implementation.” On the empirical front, Friedman (1978) noted that
33
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
recommendations that were “a matter of considerable financial consequence”
were among those more likely to be rejected in his studies of the FDA
(p. 210).
Hypothesis: The more costly that a recommendation is perceived to be,
the more problematic will be its implementation.
2. Structure
a. Communications
Discussion: This factor addresses the nature and content of interactions,
both formal and informal. In a complex environment, communications can be
the most important part of the implementation process, according to Osborne
and Hunt (pp. 235-236). Important elements of this factor can include the
frequency of communication and the nature or tone of communications
(adversarial or non-adversarial).
Hypothesis: Frequent and non-adversarial communications between a
scientific-technical advisory committee and the advised agency will make
implementation o f a recommendation less problematic.
b. Organizational Arrangements
Discussion: Organization arrangements, as a variable herein, describes
the mechanisms and structures put in place by advised agency to define the
relationship between the two entities and to attempt to provide consistent
results. As regards the relationship, it is postulated that roles and
34
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
responsibilities need to be well-defined and understood. To improve the
predictability or consistency of implementation results, processes are
established. These processes, in the case of recommendations by the DNFSB,
center around the implementation plan developed by the DOE (and approved
by the DNFSB) and the manner in which it is implemented
Hypothesis: Clearly defined roles and responsibilities within the advised
agency will make the implementation o f a recommendation less
problematic.
Further Discussion of Research Variables
Non-problematic implementation is the dependent variable in this research.
This descriptor was chosen to avoid the potentially emotion-laden impact of terms
such as “success” and “failure” in discussing the implementation of
recommendations, especially with personnel who were directly involved. Two
attributes were used to determine problematic or non-problematic nature of
completed recommendations of the DNFSB: timeliness and constancy of approach.
Both of these attributes are drawn for the enabling legislation of the DNFSB.
Congress anticipated that the recommendations of the DNFSB would be
implemented by DOE in about one year after the acceptance of the implementation
plan by the DNFSB; the one year goal has rarely been met. Therefore, in this study
the measure for timeliness will be the median of the time periods for the
implementation of completed recommendations. Further, Congress set forth a simple
35
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
structure that anticipated that DOE and the DNFSB would be able to agree on an
implementation path that would take the recommendation through to completion.
Multiple implementation plans, therefore, also are an indicator of problematic
recommendation implementation (U.S. Congress, 1988, pp. 7-10). Taken together,
these attributes were used to define the terms problematic and non-problematic, as
they will be used in this study with regard to implementation of DNFSB
recommendations:
Non-Problematic: a recommendation that was implemented in the median
amount of time (or less) and which was implemented in accordance with its
initial, approved implementation plan.
Problematic: a recommendation that was implemented in greater than the
median amount of time and which required one or more implementation plan
revisions prior to its completion.
These two definitions have been established to identify “extreme cases,” to use the
language of the GAO; that is, not all completed recommendations fit into these
categories and, in fact, the are somewhat exclusive (GAO, 1990, p. 44). It is. in
effect, a combination of “best case” and “worst case” purposive sampling methods.
It is postulated, for example, that the consistent presence of a factor in non
problematic recommendations and its absence in problematic recommendations
would be an indication of good variable explanatory power.
36
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The five independent variables that are hypothesized are briefly described in
the statement of the hypotheses, above. Some additional discussion is provided
below, for clarity.
1. Interest Group Involvement. An interest group’s participation during the
implementation of a recommendation was identified through two means,
it may have been obvious from the public record (letters, statements at
public meetings, etc.) or it could have been mentioned by interviewees
during the course of an interview. The presence of the interest group was
noted and the record analyzed for extent of involvement. The number of
interest groups involved were summed by recommendation during
qualitative analysis.
2. Uncertainty. For the purposes of this study, this variable combines
attributes of a recommendation environment commonly referred to as
‘"complexity” and “uncertainty” as discussed above. Uncertain
environments were characterized by either insufficient information or
conflicting information upon which to make decisions. Reference to
scientific or technical uncertainty or conflicts in available information
was noted in the public record and interviewees were asked to comment
of the impact of uncertainty upon the discourse during the
recommendation.
37
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
3. Cost. During the course of recommendation implementation it was the
author’s experience that detailed cost estimates were rarely performed.
That is why the investigation of this factor will center around the
perception of cost. References to cost (albeit without quantification)
were found in the public record and impressions of the impact of cost on
the dialogue regarding the recommendation remained with the
interviewees.
4. Communications. This factor was identified during interviews rather
than during the review of the public record; this was because a lack of
communications or adversarial conditions would be difficult to ascertain
from evaluating the formal communications that make up a large majority
of the public record. However, the record was searched for indication of
the number of meetings that occurred (for example, as referenced in
letters or testimony). In addition, the language used in communications
was assessed to determine if it appears adversarial in nature. Important
confirmation of these impressions came from the interviews.
5. Organizational Arrangements. Mini-organizations within DOE are
established to implement each DNFSB recommendation; these are
normally described in the implementation plan for the recommendation.
This variable will investigate the efficacy of the arrangements that were
put in place to manage the implementation of the recommendations
38
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
studied. The implementation plan for a recommendation puts in place a
specific set of decision points and execution processes. This variable
also evaluates the efficacy of these plans; the public record lays out the
plan for these processes, if they did not work out as planned, this was
often mentioned in the interviews. Information regarding problems in
executing processes was unearthed more frequently, though, during the
course of the planned interviews.
Comparison of the independent variables with the dependent variable was
accomplished thought the construction of tables on a case by case basis; when the
eight case studies had all been completed, variable summary tables for non
problematic and problematic cases were constructed. This enhanced the ability to
draw associations across cases.
Multiple-Case Study Method
In introducing his description of case study design, Yin (1994) took the
subject of the adequacy of case study methods head-on, he stated: “The case study
method has long been stereotyped as a weak sibling among social science methods”
(p. xiii). Yin, the General Accounting Office (GAO) and others have materially
improved the methodology for case studies, so that now they represent a well-
developed methodology (GAO, 1990, p. 19). However, are they the right tool for the
proposed study? To answer this question, reference is made once again to Yin
39
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(1994): “In general, case studies are the preferred strategy when 'how’ or ‘why'
questions are being posed, when the investigator has little control over events, and
when the focus is on a contemporary phenomenon within some real-life context”
(p. 1). In the proposed study we seek to understand “how” scientific-technical
advisory groups function, “how” they are impacted by their environment, and ‘"why”
some recommendations that they make are readily implemented, while others face
protracted and problematic implementation or outright rejection. Therefore, a case
study approach appears appropriate.
This dissertation uses a multiple case study design; this adds to the robustness
of the research design. As discussed by the GAO (1990), the generalizability o f the
results of case study research is less dependent on the number of cases studied than
on the right match between the methodology of case selection and the diversity of the
samples selected (pp. 22-28). Since the primary focus of this research is the potential
impact of the identified independent variables on the non-problematic
implementation of scientific-technical advisory committee recommendations, a
variable-oriented, cross-case analysis approach was selected. The detailed content
analysis will attempt to determine whether qualitative differences exist when
citations from the public record and statements from interviewees are subjected to
content analysis on a variable by variable basis. Available quantitative data will be
used in an attempt to develop further understanding of the explanatory power o f the
independent variables (Huberman & Miles, 1998, pp. 194-196).
40
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
Yin (1994) has aggressively pursued the subject of quality assurance when
using the case study method. He describes four potential threats to the validity of
case study research, along with recommendations on how to deal with these threats.
Yin's framework is used below to discuss research quality considerations associated
with case study designs and how they are addressed in the design of this present
study (pp. 32-38).
Construct Validity: This threat is aimed at ensuring that correct operational
measures are developed for the issues being studied. The case selection methodology
used in this study, which is called “best/worst” case or “extremes,” is one of the
strategies recommended by Caudle (1994, pp. 69-95) and the GAO (1990, p. 43) for
case study evaluations that address implementation issues. As recommended by Yin.
Patton and GAO, multiple sources of evidence have been incorporated: interviews
will be used to provide an initial determination of variable influence; the public
record, including agency and advisory group documentation, will be used to provide
triangulation, along with some limited statistical analysis. In addition, a clear chain
o f evidence will be maintained to ensure that one can refer back to the source for
each piece of data used.
Internal Validity: The rigorous data analysis methods recommended by
Barney Glaser and Anselm Strauss (1967), along with Patton, have been used to
ensure proper associations are developed. These include both inductive methods and
deductive methods. Specific tactics include content analysis of the public record and
41
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
interview summaries using theoretical coding (pp. 21-44). In addition, an
independent reviewer, familiar with the subject matter and research methods,
reviewed a sample of the content analysis, as suggested by Caudle (1994. p. 91).
External Validity: This attribute, also known as generalizability, is enhanced
by the use of multiple-case study designs. The definition of the variables and
research question were used to guide the development of the case study protocol; this
provides a replication logic that will also contribute to external validity.
Reliability: This attribute is enhanced by close quality control during the data
collection phase. Reliability has been enhanced by the use of a single researcher, one
who is well versed in the operation o f the particular scientific-technical advisory
committee that is under review, a factor that Patton (1990, pp. 472-477) and Caudle
(1994, p. 91) stress. Also, the process of data collection was controlled via a case
study protocol, discussed further in the next section. The interviews were conducted
using a standardized, open-ended interview format (Patton, 1990, pp. 284-287). In
addition, the data developed during interviews, document reviews and assessment of
the public record has been catalogued and maintained in a case study database.
Finally, the case summaries that have been developed were provided to personnel
from the DNFSB and DOE for comment and validation.
Case Study Protocol
The proposed research was initiated by an open-ended interview with the
Technical Director of the DNFSB, Dr. G. W. Cunningham. During this interview the
42
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
thrust of the research was discussed, the list of recommendations to be reviewed was
vetted and the interview instrument to be used with DNFSB and DOE personnel was
reviewed. In addition, a trial interview was performed with an experienced DNFSB
staff member. His comments were documented. Adjustments were made to the
interview instrument, as appropriate, based on the feedback received. Following
these initial interviews, the following protocol was followed:
1. Review of the detailed public record;
2. Scheduling of the initial interview with the DNFSB staff lead;
3. Conduct interview and obtain any additional documentation;
4. Conduct interview of DOE personnel;
5. Review of notes taken during interviews;
6. Review and detailed content analysis of interview notes and public
record;
7. Theoretical memoing, based on analysis of the public record;
8. Draft case summary;
9. Provide case summary to DNFSB and DOE for review; and
10. Finalize case summary.
Some of the above steps were, at times, performed in parallel, as frequently happens
in qualitative analysis. However, the above protocol summarizes the path that each
case study followed.
43
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data Collection and Description
An initial interview was conducted with the career head of the DNFSB, the
Technical Director, as discussed above. This interview: validated the list of potential
“extreme case” recommendations for case study review and identified the DNFSB
leads for implementation of the chosen recommendations. An open-ended format
was used for this interview. (See Appendix A.)
Prior to interviewing the DNFSB lead, a comprehensive review of the public
record for the recommendations was performed. This preliminary content analysis
was planned to support the overall thrust of the research; that is, to determine if the
set of variables hypothesized do, in fact, influence the implementation of a DNFSB
recommendation. Therefore, first the public records was reviewed to determine
whether reference to a given variable was “present” to use the GAO terminology
“[a]nalysis sometimes focuses on the mere presence of a variable in a document”
(GAO, 1996, p. 27). Standard tables were developed that listed all of the documents
in the public record, then in subsequent columns, citations that address the five
hypothesized variables were recorded. (For an example page from these tables, see
Appendix B.)
Following this preparation, a standardized, open-ended interview was
performed with the DNSFB personnel. The instrument used for this interview can be
found in Appendix C. The goal of the interview was to get these people (who were
very closely involved with the implementation of the recommendation) to discuss
44
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
their perceptions of the influence of the five variables, discussed above, on the
recommendation or recommendations that they had been associated with. These
personnel were then asked to summarize their impression of the influence of each of
the variables on the implementation of the recommendation by rating its strength on
a scale of 1 to 5. (See Appendix C.) This procedure was then repeated with DOE
personnel who had been closely associated with the implementation of the
recommendation. These interviews were performed in the first half of 2000. The
majority of the interviews were conducted face-to-face; however, due to the
geographic separation of some of the personnel, three interviews were conducted by
telephone, supplemented by E-mail. In all, twelve senior personnel were interviewed
and the interviews took approximately one to two hours each. Another four
interviews were conducted with two senior officials (one from each agency) to
confirm the accuracy of the case summaries developed from the interviews and to
review the results of the subsequent qualitative and quantitative analysis. (See
discussion of “key informants” below.)
The interviews were followed by a preliminary analysis of the public record
and interview notes. In summary, the public record and interview results were
analyzed separately. The results of this analysis were qualitatively compared with
the summary ratings provided during the interviews of the DNFSB and DOE
personnel in each case summary.
45
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The following outline was used for the case summaries:
1. A short narrative history of the recommendation,
2. Discussion of the influence of the five variables, and
3. A summary.
The results of the interviewee ratings for all eight recommendations are presented at
the end of Chapter Four in what Huberman and Miles describe as a “meta-matrix,"
that is, a table that displays multi-case data in one comprehensive chart.
Detailed Content Analysis
The preliminary content analysis, already described, provided an initial
indication of the influence that the variables had upon a given recommendation. It
was, however, only a rough and relative indication. In discussing the methodology
for content analysis, Robert Weber (1990) states: “Communication is a central aspect
of social interaction," and, further, that it is valuable to “reveal the focus of...
institutional. . . attention and describe trends in communication content” (p. 10).
There is ample reason, therefore, to further explore the data that can be mined from
the information gleaned from the larger public records by the first round of content
analysis, along with responses provided to the open-ended questions posed during
interviews (described above).
The first task that was addressed was how to establish the categories to be
used in the analysis of the content of the portions of the public record that addressed
the variables. Patton (1990) has noted: “Such constructions [categories] must be
46
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
done with considerable care to avoid creating things that are not really in the data”
(pp. 402-414). One differentiation that is made is between a “top-down” approach to
the development of content categories as opposed to a “bottom-up” approach. A top-
down approach is defined by Peter Manning and Betsy Cullum-Swan (1998) as one
where the investigator begins with a structure and “sets out to exhaust the meaning of
the text” using a set of rules based on that structure; on the other hand, a bottom-up
approach derives what are called context-dependent categories (pp. 249-250). The
approach used in this study combined the two methods. Some categories were
suggested by the definition of the independent variables being studied, while others
suggested themselves during repeated evaluations of the text.
From an overall standpoint, the methodology suggested by Weber (1990) was
used: the recording unit was defined as a “theme,” these represented a focused
statement from and interview or a coherent citation from the public record (which
could encompass several sentences); the data for a given variable was evaluated,
noting potential categories; the categories were then refined to ensure mutually
exclusivity; the text was coded and then reviewed to assess reliability; then a final
review was performed to ensure that the entire text could be coded (pp. 21-22).
Completion of categorization was not the final step; the categorization was then
reviewed with senior officials from the DOE and DNFSB to assess the validity of the
categories used.
47
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
As a further refinement, the coded statements and citations were assessed for
their potential relationship to recommendation implementation. Weber (1990)
suggests a simple grading methodology: positive, neutral and negative (pp. 73-75).
To perform this evaluation step the content categories were reviewed for attributes
that could further distinguish the statements and citations. As an example, in the
content category of “relative level of interest group involvement,” the attributes of
high or low levels of involvement were used (neutral results were not further
analyzed). This refinement to the categories produced additional information
regarding the associations within content categories and the variables that they
described.
Patton (1990) has provided some criteria to be used to determine whether the
proper categories have been applied to a content analysis. He notes that: the
categories should be internally homogeneous and externally heterogeneous; that the
set of categories should be reasonable inclusive of the data; the set should be
reproducible by another competent researcher; and that the set of categories should
be credible to those persons who provided the data—in this case the DNFSB and
DOE personnel contacted (p. 404). The first two attributes mentioned by Patton
were included in the reviews of the content analysis performed by the author. In
addition, the first and third attributes were included in the criteria assessed by an
independent reviewer (Dr. Glenn George) when he reviewed a sample of the
qualitative analyses performed for this study. Finally, throughout the research for
48
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
this study the author made use o f two “key informants,” one each at the DOE and
DNFSB. Patton describes key informants as “people who are particularly
knowledgeable and articulate, people whose insights can prove particularly useful”
(pp. 95-96). For the purposes o f this study a member of the Senior Executive Service
tasked by the Secretary of Energy with managing the interface with the DNFSB was
chosen from DOE. This individual has more than ten years experience working with
the DNFSB from within DOE. From the DNFSB a former Presidential appointee on
the DNFSB (recently retired) performed the function of key informant; he had served
eight years at the DNFSB and had previously spent 15 years in senior positions
within DOE. These key informants reviewed the categories used in the content
analysis, the interviewee ratings and the case summaries developed therefrom.
Qualitative Analysis
The qualitative analysis of data for this study has been built on two fronts.
The first front includes a qualitative summary across the cases. A preliminary
summary of associations that presented themselves after cursory review of the eight
cases was developed. It is presented at the end of Chapter Four.
The detailed content analysis of interview statements and citations from the
public record was then performed to determine if any patterns could be discerned
with respect to the five independent variables. This pattern-matching, as described
by Yin (1993), was used to see if qualitative support existed for the hypotheses
(pp. 106-110). On the basis of the consistency of the patterns observed with
49
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
predicted outcomes (e.g., hypotheses), and their consistency with available
quantitative insights, the aggregated data from the case studies has been assessed for
conclusions. Tabular presentations, consistent with guidance provided by Miles and
Huberman, assisted in identifying potential patterns. The major thrust of the analysis
has been to identify those variables (environmental and structural), that appear to be
associated with the non-problematic implementation of recommendations.
Quantitative Analysis
In an attempt to provide further information regarding the degree to which the
five independent variables were associated with the dependent variable (non-
problematic implementation of recommendations), a limited statistical analysis was
performed. Medians were determined for each variable, and for both groups of
recommendations (problematic and non-problematic). These provided one indicator
for comparison purposes. In addition, a statistical test, appropriate for use with
ordinal data (the Mann-Whitney U-test), was used to determine if there was further
evidence of any differences between the data for problematic and problematic
recommendations, on a variable-by-variable basis (Bernstein & Bernstein, 1999,
pp. 389-392). Further details of this calculation are provided in Appendix D.
For illustrative purposes, overall medians were also developed for each
variable. These overall variable medians helped to provide a relative indication of
variable influence. This inference was drawn by comparing the overall variable
50
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
medians for the five variables. The description of these results is provided at the end
of Chapter Four, they are further discussed at the end of Chapter Five.
51
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER IV
RESEARCH SETTING AND CASES
The Defense Nuclear Facilities Safety Board (DNFSB)
In the late 1980’s, a number of public health and nuclear safety issues at
aging defense nuclear facilities operated by the Department of Energy (DOE) led
Congress to create the DNFSB (U.S. Congress, 1988). The DNFSB is composed of
five “respected experts in the field of nuclear safety,” along with associated career
technical, legal and administrative staff (about 100 personnel, all told). It is
empowered to oversee the DOE's programs for the management of health and
nuclear safety at its defense nuclear facilities.
The law establishing the Board requires the Board to review and analyze
facility and system design, operations, practices and events and make
recommendations to the Secretary of Energy that are necessary to ensure adequate
protection of public health and safety (U.S. Congress, 1988, p. 2). In making its
recommendations, the DNFSB must consider the technical and economic feasibility
of their implementation. In reviewing the recommendation for acceptance, the
Secretary of Energy is also required to determine the technical and economic
feasibility of implementation and to report to the President and Congress if the
52
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
implementation o f a recommendation is deemed by the DOE to be impracticable
because of budgetary considerations.
Since its inception in 1989. the DNFSB has made 41 recommendations to the
Secretary of Energy, as of July 2001. All of the recommendations have been
accepted. An understanding of the recommendation implementation process is
important to comprehending the relationship between the DOE and the DNFSB.
After having deliberated and determined the need for a recommendation, the DNFSB
issues it to the Secretary of Energy. The Secretary has 45 days to accept or reject the
recommendation (as noted above, none of the recommendations has been rejected, to
date); after acceptance, an additional 90 days is provided for DOE to develop an
implementation plan (since all recommendations have been accepted, the process for
resolving disputes between the DNFSB and the DOE regarding the cost and
feasibility of a recommendation will not be discussed here). The implementation
plan is then submitted to the DNFSB, which reviews it for technical adequacy and
either approves it or returns it to DOE for revision. DOE’s execution of the
implementation plan for the recommendation follows DNFSB approval. After all of
the actions in the implementation plan have been completed, DOE recommends
closure of the recommendation—which must be approved in by the DNFSB.
The DNFSB recommendation, DOE acceptance letter, and DOE
implementation plan are all published in the Federal Register. Complete records of
all documents exchanged (e.g., deliverables committed to in implementations plans)
53
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and correspondence associated with each recommendation are maintained by the
DNFSB. In addition, the DOE maintains a web-site that has all major documents
pertinent to the recommendations made by the DNFSB. This official record of
recommendation implementation can entail more than a thousand pages; it formed
the “public record” reviewed in this study.
If a recommendation takes greater that one year to implement (after approval
of DOE's implementation plan), DOE must report to Congress on the status of its
actions regarding the recommendation and the reason for the protracted nature of the
implementation (U.S. Congress, 1988, p. 4). Thus, the statute establishing the
DNFSB, thus, established two criteria that can be used to judge recommendation
implementation; an implementation plan must be approved by the DNFSB and is
expected to be completed in a timely manner. As discussed below, these criteria
were used to screen recommendations for use as case studies herein.
Selection of Recommendations (Cases) To Be Reviewed
To select the recommendations to be used as case studies, all
recommendations that had been closed as of June 1999 were reviewed. First, since
review of the public record was to be a principle source of information,
recommendations that did not have any appreciable public record were not
considered further. Then, as discussed above, two screening criteria were used: the
number of implementation plans that had been required to implement the
recommendation and the time that was required to complete its implementation.
54
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
Recommendations were split into three groups: those that required a single
implementation plan and were completed in the median time period (or less) for
completed recommendations (defined as non-problematic); those that required two or
more implementation plans and were completed in greater than the median time
period required for implementation or greater (defined as problematic); and those
that did not meet either of these criteria. The first two groups mentioned formed the
recommendations considered as case studies for this research.
By June 1999,26 of the recommendations issued by the DNFSB had been
closed (DNFSB, 1999b). Review indicated that four of these recommendations had
no appreciable public record (for example, two had only a recommendation and
acceptance letter, and were implemented without a plan). Using the screening
criteria discussed above, the remaining recommendations were reviewed. It was
determined that the median time period required to implement a recommendation, up
to that point, was 1,785 days. Using this information, and the documented number of
implementation plans that had been issued, it was determined that four of the
remaining recommendations met the screening criteria for being problematic and
four recommendations met the criteria for being non-problematic. From this
information, it was determined that a total of eight recommendations would be
assessed.
The recommendations chosen were discussed with Technical Director of the
DNFSB, during a planned interview. During this interview the nature of the
55
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
recommendations that had screened for inclusion was also discussed with the
DNFSB’s Technical Director. He agreed that both groups formed a representative
sample of the DNFSB’s recommendations; one in each group applied to a single
DOE facility, while three had more „omplex-wide applicability. In addition. DNFSB
staff leads were identified for interview purposes.
The eight case studies are summarized in two groups below. In each
summary, an overview of the recommendation is provided first, then the results of
the interviews and content analyses for each of the five variables are briefly
described. The problematic recommendations are discussed first, followed by the
non-problematic recommendations and a short summary discussion follows.
Qualitative and quantitative analysis of the results, over all of the cases, is provided
in the next chapter.
Four Problematic Recommendations
Recommendation 90*2—Standards
Overview: This was the second recommendation issued by the DNFSB, in
March 1990 (DNFSB, 1990a). In this recommendation the DNFSB noted that there
was “a large degree of variability in the level of detail” in the then existent DOE
standards and that there was a “lack of uniformity among [them] as to whether they
were mandatory, non-mandatory, or referenced for information.” The subject of the
standards used in nuclear safety applications was of fundamental interest to Congress
in establishing the DNFSB; therefore, it is not surprising that it is one of the first
56
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
subjects addressed by the Board (U.S. Congress, 1988, p. 2). Overall, the DNFSB
recommended that DOE: (1) identify the specific standards that it considered
applicable to its facilities; (2) assess the adequacy of those standards from the
standpoint of nuclear safety; and (3) determine the extent to which the
recommendations had been implemented at 18 priority facilities. The DNFSB
further noted that they believed it was necessary for DOE eventually to accomplish
this review for all defense nuclear facilities under its purview.
Problems with addressing this recommendation began early. The Secretary of
Energy requested a rare 45-day extension to the time period allotted for accepting a
recommendation. He noted:
I agree with the Board that the adequacy of existing DOE
standards . . . is in need of a thorough review;. . . [however,]
partly because of the nature of information required to comply
with the Board's recommendations, the Department is unable at
this time to provide a comprehensive response. (DOE, 1990b)
In finally accepting the Board’s recommendation 45 days later, the Secretary would
state: “As I am sure the Board is aware, the effort to reconstruct, identify, and
appropriately update DOE’s site-specific nuclear facility standards presents a
considerable challenge to us over the next several years" (DOE, 1990c). The ensuing
public record bears out the Secretary's prediction.
The implementation plan for this recommendation plan went through five
revisions over the space of four years. Efforts early on were directed at the task of
simply identifying the standards that were deemed by DOE to apply to its defense
57
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
nuclear facilities. The Board took issue with the comprehensiveness of the DOE’s
review, across all three sub-recommendations (mentioned above); in addition, the
DNFSB noted that participation in the program within DOE had been limited
(DNFSB, 1991b). These Board concerns, along with others voiced in direct
interchanges, led to the first revisions to the implementation plan (DOE, 1991b).
After review of the revised implementation plan, the DNFSB issued a letter that took
substantial issue with the revised plan; over the course of eight pages of comments,
the Board stated that the actions committed to in the plan “still do not fully address
some of the substantive items recommended by the Board” and that it was
“particularly concerned because the revised plan follows nearly two years of effort”
(DNFSB, 1992a). Subsequently, the next revision was closer to the DNFSB
expectations, the Board noting (for example) that it was “pleased to note that this
plan is improved over earlier versions,” and further revisions were less contentious
(DNFSB, 1992c). The public record for the recommendation exceeded 1,200 pages
before it was closed by the DNFSB in May 1996.
Interest Groups: Several groups were mentioned as having a particular
interest in this recommendation during interviews. An interviewee noted that
Congress had mentioned the issue of standards to the Board as part of their
confirmation hearings in the Senate. Further, it was noted that the DOE contractors
industry group, the Energy Facilities Contractors Operators Group (or EFCOG), was
active in lobbying DOE for various approaches. Finally, the public record reveals
58
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
that six different public interest groups wrote to the DNFSB commenting on the
recommendation. The letters from interest groups reflected an overall agreement
with the need for operations at DOE defense nuclear facilities to be more firmly
based on industry standards. However, at least several of the groups were doubtful
of DOE’s commitment to change (NRDC, 1990).
Uncertainty: The amount of uncertainty surrounding this recommendation
was described as “big” and “mammoth” by the DOE and DNFSB personnel
interviewed (respectively). The confusion with the recommendation, as described by
the interviewees, started with defining precisely what the subject matter was, and
continued thorough determining which standards applied in a given situation and
whether compliance existed (especially based on the lack of clarity with respect to
records). It was frequently noted, by DOE (1990d), that the complex of defense
nuclear facilities was comprised of facilities that, by and large, were 30-50 years old,
and further:
As a result of isolation from commercial nuclear power and other
industries [due to secrecy concerns], modem practices and
standards were not addressed or adopted as they became
available. These are some of the reasons a well-documented body
of codes and standards has not been maintained for DOE’s
defense nuclear facilities.
Cost: The DOE and DNFSB personnel interviewed agreed that a good deal of
discussion occurred regarding how much the implementation of this recommendation
could potentially cost; however, only one formal cost estimate was mentioned. This
did not stop generalizations such as noting, 'The magnitude of the task of
59
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
reconstructing the design bases” making it clearly “a long term effort” (DOE, 1990e).
Often DOE’s site operating contractors, rather than DOE itself, were the source of
this discussion, according to the DOE interviewee. However, the situation was
probably best summarized by the Secretary of Energy in his letter that accepted the
recommendation; as discussed above, he noted that the implementation of the
recommendation presented a “considerable challenge” (DOE, 1990c).
Communications: Although the DOE interviewee rated the influence of this
variable relatively low (“Some Effect”) and the DNFSB interviewee rated it higher
(“Great Effect”), the interviewees both described the communications surrounding
the recommendation as antagonistic or contentious. Review of the public record
indicates that although the communications associated with this recommendation
began on a cordial note, soon the Board was describing portions of DOE’s approach
as “unacceptable” (DNFSB, 1992e) and DOE was committing to “resolving the
remaining issues in a responsive and timely manner” (DOE, 1993b).
Organizational Arrangements: This variable garnered substantial coverage
in the public record and high ratings from both interviewees. The DOE person
interviewed noted that, throughout, roles and responsibilities and the personnel
involved were not well defined. The DNFSB interviewee noted that the Board was
continuously asking the question “Who spoke for DOE?” This caused the DNFSB to
write at one point that there was not a clear understanding of even a fundamental
question such as the role that line and staff organizations should play in the
60
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
implementation of the recommendation (DNFSB, 1992a). In addition, substantial
discussion regarding the proper roles and responsibilities of the two organizations
was engendered in the public record by the dialogue concerning the many different
implementation plans offered by the DOE (DNFSB, 1993e).
Summary:
TABLE 3
SUMMARY OF INTERVIEWEE VARIABLE RATINGS FOR
RECOMMENDATION 90-2
Rating
Variable
Little or
No Effect
Some
Effect
Moderate
Effect
Great
Effect
Very Great
Effect
Interest Groups B E
Uncertainty E, B
Cost B E
Communications E B
Organizational
Arrangements
E B
Note: The rating provided by the DOE interviewee is indicated by "E,” and
that provided by the DNFSB interviewee is indicated by “B.”
The first thing that can be observed from Table 3 is that all five variables
were rated as having a moderate impact (or greater) on the interactions between the
DOE and the DNFSB during the implementation of Recommendation 90-2, by at
least one of the interviewees. This was the only recommendation for which this
occurred. From this elevated level, one variable stood out, having a rating of “Very
61
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Great Effect” from both interviewees, it was uncertainty. The DNFSB interviewee
noted that there was a “confused state of what the recommendation was even about.”
Regarding another variable that received at least one rating of “Very Great Effect,”
organizational arrangements, it was noted by both interviewees, and in the public
record, that it was difficult to know who was in charge within the DOE.
The three variables that received at least one mark of “Moderate Effect.” or
less, are noteworthy nonetheless. This is because the ratings received by the interest
groups and cost factors were high relative to all of the other recommendations—they
were both the second highest ratings for the factor, overall. This indicates that,
relative to the other recommendations studied, these factors appeared to have had a
disproportionate impact on the implementation of this recommendation. With regard
to communications both interviewees agreed that communications were problematic
and the words of the public record bear this out.
Recommendation 90-3 and 90-7—Characterization of the
Hanford High-Level Waste Tanks
Overview: Recommendation 90-3 was issued just three weeks after 90-2
(above); it also addressed an issue that had been brought to the DNFSB's attention
during confirmation hearings for the five Board members (DNFSB, 1990b). This
issue was, as described in the recommendation, ‘the susceptibility of the old single
shell high level waste tanks to an explosion of a spontaneous nature, with resulting
release of large amounts of radioactive material to the environment.” To address this
62
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
concern, even though the DNFSB believed the probability of an explosion to be
“low,” they recommended a that: (1) a study be undertaken to better understand the
chemical reactions that were possible within the tanks; (2) the DOE develop a
program for monitoring the conditions in the tanks; and (3) an action plan be
developed to describe the actions needed in the event that monitored conditions
indicated a dangerous situation. Further, the DNFSB noted in this recommendation
that it was still evaluating issues associated with hydrogen generation in the other
class of high-level waste tanks at Hanford, the double-shelled tanks, where
potentially explosive mixtures of hydrogen were of concern.
In its response, DOE did not take issue with the Board’s stated concerns.
They stated: “The Department agrees with the need to accelerate and expand its
programs to address [high level waste, or HLW] safety issues.. . . ” However, the
DOE noted that, due to the technical uncertainty regarding the safety of some tanks,
“... the Department cannot implement immediately one recommendation . . .
regarding single shell HLW tanks at Hanford that contain ferrocyanide.” Problems
concerning the uncertainty of the contents in several tanks at Hanford would be an
ongoing concern throughout this recommendation (DOE, 1990a).
In October 1990, the DNFSB issued Recommendation 90-7, which expanded
and elaborated upon Recommendation 90-3. It noted that, after review of DOE’s
implementation plan for Recommendation 90-3, “the Board has concluded that the
DOE proposed implementation plan for Recommendation 90-3 is not adequately
63
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
responsive. In particular, it does not reflect the urgency that the circumstances
merit.. . . ” Recommendation 90-7 added the potential for generation of flammable
gases, as hinted in Recommendation 90-3. to the list of concerns that the DOE
implementation plan needed to address (DNFSB, I990d). DOE did not take issue
with this new recommendation.
Not counting the revision required to address the issuance of
Recommendation 90-7. the implementation plan for this joint set of
recommendations underwent three revisions prior to the closure in December 1996.
The concerns that occurred which led to the need for implementation plan revisions
were often schedule-driven; as early as the first status report for the
recommendations, the Secretary of Energy would note: “A number of milestones
require minor adjustments in schedule. [Hanford] is also projecting significant
slippage of several milestones, which is of major concern to the Department of
Energy” (DOE, 1991a). The regular status reports issued by DOE and the complex,
technical nature of the subject matter of the recommendation, along with the
numerous reports required by DOE’s implementation strategy, led to a public record
that exceeded 2.300 pages.
Interest Groups: Both the DOE and DNFSB interviewees noted strong
external stakeholder interest in this set of recommendations. First and foremost, it
was noted by the DNFSB interviewee, that “it was Senator Glenn’s comments about
the ferrocyanide problem in the Hanford Tanks that caused the Board initially to
64
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
evaluate the issues.” Further, the DOE interviewee noted that a pre-existing
agreement existed between the Environmental Protection Agency (EPA), the
Washington State Department of Environment and the DOE regarding corrective
actions to be taken with regard to the these tanks; the waste in the tanks was mixed
waste and as such was subject to EPA regulation under several environmental
statutes. In addition, the Hanford Advisory Board, a citizen’s group financed by the
DOE for the purposes of community outreach, followed the progress in tank
sampling and characterization closely (as noted by DOE and DNFSB personnel
interviewed). The public record noted inputs from two private citizens when the first
recommendation was issued. In addition, during the implementation of the
recommendation. DOE often mentioned the State of Washington as a stakeholder
(DOE, 1990f).
Uncertainty: In describing the genesis of this joint set of recommendations,
both the DOE and DNFSB interviewees pointed to the uncertainty associated with
the contents of the high-level waste tanks. As the DNFSB interviewee stated: “By
their very nature these recommendations dealt with uncertainty. It was the
uncertainty of characterization information available for the waste in the tanks that
led directly to the recommendations.” This uncertainty was compounded by the
difficulty in getting samples that would be adequately representative of tanks whose
capacity was measured in the millions of gallons and waste that varied in
composition from liquid to solid (and all forms in between) (DOE, 1990d). These
65
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
problems were the foremost contributors to uncertainty; they led to this
recommendation receiving the highest overall ratings for uncertainty.
Cost: Cost was an ever-present issue on these recommendations, as noted by
both interviewees. Waste characterization costs were budgeted separately by DOE,
so their cost was highlighted, the DOE interviewee noted. This budget, even though
safety-related, was subject to the same pressures as other budget items and ‘i t was
clear that if more money had been available, more characterization could have been
performed . .. ” and the actions required by the recommendation completed earlier (as
noted by the DNFSB interviewee). Even so, both DOE and DNFSB personnel
agreed “it was very hard to argue with the safety aspects” of the recommendations, so
the cost issue was not raised too often in the public record. Overall, the ratings given
to cost were generally in line with those on other problematic recommendations.
Communications: It was noted by the DNFSB interviewee, “that
Recommendation 90-7 was issued at all was indicative of poor communication.
DOE was wholly unresponsive, both in substance and in timeliness, in answering
Recommendation 90-3, requiring the Board to issue 90-7.” This led the Secretary of
Energy to talk directly with the management of the contractor involved, “I have
discussed the importance of the Hanford High-Level Waste Tank safety issues with
senior [contractor] executives and received their commitment for managerial and
technical support” (DOE, 1990f). On the other hand, regular quarterly status reports
were credited by both organizations as leading to frequent and open dialogue
66
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
regarding issues as they arose. Overall the ratings for the impact of communications
were in line with those for other the other recommendations studied.
Organizational Arrangements: On the DOE side, alter the initial problems
that led to Recommendation 90-7 and direct intervention by the Secretary of Energy
(discussed above), a focused project team was established to address the issues raised
by the DNFSB recommendation, as discussed by the DOE interviewee. However,
frequent turnover in these positions was credited by the DNFSB interviewee with
slowing progress. The placement of senior, technically competent managers in
positions of responsibility in both the DOE and contractor organizations was noted
by both the DOE and DNFSB interviewees as being critical to finally getting the
corrective actions committed to by DOE completed. The frequent reporting
performed under this recommendation led to substantial formality. Not only was
there a regular, quarterly report issued; the public record shows that an individual
report was issued as each task was completed. This led, especially in the latter years
of implementation, to a consistent stream of formal reports being issued. The
process for developing these reports (especially the task-specific, technical reports)
involved substantial interaction between the DOE and DNFSB staffs prior to
finalizing each report, as noted by the DNFSB interviewee. Overall, the ratings for
this variable were among the lowest garnered by any of the eight recommendations.
67
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
Summary:
TABLE 4
SUMMARY OF INTERVIEWEE VARIABLE RATINGS FOR
RECOMMENDATION 90-3/7
Rating
Variable
Little or
No Effect
Some
Effect
Moderate
Effect
Great
Effect
Very Great
Effect
Interest Groups E, B
Uncertainty E, B
Cost E B
Communications E B
Organizational
Arrangements
B E
Note: The rating provided by the DOE interviewee is indicated by “E,” and
that provided by the DNFSB interviewee is indicated by “B.”
This set of recommendations had two variables that garnered ratings rated of
“Very Great Effect.” One of the variables, uncertainty, was rated as having "Very
Great Effect” by both interviewees; as noted by the DNFSB interviewee, “It was the
uncertainty of the characterization information available for the waste in the tanks
that led directly to the recommendation.” Communications was also rated as having
substantially influenced the implementation of this set of recommendations. Early on
communications were very problematic, the DNFSB interviewee noted that “DOE
was wholly unresponsive, both in substance and in timeliness, in answering
Recommendation 90-3, requiring that the Board issue Recommendation 90-7.” The
DNFSB interviewee did not lay the blame totally on DOE, though, noting that: “In
68
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
retrospect it can probably be said that the Board's expectations were never made
clear ” However, towards the end o f the recommendation, it would be noted that
good communications was ,tthe single most important factor that contributed to the
eventual successful closure."
O f the remainder of the variable ratings, one bears additional discussion. As
noted above, the ratings given to interest groups involvement were the highest
received. A full spectrum of groups participated in the implementation of this set of
recommendations: Congress, other federal and state agencies, state governments.
Native American tribal governments and individuals. This was credited by both
interviewees as complicating the implementation.
Recommendation 91-6— Radiation Protection
Overview: In addressing radiation protection, the Board took on its second
complex-wide issue. The DNFSB found DOE practices lacking in the following
areas: (1) management and leadership in radiation protection programs; (2) radiation
protection standards and practices at defense nuclear facilities; (3) training and
competence of health physics technicians and supervisors; (4) analysis of reported
occurrences and correction of radiation program deficiencies; and (5) understanding
and attention to radiation protection issues by individuals on the DOE and its
contractor organizations. In short, the DNFSB had identified major and systemic
problems with the DOE radiation protection program (DNFSB, 199Id).
69
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
To address these deficiencies, the Board recommended a seven-step program
of upgrades. These steps started with recommending that the Secretary of Energy
issue a top-level policy explicating DOE’s radiological health and safety
expectations. They included detailed recommendations regarding training programs
to be pursued and reviews to be performed of incident (or occurrence, in DOE terms)
reporting. In addition, an overall recommendation was made for DOE to assess its
infrastructure for radiation protection, identify weaknesses, and develop and execute
a corrective action program to remedy the deficiencies that were identified.
In accepting the DNFSB’s recommendation, the Secretary of Energy stated:
“1 share your concerns regarding DOE's radiation protection program at these
facilities and have recently taken steps to strengthen radiological protection
practices” (DOE, 1992a). However, the shared concerns did not necessarily transfer
to a shared understanding of an adequate implementation plan. In requesting a 45-
day extension to the 90-day time period allotted to implementation plan
development, the Secretary would note that "discussions between our two staffs” had
indicated that more time would be necessary to prepare a "complete implementation
plan” (DOE, 1992c).
Indications on the implementation plan front were not initially encouraging,
even after the additional time expired. DOE's first two attempts at implementation
plans for Recommendation 91-6 were rejected by the DNFSB, virtually out of hand.
In rejecting the first implementation plan the Board deemed it "unresponsive to many
70
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
specific recommendations contained” in the recommendation. The Board noted that
this had occurred even though they had designated senior DNFSB staff members
with which the DOE interfaced (DNFSB, 1992d).
The Board was able to narrow the areas of concern somewhat in its second
disapproval letter; they noted: “Despite the Board’s detailed comments [attached to
the first disapproval letter]... many of the deficiencies identified in the original plan
remained,. . and the four most significant problems were described by the DNFSB
(DNSFB, 1993b). This elicited a quick response from the new Secretary of Energy
(at the time, Hazel O'Leary) which emphasized that DOE was “committed to
providing an adequate implementation plan for Recommendation 91-6 and resolving
the issues identified by the Board in a responsive and timely manner” (DOE, 1993b).
After this exchange of correspondence the DOE was able to develop a satisfactory
plan within four months— 18 months after the DOE originally accepted the
recommendation. The resulting implementation plan committed to 23 specific
actions that were scheduled to be completed over the space of four years.
Interest Groups: No correspondence from outside groups was noted in the
public record for this recommendation. In the interviews, both the DOE and DNFSB
interviewees noted modest interest by outside groups. Professional societies having
to do with radiation protection were mentioned by both persons interviewed. The
DOE interviewee also remembered some interest by the unions representing workers
71
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
at several sites. Overall, the ratings for this factor were roughly in line with those for
the other problematic recommendations studied.
Uncertainty: There was some disparity in the views regarding the impact of
uncertainty on this recommendation. The DOE person interviewed ranked this factor
significantly higher than the DNFSB interviewee. The DOE interviewee noted that
the recommendation itself was complex (in addition to the subject matter), with
seven major sub-recommendations, which made the development of implementation
strategies difficult. On the other hand, the DNFSB interviewee noted that ample
guidance was available in the form of industry standards, which had been referenced
in the recommendation. Finally, the content analysis appeared to support the
description provided by the DNFSB interviewee, with no mention of complexity or
uncertainty noted.
Cost: The influence of the perception of cost on this recommendation
received one rating of “Very Great Effect” and one of “Some Effect.” It was the only
recommendation studied for which detailed cost estimates were developed on a site
by site basis (DNFSB, 1992d). Some of the estimates received significant attention
on the public record; this elicited a reaction from the DNFSB, which noted that
“contractors at some sites are attempting to justify significant increases in costs to
meet radiation protection requirements, the substance of which has been mandatory
for many years” (DNFSB, 1995b). Achieving a satisfactory resolution to just this
issue consumed approximately five months (DOE, 1995e).
72
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Communications: Communications between DOE and the DNFSB during
this recommendation were frequent, as described by both the DNFSB and DOE
personnel interviewed; it was further noted that senior representation from both
agencies were involved. Efforts at a continued dialogue included an extensive status
report by the DNFSB staff that was critical of the progress made by the DOE in the
first year of implementing the recommendation (DOE, 1995a). Review of the public
record shows that regular status reports on the progress of the implementation plan
were not initiated until almost two years after acceptance of the recommendation by
DOE (1993g). The ratings garnered for this variable on this recommendation were
somewhat lower than those garnered by the other problematic recommendations:
however, it is noted by the DOE interviewee that as the implementation of the
recommendation proceeded, these communications became increasingly formal.
Organizational Arrangements: The DOE and DNFSB personnel
interviewed indicated that ‘"who was in charge,” at both agencies, was at issue early
in the implementation of the recommendation. Further, both interviewees noted that
there “a lot of players” involved in the recommendation within the DOE. This led,
even late in the implementation of the recommendation, to an on-going dialogue
concerning who was the lead manager for DOE (DNFSB, 1995b; DNFSB, 1995c).
Further formal concern was expressed by the Board concerning what it believed to be
lack of aggressive implementation on the part of DOE's managers in the field. For
example, with respect to what it considered unduly extended implementation
73
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
schedules, the Board stated that it found some of the dates “unacceptable from a
safety perspective” and it requested that DOE bring these outliers “into conformity
with the deadlines for the vast majority of defense nuclear facilities” (DNFSB,
1995b). As part of its continued efforts to get at the information that it believed
necessary to execute its responsibilities, the DNFSB issued two statutory reporting
requirements (an action just below a recommendation in severity) in the space of
three months (DNFSB, 1995b. DNFSB, 1995c). From the standpoint o f processes
used, the DOE interviewee noted that the use of a top-level internal review group
within DOE (known as the Radiological Controls Coordinating Committee, or
RCCC) was beneficial to ensuring the quality of the DOE work performed in
implementing the recommendation. While noting the influence of the RCCC, the
DNFSB interviewee stated that the interactions between the two agencies were not
well disciplined.
74
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Summary:
TABLE 5
SUMMARY OF INTERVIEWEE VARIABLE RATINGS FOR
RECOMMENDATION 91-6
Rating
Variable
Little or
No Effect
Some
Effect
Moderate
Effect
Great
Effect
Very Great
Effect
Interest Groups B E
Uncertainty B E
Cost E B
Communications E, B
Organizational
Arrangements
E B
Note: The rating provided by the DOE interviewee is indicated by "E,” and
that provided by the DNFSB interviewee is indicated by “B.”
The most remarkable of the ratings are the ones for cost. This was the only
recommendation to receive a rating of “Very Great Effect” by either interviewee. As
noted above, the Secretary of Energy is required, prior to accepting any
recommendation, to determine that it is feasible to implement the recommendation
within the department’s budget. The DNFSB took issue with some of the formal cost
estimates that it received, considering them “exaggerated” and tantamount to
rejection of the recommendation (after formal acceptance by the Secretary), as noted
by the DNFSB interviewee. The discussion of the perception of cost in the public
record during the implementation of this recommendation was unique among the
recommendations studied in its extent, formality and contentious nature.
75
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The ratings for most of the other variables were in line with ratings received
for other problematic recommendations, with the exception of the ratings for
uncertainty, these were somewhat lower than the rest of this group. Within the
overall lower ratings for uncertainty, the DOE interviewee noted that the
recommendation itself was very complex. However, there was little discussion of
either complexity or uncertainty in the public record—this lent support to the
DNFSB interviewee's low rating.
In addition, from a qualitative standpoint, the discussion revolving around
organizational arrangements was informative. Early questioning from the DNFSB
regarding who was in charge at DOE was followed by an increase in the formality of
the interactions between the two agencies. This trend was capped by the DNFSB
using its penultimate authority, that of requiring DOE to provide a report on a given
subject within a specified time frame, twice in a very short period to gamer
information that it deemed necessary for its deliberations.
Recommendation 93-6—Maintaining Access to Nuclear Weapons Expertise
Overview: In this recommendation the Board expressed its concern that, with
the ongoing reduction in the size of the stockpile of nuclear weapons and the
associated diminution in the size of DOE's defense nuclear facilities complex, access
to capabilities in nuclear weapons safety and design would be lost. The DNFSB
noted that the individuals most needed were those who had been active in the
original design of the various nuclear weapons. It noted that many of these
76
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
individuals were being lost to the system through retirements and downsizing
(DNFSB, 1993f).
To address this concern the Board recommended specific actions to reduce
the impact of the loss. They recommended that a formal process be developed to
identify the skills and knowledge necessary to design, assemble, test and maintain
nuclear weapons while in DOE possession. Further, the Board recommended
additional proceduralization of activities within the defense nuclear facilities of the
DOE that performed nuclear weapon operations. Finally, the DNFSB requested that
several specific technical reviews be performed.
DOE’s acceptance of the recommendation was similar to several others. It
noted that DOE “shares your concern of ensuring capability to safely conduct nuclear
weapons testing operations at the Nevada Test Site and to safely dismantle nuclear
weapons at the Pantex Site.” It promised an implementation plan that capitalized “on
activities that have already been initiated and provide a structured approach for
ensuring the continued safe conduct of operations at Pantex and the Nevada Test
Site” (DOE, 1994a). However, early experience with DOE’s implementation plan
indicated that it was, in fact, not well integrated with initiatives that were underway
at the two sites mentioned.
Unique among the problematic recommendations studied, a revised
implementation plan was developed for this recommendation (almost two years into
implementation) not based on a Board disapproval, but because DOE concluded
77
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(admittedly with DNFSB prodding) that the original plan was not producing the
desired results (DOE, 1995d). However, internal negotiations within DOE took
almost nine months to develop a revised approach. This approach was more fully
integrated with other initiatives being taken to streamline the nuclear weapons
complex (DOE, 1996a). Another three years would be necessary before the Board
was prepared to close the recommendation (DNFSB, 1999a).
Interest Groups: Early on, both the interviewees and the public record
indicate little or no external interest in the recommendation. In the midst of the
recommendation, though, Congressional interest developed and several related
initiatives were launched by the DOE in response (DOE, 1995b). These outside
influences, however, were not credited with much impact on the implementation of
the recommendation by either of the interviewees. These impressions combined to
produce a pair of ratings that were below those provided by DOE and DNFSB
interviewees for other problematic recommendations. The public record notes the
presence of Congressionally-mandated initiatives; however, they do not appear to
have conflicted with efforts to implement this recommendation. On the contrary, the
efforts to implement Recommendation 93-6 were to “build on programs that are
either developed, or under development,” the most notable of which was the
Stockpile Stewardship and Management Plan (DOE, 1995b). This plan was intended
to provide an integrated approach to maintaining nuclear weapons capability under a
78
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
comprehensive nuclear testing moratorium and became an annual deliverable under
the revised implementation plan.
Uncertainty: The DNFSB interviewee probably stated it best: “The
recommendation was made because there was a great deal of uncertainty at the Board
as to whether DOE weapons program personnel would have access to highly-
perishable information when it was needed.. . . ” The DOE interviewee noted that
the DOE’s weapons laboratories believed that their efforts in this arena were
satisfactory and did not immediately see how they could be improved. The public
record would seem to support that once agreement was finally reached on a path
forward, much of the uncertainty was addressed. Discussion of uncertainty or
complexity does not figure prominently in public record over last three years of
implementing the recommendation. These elements combined to produce overall
ratings that were in the middle of the range of ratings provided for the problematic
recommendations.
Cost: The interviewees perceived a moderate influence the impact of cost
considerations on the implementation of the recommendation. This was reflected in
both interviewees’ impression that, although this was work (e.g., archiving data,
procedure development, training) that had not been budgeted, at least not
specifically, the incremental costs were not great. Therefore, cost issues were raised;
however, they were managed, even though the “unfunded mandate” flag was raised
on occasion (as noted by the DNFSB interviewee). A number of comments occur in
79
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
the public records that address resource restraints on the implementation of the
recommendation. Towards the end of the implementation of this recommendation,
the Board took the issue head-on. noting that a major portion of the implementation
strategy (the program for archiving data relevant to nuclear weapons) had been
“dependent on widely varying perceptions of the priorities within DOE.’7 The
DNFSB urged “active oversight and strategic guidance” on the part of DOE as
“essential for the program to be successful” (DNFSB, 1998).
Communications: The ratings provided for communications were in line
with other problematic recommendations. However, both interviewees noted that the
tone of the communications was, at times, adversarial. Quarterly reports were
instituted about a year into the recommendation (DOE, 1996a). Later in the
implementation of the recommendation they were supplemented with regular video
teleconferences (DOE, 1996b). This is the only recommendation reviewed which
regularly used this process. It appears that, while many processes were tried (and
discussed in the public record), the participant's impression was that effectiveness
eluded them. Early on, both DOE and DNFSB personnel interviewed noted, it was
difficult to get senior DOE management's attention; this, coupled with sometimes
active resistance by the DOE weapons laboratories, combined to require greater
emphasis on communications early in the implementation of the recommendation.
With the revised implementation plan, further resistance to implementing the
recommendation apparently abated. In fact, in approving DOE's revised
80
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
implementation plan, the Board commended DOE on actions it had already taken to
put in place the revised strategy (DNFSB, 1996a).
Organizational Arrangements: Both the DNFSB and DOE personnel
interviewed indicated that this factor was a significant problem, especially prior to
the revised implementation plan. This was particularly noticeable in the original
implementation plan, where a senior DOE manager was assigned as “integrator.” a
rather weak term, for the recommendation; early on. this indicated potential problems
(DOE, 1994e). Although, in approving the implementation plan, the DNFSB
appeared to let this pass, the Board would later note that .. the necessary level of
management attention at DOE headquarters, field offices and contractor
organizations is not being applied in meeting commitments to the Board” (DNFSB,
1995a). The processes originally specified for executing the recommendation were
somewhat indistinct, as noted by both interviewees. These concerns were addressed
by the revised implementation plan, which included clear assignment of roles and
responsibilities. The ratings were on a par with those for other problematic
recommendations.
81
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Summary:
TABLE 6
SUMMARY OF INTERVIEWEE VARIABLE RATINGS FOR
RECOMMENDATION 93-6
Rating
Variable
Little or
No Effect
Some
Effect
Moderate
Effect
Great
Effect
Very Great
Effect
Interest Groups B E
Uncertainty E B
Cost E, B
Communications E B
Organizational
Arrangements
E B
Note: The rating provided by the DOE interviewee is indicated by “E,” and
that provided by the DNFSB interviewee is indicated by “B.”
Distinctive among the ratings above are the low ratings for interest group
involvement; they are the lowest received by any problematic recommendation
studied. These ratings occurred even though, part of the way through
implementation of the recommendation, Congress turned its attention to the general
subject of maintaining the nation’s nuclear weapons capability. The ratings for cost
and uncertainty were somewhat lower than the other problematic recommendations.
In summary, this somewhat limited impact of environmental variables supports a
statement by the DOE interviewee, who noted that the problems in implementing this
recommendation were “largely internal to DOE.”
82
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The statement by the DOE interviewee is also supported by the discussion on
the communications and organizational arrangement variables. Communications
were described by both interviewees as “difficult.” This variable received ratings of
“Moderate” to “Great Effect,” in line with ratings received by other problematic
recommendations. Organizational arrangements, on the other hand, were provided
the highest ratings received by any recommendation, ratings that were consistent with
the significant level of discussion on the subject in the public record. Problems with
the assignment of roles and responsibilities for implementing the recommendation,
along with what were determined to be inappropriate implementation processes, led
to a wholesale revision of the approach to implementing the recommendation—two
years after the recommendation had been accepted by the DOE.
Four Non-Problematic Recommendations
Recommendation 91-5—Power Limits for K-Reactor Operations
at the Savannah River Site
Overview: By the time it was decided to pursue restarting the K-Reactor in
1990, it was the sole source o f tritium (a radioactive isotope of hydrogen used to
improve the performance of nuclear weapons) for the nation’s nuclear weapons; as
such it claimed a high priority upon the DOE and DNFSB’s resources. During the
interview with the Technical Director of the DNFSB, he mentioned that the public
record for this recommendation actually started with a public meeting immediately
prior to issuance of the recommendation. It had long been understood by DOE that
83
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the Board intended to place some sort of limit on the power level at which the K-
Reactor could be operated; the major question was what that power level would be.
Much of the late summer and early autumn of 1991 had been spent by the DNFSB
hearing information upon which a determination could be made.
After the information presented by the DOE at a public meeting in Aiken,
South Carolina the Board issued their recommendation. The Board recommended
that DOE not operate K-Reactor at greater than 30 percent of historical lull power
until: (1) a more detailed thermal-hydraulic analysis had been done to quantify the
performance of the emergency core cooling system; (2) a revised accident analysis,
based on the new thermal-hydraulic analysis was complete; and (3) the new analysis
had undergone quality assurance checks and was under adequate configuration
management (DNFSB, 1991e).
DOE (1992a) accepted the Board's recommendation and noted that:
The DNFSB will be informed well before any decision is made to
increase reactor power above 30 percent of the historical value of
maximum full power.. . . Evaluation of power limits to determine
the feasibility of increasing the operating power is continuing.. . .
Subsequent to acceptance of the Board's recommendation, DOE (1993e) entered into
testing of the K-Reactor, prior to placing it into operation. Limited production
operations ensued, after some mechanical problems. Based on the reduced need for
tritium (due to the end of the Cold War), and budget constraints. K-Reactor was
placed into a stand-by status in 1993 and the recommendation was closed in February
ofl994.
84
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Interest Groups: Although K-Reactor was a high profile issue overall,
neither the DOE nor the DNFSB interviewee recalled any substantial interest group
input. The DNFSB person interviewed recalled that possibly one local
environmental group was interested and also recalled some Congressional interest.
In addition, no such input from interest groups was registered on the 131 page public
record for the recommendation. Therefore, the impact of this variable was rated
relatively low, overall.
Uncertainty: Although the overall issue of the proper power to operate this
reactor was, as the DNFSB interviewee put it. “shot through with uncertainty,” by the
time the recommendation was made, much of the uncertainty had been addressed.
This permitted the Board to focus on the remaining major area of uncertainty in the
recommendation: the operation of the emergency cooling system. However, even
this remaining uncertainty was fairly substantial and the public record contains a
number of specific references to it (DNFSB, 1993c). These elements combined to
yield overall ratings that were just about on a par with those of the problematic
recommendations studied, and the highest rating received by any of the non
problematic recommendations.
Cost: The analyses that the Board recommended were not low cost items, but
as both interviewees pointed out, it was generally agreed that “the analyses had to be
done” to resolve the last remaining uncertainties. In the end, since the reactor was
never operated above 30 percent of historical full power, and was ultimately placed a
85
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
in stand-by condition, the analyses were never completed—so cost almost became a
moot point. This factor was given the lowest rating by both interviewees.
Communication: By all accounts, good communication was vital to the
information exchanges that led up to the recommendation and those that occurred as
the mission of the K-Reactor was in flux. The public record indicates a regular flow
of information to the DNFSB, both during the deliberations on the appropriate power
level and as DOE was deciding the fate of K-Reactor. Interchanges were described
as “frequent” and “non-adversarial” by both DOE and DNFSB personnel.
Communications for this recommendation received the highest rating of any variable.
Organizational Arrangements: Similar to the discussion of
communications above, positive comments on this factor were received from both
interviewees. Descriptions included “very clear roles” and “well-identified leads.”
In addition, positive comments were made about the fact that there was a DOE
headquarters group, on-site, that did nothing but pursue resolution of K-Reactor
issues. The general impression of both the DOE and DNFSB interviewees was that
the processes for information exchange, both leading up to and after the
recommendation were efficient and well organized. This was noted to be vital, due
to the massive amounts of information that had to be interchanged. Overall, this
factor was rated right in line with the ratings received by other non-problematic
recommendations.
86
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Summary:
TABLE 7
SUMMARY OF INTERVIEWEE VARIABLE RATINGS FOR
RECOMMENDATION 91-5
Rating
Variable
Little or
No Effect
Some
Effect
Moderate
Effect
Great
Effect
Very Great
Effect
Interest Groups E B
Uncertainty B E
Cost E, B
Communications B E
Organizational
Arrangements
F, B
Note: The rating provided by the DOE interviewee is indicated by "E,” and
that provided by the DNFSB interviewee is indicated by "B."
Communications stands out as having received ratings of "Great” and "Very
Great Effect.” It garnered the highest overall rating for this variable on any
recommendation. As discussed above, this was due to open dialogue that existed
between DOE and the DNFSB on this recommendation. Two additional variables
received ratings above "Moderate Effect.” The one that stands out is uncertainty,
which was rated higher than any other non-problematic recommendation. This was
explained by both the DOE and DNFSB personnel interviewed, who noted that
substantial uncertainty existed if the power level of 30 percent was exceeded, but
analyses showed this power level to be safe. The other variable, organizational
arrangements, was also notable. It was rated as having "Great Effect” by both the
87
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
DOE and DNFSB interviewees. This consistently high rating was unique among
non-problematic recommendations. The DOE organizational arrangements were
summarized by the DNFSB inteiviewee as “very clear’ and “well organized.”
The final rating of note was cost which was rated as low as possible. Such a
low rating is explained by two facts noted by the interviewees. First it was clear to
all that the additional studies needed to be done if the power level were to be
increased (therefore, cost was not considered to be an issue), and the studies, which
would have been costly, never had to be completed.
Recommendation 92-2— DOE Facility Representative Program
Overview: At contractor operated defense nuclear facilities (which includes
just about all the facilities of consequence), the DOE Facility Representative is
responsible for monitoring the performance of the facility and services, as the
primary DOE point of contact with the contractor. The DNFSB, recognizing the
importance of this position, reviewed the available DOE guidance and requirements
concerning this position. They found that the relevant standards provided only
limited detail concerning the duties and responsibilities of the Facility Representative
and that they did not provide any guidance for selection and training of these
individuals, nor did they provide any effective guidance specifying the details of the
duties and responsibilities that pertained to the position. The recommendation
included a two-page attachment that specifically addressed the last aspect of the issue
(DNFSB, 1992b).
88
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Accordingly, DNFSB Recommendation 92-2 sought to redress these
deficiencies. It recommended that DOE carry out a comprehensive analysis of the
existing Facility Representative programs and, utilizing the results of that analysis,
modify the practices and processes associated with the program across the defense
nuclear facility complex (DNFSB, 1992b).
The DOE accepted this recommendation without substantial comment. It
developed a simple implementation plan, that hinged on three major deliverables: a
standard for the management o f the program, an evaluation of the qualifications and
training required to adequately perform the role of the Facility Representative and a
resource loaded plan to implement the program. This plan was agreed to in a timely
fashion (DNFSB, 1993a). Recommendation 92-2 was closed in October 1996. after
the DNFSB had noted that DOE had “made substantial progress in its Facility
Representative program” (DNFSB, 1996b).
Interest Groups: Both the DOE and DNFSB personnel interviewed
indicated that no outside interest developed regarding this recommendation. The
public record indicates that one DOE laboratory provided generally favorable
comments on the intent and thrust of the recommendation. These elements led to a
low overall rating for the influence of interest groups on the implementation of this
recommendation.
Uncertainty: According to the DNFSB and DOE interviewees, there was
some initial uncertainty with precisely what the Board desired—largely since there
89
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
was already an existing program. The public record shows that further uncertainty
revolved around what level of staffing, on a site-by-site basis was deemed to be
technically adequate (DOE, 1993c). Overall, these indications garnered this factor
ratings that were in line with other non-problematic recommendations.
Cost: Contrary to the above discussion, cost was a factor that was rated to
have a larger influence on this recommendation than any other non-problematic
recommendation. As pointed out by the DNFSB interviewee, the costs involved in
the upgraded selection, qualification and training programs were substantial:
however, it was noted that the cost estimates that were performed were done within
the context of the approved DOE implementation plan. The DOE interviewee noted
that the costs were, however, very decentralized and managers in the field tended to
“take it out of overhead.” As to the overall size of the program, some disparities
were noted, for example: “In general the [staffing] requirements proposed by the
field have been about twice the presently authorized Facility Representative
coverage” (DOE, 1993f). This information combined to provide ratings for the cost
factor that were higher than any other non-problematic recommendation, but lower
that those garnered by the problematic recommendations studied.
Communication: Although both the DOE and DNFSB personnel
interviewed indicated that communications went smoothly, neither of them rated this
factor as having more than a moderate impact. Both interviewees noted that a major
function of the communications during the implementation of this recommendation
90
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
was the DNFSB fully explicating its vision of the role of the Facility Representative
and the DOE coming to an understanding of that vision. A review of the public
record shows a substantial amount of formal communication between the DOE and
the DNFSB; however, throughout, the nature of the communications was non-
confrontational. Overall, this factor received ratings from the interviewees that were
somewhat lower than those received by other non-problematic recommendations.
Organizational Arrangements: The DOE interviewee indicated that, early
on. the lack of DOE headquarters leadership impacted the implementation of the
recommendation. However, this problem was later solved by placing the cognizant
Associate Deputy Secretary in charge. The DNFSB interview produced no strong
comments on this subject. He noted that the "path forward was reasonably well
understood.” On the public record, the Board indicated its satisfaction with the
arrangements established by DOE: "The Board is pleased to note that the Associate
Deputy Secretary for Field Management has been tasked to provide centralized
direction for this program.. . . ” (DNFSB, 1993d). The Secretary of Energy also
indicated her pleasure: “The Department is confident that having [the Associate
Deputy Secretary for Field Management] administer the program will cause each
field organization to operate one unified FR Program” (DOE, 1993i). The
implementation plan called for the development of a DOE standard for the Facility
Representative program, a training program and a resource-loaded plan to execute
needed revisions. These deliverables provided structure to the interactions between
91
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the DOE and the Board, as noted by the DNFSB interviewee. The DOE interviewee
noted that these two deliverables formed the major decision points for the
recommendation and provided “a fairly simple structure.” This information resulted
in ratings that were in line with those received by other non-problematic
recommendations.
Summary:
TABLE 8
SUMMARY OF INTERVIEWEE VARIABLE RATINGS FOR
RECOMMENDATION 92-2
Rating
Variable
Little or
No Effect
Some
Effect
Moderate
Effect
Great
Effect
Very Great
Effect
Interest Groups E, B
Uncertainty B E
Cost E B
Communications B E
Organizational
Arrangements
B E
Note: The rating provided by the DOE interviewee is indicated by “E.” and
that provided by the DNFSB interviewee is indicated by “B.”
The only variable to receive a rating higher than moderate effect was
organizational arrangements. It was rated as having “Great Effect” by the DOE
interviewee and “Moderate Effect” by the DNFSB interviewee. Specific comments
were made regarding the implementation plan, at was considered a “fairly simple
92
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
structure.” With that said, the ratings received were in line with the ratings received
by other non-problematic recommendations.
The ratings for the communications variable were on the low end of ratings
received by non-problematic recommendations; this contrasts with the substantial
discussion in this area in the public record. This apparent disconnect was discussed
by both interviewees, they noted that, during the early part of the recommendation,
substantial dialogue was required to develop a shared vision for the facility
representative program.
One of the external variables stood out, to a degree. Cost, which was rated
higher than for any other non-problematic recommendation, presents an interesting
contrast. It should be noted, however, that estimation of the resources required to
train and qualify the required cadre of Facility Representatives was an explicit part of
the implementation plan approved by the DNFSB. This led to largely factual
discussion of requirements. This discussion did not appear to generate much in the
way of controversy between the Board and DOE based on the public record and the
impressions of the interviewees. The remaining variables, interest groups and
uncertainty, were substantially in line with the ratings received by other non-
problematic recommendations studied.
It is interesting to note that, subsequent to closing this recommendation, the
Board issued a short summary report of the implementation of the recommendation.
93
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
In the forwarding letter for the report the Board stated: “This has been one o f the
most successful recommendations implemented by DOE” (DNFSB, 1996b).
Recommendation 92-6—Operational Readiness Reviews
Overview: An operational readiness review (ORR) is a comprehensive,
independent review of all preparations that have been made to startup or restart a
facility. ORRs are a standard practice in the commercial nuclear industry and in the
naval nuclear propulsion program. Prior to issuance of this recommendation, the
Board had issued several recommendations dealing with the appropriateness of
performing ORRs for individual sites or specific facilities (DNFSB, 1990c: DNFSB,
1991a; DNFSB, 1991c). In addition, DOE had performed ORRs for several
additional facilities of its own volition. Based on two years of experience in
performing these reviews in the DOE defense nuclear facility complex, the DNFSB
believed that the time had come to standardize the approach to these reviews.
Specifically, the Board believed that guidance regarding ORRs was required
to specify the required features of an adequate ORR, and by stating specifically on
what occasions an ORR should be required. Further, the DNFSB believed that ORRs
should not serve as a substitute for line management’s responsibility to assure the
readiness of a facility for safe and reliable operation. Therefore, Recommendation
92-6 required that DOE develop uniform orders, directives and guidance to perform
ORRs which addressed a number of specific DNFSB concerns (DNFSB, 1992f).
94
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
In accepting Recommendation 92-6, the Secretary of Energy noted that DOE
had already taken several steps to systematize the practice of performing ORRs.
DOE had already held a comprehensive review of the experience that had been
gained with the major ORRs that had been performed. Further, plans had already
begun to develop some sort of DOE-wide directive with respect to performing ORRs.
DOE’s implementation plan, using the results of the above-mentioned
assessments, proposed two deliverables: a DOE order mandating the use of ORRs
and a DOE standard that would outline methods deemed to be adequate by the
Department for the execution of these reviews (DOE, 1993a). The DNFSB had two
administrative concerns with the DOE implementation plan: the need for the Board
to review and accept the DOE Order and standard prior to issuance, and the planned
timing of status reports; however, the implementation plan was approved, provided
these revisions were made. This recommendation was closed approximately two
years after its implementation plan being approved. This makes it the shortest
implementation period reviewed in this study.
Interest Groups: As succinctly stated by the DNFSB interviewee, interest
groups had ‘‘ essentially no prominence.” This evaluation was echoed by the DOE
interviewee. Review of the public record indicates no correspondence from outside
interest groups and no mention of them in either DOE or DNFSB documentation.
The rating for this variable from both interviewees was “Little or No Effect.” This
95
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
was substantiated by the absence o f any discussion of interest group involvement in
the public record.
Uncertainty: The interviewees from both the DOE and DNFSB noted, as
discussed above, that DOE and the DNFSB had already worked together for a
substantial period of time on ORRs. Thus, as described by the DNFSB interviewee,
all that remained was the details of the guidance to be issued. The public record
contains no discussion of uncertainty. This agrees with the description of the topic of
ORRs provided by the DOE interviewee, who noted that the subject matter was "well
defined.” Therefore, the ratings for uncertainty were low, below those received by
the other non-problematic recommendations studied.
Cost: Since this recommendation specifically dealt with the development of
top-level processes, there was little expense in its execution, per se; this was pointed
out by the DNFSB personnel interviewed. However, the DOE interviewee did note
that some concern existed with how expensive the reviews would be to perform.
This concern was voiced informally. However, in the opinion of the DOE
interviewee, it did have some impact on the content of the direction and guidance
that was issued. No mention of cost occurred in the documents on the public record.
These facts combined for a set of ratings that were low overall and about on par with
the other non-problematic recommendations.
Communications: The personnel interviewed from DOE and the DNFSB
agreed that the communications on this recommendation were frequent and generally
96
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
constructive. It was noted by the DNFSB interviewee that "DOE was generally
receptive to the assistance provided by the DNFSB staff and recognized that
assistance in correspondence with the DNFSB.” An example of this DOE
recognition stated “The Department appreciates the close cooperation provided by
your staff during the development of [the DOE Order and standard on ORRs]. This
enabled us to issue the documents in a timely fashion and will help to ensure
consistent performance of ORRs across the Department” (DOE, 1994b). In addition,
the periodic status report, which due to the compressed schedule for this
recommendation was issued every two months (versus the more traditional quarterly
periodicity) kept all concerned groups current on progress. This information
supported somewhat elevated ratings for this variable, and overall, the highest ratings
received by any variable on this recommendation.
Organizational Arrangements: The DOE and DNFSB personnel
interviewed were consistent in stating that the limited number of people involved in
the in the implementation of the recommendation led to very clear roles and
responsibilities. In addition, the limited time frame in which the recommendation
was implemented helped to assure stability. Space spent on this factor in the public
record was factual in tone; for example, providing points of contact, discussing
schedules for deliverables and providing authoritative context for comments. In
addition, the practice of sharing draft correspondence led to less of a need to resort to
formal processes and methods, as discussed by the DOE interviewee. Finally the two
97
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
well-defined decision points, approval of the draft DOE order and draft DOE
standard, provided clarity to the process. The ratings received by this
recommendation were indicative of a moderate impact, largely in line with the
ratings received by other the other non-problematic recommendations studied.
Summary:
TABLE 9
SUMMARY OF INTERVIEWEE VARIABLE RATINGS FOR
RECOMMENDATION 92-6
Rating
Variable
Little or
No Effect
Some
Effect
Moderate
Effect
Great
Effect
Very Great
Effect
Interest Groups E, B
Uncertainty E B
Cost B E
Communications B E
Organizational
Arrangements
E, B
Note: The rating provided by the DOE interviewee is indicated by “E,” and
that provided by the DNFSB interviewee is indicated by “B.”
The environmental variables (interest groups, uncertainty and cost) were
uniformly rated as having “Little or No Effect” or only “Some Effect.”
Organizational arrangements, among the structural variables, was rated to have only
a moderate impact, largely in line with the ratings received by non-problematic
recommendations. Of note, though, was the important role played by the routine
98
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
sharing of draft correspondence, which was instituted as an informal process between
DOE and DNFSB during the implementation of this recommendation.
Communications presented a different situation, it received higher ratings,
overall, than was the norm for non-problematic recommendations (and, in fact, the
recommendations studied overall). Perhaps more significantly, though, the positive
impact that the frequent, open communications had on the recommendation was
commented on formally by both the DOE and DNFSB on a number of occasions.
The important role played by communications is similar to the discussion for
Recommendation 91-5, above.
Recommendation 93*4—Environmental Restoration
Management Contracts
Overview: In 1992, in an attempt to deal with the process of closing sites that
contained former defense nuclear facilities, DOE developed a new contracting
strategy, known as the environmental restoration management contract (ERMC).
This strategy was put in place first at the Femald Site outside Cincinnati, Ohio,
which had processed uranium for several decades. The majority of DOE sites are
operated by contractors and thus the DNFSB sought to understand this new
contracting strategy. Also, since a number of other sites would be going through the
closure process, the Board was interested in ensuring that the new contracting
strategy could be relied upon to provide a level of safety that was at least
commensurate with previous contracting mechanisms.
99
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Recommendation 93-4 was developed by the Board after observing
operations under the ERMC strategy for about a year. The DNFSB expressed
concerns with the manner in which DOE oversight of its contractors was being
performed under the new arrangement. Consequently, the DNFSB recommended
that: (1) DOE develop a overall technical management plan for Femald and any
future sites that used the ERMC contracting mechanism: (2) the technical
management plan include a number of attributes, primarily, clear identifications of
roles and responsibilities between the DOE and its contractors and an identification
of the technical requirements and guidance that pertained to the environmental
restoration work; (3) certain specific safety-related incidents at the Femald Site be
investigated and a clear resolution path developed and (4) a Facility Representative
program, meeting DOE requirements issued under DNFSB Recommendation 92-2
(above), be instituted (DOE, 1993c).
In accepting this recommendation, the Secretary of Energy immediately
named the Assistant Secretary of Energy for Environmental Restoration and Waste
Management (at the time Thomas Grumbly) to lead DOE’s implementation planning.
DOE developed an implementation plan whose centerpiece was the development of a
technical management plan for each site undergoing environmental restoration using
the ERMC concept. DOE also committed to an invigorated facility representative
program at the Femald Site and to address the specific safety-related issues cited by
the DNFSB (DOE, 1993h). Later, DOE expanded the applicability of this approach
100
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
to the Hanford Site in Richland. Washington, which had also issued an ERMC
(DOE, 1994d). Even with this expansion of the applicability of the recommendation,
it was closed just 30 months after the DNFSB acceptance of the implementation
plan.
Interest Groups: Both the DOE and DNFSB personnel interviewed
remembered several outside interest groups, including the unions at the site, being
interested in the implementation of the recommendation. However, there was no
evidence of this interest reflected in the public record. In addition, informal inquiries
from Congressional staffers were mentioned (notably the staffs of Senator Glenn and
Congressman Dingell). The lack of any mention in the public record, coupled with
the interviewees belief that these inquiries had little impact on the progress of the
recommendation, led to fairly low ratings for the influence of interest group
involvement on this recommendation. The ratings were in line with other non
problematic recommendations.
Uncertainty: The DOE interviewee noted that this was the first use of this
type of contract (ERMC), which introduced some uncertainty into the
implementation of the recommendation. While recognizing this, the DNFSB
interviewee noted that the technical complexity involved environmental restoration
work was relatively low. Further, no mention of uncertainty was found in the more
than 240 pages of the public record. These data combined to produce ratings that
101
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
were below those received by two of the other three non-problematic
recommendations (Recommendation 92-6 being the exception).
Cost: Although the effort to reduce the cost of environmental restoration was
a concern in formulating the ERMC contracting strategy, no mention of cost is found
in the public record for the recommendation. In addition, both the DOE and DNFSB
personnel interviewed indicated that there was not a specific cost ascribed to the
implementation of this recommendation. Perhaps the DNFSB interviewee
summarized it best: "no impact due to cost.’” This factor received a rating of "Little
or No Effect” by both interviewees.
Communication: The interviewees from the DOE and DNFSB noted that
communications were open, frequent and constructive. Further, although a
significant portion of the public record is spent transmitting information the factual
nature of the information transmitted supports the impression voiced by the DOE
interviewee, namely, that there was no significant disagreement between the DOE
and DNFSB during the implementation of this recommendation. Die ratings
developed for this variable were lower than those received by two of the three other
non-problematic recommendations studied.
Organizational Arrangements: The personnel interviewed from DOE and
the DNFSB both rated the impact o f this factor fairly highly, noting that roles and
responsibilities were clear and that senior management leadership at DOE
headquarters was a factor in achieving this level of clarity. The public record has
102
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
extensive discussion of roles and responsibilities. In the discussion on the public
record they are clearly stated, reinforced and elaborated, as necessary (DOE, 1994c).
No disagreements between the DOE and DNFSB are noted. Further, the DOE
interviewee noted that the implementation plan laid out a particularly clear path
forward. The DNFSB interviewee agreed, noting that little in the area of processes,
beyond the implementation plan, was required throughout the implementation of the
recommendation. Thus, this variable received ratings that were in line with those
garnered by the other non-problematic recommendations.
Summary:
TABLE 10
SUMMARY OF INTERVIEWEE VARIABLE RATINGS FOR
RECOMMENDATION 93-4
Rating
Variable
Little or
No Effect
Some
Effect
Moderate
Effect
Great
Effect
Very Great
Effect
Interest Groups B E
Uncertainty B E
Cost E, B
Communications E B
Organizational
Arrangements
B E
Note: The rating provided by the DOE interviewee is indicated by “E,” and
that provided by the DNFSB interviewee is indicated by "B.”
103
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Many of the ratings received by this recommendation are in line with those
received by the other non-problematic recommendations. For example, the ratings
for interest groups involvement, uncertainty and organizational arrangements are
largely consistent with the other non-problematic recommendations. In addition, the
ratings for cost were the lowest possible, and in line with the comment from the
DNFSB interviewee, who noted simply “no impact due to cost.”
One of the ratings diverges somewhat from the ratings received by other non-
problematic recommendations and that is communications. This variable was rated
lower than two of the other three non-problematic recommendations. The rating
given by the DOE interviewee of “Some Effect” is consistent with his overall
impression that there was relatively little disagreement between the DOE and
DNFSB and, therefore, little cause for extensive communications, or at least formal
communications. This occurred even though both interviewees agreed that
communications were open, frequent and constructive.
Overall Summary
The overall interviewee ratings for all eight recommendations are shown in
the table at the end of this section. After a cursory inspection of the data, two
observations are offered. These observations will be briefly described below;
qualitative and quantitative assessment of any associations will be addressed in the
next chapter. In addition, in the next chapter the content analysis of information
gathered for each of the variables will be discussed.
104
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Looking at the results of the interviews, the structural variables
(communications, and organizational arrangements) appear to have been rated
consistently more highly than environmental variables. Both structural variables had
medians of “Moderate Effect” or higher. Of the environmental factors, only
uncertainty had a median rating that was equivalent to “Moderate Effect.” Interest
groups and cost were of “Some Effect.”
The next observation is the difference (or lack thereof) between ratings
received in problematic recommendations and non-problematic cases. For the
structural variables (communications and organizational arrangements) there does
not appear to be any substantial differences between the ratings received by
problematic and non-problematic cases. However, for the environmental variables
(interest groups, uncertainty and cost), there are apparently differences between the
medians and modes, based on whether the cases are problematic or non-problematic.
On the face of it, these results appear somewhat paradoxical. None of variables that
appear to be differentiators, that is, the environmental variables, were rated to have a
more than a “Moderate Effect,” overall. However, two variables that were rated
consistently to be of more than moderate effect do not appear to help us understand
why a recommendation might be problematic—at least from the numerical results.
105
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
TABLE 11
INTERVIEWEE RATINGS RESULTS FOR ALL CASES
Case Interest
Cronp
Uncertainty Cost Communications Organizational
Arrangements
Problematic
90-2
DOE 4 5 4 2 4
DNFSB 2 5 3 5 5
90-3/7
DOE 4 5 2 3 4
DNFSB 4 5 4 5 3
93-6
DOE 2 3 3 3 4
DNFSB 1 4 3 4 5
Median for
Problematic
Recommendations
2.5
4.5 3 3 4
Mode for
Problematic
Recommendations
2 or 4
5 3 3 4
Non-Problematic
91-5
DOE I 5 1 5 4
DNFSB 3 3 1 4 4
92-2
DOE 1 3 2 3 4
DNFSB 1 2 3
2
3
92-6
DOE 1 I 2 5 3
DNFSB 1 2 1 4 3
93-4
DOE 2 3 I 2 4
DNFSB 1 1 1 4 3
Median for Non
problematic
Recommendations
1 3 1 4 3 or 4
Overall Median
2
3
2
3.5 4
Overall Mode 1 5 1 3 4
106
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER V
ANALYSIS OF RESULTS
Overview
The previous chapter described the results of the research (completed as
described in the third chapter), on a case by case basis, and then provided a summary
o f the ratings provided by the interviewees for the eight cases studied. This chapter
will present the data on a variable by variable basis and provide the results of content
analysis performed on the statements made by interviewees and citations found in the
public record. The results of a statistical test appropriate for the interviewee rating
data will also be provided. Using this information, the hypotheses posed in Chapter
Three will be addressed.
This research set out to investigate whether several variables that have been
advanced by the applicable literature are associated with the effectiveness of
scientific-technical advisory committees (such as the DNFSB). The variables
selected included: environmental variables, notably interest group involvement,
uncertainty and the perception of cost; and the structural variables of
communications and organizational arrangements. Scientific-technical advisory
107
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
groups were defined as being effective when the agency which they advised accepted
their recommendations and implemented them in a timely fashion.
Hypothesis Testing Results
Five hypotheses were proposed during this research. They were based on the
literature review mentioned above and are restated below, for reference purposes,
prior to the discussion of the results for each hypothesis. The hypotheses are
discussed in the order that they were advanced in Chapter Three.
Hypothesis I. The greater the degree of interest group
involvement, the more problematic will be the implementation of
that recommendation.
Qualitative Analysis
The analysis of the results of interviews and assessment of the public record
was somewhat unique for this variable. A readily measured numerical indicator was
available to help gauge interest group involvement; that is. the number of interest
groups that were identified by interviewees and whose involvement could be
discerned from the public record. In all, thirteen individuals or groups were
mentioned in the public record for the recommendations studied, or where mentioned
by interviewees. Congress was associated with the public record of three
recommendations. No other group was mentioned with regard to more than one
recommendation. Review of the content of the interviews and the public record
108
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
revealed two relevant content categories in the interviewee statements and public
record citations; they are discussed below.
In the public record, interest group involvement was discerned in one of two
fashions. First the interest group could have written either the DNFSB or DOE
directly on the subject of the recommendation. An example of this is the Natural
Resources Defense Council (NRDC) who wrote to the DNFSB to express its opinion
of Recommendation 90-2. Alternatively, an interest group could be referred to by
either the DOE or DNFSB in correspondence between the two agencies. The
Washington State Department of Ecology, as an example, is referred to in the
correspondence from the DOE regarding Recommendation 90-3/7. Frequently,
however, the only indication of an interest group's involvement was in the memory
of the personnel interviewed; for example. Congressional interest in various
recommendations (90-2,90-3/7,93-4) was mentioned by interviewees, frequently by
both DOE and DNFSB personnel, but no indication existed on the public record.
Table 12 summarizes the numbers of interest groups that became involved in
recommendations, sorted by whether they were non-problematic cases or problematic
cases.
109
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
TABLE 12
NUMBER OF INTEREST GROUPS INVOLVED IN
NON-PROBLEMATIC AND PROBLEMATIC CASES
Topic Non-Problematic Cases Problematic Cases
Number of Interest Groups
Identified During
Interviews
4 13
Number of Interest Groups
Identified in the Public
Record
1 12
Total Number of Time
Interest Groups were
Mentioned*
5 25
•The numbers represent total numbers of times an interest group was mentioned, not
the number of interest groups mentioned, since some have been mentioned in both
interviews and the public record.
Two relevant categories of information were identified in the review of the
qualitative information available in the interviews and public record. The interviews
often had statements that referred to either high or low levels o f interest group
involvement due to the thrust of one of the questions. For example, in one case, an
interviewee noted: “There was strong interest from the Washington State Department
of Ecology throughout the implementation of the recommendation.” On the other
hand, for a different recommendation, an interviewee stated: “Interest groups had
essentially no prominence during the implementation of the recommendation.” In
addition, there were also cases where an interest group stated a position concerning a
recommendation. These statements were uniformly focused in problematic cases and
110
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
reflected positions that made the implementation of a recommendation more
difficult, such as: “The Washington Department of Ecology imposed waste
characterization requirements that included many items that were not necessary for
safety and this caused conflicts ” Table 13 summarizes the frequency with which
each category of statement or citation occurred and its source (interview or public
record).
TABLE 13
MEASURES OF THE DEGREE OF INVOLVEMENT OF INTEREST
GROUPS IN NON-PROBLEMATIC AND PROBLEMATIC CASES
Category of
Infonnatioii
Source Non-Problematic
Cases
Problematic Cases
1. Statements or
Citations Indicating
High Levels of
Interest Group
Involvement
Identified During
Interviews 0 1
Identified in the
Public Record 0 1
Sub-total 0
2
II. Statements or
Citations Indicating
Low Levels of
Interest Group
Involvement
Identified During
Interviews 4 I
Identified in the
Public Record 0 1
Sub-total 4 2
III. Statements or
Citations Indicating
an Interest Group
Position
Identified During
Interviews
0 2
Identified in the
Public Record
0 4
Sub-total 0 6
111
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Quantitative Analysis
Medians and modes were determined for responses regarding interest group
involvement for both groups of cases: non-problematic and problematic. The results
are presented below. In addition, a statistical analysis appropriate for use with the
data produced by the interviews was used to determine if the population of
interviewee responses for non-problematic cases was different from that for
problematic cases. The numerical results can be found in Appendix D; they are
summarized below.
TABLE 14
QUANTITATIVE INFORMATION FROM INTERVIEWS
CONCERNING INTEREST GROUP INVOLVEMENT
Statistic Non-Problematic Cases Problematic Cases
Median 1 2.5
Mode 1 2 or 4
Cursory inspection of the above results indicates that the median response
from interviewees concerning influence of interest group involvement on
recommendation implementation for problematic cases appears to be higher than that
received for non-problematic cases. The information available from the modes is
consistent with this trend. Statistical analysis, using the Mann-Whitney U-Test. also
supports the conclusion that the ratings for problematic cases were consistently
112
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
higher than those received by non-problematic cases for the influence of interest
group involvement.
Summary
There was support for this hypothesis. The amount of qualitative information
from interviews and from the public record was somewhat limited. However,
analysis of the information available pointed to an association between the degree of
interest group involvement and the problematic implementation of recommendations.
Most persuasive was the data which showed, based on the number of interest groups
mentioned in interviews and the public record, that the number of interest groups
involved in a recommendation showed a strong association with problematic cases
(i.e., significantly more interest groups were mentioned in problematic cases than
were mentioned in the description of non-problematic cases, by a margin of five to
one, as seen in Table 12). Information from the content analysis showed that a high
level of interest group involvement was associated with problematic cases, and a low
level of interest group involvement was associated with non-problematic cases
(Table 13, Rows I and II). Additional support was provided by the tendency of
interest groups to make their positions known predominantly in problematic cases
(Table 13, Row III).
Quantitative data available from the ratings showed that a significant
difference existed between the ratings provided for problematic cases and non
problematic cases. Problematic cases were rated consistently higher—“Moderate” to
113
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
“Some Effect” as opposed to “Little or No Effect.” When this result was discussed
with the key informants they were not surprised. They both pointed to a court case
early in the DNFSB's existence (Natural Resources Defense Council vs. United
States) that framed the public’s input to the relationship between the DNFSB and
DOE. In that case the Federal Appeals Court held that the recommendations of the
DNFSB were not rules, were not subject to the Administrative Procedures Act (U.S.
Congress, 1949), and therefore, did not need to be published for public comment
prior to being issued to DOE. This public comment stage in formal rulemaking is
ofren one where substantial public input is garnered; thus, the key informants
believed that it was unlikely that interest groups would go to the effort of becoming
involved unless they opposed the recommendation. In addition, it was noted that the
highly technical work performed by the Board, along with the deference that
Congress has shown to the Board, may have been factors in the limited involvement
by interest groups.
Hypothesis II. The greater the degree of technical uncertainty
associated with the subject of the recommendation, the more
problematic will be the implementation of the recommendation.
Qualitative Analysis
Review of the interview statements and citations from the public record
revealed two relevant categories of content. The first of the two content categories
evaluated statements and citations having to do with the resolution o f uncertainty.
114
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
These statements and citations address the reduction or elimination of uncertainty or,
conversely, can describe the difficulty in resolving sources of uncertainty. They were
associated almost exclusively with problematic cases. An example of this type of
statement would be: “Even as characterization information was obtained, the
uncertainty in records made it difficult to apply the information gained ” The
other category of statements and citations directly addressed judgements concerning
the level or amount o f technical uncertainty involved in the subject matter of a
recommendation. They include statements such as: ‘there was no particular
uncertainty involved” or, on the other hand: “The high level of uncertainty prolonged
the implementation of the recommendation by at least three years.” The statements
and citations that provided an indication of either ability to resolve uncertainty or the
level of uncertainty are summarized in Table 15.
115
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
TABLE 15
TWO MEASURES OF THE INFLUENCE OF LEVEL OF TECHNICAL
UNCERTAINTY ASSOCIATED WITH NON-PROBLEMATIC
AND PROBLEMATIC CASES
Category of
Iuformatlou
Source Non-Problematic
Cases
Problematic Cases
I. Statements or
Citations
Indicating High
Levels of Snbject
Matter
Uncertainty
Identified During
Interviews 2 1
Identified in the
Public Record 1 4
Sub-total 3 5
II. Statements or
Citations
Indicating Low
Levels of Subject
Matter
Uncertainty
Identified During
Interviews 3 0
Identified in the
Public Record 0 2
Sub-total 3
2
III. Statements or
Citations
Indicating
Difficulty
Resolving
Uncertainty
Identified During
Interviews 1 4
Identified in the
Public Record 0 6
Sub-total 1 10
Quantitative Analysis
Medians and modes were determined for responses regarding the level of
uncertainty for both groups of cases: non-problematic and problematic. The results
are presented below. In addition, a statistical analysis appropriate for use with the
data produced by the interviews was used to determine if the population of
interviewee responses for non-problematic cases was different from that for
116
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
problematic cases. The numerical results can be found in Appendix D; they are
summarized below.
TABLE 16
QUANTITATIVE INFORMATION FROM INTERVIEWS
CONCERNING THE DEGREE OF UNCERTAINTY
Statistic Non-Problematic Cases Problematic Cases
Median 2.5 4.5
Mode 3 5
Inspection of the above results indicates that the median response from
interviewees concerning influence of uncertainty on recommendation implementation
for problematic cases is higher than that received for non-problematic cases. The
information available from the modes is consistent with this trend. Statistical
analysis, using the Mann-Whitney U-Test, also supports the conclusion that the
ratings for problematic cases were consistently higher than those received by non
problematic cases for the influence of subject matter uncertainty.
Summary
There was support for this hypothesis. There was a moderate amount of data
to analyze in the public record and from the interviews. Qualitative analysis of the
available data showed that a high level of subject matter uncertainty was associated
with problematic cases and that, conversely, low levels of subject matter uncertainty
were associated with non-problematic cases (Table 15, Rows I and II). The data
117
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
involving the resolution of uncertainty provides additional support; statements
indicating difficulty resolving uncertainty were ten times more frequent in
problematic cases than in non-problematic cases (Table 15, Row 1 1 1 ) .
The quantitative data available from the ratings showed that a significant
difference existed between the ratings provided for problematic and non-problematic
cases. The median of the responses for problematic cases indicated a '‘Great’' to
“Very Great Effect,” whereas the median for non-problematic recommendations
cases was “Moderate” to “Some Effect.” This data comports well with information
gained from follow-up interviews with the key informants, one of whom noted
succinctly: “The more uncertain the subject matter of the recommendation, the more
difficult it is for DOE to get the implementation approach right the first time.” In
this one statement, the informant covered one of the attributes used to settle on the
problematic or non-problematic nature of a recommendation (number of
implementation plans) explicitly and the other (length of time to complete
recommendation implementation) implicitly.
Hypothesis III. The more costly that a recommendation is
perceived to be, the more problematic will be its implementation.
Qualitative Analysis
Review of the statements from the interviewees and the citations form the
public record revealed two relevant content categories. There were a number of
statements and citations that describe the adequacy or inadequacy o f resources that
118
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
were available or utilized to implement a recommendation. On the side of adequacy,
the public record noted that: “Field offices report that they do not need any additional
resources,” to implement one recommendation; while, for another recommendation,
an interviewee stated that: “While DOE consistently denied that delays in
implementation were due to cost, it was clear that had more money been made
available, more sampling could have been done.” There was also a category of
statements and citations that stated or clearly alluded to reactions to a
recommendation that existed to due to a perception of high cost for the
implementation of the recommendation. For example, one interviewee stated
regarding a recommendation: “The work recommended by the Board needed to be
performed regardless of the cost”; while another recommendation was described by
an interviewee as “an unfunded mandate.” The statements and citations that
provided an indication of sufficiency of resources and reactions based on the
perception o f high cost are summarized in Table 17.
119
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
TABLE 17
MEASURES OF THE INFLUENCE OF THE PERCEPTION OF COST ON
THE IMPLEMENTATION OF NON-PROBLEMATIC AND
PROBLEMATIC CASES
Category o f
Information
Source Non-Problematic
Cases
Problematic Cases
I. Statements or
Citations
Indicating
Insufficient
Resources are
Available
Identified During
Interviews 0 5
Identified in the
Public Record 1 5
Sub-total I 10
II. Statements or
Citations
Indicating
Sufficient
Resources are
Available
Identified During
Interviews 2 0
Identified in the
Public Record 2 0
Sub-total 4 0
III. Statements or
Citations
Indicating
Reactions to a
Recommendation
due to the
Perception o f High
Cost
Identified During
Interviews 3 6
Identified in the
Public Record 0 2
Sub-total 3 8
Quantitative Analysis
Medians and modes were determined for responses regarding the influence of
the perception of cost on both groups of cases: non-problematic and problematic.
The results are presented below. In addition, a statistical analysis appropriate for use
with the data produced by the interviews was used to determine if the population of
120
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
interviewee responses for non-problematic cases was different from that for
problematic cases. The numerical results can be found in Appendix D and
summarized below.
TABLE 18
QUANTITATIVE INFORMATION FROM INTERVIEWS CONCERNING
THE INFLUENCE OF THE PERCEPTION OF COST ON
NON-PROBLEMATIC AND PROBLEMATIC CASES
Statistic Non-Problematic Cases Problematic Cases
Median 1 3
Mode 1 3
Inspection of the above results indicates that the median response from
interviewees concerning influence of the perception of cost on recommendation
implementation for problematic cases is higher than that received for non
problematic cases. The information available from the modes is consistent with this
trend. Statistical analysis, using the Mann-Whitney U-Test, also supports the
conclusion that the ratings for problematic cases were consistently higher than those
received by non-problematic cases for the influence of the perception of cost on
recommendation implementation.
Summary
There was support for this hypothesis. There was limited information from
the public record on this variable, somewhat more was provided during interviews.
121
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The available information indicated that a majority o f the statements indicating that
insufficient resources were available for implementation were associated with
problematic cases, whereas, all of the statements and citations that indicated
sufficient resources were available were associated with non-problematic cases
(Table 17, Rows I and II). Statements and citations that indicated a reaction (either
positive or negative) to a recommendation based on a perception of high cost
provided additional support for the hypothesis (Table 18, Row III).
Quantitative data indicated that there was a difference between problematic
cases and non-problematic cases; problematic cases were rated consistently higher—
a median rating of “Moderate Effect” as opposed to “Little or No Effect.” The
results regarding lack of resources during recommendation implementation were
consistent with the experience of the author and the key informants, one of whom
stated: “Discussion of cost was often considered by the Board to be resistance to the
recommendation, especially after acceptance.”
As discussed more fully in Chapter Four, when DOE accepts a
recommendation, they are certifying that the recommendation is feasible to
implement. Especially with respect to funding, the Board's enabling statute requires
that the Secretary of Energy report to the President if it is determined that “the
implementation of a Board recommendation (or part thereof) is impracticable
because of budgetary consideration” (U.S. Congress, 1988, p. 9). Although, as noted
by both key informants, this section has been the subject of on-going discussion
122
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
between the DNFSB and DOE. its impact has been to limit the amount of discourse
that occurs on the subject of the cost of implementing recommendations. This
peculiarity in the relationship between the DNFSBS and DOE has the potential to
limit the generalizability of insights gained from this study with respect to the
influence of the perception of cost on recommendations of scientific-technical
advisory committees in general.
Hypothesis IV. Frequent and non-adversarial communications
between a scientific-technical advisory committee and the advised
agency will make the implementation of a recommendation less
problematic.
Qualitative Analysis
Review of the statements from interviews and the citations from the public
record produced two relevant content categories for this variable. The content
categories provide insight into the attributes of the communications between the
DOE and the DNFSB. They include statements and citations that provided an
indication as to the frequency with which communications occurred between the two
agencies and as to the adversarial or non-adversarial nature of the communications.
The first content category', frequency of communications, provides some
insight into how actively the DOE and DNFSB engaged each other during the course
o f the recommendation. Specifically, these statements and citations provide an
assessment of the frequency of communications or discuss instances of direct
123
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
communications between the DOE and DNFSB groups that were working on the
implementation of the recommendation. Several interviewees provided positive
assessments, for example, noting that there were “frequent interactions between the
staffs [of the DOE and DNFSB] that provided focus to the implementation of the
recommendation.” There were no statements or citations that noted infrequent
communications.
The other category, which addressed the nature of communications, included
statements and citations that address the tone of the communications between the
DOE and DNFSB during recommendation implementation and note whether it was
adversarial or non-adversarial. One interviewee, for example, simply responded the
“the communications between the two agencies were non-adversarial in tone.” On
the other hand, concerning a different recommendation, an interviewee would
elaborate, “Communications between the DNFSB and the DOE and its contractors
were antagonistic—it was lawyers, lawyers, everywhere.”
The statements and citations that provided an indication of whether
communications were frequent and non-adversarial are summarized in Table 19.
124
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
TABLE 19
MEASURES OF THE INFLUENCE OF COMMUNICATIONS ON
NON-PROBLEMATIC AND PROBLEMATIC CASES
Category of
Iaformatioa
Source Non-Problematic
Cases
Problematic Cases
I. Statements or
Citations
Indicating that
Communications
were Frequent
Identified During
Interviews 5 2
Identified in the
Public Record 6 4
Sub-total 1 1 6
II. Statements or
Citations
Indicating that
Communications
were Adversarial
Identified During
Interviews I 6
Identified in the
Public Record 4 10
Sub-total 5 16
III. Statements or
Citations
Indicating that
Communications
were Non-
adversarial
Identified During
Interviews 5 I
Identified in the
Public Record 3 7
Sub-total 8
--------
8
Quantitative Analysis
Medians and modes were determined for responses regarding the influence of
communications on both groups of cases: non-problematic and problematic. The
results are presented below. In addition, a statistical analysis appropriate for use with
the data produced by the interviews was used to determine if the population of
interviewee responses for non-problematic cases was different from that for
125
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
problematic cases. The numerical results can be found in Appendix D; they are
summarized below.
TABLE 29
QUANTITATIVE INFORMATION FROM INTERVIEWS CONCERNING
THE INFLUENCE OF COMMUNICATIONS ON
NON-PROBLEMATIC AND PROBLEMATIC CASES
Statistic Non-Problematic Cases Problematic Cases
Median 4 3
Mode 4 3
Inspection of the above results indicates that the median response from
interviewees concerning influence of communications on recommendation
implementation for non-problematic cases is somewhat higher than that received for
problematic cases. The information available from the modes is consistent with this
trend. However, statistical analysis, using the Mann-Whitney U-Test, did not support
a conclusion that the ratings for non-problematic cases were consistently higher than
those received by problematic cases for the influence of communications on
recommendation implementation.
Summary
There was partial support for this hypothesis. Qualitative analysis of a
substantial number of statements from interviewees and citations from the public
record revealed two content categories that assist in understanding the influence of
126
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
this variable on the non-problematic implementation of DNFSB recommendations. A
majority of the statements and citations that indicated that communications were
frequent were made with regard to non-problematic cases; no statements or citations
were found that discussed infrequent communications (Table 19, Row I). Statements
and citations that indicated adversarial communications were more than three times
as prevalent in problematic cases; on the other hand, statements and citations that
indicated that communications were non-adversarial were evenly split between
problematic and non-problematic cases (Table 19, Rows II and III). Therefore, the
information in this content category does not support the hypothesis.
Further, analysis with the appropriate statistic methodology (Appendix D)
revealed no difference in the perceived effects of communications in problematic and
non-problematic cases.
These results were discussed with the key informants. Their thoughts
regarding the quantitative results were summed up by the following statement made
by one; “That’s about right. It takes the same amount effort to keep communications
going, whether or not problems with the implementation of a recommendation have
developed.” In addition, the key informants focused on the frequency of
communications, one of them noting, “frequency of communications was vital,
especially between the principals, where this did not occur, problems ensued.”
127
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Hypothesis V. Clearly defined roles and responsibilities within
the advised agency will make the implementation of a
recommendation less problematic.
Qualitative Analysis
Assessment of the citations from the public record and statements from the
personnel interviewed produced two relevant categories of content for this variable.
These categories address two attributes of the arrangements necessary for
implementation of recommendations. These attributes are the organization of the
team tasked with implementing the recommendation, or their roles and
responsibilities, and the design o f the implementation approach described in the
implementation plan (which can be considered a subset o f role definition, but was
broken out separately for added insight).
The first content category includes statements and citations that discuss
whether nor not the roles and responsibilities of the implementation team within
DOE were clearly defined. Important aspects of this attribute included a clearly
identified lead for the recommendation within the senior management of DOE and
definition of that individuals relationship with groups within the DOE whose efforts
were necessary to execute the implementation plan. In discussing one
recommendation an interviewee noted, “roles and responsibilities were clear within
DOE and the Principle Deputy Assistant Secretary for Defense Programs was in
charge”; on the opposite side of the spectrum were citations such as, “these delays
128
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
are traceable to DOE’s failure to establish responsible ownership of implementation
plan commitments/'
Citations and statements in the next category are those that related to the
development or design of the implementation plan. They often address whether the
plan was considered simple or complex. This category also covers statements and
citations that describe formal changes needed to the implementation plan after it had
been approved. An example of such a statement is, “there was a fairly simple
structure to this implementation plan that smoothed implementation.” When there
were problems within this category, they were also given voice; for example, “the
decision points established by the implementation plan were too indistinct.”
In summary, these categories answer two important questions about the
implementation of a recommendation. Who is in charge? Is there an adequate plan?
The statements and citations that provide an indication of the answers to these
questions for non-problematic and problematic cases are summarized in Table 21.
129
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
TABLE 21
MEASURES OF THE INFLUENCE OF ORGANIZATIONAL
ARRANGEMENTS ON NON-PROBLEMATIC AND
PROBLEMATIC CASES
Category of
Iaformatioa
Soarce Non-Problematic
Cases
Problematic Cases
I. Statements or
Citations
Indicating
Unclearty Defined
Roles and
Responsibilities
Identified During
Interviews 2 5
Identified in the
Public Record 2 11
Sub-total 4 16
II. Statements or
Citations
Indicating Clearly
Defined Roles and
Responsibilities
Identified During
Interviews 6 I
Identified in the
Public Record 2 0
Sub-total 8 1
III. Statements or
Citations
Indicating
Inadeqnate Design
in an
Implementation
Plan
Identified During
Interviews 0 7
Identified in the
Public Record 0 18
Sub-total 0 25
IV. Statements or
Citations
Indicating
Adequate Design
in an
Implementation
Plan
Identified During
Interviews 7 0
Identified in the
Public Record 1 4
Sub-total 8 4
130
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Quantitative Analysis
Medians and modes were determined for responses regarding the influence of
organizational arrangements on both groups of cases: non-problematic and
problematic. The results are presented below. In addition, a statistical analysis
appropriate for use with the data produced by the interviews was used to determine if
the population of interviewee responses for non-problematic cases was different from
that for problematic cases. The numerical results can be found in Appendix D; they
are summarized below.
TABLE 22
QUANTITATIVE INFORMATION FROM INTERVIEWS CONCERNING
THE INFLUENCE OF ORGANIZATIONAL ARRANGEMENTS ON
NON-PROBLEMATIC AND PROBLEMATIC CASES
Statistic Non-Problematic Cases Problematic Cases
Median 3.5 4
Mode 3 or 4 4
Inspection of the above results indicates that the median response from
interviewees concerning influence of organizational arrangements on
recommendation implementation for problematic cases is slightly higher than that
received for non-problematic cases. The information available from the modes is
somewhat equivocal, since one of the two modes for non-problematic cases is the
same as that for problematic cases, while the other is somewhat lower. The statistical
analysis, using the Mann-Whitney U-Test, did not support a conclusion that the
131
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ratings for problematic cases were consistently higher than those received by non
problematic cases for the influence o f organizational arrangements on
recommendation implementation.
Summary
There was support for this hypothesis. Analysis of substantial qualitative
information from the interviews and public record revealed two relevant categories
of content. Statements and citations that addressed the clarity of the roles and
responsibilities assigned to the organization or team established within DOE to
implementation the recommendation was one. This data showed that statements or
citations mentioning unclear roles and responsibilities were four times more likely to
be made concerning problematic cases, conversely, positive statements and citations
regarding the clarity of roles and responsibilities of the DOE team were eight times
more likely for non-problematic recommendations (Table 21, Rows I and II). The
association regarding implementation plan design was also fairly clear. There were a
substantial number of statements and citations that indicated that implementation
plans were inadequately designed for problematic cases and no such statements or
citations for non-problematic cases (Table 21, Row III). On the other hand,
statements indicating adequate design of an implementation plan (Table 22, Row IV)
were twice as likely for non-problematic cases. Statistical testing revealed no
difference in the perceived effects of organizational arrangements on implementation
for problematic and non-problematic cases.
132
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
Another Look at the Data
In looking at the data for all cases, taken together, we can attempt to address a
slightly different question. Overall, what variables where consistently rated by the
interviewees as having a more substantial impact than others? This information will
provide a rough, relative rating o f whether a variable was important regardless of the
type of recommendation (non-problematic or problematic), or the agency of the
interviewee (DOE or DNFSB).
TABLE 23
ANOTHER LOOK AT THE EFFECT OF VARIABLES ON THE
IMPLEMENTATION OF SCIENTIFIC-TECHNICAL ADVISORY
COMMITTEE RECOMMENDATIONS—STATISTICAL DATA
CONCERNING PERCEIVED EFFECT (ALL CASES)
Data
Element
Interest
Group
Involvement Uncertainty
Perception
of Cost
Communica
tions
Organizational
Arrangements
1. Overall
Median of
Interview
Ratings
(Kale of 1
to 5)
2 3
2
3.5 4
II. Overall
Mode of
Interview
Ratings
(scale of 1
to 5)
1 5 1 3 4
133
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
TABLE 24
ANOTHER LOOK AT THE EFFECT OF VARIABLES ON THE
IMPLEMENTATION OF SCIENTIFIC-TECHNICAL ADVISORY
COMMITTEE RECOMMENDATIONS—AGGREGATE NUMBER
OF CITATIONS AND STATEMENTS (ALL CASES)
Data Element
laterest
Group
lavolvemeat Uacertaiaty
Perception
of Cost
Communi
cations
Organizational
Arrangements
1. Number of
Applicable
Statements
from
Interviews
Available for
Qualitative
Analysis
8 11 15 20 28
II. Number of
Applicable
Citations
from Public
Record
Available for
Qualitative
Analysis
6 13 8 34 38
Taken together, these two tables indicate that one variable stands out as
having been rated consistently high, on a recommendation by recommendation basis,
by both the DOE and DNFSB interviewees. That variable is organizational
arrangements. Referring back to Table 11, it will be noted that this variable received
no individual rating below “Moderate Effect.” This helped to produce both the a
median and a modal rating of “4” or “Great Effect.” This rating is consistent with
the number of citations on the public record and statements from interviewees
associated with organizational arrangements, which are both highest for this variable,
134
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
such as roles and responsibilities of the personnel assigned to implement a
recommendation and the development of the implementation plan. The
communications variable had a median rating of slightly above “Moderate Effect”
(3.5). Only three o f sixteen interview ratings (Table 11) were below “Moderate
Effect,” and all of those were “Some Effect.” Like organizational arrangements,
discussion of the extent and nature of communications between the DOE and
DNFSB garnered a fairly substantial portion of the citations analyzed from the public
record and statements made by interviewees. Therefore, both of the structural
variables garnered overall ratings higher than “Moderate Effect” from the personnel
interviewed and had a larger number of statements and citations available for
analysis.
All three of the environmental variables received overall ratings of “Moderate
Effect” or below. Uncertainty was rated the highest, garnering a median rating of
“Moderate Effect,” although there was a broad spread of ratings. (Note that “Very
Great Effect” was the most frequent rating being used by just under a third of the
interviewees.) The perception of cost and interest group involvement were both
rated even lower, with median ratings of “Some Effect,” and the most frequently
used rating for both of these variables was “Little or No Effect.” (For example, it
was used by almost half o f the interviewees to describe the effect of interest group
involvement.) Also, the number of citations from interviewees and statements on the
public record were appreciably smaller for all environmental variables.
135
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
In summary, the overall interviewee ratings indicate that the structural
variables seemed to have more than a moderate influence on the implementation of
recommendations, regardless of whether the recommendation had been screened as
non-problematic or problematic. In the case of the environmental variables, the
overall ratings that they received were, from the standpoint of central tendencies,
lower than those received by the structural variables and rated to have only a
moderate influence on recommendations (or less). The substantially greater attention
paid to structural variables, as compared to environmental variables, provides
indirect support to this apparent association.
Recommendations for Further Research
This dissertation addressed the relationship between one scientific-technical
advisory committee (the DNFSB) and the agency it advises (the DOE). Additional
case studies of the relationship between other scientific-technical advisory
committees and advised agencies would assist in further understanding the influence
that the studied variables have on the non-problematic implementation of
recommendations from such committees. Additional case studies would also allow
the influence of agency-specific structural differences (e.g., the somewhat limited
access that the public has to the DNFSB recommendation process) to be explored.
The explication of such agency-specific structural differences would be valuable in
and of itself. These further case studies could also identify additional variables of
importance.
136
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Since at least some support was found for each of the five independent
variables, additional study o f these variables within the DNFSB-DOE context might
also produce further insight. A structured interview or survey approach could be
designed to better refine the impressions gathered in this study. It could be
administered to the personnel interviewed during this dissertation to further refine the
understanding of the set of independent variables and their influence on the non
problematic implementation of recommendations by scientific-technical advisory
committees. Such a survey might also be pursued in the additional case studies
recommended above.
137
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
SELECTED BIBLIOGRAPHY
Bawn, K. (1995). Political control versus expertise: Congressional choices about
administrative procedures. American Political Science Review, 89 (1). 62-73.
Behn, R. D. (1995). The big questions of public management. Public
Administration Review, 55 (4), 313-324.
Bernstein, S., & Bernstein, R. (1999). Elements o f statistics II: Inferential statistics.
New York: McGraw Hill.
Brown, D. S. (1954). The public advisory board in the federal government: An
administrative analysis o f several boards with particular attention to the
Public Advisory Board o f the Economic Cooperation Administration and the
Mutual Security Program. Unpublished doctoral dissertation, Syracuse
University.
(1955). The Public Advisory Board as an instrument of government.
Public Administration Review, 15(2), 196-204.
________ (1972). The management of advisory committees: An assignment for the
70's. Public Administration Review, 32 (4), 334-342.
Caudle, S. (1994). Using qualitative approaches. In J. Wholey, H. Hatry, & K.
Newcomer, Handbook o f practical program evaluation (pp. 69-95). San
Francisco, CA: Jossey-Bass Publishers.
Cohen, M. D., March, J. G., & Olsen, J. P. (1972). A garbage can model of
organizational choice. Administrative Science quarterly, 17 (1). 1-25.
Defense Nuclear Facilities Safety Board (1990a, March 14). Recommendation 90-
2—Standards. Letter from J. T. Conway to J. D. Watkins.
________ (1990b, March 27). Recommendation 90-3—Future Hanford tank
monitoring. Letter from J. T. Conway to J. D. Watkins.
138
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(1990c, May 4). Recommendation 90-4— ORRs at Rocky Flats. Letter
from J. T. Conway to J. D. Watkins.
(1990d. October 12). Recommendation 90-7—Modifications required to
implementation plan fo r Recommendation 90-3. Letter from J. T. Conway to
J. D. Watkins.
(1991a, April 25). Recommendation 91-3— Waste isolation pilot plant
ORR. Letter from J. T. Conway to J. D. Watkins.
(1991b, May 20). Letter from J. T. Conway to J. D. Watkins.
(1991c, September 30). Recommendation 91-4—Rocky Flats building 559
ORR. Letter from J. T. Conway to J. D. Watkins.
(199Id, December 19). Recommendation 91-6—Radiation protection.
Letter from J. T. Conway to J. D. Watkins.
(1991 e, December 20). Recommendation 91-5—Power limits for K-
Reactor operations at the Savannah River site. Letter from J. T. Conway to
J. D. Watkins.
(1992a, January 24). Letter from J. T. Conway to J. D. Watkins.
(1992b, May 28). Recommendation 92-2—DOE's Facility Representative
Program. Letter from J. T. Conway to J. D. Watkins.
(1992c, August 4). Letter from J. T. Conway to J. D. Watkins.
(1992d, August 5). Letter from J. T. Conway to J. D. Watkins.
(1992e, August 14). Letter from J. T. Conway to J. D. Watkins.
(1992f, August 27). Recommendation 92-6—Operational readiness
reviews. Letter from J. T. Conway to J. D. Watkins.
(1993a, January 15). Letter from J. T. Conway to L. Stuntz.
(1993b, March 23). Letter from J. T. Conway to L. Stuntz.
(1993c, April 1993). Annual report to Congress (pp. 4-5).
139
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
________ (1993d, June 16, August 6). Recommendation 93-4—Environmental
restoration management contracts. Letter from J. T. Conway to H. R.
O'Leary.
________ (1993e, September 3). Letter from J. T. Conway to H. R. O'Leary.
________ (1993f, December 10). Recommendation 93-6—Maintaining access to
nuclear weapons expertise. Letter from J. T. Conway to H. R. O'Leary.
________ (1995a, April 5). Letter from J. T. Conway to V. H. Reis.
________ (1995b. April 24). Letter from J. T. Conway to H. R. O'Leary.
________ (1995c, July 24). Letter from J. T. Conway to H. R. O'Leary.
________ (1996a, March 14). Letter from J. T. Conway to V. H. Reis.
________ (1996b, July 22). Letter from J. T. Conway to H. R. O'Leary.
________ (1998, April 15). Letter from J. T. Conway to V. H. Reis.
________ (1999a, April 27). Letter from J. T. Conway to W. Richardson.
________ (1999b, June). Recommendations calendar, internal memo.
Department of Energy (1990a, March 30). Letter from J. D. Watkins to J. T.
Conway.
________ (1990b, April 18). Letter from J. D. Watkins to J. T. Conway.
________ (1990c, June 8). Letter from J. D. Watkins to J. T. Conway.
________ (1990d, August 10). Letter from J. D. Watkins to J. T. Conway.
________ (1990e, September 14). Letter from J. D. Watkins to J. T. Conway.
________ (1990f, December 3). Letter from J. D. Watkins to J. T. Conway.
________ (1991a, August 16). Letter from J. D. Watkins to J. T. Conway.
________ (1991b, November 11). Letter from J. D. Watkins to J. T. Conway.
140
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(1992a, February 7). Letter from J. D. Watkins to J. T. Conway.
(1992b, November 5). Letter from J. D. Watkins to J. T. Conway.
(1993a, January 15). Letter from J. D. Watkins to J. T. Conway.
(1993b, January 30). Letter from L. Stuntz to J. T. Conway.
(1993c, April 26). Guidelines for establishing and maintaining a Facility
Representative Program, DOE nuclear facilities.
(1993d. April 27). Letter from H. R. O'Leary to J. T. Conway.
( 1993e, July 29). Letter from H. R. O'Leary to J. T. Conway.
(1993f, September 30). Letter from H. R. O'Leary to J. T. Conway.
(1993g, October 5). Letter from P. N. Brush to J. T. Conway.
(1993h, November 8). Letter from H. R. O'Leary to J. T. Conway.
(1993i, December 30). Letter from H. R. O'Leary to J. T. Conway.
(1994a, February 2). Letter from H. R. O'Leary to J. T. Conway.
(1994b, April 1). Letter from H. R. O'Leary to J. T. Conway.
(1994c, April 11). Letter from T. Grumbly to J. T. Conway.
(1994d. April 28). Letter from T. Grumbly to J. T. Conway.
(1994e, July 5). Letter from H. R. O'Leary to J. T. Conway.
(1995a, January 25). Letter from T. O'Toole to J. T. Conway.
(1995b, February 10). Letter from V. H. Reis to J. T. Conway.
(1995c, May 1). Letter from V. H. Reis to J. T. Conway.
(1995d, July 7). Letter from V. H. Reis to J. T. Conway.
(1995e, September 19). Letter from FI. R. O'Leary to J. T. Conway.
141
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
________ (1996a, February 13). Letter from H. R. O'Leary to J. T. Conway.
________ (1996b, April 1). Letter from V. H. Reis to J. T. Conway.
Denzin, N. K., & Lincoln, Y. S. (Eds.) (1998). Collecting and interpreting
qualitative materials. Thousand Oaks. CA: Sage Publications.
Durant, R. F. (1985). When government regulates itself: EPA, TV A and pollution
control in the 1970's. Knoxville, TN: University o f Tennessee Press.
Friedman, R. S. (1978). Representation in regulatory decision making: Scientific,
industrial, and consumer inputs to the FDA. Public Administration Review.
38 (3), 205-214.
General Accounting Office (1990). Case study evaluations, GAO/PEMD-10.1.9.
Washington, DC: U.S. Government Printing Office.
________(1996). Content analysis: A methodology for structuring and analyzing
written material, GAO/PEMD-10.3.1. Washington, DC: U.S. Government
Printing Office.
________ (1998, June). Federal Advisory Committee Act: General Services
Administration's oversight o f advisory committees, GAO/GGD-98-124.
Washington, DC: U.S. Government Printing Office.
________(1998, July). Federal Advisory Committee Act: Views o f committee
members and agencies on federal advisory committee issues, GAO/GGD-98-
147. Washington, DC: U.S. Government Printing Office.
General Services Administration (1999). Twenty-seventh annual report o f the
president o f federal advisory commiitees. Washington, DC: U.S.
Government Printing Office.
George, G. R. (1995). Negotiated safety: Intragovernmental risk regulations in the
U.S. nuclear weapons complex. Unpublished doctoral dissertation, Harvard
University.
Givens, J. (1987). An evaluation o f the effectiveness o f the National Advisory
Committee on Accreditation and Institutional Eligibility. Unpublished
doctoral dissertation, The Catholic University of America.
142
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Glaser, B. G. (1992). Basics o f grounded theory analysis. Mill Valley, CA:
Sociology Press.
Glaser, B. G., & Strauss. A. L. (1967). The discovery o f grounded theory, strategies
for qualitative research. Chicago, IL: Aldine Publishing Company.
Gulick, L. (1957). Who are we to believe? Public Administration Review, 77(1),
106-110.
Huberman, A. M., & Miles, M. B. (1998). Data management and analysis methods.
In N. K. Denzin & Y. S. Lincoln (Eds.), Collecting and interpreting
qualitative materials (pp. 179-210). Thousand Oaks, CA: Sage Publications.
Jasanoff, S. (1990). The fifth branch: Science advisors as policymakers. Cambridge,
MA: Harvard University Press.
Kaufman. H. (1956). Emerging conflicts in the doctrines of public administration.
American Political Science Review, 50(4), 1057-1073.
Kirlin, J. J. (1996). The big questions of public administration in a democracy.
Public Administration Review, 56 (5), 416-423.
Lyden, F. J. (1972). Advisory committees and participatory government. Public
Administration Review, 32 (3), 256-257.
Madison, J. (1961). Federalist no. 51. In Clinton Rossiter, The federalist
papers—Alexander Hamilton, James Madison and John Jay. New York:
Mentor Books.
Manning, P. K., & Callum-Swan, B. (1998). Narrative, content, and semiotic
analysis. In N. K. Denzin & Y. S. Lincoln (Eds.), Collecting and interpreting
qualitative materials (pp. 246-274). Thousand Oaks, CA: Sage Publications.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.).
Thousand Oaks, CA: Sage Publications.
McCubbins, M. M. (1995). The legislative design of regulatory procedure.
American Journal o f Political Science, 29, 721-748.
Natural Resources Defense Council (1990, July 21). Letter from B. Costner to J. T.
Conway.
143
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Osbome, R. N., & Hunt, J. G. (1974). Environment and organizational effectiveness.
Administrative Science Quarterly, 19 (2), 231-246.
O'Toole. L., & Montjoy, R. (1986). Interorganizational policy implementation: A
theoretical perspective. Public Administration Review, 46 (6), 491-503.
Patton, M. (1990). Qualitative evaluation methods (2nd ed.). Newbury Park. CA:
Sage Publications.
Pfeffer, J., & Salancik, G. R. (1978). The external control o f organizations: A
resource dependence perspective. New York: Harper & Row.
Provan, K. G., & Milward, H. B. (1995). A preliminary theory of interorganizational
effectiveness: A comparative study of four community health systems.
Administrative Science Quarterly, 40(1). 1-33.
Sprent, P. (2000). iWon-parametric statistical methods (2nd ed.). Boca Raton, FL:
Chapman & Hall/CRC.
Turner, B. A. (1976). The organizational and interorganizational development of
disasters. Administrative Science quarterly, 21 (3), 378-397.
U.S. Congress (1949). Administrative Procedures Act. U.S. Code. Washington. DC:
U.S. Government Printing Office.
________ (1972). Federal Advisory Committee Act o f 1971, 5 U.S. Code, Appendix
A. Washington, DC: U.S. Government Printing Office.
________ (1988). Defense Nuclear Facilities Safety Board, 42 United States Code
(U.S.C.), Section 2286. Washington, DC: U.S. Government Printing Office.
Vaughan, D. (1990). Autonomy, interdependence, and social control: NASA and the
space shuttle Challenger. Administrative Science Quarterly, 35 (2), 225-258.
Weber, R. P. (1990). Basic content analysis (2nd ed.). Newbury Park, CA: Sage
Publications.
Weiss, N. (1995). Introductory statistics (4th ed.). Reading, MA: Addison-Wesley
Publishing Company.
Wholey, J., Hatry, H., & Newcomer, L. (1994). Handbook o f practical program
evaluation. San Francisco, CA: Jossey-Bass Publishers.
144
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Wilson. A. S. (1983). The National Advisory Committee on Oceans and
Atmosphere: The effectiveness o f a public advisory committee. Unpublished
doctoral dissertation. University of Delaware.
Yin, R. K. (1993). Applications o f case study research. Newbury Park, CA: Sage
Publications.
_________(1994). Case study research: Design and methods (2nd ed.). Thousand
Oaks, CA: Sage Publications.
145
R eproduced with perm ission of the copyright owner. Further reproduction prohibited without permission.
APPENDICES
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX A
INTERVIEW OUTLINE FOR INTERVIEW WITH
DR. G. W. CUNNINGHAM,
TECHNICAL DIRECTOR, DNFSB
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Interview Outline fo r Interview with Dr. G. W . Cunningham,
Technical Director, DNFSB
1. Overview o f the proposed study, emphasizing the intent to identify factors which
contribute to and detract from effective implementation of recommendations.
2. Description of, and discussion pertaining to, the criteria that was used to separate
problematic from non-problematic recommendations: single implementation
plan and implemented within the average (median) time frame for
recommendation implementation.
3. Discussion of a preliminary list of recommendations that meet the screening
criteria for problematic and non-problematic recommendations (still be
finalized).
a. Any general comments on the listed recommendations.
b. Comments concerning extent of available public record for each
recommendation.
4. Development of a list of DNFSB leads for the recommendations to be
interviewed.
5. Review of the interview guide to be used to interview DNFSB (and DOE)
recommendation leads.
6. Any closing comments.
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX B
EXAMPLE RECOMMENDATION DATA TABLE
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced w ith permission o f the copyright owner. Further reproduction prohibited without permission.
EXAMPLE OF PUBLIC RECORD ANALYSIS
(90-3/90-7 Public Record)
Dale Reference Pages lateresl
Group
Uacertaiaty Cost Commuukaliou Organisation
Arraageaieats
3/27/90 DNFSB
Rec. 90-3
3 This potential
problem had
been brought to
the attention of
the Board
during its
confirmation
hearing.
. . . one of the topics
discussed was the
question o f the
susceptibility o f the
old single shell high
level tanks to an
explosion o f a
spontaneous nature,
with resulting release
o f large amounts of
radioactive material to
the environment.
-A ll evidence
available indicates that
the conditions that
might have
contributed to a higher
probability of such an
explosion were more
prevalent in the past
than they are now, and
these conditions are
continuing to lessen as
time passes.
-However the Board
does have some
residual concerns
resulting from the
uncertainty of
information (40 words
of detail).
The Board considers
the matter o f
hydrogen generation
in the double walled
tanks to be
potentially more
serious than the
questions related to
single shell tanks,
and is pursuing it
separately.
The Board recommends
that the DOE:
-develop a program for
continuous monitoring of
those conditions in the
single shell tanks that can
serve to indicate
development o f conditions
indicating the onset of
instability in their contents
(31 more words o f
technical detail).
-that the instruments used
in monitoring the tanks be
provided with alarm
indicators at a location
where decisions can be
made to start a series of
measures to neutralize a
perceived abnormality,
-that an action plan be
developed for the measures
to be taken to neutralize the
conditions that may be
signaled by the alarms.
APPENDIX C
INTERVIEW FORM FOR RECOMMENDATION LEADS
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Interview for Recommendation Leads
Recommendation: Date:
Name: Telephone:
DNFSB or DOE E-mail:
Introduction
This research is looking into the impact, if any, of a number of factors on whether
DNFSB Recommendations are easier or more difficult to implement. What will
follow is a series of six questions that will ask you to: first, discuss the prominence
that a given factor had in the discussions that took place during the life-cycle of the
subject recommendation; then you will be asked to provide a numeric rating of the
“strength” of that factor, as regards impacting the discourse on the given
recommendation. The ratings will range from 1 to 5, with the ratings defined as
follows:
1 - Little or no impact
2 - Some impact
3 - A moderate level of impact
4 - Great impact
5 - Very great impact
1 5 2
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Environmental Factors
1. Interest Groups
a. Please describe the prominence played by interest groups (such as Congress,
the courts, various public interest groups) in the discourse during the life
cycle of Recommendation .
b. Do you recall how many were involved? What were some of their names?
c. Would you please rate the impact of interest groups on the life-cycle of this
recommendation, using the above scale (repeat ratings, as appropriate)____
2. Uncertainty
a. Please describe the prominence played by uncertainty (in other words, the
unknown nature of some information or the technical complexity of the
situation) in the discourse during the life-cycle of Recommendation .
b. Can you describe some of the sources of uncertainty or complexity?
c. Would you please rate the impact of uncertainty on the life-cycle of this
recommendation, using the above scale (repeat ratings, as appropriate)_____
Mediating Factor
Cost
a. Please describe the prominence played by the perception of the cost of
implementing Recommendation in the discourse during its life-cycle.
b. Were any actual cost estimates performed?
153
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
c. Would you please rate the impact of cost on the life-cycle of this
recommendation, using the above scale (repeat ratings, as appropriate)____
Structural Factors
1. Communications
a. Please describe the nature of communications during the life-cycle of
Recommendation (i.e., did they occur frequently, were they open and
non-adversarial?)
b. Were there any process limitations placed on the communications?
c. Would you please rate the impact of communications on the life-cycle of this
recommendation, using the above scale (repeat ratings, as appropriate)_____
2. Organization—Well Defined Roles & Responsibilities
a. Please describe the extent to which you believe roles and responsibilities of
those tasked with implementing Recommendation where clearly
understood.
b. Was there sufficient independence for the team implementing
Recommendation and did you fell you had the support of senior
decision-makers?
c. Would you please rate the impact of clarity of roles & responsibilities on the
life-cycle of this recommendation, using the above scale (repeat ratings, as
appropriate)_____
154
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
3. Organization—Operating Procedures
a. Please describe the procedures set up by DOE and the DNFSB for the
implementation of Recommendation (i.e.. were they well defined?)
b. How many major decision points were there?
c. Would you please rate the impact the operating procedures that were set up
on the life-cycle of this recommendation, using the above scale (repeat
ratings, as appropriate)_____
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX D
STATISTICAL ANALYSIS
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Statistical Analysis
The interviews with the recommendation leads produced a fairly large
database of information. At this point a determination had to be made as to which
statistical methods (both descriptive and inferential) were appropriate to use. given
the nature of the data. The discussion below provides a short description of the
statistical methods used and provides the results of the statistical analysis performed
in this study.
The data developed during the interviews is comprehensively displayed in
Table 11, “Overall Interview Ratings Results,” which is shown at the end of Chapter
Four, above. It can be seen, from this table, that there are sixteen ratings for each
variable. There are two ratings per variable for each of the eight cases studied; one
from the DNFSB lead for the recommendation studied in the case and one from the
DOE person principally involved. These sixteen ratings were split into two groups
for further analysis: the ratings for problematic cases and those for non-problematic
cases. Comparisons between these two groups provided insights into variables that
could be correlated with the problematic or non-problematic nature of the cases.
The measures o f central tendency that are appropriate for the ordinal data that
resulted from the interview ratings are the median and mode. The median is
determined by arraying the data in rank order and finding the middle-most value. On
the other hand, the mode is the most frequently occurring value. When these
157
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
assessments are performed for the interviewee rating data from Table 11, the results
are as shown therein. They are reproduced below for continuity purposes.
Table 1
Medians and Modes of Interviewee Rating Data
Statistic
laterest
Group
lavolvemeat Uncertainty
Perception
of Cost Communications
Organizational
Arrangements
Overall
Median* 2 3 2 3.5 4
Overall
Mode* 1 5 1 3 4
Median for
Problematic
Cases 2.5 4.5 3 3 4
Mode for
Problematic
Cases 2 or 4 5 3 3 4
Median for
Non-
Problematic
Cases 1 2.5 1 4 3.5
Mode for
Non-
Problematic
Cases 1 3 1 4 3 or 4
*D ata from all eight cases
The overall medians and modes (that is, the statistics that are developed using
the sixteen ratings provided for the eight cases studied) provide an indication of the
overall judgement of the interviewees as to the influence that the variable had upon
the implementation of recommendations, regardless of whether the recommendation
was deemed to be problematic or non-problematic. This information shows us that
1 5 8
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
three of the five variables are rated by the interviewees as having a moderate effect or
greater. They include both structural variables (communications and organizational
arrangements) and the perception of cost.
Looking more closely at the data, we compare the medians for problematic
recommendations to the medians for non-problematic recommendations on a
variable-by-variable basis. The medians and modes for the three environmental
variables (interest group involvement, uncertainty and the perception of cost) appear
to be different for the two groups of cases, whereas the medians and modes for the
structural variables appear similar. Further analysis requires the use of a inferential
statistic that is appropriate for the type of data, that is. ordinal data wherein a normal
distribution cannot be assumed. The Mann-Whitney U-test provides just such a test;
Bernstein and Bernstein note that the “Mann-Whitney U -test. . . is a rank order, non-
parametric test for determining whether two independent samples come from the
same population." It is similar in application to the two-sample t-test but is used
when it is not appropriate to assume a normal distribution and when the data is
ordinal-level rather than interval or ratio-level measurements (as in the present case).
The null hypothesis is that the distributions of the two populations (from which the
samples were taken) are the same, while the alternative hypothesis is that they are not
the same. The appropriate formulae are as follows:
U, = {n,n2 + [n,(n,+l)]/2} - R,
U2 = {n,n2 + [n2 (n2+l)J/2} - R2
159
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The lesser of the two values of U are compared to critical values from a table to
determine whether or not the null hypothesis is rejected (Bernstein & Bernstein,
1999, p. 389). The results of applying the Mann-Whitney U-test to the above data
are provided in Table 2.
Table 2
Results of Applying the Mann-Whitney U-Test to
Data Regarding the Independent Variables for the Present Study
Independent
Variable Calculated U
Critical U for
90% Confidence Result Comment
Interest Group
Involvement
10 15 Reject H „ Indicates that the
two population
distributions are
not the same.
Uncertainty 13.5 15 Reject H0 Indicates that the
two population
distributions are
not the same.
Perception of
Cost
5.5 15 Reject H0 Indicates that the
two population
distributions are
not the same.
Communications 29.5 15 Cannot Reject H „ Supports the null
hypothesis that
the distribution
of the two
populations is the
same.
Organizational
Arrangements
18 15 Cannot Reject H0 Supports the null
hypothesis that
the distribution
of the two
populations is the
same.
1 6 0
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The data that was collected in the eight cases that were reviewed provided a
broad variety of responses to the questions that addressed the above hypotheses. The
data were reviewed to determine whether any assumptions appeared appropriate
regarding the distribution o f ratings and if systematic biases were observed between
the responses received from DOE and DNFSB interviewees. The results of this
assessment are provided below.
Table 3
Comparison of DOE and DNFSB Interviewee Responses
Value
DNFSB Interviewees DOE Interviewees
Total
Non-
Problematic
Cases
Problematic
Cases
Non-
Problematic
Cases
Problematic
Cases
Number of
Times “ 1 ”
was used 6 0 7 2 15
Number of
Times “2”
was used 4 4 3
2
1 3
Number of
Times “3”
was used 4 7 6 4 21
Number of
Times “4”
was used 3 7 4 4 18
Number of
Times “5”
was used 3
2
0 8 1 3
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Value
DNFSB Interviewees DOE Interviewees
Total
Non-
Problematic
Cases
Problematic
Cases
Non-
Problematic
Cases
Problematic
Cases
Median of
Ratings by
Case
Grouping 2.5 3 2.5 4 3
Median
Rating by
Agency 3 N/A
It can be seen that all responses in the Likert scale were used by the
interviewees. In addition, at least in aggregate, the responses provided by personnel
from the two agencies were consistent. That conclusion was supported by
application of the Mann-Whitney U-test, described above. In addition, it is noted
that both agencies should slightly higher ratings for problematic cases. Therefore no
systematic biases from either the DOE or DNFSB are readily apparent.
R eproduced with permission of the copyright owner. Further reproduction prohibited without permission.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

An analysis of third -party regulatory audit systems for medical device safety

PDF

Defending against bioterrorism: Lessons from the 2001 anthrax attacks

PDF

A configuration study of multiagency partnerships as practiced in Taipei City government

PDF

An organizational history of the National Aeronautics and Space Administration: A critical comparison of administrative decision making in two pivotal eras

PDF

Drug treatment providers' organizational responses to implementation of California's Proposition 36

PDF

Determining acceptable seismic risk: A community participation-based approach

PDF

Federal mentor -protege programs: A pilot study on protege survival and growth

PDF

A financial audit model for entrepreneurial governments

PDF

Banking on the commons: An institutional analysis of groundwater banking programs in California's Central Valley

PDF

Interoperability among complex defense computer -based systems: The Navy case

PDF

Evaluation of professional services consultants in rural government

PDF

Bridled boldness: Reengineering the Federal Bureau of Prisons. Lessons learned

PDF

Information overload: Exploring management of electronic mail

PDF

A search for theory: Performance management to improve transportation safety

PDF

Assessing ombudsman performance: Two case studies in North America

PDF

Internal venturing in public agencies

PDF

Getting 'how' and 'why' straight: A critical discourse analysis of the National Partnership for Reinventing Government's ideological discourse on information and communication technologies

PDF

Assessing United States information assurance *policy response to computer -based threats to national security

PDF

Educational reform for private state -approved schools in California

PDF

Ideologies and champions: Universal Service from Congress to your telephone bill

Asset Metadata

Creator Krahn, Steven Lee (author)

Core Title A preliminary assessment of the influence of certain variables on the implementation of recommendations by scientific -technical advisory committees

School School of Policy, Planning and Development

Degree Doctor of Public Administration

Degree Program Public Administration

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag OAI-PMH Harvest,Political Science, public administration

Language English

Contributor Digitized by ProQuest (provenance)

Advisor Gordon, Cameron (committee chair), Kimm, Vic (committee member), Wholey, Joseph (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c16-197389

Unique identifier UC11334764

Identifier 3065810.pdf (filename),usctheses-c16-197389 (legacy record id)

Legacy Identifier 3065810.pdf

Dmrecord 197389

Document Type Dissertation

Rights Krahn, Steven Lee

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA