University of Southern California Dissertations and Theses

Identifying the point of diminishing marginal utility for cognitive task analysis surgical subject matter expert interviews

(USC Thesis Other)

Identifying the point of diminishing marginal utility for cognitive task analysis surgical subject matter expert interviews

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content IDENTIFYING THE POINT OF DIMINISHING MARGINAL UTILITY
FOR COGNITIVE TASK ANALYSIS
SURGICAL SUBJECT MATTER EXPERT INTERVIEWS

by

Patrick Douglas Crispen

A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION

May 2010

Copyright 2010 Patrick Douglas Crispen

ii

ACKNOWLEDGEMENTS

A successful doctoral dissertation requires the support and feedback of
many people. I gratefully acknowledge the people without whom this project
would not have been possible. First and foremost, I thank my wife Christine for
both her positivity and her unyielding support. From helping me collect and
organize my research to proofreading countless drafts, this dissertation is as much
her accomplishment as it is mine.
I thank Dr. Richard Clark, Dr. Ken Yates, and Dr. Maura Sullivan for
allowing me to participate in their thematic dissertation group and for the
inspiration, support, and guidance they provided throughout the past two years. I
also thank the other ‗Clarkies‘ in my group: Craig Bartholio, Julia Campbell, Eko
Canillas, Joon Kim, Leslie Tirapelle, and Maryann Tolano-Leveque. Having a
team of dedicated faculty and scholars working together towards a common goal
made this dissertation possible (and made the frequent weekend meetings in a
classroom without air conditioning a little more bearable).
Finally, I dedicate this dissertation to Dr. Marshall Schreeder and the staff
of the Clearview Cancer Institute in Hunstville, Alabama. I literally would not be
here today were it not for you.

iii

TABLE OF CONTENTS

ACKNOWLEDGEMENTS ii

LIST OF TABLES v

LIST OF FIGURES vi

ABSTRACT viii

CHAPTER 1: REVIEW OF THE LITERATURE 1
Statement of the Problem 1
Review of the Literature 5
Current Training and Assessment Methods 6
Types of Knowledge 12
The Difference between Novices and Experts 18
Cognitive Task Analysis 21
Summary of ‗How Many Experts‘ Studies 33
Summary 35

CHAPTER 2: METHODOLOGY 37
Design 37
Subjects 39
Data Collection 40
Semi-structured Interviews 40
Interview Coding 41
Creation of Six Subject Matter Experts
‗Gold Standard‘ Protocol 41
Data Analysis 43

CHAPTER 3: RESULTS 46
Co-coding and Inter-Coder Reliability 46
Research Questions 48
Research Question 1 48
Research Question 2 50
Summary 67

iv

CHAPTER 4: DISCUSSION AND CONCLUSIONS 68
Research Questions 68
Research Question 1 68
Research Question 2 70
Summary 73
Limitations and Delimitations 74
Implications 76
Conclusion 78

REFERENCES 79

v

LIST OF TABLES

Table 1: Percentage of Knowledge Acquired from One Expert
When Compared to a Six Subject Matter Expert ‗Gold
Standard‘ Protocol 49

Table 2: Average Percentage of Total Knowledge, Action
Steps, and Decision Steps Acquired from Multiple
Groups of Experts When Compared to a Six Subject
Matter Expert ‗Gold Standard‘ Protocol 52

Table 3: Average Percentage of Objectives, Reasons, and Risks
Acquired from Multiple Groups of Experts When
Compared to a Six Subject Matter Expert ‗Gold
Standard‘ Protocol 57

Table 4: Average Percentage of Indications, Contraindications,
and Standards Acquired from Multiple Groups of
Experts When Compared to a Six Subject Matter
Expert ‗Gold Standard‘ Protocol 60

Table 5: Average Percentage of Mandatory Equipment,
Recommended Equipment, and Tasks Acquired from
Multiple Groups of Experts When Compared to a Six
Subject Matter Expert ‗Gold Standard‘ Protocol 65

Table 6: Quantity of Experts Recommended if a 10% Marginal
Utility in Knowledge Acquisition is Expected 67

vi

LIST OF FIGURES

Figure 1: Percentage of Six-Expert ‗Gold Standard‘ Total Knowledge
Acquired as a Function of the Number of Experts. 52

Figure 2: Percentage of Additional Total Knowledge Acquired
as a Function of the Number of Experts. 53

Figure 3: Percentage of Six-Expert ‗Gold Standard‘ Action Steps
Acquired as a Function of the Number of Experts. 54

Figure 4: Percentage of Additional Action Steps Acquired as a
Function of the Number of Experts. 55

Figure 5: Percentage of Six-Expert ‗Gold Standard‘ Decision
Steps Acquired as a Function of the Number of Experts. 55

Figure 6: Percentage of Additional Decisions Acquired as a
Function of the Number of Experts. 56

Figure 7: Percentage of Six-Expert ‗Gold Standard‘ Risks Acquired
as a Function of the Number of Experts. 58

Figure 8: Percentage of Additional Risks Acquired as a Function
of the Number of Experts. 59

Figure 9: Percentage of Six-Expert ‗Gold Standard‘ Indications
Acquired as a Function of the Number of Experts. 60

Figure 10: Percentage of Additional Indications Acquired as a
Function of the Number of Experts. 61

Figure 11: Percentage of Six-Expert ‗Gold Standard‘ Contraindications
Acquired as a Function of the Number of Experts. 62

Figure 12: Percentage of Additional Contraindications Acquired
as a Function of the Number of Experts. 62

Figure 13: Percentage of Six-Expert ‗Gold Standard‘ Standards
Acquired as a Function of the Number of Experts. 63

vii

Figure 14: Percentage of Additional Standards Acquired as a
Function of the Number of Experts. 64

Figure 15: Percentage of Six-Expert ‗Gold Standard‘ Recommended
Equipment Acquired as a Function of the Number of
Experts. 66

Figure 16: Percentage of Additional Recommended Equipment
Acquired as a Function of the Number of Experts. 66

viii

ABSTRACT

Residents in surgical residency programs are taught through a hands-on
apprenticeship under the supervision of surgical subject matter experts despite the
fact that those experts are largely unaware of the automated strategies that guide
most of their problem-solving. In fact, experts often omit as much as 70% of the
procedural knowledge that novices need to learn. This study examines the amount
of procedural steps in an open cricothyrotomy procedure that can be learned from
multiple surgical subject matter expert interviews using Cognitive Task Analysis
(CTA) techniques. CTA focuses on measuring the mental models used in task
performance, capturing not only declarative knowledge but also procedural
knowledge.
Using CTA-based interview techniques, six expert trauma surgeons
employed by the Department of Surgery of a private, urban medical school in the
western United States were separately interviewed about how to perform an open
cricothyrotomy procedure. The interviews were coded and converted into ordered,
procedural checklists that listed all of the equipment, conditions, action steps, and
decision steps that each subject matter expert mentioned in his or her interview as
being necessary to successfully perform the procedure. Those individual checklists
were then converted into a single criterion standard, hereafter termed as the ‗gold
standard‘, for the open cricothyrotomy procedure against which each expert's
checklist was graded for completeness.
ix

This study found that, on average, each surgical subject matter expert who
participated in this study omitted 44% of the total steps, 34% of the action steps,
and 72% of the decision steps contained within a six-expert open cricothyrotomy
‗gold standard‘ procedure. The results of this study also show that interviewing
four experts was necessary to reach a 10% point of diminishing marginal utility for
the elicitation of additional procedural steps. This appears to contradict the
conclusions of previous studies that showed that interviewing three experts was
optimal for knowledge elicitation.

1

CHAPTER 1
REVIEW OF THE LITERATURE

Statement of the Problem
Residents in surgical residency programs are required to efficiently acquire,
recall, and transfer complex procedural knowledge — what to do, when to do it (or
not do it), and how to do it — within the larger framework of clinical decision-
making. Residents obtain this knowledge through a hands-on apprenticeship under
surgical subject matter experts (Halsted, 1904; Elstein, & Schwarz, 2002).
Unfortunately, expertise in a particular subject matter does not guarantee that one is
able to teach that subject to others (Bransford, Brown, & Cocking, 2000). For
example, a recent study (Clark, Pugh, Yates, Early, & Sullivan, 2008) asked ten
surgical subject matter experts to describe how to implement an emergency medical
procedure. The subject matter experts omitted nearly 70% of the how-to steps.
This 70% subject matter expert procedural step omission rate has been reported in
other studies as well as in other fields (e.g., Chao & Salvedney, 1994; Sullivan,
Ortega, Wasserberg, Kauffman, Nyquist, & Clark, 2008).
Cognitive Task Analysis (CTA) seeks to help researchers understand the
cognitive aspects of human performance and then to convert that understanding
into new teaching tools and techniques (Crandall, Klein, & Hoffman, 2006). In
particular, CTA focuses on measuring the mental models used in task performance
2

(Ryder & Redding, 1993), capturing not only declarative knowledge but also
procedural knowledge. CTA has been shown to provide considerable long-term
savings and improved efficiency in human performance (Williams, 2000), in
particular, increasing student speed, accuracy, and adaptability (Clark, Feldon, van
Merriënboer, Yates, & Early, 2008). In addition, CTA has the potential over time
to decrease the need for training to rely solely on experts (Velmahos, Konstantinos,
Sillin, Chan, Clark, Demetrios, & Maupin, 2004).
Evidence indicates that interviewing two or more subject matter experts as
part of a CTA leads to a two-fold increase in the percentage of procedural
knowledge acquired (Chao & Salvedney, 1994). However, it should be noted that
this evidence comes from studies that focused on subject matter experts in fields
other than surgery (Sullivan, Ortega, Wasserberg, Kauffman, Nyquist, & Clark,
2008). To date, there have only been a handful of studies that focused on applying
CTA to surgical training (e.g., Velmahos, Konstantinos, Sillin, Chan, Clark,
Demetrios, & Maupin, 2004; Bathalon, Martin, & Dorion, 2004; Sullivan, Brown,
Peyre, Salim, Martin, Towfigh, & Grunwald, 2007; Sullivan, Ortega, Wasserberg,
Kauffman, Nyquist, & Clark, 2008; Luker, Sullivan, Peyre, Sherman, & Grunwald,
2008) and there have been no published studies that offer evidence of how many
surgical subject matter experts must be interviewed in order to capture enough
critical information about a particular surgical procedure. More complete
information provided to students during instruction should mitigate the likelihood
3

that students taught by expert surgeons could receive incomplete or inaccurate
information and thereby contribute to surgical errors.
German writer Herman Heinrich Gossen introduced the law of diminishing
marginal utility in his 1854 book Die Entwicklung der Gesetze des Menschlichen
Verkehrs und Der Daraus Fließenden Regeln für Menschliches Handeln (Blaug,
1997; Rima, 2000). Gossen, in what would later become known as ‗Gossen‘s First
Law‘, suggested that the benefit or pleasure obtained from each additional
consumed input would diminish until satiation was reached. Applied to CTA, the
law of diminishing marginal utility suggests that, up to a certain point, each
additional surgical subject matter expert interview would elicit new information
that would be helpful in closing the 70% subject matter expert procedural step
omission gap. However, the law of diminishing marginal utility also suggests that
after reaching that certain point, known as the point of diminishing marginal utility,
the benefit of each additional surgical subject matter expert interview would be less
than the benefit of the preceding interview and potentially would not elicit any
new, missing, or significant information.
By determining how many surgical subject matter expert CTA interviews
would be needed to reach the point of diminishing marginal utility, surgical
residency programs could ensure that the knowledge elicited during CTA
interviews, and the subsequent curriculum developed based on these interviews,
was both efficient and thorough, utilizing the optimal number of subject matter
experts to avoid the omission of essential steps or decisions. The thoroughness
4

achieved by conducting a CTA with the appropriate number of subject matter
experts could also have a tangible benefit: Decreasing the quantity of surgical
errors by improving the transfer of procedural knowledge from subject matter
experts to residents through the use of CTA would increase the residents‘ technical
competence and would decrease surgical complications (Velmahos, Konstantinos,
Sillin, Chan, Clark, Demetrios, & Maupin, 2004).
In addition, one of the larger barriers to CTA is its substantial, up-front cost
(Shute, Torreano, & Willis, 2000; Crandall, Klein, & Hoffman, 2006). This cost
can be measured both in time and opportunity. Informal estimates suggest that it
takes between 30 and 35 person hours of CTA activities to produce the knowledge
content for one hour of training (Clark & Estes, 1996; Grunwald, Clark, Fisher,
McLaughlin, Narayanan, & Piepol, 2004), with knowledge elicitation being the
most time-consuming and difficult stage in conducting a CTA (Hoffman, Shadbolt,
Burton, & Klein, 1995). Every hour a surgical subject matter expert spends being
interviewed as part of a CTA is an hour that particular expert is not assisting
patients, directly teaching or supervising residents, or generating revenue. By
identifying the optimal number of surgical subject matter experts to interview, this
study will inform future medical school CTAs and possibly decrease the amount of
time, resources, and subject matter experts required to conduct these CTAs.
This study poses two research questions:
1. How much information about the open cricothyrotomy procedure does one
expert provide when compared to the combined contributions of a ‗gold
5

standard‘ summary based on interviews with six surgical subject matter
experts?
2. How much critical information is gained from each additional CTA interview
about the open cricothyrotomy procedure?
Because there was no widely accepted standard with which to compare surgical
subject matter expert interview descriptions of how to perform an open
cricothyrotomy procedure — Holmes, Panacek, Sakles, and Brofeldt (1998)
described two competing cricothyrotomy techniques and noted that it was unclear
from the literature which size tracheostomy tube was most appropriate for the
procedure — it was necessary for this study to develop its own criterion standard or
‗gold standard‘ as to how to perform the procedure. This study‘s ‗gold standard‘
was the definitive list of equipment, conditions, action steps, and decision steps that
the six subject matter experts who participated in this study considered necessary to
successfully perform an open cricothyrotomy procedure. It was against this ‗gold
standard‘ that each subject matter expert‘s description of how to perform the
procedure was compared.

Review of the Literature
To best understand the potential benefit of CTA-based medical and surgical
training, we must first examine the current training and assessment methods used
by many American medical schools. This leads directly into a discussion of the
knowledge types employed by subject matter experts. Since subject matter experts
6

asked to describe or teach a procedure had been shown to omit a large portion of
the procedure‘s steps, finding a way to efficiently elicit and capture subject matter
experts‘ implicit knowledge was essential. CTA may be beneficial in this task;
hence, this study will also discuss what CTA is, how it adds to current training
methods, and what evidence there is about CTA‘s effectiveness. We will conclude
with a summary of recent studies regarding how many experts are considered
necessary in order to conduct a thorough CTA.

Current Training and Assessment Methods
Lucas (2006) noted that by the last third of the nineteenth century, many
American institutions of higher learning were similar to English-style universities,
focusing more on learning based on the seven medieval liberal arts subjects of the
trivium (grammar, logic, and rhetoric) and the quadrivium (arithmetic, astronomy,
geometry, and harmonics) than on providing professional training for specific
technical or economic roles. By way of comparison, German universities of the
time were major research centers offering specialized graduate seminars and
lectures. Johns Hopkins University, founded in 1876, was the first American
university to adopt a completely German-style university structure, combining
subjects such as mathematics, the sciences, and history into a single, undergraduate
department and then creating specialized graduate departments organized for
advanced research and professional education. Over the following 130 years, the
7

German-style university structure first adopted by Johns Hopkins University
became the de facto structure employed by most American research universities.
Johns Hopkins‘ influence also extended to the pedagogical practices
employed to teach America‘s medical students. In 1889, Sir William Halsted
initiated a German-style, master-apprentice training model to the residency
program at the Johns Hopkins Hospital (Reznick & MacRae, 2006; Sachdeva, Bell,
Britt, Tarpley, Blair, & Tarpley, 2007; Scott, Cendan, Pugh, Mintor, Dunnington,
& Kozar, 2008). This training model remained the national standard for medical
education for the next century (Reznick & MacRae, 2006). Much like the master-
apprentice training models of the medieval period and earlier (Lucas, 2006),
residents at Johns Hopkins studied and continued to study with subject matter
experts to learn and, over an extended period of time, to master the arts of
diagnostic reasoning (Halsted, 1904; Elstein, & Schwarz, 2002) and patient care.
Residents in the master-apprentice relationship learned the principles and practice
of medicine by being immersed in the clinical environment (Halsted, 1904; Scott,
Cendan, Pugh, Mintor, Dunnington, & Kozar, 2008) under close supervision of the
faculty (Halsted, 1904; Sachdeva, Bell, Britt, Tarpley, Blair, & Tarpley, 2007). As
the residents‘ skills increased, faculty rewarded the residents with graduated levels
of responsibility and autonomy in the diagnosis and care of patients (Halsted, 1904;
Sachdeva, Bell, Britt, Tarpley, Blair, & Tarpley, 2007; Scott, Cendan, Pugh,
Mintor, Dunnington, & Kozar, 2008).
8

One example of the master-apprentice model in medical school training is
the training of surgical residents. While the workload requirements for surgical
residents has recently changed (Sachdeva, Bell, Britt, Tarpley, Blair, & Tarpley,
2007), students are still immersed in the clinical environment under close
supervision of faculty. A modern surgical residency program requires 60 months
of training with at least 54 months of that training devoted to hands-on patient
diagnosis and care (Sachdeva, Bell, Britt, Tarpley, Blair, & Tarpley, 2007).
Reznick and MacRae (2006) noted that the most significant attribute of current
surgical training is not specially designed curricula, but rather the experiencing of a
large number of clinical cases in an abbreviated amount of time. Sachdeva (2007)
reported that surgery residents are required to complete approximately 150 major
operative cases during each of their five years of residency as well as during their
final year as chief residents. As the surgical residents‘ skills increase, those
residents enjoy graduated levels of responsibility and autonomy in the diagnosis
and care of patients.
While the Halstedian master-apprentice training method is still widely used
by American medical schools, questions have arisen about the model‘s
effectiveness and efficiency. The barriers to the current training and assessment
methods are discussed in the next section.

9

Barriers to current training and assessment methods
Three common, interrelated barriers to the Halstedian master-apprentice
method have recently emerged. First, even the most advanced subject matter
experts are largely unaware of the automated strategies that guide most of their
problem-solving (Paris, Lipson, & Wixson, 1983; Cooke, 1994; Blessing &
Anderson, 1996; Clark & Estes, 1996; Bargh & Chartrand, 1999). This leads
directly to the second barrier; that is, subject matter experts often omit 70% of the
procedural knowledge that novices need to know in order to successfully complete
a procedure (Chao & Salvedney, 1994; Sullivan, Ortega, Wasserberg, Kauffman,
Nyquist, & Clark, 2008; Clark, Pugh, Yates, Early, & Sullivan, 2008). The result is
the third and final barrier to the Halstedean model; that is, medical school training
is often unorganized and disorderly (Grantcharov & Reznick, 2008).
In the Halstedian master-apprentice model, residents learn from the masters
consisting of faculty who are considered to be subject matter experts. After years
and practice, subject matter experts generally develop automated strategies that
guide most of their problem solving (Clark & Estes, 1996). Because these
strategies are automated, they are also tacit and not subject to introspection,
verbalization, or conscious process (Berry, 1987; Cooke, 1994; Clark & Estes,
1996; Velmahos, Konstantinos, Sillin, Chan, Clark, Theodorou, and Maupin, 2004;
Sullivan, Brown, Peyre, Salim, Martin, Towfigh, & Grunwald, 2007). Worse still,
subject matter experts‘ knowledge may also be compiled or proceduralized (Cooke,
10

1994; Anderson, 1996). The compilations or procedures that experts rely on may
also no longer be verbalizable (Clark & Estes, 1996; Feldon, 2006).
The first barrier contributes to the second barrier to the Halstedian master-
apprentice model. Recent research shows that subject matter area experts often
omit up to 70% of the steps necessary to complete a procedure (Chao & Salvedney,
1994; Sullivan, Ortega, Wasserberg, Kauffman, Nyquist, & Clark, 2008; Clark,
Pugh, Yates, Early, & Sullivan, 2008). Since experts may not have explicit access
to all of their implicit knowledge (Cooke, 1994), they may not be able to articulate
why certain steps in a problem-solving procedure are necessary (Clark & Estes,
1996; Feldon, 2006). In fact, experts often generate distorted, incomplete, or
inaccurate descriptions of how a task is performed (Bainbridge, 1979; Feldon,
2006). To put it another way, residents learning a new surgical procedure from an
expert should be presented with and have the opportunity to practice all of the
procedure‘s steps. Unfortunately, experts have developed shortcuts and may no
longer need to know all of the procedure‘s minute steps (Cooke, 1994; Anderson,
1996). A subject matter expert teaching a resident may teach all of the steps that
the expert remembers but not all of the steps that the expert knows (Blessing &
Anderson, 1996).
The third barrier to the Halstedian master-apprentice model is the recent
concerns that have arisen about the quality of education that residents receive under
the model. Medical subject matter experts are facing increasing demands on their
time (Grantcharov & Reznick, 2008; Luker, Sullivan, Peyre, Sherman, &
11

Grunwald, 2008; Sullivan, Ortega, Wasserberg, Kauffman, Nyquist, & Clark,
2008), potentially decreasing the time these experts are available to teach others.
Even when experts are available to teach residents, most advanced subject matter
experts are largely unaware of the automated strategies that guide most of their
problem solving. A consequence of this is that American medical school training
often focuses more on lower order behavioral skills such as how to make an
incision rather than on higher order cognitive thinking such as deciding when one
should or should not implement a certain procedure (Hall 2004). The result is that
general surgery residency programs‘ training and educational experiences vary
significantly (Sachdeva, Bell, Britt, Tarpley, Blair, & Tarpley, 2007, 1202).
The most damning barrier was described in a review by Grantcharov and
Reznick (2008) who concluded that the Halstedian master-apprentice approach to
medical training is no longer appropriate in teaching complex medical procedures.
They offered two reasons to support this assertion. First, residents‘ work weeks
had recently been shortened to 80 duty hours (Sachdeva, Bell, Britt, Tarpley, Blair,
& Tarpley, 2007), diminishing teaching time and resulting in a situation where
opportunities for residents to engage in deliberate practice were rare and decreasing
(Reznick & MacRae, 2006). It was difficult to ‗see one, do one, and teach one‘
when there was less time in the work week in which the residents could acquire,
implement, and practice their skills. Second, as hospitals emphasize efficiency in
the face of increasing costs, surgical subject matter experts were pressured to be
more efficient in their use of operating rooms, further diminishing the amount of
12

time surgical residents could spend in deliberate, situated learning (Reznick &
MacRae, 2006; Sachdeva, Bell, Britt, Tarpley, Blair, & Tarpley, 2007). A third
reason, omitted by Grantcharov and Reznick but put forward by Cooke (1994) a
decade earlier, was that the Halstedian model focused on procedural knowledge but
completely ignored the cognitive skills underlying those procedures. This is an
important point, one that reappears throughout the literature on the different types
of knowledge.
What is needed is a more efficient way to capture not only the 30% of the
surgical subject matter experts‘ explicit knowledge that is still accessible to
conscious examination but also as much of the remaining 70% of the experts‘
knowledge that is implicit and no longer consciously accessible to the experts.
However, what types of knowledge are necessary for expert performance?

Types of Knowledge
To be able to understand and eventually to be able to record the cognitive
processes employed by surgical subject matter experts, we must first examine the
different types of knowledge these experts have and understand how these
knowledge types are interrelated. Anderson (1996) wrote that ―intelligence is the
simple accrual and tuning of many small units of knowledge that, in total, reduce
complex cognition‖ (356). He added that to become competent, one had to engage
in a labor-intensive process in which one had to ―acquire one-by-one all the
knowledge components.‖ (359).
13

Anderson (1982), in describing the theory behind his Adaptive Control of
Thought (ACT) production system, proposed that knowledge is first learned in a
conscious, declarative form and is then transformed over time with use into a
largely unconscious, automated procedural form. Paris, Lipson, and Wixson
(1983), in their examination of the aspects of knowledge and motivation critical to
becoming a strategic reader, introduced a third and, as we will soon discover,
controversial knowledge type — conditional knowledge. Each of these three
knowledge types is discussed in the following sections.

Declarative knowledge
Declarative knowledge is information about why, what, or that (Paris,
Lipson, & Wixson, 1983; Clark & Estes, 1996); for example, ‗the reason why I
cannot fly Eastern Airlines is that the company no longer exists,‘ ‗I know what an
automobile looks like,‘ or ‗I know that an automobile runs on gasoline.‘ To
examine this further, declarative knowledge can be divided into concepts,
processes, and principles (Clark & Estes, 1996; Clark, Feldon, van Merriënboer,
Yates, & Early, 2008; Clark, 2008; Clark, Yates, Early, & Moulton, 2009).
Concepts are units of knowledge characterized by definitions and at least one
example, processes are descriptions of how something works in stages or phases,
and principles help you understand which cause produces what effect (Clark,
Feldon, Van Merriënboer, Yates, & Early, 2008).
14

During the acquisition of declarative knowledge, the learner receives new
instruction or new information and then encodes the instruction as a set of
concepts, process and/or principles that the learner later interprets (Anderson,
1982). Declarative knowledge is seemingly intended to help us handle novel
events (Clark & Estes, 1996) and represents abstract facts (Anderson, 1982) that
are not bound by context. The fact that a pencil writes on paper can be recalled and
applied in almost any context or environment. This lack of context is one of the
reasons why declarative knowledge is also considered to be extremely flexible
(Anderson, 1982; Anderson, & Fincham, 1994). Unfortunately, the act of
retrieving declarative knowledge from one‘s long term memory is slow and taxing
(Anderson, 1982).
The most important point to remember is that acquiring declarative, factual
knowledge is not an end unto itself but rather is the first step in the continuing
process of acquiring cognitive skill. Nearly all knowledge comes into the system in
declarative form that is committed to long-term memory and is then converted into
procedures made up of production rules (Anderson, & Fincham, 1994). For
example, concepts become classification procedures that permit the identification
of new examples of a concept. Processes and principles become procedures that
allow for the modification or change of outcomes that are controlled by the process
or procedure being applied (Clark Feldon, Van Merriënboer, Yates, & Early, 2008).

15

Procedural knowledge
Procedural knowledge is information about how to do something and
includes information about the execution of actions and decisions (Paris, Lipson, &
Wixson, 1983; Clark & Estes, 1996). Anderson (1982, 1996) suggested that
procedural knowledge is divided into production rules — simple encodings of
observed transformations in the environment. Productions are rule-bound
procedures that achieve action goals. These production rules specify when a
cognitive act should take place and are informed by action and decision steps that
consist of goals and subgoals (Anderson, 1996). In other words, concepts ―... are
instruments of the productions, which are the agents‖ for actions and decisions that
allow people to identify novel instances of familiar concepts (Anderson, 1982,
374).
Whereas declarative knowledge is flexible, procedural knowledge is bound
to the context where it was learned and applied, is use-specific, and cannot
generalize to other uses or contexts (Anderson & Fincham, 1994). Using an
example from the previous section, knowing that a pencil can write on paper can be
applied in many contexts and environments. Knowing how to use a pencil to write
on paper is much more use-specific and cannot be generalized to other writing
experiences such as writing with a typewriter or writing a text message on a cellular
telephone. The use- or domain-specificity of procedural knowledge is an important
point to make because during the 1970s there was a widely-held belief that expert
clinical skills were simply sets of domain-independent reasoning strategies or
16

heuristics that could be applied to all cases, even ones that were novel (Norman,
2002; Schmidt, Norman, & Boshuizen, 2002). However, little evidence supports
this belief (Norman, 2002).
While the act of retrieving declarative knowledge from one‘s long term
memory is slow and taxing, retrieving procedural knowledge is much quicker and
less mentally taxing, particularly as one gains experience. Eventually, over
extended periods of time and practice, goal-directed processes (e.g., ‗step 1: add the
ones column‘, ‗step 2: add the tens column‘, etc.) require less and less conscious
guidance, eventually becoming ―intentional but effortless mental processes‖ (Bargh
& Chartrand, 1999, 463). In other words, one‘s procedural knowledge, over
extended periods of time and practice, gradually becomes highly accurate,
automated, rapid, and efficient within the context or domain in which it was
developed (Clark & Estes, 1996; Bargh & Chartrand, 1999). These newly non-
conscious mental processes ‗clear the deck,‘ freeing attention from tasks in which
attention is no longer needed (Bargh & Chartrand, 1999). In addition, as people
repeatedly solve the same type of problem, they not only increase the speed at
which they solve the problem but they often come up with new problem-solving
processes that combine or skip steps (Blessing & Anderson, 1996). In effect, with
practice, one learns how to either break a single production rule into multiple
productions or to create new, personalized production rules (Blessing & Anderson,
1996)
17

While declarative memory contains factual information and procedural
knowledge contains procedural information, Paris, Lipson, and Wisxon (1983)
argued that neither knowledge type was sufficient to explain why or when to
implement a task. The authors suggested that those questions were the domain of
our third and final type of knowledge.

Conditional knowledge
Paris, Lipson, and Wixson (1983) suggested that conditional knowledge is
information about why or when to choose or not to choose to implement a particular
procedure. The authors added that while declarative knowledge lets one know
what to do and procedural knowledge lets one know how to do it, an expert with
only declarative and procedural knowledge would be unable to adjust his or her
strategies in the face of changing circumstances. According to the authors,
conditional knowledge manifests as strategic behavior or action. The learner (or
master) must exercise both intentionality and purpose to reach a goal that is
attainable using actions that the learner can use to accomplish this goal.
Willfulness is key here. The learner, with the goal in mind, chooses a specific
action from a myriad of alternative actions. The choice of action is informed by
and reveals the learner‘s intentions, perception of the value of the action, and
expectation of the action‘s success.
Unfortunately, while experts do ―possess a series of rapid and highly
efficient, domain-specific rules about the conditions that require the use of problem
18

solving strategies and the expected consequence of each stage of the use of
interventions‖ (Clark & Estes, 1996, 2), there is little evidence to support the
assertion that conditional knowledge is a third, separate type of knowledge rather
than an extension of declarative or procedural knowledge (Clark, 2008).

The Difference between Novices and Experts
The different knowledge types provide a framework that helps us
understand not only how experts differ from novices but also the reasons behind
those differences. Learning and becoming an expert in skilled performance
requires more than just following a checklist of declarative or procedural
knowledge steps. Rather, one must ―recognize which steps are important, how to
notice that the situation is different from what [is] expected, and how to adapt the
steps‖ (Crandall, Klein, & Hoffman, 2006, 195). One key is to discover the cues
that experts use to interpret a situation (Flach, 2000).
When presented with a situation or problem, Besnard and Bastien-Toniazzo
(1999) suggested that both experts and novices start by examining the situation‘s or
problem‘s surface features. However, novices use these surface features to activate
inferential processing. Experts use these surface features to recognize patterns and
to activate existing rules and responses. In other words, novices make inferences
while experts apply or create a schema. Schaafstal Schraagen, and van Berlo (2000)
reached the same conclusion, noting that trainees employed what appeared to be an
19

unsystematic approach while experts employed a symptomatic search generated
from recognizable features.
It is important to reiterate that experts‘ superior performance is domain-
specific. Outside of their particular domains, an expert‘s memory is no better than
the memory of an ordinary individual (Ericsson, Krampe, & Tesch-Römer, 1993).
Within their domains, however, experts organize information better (Chase &
Simon, 1973) and employ effective problem-solving strategies such as hypothesis
testing and pattern recognition (Elstein, & Schwarz, 2002). One example of expert
pattern description appeared in Ericcson (2004): ―When medical conditions are
frequently encountered in clinical practice, then experienced physicians will
acquire patterns that allow them to recognize each condition and access mental
models or prototypes for the corresponding disease‖ (s77).
The answer to the question ―how do I gain expertise‖ is the same as the
answer to the oft-quoted riddle ―how do I get to Carnegie Hall,‖ which is
―practice.‖ Experts gradually gain expertise through long and deliberate practice
within a particular domain (Ericsson, Krampe, & Tesch-Römer, 1993; Clark &
Estes, 1996; Ericsson, 2004; Reznick & MacRae, 2006) that requires a period of
approximately 10 years to mature (Ericsson, 2004). This is one reason why
expertise most often appears not in adolescence but rather in middle adulthood
(Kim & Hasher, 2005). Deliberate practice is highly structured and is comprised of
activities that have been found to be most effective in improving performance
(Ericsson, Krampe, & Tesch-Römer, 1993). The apparent effortless nature by
20

which an expert is able to execute domain-specific skills is simply the result of the
automation of performance skills after long hours of practice (Ericcson, 2004). The
number of hours spent in deliberate practice determines one‘s level of expertise
(Ericcson, 2004; Reznick & MacRae, 2006). The key point to remember is that ―as
individuals adapt to a domain and their performance skills become automated, they
are able to execute these skills smoothly and without apparent effort‖ (Ericcson,
2004, s70). To train residents so that they execute complex procedures quickly and
effortlessly is one of the goals of medical education. It is also one of medical
education‘s biggest barriers.
Experts are often asked to instruct novices (Feldon, 2006). Since experts
may no longer have explicit access to all of their automated, procedural expertise
(Cooke, 1994), experts may no longer be able to articulate the decision steps in a
problem-solving procedure necessary for success (Clark & Estes, 1996). Surgical
subject matter experts‘ automaticity — processes that do not require conscious
choice, intention, or intervention to become active and run to completion (Bargh &
Ferguson, 2000) — may prevent the experts from being able to provide their
students with all of the procedural knowledge needed to successfully complete a
particular task or procedure. Indeed, subject matter experts often omit up to 70% of
the steps necessary to complete a procedure (Chao & Salvedney, 1994; Sullivan,
Ortega, Wasserberg, Kauffman, Nyquist, & Clark, 2008; Clark, Pugh, Yates, Early,
& Sullivan, 2008). What is needed is a way to capture both the subject matter
expert‘s explicit knowledge and as much as his or her implicit knowledge as
21

possible; that is, knowledge generated over years and even decades of deliberate
practice.

Cognitive Task Analysis
In 1911, Fredrick Winslow Taylor published Principles of Scientific
Management, a book in which Taylor suggested that workplace managers replace
their employees‘ ineffective work methods with new methods based on the
scientific analysis of the employees‘ daily tasks. The assumption, shared by
Thorndike (1921) a few years later, was that workers performed beneath their
maximal level even for tasks that are routine. To identify which tasks were
necessary, which ones were superfluous, and how much time each task should take,
Taylor and Thorndike recommended that managers record and describe the overt,
observable behavior of their workers as the workers perform their job tasks. This
recording and description process was referred to as task analysis (Clark & Estes,
1996) or Behavioral Task Analysis (BTA). BTA was founded on the assumption
that behavior could be explained by a linear series of single Stimulus-Response (S-
R) units (Bargh & Ferguson, 2000). BTA evaluated complex tasks and then
divided those tasks into small steps (stimuli) that could be used to elicit a desired
productive outcome (responses). The result was an algorithm or formula that could
be used to train people how to perform, or to assess how well someone actually
performed most industrial-era job tasks (Arnett, 2000).
22

Unfortunately, BTA focuses on the physical movements needed to complete
a task and excludes the internal mechanisms, processes, and structures that govern
complex job task performance, advanced problem solving, and troubleshooting
(Cooke 1992a; Clark & Estes, 1996; Arnett, 2000). Behavior and other responses
are caused and some of those causes are unknowable (Bargh & Ferguson, 2000),
making the exclusive use of BTA obsolete and inappropriate in many modern
workplaces. Additionally, modern job requirements increasingly require cognitive
performance (Ryder & Redding, 1993) that relies not on a work-task algorithm or
formula but rather on the ability to recall and make use of a person‘s automated
procedural knowledge (Arnett, 2000).

Definition of CTA
Cognitive Task Analysis (CTA) emerged in the early 1980s (Crandall,
Klein, & Hoffman, 2006) as an extension of BTA (Chipman, Schraagen, & Shalin,
2000; Clark, Yates, Early, & Moulton, 2009). CTA‘s goal was ―to yield
information about the knowledge, thought processes, and goal structures that
underlie observable task performance‖ (Chipman, Schraagen, & Shalin, 2000, 3).
While CTA was primarily concerned with decision-making tasks (Arnett, 2000),
observable behaviors were no longer the focal point (Clark & Estes, 1996).
Instead, CTA focused on measuring the mental models used in task performance
(Ryder & Redding, 1993). In other words, CTA captured not only the declarative
knowledge easily measured by BTA but also the procedural knowledge heretofore
23

hidden from both experts and students because of the automaticity of expert
knowledge.
In addition, CTA helped researchers understand the cognitive aspects of
human performance and then convert that understanding into new teaching tools
and techniques (Crandall, Klein, & Hoffman, 2006). Most CTAs involved
interviews with subject matter experts whose knowledge distinguished them from
novices. ―CTA data highlight critical cues, patterns, and relationships, and can
show what makes decisions and judgments so difficult‖ (Crandall, Klein, &
Hoffman, 2006, 235).
Most of the dominant CTA methods share the same five sequential steps
(Clark, Feldon, Van Merriënboer, Yates, & Early, 2008). The first step is to collect
preliminary knowledge. This is done by identifying the tasks that are to be
analyzed and by acquiring an overview of the specific domain in which the tasks
are performed. The second step is to identify knowledge representations. These
knowledge representations include any of the knowledge types required to
implement the task. The third CTA step is to apply focused knowledge elicitation
methods. This usually involves interviewing multiple subject matter experts,
although the number of experts necessary to achieve an adequate description of a
complex task is a topic of debate. The key to a successful CTA is capturing both
procedural and declarative information (Arnett, 2000) relevant to a specific domain
from a subject matter expert. While there is a large amount of literature from the
expert systems community on the requirements that both subject matter experts and
24

CTA interviewers must meet (Chipman, Schraagen, & Shalin, 2000), there are only
four broad families of knowledge elicitation techniques — observation and
interviews, process tracing, conceptual techniques, and formal models (Wei &
Salvedney, 2004; Clark, Feldon, van Merriënboer, Yates, & Early, 2008). The
fourth CTA step is to analyze and verify the data that was acquired during the
subject matter expert interviews. Verification of the knowledge elicited during
interviews is critical. One way to be certain that the results of the knowledge
elicitation are accurate and thorough is to have multiple subject matter experts
evaluate the knowledge elicitation transcripts. The final CTA step is to format
results for the intended application. The end result is a deliverable product that can
be used to design training and assessment tools.

What CTA adds to current training methods
The use of CTA has the potential to improve current training methods in
four tangible ways. First, CTA has been shown to improve training quality (Arnett,
2000). Second, CTA has been shown to increase student speed, accuracy, and
adaptability (Clark, Feldon, van Merriënboer, Yates, & Early, 2008). Third, CTA
may decrease the need for the delivery of training to rely solely on experts
(Velmahos, Konstantinos, Sillin, Chan, Clark, Theodorou, & Maupin, 2004).
Finally, CTA may generate long-term cost savings (Means & Gott, 1998).
25

CTA seeks to improve the quality of training by capturing not only the
observable behaviors of subject matter experts but also the underlying cognitive
structures that inform those processes. As Arnett (2000) wrote:
Simply observing overt behavior can sometimes give only a
superficial understanding of what an operator is doing and why.
Furthermore, even questioning the expert does not always reveal
the underlying cognitive structure, which, if known, would be a
better predictor of performance in as yet unobserved situations,
such as a novel emergency (32).

CTA seeks to provide a much more thorough perspective of not only how to
implement a task or procedure but also why, when, and what cognitive processes
are needed for success.
CTA has been found to increase the speed, accuracy, and adaptability of
students (Clark, Feldon, van Merriënboer, Yates, & Early, 2008) by giving students
access to all of the knowledge types they need in order to successfully and
efficiently accomplish a task or implement a procedure. Without CTA, students are
often on their own to fill in the 70% of the knowledge their expert teachers
implicitly know and therefore cannot express or explain. In addition, CTA
prepares students to work in a workplace where ―jobs often require complex
thinking and the solution of problems that cannot be stated explicitly in advance‖
(Lesgold, 2000, 451) and where training is often being built for tasks that have
never before been performed (Lesgold, 2000).
CTA has also been found to decrease the need for training to rely solely on
experts over time. Subject matter experts are often called upon to teach novices
26

(Feldon, 2006) despite the fact that those subject matter experts may have no
expertise in or even experience with efficiently and effectively teaching novices
(Bransford, Brown, & Cocking, 2000). Subject matter experts may also be in short
supply (Cooke, 1994) because increasing external demands on their time
(Grantcharov & Reznick, 2008; Luker, Sullivan, Peyre, Sherman, & Grunwald,
2008; Sullivan, Ortega, Wasserberg, Kauffman, Nyquist, & Clark, 2008) prevent
them from having the time to teach others. CTA has the potential to mitigate this
situation. As more CTAs are performed, and as more expert knowledge is
obtained, training can be offered by intermediate experts with equal effectiveness
(Velmahos, Konstantinos, Sillin, Chan, Clark, Theodorou, & Maupin, 2004),
freeing experts to focus on other tasks.
Finally, CTA has the potential to generate cost savings over the long-term
period of time. While there may be a large cost associated with conducting a CTA
(a topic that will be explored in a later section) each time training materials
generated by a CTA are used it may decrease the total cost of training. In addition,
a cost savings that is often overlooked is the potential efficiency obtained through
CTA-based training. CTA-based training often takes less time than traditional
training and has been shown to yield better student performance, potentially
decreasing the need for additional training in the future to fill the students‘
knowledge gaps. In particular, Means and Gott (1998) suggested that trainees
could obtain as much as five years of advanced job knowledge after approximately
50 hours of CTA-derived training.
27

Evidence of CTA effectiveness in medicine
The use of CTA in medicine is still relatively new, so there are only a
handful of published studies that focus on the effectiveness of CTA in medicine
and medical training. Crandall and Getchell-Reiter (1993) conducted two studies
on the use of CTA in medicine. Their first study investigated the assessment
parameters reported by experienced Neonatal Intensive Care Unit (NICU) nurses in
accounts of challenging cases. Thirty-three experienced NICU nurses were
interviewed. The authors reported that significantly more information was elicited
in CTA-based interviews than in traditional interviews. The second study
investigated the sepsis indicators employed by experienced NICU nurses. Five
experienced NICU nurses were interviewed. The authors reported that cues,
indicators, and exemplars were extracted from the NICU nurses to form a guide to
early sepsis assessment that contained information not available in the current
literature.
Beard, Smith, and Denelsbeck (1996) conducted an eight-year CTA-based
iterative study to develop an interface for the electronic display of Computed
Tomography (CT) medical images with a focus on improving the speed and
accuracy that radiologists could view and interpret CT images. Four board-
certified radiologists with extensive experience with CT interpretation were
randomly selected to participate in the study. The knowledge elicited from these
subject matter experts was used to design new CT workstations that decreased CT
28

image manipulation and interpretation times by nearly 9% with no measured
decrease in accuracy.
Four additional studies used CTA methods to collect data about medical
practices. Johnson, Healey, Evans, Murphy, Crenshaw, and Gould (2005) used
CTA interview methods to generate step-by-step task descriptions and decision
protocols for five interventional radiology procedures — arterial puncture, venous
access, nephrostomy, biopsy, and percutaneous transhepatic cholangiogram.
Shackak, Hadas-Dayagi, Ziv, and Reis (2009) conducted CTA-type interviews with
25 primary care physicians in a northern Israeli Health Maintenance Organization
(HMO) about their use of an Electronic Medical Record (EMR) system. The
authors created a task diagram and knowledge audit table based on the data
collected during their interviews. Fackler, Watts, Grome, Miller, Crandall, and
Pronovost (2009) conducted CTA-type interviews with 20 Intensive Care Unit
(ICU) physicians and nurses at two teaching hospitals to identify the cognitive
aspects of the interviewees‘ practice. Five categories of cognitive activities were
identified as follows: pattern recognition; uncertainty management; strategic vs.
tactical thinking; team coordination and maintenance of common ground; and
creation and transfer of meaning through stories. Finally, Sullivan, Ortega,
Wasserberg, Kaufman, Nyquist, and Clark (2008) investigated the use of CTA to
capture the steps and decision points not articulated during the traditional teaching
of a colonoscopy procedure. Three expert surgeons participated in a CTA-based
interview. Surgeon A omitted 50% of the how-to steps and 57% of the decision
29

points. Surgeon B omitted 70% of the how-to steps and 75% of the decision points.
Surgeon C omitted 74% of the how-to steps and 62% of the decision points.
There have been only a handful of CTA studies regarding surgical training
(Sullivan, Brown, Peyre, Salim, Martin, Towfigh, & Grunwald, 2007). Velmahos,
Konstantinos, Sillin, Chan, Clark, Theodorou, and Maupin (2004) investigated the
effectiveness of a three-hour surgical skills laboratory course on Central Venous
Catheterization (CVC) taught using CTA. Twenty-six surgical residents were
randomly assigned to either a group that used CTA-based training methods (n=12)
or a control group that used traditional training methods (n=14). Surgical residents
in the CTA group scored significantly higher in the repeat test, achieved a higher
score on a 14-item procedural checklist, required fewer attempts to find the vein,
and showed a trend towards needing less time to complete the procedure.
Bathalon, Martin, and Dorion (2004) investigated the value of CTA and
kinesiology on the maintenance of surgical skills over a 12-month period. The
authors randomly divided 44 first-year medical students into three groups. The first
group (n=16) was taught to perform a cricothyroidotomy according to the
American College of Surgeons‘ Advanced Trauma Life Support (ATLS) protocol.
The second group (n=13) was taught the procedure using both CTA and
kinesiology principles. The third group (n=15) was taught the procedure using a
combination of CTA, kinesiology principles, daily mental imagery, and rapid
debriefing. The 12 month results showed that training with CTA and kinesiology
principles improved the maintenance of surgical skills over the traditional ATLS
30

technique. The group that received training using a combination of CTA,
kinesiology principles, daily mental imagery, and rapid debriefing showed the
highest acquisition and maintenance of their surgical skill.
Sullivan, Brown, Peyre, Salim, Martin, Towfigh, and Grunwald (2007)
investigated the effectiveness of using CTA to develop curriculum to teach the
behavioral skills and the cognitive strategies of a Percutaneous Tracheostomy (PT)
placement. Twenty surgical residents were randomly assigned to either a group
that used CTA-based training methods (n=9) or a control group that used traditional
training methods (n=11). The CTA group performed significantly higher on the PT
procedure at one-month and six-month periods and demonstrated superior cognitive
strategies than the control group.
Luker, Sullivan, Peyre, Sherman, and Grunwald (2008) investigated the use
of a CTA-based multimedia program to teach surgical decision making in flexor
tendon repair. Three subject matter expert surgeons were interviewed using CTA-
based methods. Multimedia curriculum was developed using the results of these
interviews, emphasizing the ―decision points and important details that are critical
for mastery and completion of the overarching surgical task‖ (12). Ten plastic
surgery residents were evaluated implementing flexor tendon repair on three
occasions, with the CTA-based multimedia curriculum used between the second
and third evaluation session. The residents‘ total knowledge and understanding of
the advantages and disadvantages of the decision points improved by a statistically
significant margin as a result of the CTA-based multimedia curriculum.
31

Lastly, Clark, Pugh, Yates, Early, and Sullivan (2008) investigated the use
of simulators for after-action reviews of medical events in Iraq. Ten surgeons were
asked to describe how to implement an emergency shunt procedure. Of the ten
surgeons, only one was interviewed using CTA methods. The surgeon who was
interviewed using CTA methods described the procedure with greater accuracy and
completeness compared to the nine remaining surgeons who, without CTA, omitted
nearly 70% of the steps.

Evidence of CTA effectiveness outside medicine
The three most common uses for CTA outside of medicine are user
interface design, work procedures design, and cross-team, collaborative business
process design (Lesgold, 2000). Williams (2000) reported that CTA can provide
considerable long-term savings and can also improve efficiency in human
performance. For that reason, CTA has also been used for assessing cost-
effectiveness and ensuring safety (Ormerod, 2000). Potter, Roth, Woods, and Elm
(2000) added that CTA had also been used in anomaly response, military aero
medical evacuation planning, railroad dispatching, situation assessment, Space
Shuttle mission control, and supervisory control. Crandall, Klein, and Hoffman
(2006) reported that CTA had been applied to aviation (including identifying any
cognitive challenges that may have contributed to an aviation accident), the military
(including assessing the impact information technology can have command and
control decision-making), national security, firefighting, emergency response,
32

manufacturing, nuclear power, consumer research, and other fields (Crandall,
Klein, & Hoffman, 2006).
Schaafstal, Schraagen, and van Berlo (2000) investigated the use of CTA to
aid in training technicians to troubleshoot. The technicians who received CTA-
based training solved twice as many malfunctions in less time than technicians
trained using traditional methods. The authors noted that it took less time to train
troubleshooting techniques using CTA-based methods than it took to train similar
techniques using traditional training methods.

Barriers to CTA
There are two significant barriers to the further adoption of CTA: (1) the
upfront expense associated with conducting a CTA; and (2) the lack of data on how
many subject matter experts need to be interviewed to capture the maximum
amount of knowledge with minimal expense.
While there are few claims and little research concerning the economic
benefits of CTA (Clark & Estes, 1996), numerous authors have raised concerns
about the expense of conducting a CTA (e.g., Shute, Torreano, & Willis, 2000;
Crandall, Klein, & Hoffman, 2006). This expense can be measured not only in
financial terms but also in terms of person-hours of the numerous personnel
required to design, conduct, and participate in the CTA. Informal estimates suggest
that it takes between 30 and 35 hours of CTA activities to produce the knowledge
content for one hour of training (Clark & Estes, 1996; Grunwald, Clark, Fisher,
33

McLaughlin, Narayanan, & Piepol, 2004). One reason for this is that it takes more
time to elicit knowledge from experts than it does to create the training or expert
system. Hoffman, Crandall, and Shadbolt (1998) referred to this inequality as the
knowledge acquisition bottleneck. Knowledge elicitation can be the most time-
consuming and difficult stage in constructing a CTA (Hoffman, Shadbolt, Burton,
& Klein, 1995) because experts are often scarce, expensive, and inconsistent
(Cooke, 1994). In addition, because CTAs require qualified researchers, and
because the quantity of these researchers is still relatively low, the number of CTAs
that have been performed is relatively low in relationship to the number of other
task analyses conducted in industry (Seamster, Redding, & Kaempf, 2000).
The second, and in terms of this study, more pressing barrier to CTA is the
fact that little research exists about the effectiveness of multiple experts and no
empirical evidence exists to provide the selection and comparison of the most
appropriate expert number for different knowledge elicitation methods and tasks
(Chao & Salvedney, 1994). This topic is discussed further in the next section.

Summary of ‘How Many E xp e r ts’ Studies
Knowledge elicitation through interviews plays a critical role in the CTA
process. However, how many subject matter experts should be interviewed? Chao
and Salvedney (1994) interviewed 24 subjects from the top 9% of the computer
science students at Purdue University and discovered that acquired procedural
knowledge increased two-fold from using one to six experts. The authors
34

concluded that the optimal cost-benefit utility could be achieved by using three
experts for knowledge elicitation. Clark, Pugh, Yates, Early, and Sullivan (2008)
added that the use of CTA methods with a single subject matter expert had been
demonstrated to provide a 28% increase in the amount of information captured
from experts during the performance of a task. The authors also noted that when
multiple experts were interviewed, the percent of information captured increased
proportionately. No other published articles that empirically answered the ‗how
many experts‘ question could be located short of occasional recommendations that
two or more (e.g., Clark, Feldon, Van Merriënboer, Yates, & Early, 2008), three to
five (DuBois & Shalin, 2000), or six to eight (Shackak, Hadas-Dayagi, Ziv, & Reis,
2009) subject matter experts should be used.
Turban (1991) suggested that eliciting knowledge from multiple experts
yielded three positive outcomes. First, multiple experts increased the validity,
completeness, and importance of the acquired knowledge. Second, multiple
experts enhanced productivity. Third, multiple experts decreased and corrected the
wrong answers. Flach (2000) supported these conclusions, adding that:
Because the goal of ecological task analysis is to understand the
meaning within the domain, not in any particular head, the input
from many different perspectives is essential. Talking to multiple
experts helps differentiate those aspects that are significant with
respect to the work domain from those aspects that are significant
to a particular perspective or level of awareness (94).

Turban‘s and Flach‘s ‗more is better‘ approach seems to be supported by the
existing CTA data. Most CTAs interview two (Velmahos, Konstantinos, Sillin,
35

Chan, Clark, Theodorou, & Maupin, 2004), three (Sullivan, Brown, Peyre, Salim,
Martin, Towfigh, & Grunwald, 2007; Luker, Sullivan, Peyre, Sherman, &
Grunwald, 2008), or even more (Crandall & Gretchell-Leiter, 1993; Beard, Smith,
& Denelsbeck, 1996; Shackak, Hadas-Dayagi, Ziv, & Reis, 2009) subject matter
experts.

Summary
The Halstedian master-apprentice training and assessment model currently
employed by many American medical schools faces several barriers. Subject
matter experts teach and are largely unaware of the automated strategies that guide
most of their problem-solving, resulting in their often omitting 70% of the
procedural knowledge that novices need to know in order to successfully complete
a procedure. The result is that American medical school education has been
described as ‗unsystematic and unstructured.‘ What is needed is a way to capture
not only the 30% of the subject matter experts‘ explicit knowledge but also the
remaining 70% that is implicit and is no longer consciously accessible. This
includes the experts‘ declarative and procedural knowledge.
Cognitive Task Analysis (CTA) may hold the key. CTA focuses on
measuring the mental models used in task performance with the goal of converting
that information into new teaching tools and techniques. CTA has been shown to:
(1) improve training quality; (2) increase student speed, accuracy, and adaptability;
(3) decrease the need for training to rely solely on experts; and (4) potentially
36

generate long-term cost savings. CTA-based methods have been successfully and
effectively employed both inside and outside of medicine. What remains
unanswered, however, is how many subject matter experts should be interviewed in
a CTA for maximum effectiveness and efficiency. This study will contribute to the
research to suggest an answer to that question.

Research Questions
This study poses two research questions:
1. How much information about the open cricothyrotomy procedure does one
expert provide when compared to the combined contributions of a ‗gold
standard‘ summary based on interviews with six surgical subject matter
experts?
2. How much critical information is gained from each additional CTA interview
about the open cricothyrotomy procedure?

37

CHAPTER 2
METHODOLOGY

The open cricothyrotomy procedure is an emergency medical procedure
designed to gain and provide stable airway access when nasal tracheal intubation
and oral tracheal intubation fails.

Design
In order to answer the two questions posed for this study, the design
replicated a portion of the design of a study summarized in articles by Chao and
Salvedney (1994) and Chao, Salvedney, and Lightner (1999). Chao and Salvedney
(1994) randomly assigned 24 computer science students to one of four groups.
Each group was taught a different knowledge elicitation technique; they are
interview, protocol analysis, repertory grid, or induction. Each student was then
asked to complete a set of three tasks in random order: (1) error detection in a 278
line FORTRAN program; (2) correcting a set of invalid conditions in a similar
FORTRAN program; and (3) describing the various causes of an ‗error in format‘
FORTRAN program error message. The study then calculated the effectiveness of
the four different knowledge elicitation methods using five criteria, as follows:

38

1. The percentage of total knowledge captured [completeness].
2. Total elapsed time from the start of the knowledge elicitation session to the
completion of the rule review by an expert [time].
3. The total number of conflicting rules in the final analysis result
[inconsistency].
4. The importance of each elicited rule based on weights assigned by three
‗super experts‘ [importance of data].
5. The total number of correct rules elicited divided by the total time elapsed
[efficiency].
The mean and standard deviation were calculated for the completeness, time,
inconsistency, importance of data, and efficiency data for each task and each
knowledge elicitation method. To compare similar data across different treatment
combinations, Chao and Salvedney used a nested factorial design. The
Multivariate Analysis of Variance (MANOVA) was used to analyze the results of
this nested factorial design. A Student-Newman-Keuls (SNK) multiple range test
was also performed to investigate all possible pairs of means in a sequential
manner.
While Chao and Salvedney‘s study used five discrete criteria to measure the
results of three discrete procedures under four discrete treatments, this study had
only one criterion (completeness), one procedure (the open cricothyrotomy
procedure), and one treatment (CTA). In other words, this study focused solely on
the percentage of total knowledge captured about the open cricothyrotomy
39

procedure using a single knowledge elicitation technique — CTA. This data was
calculated using simple descriptive statistics.
Because, at present, no checklist or ‗gold standard‘ as to how to perform an
open cricothyrotomy exists against which surgical subject matter experts‘
descriptions of the procedure‘s steps can be assessed quantitatively, this study
began by collecting qualitative data from semi-structured interviews. The experts‘
qualitative interview data were then converted into a quantitative checklist, a ‗gold
standard‘, against which each expert‘s original procedural descriptions were
assessed, enabling the computation of the means and standard deviations for the
total knowledge acquired by each subject matter expert interview. This mirrored
the way that Chao and Salvedney (1994) and Chao, Salvedney, and Lightner (1999)
calculated the mean and standard deviation for completeness of data of their study.

Subjects
Six expert trauma surgeons employed by the Department of Surgery of an
urban, private medical school located in the western United States were interviewed
separately to educe the actions, decisions, analyses, and judgments that each expert
employed to perform an open cricothyrotomy procedure. In particular, the experts
were asked to describe the how-to action steps and critical decision points in the
procedure. This study was submitted to that medical school‘s institutional review
board who determined that board review was unnecessary because no identifiable
private information was recorded about the study‘s participants.
40

Data Collection
Using a design discussed in Clark, Pugh, Yates, Early, and Sullivan (2008)
and outlined in a proprietary training aid produced by Expert Knowledge Solutions
(2009), data collection for this study was conducted in four phases: (1) semi-
structured interviews of surgical subject matter experts; (2) interview coding;
(3) creation of a six subject matter expert ‗gold standard‘; and (4) comparison of
the contents of each expert‘s interview against the ‗gold standard‘.

Semi-structured Interviews
During the semi-structured qualitative interviews, the experts were asked to
answer a series of questions or to respond to a series of prompts that focused on the
key tasks and potential problems that a surgeon could encounter during an open
cricothyrotomy procedure. Attention was focused on eliciting the indications and
contraindications for performing the procedure. Each expert was asked to describe
the time and quality standards for the procedure and to list the equipment that was
required. The experts were then asked how to perform each specific task and sub-
task in the open cricothyrotomy procedure, step-by-step. Attention was focused on
eliciting who did what, when, and where. Attention was also focused on eliciting
the procedure‘s action and decision steps, alternatives to those steps, and the
criteria used for decision making. Each interview was recorded in audio.

41

Interview Coding
Each qualitative interview was transcribed verbatim from the audio
recordings. Two trained coders independently coded the transcripts using a coding
scheme designed by Expert Knowledge Solutions (2009). In particular, the coders
were asked to carefully review each interview transcript and to note anything said
by the expert that could be considered to be an action step, a decision step, an
objective, an indication, a contra-indication, and so forth. The coders then met to
compare their coding results and resolve discrepancies. Inter-rater reliability was
calculated to assess the consistency of the coding.

Creation of a Six Subject Matter Expert s’ ‘Gold Standard ’ Protocol
The coded data for each individual subject matter was summarized in a
structured CTA report. A CTA report is an ordered, procedural checklist that listed
all of the equipment, conditions, action steps, and decision steps that the subject
matter expert mentioned in his or her interview as being necessary to successfully
perform an open cricothyrotomy procedure. Each surgical subject matter expert
was then asked to review the results of his or her own CTA report for accuracy.
Any additions or deletions were incorporated into that expert‘s final CTA report.
The edited CTA reports for the first three subject matter experts were
combined into a single, three-expert CTA report. Each surgical subject matter
expert was asked to review the three-expert CTA report for accuracy and to identify
42

any of the report‘s items that were unnecessary or incorrect. Those additions or
deletions were incorporated into the final three-expert CTA report.
Concurrent with this, an additional three subject matter experts underwent
the exact same process. They were separately interviewed, their interviews were
transcribed and coded, and their coded interviews were converted into individual
CTA reports. Each subject matter expert was asked to review and edit his or her
own CTA report. Additions or deletions were incorporated into that expert‘s final
CTA report. Each subject matter expert‘s final CTA report was combined into a
three-expert CTA report. Each subject matter expert was asked to review the three-
expert CTA report, and additions or deletions were incorporated into the final
three-expert CTA report.
Once a three-expert CTA report was generated for both groups of subject
matter experts, both CTA reports were then combined into a six-expert CTA report.
All six surgical subject matter experts were asked to review the six-expert CTA
report for accuracy and to identify any of the report‘s items that were unnecessary
or incorrect. Those additions or deletions were incorporated into the final six-
expert CTA report. This final report became the open cricothyrotomy procedure‘s
‗gold standard‘, which is the definitive list of equipment, conditions, action steps,
and decision steps that the six subject matter experts who participated in this study
considered necessary to successfully perform an open cricothyrotomy procedure.

43

Comparison against the ‘gold standard ’
Using the ‗gold standard‘ as a checklist, each expert‘s final CTA report was
manually graded, item-by-item, for completeness. One point was assigned for each
discrete item (e.g., condition, action step, decision step) on a subject matter expert‘s
CTA report that matched the same item on the ‗gold standard‘. Zero points were
assigned for any item that appeared on the ‗gold standard‘ but not on the subject
matter expert‘s CTA report.

Data Analysis
To answer the first research question, ―How much information about the
open cricothyrotomy procedure does one expert provide when compared to the
combined contributions of a ‗gold standard‘ summary based on interviews with six
surgical subject matter experts?,‖ a knowledge acquired score for each subject
matter expert‘s CTA report was calculated. This measurement unit was calculated
using the formula

𝑆 𝑀 𝐸 𝐺 𝑆 ∗ 100
where ―SME‖ equals the total number of the procedure‘s items captured from each
individual subject matter expert and ―GS‖ equals the total number of items in the
‗gold standard‘. Knowledge-acquired scores were calculated for total knowledge
(all items in the ‗gold standard‘). Knowledge acquired scores were also calculated
for all subsets of the ‗gold standard‘ including action steps, decision steps,
44

objectives, reasons for performing the procedure, risks of not performing well,
indications, contraindications, standards as to how to judge if the procedure was
successful, mandatory equipment items, recommended equipment items, and
overall procedure tasks. Descriptive statistics were also calculated for both total
knowledge and all subsets.
To answer the second research question, ―How much critical information is
gained from each additional CTA interview about the open cricothyrotomy
procedure?,‖ an additional knowledge-acquired score for each possible non-
repeating two, three, four, five, and six subject matter expert groups was calculated.
This measurement unit was represented by the formula:

𝑆 𝑀 𝐸 𝑔 𝐺 𝑆 ∗ 100
where ―SMEg‖ equals the number of the procedure‘s items captured from each
non-repeating group of subject matter experts and ―GS‖ equals the total number of
items in the ‗gold standard‘. As with the first research question, knowledge
acquired scores were calculated for total knowledge (all items in the ‗gold
standard‘). Knowledge acquired scores were also calculated for the ‗gold
standard‘s‘ action steps, decision steps, objectives, reasons for performing the
procedure, risks of not performing well, indications, contraindications, standards as
to how to judge if the procedure was successful, mandatory equipment items,
recommended equipment items, and overall procedure tasks. Descriptive statistics
were also calculated.
45

The marginal utility of acquiring knowledge from each additional subject
matter expert was caculated using the formula:
Δ𝑈 Δ𝑄
where ―U‖ is the knowledge acquired score and ―Q‖ is the quantity of subject
matter experts that were utilized. This marginal utility calculation represents the
percentage of acquired-knowledge change when each additional expert is utilized
(Chao & Salvedney, 1994). The point of diminishing marginal utility was
calculated as the point where the marginal utility of the next expert added was less
than 10% (Chao & Salvedney, 1994).

46

CHAPTER 3
RESULTS

Co-coding and Inter-coder Reliability
Six expert trauma surgeons employed by the Department of Surgery of a
private, urban medical school located in the western United States, were separately
interviewed about how to perform an open cricothyrotomy procedure, an
emergency medical procedure designed to gain and provide stable airway access
when nasal tracheal intubation and oral tracheal intubation fails. Those interviews
were recorded and transcribed. Pairs of doctoral students in education who had
been trained in Cognitive Task Analysis (CTA) methods coded the transcripts using
a coding scheme designed by Expert Knowledge Solutions (2009). In particular,
the coders independently reviewed each transcript and noted anything said by the
expert that could be considered to be an action step, a decision step, an objective,
an indication, a contra-indication, and so forth. The coders then met to compare
their coding results and resolve discrepancies.
Co-coding of the six interview transcripts resulted in a mean inter-coder
reliability of 96%. Of the six pairs of co-coded interview transcripts, the minimum
inter-coder reliability was 91.53% and the maximum inter-rater reliability was
100%.
47

The coded results from each subject matter expert‘s interviews were
converted into individual CTA reports. CTA reports are ordered, procedural
checklists that listed all of the equipment, conditions, action steps, and decision
steps that that particular subject matter expert mentioned in his or her interview as
being necessary to successfully perform an open cricothyrotomy procedure. Each
subject matter expert then reviewed and edited his or her own CTA report for
accuracy. The edited CTA reports for the first three subject matter experts were
then combined into a new, three-person CTA report that was then reviewed and
edited by the report‘s respective subject matter experts. The same process was
repeated with the remaining three subject matter experts. The two resulting three-
person CTA reports were then combined into a six-person CTA report that was
reviewed and edited by each of the six subject matter experts. The final edited and
reviewed six-person CTA report became this study‘s open cricothyrotomy
procedure‘s ‗gold standard‘, which is the definitive list of equipment, conditions,
action steps, and decision steps that the six subject matter experts who participated
in this study considered necessary to successfully perform an open cricothyrotomy
procedure.

48

Research Questions
Research Question 1
How much information about the open cricothyrotomy procedure does one expert
provide when compared to the combined contributions of a ‘gold standard’
summary based on interviews with six surgical subject matter experts?
Single expert CTA interviews elicited an average of 56% of the total
knowledge, 66% of the action steps, and 28% of the decision steps found in the
open cricothyrotomy six-person ‗gold standard‘. In addition, while single expert
CTA interviews elicited, on average, 100% of the objectives, mandatory
equipment, and tasks found in the open cricothyrotomy six-person ‗gold standard‘,
those same interviews elicited, on average, 51% of the ‗gold standard‘s‘
recommended equipment items, 33% of the risks for not performing well, and 20%
of the contraindications.
Using the six-expert ‗gold standard‘ as a checklist, each individual expert‘s
final, edited CTA report was manually graded, item-by-item, for completeness.
One point was assigned for each discrete item on a subject matter expert‘s CTA
report that matched the same item on the ‗gold standard‘. Dividing the cumulative
completeness score of each expert‘s CTA report by the total number of items on the
‗gold standard‘ resulted in a knowledge-acquired percentage for each subject matter
expert. Knowledge-acquired percentages were computed for total knowledge (all
items in the ‗gold standard‘) as well for all of the procedure‘s action steps, decision
steps, objectives, reasons for performing the procedure, risks of not performing
49

well, indications, contraindications, standards as to how to judge if the procedure
was successful, mandatory equipment items, recommended equipment items, and
overall procedure tasks. Table 1 shows the total knowledge, action steps, and
decision steps elicited from a single expert CTA interview when compared to a six-
person ‗gold standard‘. The last column, 𝑥 , shows the mean percentage of
knowledge acquired in each particular category.

Table 1: Percentage of Knowledge Acquired from One Expert When Compared
to a Six Subject Matter Expert ‗Gold Standard‘ Protocol

SME 1 SME 2 SME 3 SME 4 SME 5 SME 6
𝒙

Total Knowledge Acquired 59 62 60 43 55 59 56
Action Steps Acquired 76 79 64 39 58 82 66
Decision Steps Acquired 15 31 54 31 23 15 28
Objectives Acquired 100 100 100 100 100 100 100
Reasons Acquired 100 100 0 100 100 100 83
Risks Acquired 33 0 67 67 33 0 33
Indications Acquired 100 100 67 33 100 67 78
Contraindications Acquired 0 60 0 0 20 40 20
Standards Acquired 78 44 67 33 67 33 54
Mandatory Equipment
Acquired
100 100 100 100 100 100 100
Recommended Equipment
Acquired
45 50 55 45 55 55 51
Tasks Acquired 100 100 100 100 100 100 100

50

Research Question 2
How much critical information is gained from each additional CTA interview about
the open cricothyrotomy procedure?
The amount of information acquired from each additional CTA interview
varied dependent upon the amount of experts that were interviewed and the type of
knowledge that was being measured. For example, with three experts, the amount
of total knowledge acquired increased from 56% (with one expert) to 85%, the
amount of action steps acquired increased from 66% to 93%, and the amount of
decision steps acquired increased from 28% to 67%. Similar increases were also
recorded for most of the other knowledge types, with three exceptions. Because
each individual expert was able to correctly identify all of the procedure's ‗gold
standard‘ objectives, mandatory equipment items, and tasks, no additional
information was acquired with the addition of extra experts.
Completeness scores for each possible non-repeating two, three, four, five,
and six subject matter expert groups were also calculated. Dividing the cumulative
completeness score of each subject matter expert group by the total number of
items on the ‗gold standard‘ resulted in a knowledge-acquired percentage for each
subject matter expert group. Knowledge-acquired percentages were computed for
total knowledge (all items in the ‗gold standard‘) as well for all of the procedure‘s
action steps, decision steps, objectives, reasons for performing the procedure, risks
of not performing well, indications, contraindications, standards as to how to judge
if the procedure was successful, mandatory equipment items, recommended
51

equipment items, and overall procedure tasks. Following Chao and Salvedney
(1994), marginal utility was calculated as the percentage of acquired knowledge
change when each additional expert is utilized. The point of diminishing marginal
utility was calculated as the point where the marginal utility of the next expert
added was less than 10%.
While Table 1 contains a dozen different categories (from Total Knowledge
Acquired to Tasks Acquired) for usability and readability purposes, the remaining
tables in this chapter each contain only three categories. Table 2 shows the average
percentage of knowledge, actions steps, and decision steps acquired as the number
of experts increases from one to six. In particular, Table 2 shows that the use of
multiple experts increased the total knowledge, action steps, and decision steps
acquired although there was a diminishing return from the addition of each
additional expert.
Please note that the first column each category (

x
) in Table 2 and all
subsequent tables in this chapter show the mean percentage of knowledge acquired
in that particular category with the addition of each extra subject matter expert.
The second column (

) shows the standard deviation of the scores within the six
non-repeating one-expert groups, the fifteen non-repeating two-expert groups, the
twenty non-repeating three-expert groups, the fifteen non-repeating four-expert
groups, and the six non-repeating five-expert groups. Because there was only one
non-repeating six-expert group, a standard deviation could not be calculated for the
percentage of knowledge acquired from six experts.
52

Table 2: Average Percentage of Total Knowledge, Action Steps, and Decision
Steps Acquired from Multiple Groups of Experts When Compared to a
Six Subject Matter Expert ‗Gold Standard‘ Protocol

Number of
Experts
Total Knowledge Action Steps Decision Steps
𝑥

 𝑥

 𝑥


1 56 6.5 66 5.3 28 1.9
2 74 3.8 85 3.0 50 2.2
3 85 3.4 93 1.9 67 1.9
4 91 3.2 97 1.6 80 1.5
5 96 3.0 98 1.2 91 1.2
6 100 NA 100 NA 100 NA

Figure 1 shows the percentage of total knowledge acquired (as compared to
the six-expert ‗gold standard‘) with the addition of each extra subject matter expert.
The average total knowledge acquired increased from 56% with one expert to 85%
with three experts to 100% with six experts.

Figure 1: Percentage of six-expert ‗gold standard‘ total knowledge acquired as
a function of the number of experts.
56%
74%
85%
91%
96%
100%
55
60
65
70
75
80
85
90
95
100
1 2 3 4 5 6
% Total Knowledge Acquired
Number of Experts
53

Figure 2 shows the marginal utility of the percent of total knowledge
acquired with the addition of each extra subject matter expert. By interviewing one
subject matter expert and zero additional experts, this study was able to capture
56% of the ‗gold standard‘s‘ total knowledge items. The marginal benefit of
adding one additional expert was 18%. In other words, that one additional expert
provided an additional 18% of the ‗gold standard‘s‘ total knowledge. Figure 2 also
shows that four experts are recommended if a 10% marginal utility in total
knowledge acquired is expected.

Figure 2: Percentage of additional total knowledge acquired as a function of the
number of experts.

Figure 3 shows the percentage of action steps acquired (as compared to the
six-expert ‗gold standard‘) with the addition of each extra subject matter expert.
The average percentage of action steps acquired increased from 56% with one
expert to 100% with six experts.
56%
18%
11%
6%
5%
4%
0
10
20
30
40
50
60
1 2 3 4 5 6
% Additional Total Knowledge Acquired
Number of Experts
54

Figure 3: Percentage of six-expert ‗gold standard‘ action steps acquired as a
function of the number of experts.

Figure 4 shows the marginal utility of the percent of action steps acquired
with the addition of each extra subject matter expert. In particular, Figure 4 shows
that three experts are recommended if a 10% marginal utility in action steps
acquired is expected.
Figure 5 shows the percentage of decision steps acquired (as compared to
the six-expert ‗gold standard‘) with the addition of each extra subject matter expert.
The average percentage of decision steps acquired increased from 28% with one
expert to 67% with three experts, to 100% with six experts.

66%
85%
93%
97%
98%
100%
65
70
75
80
85
90
95
100
1 2 3 4 5 6
% Action Steps Acquired
Number of Experts
55

Figure 4: Percentage of additional action steps acquired as a function of the
number of experts.

Figure 5: Percentage of six-expert ‗gold standard‘ decision steps acquired as a
function of the number of experts.

Figure 6 shows the marginal utility of the percent of decision steps acquired
with the addition of each additional subject matter expert. In particular, Figure 6
66%
19%
8%
4%
1%
2%
0
10
20
30
40
50
60
70
1 2 3 4 5 6
% Additional Action Steps Acquired
Number of Experts
28%
50%
67%
80%
91%
100%
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6
% Decision Steps Acquired
Number of Experts
56

shows that a total of six experts are recommended if a 10% marginal utility in
decision steps acquired is expected.

Figure 6: Percentage of additional decisions acquired as a function of the number
of experts.

Table 3 shows the average percentage of objectives, reasons, and risks of
not performing the procedure well (e.g., death, anoxic brain injury) acquired as the
number of experts increases from one to six. In particular, Table 3 shows that the
addition of multiple experts had no impact on the percentage of ‗gold standard‘
objectives acquired. Using multiple experts did increase the percentage of ‗gold
standard‘ reasons acquired, although that benefit ceased after the addition of a
second expert. Using multiple experts also increased the percentage of ‗gold
standard‘ risks acquired, although that benefit ceased after the addition of a fifth
expert.
28%
22%
17%
13%
11%
9%
0
5
10
15
20
25
30
1 2 3 4 5 6
% Additional Decision Steps Acquired
Number of Experts
57

Table 3: Average Percentage of Objectives, Reasons, and Risks Acquired from
Multiple Groups of Experts When Compared to a Six Subject Matter
Expert ‗Gold Standard‘ Protocol

Number of
Experts
Objectives Reasons Risks
𝑥

 𝑥

 𝑥


1 100 0 83 0.4 33 0.9
2 100 0 100 0 60 0.9
3 100 0 100 0 80 0.7
4 100 0 100 0 93 0.4
5 100 0 100 0 100 0
6 100 NA 100 NA 100 NA

Because the addition of extra subject matter experts did not increase the number of
objectives acquired, graphs of this data are not needed. A total of one expert is
recommended if a 10% marginal utility in objectives acquired is expected. Graphs
are also not needed for the ‗Reasons‘ category as the number of reasons acquired
reaches 100% with the addition of a single extra subject matter expert. A total of
two experts are recommended if a 10% marginal utility in reasons acquired is
expected.
Figure 7, however, shows the percentage of risks acquired (as compared to
the six-expert ‗gold standard‘) with the addition of each extra subject matter expert.
The average percentage of risks acquired increased from 33% with one expert to
80% with three experts, to 100% with five experts.

58

Figure 7: Percentage of six-expert ‗gold standard‘ risks acquired as a function
of the number of experts.

Figure 8 shows the marginal utility of the percent of risks acquired with the
addition of each extra subject matter expert. In particular, Figure 8 shows that a
total of five experts are recommended if a 10% marginal utility in risks acquired is
expected.

33%
60%
80%
93%
100% 100%
30
40
50
60
70
80
90
100
1 2 3 4 5 6
% Risks Acquired
Number of Experts
59

Figure 8: Percentage of additional risks acquired as a function of the number of
experts.

Table 4 shows the average percentage of indications, contraindications, and
standards acquired as the number of experts increases from one to six. In
particular, Table 4 shows that the addition of multiple experts increased the
percentage of ‗gold standard‘ indications, contraindications, and standards
acquired, although no new indications were acquired after the addition of a fourth
expert. By way of comparison, it took six experts to capture all of the ‗gold
standard‘s‘ contraindications and standards.

33%
27%
20%
13%
7%
0%
0
5
10
15
20
25
30
35
1 2 3 4 5 6
% Additional Risks Acquired
Number of Experts
60

Table 4: Average Percentage of Indications, Contraindications, and Standards
Acquired from Multiple Groups of Experts When Compared to a Six
Subject Matter Expert ‗Gold Standard‘ Protocol

Number
of experts
Indications Contraindications Standards
𝑥

 𝑥

 𝑥


1 78 0.8 20 1.3 54 1.7
2 93 0.4 39 1.4 73 1.5
3 98 0.2 56 1.3 85 1.2
4 100 0 72 1.1 92 1
5 100 0 87 0.8 96 0.8
6 100 NA 100 NA 100 NA

Figure 9 shows the percentage of indications acquired (as compared to the
six-expert ‗gold standard‘) with the addition of each extra subject matter expert.
The average percentage of indications acquired increased from 78% with one
expert to 98% with three experts, to 100% with four experts.

Figure 9: Percentage of six-expert ‗gold standard‘ indications acquired as a
function of the number of experts.

78%
93%
98%
100% 100% 100%
75
80
85
90
95
100
1 2 3 4 5 6
% Indications Acquuired
Number of Experts
61

Figure 10 shows the marginal utility of the percent of indications acquired
with the addition of each extra subject matter expert. In particular, Figure 10
shows that a total of three experts are recommended if a 10% marginal utility in
indications acquired is expected.

Figure 10: Percentage of additional indications acquired as a function of the
number of experts.

Figure 11 shows the percentage of contraindications acquired (as compared
to the six-expert ‗gold standard‘) with the addition of each extra subject matter
expert. The average percentage of contraindications acquired increased from 20%
with one expert to 56% with three experts, to 100% with six experts.

78%
15%
5%
2%
0% 0%
0
10
20
30
40
50
60
70
80
1 2 3 4 5 6
% Additional Indications
Acquired
Number of Experts
62

Figure 11: Percentage of six-expert ‗gold standard‘ contraindications acquired
as a function of the number of experts.

Figure 12: Percentage of additional contraindications acquired as a function of
the number of experts.

Figure 12 above shows the marginal utility of the percent of
contraindications acquired with the addition of each extra subject matter expert. In
20%
39%
56%
72%
87%
100%
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6
% Contraindications Acquired
Number of Experts
20% 19%
17%
16%
15%
13%
10
12
14
16
18
20
1 2 3 4 5 6
% Additional Contraindications
Acquired
Number of Experts
63

particular, Figure 12 shows that at no point did the marginal utility of any
additional expert fell beneath 10%.
Figure 13 shows the percentage of standards acquired (as compared to the
six-expert ‗gold standard‘) with the addition of each extra subject matter expert.
The average percentage of standards acquired increases from 54% with one expert
to 85% with three experts, to 100% with six experts.

Figure 13: Percentage of six-expert ‗gold standard‘ standards acquired as a
function of the number of experts.

Figure 14 shows the marginal utility of the percent of standards acquired
with the addition of each extra subject matter expert. In particular, Figure 14
shows that a total of four experts are recommended if a 10% marginal utility in
standards acquired is expected.

54%
73%
85%
92%
96% 100%
50
55
60
65
70
75
80
85
90
95
100
1 2 3 4 5 6
% Standards Acquired
Number of Experts
64

Figure 14: Percentage of additional standards acquired as a function of the
number of experts.

Table 5 shows the average percentage of mandatory equipment items,
recommended equipment items, and tasks acquired as the number of experts
increased from one to six. In particular, using additional experts had no impact on
the percentage of ‗gold standard‘ mandatory equipment items or tasks acquired,
although it did take six experts to acquire all of the ‗gold standard‘s‘ recommended
equipment items. One expert is recommended if a 10% marginal utility in
mandatory equipment items or tasks acquired is expected.

54%
19%
12%
7%
4% 4%
0
10
20
30
40
50
1 2 3 4 5 5
% Additional Standards Acquired
Number of Experts
65

Table 5: Average Percentage of Mandatory Equipment, Recommended
Equipment, and Tasks Acquired from Multiple Groups of Experts When
Compared to a Six Subject Matter Expert ‗Gold Standard‘ Protocol

Number of
Experts
Mandatory
Equipment
Recommended
Equipment
Tasks
𝑥

 𝑥

 𝑥


1 100 0 51 1.0 100 0
2 100 0 71 1.1 100 0
3 100 0 83 1.1 100 0
4 100 0 90 1.0 100 0
5 100 0 96 0.8 100 0
6 100 NA 100 NA 100 NA

Because the addition of extra subject matter experts did not increase either the
number of mandatory equipment items or the number of tasks acquired, graphs of
this data are not needed.
Figure 15 shows the percentage of recommended equipment items acquired
(as compared to the six-expert ‗gold standard‘) with the addition of each extra
subject matter expert. The average percentage of recommended equipment items
acquired increased from 51% with one expert to 83% with three experts to 100%
with six experts.

66

Figure 15. Percentage of six-expert ‗gold standard‘ recommended equipment
acquired as a function of the number of experts.

Figure 16: Percentage of additional recommended equipment acquired as a
function of the number of experts.

Figure 16 above shows the marginal utility of the percent of recommended
equipment items acquired with the addition of each extra subject matter expert. In
51%
71%
83%
90%
96% 100%
50
55
60
65
70
75
80
85
90
95
100
1 2 3 4 5 6
% Recommended Equipment
Acquired
Number of Experts
51%
20%
12%
7%
6%
4%
0
10
20
30
40
50
1 2 3 4 5 6
% Additional Recommended Equipment
Acquired
Number of Experts
67

particular, Figure 16 shows that a total of four experts are recommended if a 10%
marginal utility in recommended equipment items acquired is expected.

Summary
Single expert CTA interviews elicited an average of 56% of the total
knowledge, 66% of the action steps, and 28% of the decision steps found in an
open cricothyrotomy six-person ‗gold standard‘. The amount of information
acquired from each additional CTA interview varied depending upon the amount of
experts that were interviewed and the type of knowledge that was being measured.
Table 6 shows the point of diminishing marginal utility for both total knowledge
acquired as well as for each of the subsets of knowledge contained in the six-person
‗gold standard‘.

Table 6: Quantity of Experts Recommended if a 10% Marginal Utility in
Knowledge Acquisition is Expected

Number of Experts
Total Knowledge Acquired 4
Action Steps Acquired 3
Decision Steps Acquired 6
Objectives Acquired 1
Reasons Acquired 2
Risks Acquired 5
Indications Acquired 3
Contraindications Acquired NA
Standards Acquired 4
Mandatory Equipment Acquired 1
Recommended Equipment Acquired 4
Tasks Acquired 1

68

CHAPTER 4
DISCUSSION AND CONCLUSIONS

This purpose of this study was to identify the optimal number of experts
needed to teach a surgical resident how to perform an open cricothyrotomy
procedure. While no formal hypotheses were stated, two research questions guided
this study. Previous studies had measured the maximum percentage of procedural
knowledge captured using Cognitive Task Analysis (CTA) knowledge elicitation
techniques; hence, this study‘s first research question sought to confirm those prior
findings. Using that data as a starting point, this study‘s second research question
sought to confirm the findings of previous studies that the marginal utility of the
percentage of procedural knowledge acquired does indeed decrease as the number
of experts increase. This study‘s second research question also sought to quantify
the total number of experts necessary to reach the point of diminishing marginal
utility for knowledge acquisition in surgical subject matter expert interviews.

Research Questions
Research Question 1
How much information about the open cricothyrotomy procedure does one expert
provide when compared to the combined contributions of a ‘gold standard’
summary based on interviews with six surgical subject matter experts?
69

The results of this study support the findings of previous studies that show
that, even with CTA knowledge elicitation techniques, experts have a propensity to
provide imprecise, partial, or even erroneous descriptions and often omit a sizable
portion of a procedure‘s steps when asked to describe how to perform that
procedure. Each individual surgical subject matter expert in this study, on average,
provided only 56% of the total steps in a six-expert ‗gold standard‘ on how to
perform an open cricothyrotomy procedure. In addition, this study found a distinct
difference between the amount of the procedure‘s overt actions that were elicited
from each expert and the amount of covert cognitive processes that informed those
actions that were also elicited. For example, each expert interview provided, on
average, 66% of the ‗gold standard‘s‘ action steps but only 28% of the decision
steps, which confirms the findings of other studies that experts omit as much as
57% of the procedure‘s critical decision steps.
Action steps are directly observable behaviors while decision steps involve
cognitive processes and structures that inform when or when not to perform an
action or procedure. Because the cognitive processes that inform the experts‘
decisions are based on automated, procedural knowledge, the experts in this study
were unable to verbalize almost two-thirds of the decision steps they employed
when performing the procedure. The difference between the amount of knowledge
the experts provided about observable items (e.g., objectives, indications,
mandatory equipment, tasks) and unobservable items (e.g., risks of not performing
well, contraindications) were reflected throughout the data and are noteworthy.
70

While there have been only a handful of published studies that focus on the
effectiveness of CTA in medicine and medical training, most of those studies
provide data about only the total procedure steps, action steps, and decision steps
acquired from each expert. This study appears to be the first study about the
effectiveness of CTA in medicine that also includes data about all of the subsets of
knowledge (e.g., risks of not performing the procedure well, contraindications,
recommended equipment items) within a larger medical procedure. For example,
each of the experts in this study was able to provide 100% of the objectives,
mandatory equipment items, and tasks found in a six-expert ‗gold standard‘.
However, each expert in this study also, on average, omitted 67% of the six-expert
‗gold standard‘ risks, 80% of the contraindications, 46% of the standards, and 59%
of the recommended equipment items.

Research Question 2
How much critical information is gained from each additional CTA interview about
the open cricothyrotomy procedure?
The results of this study support previous findings that the marginal utility
of the percentage of procedural knowledge acquired decreases as the number of
experts increase. The results of this study also partially support previous findings
that interviews with three to five subject matter experts typically result in a
converging set of job duties and task characteristics because multiple experts often
provide different subsets of knowledge. Three experts were required to reach a
71

10% point of diminishing marginal utility for the acquisition of action steps and
indications and four experts were needed to reach the point of diminishing marginal
utility for the acquisition of total knowledge, standards, and recommended
equipment items.
The results of this study appear to conflict with Chao and Salvedney‘s
(1994) finding that using three experts is optimal for knowledge elicitation. Rather,
this study‘s data show that four experts are needed if a 10% marginal utility in total
knowledge acquisition is expected. However, it is unclear if the declarative and
procedural FORTRAN programming knowledge elicited by Chao and Salvedney is
analogous to the declarative and procedural knowledge elicited by this study. For
example, while Chao and Salvedney assigned importance scores to each item they
measured, each item on this study's six-expert ‗gold standard‘ was given equal
weight when calculating both completeness scores and marginal utility. In
addition, while neither Chao and Salvedney (1994) nor Chao, Salvedney, and
Lightner (1999) specifically stated how many decision steps their study measured,
one possible reason for the difference between the point of diminishing marginal
utility found in this study and the one found in previous studies is that the number
of decision steps in the six-expert open cricothyrotomy ‗gold standard‘ (13) may
have exceeded the number of decision steps in those previous studies. Research
shows that surgical subject matter experts omit over half of the procedure‘s critical
decision steps (Sullivan et al., 2008). Further research is needed to determine if
there is a positive correlation between the number of decision steps in a procedure,
72

the nature of the domain and task steps, and that specific procedure‘s point of
diminishing marginal utility for total knowledge acquired. In this study, each
surgical subject matter expert, on average, included only 28% of the six-expert
open cricothyrotomy procedure ‗gold standard‘ decision steps, and the point of
diminishing marginal utility for the percentage of additional decision steps acquired
was not four experts but rather six experts. Further research is needed to determine
if the point of diminishing marginal utility for total knowledge acquisition found in
this study is limited to this study or extends to studies of other medical procedures
as well.
This leads to another issue; that is, the point of diminishing marginal utility
in knowledge acquisition was not uniform for all subsets of knowledge within this
study‘s six-expert ‗gold standard‘. As mentioned in the previous section, this study
appears to be the first study about the effectiveness of CTA in medicine that also
includes data about the subsets of knowledge within a larger medical procedure.
Similar to what was found when analyzing the contributions of one expert, this
study also found a distinct difference between the amount of the procedure‘s overt
actions that were elicited from multiple experts and the amount of covert cognitive
processes that informed those actions. The point of diminishing marginal utility in
knowledge acquisition varied widely depending upon the subset of knowledge
being measured from only one expert for objectives, mandatory equipment, and
tasks, to six experts for decision steps. In fact, it is possible that using the number
of experts suggested by this study‘s diminishing marginal utility function for total
73

knowledge acquired may give one an incomplete picture of how to perform an open
cricothyrotomy procedure. This study‘s data shows that while interviewing four
experts captures 91% of the six-expert ‗gold standard‘s‘ total knowledge,
interviewing that same number of experts captures only 80% of the procedure‘s
decision steps and 72% of the procedure‘s contraindications. Because groups of
experts omitted more covert decision steps than overt action steps, to maximize the
effectiveness and efficiency of a CTA the number of experts to be interviewed may
need to be determined not by how many experts are necessary to reach a point of
diminishing marginal utility for all of the procedure‘s steps but rather by how many
experts are needed to reach the point of diminishing marginal utility for covert
items such as decision steps, risks of not performing well, and contraindications.

Summary
Six expert trauma surgeons were separately interviewed about how to
perform an open cricothyrotomy procedure. Through an iterative process, those
interviews were coded and converted into individual CTA reports. CTA reports are
ordered, procedural checklists that listed all of the equipment, conditions, action
steps, and decision steps that the subject matter expert mentioned in his or her
interview as being necessary to successfully perform an open cricothyrotomy
procedure. Following a lengthy analysis, aggregation, and validation process, those
individual CTA reports were converted into a single, six-expert ‗gold standard‘
74

protocol. It was against this ‗gold standard‘ that the CTA report for each expert
was graded for completeness.
The results of this study indicate that, on average, each surgical subject
matter expert who participated in this study provided a little more than half of the
total knowledge, two-thirds of the action steps, and a little more than one-quarter of
the decision steps contained within a six-expert open cricothyrotomy procedure
‗gold standard‘. This data is congruent with knowledge acquisition data found in
similar CTA studies. The results of this study also show that four experts were
needed to reach a 10% point of diminishing marginal utility for the acquisition of
additional total knowledge contained within a six-expert open cricothyrotomy
procedure ‗gold standard‘. This appears to contradict the conclusions of previous
studies that showed that using three experts is optimal for knowledge elicitation.
However, it remains unclear if the declarative and procedural knowledge elicited in
those studies are analogous to declarative and procedural knowledge in a medical
procedure.

Limitations and Delimitations
A limitation of this study is that each item on the six-expert ‗gold standard‘
was given equal weight when calculating both the scores‘ completeness and
marginal utility. By way of comparison, Chao, Salvedney, and Lightner (1999)
asked three ‗super experts‘ to use a five point Likert scale to estimate each acquired
rule‘s importance from 1 - not important to 5 - extremely important. The inclusion
75

of unimportant items on this study‘s six-expert ‗gold standard‘ could have
artificially deflated each subject matter expert‘s score‘s completeness and/or
artificially inflated the point of diminishing marginal utility for the percentage of
knowledge acquired.
The 10% threshold for diminishing marginal utility employed in this study
was chosen solely because that same threshold was used in a prior study, not
because of any statistically-based research showing that the 10% threshold was
meaningful. However, considering that the opportunity cost of interviewing a
subject matter expert, coding the interview, converting the interview into a CTA
report, having the expert review and correct the CTA report, and then incorporating
the expert‘s feedback into a final, edited CTA report may greatly exceed the benefit
of that process if the result is the acquisition of only a few percentage points of new
knowledge. In this regard, the 10% threshold seems reasonable.
Another limitation of this study is that it did not exactly mirror the subject
matter expert review process of other CTA studies. In other CTA studies, subject
matter experts reviewed their own CTA reports, then reviewed an anonymous CTA
report of a fellow subject matter expert, and then reviewed an aggregated CTA
report containing the contributions of all participating subject matter experts. In
this study, subject matter experts reviewed their own CTA reports, then reviewed
an aggregated CTA report containing their contributions and the contributions of
two other subject matter experts, and then reviewed an aggregated CTA report
containing the contributions of all six participating subject matter experts. One
76

consequence of this may be that the experts, when reviewing the three-expert CTA
report, may have been less likely to mark for deletion items that were unnecessary
or incorrect than if the CTA report only contained information from a single expert.
An additional consequence is that self-review of a CTA report may lead to the same
inaccuracies and incompleteness evidenced in single subject matter expert CTA
interviews.
A final limitation of this study is that while the participants in this study
were all trauma surgeons employed by the Department of Surgery of a private,
urban medical school located in the western United States, the surgeons were of
different ages, varied levels of expertise, and therefore may have had different skill
levels. In addition, it is unknown if the inclusion of subject matter experts from
other institutions would have impacted this study‘s data.

Implications
The major implication of this study is that, if this study‘s findings are
confirmed by CTA studies of other surgical procedures, it may encourage a re-
examination of the belief that only three subject matter experts need to be
interviewed in order to capture enough information about a surgical procedure for a
resident to be able to perform that procedure successfully. While the addition of a
fourth expert may increase the cost of conducting a CTA compared with the cost of
a three-expert CTA, the benefit may be the inclusion of additional critical
procedural knowledge that might otherwise be overlooked or excluded.
77

An additional implication is that only one expert may need to be
interviewed to capture all of the procedure‘s objectives, mandatory equipment
items, and tasks. If additional research supports this finding, this would free
cognitive task analysts in subsequent interviews to largely focus on capturing the
procedure‘s cognitive processes that are often not verbalizable (e.g., decision steps,
risks of not performing well, contraindications). This would fundamentally change
the way that we conduct CTAs. One expert would be interviewed to elicit the
procedure‘s observable items (e.g., action steps) and subsequent interviews would
focus primarily on eliciting the covert, automated knowledge that is often most
difficult to capture.
We may also need to examine the process of allowing experts to review
their own CTA reports, the CTA reports of the colleagues, or the combined ‗gold
standard‘ reports. The differences between observable actions and unobservable
cognitive processes are significant and it is possible that self-review of a CTA
report may result in the same inaccuracies and incompleteness of other self-reports.
Furthermore, should this study‘s findings be confirmed in future studies,
this study may encourage a deeper analysis of the subsets of knowledge contained
within a particular procedure. While prior studies have focused on total
knowledge, action steps, and decision steps, this study indicates that there may be
great variations in the points of diminishing marginal utility for subsets of
knowledge including contraindications and risks of not performing the procedure
well. Careful analysis of these subsets of knowledge may be warranted.
78

A final implication of this study is that it has produced a six-expert ‗gold
standard‘ on how to perform an open cricothyrotomy procedure. This ‗gold
standard‘ can be used to inform the development of future medical training
materials, potentially increasing the content coverage and instructional efficiency of
those materials.

Conclusion
In conclusion, this study is only a first step towards exploring and better
understanding how much knowledge is — and should expected to be — acquired
from each surgical subject matter expert using Cognitive Task Analysis (CTA)
knowledge elicitation techniques. In addition, while this study indicates that four
experts need to be interviewed, further studies will continue to investigate the 'how
many experts' question as it applies to medical education. In those future studies,
the methods that will be used will need to measure not only total knowledge
acquired but also subsets of knowledge contained within a particular medical
procedure to ensure that the calculation of the point of diminishing marginal utility
is more robust.

79

REFERENCES

Anderson, J.R. (1982). Acquisition of cognitive skill. Psychological Review
89(4), 369-406.
Anderson, J.R. (1996). ACT: A simple theory of complex cognition. American
Psychologist 51(4), 355-365.
Anderson, J.R. & Fincham, J.M. (1994). Acquisition of procedural skills from
examples. Journal of Experimental Psychology: Learning, Memory, and
Cognition 20(6), 1322-1340.
Arnett, J. (2000). Theoretical and pragmatic influences on task analysis methods.
In J.M. Schraagen, S.F. Chipman, & V.L. Shalin (Eds.), Cognitive Task
Analysis (pp. 25-37). Mahwah, NJ: Lawrence Erlbaum Associates.
Bainbridge, L. (1979). Verbal reports as evidence of the process operator's
knowledge. International Journal of Man-Machine Studies 11, 411-436.
Bargh, J.A. & Chartrand, T.L. (1999). The unbearable automaticity of being.
American Psychologist 54(7), 462-479.
Bargh, J.A. & Ferguson, M.J. (2000). Beyond behaviorism: On the automaticity
of higher mental processes. Psychological Bulletin 126, 925-945.
Bathalon, Martin, & Dorion. (2004). Cognitive task analysis, kinesiology, and
mental imagery: Challenging surgical attrition. Surgical Education 199(3),
S73.
Beard, D.V., Smith, D.K., & Denelsbeck, K.M. (1996). Quick and dirty GOMS:
A case study of computed tomography interpretation. Human-Computer
Interaction 11, 157-180.
Besnard, D. & Bastien-Toniazzo, M. (1999). Expert error in trouble-shooting: An
exploratory study in electronics. International Journal of Human-Computer
Studies 50, 391-405.
Blaug, M. (1997). Economic Theory in Retrospect (5th ed.). Cambridge:
Cambridge University Press.
Blessing, S.B. & Anderson, J.R. (1996). How people learn to skip steps. Journal
of Experimental Psychology: Learning, Memory, and Cognition 22(3),
576- 598.
80

Bransford, J.D., Brown, A.L., & Cocking, R.R. (2000). Chapter 2: How experts
differ from novices. In J.D. Bransford, A.L. Brown, & R.R. Cocking
(Eds.), How People Learn: Brain, Mind, Experience, and School
(Expanded ed., pp. 31-50). Washington, D.C.: National Academy Press.
Chase, W.G. & Simon, H.A. (1973). Perception in chess. Cognitive Psychology 4,
55-81.
Chao, C. & Salvedney, G. (1994). Percentage of procedural knowledge acquired
as a function of the number of experts from whom knowledge is acquired
for diagnosis, debugging, and interpretation tasks. International Journal of
Human-Computer Interaction 6 (3), 221-233.
Chao, J.C., Salvedney, G., & Lightner, N. (1999). Development of a methodology
for optimizing elicited knowledge. Behaviour & Information Technology
18(6), 413-430.
Chipman, S.F., Schraagen, J.M., & Shalin, V.L. (2000). Introduction to cognitive
task analysis. In J.M. Schraagen, S.F. Chipman, & V.L. Shalin (Eds.),
Cognitive Task Analysis (pp. 3-23). Mahwah, NJ: Lawrence Erlbaum
Associates.
Clark, R. (2008, October 26). Knowledge and Type Tests. Presented at Education
790. University of Southern California, Los Angeles, CA.
Clark, R.E. & Estes, F. (1996). Cognitive task analysis. International Journal of
Educational Research 25(5), 403-417.
Clark, R.E., Feldon, D., Van Merriënboer, J.J.G., Yates, K., & Early, S. (2008).
Cognitive task analysis. In J.M. Spector, M.D. Merrill, J.J.G. van
Merriënboer, & M.P. Driscoll (Eds.), Handbook of Research on
Educational Communications and Technology (3rd ed.). Mahwah, NJ:
Lawrence Erlbaum Associates.
Clark, R.E., Pugh, C.M., Yates, K., Early, S., & Sullivan, M. (2008). The use of
cognitive task analysis and simulators for after action review of medical
events in Iraq. Technical Report 5-21-2008 developed for the Center for
Cognitive Technology, Rossier School of Education, University of Southern
California. Retrieved from http://www.cogtech.usc.edu.

81

Clark, R.E., Yates, K., Early, S. & Moulton, K. (2009). An analysis of the failure
of electronic media and discovery-based learning: Evidence for the
performance benefits of guided training methods. In K.H. Silber &
R.Foshay, (Eds.), Handbook of Training and Improving Workplace
Performance, Volume I: Instructional Design and Training Delivery.
Washington, DC: International Society for Performance Improvement.
Cooke, N.J. (1994). Varieties of knowledge elicitation techniques. International
Journal of Human-Computer Studies 41, 801-849.
Crandall, B. & Gretchell-Leiter, K. (1993). Critical decision method: A technique
for eliciting concrete assessment indicators from the "intuition" of NICU
nurses. Advances in Nursing Science 16(1), 42-51.
DuBois, D & Shalin, V.L. (2000). Describing job expertise using cognitively
oriented task analyses (COTA). In J.M. Schraagen, S.F. Chipman, & V.L.
Shalin (Eds.), Cognitive Task Analysis (pp. 41-55). Mahwah, NJ:
Lawrence Erlbaum Associates.
Elstein, A.S. & Schwarz, A. (2002). Evidence base of clinical diagnosis: Clinical
problem solving and diagnostic decision making: Selective review of the
cognitive literature [Electronic version]. BMJ: British Medical Journal
324, 729-732.
Ericsson, K.A., Krampe, R.T., & Tesch-Römer, C. (1993). The role of deliberate
practice in the acquisition of expert performance. Psychological Review
100(3), 363-406.
Ericsson, K.A. (2004). Deliberate practice and the acquisition and maintenance of
expert performance in medicine and related domains. Academic Medicine
79(10), S1-S12.
Expert Knowledge Solutions. (2009). Cogntitive Task Analysis Training Aid.
Feldon, D.F. & Clark, R.E. (2006). Instructional implications of cognitive task
analysis as a method for improving the accuracy of experts‘ self-report. In
G. Clarebout & J. Elen (Eds.), Avoiding Simplicity, Confronting
Complexity: Advances in Studying and Designing (computer-based)
Powerful Learning Environments. Rotterdam, The Netherlands: Sense
Publishers, pp. 109-116.
Flach, J.M. (2000). Discovering situated meaning: an ecological approach to task
analysis. In J.M. Schraagen, S.F. Chipman, & V.L. Shalin (Eds.), Cognitive
Task Analysis (pp. 87-100). Mahwah, NJ: Lawrence Erlbaum Associates.
82

Guba, E.G. (1978). Toward a methodology of naturalistic inquiry in educational
evaluation. CSE Monograph Series in Evaluation, No. 8. Los Angeles:
Center for the Study of Evaluation, University of California, Los Angeles
Grantcharov, T.P. & Reznick, R.K. (2008). Teaching rounds: Teaching
procedural skills [Electronic Version]. BMJ: British Medical Journal 336,
1129-1131.
Grunwald, T., Clark, D., Fisher, S.S., McLaughlin, M., Narayanan, S., & Piepol, D.
(2004). Using cognitive task analysis to facilitate collaboration in
development of simulator to accelerate surgical training. Proceedings of the
12th Medicine Meets Virtual Reality Conference, 114–120.
Hall, J.C. (2004). One surgeon's philosophy of surgical education. The American
Journal of Surgery 187, 486-490.
Halsted, W.S. (1904). The training of the surgeon. Bulletin of the Johns Hopkins
Hospital 15(162), 267-275.
Hoffman, R.R., Shadbolt, N.R., Burton, A.M., & Klein, G. (1995). Eliciting
knowledge from experts: A methodological analysis. Organizational
Behavior and Human Decision Process 62(2), 129-158.
Hoffman, R., Crandall, B. & Shadbolt, N. (1998). Use of the critical decision
method to elicit expert knowledge: A case study in the methodology of
cognitive task analysis. Human Factors 40(2), 254-276.
Holmes, J.F., Panacek, E.A., Sakles, J.C., & Brofeldt, B.T. (1998). Comparison of
2 cricothyrotomy techniques: Standard method versus rapid 4-step
technique. Annals of Emergency Medicine 32(4), 442-446.
Johnson, S., Healey, A., Evans, J., Murphy, M., Crenshaw, M., & Gould, D.
(2006). Physical and cognitive task analysis in interventional radiology.
Clinical Radiology 61, 97-103.
Kim, S. & Hasher, L. (2005). The attraction effect in decision making: Superior
performance by older adults. Quarterly Journal of Experimental
Psychology 58A, 120-133.
Lesgold, A. (2000). On the future of cognitive task analysis. In J.M. Schraagen,
S.F. Chipman, & V.L. Shalin (Eds.), Cognitive Task Analysis (pp. 451-465).
Mahwah, NJ: Lawrence Erlbaum Associates.
Lucas, C.J. (2006). American Higher Education: A History (2nd ed.). New York:
Palgrave Macmillan.
83

Luker, K.R., Sullivan, M.E., Peyre, S.E., Sherman, R., & Grunwald, T. (2008).
The use of a cognitive task analysis-based multimedia program to teach
surgical decision making skills in flexor tendon repair. The American
Journal of Surgery 195, 11-15.
Means, B. & Gott, S. (1988). Cognitive task analysis as a basis for tutor
development: articulating abstract knowledge representations. In J. Psotka,
L.D. Massey, & S.A. Mutter (Eds.), Intelligent Tutoring Systems: Lessons
Learned (pp. 35-58). Hillsdale, NJ: Lawrence Erlbaum.
Norman, G. (2002). Research in medical education: three decades of progress
[Electronic version]. BMJ: British Medical Journal 324, 1560-1562.
Ormerod, T.C. (2000). Using task analysis as a primary design method: The SGT
approach. In J.M. Schraagen, S.F. Chipman, & V.L. Shalin (Eds.),
Cognitive Task Analysis (pp. 165-180). Mahwah, NJ: Lawrence Erlbaum
Associates.
Paris, S.G., Lipson, M.Y., & Wixson. (1983). Becoming a strategic learner.
Contemporary Educational Psychology 8, 293-316.
Patton, M.Q. (2002). Qualitative Research & Evaluation Methods (3rd ed.).
Thousand Oaks, CA: Sage Publications.
Potter, S.S., Roth, E.M., Woods, D.D., & Elm, W.C. (2000). Bootstrapping
multiple converging cognitive task analysis techniques for system design.
In J.M. Schraagen, S.F. Chipman, & V.L. Shalin (Eds.), Cognitive Task
Analysis (pp. 317-340). Mahwah, NJ: Lawrence.
Reznick, R.K. & MacRae, H. (2006). Teaching surgical skills — Changes in the
wind. The New England Journal of Medicine 355 (25), 2664-2669.
Rima, I.H. (2000). Development of Economic Analysis (6th ed). London:
Routledge.
Ryder, J.M. & Redding, R.E. (1993). Integrating cognitive task analysis into
instructional design systems development. Educational Technology
Research & Development 41(2), 75-96.
Sachdeva, A.K., Bell, R.H., Britt, L.D. Tarpley, J.L., Blair, P.G., & Tarpley, M.J.
(2007). National efforts to reform residency education in surgery.
Academic Medicine 82(12), 1200-1210.

84

Schaafstal, A. & Schraagen, J.M. (2000). Training of troubleshooting: a
structured, task analytical approach. In J.M. Schraagen, S.F. Chipman, &
V.L. Shalin (Eds.), Cognitive Task Analysis (pp. 57-70). Mahwah, NJ:
Lawrence Erlbaum Associates.
Schaafstal, A., Schraagen, J.M., & van Berlo, M. (2000). Cognitive task analysis
and innovation of training: The case of structured troubleshooting. Human
Factors 42(1), 75-86.
Shackak, Hadas-Dayagi, Ziv, & Reis. (2009). Primary care physicians‘ use of an
electronic medical record system: A cognitive task analysis. Journal of
General Internal Medicine 24(3), 341-348.
Schmidt, H.G., Norman, G.R., & Boshuizen, H.P.A. (1990). A cognitive
perspective on medical expertise: Theory and applications. Academic
Medicine 65(10), 611-621.
Scott, D.J., Cendan, J.C., Pugh, C.M., Mintor, R.M., Dunnington, G.L. & Kozar,
R.A. (2008). The changing face of surgical education: Simulation as the
new paradigm. Journal of Surgical Research 147, 189-193.
Seamster, T.L., Redding, R.E., & Kaempf, G.L. (2000). A skill-based cognitive
task analysis framework. In J.M. Schraagen, S.F. Chipman, & V.L. Shalin
(Eds.), Cogntive Task Analysis (pp. 135-146). Mahwah, NJ: Lawrence
Erlbaum Associates.
Shute, V.J., Torreano, L.A., & Willis, R.E. (2000). DNA: Providing the blueprint
for instruction. In J.M. Schraagen, S.F. Chipman, & V.L. Shalin (Eds.),
Cogntive Task Analysis (pp. 71-86). Mahwah, NJ: Lawrence Erlbaum
Associates.
Sullivan, M.E., Brown, C.V., Peyre, S.E., Salim, A., Martin, M., Towfigh, S., &
Grunwald, T. (2007). The use of cogntive task analysis to improve the
learning of precutaneous tracheostomy placement. The Journal of
American Surgery 193, 96-99.
Sullivan, M.E., Ortega, A., Wasserberg, N., Kauffman, H., Nyquist, J., & Clark, R.
(2008). Assessing the teaching of procedural skills: Can cognitive task
analysis add to our traditional teaching methods? The American Journal of
Surgery 195, 20-23.
Thorndike, E.L. (1921). Measurement in education. Teachers College Record
22(5), 371-379.
85

Velmahos, G.C., Konstantinos, G.T., Sillin, L.F., Chan, L., Clark, R.E., Demetrios,
T., & Maupin, F. (2004). Cognitive task analysis for teaching technical
skills in an inanimate surgical skills laboratory. The American Journal of
Surgery (187), 114-119.
Williams, K.E. (2000). An automated aid for modeling human-computer
interaction. In J.M. Schraagen, S.F. Chipman, & V.L. Shalin (Eds.),
Cogntive Task Analysis (pp. 165-180). Mahwah, NJ: Lawrence Erlbaum
Associates.
Willingham, D.T. (2009). Why Students Don’t Like School: A Cognitive Scientist
Answers Questions About How the Mind Works and What It Means for the
Classroom. San Francisco: Jossey-Bass.

Abstract (if available)

Abstract Residents in surgical residency programs are taught through a hands-on apprenticeship under the supervision of surgical subject matter experts despite the fact that those experts are largely unaware of the automated strategies that guide most of their problem-solving. In fact, experts often omit as much as 70% of the procedural knowledge that novices need to learn. This study examines the amount of procedural steps in an open cricothyrotomy procedure that can be learned from multiple surgical subject matter expert interviews using Cognitive Task Analysis (CTA) techniques. CTA focuses on measuring the mental models used in task performance, capturing not only declarative knowledge but also procedural knowledge.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

The use of cognitive task analysis for identifying the critical information omitted when experts describe surgical procedures

PDF

The effect of cognitive task analysis based instruction on surgical skills expertise and performance

PDF

Using cognitive task analysis to determine the percentage of critical information that experts omit when describing a surgical procedure

PDF

Employing cognitive task analysis supported instruction to increase medical student and surgical resident performance and self-efficacy

PDF

The use of cognitive task analysis to capture exterptise for tracheal extubation training in anesthesiology

PDF

Cognitive task analysis for instruction in single-injection ultrasound-guided regional anesthesia

PDF

Towards a taxonomy of cognitive task analysis methods: a search for cognition and task analysis interactions

PDF

The use of cognitive task analysis to determine surgical expert's awareness of critical decisions required for a surgical procedure

PDF

Using cognitive task analysis for capturing expert instruction of food safety training for novice employees

PDF

The use of cognitive task analysis to investigate how many experts must be interviewed to acquire the critical information needed to perform a central venous catheter placement

PDF

The use of cognitive task analysis to capture expert patient care handoff to the post anesthesia care unit

PDF

The use of cognitive task analysis to capture expert instruction in teaching mathematics

PDF

The use of cognitive task analysis for the postanesthesia patient care handoff in the intensive care unit

$Using cognitive task analysis to capture expert instruction in division of fractions$

PDF

Using cognitive task analysis to capture expert instruction in division of fractions

PDF

Using cognitive task analysis to capture expert reading instruction in informational text for students with mild to moderate learning disabilities

PDF

Using incremental cognitive task analysis to capture expert instruction in expository writing for secondary students

PDF

Using cognitive task analysis to capture how expert anesthesia providers conduct an intraoperative patient care handoff

PDF

Using cognitive task analysis to capture how expert principals conduct informal classroom walk-throughs and provide feedback to teachers

PDF

Using individual cognitive task analysis to capture expert writing instruction in expository writing for secondary students

PDF

Using cognitive task analysis to capture palliative care physicians' expertise in in-patient shared decision making

Asset Metadata

Creator Crispen, Patrick Douglas (author)

Core Title Identifying the point of diminishing marginal utility for cognitive task analysis surgical subject matter expert interviews

School Rossier School of Education

Degree Doctor of Education

Degree Program Education (Leadership)

Publication Date 03/23/2010

Defense Date 02/08/2010

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag cognitive task analysis,cricothyrotomy,CTA,Medical education,OAI-PMH Harvest,Surgery

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Clark, Richard E. (committee chair), Sullivan, Maura E. (committee member), Yates, Kenneth A. (committee member)

Creator Email crispen@gmail.com,crispen@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-m2873

Unique identifier UC1289602

Identifier etd-Crispen-3508 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-291454 (legacy record id),usctheses-m2873 (legacy record id)

Legacy Identifier etd-Crispen-3508.pdf

Dmrecord 291454

Document Type Dissertation

Rights Crispen, Patrick Douglas

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Repository Name Libraries, University of Southern California

Repository Location Los Angeles, California

Repository Email cisadmin@lib.usc.edu