University of Southern California Dissertations and Theses

The use of cognitive task analysis to investigate how many experts must be interviewed to acquire the critical information needed to perform a central venous catheter placement

(USC Thesis Other)

The use of cognitive task analysis to investigate how many experts must be interviewed to acquire the critical information needed to perform a central venous catheter placement

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content THE USE OF COGNITIVE TASK ANALYSIS TO INVESTIGATE HOW MANY
EXPERTS MUST BE INTERVIEWED TO ACQUIRE THE CRITICAL
INFORMATION NEEDED TO PERFORM A CENTRAL VENOUS CATHETHER
PLACEMENT

by

Craig W. Bartholio

A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION

December 2010

Copyright 2010 Craig W. Bartholio

ii
Dedication
There are many people that help support me during the entire three years of
completing this doctoral program and dissertation. I want to first thank my parents
for believing in me and supporting the whole process through multiple means. They
have always been a huge believer in my abilities even when I had my own doubts.
Secondly, I would like to thank my extended family for all their verbal
encouragement, baby sitting, inspiring phone conversations, editing, and all around
understanding of my multiple priorities and commitments.
To my three children, Chloe, Noah, and Hannah. Each of you has spent
several nights sleeping next to Daddy while he typed late into the night, and
sometimes into the next morning. I want to thank you for understanding all the times
I could not “play” with you because Daddy had to “study” or “write”. I have a lot of
playing activities to catch up on with each of you.
Lastly, I want to thank my wife for all her continued support. Without her
encouragement, frank realism, and editing abilities (far superior to mine), and the
ability to always understand the real perspective of importance in life, I share this
accomplishment with you.

iii
ACKNOWLEDGEMENTS
I would like to take this time to express my deepest gratitude, appreciation,
admiration, and thankfulness to the faculty on my dissertation committee: Dr.
Richard Clark, Dr. Kenneth Yates, and Dr. Maura Sullivan without whom I would
not have been able to accomplish this huge undertaking.
To Dr. Richard Clark, I am in continual awe of your knowledge and ability to
express the complex in simple understandable terms. I want to offer sincere
gratitude for all of the time, patience, and sharing of his knowledge that allowed me
to accomplish this feat. I am thirsty for knowledge in the area of cognitive task
analysis and look forward to continuing to add to the current body of knowledge in
this field.
To Dr. Kenneth Yates, first of all thank you for inspiring me to switch from
the K-12 Leadership concentration to the Educational Psychology concentration. It
has been a wonderful and thrilling ride. I appreciate all the significant amount of
time effort you have provided me in achieving this accomplishment. You have
provided many wonderful insights into what it takes to write a dissertation. I am
deeply indebted to you.
To Dr. Maura Sullivan, without you initial support and guidance, I’m not sure
I would be writing this acknowledgement. Thank you for your patience and
understanding of this dissertation process. The insights on organizing my writing,

iv
continual support and the sharing of your significant other to the good of the cause,
thank you.
Finally, to all my fellow cohort members Eko, Joon, Julia, Mary Ann, and
Lesile, and especially Patrick, thanks for your wonderful continual support.

v
TABLE OF CONTENTS
Dedication ii
Acknowledgements iii
List of Tables vii
List of Figures viii
Abstract x
Chapter 1 1
Statement of the Problem 1
Review of the Literature 5
Current Trends in Surgical Training 5

Development of Expertise 7
Expertise in Medicine 8
Expertise in Surgery 9

Knowledge Types 10
Declarative Knowledge 11
Procedural Knowledge 13
Automaticity 14
Automaticity and Expert Recall 16
Automaticity and Expert Recall in Medicine and
Surgery 18

Cognitive Task Analysis 20
Defining Cognitive Task Analysis 20
CTA Methodology 21

Effectiveness Studies using CTA 23
Effectiveness of CTA in Surgical Training 29
Limitations of CTA 35
Number of Experts Required for CTA 37
Summary 39

Purpose of the Study 40

vi

Chapter 2: Method 42
Design 42
Subjects 43
Data Collection 44
Semi-structured CTA Interviews 44
CTA Coding Scheme and Procedure 45
CTA Protocol and creating a six-Subject Matter “Gold Standard”
CVC Protocol 46
Data Analysis 47

Chapter 3: Results 52
Coding and Inter-rater reliability 52
Summary 74

Chapter 4: Conclusion s 76
Research Questions 76
Summary 84
Limitations and Implications 85
Conclusion 87

References 89

Appendices
Appendix A: Non-Repeating Combinations of SME Protocols 98
Appendix B: CTA Coding Scheme 99
Appendix C: CTA Gold Standard for Central Venous Catheter 100

vii

LIST OF TABLES
Table 1: Excerpt from Central Venous Catheter Protocol Spreadsheet,
Single Expert 48

Table 2: Excerpt from Central Venous Catheter Placement Protocol
Spreadsheet: Multiple Experts 50

Table 3: CVC Gold Standard Sections and Corresponding Number
Of Items per Section 53

Table 4: Percentage of Knowledge Acquired from an Expert When
Compared to the Six-Expert CVC Gold Standard Protocol
(Experts A-F) 54

Table 5: The average percent of Total Knowledge, Action Steps,
Decision Steps per Number of Experts when compared to a
Six-SME Gold Standard Protocol 56

Table 6: Average Percentage Increase of Total Knowledge, Action Steps,
And Decision Steps Acquired from Multiple Groups of Expert
When compared to a Six-expert Gold Standard CVC Protocol 57

Table 7: Average Percentage of Objectives, Reasons, and Risks Acquired
From Multiple Groups of Experts When Compared to a Six
Subject Matter Expert Gold Standard Protocol 63

Table 8: Average Percentage of Indications, Contraindications, and
Standards Acquired from Multiple Groups of Experts When
Compared to a Six Subject Matter Expert Gold Standard Protocol 67

Table 9: Average Percentage of Equipment and Tasks acquired from
Multiple Groups of Experts when compared to a Six Subject
Matter Expert Gold Standard Protocol 72

Table 10: Quantity of Experts Recommended if a 10% Marginal
Utility in Knowledge Acquisition is Expected 75

viii
LIST OF FIGURES
Figure 1: Percentage of Acquired Knowledge Based on Six-Expert
Gold Standard CTA CVC Procedure Protocol as a Function
Of the Number of Experts 58

Figure 2: Average Percentage Increase of Knowledge as a Function
Of an Additional Expert for Total Possible Items in the
Gold Standard CVC Procedure Protocol 59

Figure 3: Average Percentage Increase in Action Step Acquired
Knowledge as a Function of the Number of Experts 60

Figure 4: Average Percentage Increase of Action Steps as a
Function of Additional Experts 60

Figure 5: Average Percentage Increase of Gold Standard
Decision Steps as a Function of the Number of Experts 61

Figure 6: Average Percentage Increase of Decision Steps as a
Function of Additional Experts 62

Figure 7: Percentage of Knowledge Acquired of Risks as a
Function of the Number of Experts 63

Figure 8: Average Increase of Acquired Knowledge of Risks as a
Function of Added Experts 65

Figure 9: Percentage of Knowledge Acquired for Reasons to Perform
The CVC Procedure Correctly 65

Figure 10: Average Percentage Increase in Acquired Knowledge in the
Area of Reasons Based on a Six-Expert Gold Standard
CVC protocol 66

Figure 11: Percentage of Indications to Perform the CVC Procedure
Based on a Six-Expert Gold Standard as a Function of the
Number of Experts 67

ix
Figure 12: The Average Percentage of Acquired Knowledge Gained as a
Function of Additional Experts in the Area of Indications to
Perform a CVC Procedure 68

Figure 13: Percentage of Knowledge Acquired in the Area of
Contraindications as a Function of the Number of Experts 69

Figure 14: The Average Percentage Increase in the Amount of
Acquired Knowledge in the Area of Contraindications
As a Function of the Number of Experts 70

Figure 15: Percentage of knowledge acquired for standards of CVC
Procedure as a function of the number of experts 71

Figure 16: Average Percentage Increase of Acquired Knowledge for
Standards Based on a Six-Expert Gold Standard CVC Protocol 71

Figure 17: The Average Percentage Gain of Acquired Knowledge of
Needed Equipment as a Function of Additional Experts 73

Figure 18: The Average Percentage Gain of Acquired Knowledge of
Needed Equipment as a Function of Additional Experts 74

x
ABSTRACT
The purpose of this study was to examine the amount of relevant information
experts provide and fail to provide when asked to describe how to perform a
complex task in enough detail for students to perform the task. In this study,
medical experts where interviewed because their past successes and failures at
the task are known and so it could be determined that all experts had succeeded
consistently at the task being described. Past research has suggested that because
experts have both conscious and unconscious automated knowledge they may not
“know what they know” and so not be able to completely describe how to make
critical decisions during task performance. A version of Cognitive Task Analysis
designed to support training was used to interview medical school faculty and
analyze their description of a controversial trauma procedure in order to
determine the average percentage of knowledge that was acquired from a single
expert and how much additional knowledge is acquired from each succeeding
expert interviewed. After analysis, it was determined that the amount of
knowledge acquired from one expert was about 63 percent and the average
increase of acquired knowledge with the second expert was about 16% and the
third expert added another 8 percent. Past studies have reported considerably
lower percentages of relevant information captured with Cognitive Task Analysis
– about 30 percent of decisions are typically reported for example. Thus it was

xi
hypothesized that information about the controversial procedure examined in
this study may have become more conscious to the experts interviewed because
they had been discussing it among themselves and reading accounts in journals.

1
CHAPTER 1

STATEMENT OF THE PROBLEM

Surgical skills training has evolved in recent years from the traditional “see one-
do one-teach one” model (Halsted, 1904) to methods that focus more on instruction
by expert surgeons and various forms of technology, such as computer based
learning; virtual reality (VR); and high-fidelity bench models (Vozenilek, Huff,
Reznek, & Gordon, 2004). These technologies increase surgical residents exposure to
a wide array of procedures while practicing their proficiency of technical skills prior
to real surgical exposure (Aggarwal, Grantcharov, & Darzi, 2007; Reznick &
MacRae, 2006;Vozenilek et al., 2004).
The common thread for both the traditional apprenticeship and current surgical
skills training programs is that expert surgeons are often relied upon as the “master
teacher” to inform and the curriculum content, training procedures, and simulation
software. However, research indicates that experts can omit up to 70% of
information when explaining to others how to perform a task (Clark, Pugh, Yates,
Early, & Sullivan, 2008). Additionally, the transfer of knowledge from experts to
novices is not easily accomplished (Hinds, Patterson, & Pfeffer, 2001). This may
have serious consequences for surgical training in which surgical experts may not
provide a full account of the knowledge and skills required to perform a procedure
(Abernethy, Poolton, Masters, and Patil, 2008). Thus, surgical residents may be
provided incomplete knowledge about the behavioral tasks of a procedure, but, more

2
important, the decision-making and cognitive processes that are inherently involved
with performing these procedures (Jacklin, Sevdalis, Darzi, & Vincent, 2008).
Cognitive task analysis is a generic reference to a variety of methods that have
been demonstrated to effectively capture the knowledge and skills experts use to
perform complex tasks (Crandall, Klein, & Hoffman, 2006; Hoffman & Militello,
2009; Clark, Feldon, van Merrienboer, Yates, & Early, 2008). There are many
definitions of cognitive task analysis (CTA). For example, Crandall, Klein, and
Hoffman (2006) define CTA as a “family of methods used for studying and
describing reasoning and knowledge [including] the activities of perceiving and
attending that underlie performance of tasks [and] the cognitive skills and strategies
needed to respond adeptly to complex situations” (p. 3). Whereas, Clark et al. (2008)
define CTA as the usage of “interviews and observation strategies to capture a
description of knowledge that experts use to perform complex tasks.” The common
thread among these definitions of CTA is that the goal of CTA is to capture the
underlying knowledge and skills experts use to solve complex tasks.
Information captured from experts through CTA has been shown to be effective
for training novice physicians (Velmahos, Toutouzas, Silin, Chan, Clark, Theodorou,
& Maupin, 2004; Sullivan, Brown, Peyre, Salim, Martin, Toowfugh, & Grunwald,
2007; and Luker, Sullivan, Peyre, Sherman, & Grunwald, 2008). For example,
Velmahos et al. (2004) found that surgical residents who received CTA based
instructions on how to perform a CVC procedure performed better on a 14 point
procedure checklist when compared to a control group taught through traditional

3
methods. Additionally, Sullivan et al. (2007) found that general surgery residents
performed significantly better on post -instruction technical assessments at both a
one-month and six-month benchmarks when instructed on how to perform a
percutaneous tracheostomy (PT) utilizing a CTA based curriculum. In another study,
Luker et al. (2008) found that although technical skills are developed through task
exposure, the insertion of a CTA based multimedia instructional program
significantly increased the knowledge and skills of the surgical residents completing
flexor tendon repair procedure. And, in a recent study, Tirapelle (2010) found that
randomly assigned surgical residents who were provided CTA-based instruction
improved in their knowledge and skills on how to conduct an Open Cricothyrotomy
when compared to a control group taught with traditional methods. These studies
emphasize the positive impact a CTA based curriculum has on surgical residents’
knowledge and skills in performing complex procedures.
It is often recommended in the CTA literature that multiple experts be consulted
to increase the validity and reliability of CTA results (Yates, 2007; Yates & Feldon,
in press). However, many researchers note that the recommendation to use multiple
experts creates a “bottleneck” due to the additional time and resources required
during the CTA process (Hoffman, Crandall, and Shadbolt, 1998; Hoffman, Shadbolt,
Burton, & Klein, 1995). As such, a question often asked is how many experts are
required to capture the optimal knowledge needed to perform a complex task? There
is a paucity of research that has studied this problem. An exception is Chao and
Salvendy (1994), who examined different techniques of acquiring knowledge to

4
diagnose computer software programs. While Chao and Salvendy (1994) found that
a single expert provided an average of 27% to 40%, depending on the software-
debugging task, the percentage of knowledge and skills increased as they interviewed
additional subject matter experts up to a total of six individuals. Chao & Salvendy
(1994) concluded three experts were needed to acquire the optimum critical
knowledge and skills needed to solve a complex task.
To date, there have been no published studies within the field of surgical training
that examine the number of surgical experts required to capture the optimum
knowledge and skills required to perform complex procedures. A study conducted
concurrently with this study sought to examine this question for the cricothyrotomy
procedure and found that 3 experts were required to capture the actions and decision
steps necessary to perform the procedure (Crispen, 2010). The current investigation
seeks to replicate the methodology used by Crispen (2010) to determine the optimal
number of experts required for the procedure to place a central venous catheter.
As such, this study poses two research questions: 1) How much information about
a central venous catheter placement procedure does a single expert provide when
compared to the combined contributions of a comprehensive “gold standard” protocol
based on interviews with six physician subject matter experts; 2) How much critical
information is gained from each additional interview about a central venous catheter
placement?

5
REVIEW OF THE LITERATURE
To understand the contributions of cognitive task analysis and its potential benefit
to medical education, we must first review the current trends in surgical training and
the inherent complexities of relying on expert physicians as instructors/mentors for
training surgical residents. A review of both declarative and procedural knowledge
types is critical to understanding how experts construct knowledge and sometimes
have difficulty communicating acquired knowledge. Often, subject matter experts
(SMEs) omit critical information when describing how to perform a task thus.
Research shows that surgeons can omit up to 70% of the critical information needed
to perform a procedure (Clark, Pugh, Yates, Sullivan, 2008). Thus, capturing this
information is a critical component to helping train surgical novices. Cognitive task
analysis provides a method to capture this omitted critical information. Therefore, a
review of the use of CTA and it’s reported effectiveness both within and outside the
medical field is presented. The review of the literature section will conclude with a
summary of recent studies that examined how many experts are required to capture
the critical knowledge needed to perform a complex task.
Current Trends in Surgical Training
Surgical training has experienced a paradigm shift over the last several decades
from the Halstedain-based apprenticeship model to the development of surgical skills
training facilities (Hamdorf and Hall, 2000). This shift is due to advancements in
technology and the need to expose surgical residents to multiple procedures to
increase the knowledge and skills of surgical residents outside the operating room

6
(Scott, Cendan, Pugh, Minter, Dunnington, & Kozar, 2008). As a result, university
medical education facilities are modifying their programs to utilize discrete,
procedure specific trainings, in an effort to provide the needed skills training for
proficient physicians (Sachdeva, Bell, Britt, Tarpley, Blair, & Tarpley, 2007). A
surgeon’s developing knowledge and skills in completing a variety of medical
procedures are determined by the frequency of practice. Surgical skills training
centers provide an opportunity for surgical residents to develop their skills to a level
of automaticity outside the operative room. The use of virtual reality and other human
performance simulators can help develop the knowledge and skills of surgical
residents (Reznick and MacRae, 2006).
Evidence of increased surgical knowledge and skills has been validated through
studies in short-term, specific procedure concentrated classes. Surgical interns who
were trained using a human patient stimulator (HPS) saw an increase the participant’s
trauma management skills and overall self-confidence in their abilities (Marshall,
Smith, Gorman, Krummel, Haluck, & Cooney, 2001). Additionally, third-year
residents indicated through self-reports that their knowledge and skills generally
increased as a result of participating in a small group training sessions covering four
surgical procedures [foley catheter placement; nasogastric tube insertion/removal;
I.V. placement; and arterial stick] (Meyers, Meyer, Stewart, Dreesen, Barrick, Lange,
& Farrell, In press). Grantcharov and Reznick (2008) indicate two common elements
within teaching procedural technical skills for surgery: pre-patient training and
training in a clinical situation. While the pre-patient focuses theory and simulation

7
curriculum, clinical situation training incorporates an expert instructor for
demonstration, instruction, and evaluation purposes.
The common element of all medical education is the reliance on surgical experts.
Experts are often characterized by the coordination of their highly organized
knowledge structure and their advanced technical skills within a specific domain
(Ericsson, Charness, Feltovich, & Hoffman, 2006). However, Sleeman and Brown
(1982) indicate that much of an expert’s knowledge is tacit and not readily available
to the expert during retrieval tasks. Therefore, relying on experts for educating
surgical residents is a major limitation to the current training methods. A brief review
of the characteristics of experts, experts in medicine, and the affect of experts on
medical training will be reviewed to establish an understanding of the complexity
involved with experts in the field of medicine and the underlying problems with the
expert’s ability to transfer their knowledge to novices.
Development of Expertise
Although expertise can be defined in many ways (Feldon, 2007), it is commonly
agreed upon that expertise is defined as the possession of a large body of knowledge
and procedural skills within a specific domain (Chi, Glaser, & Rees, 1982; Ericsson et
al., 2006; Feldon, 2007). Expertise includes the development of automaticity through
deliberate practice (Ericsson et al., 2006) that reduces the cognitive demand in
performing procedural tasks. Cognitive demand is lowered through automaticity by
reducing the number of decisions that require conscious thought. Therefore,
automaticity provides the benefit of processing speed in problem solving, but inhibits

8
conscious monitoring and modification of skills during task performance. The
inability to adapt routine behavior could lead to lower performance levels when faced
with atypical tasks. Feldon (2007) discussed the need for experts to have a developed
level of adaptivity when faced with atypical tasks (adaptive expertise). Adaptivity
provides the ability to adapt to novel task constraints while consistently demonstrate
optimal performance. In contrast, routine experts fail to maintain high levels of
performance when faced with changing task conditions. As a result of their high
performance level and cognitive efficiency in problem solving abilities, experts are
often called upon to provide information for training and expert systems, and are
revered socially as an authority figure within a specific field.
Expertise in Medicine
According to Ericsson et al. (2006) a significant portion of the research literature
on medical expertise focuses on diagnosis on medical problems. Boshuizen and
Schmidt (1992) investigated the different use of biomedical and clinical knowledge
by novice, intermediate, and expert-level participants in a study in clinical reasoning.
They defined biomedical knowledge as understanding the principle processes
underlying the manifestation of a disease (e,g, virus or bacteria, organ or organ
systems) and clinical knowledge as knowledge of the attributes in which diseases can
manifest themselves, the related overt and covert signs of the disease and overall case
management heuristics. Boshuizen and Schmidt (1992) found that with increased
levels of expertise, individuals utilized less biomedical knowledge and more clinical
knowledge while sequencing a clinical diagnosis. The increase in clinical knowledge

9
is due to an experts developed “illness scripts” which provide the ability of experts
to mediate the current presented task based on prior actual worked cases.
In a study to analyze how subject matter experts use personal causal models for
diagnostic purposes, Patel and Groen (1986) provided a written medical case to seven
cardiologist specialists. After a brief review of the case, the participants were asked
to provide a written account of the case and also provide an underlying
pathopysiology of the case including a diagnosis. Patel and Groen (1986) found that
the experts who provided an accurate diagnosis utilized forward reasoning in their
causal models, while participants who provided inaccurate diagnosis utilized
hypothetical testing, including backwards reasoning. Similarly, in their research on
expertise and problem solving, Chi et al. (1982) noted that one of the qualitative
differences between experts and novices is their approach to a presented task. While
novices solve problems while working backwards with hypothesis testing heuristics,
experts tend to perform an initial qualitative analysis of the problem and utilize
forward reasoning strategies mediating the current task with similarly stored
representations based on prior experience.
Expertise in Surgery
Acquiring surgical skills requires the development of cognitive and psychomotor
abilities (Hamdorf & Hall, 2000; Peters, Fired, Swanstrom, Soper, Sillin, Schirmer et
al., 2003). According to Hamdorf and Hall (2000), motor dexterity is developed
through three phases: cognition, integration, and automation. The cognition phase
consists of understanding the task. The integration stage incorporates the motor

10
movements specific to the procedure. Lastly, automation infers that the movements
needed for a procedure have become non-conscious and do not rely on continuous
external stimuli. As individuals develop surgical motor and cognitive skills it is
imperative to minimize error development (Porte, Xeroulis, Reznick, & Dubrowski,
2007; Abernethy et al., 2008).
In summary, the attainment of expertise requires the acquisition of extensive
conceptual knowledge and automated procedural skills that allows experts to solve
problems within minimal cognitive demand (Feldon, 2007). To obtain the level of
expertise, in any domain, takes about ten years of deliberate practice with continuous
feedback (Ericsson, 1996; Pavlik and Anderson, 2008). Therefore, to acquire the level
of expertise in medicine and surgery, in particular, requires extensive practice. Two
underlying themes that distinguish experts from non-experts are their knowledge base
and their rapid cognitive processing abilities (Chi et al., 1982).
Due to the automaticity of an expert’s knowledge and skills, conscious monitoring
of thought processes is difficult, therefore an expert’s description of problem solving
is typically limited to observable actions. An understanding of knowledge types
(declarative and procedural) and their function in developing expertise provides a
further understanding of the requirements for capturing needed information from
experts for training complex tasks, such as surgical procedures.
Knowledge Types
Knowledge is the representation of learned information that can be expressed in a
variety of settings for different purposes, actions, or means (Markus, 2001; Spender

11
1996; de Jong & Ferguson-Hessler, 1996; & Anderson, Krathwol, Airasian,
Cruikshank, Mayer, Pintirich, Raths, & Wittrock, 2001). Research indicates that
individuals have a limited capacity in the ability to consciously process information
and there is evidence that people have the ability to process four plus or minus one
bits of information within their working memory (Cowan, 2001). Working memory
is the theorized space that individuals “pay attention to” and encode new information
into long-term memory. Long-term memory is considered unlimited in its capacity
and is associated to encoded chunks of information and are accessible based on their
retrieval structures (Ericcson and Kintsch, 1995). In this section, two types of
knowledge – declarative and procedural – will be discussed and the interactions
between the two that comprise expertise
Declarative Knowledge
Cognitive psychologists have defined declarative knowledge in many ways.
Ormrod (2008) describes declarative knowledge as information retrievable from an
individual’s long-term memory into their short-term memory as concepts, principles,
general knowledge, and recollections of life events/ experiences. Schunk (2000)
suggests that declarative knowledge is comprised of facts, actions within a story, and
the organization of a passage, with reference to information obtained from a novel or
piece of literature. Another definition by Schraw (2006), postulates that declarative
knowledge encompasses facts, concepts, and the mediated relationships among
learned concepts that create an integrated holistic conceptualization within a specific
domain of knowledge. Declarative knowledge is often subdivided into two

12
categories, semantic knowledge comprised of organized schemata regarding
individual concepts (e.g., vascular system) and episodic knowledge (time specific
events or episodes in life, e.g. prom, wedding, & birth of a child) (Schraw, 2006).
Declarative knowledge is considered explicit knowledge (which is easily
retrievable) that is developed through encoding information from the environment.
Declarative knowledge is retrievable information that can articulate the answers to
question of what and why with regards to concepts and facts (Paris, Lipson &
Wixson, 1983; Anderson & Schunn, 2000; Hoffman & Militello, 2009). These
encoded chunks are retrievable from an individual’s long-term memory into their
short-term (i.e. working memory) for immediate needed purposes. Retrieved
declarative knowledge guides individuals in completing performance-based problem
solving tasks (Anderson, Bothell, Byrne, Douglas, Lebiere, & Qin, 2004). For
example, a surgeon knows what equipment to use for a procedure and why the
surgery is needed and has the ability to describe what and why
According to Anderson’s (1996) Adaptive Character of Thought (ACT-R) theory
declarative knowledge is developed as one encodes chunks of information from their
environment. Anderson (1996) describes the declarative chunks of knowledge as
“schema-like structures” that are categorically specific with tangible connections to
other related encoded content. Knowledge structures develop through the
accumulation and toning of individual units of information (chunks) that in whole
produce complex cognition. Although, these chunks of knowledge represent factual
information that an individual can express (Anderson & Schunn, 2000), the

13
acquisition of declarative knowledge does not occur in a vacuum, but rather in
conjunction with the acquisition of procedural knowledge.
Procedural Knowledge
Anderson (1996) contrasts declarative and procedural knowledge by describing
declarative knowledge as “chunks” of information, and procedural knowledge as
“production rules” that are based on the encoded chunks of information. Specifically,
Anderson (1996) suggests that production rules represent procedural knowledge of
the conditions (when) and actions (how) to perform a goal-oriented task that are
based on the individual’s declarative knowledge structures. Further, a production rule
can only be activated once a precise environmental state meets the set conditions for
the production rule to be carried out. For example, a surgical procedure consists of
multiple decisions that need to be made based on the presenting conditions within that
surgery. Thus, production rules symbolize the condition and action needed in
response to the attainment of specific goals.
Production rules consist of IF, THEN statements and are context embedded
(Anderson, 1996) In medical procedures, for example, IF there is bright red pulsating
blood in a syringe, THEN the needle must have punctured an artery. Thus, procedural
knowledge is “when and how to” perform an action or “make a decision” while
completing a task or solving a problem. These production rules (procedural
knowledge) are tacit and not easily retrievable to the individual. In surgery,
procedural knowledge consists of the decisions a physician makes on “when and
how” to perform certain actions during the surgery. In contrast, declarative

14
knowledge is the information the individual knows about the surgery and its
components (knowledge of the anatomy and the instruments used for a procedure.)
When they practice, individuals fine-tune their skills and declarative knowledge
gradually evolves into procedural knowledge (Pirolli & Anderson, 1985; Stadler,
1989; Anderson & Fincham, 1994; Anderson, Fincham, & Douglass, 1997;
Anderson, Fincham & Douglass, 1999; Pavlik & Anderson, 2008; and Knowlton &
Moody, 2008.). Ericsson et al. (2006) indicate that procedural knowledge is
considered to be the final state of the learning process: the combination of acquired
declarative knowledge used to solve problems that become automated through
deliberate practice.
In medicine, through deliberate practice, a highly experienced surgeon can
process the cues of a medical procedure and perform the tasks within the procedure to
a level of automaticity. This reduces the attention demands on the surgeon (What is
the next step in this procedure?) and frees up cognitive resources and enabling the
physician to shift attention to other cognitive actions such as directing the supporting
nurses, self-monitoring effectiveness of the actions taken, and the status of the patient
(changing conditions).
Automaticity
Automaticity is a means of restructuring same procedures so that working
memory is largely circumvented, freeing cognitive resources for other cognitive
demanding tasks (Ericsson, Charness, Feltovich, & Hoffman, 2006). Ericsson et al.,
(2006) discuss how automaticity is central to the development of expertise and

15
practice is the means to achieving automaticity. Expertise involves automated basic
strokes. Experts perform an extreme amount of practice. Such experience,
appropriately conducted, can yield effective, major behavior and brain changes.
Through the act of practice (with reflective feedback, self-regulation, & monitoring of
one’s progress) the character of cognitive operations changes in a manner that:
a) Improves the speed of the operations
b) Improves the smoothness of the operations
c) Reduces the cognitive demands of the operations.
Thus releasing cognitive resources (attention) for other functions such as
planning, self-monitoring, situational awareness. Appropriate planning includes the
use of a developed algorithm to problem solve. Self-monitoring focuses on the error
correction procedures conducted within a task. Situational awareness refers to being
observant of the current conditions during a task.
According to Ericsson et al. (2006), two major difficulties exist with the
development of automaticity. First, individual’s performing at this level of cognitive
automaticity are not as attentive to new cues that fall outside their learned procedure
and may overlook the otherwise salient cue, nor are individuals cognizant of all the
discrete steps involved within their actions. For example, while performing a medical
procedure, a surgeon may continue to follow previously learned behaviors when the
present conditions indicate alternative techniques may be more beneficial.
Additionally, when describing their actions multiple steps may be omitted due to their
automated knowledge. Second, once automaticity has been achieved, the ability for

16
individuals to increase their skills diminishes. Once individuals learn a task to the
level of automaticity, they lose the ability to augment their performance without
purposeful actions, both physically and mentally, which Ericsson calls deliberate
practice. Deliberate practice is 1) actively seeking out novel demanding tasks, and 2)
monitoring, through guided introspection, actions and decisions while performing a
task.
In summary, expertise is the interaction of declarative knowledge (the What, the
concepts, process, and principles involved in labeling items in the world) and
procedural knowledge (the how and when in terms of actions and decisions necessary
in completing a task). In terms of surgical residents, their declarative knowledge
consists of the conscious academic preparation and skills they have learned up to an
intermediate level of proficiency. Their procedural knowledge consists of utilizing
their declarative knowledge in completing medical procedures. As individuals
develop expertise, their knowledge becomes highly organized and integrated
structures within their domain. Their skills become highly developed through
deliberate practice, reflection, and corrective feedback.
Automaticity and Expert Recall
Cognitive psychology describes how individuals transform encoded information
through practice to develop heuristics in problem solving. These heuristics are often
automated procedures that are tacit and unconscious to the individual. Polanyi (1962)
initially referred to tacit knowledge as certain cognitive processes in context with
problem-solving behaviors that are inaccessible to the conscious mind. Tacit

17
knowledge refers to the individual’s inability to recall the explicit knowledge used
during a series of events while completing a task (Gourlay, 2004).
Expert instructors are often called up to provide needed information and resources
to novices through curriculum development and teaching. Hinds, Patterson, and
Pferrer (2001) found that due to the cognitive characteristics of experts, that is, a
highly organized, hierarchal system of interconnected abstract concepts, accompanied
by advanced procedural automated knowledge, the transfer of knowledge from
experts to novices has severe limitations. In their first study, Hinds et al. (2001)
investigated the different instructional styles used by beginner and expert instructors
when training novice individuals about how to understand simple electronic concepts
by completing an electronic circuit. Hinds et al. (2001) found that experts utilized
more abstract concepts and statements when compared to beginner instructors who
trained novices with more concrete statements. For instance, only 9% of experts
provided direct information about the requirement of properly connecting the wires to
the spring coil, which is essential for the electrical connection to be completed, while
90% of beginners provided this information. Hinds et al. suggest that experts
educating individuals in this process may omit critical information that novices need
to develop their conceptual knowledge within this domain. According to Hinds
(1999), as experts develop their automaticity, their simplification of facts and
concepts precludes them from recalling specific details and task imbedded
complexities that novices require for problem solving in this domain.

18
Automaticity and expert recall in medicine and surgery
An individual’s automated procedural (tacit) knowledge has systemic
consequences in the medical field. Surgical experts usually train novice physicians
(Vadcard & Luengo, 2004). Research indicates experts have greater perceptual,
cognitive, motor, attention and personal reflective feedback capabilities when
compared with non-experts (Abernethy, Poolton, Masters, & Patil, 2008). Physicians
develop their declarative and procedural knowledge through years of experience and
practice (Cauraugh, Martin, & Martin, 1999). Physicians develop and hone their skills
by conducting multiple procedures and develop ‘rules of thumb’ (Andre, Borgquist,
Foldevi, & Molstad, 2002) which are tacit and unconscious to the individual. As a
result of their continued practice, their skills becomes automated and transition from
being knowledge that is on a conscious level to knowledge that is tacit and illicit
without deliberate probing (Abernethy, Poolton, Masters, & Patil, 2008). Novice
medical students relying on the attending physician for critical information regarding
a procedure may not receive the knowledge required to perform adequately due to the
attending physician’s inability to recall the needed information (Hamdorf & Hall,
2000).
The consequences of expertise and automaticity on expert recall was clearly
establish through the research of Clark, Pugh, Yates, Early, & Sullivan (2008) in their
investigation of capturing the declarative and procedural knowledge from trauma
surgeons on how to perform an emergency femoral artery shunt procedure. Nine
trauma surgeons provided an unaided (no- CTA condition) account of the shunt

19
procedure. When their description of the surgery was compared to a gold standard
femoral shunt surgical protocol, it was determined that they omitted an average of
68.75% of the procedural steps performed in the surgery (Clark et al., 2008). As a
result of their automated knowledge and skills, the trauma surgeons within the study
were not able to recall the full accounts of the procedure. Theoretically, experts may
omit up to 70% of the procedural knowledge involved within a surgical procedure
during the instruction of novices. However, differing results were obtained when one
trauma surgeon was interviewed using CTA-based methodology. When the acquired
knowledge from the one trauma surgeon (CTA condition) was compared to the gold
standard protocol, Clark et al., (2008) calculated only a 30% omission of the
procedural steps. The significant variable in capturing the procedural steps from the
trauma surgeons was the use of an interview based on cognitive task analysis
methodologies, suggesting that cognitive task analysis shows promise as a
methodology to capture the automated, procedural (tacit) knowledge from experts.
The following sections includes a review of how cognitive task analysis is
defined; the appropriate CTA methodologies to capture the intended critical
knowledge and skills sought; the effectiveness of CTA outside medicine and within
surgical training; and the barriers and limitations to using CTA. Finally, the
importance in determining the number of experts and the methods utilized in
determining the optimal number of experts that are the most reliable and valid will be
reviewed.

20
Cognitive Task Analysis
Until the second half of the 20
th
century, behavioral task analysis was the primary
method to understand task performance by detailing the observable behaviors as an
individual performs a task. Although specific physical attributes of problem solving
skills could be quantified, there was little consideration for the unobservable thought
processes taking place inside the individual’s mind. Prior to the use of cognitive task
analysis, individuals developed and used a behavior task analysis to capture the
needed information from experts on how to complete tasks (Clark et al., 2008).
Although a behavior task analysis can describe the observable steps involved in
completing a task, it fails to capture the critical information about the cognitive
processes required for how to complete the task. Cognitive task analysis developed
out of the need to capture these previously unobservable decisions, analyses,
judgments, and other cognitive processes.
Defining cognitive task analysis
There are a considerable number of definitions for cognitive task analysis.
Cognitive task analysis (CTA) refers to a variety of tools and techniques for
describing the knowledge and skills required to perform a task that yields information
about the unconscious thought processes and goal structures that underlie the
observable task performance (Schraagen, Chipman, & Shalin, 2000). CTA
techniques capture a representation of the knowledge and strategies that have been
retained by the individual until some time after the event in question (Feldon, 2007).
Furthermore, cognitive task analysis (CTA) uses a variety of methods to capture

21
experts conscious and non-conscious automated knowledge needed to perform a
complex or critical task (Crandall, Klein, & Hoffman, 2006). The common theme
found in these definitions is the use of multiple techniques that elicit knowledge,
facilitate data analysis, and represent the content and structure of the participant’s
knowledge within a specific task domain.
CTA has been used to capture expertise in a variety of working environments.
Researchers have used a cognitive task analysis framework to examine expert
performance to capture the cognitive task demands on an Intensive Care Unit team
(Fackler, Watts, Grome, Miller, Crandall, & Pronovost, 2009); to capture the
cognitive demands and indicators of business pilot expertise when faced with various
weather conditions (Latorella, Pliske, Hutton, & Chrenka, 2001); to investigate the
characteristics and decision making of expert warning forecasters of severe weather
(Hahn, Rall, & Klinger, 2003); and to develop a preliminary visual design aide
(heuristic) and prescriptive evaluation criteria to assist intelligence analyst efficiently
sift through data overload and develop an appropriate response to situational
questions outside their area of expertise (Patterson, Woods, Tinapple, & Roth, 2001).
CTA methodology
Cooke (1994) identified over 100 different types of cognitive task analysis
methods. This enormous variety of CTA methodologies can make it difficult for
practitioners to choose the correct method for the end result purposes of the research.
Cooke (1994) grouped the CTA methods reviewed into three broad families of CTA
techniques: 1) Observations and interviews; 2) process tracing; and 3) conceptual

22
techniques. As noted in Clark et al. (2008), observations and interviews includes
talking with experts and watching them perform tasks. Process tracing involves the
use of think-aloud protocol or subsequent recall of an expert’s performance in a
specific task, while conceptual techniques are used to capture structured conceptual
knowledge within a domain (Clark et al., 2008).
Clark et al., (2008) reviewed the CTA literature and found that individual’s who
conduct CTA research typically follow a five-step procedure: 1) Collect preliminary
knowledge; 2) Identify knowledge representation; 3) Apply focused knowledge
elicitation methods; 4) Analyze and verify data acquired; and 5) Format results for the
intended application (Schraagen et al., 2000 and Clark et al., 2008,).
Collecting preliminary knowledge incorporates gaining initial knowledge of the
domain (bootstrapping) and developing a sequence of main steps that will guide the
CTA investigation. This initial step includes the analysis of documents and
identification of potential experts to be utilized within the study. The information
gained in the preliminary stage allows the analyst to examine the overall task in
question and identify possible sub-tasks and types of knowledge required for experts
to perform the complex task. Once the analyst has identified the knowledge
representations involved in the complex task, it is important to choose the appropriate
CTA method of knowledge elicitation (Clark et al., 2008; Yates, 2007) depending on
the knowledge type the analyst is trying to capture. Differing CTA methods elicit
either conceptual or procedural knowledge (or both) (Yates, 2007) with varying
efficiency (Hoffman, Crandall, and Shadbolt, 1998). One of the more effective ways

23
to conduct a CTA and capture both declarative and procedural knowledge is
through a semi-structured interview (Yates and Clark, in review).
After applying the appropriate focused knowledge elicitation methods, acquired
data analysis and verification involves multiple steps. This process includes a)
coding the transcribed interviews to identify the conceptual knowledge (concepts,
processes, and principles), procedural knowledge (action and decision steps), goals,
standards, equipment and materials (Clark, 2004); b) formatting the output for review
and verification by the interviewed subject matter experts (SME); and c) finally the
analyst aggregates all documents into one standard “gold standard” format (Clark et
al., 2008). The final CTA document details the declarative and procedural knowledge
required to complete the actions and decision steps involved with performing the task.
Often, these products are utilized in the development of instructional materials (see
Velmahos et al., 2004) and expert systems (Schraagen et al., 2000).
Research has demonstrated that CTA is effective to acquire the critical conceptual
knowledge and automated skills utilized by experts in task performance (Clark et al.,
2008; Yates, 2007). The following sections will discuss the effectiveness of using
cognitive task analysis methods within a variety of settings, concluding with a review
of CTA within surgical training.
Effectiveness Studies Using Cognitive Task Analysis
Webster’s dictionary (1999) defines effectiveness as “producing a definite or
desired result: efficient” (p.454). Research in the effectiveness of CTA has a variety
of components, such as the chosen knowledge elicitation technique (domain specific),

24
the type of knowledge captured or the usefulness of the knowledge and its
application capabilities (Hoffman, Crandall, & Shadbolt, 1998). In addition, the
effectiveness of a CTA study depends on the results meeting or surpassing some set
criteria in order to be considered effective. In experimental CTA research, the
effectiveness of a study is dependent on the acquired knowledge types utilizing a
specific methodology (Clark et al., 2008; Yates, 2007; Yates and Feldon, in press).
The use of cognitive task analysis has shown to be effective within a variety of areas,
including addressing data overload and human-computer interaction (Patterson,
Woods, Tinapple, & Roth, 2001), piloting under various weather conditions
(Latorella, Pliske, Hutton, & Chrenka, 2001); emergency response teams and white-
water rafting (O’Hare, Wiggins, Williams, & Wong, 1998); air traffic control
knowledge (Redding, Cannon, Lierman, Ryder, Purcell, & Seamster, 1991);
information retrieval through library resources (Pejtersen, 1989); and medical
education (Crandall & Getchell-Reiter, 1993; Johnson, Healey, Evans, Murphy,
Crawshaw, & Golud, 2005; Sullivan, Ortega, Wasserberg, Kaufman, Nyquist, &
Clark, 2008).
Redding et al. (1991) investigated the use of an integrated task analysis
methodology to extrapolate the expert knowledge structures of air traffic controllers
in an effort to produce instructional content for new curriculum. Five full
performance level air traffic controllers were videotaped solving four different
problem scenarios. Each participant provided a verbal protocol while reviewing their
performance giving a description of their immediate goals and decision-making

25
processes while solving each of the four problems. As a result of the collective
verbal protocols, mental models, task decompositions, and individual task models
were constructed. The results of the CTA indicated a significant difference between
expert and non-expert air traffic controllers. In comparison to novices, experts
utilized fewer management strategies at a greater effective rate than novices.
Although it was noted that experts had a vastly greater repertoire of strategies to draw
upon, their ability to synthesize the situation better than novices allowed them to
choose specific appropriate strategies to solve the problem at a faster effective rate,
unlike novices who sometimes exhausted their limited strategies to find a workable
solution. The results from the cognitive task analysis were integrated into a
standardized curriculum that provided novice air traffic controllers a consistent and
deeper understanding to solving problems within their field (Redding et al., 1991).
While the CTA conducted within the Redding et al., (1991) study resulted in a
standardized curriculum for future trainings, Johnson, Cumming, & Omodei (2008)
concluded that the knowledge and skills captured from bushfire fighters would be
inappropriately used in a universal training guide, but would be more appropriate for
training at different levels of leadership. Johnson et al., (2008) interviewed 90
experienced bushfire fighters from 2005-2006 in order to capture the conceptual
knowledge and the decision making process that occurred at various levels of
firefighting leadership positions with regards to a worst case scenario. Johnson et
al.’s, (2008) investigation was in reaction to the post hoc reality of poor decision
making in recent bushfires in the Australian countryside that lead to two fatalities.

26
Semi-structured interviews (adapted from the Critical Decision Method: CDM;
Klein, Calderwood, & MacGregor, 1989) were conducted in multiple probes. Initially
to capture the series of events with additional probes aimed at capturing the actions
and decisions occurring within specific incidents.
An analysis of the transcribed interviews indicated that the responses from the
bushfire fighters could be classified into three major themes: A fire focus; a fire
fighter focus; and a community focus. Additionally, Johnson et al. (2008) indicated
the results followed a trend that matched the level of leadership to the respondent’s
primary focus (i.e. higher levels of leadership displayed more of a community focus).
The CTA utilized captured the decision making process at all levels of leadership, but
due to the complexity at each level, a universal training protocol would not meet the
varied needs of the leadership positions. An interesting finding during the
investigation was the occurrence of experienced bushfire fighters omitting of any
references to a “worst case scenario” during their interviews which could lead to poor
decision making under certain contexts (Johnson et al., 2008). The omitted reference
to a ‘worst case scenario’ from experienced bushfire fighters is consistent with the
research literature on expertise and automated knowledge (Ericsson et al., 2006).
Cognitive task analysis provided a means for Johnson et al., (2008) to capture the
automated decision-making bushfire fighters would execute within a specific context
and for different leadership positions.
Cognitive task analysis has been shown to be effective in capturing decision-
making made by medical professionals (Crandall et al., 1993; & Jacklin, Sevdalis,

27
Darzi, & Vincent, 2008). For example, in an effort to elicit the critical cues
neonatal intensive care unit nurses used to evaluate the health and immediate future
needs of newborns within their care, Crandall and Getchell-Reiter (1993) conducted
semi-structured interviews with nineteen registered nurses. The semi-structured
interviews produced 33 incidents of immediate, life threatening situations, most often
cases of sepsis or systemic bacterial infection, requiring urgent care within minutes or
over the next few hours. The results indicated NICU nurses were alert to a number of
critical indicators of sepsis not found in the medical literature used to train neonatal
care nurses.
In a second study, Crandall & Getchell-Reiter (1993) conducted semi-structured
interviews with five experienced female NICU nurses utilizing a critical decision
method (CDM). Critical decision method is a retrospective knowledge elicitation
technique developed by Klein, Calderwood, & Macgregor (1989). Each nurse was
interviewed about three incidents that lead her to the correct diagnosis of infant
septsis, an incorrect diagnosis of the same nature, and a time when the baby
developed sepsis when not originally detected by the current methods. The
information obtained from the initial study along with the secondary study of sepsis-
related incidents was used to create a sepsis assessment guide that became a learning
aid within training NICU nurses.
As individuals gain expertise, their ability to communicate their knowledge to
novices diminishes (Hinds, Patterson, and Pferrer, 2001). Sullivan et al. (2008)
investigated whether expert surgeons omitted actions and decisions when teaching a

28
colonoscopy to second -year postgraduate residents. Three expert surgeons were
videotaped during an instructional session during which they provided a detailed
account of both the action and decision steps involved in a colonoscopy procedure.
Shortly after the expert’s instructional session, the experts participated in a free recall
of the colonoscopy procedure, focusing on the detail and completeness of the
information provided during the prior teaching session. Following their free recall, an
analyst conducted a CTA with each expert individually in a series of structured
interviews to capture the conceptual and procedural knowledge required to perform
the procedure. As a result of the knowledge captured through the CTA, a 26-step
procedural checklist and a 14-point ‘cognitive demands checklist’ (p.21) were
created. In comparing the experts free recall to the CTA checklist, Sullivan et al.,
(2008) found that the experts omitted 50% to 74% of the ‘essential how to steps’ and
57-75% of the critical decisions involved in the colonoscopy procedure. Although
the expert surgeons in the study were recognized as outstanding surgical educators,
they were not able to articulate all of the action and decision steps involved in the
task. This is consistent with the expertise literature that suggests the knowledge and
skills of experts are highly automated and not easily retrievable when prompted.
(Ericsson et al., 2006), and, moreover, because experts often make errors when
describing how to perform a task (Clark and Estes, 1996, Clark et al., 2008), they
often omit critical information they believe that they have communicated to their
students (Clark, 2006; Feldon, 2004).

29
In summary, the effectiveness of cognitive task analysis is often measured by
the type of knowledge captured and sufficiency for the intended application. The
studies reviewed provide insight about how the use of CTA successfully captures
both the conceptual knowledge and the procedural skills experts use to perform
complex tasks. While there are numerous studies that examine CTA for the purposes
of capturing expertise (See Cooke, 1994 and Yates, 2007) there is a paucity of studies
that investigate integrating the knowledge and skills of experts into training protocols
in the surgical domain (exceptions- see Luker et al., 2008; Sullivan et al., 2007;
Velmahos et al., 2004; and Tirapelle, 2010).
Effectiveness of CTA in Surgical Training
Cognitive task analysis (CTA) has been shown to make a significant difference in
increasing the knowledge and skills of surgical residents (Velmahos et al., 2004;
Bathalon, Martin, and Dorin, 2004; Johnson et al., 2005; Luker et al., 2008; Sullivan
et al., 2007; and Tirapelle, 2010). The following review of studies illustrates the
significant benefit CTA brings to surgical training. The proceeding sections will
discuss the benefits CTA provides to surgical training including capturing both action
and decision steps within a procedure; utilizing CTA to create training materials that
produce immediate acquisition of knowledge and skills when comparing CTA- based
curriculum versus traditional methods; and CTA and long term maintenance of
acquired knowledge and skills.
Cognitive task analysis is often utilized to capture the expertise in order to create
training protocols for novices. Johnson et al. (2005) followed CTA principles to

30
create training guides for five commonly performed interventional radiology
procedures. Each task analysis captured the action and decision steps involved in the
procedure. For example, the CTA conducted for an arterial needle puncture captured
101 action steps and 24 decision steps from the experts interviewed. Johnson et al.,
(2005) indicated the collective knowledge gained through conducting the five CTA’s
would provide a basis for establishing a standard of practice in conducting these
interventional radiology procedures. Johnson et al. (2005) proposed that the
knowledge and skills captured through CTA could be utilized to develop novice
training and assessment simulations focusing on both the action and decision steps
conducted during a procedure. As a result, CTA protocols would rapidly increase a
novice’s knowledge and skill development. Additionally, the CTA could be used as a
cue for experienced operators before they conduct a rarely performed procedure. An
expert’s automated knowledge and skills impacts their ability to modify ingrained
task performance (Ericsson et al., 2006). Therefore, reviewing the action and
decision steps within a procedure would help create a revised mental model to follow
during task performance. Overall, Johnson et al. (2005) hoped that their investigation
would lead to additional studies that focused on less common interventional radiology
procedures, thus highlighting the benefits CTA in its ability to capture automated
decision steps for training purposes.
In a study to examine the efficacy of CTA-based instructional protocols,
Velmahos et al. (2004) sought to determine if training new interns through instruction
developed with the results of cognitive task analysis made a difference in the

31
knowledge and technical skills of interns when compared to those trained with
traditional methods. Twenty-six new surgical interns were randomly assigned to
receive instruction on how to perform a central venous catheterization (CVC) through
traditional methods or through training based on a cognitive task analysis. Pre-test
scores indicated no differences between control (n=12) and experimental (n=14)
groups. Velmahos et al. (2004) found a significant difference between groups in the
posttest mean score when comparing the CTA group with the traditional group (11.0
+- 1.86 versus 8.64 +- 1.82, P=0.03). Velmahos et al. (2004) concluded that the
course taught with the results of cognitive task analysis was more effective in
increasing the cognitive knowledge and technical skills of interns on how to conduct
a CVC procedure.
In another study, Luker et al. (2008) investigated the use of a CTA based multi-
media instructional program and its effect on increasing the surgical skills knowledge
of residents in repairing a flexor tendon. Luker et al. (2008) utilized three flexor
tendon repair subject matter experts to construct a CTA detailing the knowledge and
skills needed to perform the surgery. The principal investigator designed a CTA-
based multimedia curriculum highlighting the critical decision points and skills
needed to complete a flexor tendon repair surgery. Ten surgical residents were asked
to perform flexor tendon repairs on three different occasions with a post-procedure
assessment to assess the residents understanding of the tasks and the critical decision
points within the procedure. Any increase in the participant’s knowledge and skills
from the first and second performance was deemed a result of practice and was

32
considered the “control group” within the study. A learning session using a CTA-
based multi-media instructional presentation was conducted in between the second
and third flexor tendon procedure. Luker et al., (2008) concluded that changes in the
post procedure scores resulted from the CTA-based presentation. The results
indicated that the mean difference between the first and second procedure (control
group) was 13.2 for conceptual knowledge and 8.5 for understanding the advantages
and disadvantages of each decision within the procedure. However, the mean
difference between the second and third procedure was 34.0 for conceptual
knowledge and 19.4 for decision points. Luker et al., (2008) discovered that although
the group improved due to shear practice, the use of CTA in surgical training
significantly increased the knowledge and skills needed to perform a flexor tendon
surgery.
In a recent study, Tirapelle (2010) investigated the effects of a CTA based
curriculum versus traditional surgical training methods on how to perform an Open
Cricothyrotomy. The composition of the twenty-six medical student participants
included 3
rd
year medical students, 2
nd
year post-graduate students, and 3
rd
year post-
graduate students. Pre-test measures showed no differences between the randomly
assigned experimental (n=12) and the control (n=14) groups in terms of experience
F(6,19) = .414; p = .860) or gender F(6,19) = .396; p = .873, or pre-test assessments
=5.4 (experimental) and =5.5 (control) out of 17 possible points. All assessment
and training materials for this investigation were developed from a gold standard
CTA report derived from interviewing six subject matter experts on how to perform

33
an Open Cricothyrotomy. Tirapelle (2010) utilized Clark’s (2004) Guided
Experiential Learning (GEL) protocol in the development of the training materials.
Both groups (experimental and control) received a 30-minute instructional session on
how to conduct the procedure. The control group received instructions through
existing course materials and course structure, while the experimental group was
provided a CTA-based curriculum. Both groups were provided opportunities to
practice an Open Cricothyrotomy on inanimate models following the instruction in a
guided practice format. Post-instruction analysis was conducted on the participant’s
acquired knowledge and skills through individual assessments and ratings from expert
instructors and surgeons participating in the presentation of the training materials.
While the post instruction assessment showed no significant differences in the
acquired conceptual knowledge between the groups ( = 14.5, experimental and 13.9,
control; (t(22) = 0.55, p = .59), a significant difference was found when Tirapelle
(2010) measured the acquired procedural knowledge ( =17.75, experimental and
=16, control; t(21) = 2.08, p = .050) between the groups. Tirapelle (2010)
concluded that CTA-based instruction had a significant positive effect on the
acquisition of procedural knowledge and performance when compared to traditional
expert-led surgical skills instruction.
While the research of Velmahos et al. (2004), Luker et al. (2008) and Tirapelle
(2010) all found CTA- based curriculum produced a greater immediate increase in the
knowledge and skills of surgical interns, Bathalon et al. (2004) provided evidence that
the use of CTA for surgical instruction has long-term benefits as well. Bathalon et al.

34
(2004) randomly assigned 44 first year medical students into three different
instructional groups on how to perform a cricthyroidotomy procedure. The first group
(n=16) received instruction from the traditional ATLS protocol. The second group
(n=13) learned the same procedure using both cognitive task analysis and kinesiology
principles. The third group (n=15) was instructed with a combination of CTA,
kinesiology principles, mental imagery practiced daily, and debriefing. The results
indicated that the groups taught with a CTA-based curriculum performed better
initially and also maintained their skill level over 12 months when compared to the
first group. The group who received training using a combination of CTA,
kinesiology principles, mental imagery practice, and debriefing showed the highest
skill acquisition and long-term maintenance.
In another study, Sullivan et al. (2007) investigated the effectiveness of a CTA-
based curriculum to instruct novices on the knowledge and skills required to
successfully complete a percutaneous tracheostomy (PT) placement as compared to
students trained by traditional methods. Sullivan et al. (2007) randomly assigned 20
postgraduate surgery residents into either the control group (N=11) and provided
traditional PT training, or the CTA group (N=9) and provided CTA-based training.
The results from Sullivan et al. (2007) study indicated that there were no significant
differences between the groups prior to instruction. However, post-instruction
assessment results indicated that a significant difference existed between the control
and experimental (CTA) group after instruction. The CTA group scored significantly
higher mean averages than the control group in the technical aptitude assessment at 1

35
month (CTA: 43.5+- 3.7, control 35.2 +- 3.9, P=0.001). Although an attrition of
skills was indicated at the 6-month reevaluation assessment for both groups (CTA: 39
+- 4.2, Control: 31.8 +- 5.8, P=0.004), the group that received a CTA-based
curriculum still performed significantly better than the group who received traditional
instructional methods. Therefore, the expertise CTA captures, when utilized in
instructional materials, plays a significant role in encoding knowledge and skill
development into long-term memory.
In summary, research has shown using CTA is effective for surgical training.
Cognitive task analysis is effective in capturing expertise to create training protocols
for standards of practice (Johnson et al., 2005). Additionally, others have developed
CTA-based instructional materials where the results showed significant gains in
knowledge and skills of surgical interns when compared to traditional methods
(Velmahos et al., 2004; Luker et al., 2008; and Tirapelle, 2010). While others
provided evidence CTA has beneficial acquisition of knowledge and skills with long-
term maintenance (Bathalon et al., 2004 and Sullivan et al., 2007). CTA can help
obtain insights into the automated expert conceptual and procedural knowledge of
domain specific tasks and separate the steps into individual discrete teachable units
for others to learn. While there are several positive outcomes when utilizing cognitive
task analysis methods, there are some limitations with CTA.
Limitations of Cognitive Task Analysis
Although the research on cognitive task analysis consistently shows that CTA
training is more effective than traditional training methods, there are several

36
limitations to using CTA techniques. An important limitation and the one most
commonly identified in the literature is the amount of time and effort involved in
conducting a CTA. Additional limitations include the automaticity of expert’s
knowledge and the ability to acquire their expertise. Next, these limitations are
discussed in detail.
The time necessary to conduct a cognitive task analysis can be time intensive
(Chao & Salvendy, 1994). According to Clark & Estes (2008), one hour of capturing
expertise requires approximately 30-35 hours of effort (Grunwald, Clark, Fisher,
McLaughlin, Piepol, 2004). Sullivan et al., (2008) estimated it took approximately 30
hours to complete the cognitive task analysis in their study. Additionally, Hoffman et
al., (2004) solidified the human cost of knowledge acquisition by documenting the
human hours required to complete components of their investigation utilizing CTA
methods. For instance, Hoffman et al., (2004) utilized CTA methods in transferring
the information in the Terrain Analysis Data Base (TADB) into 150 concept maps.
The total person hours to complete the 150 concept maps was calculated at 187.5-225
person hours. Each individual concept map took approximately 75-90 person hours
to complete.
One of the major limitations of cognitive task analysis focuses on the process of
knowledge acquisition (Hoffman et al., 1998). There are two main components to
knowledge acquisition including acquiring experts to interview and capturing their
expertise. An overall factor involved in these components is the time and effort
required in conducting a CTA. While there are enough experts to interview and

37
capture the desired procedural knowledge for training purposes, their time to
participate in a CTA may be limited. Patience, adaptivity to the expert’s schedule,
and the ability to mediate logistic situations are essential to obtaining an environment
to conduct the CTA. Once these issues have been resolved, a bottleneck still exists in
capturing the expertise due to their highly organized cognitive structures.
The automated knowledge and skills of experts is a natural barrier to acquiring
their expertise. Sleeman and Brown (1982) indicate that much of an expert’s
knowledge is tacit and not readily available to the expert during retrieval tasks. Glaser
(1985) derived that experts have the ability to discuss the “what” and “why” in
performing a task, but a conscious analysis of their verbal recall is required to capture
the “how and when”, that is the decisions and reasoning necessary to perform a task.
As a result experts automated knowledge may lead to retrieval inaccuracies during the
knowledge elicitation process (Feldon, 2007). As a consequence, dependence on a
single expert for a CTA investigation can provide incomplete results for intended
purposes. Therefore, the necessity of acquiring multiple experts may be an additional
limitation to capture the required knowledge.
Number of Experts Required for CTA
There is a scarcity of research indicating how many experts are required to
optimally capture the critical information necessary to problem-solve tasks. A notable
exception is Chao and Salvendy’s (1994) study recommending the use of three
experts. They randomly assigned twenty-four expert computer science students to
one of four knowledge elicitation techniques. The dependent variable was the

38
percentage of procedural knowledge captured through a knowledge elicitation
technique. The independent variables included the three computer programming
tasks (diagnosis, debugging, and interpretation) and the four knowledge elicitation
methods utilized (protocol, interview, induction, and repertory grid). The diagnosis
task combined with the repertory grid knowledge elicitation method indicated the
greatest gains in procedural knowledge from 40% from a single expert to 87% from a
total of six experts. For the other two tasks, the use of protocol knowledge elicitation
method showed the greatest gains in procedural knowledge obtained from one to six
subject matter experts: Debugging: 37% to 88%; Interpretation: 27% to 62% (Chao &
Salvendy, 1994). Chao and Salvendy (1994) found that the percentage of procedural
knowledge increased as they interviewed additional subject matter experts up to a
total of six individuals. As a result of their findings, Chao and Salvendy (1994)
recommended interviewing three subject matter experts based on a 10% marginal
utility cost-benefit analysis.
A review of the surgical training literature indicates a variability of the number of
experts utilized for CTA investigations. For example, Velmahos et al. (2004) made
use of two subject matter experts; Sullivan et al. (2007) gained knowledge from three
subject matter experts; Luker et al. (2008) utilized three subject matter experts in their
investigation; Johnson et al. (2006) employed two or three subject matter experts to
create their CTA protocols; and Sullivan et al. (2008) developed their CTA from three
subject matter experts. In an effect to create a decision map based on a diagnosis of
symptomatic gallstones, Jacklin, Sevdalis, Darzi, and Vincent (2008) utilized a

39
structured interview technique with 10 experienced physicians in the area of
gallstones. It appears common in the CTA research to utilize two to three experts for
the purposes of capturing expertise.
Currently, only two known studies exist recommending the number of subject
matter experts needed to conduct a cognitive task analysis: for solving computer
related problems (Chao and Salvendy, 1994): and for surgical procedures (Crispen,
2010). In a concurrent investigation, Crispen (2010) investigated the optimal number
of subject matter experts to be interviewed in acquiring the critical expertise required
to complete a surgical procedure. Crispen (2010) found that four subject matter
expert’s were optimal in capturing the knowledge and skills needed to perform an
Open Cricothyrotimy procedure. This was the first empirical study that has identified
a recommended number of subject matter experts to be interviewed to capture the
expertise needed to complete a surgical procedure.
Summary
Medical training centers are charged with producing competent physicians who
are technically proficient in a variety of procedures (Aggarwal, Grantcharo, & Darzi,
2007). Within every educational program, expert instructors are charged with
providing needed information and resources to help novices develop their expertise.
Due to an experts highly developed declarative and procedural knowledge, their
ability to share their expertise with novices is limited. Cognitive task analysis (CTA)
has been shown to be effective in capturing expert’s automated knowledge in a
variety of fields (O’Hare, Wiggins, Williams, & Wong, 1998; Redding, Cannon,

40
Lierman, Ryder, Purcell, & Seamster, 1991; Pejtersen, 1989; and Chao &
Salvendy, 1994). CTA investigations within the field of medicine are aimed at
capturing critical decisions an expert physician makes during a procedure. This
acquired knowledge has been proven beneficial in the development of expert-based
instructional protocols (Johnson et al., 2005). These CTA based training aids help
novice physicians develop declarative and procedural knowledge for a variety of
medical procedures.
Although there are significant benefits of CTA techniques, there are certain
limitations of this methodology. The two major limitations of cognitive task analysis
are 1) the process of knowledge acquisition and 2) the time required to conduct CTA
to provide a useable product for training purposes. The research literature has
multiple examples of studies indicating the benefits of using experts in capturing
expertise needed to perform a task/ procedure (Lyons, 2009). Currently, there is no
established recommended number of subject matter experts required when conducting
a CTA capturing the necessary conceptual and procedural knowledge for intended
purposes. Only two studies have provided recommended number of SMEs in CTA
research: a study conducted by Chao and Salvendy (1994) resulted in their
recommendation of three experts in solving computer based problems; and Crispen
(2010) recommended interviewing four experts for a surgical procedure.
Purpose of the study
Informed by the study conducted by Chao & Salvendy (1994), the purpose of this
investigation is to explore critical information that is gained from a single expert

41
position and the information gained from each additional expert CTA interview.
The information acquired from this investigation will allow us to propose an answer
to the following question: How many experts are recommended to interview in order
to collect the critical knowledge (procedural steps and decisions) needed to perform a
(surgical) procedure? The results of this investigation aims to establish a
recommended number of experts one would need to conduct CTA interviews with in
order to capture their expertise to develop a gold standard protocol to conduct a
medical procedure. Establishing a recommended number of experts to interview has
significant implications in long-term cost-benefit savings of time and effort inherent
in conducting CTA investigations.
To review, the research questions are:
1) How much information about a central venous catheter placement procedure
does a single expert provide when compared to a six-subject matter expert gold
standard protocol?
2) How much Critical Information is gained from each additional CTA interview
about the Central Venous Catheter Procedure?

42
CHAPTER 2: METHOD
The central venous catheter placement is a procedure that can be performed in
emergency and non-emergency situations. In medical procedures, a central venous
catheter ("central line", "CVC", "central venous line" or "central venous access
catheter") is a catheter placed into a large vein in the neck (internal jugular), chest
(sublcavian) or groin (femoral) . A CVC is used to administer medication or fluids,
and directly obtain cardiovascular measurements such as the central venous pressure.
As noted, the procedure is designed to gain access to a central vein for multiple
reasons, including rapid fluid infusion. The central venous catheter procedure was
used as the sample procedure to answer the proposed research questions. The current
study utilized a CTA based methodology. The proceeding sections review the overall
study design, how the subject matter experts were acquired, and how the data was
collected and analyzed.
Design
The current descriptive investigation, in general, replicates the research questions
proposed by Chao and Salvendy (1994), While Chao and Salvendy (1994) had 100%
knowledge of all possible errors within their study, the current investigation started
without a gold standard to measure expert’s elicited knowledge. Prior research
(Velmahos et al., 2004) on the central venous catheter placement produced a 14-point
checklist of tasks to complete during the procedure. This was created through the
collaborative work between two CTA experts and two experts on CVC. Although a
checklist has previously been created, it was never qualified as a “gold standard” for

43
how to perform the CVC procedure. The current investigation began by collecting
conceptual and procedural knowledge from experts discussing how to complete a
CVC procedure through semi-structured interviews. The combined experts’ elicited
knowledge was aggregated into a gold standard CVC protocol. The gold standard
protocol represented 100% of total knowledge about the CVC procedure. The gold
standard was utilized to quantify the total conceptual and procedural knowledge
acquired from each expert to answer the research questions. As indicated above this
data analysis was in part a replication of the work completed by Chao & Salvendy
(1994).
Subjects
A convenience sampling procedure was used to acquire the six expert physicians
who participated in this investigation. The participating physicians included four
trauma physicians; one critical care internist; and one anesthesiologist at whom work
in large medical centers in the Los Angeles area. The participants were considered
experts due to their extensive years of successfully performing the CVC procedure.
Prior CTA based investigations used a range between two to ten experts. Six experts
were utilized in the current investigation to determine the optimal number of experts
needed to capture the needed critical information and to have a large enough sample
to validate the average amount of information gained from additional experts without
expanding undue human time, effort, and cost. The research literature indicates
between two and five subject matter experts are recommended to complete a valid
CTA investigation.

44
Data Collection
The procedures on how to conduct the CTA for this investigation followed the
description provided by the work of Clark, Feldon, van Merrienboer, Yates, and Early
(2008) who provided details on the five most common elements of a CTA
investigation. The data collection for this investigation was conducted in four parts::
1) semi-structured CTA interviews of physicians with expertise on conducting CVC
procedure; 2) coding of interview manuscript; 3) creation of a six-expert based gold
standard CVC procedure protocol; and 4) analysis of the elicited knowledge from
both (a) single-expert and (b) non-repeated grouping of experts (i.e 2 SME’s, 3
SME’s up to 6 SME’s) against the gold standard.
Semi-structured CTA interviews
During the semi-structured cognitive task interview, the experts were asked a
series of questions that focused on the major tasks and potential problems a surgeon
could encounter when conducting a CVC procedure. Attention was focused on
obtaining the indications and contraindications on when to and when not to perform
the procedure. In addition to obtaining the overall procedural objective, the analyst
inquired about the expert’s knowledge of the benefits and any potential risks
performing the procedure. During their description of the CVC procedure, each
expert was asked to identify the equipment needed at the various stages within the
procedure. Upon confirmation of the major tasks, the experts were asked to describe
the specific actions they perform at each major task and sub task(s) within the CVC
procedure. During the interview process, the analyst asked probing questions about

45
the action and decision steps involved in the procedure to uncover any alternatives
to the decision steps being made and the criteria for choosing such alternative actions.
Lastly, the experts were asked to provide sensory information (touch, hear, or smell)
that a surgeon utilizes for an action or decision step. All interviews were recorded in
audio and transcribed into a manuscript.
CTA Coding Scheme and Procedure
Groups of two to three trained coders utilized a coding scheme developed by
Expert Knowledge Solutions (2009), to code the interview manuscripts. The analysts’
reviewed the transcriptions and coded items (words or phrases) into one of the
following categories: action or decision steps, equipment, indications to perform the
procedure, contra-indications on when not to perform the procedure, benefits, time
and accuracy details, along with any sensory information. Additional attention was
paid to uncover if any statements concealed covert action or decision steps or other
items that were considered critical information that required coding. Overall, The
coders compared their individual coding results and resolved coding disagreements
through discussion. Inter-rater coding reliability was calculated using Cohen’s Kappa
to assess the consistency of the coding process. Once the coding of the document was
completed and an inter-rater reliability calculation indicated a 99% inter-rater
reliability, the coded information was transferred into a CTA protocol.

46
CTA Protocol and creating a six-subject matter “gold
standard” CVC protocol

The coded items within each of the transcripts were formatted into individual
CTA protocols that listed all the relevant information involved in conducting the
central venous catheter placement (CVC) procedure from each expert interviewed.
Each expert reviewed their own CTA protocol for data verification and agreement.
Any necessary modification, deletions, or additions to the document were made at
that time. Upon inputting the changes provided by each expert, the experts reviewed
their own CTA protocols for verification of the edits and any other possible
corrections made by the expert. Upon final edits from the experts, the six individual
protocols were aggregated to create a gold standard CTA protocol.
The aggregation process started with combining similar statements that were
made by the experts regarding either action or decision steps. When different
statements were made regarding a similar task, they were molded into one action or
decision step (for example, “advance needle forward’ and ‘walk your needle’ are two
similar statements indicating the same action- pushing the needle forward into the
patient). Partial action steps were combined into larger steps to create complete and
efficient action steps. For example, the individual action steps of “gather materials’,
‘prepare equipment’, and ‘validate usability’ were combined to create one complete
action step. This aggregation process was completed for every section of the CTA
protocol.

47
After the six-subject matter expert CVC CTA protocol was created, it went
through two rounds of editing. In the first round, each expert reviewed the six-expert
CTA protocol independently for verification of data and editing purposes. All edits
were incorporated into the final CTA protocol by the researchers. A final editing
review was conducted by one of the original experts in the presence of an analyst for
immediate verification of data and sequencing of events within the CVC procedure.
These final edits were incorporated into the six-expert CTA job aid protocol. This
final version became the ‘gold standard’, which encompassed the complete details of
the central venous catheter placement procedure.
Data Analysis
The gold standard CTA protocol was used to calculate the percentage of
agreement from each of the individual subject matter expert CTA protocols. The
individual items within the CVC gold standard CTA protocol were individually
entered into a Microsoft word spreadsheet document. Each item was manually
graded for completeness in quantifying the information obtained from each expert.
Every discrete item that appears on the individual expert’s CTA protocol and matches
the ‘gold standard’ received one point. The discrete items included conditions,
equipment, action steps, and decision steps for example. Any item that was included
on the gold standard and not the subject matter expert’s CTA protocol, received zero
points. Zero points are awarded for any item that appears on the gold standard but not
on the subject matter expert’s initial CTA report (See table 1 for an example).

48
Table 1
Equipment CTA Gold
Standard
Expert A Expert B Expert C Expert D Expert E Expert F
X-ray
(A72;
B1066;
C48; F175;
D381; E92)

1 1 1 1 1 1 1
Personal
protectiv
e gear
(A91;
D60)

1 1 0 0 1 0 0
(Excerpt from CVC CTA protocol spreadsheet document.)

In an effort to provide evidence to answer the first research question, “How much
information does one expert provide when compared to a six subject matter gold
standard protocol? – an ‘acquired knowledge’ score for each subject matter expert’s
CTA protocol was calculated. An acquired knowledge score is the calculated
percentage of total knowledge obtained from each expert, per section of CTA
protocol, that was calculated based on the number of items captured from each expert
in comparison to the six subject matter expert gold standard protocol. For example,
per CTA interview, there are 30 pieces of equipment needed to complete the CVC
procedure, as identified by the six-SME gold standard CTA protocol. Each expert
provided varied amounts of equipment items in their initial interviews (i.e., Expert
“A”-23 equipment items; Expert “B”-7 equipment items; and Expert “C”-12
equipment items.) An acquired knowledge score for total equipment items was
calculated by dividing the number of items obtained by the total possible items, i.e. (#
of items per expert)/ 30). Therefore, the acquired knowledge from Expert A was

49
23/30 or 77%. In other words, Expert “A” provided 77% of the total number of
equipment items needed when compared to the CTA gold standard protocol.
Acquired knowledge scores were calculated for each of the sub sections within the
gold standard protocol, including action steps, decision steps, equipment, reasons for
performing the procedure, risks for not performing the procedure correctly,
indications to perform the procedure, and contraindications on when not to perform
the procedure or in determining site selection for the CVC procedure.
To answer the second research question- how much critical information is gained
from the addition of subsequent experts- an ‘acquired knowledge’ score was
calculated for each non-repeating groups of two, three, four, five, and six subject
matter expert groups for all sub-sections of the CTA CVC protocol. This calculation
will be represented by the cumulative total number of the procedure’s items captured
from each non-repeating group of subject matter experts divided by the total number
of items in the gold standard. The groupings were created by pairing the protocols
from the experts in non-repeating combinations, A SME combination refers to the
combined captured knowledge of the stated number of experts per grouping (i.e.,
Two-pair combinations: (Experts AB; AC; AD); Three SME combinations (Experts
ABC; ABD; ABE). There were 15 non-repeating two SME pair combinations, 20
non-repeating three SME combinations, 15 non-repeating four SME combinations, 6
non-repeating five SME combinations, and one six-SME combination (see appendix
‘A’ for complete list of two through six SME non-repeating group combinations).

50
Acquired knowledge scores were calculated for all subject matter combinations
by using the same procedure for the first research question: if a discrete item was
present in either of the experts CTA, that item was awarded one point and if not
present a zero will be inputted into the corresponding spreadsheet cell. Using an
excel spread sheet the total acquired knowledge for each paired SME combinations
were calculated for all 10 sections (i.e. objective, conditions, equipment, etc.) within
the CTA CVC protocol and the site location subsections (Internal jugular, subclavian,
and femoral sites). For each SME combination group an average was calculated from
adding up all the raw scores from the paired SMEs and dividing by the total number
possible for each sub-section (example- Standards). The same calculations were
performed for each subject matter combinations of three, four, five, and six expert
groups (see Table 2 for a two SME calculation for the sub-section: Standards). A sum
of the points from each of the subject matter expert pairings were divided by the total
number of gold standard items to provide the total percentage agreement with the
gold standard.
Table 2
Standards
CTA Gold Standard SME A SME B Combined
The time frame for the CVC
procedure ranges from 2 minutes
(F50) -10 minutes (E86) with an
average of 5 minutes (A445;
B1101; E86)

1 1 1 1
Observable indications of success:
Chest x-ray (E92) should show:

1 0 0 0
1. The catheter is in the superior
vena cava. (B1081)

1 0 1 1
2. Clear lung fields (B1076) 1 0 1 1

51

3. Easily draw blood back from the
catheter (E89)

1 0 0 0
4. Easily flush fluid into the
catheter (E90)

1 0 0 0
Total # of Standards 5 1 3 3
Percent of Standard per SME 100 20 60 60
(Excerpt from CVC CTA protocol spreadsheet document.)
(Table 2, Continued)
Since the ‘gold standard’ CTA protocol is a combination of the acquired
knowledge from all six physicians with expertise in conducting a CVC procedure, the
amount of total information acquired with each additional expert will increase up
until the sixth expert. In conducting cognitive task analysis interviews, there exists a
point of diminishing marginal return on the investment of time and human effort.
Chao & Salvendy (1994) utilized a marginal utility of 10% representing this point of
diminishing marginal return. The marginal utility of acquired knowledge was
distinguished as the change in increased acquired knowledge due to the additional
expert. In formula form, marginal utility = Change in total utility/ change in quantity.
In alignment with Chao & Salvendy (1994), the marginal utility for the current study
was set at the position where the additional acquired knowledge from an additional
expert was calculated to be less than 10%.

52
CHAPTER 3: RESULTS
Conducting CTA investigations constitutes significant investments in time and
effort to complete. An unanswered barrier to conducting a CTA research project is
deciding on the number of experts one would need to interview. Each expert
interviewed consists of a dedicated amount of time and effort to procure an expert,
capture their knowledge, analyze the findings, and represent their knowledge in a
working document. Establishing a standard number of experts to interview will
reduce the cost-benefit of conducting CTA investigations by providing future analyst
a set standard equated with reliable data. Knowing a set number of SMEs to interview
will minimize extraneous cost and effort from interviewing too many experts.
Coding and Inter-Rater Reliability
Each coder was instructed to review the transcript to capture the overall objective
of the procedure, any conditions for performing or contraindications, as well as noting
required action steps, decision steps, equipment, and the major tasks necessary to
perform the procedure. The analysts compared all documentation coding and the
inter-rater reliability was calculated at 99.46%. Each protocol document was
formatted into separate CTA protocols. Each CTA protocol was reviewed and edited
by the expert who was interviewed. The six individual CTA job aids were aggregated
to create a six-expert gold standard CTA CVC placement procedure protocol. Table 3
indicates the CVC gold standard CTA protocol subsections and the number of items
within each sub-section. The total items were calculated through adding the number
of individual concepts, actions, or decisions within each sub-section.

53
Table 3: CVC Gold Standard Sections and Corresponding Number of Items per
Section.
Gold Standard Item Total Number Gold Standard
Item
Total Number
Objective 1 Standards 5
Risks 17 Equipment 30
Reasons 3 Tasks 8
Indications 4 Action Steps 44
Contraindications 4 Decision Steps 14

Research question #1: How much information about a central venous
catheter placement procedure does a single expert provide when compared to a six-
subject matter expert gold standard protocol?
The six-expert gold standard protocol was transferred to an Excel spreadsheet for
analysis. For each individual expert CTA protocol, a value of 1 was given to each
discrete item that represented a single action, decision step, condition, equipment, etc.
A column in the excel spreadsheet was established for each of the six expert CTA
protocols. The total score from each expert was calculated by adding all the values
together. The percentage of acquired knowledge was calculated by dividing the
obtained total for each of the six individual scores by the total possible number of
items from the six-expert CVC gold standard protocol. Sub-totals were also
calculated for the procedures action steps, decision steps, conditions, equipment,
reasons for performing the procedure, and risks for not performing the procedure
correctly.
To answer the first research question, the results of the study are displayed in
Table 4, which indicates the percentage of acquired knowledge obtained from

54
individual experts in comparison to the six-expert CTA CVC “gold standard”
protocol based on a single CTA interview. The range of total knowledge acquired
from a single expert when compared to a six-subject matter expert gold standard was
43% to 73% with an average of 57%. The range of acquired action steps from a
single expert when compared to a six-subject matter expert gold standard from a
single from 50% to 89% with an average of 70%. The average acquired knowledge
for decision steps from a single expert when compared to a six-subject matter expert
gold standard was 65% with a range of 57% to 71%.
Table 4
Percentage of Knowledge Acquired from One Expert When Compared to a Six
Subject Matter Expert Gold Standard Protocol
SME 1 SME 2 SME 3 SME 4 SME 5 SME 6

Total
Knowledge
Acquired
65 48 43 73 61 54 57
Action Steps 75 66 50 89 70 70 70
Decision Steps 57 71 64 64 71 64 65
Objectives 100 100 100 100 100 100 100
Reasons 0 0 67 0 67 33 28
Risks 35 0 0 82 0 0 20
Indications 75 75 75 75 75 75 75
Contra-
indications
50 50 25 25 50 75 46
Standards 20 60 0 0 60 0 23
Recommended
Equipment
77 23 40 67 63 50 53

55
SME 1 SME 2 SME 3 SME 4 SME 5 SME 6

Tasks 100 100 75 100 100 88 94
(Table 4, Continued)

Research Question #2: How much Critical Information is Gained from Each
Additional CTA Interview about the Central Venous Catheter Procedure?
In order to calculate the added percentage of acquired knowledge from each
additional subject matter expert (SME) a total composite score for each SME non-
repeating two, three, four, five, and six expert combination group was calculated. An
average percentage was determined by dividing the raw combination score obtained
from each paired SME combination by the total possible score based on the gold
standard. This calculation was completed for every section (Objective, Risks,
Reasons, Conditions, Equipment, Standards, Tasks, Action steps, and Decision steps)
including total number of items. With respect to Chao and Salvendy (1994), the
marginal utility was calculated as the change in an increased percentage of acquired
knowledge with the supplement of an additional expert. The point of diminishing
marginal return was calculated at the point where the marginal utility of the additional
expert was calculated at less than 10%.
Within this section, both tables and figures will be utilized to illustrate the
additional percentage of acquired knowledge for each of the sections within the CVC
gold standard. A table illustrating three categories will be followed by multiple
figures that help illustrate both the increase in acquired knowledge and the decrease
in marginal utility with each additional expert for individual categories. Table 5

56
indicates the total average percent of total acquired knowledge (all items on the
CTA protocol), action steps, and decision steps for each non- repeating group’s when
compared to the six-SME gold standard protocol. As anticipated, the average
percentage increases steadily with each additional expert up to six experts, whom
account for 100% of the items within the gold standard CTA protocol.
Table 5
The average percent of Total knowledge, Action steps, Decision steps per number of
experts when compared to a six-SME gold standard protocol.

Table 6 illustrates the average percentage of acquired knowledge from a single
expert and the increased percentage of total knowledge, action steps, and decision-
steps as a result of an additional SME. Also noted within table 6 is the diminishing
marginal utility with each additional expert. The average amount of total knowledge
gained from zero experts to one expert was a 57.44% increase. With the addition of a
second expert, an average increase of 13.28% of total knowledge was noted. A third
expert provides and average increase of 8.59%, while a fourth expert provided an
increase of 4.59%. With the addition of the fifth and sixth experts, the average
increase of acquired knowledge was 6.62% and 9.49%, respectively. A single expert
provided an average increase of 70% for action steps and an average increase of 65%
Number of
Experts
Total
Knowledge
Action Steps Decision Steps
Per group Per group Per group
1 57% 70% 65%
2 71% 72% 88%
3 79% 78% 91%
4 84% 82% 92%
5 91% 84% 93%
6 100% 100% 100%

57
for decision steps. The addition of subsequent experts produces an increase in
acquired action steps and decision steps at varying increased rates. However, there is
clearly a diminishing utility with each additional expert.
Table 6
Average Increase in Percentage of Total Knowledge, Action Steps, and Decision
Steps as a function of an additional Expert When Compared to a Six-Subject Matter
Expert Gold Standard Protocol

Figure 1 shows the increase in percentage of total knowledge acquired as a function
of additional experts when compared to a six-expert gold standard CTA CVC procedure
protocol. Total knowledge consists of all the subsections measured, including action
steps, decision steps, conditions when to perform and contra-indications on when not to
perform the surgery. The average total knowledge increases from 57.44% with one
expert to 79.31% with three experts and 100% total knowledge gained from the six SMEs
interviewed.
Number of
Experts
Total Knowledge Action Steps Decision Steps
increase increase increase
1 57.44 70.08 65.48
2 13.28 1.59 22.62
3 8.59 6.52 2.98
4 4.59 3.79 1.31
5 6.62 2.5 0.48
6 9.49 15.53 7.14

58

Figure 1. Percentage of acquired knowledge based on six-expert gold standard
CTA CVC procedure protocol as a function of the number of experts.

Figure 2 shows the marginal utility or the percent of total knowledge acquired
with the addition of an additional expert, up to six total experts. One subject matter
expert provided an average increase of 57.44% of knowledge. When the data from
the second subject matter expert is combined with the first subject matter, there is an
average increase of 13.28% of acquired knowledge. When a third expert is added, an
average of 8.59% additional acquired knowledge is obtained. A fourth expert
provided an average increase of 4.59%. While a fifth expert added an additional
6.62%% increase in acquired knowledge. The sixth expert provided just an additional
9.49% of increased knowledge. Utilizing the established standard set by Chou and
Salvendy (1994) of 10% marginal utility to quantify how many experts are
recommended, figure 2 indicates that three experts are recommended to meet this
threshold level of acquired knowledge.

59

Figure 2. Average percentage increase of knowledge as a function of an
additional expert for total possible items in the gold standard CVC procedure
protocol.

Figure 3 shows the increase of percentage of action steps acquired [based
on a six-expert gold standard CVC procedure protocol] with each additional subject
matter expert. The accumulation of knowledge from additional experts indicates a
greater percentage of actions steps were acquired as a result of subsequent experts.
On average, the knowledge gained from additional experts exceeded the total
combined knowledge of the prior number of experts. For example, the amount of
acquired knowledge from two experts is greater than a single expert. This pattern of
increased acquired knowledge is repeated in future analysis of different subsections
within the gold standard protocol. A single expert provided an increase of 70.08% of
action steps while three experts provided an average of 78.18%, while six experts
provided 100% of action steps within the CVC gold standard procedure protocol.

60

Figure 3. Average percentage increase in action step acquired knowledge as a
function of the number of experts.

Figure 4 indicates the marginal utility for action steps in respect to
additional experts, based on a six-expert gold standard CVC procedure protocol. A
single expert provided an average increase of 70.08% of new knowledge. Figure 4
illustrates that marginal utility is achieved at two experts. The greatest percentage of
acquired knowledge is achieved with the initial expert. When the acquired
knowledge from additional experts is added, there is an increase in overall captured
knowledge. The variability of the average increase in acquired knowledge was a
product of the varying degrees of information acquired from the experts within the
investigation and their agreement with the gold standard protocol.

Figure 4. Average percentage increase of action steps as a function of additional
experts.

61

Figure 5. Average Percentage increase of Gold Standard Decision Steps as a
function of the number of experts.

Figure 5 shows the increase of acquired knowledge of gold standard
decision steps with each additional subject matter expert. One expert provided
65.48% of total gold standard decision steps, while three experts provide an average
of 91.07% and six experts provide 100% of gold standard decision steps. There were
a total of fourteen decision steps within the six-expert based gold standard protocol.
The majority of the decision steps were acquired with the information captured from
the first two experts, on average. The information captured from groups of three,
four, and five experts did not provide a significant increase of acquired decision steps
beyond the average of two experts.

62

Figure 6. Average percentage increase of decision steps as a function of
additional experts.

Figure 6 shows the average percentage increase of decision steps as a function of
additional experts. A single expert provided an average increase of 65.48% of the
gold standard decision steps that are within the CVC placement procedure. With the
addition of a second expert, an average of 22.62% increase was noted. The acquired
knowledge from the third expert provided an additional 2.98% increase. Figure 6
illustrates that diminishing marginal utility of 10% was achieved with three experts.
As noted in figure six, the greatest percentage of acquired knowledge in decision
steps is achieved with two experts. Groups of three, four, and five experts did not
produce much of an increase in acquired knowledge. While six experts provided a
increase of acquired knowledge due to comprising of all captured knowledge.

63
Table 7
Average Percentage of Objectives, Reasons, and Risks Acquired from Multiple
Groups of Experts When Compared to a Six Subject Matter Expert Gold Standard
Protocol

Table 7 indicates that the addition of subsequent experts did not increase the
percentage of information regarding the overall objective of performing the CVC
procedure; therefore a figure is not required. A total of one expert is recommended
based on a 10% marginal utility for the overall objective. The average percentage of
acquired knowledge in the areas of Risks and Reasons increases with each additional
expert up to six.

Figure 7. Percentage of knowledge acquired of risks as a function of the number
of experts.
Number of
Experts
Objective Reasons Risks

1 100 27.78 19.61
2 100 48.89 38.04
3 100 65 55.29
4 100 77.78 67.45
5 100 88.89 86.27
6 100 100 100

64
Figure 7 shows the percentage of acquired knowledge in the area of risks of not
performing a CVC placement procedure correctly. The percentage of acquired
knowledge of risks increases steadily with each additional expert. A single expert
provided an average of 19.61% of acquired knowledge. Two experts provide an
average of 38.04% while three experts provided an average of 55.29%. Four experts
provided 67.45% and five experts provided 86.27% of acquired knowledge. Six
experts provided 100% of total acquired knowledge in the area of risks of not
performing the CVC procedure correctly.
Figure 8 indicates the average percentage increase of acquired knowledge in the
area of risks as a function of added experts up to six experts. The percentage of
knowledge captured from experts, increases steadily as a result of additional experts.
The average acquired knowledge from a single expert was 19.61%. The addition of a
second and third expert provided an increase of 18.43% and 17.25% respectively.
The addition of a fourth expert indicated an average of 12.26%. The addition of a
fifth expert showed an elevated increase of 18.82% of acquired knowledge. This
increase was caused by the total inclusion of information from expert four, which had
the greatest agreement overall with the six-expert based gold standard. The sixth
expert provided an additional 13.73% increase. The marginal utility of 10% was
achieved at the sixth expert.

65

Figure 8. Average increase of acquired knowledge of risks as a function of added
experts.

Figure 9 shows the percentage of acquired knowledge in the area of reasons and
benefits of performing a CVC procedure correctly. On average, a single expert
provided 27.78% of acquired knowledge. The addition of a second and third expert
increased the acquired knowledge to 48.89% and 65% respectfully. Four experts
provided an average of 77.78% and five experts provided 88.89% of acquired
knowledge. Six experts provided 100% of total knowledge in the area of reasons to
perform the CVC procedure correctly.

Figure 9. Percentage of knowledge acquired for reasons to perform the CVC
procedure correctly.

66

Figure 10. Average percentage increase in acquired knowledge in the area of
reasons based on a six-expert gold standard CVC protocol.

Figure 10 displays the average percentage increase of reasons and benefits for
performing a CVC procedure correctly based on a six-expert CTA CVC procedure
gold standard protocol. On average, a single expert provided an increase of 27.78%
of new knowledge. Adding a second and third expert provided a 21.11% and 16.11%
percentage increase in acquired knowledge respectively. Four experts provided an
average of 12.78% increase, while both groups of five and six experts gave an
increase of 11.11% each. The marginal utility value of 10% was achieved at the sixth
expert.

67
Table 8
Average Percentage of Indications, Contraindications, and Standards Acquired from
Multiple Groups of Experts When Compared to a Six Subject Matter Expert Gold
Standard Protocol
Number of
Experts
Indications Contraindications Standards

1 75 45.83 23.33
2 92.5 65 42.67
3 92.5 71.25 59
4 95 78.33 73.33
5 100 91.67 86.67
6 100 100 100

Figure 11. Percentage of Indications to perform the CVC procedure based on a
six-expert gold standard as a function of the number of experts

Figure 11 shows the average increase of acquired knowledge in the area of
indications to perform a CVC placement procedure as a function of the number of
experts up to six experts. Based on a six-expert gold standard, a single expert gave an
average of 79.17% of acquired knowledge. With the addition of a second and third
expert, the knowledge acquired increased to 96.67% and 98.75% respectfully. 100%
total acquired knowledge was achieved with the addition of expert four.

68

Figure 12. The average percentage of acquired knowledge gained as a function of
additional experts in the area of indications to perform a CVC procedure.

Figure 12 shows the average increase of acquired knowledge as a function of an
additional expert. A single expert provided an average increase of 75%. The second
expert provided an increase of 17.5% of acquired knowledge. The third expert did
not provide any increase in acquired knowledge. The fourth expert provided only an
added 2.5% of acquired knowledge. Expert five provided 5% of additional acquired
knowledge while expert six did not provide any additional information. The marginal
utility was achieved with three experts.

69

Figure 13. Percentage of knowledge acquired in the area of contraindications as
a function of the number of experts

Figure 13 displays the average acquired knowledge of contraindications as a
function of the number of experts based on a six-expert gold standard CTA CVC
placement protocol. A single expert provided an average of 45.83% of acquired
knowledge in contraindications on when not to perform the CVC placement
procedure or a particular side of patient to insert the CVC catheter. The third expert
provided an average of 71.25%. With five experts, an average of 91.67% of acquired
knowledge was obtained. Expert six provided 100% of total knowledge in
contraindications.

70

Figure 14. The average percentage increase in the amount of acquired
knowledge in the area of contraindications as a function of the number of experts.

Figure 14 shows the average increase in acquired knowledge as a function of the
number of experts, based on a six-expert gold standard CVC placement procedure in
the area of contraindications (when not to perform a procedure). The average
increase of acquired knowledge from the first expert was 45.83%. When the acquired
knowledge from a second expert was added, an additional 19.17% increase was
observed. The third and fourth expert provided an increase of 6.25% and 7.08%
respectfully. Interestingly enough, the addition of a fifth expert provided an average
increase of 13.33% of new acquired knowledge. The sixth expert provided an
increase of 8.33% of additional knowledge of contraindications. The marginal utility
of 10% threshold was achieved at the third expert.

71

Figure 15. Percentage of knowledge acquired for standards of CVC procedure as
a function of the number of experts.

Figure 15 displays the average increase of acquired knowledge in the area of
standards indicating that the CVC placement procedure was performed correctly. The
first expert provided an average of 23.33% of acquired knowledge. The addition of
the second expert provided 42.67% of acquired knowledge. The addition of expert
three and four provided 59% and 73.33% of acquired knowledge of the standards. A
total of 86.67% of knowledge was noted with the fifth expert and 100% of acquired
knowledge was achieved with the sixth expert.

Figure 16. Average percentage increase of acquired knowledge for standards
based on a six-expert gold standard CVC protocol.

72

Figure 16 indicates the average percent increase of acquired knowledge in the
area of standards as a function of additional experts. A single expert provided
23.33% of new knowledge. The addition of the second expert provided an increase of
19.33% and expert three provided an average increase of 16.33%. The fourth and
fifth experts added 14.33% and 13.33% of acquired knowledge. The sixth expert
provided an increase of 13.33% of acquired knowledge in the area of standards
indicating the procedure was performed correctly. The 10% marginal utility was
achieved at the sixth expert.
Table 9
Average Percentage of Equipment and Tasks acquired from Multiple Groups of
Experts When Compared to a Six Subject Matter Expert Gold Standard Protocol
Number of Experts Equipment Tasks

1 65.48 93.75
2 88.10 100
3 91.07 100
4 92.38 100
5 92.86 100
6 100 100

Table 9 indicates that the addition of the second expert increased the percentage
of information regarding the amount of tasks involved with performing the CVC
procedure to 100%; therefore a figure is not required. A total of two experts are
recommended based on a 10% marginal utility for the required tasks of the CVC
procedure.

73

Figure 17. Average Percentage increase of Gold Standard Equipment as a
function of the number of experts.

Figure 17 shows the average increase of acquired knowledge of necessary
equipment for the CVC procedure, based on the six-expert gold standard protocol. A
single expert gave an average of 65.48% of total equipment recommended to perform
the CVC placement procedure. Three experts gave an average of 91.07% while six
experts gave 100% of total acquired knowledge in the area of recommended
equipment for the CVC placement procedure. There was a significant increase
between a single expert and an additional expert, but there were only small changes
noticed from experts two through five and a slight jump of acquired knowledge of
equipment with the sixth expert.

74

Figure 18. The Average Percentage gain of acquired knowledge of needed
equipment as a function of additional experts

Figure 18 indicates the average percentage increase of acquired knowledge as a
result of adding the knowledge from an additional expert. With the assumption you
have no information with zero expert, a single expert will provide an increase of
53.33% of knowledge acquired in the area of recommended equipment for the CVC
placement procedure. When a second expert is added to the first expert, an average
of 22.22% of acquired knowledge was observed. A third expert added an average of
11.44% of additional knowledge, while the fourth and fifth experts added 2.11% and
8.11% respectfully. The sixth expert added an average of 2.78% of additional
acquired knowledge. The marginal utility of 10% was achieved at four experts.
Summary
On average, a single expert provided 57% of the total knowledge, 70% of the
action steps, and 65% of the decision steps in a central venous catheter six – SME
gold standard CTA protocol.. Table 10 indicates the point of diminishing marginal

75
utility for total knowledge acquired and all the other subsections of items acquired
within a six-expert CVC gold standard protocol.
Table 10: Quantity of Experts Recommended if a 10% Marginal Utility in Knowledge
Acquisition is Expected
Item Measured Number of Experts for
10% Marginal Utility
Total Knowledge Acquired 3
Action Steps 2
Decision Steps 3
Objective 1
Reasons 6
Risks 6
Indications 3
Contraindications 3
Standards 6
Equipment 4
Tasks 2
Total Average 3.5

76
CHAPTER 4: CONCLUSIONS
The purpose of this study was two-fold: to investigate the average amount of
information obtained from interviewing an expert and the average amount of acquired
knowledge from each additional expert up to six experts. As a result of answering the
two research questions, a benchmark will be established for the optimal number of
experts one needs to interview to capture the knowledge and skills required to
complete a central venous catheter placement. The research literature that focuses on
the use of cognitive task analysis and experts has expressed the advantages and
benefits of utilizing experts and the knowledge captured through CTA knowledge
elicitation techniques. However, prior research has not focused on quantifying the
amount of knowledge CTA capture’s with respect to an overall task performance goal
and additionally, there is only one known study that specifically quantifies the
average amount of knowledge and skills captured from interviewing additional
experts (Crispen, 2010). As a result, there is limited research on the number of subject
matter experts needed to capture the acquired knowledge from experts (Chao and
Salvendy, 1994; Crispen, 2010).
Research Questions
Research question #1: How much information about a central venous
catheter placement procedure does a single expert provide when compared to a six-
subject matter expert gold standard protocol?
The results of this study indicate that experts omitted critical information on how
to conduct a central venous catheter placement procedure. The range of acquired

77
knowledge obtained from the experts for total information in comparison to the six
–expert gold standard protocol was 43% to 73%, which means that the experts within
this study omitted a range of ~57%-27% of total information. Although this is not a
focus of this investigation, it is important to note that other research has reported
similar results. For example, Clark, Pugh, Yates, Early, and Sullivan (2008) found
that physicians omitted an average of 68.25% of procedural steps when the data from
their initial interviews was compared to a gold standard. Information regarding
subsets of knowledge was acquired for overall objective, risks of not performing the
procedure correctly, benefits and reasons to performing the procedure correctly,
conditions of when to perform the procedure (indications) and when not to perform
the procedure or a specific side/location (contraindications) and recommended
equipment. The results indicated that, on average, each of the experts were able to
provide the overall objective for the procedure (100%), an average of 70% of the
action steps, and 65% of decision steps. Additionally, each of the experts
interviewed provided an average of 28% of reasons and benefits, 20% of risks of not
performing the procedure correctly, 79% of indications of when to perform the
procedure, 46% of contraindications of when not to perform the procedure or choose
a side/location, and 56% of total equipment for the procedure, when compared to a
six-expert based gold standard protocol.
The findings of the current study are supported by previous research. Crispen
(2010) investigated the effects of the use of CTA in capturing the critical knowledge
from experts on how to conduct an open cricothyrotomy procedure. Crispen (2010)

78
found that a single expert provided only 56% of total knowledge, 66% of action
steps, and 28% of decision steps when compared to a six-expert gold standard on how
to perform an open cricothyrotomy procedure. The results of the current
investigation support previous research that utilized CTA knowledge elicitation
techniques in capturing critical knowledge from experts on specific tasks, in that
experts omit a significant amount of information when asked to describe how to
conduct a procedure (Clark et al., 2008). And yet the experts in this study reported
many more decision steps (65 percent) than in previous studies (28 percent in
Crispen, 2010 and Velmahos et al, 2004). It is possible that the difference is due to
the fact that the catheter procedure examined is highly controversial and undergoing
changes because of medical errors. This controversy may have led the experts
interviewed to become more conscious of the decisions necessary to perform.
Research question 2: How much average ( ) critical information is gained
from each additional cognitive task analysis (CTA) interview in regards to a medical
procedure?
With each additional expert, the amount of acquired knowledge increased overall,
but at a diminishing rate of utility. When comparing the total knowledge acquired
from one to two subject matter experts, there was an average of 16 percent gain in
acquired knowledge. Three subject matter experts provided an average gain of 8% of
additional knowledge. The diminishing marginal utility effect of increased
percentage of acquired knowledge continued with the addition of the fourth expert
(4% increase), but jumped to a 5% increase when 5 subject matter experts were

79
utilized. This unexpected jump in percentage of additional acquired knowledge
from 5 experts could be explained by looking at the individual experts percentage of
acquired knowledge when compared to the gold standard. Expert 4 had the highest
agreement with the six-expert CTA CVC gold standard protocol at 73%. When the
groups of the five subject matter experts were created, the data from expert 4 was
included in five of the six of the averages created with this group. On the other hand,
when groups of 4 subject matter experts were created, expert 4 was include in only 10
of the 15 total groups of non-repeating experts possible. Therefore, the data expert 4
was present in only 75% of the total 4-expert possible groupings. As expected, when
the data from expert 4 was added to any grouping, the average of that group was
higher than groups that did not contain expert 4. The groups were created so no single
expert appeared in any combination more than once. No combinations appeared more
than once in any grouping level (2 SME, 3 SME, 4 SME, and 5 SME combinations).
One of the possible reasons for this unexpected elevated score from expert 4 could be
caused by a recency effect (Nairne, 1988) of performing the CVC procedure close in
approximation prior to their scheduled interview. Other factors include their
involvement with the overall training of resident physicians in how to perform the
CVC procedure. At the hospital where the investigation occurred, surgical residents
are not able to perform a CVC placement without completing a required surgical
skills course. In reviewing the interview transcripts, it was noted that one of the
experts indicated involvement as an instructor within a central venous catheter
placement surgical skills course. Due to the amount of information this expert

80
provided in agreement with the creation of the six-expert gold standard protocol
(73% agreement), this could lead to a possible criteria of who should be considered
an expert for the purposes of a cognitive task analysis interview. Future research
could investigate the relationship between teaching physicians versus non-teaching
physicians and the percentage of agreement with an expert based gold standard
protocol on how to conduct a medical procedure. Although research indicates
teachers often describe ‘what to do’ and not necessarily ‘how to do it’. Sullivan et al.,
(2008) found that while expert surgeons described between 46 to 61% on what to do
in a colonoscopy procedure: they reported only between 26-50% of how to perform
the same procedure. Additionally, Feldon (2004) found that experts have significant
omissions (48-89%) in their accuracy of self-reports. Therefore, honoring an
individual as an expert due to their position as a teacher needs to also include other
contributing factors as well. While the literature provides clear differences in both
the cognitive and task performance abilities between experts and novices (Ericsson et
al., 2006) future research could investigate other characteristics that qualify
individuals as experts.
In reviewing the data collected from each expert, it is noted that not every expert
interviewed provided information in all subset categories. This may be a human
variable associated with semi-structured interviews. While every semi-structured
interview used the same CTA interview protocol that provided the experts an
opportunity to share their knowledge and skills regarding the CVC procedure,
variance due to the individual characteristic differences between the multiple analyst

81
utilized in this investigation might have been a “contaminating variable” (Crandall
et al., 2006 p.261). An analyst could become a contaminating variable during the
interview process by not presenting themselves in a neutral presence and utilizing the
same tone and interpersonal communication actions equally to all individuals
involved in the research project. Since multiple analysts were utilized in this story
certain individual human differences and CTA interview skill level may have been a
contributing factor in the initial capturing of experts knowledge and skills. However,
each expert reviewed the information they provided, represented in a CTA protocol,
for verification of data and agreement purposes. As a result of reviewing the CTA
protocol, five out of six experts provided additional inserts of conceptual and
procedural knowledge and/or modifications to their initial interview data. Since the
experts were given an equal opportunity to provide such detail initially, their additive
and subtractive actions support previous research that experts tend to omit
information when explaining how to conduct a procedure (Clark & Estes, 1996).
Chao and Salvendy (1994) utilized a 10% marginal utility to determine when to
stop obtaining information from experts to solve computer application problems.
They indicated that although additional experts, beyond three, will provide an
increase in acquired knowledge, it is not worth the additional time and effort required
to gain the increase in knowledge from additional experts. The author utilized the
same 10% marginal utility threshold as the point where you would not need to
interview additional experts to acquire the needed knowledge on how to conduct a
central venous catheter placement. The results of the current investigation support

82
Chao and Salvendy (1994) given the significant costs human resources and time
involved with completing the current investigation and the average total acquired
knowledge from experts beyond three experts.
Table 7 indicates the recommended number of experts needed to interview to
capture the conceptual and/or procedural knowledge for each sub-section (i.e. risks,
standard, equipment, etc.) based on a 10% marginal utility. The data analysis
indicated three experts are recommended to capture all items (total acquired
knowledge in completing a CVC procedure. However, the data indicated different
amounts of experts are recommended to capture the knowledge and skills for each
subsection. As a result of averaging all the required experts for each subsection,
excluding total acquired knowledge, the average number of needed experts was 3.6.
For all realistic purposes, when reviewing the calculated average number of experts
needed for all subsections combined, four experts would be required. Similar results
were found by Crispen (2010). Crispen (2010) conducted a CTA investigation
focused on an open cricothyrotomy procedure in investigating how many experts are
needed to capture the critical information. Crispen (2010) found, on average, four
experts were recommended to interview in order to capture all critical knowledge and
skills needed to complete an Open Cricothyrotomy procedure.
Although there are several components to the central venous catheter placement
procedure, an argument can be made that the most critical components of the
procedure are the action and decision steps involved with the CVC protocol. When
reviewing the needed number of experts for both action and decision steps, based on a

83
10% marginal utility, the average number of experts needed is 2.5 experts.
Therefore, interviewing three experts is recommended to capture the desired
expertise.
The results of the current investigation support previous findings that Chao and
Salvendy (1994) obtained in their study. Chao and Salvendy (1994) focused on the
percentage of procedural knowledge (Action and Decision steps) required to
diagnose, debug, and or interpret a computer problem. When reviewing the required
number of experts needed for the action and decision steps for the CVC procedure it
is noted that three experts are required based on a 10% marginal utility. Additionally,
the current investigation provides support to the work previous conducted by Ashton
(1986). Ashton (1986) who built upon the work of R. M Hogarth (1978),
investigating how many experts and which ones are needed to be included in a static
group. Ashton (1986) concluded that group validity increases as a function of
increasing experts, however on average only three experts are needed to achieve the
most improvement of validity possible.
Performing a central venous catheter placement has greater human complications
than fixing a computer problem. Therefore, the analysis conducted within the current
investigation looked beyond just the procedural knowledge acquired from experts.
The current investigation looked critically at the contributions of each subsection
including: indications; contraindications; risks of not performing the surgery
correctly; benefit of performing the surgery correctly; and standards indicating a
correctly performed CVC placement procedure.

84
Summary
A cognitive task analysis interview was conducted with six expert physicians
regarding how to conduct a central venous catheter placement. The transcribed
interviews were coded and converted into individual CTA protocols listing the
knowledge and skills necessary to complete a central venous catheter placement. The
individual expert CTA protocols were combined into a six-expert gold standard
protocol through an iterative process that included analysis by both the experts and
analysts;, aggregation of individual reports into one, and a final validation of a
completely detailed procedure through a final expert review.
The results of the study indicated that on average experts provided approximately
68% of total acquired knowledge when compared to a six-person gold standard
protocol. When the acquired knowledge is added to the initial expert, it was found
that a second expert provided an average of additional 16% of acquired knowledge
and the third expert provided approximately 7% more information. The results of the
student indicate that four experts were needed to reach a 10% point of diminishing
marginal utility for the acquisition of additional knowledge based on combining all
sub-sections of a six-expert gold standard CTA CVC protocol, however only three
experts are recommended to acquire the most critical action and decision steps. This
study is one of the first investigations that have determined a required number of
experts needed to capture all the required knowledge to conduct a CVC procedure.
Future research will be needed to validate these findings.

85
Limitations and Implications for Future Research
The physicians who participated within the study were considered experts by their
longevity of being a physician, the number of times they have performed the central
venous catheter placement, and their activities in training new surgical residents.
However, no established criteria of how many times a physician needed to perform
this CVC procedure, in combination with years of experience, were set as parameters
in establishing quantitative data signifying who is an expert. Although each of the
physicians verbally expressed how many times they conducted the CVC procedure
and the approximate time they last performed the procedure, the exact quantity of
procedures performed and the exact date of their last performance of the procedure
was not obtained from all physicians. Therefore any correlation or regression analysis
that focused on the number of procedures conducted and future percentage of total
gold standard items or last date of performing the procedure and the percentage of
total gold standard items to measure the effects of time and recall was not available to
the current investigation. Future research may incorporate these demographic
characteristics to better quantify qualities of an expert physician in relation to other
measures of expertise for a more holistic view of what constitutes a medical expert.
Another limitation to this study was the expert’s ability to review documents with
a discerning eye checking for agreements and disagreements with their original and
gold standard protocols. If research indicates that experts have limited capacity to
recognize information they omit when providing instruction, there is reason to believe
that they would have similar difficulty recognizing omissions in a self-report based

86
protocol. Although every physician was notified that they would have the
opportunity to review the formatted protocols to verify accuracy and their ability to
make addition or deletions, only five out of six experts added or made any
modifications. One expert indicated the protocol looked fine the way it was. This
leads to a question; does the level of complexity cause the experts to have difficulty
in reviewing the document, or is time and effort on the part of the expert a
confounding variable? Johnson et al., (2006) acknowledged that in their iterative
verification and editing process of having the experts validate the acquired
knowledge; minimal to no response was provided by the experts in the second round
of editing. They concluded that this was primarily due to the busy schedules of the
expert radiologists and a limited understanding of the potential benefits CTA can
offer to training.
Finally, a significant limitation to this investigation could be tied to the context of
the procedure. While the current investigation indicates that three expert surgeons
need to be interviewed to capture the necessary knowledge and skills for a CVC
procedure, Crispen (2010) recommended that four expert surgeons are sufficient for
the Open Cricothyrotomy procedure. If the same CTA methodologies were utilized
for both the current study and Crispen (2010), then why were different numbers of
experts recommended? One of the contributing factors could be the frequency of the
procedures performed by the experts. While the Open Cricothyrotomy’s is
infrequently performed, the time delay in between procedures may affect the experts
ability to recall specific details conducted during the procedure.

87
The central venous catheter procedure is more frequently performed in
comparison to the Open Cricothyrotomy. Due to the frequency surgeons perform the
CVC procedure, it would be assumed that the experts would omit a greater percentage
of their knowledge and skills as a consequence of their automated knowledge.
However, the data indicated that the experts provided more critical information (70%
- action steps, and 65%- decision steps) than prior research has found (Clark et al.,
2008). Clark et al., (2008) found that expert trauma surgeons provided only 30% of
the critical information necessary to complete a medical procedure. A confounding
variable could be the increase of interest in the medical field related to the central
venous catheter procedure within the past year. The central venous catheter
procedure has recently become a procedure that requires physicians to pass a surgical
skills course in order to complete the procedure on a live patient and is continually a
topic of interest in the research literature.
Conclusion
This study aimed to contribute to the growing body of knowledge on the use of
cognitive task analysis and the benefits of working towards quantifying the optimal
number of experts needed to capture expert knowledge and the skills required to
perform complex medical procedures. Although this study indicates that three
experts are required to capture the critical information (action and decision steps)
needed to perform a central venous catheter placement procedure, future research will
be required to validate these findings. These findings could benefit future curriculum
development and additional research in the medical/ surgical education field by

88
limiting the costs of CTA by knowing how many experts are recommended to
capture critical information for a procedure. Additionally, Crispen (2010) found four
experts are recommended to capture the critical information related to the Open
Cricothyrotomy procedure, which differs from the results of the current investigation.
Since the primary factor involved with the current CTA research focused primarily on
human factors, future research needs to investigate and establish a standard for time
as a cost of conducting CTA investigations. While there are different CTA techniques
that capture the knowledge and skills from experts, reducing the time and effort
associated with the knowledge bottleneck inherent in the capturing of expert
knowledge to a usable form of knowledge representation- training protocols for
novice surgeons, is a compounding variable in CTA research that needs attention.

89
REFERENCES
Abernethy, B., Poolton, J. M., Masters, R. S. W., & Patil, N. G. (2008). Implications
of an expertise model for surgical skills training. ANZ Journal of Surgery,
78(12) 1092-1095.

Aggarwal, R., Grantcharov, T. P., & Darzi, A. (2007). Framework for Systematic
Training and assessment of technical skills. Journal of American College of
Surgeons, 697-705.

Anderson, J.R. (1996). ACT: A simple theory of complex cognition. American
Psychologist, 51(4) 355-365.

Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., Qin, Y. (2004).
An integrated theory of mind. Psychological Review, 111(4) 1036-1060.

Anderson, J. R. & Fincham, J. M. (1994). Acquisition of procedural skills from
examples. Journal of Experimental Psychology: Learning, Memory, and
Cognition, 20(6) 1322-1340.

Anderson, J. R., Finchman, J. M., Douglass, S. (1997). The role of examples and
rules in the acquisition of a cognitive skill. Journal of Experimental
Psychology: Learning, Memory, and Cognition, 23(4), 932-945.

Anderson, L.W., Krathwohl, D. R., Airasian, P.W., Cruikshank, K. A., Mayer, R. E.,
Pintrich, P. R., Raths, J., & Wittrock, M. C. (2001). A Taxonomy for
Learning, Teaching, and Assessing: A revision of Bloom’s Taxonomy of
Educational Objectives. New York: Longman.

Anderson, J.R., & Schunn, C.D. (2000). In R. Glaser (Ed.), Advances in instructional
psychology (5). Mahwah, NJ: Erlbaum.

Andre, M., Borgquist, L., Foldevi, M., & Molstad, S. (2002). Asking for rules of
thumb: A way to discover tacit knowledge in general practice. Family
Practice, 19(6), 617-622.

Ashton, R. H. (1986). Combining the judgments of experts: How many and which
ones? Organizational Behavior and Human Decision Processes, 38, 405-
414.

Bathalon, S., Martin, M., & Dorion, D. (2004). Cognitive task analysis, kinesiology
and mental imagery: challenging surgical attrition. Journal of American
College of Surgeons, S73.

90

Boshuizen, H.P.A., & Schmidt, H.G. (1992). On the role of biomedical knowledge in
clinical reasoning by experts, intermediates and novices. Cognitive Science,
16, 153-184.

Cauraugh, J. H., Martin, M., Martin, K. K. (1999). Modeling Surgical Expertise for
motor skill acquisition. The American Journal of Surgery, 177, 331-336.

Chao, C-J. & Salvendy, G. (1994). Percentage of procedural knowledge acquired as a
function of the number of experts from whom knowledge is acquired for
diagnosis, debugging, and interpretation tasks. International Journal of
Human-Computer Interaction, 6(3), 221-233.

Chi, M., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. Sternberg
(Ed.), Advances in the psychology of human intelligence (Vol. 1). Hillsdale,
NJ: Erlbaum.

Clark, R. E. (2004). Design Document For A Guided Experiential Learning Course.
Final report on contract DAAD 19-99-D-0046-0004 from TRADOC to the
Institute for Creative Technology and the Rossier School of Education.

Clark, R. E. (2008, October 26). Knowledge and test types. Unpublished manuscript.
University of Southern California, Los Angeles, CA.

Clark, R. E. & Estes, F. (1996). Cognitive task analysis. International Journal of
Educational Research, 25, 2, 403-417.

Clark, R.E., Feldon, D., Van Merriënboer, J.J.G., Yates, K., & Early, S. (2008).
Cognitive task analysis. In J.M. Spector, M.D. Merrill, J.J.G. van
Merriënboer, & M.P. Driscoll (Eds.), Handbook of Research on Educational
Communications and Technology (3rd ed.). Mahwah, NJ: Lawrence Erlbaum
Associates.

Clark, R. E., Pugh, C. M., Yates, K. A. and Sullivan, M. (2008). The use of cognitive
task analysis and simulators for after action review of medical events in Iraq.
Technical Report 5-21-2008 developed for the Center for Cognitive
Technology, Rossier School of Education, University of Southern
California. Retrieved from http://www.cogtech.usc.edu

Cooke, N. J. (1994). Varieties of knowledge elicitation techniques. International
Journal of Human-Computer Studies, 41, 801-849.

91
Cowan, N. (2001). The magical number four in short-term memory: A
reconsideration of mental storage capacity. Behavioral & Brain Sciences, 24,
87-114

Crandall, B. & Getchell-Reiter. K. (1993). Critical decision method: A technique for
eliciting concrete assessment indicators from the intuition of NICU nurses.
Advances in Nursing Science, 16(1), 42-51.

Crandall, B., Klein, G., & Hoffman, R.R. (2006). Working Minds: A practitioners
guide to cognitive task analysis. MIT Press: Cambridge, Massachusetts.

Crispen, P. D. (2010). Identifying the Point of Diminishing Marginal Utility for
Cognitive Task Analysis Surgical Subject Matter Expert Interviews.
Unpublished doctoral dissertation presented to the faculty of the USC Rossier
School of Education, University of Southern California.

De Jong, T. & Ferguson-Hessler, M. G. M. (1996). Types and qualities of knowledge.
Educational Psychologist, 31, 2, 105-113.

Ericsson, K. A. (2006). The influence of experience and deliberate practice on the
development of superior expert performance. In K. A. Ericsson, N.
Charness, P. Feltovich, and R. R. Hoffman, R. R. (Eds.). The Cambridge
Handbook of Expertise and Expert Performance (pp. 685-706). Cambridge,
UK: Cambridge University Press.

Ericsson, K. A., Charness, N., Feltovich, P. J., Hoffman, R. R. (2006). The
Cambridge Handbook of Expertise and Performance. Cambridge University
Press: New York, NY.

Ericsson, K. A. & Kintsch, W. (1995). Long-term working memory.
Psychology. Review. 102, 211–245.

Expert Knowledge Solutions (2009). Cogntitive task analysis training aid.

Fackler, J.C., Watts, C., Grome, A, Miller, T., Crandall, B., Pronovost, P. (2009).
Critical care physician cognitive task analysis: an exploratory study. Critical
Care, 13(2), 1-8.

Feldon, D. F. (2004) Inaccuracies in expert self report: Errors in the description of
strategies for designing psychology experiments. Unpublished doctoral
dissertation, Rossier School of Education, University of Southern California,
USA.

92

Feldon, D. (2007). The implications of research on expertise for curriculum and
pedagogy. Educational Psychology Review, 19, 91-110.

Gourlay, S. N. (2004). ‘Tacit knowledge’: The variety of meanings in empirical
research.. 5th European Organizational Knowledge, Learning and
Capabilities conference; Innsbruck, Austria, 2-3 April.

Grantcharov, T.P. & Reznick, R.K. (2008). Teaching rounds: Teaching procedural
skills [Electronic Version]. BMJ: British Medical Journal 336, 1129-1131.

Grunwald, T. Clark, D. Fisher S.S., McLaughlin, M. Narayanan, S., Piepol, D.
(2004). Using cognitive task analysis to facilitate collaboration in
development of simulator to accelerate surgical training. Proceedings of the 
12th Medicine Meets Virtual Reality Conference, 114‐120. 
 
Hahn, B. B., E. Rall, and D. W. Klinger, (2003). Cognitive task analysis of the
warning forecaster task. Report RA1330-02-SE-0280. [Available from Klein
Associates Inc., 1750 Commerce Center Blvd., North Fairborn, Ohio 45324-
6362.]

Halsted, W.S. (1904). The training of surgeons. Bulletin of Johns Hopkins Hospital,
15, 267-275.

Hamdorf, J. M. & Hall, J.C. (2000). Acquiring surgical skills. British Journal of
Surgery, 87, 28-37.

Hinds, P. J. (1999). The curse of expertise: The effects of expertise and debiasing
methods on prediction of novice performance. Journal of Experimental
Psychology: Applied, 5, 205-221.

Hinds, P. J., Patterson, M. & Pfeffer, J. (2001). Bothered by abstraction: The effect of
expertise on knowledge transfer and subsequent novice performance. Journal
of Applied Psychology, 86(6), 1232-1243.

Hoffman, R. R. & Militello, L. G. (2009) Perspectives on Cognitive Task Analysis:
Historical origins and modern communities of practice. Psychology Press.
New York, NY.

Hoffman, R.R., Crandall, B. W., & Shadbolt, N.R. (1998). Use of the critical decision
method to elicit expert knowledge: A case study in cognitive task analysis
methodology. Human Factors, 40(2), 254-276.

93
Hoffman, R.R., Shadbolt, N. R., Burton, A.M., & Klein, G. (1995). Eliciting
knowledge from experts: A methodological analysis. Organizational
Behavior and Human Decision Processes, 62(2), 129-158.

Jacklin, R., Sevdalis, N., Darzi, A., & Vincent, C. (2008). Mapping surgical practice
decision making: an interview study to evaluate decision in surgical care. The
American Journal of Surgery, 195, 689-696.

Johnson, C., Cumming, G., Omodei, M. (2008). How worst case scenarios are
considered by bushfire fighters: An interview study. Presented at the
International Bushfire Research Conference, Adelaide, Australia.

Johnson, S., Healey, A., Evans, J., Murphy, M., Crawshaw, M., & Gould, D., (2006).
Physical and cognitive task analysis in interventional radiology. Clinical
Radiology, 61, 97-103.

Klein G. A., Calderwood, R. & MacGregor, D. (1989). Critical decision method for
eliciting knowledge. IEEE Transactions on Systems, Man, Cybernetics, 19,
462-472.

Knowlton, B. J. & Moody, T. D. (2008). Procedural learning in humans. Learning
and Memory: A Comprehensive Reference, 3, 321-340.

Latorella, K.A., Pliske, R., Hutton, R., & Chrenka, J. (2001). Cognitive task analysis
of business jet pilots' weather flying behaviors: Preliminary results. NASA-
TM-2001211034. NASA Langley Research Center. Hampton, VA.

Luker, K. R., Sullivan, M. E., Peyre, S. E., Sherman, R., & Grunwald, T., (2008). The
use of cognitive task analysis-based multimedia program to teach surgical
decision making in flexor tendon repair. The American Journal of Surgery,
195, 11-15.

Markus, M. L. (2001). Toward a theory of knowledge reuse: Types of knowledge
reuse situations and factors in reuse success. Journal of Management
Information Systems, summer, 18(1), 57-93.

Marshall, R.L., Smith, J.S, Gorman, P.J,, Krummel, T. M., Haluck, R. S., & Cooney,
R. N. (2001) . Use of a human patient simulator in the development of
resident trauma management skills. Journal of Trauma Injury, Infection and
Critical Care, 5, 17–21.

94
Meyers M.O., Meyer A.A., Stewart, R.D., Dreesen, E.B., Barrick J., Lange, P.A.,
& Farrell, T.M, (In press). Teaching technical skills to medical students
during a surgery clerkship: Results of a small group curriculum. Journal of
Surgical Research.

Nairne, J. S. (1988). A framework for interpreting recency effects in immediate serial
recall. Memory & Cognition, 16, 4, 343-352.

O’Hare, D., Wiggins, M., Williams, A., & Wong, W. (1998). Cognitive task analysis
for decision centered design and training. Ergonomics, 41(11), 1698-1718.

Ormrod, J. E. (2008). Educational Psychology: Developing learners. New Jersey:
Pearson.

Paris, S. G., Lipson, M. Y., & Wixson (1983). Becoming a strategic learner.
Contemporary Educational Psychology, 8, 293-316.

Patel, V.L. & Groen, G.J. (1986). Knowledge based solution strategies in medical
reasoning. Cognitive Science, 10, 91-116.

Patterson, E.S., Woods, D. D., Tinapple, D., Roth, E. M. (2001). Using cognitive task
analysis (CTA) to seed design concepts for intelligence analysts under data
overload. In Proceedings of the Human Factors and Ergonomics Society 45th
Annual Meeting, Minneapolis, MN: October 8-12, 2001.

Pavlik, P.I., & Anderson, J.R. (2008). Using a model to compute the optimal schedule
of practice. Journal of Experimental Psychology: Applied, 14(2), 101-117.

Peters, J. H., Fried, G. M., Swanstrom, L. L., Soper, N. J., Sillin, L. F., Schirmer, B.,
& Hoffman, K. (2003). Development and validation of a comprehensive
program of education and assessment of the basic fundamentals of
laparoscopic surgery. Surgery, 135(1), 21-27.

Pejtersen, A. M. (1989). A Library system for information retrieval on a cognitive
task analysis and supported by an icon-based interface. Association for
Computing Machinery, 40-47.

Pirolli, P. L. & Anderson, J. R. (1985). The role of practice in fact retrieval. Journal
of Experimental Psychology: Learning, Memory, and Cognition, 11 (1), 136-
153.

95
Porte, M. C., Xeroulis, G., Reznick, R. K., Dubrowski, A. (2007). Verbal feedback
from an expert is more effective than self-accessed feedback about motion
efficiency in learning new surgical skills. The American Journal of Surgery,
193, 105-110.

Polanyi M (1962) Personal Knowledge: Towards a Post-Critical Philosophy
University of Chicago Press.

Redding, R. E., Cannon, J. R., Lierman, B. C., Ryder, J. M., Purcell, J. A. &
Seamster, T. L. (1991). The Analysis of Expert Performance in the Redesign
of the en route Air Traffic Control Curriculum. In: Proceedings of the Human
Factors Society, 35
th
Annual Meeting.

Reznick, R. K., & MacRae,H. (2006). Teaching surgical skills-Changesin the wind.
New England Journal of Medicine, 355, 2664-2669.

Sachdeva, A. K.,Bell, R. H.,Britt, L.D., Tarpley, J. L., Blair, P.G., Tarpley, M. J.
(2007). National efforts to reform residency education in surgery. Academic
Medicine, 82 (12), 1200-1210.

Schraagen, J. M., Chipman, S. F., and Shalin, V. (2000). Introduction to Cognitive
Task Analysis. In S. Chipman, V. Shalin (Eds.) Cognitive Task Analysis.
Mahwai, NJ, Lawrence Erlbaum Associates.

Schraw, G. (2006). Knowledge: Structures and processes. In P. Alexander & P.
Winne, (Eds.) (2006). Handbook of educational psychology (2
nd
ed.).
Mahwaj, NJ, Lawrence Erlbaum Associates.

Schunk, D. H. (2000). Learning theories: An educational perspective. Upper Saddle
River, NJ: Prentice-Hall.

Scott, D. J., Cendan, J. C., Pugh, C. M., Minter, R. M., Dunnington, G. L., and Kozar,
R. A. (2008). The changing face of surgical education: Simulation as the new
paradigm. Journal of Surgical Research, 147(2),189–193.

Sleeman, D.H. & Brown, J.S. (Eds.). (1982). Intelligent tutoring systems. New York:
Academic Press.

Spender, J. C. (1996). Organizational knowledge, learning and memory: Three
concepts in search of a theory. Journal of Organizational Change
Management, 9(1), 63-78.

96
Stadler, M. A. (1989). On learning complex procedural knowledge. Journal of
experimental psychology: Learning, Memory, and Cognition, 15(6) 1061-
1069.

Sullivan, M. E., Brown, C.V. R., Peyre, S. E., Salim, A., Martin, M., Towfigh, S., &
Grunwald, T. (2007).The use of cognitive task analysis to improve the
learning of percutaneous tracheostomy placement. The American Journal of
Surgery, 193, 96-99.

Sullivan, M. E., Ortega, A., Wasserberg, N., Kaufman, H., Nyquist, J., & Clark, R.
(2008). Assessing the teaching of procedural skills: can cognitive task
analysis add to our traditional methods? The American Journal of Surgery,
195, 20-23.

Tirapelle, L.A. (2010). The effect of cognitive task analysis based instruction on
surgical skills expertise and performance. Unpublished doctoral dissertation
presented to the faculty of the Rossier School of Education, University of
Southern California.

Vadcard, L. & Luengo, V. (2004). Embedding knowledge in the design of an
orthopedic surgery learning environment. In: CALIE04, International
Conference on Computer Aided Learning in Engineering education,
Grenoble: 16-18 fevier, 2004, 1-6.

Velmahos, G. C., Toutouzas, K. G., Sillin, L. F., Chan, L., Clark, R. E., Theodorou,
D., & Maupin, F. (2004). Cognitive task analysis for teaching technical skills
in an inanimate surgical skills laboratory. The American Journal of Surgery,
187, 114-119.

Vozenilek, J., Huff, S. Reznek, M., & Gordon, J. A. (2004) See one, do one, teach
one: advanced technology in medical education. Academic Emergency
Medicine, 11, 1149-1154.

Webster’s New World College Dictionary (4
th
ed.). (1999). New York, NY:
Macmillan.

Yates, K. (2007). Towards a taxonomy of task analysis methods: A search of
cognition and task analysis interactions. Unpublished doctoral dissertation
presented to the faculty of the USC Rossier School of Education, University
of Southern California.

Yates, K. A. and Clark, R. E. (In review). Cognitive Task Analysis. International
Handbook of Student Achievement.

97

Yates, K. A., & Feldon, D. F. (in press). Towards a taxonomy of cognitive task
analysis methods for instructional design: Interactions with cognition.
Theoretical Issues in Ergonomics Science.

98
APPENDIX A

Non- Repeating combinations of SME Protocols

Two Subject Matter Expert Non- Repeating Pairs:

AB, AC, AD, AE, AF, BC, BD, BE, BF, CD, CE, CF, DE, DF, EF

Three Subject Matter Expert Non- Repeating Groups:

ABC, ABD, ABE, ABF, BCD, BCE, BCF, CDE, CDF, ACD, ACE, ACF, ADE,
ADF, AEF, BDE, BDF, BEF, CEF, DEF

Four Subject Matter Expert Non- Repeating Groups:

ABCD, ABCE, ABCF, ACDE, ACDF, ADEF, BCDE, BCDF, BDEF, CDEF,
ACEF, ABDE, ABDF, ABEF, BCEF

Five Subject Matter Expert Non-Repeating Groups:

ABCDE, ABCDF, ABDEF, ACDEF, ABCEF, BCDEF

Six Subject Matter Expert Group:

ABCDEF

99
APPENDIX B

CTA Coding Scheme

Action = OA
Conditions:
Indications = OCI
Contraindications = OCC
Accuracy = OSA
Time = OST
Overall Reasons for Conducting Procedure = R
Reason for Action or Decision step = r
Equipment and Materials = EM
Declarative Knowledge:
Concepts = DC
Processes = DPR
Principles = DPN
Procedural Knowledge:
Classification = PDC
Modification = PDM
Sensory Cues:
Hearing = SH
Seeing = SS
Touching = ST
Steps:
Action = A
Decision step criteria for deciding = DSC (“IF” statements)
Decision step alternatives = DSA (“Then” statements)
Uncertain = Δ (Delta triangle)

100
APPENDIX C

Six-Expert Gold Standard Central Venous Catheter Placement Procedure Protocol

Cognitive Task Analysis:
Central Venous Catheter Placement

Gold Standard Protocol
Procedure Description

Task Analysis

January 2010

Procedure Title: Central Venous Catheter Placement

Experts: Surgeons A, B, C, D, E, F

Task Analysts: Craig Bartholio, Joon Kim, Eko Canillas

Interview Location: USC Keck Medical Center

1. Objective
(Write the overall terminal (end) performance objective of this procedure, using an
“action” verb that describes what a resident does.)
• Perform a central venous catheter placement (A14; C8 F2; D10) to provide
central venous access (A61; B42; E21) quickly administer drugs (D16), fluids, or bloods to a
patient (B72) and to monitor the patient (D16).
2. Reasons
(What is the risk for not performing well? What are the benefits to the trainees if they do
a good job?)
The risks of not inserting a central venous catheter properly could be:
• Inadvertent carotid injury (IJ site) (A123)
• Inadvertent pneumothorax (A132; D24)
• Artery is inadvertently cannulated (subclavian site) (A308-309; D25)
• Ectopic beats (A358)
• Widening of the mediastinum (A420)
• A hematoma (A420; D27)
• Death (D33)
• Damage to vein (D326)
• Air embolus (D526)
• Sepsis (D32)
• Bleeding (D24)

101
• Physical damage to the area itself (D27)
• Allergic reaction (D29)
• Infection (D32)
• Trauma to the vein (D326; D27)
• Getting into the femoral artery by mistake (D391)
• Guide wire becoming embolized (D545)

The benefits of inserting a central venous catheter properly are:
• Venous access (C8;F5) to provide fluids and drugs (C8;E24)
• Access to central circulation to allow for hemodynamic monitoring (C8)
• Deliver blood and nutrition (E24)
Indications: Perform the procedure if the patient exhibits one or more of these
indications. .
• Patient requires long-term IV access (i.e. for administering medications and
nutrition, etc.) (A45; B80; C25; D28; D42; E61; F6)
• Patient requires large volume resuscitation (A48; B75; E33; F8)
• Patient has inadequate peripheral IV access (A54; B83; C91; D16; F5)
• Patient requires direct access into central circulation system for Hemodynamic
monitoring (C96; D16;D44; E53; F8)
Contraindications: Do not perform the procedure if the patient exhibits one or more of
these contraindications. List the symptoms or conditions that make the particular treatment or
procedure inadvisable.)
• Peripheral access is available for the patient (D49)
• Patient is coagulopathic (A56; B101; C126; E79; F15)
o Standard: First correct coagulopathy before performing CVC placement (E80)
4. Standards
(Are there any time limits, efficiency, or quality standards that must be met when
performing this procedure? What are the observable indications of a successful procedure?)
• The time frame for the CVC procedure ranges from 2 minutes (F50) -10 minutes
(E86) with an average of 5 minutes (A445; B1101; E86)
• Observable indications of success: Chest x-ray (E92) should show:
1. The catheter is in the superior vena cava. (B1081)
2. Clear lung fields (B1076)
3. Easily draw blood back from the catheter (E89)
4. Easily flush fluid into the catheter (E90)
5. Equipment
(What supplies and materials must be available to perform the task?)

102
• X-ray (A72; B1066; C48;F175; D381; E92)
• Personal protective gear (A91; D60)
o Cap (A91;C387; F309; D174; E97)
o Gown (A91;C387;F317; D174; E97)
o Sterile Gloves (A91;B171;C388;F310; D174; E97)
o Mask (A91;C387; D 174;F309; E97)
o Eye protection (C411; F310; D174; E99)
o Full sheet drape (C770; D182; F310; E100)
o Sterile Prep (includes chlorhexidine) (A92; B172; C383; D180; E101)
• Stethoscope (C719)
• Ultrasound machine (A123; B176; C232;; D112; E264; F408)
• Local anesthesia/one percent lidocaine (A188, A196;F527;D179; E298)
• Central Line Kit (A196;B118;C40;F309; D177; E97)
o Introducer needle (A69; A101; D305)
o Guide wire (A69; D255; E105)
o Central Venous Catheter (A70; D204; E105)
 Single lumen central venous catheter-large lumen for large volume fluid or blood
administration (A638, A644-A645; E44; D209)
 Triple lumen- when large volume is no longer needed, when patient needs
multiple medications (A647-A649;C577; D222; E369)
o Syringe (A101; D178; E104; F629)
o Tubing (B1054)
o Finder needle/20 gauge needle (A527)
o Dilator (A656-A657; D537; E105)
• Cardiac monitor (A355; B812)
• Scalpel (A380; F598; E105)
• Sterile saline (A409; F309; D192)
• Sutures (A411; D194; F631; E108)
• Sterile Dressing (A412; B1045; C773; D639; E107)
• Sterile towels, (F309), Extra towels (F309; D157), Large sheet (F310)
• Table to place equipment (F311)
• External lighting (D182)
• Non-scrubbed assistant (D176) (Optional)

103
• Extra kit (D183) (Optional)

6. Task List
1. Inform patient (B499; C38; D155; F120), Prepare Yourself (A90; B251; C39; D60;
E153; F133) & Equipment (B156; C39; D59; E153; F381)
2. Decide where to place the CVC from among three anatomic locations (Internal Jugular,
Subclavian, Femoral Vein) (A92; B254; C34; D49; E131; F82)
3. Prepare Patient (A92; B205;C40; D167; E275; F120)
4. Locate the vessel
5. Insert Introducer needle into the vein (A528; B257; D305; E104; F154)
6. Insert central venous catheter into the vessel (A70; B285; C44; D62; E154; F170)
7. Verify the placement of CVC (A71; B1066; C43; D64; E389; F175)
8. Secure catheter to the skin (A41; B296; C44; D63; E155; F179)

7. Procedure for Central Venous Catheter Placement

Task 1: Inform patient (B499; C38; D155; F120), Prepare Yourself (A90; B251;
C39; D60; E153; F133) & Equipment (B156; C39; D59; E153; F381)

For both single lumen and multi-lumen catheters

Step 1.1: Determine the conscious state of the patient (A92; B205; C441; D286; E208,
E212; F286)

IF the patient is awake, THEN inform the patient of the procedure (B499; C38; D155;
F120) AND obtain oral and/or written consent from the patient (D154; F118), AND go to
step 1.2

IF the patient is unconscious OR non responsive, THEN go to Step 1.2

Step 1.2: Prepare yourself by donning protective gear (C39), i.e. cap, gown, (A91; D174;
E284; F133) gloves (A91; B171; F317), mask (A91; C39), and eye protection (D Edited
Text) using universal precautions (A91; D174; E284)

Step 1.3: Determine the type of catheter to use (B156; C577; D493; E369; F109)

IF it is an emergency situation, THEN use whatever kit is available (E167) AND go to
Step 1.4

Otherwise determine whether to use a single or multi-lumen kit (E168) based on the
following:
IF:
• The patient requires large volumes of blood or medication (F282), OR
• The patient is unstable (F286), OR

104
• A Swan Ganz is required (F284) to monitor fluid and hemodynamic
parameters or temporary pacemaker, or for general resuscitation (F287),
THEN use a single lumen (B157) AND go to Step 1.4

IF the patient requires:
• Administering multiple medications over time (B158; F287), OR
• Total Parental Nutrition (TPN) therapy (B160), OR
• Antibiotic therapy (B160), OR
• The patient is stable (B163) and requires medications that can only be given
through a large bore vein (F287),
THEN use a multiple lumen catheter AND go to Step 1.4

IF you are unsure whether to use a multi-lumen catheter OR a single lumen catheter,
THEN choose a single lumen catheter (E185)
Standard: A single lumen catheter can accommodate a multi-lumen catheter (D236;
E183)

Step 1.4: Gather (D262; F308) and prepare all equipment needed for CVC procedure
(B478; C40; D175; E283; F381) by checking contents and usability (C40; B478; D175;
E283; F308)

For single lumen catheter:

Step1.4.1: Insert the dilator through the back of the catheter

For Multi-lumen catheters:

Step 1.4.2: Unlock the brown port

For both single lumen and multi-lumen catheters:

Step 1.5: Pull scalpel in and out of protective sheath (E293; F393)

Step 1.6: Move wire back and forth from roll (E288; F389)

Step 1.7: Determine if the Introducer needle will accommodate guide wire (C468), by
passing the wire through the needle.

IF wire successfully passes through the Introducer needle, THEN assemble a 10 ml
syringe to the Introducer needle A549; F382), AND go to step 1.9.

IF the wire does not pass through the Introducer needle, then start with a new kit (R).

Step 1.8: Place a 5 ml syringe on your Finder needle (A528; C464; D288)

Step 1.9: Prime catheter line(s) with saline (D526; F395)

105

Step 1.10: Place all equipment near patient or on a table (F383).

Task 2: Decide where to place the CVC from among three anatomic locations (Internal
Jugular, Subclavian, Femoral Vein) (A92; B254; C34; D49; E131; F82)

For both single lumen and multi-lumen catheters

Step 2.1: Decide where to place the central venous catheter (A92, B334; C34; D149;
E131; F82)

Select the subclavian location, unless any of the following occurs:

IF
• You are not familiar with the anatomy or anatomy is distorted (i.e. swelling,
trauma, broken bone, infection, burns) (A57; B110; E76; F30)
• You can’t find subclavian vein (A140), OR
• It’s impossible to cannulate the subclavian vein (A140), OR
• You cannot pass the guide wire during the subclavian procedure (A140), OR
• You have a coagulopathic patient and are concerned with subclavian bleeding
(C279), OR
• You are concerned about possible pneumothorax (C217),

THEN choose internal jugular location unless any of the following occurs:

IF
• You are not familiar with the anatomy or anatomy is distorted (i.e. swelling,
trauma, broken bone, infection, burns) (A57; B110; E76; F30)
 The patient’s internal jugular is unavailable (A140; B363), OR
 The patient has injury to internal jugular vein, OR
 The patient has thrombosis to the internal jugular vein, OR
 The patient has a C-Spine collar/injury (D150; F329),

THEN, choose the femoral vein location, unless the following:

IF
• You are not familiar with the anatomy or anatomy is distorted (i.e. swelling,
trauma, broken bone, infection, burns) (A57; B110; E76; F30)
• The femoral vein is not available, OR
• There is pelvic trauma,

THEN, you are unable to obtain central venous access.

106

Step 2.2: Decide between left OR right side of the site selected for patient to begin CVC
procedure (B208; D125; E249)

For the subclavian location, the left side is preferred, unless (R):

IF
• The patient has a right-sided pneumothorax, OR
• The patient has a venous thrombosis on the left side, OR
• There is an arterial or venous injury to the left side,
THEN use the right side.

For the internal jugular location, the right side is preferred (R), unless:

IF the patient has a left-sided pneumothorax, THEN use the left side.

For the femoral vein location, there is no left or right preference, unless:

If there is injury or thrombosis on one side, THEN use the other side.

Task 3: Prepare Patient for CVC procedure (A92; B205; C40; D167; E275; F120)

For both single lumen and multi-lumen catheters

Step 3.1: Determine how to position the patient
IF the subclavian location is selected, THEN position patient in a Trendelenburg position
(A169; F121), placing a roll of towels underneath the shoulder to open up the area (D157)
and have assistant hold down patient arm at side (A169; B399; C394) AND go to Step 3.2
IF the internal jugular location is selected, THEN position patient in a slight
Trendelenburg, head turned away from you (B398; D159; E433; F328) with their arms at
their side (A169; B397; C394; E197) AND go to Step 3.2
IF the femoral location is selected, THEN position patient in a supine position (A169;
B409; E228; F121) with legs rotated out to open up the area (A568; F441), AND go to Step
3.2
Step 3.2: Remove patient’s gown (D161; F315); drape the patient from head to toe
(C389; D277; F123) lower guard rails (E270) and position bed at a height at which you are
comfortable to perform the procedure (E271).
Step 3.3: Clean area with a sterile prep solution. (A92; B475; D167; E275; F316)
Step 3.4: Administer local anesthesia (A194; B503; C441; D284; F527) unless the
patient is unconscious.

107
Procedure for Subclavian Vein Location (A)

Task 4A: Locate the subclavian vein using physical landmarks

For both single-lumen and multi-lumen catheters

Step 4A.1: Palpate the clavicle and identify a point 2/3 distal from the sternal notch
(A245; C515; D281) on the clavicle and place thumb of non-dominate hand just inferior to
the clavicle at that position (A245; B1128; C515; D281; F136) into the Delta-pectoral groove
(E297), one to two finger-breadth(s) below the clavicle (D281; E297; F136)

Task 5A: Insert Introducer needle into the subclavian vein (A528; B257; D305;
E104; F154)

For both single lumen and multi-lumen catheters

Step 5A.1: Insert the Introducer needle at the location of your thumb.
Standard: Subclavian vein sits deep (A283) underneath the clavicle (D372) therefore
keep the needle in a plane parallel to the floor to avoid inadvertent puncture of the apex of the
lung (A248; C519; F476)

Step 5A.2: Advance the Introducer needle, under aspiration (A257; C522; D450; E321;
F160, F480), towards the sternal notch (A257-258; B1128; E316a) until it strikes the inferior
surface of the clavicle (A253; C519; E316)

Step 5A.3: Pull back 1-2cm, angle the needle down, and guide the Introducer until you
can feel it pass underneath the clavicle (A254; B1134; C520; E317). Move forward to the
subclavian vein (D373) until you see a nice drawback of blood into your syringe (D294)
AND go to step 5B.6

IF you don’t get a flash of blood in your syringe, go back to Step 5A.1 (B1141)

Step 5A.4: Confirm proper placement of Introducer needle in the subclavian vein by
removing the syringe from the needle (A347; B670; C563; D393; E336; F161) and visually
assess presence of blood and blood flow

IF
You have continuous and steady (not pulsating) bleeding (A348), OR
You have dark slow dripping blood (D396; E337)
THEN, assume that the needle is in the subclavian vein (A348; D396; E337)
AND go to step 5A.5

IF
You see pulsatile, red blood (D398; E337)

108
THEN assume your needle is in the artery AND pull needle out (E340), apply
pressure (E342) with a 4 by 4 dressing (C617) for 5 minutes (C618), go back to Task 5B
AND try again on the same side without re-prepping (D399), OR repeat procedure using the
other side (B670; C613)

Step 5A.5: Once the subclavian vein is cannulated, maintain a still (D461) position of
needle by firmly holding the needle at the hub with your non-dominate hand (D459; F536) to
keep it in position (A345; B789; D459; F162)

Step 5A.6: Place thumb over top of needle to avoid air being introduced into the vein
(A350; C563; D459)

Task 6A: Insert central venous catheter into the vessel. (A70; B285;
C44; D62; E154; F170)

For both single lumen and multi-lumen catheters

Step 6A.1: Insert the flexible side of guide wire (D558) easily (A354; B744; D672) into
the Introducer needle (A353; B675; C566; D462; E350; F154) while always maintaining one
hand on the guide wire (C566; E edited text)
Standard: Easily is a standard Leave a portion of the guide wire showing at the top of
the Introducer needle. Do not insert the guide wire completely (A361; B844; D565; E353) or
it may cause heart arrhythmias (D558).

IF the guide wire induces arrhythmias in the patient (B816; C550; E361), THEN remove
guide wire several centimeters until arrhythmias cease

IF the guide wire doesn’t go through the Introducer needle OR gets stuck (F456), THEN
remove guide wire and Introducer needle and go back to Step 5A.1
Standard: Persisting with a stuck wire may shred the wire and embolize bits of the
wire. (D587)

Step 6A.2: Remove the Introducer needle leaving only guide wire in place (A359; B233;
C569; D492; E354; F162) by holding wire in one hand (B882; E Edited text; F602) and
sliding the Introducer needle off wire with your other hand (B883) so guide wire does not exit
the vein (B885; C571)

Step 6A.3: Make a quarter inch incision with a scalpel where the wire enters the skin
Standard: Incision is wide enough (3 millimeters) (A391; D488) ensuring there is no
skin bridge (D596) to allow the connective tissue around the clavicle to be accessed by the
dilator (A382; B899) for passage of the central venous catheter

Step 6A.4: Insert central venous catheter over guide wire into the vessel. (A70; B285;
C44; D62; E154; F170)

For single lumen catheters:

109
Step 6A.4.1: Place the dilator through the single lumen catheter (R)

Step 6A.4.2: Thread the single lumen catheter with dilator over the guide wire until the
hub contacts the skin (R)

Step 6A.4.3: Remove the guide wire from the subclavian vein through the catheter (A70;
B277; D621; E382; F170)

Step 6A.4.3.1: Use non-dominant hand to hold catheter and use dominant hand to remove
guide wire and dilator, together in one motion, from the catheter (A401; B979; D621; E382;
F170). Go to Task 7A.

For multi-lumen catheters:

Step 6A.4.4: Hold the wire steady (C582; D535) and lead the dilator over the wire
(A381: B955; C583; D538; E365; F162) and run it back and forth to establish track to pass
catheter (B915; C583; D497) and then remove the dilator (A386; E368; F163)
Standard: Do not let the wire get embolized (D545)

Step 6A.4.5: Remove the dilator (B386; D539; E368) while holding the wire in place
(D538; F603)

Step 6A.4.6: Guide the multi-lumen catheter (F604) (A396) over the wire until the wire
exits the brown port of the catheter (A399; C588; D523; E373). While maintaining control of
the wire (C582; D535), advance catheter until the hub is at the skin (A398; B241)

Step 6A.4.7: Remove the guide wire from the subclavian vein through the catheter
(A70; B277; D621; E382; F170)

Step 6A.4.7.1: Use non-dominant hand to hold catheter and use dominant hand to remove
guide wire from the catheter (A401; B979; D621; E382; F170). Go to Task 7A.

Task 7A: Verify the placement of CVC in the subclavian vein (A71; B1066; C43;
D64; E389; F175)

For both single lumen and multi-lumen catheters

Step 7A.1: Verify the correct placement of the central venous catheter by attaching a
syringe on all ports of the catheter and aspirating (A408) to verify that the catheter is in
position in the vein (A401; B995; D622; E383; F179)

Step 7A.2: Attach a syringe to the port (single-lumen) OR ports (multi-lumen) and
aspirate. (R)

IF you get blood flow, THEN proceed to Step 9A.2

110
IF you don’t get blood flow, THEN leave it and go back to Step 5A.1

Step 7A.2: Flush each port with a small amount of sterile saline, one to two cc’s per port
(A409; D622; E383; F178)

Step 7A.3: Replace the caps on the catheter (A410; D626; F628)

Step 7A.4: Verify the proper placement of the catheter inside the body through a chest x-
ray (A414; B1066; C753; D647; E389; F175).

IF the chest x-ray indicates that the catheter is in the superior vena cava (A417; C753;
D647), THEN it has been placed properly AND go to Task 8A.

IF the chest x-ray indicates a pneumothorax on the side of the central line (A418), OR a
widening of the mediastinum (A419), OR a hematoma indicating arterial injury (A420), OR
indicates extravasations of blood into the right chest (A421), OR you see blood or air
(B1086), OR you see the catheter in the wrong place, THEN notify the patient they have
experienced a complication and will need further intervention, and notify attending physician.

Task 8A: Affix catheter to the skin (A41; B296; C44; D63; E155; F179)

For both single lumen and multi-lumen catheters

Step 8A.1: Affix the central venous catheter to the skin (C44) using two interrupted
sutures through the skin to the tabs on the hub of the catheter (A410; B1039; D360; E387;
F179)

Step 8A.2: Place an antibiotic-coated disk directly on the insertion site (D636) and some
sterile ointment with a plastic occlusive dressing (C44; D639; E387) over the wound (B1044)

Step 8A.3: Remove the drapes (E388) and place patient in a comfortable position (E388).

End of Subclavian procedure.
Procedure for the Internal Jugular Location (B)
Task 4B: Locate the internal jugular vein (A68; B220; C42; D291; F154) using
ultrasound (R)

For both single lumen and multi-lumen catheters

Step 4B.1: Place ultrasound probe near the head of the clavicle and move up the neck
(E439) to the apex of the sternocleidomastoid heads triangle (F411), positioning the probe so
the internal jugular vein can be distinguished from the carotid artery by size and
compressibility of the vein (A514; B1227; C416; D339; E450).

111
Step 4B.2: Once you have found the internal jugular through use of ultra-sound,
verify internal jugular vein by compressing the neck at the site you have located the vein
(A517; C643)

IF the compressed vein is larger, oblong, changes with patient’s respiration (A519) AND
compresses (A520; C643) under pressure from the ultrasound probe, THEN it is the internal
jugular (A517) AND proceed to Task 5B.

IF the vein appears completely round (A516) and remains patent under compression
(A517), THEN the vein is the carotid artery, Repeat Step 4A.2 until you have located the
internal jugular, BUT, if you are unable to locate it after three attempts (B654), ask for
assistance.

Task 5B: Insert Introducer needle into the internal jugular vein (A528; B257; D305;
E104; F154)

For both single lumen and multi-lumen catheters

Step 5B.1: Under ultrasound visualization, (R) advance the needle at an angle 30-60
degrees perpendicular to the skin into the internal jugular vein providing continuous
aspiration on the syringe (A532; B561; C522; D293; E321; F161) until there is a flash or
flow of blood (B561; D294; E325; F462)

IF you don’t get a flash of blood in your syringe, go back to Step 4B.1 (B1141)

Step 5B.2: Confirm proper placement of Introducer needle in the vein by removing the
syringe from the needle (A347; B670; C563; D393; E336; F161) and visually assess presence
of blood and blood flow (R)

IF
You have continuous and steady (not pulsating) bleeding (A348), OR
You have dark slow dripping blood (D396; E337)
THEN, assume that the needle is in the vein (A348; D396; E337) AND go to
step 5B.3

IF
You see pulsatile, red blood (D398; E337)
THEN assume your needle is in the artery and pull needle out (E340), apply pressure
(E342) with a 4 by 4 dressing (C617) for 5 minutes (C618), go back to Task 4B and try
again on the same side without re-prepping (D399), or repeat procedure using the other side
(B670; C613)

Step 5B.3: Maintain the position of the needle by firmly holding the hub of the needle
next to the skin to keep it in position (A345; B789; D461; F162) with non-dominant hand
(D459; F536)

112
Step 5B.4: Place thumb over end of needle to avoid any air embolus introduced into
the vein (A350; C563; D459)

Task 6B: Insert central venous catheter into the internal jugular.
(A70; B285; C44; D62; E154; F170)

For both single lumen and multi-lumen catheters

Step 6B.1: Insert the flexible side of guide wire (D558) easily (A354; B744; D672) into
the Introducer needle (A353; B675; C566; D462; E350; F154) while always maintaining one
hand on the guide wire (C566; E edited text)
Standard: Easily is a standard Leave a portion of the guide wire showing at the top of
the Introducer needle. Do not insert the guide wire completely (A361; B844; D565; E353) or
it may cause heart arrhythmias (D558).

IF the guide wire induces arrhythmias in the patient (B816; C550; E361), THEN remove
guide wire several centimeters until arrhythmias cease

IF the guide wire doesn’t go through the Introducer needle OR gets stuck (F456), THEN
remove guide wire and Introducer needle and go back to Step 5A.1
Standard: Persisting with a stuck wire may shred the wire and embolize bits of the
wire. (D587)

Step 6B.2: Remove the Introducer needle leaving only guide wire in place (A359;
B233; C569; D492; E354; F162) by holding wire in one hand (B882; E Edited text; F602)
and sliding the Introducer needle off wire with your other hand (B883) so guide wire does not
exit the vein (B885; C571)

Step 6B.3: Make a quarter inch incision with a scalpel where the wire enters the skin
Standard: Incision is wide enough (3 millimeters) (A391; D488) ensuring there is no
skin bridge (D596) to allow the connective tissue around the clavicle to be accessed by the
dilator (A382; B899) for passage of the central venous catheter

Step 6B.4: Insert central venous catheter over guide wire into the internal jugular. (A70;
B285; C44; D62; E154; F170)

For single lumen catheters:

Step 6B.4.1: Place the dilator through the single lumen catheter (R)

Step 6B.4.2: Thread the single lumen catheter with dilator over the guide wire until the
hub contacts the skin (R). Go to Step 6B.5

Step 6B.4.3: Remove the guide wire from the internal jugular vein through the catheter
(A70; B277; D621; E382; F170)

113
Step 6B4.3.1: Use non-dominant hand to hold catheter and use dominant hand to
remove guide wire and dilator, together in one motion, from the catheter (A401; B979; D621;
E382; F170). Go to Task 7B.

For multi-lumen catheters:

Step 6B.4.4: Hold the wire steady (C582; D535) and lead the dilator over the wire
(A381: B955; C583; D538; E365; F162) and run it back and forth to establish track to pass
catheter (B915; C583; D497) and then remove the dilator (A386; E368; F163)
Standard: Do not let the wire get embolized (D545)

Step 6B.4.5: Remove the dilator (B386; D539; E368) while holding the wire in place
(D538; F603)

Step 6B.4.6: Guide the multi-lumen catheter (F604) (A396) over the wire until the wire
exits the brown port of the catheter (A399; C588; D523; E373). While maintaining control of
the wire (C582; D535), advance catheter until the hub is at the skin (A398; B241). Go to Step
6B.5

Step 6B.4.7: Remove the guide wire from the internal jugular vein through the catheter
(A70; B277; D621; E382; F170)

Step 6B.4.7.1: Use non-dominant hand to hold catheter and use dominant hand to remove
guide wire from the catheter (A401; B979; D621; E382; F170) Go to Task 7B.

Task 7B: Verify the placement of CVC in the internal jugular vein (A71; B1066;
C43; D64; E389; F175)

For both single lumen and multi-lumen catheters

Step 7B.1: Verify the correct placement of the central venous catheter by attaching a
syringe on all ports of the catheter and aspirating (A408) to verify that the catheter is in
position in the vein (A401; B995; D622; E383; F179)

IF you get blood flow, THEN proceed to Step 7B.2

IF you don’t get blood flow, THEN leave it and go back to Step 5B.1

Step 7B.2: Flush each port with a small amount of sterile saline, one to two cc’s per port
(A409; D622; E383; F178)

Step 7B.3: Replace the caps on the catheter (A410; D626; F628)

Step 7B.4: Verify the proper placement of the catheter inside the body through a chest x-
ray (A414; B1066; C753; D647; E389; F175).

114
IF the chest x-ray indicates that the catheter is in the superior vena cava (A417;
C753; D647), THEN it has been placed properly AND go to Task 8B.

IF the chest x-ray indicates a pneumothorax on the side of the central line (A418), OR a
widening of the mediastinum (A419), OR a hematoma indicating arterial injury (A420), OR
indicates extravasations of blood into the right chest (A421), OR you see blood or air
(B1086), OR you see the catheter in the wrong place, THEN notify the patient they have
experienced a complication and will need further intervention, and notify attending physician.

Task 8B: Affix catheter to the skin (A41; B296; C44; D63; E155; F179)

For both single lumen and multi-lumen catheters

Step 8B.1: Affix the central venous catheter to the skin (C44) using two interrupted
sutures through the skin to the tabs on the hub of the catheter (A410; B1039; D360; E387;
F179)

Step 8B.2: Place an antibiotic-coated disk directly on the insertion site (D636) and some
sterile ointment with a plastic occlusive dressing (C44; D639; E387) over the wound (B1044)

Step 8B.3: Remove the drapes (E388) and place patient in a comfortable position (E388).

End of Internal Jugular procedure.

Procedure for the Femoral Vein Location (C)

Task 4C: Locate the femoral vein using physical landmarks and
ultrasound

Step 4C.1: Identify anatomic landmarks (D320) using Nerve-Artery-Vein-Empty
Space-Lymphatic’s from lateral to medial (A570)

Step 4C.2: Palpate the groin (E466) and femoral artery to locate femoral vein (A574;
B1157; C646; D331)
Standard: Feel for arterial pulse and go medial (about 1 centimeter) (D331). The
femoral artery is just inferior to the inguinal ligament (A574; B1158; E467). Work your way
away from the artery to find the insertion point (D331). The vein is one fingerbreadth (C425;
E469) medial to this palpable pulsation (A574; B1158; E467). Stay below the inguinal
ligament (D324)

Step 4C.3: Place the ultrasound at the location in step 4C.2 to confirm the location of the
femoral vein.

Step 4C.4: Once you have found the femoral vein through use of ultra-sound, verify the
location of the vein by compressing the site where you have located the vein (A517; C643)

115
IF the compressed vein is larger, oblong, changes with patient’s respiration (A519)
AND compresses (A520; C643) under pressure from the ultrasound probe, THEN it is the
femoral vein (A517) AND proceed to step 4A.5.

IF the vein appears completely round (A516) and remains patent under compression
(A517), THEN the vein is the femoral artery, Repeat Step 4A.2 until you have located the
femoral vein, BUT if you are unable to locate it after three attempts (B654), ask for
assistance.

Task 5C: Insert Introducer needle into the femoral vein (A528; B257;
D305; E104; F154)
For both Single-lumen Introducer and Multi-lumen catheters

Step 5C.1: Using ultrasound visualization (R), insert Introducer needle into the skin at
the insertion point (D284) to cannulize the femoral vein (A581; B224; D291; E451)

Step 5C.2: Use continuous aspiration on the syringe until there is a return of venous
blood (A583; C1141; D294)

IF you don’t get a flash of blood in your syringe, go back to Step 4C.4 (R)

Step 5C.3: Confirm proper placement of Introducer needle in the vein by removing the
syringe from the needle (A347; B670; C563; D393; E336; F161) and visually assess presence
of blood and blood flow

IF
You have continuous and steady (not pulsating) bleeding (A348), OR
You have dark slow dripping blood (D396; E337)
THEN, assume that the needle is in the vein (A348; D396; E337) AND go to
step 5C.4

IF
You see pulsatile, red blood (D398; E337)
THEN assume your needle is in the artery and pull needle out (E340), apply pressure
(E342) with a 4 by 4 dressing (C617) for 5 minutes (C618), go back to Task 4C and try again
on the same side without re-prepping (D399), or repeat procedure using the other side (B670;
C613)

Step 5C.4: Once the femoral vein is cannulated, maintain a still (D461) position of
needle by firmly holding the needle at the hub with your non-dominate hand (D459; F536) to
keep it in position (A345; B789; D459; F162)

Step 5C.5: Place thumb over top of needle to avoid air being introduced into the vein
(A350; C563; D459)

116
Task 6C: Insert central venous catheter into the vessel. (A70;
B285; C44; D62; E154; F170)

For both single lumen and multi-lumen catheters
Step 6C.1: Insert the flexible side of guide wire (D558) easily (A354; B744; D672) into
the Introducer needle (A353; B675; C566; D462; E350; F154) while always maintaining one
hand on the guide wire (C566; E edited text)
Standard: Easily is a standard Leave a portion of the guide wire showing at the top of
the Introducer needle. Do not insert the guide wire completely (A361; B844; D565; E353) or
it may cause heart arrhythmias (D558).

IF the guide wire induces arrhythmias in the patient (B816; C550; E361), THEN remove
guide wire several centimeters until arrhythmias cease

IF the guide wire doesn’t go through the Introducer needle OR gets stuck (F456), THEN
remove guide wire and Introducer needle and go back to Step 5C.1
Standard: Persisting with a stuck wire may shred the wire and embolize bits of the
wire. (D587)

Step 6C.2: Remove the Introducer needle leaving only guide wire in place (A359; B233;
C569; D492; E354; F162) by holding wire in one hand (B882; E Edited text; F602) and
sliding the Introducer needle off wire with your other hand (B883) so guide wire does not exit
the vein (B885; C571)

Step 6C.3: Make a quarter inch incision with a scalpel where the wire enters the skin
Standard: Incision is wide enough (3 millimeters) (A391; D488) ensuring there is no
skin bridge (D596) to allow the connective tissue around the clavicle to be accessed by the
dilator (A382; B899) for passage of the central venous catheter

Step 6C.4: Insert central venous catheter over guide wire into the vessel. (A70; B285;
C44; D62; E154; F170)

For single lumen catheters:

Step 6C.4.1: Place the dilator through the single lumen catheter (R)

Step 6C.4.2: Thread the single lumen catheter with dilator over the guide wire until the
hub contacts the skin (R).

Step 6C.4.3: Remove the guide wire from the femoral vein through the catheter (A70;
B277; D621; E382; F170)

Step 6C.4.3.1: Use non-dominant hand to hold catheter and use dominant hand to remove
guide wire and dilator, together in one motion, from the catheter (A401; B979; D621; E382;
F170). Go to Task 7C

117
For multi-lumen catheters:

Step 6C.4.4: Hold the wire steady (C582; D535) and lead the dilator over the wire
(A381: B955; C583; D538; E365; F162) and run it back and forth to establish track to pass
catheter (B915; C583; D497) and then remove the dilator (A386; E368; F163)
Standard: Do not let the wire get embolized (D545)

Step 6C.4.5: Remove the dilator (B386; D539; E368) while holding the wire in place
(D538; F603)

Step 6C.4.6: Guide the multi-lumen catheter (F604) (A396) over the wire until the wire
exits the brown port of the catheter (A399; C588; D523; E373). While maintaining control of
the wire (C582; D535), advance catheter until the hub is at the skin (A398; B241). Go to Step
6C.5

Step 6C.4.7: Remove the guide wire from the femoral vein through the catheter (A70;
B277; D621; E382; F170)

Step 6C.4.7.1: Use non-dominant hand to hold catheter and use dominant hand to remove
guide wire from the catheter (A401; B979; D621; E382; F170). Go to Task 7C

Task 7C: Verify the placement of CVC in the femoral vein (A71; B1066; C43; D64;
E389; F175)

For both single lumen and multi-lumen catheters

Step 7C.1: Verify the correct placement of the central venous catheter by attaching a
syringe on all ports of the catheter and aspirating (A408) to verify that the catheter is in
position in the vein (A401; B995; D622; E383; F179)

IF you get blood flow, THEN proceed to Step 7C.2

IF you don’t get blood flow, THEN leave it and go back to Step 5C.1

Step 7C.2: Flush each port with a small amount of sterile saline, one to two cc’s per port
(A409; D622; E383; F178)

Step 7C.3: Replace the caps on the catheter (A410; D626; F628)

Task 8C: Affix catheter to the skin (A41; B296; C44; D63; E155; F179)

For both single lumen and multi-lumen catheters

Step 8C.1: Affix the central venous catheter to the skin (C44) using two interrupted
sutures through the skin to the tabs on the hub of the catheter (A410; B1039; D360; E387;
F179)

118
Step 8C.2: Place an antibiotic-coated disk directly on the insertion site (D636) and
some sterile ointment with a plastic occlusive dressing (C44; D639; E387) over the wound
(B1044)

Step 8C.3: Remove the drapes (E388) and place patient in a comfortable position (E388).

End of Femoral Vein procedure.

Abstract (if available)

Abstract The purpose of this study was to examine the amount of relevant information experts provide and fail to provide when asked to describe how to perform a complex task in enough detail for students to perform the task. In this study, medical experts where interviewed because their past successes and failures at the task are known and so it could be determined that all experts had succeeded consistently at the task being described. Past research has suggested that because experts have both conscious and unconscious automated knowledge they may not “know what they know” and so not be able to completely describe how to make critical decisions during task performance. A version of Cognitive Task Analysis designed to support training was used to interview medical school faculty and analyze their description of a controversial trauma procedure in order to determine the average percentage of knowledge that was acquired from a single expert and how much additional knowledge is acquired from each succeeding expert interviewed. After analysis, it was determined that the amount of knowledge acquired from one expert was about 63 percent and the average increase of acquired knowledge with the second expert was about 16% and the third expert added another 8 percent. Past studies have reported considerably lower percentages of relevant information captured with Cognitive Task Analysis – about 30 percent of decisions are typically reported for example. Thus it was hypothesized that information about the controversial procedure examined in this study may have become more conscious to the experts interviewed because they had been discussing it among themselves and reading accounts in journals.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Using cognitive task analysis to determine the percentage of critical information that experts omit when describing a surgical procedure

PDF

The use of cognitive task analysis for identifying the critical information omitted when experts describe surgical procedures

PDF

The use of cognitive task analysis to determine surgical expert's awareness of critical decisions required for a surgical procedure

PDF

The effect of cognitive task analysis based instruction on surgical skills expertise and performance

PDF

Using cognitive task analysis to capture expert reading instruction in informational text for students with mild to moderate learning disabilities

PDF

The use of cognitive task analysis to capture expert instruction in teaching mathematics

PDF

The use of cognitive task analysis to capture exterptise for tracheal extubation training in anesthesiology

PDF

The use of cognitive task analysis for the postanesthesia patient care handoff in the intensive care unit

PDF

Towards a taxonomy of cognitive task analysis methods: a search for cognition and task analysis interactions

PDF

Identifying the point of diminishing marginal utility for cognitive task analysis surgical subject matter expert interviews

PDF

Using cognitive task analysis to capture how expert principals conduct informal classroom walk-throughs and provide feedback to teachers

PDF

Using cognitive task analysis for capturing expert instruction of food safety training for novice employees

PDF

Using incremental cognitive task analysis to capture expert instruction in expository writing for secondary students

PDF

Cognitive task analysis for instruction in single-injection ultrasound-guided regional anesthesia

PDF

Employing cognitive task analysis supported instruction to increase medical student and surgical resident performance and self-efficacy

PDF

Using cognitive task analysis to capture how expert anesthesia providers conduct an intraoperative patient care handoff

PDF

Using individual cognitive task analysis to capture expert writing instruction in expository writing for secondary students

$Using cognitive task analysis to capture expert instruction in division of fractions$

PDF

Using cognitive task analysis to capture expert instruction in division of fractions

PDF

The use of cognitive task analysis to capture expert patient care handoff to the post anesthesia care unit

PDF

Using cognitive task analysis to capture palliative care physicians' expertise in in-patient shared decision making

Asset Metadata

Creator Bartholio, Craig W. (author)

Core Title The use of cognitive task analysis to investigate how many experts must be interviewed to acquire the critical information needed to perform a central venous catheter placement

School Rossier School of Education

Degree Doctor of Education

Degree Program Education (Leadership)

Degree Conferral Date 2010-12

Publication Date 09/07/2010

Defense Date 06/17/2010

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag automaticity,cognitive task analysis,expertise,interview,knowledge types,Medical education,OAI-PMH Harvest,subject matter expert,Surgery,Training

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Clark, Richard E. (committee chair), Sullivan, Maura E. (committee member), Yates, Kenneth A. (committee member)

Creator Email cbartholio@apu.edu,cbartholio@prodigy.net

Permanent Link (DOI) https://doi.org/10.25549/usctheses-m3424

Unique identifier UC1167657

Identifier etd-Bartholio-4088 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-385767 (legacy record id),usctheses-m3424 (legacy record id)

Legacy Identifier etd-Bartholio-4088.pdf

Dmrecord 385767

Document Type Dissertation

Rights Bartholio, Craig W.

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Repository Name Libraries, University of Southern California

Repository Location Los Angeles, California

Repository Email cisadmin@lib.usc.edu