Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
The use of cognitive task analysis for identifying the critical information omitted when experts describe surgical procedures
(USC Thesis Other)
The use of cognitive task analysis for identifying the critical information omitted when experts describe surgical procedures
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
THE USE OF COGNITIVE TASK ANALYSIS FOR IDENTIFYING THE
CRITICAL INFORMATION OMITTED WHEN EXPERTS DESCRIBE
SURGICAL PROCEDURES
by
Eko Natividad Canillas
A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION
May 2010
Copyright 2010 Eko Natividad Canillas
ii
DEDICATION
This work could not have been completed without the love, patience
and support of all my family and friends.
First and foremost, Marco Lim Canillas, I dedicate this to you for
encouraging “Mommy” to continue her own learning to make this world the
best place possible for you. I love you, Son.
To my mother, Patricia D. Canillas and Cecil A. Canillas, I love you
beyond words. Not only for your strength, wisdom, love, and support, but for
also instilling within me the opportunities and access education can bring to
those who seek it.
To my partner, friend, and confidante, O’Shea” Myles, thank you for
never letting me fall and for believing in me when I couldn’t believe in
myself. You are my rock and I love you.
To my brother, Charleston Wilson, thank you for reminding me that a
comprehensive and meaningful education is not limited to classroom walls.
To my grandmothers Ferrell L. Beals and Roberta G. Canillas, my
grandfather Narciso L. Canillas, my aunt Eddye R. Washington, my brother
Timothy D. Wilson, my aunt Sylvia E. Finley, and my uncle Gregorio N.
Canillas. I miss you so much and dedicate this in honor of your memory.
Finally, to the countless friends, family, youth, mentors, and
colleagues who created the conditions for me to succeed, served as the
iii
models for me to aspire to, and provided the encouragement for me to finish
this program, this is also for you.
iv
ACKNOWLEDGEMENTS
I would like to express my gratitude to the following affiliates of the USC
Rossier School of Education whose support and guidance made the completion of
this journey possible:
To Dr. Richard Clark, who shared his vision of the role CTA can play in
improving the treatment and lives of those individuals in need of quality medical
care.
To Dr. Maura Sullivan, who not only demystified this dissertation process,
but provided the access to the resources that supported the completion of this work.
To Dr. Kenneth Yates, who introduced me to the gap analysis and played a
pivotal role in providing the encouragement and high-level of expectations to make
the completion of this project both feasible, yet challenging. I am extremely grateful
for your patience, knowledge, and expertise.
To the CTA dissertation team consisting of fellow students of the 2010 Ed.D.
cohort, I thank you for your support and your constant commitment to ensuring an
exceptional degree of professionalism (and humanism) to see this process through as
a team.
Lastly, to the Ed.D. program faculty and student support team of the USC
Rossier School of Education. This was a rigorous program and with your collective
dedication to student success, I was able to complete this journey.
v
TABLE OF CONTENTS
Dedication ii
Acknowledgements iv
List of Tables vi
Abstract vii
Chapter 1: Review of the Literature 1
Statement of the Problem 1
Purpose of the Study 4
Review of the Literature 4
Chapter 2: Method 26
Study Design 26
Task 26
Sample 26
Procedure for Data Collection 27
Analysis of the Data 30
Chapter 3: Results 34
Inter-rater Reliability 34
Data Analysis 35
Research Questions 36
Research Question 1 36
Research Question 2 43
Summary 45
Percentage of Knowledge Omissions 45
Omissions based on Prior Knowledge 46
Chapter 4: Discussion 48
Research Question 1 48
Research Question 2 50
Study Limitations 52
Summary 54
Implications 55
Conclusion 58
References 60
vi
LIST OF TABLES
Table 1: Indication omissions 36
Table 2: Contraindication omissions 37
Table 3: Equipment omissions 38
Table 4: Action step omissions 40
Table 5: Decision step omissions 41
Table 6: Overall step omissions for entire CVC procedure 42
Table 7: Prior knowledge and action step omissions 44
Table 8: Prior knowledge and decision step omissions 44
Table 9: Summary of all omissions by CTA item type 45
vii
ABSTRACT
Evidence in research suggests experts omit 70% of procedural steps when describing
a surgical procedure. Additionally, prior knowledge of a procedure may have an
affect on the percentage of information omitted when describing a procedure. To
examine this, the current study used CTA methods to conduct semi-structured
interviews with six subject matter experts describing a central venous catheter (CVC)
placement procedure. Quantitative methods were used to analyze data regarding the
amount of information omitted when describing a CVC and the amount of
information omitted as it relates to prior knowledge of the procedure. Study findings
did not confirm experts omit 70% of procedural steps, but approximately 30% of
action steps and 35% of decision steps were omitted when describing a CVC.
Failure to confirm prior estimates of omissions in existing research may be due to
study limitations including (a) “rolling-up” steps of CTA items to achieve better
organization in the gold standard for data analysis (b) issues with crediting SMEs
and (c) challenges to SME participation. Moreover, participants omitted a lower
percentage of action and decisions steps when describing a CVC, yet possessed more
prior knowledge of the procedure. This was compared to data of a similar study
using an open cricothyrotomy (CRIC) procedure where participants had less prior
knowledge of the CRIC, but more knowledge omissions. This outcome may stem
from the type of prior knowledge examined in this study and heightened awareness
resulting from institutional responses targeting the CVC. Future research in the field
of CTA is discussed.
1
CHAPTER 1: REVIEW OF THE LITERATURE
Statement of Problem
In a perfect world, physicians would never get tired and never get stuck on
the wrong solution. They wouldn’t have to hand off their patients to the
physician working the next shift. Nurses would communicate instructions to
each other with perfect clarity. Not so in the real world. Most medical errors
are small, procedural slip-ups. But occasionally, they blossom into full-
blown tragedies. (Price, 2010, p.1)
There is increasing pressure to improve training and instruction in all
professional fields, but the field of surgical education has experienced unusual
challenges. This pressure arises from the need to better respond to the changing
demands and evolving problems in surgical practice. Issues of patient safety,
modifications in accreditation requirements, decreases in the work week (Scott,
Cendan, Pugh, Minter, Dunnington, & Kosar, 2008) and ethical concerns
surrounding the use of live animals (Barnes, Lang, & Whiteside, 1989) has proved
challenging to the type and amount of teaching time and/or clinical exposure surgical
learners receive (Reznick & MacRae, 2006; Scott et al., 2008).
There has been a shift from the educational and training strategies of
“yesteryear” towards evolving learning technologies and instructional strategies
which incorporate innovations, such as web-based education, use of virtual reality,
and high-fidelity human patient simulation (Vozenilek, Huff, Reznek, & Gordon,
2004). These developments seek to respond to the need of the sector to meet, if not,
enhance the technical skills and competencies of its professionals (Vozenilek et al.,
2004). The use of these emerging tools train students by providing opportunities for
2
expertise development in skills that may not easily be supported by surgical
challenges amidst decreased teaching time and decreased opportunities for deliberate
practice (Grantcharov & Reznick, 2008; Reznick & MacRae, 2006). Nevertheless,
the tools by themselves can not enhance learning outcomes if they fail to capture the
cognition that drives performance.
Training should provide the “how-to” knowledge and skills to support the
achievement of performance goals (Clark & Estes, 2002). In improving practice, the
surgical community is recognizing that task performance is linked to recognizing
both the observable and unobservable cognition used to perform procedures (Flin,
Youngson, & Yule, 2007). Additionally, experts are relied upon heavily to transfer
the information that train emerging surgical practitioners. However, there is
evidence in the research that experts unintentionally omit information when
describing a surgical procedure (Sullivan, Ortega, Wasserberg, Kaufman, Nyquist, &
Clark, 2008) due to the nature of expertise and automated knowledge. Hence, it is
critical that information that informs training and instruction be complete and
accurate by including both the observable action and unobservable decisions that
inform performance.
Clark and Estes (1996) suggest as expertise develops through repeated
practice, the underlying knowledge and skills experts use to perform complex tasks
can automate. Automated knowledge is described as cognitive processes and
behaviors that occur “without the need for conscious guidance or monitoring”
(Wheatley & Wegner, 2001, p. 991). It can include the technical skills and
3
unobservable critical decisions required in performing a complex task. This includes
the “when, how, and why” knowledge experts use when decision-making and/or
problem-solving. Consequently, the non-conscious nature of automated knowledge
can impede an expert’s ability to deliberately access this critical information
(Ericsson, 2004). Considering this, experts’ knowledge is vulnerable to omissions
when describing how they perform complex tasks, such as those found in medical
procedures. If such is the case, the lack of complete, thorough, and accurate
information can compromise the instruction and learning that informs surgical
competence and practice. As such, more research is needed on the unobservable,
cognition that informs task performance. In addition, an understanding of the tools
and technologies that can achieve this are pertinent.
Cognitive Task Analysis (CTA) is a knowledge elicitation tool to extract
knowledge. CTA can capture the unobservable knowledge that informs observable
action and is currently utilized within the surgical community for this purpose. By
offering a mechanism for which to capture advanced, expert performance (Ericsson,
2004), CTA methods can be used to enhance the current training needs and
performance efforts in the surgical community. As such, CTA as a tool to capture
critical knowledge may have valuable implications for improving the effectiveness
of training design and programs within the field of surgery.
4
Purpose of the Study
The objective of this descriptive study is to build upon research suggesting
that experts omit critical information when describing surgical procedures. Using
CTA, the purpose of this study is to elicit and collect data to examine the amount and
types of knowledge expert’s use and may omit when describing a surgical procedure.
Accordingly, the research question this study addresses is:
What percentage of critical information do surgeons omit when describing a central
venous catheter (CVC) placement surgical procedure?
Arising from this study is a sub-area of research that intends to explore the
amount of an expert’s prior knowledge of a procedure and its relationship to
knowledge omissions when describing the procedure. Thus, the second research
question of this study is:
Does the amount of an expert's prior knowledge and experience with a central
venous catheter (CVC) placement procedure influence the amount of knowledge
about the procedure they omit when describing how to perform the procedure?
Review of the Literature
Issues in Surgical Education
Surgical education is seeking to improve the instruction and practice of
surgery. This effort stems from the sector’s need to respond to recent critical issues
threatening the integrity and safety of surgical practice. Issues of patient safety
5
resulting from medical errors has proven to be costly to the healthcare system, in
addition to its threat of triggering public scrutiny (Kohn, Corrigan, & Donaldson,
2000).
The introduction of web-based education, use of virtual reality, and high-
fidelity human patient simulation are some examples of the medical communities
shift from traditional tools of delivering instruction (Vozenilek et al., 2004). The use
of such technology provides a learning environment in which patient risk is lowered
and time factors regarding teaching time are less of an obstacle (Grunwald, Clark,
Fisher, McLaughlin, Narayanan, & Piepol, 2004).
With the advent of technology and the multitude of changes in the surgical
community, critical attention must be given towards the types of improvements
needed to optimize training. In an article by Kneebone (2003), the author reviews
the educational issues and practical implications of simulation in surgical training.
In his discussion, he proposes that the technologies used must not only effectively
teach technical skills, but must include the principles of education and clinical
practice in providing an integrative learning environment. This integrative learning
environment, he suggests, can enrich the tools to better meet the learning outcomes
directed towards acquisition, assessment, and real-life application of comprehensive
surgical skills (i.e. cognitive and motor).
Sharing in this viewpoint, Aggarwal, Grantcharov, and Darzi (2007) propose
that the use of simulation and similar tools reproduce safe and efficient training
experiences for learners to acquire technical skills. However, the authors caution
6
experience alone is not enough. Instead, these tools must also embody both the
technical and non-technical skills that generate the proficiency to support learning,
expertise development, and achieve performance goals.
Effective instruction is more than the innovation of information delivery.
Clark, Yates, Early, and Moulton (2009) define instruction as a process of providing
information (instructional content) and structuring the information to deliver learning
(instructional method). As such, it is no surprise that national efforts are underway
to reform surgical residency education programs to provide competency-based
instruction that better prepares learners (Sachdeva, Bell, Britt, Tarpley, Blair, &
Tarpley, 2007). With the focus of competency and proficiency being priority,
training and instruction must be at the forefront in effectively responding to issues
that meet the needs of its learners. Nonetheless, the tools and strategies emerging
will fail in meeting their goals if not prepared to address all aspects necessary “to
facilitate learning and accelerate the acquisition of expertise” (Grunwald et al.,
2004). This includes both the observable and unobservable decision-making process
of cognition. If such is the case, what elements can influence the quality and
effectiveness of training and instruction?
Different Types of Knowledge About Surgery
Performing complex tasks involves interaction between two types of
knowledge: declarative and procedural (Clark, in press). Declarative knowledge is
described as controllable knowledge. We are aware of it and we have the capacity to
7
modify or change it (Clark & Elen, 2006); hence, we can control it. This type of
knowledge is stored in memory and is consciously accessible. When we “think” or
“remember” we are using declarative knowledge.
Declarative knowledge is learned in the form of facts, concepts, processes,
and principles (Merrill, 1983). It is mentally represented and episodic or semantic in
form (Clark, in press). If an individual confronts a goal in which existing automated
procedural knowledge is incomplete or inadequate, declarative knowledge is
employed to fill the gaps (Clark, in press).
Procedural knowledge on the other hand, is about “when and how” to do
something mentally or physically. Estimates in research suggest that 90 to 70
percent of procedural knowledge account for our total knowledge (Clark & Elen,
2006). It can include “the form of step-by-step action and decision steps required to
achieve task and sub goals under specific conditions” (Clark & Elen, 2006, p.286).
In surgery, procedural knowledge is representative of the action and decision steps
used to complete tasks for a given procedure. As discussed in detail later in this
chapter, when procedural knowledge is learned and applied, it gradually becomes
automated and non-conscious (Clark & Estes, 1996). Thus, automated procedures
can be used effectively to solve problems or perform complex tasks without the
performer being aware of what they did to succeed (Clark et al., 2008).
Declarative and procedural knowledge are both necessary to perform most
tasks. Yet, Clark (2008) suggests that research has traditionally been concerned with
8
focusing on the forms of knowledge which we can consciously inspect. If such is the
case, what is the impact on our understanding of expertise development?
Expertise
Expertise is no longer considered just the result of innate talent. Current
views on the subject propose that expertise is indicative of high levels of
performance in problem-solving, problem detection strategies, efficient use of
resources, and minimal cognitive effort in performing skills (Chi, 2006). This
expanded view is the result of what has been observed in the research on the subject
of expert performance.
Ericsson, Krampe, and Tesch-Romer (1993) propose that individuals develop
levels of expertise through the repetition of performing a task during deliberate
practice, that is, attending to critical aspects of the performance situation and
improving task performance by adapting behavior and receiving feedback. This is
acquired through many years of experience (Glaser, 1992). Furthermore, improving
performance at an expert level is an ongoing process in which superiority in
performance is measured by both consistent and effortful practice of challenging
features of tasks (Ericsson, 2004). Thus, expertise is not something that an
individual is born with, but instead is a measure of performance that must be
acquired.
In the context of learning and training, the body of work done on the subject
of expertise has and continues to provide valuable insight to those professional fields
9
and industries concerned with replicating and reproducing high levels of proficiency
and performance. Where this work can continue to be relevant is in continuing
discussions that not only recognize the benefits of expertise, but also include the
challenges that can result from expertise. Doing so increases the usability of
research in this area by widening our understanding of human performance and the
resources that complement it.
Nature and challenges of expertise
Expertise is characterized by “highly organized, integrated structures of
knowledge” (Glaser, 1992, p.64). According to Feldon (2007), as experts develop
their conscious declarative knowledge it becomes gradually more automated.
Automated knowledge can involve the “when and how” knowledge experts use when
solving problems and can also include the manual skills and critical decisions
required in performing a surgical procedure. Knowledge becomes automated
through repeated performance of a task to the point that the speed in performing the
tasks increases while the amount of active mental effort decreases (Feldon, 2007).
Automated knowledge helps to alleviate cognitive overloads and/or processes
that can impede the efficiency of working memory. This is important because the
length and amount of information that can be retained and processed in working
memory is limited (Kirschner, Sweller, & Clark, 2006). As a result, when experts
develop their expertise by building their knowledge base (or schemas) and
automating processes in long-term memory, this allows “space” or increased
10
capacity for working memory (Kalyuga, Ayres, Chandler, & Sweller, 2003) to attend
to novel and/or unfamiliar problems.
Consider the task of medical diagnosis. Physicians use available working
memory capacity to make a medical diagnosis based on the condition (symptoms)
that are presented. During a diagnosis, the medical expert must attend to the
information being presented and retrieve knowledge stored in long-term memory to
construct a diagnosis (Ericsson & Kintsch, 1995). Essentially, automated knowledge
from long-term memory allows the physician to perform at an expert level by
increasing the amount, speed and accuracy of knowledge retrieved from long-term
memory.
As stated, automated knowledge can be understood as those ideas,
knowledge, and reasoning that informs the completion of a task in responding to a
complex situation (Crandall, Klein, & Hoffman, 2006). At first glance, the
automation process is advantageous to expertise as it supports the capacity to
respond to novel problems with speed, accuracy, and consistency within an expert’s
domain. Yet, automaticity has a downside. Research has suggested that experts are
unaware of the information they use to complete complex tasks as a consequence of
automaticity (Clark and Elen, 2006). This is particularly significant for fields in
which task description is essential to transferring the knowledge that directs
performance. As a result, although automated knowledge is a benefit for the expert
performer, it can produce adverse effects when experts are needed to recall their
information when describing how they perform a procedure.
11
Knowledge Omissions
In surgical education, automated knowledge can be in the form of procedural
knowledge and can include critical information of the actions and decisions surgical
experts use to carry out a procedure. Instructional content depends upon the
unobservable knowledge and critical decision points that accompany-if not inform-
the observable action involved in performing a task (Clark et al., 2009). Yet, due to
the non-conscious nature of automated knowledge, it is susceptible to not presenting
itself when performing a procedure. As a result, instruction and/or training can fail if
information given is omitted resulting in incomplete or inaccurate descriptions
(Clark et al., 2009) of surgical task performance.
Feldon and Clark (2006) examined the accuracy of expert self-reports and
instructional efficacy by examining two dissertation studies conducted in the field of
surgery and experimental psychology. The first study revealed that CTA elicitation
strategies supported improved performance of students exposed to a training
curriculum that utilized CTA versus the control group exposed to the non-CTA
curriculum. Within the data provided, there was evidence that expert omissions
during self-reportable situations occurred when describing a surgical procedure. In
the second study, researchers identified that study participants omitted information
ranging from 48% to 88% when describing domain-specific procedures from self-
reporting.
In a separate study conducted, Clark, Pugh, Yates, Early, and Sullivan (2008)
used CTA methods to reveal that trauma surgeons performing a femoral shunt
12
procedure omitted 70% of procedural steps when describing a femoral shunt
procedure. The relevance of these studies provides evidence that experts,
specifically surgical experts, unintentionally omit information when describing
surgical procedures. With such as the case, identifying the types, levels, and amount
of knowledge that establish competency are an important facet of research in surgical
education. This is especially important when considering the critical information
that should be included, and may omitted, when describing procedures.
Complexity and Prior Knowledge
In addition to developing expertise, experts use complex knowledge to
perform complex skills (Ericsson & Kintsch, 1995). This can also have implications
for the amount of information that may be omitted when describing a complex,
surgical procedure. Thus far, distinguishing and measuring of task complexity is a
challenging area for researchers. Operating from cognitive load theory, Sweller
(2006) once approached the undertaking of defining complexity by examining the
level of cognitive element interactivity that occurs simultaneously for the
performance of the task.
Sweller’s (2006) view is no longer the current understanding of complexity.
Instead, complexity is currently understood as a characteristic that can vary from
individual to individual so that what is complex to one person may prove less
complex to another (Clark & Elen, 2006; Sweller, 2006). This deviates from the
argument that the number of elements and its interactions determine complexity and
13
shifts the attention to focus on the learner, the task and the role of prior knowledge in
understanding the variation.
Prior knowledge is past learning or pre-existing information that may include
highly automated procedural knowledge (Clark & Elen, 2006). This type of
knowledge can be employed to connect new information towards accruing new
learning (Clark & Elen, 2006) in addition to supporting and heightening the problem-
solving capacity characteristic of expertise (Glaser, 1992). Considering the nature of
automaticity, if the prior knowledge being examined includes automated procedural
knowledge, exploring the type and amount of prior knowledge an expert possesses
may serve to enhance discussions around the amount of information omitted when
describing a procedure.
What is CTA?
One condition under which instruction can fail is when the information given
is incomplete or inaccurate. To resolve these types of challenges in instructional
content, the use of task analysis methods are strategies that can help to “to identify
the conceptual and procedural knowledge required to perform a task” (Clark, 2008,
p. 13-14). One such tool currently being used in research related to cognitive
architecture and human performance is cognitive task analysis or CTA.
CTA is a type of knowledge elicitation, analysis and representation tool to
extract expert knowledge to “capture the way the mind works” (Crandall et al., 2006,
p. 9). Knowledge elicitation involves obtaining the cognition processes (e.g.
14
decisions, processes, concepts) that inform the performance of a job from a human
source (Fowlkes, Salas, Baker, Cannon-Bowers, & Stout, 2000). It uses strategies,
such as interviews and observations, to capture the accurate and complete knowledge
descriptions of how experts perform a task (Clark, Feldon, van Merriënboer, Yates,
& Early, 2008). Although traditional techniques of analysis have sought to capture
the observable behaviors in task performance, CTA sets out to capture the
unobservable cognitive processes that preface such behavior (Schraagen, Chipman,
& Shalin, 2000).
CTA Varieties
There are over 100 methods under the general classification of CTA (Yates,
2007). For example, there are CTA protocols that aim to capture the decision and
action steps experts use to perform tasks. Such protocols provide a mechanism to
capture critical information that can improve performance to better train novices
and/or intermediates in job tasks (Clark & Estes, 1996). Other forms use critical
events that experts encounter to discover the cues, challenges, and details (Crandall
et al., 2006) used to problem-solve. Accordingly, CTA will vary by its purpose.
Therefore, the appropriate selection of a CTA type or method should be carefully
considered to align with the type of analysis and its intended use (Clark & Estes,
1996; Cooke, 1999; Crandall et al., 2006; Schraagen et al., 2000).
15
CTA in Practice
According to Crandall et al. (2006), CTA involves a three-step approach
which includes knowledge elicitation, data analysis and knowledge representation.
Knowledge elicitation is representative of the techniques employed to extract and
capture knowledge. Essentially, it seeks to access information “about what people
know and how they know it” (Crandall et al., 2006, p.10). The data analysis and
knowledge representation activities of CTA are designed to systematize and give
meaning to the knowledge, or data, elicited and to provide an organized depiction of
the cognitive elements underlying the execution and/or performance of a task.
Clark et al. (2008) propose that the most common stages in the CTA process
include (a) collection of preliminary knowledge (b) identifying knowledge
representations (c) application of knowledge elicitation methods (d) data analysis
and verification of data collected and (e) formatting of results for its intended
application.
Collect Preliminary Knowledge. Collection of preliminary knowledge
involves becoming familiar with the domain in which the CTA will be conducted.
This process is referred to in the research on CTA as “bootstrapping” (Crandall et al.,
2006). Bootstrapping functions to bring the analyst “up-to-speed” by allowing the
analyst to become familiar with the general knowledge of the area (Schraagen et al.,
2000) and acquire information that will support the appropriate elicitation strategies.
Collecting preliminary knowledge can include document analysis, observations, and
16
unstructured interviews (Clark et al., 2008; Crandall et al., 2006; Schraagen et al.,
2000).
Identifying Knowledge Representations. The data collected from the
preliminary knowledge will allow the analyst to identify the knowledge
representations that support task performance. Tools such as concept mapping and
flow charts provide organization in delineating the types of knowledge involved with
performing the task. Identifying knowledge representations can also support the
analyst to better structure the elicitation methods to appropriately capture types of
knowledge (e.g. declarative, procedural) required to perform the task.
Application of Knowledge Elicitation Method. CTA utilizes multiple subject
matters experts (SMEs) or individuals that possess expert level knowledge in a
specific domain. As a result, the CTA analyst seeks to elicit the essential decisions
or steps that will result in successful performance of the task. Strategies that are
selected must be structured appropriately for the types of knowledge the analyst is
seeking to elicit from the SME. For instance, elicitation methods to capture
concepts, principles, and processes will differ from the strategies used to capture the
critical cognitive decisions involved in performing a task. Semi-structured
interviews are often selected as the primary method for knowledge elicitation for the
facility it provides novice CTA users (Clark et al., 2008).
Data Analysis and Verification. Although analysis approaches depend upon
the product and/or outcome leading to conducting a CTA, Crandall et al., (2006)
identify the following activities to be central to CTA analysis: preparing data,
17
structuring data, discovering meaning, and representation of the results. Each
component contributes to verifying, developing, structuring, and organizing the data
to completely and accurately represent the essential cognitive components involved
in performing a given task. Multiple SMEs are used in both the elicitation and the
review process to ensure completeness and accuracy of the results (Hoffman, 1987).
More so, a study conducted by Chao and Salvendy (1994) revealed that increasing
the number of SMEs increased the amount of knowledge elicited for a
troubleshooting task.
Formatting results. Data collected from the elicitation stage of CTA is
developed into protocols that reflect the necessary action and decision steps required
to perform a task, along with the required conceptual knowledge, equipment, and
performance standards. Once formatted, CTA protocols can be used to serve varying
purposes, including those of training and instruction.
CTA Effectiveness Studies
There several professional fields that have conducted studies utilizing CTA
methodologies. Hoffman, Shadbolt, Burton, and Klein (1995) cite research on
expertise that has been done in the field of accounting, finance, and taxicab drivers in
which capturing and “preserving” the critical information of expert knowledge was
achieved using CTA. Crandall et al. (2006) referenced weather-forecasting studies
to demonstrate the ways in which CTA, used as an incident-based tool, supported the
investigation of underlying cognitive aspects associated with expert weather
18
forecasting. Industries within the military and medical sciences are some additional
sectors that have used CTA methods and tools to increase the quality in delivery of
services and/or training and instruction by “providing a description of the tasks or
functions that need to be performed“(Crandall et al., 2006, p.37).
Lee (2004) conducted a meta-analytical study of research in the field that
used CTA to test its genrealizability in improving training and performance. The
researcher looked at a wide range of disciplines and identified 318 studies of which 7
studies were selected for the investigation. Selected studies required the following:
an analyst to conduct CTA, the use of CTA methods, pre- and post-test measures as
part of the studies’ design, and studies conducted within the specified time frame of
1985 to 2003. The researchers aggregated data from these studies to find evidence
that post-training performance gains in learner outcomes occurred (mean=75.2%)
when CTA-based instruction was utilized.
There is also evidence of the effectiveness of CTA in a number of studies
concerned with improving practice and/or performance by capturing cognitive
aspects of expertise. O’Hare, Wiggins, Williams, and Wong (1998) conducted a
study of three case studies in which CTA methods were used to develop systems
designs and training requirements for the dynamic and complex environments of
white-water rafting, general aviation, and emergency ambulance dispatching.
Throughout the studies, the authors highlight that CTA methods were used to elicit
the information that went into the design content and systems development of tools
that provided training and/or enhanced human performance in job tasks.
19
Hillburn (2007) used CTA to examine the cognitive elements and error
probabilities involved in two types of operational models involving procedures to
prevent aircraft mid-air collisions. CTA was used to conduct semi-structured
interviews with licensed air traffic controllers to “decompose”, collect, and map
cognitive elements associated with task performance considering the two models.
Though the study did not intend to explain the cause of why errors occur, CTA was
highlighted as a useful tool in facilitating a systemic way to elicit and capture
unobservable cognitive processes and information relevant to decisions, tasks, and
errors possibilities.
The studies presented provide a brief overview of CTA’s use as a tool to
capture unobservable knowledge. These studies are not entirely specialized to the
function of CTA in examining expert knowledge omissions. Yet in still, the studies
do supply the use and effectiveness of CTA to impact areas of instructional design,
human performance, and systems development to improve performance and practice.
Understanding the tools and technologies, such as cognitive task analysis, which can
draw out, capture, and analyze the unobservable knowledge associated with task
performance, provides potential benefits to impacting the instructional content that
affects training, learning, and performance.
CTA Studies in Medicine
In medicine, CTA has already been used to elicit expertise to improve training
and/or performance. Crandall and Getchell-Reiter (1993) conducted 2 studies using
20
Critical Decision Method (CDM) as a tool to elicit and document expert knowledge
from nurses working in the neonatal intensive care unit (NICU). Researchers used
CDM to capture nursing decisions used to assess critically-ill infants. The data
collected from the study resulted in an increase in the number of accurate assessment
indicators nurses provided when making assessments of NICU infants. Additionally,
prior to the study, the knowledge captured from study participants was information
absent from pre-existing training materials.
Surgical education is core to preparing learners for their profession (Aggarwal et
al., 2007) and experts are often utilized to inform the content of instruction and
training materials. Experts can inform instruction because of the information they
contribute to instructional content via lectures, seminars and other pathways for
training and learning (Feldon, 2007). In study conducted in a teaching hospital
setting, Velhamos, Toutouzas, Sillin, Chan, Clark, Theodorou, and Maupin (2004)
used CTA to capture procedural knowledge associated with performing a central
venous catheter placement procedure (CVC) and tracking its effect on training
performance. 26 surgical interns were split into two groups. One group was trained
in a surgical skills laboratory using the results of CTA. The comparison group of
study participants were trained using the traditional “see one, do one, teach one”
method in which surgical interns observed the CVC procedure. Baseline factors
were considered to detect and ensure that there were no differences that may corrupt
the study findings (e.g. experience with CVC) amongst the two groups of surgical
interns and/or patients involved in the application of CVC. Acquisition of technical
21
and theoretical knowledge of CVC by surgical interns was the study focus. The
study results showed that the group trained using the CTA-based curriculum
exhibited statistical gains in post-test scores measuring theoretical knowledge and
technical competence in performing CVC surgical steps correctly as compared to the
study group trained in the curriculum that did not use CTA. Such gains included
taking less time to find the vein, less time for procedure completion, and higher
scores in tests compared to the interns trained under traditional methods.
Research studies in surgical education also present evidence that experts omit
information when describing a task (Feldon and Clark, 2006; Sullivan et al., 2008)
possibly as a result of the nature and development of expertise. Clark et al. (2008)
conducted a study using 10 trauma surgeons with expert knowledge in performing an
emergency shunt procedure using CTA. Researchers examined 3-4 conditions in
which surgical experts described the surgical procedure including: unaided surgical
descriptions, technology-aided descriptions, and descriptions using CTA methods.
Unaided surgical descriptions resulted in 68.75% of omissions of procedural steps;
whereas when CTA methods were used to develop a gold standard a 37.50% gain in
agreement (from 31.25% without CTA to 68.75%) of accuracy and completeness of
surgical descriptions resulted.
CTA has also been used to examine the amount of knowledge omissions
occurring when experts describe a colonoscopy from free-recall. For instance,
Sullivan et al. (2008) conducted a study to elicit and capture critical decision points
and steps involved in a colonoscopy surgical procedure. CTA was used to capture
22
critical information from 3 subject matter experts (SMEs) in the field of colorectal
surgery in performing a colonoscopy surgical procedure. Using CTA processes and
principles, a checklist was developed from the knowledge data yielded. This
checklist served as a guide to assess the number of knowledge omissions when
describing a surgical procedure using traditional teaching methods. The study
revealed that when SMEs provided “free-recall” descriptions of a colonoscopy, at
least 50% of “how-to” steps and at least 57% of critical decisions were omitted from
their descriptions.
Lastly, CTA can add to current training methods by improving instruction and
proficiency in acquisition of procedural skills. Sullivan, Brown, Peyre, Salim,
Martin, Towfigh, and Grunwald (2007) used CTA to present evidence of expert
knowledge omissions through a comparative study of instructional methods for
percutaneous tracheostomy placement (PT) instruction for surgical residents. The
group of surgical residents exposed to a curriculum developed by CTA methods
improved performance on assessments as compared to the group exposed to
traditional teaching and instruction of lecture and demonstration. Interestingly, when
comparing the improved performance of surgical residents receiving the CTA
curriculum with the group exposed to traditional teaching methods, data results from
study findings give persuasive evidence that the lack of performance gains for the
group trained under traditional teaching methods may have been the result of
knowledge omissions of the expert instructor.
23
In short, several studies in the surgical community lend evidence that experts
omit information when describing surgical procedures. In addition, CTA has proven
to be successful in capturing unintentional omissions by experts when describing the
critical information used to perform surgical procedures (Clark et al., 2008; Sullivan
et al., 2008; Velhamos et al., 2004). These implications inform the current study in
using CTA as a data collection tool to examine and describe the percentage of
critical information omitted by surgeons, if any, when describing a surgical
procedure to perform a central venous catheter (CVC) placement procedure. By
capturing the critical information that informs expert performance, CTA can provide
valuable benefits in capturing the knowledge that can improve practice.
The Current Study
Complications and challenges can arise if experts do omit and/or inaccurately
report information that informs content for training. Investigations conducted around
the issue of expert knowledge omissions are an important area of research in many
professional fields, including medicine. Additionally, exploring expert knowledge
omissions requires the appropriate tools with the capacity to access, capture, and
structure the integrative, extremely organized nature of expertise. Hence, research
that can contribute to the field regarding the tools that can unpack and elicit critical
information that drives performance is pertinent to improving practice.
The current descriptive study seeks to build upon research in the field that
suggests surgical experts omit critical information when describing a surgical
24
procedure. The study uses a mixed methods approach in which the CTA
methodology described in Clark et al. (2008) is utilized to collect data from medical
experts through interviews. Quantitative tools are used to analyze the interview data
for evidence of gaps, knowledge omissions, and/or the amount of prior knowledge.
To do this, surgical subject matter experts, or SMEs, are queried about
actions and decision steps required to complete a central venous catheter (CVC)
placement procedure. CTA interview techniques are used to elicit and capture this
information from each SME. Subsequently, the data is analyzed using quantitative
methods to describe conclusions about the presence or absence of critical knowledge
omissions by experts when describing a central venous catheter (CVC) placement
procedure.
The findings of the current study contribute to the body of research related to
expert knowledge omissions as well as the benefits and/or challenges that arise when
using CTA. This is an important area of investigation as both the content of training
programs and the assessment of surgical competence often use observable behaviors
as performance indicators. Furthermore, the instructional content provided relies on
medical expertise. Yet, there is evidence that experts often omit critical information
when describing surgical procedures. Therefore, if training and learning is designed
solely around observable actions, the unobservable cognition that informs task
performance is the additional critical pieces of knowledge subject to omission. This
can lead to serious and/or fatal errors in the field. As such the current study seeks to
answer the following question:
25
Research Question #1: Using CTA, what percentage of critical information
do surgeons omit when describing a central venous catheter (CVC) surgical
procedure?
In addition to the main research question, there was a second research
question that accompanied this study. As presented in the literature, CTA comes in a
variety of forms, strategies, applications, and use (Clark & Estes, 1996: Cooke, 1999;
Crandall et al., 2006; Schraagen et al., 2000). Therefore, CTA’s effectiveness and
appropriateness may vary by task. This is an important distinction for both CTA
practitioners and the sectors for which CTA is utilized. More importantly, is
determining the types of tasks for which CTA can be used. To contribute to this area
in the research, this study uses CTA methodology to examine the second research
question:
Research Question # 2: Does the amount of an expert's prior knowledge and
experience with a central venous catheter (CVC) placement procedure
influence the amount of knowledge about the procedure they omit when
describing how to perform the procedure?
26
CHAPTER 2: METHOD
Study Design
Task
A central venous catheter (CVC) placement surgical procedure was the target
task of this study. It was selected to contribute to the CTA studies currently in the
field that focus on examining challenges in surgical practice regarding expertise
when describing surgical procedures. The objective of a central venous catheter
(CVC) placement procedure is to provide central venous access for multiple reasons
which can include large volume resuscitation and/or to administer medications and
fluids. Although it is a common surgical procedure it involves complex procedural
steps which can result in unnecessary risks and complications for the patient if the
treatment is inaccurately performed. In contrast, the open cricothyrotomy-which is
considered for the second research question- is performed to provide an airway for
patient breathing and involves less procedural steps.
Sample
The study sample included interviews with 4 trauma surgeons from the Keck
School of Medicine at the University of Southern California, 1 critical care internist
also from the Keck School of Medicine at the USC, and 1 anesthesiologist from City
of Hope Medical Center. The surgeons were selected based upon their expertise in
performing a central venous catheter (CVC) surgical procedure. For this study, an
expertise criterion was indicative of the surgical expert’s years of experience,
27
domain expertise, and performance rate in performing a central venous catheter
(CVC) placement procedure. As suggested by Patton (2002), interviews can yield
data rich in quantity and depth of description. As such, the sample size was small
and the sample selection was intentional to allow for depth in data collection and
analysis of data that informs the answer to the study’s research questions. Given
this, participant selection utilized purposeful sampling techniques. This study was
determined EXEMPT by the Institutional Review Board (IRB) of the University of
Southern California’s Health Science Campus.
Procedure for Data Collection
According to Clark et al. (2008), CTA is conducted in five stages which
include (a) collection of preliminary knowledge (b) identify knowledge
representations (c) application of knowledge elicitation methods (d) data analysis
and verification of data collected and (e) formatting of results for its intended
application.
In regards to research question #1 and following the approach described by
Clark et al. (2008), the data collection procedure was delineated as follows:
Phase 1: Collection of preliminary knowledge
The activities in this stage utilized document analysis to research the central
venous catheter (CVC) placement procedure. Resources included medical books and
other materials to familiarize the interviewers with the procedure. This phase also
28
included identifying additional names for a central venous catheter placement
procedure which include central line placement and CVC.
Phase 2: Identify knowledge representations
Preliminary research from Phase 1 was used to generate a preliminary list of
knowledge types and/or structures used in performing a central venous catheter
(CVC) placement surgical procedure. Following a previous interview conducted in
conjunction with this study, a flow chart was utilized to organize the information
captured from the SMEs involving a central venous catheter (CVC) placement
procedure. An online medical dictionary was accessible to familiarize the researcher
with unfamiliar terms that may impede the productivity of interviews and/or hinder
the process of organizing knowledge types due to medical terminology.
Phase 3: Application of knowledge elicitation methods
CTA semi-structured interviews were conducted with six subject matter
experts. These experts are referred to as Surgeon A, B, C, D, E, and F within this
study. This phase was repeated for each SME. Audio recording equipment was used
during the interviews. Semi-structured interviews were used to elicit major tasks
performed by the SME as well as information about performing key sub-tasks and
other information from each SME (Clark et al., 2008). Major tasks elicited include:
29
1. The order of actions or steps required in performing the sub-task.
2. The decisions that inform performance of the sub-task, including
alternatives and the decisions that inform using alternatives.
3. The conceptual knowledge SMEs use to move toward the sub-tasks which
include all relevant concepts, processes, and principles; this step will include clarity
of new concepts and/or processes presented.
4. The conditions or events that precede and initiate the correct procedure
in addition to any equipment and materials needed for task completion.
5. The sensory experiences that contribute to completion of the task (i.e. the
researcher will inquire if the SME uses smell, touch, taste or other sensory cues to
perform each sub-task).
6. The performance standards required (e.g. time, accuracy, speed, or quality
indicators.
Phase 4: Data analysis and verification
This phase involved the researcher transcribing and coding the data provided
by the SMEs. The purpose of coding was to assist in developing a final document,
or CTA protocol, free of inaccurate and/or unnecessary steps and indicative of only
the essential tasks and steps in performing a central venous catheter (CVC)
placement procedure. Inter-coder reliability methods were achieved by using a team
of 2-3 raters to code the data. Rater coding was compared and analyzed for
30
agreement by tallying the total number of agreements and dividing this by the total
number of coded items.
During Round 1 of the data collection process, the coded document was
formatted as a CTA protocol and given back to each SME for review to verify
completion and accuracy of the information. This was done by instructing SMEs to
review the draft CTA protocol as a Microsoft Word document in which the
track/changes function was activated. Each SME reviewed the draft protocol and
provided feedback regarding the completeness and accuracy of information. Any
additions made be the SME were documented in excel as Round 2 of the data. As a
note, the researcher accepted the tracked changes resulting from the SME review of
their draft protocol.
Phase 5: Formatting of results
Each SME corrected CTA protocol was combined to develop one draft
protocol, which is indicative of the essential information and steps in performing a
central venous catheter (CVC) placement procedure. This protocol served as a “gold
standard” draft protocol and was given to another senior expert to review and edit for
completeness and accuracy to produce a corrected and final gold standard protocol.
Analysis of the Data
The current study was designed to capture expert knowledge using CTA to
describe omissions of critical knowledge when describing a central venous catheter
31
(CVC) surgical procedure. The data analysis involved comparing the CTA protocols
with the gold standard protocol to answer the study’s research questions.
Research Question 1: What percentage of critical information do surgeons omit
when describing a central venous catheter (CVC) placement surgical procedure?
A Microsoft excel spreadsheet was created which reflects information from
the gold standard protocol. Using frequency counts, Round 1 data provided from
each surgeon CTA protocol was compared and analyzed with Round 2 data to
determine any additions affecting the percentage of omissions. Subsequently, Round
2 data was used to develop a draft protocol which included the cumulative count of
information amongst the study participants. After developing, a final gold standard
protocol from all data, this data was compared and analyzed with the gold standard
protocol to examine the cumulative percentage of omissions. “1” was used to
indicate inclusion of the knowledge each SME provides that was in agreement with
the gold standard. “0” was used to indicate the omissions or exclusion of the
knowledge. An omission is described as an indication on the gold standard protocol
that is not included in the CTA protocols of the SMEs’ descriptions. As, the central
venous catheter placement (CVC) procedure involved three possible surgical sites
from which to begin the procedure, decision steps and action steps were calculated at
the first site mentioned, but were considered repeated steps and zeroed out for
subsequent reoccurrences in other sites. Frequency counts were used to convert to
percentages and calculate the percentage of agreement. This percentage was
32
subtracted from 100% to reveal the total percentage of knowledge omissions for
Round 1 and Round 2 data provided from the SMEs’ descriptions. Additionally,
data analysis included analyzing the knowledge omission per CTA item of the
protocol (e.g. # of items in equipment and materials, # of items in task 1, etc.) in
addition to the average decision step agreement amongst all SMEs regarding the
entire surgical procedure. The average step of agreement is equal to the following:
[(Total SME items for CVC procedure)/Total number of SMEs]/Total GS items
Research Question 2: Does the amount of an expert's prior knowledge and
experience with a central venous catheter (CVC) placement procedure influence the
amount of knowledge about the procedure they omit when describing how to perform
the procedure?
As stated, the second research question of this study involves examining the
amount of expert prior knowledge and experience with a central venous catheter
(CVC) placement procedure and the amount of knowledge omissions. To answer the
second research question, data and results from Phase 4 of a similar CTA study
conducted on an open cricothyrotomy procedure (Tolano-Leveque, 2010) were
compared to the results of this study. The researcher used the average step
agreement and omissions of Round 1 data of decision steps and action steps to
compare and analyze the two studies. The following data was collected from both
studies as predictors of prior knowledge: (a) number of years as a surgeon, (b)
number of times performed the study’s procedure, (c) number of times procedure
33
was performed within the last 12 months, and (d) number of times the procedure was
taught in the last 12 months. From here, the percentage of knowledge omissions
between the two procedures was calculated to describe any differences in the results
that could answer the percentage of knowledge omissions based on prior knowledge.
34
CHAPTER 3: RESULTS
This chapter presents the results to answer the two questions concerning
expert knowledge omissions when describing surgical procedures. Following the
methodology outlined in chapter 2, six semi-structured interviews were conducted to
elicit surgical descriptions from subject matter experts regarding a central venous
catheter placement procedure. Participant selection was not randomized and the
study participants were selected based on their expertise in performing a central
venous catheter placement. The participants were all male.
Inter-rater Reliability
Semi-structured interviews were used to elicit expert descriptions in
performing a central venous catheter placement. Interview data was transcribed and
coded. Coding was conducted in research teams and interrater reliability was
completed. Interrater reliability was calculated by tallying the total coded items of
agreement amongst the rating team and dividing this number by the total items coded
(which was equivalent to the sum of tallied agreements and disagreements). The
resulting percentage was the interrater reliability. The mean percentage for interrater
reliability equaled 96.46 with 99.61% being the highest percentage of rater
agreement and 92% being the lowest percentage of rater agreement.
Six CTA protocols were developed from coded interview data (Round 1).
Protocols were returned to each respective SME for review. Any edits provided by
SMEs went into developing a second CTA draft protocol (Round 2 data). The draft
35
protocol was reviewed by one additional senior expert for accuracy and
completeness. After the senior experts review, a final CVC gold standard (GS)
protocol was developed.
Data Analysis
Microsoft Excel was used for analysis of the percentage of critical
information omitted when describing a central venous catheter placement procedure.
SME descriptions were compared to the gold standard-first, using Round 1 data-then
aggregating Round 1 and Round 2 data to reanalyze for agreement. SMEs additions
are defined as edits made during the review of the initial CTA report. 1’s were used
to indicate SME items in agreement with the gold standard. O’s were used to
indicate items that SMEs did not include when compared to the gold standard. The
total number of categorical items per SME were added and divided by the number of
possible items per CTA category to yield the percentage of agreement. This number
was subtracted from 100% to calculate the presence of knowledge omissions. As a
central venous catheter placement (CVC) procedure involved three choices of
surgical sites from which to start the procedure, repeated decision steps and action
steps were zeroed out. Furthermore, CTA categories for analysis included:
conditions, equipment, action steps, and decision steps.
36
Research Questions
Research Question 1:
What percentage of critical information do surgeons omit when describing a central
venous catheter (CVC) placement surgical procedure?
In the data analysis, critical information is defined as the conditions,
equipment, action steps, and decision steps necessary to perform a CVC placement
procedure. Tables 1-6 present the descriptive statistics to answer research question
#1. The data has been disaggregated per CTA item for more detail. Round 2 data, if
applicable, is discussed in the narrative following each table. Round 2 refers to any
edits made by SMEs during the review phase of their initial draft CTA report.
Conditions
Gold standard items coded as condition (e.g. indications, contraindications)
are indicative of the knowledge SMEs use to decide when (or when not) to perform a
central venous catheter (CVC) placement procedure. Tables 1 and 2 provide the
results of this data.
Table 1: Indication omissions
Round 1 Round 2
# of GS items: 4
SME Omissions
Median 1 1
Mode 1 1
SD 0 .408
Range
Mean
0
1
1
.833
% of omissions 25.00% 20.83%
37
There were 4 indications listed on the gold standard for when to perform a
CVC procedure. During Round 2, when describing a CVC surgical procedure, only
1 SME made an addition to their description of indications. After the addition, this
same SME had 0% knowledge omissions when compared to the gold standard. The
mean score for omissions of indications changed from Round 1 to Round 2. There
was also a 4.17% decrease in the percentage of knowledge omissions from Round 1
to Round 2 data.
Table 2: Contraindication omissions
Round 1 Round 2
Total GS items: 4
SME Omissions
Median 2 2
Mode 2 2
SD .753 .753
Range 2 2
Mean 2.167 2.167
% of omissions 54.17% 54.17%
According to the CVC final gold standard, there are 4 contraindications for a
CVC procedure. 3 out of the 6 SME study participants omitted 50% of
contraindications relevant for when not to perform a central venous catheter (CVC)
placement procedure when compared to the gold standard. 2 of 3 SMEs omitted
75% of contraindications regarding when not to perform a CVC procedure and 1
SME omitted 25% of critical information associated with a CVC procedure as
compared to the gold standard. There were no additions made during the Round 2
review.
38
The average mean score for omissions regarding contraindication items is
2.167 (SD ± .753). There were no additions made by SMEs from Round 1 to Round
2. The average percentage of omissions is 54.17%. Cumulatively, when describing
critical information related to the conditions of when to (or not to) perform a central
venous catheter (CVC) placement procedure, more than ½ of gold standard
contraindications were omitted.
Equipment
For the current study, equipment refers to the materials, tools, and
technological items required for performing a CVC procedure. The CVC protocol
generated for this study indicated 30 items including: ultrasound, personal protective
gear (such as surgical mask, cap, and gown), and items such as a central line kit and
its contents. The following Table 3 presents the results associated with the CTA
section pertaining to equipment.
Table 3: Equipment omissions
Round 1 Round 2
Total GS items: 30
SME Omissions
Median 13 13
Mode - -
SD 5.865 6.593
Range 16 17
Mean 14 13.33
% of omissions 46.67% 44.44%
There were 30 items indicated on the gold standard regarding the necessary
equipment for performing a CVC procedure. For 5 of the 6 SMEs, there was no
39
change in percentage of agreement and percentage of knowledge omissions after
review of Round 1 data. However, 4 additions were made by 1 SME after review of
their Round 1 data. Respectively, this increased their percentage of agreement to the
gold standard (from 66.67% to 80.00%) and decreased their percentage of
knowledge omission (from 33.33% to 20.00%).
On average, Round 1 data reflects that 46.67% of items were omitted when
describing the equipment involved in performing a central venous catheter (CVC)
placement procedure. During Round 1, the mean score of omission was 14 (SD ±
5.865) which is roughly half the total number of equipment listed on the gold
standard. There is a decrease during Round 2 in the mean score of omissions (from
14 to 13.33). This change in mean score is the result of 4 additions made by SME D
during Round 2 data. Additionally, there is a decrease in the percentage of
omissions for Round 2 (from 46.67% during Round 1 to 44.44% in Round 2).
Action and decision steps
Action steps and decision steps refer to those items that provide “how-to”
procedural information for performing a central venous catheter (CVC) placement
procedure. Action steps refer to the items that SMEs observably perform while
decision steps refer to the unobservable cognition and/or decisions that can inform an
action. Table 4 provides the data results for action steps while the following Table 5
gives the results for items relating to decision steps.
40
Table 4: Action step omissions
Round 1 Round 2
Total GS items: 44
SME Omissions
Median 13 12.5
Mode 13 -
SD 5.529 5.231
Range 17 16
Mean 13.167 12.833
% of omissions 29.92% 29.17%
There were a total of 44 items on the gold standard regarding action steps for
performing a central venous catheter placement procedure. The lowest percentage of
knowledge omissions during Round 1 was 11.36%; the highest percentage was
50.00%. There was 1 addition each made by 2 SMEs during Round 2 which
decreased the percentage of knowledge omissions by 2.27% and 2.28% respectively.
Any additions that were subject to the repeated action and/or decision step criteria
were not calculated. As such, there were no calculated additions provided by the
remaining 4 SMEs during Round 2. When describing a central venous catheter
(CVC) placement procedure, the average mean percentage of omissions of action
steps in Round 1 was 29.92% (SD ± 5.529). These additions reflected a 0.75%
decrease of knowledge omissions in Round 2 (from 29.92% to 29.17%).
41
Table 5: Decision step omissions
Round 1 Round 2
Total GS items: 14
SME Omissions
Median 5 5
Mode 5 5
SD .753 .753
Range 2 2
Mean 4.833 4.833
% of omissions 34.52% 34.52%
14 decision steps are indicated on the CVC gold standard pertaining to
decision steps. In Round 1, 3 out of 6 SMEs omitted 35.71% of decision steps when
describing a central venous catheter (CVC) placement procedure. 2 SMEs omitted
28.57% of decision steps and 1 SME omitted 42.86% of decision steps. The highest
reoccurring percentage of knowledge omission was 35.71%. Additions that were
subject to the repeated step criteria were not tabulated. As a result, there were no
additions made to decision steps reflected in the Round 2 data.
Overall STEP agreement and omissions for entire CVC procedure
Table 6 presents data results for the overall percentage of information omitted
(or not omitted) when describing a central venous catheter (CVC) placement
procedure. This data is a cumulative total of the entire critical information involved
in performing a central venous catheter (CVC) placement procedure according to the
items listed on the CTA gold standard.
42
Table 6: Overall step omissions for entire CVC procedure
Round 1 Round 2
Total GS items: 96
SME Omissions
Median 33 32
Mode - -
SD 9.968 10.696
Range 25 28
Mean 35.167 34
% of omissions 36.63% 35.42%
There were 96 total steps indicated on the gold standard regarding the
conditions, equipment, actions, and decisions steps necessary to performing a central
venous catheter (CVC) placement procedure as described by six SMEs. In Round 1
data, the lowest number of knowledge omissions (25.00%) belonged to SME D. The
highest number of knowledge omissions respective to the critical information
involved with a CVC procedure belonged to SME C (51.04%). The remaining 4
SMEs showed knowledge omissions within a range of 30%-50% when describing a
central venous catheter (CVC) placement procedure.
For all SMEs, the data analysis conducted for Round 1 reveals that the
average step of knowledge omissions of all critical items (e.g. conditions, equipment,
actions, decisions) related to the CVC is 36.63%. The mean score of omissions of
gold standard items was 35.167 (SD ± 9.968). Considering the cumulative additions
made during Round 2, the percentage of knowledge omissions decreased to 35.42%.
As such, the mean score of items omitted for the entire procedure decreased to 34
(SD ± 10.696). The range of difference between Round 1 data and the aggregated
total is 1.21%.
43
Research Question 2:
Does the amount of an expert's prior knowledge and experience with a central
venous catheter (CVC) placement procedure influence the amount of knowledge
about the procedure they omit when describing how to perform the procedure?
Another area of analysis for this study was to examine the amount of an
expert's prior knowledge and experience with a central venous catheter (CVC)
placement procedure and its influence in the amount of knowledge about the
procedure they omit when describing how to perform the procedure. Data was used
from a similar study involving an open cricothyrotomy procedure (Tolano-Leveque,
2010). In both studies, prior knowledge predictors include: (a) number of years as a
surgeon, (b) number of times performed the study’s procedure, (c) number of times
procedure was performed within the last 12 months, and (d) number of times the
procedure was taught in the last 12 months.
Table 7 provides data regarding SME agreement and knowledge omissions
related to action steps when describing a central venous catheter (CVC) placement
procedure and the data from the open cricothyrotomy (CRIC) study (Tolano-
Leveque, 2010). Table 8 presents the data results for decision steps. Included in
Table 7 and Table 8 are the mean data scores regarding the predictors of prior
knowledge. Table 7 and Table 8 data results are reflective of post-Round 1 data
only. 4 of the 6 SMEs that were used during the study regarding the central venous
catheter placement procedure also participated in the open cricothyrotomy study. In
44
total, 8 SMEs were interviewed collectively amongst the two studies. However, only
6 of the 8 SMEs were available during this portion of the data collection.
Table 7: Prior knowledge and action step omission
CVC CRIC
# SMEs (N)
Years as a surgeon
# times performed procedure
# performed procedure within last 12 months
# times taught procedure within last 12 months
5
17
-
17.2
19.4
4
12.75
6.5
.75
5.75
Total # of GS action steps
SME mean agreement
SME mean omissions
44
30.833
13.167
33
19.167
13.833
Mean percentage of SME agreement per
procedure
70.08% 58.08%
Percentage of SME omission per procedure
29.92% 41.92%
Table 8: Prior knowledge and decision step omissions
CVC CRIC
# SMEs (N)
Years as a surgeon
# times performed procedure
# performed procedure within last 12 months
# times taught procedure within last 12 months
5
17
-
17.2
19.4
4
12.75
6.5
.75
5.75
Total # of GS decision steps
SME mean agreement
SME mean omissions
14
30.833
13.167
13
3
10
Mean percentage of SME agreement per
procedure
65.48%
23.08%
Percentage of SME omission per procedure 34.52% 76.92%
Using Table 7 and Table 8, SMEs that participated in the central venous
catheter placement procedure study possess more prior knowledge of the CVC
procedure when compared to data regarding the prior knowledge indicators for
45
participants of the open cricothyrotomy procedure. However, the average percentage
of knowledge omissions is higher for the open cricothyrotomy procedure in both
decision steps (76.92%) and action steps (41.92%) when compared to the data
regarding knowledge omissions for the central venous catheter placement study.
When queried about the number of times performing the CVC procedure, SMEs
gave nominal data reports (i.e. between 100-1000 times) instead of interval data. As
a result, the number of times performing the CVC procedure was not calculated.
Summary
Percentage of Knowledge Omissions
Table 9 provides an overview of the amount of omissions that occurred for 6
SMEs when describing a central venous catheter placement procedure by CTA item
type. Data for both Round 1 and Round 2 are included. Contraindications and
indications have been combined into one item and referred to in the table as
“Conditions”. Aggregated data for the percentage of action and decision steps has
also been included.
Table 9: Summary of all omissions by CTA item type
Round 1
% Omissions
Round 2
% Omissions
Change
Conditions 39.58% 37.50% 2.08%
Equipment 46.67% 44.44% 2.23%
Action 29.92% 29.17% 0.75%
Decision 34.52% 34.52% 0.00%
Action and Decision 31.03% 30.46% 0.57%
All Steps 36.63% 35.42% 1.21%
46
In summary, the highest mean score of knowledge omissions occurred in
equipment. The lowest mean score of knowledge omissions occurred in action steps.
There are no Round 2 additions made for decisions steps, so the percentage of
omissions did not change.
Cumulatively, there were 58 action and decision steps indicated on the gold
standard for performing a central venous catheter placement. The highest percentage
of knowledge omissions (46.55%) occurred for 1 SME during Round 1; the lowest
percentage (17.24%) occurred for 1 SME. The remaining 4 SMEs omitted between
29% and 32% of action steps and decision steps. There was a percentage decrease
for 2 SMEs (less than 2%) as a result of additions made in action steps during Round
2. Additionally, there is a higher percentage of knowledge omissions of decision
steps when compared to action steps.
Excluding decision steps, for each item type listed on the CTA, there was at
least one addition made to Round 2. The average percentage of decrease as result of
Round 2 additions was 1.37%.
Omissions based on Prior Knowledge
The central venous catheter (CVC) placement procedure involved 14 decision
steps and 44 action steps. There were 13 decision steps and 33 action steps
indicative of gold standard items to perform an open cricothyrotomy procedure.
Between the central venous catheter placement and the open cricothyrotomy, there
was a difference of 12.00% for action steps omitted and 42.40% for decision steps
47
omitted in which the open cricothyrotomy had the highest amount of knowledge
omissions when comparing the data between the two studies. Additionally, SMEs
data for the CVC study reflect that participants possessed a higher amount of prior
knowledge when compared to the prior knowledge data for the open cricothyrotomy
study.
48
CHAPTER 4: DISCUSSION
The purpose of this descriptive study was to use cognitive task analysis to
identify the critical information omitted when experts describe surgical procedures.
The first research question of this study sought to describe the percentage of critical
information omitted when describing a central venous catheter placement procedure.
The second research question sought to compare prior knowledge and knowledge
omissions between two surgical procedures. The current discussion provides both an
interpretation of the data and explanations of the results. A review of the study
limitations, possible recommendations, and future next steps for this line of research
are also discussed.
Research Question 1
The first research question posed for analysis in the current study was:
What percentage of critical information do surgeons omit when describing a central
venous catheter (CVC) placement surgical procedure?
When compared to the gold standard, 36.46% of information was omitted
during Round 1 for the overall steps involved in performing a CVC. During Round
2, 34.52% of overall steps were omitted. Using CTA, the lowest percentage of
knowledge omissions was reported in indications for both Round 1 and Round 2
data. In total, knowledge omissions occurred at every gold standard item when
describing a central venous catheter placement procedure with the exception of
49
decision steps. When compared to action steps, there was a higher percentage of
decisions step omissions.
The results of the data confirm existing literature suggesting CTA can be
used to capture the unobservable cognitions involved in performing a task.
Moreover, experts do omit critical information when describing a central venous
catheter placement surgical procedure. In addition, lessons were learned regarding
the benefits and challenges of using CTA that may be considered for similar, future
studies.
Number of items
As a group, omissions by SMEs were greater than 40% (with the exception of
the “Equipment” category) when 2 or more items were omitted for gold standard
categories with 4 or less gold standard items. This was the case when disaggregating
the data for conditions (indication and contraindications). However, interpretation of
this data suggests the possibility that a lesser number of items per CTA category may
increase the occurrence of knowledge omissions when describing a central venous
catheter placement procedure. Still, as this range was not consistent throughout the
study with all categories, it is feasible to suggest that the low number of gold
standard items may have inflated the knowledge omissions percentage for the data
relative to the contraindications in the CTA items of conditions. Disaggregating the
sub-items within items may prove useful in data analysis for future studies.
50
Generalizing knowledge omissions
Based on the results from the current study, the findings reveal that 30%-60%
of knowledge omissions occur when describing a central venous catheter (CVC)
placement procedure. This range is smaller than what has been proposed in literature
regarding knowledge omission when describing a surgical procedure. Yet, estimates
in the literature often referred to instances of free-recall, where as the current study
used CTA methods to elicit information upon the start of SME descriptions.
Furthermore, the evidence of knowledge omissions presented in the data confirms
existing literature that proposes experts do omit a percentage of information via
descriptions of surgical procedures. Lastly, this study affirms evidence that CTA
methods can be useful as a data collection device to elicit unobservable cognition
that informs performance.
Research Question 2
The second question for analysis in this study was: Does the amount of an
expert's prior knowledge and experience with a central venous catheter (CVC)
placement procedure influence the amount of knowledge about the procedure they
omit when describing how to perform the procedure?
In the “Results” chapter, there was a 42.40% difference in the number of
knowledge omissions for decision steps and a 12.00% difference for action step
omissions reflected in the data between a CVC procedure and an open
cricothyrotomy procedure. In both instances, there were a higher percentage of
51
knowledge omissions in the open cricothyrotomy study when compared to the
amount of knowledge omissions for the central venous catheter placement study.
Additionally, SMEs for the open cricothyrotomy study had less prior knowledge of
the procedure when compared to SMEs from this study. The following has been
considered to explain these results.
Prior Knowledge
Several of the SMEs reported during their interviews that the open
cricothyrotomy is a rarely performed procedure as compared to the central venous
catheter placement which is performed on a more regular basis. This is reflected in
the action and decision steps for Table 8 and Table 9 of the “Results” section. This
would suggest that although the SMEs have procedural knowledge of the open
cricothyrotomy and could perform it an expert level, their ability to describe the
procedure resulted in more knowledge omissions when compared to the data of the
central venous catheter placement-a more routinely performed procedure.
Research on expertise theorizes the more automation that occurs from
deliberate practice in developing expertise, the higher possibility of omissions of
critical information. Moreover, the higher amount of automated prior knowledge an
expert possesses, would suggest the greater the possibility of omitting information
due to the nature of expertise and the challenges of automaticity. This proposes that
the central venous catheter placement procedure should have yielded a higher rate of
52
omission. On the contrary, when the results were compared for analysis, this was not
the case.
One possible explanation of the results may be that the type of prior
knowledge SMEs possess could serve as an indicator of the amount of knowledge
omissions present-or not present-for the respective procedures. For instance, 5 of the
6 SMEs work within a medical institution in which an organizational policy was
implemented in 2009 in response to issues of patient care resulting from
complications associated with central venous catheter placement procedures. There
is no policy that has been implemented for the open cricothyrotomy procedure within
this same institution. Given this, the type of prior knowledge SMEs had regarding
the complications and issues with the central venous catheter placement procedure
may have affected their omissions rate by allowing SMEs to mentally attend to those
areas that have received popular attention. Unfortunately, study limitations did not
allow for the elicitation of this additional data. More research is needed in
understanding how the type of prior knowledge in the form of heightened and/or
popular awareness of a procedure affects descriptions of knowledge omissions.
Study Limitations
In general, the study findings supported existing literature that suggests
experts omit information when describing a surgical procedure. Yet as presented in
the earlier discussion, the results do not completely coincide with all that has been
reviewed in the area of expert knowledge omissions. What follows is a discussion of
53
several study limitations and possible recommendations that may enrich future
studies interested in continuing similar types of research.
Crediting SMEs/Partial Credit
The current analysis did not accommodate instances for SMEs to receive
partial credit for gold standard items. As a result, there were additions made by
SMEs that were not indicated in the data analysis. This is the result of a SME
already receiving item credit during Round 1 and providing an addition that could
not be reflected in the current structure of the analysis during Round 2 because the
SME had already received credit. This situation mainly occurred during items in
which similar steps were “rolled-up” into one item step for the purpose of better
organization and clarity in the gold standard. Thus, there were several instances in
which a SME made a contribution of additional information in Round 2 to
preexisting Round 1 information. As Round 1 credit was already given, there was no
mechanism to account for the added information (e.g. double-crediting).
Furthermore, double-crediting SMEs for additions was tested by the researcher and
resulted in inconsistencies in the data. As a result, initial credit was given to SMEs,
but secondary credit was not for Round 2 additions to the same, respective step.
Round 2 additions were only given to SMEs in instances of “pure” additions
meaning no Round 1 credit was present. Finally, in situations where a SME made an
addition that met the criteria of a repeated decision step, the addition was not
counted, but instead zeroed-out during analysis.
54
SME Participation
The study participants were required to participate in multiple processes that
required both their time and mental effort. From the initial interview process to
collect data, to several phases of reviewing information and providing feedback, a
high commitment and level of participation was required from SMEs. Without
isolating and/or eliminating the daily responsibilities of each study participant, the
probability exists that some SMEs were unable to dedicate their full mental effort
during the editing and review of CTA draft and protocols. As such, there exists the
possibility that additional data and information by SMEs may have been overlooked
that would serve to enrich the current data in understanding expert knowledge
omissions regarding surgical procedural descriptions. The researcher recommends
that further studies seeking to replicate the current one-if available-utilize a separate
population of SMEs allocated to the review and edit phase with the purposes of
ensuring a thorough and complete review.
Summary
CTA methods were used to collect data to conduct the current study’s
examination of the percentage of critical information omitted when describing a
central venous catheter placement procedure. Although, the results yielded did not
fall routinely within the knowledge omissions estimates outlined in earlier studies,
using CTA did create access to eliciting information critical to performing a CVC
procedure. As a result, data analysis provided results that suggest that experts do
55
omit critical information when describing a surgical procedure. This may have
implications for future studies.
Implications
Training, Instruction
Automaticity is a central component of expertise (Clark & Estes, 2006) with
both benefits and challenges. Automaticity supports increases in the capacity of
working memory to attend to novel problems, but impedes experts from accessing
this knowledge because it has become non-conscious to the expert knowledge
holder. Considering this, experts’ knowledge is not mentally attended to by the
experts and vulnerable to unintentional omissions of information when reporting out.
Consequently, there is potential for surgical experts to omit significant amounts of
critical “when, how, and why” information when describing surgical procedures
and/or tasks.
Lack of knowledge can affect performance (Clark & Estes, 2002), but
improving surgical education to better prepare surgical practitioners is not an easy
feat. Especially, amongst the many demands that the sector is currently facing (e.g.
accreditation requirements, patient safety, decreases in the work week) in addition to
the challenges concerning the type and amount of teaching and/or clinical exposure
surgical learners receive. Innovation in the field is making way for evolving learning
technologies to address sector needs to enhance technical skills and competencies of
its professionals. With teaching and instruction serving as pertinent forces for
56
developing and continuing surgical practitioners, the resources that inform the
instructional content of cutting-edge teaching tools must not only be efficient, but
must also harness the complete and accurate information to provide surgical novices
with the tools for good practice.
Expertise
Expertise has played a pivotal role in surgical education. Yet, the cognitive
nature of expertise has inherent challenges that may compromise training and
instruction. Additionally, capturing expertise may provide the surgical community
with an increased capacity to develop “gold standards” that preserve the critical
information of expertise. These “gold standards” provide measurements from which
to frame and assess advanced performance (Ericsson, 2004) while establishing
benchmarks of success for surgical novices to pursue. For a sector that exists on the
philosophy of “doing no harm”, this is an important area of research. As such, the
field of medicine-in general-should consider tools, such as CTA, to capture the level
of knowledge that can ensure the skills and competencies towards quality of care.
Using CTA to examine knowledge omissions
This study’s main concern was to reflect the most accurate data involving
knowledge omissions. As presented in Chapter 1, there is an array of CTA types
available for use. As a result, careful attention to the type of CTA selected should
align appropriately with its intended use (Clark & Estes, 1996: Cooke, 1999;
57
Crandall et al., 2006; Schraagen et al., 2000). For instance, a CTA protocol
developed for training purposes may differ from a protocol designed to explicitly
analyze, present, and credit knowledge holders in the cognitive elements necessary to
performing a task. For a CTA training protocol, organization of information can
optimize or impede learning. Strategies such as “rolling-up” information or
combining action steps into one action step may prove beneficial in effectively
formatting the amount of information to maximize processing new information by
the learner. In effect, “chunking” material may reduce information loads on working
memory. However, for a study such as this which proposes to explicitly examine
knowledge omissions, it is essential to disaggregate and not combine items to
improve analysis efforts in identifying knowledge omissions per SME. Given this, it
is recommended that data analysis tools and methodology require flexibility to
accurately and appropriately optimize CTA methods (Roth, 2008) for the desired
use. Finally, CTA methods as a device for research data collection and analysis are
promising, yet more research is needed in understanding how best to align CTA
methods with the type of research and analysis being conducted.
Critical Information
The current study looked at the critical information involved in performing a
CVC procedure. The items for review included the conditions, equipment, action,
and decisions steps that informed the selected task of a central venous catheter
placement procedure. However, these were not the only items elicited from study
58
participants. For instance, risks and benefits were another area elicited during
interviews, but the data was not applicable for use in this study. Hence, future
research studies within this field of interest may consider examining additional types
and amounts of knowledge omissions that inform task performance beyond the ones
examined within this paper.
Conclusion
Surgical education has been undergoing an instructional renovation for 20
years (Scott et al., 2008). Emerging instructional methods, such as simulation, train
students by providing opportunities for expertise development in skills that may not
easily be supported and/or not suitable for the current structure and demands of
learning in the surgical field. The changing environment of surgical education
coupled with the push to improve competency and professional practice makes the
need to capture accurate and complete descriptions of expert knowledge pertinent to
better support the strategies for learning and/or instruction.
The purpose of this descriptive study was to use CTA methods to build upon
research in the field that suggests surgical experts omit critical information when
describing a surgical procedure. The researcher tested and found evidence that
experts omit an average of 37% when describing critical information involved in
performing a central venous catcher placement surgical procedure. Although this is
less than the amount of percentage described in pervious studies regarding this topic,
it is still more than 1/3 of information that may be absent from training and
59
instruction. Consequently, more research is required in understanding how best to
capture both the observable actions and unobservable decisions that drive learning,
instruction, and performance.
60
REFERENCES
Aggarwal, R., Grantcharov, T., & Darzi, A. (2007). A framework for systematic
training and assessment of technical skills. American College of Surgeons,
204 (4), 697-705.
Barnes, R., Lang, N., & Whiteside, M. (1989). Halsteadian Technique Revisited:
Innovations in teaching surgical skills. Annals of Surgery, 210(1), 118-121
Chao, C. & Salvendy, G. (1994). Percentage of procedural knowledge acquired as a
function of the number of experts from whom knowledge is acquired for
diagnosis, debugging, and interpretation tasks. International Journal of
Human-Computer Interaction, 6 (3), 221-223.
Chi, M. T. H. (2006). Two approaches to the study of experts’ characteristics. In K.
A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The
Cambridge handbook of expertise and expert performance (pp. 21–30). New
York: Cambridge University Press.
Clark, R. (in press). How much and what type of guidance is optimal for learning
from instruction? In S.Tobias, & T. M. Duffy (Eds.), Constructivist Theory
Applied to instruction: Success or Failure? (pp. 158-183). New York:
Routledge, Taylor and Francis. Retrieved from:
http://www.cogtech.usc.edu/recent_publications.php
Clark, R. (2008). Resistance to change: Unconscious knowledge and the challenge of
unlearning. In D.C. Berliner & H. Kupermintz (Eds.), Changing institutions,
environments and people. New York: Rutledge (pp. 75-94). Retrieved from:
http://www.cogtech.usc.edu/recent_publications.php
Clark, R. & Elen, J. (2006). When less is more: Research and theory insights about
instruction for complex learning. In J. Elen & R. E. Clark (Eds.), Handling
complexity in learning environments: Theory and research (pp. 283–295).
New York: Elsevier.
Clark, R. & Estes, F. (2002) Turning research into results: A guide to selecting the
right performance solutions. CEP Press: Atlanta, Georgia.
Clark, R. & Estes, F. (1996). Cognitive Task Analysis for Training. International
Journal of Educational Research, 25(5), 403-417. Retrieved from:
http://www.cogtech.usc.edu/recent_publications.php
61
Clark, R. Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2008). Cognitive
task analysis. In J.M. Spector, M.D. Merrill, J.J.G. van Merrienboer, & M.P.
Driscoll (Eds.). Handbook of research on educational communications and
technology (3rd ed.). Mahwah, NJ: Lawrence Erlbaum Associates. Retrieved
from: http://www.cogtech.usc.edu/recent_publications.php
Clark, R., Pugh, C., Yates, K., & Sullivan, M. (2008). The use of cognitive task
analysis and simulators for after action review of medical events in Iraq.
Technical Report 5-21-2008 developed for the Center for Cognitive
Technology, Rossier School of Education, University of Southern California.
Clark, R., Yates, K., Early, S. & Moulton, K. (2009). An Analysis of the Failure of
Electronic Media and Discovery-based learning: Evidence for the
performance benefits of Guided Training Methods. In Silber, K.H. & Foshay,
R. (Eds). Handbook of Training and Improving Workplace Performance,
Volume 1: Instructional Design and Training Delivery. Somerset, N.J.:
Wiley. Retrieved from: http://www.cogtech.usc.edu/recent_publications.php
Cooke, N. J. (1999). Knowledge elicitation. In F. T. Durso (Ed.), Handbook of
Applied Cognition, (pp. 479-509). U.K.: Wiley.
Crandall, B., & Gretchell-Leiter, K. (1993). Critical decision method: A technique
for eliciting concrete assessment indicators from the "intuition" of NICU
nurses. Advances in Nursing Science, 16(1), 42-51.
Crandall, B., Klein, G., & Hoffman, R. R. (2006). Working minds: A practitioner's
guide to cognitive task analysis. Cambridge, MA: MIT Press.
Ericsson, K. (2004). Deliberate practice and the acquisition and maintenance of
expert performance in medicine and related domains. Academic Medicine,
79(10), S70-S81.
Ericsson, K. & Kintsch, W. (1995). Long-term working memory. Psychological
Review, 102, 211-245.
Ericsson, K., Krampe, R. & Tesch-Romer, C. (1993). The role of deliberate practice
in the acquisition of expert performance. Psychological Review, 100(3), 363-
406.
Feldon, D. (2007). Implications of research on expertise for curriculum and
pedagogy. Educational Psychology Review, 19(2), 91-110.
62
Feldon, D. & Clark, R. (2006). Instructional implications of cognitive task analysis
as a method for improving the accuracy of experts’ self-report. In G.
Clarebout & J. Elen (Eds.), Avoiding simplicity, confronting complexity:
Advances in studying and designing (computer-based) powerful learning
environments (pp. 109-116). Rotterdam, The Netherlands: Sense Publishers.
Flin, R., Youngson, G., & Yule, S. (2007). How do surgeons make intraoperative
decisions? Qual Saf Health Care, 16 (3), 235-239.
doi:10.1136/qshc.2006.020743
Fowlkes J., Salas E., Baker D., Cannon-Bowers J., & Stout R. (2000). The utility of
event based knowledge solicitation. Human Factors, 42(1), 24-35.
Glaser, R. (1992). Expert knowledge and processes of thinking. In D. F. Halpern
(Ed.), Enhancing thinking skills in the sciences and mathematics (pp. 63–76).
Hillsdale, NJ: Lawrence Erlbaum Associates.
Grantcharov, T. & Reznick, R. (2008). Teaching procedural skills. British Medical
Journal. 336(7653), 1129-31. doi:10.1136/bmj.39517.686956.47
Grunwald T., Clark D., Fisher S., McLaughlin M., Narayanan S., & Piepol D.
(2004). Using cognitive task analysis to facilitate collaboration in
development of simulators to accelerate surgical training. In: Westwood JD
(Eds.) Medicine Meets Virtual Reality, (pp. 114-120). IOS Press.
Hillburn, B. (2007). ATC CTA: Cognitive Task Analysis of Future Air Traffic
Control Concepts. In Harris (Ed.): Engineering Psychology. and Cognitive.
Ergonomics, (pp. 497-503). doi:10.1007/978-3-540-73331-7
Hoffman, R., Shadbolt, N., Burton, A., & Klein,G. (1995). Eliciting Knowledge
from Experts: A Methodological Analysis. Organizational Behavior and
Human Decision Processes, 62(2). 129-158.
Hoffman, R. (1987). The problem of extracting the knowledge of experts from the
perspective of experimental psychology. AI Magazine, 8, 53-67.
Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal
effect. Educational Psychologist, 38(1), 23-31. doi:
10.1207/S15326985EP3801_4
63
Kirschner, P., Sweller, J., & Clark, R. (2006). Why minimal guidance during
instruction does not work: An analysis of the failure of constructivist,
discovery, problem-based experiential and inquiry-based teaching.
Educational Psychologist, 41(2), 75-86.
Kneebone, R. (2003). Simulation in surgical training: educational issues and
practical implications. Medical Education, 3, 267-277.
Kohn L., Corrigan J., & Donaldson M. (2000). To Err is Human: Building a Safer
Health System. Washington, D.C.: National Academy Press.
Lee, R. (2004). The impact of cognitive task analysis on performance: A meta
analysis of comparative studies. Unpublished doctoral dissertation,
University of Southern California.
Merrill, M. (1983). Component Display Theory. In C. M. Reigeluth (Ed.),
Instructional-design theories and models: An overview of their current status
(pp. 279-333). Hillsdale, NJ: Lawrence Erlbaum Associates.
Patton, M. (2002). Qualitative Research and Evaluation Methods. Thousand Oaks,
CA: Sage Publications.
Price, Michael. (2010). The antidote to medical errors. American Psychological
Association, 41(1), 1-2.
Reznick, R. & MacRae, H. (2006). Teaching surgical skills--changes in the wind.
New England Journal of Medicine, 355, 2664-2669.
Roth, E. (2008). Uncovering the requirements of cognitive work. Human Factors,
50(3), 475-478. doi: 10.1518/001872008X288556
Sachdeva, A., Bell, R., Britt, L., Tarpley, J., Blair, P., & Tarpley, M. (2007).
National efforts to reform residency education in surgery. Academic
Medicine, 82(12), 1200-1210.
Scott, D., Cendan, J., Pugh, C., Minter, R., Dunnington, G., & Kozar, R. (2008). The
changing face of surgical education: Simulation and the new paradigm.
Journal of Surgical Research, 147, 189-193.
Schraagen, J., Chipman, S., & Shalin, V. (2000). Cognitive task analysis. Mahwah,
NJ: Lawrence Erlbaum Associates.
64
Sullivan, M., Brown, C., Peyre, S., Salim, A., Martin, M., Towfigh, S., & Grunwald,
T. (2007). The use of cognitive task analysis to improve the learning of
percutaneous tracheostomy placement. The American Journal of Surgery, 93,
96-99.
Sullivan, M., Ortega, A., Wasserberg, N., Kaufman, H., Nyquist, J., & Clark, R.
(2008). Assessing the teaching of procedural skills: Can cognitive task
analysis add to our traditional teaching methods? The American Journal of
Surgery, 195, 20-23.
Sweller, J. (2006). How the human cognitive system deals with complexity. In J.
Elen & R. E. Clark (Eds.), Handling complexity in learning environments:
Theory and research (pp. 13–26). New York: Elsevier.
Tolano-Leveque, M. (2010). Using cognitive task analysis to determine the
percentage of critical information that experts omit when describing a
surgical procedure. Unpublished doctoral dissertation, Rossier School of
Education, University of Southern California.
Velhamos, G., Toutouzas, K., Sillin, L., Chan, L., Clark, R., Theodorou, D., &
Maupin, F. (2004). Cognitive Task Analysis for teaching technical skills in
an inanimate surgical skills laboratory. The American Journal of Surgery,
187, 114-119.
Vozenilek, J., Huff, S., Reznek, M., & Gordon, J. (2004). See one, do one, teach one:
Advanced Technology in Medical Education. American Emergency
Medicine, 11(11), 1149-1154.
Wheatley, T. & Wegner D. (2001). Automaticity in action. In N. J. Smelser & P. B
Baltes (Eds.), International encyclopedia of the social and behavioral ,
(pp.991-993). Oxford: Elsevier Science Limited.
Yates, K. (2007). Towards a taxonomy of cognitive task analysis methods: A search
for cognition and task analysis interactions. Unpublished doctoral
dissertation, Rossier School of Education, University of Southern California.
Abstract (if available)
Abstract
Evidence in research suggests experts omit 70% of procedural steps when describing a surgical procedure. Additionally, prior knowledge of a procedure may have an affect on the percentage of information omitted when describing a procedure. To examine this, the current study used CTA methods to conduct semi-structured interviews with six subject matter experts describing a central venous catheter (CVC) placement procedure. Quantitative methods were used to analyze data regarding the amount of information omitted when describing a CVC and the amount of information omitted as it relates to prior knowledge of the procedure. Study findings did not confirm experts omit 70% of procedural steps, but approximately 30% of action steps and 35% of decision steps were omitted when describing a CVC. Failure to confirm prior estimates of omissions in existing research may be due to study limitations including (a) “rolling-up” steps of CTA items to achieve better organization in the gold standard for data analysis (b) issues with crediting SMEs and (c) challenges to SME participation. Moreover, participants omitted a lower percentage of action and decisions steps when describing a CVC, yet possessed more prior knowledge of the procedure. This was compared to data of a similar study using an open cricothyrotomy (CRIC) procedure where participants had less prior knowledge of the CRIC, but more knowledge omissions. This outcome may stem from the type of prior knowledge examined in this study and heightened awareness resulting from institutional responses targeting the CVC. Future research in the field of CTA is discussed.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Using cognitive task analysis to determine the percentage of critical information that experts omit when describing a surgical procedure
PDF
Identifying the point of diminishing marginal utility for cognitive task analysis surgical subject matter expert interviews
PDF
The use of cognitive task analysis to determine surgical expert's awareness of critical decisions required for a surgical procedure
PDF
The use of cognitive task analysis to investigate how many experts must be interviewed to acquire the critical information needed to perform a central venous catheter placement
PDF
Using incremental cognitive task analysis to capture expert instruction in expository writing for secondary students
PDF
Towards a taxonomy of cognitive task analysis methods: a search for cognition and task analysis interactions
PDF
The use of cognitive task analysis for the postanesthesia patient care handoff in the intensive care unit
PDF
The effect of cognitive task analysis based instruction on surgical skills expertise and performance
PDF
Employing cognitive task analysis supported instruction to increase medical student and surgical resident performance and self-efficacy
PDF
Using individual cognitive task analysis to capture expert writing instruction in expository writing for secondary students
PDF
Using cognitive task analysis to capture expert reading instruction in informational text for students with mild to moderate learning disabilities
PDF
The use of cognitive task analysis to capture expert instruction in teaching mathematics
PDF
The use of cognitive task analysis to capture expert patient care handoff to the post anesthesia care unit
PDF
Using cognitive task analysis to capture expert instruction in division of fractions
PDF
Using cognitive task analysis to capture how expert principals conduct informal classroom walk-throughs and provide feedback to teachers
PDF
Using cognitive task analysis for capturing expert instruction of food safety training for novice employees
PDF
Using cognitive task analysis to capture palliative care physicians' expertise in in-patient shared decision making
PDF
Cognitive task analysis for instruction in single-injection ultrasound-guided regional anesthesia
PDF
The use of cognitive task analysis to capture exterptise for tracheal extubation training in anesthesiology
PDF
Using cognitive task analysis to capture how expert anesthesia providers conduct an intraoperative patient care handoff
Asset Metadata
Creator
Canillas, Eko Natividad
(author)
Core Title
The use of cognitive task analysis for identifying the critical information omitted when experts describe surgical procedures
School
Rossier School of Education
Degree
Doctor of Education
Degree Program
Education
Publication Date
04/27/2010
Defense Date
03/15/2010
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
cognitive task analysis methods,CTA methods,knowledge omissions,OAI-PMH Harvest,surgical subject matter experts
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Clark, Richard E. (
committee chair
), Sullivan, Maura E. (
committee member
), Yates, Kenneth A. (
committee member
)
Creator Email
ncani13@gmail.com
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-m2953
Unique identifier
UC1273689
Identifier
etd-Canillas-3647 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-309759 (legacy record id),usctheses-m2953 (legacy record id)
Legacy Identifier
etd-Canillas-3647.pdf
Dmrecord
309759
Document Type
Dissertation
Rights
Canillas, Eko Natividad
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Repository Name
Libraries, University of Southern California
Repository Location
Los Angeles, California
Repository Email
cisadmin@lib.usc.edu
Tags
cognitive task analysis methods
CTA methods
knowledge omissions
surgical subject matter experts