Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Using cognitive task analysis to capture how expert principals conduct informal classroom walk-throughs and provide feedback to teachers
(USC Thesis Other)
Using cognitive task analysis to capture how expert principals conduct informal classroom walk-throughs and provide feedback to teachers
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
Running Head: INFORMAL CLASSROOM WALK-THROUGHS 1
USING COGNITIVE TASK ANALYSIS TO CAPTURE HOW EXPERT PRINCIPALS
CONDUCT INFORMAL CLASSROOM WALK-THROUGHS AND PROVIDE FEEDBACK
TO TEACHERS
By
Chad S. Hammitt
______________________________________________________________________________
A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION
December 2014
Copyright 2014 Chad S. Hammitt
INFORMAL CLASSROOM WALK-THROUGHS 2
Dedication
This dissertation is dedicated to my best friend and biggest supporter, my wife Wendy;
and to my two biggest cheerleaders, Victoria and Brenden. Without your support and insistence
that I not only complete this dissertation but also live to tell about it, I never would have
completed this journey. I am extremely grateful for your sacrifice and understanding, and now I
look forward to being a regular part of the family again.
INFORMAL CLASSROOM WALK-THROUGHS 3
Acknowledgements
First and most I must acknowledgement Dr. Kenneth Yates, who through this process not
only served as my dissertation chairperson but also my motivator, logical reasoning and thinking
coach, counselor, mentor, and most importantly friend. Dr. Yates I thank you from the bottom
of my heart for being available around my schedule, even if that meant 6:00 am phone
conversations, and no matter what time of the day you pushed my thinking to new levels and
always provided encouragement and support. You are model of what an educator should be;
someone who goes above and beyond not because it is his job but because it is his passion.
Without your guidance and assistance this dissertation journey would have been nearly
impossible. I would also like to thank Dr. Artineh Samkian and Dr. Joan Lucid for their
willingness to sacrifice their time, energy, and invaluable feedback as members of my
dissertation committee.
I am also extremely grateful to the four principals who participated in this study. I am
very appreciative for your sacrifice of time and for persisting with me through the CTA
interview process for this study. You are incredible instructional leaders and your leadership and
commitment to student achievement is making a difference in the lives of the students, teachers,
and families served by you in your schools.
I am also grateful to my fellow Thursday night cohort of 2012 classmates. We shared
many great hours together and I will miss the interaction, support, and mental stimulation of our
weekly meetings. However, I will not miss the homework!
Finally, I am forever grateful to my family and friends who supported me and understood
my absence throughout this doctoral program. Not once did you ask me to delay my studies or
put things off, and I am very appreciative for your understanding, support, and love.
INFORMAL CLASSROOM WALK-THROUGHS 4
Table of Contents
List of Tables 7
List of Figures 8
List of Abbreviations 9
Abstract 10
Chapter One: Overview of the Study 11
Statement of the Problem 11
Purpose the Study 15
Methodology of the Study 15
Definition of Terms 16
Organization of the Study 18
Chapter Two: Review of the Literature 19
The Changing Educational Landscape 19
No Child Left Behind Act 19
High-Stakes Accountability 19
Unintended Consequences of NCLB 20
Common Core State Standards 21
Background 21
Implications 22
Implementation 22
Increased Accountability for Student Achievement 23
Increased Pressure on Teachers for Student Achievement Growth 23
Accountability of Principals for Student Achievement 25
Improving Classroom Instruction 27
Importance of Teacher’s Instructional Practices for Improving Student
Achievement 27
Principal’s Role in Improving Classroom Instruction and Instructional
Leadership 30
Preparing Principals as Instructional Leaders 31
Formal and Informal Observations 32
Effective and Ineffective Formal Observations 32
Effective and Ineffective Informal Observations 35
Providing Teachers with Formative Feedback 37
Conducting Informal Classroom Walk-throughs and Providing
Feedback to Teachers 38
Professional Development for Principals on Informal Classroom
Walk-throughs 40
Summary 41
Using Subject Matter Experts (SMEs) to Train Non-experts 41
Knowledge Types 42
Declarative Knowledge 43
Procedural Knowledge 44
Automaticity 45
Expertise 48
Characteristics of Experts 48
INFORMAL CLASSROOM WALK-THROUGHS 5
Building Expertise 50
Consequences of Expertise 52
Expert Omissions 53
Cognitive Task Analysis (CTA) 54
Definition of CTA 54
CTA History 54
Cognitive Task Analysis Methodology 55
Effectiveness of CTA for Capturing Expert Knowledge 56
Effectiveness of CTA-based Training 57
Meta-analysis of Studies 58
Summary 60
Chapter Three: Methods 61
Participants 61
Data Collection for Question 1 63
Phase 1: Collect Primary Knowledge 64
Phase 2: Identify Knowledge Types 64
Phase 3: Apply Knowledge Elicitation Techniques 64
Instrumentation 64
Interviews 65
Phase 4: Data Analysis 66
Coding 66
Inter-rater Reliability 66
Subject Matter Expert Protocol and Verification 66
Phase 5: Formatting the Results 67
Gold Standard Protocol (GSP) 67
Summary 68
Data Collection for Question 2 68
Spreadsheet Analysis 68
Chapter 4: Results 69
Overview of Results 69
Research Questions 69
Question 1 69
Inter-rater Reliability 69
Flowchart Analysis 69
Gold Standard Protocol 70
Recalled Action and Decision Steps 72
Action and Decision Steps Contributed by Each SME 73
Action and Decision Steps Captured in Follow-up Interviews 75
Alignment of SMEs in Describing the Same Action and Decision
Steps 75
Question 2 76
Total Knowledge Omissions 76
Analysis of Action and Decision Step Omissions 78
INFORMAL CLASSROOM WALK-THROUGHS 6
Chapter 5: Discussion 80
Overview of Study 80
Process of Conducting Cognitive Task Analysis 81
Selection of Experts 81
Collection of Data 84
Discussion of Findings 86
Question 1 86
Action Steps Versus Decision Steps 86
Action and Decision Steps Captured During the Review of the
Initial Individual Protocols and Preliminary Gold Standard
Protocol 88
Use of Multiple Experts 90
Question 2 91
Expert Knowledge Omissions 91
Limitations 92
Confirmation Bias 92
Internal Validity 93
External Validity 93
Implications 94
Future Research 94
Conclusion 96
References 98
Appendix A: Cognitive Task Analysis Interview Protocol 114
Appendix B: Inter-rater Reliability Code Sheet 117
Appendix C: Job Aid for Developing a Gold Standard Protocol 118
Appendix D: SME A Initial Individual Protocol Flowchart 120
Appendix E: Gold Standard Protocol 130
Appendix F: Incremental Coding Spreadsheets 147
INFORMAL CLASSROOM WALK-THROUGHS 7
List of Tables
Table 1: Potential SMEs as Recommended by a Consultant for the Current Study 63
Table 2: Cumulative Action and Decision Steps Captured for Each SME in the Initial
Individual Protocols 73
Table 3: Additional Expert Knowledge Captured, in Action and Decision Steps, During
Follow-up Interviews 75
Table 4: Number and Percentage of Action and Decision Steps that are Highly Aligned,
Partially Aligned, and Slightly Aligned 76
Table 5: Total Action and Decision Steps, or Expert Knowledge, Omissions by SME
when Compared to the Gold Standard Protocol 77
INFORMAL CLASSROOM WALK-THROUGHS 8
List of Figures
Figure 1: 3i + 3r CTA Method 68
Figure 2: Example of Aggregating Action and Decision Steps for the Preliminary Gold
Standard Protocol (PGSP) 71
Figure 3: Number of Action Steps, Decision Steps, and Action and Decision Steps
for SME A, SME B, and SME C Captured through CTA 74
Figure 4: Total SME Knowledge Omissions when Compared to the Gold Standard
Protocol 78
INFORMAL CLASSROOM WALK-THROUGHS 9
List of Abbreviations
CCSS: Common Core State Standards
CDM: Critical Decision Method
CPP: Concepts, Processes, and Principles
CTA: Cognitive Task Analysis
GSP: Gold Standard Protocol
IRB: Institutional Review Board
K – 12: Kindergarten through Twelfth Grade
MDL: Model of Domain Learning
NAEP: National Assessment of Educational Progress
NCLB: No Child Left Behind Act
NICU: Neo-Natal Intensive Care Unit
OECD: Organisation for Economic Co-operation and Development
PARI: Precursors, Action, Result, and Interpretation
PGSP: Preliminary Gold Standard Protocol
SAM: School Administration Manager
SBA: Smarter Balanced Assessments
SES: Socio-economic Status
SME: Subject Matter Expert
3i + 3r: Three Independent Interviews + Three Reviews
INFORMAL CLASSROOM WALK-THROUGHS 10
Abstract
Informal classroom walk-throughs conducted by school principals with feedback provided to
teachers has been demonstrated to improve learning achievement in kindergarten through twelfth
grade (K-12) education. Principals are often trained by experts to conduct these walk-throughs.
Unfortunately, research shows that experts may omit up to 70% of the critical information
needed by trainees to replicate their expertise. The purpose of this study was to capture the
knowledge and skills expert K-12 principals use when they conduct informal classroom walk-
throughs and provide feedback to teachers. Cognitive Task Analysis (CTA) semi-structured
interviews were conducted with three principals who were qualified as experts using both
qualitative and quantitative measures. Action and decision steps, as well as standards,
equipment, and conceptual knowledge from individual subject matter experts (SMEs) were
captured and aggregated into a gold standard protocol which was reviewed by a fourth expert.
The study also sought to identify and quantify the number and percentage of expert knowledge
and skills omissions when the principals described how they conducted classroom walk-throughs
and provided feedback to teachers. Findings indicate that expert principals omitted an average of
54.76% of the action and decision steps when compared to the gold standard protocol. This
study extends the potential negative effects of relying on experts for instruction and curriculum
development. The expert knowledge and skills captured by CTA methods may be used to train
pre-service and in-service principals in performing the complex instructional leadership task of
informal walk-throughs and providing feedback to teachers, which may ultimately improve
teachers’ classroom instruction and student achievement.
INFORMAL CLASSROOM WALK-THROUGHS 11
CHAPTER ONE: OVERVIEW OF THE STUDY
Statement of the Problem
The adoption and implementation of No Child Left Behind (NCLB) in addition to the
Common Core State Standards (CCSS) and the aligned Smarter Balanced Assessments (SBA)
have brought a new level of accountability for student achievement to schools and school
districts across the country (California Department of Education [CDE], 2010; Davidson,
Reback, Rockoff, & Schwartz, 2013; Dee & Jacob, 2011; Elmore, 2002; Fuhrman & Elmore,
2004; Kim & Sunderman, 2005; McDonnell & Weatherford, 2013; Phillips & Wong, 2010).
NCLB was signed into law by President George Bush on January 1, 2002 and launched a new
era of Federal educational accountability through high stakes testing. The results of the testing
were made public, all students were required to score proficient on the testing by 2014, and
sanctions were placed upon schools and districts that did not meet the predetermined
achievement thresholds (Davidson et al., 2013; Dee & Jacob, 2011; Kim & Sunderman, 2005).
Since 2007, NCLB has been due for reauthorization, but has not received sufficient bi-
partisan support from the Congress. In an attempt to rectify some of the shortcomings of NCLB,
the Council of Chief State School Officers and the National Governors’ Association Center for
Best Practices with representatives from 48 states, two territories, and the District of Colombia;
voluntarily created the CCSS to provide national academic standards for all students in grades
kindergarten through 12 to ensure that all students regardless of where they live within the
United States would be held to the same academic standards with the goal of being college and
career ready (CDE, 2010; McDonnell & Weatherford, 2013; Phillips & Wong, 2010). To assess
and monitor student achievement toward mastering the CCSS, the Smarter Balanced Assessment
INFORMAL CLASSROOM WALK-THROUGHS 12
(SBA) Consortium has designed a national assessment which schools and districts will begin to
take in the 2014-2015 school year (CDE, 2010).
As a result, school principals and teachers are under increasing pressure to foster and
maintain high quality classroom instruction. The pressure to meet accountability measures rests
primarily upon each classroom teacher who has the single greatest impact upon the individual
student, the quality of the instruction provided, and the classroom environment; even greater than
the school that the child attends (Nye, Konstantopoulos, & Hedges, 2004; Rivkin, Hanushek, &
Kain, 2005). In a study conducted by Murnane and Papay (2010), a majority of teachers
reported that they were in support of the goals of NCLB but also reported much higher levels of
pressure put upon them, especially when working in schools with greater numbers of
disadvantaged students. A majority of teachers in the Murnane and Papay study reported this
higher level of pressure due to NCLB had led to: (a) score inflation due to a focus on test
preparation; (b) increased concentration on students that were just short of the proficiency level
on tests at the exclusion of other students; (c) narrowing of the curriculum to tested areas; and (d)
incentives and sanctions that did not adequately represent a school’s achievement. As of the
NCLB testing in 2014, if just one student failed to reach proficient status the entire school was
determined to be failing and the pressure to ensure that all students score proficient is most often
placed directly onto the school’s principal. NCLB requires that one of the sanctions placed upon
a school for not reaching the achievement targets for two consecutive years is removal and
replacement of the principal (Davidson et al., 2013).
The ultimate responsibility at the school level to achieve consistent high quality
instruction lies with the principal who must serve as the instructional leader of the
school. Instructional leadership has been identified over the past two decades as the most
INFORMAL CLASSROOM WALK-THROUGHS 13
important skill for site principals to have (Marzano, Waters, & McNulty, 2005). Marzano et al.
conducted a meta-analysis to look at the last 35 years of research on principal leadership and
determined that increased skillful principal leadership, especially in the area of instruction, is
positively correlated to a school’s improvement. A school principal sets the climate for a school
and determines the extent to which a school is focused upon student learning through effective
classroom instruction (DuFour & Mattos, 2013).
The school principal must use multiple methods to evaluate instructional quality. The
most popular method of evaluation, which is often a legal or contractual requirement, is the
formal process of clinical supervision. Clinical supervision is a summative process meant to
evaluate or judge a teacher’s performance. In contrast, Range, Young, and Hvidston (2013)
reported informal formative instructional evaluation methods focusing upon constructive
principal feedback such as informal observations, also known as walk-throughs, and providing
feedback to teachers, enable teachers to plan and reflect on instruction.
When conducted correctly, informal observations can increase the quality of teachers'
instruction; when performed incompletely and/or incorrectly, informal observations can result in
no change or even negative consequences to teachers’ instruction, student achievement, and a
school’s climate (Bloom, 2007; Grissom, Loeb, & Master, 2013). Principals conduct informal
observations and walk-throughs for a variety of purposes including creating a presence,
monitoring school operational issues, and complying with district-level expectations or
mandates. Informal observations, specifically walk-throughs, help facilitate formative
assessment which has been characterized as events that encourage assessment “for learning”
rather than assessment “of learning” (Stiggins, 2002). Providing feedback is fundamental to
formative assessment. Informal walk-throughs without follow-up feedback between the
INFORMAL CLASSROOM WALK-THROUGHS 14
principal and classroom teacher to provide the opportunity for improving instruction have little
to no effect (Ing, 2009).
The primary source for training principals in informal walk-throughs has traditionally
been through expert led formal professional development, in the form of consultants. This
formal professional development is most often conducted in a seminar or lecture format whereby
the presenter, or expert, transmits the information through a presentation to the learners. This
method of professional development has not been shown to be as effective as on-the-job training
which promotes learning in one’s own work environment using a mentor or expert as a guide
using real work situations as learning experiences (Mesler Parise & Spillane, 2010).
Experts, such as those who train principals, are at a disadvantage as trainers in that their
knowledge and skills are often automated and unconscious; and, thus, not easily accessible to
accurately describe how to perform complex tasks, such as conducting informal walk-throughs.
Studies show that when training, experts may omit up to 70% of the key actions and decisions
they make when describing how to perform a complex task (Clark, Feldon, van Merriënboer,
Yates, & Early, 2008; Sullivan, Yates, Baker, & Clark, 2010). Expert knowledge becomes
automated to free-up mental resources to handle novel problems. This expert knowledge is of
two types: declarative knowledge, or “why or that” information, which is characterized by
concepts, principles, and processes; and procedural or production knowledge which is “how and
when” to apply knowledge. Experts who provide training can omit one or both of these types of
knowledge, leaving the learner with incomplete knowledge or “holes” in their understanding
(Clark & Estes, 1996).
Cognitive Task Analysis (CTA) is a method that has been demonstrated to be useful to
capture the knowledge and skills experts use to perform complex tasks and solve difficult
INFORMAL CLASSROOM WALK-THROUGHS 15
problems (Clark et al., 2008). CTA is a qualitative research approach whereby three to five
subject matter experts are interviewed, using a semi-structured interview protocol, in order to
capture the conceptual and procedural knowledge required to perform a complex task.
Conducting informal classroom walk-throughs and providing feedback to teachers are such
complex tasks. The results of a CTA study could potentially be used for professional
development of existing and aspiring principals.
Purpose of the Study
The purpose of this study is to conduct a CTA with principals who have been identified
as experts, to capture the knowledge and skills they use when conducting informal walk-throughs
and providing feedback to teachers to improve teachers’ classroom instruction.
The questions that guide the study are:
• What are the action and decision steps that expert principals recall when they describe
how they conduct informal classroom walkthroughs and provide feedback to
teachers?
• What percentage of actions and/or decision steps, when compared to a gold standard,
do expert principals omit when they describe how they conduct informal classroom
walk-throughs and provide feedback to teachers?
Methodology of the Study
Cognitive Task Analysis was used in this study to determine the knowledge and skills of
kindergarten through twelfth grade (K-12) principals from school districts in Southern California
identified as experts (subject matter experts or SMEs) in conducting informal classroom walk-
throughs and providing feedback to teachers. Four SMEs were selected, three to participate in
INFORMAL CLASSROOM WALK-THROUGHS 16
interviews and the fourth to verify the data collected from the three SMEs on informal classroom
walk-throughs. The CTA followed a five step process
1) a preliminary phase to build general familiarity frequently called “bootstrapping;”
2) the identification of declarative and procedural knowledge and any hierarchal
relationships in the application of these knowledge types;
3) knowledge elicitation through semi-structured interviews;
4) data analysis involving coding, inter-rater reliability, and individual SME protocol
verification; and
5) the development of a gold standard protocol that was used to analyze and determine
expert omissions and ultimately for use in the training of novice principals.
Definition of Terms
The following are definitions of terms related to cognitive task analysis as suggested by
Zepeda-McZeal (2014).
Adaptive expertise: When experts can rapidly retrieve and accurately apply appropriate
knowledge and skills to solve problems in their fields or expertise; to possess cognitive
flexibility in evaluating and solving problems (Gott, Hall, Pokorny, Dibble, & Glaser, 1993;
Hatano & Inagaki, 2000).
Automaticity: An unconscious fluidity of task performance following sustained and
repeated execution; results in an automated mode of functioning (Anderson, 1996a; Ericsson,
2004).
Automated knowledge: Knowledge about how to do something: operates outside of
conscious awareness due to repetition of task (Wheatley & Wegner, 2001).
INFORMAL CLASSROOM WALK-THROUGHS 17
Cognitive load: Simultaneous demands placed on working memory during information
processing that can present challenges to learners (Sweller, 1988).
Cognitive tasks: Tasks that require mental effort and engagement to perform (Clark &
Estes, 1996).
Cognitive task analysis: Knowledge elicitation techniques for extracting implicit and
explicit knowledge from multiple experts for use in instruction and instructional design (Clark et
al., 2008; Schraagen, Chipman, & Shalin, 2000).
Conditional knowledge: Knowledge about why and when to do something; a type of
procedural knowledge to facilitate the strategic application of declarative and procedural
knowledge to problem solve (Paris, Lipson, & Wixson, 1983).
Declarative knowledge: Knowledge about why or what something is; information that is
accessible in long-term memory and consciously observable in working memory (Anderson,
1996a; Clark & Elen, 2006).
Expertise: The point at which an expert acquires knowledge and skills essential for
consistently superior performance and complex problem solving in a domain; typically develops
after a minimum of 10 years of deliberate practice or repeated engagement in domain-specific
tasks (Ericsson, 2004).
Procedural knowledge: Knowledge about how and when something occurs; acquired
through instruction or generated through repeated practice (Anderson, 1982; Clark & Estes,
1996).
Subject matter expert: An individual with extensive experience in a domain who can
perform tasks rapidly and successfully; demonstrates consistent superior performance or ability
to solve complex problems (Clark et al., 2008).
INFORMAL CLASSROOM WALK-THROUGHS 18
Organization of the Study
Chapter Two of this study reviews the literature in two main sections, the first section of
the literature review examines the relevant literature related to conducting informal classroom
walk-throughs and the second section focuses on the relevant literature related to Cognitive Task
Analysis as a knowledge elicitation technique for subject matter expertise. Chapter Three
addresses the methods of this study and how the approach to the research answers the research
questions. Chapter Four reviews the result of the study and compares the findings to each of the
research questions. Chapter Five serves as a discussion of findings, the implication of the
findings upon informal classroom walk-throughs and CTA, limitations of the study, and
implications for future research.
INFORMAL CLASSROOM WALK-THROUGHS 19
CHAPTER TWO - LITERATURE REVIEW
The Changing Educational Landscape
Over the course of the past 20 years, education has experienced significant changes
related to nearly every aspect of teaching, learning, and accountability with the adoption and
implementation of rigorous standards, testing, and accountability measures through No Child
Left Behind and more recently the Common Core State Standards. The result has been increased
pressure upon teachers and principals to improve student performance. The following sections
examine these issues in greater depth by discussing the No Child Left Behind Act and Common
Core State Standards, and the increased pressure they have placed upon teachers and principals.
No Child Left Behind Act
The No Child Left Behind (NCLB) Act was signed into law in January 2002 by President
George Bush and had far reaching effects on education throughout the United States. The
theoretical underpinnings of NCLB were that schools should be forced to improve student
achievement and be held accountable if students were not achieving. Student achievement under
NCLB was measured through large-scale assessment systems with consequences for student
achievement designed to motivate school leaders, teachers, and students.
High stakes accountability. Accountability under NCLB demonstrated a “contractual”
relationship between a “director” who rewards, punishes, or replaces and a “provider” who
provides a good or service on many levels: federal to state, state to county, county to district,
district to principal, principal to teacher, and teacher to students (Hentschke & Wohlstetter,
2004). All students and numerically significant subgroups as described by Kim and Sunderman
(2005), were expected to score at the proficient level in English language arts and math by 2014
on state assessments, even though as Elmore (2002) and Linn (2005) described a 10% increase
INFORMAL CLASSROOM WALK-THROUGHS 20
each year in proficiency from the implementation of NCLB up to 2014 was ungrounded in any
research base or theory of school improvement. The results of NCLB testing were made public
as were the sanctions for schools and districts that did not achieve “proficient” levels (Dee &
Jacob, 2011). As Elmore (2002) explained the sanctions were developed by policy makers that
were not involved in education or local communities and included forcing schools to use Federal
funds to provide tutoring, offering parents transfers to non-failing schools, removing the
principal, and restructuring of closing the school altogether (Davidson et al., 2013). The rewards
and sanctions under NCLB have proven to be ineffective, both in practice and the research
literature, in turning around or improving professional practice and student achievement in
failing schools in urban areas, with higher total student enrollments, which contain large
numbers of students who are low Socio-economic Status (SES), minority, and designated as Title
I eligible (Davidson et al., 2013; DuFour & Mattos, 2013). Furhman (2004) states that NCLB
was intended to result in more effective instruction with minimal unintended consequences,
however as seen in the next section, NCLB had several unintended consequences.
Unintended consequences of NCLB. Under NCLB each state was given the authority
to establish teaching standards, assessments, and scores that would determine proficiency levels;
leading to a large degree of variability between “proficient” among states. As a result, failing
schools and students in one state would likely be proficient in another state and there was no way
to accurately and fairly compare results across states due to inconsistent standards, assessments,
levels of rigor, and the inability to correlate NCLB testing with nationally normed tests such as
the National Assessment of Educational Progress (NAEP) (Cawelti, 2006; Davidson et al., 2013;
Linn, 2005; McDonnell & Weatherford, 2013). Teachers supported the aims of NCLB, however
teachers also made changes to their instructional practices including: shifting instructional time
INFORMAL CLASSROOM WALK-THROUGHS 21
to the tested areas of language arts and math at the exclusion of other subject areas, increased
time teaching test preparation, and paying more attention to student just below the “proficient”
level or “bubble kids” (Cawalti, 2006; Murnane & Papay, 2010).
Another unintended consequence of NCLB was the incentive and sanction system did not
provide rewards for progress or focus on growth that did not reach the predetermined levels of
proficiency. Schools with high concentrations of disadvantaged students, students with special
needs, and English language learners were more likely to be labeled “failing” despite
demonstrating consistent and regular student achievement gains (Linn, 2005; Murnane & Papay,
2010). Teacher morale and school climate were also impacted by demoralizing patterns of
repeated failure, perceived inability to reach NCLB goals, and removal of school principals
(Cawalti, 2006; Murnane & Papay, 2010). Educational gains as measured by student
achievement for all student groups have fallen short of expectations, and in most cases, remained
flat despite the money and energy spent to implement the “top down” accountability measures of
NCLB (Adams, 2010; Council for Exceptional Children, 2005; Harkin, 2013). The unintended
consequences of NCLB, in addition to the inability of many schools and districts to reach the
student achievement goals has led to the development and adoption of a new set of learning
standards, the Common Core State Standards.
Common Core State Standards
In 2009, the Council of State School Officials and the National Governors Association
Center for Best Practices coordinated the creation of the Common Core State Standards (CCSS)
(CDE, 2010).
Background. The concept of developing “National Standards” dates back more than two
decades, not long after the publication of A Nation at Risk. The CCSS represent a previously
INFORMAL CLASSROOM WALK-THROUGHS 22
unknown shift from the varied and inconsistent standards across each state in the country, and
the CCSS were developed to be the next generation of learning standards to provide for a
common set of expectations for the entire United States based upon research, international
models, college and career readiness skills, and 21
st
century skills (California Department of
Education [CDE], 2013; McDonnell & Weatherford, 2013; Phillips & Wong, 2013; Porter,
McMaken, Hwang, & Yang, 2011).
Implications. O’Day and Smith (1993) explained the development of a “national
curriculum,” the CCSS, attempted to address the achievement gap and focused on impacting
classroom instruction which has a profound effect upon the achievement of minority,
disadvantaged, and low-SES students. The movement to a “national curriculum” is anticipated
to lead to: (a) common understanding of what all students should know and be able to do; (b)
structures and supports for schools to deliver state adopted curriculum through sound teaching by
teachers who are appropriately educated and licensed; and (c) governance structures which
provide for sufficient resources, flexibility, and accountability to ensure all students are meeting
the goals (O’Day & Smith, 1993).
Implementation. The focus of the CCSS, which have been adopted in 48 states, is on
what students should know and do, but not on how they are taught or assessed, so the next step in
implementing the CCSS will be to develop aligned instructional tools and assessments (Phillips
& Wong, 2013; Porter et al., 2011). The CCSS are “fewer, higher, and clearer,” however the
number of standards and amount of content that needs to be taught will require alignment of the
CCSS in all content areas. Adding to this challenge, textbooks and instructional materials
created on a state-by-state basis need to be aligned to the CCSS through rewriting and
redesigning by the textbook companies because the alignment between existing standards and the
INFORMAL CLASSROOM WALK-THROUGHS 23
CCSS is only between 25% and 52% for the two primarily assessed areas of English language
arts and Math (Phillips & Wong, 2013; Porter et al., 2011). Additionally, state and local
assessment systems will not just have to be readjusted, but entirely redesigned as current state
assessment systems based upon existing state standards show a 19% alignment with the CCSS
(Phillips & Wong, 2013; Porter et al., 2011). The adoption and implementation of the new and
more challenging Common Core State Standards will not raise student achievement alone;
ultimately the pressure is on classroom teachers and school principals to change the content,
delivery, and assessment of classroom instruction.
Increased Accountability for Student Achievement
Changes to educational policy and standards brought about by NCLB and the CCSS
affect classroom teachers and school principals as they are the individuals who must ultimately
implement these policies and standards, and who are held accountable for demonstrating results
in the form of student achievement.
Increased pressure on teachers for student achievement growth. The theory behind
NCLB and other accountability measures is high-stakes accountability and testing influence
classroom activities. The greatest impact upon teachers across the research was the increased
pressure they were under to produce student achievement gains on high-stakes testing (Diamond
2007; Hannaway & Hamilton, 2008; Plank & Falk Condliffe, 2013; Reback, Rockoff, &
Schwartz, 2011). This increased pressure also manifested itself in changes to instructional
content wherein teachers mediated the changes to instructional content through focusing on
English language arts and Math at the exclusion of other non-tested subject areas (Diamond,
2007; Hannaway & Hamilton, 2008; Murnane & Papay, 2010). These changes to instructional
content were even greater for minority and disadvantaged students who received significantly
INFORMAL CLASSROOM WALK-THROUGHS 24
more didactic instruction, as opposed to interactive instruction, than their predominately white
middle class peers (Diamond, 2007). In schools with a greater number of students below
proficiency benchmarks, teachers were 10% less likely to teach a science lesson and 6% less
likely to teach a Social Studies lesson during a week of instruction (Reback et al., 2011). In
addition, lesson pacing was changed, some instructional content was covered hastily to ensure
students were exposed to tested curricular areas prior to testing, and students were instructed
how to approach types of problems on a test without teaching the underlying subject area
content, most commonly referred to as test preparation or “teaching to the test” (Diamond, 2007;
Hannaway & Hamilton, 2008; Plank & Falk Condliffe, 2013).
Pressure from high stakes accountability also changed the way teachers delivered content
and their instructional practices. During instructional periods prior to high-stakes testing,
accountability pressures impacted the instructional practices of emotional support, classroom
organization, and instructional support; and once released from the pressure of high stakes
testing, tested classrooms returned to the same levels of organization and instructional and
emotional support as non-tested classrooms (Plank & Falk Condliffe, 2013). The amount of time
and attention students received on instructional content was varied due to high-stakes
accountability with students who were just below testing proficiency goals, or “bubble-kids,”
receiving greater instructional time, specialized instruction, and resources at the expense of
students significantly above or below the proficiency goals (Hannaway & Hamilton, 2008;
Murnane & Papay, 2010). Hannaway and Hamilton (2008) found that in extreme cases teachers
would even allow cheating, or at a minimum, turn a blind-eye to cheating when they knew it was
taking place.
INFORMAL CLASSROOM WALK-THROUGHS 25
Pressure from high-stakes testing was also found to reduce teacher’s feelings of having
job security, to increase the percentage of teacher who reported that they did not plan on working
in education until they retired, and to result in longer work hours for teachers. These results
were even more significant for non-tenured teachers who lacked the same job protections as
tenured teachers (Reback et al., 2011). Principals strategically placed teachers in tested grade
levels, classrooms, and subjects adding to these pressures. The pressure of high-stakes
accountability is not limited to teachers, but also impacts school principals as will be discussed in
the following section.
Accountability of principals for student achievement. High-stakes accountability
measures like NCLB call for the implementation and use of research-based strategies, like more
intensive formal teacher evaluation, even though the research literature demonstrates that more
intensive formal evaluations do not improve instruction or student achievement (DuFour &
Mattos, 2013). Principals have been under increased pressure to improve instruction and student
achievement through teacher evaluation. Effective teacher evaluation involves teachers knowing
what to do and being motivated to actually do it, and if they are not principals are expected to
teach them how and motivate them to change their practice. For effective teacher evaluation to
occur, principals must have the time to conduct teachers’ evaluations and the expertise to help
improve teachers’ practices (DuFour & Mattos, 2013). The principal’s role in improving
teachers’ practice and motivating the teachers to implement these improvements are commonly
referred to as “instructional leadership,” a concept popularized in the 1980’s when the “Effective
Schools” movement began based on the reasoning that principals of “instructionally effective”
schools are leaders (Hallinger, 2007).
INFORMAL CLASSROOM WALK-THROUGHS 26
The role of instructional leader places principals under increased pressure to demonstrate
that a school is both legitimate and has organizational integrity (Spillane & Kenney, 2012). This
need for legitimacy and integrity creates conflict between the teachers and principal as the
teachers are accustomed to autonomy and personal choice, and principals are accountable for
implementing policies with fidelity (Spillane & Kenney, 2012). Spillane and Kenney explain the
principal is caught between being the primary unit for implementing federal, state, and district
mandates and policies, and supporting classroom teachers as they work to improve student
achievement; while convincing them to implement policies and mandates that may be well
intentioned but lacking detail, untested, and in the worst case, not educationally sound. One of
the primary ways principals increase legitimacy and integrity is through improving teaching and
learning with a more coherent instructional program. The challenge for many school principals
is that instructional leadership is not an area that is taught in many pre-service administrative
programs or that has a large body of supporting research (Spillane & Kenney, 2012). Moreover,
despite the increased pressure for instructional leadership and improved instruction, the
classroom remains the teacher’s domain where the teacher works primarily in isolation and the
presence of the principal may not be welcomed
Under high-stakes accountability, another area of pressure for principals is the possibility
of being removed from their position if their school, especially if a school is low-performing
with a high number for students from disadvantaged populations, fails to meet growth
benchmarks based upon student assessment scores. Branch, Hanushek, and Rivkin (2012) stated
that low-performing schools have the highest percentage of first year principals with no previous
administrative experience, whereas high-performing schools without students from
disadvantaged backgrounds have 50% more principals with at least six years of experience. The
INFORMAL CLASSROOM WALK-THROUGHS 27
researchers found that a principal in the top 16% of quality measures will lead to a .05 standard
deviation increase in student achievement above the average growth for students with similar
demographics. In short, Branch et al. (2012) found that school principals have an impact on
student achievement outcomes. The pressure to increase student achievement has had
unintended consequences on both teachers and principals, primarily increased pressure and
stress, leading educators to question what is the most effective way to improve classroom
instruction and student achievement.
Improving Classroom Instruction
The interaction between teachers and students in the classroom around instructional
activities is the primary mechanism for learning and increasing student achievement. This
section examines the importance of the classroom teacher’s instructional practices on student
achievement, the principal’s role in improving classroom instruction, and instructional
leadership.
Importance of Teacher’s Instructional Practices for Improving Student Achievement
Educational policy makers have implemented a variety of policy initiatives with the goal
of increasing student achievement such as: increased resources, increased instructional time,
smaller class sizes, prescriptive instructional curriculums, and incentives and sanctions among
many others (Cohen, Raudenbush, & Loewenberg Ball, 2003). The difficulty researchers have
had in narrowing down the effects of the teacher on student learning and achievement is isolating
the teacher’s input as a variable and gathering a large enough sample size to mitigate other
external student factors such as SES, language dominance, minority status, and assuring the non-
random assignment of students to teachers’ classrooms. This analysis is known as “value-added”
INFORMAL CLASSROOM WALK-THROUGHS 28
analysis, or the value a teacher’s input adds to a student’s achievement (Wright, Horn, &
Sanders, 1997).
A number of studies have found the greatest influence on student achievement remains
effective classroom instruction and the effect of a teacher is significantly higher than all other
factors measured (Cohen et al., 2003; Kane, McCaffrey, Miller, & Staiger, 2013; OECD, 2005;
Rivkin et al., 2005; Rowan, Correnti, & Miller, 2002; Wright et al., 1997). Rowan et al. (2002)
used a statistical model that overcame the weaknesses of most “value-added” models and found
the effect size of a teacher upon a student’s individual achievement differs between 4-18% of the
variance per year, and when students are provided instruction in two subject areas by the same
teacher they will demonstrate higher achievement in one subject over the other. The amount of
increased achievement a student experiences with an effective teacher is offset if the student has
an ineffective teacher the following year or even worse two years of an ineffective teacher in a
row; however the data indicate that students only have a 3% chance of having two years of an
effective or ineffective teacher in a row, and a 1% chance or having three consecutive years of
either type of teacher (Rowan et al., 2002). Rowan et al. conducted interviews with the teachers
in their study to further investigate the difference in achievement across classrooms and
concluded that teachers use a variety of research-based effective instructional content and
teaching strategies but not all. This was due mainly to teacher autonomy and the lack of
supervision of teachers and classrooms to reinforce, support, and monitor the use of research-
based effective instructional practices. Autonomy and lack of supervision was compounded by
the effect of randomization, naturally occurring talent, and voluntary adoption of techniques and
practices, causing researchers to conclude, “As a direct result of teacher to teacher differences in
instructional effectiveness, some students make far less academic progress than they would
INFORMAL CLASSROOM WALK-THROUGHS 29
otherwise be expected to make simply by virtue of placements in ineffective classrooms”
(Rowan et al., 2002, p. 10).
Similar value added studies which have narrowed the scope of student achievement down
to the effectiveness of the teacher, eliminating the factors of student selection and teacher
assignment, have found the greatest influence on student achievement is effective classroom
instruction (Cohen et al., 2003; Kane et al., 2013; Rivkin et al., 2005; Wright et al., 1997).
Further, a teacher’s education level, years of experience after the first three years, class size, and
student’s family income made no measureable difference in a teacher’s impact on student
achievement. The results of these studies support two important findings regarding student
achievement: (a) the teacher has a greater influence on student achievement than all other
factors; and (b) standardized tests accurately report academic progress.
Effective teachers were effective with all students due to common components of
effective instructional practices regardless of the combination of all other factors, and the
variance in teacher quality was just as strong within schools as it was between schools, making
teacher quality a much larger issue for every school in every district (Cohen et al., 2003; Kane et
al., 2013; Rivkin et al., 2005; Wright et al., 1997). As a result, these studies call for teacher
evaluation which includes reliable and valid measures of a teacher’s ability to provide effective
classroom instruction which raises student academic achievement. They also support coaching
and training for principals and teachers to increase teacher quality. “In comparison to gains from
higher teacher quality, however, the estimates indicate that even a very costly ten student
reduction in class size, such as those undertaken in some U.S. states, produce smaller benefits
than a one standard deviation improvement in teacher quality” (Wright et al., 1997, p. 419).
INFORMAL CLASSROOM WALK-THROUGHS 30
In 2005, the Organisation for Economic Co-operation and Development (OECD)
conducted a study which looked at data across 25 countries and focused on issues related to
teacher quality and found that teachers are essential to school improvement and the most costly
component. A teacher’s educational level, previous experience, academic ability, and
knowledge of subjects taught made little difference in student achievement, but effective
instructional practices impacted student achievement (OECD, 2005). The study found that
teaching, unlike many other professions, does not have a built in cultural expectation of change
and life-long learning brought about by changes in knowledge, technology, and research. As a
result, teachers expect to do what they learned in college for their entire career; however, what
has been learned from the research on teaching and learning in the past 20 years should change
new and veteran teacher’s practice (OECD, 2005). The principal, as the instructional leader,
plays an important role in improving classroom instruction and changing teacher practice.
Principal’s Role in Improving Classroom Instruction and Instructional Leadership
With NCLB and now CCSS, the role of a principal has changed from that of a school
manager to an instructional leader (Grissom et al., 2013; Hallinger, 2007; Ing, 2009; Louis,
Dretzke, & Wahlstrom, 2010; May, Huff, and Goldring, 2012; Ovando, 2006; Turnball et al.,
2009). The research literature has focused on three key components related to instructional
leadership: defining instructional leadership and what it looks like; explaining the current model
in practice in schools; and describing the effects of instructional leadership on student
achievement. Instructional leadership is defined as transforming a school’s culture and
instructional practices so that effective teaching and learning are taking place and accessible to
all students. Instructional leadership is further characterized by working publically to improve
instructional practices and engaging teachers in reflective discourse about improving their
INFORMAL CLASSROOM WALK-THROUGHS 31
instructional practices without focusing on the principal’s personal attributes (Elmore, 2002;
Hallinger, 2007; Louis et al., 2010; Ing, 2009; May et al., 2012; Marzano et al., 2005).
The shift to principal as instructional leader has led researchers to not just look at what an
instructional leader does, but when, how, and why it is done. Marzano et al. (2005) conducted a
meta-analysis on research studies related to principal instructional leadership conducted since the
1970’s and looked at the effect of principal instructional leadership on student achievement.
They found the correlation between principal leadership and student achievement to equal a 10%
difference in achievement between effective and ineffective instructional leadership practices
(Marzano et al., 2005).
Preparing principals as instructional leaders. The impact of instructional leadership
upon student achievement is limited by a principal’s ability to actually become an instructional
leader. Louis et al. (2010) examined the impact of instructional leadership on student
achievement and determined that in addition to instructional leadership practices, principals also
need to be able to implement shared leadership and develop trust with the teaching staff.
However, principals report that they avoided the role of instructional leader for other school
management activities that were more tangible and managerial, especially in higher performing
schools, and instructional leadership activities changed and often decreased as a school’s
performance increased (Hallinger, 2007; May et al., 2012). Although principals are more
socialized in the role of instructional leader, they do not feel competent to perform this role and
face the obstacles to getting into the classrooms due in part to the diverse nature of instructional
leadership activities which can vary from one school year to the next (May et al., 2012). As
such, “there is little evidence to support the view that on a broad scale at either the elementary or
secondary school level that principals have become more engaged in hands-on directed
INFORMAL CLASSROOM WALK-THROUGHS 32
supervision of teaching or learning in the classrooms. The classroom doors appear to remain as
impermeable as a boundary line for principals in 2005 as 1980” (p. 230). This presents a
challenge to principals who are expected to take a hands-on role as an instructional leader
coaching, monitoring, and evaluating a teacher’s instructional performance which is primarily
accomplished through the process of formal and informal observations.
Formal and Informal Observations
As noted in the previous sections, the principal is expected to be in the classrooms to
observe teachers providing instruction in order to be able to determine the teachers’
effectiveness, areas where teachers need assistance, and to monitor student learning. There are
two main approaches to a principal’s observations in a teacher’s classroom, formal and informal.
The next section examines what the literature indicates about both effective and ineffective
formal and informal observations and how conducting informal classroom walk-throughs and
providing feedback to teachers by the principal influences teachers’ classroom instructional
practices.
Effective and Ineffective Formal Observations
Formal teacher evaluations in most school systems are required by union contracts and
local and state laws. The formal supervision model stems from an industrial model focused on
compliance which is generally normative, punitive, and disciplinary reinforcing the hierarchical
relationship between the teacher and the principal often thought of as a parent-child relationship
in which the parent is in charge, tells the child what to do, and scolds the child when it is not
done correctly (Downey, Steffy, English, Frase, & Poston, 2004).
Most frequently teachers are evaluated as “excellent” or “effective,” on average 99%,
with very few examples of teachers receiving negative comments and only when they are in
INFORMAL CLASSROOM WALK-THROUGHS 33
danger of being dismissed (Downey et al., 2004; Weisberg, Sexton, Mulhern, & Keeling, 2009).
For example, during the 2002-2003 and 2003-2004 school years, 96% of nearly 2,650 teachers in
the San Bernardino City Unified School District (SBCUSD) were rated as “meeting or
exceeding” performance standards and 28% to 32% of the teachers did not receive comments
providing suggestions for improvement on their performance evaluations, although 40 out of 54
campuses had not met NCLB proficiency standards (Peyton, 2005).
The FDR Group conducted a survey of 1,010 teachers who were representative of the
teacher population nationwide who were formally evaluated, and then facilitated six focus
groups to follow-up on the findings of the survey (Duffett, Farkas, Rotherham, & Silva, 2008).
Only 26% of teachers surveyed said their most recent formal evaluation was useful and
meaningful for their professional improvement, 44% called it “just a formality,” and 70%
reported that once a teacher became tenured it was merely a formality and not reflective of
teacher quality. Duffett et al. (2008) asked the surveyed teachers about untenured teachers and
83% responded that they should have a more rigorous evaluation process. When asked about
tenured or veteran teachers nearly one-half of the teachers reported that they knew a teacher who
should be dismissed for being ineffective and 76% reported that they knew veteran teachers who
were burned out and ineffective but “locked” in their teaching jobs due to pay and benefits
(Duffett et al., 2008).
Evaluating all teachers with the same undifferentiated process with little to no clear goals
for improvement or with no differentiated professional development or training based on
professional experience has been referred to as the “Widget Effect” (Weisberg et al., 2009).
Weisberg et al. state that the “Widget Effect” stems from the mindset that all teachers, “widgets,”
have the same effectiveness in the classroom and can be placed interchangeably with any
INFORMAL CLASSROOM WALK-THROUGHS 34
students without any negative impact to student achievement. Furthermore, the formal
evaluation process is flawed by limiting administrators to one or two short and infrequent formal
observations with little to no training and an engrained culture which views anything other than
an average or above average rating as a personal attack. Weisberg et al. (2009) ultimately called
for a formal teacher evaluation system that recognizes teachers who are truly excellent; provides
for formative assessment and differentiated professional development and training for teachers;
requires training and calibration for evaluators which holds them accountable for timely and
accurate evaluations; uses evaluations to make personnel decisions such as assignment,
promotion, and dismissal; and implements teacher dismissal policies that are efficient and timely,
but that also allow teachers who are evaluated poorly to exit the profession easily.
Donaldson (2009) reviewed teacher evaluation processes and instruments finding that
rating inflation was rampant in education which did not allow school systems to remove
ineffective teachers because almost all teachers were rated as “satisfactory,” and did not
recognize teachers who truly were satisfactory and outstanding. Additionally, many evaluation
instruments worked against improving and evaluating instruction, also known as “instrument
failure,” due to the focus on a checklist or working counter to the instructional focus of a district.
District policies and teacher contracts further supported “instrument failure” by placing
restrictive rules on what is evaluated, how it is evaluated, and limiting evaluation sessions to one
class period as infrequently as every five years with no real consequences for poor evaluation;
even though the instruments state that the goal was to improve instruction and student
achievement. Principals face additional difficulties in making the formal teacher evaluation
process robust as between 7% and 10% of a principal’s total work time is needed to evaluate 20
teachers. Furthermore, principals did not receive professional development which included
INFORMAL CLASSROOM WALK-THROUGHS 35
immediate feedback and coaching to increase effectiveness, were evaluating subjects or grades
they were unfamiliar with, and did not have a grasp of the components of great teaching
(Donaldson, 2009; Ho and Kane, 2013). Added to these difficulties with the formal teacher
evaluation process was the social issue of pushback from teachers when the process was made
more rigorous and accurate; and the risk of losing the teachers’ cooperation which was needed
for the principal to run the school, and if a teacher was moved out, there was a strong likelihood
of getting an equal or worse replacement (Donaldson, 2009).
In sum, principals’ formal evaluations of teachers have been shown to be ineffective as a
formative assessment for improving teacher’s instructional practices and ultimately student
achievement (Donaldson, 2009; Duffett et al., 2008; Jacob & Lefgren, 2006; Peyton, 2005;
Weisberg et al., 2009). As will be shown in the next section, principals also conduct informal
observations and researchers have studied their effectiveness.
Effective and Ineffective Informal Observations
Informal observations are most frequently referred to as classroom walk-throughs which
are “brief, structured, non-evaluative observations followed by collaborative conversations”
designed to support classroom instruction and student achievement, the goal of instructional
leadership (Feeney, 2014; Grissom et al., 2013). When informal classroom walk-through
observations are done well, the principal prepares teachers for the process by creating clarity of
purpose, describing who will participate, and outlining the process (Bloom, 2007; Grissom et al.,
2013). Bloom emphasizes that these initial steps followed by walk-throughs aligned with these
initial steps, lead to informal observations that are tied to a school’s instructional focus. Further,
these walk-throughs build professional learning communities which lead to continuous
INFORMAL CLASSROOM WALK-THROUGHS 36
improvement and transformative processes that improve classroom instruction and student
achievement.
Grissom et al. (2013) shadowed principals during their work day to observe their
instructional leadership practices, in particular informal classroom walk-throughs, to determine if
this strategy increased learning and academic achievement. Even though principals reported
that walk-throughs served as the primary source of information about classroom instructional
practices, the researchers (Grissom et al., 2013) found when a principal simply visited
classrooms without providing feedback to teachers there was a negative association to school
improvement. Bloom’s (2007) findings further supported Grissom et al. (2013) by showing that
when done poorly, classroom walk-throughs led to hostility and distrust among teachers and
principals, and became one more school reform activity that did not increase student
performance and caused teachers to be skeptical and locked into their existing instructional
practices whether or not they were effective for improving student learning. A major limitation
related to this study was not investigating the knowledge and skills that the principals had related
to classroom walk-throughs and without training on classroom walk-throughs, which takes 50
hours of ongoing training to build proficiency. Principals may not know how to use the practice
to support instruction and school improvement (Bloom, 2007; Feeney 2014).
The Wallace Foundation funded a study that provided a School Administration Manager
(SAM) to principals who agreed to spend the time in the classrooms that the SAM freed them
from managerial tasks (Turnbull et al., 2009). Turnbull et al. found with a SAM, principals were
exercising greater instructional leadership through spending more time in the classrooms, but
principals held few if any meetings with teachers to provide feedback and assistance to teachers
on what they were observing. Specifically, on average principals increased their time in the
INFORMAL CLASSROOM WALK-THROUGHS 37
classrooms during instruction by five hours per week, but only four minutes per day were added
to principals’ time spent providing teachers with feedback, modeling, and coaching (Turnbull et
al., 2009). Additionally, increased participation in school team meetings that looked at student
data, achievement results, and instructional practices were sporadic. Increasing principals’
available time in the classroom increased informal classroom observations providing compliance
with quickly observable items such as posted objectives or checklist items. Informal classroom
walk-throughs that are transformative to instructional practices will require more intense
principal training and support. “Principals expressed interest in providing coaching and ‘deep
feedback’ which would build teacher capacity. However, these principals also said this role
would require specific knowledge and skills, which they were working to develop” (Turnbull et
al., 2003, p. 53).
Providing teachers with formative feedback. Principals who provide professional
development to teachers in the form of instructional coaching see results in improved
instructional practices. Range et al. (2012) correlated formative supervision, or the process
whereby principals provide feedback and assistance to teachers in order to impact instructional
practices, with continuous improvement and a proactive stance toward improving teacher’s
performance. For classroom walk-throughs and the information collected during these walk-
throughs to be useful as formative supervision, informal classroom walk-throughs must be linked
with structured and purposeful coaching activities (Range et al., 2012). However, one of the
difficulties in many school districts is formative supervision and formal evaluations are treated as
the same process. Effective formative observations and supervision should not be evaluative but
rather non-threatening and solely for the purpose of improving classroom practice (Range et al.,
2012). Range et al. found that teachers valued open, honest, and sincere constructive feedback,
INFORMAL CLASSROOM WALK-THROUGHS 38
built on a relationship of trust, which included time for reflection and discussion of areas of
improvement directly related to their instructional practices, was the most valuable component of
a professional growth program.
Classroom walk-throughs which include the components of formative observations,
coaching, and teacher feedback are important for effective and meaningful interactions between
the principal and classroom teacher and result in improved instructional practices leading to
increased student achievement.
Conducting informal classroom walk-throughs and providing feedback to teachers.
Researchers have yet to make a direct definitive link between instructional leadership activities
and increased student achievement, however when principals set a specific focus for the walk-
throughs, conduct them regularly, and provide teachers with specific feedback; there is a positive
correlation to the teachers’ perceptions of the school’s instructional climate (Ing, 2010). Further,
Ing (2010) determined that conducting classroom walkthroughs without providing feedback to
teachers has no impact upon the teacher’s perception of the school’s instructional climate.
Teacher training most often focuses upon formal lecture-style generic professional
development with very few opportunities for on-the-job personalized training. Mesler Parise and
Spillane (2010) conducted a study on the impact of on-the-job training through the practice of
informal classroom walk-throughs to change teachers’ instructional practices. The researchers
had two primary findings from their study; first, teachers’ perceptions of their school’s
conditions and the need for change are very different from a teacher’s willingness to change their
on-the-job performance. Meaning that on-the-job training has the potential of having the theory
learned in formal professional development actually impact classroom practice. Second,
teachers’ learning with on-the-job training was not different between math and English language
INFORMAL CLASSROOM WALK-THROUGHS 39
arts. Teachers are generally more comfortable and effective in one of these two subject areas,
and on-the-job training has the potential to balance out performance differences and improve
instruction in both content areas (Mesler Parise & Spillane, 2010).
The classroom walk-through process was developed from the concept of “management
by walking around” which was popularized in the 1980’s and supported by the idea that the most
effective managers were those that were out among their employees and involved in “the work”
(Downey et al., 2004). There are at least 18 documented models of classroom walk-throughs and
despite a large amount of professional development on classroom walk-throughs, there is not one
agreed upon most effective model or best practice (Kachur, Stout, & Edwards, 2010). Classroom
walk-throughs have four components in common:
1. They are short, informal, on-going, and paperless conversations to improve
instruction and professional practice.
2. The teacher becomes the main audience for the principal, like the students are for the
teacher, and the principal gives teachers information on their instructional practices
for self-reflection.
3. They are conducted in the classroom where the instruction occurs in order to get a
brief snapshot of the teacher’s instructional practices.
4. Separate from formal observations and evaluations, it’s a process that a principal can
accomplish through clear communication and building trust (Downey et al., 2004,
Kachur et al., 2010).
All models of classroom walk-throughs incorporate reflective conversations and dialogue
based upon the data gathered during principal walk-throughs (Downey et al., 2004; Kachur et al.,
2010). Downey et al. (2004) emphasize that follow-up conversations are essential to the
INFORMAL CLASSROOM WALK-THROUGHS 40
classroom walk-through; simply walking in and out of classrooms does not improve instructional
quality or student achievement. The follow-up conversations should: occur in relative proximity
to the walk-through, be done in private, be paperless, focus on the knowledge that teachers
possess regarding their classroom practices, end with a reflective question designed to help the
teacher think about his/her practice without having the principal tell the teacher what to do, and
continue ongoing conversations regarding professional practice between the teacher and
principal (Downey et al., 2004; Kachur et al., 2010). Kachur et al. emphasize three “T’s” during
this process “Trust, Transparency, and Truthfulness.”
Professional Development for Principals on Informal Classroom Walk-throughs
Principals have to assume the role of mentor and coach to effectively conduct classroom
walk-throughs, this is a role that principals are often not trained to take, does not come naturally,
and requires time and coaching in order to reach a proficient level (Downey et al., 2004; Kachur
et al., 2010). Training principals to conduct effective classroom walk-throughs involves using
adult learning theory which includes the following key concepts: recognizing and reducing
anxiety, defining and elaborating expectations, identifying and employing experience, allowing
and supporting active participation, identifying and utilizing relevant information, and providing
opportunities for change and growth (Kachur et al., 2010). Downey et al. (2004) found that it is
extremely difficult for adults to assume the responsibility for and actually accomplish their own
professional growth. As such, experts, in the form of consultants, are often relied upon to
provide this professional growth to principals.
Nagel (2014) found that principals were asked to implement teacher observation and
evaluation systems without sufficient tools, resources, and knowledge. When observing and
evaluating teachers, principals should receive on-going professional development, including
INFORMAL CLASSROOM WALK-THROUGHS 41
technology and related tools, and credentialing programs should include explicit instruction
regarding effective teacher observation and evaluation (Nagel, 2014).
Summary
Formal observations are required in many states and school systems. Results from formal
observations indicate that nearly all except the absolutely worst teachers receive satisfactory or
excellent ratings despite teachers and principals reporting that anywhere from 10 to 20 percent of
the teachers at their school need to improve or even be terminated. Informal observations,
specifically classroom walk-throughs, when conducted by a principal with follow-up activities
change a school’s culture and climate as the principal serves as the instructional leader, breaks
down the hierarchal structure of principal-teacher supervision, and improves teachers’ classroom
instructional practices. However, when a principal conducts informal classroom walk-throughs
without providing feedback to teachers or receiving proper training and support, the informal
classroom walk-throughs can do greater harm than good. Thus it is important to determine how
principals, especially those who lead high performing schools, conduct informal classroom walk-
throughs and provide feedback to teachers and use this knowledge to develop training,
instructional materials, and coaching and mentoring for other principals. In order to effectively
train novice principals, training must be based on capturing the knowledge and skills of these
successful principals.
Using Subject Matter Experts to Train Non-experts
Experts are frequently called upon for their knowledge and skills to teach, to inform
curriculum content and instructional materials, and to mentor and coach others to perform
complex tasks and solve difficult problems. The purpose of education is to replicate knowledge
(Jackson, 1985). Education started with the traditional master and apprentice model in which the
INFORMAL CLASSROOM WALK-THROUGHS 42
objective was for the apprentice to imitate the master. As the concept of education evolved into
a model of one instructor to many students, the objective of imitating the master remained.
Contemporary education seeks to maintain this traditional model (Jackson, 1985).
Current research shows, however, that experts may omit up to 70% of the critical
knowledge and skills novices need to replicate expert performance. Feldon and Clark (2006)
found that experts’ self-reports on critical knowledge and skills are often incomplete and
inaccurate, and errors and omissions are often introduced that can hinder novices’ subsequent
performance of the target task. The errors and omissions made by experts cause novices to
receive incomplete information which results in a void that the novices fill with their own
information, which often contains misconceptions about successful strategies (Feldon & Clark,
2006).
Cognitive Task Analysis has been shown to be an effective method for capturing both the
conscious and automated knowledge experts use to perform complex skills and solve difficult
problems. To further understand why CTA is effective, the following sections examine two
types of knowledge, the nature of automaticity, and the characteristics of expertise.
Knowledge Types
Merrill (1994) suggests that there are four types of knowledge: concept, processes,
principles, and procedures. Further, Merrill stated that there were two uses of these four
knowledge types; first, knowledge that you say or tell, or declarative knowledge; and knowledge
that you use or apply, or procedural knowledge. Declarative knowledge, or factual knowledge, is
recalled from long-term memory and is consciously observable in working memory, however it
in not sufficient for performance. Procedural knowledge, or production knowledge, is required
INFORMAL CLASSROOM WALK-THROUGHS 43
for skilled performance. Automaticity of procedural knowledge develops as skills are practiced
and automated knowledge is outside of the consciousness of the expert (Clark et al., 2008).
Declarative Knowledge
Declarative knowledge is knowledge that is controlled and can be changed abruptly in the
working memory. Declarative knowledge is described as overt knowledge about facts, events,
and objects which includes concepts, principles, and processes that can be used to articulate the
answers to questions of “what, why, and that” (Anderson & Schunn, 2000; Clark & Estes, 1996).
Nearly all knowledge is first learned in a declarative form characterized by the speed with which
it can be learned and modified when it is retrieved from the long-term memory seemingly
intended to help solve novel tasks (Anderson & Fincham, 1994; Clark & Estes, 1996).
Declarative knowledge is also factual and goal-independent which includes beliefs about tasks or
personal abilities which may influence goal setting (Corbett & Anderson, 1995; Paris et al.,
1983). These characteristics of declarative knowledge led Anderson (1996a) to suggest that the
proposition was the “atomic unit of thought,” or in other words, at the basic level declarative
knowledge is the relationship between two things or “this is that.”
Anderson (1982) described that declarative knowledge by itself is insufficient to execute
skilled performance. Knowledge is first learned in a conscious, declarative form represented in
chunks which are then transformed over time with repeated use into a largely unconscious,
automated procedural form freeing-up the working memory (Anderson, 1982; Anderson, 1993;
Anderson & Schunn, 2000). The declarative stage of knowledge is where the domain knowledge
is directly embodied in procedures for performing a skill made up of production rules (Anderson,
1982; Anderson & Fincham, 1994). Declarative knowledge, knowing why and what something
is enables and supports the attainment of procedural knowledge, the how and when something is.
INFORMAL CLASSROOM WALK-THROUGHS 44
Procedural Knowledge
Procedural knowledge is knowledge about “when and how” to perform a task and
apply specific procedures, skills, or methods which include sequences and steps to be followed
during a simple or complex task (Anderson & Krathwohl, 2001; Ambrose, Bridges, DiPietro,
Lovett, & Norman, 2010; Clark & Estes, 1996). Anderson (1982) explained procedural
knowledge consists of “IF/THEN” statements that rely on facts generated from declarative
knowledge (IF) that are converted to decision steps from procedural knowledge (THEN).
Cognitive skill is a function of translation or transition from the declarative stage to the
procedural stage (Anderson, 1982). Procedural knowledge is goal oriented, in that it promotes
problem solving, and requires repetition, practice, and direct instruction (Corbett & Anderson,
1995; Paris et al., 1983).
Procedural knowledge involves knowing how to perform a task and includes the steps
and actions taken during goal attainment (Paris et al., 1983). It is knowledge of different
processes that result from subject-specific declarative or factual knowledge, for example
knowledge of the skills and techniques in performing a complex task such as surgery
(Anderson & Krathwohl, 2001). Clark and Estes (1996) suggested that procedural knowledge
is difficult to learn and fast to execute. Further, it requires practice and feedback but once
learned it demonstrates a high level of expertise and is automated requiring little conscious
thought, like driving an automobile. Once procedural knowledge is developed and automated,
it is very difficult to change or revise (Anderson, 1982).
Anderson and Schunn (2000) found that procedural knowledge is production
knowledge represented by condition-action or procedural rules. Additionally, procedural rules
and factual chunks combine to form competence in a given domain. The activation or retrieval
INFORMAL CLASSROOM WALK-THROUGHS 45
of factual chunks is influenced by how well the production knowledge is learned (base-level
activation) and by the context (association activation) in which factual chunks are retrieved.
Practice is needed to strengthen both the factual knowledge and production rules and retention
is a function of how well information was learned and the pattern of practice (Anderson and
Schunn, 2000).
Conditional knowledge is a subset of procedural knowledge. Conditional knowledge is
described as knowing when or why to use or not use a given procedure, providing the
circumstances or rational for various actions, including value judgments, and helping modulate
procedural and declarative knowledge (Anderson & Krathwohl, 2001; Paris et al., 1983).
Conditional knowledge modulates the fact-to-action process (Anderson, 1982). Paris et al.
(1983) emphasized both “skill and will” as components of conditional knowledge and task
execution as the amount of knowledge or skill may be an indication of capability but not
necessarily likelihood of actual task execution. It is “will” or motivation that directs an
individual’s actions.
Declarative, procedural, and conditional knowledge are acquired as one transitions
from novice to expert (Paris et al., 1983). With repetition and practice, both declarative and
procedural knowledge become stronger and performance becomes more fluid, consistent, and
automated.
Automaticity
Through repeated performance and deliberate practice of a task, declarative and
procedural knowledge becomes automated and unconscious in nature and speed increases in
performing a task while the amount of active mental effort decreases (Feldon, 2007). Anderson
(1996b) identified three stages of automaticity:
INFORMAL CLASSROOM WALK-THROUGHS 46
1. The first stage is the interpretive stage or cognitive stage in which a learner is able
to complete a task or at least a close approximation of the task with initial
instructions that are often verbal. This stage frequently involves talking to oneself
when performing the action.
2. The second stage is the knowledge compilation or associative stage. In this stage
the learner works through the procedure and applies or learns the declarative
knowledge necessary to correct procedural errors. As the learner corrects errors
and develops stronger procedural knowledge the verbal cueing of talking to
oneself decreases and ultimately disappears.
3. The third stage is the strengthening and tuning or autonomous stage where the
learning performs the procedure automatically without verbal cueing and any
changes made to the procedure serve to strengthen or make the process more
efficient.
A fourth stage of automaticity was identified by Ericsson, Krampe, and Tesch-Römer (1993)
which is reserved for expert performance only. In this fourth and final stage, experts have
mastered most of the training provided by their teachers and coaches and add their own
innovative and creative contributions to the field of expertise.
Automated processes often initiate without prompting and once initiated, they run to
completion without being available for conscious monitoring (Feldon, 2007). Feldon found even
when teachers are made aware of omissions in their automated teaching processes or are
provided with goals to change these automated processes, they ironically fail to make changes
because the working memory becomes quickly occupied with the changes and the automated
processes begin and run to completion because the working memory is occupied. This results in
INFORMAL CLASSROOM WALK-THROUGHS 47
the inability to attend to and monitor automated processes in order to bring about desired
changes. Further, Feldon determined as experts develop their conscious declarative knowledge,
it becomes gradually more automated.
Automated knowledge helps to alleviate cognitive overload and/or processes that can
impede the efficiency of working memory because the length and amount of information that can
be retained and processed in working memory is limited (Kirschner, Sweller, & Clark, 2006).
Kirschner et al. go on to say that procedural knowledge is difficult to articulate because it has
become automated, non-conscious expert action, so critical information omitted by experts may
confound effective knowledge sharing. With repeated practice, cognitive tasks become fluid and
automatic and SMEs are able to deploy strategies to solve problems with ease (Clark, 1999).
Automaticity is a double-edged sword, on one hand, it frees up working memory by
unconsciously processing and running procedures, which then is available to attend to novel
information and tasks (Wheatley & Wegner, 2001). When experts encounter new problems, they
have the working capacity to see what is novel in new problems and figure out what to do
consciously. On the other hand, once a skill or task is automated it is very resistant to change or
modification and takes considerable sustained monitoring of mental processes to modify or
eliminate (Clark, 2008; Wheatley & Wegner, 2001).
Automaticity enables the unconscious performance of tasks requiring declarative and
procedural knowledge, freeing up working memory to address novel tasks, however due to the
unconscious nature of automaticity it is resistant to change and difficult to modify, eliminate, or
express to others; thus making automaticity an important, albeit complex, characteristic of
expertise.
INFORMAL CLASSROOM WALK-THROUGHS 48
Expertise
Characteristics of Experts
The characteristics of expertise include extensive and highly structured knowledge of the
domain, effective strategies for solving problems within the domain, and expanded working
memory that utilizes elaborated schemas to organize information effectively for rapid storage,
retrieval, and manipulation (Chi, 2006; Glaser & Chi, 1998). An expert is defined as a brilliant
or distinguished journeyman, due to a track record of strong performance or years of working in
a specific domain, who is highly regarded by peers and whose judgments are uncommonly
accurate and reliable (Chi, 2006; Feldon, 2007). Further, an expert is able to consistently
demonstrate superior performance reliability upon demand due to special knowledge or skills
acquired from extensive experience with subdomains (Chi, 2006; Ericsson & Lehmann, 1996).
Expert performance was initially thought to be a gift from the gods and was first
approached be theorists as something that an individual was “born” with (Ericsson & Charness,
1994). However, the more modern view supports the idea that expert performance is the result
of skill that develops over time and with increased exposure to a task, however sufficient
experience and practice will not lead to improved performance beyond any innate physical
barriers prohibiting expertise (Ericsson, 2004; Ericsson & Charness, 1994; Ericsson et al., 1993).
Studies of expertise have focused in two areas: (a) an “absolute approach” which starts with the
tacit assumption that expertise happens by chance and is innate so the focus of these studies are
“exceptional” individuals; and (b) a “relative approach” in which experts are compared to non-
experts or novices which assumes that expertise is a level of proficiency that novices can achieve
(Chi, 2006). The “relative approach” has been shown to be more accurate and is used more
frequently in studies.
INFORMAL CLASSROOM WALK-THROUGHS 49
Experts are set apart from novices in that they have developed schemas allowing them to
efficiently organize information so it is quickly and efficiently retrieved with minimal effort
(Bedard & Chi, 1992). Experts can optimize normal working memory capacity limitations by
seeing beyond function and simple schemas using principles, concepts, connections, and deeper
learning allowing them to create mental models so they can anticipate based on advanced
perceptual cues and forward thinking (Bedard & Chi, 1992; Ericsson & Lehman, 1996). This
forward thinking enables an expert to view problems differently from novices who are more
literal, predictable, and use surface information. An expert has the ability to determine the
relevance of a situation through better developed working memory, short-term and long-term
memory, and reflective memory skills between the long-term and working memory to detect
with much greater speed and accuracy that a problem exists; enabling the expert to assess the
degree of difficulty to generate a domain-relevant effective solution strategy through planning,
reasoning, and anticipation of future events (Chi, 2006; Ericsson & Charness, 1994; Ericsson &
Lehmann, 1996; Feldon, 2007; Glaser & Chi, 1988).
In sum, experts leverage their highly structured knowledge of relevant concepts and
processes, or declarative and procedural knowledge, not as a consequence of domain-specific
experience but rather through the acquisition of specific memory skills and highly structured
knowledge of relevant concepts and principles tailored to the demands of working memory by a
specific activity (Ericsson & Lehmann, 1996; Feldon, 2007).
Experts are limited when their expertise does not match the task that they are trying to
solve, in other words, expertise does not transfer from one domain to another (Bedard & Chi,
1992; Ericsson et al., 1993). Bedard and Chi emphasized in cases where an expert’s domain
knowledge does not match the task, novices can outperform experts, supporting the importance
INFORMAL CLASSROOM WALK-THROUGHS 50
of declarative and procedural knowledge types when addressing expertise. Subject matter
experts possess distinct and extensive declarative and procedural knowledge that enabled them to
perform tasks expertly in their specific subject area much differently than novices, and this
expertise is developed through specific actions.
Building Expertise
Expertise, by its nature, is acquired as a result of continuous and deliberate practice in
solving problems in a domain. Alexander (2003) developed the Model of Domain Learning
(MDL) which focuses on the journey that an individual takes from novice to expert specifically
in academic domains to bridge the understanding of expertise and educational practice. The
MDL focuses on three components: knowledge, strategic processing, and interest; all three of
which play a role toward developing expertise as they influence each other at every stage, but
differently at each stage. According to the MDL, the components of knowledge, strategic
processing, and interest configure differently as individuals progress through the three stage of
domain learning which are:
1) Acclimation: the initial stage of domain expertise. The term signifies the demands
placed on student as they acclimate to a complex, unfamiliar domain.
2) Competence: competent individuals not only demonstrate a foundational body of
domain knowledge, but that knowledge is also more cohesive and principled in
structure.
3) Proficiency: a synergy among components is required for movement from competence
into expertise. Not only is the knowledge base of experts both broad and deep, but the
experts are also contributing new knowledge to the domain.
INFORMAL CLASSROOM WALK-THROUGHS 51
Ericsson (2004) also developed a theory of skill acquisition based on an individual’s
primary goal being to reach a level of mastery that will allow him/her to perform everyday tasks
at an acceptable level or to engage proficiently in recreational activities with his/her friends.
During the initial phase of learning, novices try to understand the activity and concentrate on
avoiding mistakes. With more experience in the middle phase of learning, gross mistakes
become rare, performance appears smoother, and learners no longer need to concentrate as hard
to perform at an acceptable level. After a limited period of training and experience, frequently
less than 50 hours for most recreational activities such as typing, playing tennis, or driving a car;
an acceptable standard of performance is typically attained. As individuals adapt to a domain
and their skills become automated, they are able to execute skills smoothly and without apparent
effort. As a consequence of automation, performers lose conscious control over execution of
those skills, making intentional modifications difficult. Once the automated phase of learning
has been attained, performance reaches a stable plateau with no further improvements (Ericsson,
2004). In the absence of innate physical barriers, the challenge for aspiring expert performers is
to avoid the arrested development associated with automaticity and to acquire cognitive skills to
support their continued learning and improvement which is accomplished with more experience
coupled with deliberate practice (Ericsson, 2004).
Deliberate practice leads to elite expert performance and to physical change (Ericsson &
Charness, 1994). Deliberate practice is characterized by the aspiring expert’s motivation to
attend to the task and exert effort to improve his/her performance. The design of the task should
take into account the preexisting knowledge of the aspiring expert so that the task can be
correctly understood after a brief period of instruction. This ensures that the deliberate practice
is designed to improve specific aspects of performance in a manner that attained changes can be
INFORMAL CLASSROOM WALK-THROUGHS 52
successfully integrated into representative performance (Ericsson, 2004a). The aspiring expert
should receive immediate feedback and knowledge of results related to his/her performance, and
finally the subject should repeatedly perform the same or similar tasks (Ericsson & Charness,
1994). Once expertise is developed, it has significant consequences.
Consequences of Expertise
As new knowledge becomes automated and unconscious, experts are often unable to
completely and accurately recall the knowledge and skills that comprise their expertise,
negatively impacting instructional efficacy and leading to subsequent difficulties for learners.
Once knowledge and skills become automated they are deeply ingrained and difficult to modify
or change, and evidence suggests that habitual approaches to problems are goal activated and
start automatically (Feldon, 2007).
Feldon (2007) reviewed studies on the accuracy of expert’s self-reports and determined
that errors are prevalent in expert’s self-reports of problem-solving processes and omissions
increased as skills improved especially in the expert’s domain. Feldon found that individuals
tend to attribute most, if not all, of their actions to intentional decision making processes, and the
strength of this belief can lead them unintentionally to fabricate consciously reasoned
explanations for their automated behaviors. Hence, reports may be inaccurate when participants
rely on incorrect preexisting causal theories to explain their processes and experts commonly fail
to articulate relevant cues seen in problem states (Feldon, 2007). Consequently, the most
frequent employed elements – presumably those of greatest utility within a domain of expertise –
would be the most difficult to articulate through recall (Feldon, 2007). The automaticity of
experts impairs their ability to consciously identify many of the decisions they make thereby
INFORMAL CLASSROOM WALK-THROUGHS 53
omitting key details and process information necessary to provide instruction for optimal
performance.
Expert Omissions
Experts in an instructional role may unintentionally leave out information that students
must master when learning procedural skills. Feldon (2004) concluded that automaticity and the
accuracy of expert’s self-reporting were negatively correlated. Recent research has shown that
when experts describe how they perform a difficult task, they may unintentionally omit up to
70% of the critical information novices need to learn to successfully perform the procedure
(Feldon & Clark, 2006). This is a serious problem because it forces novices to “fill in the
blanks” using less efficient and error-prone trial-and error methods. Moreover, as these errors
are practiced over time, they become more difficult to “unlearn” and correct (Clark, 2008).
There are two reasons for this problem. First, as individuals gain expertise, their skills
become automated and the steps of the procedure blend together due to unconscious knowledge
as a result of years of practice and experience (Clark & Estes, 1996). This causes experts to omit
specific steps when trying to describe a procedure because this information is no longer
accessible to conscious processes (Clark & Elen, 2006). Secondly, many experts are not able to
share the complex thought processes of behavioral execution of technical skills. Even experts
who make attempts to “think out loud” during procedures often omit essential information and
have difficulty identifying points during a procedure where they make decisions (Clark & Elen,
2006). Errors are not often recognized by experts because of the automated and unconscious
nature of the knowledge described and are likely to increase in number and impact under
stressful situations (Hunt and Joslyn, 2000; Wheatley & Wegner, 2001). Due to expert
INFORMAL CLASSROOM WALK-THROUGHS 54
omissions in self-reports, knowledge elicitation techniques such as Cognitive Task Analysis are
necessary to elicit expert knowledge.
Cognitive Task Analysis (CTA)
Definition of CTA
Cognitive Task Analysis (CTA) has evolved from traditional task analysis methods, and
is utilized in order to elicit and explicate expert knowledge within a specific domain (Hoffman &
Militello, 2009). CTA uses a variety of interview and observation strategies to capture a
description of the explicit and implicit knowledge that experts use to perform complex tasks, as
well as overt and covert cognitive functions that form the integrated whole (Clark et al., 2008).
CTA captures both conscious and automated knowledge of complex tasks that can extend over
several hours or days (van Merriënboer, Clark, & de Croock, 2002). CTA has been defined as an
“extension of traditional task analysis techniques to yield information about the knowledge,
thought processes, and goal structures that underlie observable task performance” (Chipman,
Schraagen, & Shalin, 2000, p. 3). Modern work relies upon automation to support human action
and is focused on the cognitive aspects of work that are not directly observable, so the products
of CTA are used for teaching, individual training, performance assessment, and the development
of expert systems (Chipman et al., 2000).
CTA History
The origins of CTA date back as far as 1880 and encompass many aspects of the study of
cognitive engineering and task analysis. Studying work via task analysis is firmly rooted in
Taylor’s time and motion studies as well as the bricklaying studies of Gilbreth in the early
1900’s whereby the physical cognitive elements of work were studied in order to improve human
performance and system design to decrease fatigue, injuries, and labor costs (Annett, 2000;
Hoffman & Militello, 2009; Militello & Hoffman, 2008). In the 1950s the foundations for
INFORMAL CLASSROOM WALK-THROUGHS 55
cognitive psychology began to take hold as the concept of mental workload and information
processing became more important. As a result of this, there became increased interest in
capturing human expertise, which time and motion studies could never capture; the mental
processes and decisions behind expert performance, knowledge elicitation of complex tasks, and
the explication of declarative and procedural knowledge (Annett, 2000).
This was the foundation of modern CTA which began in the late 1960’s as a response to
applied psychology, industrial engineering, human factors, and social and cognitive activities in
the workplace (Hoffman & Woods, 2000; Militello & Hoffman, 2008). CTA methods emerged
in the 1980s in response to workplace demand for expert systems and other applications of
artificial intelligence however the term cognitive task analysis came into being as early as the
1970s (Annett, 2000; Hoffman & Woods, 2000). CTA has been long in evolution and over the
past century has become one of the most successful methods of elicitation of expert knowledge
(Clark & Estes, 1996). Modern CTA is the advanced task analysis system that fills in the gap of
determining unobservable mental processes and decisions steps which evolved from traditional
behavioral task analysis and the study of cognitive engineering.
Cognitive Task Analysis Methodology
A number of researchers have identified the stages through which a typical, ideal CTA
would proceed. The ideal model of CTA, one that is not subject to resource restrictions, is
typified by a series of five discrete steps; which are
1) A preliminary phase;
2) The identification of knowledge representations;
3) Knowledge elicitation techniques;
4) A review and possible modification of the knowledge elicited to date by experts; and
INFORMAL CLASSROOM WALK-THROUGHS 56
5) Using the results of the analysis as a basis for an expert system or expert cognitive
model (Chipman et al., 2000; Clark et al., 2008).
Over 100 types of CTA have been identified (Cooke, 1994; Yates, 2007). Despite the varieties
of CTA, in a general sense, most varieties follow the five-stage process.
Clark et al. (2008) and Clark (2014) state that concepts, processes, and principles (CPP)
have been one of the most commonly used “evidenced-based” CTA methods. CPP draws from
two methods: Precursors, Actions, Results, and Interpretations (PARI) which is a process in
which experts, frequently in pairs, identify complex cognitive and behavioral demands in each of
the four categories to complete a task as they think aloud, use diagrams and drawings, and ask
probing questions; and Merrill’s (2002, 2006) recommendations regarding instructional design
(Clark, 2014; Hoffman & Militello, 2008; Tofel-Grehl & Feldon, 2013; Yates, 2007; Yates &
Feldon, 2011). Despite the number of methods for conducting CTA, it has been shown across
different disciplines to be an effective way to elicit expert knowledge.
Effectiveness of CTA for Capturing Expert Knowledge
Cognitive Task Analysis has proven to be an effective method for capturing the explicit
observable behaviors, as well as the tacit, unobservable knowledge of experts. CTA is regarded
as a necessary component of research in complex cognitive work because CTA addresses the
issues of research into the interaction of people, technology, and work (Hoffman & Militello,
2009; Yates, 2011). CTA is able to identify the explicit and implicit knowledge of experts to use
for training and technology. Knowledge that can be captured from experts includes domain
content, concepts and principles, experts’ schemas, reasoning and heuristics, mental models, and
sense making (Hoffman & Militello, 2009; Yates, 2011).
INFORMAL CLASSROOM WALK-THROUGHS 57
CTA methodology uses multiple experts to capture explicit knowledge and skills needed
to perform complex tasks and solve difficult problems. The use of CTA knowledge elicitation
techniques is able to capture expert declarative and procedural knowledge for use in training and
technology. A number of studies have shown that although experts individually omit critical
information when describing how to perform complex task and solve difficult problems, this gap
in knowledge can be meditated for by eliciting expert knowledge from multiple experts using
CTA procedures (Tirapelle, 2010; Zepeda-McZeal, 2014). These studies have consistently
revealed the effectiveness of aggregating multiple individual protocols of expert action and
decision steps for performing complex tasks into a gold standard protocol. The gold standard
protocol upon expert review is considered to be the most complete representation of how to
perform the complex task or solve a difficult problem. Moreover, Crispen (2010) and Bartholio
(2010) replicated the methodology of Chao and Salvendy (1994) and demonstrated that using
three to five experts captured up to 70% of action and decision steps, and eliciting knowledge
from additional experts beyond three to five results in only marginal increases (less than 10%) of
action and decision steps.
Effectiveness of CTA-Based Training
Studies that have applied Cognitive Task Analysis to capture knowledge and deliver
instruction have uncovered several benefits and useful design strategies as compared to other
forms of instruction. CTA is able to identify the explicit and implicit knowledge of experts to
use for training and technology, and the knowledge that can be captured from experts includes:
domain content, concepts and principles, experts’ schemas, reasoning and heuristics, mental
models, and sense making (Hoffman & Militello, 2009). Data captured from CTA supports
effective and efficient training and instructional activities in complex systems. CTA can be used
INFORMAL CLASSROOM WALK-THROUGHS 58
for training in a variety of ways, such as “cognitive training requirements, scenario design,
cognitive feedback, and on-the-job training” (Crandall, Klein, & Hoffman, 2006, p .196).
CTA-based training has a long and rich history of result in the field of medicine.
Tirapelle (2010) conducted a study to determine the increase in declarative knowledge and
procedural accuracy and recall of critical decision steps for CTA-based training on Open
Cricothyrotomy for medical students. Tirapelle found that CTA-based training resulted in
significant increases in both procedural accuracy and recall of critical decision steps for medical
students on Open Cricothyrotomy. Embry (2012) and Gucev (2012) conducted blind studies
using CTA-based anesthesia training and found increases in the amount of declarative and
procedural knowledge and speed of performing procedures for trainees receiving CTA-based
training. Crandall and Getchell-Reiter (1993) conducted a study in which the findings were
favorable in supporting CTA-based training methods as more effective for training in nursing
practice. Fackler et al. (2009) demonstrated that CTA-based training was shown useful for
understanding communication and decision making among physicians, resulting in the
possibility of preventing harm to patients. CTA-informed learners, or employees, may be
considered better trained and perhaps more appealing to employers throughout the medical field
(Clark, 2014).
CTA-based instruction has also been successfully used by the military and Federal
Government. Crandall et al. (2006) conducted a CTA-based training study with firefighters for
the National Emergency Training Center to enhance their training systems in the areas of critical
decision points, judgments, and patterns, which are essential to the training of firefighters. The
CTA-based training proved to be more effective than the then current method of training
firefighters. A research study by Schaafstal et al. (2000) was conducted on a group of students
INFORMAL CLASSROOM WALK-THROUGHS 59
who were attending the Naval Weapon Engineering School, and demonstrated the effectiveness
of CTA-based training. The key success indicators of the CTA-based training were: systematic
processing, understanding of the troubleshooting functions, and the correct identification of
problems with a reduction in training time. The CTA method proved to be effective in
increasing expertise from a theoretical perspective as well as a practical level with decreased
costs (Schaafstal et al., 2000).
Meta-analysis of studies. Two meta-analyses of CTA-based studies have investigated
the effectiveness of CTA-based training over more traditional non CTA-based training methods
(Lee, 2004; Tofel-Grehl & Feldon, 2013). Lee (2004) reviewed 39 studies based on pretest and
posttest results after CTA-based training and found performance gain of 75.2%. Tofel-Grehl and
Feldon (2013) found an effect size for CTA-based training that is three time that of non CTA-
based training, such as traditional behavioral task analysis and expert self-reporting. According
to Clark (2014) CTA-based instruction contributes to posttest learning gains of 31% versus
traditional training methods.
Critical Decision Method (CDM), a CTA method in which experts recall the critical
decisions that they made during incidents which usually involve life or property (Klein,
Calderwood, & MacGregor, 1989), and PARI CTA elicitation methods have been shown to be
more effective toward improved training outcomes than other CTA methods. The effect size for
CDM was .329 and for PARI was 1.598, resulting in a 13% increase in learning for CDM and a
45% increase in learning for PARI (Clark, 2014; Tofel-Grehl & Feldon, 2013). CTA-based
training consistently demonstrates increased learning over more traditional non CTA-based
training methods (Clark, 2014; Lee, 2004; Tofel-Grehl & Feldon, 2013).
INFORMAL CLASSROOM WALK-THROUGHS 60
Summary
CTA is a knowledge elicitation method that uses interview and observation techniques to
capture a description of the explicit and implicit knowledge that experts use to perform complex
tasks. When experts are asked to describe how to perform domain-specific tasks, they
unintentionally omit up to 70% of critical information novices need to learn to successfully
perform complex tasks. There are at least 18 different documented methods for conducting the
complex domain-specific task of informal classroom walk-throughs and providing feedback to
teachers, and traditional training relies upon subject matter experts, in the form of consultants, to
provide instruction to novice principals. The purpose of this present study was to conduct a CTA
to elicit school principals’ expert knowledge, that is, the critical action and decision steps of
subject matter experts, in order to develop a gold standard protocol that can be used as the basis
for training novices on the task of informal classroom walk-throughs and providing feedback to
teachers.
INFORMAL CLASSROOM WALK-THROUGHS 61
CHAPTER THREE: METHODS
The purpose of this study was to conduct a Cognitive Task Analysis to determine the
knowledge and skills that expert principals (subject matter experts or SMEs) rely upon when
conducting informal classroom walk-throughs and providing feedback to teachers to improve
teachers’ classroom instruction. The task examined in the study was conducting informal
classroom observations of teachers and providing feedback by principals to improve teachers’
classroom instruction. Given the nature of expertise, it was assumed that these subject matter
experts had both declarative and procedural knowledge that was highly automated and often
unconscious making it difficult for them to give detailed or accurate descriptions of what, why,
how, and when they conducted informal classroom walk-throughs and then provided feedback to
teachers.
As such, the questions that guided the study were:
• What are the action and decision steps that expert principals recall when they
describe how they conduct informal classroom walkthroughs and provide
feedback to teachers?
• What percentage of actions and/or decision steps, when compared to a gold
standard, do expert principals omit when they describe how they conduct
informal classroom walk-throughs and provide feedback to teachers?
Participants
The participants in this study were K - 12 principals identified as experts in conducting
informal classroom walk-through observations followed by providing feedback to teachers on
what was observed. As operationalized for this study, expert principals had at least five years of
recent, consistent, and recognized success conducting informal classroom walk-throughs and
providing feedback to teachers. These expert principals had reliable and commonly recognized
INFORMAL CLASSROOM WALK-THROUGHS 62
performance that could be validated, not mere social recognition or time on the job (Ericsson &
Charness, 1994). Further the expert principals had a wide range of experiences, settings, and
problems that they address on a regular basis; and did not have experience as instructors or
trainers in the area of the study (Yates, 2007). Yates (2007) suggests that instructors or trainers
often will describe how they train a task as opposed to how they actually perform the task on the
job.
In order to select these experts, the researcher initially contacted school districts and
county offices of education for names of experts in conducting informal classroom walk-
throughs and providing feedback to teachers. However, this strategy was not successful. As an
alternative path for identifying experts, the researcher contacted a consultant who for the past 10
years has provided training, on-site coaching, and observation of principals when conducting
informal classroom walk-throughs and providing feedback to teachers. The consultant has
worked with over 50 school districts in three states and with approximately 1,200 administrators
walking through more than 5,000 classrooms. This consultant provided the researcher with a list
of nine potential SMEs, principals of kindergarten through twelfth grade schools in Southern
California with diverse student populations, for participation in the study and ultimately four
responded to an invitation from the researcher and agreed to participate in this CTA study. The
recommended participants were qualified as experts by the consultant based on: reputation; years
of experience; superior performance of the task based on the consultant’s personal observations
and experience, and a track record of student achievement based on the annual state assessments
verified on the California Department of Education (CDE) website. Table 1 provides a complete
list of the potential SMEs whom were invited to participate in the current study.
INFORMAL CLASSROOM WALK-THROUGHS 63
Table 1
Potential SMEs as Recommended by a Consultant for the Current Study
______________________________________________________________________________
Potential SME Grades Years of Experience % of ELLs % of FRLP API Ranking
SME 1 K-8 7 40.8% 83.9% 719
SME 2 K-6 13 70.0% 80.0% 835
SME 3 K-6 20 31.8% 82.2% 878
SME 4 6-8 6 4.5% 54.0% 843
SME 5 9-12 22 2.9% 30.5% 839
SME 6 6-8 20 4.5% 35.9% 865
SME 7 K-5 15 31.1% 72.4% 846
SME 8 K-6 14 15.6% 9.8% 917
SME 9 6-8 22 60.6 97.6% 642
______________________________________________________________________________
Note. All data in Table 1 is de-identified. Each SME is numbered for demonstration purposes only and numbering
does not represent any rank order or selection criteria. ELLs means English Language Learners or students who
have a dominant language other than English upon entering school in the United States and have yet to demonstrate
English Fluency as demonstrated by the California English Language Development Test (CELDT). FRLP represents
the Free and Reduced Lunch Program which provides free or reduced price school meals to students who are
considered socio-economically disadvantaged. API is the Academic Performance Index which is a scaled score,
from 200 to 1000, representing a school’s ranking as compared to all other schools in the state based on student
achievement as measured by students’ annual state test scores.
Data Collection for Question 1: What are the action and decision steps that expert principals
recall when they describe how they conduct informal classroom walkthroughs and provide
feedback to teachers?
Clark et al. (2008) describe a five-stage process to conduct Cognitive Task Analysis:
1. Collect preliminary information which builds general familiarity with the topic of the
study through document analysis, observations, and informal interviews.
INFORMAL CLASSROOM WALK-THROUGHS 64
2. The second stage identifies knowledge types used when performing the task which
requires the researcher to identify declarative and procedural knowledge and any
hierarchal relationships in the application of these knowledge types.
3. The third stage applies the knowledge elicitation techniques best suited to the study.
4. The fourth stage verifies and analyzes the data gathered through the use of qualitative
data analysis techniques.
5. A fifth and final stage formats the results into a training tool.
In this study, the five-stage process was implemented as described in the next sections.
Phase 1: Collect preliminary knowledge. Because the researcher is a former school
principal and has experience conducting informal classroom walk-throughs, a thorough literature
review was conducted to help gather preliminary information and build general familiarity on
informal classroom walk-throughs.
Phase 2: Identify knowledge types. In the process of completing the literature review
the researcher was able to develop a thorough understanding of the distinction between
declarative and procedural knowledge. To practice distinguishing between these two knowledge
types and to understand hierarchal relationship the researcher participated in practice activities
with other researchers under the guidance of a senior researcher to identify action steps, decision
steps, as well as conceptual knowledge types such as concepts, processes, and principles. These
knowledge types were used in the interview protocol.
Phase 3: Apply knowledge elicitation techniques.
Instrumentation. A semi-structured interview protocol was used to capture the
knowledge and skills from the subject matter experts using the concepts, processes and principles
INFORMAL CLASSROOM WALK-THROUGHS 65
(CPPs) technique (Clark, 2014). The semi-structured interview protocol is attached as Appendix
A.
The CPPs techniques were used to develop the semi-structured interview protocol for this
CTA. The CPPs technique uses layered interview techniques to gather the automated and often
unconscious knowledge used by experts to accomplish complex tasks (Clark, 2006). The
process begins with the researcher explaining the CTA process to the participant and asking the
participant to list stages required to accomplish the larger task being studied. The expert then
identifies five problems that an expert should be able to solve if the primary task has been
mastered, and for the final step, the researcher leads the expert through a semi-structured
interview that focus on each of the subtasks and how the expert completes the subtasks. The
semi-structured interview focuses on items such as: action steps; decisions; concepts, processes,
and principles; initiating conditions to start a subtask; equipment and materials needed; sensory
experiences needed; and performance standards required among other relevant areas (Clark et al.,
2008). The action and decision steps are considered the critical information novice need to
perform the task. Action steps begin with a verb and are statements about what a person should
do, such as “When driving a car, make a full stop at every stop sign.” Decision steps contain two
or more alternative to consider before taking an action, such as “When driving IF a traffic light is
red, THEN stop; IF the traffic light is yellow, THEN proceed only if it is safe; IF the traffic light
is green, THEN continue through the intersection.”
Interviews. Following Institutional Review Board (IRB) approval from the University of
Southern California, three of the four principals from school districts in Southern California were
asked to participate in the semi-structured interview per the protocol described above. Each
interview took approximately two hours and with prior approval of the subject matter experts,
INFORMAL CLASSROOM WALK-THROUGHS 66
the interviews were audio recorded to aid with transcription. The researcher used the semi-
structured interview protocol to capture the unconscious and non-observable action steps,
decisions, judgments, cognitive processes, and knowledge that the subject matter experts use
when they conduct informal classroom walk-throughs and provide feedback to teachers.
Phase 4: Data analysis. Audio recording the interviews of the subject matter experts
permitted verbatim transcription, using an outside service, of each interview for deep analysis of
the interview data.
Coding. Once the interviews were transcribed verbatim, a coding scheme previously
developed based Clark’s (2006) CPP method was used to code the data from the semi-structured
interviews. The coding scheme was used for inter-rater reliability and is included as part of
Appendix B.
Inter-rater reliability. Using the coding scheme described above, the transcription of one
complete interview was coded by the researcher and a fellow researcher as a second coder and
compared for inter-rater reliability. The double coding was analyzed and a standard inter-rater
reliability was calculated as a percentage of correspondence between the two coders. Hoffman,
Crandall, and Shadbolt (1998) determined that once there is an 85% or higher agreement in inter-
rater reliability, the coding process is consistent and reliable among different coders. If the inter-
rater reliability is less than 85%, Crandall et al. (2006) recommend that the coding scheme and
the function-unit categories may need to be further refined. The results of the inter-rater
reliability are presented in Chapter Four.
Subject matter expert protocol and verification. The coding of the transcriptions led to
the creation of a step-by-step protocol for each subject matter expert who was interviewed.
These individual step-by-step protocols were generated and reviewed by each subject matter
INFORMAL CLASSROOM WALK-THROUGHS 67
expert with the researcher. During the second interview the researcher asked the SME to make
any corrections, additions, or deletions to the individual protocol.
Phase 5: Formatting the results.
Gold standard protocol (GSP). The revised and corrected individual subject matter
expert protocols were aggregated in order to generate a preliminary gold standard protocol
(PGSP). The aggregation was accomplished by identifying the individual protocol that was most
complete, clear, and used the most accurate language and terminology. Then each action and
decision from each individual protocol was compared to the initial individual protocol
considered to be most complete. If the action and decision steps had the same meaning, then it
was attributed to both SMEs. If the language action or decision step was more accurate or
complete from one of the other two individual protocols, then the action or decision step was
modified and attributed to both SMEs. If it was a new step which was not listed in the most
complete individual protocol, it was added to the initial protocol in order to build the aggregated
initial gold standard protocol and attributed to the SME who had described it in their individual
protocol. See Appendix C for a description for the complete procedure for creating a GSP. The
PGSP was returned to the three subject matter experts who participated in the semi-structured
interviews and provided to the fourth subject matter expert selected as a participant in the study
but who was not interviewed as part of this CTA.
Summary. The five phase process noted above has been is called the 3i +3r method,
which stands for three initial interviews and three reviews, and is represented visually in Figure
1.
INFORMAL CLASSROOM WALK-THROUGHS 68
Figure 1.
3i + 3r CTA Method
Figure 1: Provides a visual representation of the five stages of the CTA 3i + 3r Method.
Data Analysis for Question 2: What percentage of actions and/or decision steps, when
compared to a gold standard, do expert principals omit when they describe how they conduct
informal classroom walk-throughs and provide feedback to teachers?
Spreadsheet analysis. The final stage of the data collection was completed by
transferring the action and decision steps identified in the GSP to a spreadsheet. Each subject
matter expert’s protocol, as a result of the interview, was reviewed and compared to the GSP. If
the individual protocol included an action or decision step that was also included in the GSP,
then a “1” was placed in the cell corresponding to that action or decision step. If the action of
decision step in the GSP was not in the individual subject matter expert’s protocol, then a “0”
was placed in the cell corresponding to that action or decision step. This analysis enabled the
researcher to convert the frequency counts to percentages which represented the total number of
agreements and omissions between the subject matter expert’s individual protocol and the GSP.
Researcher Conducts Semi-
structured Interviews
SME A SME B
SME C
Individual Protocol Individual Protocol Individual Protocol
Review Interview
Review Interview
Review Interview
Preliminary Gold Standard Protocol (PGSP)
SME D
Review of
PGSP
Gold Standard Protocol (GSP)
INFORMAL CLASSROOM WALK-THROUGHS 69
CHAPTER FOUR: RESULTS
Overview of Results
This study examines the declarative and procedural knowledge, which are expressed as
objectives, standards, cues, conceptual knowledge, and action and decision steps of four expert
K- 12 grade principals as a result of using CTA methods to capture their expertise. The results of
the data analysis are organized by research question.
Research Questions
Question 1
What are the action and decision steps that expert principals recall when they describe
how they conduct informal classroom walkthroughs and provide feedback to teachers?
Inter-rater reliability. As described in Chapter Three, inter-rater reliability was
determined by tallying the number of coded items that were in agreement and dividing that
number by the total number of coded items. The results are shown in Appendix B. The inter-
rater reliability was established at 97%. Given the high inter-rater reliability, this researcher
coded the remaining two SME’s interviews prior to creation of an initial individual protocol for
each SME.
Flowchart analysis. SME A’s initial individual protocol was then used to create a
flowchart which is attached as Appendix D. The flowchart was carefully analyzed and reviewed
by the researcher to ensure that SME A’s knowledge flowed logically and that there were no
decision steps without appropriate potential actions. The flowcharting process revealed
additional questions regarding the knowledge captured in SME A’s first interview which were
asked during a follow-up interview, resulting in the final individual protocol for SME A. For
example, the flowchart revealed several decision steps that did not provide two or more
alternatives and several action steps that did not make a logical progression. The process of
INFORMAL CLASSROOM WALK-THROUGHS 70
flowcharting SME A’s protocol and the subsequent round two interview, also informed the
researcher about additional questions to consider during the protocol review with SME B and
SME C.
Gold standard protocol. As described in Chapter Three, the researcher reviewed all
three SME individual protocols and aggregated the data to create an initial gold standard protocol
for conducting informal classroom walk-throughs and providing feedback to teachers. Upon
review of the three individual protocols, SME C was determined to have the most complete
individual protocol and as a result was used as the foundation for the preliminary gold standard
protocol. Upon review of the remaining two protocols SME A was determined to be more
complete than SME B. The individual action and decision steps for SME A were compared to
each action and decision step of SME C and where the meaning of the step was identical, then
SME A was also associated with describing that step. In the event there was an action and
decision step in SME A’s individual protocol that was not captured by SME C, it was added to
SME C’s foundational protocol; however, it was only associated with SME A. Following this
process, the individual protocol for SME B was also aggregated in a similar manner to the
aggregated preliminary gold standard protocol. An example of the process is shown in Figure 2.
INFORMAL CLASSROOM WALK-THROUGHS 71
Figure 2
Example of Aggregating Action and Decision Steps for the Preliminary Gold Standard Protocol
(PGSP)
Figure 2. The figure show the progression of an action step as data described by each SME from their individual
protocols is aggregated to create an action step found in the GSP.
The researcher sent an explanatory email and copy of the initial gold standard protocol to
a fourth SME, identified as SME D, to allow SME D time to review the initial gold standard
protocol before meeting with SME D in-person to capture additions, modification, and deletions
to the initial gold standard protocol.
The response to Research Question one is the final gold standard protocol, attached as
Appendix E, and represents the action steps and decision steps that expert principals use to
improve teachers’ classroom instruction through the informal observation and feedback process.
Overall there were found to be eight stages in the process of conducting informal classroom
walk-throughs and providing feedback to teachers. These eight stages are:
1. Build rapport, relationships, and trust.
SME C –Action Step:
Communicate to the staff the
overarching focus areas of the
informal walkthroughs for the
school year. (C)
SME A – Additions (in Bold)
to SME C’s Action Step:
At the first staff meeting at
the beginning of the school
year, tell the staff what your
expectations (Premier
Instruction, overarching
focus areas) are when you
walk-through the classroom
and provide them with these
expectations in writing from
step 1.1. (A, C)
SME B – Additions
(underlined) to PGSP (As
Step reads in final GSP):
At the first staff meeting at
the beginning of the school
year, tell the entire teaching
staff what your expectations
(Premier Instruction, OWL,
non-negotiables, overarching
focus areas, or other
instructional practices) are
when you walk-through the
classroom and provide them
with these expectations in
writing from Step 1.2. (A, B,
C)
INFORMAL CLASSROOM WALK-THROUGHS 72
2. Set clear expectations.
3. Plan for the walk-throughs.
4. Observe the teacher in the classroom.
5. Provide feedback to individual teachers.
6. Provide feedback to the entire teaching staff.
7. Communicate with the Leadership Team and plan next steps.
8. Facilitate personal reflection.
The disaggregated results are described in the following sections.
Recalled action and decision steps. A behavior that is observable is an action step.
Unobservable cognitive processes are represented as decision steps that are critical to performing
tasks and serve as cues or prompts for the SME to evaluate, interpret, analyze, and decide among
alternatives. The action and decision steps captured from the experts interviewed comprise the
critical information novices need to replicate expert performance. As such, the researcher
conducted an analysis of each SME’s action and decision steps to determine the number of steps
recalled by each.
To conduct this analysis, the researcher listed each action and decision step in the final
gold standard protocol in its own row of a Microsoft Excel spreadsheet. The number of action
and decision steps was determined through frequency counts. In the first column on the
spreadsheet, each step from the gold standard protocol was coded with “A” for action step or
“D” for decision step. Some steps were not action or decision steps, for example standards in the
gold standard protocol were labeled “S” for “Standard” but had no numerical value as part of the
data analysis. An example of a standard is Step 2.5.5, “All professional development is aligned
with the Board’s goals and research-based.” Each SME was assigned a letter, “SME A,” SME
INFORMAL CLASSROOM WALK-THROUGHS 73
B,” or “SME C” for identification purposes based on the order in which the SMEs were
interviewed, and a column on the spreadsheet was used to track the inclusion of action and
decision steps. Action and decision steps that were included in the individual SMEs protocol and
the gold standard protocol were marked with a “1.” The number of actions and decisions for
each SME were totaled at the bottom of the SME’s column. The spreadsheet analysis is attached
as Appendix F. Table 2 provides a total of each SME’s action and decision steps.
Table 2
Cumulative Action and Decision Steps Captured for Each SME in the Initial Individual
Protocols.
______________________________________________________________________________
Steps
Action Steps Decision Steps Total Steps
SME A 35 47 82
SME B 49 35 84
SME C 58 42 100
Action and decision steps contributed by each SME. Table 2 reports action and
decision steps recalled by each SME. The action and decision steps in Table 2 which were
elicited through CTA may not be solely attributed to one SME, as a result the action and decision
steps reported in Table 2 when added together do not equal the total number of action and
decision steps in the gold standard protocol reported in Table 2. In other words, SMEs in many
cases provided the same action or decision steps through the CTA process.
INFORMAL CLASSROOM WALK-THROUGHS 74
Figure 3
Number of Action Steps, Decision Steps, and Action and Decision Steps for SME A, SME B, and
SME C Captured through CTA.
Figure 3. Total non-repeating action and decision steps from the CTA process represented in the gold standard
protocol: action and decision steps – 196; action steps – 105; decision steps 91.
The SMEs collectively described a total of 197 action and decision steps, however no
individual SME described more than 100 action and decision steps and the percentage of total
recalled action and decision steps across all SMEs was between 41.84% and 51.02%. Only one
of the three SMEs recalled more decision steps than actions steps, SME A recalled 35 action
steps and 47 decision steps, a difference of 18.32 %. While SMEs B and C recalled more action
steps than decision steps, SME B recalled 49 action steps and 35 decision steps, a difference of
8.21%; and SME C recalled 58 action steps and 42 decision steps, a difference of 9.09%. The
range of the percentages of action steps identified between each SME was nearly 22%. The least
percentage identified was 33.33% versus 55.24% for the most number identified; and the range
35
49
58
47
35
42
82
84
100
0
20
40
60
80
100
120
SME A SME B SME C
Action Steps
Decision Steps
Total
INFORMAL CLASSROOM WALK-THROUGHS 75
of decision steps identified was 13%, between 38.46% for the least number identified and
51.65% for the most.
Action and decision steps captured in the follow-up interviews. As an additional
analysis, the researcher wanted to know how many action and decision steps were added,
modified, or deleted during the follow–up interview with each SME. The results of this analysis
are shown in Table 3.
Table 3
Additional Expert Knowledge Captured, in Action and Decision Steps, During Follow-up
Interviews
_____________________________________________________________________________
Additional Steps Captured
SME Action Decision
A 3 7
B 2 1
C 31 12
D 0 2
Note: SME D did not participate in the CTA semi-structured interviews and only reviewed the initial gold standard
protocol for additions, modifications, and deletions after it was reviewed by the other 3 SMEs.
In all cases, when the initial SMEs reviewed their individual protocols and participated in
a follow-up in-person interview, the process resulted in increased action and decision steps.
Alignment of SMEs in describing the same action and decision steps. The
spreadsheet analysis was also used to determine the number and percentage of action and
decision steps described by each SME that were highly aligned, partially aligned, or slightly
aligned. For each action and decision step, if the step was only included by one SME, it was
identified as being “slightly aligned” then the number “1” was added in a separate column. If an
INFORMAL CLASSROOM WALK-THROUGHS 76
action or decision step was described by two of the three SMEs, then the number was “2” was
added in the column indicating that the step was “partially aligned.” If an action or decision step
was described by all three SMEs the number was “3” was added in the column indicating the
step was “highly aligned.” Table 4 shows the results of this analysis.
Table 4
Number and Percentage of Action and Decision Steps that are Highly Aligned, Partially Aligned,
and Slightly Aligned
_____________________________________________________________________________________
Number Percentage
Highly Aligned 15 7.69%
Partially Aligned 44 22.56%
Slightly Aligned 136 69.74%
_____________________________________________________________________________
Collectively the SMEs were “highly aligned” on 15 total or 7.69%, “partially aligned” on
44 total or 22.56%, and “slightly aligned” on 136 or 69.74% of action and decision steps. The
implications of these differences are discussed in Chapter 5.
Question 2
What percentage of actions and/or decision steps, when compared to a gold standard, do
expert principals omit when they describe how they conduct informal classroom walk-throughs
and provide feedback to teachers?
Total knowledge omissions. To answer Question 2, the Microsoft Excel spreadsheet for
analysis of the percentage of action and decision step captured was also used to determine the
number of action and decision steps omitted by the individual SMEs when describing the
informal classroom walk-through process and providing teachers feedback. Action and decision
steps which were included in the gold standard protocol but omitted by the SME were marked
INFORMAL CLASSROOM WALK-THROUGHS 77
“0.” The total number of action and decision steps omitted was added and divided by the total
number of cumulative action and decision steps for all SMEs in the gold standard protocol which
produced a percentage of knowledge omissions for action and decision steps and total steps.
As shown in Table 5, SMEs omitted a significant number of action and decision steps. Between
48.98% and 58.16% of total action and decision steps were omitted. Table 5 provides a
comparison of action and decision steps omitted by each SME when compared to the gold
standard protocol including the range and standard deviation.
Table 5
Total Action and Decision Steps, or Expert Knowledge, Omissions by SME when Compared to
the Gold Standard Protocol
______________________________________________________________________________
Steps Omitted
Total Action & Action Decision
Decision Steps Steps Steps
Omitted % Omitted % Omitted %
SME A 114 58.16% 70 66.67% 44 48.35%
SME B 112 57.14% 56 53.33% 56 61.54%
SME C 96 48.94% 47 44.76% 49 53.85%
Mean
Omissions 107.34 54.76% 57.67 54.92% 49.67 54.58%
Range 18 23 12
SD 8.06 9.46 4.92
Note. Total non-repeating action and decision steps from the CTA process represented in the gold standard
protocol: action and decision steps – 196; action steps – 105; decision steps 91.
Across all SMEs, the percentage of expert omissions when describing how to conduct
informal classroom walk-throughs and provide feedback to teachers was on average: 107.34
average total action and decision steps omitted, or 54.76% (SD ± 8.06); 57.67 average total
INFORMAL CLASSROOM WALK-THROUGHS 78
action steps omitted, or 54.92% (SD ± 9.46); and 49.67 average total decision steps omitted, or
54.58% (SD ± 4.92).
Individually, the percentage of both action and decision step omissions, or expert
knowledge omissions, varied moderately among the SMEs when compared to the gold standard
protocol, from 48.94% to 58.16%. However, the percentage of action and decision steps when
compared individually to the gold standard protocol varied significantly among the SMEs from:
44.76% to 66.67% for action steps; and 48.35% to 61.54% for decision steps.
Analysis of action and decision step omissions. Figure 4 represents the action and
decision step omissions, or expert knowledge omissions, data for SME A, SME B, and SME C
when compared to the cumulative gold standard protocol.
Figure 4
Total SME Knowledge Omissions when Compared to the Gold Standard Protocol
Figure 4. Total non-repeating action and decision steps from the CTA process represented in the gold standard
protocol: action and decision steps – 196; action steps – 105; decision steps 91.
70
56
47
44
56
49
114
112
96
0
20
40
60
80
100
120
SME A SME B SME C
Omitted Action Steps
Omitted Decision Steps
Combined Omitted Action and
Decision Steps
INFORMAL CLASSROOM WALK-THROUGHS 79
The next chapter will include an overview of the study, a discussion of the findings,
limitations, implications, and future research.
INFORMAL CLASSROOM WALK-THROUGHS 80
CHAPTER FIVE: DISCUSSION
Overview of Study
The primary purpose of this study was to capture the knowledge and skills expert
kindergarten through twelfth grade principals use to conduct informal classroom walk-throughs
and provide feedback to teachers. This study also sought to identify the critical knowledge and
skills omissions by expert principals when they describe the process of conducting informal
classroom walk-through observations and providing teachers feedback when compared to a gold
standard protocol. Research has shown that when experts communicate the knowledge and skills
they use to conduct complex tasks, they may omit up to 70% of the critical information novices
need to replicate the activity though experts can perform these complex tasks automatically and
without conscious thought (Clark, 2014; Clark et al., 2008; Clark & Feldon, 2006). Automaticity
is a key characteristic and consequence of expertise, and has been referred to as a “double-edged
sword” (Feldon, 2007).
Consultants, who are deemed experts in the field, often train principals in the complex
task of informal classroom walk-throughs and providing feedback to teachers. Although these
consultants had the advantage of expertise acquired over time, unfortunately, these consultants
may be disadvantaged by the 70% rule as a result of the effects of expertise and automaticity
when recalling critical knowledge and skills. Nonetheless, training principals to conduct this
complex task with expert skill has been demonstrated as an ongoing need (Donaldson, 2009;
Hallinger, 2007; Ho and Kane, 2013; May et al., 2012). Although CTA has been used
effectively to elicit and capture expert knowledge across a number of disciplines (Clark, 2014),
this is the first study to use CTA to elicit and capture expert principals’ knowledge and skills as
they exercise instructional leadership when conducting informal classroom walk-throughs and
providing feedback to teachers. The expert knowledge and skills captured through this CTA
INFORMAL CLASSROOM WALK-THROUGHS 81
study may provide valuable input to help design more effective professional development
programs for both aspiring and in-service principals than those currently provided, which are
often based on behavioral task-analysis and experts’ self-reports (Clark et al., 2008; Feldon &
Yates, 2011).
Process of Conducting Cognitive Task Analysis
Selection of Experts
The literature related to the selection of experts from whom knowledge and skills are
captured through CTA recommends the use of three to five experts before diminishing marginal
returns set-in. The investment of time and effort in conducting a CTA study with more than three
to four experts yields less than 10% additional action and decision steps (Bartholio, 2010; Chao
& Salvendy, 1994). Replicating the methods of Chao and Salvendy, Crispen (2010) also found
that it takes three to five experts to yield an acceptable amount of expert knowledge to create a
gold standard protocol that can be used for training purposes, thus reversing the 70% rule (Clark
& Feldon, 2006). In addition to the number of experts, the identification of experts that were
selected for this CTA study was another critical element.
The selection criteria for a subject matter experts in a CTA study includes four key
criteria identified in the literature as: (a) three to five years of consistent success in performing
the task to be studied; (b) recent expert performance of the task as recognized by colleagues in
the field; (c) history of expert performance of the task in a wide variety of settings; and (d) not
having provided instruction to others on the performance of the task within the past year or
longer (Clark, 2014; Clark et al., 2008; Flynn, 2012). The SMEs selected for this study met the
criteria as described in the CTA literature above, however identifying the SMEs proved
challenging.
INFORMAL CLASSROOM WALK-THROUGHS 82
There are at least 18 different documented models for informal classroom walk-throughs
(Kachur et al., 2010), however, there is no standardized “best practice” or evaluation tool that
scores or rates principals to determine their effectiveness when they conduct classroom walk-
throughs and provide feedback to teachers. As such, the researcher did not have any success in
contacting school districts or county offices of education for names of experts in performing this
task. As an alternative path for identifying experts, the researcher contacted a consultant who for
the past 10 years has provided on-site coaching and observation of principals when conducting
informal classroom walk-throughs and providing feedback to teachers. The consultant has
worked with over 50 school districts in three states and with approximately 1,200 administrators
walking through more than 5,000 classrooms. This consultant provided the researcher with a list
of eight potential SMEs for participation in the study and ultimately four responded to an
invitation from the researcher and agreed to participate in this CTA study. The SMEs who
responded represent schools with students enrolled from kindergarten through twelfth grade,
with diverse student populations, and with a track record of student achievement based on the
annual state assessments, which are posted on the state’s department of education website.
Identifying experts and the characteristics of expertise for teachers and principals is
difficult. The difficulty lies in not only the characteristics of expertise but also in how expertise
is developed in teaching (Berliner, 1986, 1988; Bullough & Baughman, 1995; Smith & Strahan,
2004; Sternberg & Horvath, 1995). Studies of teaching expertise have focused on comparisons
of novice teachers with “expert” teachers, on teaching expertise based on experimental or
simulated tasks as opposed to real classrooms, on equating experience with expertise, on
checklists or tables of expert behavior and rating expertise based on the number of identified
“characteristics,” and on student assessment outcomes (Smith & Strahan, 2001). However, in
INFORMAL CLASSROOM WALK-THROUGHS 83
the end these approaches have done little to help determine expertise in teaching (Sternberg &
Horvath, 1995) and the complexities related to “expert” school principals have not been
addressed in the research at this point. As seen in the section that follows, the difficulty of
identifying expertise and how to develop it has been an issue for other professions as well.
For example, determining expertise in psychotherapy has proven “elusive” due to the
unpredictable nature of the outcomes and the limited amount of quality feedback provided to
psychotherapists. Expertise for psychotherapists, much like teachers and school principals, is
based on reputation, performance, and client outcomes (Tracey, Wampold, Lichtenberg, &
Goodyear, 2014). Tracey et al. explain measuring expertise through reputation is based on peer
nomination or degree attainment; performance is difficult to define and quantify in order to
distinguish “expertise from competence;” and client outcomes are not only dependent upon the
psychotherapist’s skills but also the patient’s willingness and motivation to participate in
treatment. “Instruction is the instructor’s manipulation of the learner’s environment in order to
foster learning” (Mayer, 2011, p. 53). Using this definition, psychotherapists, teachers, and
school principals all provide instruction with the goal of changing a learner’s environment to
foster learning but the challenge remains identifying expertise during this instructional process.
Future studies may consider the suggestion by Tracey et al. (2014) for an approach to
identify expertise in psychotherapy which could similarly be used for the identification of
expertise in teachers and school principals described as the “disconfirmatory approach.” In this
approach expertise is identified by determining which knowledge, skills, and characteristics
would be present if one were an expert and seeking disconfirming information to determine the
expert status of a practitioner (Tracey et al., 2014).
INFORMAL CLASSROOM WALK-THROUGHS 84
Collection of Data
The data were collected for this study by conducting semi-structured interviews with
three of the four SMEs who participated in this study. The interview method is referred to as 3i
+ 3r in the research literature which stands for three independent interviews and three reviews of
the data collected. The 3i + 3r method has been shown as an effective way to capture expert
knowledge to inform instructional design and the development of related instructional materials
(Clark et al., 2008; Zepeda-McZeal, 2014). The initial and follow-up interviews with the three
SMEs took over two hours each to complete, resulting in each SME dedicating at least four
hours of time and mental effort to this CTA process. Despite the commitment of time and
mental effort, each SME fully participated in the process. The time and mental effort expended
by the SMEs refutes the recommendations of Canillas (2010) who calls for using a separate
group of SMEs for review of the initial protocol because the initial group of interviewed SMEs
in her study demonstrated difficulty committing the time and sustained mental effort necessary to
review the initial individual protocols. Additionally, the methodology of this CTA study calls
for the researcher to serve as a knowledge analyst who must code, organize, and interpret the
data captured from the SME. Therefore, the researcher must review each step individually in the
initial individual protocol with each SME to ensure that the expert knowledge and skills are
accurately represented,, thereby making the recommendations of Canillas (2010) further
inappropriate for this CTA study and other CTA studies using similar methodology.
During the semi-structured interview process, the SMEs began by identifying the main or
primary procedures or steps of an informal classroom walk-through and providing teachers
feedback. This was followed by the SMEs then providing the individual steps within each main
or primary procedure. Each of the three SMEs began by explaining how they walk into the
INFORMAL CLASSROOM WALK-THROUGHS 85
classroom through the door and begin to conduct an informal classroom walk-through.
Individually, the SMEs appeared to recall the steps in the process of conducting an informal
classroom walk-through and providing teachers feedback as if they were actually in the
classroom at that moment. During the CTA interviews, each SME stopped at one point during
the interview due to the realization that the knowledge and skills they were describing didn’t
actually occur at that time but in an earlier previously unidentified main or primary procedure or
step that the SME now needed to add. A possible explanation of this may be found in
Anderson’s (1996b) model for the acquisition of automated knowledge. It may be that when the
SMEs recalled knowledge and skills which were affected by their automated and unconscious
knowledge (the third stage of automaticity), the process triggered the less automated and more
conscious knowledge recall (associative or knowledge compilation stage). The associative or
knowledge compilation stage of automaticity enabled the SMEs to correct procedural errors by
applying declarative knowledge to procedural tasks (Anderson, 1996b).
Another possible explanation may be found in the method for conducting the CTA. A
CTA study by Crandall and Getchell-Reiter (1993) captured the expert knowledge and skills
neonatal intensive care unit (NICU) nurses used to identify and diagnose early sepsis in
newborns. The NICU nurses’ expert knowledge and skills were captured by “being in the
moment” through the use of the Critical Decision Method (CDM). The CDM is a CTA method
in which experts recall, often an extended period of time after a critical event, the critical
decisions that they made during incidents which usually involve life or property (Klein,
Calderwood, & MacGregor, 1989). The NICU nurses were able to recall a greater number of
critical sepsis assessment techniques than were being used in the current NICU nurse pre-service
sepsis assessment training (Crandall & Getchell-Reiter, 1993). Due to the NICU nurses placing
INFORMAL CLASSROOM WALK-THROUGHS 86
themselves “in the moment” of providing care to a newborn suspected of having sepsis,
additional knowledge and skills were captured by asking expert nurses to focus on goals, cues,
contextual conditions, and place. It is possible that this same process occurred during the CTA
interview process in the current study, which also incorporates elements of CDM.
Discussion of Findings
No formal hypotheses were developed for this research study. Rather the study was
guided by two main research questions.
Question 1
What are the action and decision steps that expert principals recall when they describe
how they conduct informal classroom walkthroughs and provide feedback to teachers?
Two of the three experts in this study described a greater number of action steps than
decision steps. The recall of a greater number of action steps over decision steps indicates the
impact of expertise upon knowledge recall. Expert knowledge becomes automated through
repeated deliberate practice of complex tasks. Automaticity frees up experts’ working memory
to address novel tasks by transferring knowledge into the long-term memory making the
execution of complex tasks unconscious but the capturing of critical action and decision steps
difficult (Clark, 2014; Clark & Estes, 1996; Ericsson, 2004).
Action steps versus decision steps. As indicated in Chapter Four: Results, two SMEs
(SME B and SME C) described 14 and 16 more action steps than decision steps. The tendency
of experts to describe a greater number of actions than decision steps has been noted in other
CTA studies (Canillas, 2010; Crispen, 2010; Embry, 2012; Tolano-Leveque, 2010; Zepeda-
McZeal, 2014). However, SME A described 12 more decision steps than action steps. This was
an unexpected result, so the researcher reviewed the interview transcript, recorded interview, and
INFORMAL CLASSROOM WALK-THROUGHS 87
data collected and organized into the initial individual protocol for SME A. One possible
explanation for the higher number of decision steps than action steps for SME A is the seemingly
more directive or “top-down” leadership style of SME A. In a research study by Harris (2004),
the researcher examined the impact of distributed leadership upon school improvement and
determined that not only with distributive leadership, but all forms of leadership, a principal
takes either a “top-down” or “bottom-up” approach. Harris (2004) describes examples of
successful “top-down” and “bottom-up” leadership which are dependent upon the leader, the
school staff, and the specific situational context of the school. In this current study SME A was
responsible for the initiation, implementation, and expectations related to conducting informal
classroom walk-throughs and providing feedback to teachers. Due to this “top-down” approach
to the informal classroom walk-through process there are fewer action steps and/or input
required from teachers in the process. The task for SME A involves making decisions on what
was observed and whether or not to provide feedback and in what form. In contrast, SME B and
SME C demonstrate a more “bottom-up” approach to leadership that involves the teachers in the
development of the implementation and expectations for informal classroom walk-throughs both
initially and throughout the process which results in a greater number of action steps to involve
the staff and develop consensus.
Additionally, SMEs B and C recalled more action than decision steps, indicating that
subject matter experts have greater difficulty recalling the unobservable decisions made and find
it easier to recall the observable and thereby more tangible tasks they perform. Clark (2014), in
reviewing CTA studies related to healthcare, found that experts tend to recall more action steps
because they form a “mental image” which is more easily recalled than decision steps which do
not to create the same “mental images” as action steps.
INFORMAL CLASSROOM WALK-THROUGHS 88
As demonstrated in Table 2 in Chapter Four: Results, the total number of action and
decision steps described by each SME increased with each interview (SME A - 82, SME B - 84,
and SME C - 100). An explanation for this increase in the total number of action and decision
steps may be the increase in interview skills of this researcher. By the third SME interview, this
researcher had a higher degree of confidence with the CTA interview process and had a more
intuitive sense of what type of follow-up questions to ask and when. The influences of practice
upon the development of this researcher’s expertise as a CTA researcher were being
demonstrated through the process of conducting this CTA study.
Action and decision steps captured during the review of the initial individual
protocols and preliminary gold standard protocol. As discussed in Chapter Four: Results,
each SME described additional action and decision steps which were integrated into their initial
individual protocols, as well as, additional action and decision steps which were described by the
fourth SME who reviewed the preliminary gold standard protocol. After the researcher had
analyzed the interview transcripts for each SME, the data were used to create an initial individual
protocol for each SME. This initial individual protocol was then reviewed in an in-person
meeting with each SME in which each action and decision step was reviewed individually in
order and the SME was asked to make additions, modifications, and deletions while the
researcher took notes. Based on the research of Zepeda-McZeal (2014), the reviews of the initial
individual protocols were done in-person because Zepeda-McZeal found the in-person review
increased the number of additional action and decision steps described by the SMEs due to the
increased communication from the personal face-to-face interaction. The three SMEs that
participated in the initial CTA process added between 3 to 31 additional action steps and 1 to 12
additional decision steps.
INFORMAL CLASSROOM WALK-THROUGHS 89
SME C added 31 additional action steps and 12 additional decision steps when reviewing
the initial individual protocol; more additions than all of the other SMEs combined. This SME
was one month from retirement at the time of the CTA interview and review, and one possible
explanation of the number of additional action and decision steps may be the proximity of the
CTA process to the retirement date for this SME. SME C, being near retirement, may have
exerted greater mental effort to retrieve automated knowledge and skills from the long term
memory in order to leave something behind for future principals who will need to learn the skill
of informal classroom walk-throughs and providing feedback to teachers. Erikson (1987) and
Erikson, Erikson, and Kivnick (1986) referred to eight stages of psychosocial development,
which a person experiences during a lifetime. The seventh stage of psychosocial development is
experienced during mid-life and is defined as generativity versus stagnation, or contributing
something meaningful that will provide guidance and continuity to the next generation versus
being unable to contribute to the next generation due to selfish pursuits or limited productivity
(Erikson, 1987; Erikson et al., 1986). For SME C the development of the individual protocol
may have served as a “generativity script” that enabled SME C to make meaning of a long
educational career and construct a product to help future generations of educators and
specifically school administrators (McAdams, de St. Aubin, & Logan, 1993). Future research
could identify if the amount of expert knowledge captured from SMEs nearing retirement, or in
Erikson’s generativity versus stagnation stage of psychosocial development, is greater than that
for experts who are still in the midst of their career and an earlier stage of Erikson’s psychosocial
development.
A fourth SME, SME D, was asked to review the preliminary gold standard generated
from the protocols of the previous three SMEs. SME D had not participated in any of the earlier
INFORMAL CLASSROOM WALK-THROUGHS 90
stages of the CTA process, so SME D was the only SME who was emailed the initial gold
standard protocol in advance to allow time to review protocol. SME D was asked in the email to
make additions, modifications, and deletions to the initial gold standard protocol in advance prior
to the in-person meeting with the researcher. At the onset of the in-person meeting to review the
initial gold standard protocol SME D said, “I have had over 60 hours of training on informal
walk-throughs and have conducted hundreds of classroom walk-throughs on my own, but I have
never seen such a complete and detailed explanation of the process.” As indicated in Table 3 in
Chapter Four: Results, SME D contributed two additional decision steps to the gold standard
protocol. It may be that the extensive number of steps generated by the previous three SMEs
caused SME D to experience cognitive overload (Sweller, 1988). Due to the mental effort
involved in reviewing the initial gold standard protocol and not having participated in the CTA
semi-structured interview process, it was likely difficult for SME D to commit the sustained
mental effort required to both fully capture the depth of expert knowledge and then make
substantial additions, modifications, and deletions to the initial gold standard protocol without
the advantage of having participated in the entire CTA process.
Use of multiple experts. As previously noted in the literature (Bartholio, 2010; Crispen,
2010; Chao & Salvendy, 1994), it is recommended that multiple experts be used to capture a
more accurate representation of the action and decision steps required to perform a task. In this
study, as the data shows only 7.69% of action and decision steps were identified by all three
SMEs, thus being “highly aligned;” 22.56 % were in “partial alignment,” indicating that two
SMEs described the action or decision step; and 69.74% were only “slightly aligned,” meaning
only one SME included that action or decision step. These data shows the importance of using
more than one expert to capture expertise. As such, current methods of using one expert to teach
INFORMAL CLASSROOM WALK-THROUGHS 91
aspiring and current administrators the instructional leadership practice of conducting informal
classroom walk-throughs and providing feedback to teacher is certain to omit crucial expert
knowledge that school administrators will need to perform this task. Instruction based on of
results CTA will help administrators who are new to this task to be more successful.
Question 2
What percentage of actions and/or decision steps, when compared to a gold standard, do
expert principals omit when they describe how they conduct informal classroom walk-throughs
and provide feedback to teachers?
Expert knowledge omissions. The final gold standard protocol was compared to each
SME’s individual protocol to determine expert knowledge and skills omissions, derived from
omissions of action and decision steps, for the task of informal classroom walk-throughs and
providing feedback to teachers. Experts, who are impacted by the effects of automaticity, may
omit up to 70% of critical action and decision steps when asked to describe complex tasks in
their area of expertise (Feldon, 2004; Clark & Feldon, 2006).
When SMEs were asked to describe the critical action and decision steps for informal
walk-throughs and providing feedback to teachers, on average SMEs omitted 54.76%, ranging
between 48.98% and 58.16%, of action and decision steps as compared to the gold standard
protocol. SMEs on average omitted 54.2% of action steps, ranging between 44.76% and
66.67%; and omitted 54.58% of decision steps, ranging between 48.35% and 61.54%. The total
aggregate number of captured action steps was greater than the total aggregate number of
captured decision steps, however the difference between the average percentage of total omitted
action and decision steps was only .34%, indicating no significant difference in the omission of
action and decision steps. These findings are contrary to the findings of Canillas (2010) and
INFORMAL CLASSROOM WALK-THROUGHS 92
Tolano-Levenque (2010) who found greater omissions in decision steps than action steps in their
study of medical procedures. As Hoffman (1987) noted, it may be that the nature of the task
itself, in particular, the complexity and length of time required to perform the task may influence
the number of action and decision steps captured with CTA. Zepeda-McZeal (2014) also
observed that the complexity of the task and time required to complete the task may influence the
CTA knowledge elicitation techniques which used to conduct a CTA study.
Limitations
The present study produced findings which are consistent with findings of previous CTA
research studies related to expert knowledge captured in the form of action and decision steps
and expert knowledge omissions. The next section will discuss the limitations of the present
study.
Confirmation Bias
The first limitation of this present study is the researcher has 12 years of experience as a
school principal and district human resources administrator. This background and experience
required the researcher to be cognizant of researcher bias when conducting this CTA research
study. When a CTA analyst has experience in a task domain, the analyst has a natural tendency
to edit the knowledge captured from SMEs to align with the analyst’s own experiences (Clark,
2014). The knowledge analyst had knowledge of the task domain and participated in minimal
bootstrapping (Schraagen et al., 2000), however extra effort was required by the researcher to not
place his own preexisting expectations and experiences onto the data collected by constantly
being mindful of not placing the researcher’s own background and experiences onto the data
collected from each SME.
INFORMAL CLASSROOM WALK-THROUGHS 93
Internal Validity
The second limitation of this present study is the results of this CTA have not been
validated against the actual practice of the SMEs. To ensure internal validity, the gold standard
protocol developed for informal classroom walk-throughs and providing feedback to teachers
would need to be validated against the actual practice of the SMEs when they perform this task.
The expert knowledge and skills would be validated through direct observation as they were
performed by the SMEs, ensuring internal validity of the data captured in the gold standard
protocol. The scope of this study did not include the validation of the gold standard protocol by
observed, however this would be an appropriate study for future research.
External Validity
Another limitation of the present study is the threat to generalizability due to the limited
sample size of four SMEs and the domain specificity of kindergarten through twelfth grade
principals in the Southern California area. The principals interviewed for this study included two
elementary principals, one middle school principal, and one high school principal with a range of
students identified as English Language Learners and who participated in the Free and Reduced
Lunch Program. Further CTA studies in the same domain which include a greater number and
variety of SMEs would help to increase the generalizability and external validity of the current
study.
The final limitation of the present study that affects generalizability is the reliance on one
consultant’s opinion for the recommendation of experts. Generalizability could have been
increased by identifying experts from a greater number of sources.
INFORMAL CLASSROOM WALK-THROUGHS 94
Implications
Across a variety of domains CTA has been demonstrated to be an effective means of
capturing expert declarative and procedural knowledge required for the performance of complex
tasks (Canillas, 2010; Embry, 2012; Tolano-Leveque, 2010; Zepeda-McZeal, 2014). The
declarative and procedural knowledge captured from CTA research when applied to training and
instruction increases novice performance and decreases the amount of time and expense required
for training. Much of the training for novice and in-service principals in complex tasks such as
classroom walk-throughs and providing feedback to teachers is conducted by consultants,
serving as experts, who may omit up to 70% of the critical information novices need to replicate
complex tasks and activities (Clark, 2014; Clark et al., 2008; Clark & Feldon, 2006). The
current study supports the use of CTA research to capture expert knowledge and skills in
complex instructional leadership tasks, such as informal classroom walk-throughs and providing
feedback to teachers, for not only training and instruction but to ultimately increase student
achievement.
Future Research
The use of CTA research and CTA-based instruction for pre-service and in-service
training of principals is unknown and a search of the research did not result in any studies in this
area. As a result of this study, future research may consider taking the gold standard protocol
generated by this research and implementing a randomized experimental design study with
principals who are novices to informal classroom walk-throughs and providing feedback to
teachers. The randomized experimental study would analyze learning gains from using CTA-
guided instruction as compared to traditional instructional methods. Longitudinal research may
INFORMAL CLASSROOM WALK-THROUGHS 95
also provide useful data regarding the short-term and long-term learning gains resulting from
expert implementation of informal classroom walk-throughs and providing feedback to teachers.
Based upon the findings of this study and the findings of Zepeda-McZeal (2014), future
research is needed to determine which CTA methods result in a fewer omissions of decision
steps than action steps in complex and time intensive K-12 instructionally related tasks. Yates
(2007) and Yates and Feldon (2011) suggested different CTA methods may be appropriate to
elicit different knowledge types. Other CTA based studies in non K-12 educational task
domains, such as Canillas (2010) and Tolano-Leveque (2010) in medical procedures, have
shown the opposite tendency as experts in that domain omit fewer actions steps than decision
steps. Additional studies related to K-12 instructional practices will be required to determine the
overall effectiveness of CTA for capturing key action and decision steps in complex and time
intensive instructional tasks.
Identifying SMEs to participate in CTA studies on instructional and instructional
leadership tasks is difficult due to the lack of standardized non-biased assessments or metrics.
The SMEs were selected to participate in this CTA study based on their professional reputation,
their performance as rated by colleagues in their domain, and student assessment results. Tracy
et al. (2014), in looking for ways to determine expertise in psychotherapy, called for the use of a
“disconfirmatory approach;” which determines expertise by identifying knowledge, skills, and
characteristics which would be present if one were an expert and then seeks disconfirming
information to determine the expert status of a practitioner. Future research may examine the use
of a “disconfirmatory approach” to identify SMEs for CTA studies.
The data collected as part of this study may have been influenced by the leadership styles
of the SMEs who participated in this CTA research. Specifically, a more “top-down” orientation
INFORMAL CLASSROOM WALK-THROUGHS 96
towards leadership may have resulted in a SME describing more decision steps than action steps,
whereas a more “bottom-up” leadership style may have led a SME to describe a greater number
of action steps than decision steps. If feasible, future research which examines the impact of an
individual’s leadership style upon the capturing of expert knowledge and skills would be needed
to explore this area.
Additionally, it may be that an individual’s stage of Psychosocial Development as
defined by Erikson (1987) and Erikson et al. (1986) affect the amount of expert knowledge and
skills which are captured using CTA methodology. Specifically, in the present study Erikson’s
Generativity versus Stagnation stage of Psychosocial Development may have resulted in the
capturing of a greater amount of expert knowledge and skills from one specific SME who may
have been negotiating the Psychosocial Development Stage. However, further research would be
needed to establish a relationship between a SME’s Psychosocial Development Stage and the
amount of expert knowledge and skills described by the SME.
Conclusion
The purpose of this study was to add to the body of research about the benefits of CTA
for capturing more complete descriptions of the knowledge and skills that experts use when
solving difficult problems and performing complex tasks and the omissions experts make when
describing their knowledge and skills. The complex task of capturing the expert knowledge and
skills that K-12 principals use when performing informal classroom walk-throughs and providing
feedback to teachers is the first of its kind, however there are other similar studies that examine
the knowledge and skills captured and omitted by experts through CTA methods. As found in
other similar studies, expert principals in this study omitted nearly 70% of the critical action and
decision steps when describing how to perform informal classroom walk-throughs and provide
INFORMAL CLASSROOM WALK-THROUGHS 97
feedback to teachers. However, the result of this study show that CTA methods can be useful to
capture the unconscious, automated knowledge of expert principals when they perform the
complex task of informal classroom walk-throughs and provide feedback to teachers.
Meta-analyses conducted by Marzano et al. (2005) have clearly demonstrated the
correlation between effective instructional leadership practices by a school principal and up to a
10% increase in student achievement gains. The expert knowledge captured and accumulated
into a gold standard protocol in this study may be used to train pre-service and in-service
principals in performing the complex instructional leadership task of informal walk-throughs and
providing feedback to teachers, which may ultimately improve teachers’ classroom instruction
and student achievement.
INFORMAL CLASSROOM WALK-THROUGHS 98
References
Adams, J. E. (2010). Introduction. In J. E. Adams (Ed.). Smart money: Using educational
resources to accomplish ambitious learning goals (pp. 1-26). Cambridge: Harvard
Education Press.
Alexander, P.A. (2003). The development of expertise: The journey from acclimation to
proficiency. Educational Researcher, 32(8), 10-14.
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How
learning works, 7 research-based principles for smart teaching. San Francisco, CA:
Jossey-Bass.
Anderson, J.R. (1982). Acquisition of cognitive skill. Psychological Review, 89(4), 369-406.
Anderson, J. R. (1993). Problem solving and learning. American Psychologist, 48(1), 35-44.
Anderson, J.R. (1996a). ACT: A simple theory of complex cognition. American Psychologist,
51(4), 355-365.
Anderson, J. R. (1996b). The Architecture of Cognition. Mahwah, NJ: Lawrence Erlbaum
Associates.
Anderson, J.R., & Fincham, J.M. (1994). Acquisition of procedural skills from examples.
Journal of experimental psychology: Learning, memory, and cognition 20(6), 1322-1340.
Anderson, J. R. & Schunn, C.D. (2000). Implications of the ACT-R learning theory: No magic
bullets. In R. Glaser (Ed.), Advances in instructional psychology (Vol. 5). Mahwah,
NJ: Erlbaum.
Anderson, L.W., & Krathwohl (Eds.). (2001). A Taxonomy for learning, teaching, and
assessing: A revision of Bloom's taxonomy of educational objectives. New York, NY:
Longman.
INFORMAL CLASSROOM WALK-THROUGHS 99
Annett, J. (2000). Theoretical and pragmatic influences on task analysis methods. In J.M.
Schraagen, S.F. Chipman & V.L. Shalin (Eds.), Cognitive task analysis (pp. 24-
36). Mahwah, NJ: Lawrence Erlbaum Associates.
Bartholio, C. W. (2010). The use of cognitive task analysis to investigate how many experts must
be interviewed to acquire the critical information needed to perform a central venous
catheter placement (Doctoral dissertation). Retrieved from
http://digitallibrary.usc.edu/cdm/ref/collection/p15799coll127/id/385767
Bedard, J. & Chi, M.T.H. (1992). Expertise. Current Directions in Psychological Science, 1(4),
135-139.
Berliner, D. C. (1986). In pursuit of the expert pedagogue. Educational Researcher, 15(7), 5-
13.
Berliner, D. C. (1988). Implications of expertise in pedagogy for teacher education and
evaluation. In New directions for teacher assessment: Proceedings of the 1988 ETS
invitational conference (pp. 39-67). Princeton, NJ: Educational Testing Service.
Bloom, G. (2007). Classroom visitations done well. Leadership, 36(4), 40-44.
Branch, G. F., Hanushek, E. A, & Rivkin, S. G. (2012). Estimating the effect of leaders on
public sector productivity: The case of school principals (NBER Working Paper No.
17803).
Bullough, R. V., & Baughman, K. (1995). Changing contexts and expertise in teaching: First-
year teacher after seven years. Teaching and Teacher Education, 11(5), 461-477.
California Department of Education (2010). Common core state standards: Frequently asked
questions. Retrieved from http://www.cde.ca.gov/re/ee/ccssfaqs2010.asap
INFORMAL CLASSROOM WALK-THROUGHS 100
California Department of Education (2013). Common core state standards for English language
arts and literacy in history/social science, science, and technical subjects for California
public schools: Kindergarten through grade twelve. Retrieved from:
http://www.cde.ca.gov/be/st/ss/documents/finalelaccssstandards.pdf
Canillas, E. N. (2010). The use of cognitive task analysis for identifying the critical
information omitted when experts describe surgical procedures
(Unpublished doctoral dissertation). University of Southern California, Los Angeles.
Cawelti, G. (2006). The side effects of NCLB. Educational Leadership, 64(3), 64-68.
Chao, C-J., & Salvendy, G. (1994). Percentage of procedural knowledge acquired as a function
of the number of experts from whom knowledge is acquired for diagnosis, debugging,
and interpretation of tasks. International Journal of Human-Computer Interaction, 6(3),
221-233.
Chi, M. T. H. (2006). Two approaches to the study of experts’ characteristics. In K. A.
Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge
handbook of expertise and expert performance (pp. 21–30). New York: Cambridge
University Press.
Chipman, S. F., Schraagen, J. M., & Shalin, V. L. (2000). Introduction to cognitive task
analysis. In J. M. Schraagen, S. F. Chipman, & V. L. Shalin (Eds.), Cognitive task
analysis (pp. 3-23). Mahwah, NJ: Lawrence Erlbaum Associates.
Clark, R.E. (1999). Ying and yang: Cognitive motivational processes in multimedia learning
environments. In J. van Merriënboer (ed.), Cognition and multimedia design. Herleen,
Netherlands: Open University Press.
Clark, R.E. (2008). Resistance to change: Unconscious knowledge and the challenge of
unlearning. In D.C. Berliner, & H. Kupermintz (Eds.), Changing institutions,
INFORMAL CLASSROOM WALK-THROUGHS 101
environments, and people. Mahwah, NJ: Lawrence Erlbaum Associates.
Clark, R.E. (2014). Cognitive task analysis for expert-based instruction in healthcare. In
Spector, J.M. Merrill, M.D. Elen, J. & Bishop, M.J. (Eds.), Handbook of research on
educational communications and technology (4
th
ed., pp.541-551). New York, NY:
Springer.
Clark, R., & Elen, J. (2006). When less is more: Research and theory insights about instruction
for complex learning. In J. Elen & R.E. Clark (Eds.), Handling complexity in learning
environments: Theory and research (pp. 283-295). New York, NY: Elsevier.
Clark, R. E., & Estes, F. (1996). Cognitive task analysis. International Journal of Educational
Research, 25(5), 403-417.
Clark, R. E., Feldon, D. F., van Merriënboer, J. J. G., Yates, K. A., & Early, S. (2008). Cognitive
task analysis. In J. M. Spector, M. D. Merrill, J. van Merriënboer, & M. P. Driscoll
(Eds.), Handbook of research on educational communications and technology (3rd ed.,
pp. 577-593). Mahwah, NJ: Lawrence Erlbaum Associates.
Cohen, D. K., Raudenbush, S. W., & Loewenberg Ball, D. (2003). Resources, instruction, and
research. Educational Evaluation and Policy Analysis, 25(2), 119-142.
Cooke, N. J. (1994). Varieties of knowledge elicitation techniques. International Journal of
Human-Computer Studies, 41(6), 801-849.
Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of
procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253-278.
Council for Exceptional Children (2005). CEC’s ESEA reauthorization recommendations.
Retrieved from
INFORMAL CLASSROOM WALK-THROUGHS 102
http://www.cec.sped.org/~/media/Files/Policy/ESEA/Recommendations/ESEA%20Reco
mmendations.pdf
Crandall, B., & Getchell-Reiter, K. (1993). Critical decision method: A technique for eliciting
concrete assessment indicators from the intuition of NICU nurses. Advances in Nursing
Science, 16(1), 47-51.
Crandall, B., Klein, G., & Hoffman, R. R. (2006). Working minds: A practitioner’s guide to
cognitive task analysis. Cambridge, MA: MIT Press.
Crispen, P. D. (2010). Identifying the point of diminishing marginal utility for cognitive task
analysis surgical subject matter expert interviews (Doctoral dissertation). Available from
ProQuest Dissertations and Theses database. (UMI No. 3403725)
Davidson, E., Reback, R., Rockoff, J., & Schwartz, H. L. (2013). Fifty ways to leave a child
behind: Idiosyncrasies and discrepancies in states’ implementation of NCLB. Article in
progress for publication. Retrieved from
http://www.columbia.edu/~ekd2110/Fifty_Ways_4_5_2013.pdf
Dee, T. S., & Jacob, B. (2011). The impact of No Child Left Behind on student achievement.
Journal of Policy Analysis and Management, 30(3), 418-446.
Diamond, J. B. (2007). Where the rubber meets the road: Rethinking the connection between
high-stake testing policy and classroom instruction. Sociology of Education, 80(4), 285-
313.
Donaldson, M. L. (2009). So long Lake Wobegon? Using teacher evaluation to raise teacher
quality. Washington, DC: Center for American Progress. Retrieved from
http://www.americanprogress.org/issues/2009/06/pdf/teacher_evaluation.pdf
INFORMAL CLASSROOM WALK-THROUGHS 103
Downey, C. J., Steffy, B. E., English, F. W., Frase, L. E., & Poston, W. K., Jr., (2004). The
three-minute classroom walk-through: Changing school supervisory practice one teacher
at a time. Thousand Oaks, CA: Corwin Press.
Duffett, A., Farkas, S. Rotherham, A. J., & Silva E. (2008). Waiting to be won over: Teachers
speak on the profession, unions, and reform. Washington, DC: Education Sector.
DuFour, R. & Mattos, M. (2013). How do principals really improve schools? Educational
Leadership, 70(7), 34-40.
Elmore, R. F. (2002). Bridging the gap between standards and achievement (Research report
written for the Albert Shanker Institute). Retrieved from
http://www.shankerinstitute.org/Downloads/Bridging_Gap.pdf
Embry, K. K. (2012). The use of cognitive task analysis to capture expertise for tracheal
extubation training in anesthesiology (Doctoral dissertation). Retrieved from
http://digitallibrary.usc.edu/cdm/ref/collection/p15799coll127/id/678652
Ericsson, K. A. (2004). Invited Address: Deliberate practice and the acquisition and
maintenance of expert performance in medicine and related domains. Academic
Medicine, 79(10), s70-s81.
Ericsson, K.A. & Charness, N. (1994). Expert performance its structure and acquisition.
American Psychologist, 49(8), 725-747.
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the
acquisition of expert performance. Psychological Review, 100(3), 363-406.
Ericsson, K. A., & Lehmann, A. C. (1996). Expert and exceptional performance: Evidence of
maximal adaptation to task constraints. Annual Review of Psychology, 47(1), 273-305.
INFORMAL CLASSROOM WALK-THROUGHS 104
Erikson, E. H. (1987). The human life cycle. In S. Schlein (Ed.), A way of looking at things (pp.
595-610). New York, NY: W. W. Norton & Company.
Erikson, E. H., Erikson, J. H., & Kivnick, H. Q. (1986). Vital involvement in old age. New York,
NY: W. W. Norton.
Fackler, J. C., Watts, C., Grome, A., Miller, T., Crandall, B., & Pronovost, P. (2009). Critical
care physician cognitive task analysis: an exploratory study. Critical Care, 13(2), R33.
doi: 10.1186/cc7740
Feeney, E. (2014). Design principles for learning to guide teacher walk-throughs. The Clearing
House: A Journal of Education Strategies, Issues and Ideas, 87(1), 21-29.
Feldon, D. F. (2004). Inaccuracies in expert self-report: Errors in the description of strategies
for designing psychology experiments (Doctoral dissertation). Available from ProQuest
Dissertations and Theses database. (UMI No. 3145196)
Feldon, D. (2007). Implications of research on expertise for curriculum and pedagogy.
Educational Psychology Review, 19(2), 91-110.
Feldon, D. F., & Clark, R, E. (2006). Instructional implications of cognitive task analysis as a
method for improving the accuracy of experts’ self-report. In G Clareabout & J. Elen
(Eds.), Avoiding simplicity, confronting complexity: Advances in studying and designing
(computer-based) powerful learning environments (pp. 109-116). Rotterdam,
Netherlands: Sense Publishers.
Flynn, C. L. (2012). The relative efficiency of two strategies for conducting cognitive task
analysis. (Doctoral dissertation). Retrieved from
http://gradworks.umi.com/35/61/3561771.html
INFORMAL CLASSROOM WALK-THROUGHS 105
Furhman, S. H. (2004). Introduction. In S. H. Furhman & R. F. Elmore (Eds.), Redesigning
accountability systems for education (pp. 3-14). New York, NY: Teachers College Press.
Glaser, R., & Chi, M. T. H. (1988). Overview. In M. T. H. Chi, R. Glaser, & M. Farr (Eds.), The
nature of expertise (pp. xv-xxviii). Mahwah, NJ: Lawrence Erlbaum Associates.
Gott, S. P., Hall, E. P., Pokorny, R. A., Dibble, E, & Glaser, R. (1993). A naturalistic study of
transfer: Adaptive expertise in technical domains. In D. K. Detterman & R. J. Sternberg
(Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 255-288).
Norwood, NJ: Ablex.
Grissom, J. A., Loeb, S., & Master, B. (2013). Effective instructional time use for school
leaders: Longitudinal evidence from observations of principals. Educational Researcher,
42(8), 433-444.
Gucev, G. V. (2012). Cognitive task analysis for instruction in single-injection ultrasound
guided-regional anesthesia (Doctoral dissertation). Available from ProQuest
Dissertations and Theses database. (UMI No. 3513770)
Hallinger, P. (2007). Instructional leadership and the school principal: A passing fancy that
refuses to fade away. Leadership and Policy in Schools, 4(3), 221-239.
Hannaway, J., & Hamilton, L. (2008). Performance-based accountability policies: Implications
for school and classroom practices. Retrieved from Urban Institute website:
http://www.urban.org/UploadedPDF/411779_accountability_policies.pdf
Harkin, T. (2013, June). Harkin: Why we must move forward on ESEA reauthorization.
Education Week. Retrieved from:
http://www.edweek.org/ew/articles/2013/06/27/36harkin.h32.html?qs=esea+reauthorizati
on
INFORMAL CLASSROOM WALK-THROUGHS 106
Harris, A. (2004). Distributed leadership and school improvement: Leading or misleading?
Educational Management Administration and Leadership, 32(1), 11-24.
Hatano, G. & Inagaki (2000). Practice makes a difference: Design principles for adaptive
expertise. Presented at the Annual Meeting of the American Education Research
Association. New Orleans, LA: April, 2000.
Hentschke, G. C., & Wohlstetter, P. (2004). Cracking the code of accountability. Urban Ed.
(Spring/Summer), 17-19.
Ho, A. D., & Kane, T. J. (2013). The reliability of classroom observations by school personnel
(Measures of Effective Teaching Research Paper). Retrieved from MET website:
http://www.metproject.org/downloads/MET_Reliability%20of%20Classroom%20Observ
ations_Research%20Paper.pdf
Hoffman, R. R., Crandall, B. W., & Shadbolt, N. R. (1998). Use of the critical decision method
to elicit expert knowledge: A case study in cognitive task analysis methodology. Human
Factors, 40(2), 254-276.
Hoffman, R. & Militello, L. (2009). Perspectives on cognitive task analysis: Historical origins
and modern communities of practice. New York: Psychology Press.
Hoffman, R., & Woods D. D. (2000). Studying cognitive systems in context: Preface to the
special section. Human Factors: The Journal of Human Factors Ergonomics Society, 42
(1), 1-7.
Hunt, E. & Joslyn, S.L. (2000). A functional task analysis of time-pressured decision making.
In J.M. Schraagen, S. Chipman & V.J. Shalin (Eds.), Cognitive task analysis (pp. 119-
132). Mahwah, NJ: Erlbaum.
INFORMAL CLASSROOM WALK-THROUGHS 107
Ing, M. (2010). Using informal classroom observations to improve instruction. Journal of
Educational Administration, 48(3), 337-358.
Jackson, P. W. (1985). Private lessons in public schools: Remarks on the limits of adaptive
instruction. In M. C. Wang & H. J. Walberg (Eds.), Adapting instruction to individual
differences (pp. 66–81). Berkeley, CA: McCutchan.
Jacob, B., & Lefgren, L. (2006). When principals rate teachers: The best and the worst stand out.
Education Next, 6(2), 59-64.
Kachur, D. S., Stout, J. A., & Edwards, C. L. (2010). Classroom walkthroughs: To improve
teaching and learning. Larchmont, NY: Eye on Education.
Kane, T. J., McCaffrey, D. F., Miller, T, & Staiger, D. O. (2013). How we identify effective
teachers: Validating measures of effective teaching using random assignment (Measures
of Teacher Effectiveness Project Research Paper). Retrieved from MET website:
http://www.metproject.org/downloads/MET_Validating_Using_Random_Assignment_R
esearch_Paper.pdf
Kim, J. S., & Sunderman, G. L. (2005). Measuring academic proficiency under the No Child
Left Behind Act: Implications for educational equity. Educational Researcher, 34(8), 3-
13.
Kirschner, P. A, Sweller, J., & Clark, R. E. (2006). Why minimally guided instruction does not
work: An analysis of the failure of constructivist, discovery, problem based, experiential,
and inquiry based teaching. Educational psychologist, 41(2), 75-86. doi:
10.1207/s15326985ep4102_1
Klein, G., Calderwood, R., & MacGregor, D. (1989). Critical decision method for eliciting
knowledge. IEEE Transactions on Systems, Man, and Cybernetics, 19(3), 462-473.
INFORMAL CLASSROOM WALK-THROUGHS 108
Lee, R. L. (2004). The impact of cognitive task analysis on performance: A meta-analysis of
comparative studies (Unpublished doctoral dissertation). University of Southern
California, Los Angeles, CA.
Linn, R. L. (2005). Fixing the NCLB accountability system (CREST Policy Brief 8). Los
Angeles: University of California Los Angeles, Center for the Study of Evaluation.
Louis, K. S., Dretzke, B., & Wahlstrom, K. (2010). How does leadership affect student
achievement? Results from a national US survey. School Effectiveness and School
Improvement, 21(3), 315-336.
Marzano, R. J., Waters, T., & McNulty, B. A. (2005). School leadership that works: From
research to results. Alexandria, VA: ASCD.
May, H., Huff, J., & Goldring, E. (2012). A longitudinal study of principals’ activities and
student performance. School effectiveness and school improvement, 23(4), 417-439.
McAdams, D. P., de St. Aubin, E., & Logan, R. L. (1993). Generativity among young, midlife,
and older adults. Psychology and Aging, 8(2), 221-230.
McDonnell, L. M., & Weatherford, M. S. (2013). Evidence use and the common core state
standards movement: From problem definition to policy adoption. American Journal of
Education, 120(1), 1-25.
Merrill, M. D. (1994). Instructional Design Theory. Englewood Cliffs, NJ: Educational
Technology Publications.
Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and
Development, 50(3), 43-49.
Merrill, M. D. (2006). Hypothesized performance on complex tasks as a function of scaled
instructional strategies. In J. Elen & R. E. Clark (Eds.), Handling complexity in learning
INFORMAL CLASSROOM WALK-THROUGHS 109
environments: Research and theory (pp. 265-281). Oxford, UK: Elsevier Science
Limited.
Mesler Parise, L., & Spillane, J. P. (2010). Teacher learning and instructional change: How
formal and on-the-job learning opportunities predict change in elementary school
teachers’ practice. The Elementary School Journal, 110(3), 323-346.
Militello, L. G., & Hoffman, R. R. (2008). The forgotten history of cognitive task analysis.
Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 52, 383-
387.
Murnane, R. J., & Papay, J. P. (2010). Teacher’s views on No Child Left Behind: Support for
the principles, concern about the practices. Journal of Economic Perspectives, 24(3),
151-166.
Nagel, D. (2014). Teacher evaluations: Principals need more support of effective
implementation. Technology Horizons in Education Journal. Retrieved from:
http://thejournal.com/articles/2014/02/12/teacher-evaluations-principals-need-more-
support-for-effective-implementation.aspx
Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004). How large are teacher effects?
Educational Evaluation and Policy Analysis, 26(3), 237-257.
O’Day, J. A., & Smith, M. S. (1993). Systematic reform and educational opportunity. In S. H.
Fuhrman (Ed.), Designing coherent education policy: Improving the system (pp. 250-
312). San Francisco, CA: Jossey-Bass.
OECD (2005). Teachers matter: Attracting, retaining, and developing effective teachers.
Retrieved from http://www.oecd.org/edu/teacherpolicy
INFORMAL CLASSROOM WALK-THROUGHS 110
Ovando, M. N. (2006). Building instructional leaders capacity to deliver constructive feedback
to teachers. Journal of Personnel Evaluation, 18, 171-183. doi:10.1007/s11092-006-
9018-z
Paris, S.G., Lipson, M.Y., & Wixson (1983). Becoming a strategic reader. Contemporary
Educational Psychology, 8(3), 293-316.
Peyton, C. (2005, July 26). District teachers get good grades - S.B. Unified: 96 percent of
instructors met performance standards, audit findings say. The Press Enterprise.
Retrieved from: http://www.pe.com
Phillips, V., & Wong, C. (2013). Tying together common core of standards, instruction, and
assessment. The Phi Delta Kappan, 91(5), 37-42.
Plank, S. B., & Falk Condliffe, B. (2013). Pressures of the season: An examination of classroom
quality and high-stakes accountability. American Educational Research Journal, 50(5),
1152-1182.
Porter, A., McMaken, J., Hwang, J., & Yang, R. (2011). Common core standards: The new U.S.
intended curriculum. Educational Researcher, 40(3), 103-116.
Range, B. G, Young, S., & Hvidston, D. (2013). Teacher perceptions about observation
conferences: What do teachers think about their formative supervision in on US district
school? School Leadership & Management, 33(1), 61-77. doi:
http://dx.doi.org/10.1080/13632434.2012.724670
Reback, R., Rockoff, J., & Schwartz, H. L. (2011). Under pressure: Job security, resource
allocation, and productivity in schools under NCLB (NBER Working Paper No. 16745).
Retrieved from the National Bureau of Economic Research website:
http://www.nber.org/papers/w16745
INFORMAL CLASSROOM WALK-THROUGHS 111
Rivkin, S. G., Hanushek, E. A., & Kain, J. F. (2005). Teachers, schools, and academic
achievement. Econometrica, 73(2), 417-458.
Rowan, B., Correnti, R., & Miller, R. J. (2002). What large-scale, survey research tells us about
teacher effects on student achievement: Insights from the Prospects Study of elementary
schools. CPRE Research Report Series, RR-051.
Schaafstal, A., Schraagen, J. M., & Marcel, v. B. (2000). Cognitive task analysis and innovation
of training: The case of structured troubleshooting. Human Factors, 42(1), 75. Retrieved
from http://search.proquest.com/docview/216446753?accountid=14749
Schraagen, J. M., Chipman, S. F., & Shalin, V. L. (2000). Cognitive Task Analysis. Mahwah,
NJ: Lawrence Erlbaum Associates.
Spillane, J. P., & Kenney A. W. (2012). School administration in a changing education sector:
The US experience. Journal of Education Administration, 50(5), 541-561.
Sternberg, R. J., & Horvath, J. A. (1995). A prototype view of expert teaching. Educational
Researcher, 24(6), 9-17.
Stiggins, R. J. (2002). Assessment crisis: The absence of assessment for learning. Phi Delta
Kappan, 83(10), 758-765.
Sullivan, M. E., Yates, K. A., Baker, C. J., & Clark, R. E. (2010). Cognitive task analysis and its
role in teaching technical skills. In Tsueda, S., Scott, D., & Jones, D. (Eds.), Textbook of
Simulation, Skills, and Team Training (pp.85-94). Woodbury, CT: Cine-Med.
Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive
Science, 12(2), 257-285.
INFORMAL CLASSROOM WALK-THROUGHS 112
Tirapelle, L. A. (2010). The effect of cognitive task analysis based instruction on surgical skills
expertise and performance. Available from ProQuest Dissertations and Theses database.
(UMI No. 3403766)
Tofel-Grehl, C., & Feldon, D. F. (2013). Cognitive task analysis-based training: A meta-analysis
of studies. Journal of Cognitive Engineering and Decision Making, 7(3), 293-304.
Tolano-Leveque, M. (2010). Using cognitive task analysis to determine the percentage of
critical information that experts omit when describing a surgical procedure (Doctoral
dissertation). Available from ProQuest Dissertations and Theses database. (UMI No.
3418184)
Tracey, T. J. G., Wampold, B. E., Lichtenberg, J. W., & Goodyear, R. K. (2014). Expertise in
psychotherapy: An elusive goal? American Psychologist, 69(3), 218-229.
Turnbull, B. J., Haslam, M. B., Arcaira, E. R., Riley, D. L., Sinclair, B., & Coleman, S. (2009).
Evaluation of the school administration manager project. Washington, DC: Policy
Studies Associates.
van Merriënboer, J. J. G., Clark, R. E., & de Croock, M. B. M. (2002). Blueprints for complex
learning: The 4C/ID-model. Educational Technology, Research and Development, 50(2),
39-64. doi: 0.1007/BF02504993
Wei, J. & Salvendy, G. (2004). The cognitive task analysis methods for job and task design:
review and reappraisal. Behavior & Information Technology, 23(4), 273–299.
Weisberg, D., Sexton, S., Mulhern, J., & Keeling, D. (2009). The widget effect: Our national
failure to acknowledge and act on differences in teacher effectiveness (Research Report
of The New Teacher Project). Retrieved from The New Teacher Project website:
http://widgeteffect.org/downloads/TheWidgetEffect.pdf
INFORMAL CLASSROOM WALK-THROUGHS 113
Wheatley, T., & Wegner, D. M. (2001). Automaticity of action, psychology of. In N. J. Smelser
& P. B. Baltes (Eds.), International encyclopedia of the social and behavioral sciences
(pp. 991-993). Oxford, UK: Elsevier Science Ltd.
Winslow, F. T. (1911). The principles of scientific management. New York, NY: Harper &
Brothers.
Woods, D. D., & Roth, E. M. (1988). Cognitive engineering: Human problem solving with
tools. Human Factors: the Journal of the Human Factors and Ergonomics Society, 30(4),
415-430.
Wright, S. P., Horn, S. P., & Sanders, W. L. (1997). Teacher and classroom context effects on
student achievement: Implications for teacher evaluation. Journal of Personnel
Evaluation in Education, 11(1), 57-67.
Yates, K. A. (2007). Towards a taxonomy of cognitive task analysis methods: A search for
cognition and task analysis interactions. (Doctoral dissertation). Retrieved from
http://search.proquest.com/docview/304828231?accountid=14749
Yates, K. A., & Feldon, D. F. (2011). Advancing the practice of cognitive task analysis: A call
for taxonomic research. Theoretical Issues in Ergonomics Science, 12(6), 472-495.
Zepeda-McZeal, D. (2014). Using cognitive task analysis to capture expert reading instruction
in informational text for students with mild to moderate learning disabilities
(Unpublished doctoral dissertation). University of Southern California, Los Angeles,
CA.
INFORMAL CLASSROOM WALK-THROUGHS 114
Appendix A
Cognitive Task Analysis Interview Protocol
Begin the Interview: Meet the Subject Matter Expert (SME) and explain the purpose of the
interview. Ask the SME for permission to record the interview. Explain to the SME the
recording will be only used to ensure that you do not miss any of the information the SME
provides.
Name of task(s): How to conduct informal classroom walk-throughs and provide feedback
to teachers.
Performance Objective:
Ask: “What is the objective of informal classroom walk-throughs and providing feedback to
teachers? What action verb should be used?”
Step 1:
Objective: Capture a complete list of outcomes for informal classroom walk-throughs and
providing feedback to teachers.
A. Ask the Subject Matter Expert (SME) to list outcomes when the task is complete. Ask
them to make the list as complete as possible
B. How are the outcomes assessed?
Step 2:
Objective: Provide practice exercises that are authentic to the teaching context in which the
tasks are performed.
A. Ask the SME to list all the contexts in which these tasks are performed (i.e., primary,
middle school, high school, English language arts, math, science, social studies, special
ed., ELD, etc.).
B. Ask the SME how the tasks would change for each teaching context.
Step 3:
Objective: Identify main steps or stages to accomplish the task.
A. Ask SME the key steps or stages required to accomplish the task.
B. Ask SME to arrange the list of main steps in the order they are performed, or if there is
no order, from easiest to difficult.
Step 4:
Objective: Capture a list of “step by step” actions and decisions for each task.
A. Ask the SME to list the sequence of actions and decisions necessary to complete the task
and/or solve the problem.
B. Ask: “Please describe how you accomplish this task step-by-step, so a novice could
perform it.”
INFORMAL CLASSROOM WALK-THROUGHS 115
For each step the SME gives you, ask yourself, “Is there a decision being made by the
SME here?” If there is a possible decision, ask the SME if there is a decision being
made.
If SME indicates that a decision must be made…
Ask: “Please describe the most common alternatives (up to a maximum of three) that
must be considered to make the decision and the criteria trainees should use to decide
between the alternatives”.
Step 5:
Objective: Identify prior knowledge and information required to perform the task
A. Ask SME about the prerequisite knowledge and other information required to perform the
task.
1. Ask the SME about Cues and Conditions
Ask: “For this task, what must happen before someone starts the task? What prior task,
permission, order, or other initiating event must happen? Who decides?”
2. Ask the SME about New Concepts and Processes
Ask: “Are there any concepts or terms required for this task that may be new to the
novice?”
Concepts – terms mentioned by the SME that may be new to the novice.
Ask for a definition and at least one example
Processes - How something works
Where do informal classroom walkthroughs and providing teachers with feedback
fit within a larger process? If the novice needs to understand a component of a
larger process, then prompt the SME “Processes usually consist of different
phases and within each phase, there are different activities – think of it as a flow
chart.”
Ask: “Must novices know this process to do the task?” “Will they have to use it
to change the task in unexpected ways?”
IF the answer is NO, do NOT collect information about the process.
3. Ask the SME about Equipment and Materials
INFORMAL CLASSROOM WALK-THROUGHS 116
Ask: “What equipment and materials are required to succeed at this task in
routine situations? Where are they located? How are they accessed?
4. Performance Standard
Ask: “How do we know the objective has been met? What are the criteria, such
as time, efficiency, quality indicators (if any)?”
5. Sensory experiences required for the task
Ask: “Must trainees see, hear, smell, feel, or taste something in order to learn
any part of the task? For example, are there any parts of this task they could not
perform unless they could see or hear something?”
Step 6:
Objective: Identify problems that can be solved by using the procedure.
A. Ask the SME to describe at least one simple or routine problem and two to three
complex problems that the trainee should be able to solve if they can perform each of
the tasks on the list you just made.
Ask: “Of the task we just discussed, describe at least one simple or routine problem and
two to three complex problems that the trainee should be able to solve IF they learn to
perform the task”.
INFORMAL CLASSROOM WALK-THROUGHS 117
Appendix B
Inter-rater Reliability Code Sheet for SME A
INFORMAL CLASSROOM WALK-THROUGHS 118
Appendix C
Job Aid for Developing a Gold Standard Protocol
Richard Clark and Kenneth Yates (2010, Proprietary)
The goals of this task are to 1) aggregate CTA protocols from multiple experts to create a “gold
standard protocol” and 2) create a “best sequence” for each of the tasks and steps you have
collected and the best description of each step for the design of training.
Trigger: After having completed interviews with all experts and capturing all goals, settings,
triggers, and all action and decision steps from each expert – and after all experts have edited
their own protocol.
Create a gold standard protocol
STEPS Actions and Decisions
1. For each CTA protocol you are aggregating, ensure that the transcript line number is
present for each action and decision step.
a. If the number is not present, add it before going to Step 2.
2. Compare all the SME’s corrected CTA protocols side-by-side and select one protocol
(marked as P1) that meets all the following criteria:
a. The protocol represents the most complete list of action and decision steps.
b. The action and decision steps are written clearly and succinctly.
c. The action and decision steps are the most accurate language and terminology.
3. Rank and mark the remaining CTA protocols as P2, P3, and so forth, according to the
same criteria.
4. Starting with the first step, compare the action and decision steps of P2 with P1 and
revise P1 as follows:
a. IF the step in P2 has the same meaning as the step in P1, THEN add “(P2)” at the
end of the step.
b. IF the step in P2 is a more accurate or complete statement of the step in P1,
THEN revise the step in P1 and add “(P1, P2)” at the end of the step.
c. IF the step in P2 is missing from P1, THEN review the list of steps by adding the
step to P1 and add “(P2N)”* at the end of the step.
5. Repeat Step 4 by comparing P3 with P1, and so forth for each protocol.
6. Repeat Steps 4 and 5 for the remaining components of the CTA report such as triggers,
main procedures, equipment, standards, and concepts to create a “preliminary gold
standard protocol” (PGSP).
7. Verify the PGSP by either:
a. Asking a senior SME, who has not been interviewed for a CTA, to review the
PGSP and note any additions, deletions, revisions, and comments.
b. Asking each participating SME to review the PGSP, and either by hand or using
MS Word Track Changes, note any additions, deletions, revisions, or comments.
i. IF there is disagreement among the SMEs, THEN either
1. Attempt to resolve the differences by communicating with the
SMEs, OR
INFORMAL CLASSROOM WALK-THROUGHS 119
2. Ask a senior SME, who has not been interviewed for a CTA, to
review and resolve the differences.
8. Incorporate the final revisions in the previous Step to create the “gold standard protocol”
(GSP).
INFORMAL CLASSROOM WALK-THROUGHS 120
Appendix D
SME A Initial Individual Protocol Flowchart
No
Yes
Legend
Begin and End
Process
Decision
Point
My Actions
Off-page
connector
Begin Procedure 1:
Discuss classroom
walk-throughs and
norms @ 1
st
staff
meeting of year
Communicate focus
area at biweekly
faculty meetings
Will notes be
taken in
classroom?
Inform staff,
describe notes,
provide form
Provide professional
development on
focus areas
Build a climate of
trust through your
ongoing actions
End Procedure 1
On-page
connector
INFORMAL CLASSROOM WALK-THROUGHS 121
No
Yes
Begin Procedure 2:
Determine focus
element for this
walk-through
Planning on
taking notes in
classroom?
Prepare form and
collect data as staff
was told
Prepare to take
notes outside
classroom
Select number of
rooms to visit and
how long to stay
End Procedure 2
INFORMAL CLASSROOM WALK-THROUGHS 122
No
Yes
No
Yes
Begin Procedure 3:
Walk into classroom,
make eye contact
with teacher
Take notes
in the
classroom?
Begin taking notes
Observe students
and identify
objective.
Is it a
lesson or
activity?
Lesson
Activity
Take notes in
classroom feedback
if applicable
Are students
working in
groups?
Observe teacher,
note observations
Procedure 3
continued on
page 116
See Step 3C on
page 7
See step 3B on
page 6
Identify whether
students can fill any
role and do they
know the objective
for the group work?
INFORMAL CLASSROOM WALK-THROUGHS 123
No
Yes
No
Yes
Is the teacher
using
technology?
Level and type of
usage, ask students
Is the teacher
on an
improvement
plan?
Note if meeting
objectives or note
deficiencies toward
improvement plan
Procedure 3
continued from
page 115
Take notes in
classroom if
applicable for
feedback
Take notes in
classroom for
feedback if
applicable
Take notes in the
classroom or
feedback if
applicable
Procedure 3
continued on
page 117
INFORMAL CLASSROOM WALK-THROUGHS 124
Yes
No
No
Yes
Yes
No
Yes
No
Is the teacher
demonstrating an
exemplary practice?
Ask teacher if you
can take a picture or
video
Is classroom
running
smoothly?
Are there parent
volunteers in the
classroom?
Exit Classroom
Acknowledge that
you are aware of the
issues but support
the teacher.
Thank parent for
helping the
classroom
End Procedure 3
Procedure 3
continued from
page 116
Take notes in the
classroom for
feedback if
applicable
Will you provide
feedback to the
teacher?
Think about the
feedback you will
provide to the
teacher and confirm
your observations.
INFORMAL CLASSROOM WALK-THROUGHS 125
No
No
No
Yes
Yes
No
Is objective
posted in student
friendly language?
Are students
meeting the
objective?
Ask another
student if he/she
knows the
objective?
Ask a student if
he/she knows
the objective?
Take notes in
classroom for
feedback if
applicable
Step 3B from
page 115
Return to step
3B on page 115
Ask more
students until you
find one who
knows, if any do
Yes
Yes
INFORMAL CLASSROOM WALK-THROUGHS 126
Yes
No
Step 3C from
page 115
Ask a student
what they’re
doing and why
Can a student tell
you the objective
and does the
activity match?
Ask another
student about the
objective and see
if activity matches
until you find one
who knows
Take notes in
classroom for
feedback if
applicable
Return to step
3C on page 115
INFORMAL CLASSROOM WALK-THROUGHS 127
No
No Yes
Yes
Begin Procedure 4:
Record notes
outside of
classroom
What is the quality of
the instruction
observed?
Positive feedback to
teacher in hallway or
write a short note
Informal hallway
conversation with
reflective questions
Schedule a meeting
in your office with
the teacher
Be supportive, offer
professional dev., address
operational issues
Were there
student issues
left unresolved
by the teacher?
Are students
in a harmful/
dangerous
setting?
Address teacher’s
practices and the
impact on student
learning
Investigate, meet
with union rep.,
document
Procedure 4
continued on
page 9
Low quality
implementation
Average
implementation
High quality
implementation
Discuss student
behavior with the
teacher and
develop strategies
to address the
student behavior
INFORMAL CLASSROOM WALK-THROUGHS 128
Yes
No
No
Yes
Yes
No
No
Yes
End Procedure 4
Is the teacher on
an improvement
plan?
Does the teacher
approach you
first?
Ask reflective
question or reinforce
positive behaviors
Does walk-through
data match formal
observation data?
Look for extenuating
circumstances to
enable redo
Procedure 4
continued from
page 8
Is the teacher
meeting the
objectives of the
improvement plan?
Investigate, meet
with union rep.,
document
Meet with teacher
to set clear
expectations for all
instruction
INFORMAL CLASSROOM WALK-THROUGHS 129
No Yes
No
No
Yes
Yes
Share data from
walk-throughs with
leadership team.
Were pictures
taken in classrooms
that could be used
as examples?
Are there teachers
who could provide
individual or group
model lessons?
Begin Procedure 5:
Meet with
leadership team to
receive feedback.
Does staff need
more professional
development on
same topic?
Provide professional
development in
same focus area.
Select a different
element of focus for
prof. development.
Work with
Leadership Team to
plan prof. dev.
End Procedure 5
Ask the teacher if the
pictures from his/her
class can be used and
use them as an
example.
Ask the teacher teach
a strategy as an
example for
individual or groups
of teachers.
INFORMAL CLASSROOM WALK-THROUGHS 130
Appendix E
Gold Standard Protocol
Principal’s Informal Classroom Observation Walk-throughs and Providing Feedback to
Teachers
Conducting an informal classroom observation walk-through and providing feedback to teachers
is outlined in this protocol. The informal classroom observation walk-through is conducted in
one single day however the follow-up activities may occur over multiple days.
Main Procedures:
1) Build rapport, relationships, and trust.
2) Set clear expectations.
3) Plan for the walk-throughs.
4) Observe the teacher in the classroom.
5) Provide feedback to individual teachers.
6) Provide feedback to the entire teaching staff.
7) Communicate with the Leadership Team and plan next steps.
8) Facilitate personal reflection.
Procedure 1. Build rapport, relationships, and trust.
1.1. IF you are a new principal to a school OR there are new staff members at the
school, THEN contact each staff member individually by telephone OR in person
before the school year starts to welcome him/her to a new school year and
introduce yourself as the principal. (B)
1.2. Starting with Step 1.1 and prior to the start of the school year and throughout the
school year, build rapport, relationships, credibility, and trust with the teaching
staff through: (B, C)
1.2.1. Conducting staff meetings in which at least one teacher shares how he/she
has implemented a component of your expectations (Premier Instruction,
OWL, non-negotiables, over-arching focus area, or other instructional
practices). (B, C)
1.2.2. Attending social events. (B)
1.2.3. Making one-on-one connections with teachers. (B)
1.2.4. Recognizing teachers’ individual events (birthdays, anniversaries,
achievement, etc.). (B)
1.2.5. Celebrating and honoring holidays. (B)
1.2.6. Maintaining a positive tone with the staff focusing on what is going well
by providing support and positive feedback at least twice as frequently as
constructive feedback. (B, C)
1.2.7. Writing individualized notes providing positive feedback. (B)
1.2.8. Maintaining open and frequent communication with the teachers about
what you are seeing during your walk-throughs. (C)
INFORMAL CLASSROOM WALK-THROUGHS 131
1.2.9. Being honest about what you observe and sharing your observations with
the teachers frequently. (C)
1.2.10. Validating the teachers as professionals by asking questions, listening, and
mirroring back their words. (C)
1.2.11. Using directive or telling language only in cases where all other strategies
are not changing a teacher’s practice. (C)
Procedure 2. Set clear expectations.
2.1. Communicate orally and in writing what your expectations are for the informal
classroom walk-throughs (Premier Instruction, OWL, non-negotiables, over-
arching focus area, or other instructional practices). (A, B)
2.1.1. STANDARD: These strategies will likely change every year as the school
improves and the proficiency level of the staff increases, and remember to
establish realistic expectation based on the staff’s current proficiency
level. (B)
2.2. Meet with the Leadership Team (staff members at the school who are respected
by their peers and recognized as understanding and delivering quality instruction)
between the beginning of the school year the first staff meeting of the school year
to discuss the informal walk-through observation process. (B)
2.2.1. Reinforce that as the instructional leader you will be walking through the
classrooms at different times during the school day to provide support and
help to the teachers. (B, C)
2.2.2. Develop the agenda for the first staff meeting of the school year with the
Leadership Team which includes informal walk-through observations as
one of the agenda items. (B)
2.3. Meet with the office staff and explain the informal walk-through process so they
understand that you should not be interrupted during walk-throughs unless it is an
emergency. (B)
2.4. At the first staff meeting at the beginning of the school year, tell the entire
teaching staff what your expectations (Premier Instruction, OWL, non-
negotiables, over-arching focus area, or other instructional practices) are when
you walk-through the classrooms and provide them with these expectations in
writing from step 1.2. (A, B, C)
2.4.1. IF most of your teachers are returning teachers AND they know your
expectations (Premier Instruction, OWL, non-negotiables, over-arching
focus area, or other instructional practices) AND have heard you explain
them before, THEN keep the explanation of your expectations more
general. (A)
2.4.2. IF you have a large number of new teachers or transferring teachers,
THEN discuss your expectations (Premier Instruction, OWL, non-
negotiables, over-arching focus area, or other instructional practices) in
detail. (A, B)
2.4.3. Emphasize with the teacher staff your role as instructional leader is to
provide support and help to the teachers. (B)
INFORMAL CLASSROOM WALK-THROUGHS 132
2.4.3.1. Open and close the agenda item with this emphasis because
due to “primacy/recency” teachers will best remember the first
the last items you discuss of an agenda item. (B)
2.4.4. Ensure the teachers understand that you will be walking through the
classrooms frequently to identify and support your expectations (Premier
Instruction, OWL, non-negotiables, over-arching focus area, or other
instructional practices). (B)
2.4.5. Set the norms with your teachers for the walk-throughs: (C)
2.4.5.1. Tell the teachers when you walk in the classroom to continue
teaching and do not stop instruction to recognize your
presence. (C)
2.4.5.2. Tell the teachers that you will be looking for specific evidence
that your expectations (Premier Instruction, OWL, non-
negotiables, over-arching focus area, or other instructional
practices) are being addressed. (C)
2.4.5.3. Reinforce with the teachers that the informal walk-through is
just a snapshot and not intended to be an observation of an
entire lesson. (C)
2.4.5.4. Stress with the teachers that this informal walk-through is not
evaluative and is solely for the purpose of looking at the
progress towards implementing your expectations (Premier
Instruction, OWL, non-negotiables, over-arching focus area, or
other instructional practices). (C)
2.4.5.5. Differentiate informal walk-throughs from formal observations
by not conducting informal walk-throughs during the time of
the year when formal observations occur. (C)
2.5. Use Leadership Team meetings and staff meetings throughout the school year to
reinforce and provide professional development on techniques, methods, or ideas
related to your expectations (Premier Instruction, OWL, non-negotiables, over-
arching focus area, or other instructional practices). (A, B, C)
2.5.1. Instruct the staff so they understand and identify the difference between
lessons and activities. (B)
2.5.2. Develop a common language related to your expectations (Premier
Instruction, OWL, non-negotiables, over-arching focus area, or other
instructional practices). (C)
2.5.3. Build a common understanding of what each element of your expectations
(Premier Instruction, OWL, non-negotiables, over-arching focus area, or
other instructional practices) should look like when used in the classroom.
(C)
2.5.4. Tell the staff that this is an ongoing process of continuous learning that
will continue to develop and change based on the students assigned to the
teachers’ classrooms, the teachers’ professional growth, and the school’s
achievement data. (B)
2.5.5. STANDARD: All professional development is aligned with the Board’s
goals and research based. (B)
INFORMAL CLASSROOM WALK-THROUGHS 133
2.6. IF you have new teachers on your staff, THEN meet with the new teachers every
month at a separate meeting to develop a clear understanding and common
vocabulary regarding your expectations (Premier Instruction, OWL, non-
negotiables, over-arching focus area, or other instructional practices) during your
informal walk-throughs so the new teachers will clearly understand what you are
talking about when you provide feedback after your informal walk-throughs. (A)
2.6.1 Provide new teachers with clear examples of the types of positive
feedback or suggestions that you will provide and what they will sound
like so they are not surprised when they receive them or left wondering
what the feedback sounds like. (A)
2.6.2. Set a clear expectation that teachers will know where they stand and
receive clear feedback from you on their performance. (A)
2.7. IF there are multiple administrators at the site who will be conducting informal
classroom walk-through observations, THEN ensure all administrators hold the
same expectations when they conduct their informal walk-throughs. (A)
2.8. Communicate with the staff at the regularly scheduled staff meeting what
component of your expectations (Premier Instruction, OWL, non-negotiables,
over-arching focus area, or other instructional practices) you will look for during
the informal walk-throughs for the each subsequent period between staff
meetings. (C)
2.8.1 IF notes will be taken while you are inside the classroom during the period
of time between staff meetings, THEN inform teachers in advance at the
staff meeting that this will be taking place AND exactly what type of
information will be recorded in the notes (i.e. tallying, scripting, asking
questions, etc.) AND provide a copy of any form that will be used to
record these notes so staff may provide feedback and offer ideas for
editing the form. (C)
2.9. Create and email a brief weekly newsletter (Monday Morning Message) to all
school staff, key district office staff, select community members, and some
retirees that is two or three paragraphs long and reinforces and reviews a
component of your expectations (Premier Instruction, OWL, non-negotiables,
over-arching focus area, or other instructional practices) among other school
related topics. (A)
2.10. Convey the same message regarding instructional leadership and informal
classroom walk-throughs shared with staff in Step 1.5.5 through 1.5.5.5 with
various parent and community groups throughout the school year so they will
understand your expectations (Premier Instruction, OWL, non-negotiables, over-
arching focus area, or other instructional practices) and the informal walk-through
process. (B)
2.11. Anticipate push-back from the teacher’s union due to increased expectations,
claims that the informal walk-through process is evaluative, and addition of extra
duties to the teachers’ workloads. (B)
Procedure 3. Plan for the walk-throughs.
INFORMAL CLASSROOM WALK-THROUGHS 134
3.1. (a) IF you are the only one administrator assigned to the school, THEN proceed to
step 3.2. (b) IF there is more than one administrator assigned to the school,
THEN schedule and hold a weekly administrator’s meetings. (A)
3.1.1. Divide the campus evenly among the administrative team so every week at
least one administrator is in every classroom. (A)
3.1.1.1. Rotate the way the classrooms are divided so that each
administrator has the opportunity to see all of the classrooms
on the campus. (A)
3.1.1.2. IF during the previous week you did not have an opportunity to
conduct informal walk-throughs in all of your assigned
classrooms, THEN discuss with the rest of the administrative
team not rotating the assigned classrooms that week AND
visiting the same set of classrooms for two weeks in a row. (A,
B)
3.1.1.3 IF a specific teacher, grade level, or department have requested
you to walk-through their classrooms, THEN ensure that your
scheduled informal walk-throughs will enable you to visit those
classrooms, AND even if that classroom(s) is not in your
weekly rotation of classrooms to visit. (A, B)
3.1.1.4 IF you had a follow-up meeting with a teacher as described in
step 5.11.5, THEN make sure to visit that classroom during the
week, especially if/when invited by the teacher, AND even if
that classroom is not in your weekly rotation of classrooms to
visit. (A, B)
3.1.2. For a high performing teacher, IF the administrative team has seen this
teacher providing a high quality lesson or activity that is in the area where
another teacher needs assistance, THEN ask the high performing teacher
to teach a model lesson OR support another teacher on learning how to
better deliver the lesson or activity. (A)
3.1.3. For an average performing teacher, IF you or another administrator
observed something during your informal walk-through that was not
consistent with the teacher’s normal performance, THEN share the
concern with the other administrators to get their input from their informal
observations AND ensure the administrator who walks-through that
classroom during the week looks for the same behavior to determine if
there is a pattern of behavior. (A)
3.1.4. For a low performing teacher, discuss what has been observed in his/her
classroom every week and whether that teacher is improving toward
meeting the expectations (Premier Instruction, OWL, non-negotiables,
over-arching focus area, or other instructional practices). (A)
3.1.5. IF the administrative team thinks a specific teacher who is being discussed
needs feedback to meet the expectations (Premier Instruction, OWL, non-
negotiables, over-arching focus area, or other instructional practices),
THEN determine among the administrative team what type of feedback to
provide and who will provide that feedback to the teacher. (A)
INFORMAL CLASSROOM WALK-THROUGHS 135
3.1.6. IF an administrator has received a parent or student complaint regarding a
teacher, THEN discuss what the administrative team has seen during
informal walk-throughs to confirm or refute the complaint AND focus on
that teacher’s performance during the next week’s informal walk-
throughs. (A)
3.1.7. IF a teacher has sent a greater than normal number of students to the office
for discipline referrals OR a greater than normal number of students for
detention, THEN discuss what the administrative team has seen during
informal walk-throughs to explain the referrals and detentions AND focus
on that teacher’s performance during the next week’s informal walk-
throughs. (A)
3.2. IF you are the only administrator assigned to the school, THEN consider the
following prior to conducting your walk-throughs: (A, B)
3.2.1. IF certain classrooms were not visited during the last time you conducted
informal walk-throughs, THEN start with the classrooms that were not
visited last time. (A, B)
3.2.2. If a specific teacher or grade level has requested you to walk-through their
classrooms, THEN ensure that your scheduled informal walk-throughs
will enable you to visit that classroom(s). (A, B)
3.2.3. IF you had a follow-up meeting with a teacher as described in step 5.1.4.2,
THEN make sure to visit that classroom during the week, especially
if/when invited by the teacher, AND even if that classroom is not in your
scheduled classrooms to visit that day. (A, B)
3.2.4. For an average performing teacher, IF you observe something during your
informal walk-through that was not consistent with the teacher’s normal
performance, THEN ensure you look for the same behavior to determine if
there is a pattern of behavior. (A)
3.2.5. For a low performing teacher, think about what has been observed in
his/her classroom every week and whether that teacher is improving
toward meeting the expectations (Premier Instruction, OWL, non-
negotiables, over-arching focus area, or other instructional practices). (A)
3.2.6. For a high performing teacher, IF you have seen this teacher providing a
high quality lesson or activity that is in the area where another teacher
needs assistance, THEN ask the high performing teacher to teach a model
lesson OR support another teacher on learning how to better provide the
lesson or activity. (A)
3.2.7. IF you think a specific teacher who you are reflecting on needs feedback
to meet the expectations (Premier Instruction, OWL, non-negotiables,
over-arching focus area, or other instructional practices), THEN determine
what type of feedback to provide and when to provide that feedback to the
teacher. (A)
3.2.8. IF you have received a parent or student complaint regarding a teacher,
THEN think about what you have seen during informal walk-throughs to
confirm or refute the complaint AND focus on that teacher’s performance
during the next week’s informal walk-throughs. (A)
INFORMAL CLASSROOM WALK-THROUGHS 136
3.2.9. IF a teacher has sent a greater than normal number of students to the office
for discipline referrals OR a greater than normal number of students for
detention, THEN think about what you have seen during informal walk-
throughs to explain the referrals and detentions AND focus on that
teacher’s performance during the next week’s informal walk-throughs.
(A)
3.3. Schedule daily classroom walk-throughs into your calendar, but do not tell the
staff when you will be conducting informal walk-throughs as they are
unannounced. (A)
3.3.1. IF the informal walk-through observations were conducted at a certain
time of day during the prior observations, THEN try to conduct them at a
different time of the day. (B)
3.4. Determine the focus from your expectations (Premier Instruction, OWL, non-
negotiables, over-arching focus area, or other instructional practices) for the day’s
informal classroom walk-throughs based on what was communicated with the
teachers in Step 2.8 and what you have determined to be the highest priority based
on student achievement in Steps 3.1 or 3.2. (B, C)
3.4.1. Based on the time of the school year, select a part of your expectations
(Premier Instruction, OWL, non-negotiables, over-arching focus area, or
other instructional practices) that are appropriate for the time of year (i.e.
beginning of the year, establishing routines and patterns; middle of the
year, maximizing learning and building on previous learning; etc.). (B)
3.5. Select the number of classrooms to walk-through and how long to stay in each
classroom based on time available to conduct walk-throughs and focus from your
expectations (Premier Instruction, OWL, non-negotiables, over-arching focus
area, or other instructional practices). (C)
3.6. IF the informal walk-through observation requires specific data that has to be
recorded in the classroom in the moment in a specific way, THEN prepare to
record the data according to the manner that was indicated to the staff in Step
2.8.1. (C)
3.6.1. If a specific form or note-taking aide will be used, THEN make sufficient
copies of the form or note taking aide. (C)
3.7. IF notes will not be taken while in the classroom, THEN prepare to take notes
outside the classroom after conducting the informal walk-through observation by
carrying a clipboard with paper for taking notes OR using you cell phone or IPad
and send yourself an email with your notes. (A, B, C)
3.8. Inform the office staff that you will be conducting informal walk-throughs and for
what length of time. (B)
3.8.1. If you have been interrupted over the past one or two informal walk-
throughs for non-emergency situations, THEN remind the office staff that
this time is dedicated to informal walk-throughs and you should only be
interrupted in an emergency. (B)
Procedure 4. Observe the teacher in the classroom.
4.1. Open the door and walk into the classroom. (A, B, C)
INFORMAL CLASSROOM WALK-THROUGHS 137
4.1.1. IF there are multiple entrances into a classroom, THEN use a different
entrance and exit for each informal classroom walk-through. (B)
4.2. Make eye contact with the teacher and smile so that the teacher will not think you
are trying to be sneaky and to acknowledge that this is an informal walk-through.
(A, C)
4.3. IF you will be taking notes in the classroom during the informal walk-through,
THEN begin to note observable data and write it down once observed. (C)
4.4. Do not talk to the teacher or interrupt what the teacher is doing. (A, C)
4.4.1. IF you do need to talk to the teacher, THEN move to the front of the
classroom AND keep eye contact AND wait for the teacher to ask if you
need something AND depending on the content of the conversation talk to
the teacher either inside or outside of the classroom. (A)
4.4.2. Keep a “poker face” and neutral demeanor while in the classroom. (B)
4.5. Walk to the back of the classroom or where you can observe the most from one
location but be as least disruptive as possible. (A, B)
4.5.1. Move throughout the classroom and observe from multiple vantage points.
(B)
4.6. Look around the entire classroom taking in all four walls, the students, how the
room is organized, what is being presented and displayed, and the teacher. (A, B)
4.6.1. Evaluate if the items displayed are accessible and available for student
use, preferably student work product. (A)
4.7. Scan the front of the classroom for a posted daily agenda and identify that the
lesson or activity matches what the teacher has posted on the daily agenda. (A, B)
4.8. Look at what is projected, written on the Board, or being given to the students for
evidence of the objective. (A, B)
4.9. Observe the students, where the action is taking place in the classroom, and look
at what the students are doing. (A, C)
4.10 Observe whether the students are participating in a lesson or an activity and
identify if what the students are doing matches the objective. (A, C)
4.10.1. Identify whether it is an instructional day or activity day by trying to
determine where the teacher is in the instructional process. (A, C)
4.10.2. IF it is an instructional day, THEN the teacher is teaching a lesson. (C)
4.10.2.1. IF the teacher is teaching a lesson, THEN look for the objective
of the lesson to be posted at the front of the classroom in
student friendly language AND identify if the instruction is
enabling the students to reach the objective. (C)
4.10.2.2. IF you determine the students are meeting the posted objective,
THEN do not proceed to Step 4.11 indicating to the teacher
that you recognize the lesson is going well and you are
observing instruction that meets the posted objective. (C)
4.10.2.3. IF the teacher is teaching a lesson AND no objective is posted,
THEN remember this information to record in your notes AND
to provide feedback to the teacher. (C)
4.10.3 IF you cannot determine where the teacher is in the instructional process
within approximately five minutes, THEN it is an activity day. (A)
INFORMAL CLASSROOM WALK-THROUGHS 138
4.10.3.1. IF it is an activity day OR you cannot determine the objective
of the lesson, THEN move to Step 4.11. (C)
4.11. IF it will not interrupt a student’s learning or a group activity, unobtrusively speak
with a student and ask them what they are learning today and why. (A, B, C)
4.11.1. IF the student cannot tell you what he/she is learning OR the task does not
match the objective, THEN unobtrusively ask another student what they
are learning today AND keep asking additional students until you get an
answer. (A, B, C)
4.11.1.1. Remember the names of the students or where they are sitting
to discuss with the teacher during feedback. (B)
4.11.1.2. IF multiple students do not know the objective OR the task
does not match the objective AND the students are not learning
the intended materials, THEN remember to record this
information in your notes AND remember to send the teacher
an email outside the classroom or give the teacher a hand
written note to see you after school in order to discuss the
observation. (B, C)
4.11.2. IF the student understands the objective and the task matches the
objective, THEN continue the informal observation AND speaking to
additional student(s) is optional. (B, C)
4.11.2.1. While speaking with the student, observe what work the
student is producing and determine if it matches your
expectations for that portion of the lesson. (A)
4.12. Look for student engagement by determining what the students are doing and/or
producing and if it is connected to what the teacher intends for them to learn/do,
in other words, the activities match the objective. (A)
4.13. Observe the balance between students working independently and depending
upon their peers. (C)
4.14. IF students are working in cooperative groups, THEN distinguish whether or not
all students understand the materials AND whether students could serve in any
role required to complete the task. (C)
4.15. Observe the teacher’s actions. (A, C)
4.16. Look for evidence of your expectations (Premier Instruction, OWL, non-
negotiables, over-arching focus area, or other instructional practices). (A, C)
4.17. Observe classroom management and discipline issues. (C)
4.18. Observe the rapport and positive classroom environment that is being built
between the teacher and the student by looking for the following: (C)
4.18.1. Distinguish if student feel safe and supported to take risks and try new
experiences by observing if they will answer questions when they are not
completely certain of the answer. (C)
4.18.2. Observe which students are asked to answer question and how the teacher
responds to the student s in order to see if even incorrect responses are
received positively (not using terms such as “wrong answer”), questions
are open-ended versus “yes/no,” and students do not feel ridiculed. (C)
INFORMAL CLASSROOM WALK-THROUGHS 139
4.18.3. Identify the teacher’s style of interacting with the students (i.e. humor,
caring, etc.) and look for visual cues of how the students respond to this
interaction. (C)
4.18.4. Identify any students who are “pushing a teacher’s buttons” and remember
these occurrences to be recorded in your notes outside the classroom for
later discussion with the teacher noting the triggers for the student’s
behavior and the teacher’s response(s). (C)
4.18.5. Identify students who are responding with “I don’t know,” or shrugging
their shoulders and how the teacher encourages their participation. (C)
4.18.6. Observe students who do not seem to understand the instruction or the
activity and how the teacher encourages or support them. (C)
4.18.7. Look for students who do not raise their hands or try to answer questions
to see if the teacher involves them and encourages their participation. (C)
4.18.8. Observe students’ body language and how the teacher reacts to the
different types of body language in the classroom. (C)
4.18.9. Identify how and what students use in the room environment as a tool to
assist them with learning during lesson and activities. (C)
4.18.10. Notice how long the teacher waits between asking a question and calling
on a student to answer (wait time) and how that impacts the number of
students volunteering to answer questions. (C)
4.19 IF you identify students in the classroom who are identified for intervention or
specialized instruction (Special Education students, English Language Learners,
GATE students, etc.), THEN identify the strategies (preview/review, collaborative
groups, total physical response, sentence frames, depth and complexity icons,
etc.) that the teacher is using and how they are helping the identified students.
(D)
4.19.1 IF the teacher is not using any differentiated strategies to assist students
identified for intervention or specialized instruction OR the strategies are
not being used appropriately, THEN note the information and remember
the information to record in your notes once outside the classroom. (D)
4.20. Look for the use of classroom technology. (C)
4.20.1. IF classroom technology is being used, THEN determine the level and
type of usage AND remember the information to record in your notes once
outside the classroom. (C)
4.20.2. Ask the students about the technology to determine if it is enhancing their
learning, working appropriately, and creating minimal distractions. (C)
4.21. IF the teacher has been placed on a Plan for Improvement based on a history of
performance problems, THEN observe whether the areas addressed in the Plan for
Improvement are being implemented in the classroom AND note the deficiencies
and improvements AND provide feedback in Steps 5.13 and 5.14. (C)
4.22. If you observe an extreme situation in the classroom in which students are at risk
of harm, there is extreme misbehavior, there is a violation of school or district
rules, or the teacher is presenting information inaccurately so that students will
learn the information incorrectly, THEN discreetly approach the teacher and
inform the teacher that there is a serious situation which needs to be addressed
immediately. (B)
INFORMAL CLASSROOM WALK-THROUGHS 140
4.23. If there is exemplary implementation of one or your expectations (Premier
Instruction, OWL, non-negotiables, over-arching focus area, or other instructional
practices), THEN ask the teacher if you can take a picture or video AND use your
IPad or your phone to take a picture to provide positive feedback to the
teacher/grade level OR present to the entire staff during a staff meeting. (B, C)
4.24. Observe to see if there are any parents in the classroom. (C)
4.24.1. IF parent volunteers are in the classroom, THEN acknowledge their
presence by smiling, making eye contact, and speaking with them in an
unobtrusive manner to thank them for volunteering. (C)
4.24.2. IF parent volunteers are in the classroom and the lesson or activity is not
going smoothly, THEN speak to the parent in an unobtrusive and non-
judgmental manner letting the parent know that you have recognized the
current classroom condition but are supporting the teacher’s
professionalism with a phrase such as, “It looks like (Student Name) may
be having a difficult day and (Teacher Name) has his/her hands full.
He/she is lucky to have such dedicated parent volunteers like you.” (C)
4.25. Spend at least 5 but not more than 15 minutes in each classroom. (A, B)
4.25.1. IF you have gathered data from observing where the teacher is in the
instructional process, looking at teacher behaviors, talking to a student(s),
looking at what a student(s) is producing, and looking at the room
environment; THEN prepare to exit the classroom. (A)
4.26. Prior to leaving the classroom, double-check the data that you have from your
informal walk-through and decide whether or not to provide feedback to the
teacher based on specific observations from the classroom that have the highest
possibility of affecting student learning and remember specific information to
note once outside of the classroom. (A, B, C)
4.26.1. IF you will not be providing the teacher with feedback from your informal
walk-through because the teacher is meeting your expectations, THEN
unobtrusively get the teacher’s attention and give him/her a “thumbs-up”
or a nod on your way out of the door AND remember to send an email to
yourself or take written notes when you are outside the classroom for your
own notes or journaling. (A, B)
4.26.2. IF you will be positive feedback to the teacher, THEN decide what
positive information you will share with the teacher from what you have
observed AND what the next suggested steps are for the teacher to work
on AND remember to send yourself an email or take written notes when
you are outside the classroom for your own notes or journaling. (B)
4.26.3. IF you will be providing suggestions or asking reflective questions of the
teacher based on a practice that needs improvement, THEN unobtrusively
get the teacher’s attention and give him/her a “thumbs up” or a nod on
your way out of the door AND remember to send an email to yourself or
take written notes with a suggestion or reflective question when you are
outside the classroom. (A, B)
4.26.4. IF you will be providing feedback that is critical or urgent, THEN make
eye contact with the teacher when you exit the classroom if you are able
BUT do not give a “thumbs-up” AND remember to send the teacher an
INFORMAL CLASSROOM WALK-THROUGHS 141
email or hand the teacher a written note to see you in your office after
school. (A)
4.27 Exit the classroom if one or more of the following conditions occur: (A, B, C)
4.27.1. IF you have collected enough data to make a decision about providing
follow-up information to the teacher, THEN exit the classroom. (A, B, C)
4.27.2. IF you have been called on the radio and are need elsewhere, THEN exit
the classroom. (C)
4.27.3. IF the classroom environment is uncomfortable for the teacher or the
students are having an awkward moment due to what is happening in the
classroom, THEN exit the classroom. (C)
4.27.4. IF it is not a dangerous or harmful situation, THEN exit the classroom.
(C)
Procedure 5. Provide feedback to individual teachers.
5.1. Once outside the classroom, use paper to take notes or use your IPad or cell phone
to send a short email (2-3 sentences) that will be for your reference, which will
include: what was observed, what needs to be reinforced with the teacher, how it
will be reinforced, what support will be provided through professional
development and coaching, what positive feedback will be provided to the
individual or entire staff, and what connections should be made among staff for
collaborative support. (A, B, C)
5.1.1. IF there is more than one administrator assigned to the school AND you
saw, heard, or felt something that you want to remember BUT are not sure
what you will do with the information, THEN share the note or email
during the weekly administrator’s meeting AND decide what to do with
the information during the next administrator’s meeting (Steps 3.1.2
through 3.1.5). (A)
5.1.2. If you are the only administrator assigned to the school AND you saw,
heard, or felt something that you want to remember BUT are not sure what
you will do with the information, THEN review the notes or email AND
decide what to do with the information after reflecting on it. (A, B)
5.2. IF you will provide feedback to the teacher, THEN make every effort to start
verbal or written (email) feedback with something you observed during your
walk-through of the classroom that was positive. (A, B)
5.3. IF you will provide feedback to the teacher that involves areas of improvement,
THEN offer ways that you will help the teacher AND show your willingness to be
a resource or provide resources to work with the teacher to find solutions so the
feedback is not viewed as a “gotcha.” (B)
5.3.1. Ensure that the teaching staff hears you say and sees your actions which
demonstrate you do not have all of the answers but that you are also a
learner and are willing to work with them to find answers. (B)
5.3.2. Based on the time of the school year, provide feedback to the teacher from
your informal walk-throughs based on what you would expect to see at
INFORMAL CLASSROOM WALK-THROUGHS 142
that time of the year and the amount of stress that teachers are
experiencing. (A)
5.3.2.1 IF a teacher is under pressure due to the time of the year OR a
personal situation (illness, family illness, divorce, death, etc.)
AND the teacher’s performance is not significantly impacting
student learning, THEN write a note or send an email to
yourself AND save the note or email for a less stressful time to
discuss with the teacher and for your own journaling. (A)
5.3.2.2 IF a teacher is under pressure due to the time of the year OR a
personal issue (illness, family illness, divorce, death, etc.)
AND the teacher’s performance is significantly impacting
student learning over more than one day, THEN give the
teacher a note or send an email to the teacher to come see you
after school and discuss what can be done to lessen the impact
upon student learning. (A)
5.4. IF the teacher approaches you in an informal (hallway) setting for feedback on
what was occurring in his/her classroom when you walked-through, THEN try to
find a more private location in which to talk AND let the teacher do the majority
of the talking AND attempt to build the teacher’s confidence by asking reflective
questions (no more than 2) that will help strengthen and refine the teacher’s
strategies to address your expectations (Premier Instruction, OWL, non-
negotiables, over-arching focus area, or other instructional practices). (A, B, C)
5.4.1. IF the conversation with the teacher will take more than a few minutes or
is sensitive in nature, THEN politely ask the teacher to see you at the end
of the school day in your office in order to give you the “gift of time” to
meet the teacher in an unrushed manner and develop your response in
advance. (B)
5.5. IF the informal walk-through observation results in information that needs
immediate attention, is dangerous or harmful, a safety hazard, breaks school or
district rules, or is critical or urgent in nature because it impacts student learning,
THEN give the teacher a written note and send an email to the teacher to meet
with you after school that day in your office with a union representative AND
schedule the meeting on your calendar. (A, B, C)
5.5.1. IF a negative situation needs to be discussed with a teacher, THEN have
that discussion behind closed doors with the individual teacher directly
involved. (B)
5.5.2. IF the teacher shows an area of strength, THEN begin by discussing that
area of strength in order to build the teacher’s confidence and open a door
to address the areas where the teacher needs to improve. (C)
5.5.3. Attempt to discover what may have led to this teacher’s actions or lack of
action, and provide support and coaching to enable the teacher to improve
in order to not repeat the same behavior. (C)
5.5.4. Be honest and use observation data collected during the informal walk-
throughs when addressing areas that need improvement. (B)
INFORMAL CLASSROOM WALK-THROUGHS 143
5.6. IF a student is “pushing a teacher’s buttons,” THEN discuss the situation with the
teacher AND brainstorm strategies for dealing with students with challenging
behaviors, reacting appropriately, and recognizing triggers. (C)
5.7. IF you are in a classroom for longer than five to seven minutes, THEN inform the
teacher what captured your interest and why you stayed longer than normal. (B)
5.8. IF you observed exemplary implementation of your expectations, THEN send an
email, give the teacher positive feedback when passing him/her in the hallway, or
give a short personalized note (the more preferred method) to the teacher
specifically noting the teacher’s effective use of strategies AND how they relate
back to your expectations. (A, B, C)
5.8.1. IF you are sending an email AND you took a picture in the classroom,
THEN attach that picture to the email as evidence of excellent
implementation of your expectations. (B).
5.9. IF you observed above average implementation of your expectations AND you
have provided positive feedback to the teacher recently, THEN provide feedback
to the teacher if/when you pass him/her in the hallway. (A)
5.10. IF you observed average implementation of your expectations AND the teacher
showed areas that need small modifications related to your expectations THEN
have an informal conversation with the teacher in the hallway, his/her classroom,
or through email AND provide encouragement to try something new or different
in order to improve his/her skills related to your expectations. (C)
5.10.1 IF you decide to provide the teacher with feedback in the hallway or in
his/her classroom, THEN without students present ask the teacher one or
two reflective questions designed to help the teacher reflect on his/her
practices related to your expectations AND do not expect an answer. (A,
C)
5.10.2. IF you decide to provide the teacher with feedback through email, THEN
provide specific suggestions in your email that you think would help the
teacher more effectively meet your expectations OR reflective questions
(no more than two) for the teacher to consider AND do not expect a
response from the teacher. (A, C)
5.11. IF you observed a poor implementation of your expectations, THEN give the
teacher a written note and send an email to the teacher to meet with you after
school within the next 24 hours in your office AND schedule the meeting on your
calendar. (C)
5.11.1. Hold a meeting with the teacher that is positive and supportive in nature
using a conversational style of communicating to explain what you
observed in the classroom; discuss your expectations with the teacher; and
establish a timeline for support, coaching, and implementing your
expectations. (A, C)
5.11.2. Provide the teacher with coaching and individual professional
development in the areas where it is needed. (C)
5.11.3. IF operational issues (library time, special education mainstreaming, etc.)
are impacting instruction, THEN find a solution to make changes that will
maximize the best use of students’ time in the classroom. (C)
INFORMAL CLASSROOM WALK-THROUGHS 144
5.11.4. IF multiple students indicated that they were having difficulty
understanding the material that was being presented in the lesson (Step
4.10.2.3) OR they indicated that they did not understand the objective of
the activity (Step 4.11.1.2), THEN ask the teacher how he/she will provide
support or reteach the students who do not understand. (C)
5.11.5. Tell the teacher, you are going to come back to walk-through the
classroom again and suggest that the teacher invite you into the classroom
the next time he/she will be providing a lesson that he/she would like you
to see, demonstrating that the teacher is attempting to improve his/her
practice. (A)
5.11.6. IF the teacher is showing a repeated pattern of poor performance AND
attempts to coach and assist the teacher are not changing behavior, THEN
save the emails with this observational information to note reoccurring
patterns (positive or negative) or behaviors that are not changing for
documentation to begin the progressive discipline process. (A)
5.12. IF a teacher was identified to receive feedback in Step 3.1.5 or 3.2.7, THEN
ensure that teacher receives the feedback AND reflect on how the feedback was
received in order to plan next steps. (A)
5.13. IF a teacher has been placed on a Plan for Improvement based on a history of poor
performance AND is implementing your expectations in an average to above
average manner, THEN hold a conference with the teacher to reinforce the
positive behavior observed during the informal walk-through observations and
encourage more of this behavior. (C)
5.14. If a teacher has been placed on a Plan for Improvement based on a history of poor
performance AND is not implementing your expectations, THEN give the teacher
a written note and send an email to the teacher to meet with you after school that
day in your office with a union representative AND provide a written disciplinary
document to the teacher. (C)
5.15. IF the informal walk-through observation data does not match what has been
observed during formal observations, THEN discuss this disparity between the
two pieces of observation data with the teacher. (A, C)
5.15.1. IF the teacher’s performance during the formal observation was not up to
the same level as the informal walk-through data AND the teacher
communicates that there were extenuating circumstances (i.e. personal
tragedy, medical issues, higher than normal stressors, etc.), THEN permit
the teacher to redo his/her formal observation. (C)
Procedure 6. Provide feedback to the entire teaching staff.
6.1. Avoid providing negative feedback to the entire staff when feedback should be
directed toward specific staff members. (B)
6.2. Provide positive feedback publically to the entire staff through the weekly bulletin
(Monday Morning Message) or at the staff meeting without using the specific
INFORMAL CLASSROOM WALK-THROUGHS 145
teacher’s name to reinforce your expectations and to enable multiple staff
members to potentially recognize that you are complimenting them. (A)
6.3. IF you have taken picture(s) or video(s) in individual teacher’s classrooms AND
he/she has given you permission to show the picture(s) or video(s) to the entire
staff, THEN use the picture(s) or video(s) to reinforce the positive example of the
implementation of your expectations. (B, C)
Procedure 7. Communicate with the Leadership Team and plan next steps.
7.1. Meet regularly (at least monthly) with your Leadership Team (Staff members at
the school who are respected by their peers and recognized as understanding and
delivering quality instruction) to discuss the schools’ current progress toward
meeting your expectations and new ideas, techniques, or methods that may be
introduced to the staff through professional development. (A, B, C)
7.1.1. IF informal walk-through data OR school assessment data OR feedback
from the Leadership Team indicates that more than 50% of the teachers
are not effectively implementing your expectations, THEN provide
ongoing professional development for this specific expectation. (B, C)
7.1.2. IF informal walk-through data OR school assessment data OR feedback
from the Leadership Team indicates that less than 50% of the teachers are
not effectively implementing your expectations, THEN meet with the
teachers not effectively implementing your expectations AND establish an
individualized professional development plan which may include one of
the following strategies: (C)
7.1.2.1. Release the teacher from his/her daily teaching duties with a
substitute during the instructional day to observe another
teacher who implements your expectations well based on what
you have observed during your informal walk-through
observations. (C)
7.1.2.2. Release the teacher from his/her daily teaching duties with a
substitute during the instructional day to walk-through
classrooms with you or an instructional coach to see examples
of your expectations in application. (B, C)
7.1.2.3. Schedule a one-on-one meeting with the teacher and you or an
instructional coach to review your expectations and develop an
implementation plan. (B, C)
7.1.2.4. Send the teacher to a professional development workshop
outside the school specific to your expectations. (C)
7.1.2.5. Use teacher preapproved pictures and videos collected during
your informal walk-through observations to give the teacher
examples from your own staff of how to implement your
expectations. (B, C)
7.1.2.5.1. Refrain from showing a picture or video from a
teacher’s classroom to anyone else unless
permission is obtained from the teacher. (B)
7.1.3. Discuss and plan the next steps for new professional development related
to demonstrating greater mastery of your expectations. (B, C)
INFORMAL CLASSROOM WALK-THROUGHS 146
7.1.3.1. Identify patterns of teacher performance related to the elements
of your expectations by teacher, grade level, or school that will
be used in Step 7.1.3. (B, C)
7.1.4. Provide professional development in Step 2.5. (A, B, C)
7.1.4.1. STANDARD: Remember not to move too quickly and assume
that since you are moving to new professional development
that teachers have completely mastered your expectations. (B)
Procedure 8. Facilitate personal reflection.
8.1. Use recurring trends, informal walk-through observation data, the content of the
notes and emails you have written, and the input of the Leadership Team for
personal reflection on where the school has come from, where the school is
headed, and how you as a leader will help the school continue to move forward by
providing greater support to the teachers. (B)
8.1.1. Discuss recurring trends with the assistant superintendent of instruction to
see if your observations and reflections are a larger district-wide issue
which requires district-wide professional development. (B)
INFORMAL CLASSROOM WALK-THROUGHS 147
Appendix F
Incremental Coding Spreadsheets
Spreadsheet Analysis: Gold Standard Protocol Procedures, Action and Decision Steps
Subject Matter Expert Section
Steps
Step Type Final Gold Standard Protocol
Analysis
A B C A D
Procedure 1. Build rapport,
relationships, and trust.
12 1
1 D 1.1. IF you are a new principal to a
school OR there are new staff
members at the school, THEN
contact each staff member
individually by telephone OR in
person before the school year starts
to welcome him/ her to a new school
year and introduce yourself as the
principal. (B)
0 1 0
2 A 1.2. Starting with Step 1.1 and prior
to the start of the school year and
throughout the school year, build
rapport, relationships, credibility,
and trust with the teaching staff
through: (B, C)
0 1 1
3 A 1.2.1. Conducting staff meetings in
which at least one teacher shares
how he/she has implemented a
component of your expectations
(Premier Instruction, OWL, non-
negotiables, over-arching focus area,
or other instructional practices). (B,
C)
0 1 1
4 A 1.2.2. Attending social events. 0 1 0
5 A 1.2.3. Making one-on-one
connections with teachers. (B)
0 1 0
6 A 1.2.4. Recognizing teachers’
individual events (birthdays,
anniversaries, achievements, etc.).
(B)
0 1 0
7 A 1.2.5. Celebrating and honoring
holidays. (B)
0 1 0
8 A 1.2.6. Maintaining a positive tone
with the staff focusing on what is
going well by providing support and
0 1 1
INFORMAL CLASSROOM WALK-THROUGHS 148
feedback at least twice as frequently
as constructive feedback. (B, C)
9 A 1.2.7. Writing individualized notes
providing positive feedback. (B)
0 1 0
10 A 1.2.8. Maintaining open and frequent
communication with the teachers
about what you are seeing during
your walk-throughs. (C)
0 0 1
11 A 1.2.9. Being honest about what you
observe and sharing your
observations with the teachers
frequently. (C)
0 0 1
12 A 1.2.10. Validating the teachers as
professionals by asking questions,
listening, and mirroring back their
words. (C)
0 0 1
13 A 1.2.11. Using directive or telling
language only in cases where all
other strategies are not changing a
teacher’s practice. (C)
0 0 1
Procedure 2. Set clear
expectations.
26 5
14 A 2.1. Communicate orally and in
writing what your expectations are
for the informal classroom walk-
throughs (Premier Instruction, OWL,
non-negotiables, over-arching focus
area, or other instructional practices).
(A, B)
1 1 0
2.1.1. STANDARD: These strategies
will likely change every year as the
school improves and the proficiency
level of the staff increases, and
remember to establish realistic
expectations based on the staff’s
current proficiency level. (B)
15 A 2.2. Meet with the Leadership Team
(staff members at the school who are
respected by their peers and
recognized as understanding and
delivering quality instruction)
between the beginning of the school
year and the first staff meeting of the
school year to discuss the informal
walk-through observation process.
(B)
0 1 0
INFORMAL CLASSROOM WALK-THROUGHS 149
16 A 2.2.1. Reinforce that as the
instructional leader you will be
walking through the classrooms at
different times during the school day
to provide support and help to the
teachers. (B, C)
0 1 1
17 A 2.2.2. Develop the agenda for the
first staff meeting of the school year
with the Leadership Team which
includes informal walk-through
observations as one of the agenda
items. (B)
0 1 0
18 A 2.3. Meet with the office staff and
explain the informal walk-through
process so they understand that you
should not be interrupted during
walk-throughs unless it is an
emergency. (B)
0 1 0
19 A 2.4. At the first staff meeting at the
beginning of the school year, tell the
entire teaching staff what your
expectations (Premier Instruction,
OWL, non-negotiables, over-arching
focus area, or other instructional
practices) are when you walk-
through the classrooms and provide
them with these expectations in
writing from Step 1.2. (A, B, C)
1 1 1
20 D 2.4.1. IF most of your teachers are
returning teachers AND they know
your expectations (Premier
Instruction, OWL, non-negotiables,
over-arching focus area, or other
instructional practices) AND have
heard you explain them before,
THEN keep the explanation of your
expectations more general. (A)
1 0 0
21 D 2.4.2. IF you have a large number of
new teachers or transferring teachers,
THEN discuss your expectations
(Premier Instruction, OWL, non-
negotiables, over-arching focus area,
or other instructional practices) in
detail. (A, B)
1 1 0
22 A 2.4.3. Emphasize with the teaching
staff your role as instructional leader
0 1 0
INFORMAL CLASSROOM WALK-THROUGHS 150
is to provide support and help to the
teachers. (B)
23 A 2.4.3.1. Open and close the agenda
item with this emphasis because due
to “primacy/recency” teachers will
best remember the first and the last
items you discuss of an agenda item.
(B)
0 1 0
24 A 2.4.4. Ensure the teachers understand
that you will be walking through the
classrooms frequently to identify and
support your expectations (Premier
Instruction, OWL, non-negotiables,
over-arching focus area, or other
instructional practices). (B)
0 1 0
25 A 2.4.5. Set the norms with your
teachers for the walk-throughs: (C)
0 0 1
26 A 2.4.5.1. Tell the teachers when you
walk in the classroom to continue
teaching and do not stop instruction
to recognize your presence. (C)
0 0 1
27 A 2.4.5.2. Tell the teachers that you
will be looking for specific evidence
that your expectations (Premier
Instruction, OWL, non-negotiables,
over-arching focus area, or other
instructional practices) are being
addressed. (C)
0 0 1
28 A 2.4.5.3. Reinforce with the teachers
that the informal walk-through is just
a snapshot and not intended to be an
observation of an entire lesson. (C)
0 0 1
29 A 2.4.5.4. Stress with the teachers that
this informal walk-through is not
evaluative and is solely for the
purpose of looking at the progress
towards implementing your
expectations (Premier Instruction,
OWL, non-negotiables, over-arching
focus area, or other instructional
practices). (C)
0 0 1
30 A 2.4.5.5. Differentiate informal walk-
throughs from formal observations
by not conducting informal walk-
through during the time of the year
when formal observations occur. (C)
0 0 1
INFORMAL CLASSROOM WALK-THROUGHS 151
31 A 2.5. Use Leadership Team meetings
and staff meetings throughout the
school year to reinforce and provide
professional development on
techniques, methods, or ideas related
to your expectations (Premier
Instruction, OWL, non-negotiables,
over-arching focus area, or other
instructional practices). (A, B, C)
1 1 1
32 A 2.5.1. Instruct staff so they
understand and identify the
difference between lessons and
activities. (B)
0 1 0
33 A 2.5.2. Develop a common language
related to your expectations (Premier
Instruction, OWL, non-negotiables,
over-arching focus area, or other
instructional practices). (C)
0 0 1
34 A 2.5.3. Build a common
understanding or what each element
of your expectations (Premier
Instruction, OWL, non-negotiables,
over-arching focus area, or other
instructional practices) should look
like when used in the classroom. (C)
0 0 1
35 A 2.5.4. Tell the staff that this is an
ongoing process of continuous
learning that will continue to develop
and change based on the students
assigned to the teachers’ classrooms,
the teachers’ professional growth,
and the school’s achievement data.
(B)
0 1 0
2.5.5. STANDARD: All professional
development is aligned with the
Board’s goals and research-based.
(B)
36 D 2.6. IF you have new teachers on
your staff, THEN meet with the new
teachers every month at a separate
meeting to develop a clear
understanding and common
vocabulary regarding your
expectations (Premier Instruction,
OWL, non-negotiables, over-arching
focus area, or other instructional
1 0 0
INFORMAL CLASSROOM WALK-THROUGHS 152
practices) during your informal
walk-throughs so the new teachers
will clearly understand what you are
talking about when you provide
feedback after your informal walk-
throughs. (A)
37 A 2.6.1. Provide new teachers with
clear examples of the types of
positive feedback or suggestions that
you will provide and what they will
sound like so they are not surprised
when they receive them or left
wondering what the feedback sounds
like. (A)
1 0 0
38 A 2.6.2. Set a clear expectation that
teachers will know where they stand
and receive clear feedback from you
on their performance. (A)
1 0 0
39 D 2.7. IF there are multiple
administrators at the site who will be
conducting informal classroom walk-
through observations, THEN ensure
all administrators hold the same
expectations when they conduct their
informal walk-throughs. (A)
1 0 0
40 A 2.8. Communicate with the staff at
the regularly scheduled staff meeting
what component of your
expectations (Premier Instruction,
OWL, non-negotiables, over-arching
focus area, or other instructional
practices) you will look for during
the informal walk-throughs for each
subsequent period between staff
meetings. (C)
0 0 1
41 D 2.8.1. IF notes will be taken while
you are inside the classroom during
the period of time between staff
meetings, THEN inform teachers in
advance at the staff meeting that this
will be taking place AND exactly
what type of information will be
recorded in the notes (i.e. tallying,
scripting, asking questions, etc.)
AND provide a copy of any form
that will be used to record these
0 0 1
INFORMAL CLASSROOM WALK-THROUGHS 153
notes so staff may provide feedback
and offer ideas for editing the form.
(C)
42 A 2.9. Create and email a brief weekly
newsletter (Monday Morning
Message) to all school staff, key
district office staff, select community
members, and some retirees that is
two or three paragraphs long and
reinforces and reviews a component
of your expectations (Premier
Instruction, OWL, non-negotiables,
over-arching focus area, or other
instructional practices) among other
school related topics. (A)
1 0 0
43 A 2.10. Convey the same message
regarding instructional leadership
and informal classroom walk-
throughs shared with the staff in Step
1.5.5 through 1.5.5.5 with various
parent and community groups
throughout the school year so they
will understand your expectations
(Premier Instruction, OWL, non-
negotiables, over-arching focus area,
or other instructional practices) and
the informal walk-through process.
(B)
0 1 0
44 A 2.11. Anticipate push-back from the
teacher’s union due to increased
expectations, claims that the
informal walk-through process is
evaluative, and addition of extra
duties to the teachers’ workloads. (B)
0 1 0
Procedure 3. Plan for the walk-
throughs.
9 24
45 D 3.1(a). IF you are the only
administrator assigned to the school,
THEN proceed to Step 3.2. (A)
3.1(b). IF there is more than one
administrator assigned to the school,
THEN schedule and hold a weekly
administrator’s meeting. (A)
1 0 0
46 A 3.1.1. Divide the campus evenly
among the administrative team so
every week at least one administrator
1 0 0
INFORMAL CLASSROOM WALK-THROUGHS 154
is in every classroom. (A)
47 A 3.1.1.1. Rotate the way the
classrooms are divided so that each
administrator has the opportunity to
see all of the classrooms on the
campus. (A)
1 0 0
48 D 3.1.1.2. IF during the previous week
you did not have the opportunity to
conduct informal walk-throughs in
all or your assigned classrooms,
THEN discuss with the rest of the
administrative team not rotating the
assigned classrooms that week AND
visiting the same set of classrooms
for two weeks in a row. (A, B)
1 1 0
49 D 3.1.1.3. IF a specific teacher, grade
level, or department has requested
you to walk-through their
classrooms, THEN ensure that your
scheduled informal walk-throughs
will enable you to visit those
classrooms, AND even if that
classroom(s) is not in your weekly
rotation of classrooms to visit. (A, B)
1 1 0
50 D 3.1.1.4. IF you had a follow-up
meeting with a teacher as described
in Step 5.11.5, THEN make sure to
visit that classroom during the week,
especially if/when invited by the
teacher, AND even if that classroom
is not in your weekly rotation of
classrooms to visit. (A, B)
1 1 0
51 D 3.1.2. For a high performing teacher,
IF the administrative team has seen
this teacher providing a high quality
lesson or activity that is in the area
where another teacher needs
assistance, THEN ask the high
performing teacher to teach a model
lesson OR support another teacher
on learning how to better deliver the
lesson or activity. (A)
1 0 0
52 D 3.1.3. For an average performing
teacher, IF you or another
administrator observed something
during your informal walk-through
1 0 0
INFORMAL CLASSROOM WALK-THROUGHS 155
that was not consistent with the
teacher’s normal performance,
THEN share the concern with the
other administrators to get their input
from their informal observations
AND ensure the administrator who
walks-through that classroom during
the week looks for the same behavior
to determine if there is a pattern of
behavior. (A)
53 A 3.1.4. For a low performing teacher,
discuss what has been observed in
his/her classroom every week and
whether that teacher is improving
toward meeting the expectations
(Premier Instruction, OWL, non-
negotiables, over-arching focus area,
or other instructional practices). (A)
1 0 0
54 D 3.1.5. IF the administrative team
thinks a specific teacher who is being
discussed needs feedback to meet the
expectations (Premier Instruction,
OWL, non-negotiables, over-arching
focus area, or other instructional
practices), THEN determine among
the administrative team what type of
feedback to provide and who will
provide the feedback to the teacher.
(A)
1 0 0
55 D 3.1.6. IF an administrator has
received a parent or student
complaint regarding a teacher,
THEN discuss what the
administrative team has seen during
the informal walk-throughs to
confirm or refute the complaint AND
focus on the teacher’s performance
during the next week’s informal
walk-throughs. (A)
1 0 0
56 D 3.1.7. IF a teacher has sent a greater
than normal number of students to
the office for discipline referrals OR
a greater than normal number
students for detention, THEN discuss
what the administrative team has
seen during the informal walk-
1 0 0
INFORMAL CLASSROOM WALK-THROUGHS 156
throughs to explain the referrals and
detentions AND focus on that
teacher’s performance during the
next week’s informal walk-throughs.
(A)
57 D 3.2. IF you are the only administrator
assigned to the school, THEN
consider the following prior to
conducting your walk-throughs: (A,
B)
1 1 0
58 D 3.2.1. If certain classrooms were not
visited during the last time you
conducted informal walk-throughs,
THEN start with the classrooms that
were not visited last time. (A, B)
1 1 0
59 D 3.2.2. IF a specific teacher or grade
level has requested you to walk-
through their classrooms, THEN
ensure that your scheduled informal
walk-throughs will enable you to
visit that classroom(s). (A, B)
1 1 0
60 D 3.2.3. IF you had a follow-up
meeting with a teacher as described
in Step 5.1.4.2, THEN make sure to
visit that classroom during the week,
especially if /when invited by the
teacher, AND even if that classroom
is not in your scheduled classrooms
to visit that day. (A, B)
1 1 0
61 D 3.2.4. For an average performing
teacher, IF you observe something
during your informal walk-through
that was not consistent with the
teacher’s normal performance,
THEN ensure you look for the same
behavior to determine if there is a
pattern of behavior. (A)
1 0 0
62 A 3.2.5. For a low performing teacher,
think about what has been observed
in his/her classroom every week and
whether that teacher is improving
toward meeting the expectations
(Premier Instruction, OWL, non-
negotiables, over-arching focus area,
or other instructional practices). (A)
1 0 0
63 D 3.2.6. For a high performing teacher, 1 0 0
INFORMAL CLASSROOM WALK-THROUGHS 157
IF you have seen this teacher
providing a high quality lesson or
activity that is in the area where
another teacher needs assistance,
THEN ask the high performing
teacher to teach a model lesson OR
support another teacher on learning
how to better provide the lesson or
activity. (A)
64 D 3.2.7. If you think a specific teacher
who you are reflecting on needs
feedback to meet the expectations
(Premier Instruction, OWL, non-
negotiables, over-arching focus area,
or other instructional practices),
THEN determine what type of
feedback to provide and when to
provide the feedback to the teacher.
(A)
1 0 0
65 D 3.2.8. IF you have received a parent
or student complaint regarding a
teacher, THEN think about what you
have seen during informal walk-
throughs to confirm or refute the
complaint AND focus on that
teacher’s performance during the
next week’s informal walk-throughs.
(A)
1 0 0
66 D 3.2.9. IF a teacher has sent a greater
than normal number of students to
the office for discipline referrals OR
a greater than normal number of
students for detention, THEN think
about what you have seen during
informal walk-throughs to explain
the referrals and detentions AND
focus on that teacher’s performance
during the next week’s informal
walk-throughs. (A)
1 0 0
67 A 3.3. Schedule daily classroom walk-
throughs into your calendar, but do
not tell the staff when you will be
conducting informal walk-throughs
as they are unannounced. (A)
1 0 0
68 D 3.3.1. IF the informal walk-through
observations were conducted at a
0 1 0
INFORMAL CLASSROOM WALK-THROUGHS 158
certain time of day during the prior
observations, THEN try to conduct
them at a different time of day. (B)
69 A 3.4. Determine the focus from your
expectations (Premier Instruction,
OWL, non-negotiables, over-arching
focus area, or other instructional
practices) for the day’s informal
classroom walk-throughs based on
what was communicated with the
teachers in Step 2.8 and what you
have determined to be the highest
priority based on student
achievement in Steps 3.1 or 3.2.(B,
C)
0 1 1
70 A 3.4.1. Based on the time of the
school year, select a part of your
expectations (Premier Instruction,
OWL, non-negotiables, over-arching
focus area, or other instructional
practices) that are appropriate for the
time of year (i.e. beginning of the
year, establishing routines and
patterns; middle of the year,
maximizing learning and building on
previous learning; etc.). (B)
0 1 0
71 A 3.5. Select the number of classrooms
to walk-through and how long to stay
in each classroom based on time
available to conduct walk-throughs
and focus from your expectations
(Premier Instruction, OWL, non-
negotiables, over-arching focus area,
or other instructional practices). (C)
0 0 1
72 D 3.6. If the informal walk-through
observation requires specific data
that has to be recorded in the
classroom in the moment in a
specific way, THEN prepare to
record the data according to the
manner that was indicated to the staff
in Step 2.8.1. (C)
0 0 1
73 D 3.6.1. If a specific form or note-
taking aide will be used, THEN
make sufficient copies of the form or
note-taking aide. (C)
0 0 1
INFORMAL CLASSROOM WALK-THROUGHS 159
74 D 3.7. IF notes will not be taken while
in the classroom, THEN prepare to
take notes outside the classroom
after conducting the informal walk-
through observation by carrying a
clipboard with paper for taking notes
OR using your cell phone or IPad
and sending yourself an email with
your notes. (A, B, C)
1 1 1
75 A 3.8. Inform the office staff that you
will be conducting informal walk-
throughs and for what length of time.
(B)
0 1 0
76 D 3.8.1. IF you have been interrupted
over the past one or two informal
walk-throughs for non-emergency
situations, THEN remind the office
staff that this time is dedicated to
informal walk-throughs and you
should only be interrupted in an
emergency. (B)
0 1 0
Procedure 4. Observe the teacher
in the classroom.
36 30
77 A 4.1. Open the door and walk into the
classroom. (A, B, C)
1 1 1
78 D 4.1.1. IF there are multiple entrances
into a classroom, THEN use a
different entrance and exit for each
informal classroom walk-through.
(B)
0 1 0
79 A 4.2. Make eye contact with the
teacher and smile so that the teacher
will not think you are trying to be
sneaky and to acknowledge that this
is an informal walk-through. (A, C)
1 0 1
80 D 4.3. IF you will be taking notes in the
classroom during the informal walk-
through, THEN begin to note
observable data and write it down
once observed. (C)
0 0 1
81 A 4.4. Do not talk to the teacher or
interrupt what the teacher is doing.
(A, C)
1 0 1
82 D 4.4.1. IF you need to talk to the
teacher, THEN move to the front of
the classroom AND keep eye contact
1 0 0
INFORMAL CLASSROOM WALK-THROUGHS 160
AND wait for the teacher to ask if
you need something AND depending
on the content of the conversation
talk to the teacher either inside or
outside of the classroom. (A)
83 A 4.4.2. Keep a “poker face” and
neutral demeanor while in the
classroom. (B)
0 1 0
84 A 4.5. Walk to the back of the
classroom or where you can observe
the most from one location but be as
least disruptive as possible. (A, B)
1 1 0
85 A 4.5.1. Move throughout the
classroom and observe from multiple
vantage points. (B)
0 1 0
86 A 4.6. Look around the entire
classroom taking in all four walls,
the students, how the room is
organized, what is being presented
and displayed, and the teacher. (A,
B)
1 1 0
87 A 4.6.1. Evaluate if the items displayed
are accessible and available for
student use, preferably student work
product. (A)
1 0 0
88 A 4.7. Scan the front of the classroom
for a posted daily agenda and
identify that the lesson or activity
matches what the teacher has posted
on the daily agenda. (A, B)
1 1 0
89 A 4.8. Look at what is projected,
written on the board, or being given
to the students for evidence of the
objective. (A, B)
1 1 0
90 A 4.9. Observe the students, where the
action is taking place in the
classroom, and look at what the
students are doing. (A, C)
1 0 1
91 A 4.10. Observe whether the students
are participating in a lesson or an
activity and identify if what the
students are doing matches the
objective. (A, C)
1 0 1
92 A 4.10.1. Identify whether it is an
instructional day or activity day by
trying to determine where the teacher
1 0 1
INFORMAL CLASSROOM WALK-THROUGHS 161
is in the instructional process. (A, C)
93 D 4.10.2. IF it is an instructional day,
THEN the teacher is teaching a
lesson. (C)
0 0 1
94 D 4.10.2.1. IF the teacher is teaching a
lesson, THEN look for the objective
of the lesson to be posted at the front
of the classroom in student friendly
language AND identify if the
instruction is enabling the students to
reach the objective. (C)
0 0 1
95 D 4.10.2.2. IF you determine that
students are meeting the posted
objective, THEN do not proceed to
STEP 4.11 indicating to the teacher
that you recognize the lesson is
going well and you are observing
instruction that meets the posted
objective. (C)
0 0 1
96 D 4.10.2.3. IF the teacher is teaching a
lesson AND no objective is posted,
THEN remember this information to
record in your notes AND to provide
feedback to the teacher. (C)
0 0 1
97 D 4.10.3. IF you cannot determine
where the teacher is in the
instructional process within
approximately five minutes, THEN it
is an activity day. (A)
1 0 0
98 D 4.10.3.1. IF it is an activity day OR
you cannot determine the objective
of the lesson, THEN move to Step
4.11. (C)
0 0 1
99 D 4.11. IF it will not interrupt a
student’s learning or a group activity,
unobtrusively speak with a student
and ask them what they are learning
today and why. (A, B, C)
1 1 1
100 D 4.11.1. IF the student cannot tell you
what he/she is learning OR the task
does not match the objective, THEN
unobtrusively ask another student
what they are learning today AND
keep asking additional students until
you get an answer. (A, B, C)
1 1 1
101 A 4.11.1.1. Remember the names of the 0 1 0
INFORMAL CLASSROOM WALK-THROUGHS 162
students or where they are sitting to
discuss with the teacher during
feedback. (B)
102 D 4.11.1.2. IF multiple students do not
know the objective OR the task does
not match the objective AND the
students are not learning the intended
materials, THEN remember to record
this information in your notes AND
remember to send the teacher an
email outside of the classroom or
give the teacher a hand written note
to see you after school in order to
discuss the observation. (B, C)
0 1 1
103 D 4.11.2. IF the student understands the
objective and the task matches the
objective, THEN continue the
informal observation AND speaking
to additional student(s) is optional.
(B, C)
0 1 1
104 A 4.11.2.1. While speaking with the
student, observe what work the
student is producing and determine if
it matches your expectations for that
portion of the lesson. (A)
1 0 0
105 A 4.12. Look for student engagement
by determining what the students are
doing and/or producing and if it is
connected to what the teacher
intends for them to learn/do, in other
words, the activities match the
objective. (A)
1 0 0
106 A 4.13. Observe the balance between
students working independently and
depending upon their peers. (C)
0 0 1
107 D 4.14. IF students are working in
cooperative groups, THEN
distinguish whether or not all
students understand the materials
AND whether students could serve
in any role required to complete the
task. (C)
0 0 1
108 A 4.15. Observe the teacher’s actions.
(A, C)
1 0 1
109 A 4.16. Look for evidence of your
expectations (Premier Instruction,
1 0 1
INFORMAL CLASSROOM WALK-THROUGHS 163
OWL, non-negotiables, over-arching
focus area, or other instructional
practices). (A, C)
110 A 4.17. Observe classroom
management and discipline issues.
(C)
0 0 1
111 A 4.18. Observe the rapport and
positive classroom environment that
is being built between the teacher
and the students by looking for the
following: (C)
0 0 1
112 A 4.18.1. Distinguish if students feel
safe and supported to take risks and
try new experiences by observing if
they will answer questions when
they are not completely certain of the
answer. (C)
0 0 1
113 A 4.18.2. Observe which students are
asked to answer questions and how
the teacher responds to the students
in order to see if even incorrect
responses are received positively
(not using terms such as “wrong
answer”), questions are open-ended
versus “yes/no,” and students do not
feel ridiculed. (C)
0 0 1
114 A 4.18.3. Identify the teacher’s style
for interacting with the students (i.e.
humor, caring, etc.) and look for
visual cues of how the students
respond to this interaction. (C)
0 0 1
115 A 4.18.4. Identify any students who are
“pushing a teacher’s buttons” and
remember these occurrences to be
recorded in your notes outside the
classroom for later discussion with
the teacher noting the triggers for the
students’ behavior and the teacher’s
response(s). (C)
0 0 1
116 A 4.18.5. Identify students who are
responding with “I don’t know,” or
shrugging their shoulders and how
the teacher encourages their
participation. (C)
0 0 1
117 A 4.18.6. Observe students who do not
seem to understand the instruction or
0 0 1
INFORMAL CLASSROOM WALK-THROUGHS 164
the activity and how the teacher
encourages or supports them. (C)
118 A 4.18.7. Look for students who do not
raise their hands or try to answer
question to see if the teacher
involves them and encourages their
participation. (C)
0 0 1
119 A 4.18.8. Observe students’ body
language and how the teacher reacts
to the different types of body
language in the classroom. (C)
0 0 1
120 A 4.18.9. Identify how and what
students use in the room
environment as a tool to assist them
with learning during lessons and
activities. (C)
0 0 1
121 A 4.18.10. Notice how long the teacher
waits between asking a question and
calling on a student to answer (wait
time) and how that impacts the
number of students volunteering to
answer questions. (C)
0 0 1
122 D 4.19. IF you identify students in the
classroom who are identified for
intervention or specialized
instruction (Special Education
students, GATE students, etc.),
THEN identify the strategies
(preview/review, collaborative
groups, total physical response,
sentence frames, depth and
complexity icons, etc.) that the
teacher is using and how they are
helping the identified students. (D)
0 0 0
123 D 4.19.1. IF the teacher is not using
any differentiated strategies to assist
students identified for intervention or
specialized instruction OR the
strategies are not being used
appropriately, THEN note the
information and remember the
information to record in your notes
once outside the classroom. (D)
0 0 0
124 A 4.20. Look for the use of classroom
technology. (C)
0 0 1
125 D 4.20.1. IF classroom technology is 0 0 1
INFORMAL CLASSROOM WALK-THROUGHS 165
being used, THEN determine the
level and type of usage AND
remember the information to record
in your notes once outside the
classroom. (C)
126 A 4.20.2. Ask the student about the
technology to determine if it is
enhancing their learning, working
appropriately, and creating minimal
distractions. (C)
0 0 1
127 D 4.21. IF the teacher has been placed
on a Plan for Improvement based on
a history of performance problems,
THEN observe whether the areas
addressed in the Plan for
Improvement are being implemented
in the classroom AND note the
deficiencies and improvements AND
provide feedback in Steps 5.13 and
5.14. (C)
0 0 1
128 D 4.22. IF you observe an extreme
situation in the classroom in which
student are at risk of harm, there is
extreme misbehavior, there is a
violation of school or district rules,
or the teacher is presenting
information inaccurately so the
students will learn the information
incorrectly, THEN discreetly
approach the teacher and inform the
teacher that there is a serious
situation which needs to be
addressed immediately. (B)
0 1 0
129 D 4.23. IF there is exemplary
implementation of one or more of
your expectations (Premier
Instruction, OWL, non-negotiables,
over-arching focus area, or other
instructional practices), THEN ask
the teacher if you can take a picture
or video AND use you r IPad or your
phone to take a picture to provide
positive feedback to the
teacher/grade level OR present to the
entire staff during a staff meeting.
(B, C)
0 1 1
INFORMAL CLASSROOM WALK-THROUGHS 166
130 A 4.24. Observe to see if there are any
parents in the classroom. (C)
0 0 1
131 D 4.24.1. If parent volunteers are in the
classroom, THEN acknowledge their
presence by smiling, making eye
contact, and speaking with them in
an unobtrusive manner to thank them
for volunteering. (C)
0 0 1
132 D 4.24.2. IF parent volunteers are in
the classroom and the lesson or
activity is not going smoothly,
THEN speak to the parent in an
unobtrusive and non-judgmental
manner letting the parent know that
you have recognized the current
classroom condition but are
supporting the teacher’s
professionalism with a phrase such
as, “It looks like (Student Name)
may be having a difficult day and
(Teacher Name) has his/her hands
full. He/she is lucky to have such
dedicated parent volunteers like
you.” (C)
0 0 1
133 A 4.25. Spend at least 5 minutes but
not more than 15 minutes in each
classroom. (A, B)
1 1 0
134 D 4.25.1. IF you have gathered data
from observing where the teacher is
in the instructional process, looking
at teacher behaviors, talking to a
student(s), looking at what a
student(s) is producing, and looking
at the room environment; THEN
prepare to exit the classroom. (A)
1 0 0
135 A 4.26. Prior to leaving the classroom,
double-check the data that you have
from your informal walk-through
and decide whether or not to provide
feedback to the teacher based on
specific observations from the
classroom that have the highest
possibility of affecting student
learning and remember specific
information to note once outside of
the classroom. (A, B, C)
1 1 1
INFORMAL CLASSROOM WALK-THROUGHS 167
136 D 4.26.1. If you will not be providing
the teacher with feedback from your
informal walk-through because the
teacher is meeting your expectations,
THEN unobtrusively get the
teacher’s attention and give him/her
a “thumbs-up” or a nod on your way
out the door AND remember to send
an email to yourself or take written
notes when you are outside the
classroom for you own notes or
journaling. (A, B)
1 1 0
137 D 4.26.2. IF you will be providing
positive feedback to the teacher,
THEN decide what positive
information you will share with the
teacher from what you have
observed AND what the next
suggested steps are for the teacher to
work on AND remember to send
yourself an email or take written
notes when you are outside the
classroom for your own notes or
journaling. (B)
0 1 0
138 D 4.26.3. IF you will be providing
suggestions or asking reflective
questions of the teacher based on a
practice that needs improvement,
THEN unobtrusively get the
teacher’s attention and give him/her
a “thumbs-up” or a nod on your way
out of the door AND remember to
send an email to yourself or take
written notes with a suggestion or
reflective question when you are
outside the classroom. (A, B)
1 1 0
139 D 4.26.4. IF you will be providing
feedback that is critical or urgent,
THEN make eye contact with the
teacher when you exit the classroom
if you are able BUT do not give a
“thumbs-up” AND remember to send
the teacher an email or hand the
teacher a written note to see you in
your office after school. (A)
1 0 0
4.27. Exit the classroom if one or 1 1 1
INFORMAL CLASSROOM WALK-THROUGHS 168
more of the following conditions
occur: (A, B, C)
140 D 4.27.1. IF you have collected enough
data to make a decision about
providing follow-up information to
the teacher, THEN exit the
classroom. (A, B, C)
1 1 1
141 D 4.27.2. IF you have been called on
the radio and are needed elsewhere,
THEN exit the classroom. (C)
0 0 1
142 D 4.27.3. IF the classroom environment
is uncomfortable for the teacher or
the students are having an awkward
moment due to what is happening in
the classroom, THEN exit the
classroom. (C)
0 0 1
143 D 4.27.4. IF it is not a dangerous or
harmful situation, THEN exit the
classroom. (C)
0 0 1
Procedure 5. Provide feedback to
the individual teachers.
144 A 5.1. Once outside the classroom, use
paper to take notes or use your IPad
or cell phone to send a short email
(2-3 sentences) that will be for your
reference, which will include: what
was observed, what needs to be
reinforced with the teacher, how it
will be reinforced, what support will
be provided through professional
development and coaching, what
positive feedback will be provided to
the individual or entire staff, and
what connections should be made
among the staff for collaborative
support. (A, B, C)
1 1 1
145 D 5.1.1. IF there is more than one
administrator assigned to the school
AND you saw, heard, or felt
something that you want to
remember BUT are not sure what
you will do with the information,
THEN share the note or email during
the weekly administrator’s meeting
AND decide what to do with the
information during the next
1 0 0
INFORMAL CLASSROOM WALK-THROUGHS 169
administrator’s meeting (Steps 3.1.2
through 3.1.5). (A)
146 D 5.1.2. IF you are the only
administrator assigned to the school
AND you saw, heard, or felt
something that you want to
remember BUT are not sure what
you will do with the information,
THEN review the notes or email
AND decide what to do with the
information after reflecting on it. (A,
B)
1 1 0
147 D 5.2. IF you will provide feedback to
the teacher, THEN make every effort
to start verbal or written (email)
feedback with something you
observed during your walk-through
of the classroom that was positive.
(A, B)
1 1 0
148 D 5.3. IF you will provide feedback to
the teacher that involves areas of
improvement, THEN offer ways that
you will help the teacher AND show
your willingness to be a resource or
provide resources to work with the
teacher to find solutions so the
feedback is not viewed as a
“gotcha.” (B)
0 1 0
149 A 5.3.1. Ensure that the teaching staff
hears you say and sees you actions
which demonstrate you do not have
all the answers but that you are also a
learner and are willing to work with
them to find answers. (B)
0 1 0
150 A 5.3.2. Based on the time of the
school year, provide feedback to the
teacher from your informal walk-
throughs based on what you would
expect to see at that time of the year
and the amount of stress that teachers
are experiencing. (A)
1 0 0
151 D 5.3.2.1. IF a teacher is under pressure
due to the time of the year OR a
personal situation (illness, family
illness, divorce, death, etc.) AND the
teacher’s performance is not
1 0 0
INFORMAL CLASSROOM WALK-THROUGHS 170
significantly impacting student
learning, THEN write a note or send
an email to yourself AND save the
note or email for a less stressful time
to discuss with the teacher and for
your own journaling. (A)
152 D 5.3.2.2. IF a teacher is under pressure
due to the time of the year OR a
personal issue (illness, family illness,
divorce, death, etc.) AND the
teacher’s performance is
significantly impacting student
learning over more than one day,
THEN give the teacher a note or
send an email to the teacher to come
see you after school and discuss what
can be done to lessen the impact
upon student learning. (A)
1 0 0
153 D 5.4. IF the teacher approaches you in
an informal (hallway) setting for
feedback on what was occurring in
his/her classroom when you walked-
through, THEN try to find a more
private location in which to talk
AND let the teacher do the majority
of the talking AND attempt to build
the teacher’s confidence by asking
reflective questions (no more than 2)
that will help strengthen and refine
the teacher’s strategies to address
your expectations (Premier
Instruction, OWL, non-negotiables,
over-arching focus area, or other
instructional practices). (A, B, C)
1 1 1
154 D 5.4.1. IF the conversation with the
teacher will take more than a few
minutes or is sensitive in nature,
THEN politely ask the teacher to see
you at the end of the school day in
your office in order to give you the
“gift of time” to meet the teacher in
an unrushed manner and develop
your response in advance. (B)
0 1 0
155 D 5.5. IF the informal walk-through
observation results in information
that needs immediate attention, is
1 1 1
INFORMAL CLASSROOM WALK-THROUGHS 171
dangerous or harmful, a safety
hazard, breaks school or district
rules, or is critical or urgent in nature
because it impacts student learning,
THEN give the teacher a written note
and send an email to the teacher to
meet with you after school that day
in your office with a union
representative AND schedule the
meeting on your calendar. (A, B, C)
156 D 5.5.1. IF a negative situation needs to
be discussed with a teacher, THEN
have that discussion behind closed
doors with the individual teacher
directly involved. (B)
0 1 1
157 D 5.5.2. IF the teacher shows an area of
strength, THEN begin by discussing
that area of strength in order to build
the teacher’s confidence and open a
door to address the areas where the
teacher needs to improve. (C)
0 0 1
158 A 5.5.3. Attempt to discover what may
have led to this teacher’s actions or
lack of action, and provide support
and coaching to enable the teacher to
improve in order to not repeat the
same behavior. (C)
0 0 1
159 A 5.5.4. Be honest and use observation
data collected during the informal
walk-throughs when addressing
areas that need improvement. (B)
0 1 0
160 D 5.6. IF a student is “pushing a
teacher’s buttons,” THEN discuss
the situation with the teacher AND
brainstorm strategies for dealing with
students with challenging behaviors,
reacting appropriately, and
recognizing triggers. (C)
0 0 1
161 D 5.7. IF you are in a classroom for
longer than five to seven minutes,
THEN inform the teacher what
captured your interest and why you
stayed longer than normal. (B)
0 1 0
162 D 5.8. IF you observed exemplary
implementation of your expectations,
THEN send an email, give the
1 1 1
INFORMAL CLASSROOM WALK-THROUGHS 172
teacher positive feedback when
passing him/her in the hallway, or
give a short personalized note (the
more preferred method) to the
teacher specifically noting the
teacher’s effective use of strategies
AND how they relate back to your
expectations. (A, B, C)
163 D 5.8.1. IF you are sending an email
AND you took a picture in the
classroom, THEN attach that picture
to the email as evidence of excellent
implementation of your expectations.
(B)
0 1 0
164 D 5.9. IF you observed above average
implementation of your expectations
AND you have provided positive
feedback to the teacher recently,
THEN provide feedback to the
teacher if/when you pass him/her in
the hallway. (A)
1 0 0
165 D 5.10. IF you observed average
implementation of your expectations
AND the teacher showed areas that
need small modifications related to
your expectations THEN have an
informal conversation with the
teacher in the hallway, his/her
classroom, or through email AND
provide encouragement to try
something new or different in order
to improve his/her skills related your
expectations. (C)
0 0 1
166 D 5.10.1. IF you decide to provide the
teacher with feedback in the hallway
or in his/her classroom, THEN
without students present ask the
teacher one or two reflective
questions designed to help the
teacher reflect on his/her practices
related to your expectations AND do
not expect an answer. (A, C)
1 0 1
167 D 5.10.2. IF you decide to provide the
teacher with feedback through email,
THEN provide specific suggestions
in your email that you think would
1 0 1
INFORMAL CLASSROOM WALK-THROUGHS 173
help the teacher more effectively
meet your expectations OR reflective
questions (no more than two) for the
teacher to consider AND do not
expect a response from the teacher.
(A, C)
168 D 5.11. IF you observed a poor
implementation of your expectations,
THEN give the teacher a written note
and send an email to the teacher to
meet with you after school within the
next 24 hours in your office AND
schedule the meeting on your
calendar. (C)
0 0 1
169 A 5.11.1. Hold a meeting with the
teacher that is positive and
supportive in nature using a
conversational style of
communicating to explain what you
observed in the classroom; discuss
your expectations with the teacher;
and establish a timeline for support,
coaching, and implementing your
expectations. (A, C)
1 0 1
170 A 5.11.2. Provide the teacher with
coaching and individual professional
development in the areas where it is
needed. (C)
0 0 1
171 D 5.11.3. IF operational issues (library
time, special education
mainstreaming, etc.) are impacting
instruction, THEN find a solution to
make changes that will maximize the
best use of students’ time in the
classroom. (C)
0 0 1
172 D 5.11.4. IF multiple students indicated
that they were having difficulty
understanding the material that was
being presented in the lesson (Step
4.10.2.3) OR they indicated that they
did not understand the objective of
the activity (Step 4.11.1.2), THEN
ask the teacher how he/she will
provide support or reteach the
materials to the students who do not
understand. (C)
0 0 1
INFORMAL CLASSROOM WALK-THROUGHS 174
173 A 5.11.5. Tell the teacher, you are
going to come back to walk-through
the classroom again and suggest that
the teacher invite you into the
classroom the next time he/she will
be providing a lesson that he/she
would like you to see, demonstrating
that the teacher is attempting to
improve his/her practice. (A)
1 0 0
174 D 5.11.6. IF the teacher is showing a
repeated pattern of poor performance
AND your attempts to coach and
assist the teacher are not changing
behavior, THEN save the emails
with this observational information
to note reoccurring patterns (positive
or negative) or behaviors that are not
changing for documentation to begin
the progressive discipline process.
(A)
1 0 0
175 D 5.12. IF a teacher was identified to
receive feedback in Step 3.1.5 or
3.2.7, THEN ensure that teacher
receives the feedback AND reflect
on how the feedback was received in
order to plan next steps. (A)
1 0 0
176 D 5.13. IF a teacher has been placed on
a Plan for Improvement based on a
history of poor performance AND is
implementing your expectations in
an average to above average manner,
THEN hold a conference with the
teacher to reinforce the positive
behavior observed during the
informal walk-through observations
and encourage more of this behavior.
(C)
0 0 1
177 D 5.14. IF a teacher has been placed on
a Plan for Improvement based on a
history of poor performance AND is
not implementing your expectations,
THEN give the teacher a written note
and send an email to the teacher to
meet with a union representative
AND provide a written disciplinary
document to the teacher. (C)
0 0 1
INFORMAL CLASSROOM WALK-THROUGHS 175
178 D 5.15. IF the informal walk-through
observation data does not match
what has been observed during
formal observations, THEN discuss
this disparity between the two pieces
of observation data with the teacher.
(A, C)
1 0 1
179 D 5.15.1. IF the teacher’s performance
during the formal observation was
not up to the same level as the
informal walk-through data AND the
teacher communicates that there
were extenuating circumstances (i.e.
personal tragedy, medical issues,
higher than normal stressors, etc.),
THEN permit the teacher to redo
his/her formal observation. (C)
0 0 1
Procedure 6. Provide feedback to
the entire teaching staff.
2 1
180 A 6.1. Avoid providing negative
feedback to the entire staff when
feedback should be directed toward
specific staff members. (B)
0 1 0
181 A 6.2. Provide positive feedback
publically to the entire staff through
the weekly bulletin (Monday
Morning Message) or at the staff
meeting without using the specific
teacher’s name to reinforce your
expectations and to enable multiple
staff members to potentially
recognize that you are
complimenting them. (A)
1 0 0
182 D 6.3. IF you have taken a picture(s) or
video(s) in an individual teacher’s
classrooms AND he/she has given
you permission to show the
picture(s) or video(s) to the entire
staff, THEN use the picture(s) or
video(s) to reinforce the positive
example of the implementation of
your expectations. (B, C)
0 1 1
Procedure 7. Communicate with
the Leadership Team and plan
next steps.
183 A 7.1. Meet regularly (at least monthly) 1 1 1
INFORMAL CLASSROOM WALK-THROUGHS 176
with your Leadership Team (Staff
members at the school who are
respected by their peers and
recognized as understanding and
delivering quality instruction) to
discuss the school’s current progress
toward meeting your expectations
and new ideas, techniques, or
methods that may be introduced to
the staff through professional
development. (A, B, C)
184 D 7.1.1. IF informal walk-through data
OR school assessment data OR
feedback from the Leadership Team
indicates that more than 50% of the
teachers are not effectively
implementing your expectations,
THEN provide ongoing professional
development for this specific
expectation. (B, C)
0 1 1
185 D 7.1.2. IF informal walk through data
OR school assessment data OR
feedback from the Leadership Team
indicates that less than 50% of the
teachers are not effectively
implementing your expectations,
THEN meet with the teachers who
are not effectively implementing
your expectations AND establish an
individualized professional
development plan which may include
one of the following strategies: (C)
0 0 1
186 A 7.1.2.1. Release the teacher from
his/her daily teaching duties with a
substitute during the instructional
day to observe another teacher who
implements your expectations well
based on what you have observed
during your informal walk-through
observations. (B, C)
0 1 1
187 A 7.1.2.2. Release the teacher from
his/her daily teaching duties with a
substitute during the instructional
day to walk-through classrooms with
you or an instructional coach to see
examples of your expectations in
0 1 1
INFORMAL CLASSROOM WALK-THROUGHS 177
application. (B, C)
188 A 7.1.2.3. Schedule a one-on-one
meeting with the teacher and you or
an instructional coach to review your
expectations and develop an
implementation plan. (B, C)
0 1 1
189 A 7.1.2.4. Send the teacher to a
professional development workshop
outside the school specific to your
expectations. (C)
0 0 1
190 A 7.1.2.5. Use teacher preapproved
pictures and videos collected during
your informal walk-through
observations to give the teacher
examples from your own staff of
how to implement your expectations.
(B, C)
0 1 1
191 A 7.1.2.5.1. Refrain from showing a
picture or video from a teacher’s
classroom to anyone else unless
permission is obtained from the
teacher. (B)
0 1 0
192 A 7.1.3. Discuss and plan the next steps
for new professional development
related to demonstrating greater
mastery of your expectations. (B, C)
0 1 1
193 A 7.1.3.1. Identify patterns of teacher
performance related to the elements
of your expectations by teacher,
grade level, or school that will be
used in Step 7.1.3. (B, C)
0 1 1
194 A 7.1.4. Provide professional
development in Step 2.5. (A, B, C)
1 1 1
Procedure 8. Facilitate personal
reflection.
2 0
195 A 8.1. Use recurring trends, informal
walk-through observation data, the
content for the notes and emails you
have written, and the input of the
Leadership Team for personal
reflection on the where the school
has come from, where the school is
headed, and how you as a leader will
help the school continue to move
forward by providing greater support
to the teachers. (B)
0 1 0
INFORMAL CLASSROOM WALK-THROUGHS 178
196 A 8.1.1. Discuss recurring trends with
the assistant superintendent of
instruction to see if your
observations and reflections are a
larger district-wide issue which
requires district-wide professional
development. (B)
0 1 0
196 Total Action and Decision Steps 82 84 100 105 91
105 Action Steps 35 49 58
91 Decision Steps 47 35 42
Total Action and Decisions Steps 41.84% 42.86% 51.02%
Action Steps 33.33% 46.67% 55.24%
Decision Steps 51.65% 38.46% 46.15%
Action and Decision Steps Omitted 114 112 96
Action Steps Omitted 70 56 47
Decision Steps Omitted 44 56 49
Action and Decision Steps Omitted 58.16% 57.14% 48.98%
Action Steps Omitted 66.67% 53.33% 44.76%
Decision Steps Omitted 48.35% 61.54% 53.85%
Average Captured Omitted
Total Action and Decision Steps 45.24% 54.79%
Action Steps 45.08% 54.92%
Decision Steps 45.42% 54.58%
Abstract (if available)
Abstract
Informal classroom walk-throughs conducted by school principals with feedback provided to teachers has been demonstrated to improve learning achievement in kindergarten through twelfth grade (K-12) education. Principals are often trained by experts to conduct these walk-throughs. Unfortunately, research shows that experts may omit up to 70% of the critical information needed by trainees to replicate their expertise. The purpose of this study was to capture the knowledge and skills expert K-12 principals use when they conduct informal classroom walk-throughs and provide feedback to teachers. Cognitive Task Analysis (CTA) semi-structured interviews were conducted with three principals who were qualified as experts using both qualitative and quantitative measures. Action and decision steps, as well as standards, equipment, and conceptual knowledge from individual subject matter experts (SMEs) were captured and aggregated into a gold standard protocol which was reviewed by a fourth expert. The study also sought to identify and quantify the number and percentage of expert knowledge and skills omissions when the principals described how they conducted classroom walk-throughs and provided feedback to teachers. Findings indicate that expert principals omitted an average of 54.76% of the action and decision steps when compared to the gold standard protocol. This study extends the potential negative effects of relying on experts for instruction and curriculum development. The expert knowledge and skills captured by CTA methods may be used to train pre-service and in-service principals in performing the complex instructional leadership task of informal walk-throughs and providing feedback to teachers, which may ultimately improve teachers’ classroom instruction and student achievement.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Using cognitive task analysis to capture expert instruction in division of fractions
PDF
Using incremental cognitive task analysis to capture expert instruction in expository writing for secondary students
PDF
Using cognitive task analysis for capturing expert instruction of food safety training for novice employees
PDF
Using cognitive task analysis to capture how expert anesthesia providers conduct an intraoperative patient care handoff
PDF
The use of cognitive task analysis to capture expert instruction in teaching mathematics
PDF
Using individual cognitive task analysis to capture expert writing instruction in expository writing for secondary students
PDF
The use of cognitive task analysis for identifying the critical information omitted when experts describe surgical procedures
PDF
Using cognitive task analysis to capture palliative care physicians' expertise in in-patient shared decision making
PDF
The use of cognitive task analysis to capture expert patient care handoff to the post anesthesia care unit
PDF
The use of cognitive task analysis for the postanesthesia patient care handoff in the intensive care unit
PDF
Using cognitive task analysis to capture expert reading instruction in informational text for students with mild to moderate learning disabilities
PDF
The use of cognitive task analysis to capture exterptise for tracheal extubation training in anesthesiology
PDF
Towards a taxonomy of cognitive task analysis methods: a search for cognition and task analysis interactions
PDF
The effect of cognitive task analysis based instruction on surgical skills expertise and performance
PDF
Using cognitive task analysis to determine the percentage of critical information that experts omit when describing a surgical procedure
PDF
The use of cognitive task analysis to determine surgical expert's awareness of critical decisions required for a surgical procedure
PDF
Employing cognitive task analysis supported instruction to increase medical student and surgical resident performance and self-efficacy
PDF
Identifying the point of diminishing marginal utility for cognitive task analysis surgical subject matter expert interviews
PDF
The use of cognitive task analysis to investigate how many experts must be interviewed to acquire the critical information needed to perform a central venous catheter placement
PDF
Cognitive task analysis for instruction in single-injection ultrasound-guided regional anesthesia
Asset Metadata
Creator
Hammitt, Chad S.
(author)
Core Title
Using cognitive task analysis to capture how expert principals conduct informal classroom walk-throughs and provide feedback to teachers
School
Rossier School of Education
Degree
Doctor of Education
Degree Program
Education (Leadership)
Publication Date
10/30/2014
Defense Date
10/20/2014
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
classroom walk-throughs,cognitive task analysis,feedback to teachers,informal classroom walk-throughs,OAI-PMH Harvest
Format
application/pdf
(imt)
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Yates, Kenneth A. (
committee chair
), Lucid, Joan (
committee member
), Samkian, Artineh (
committee member
)
Creator Email
hammitt@usc.edu,hammitts@sbcglobal.net
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c3-511509
Unique identifier
UC11297583
Identifier
etd-HammittCha-3048.pdf (filename),usctheses-c3-511509 (legacy record id)
Legacy Identifier
etd-HammittCha-3048.pdf
Dmrecord
511509
Document Type
Dissertation
Format
application/pdf (imt)
Rights
Hammitt, Chad S.
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
classroom walk-throughs
cognitive task analysis
feedback to teachers
informal classroom walk-throughs