Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Building data use capacity through school leaders: an evaluation study
(USC Thesis Other)
Building data use capacity through school leaders: an evaluation study
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
Building Data use Capacity through School Leaders:
An Evaluation Study
by
Daniel Adam Bennett
A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION
December 2020
Copyright 2020 Daniel Adam Bennett
ii
TABLE OF CONTENTS
LIST OF TABLES .......................................................................................................................... v
LIST OF FIGURES ...................................................................................................................... vii
ABSTRACT ................................................................................................................................. viii
CHAPTER ONE: INTRODUCTION ............................................................................................. 1
Introduction of the Problem of Practice .............................................................................. 1
Organizational Context and Mission .................................................................................. 2
Organizational Goal ............................................................................................................ 3
Related Literature................................................................................................................ 3
Importance of the Evaluation .............................................................................................. 4
Description of Stakeholder Groups ..................................................................................... 5
Stakeholders Performance Goals ........................................................................................ 6
Stakeholder Group for the Study ........................................................................................ 6
Purpose of the Project and Questions ................................................................................. 7
Methodological Framework ................................................................................................ 8
Definitions........................................................................................................................... 8
Organization of the Project ................................................................................................. 8
CHAPTER TWO: REVIEW OF THE LITERATURE ................................................................ 10
Introduction ....................................................................................................................... 10
Influences on the Problem of Practice .............................................................................. 10
Student Achievement Improvement Through Data-based Decisions ............................... 10
The Role of School Leaders in Data use Within Their School ......................................... 12
Role of Stakeholder Group of Focus ................................................................................ 14
Clark and Estes’ (2008) Knowledge, Motivation and Organizational Influences
Framework ........................................................................................................................ 14
Stakeholder Knowledge, Motivation and Organizational Influences ............................... 15
Knowledge Influences ...................................................................................................... 15
Motivation Influences ....................................................................................................... 21
Organization Influences .................................................................................................... 26
Conceptual Framework: The Interaction of Stakeholders’ Knowledge, Motivation, and
Organizational Context ..................................................................................................... 29
Conclusion ........................................................................................................................ 32
CHAPTER THREE: METHODS ................................................................................................ 33
Participating Stakeholders ................................................................................................ 34
Survey Sampling Criteria and Rationale........................................................................... 34
Interview Sampling Criteria and Rationale....................................................................... 36
Data Collection and Instrumentation ................................................................................ 38
Surveys .............................................................................................................................. 39
Interviews .......................................................................................................................... 41
Data Analysis .................................................................................................................... 42
iii
Credibility and Trustworthiness ........................................................................................ 43
Validity and Reliability ..................................................................................................... 43
Ethics................................................................................................................................. 44
Limitations and Delimitations ........................................................................................... 47
Conclusion ........................................................................................................................ 48
CHAPTER FOUR: RESULTS AND FINDINGS ........................................................................ 50
Participating Stakeholders ................................................................................................ 51
Determination of Assets and Needs .................................................................................. 53
Results and Findings for Knowledge Causes.................................................................... 54
Factual Knowledge ........................................................................................................... 55
Conceptual Knowledge ..................................................................................................... 58
Procedural Knowledge ...................................................................................................... 61
Metacognitive Knowledge ................................................................................................ 65
Results and Findings for Motivation Causes .................................................................... 69
Attribution Theory ............................................................................................................ 70
Goal-Orientation ............................................................................................................... 73
Results and Findings for Organization Causes ................................................................. 76
Cultural Models ................................................................................................................ 77
Cultural Settings................................................................................................................ 85
Summary of Influence Assets and Needs ......................................................................... 95
Knowledge ........................................................................................................................ 95
Motivation ......................................................................................................................... 95
Organization ...................................................................................................................... 96
CHAPTER FIVE: RECOMMENDATIONS, IMPELMENTAITON, AND EVALUTION ....... 98
Organizational Context and Mission ................................................................................ 99
Organizational Performance Goal ..................................................................................... 99
Description of Stakeholder Groups ................................................................................. 100
Goal of the Stakeholder Group for the Study ................................................................. 101
Purpose of the Project and Questions ............................................................................. 102
Introduction and Overview ............................................................................................. 102
Recommendations for Practice to Address KMO Influences ......................................... 103
Knowledge Recommendations ....................................................................................... 103
Motivation Recommendations ........................................................................................ 109
Organization Recommendations ..................................................................................... 113
Integrated Implementation and Evaluation Plan ............................................................. 120
Implementation and Evaluation Framework ................................................................... 120
Organizational Purpose, Need, and Expectations ........................................................... 121
Level 4: Results and Leading Indicators ......................................................................... 121
Level 3: Behavior ............................................................................................................ 123
Level 2: Learning ............................................................................................................ 127
Level 1: Reaction ............................................................................................................ 131
Data Analysis and Reporting .......................................................................................... 133
Summary ......................................................................................................................... 134
Strengths and Weaknesses of the Approach ................................................................... 134
iv
Limitations and Delimitations ......................................................................................... 135
Future Research .............................................................................................................. 136
Conclusion ...................................................................................................................... 137
REFERENCES ........................................................................................................................... 140
APPENDIX A ............................................................................................................................. 148
Survey Items ................................................................................................................... 148
APPENDIX B ............................................................................................................................. 151
Interview Protocol ........................................................................................................... 151
APPENDIX C ............................................................................................................................. 154
Initial Workshop Evaluation Tool .................................................................................. 154
APPENDIX D ............................................................................................................................. 156
Delayed Workshop Evaluation Tool ............................................................................... 156
APPENDIX E ............................................................................................................................. 158
Data Visualization Concept for Displaying Progress Towards Desired Outcomes ........ 158
v
LIST OF TABLES
Table 1: Organizational Mission, Organizational Performance Goal, and Stakeholders’ Goals
Table 2: Knowledge Influences, Types, and Assessments for Knowledge Gap Analysis
Table 3: Motivation Influences, and Assessments for Motivation Gap Analysis
Table 4: Organizational Influences, and Assessments for Organizational Gap Analysis
Table 5: Demographic Information of Survey Participants’ Service Years in Education
Table 6: Demographic Information of Survey Participants’ Service Years as an Administrator
Table 7: Demographic Information of Interview Participants’ Service Years in Education
Table 8: Demographic Information of Interview Participants’ Service Years as an Administrator
Table 9: Knowledge and Skill Influences Assets and Needs
Table 10: School Leader’s Procedural Knowledge of Practices that Support Data-Driven Schools
Table 11: School Leader’s Metacognitive Knowledge of Practices that Support Data-Driven
Schools
Table 12: Summary of Assumed Knowledge and Skill Influences Identified as Assets or Needs
Table 13: Motivational Influences Identified as Assets or Needs
Table 14: Summary of Assumed Motivational Influences Identified as Assets or Needs
Table 15: Organizational Influences Identified as Assets or Needs
Table 16: School Leader’s Cultural Model of Practices Associated with Influence 1
Table 17: School Leader’s Cultural Model of Practices Associated with Influence 2
Table 18: School Leader’s Cultural Setting of Practices Associated with Influence 1
Table 19: School Leader’s Cultural Setting of Practices Associated with Influence 2
Table 20: Summary of Assumed Organizational Influences Identified as Assets or Needs
Table 21: Summary of Assumed Knowledge and Skill Influence Assets and Needs
Table 22: Summary of Assumed Motivation Influence Assets and Needs
vi
Table 23: Summary of Assumed Organization Influence Assets and Needs
Table 24: Organizational Mission, Organizational Performance Goal, and Stakeholders’ Goals
Table 25: Summary of Knowledge Influences and Recommendations
Table 26: Summary of Motivation Influences and Recommendations
Table 27: Summary of Organization Influences and Recommendations
Table 28: Outcomes, Metrics, and Methods for External and Internal Outcomes
Table 29: Critical Behaviors, Metrics, Methods, and Timing for Evaluation
Table 30: Required Drivers to Support Critical Behaviors
Table 31: Evaluation of the Components of Learning for the Program
Table 32: Components to Measure Reactions to the Program
vii
LIST OF FIGURES
Figure 1: Building Data Use Capacity Conceptual Framework
Figure C1: Section one of the survey that will immediately follow the first workshop
implementing a program for addressing the knowledge, motivational, and
organizational influences identified by the study.
Figure C2: Section two of the survey that will immediately follow the first workshop
implementing a program for addressing the knowledge, motivational, and
organizational influences identified by the study
Figure D1: Section one of the survey that will follow the first workshop after six to eight weeks
of implementation of critical behaviors and drivers.
Figure D2: Section two of the survey that will follow the first workshop after six to eight weeks
of implementation of critical behaviors and drivers.
Figure E1: Mock dashboard charts for visualizing the status of implementation progress as
determined through surveys of school leaders throughout the process.
viii
ABSTRACT
Schools regularly collect data regarding student progress and learning gaps. However. the
support, structures, and value school leaders place on the use of data for informing instruction
and intervention influences whether data informs student achievement improvement. This
evaluation study examines how Sunshine Unified School District (SUSD) school leaders support
data use by teachers to improve student achievement in English Language Arts and Mathematics.
A gap analysis framework was utilized, focusing on the knowledge and skill, motivation, and
organizational barriers influencing school leaders at the SUSD schools. Quantitative and
qualitative data was collected through a survey and interviews. The findings suggest that school
leaders must have knowledge and organization structures that establish dedicated time for
teachers to collaborate around data, as well as consistently reinforce the belief that the purpose of
data use is for student achievement improvement. Successful implementations of the
recommendations is anticipated to develop skills and structures for school leaders to effectively
build the capacity for teachers to use data to inform their instructional practices and
interventions, and ultimately lead to the accomplishment of the organization’s goal to increase
student achievement.
1
CHAPTER ONE: INTRODUCTION
Introduction of the Problem of Practice
An essential step towards academic achievement for all students involves timely
identification of students that are at-risk in order to provide support (McComas, Burns, &
Helman, 2015). Schools regularly collect data regarding student progress that may be leveraged
to inform the identification of students in need of intervention. Processing the data for insights
involves multiple variables to account for motivation, understanding of learning goals, and
environmental factors (Harwati, Alfiani, & Wulandari, 2014). The student data that schools
collect is often found in separate systems and formats, requiring data processing or integration to
examine multiple variables (Romero & Ventura, 2013). Faced with the challenge of accessing
and interpreting multiple data points for a student, educators may instead choose to rely on
professional judgement. However, while it is not universal, studies support that combining data
from multiple assessments or measures can improve accuracy (McComas, Burns, & Helman,
2015).
Education in the US places increasing importance on data use to inform instruction, as
well as to comply with policies for accountability (Jimerson, 2014). Jimerson (2014) explains
that mental models include assumptions, beliefs, biases that influence understanding, decisions,
and actions. School leaders and teachers can use data to further inform their understanding of
students, tempering their beliefs and bias as they seek to support each student. The study also
warns that data use historically has been used for two purposes: improvement or compliance.
Accountability and compliance practices can make teachers wary of data use, creating a negative
association with teacher performance evaluation, and a fear of taking responsibility for student
achievement growth (Jimerson, 2014). Wayman and Jimerson (2013) cite that their review of
2
research suggests that educators lack in confidence and skill in using data to inform instruction.
The reasons for this deficit include lack of technical training with data tools, guidance and
leadership in analyzing and utilizing data, inadequate time to review data in a timely manner, and
lack of knowledge of strategies for responding to data observations to improve achievement.
The study concluded that teachers do not receive adequate professional development, aligned to
build on itself through multiple training sessions and application of data use. The training they
receive is often not relevant to their classroom context or immediately applicable to their
classrooms, nor does it provide opportunity to develop language and understanding through
conversation about data use. With the transition away from No Child Left Behind measures,
school accountability has begun to examine multiple formative data points to provide a more
holistic understanding of student achievement. It is not enough to base growth on a single
summative measure (Portz, 2017).
Organizational Context and Mission
The organization, Sunshine Unified School District (SUSD), is a PreK-12 school district
of approximately 9,000 students in Southern California. During the study, the mission of the
district focused on providing a caring, respectful, and encouraging environment where students
thrive and demonstrate academic excellence and develop unique talents to prepare them for their
future goals. Within the student population, forty-eight percent self-identified as White, twenty-
five percent as Hispanic or Latino, thirteen percent as Asian, seven percent as two or more races,
three percent as African American, and three percent as Filipino. The district’s schools consisted
of six elementary schools, grades preschool to five, and three secondary schools, grades six to
twelve.
3
Organizational Goal
During the year prior to the study, the district demonstrated that eighty percent of
students assessed through the CAASPP state assessments were at or above proficiency in English
Language Arts (ELA) and fifty percent of students were proficient in Mathematics. The skills
associated with proficiency are broken down through the California Common Core standards and
are further assessed through screening and formative assessments during the year. The
proficiency suggests that twenty percent of graduating students would not demonstrate ELA
proficiency and fifty percent would not demonstrate Mathematics proficiency necessary to
prepare them for success in college and future careers. The organizational goal was to provide
an education of academic excellence and prepare students for their future which includes college
and career readiness. Failure to further increase student proficiency would mean that students
graduate without preparation for success.
The organizational performance goal during the study was to improve student
achievement in ELA and Mathematics proficiencies as measured by the state summative
assessment, CAASPP. By June of 2022, the district sought to increase student performance on
the CAASPP by six percent or two percent per year. Student proficiency in ELA and
Mathematics is essential for college and career readiness. Failure to achieve the district goal
would result in less students graduating with the skills and proficiency needed to succeed in
college and future careers.
Related Literature
California requires schools to submit, and revise annually, a Local Control and
Accountability plan designating how they will use funding to improve student outcomes (Local
Control and Accountability Plan (LCAP) - Resources, 2018). As part of their plan to improve
4
student proficiency, SUSD designated funds to improving data-based practices, providing insight
into the improvement of instruction, targeting intervention more effectively, and developing
more informative assessments. Investing in data-based practices is supported by Black and
Wiliam (2010) who describe a balance that must exist between teaching and learning. To close
achievement gaps and meet student needs teachers must monitor student performance and adapt
learning opportunities to compensate (Black & Wiliam, 2010; Katz & Dack, 2014). The
classroom described involves frequent formative assessments that inform upcoming lessons,
allowing students additional opportunities to improve their understanding of challenging
concepts. However, as schools develop more effective formative assessments and data collection
strategies, the next challenge is incorporating data-based decision making into school practices
such as lesson planning, resource allocation, and professional development (Marsh, 2012; Marsh
& Farrell, 2014). The study also highlights that beyond the knowledge-based needs school
leaders and teachers have for learning to use data, there are also motivation-based needs. The
beliefs and expectations of both school leaders and teachers may be challenged as they adjust
their practices informed by data. It is essential to provide tools, workshops, and other supports
that build up a deep theoretical understanding of how data can be used dynamically throughout
the year to guide choices both in the classroom as well as in school intervention programs (Katz
& Dack, 2014; Marsh & Farrell, 2014; Wayman & Jimerson, 2014).
Importance of the Evaluation
The problem of utilizing multiple measures to inform the support of student achievement
is important to solve for a variety of reasons. Currently, schools collect an abundance of data
both to support instruction as well as to meet the requirements of accountability to State and
Federal governments, as well as grants and other financial supports (Jimerson, 2014).
5
Connecting multiple measures for analysis in ways that increase student achievement will
provide deeper value for data collection (Portz, 2017). The process can reveal data collections
that were previously unconsidered for potential to inform improvement efforts. Redundant data
collection steps can also be revealed providing increased levels of efficiency (Bin Mat,
Buniyamin, Arsad, & Kassim, 2013). Schools have limited resources with which to provide
support for students. Opportunities to improve the utilization and efficiency of data that is
consistently collected will enable schools to make more informed decisions on the allocation of
resources yielding greater achievement for all students.
Description of Stakeholder Groups
The stakeholder groups directly invested and benefiting from the organizational goal are
the students, parents, teachers, and school leaders. All four groups participate on the advisory
committee and provide input into the goals that are set and how organizational resources are
allocated for support. Students and parents invest in the education of the student, and directly
benefit from improvements in how the district supports their growth and success. Teachers serve
as the primary facilitators of student learning and are directly responsible for proficiency
improvement. School leaders, such as principals and assistant principals, are responsible for
overseeing improvement efforts, providing support and intervention resources for students that
teachers identify as in need, and leading data-based decision making to support student learning
at their school. Overall, it was the goal of school leaders to facilitate and support the efforts of
other stakeholder groups in order to develop incremental progress to the organizational goal for
improving student achievement.
6
Stakeholders Performance Goals
Table 1 below describes the connections between SUSD’s organizational mission for supporting
students and the organizational and school leader goals for improving student achievement.
Table 1.
Organizational Mission, Organizational Performance Goal, and Stakeholders’ Goals
Organizational Mission
The mission of SUSD is that students will thrive in a caring, respectful, and encouraging
environment where they demonstrate academic excellence and develop unique talents in
preparation for their future goals.
Organizational Performance Goal
By June 2022, SUSD will use data-based decision making to improve learning
opportunities, closing the proficiency gap by 6%, or 2% per year.
School Leader Stakeholder Goal
By June of 2020, School leaders will adapt site messaging and support for data-based decision
making using student data from multiple measures to increase student proficiency towards the
2% annual growth goal.
Stakeholder Group for the Study
While the combined efforts of all stakeholders will join to achieve the organizational goal
of a 2% annual increase in student proficiency, school administrators are responsible for the
overall leadership of improvement efforts towards the performance goal. Therefore, the
stakeholders of focus for this study were the administrators overseeing instruction and learning at
each of the nine schools. The specific stakeholder goal for school administrators, which are
referred to as school leaders in this study, was to incrementally increase student proficiency by
2% each year, using data from multiple measures to inform the allocation of resources for first
7
time instruction and intervention based on student need and growth. Failure to accomplish this
goal would impact student achievement, which would eventually impact graduation rates, post-
secondary careers and education, and district enrollment. Inter-district transfer students could
choose to return to their home district, and the SUSD students within the community could begin
to transfer to another district. Both outcomes would lead to a loss of funding for the district.
Purpose of the Project and Questions
The purpose of this study was to evaluate the knowledge, motivation and organizational
influences on the capacity of school leaders to lead the use of data-based decision making to
increase student achievement. The analysis began by generating a list of possible or assumed
interfering influences to be examined systematically, focusing on actual or identified interfering
influences. While a complete evaluation would focus on all stakeholders, for practical purposes
the study focused on school leaders that oversaw instruction and learning, exploring their
leadership in the use of data to inform the improvement efforts.
As such, the questions that guided the evaluation study were:
1. To what extent did school leaders improve student achievement through the
promotion of data-based decision making in their schools?
2. What knowledge and motivation did school leaders have related to increasing
student achievement through data-based decision making?
3. What was the interaction between Sunshine Unified School District’s culture and
context and school leader knowledge and motivation for increasing student
achievement through data-based decision making?
8
4. What are the recommendations for organizational practice in the areas of
knowledge, motivation, and organizational resources for achieving increases in
student proficiency through data-based decision making?
Methodological Framework
Clark and Estes’ (2008) gap analysis, a systematic, analytical method that helps to clarify
organizational goals and identify the gap between the actual performance level and the preferred
performance level within an organization, was implemented as the conceptual framework. The
methodological framework was a mixed methods study. Assumed knowledge, motivation and
organizational influences that interfered with organizational goal achievement was generated
based on personal knowledge and related literature. The influences were assessed through
surveys, interviews, literature review and content analysis. Research-based solutions were then
recommended and evaluated in a comprehensive manner.
Definitions
Data-Based Decision: Decisions that are informed by or consider multiple measures,
such as student attendance, screening results, and benchmark scores.
Data Inquiry Model: A model or procedure for approaching data analysis through the
process of asking critical questions. The goal was to identify an area of improvement that can be
tested through further data collection.
Professional Learning Community (PLC): Groups of teachers in the same grade-level or
teaching the same subjects that meet regularly to collaborate on lesson design and improvement.
Organization of the Project
Five chapters organize this study. This chapter provided the reader with the key concepts
and terminology commonly found in a discussion about utilizing data to inform decisions when
9
improving student achievement. The organization’s mission, goals and stakeholders and the
framework for the project were introduced. Chapter Two provides a review of current literature
surrounding the scope of the study. Topics of school leader beliefs about data use, how they
support teachers in using data, and organizational supports for data use are addressed. Chapter
Three details the knowledge, motivation and organizational elements to be examined as well as
methodology when it comes to the choice of participants, data collection, and analysis. In
Chapter Four, the data and results are assessed and analyzed. Chapter Five provides solutions,
based on data and literature, for closing the perceived gaps as well as recommendations for an
implementation and evaluation plan for the solutions.
10
CHAPTER TWO: REVIEW OF THE LITERATURE
Introduction
According to the 2017 National Assessment for Educational Progress (NAEP), average
scores in reading and math proficiency for students in 4
th
and 8
th
grade have been below
proficient for the last 15 years (NAEP Data Explorer). The same assessment data also
demonstrates that while California student scores have been improving, they are still below the
national averages, and proficiency gaps are greatest for students of socio-economically
disadvantaged, Hispanic, and African American populations. While in 2019, over eighty percent
of students tested in Sunshine Unified School District (SUSD) demonstrated ELA proficiency
and over fifty percent of students were proficient in Mathematics; literacy and math skills remain
a major focus for improving student achievement. To close achievement gaps in all student
groups, SUSD school leaders must improve how data-driven decision making is utilized
throughout their school to identify specific student needs. This chapter reviews the influences on
data-driven decision making within a school. Then, the role of school leaders within their school
are reviewed, followed by the explanation of the knowledge, motivation, and organizational
influences’ framework used in this study. Next, the chapter focuses on school leader knowledge,
motivation, and organizational influences and then the chapter concludes by explaining the
conceptual framework.
Influences on the Problem of Practice
Student Achievement Improvement Through Data-based Decisions
Belief in the potential for student achievement. Datnow and Park (2015) explain that
the experience most school leaders and teachers have with data use are quick or incomplete
reactive looks at data that lead to simplification of complex issues. The analysis is often
11
superficial and influenced heavily by beliefs in instruction and student achievement. The study
stresses that schools need to move beyond standardized student achievement data and instead
look at data in terms of opportunities-to-learn. This includes considering multiple assessments
and data collections such as attendance. To challenge and shift beliefs tied to student
achievement, Datnow and Park (2014) suggest that school leaders should frame data analysis as
the search for opportunities for students to learn, closing gaps in understanding. According to
Miranda and Jaffe-Walter (2018), school leaders seeking to challenge beliefs through data
analysis should be cautious. The study explains that data discussions focused on deficits can
lead to misuse of data. School leaders should avoid attributing low achievement to families or
student ability as it leads to teachers taking less responsibility for student growth (Miranda &
Jaffe-Walter, 2018). Instead, Miranda and Jaffe-Walter (2018) recommend that school leaders
should build on the belief that students can grow with the right curriculum or pedagogical shifts.
The study also warns school leaders to avoid data meetings that take on all student achievement
concerns at once; meetings should focus on specific targets. Through focused data-based
discussions Miranda and Jaffe-Walter (2018) suggest that school leaders can build an
understanding that no one is blamed for low achievement. Students can achieve, so the goal is to
identify and reflect on the challenges that are revealed by the analysis.
Goals Orientation of Data Use. Black and Wiliam (2010) identify that the goal
orientation of school data use efforts should focus on identifying student progress and
difficulties, so that teachers can adapt or identify needs for support. School leaders should help
teachers access and analyze formative data tied to the desired area of improvement and then
adjust learning opportunities to meet student needs (Black & Wiliam, 2010). The study supports
this strategy based on the finding that frequent use of formative assessments improves students
12
with low achievement the most, providing frequent feedback, and raising achievement overall.
Black and Wiliam (2010) summarize the change in goal orientation as shifting the focus from
scoring and grades, towards the analysis of work to identify learning needs. Their method is
supported by Dowd’s (2005) argument that developing a culture of inquiry engages teachers in
the change effort. If schools want to improve student learning, creating a culture of inquiry
means asking questions about what students are learning and what they are struggling with
understanding. Dowd (2005) concludes that determining what information to collect, the process
of collection, who will participate in the analysis, and how the results and action steps will be
communicated are all important to defining the purpose and goal. School leaders should also be
clear about the how the goals will be used and what the school can do to support teachers in
pursuing their goals (Dowd, 2005). To begin the goal setting process, Marsh (2012)
recommends that school leaders need to model data use in making educational decisions.
Alongside the development of school-wide data-based improvement goals, teachers can create
their own goals aligned to the school’s focus for improving student proficiency through data-
based decisions. As teachers create goals they are influenced by the resources that they perceive
will be available, so it is important for school leaders to be clear about the role they play in
providing support, interventions, and other resources (Marsh, 2012).
The Role of School Leaders in Data use Within Their School
Support and accountability for data analysis. The support for developing goals to
improve student learning comes not only from school leaders, but also from teachers working
together. Collaborative inquiry enables teachers to change practices and influence student
learning (Katz & Dack, 2014). According to Katz and Dack (2014), teacher collaboration in
groups, such as PLCs, should engage in a process that questions current learning opportunities,
13
teaching routines, and methods of intervention, while looking at new options for supporting
student learning. The study suggests that such discussions should encourage action steps that
lead to professional growth and student growth. To develop capacity for improvement, Katz and
Dack (2014) argue that a group must generate new knowledge and understanding through the
collaborative inquiry process. This argument is supported by Huguet, Farrell, and Marsh (2017)
with evidence that teacher collaboration with grade-level or content peers is essential to the
process of building capacity for improvement. It is recommended that school leaders support the
process by providing tools and requiring artifacts to be generated, implementing accountability.
However, Huguet, Farrell, and Marsh (2017) urge leaders to be cautious of requiring too much
paperwork which can limit the time and space teachers have available for productive dialogue
and analysis. The importance of providing time and space for teacher collaboration groups to
adequately examine data is also supported by Miranda and Jaffe-Walter (2018). The study also
emphasizes that school leaders ensure that teacher collaboration occurs in like groups, such as
grade level teams, to prevent individuals from hiding or getting defensive or dismissive. Like
groups have a shared understanding of the data that is necessary for engaging in deep inquiry of
the data and the insights it can provide (Miranda & Jaffe-Walter, 2018).
Structure and protocols for data analysis. Huguet, Farrell, and Marsh (2017) suggest
that it is important for school leaders to purposefully define the data inquiry process, including
thoughtful formation of collaborative groups and holding each group accountable for creating
action steps at the end of the process. The school leaders should be clear in establishing
conditions for teachers to examine student data deeply (Datnow & Park, 2015). Datnow and
Park (2015) explain that big decisions may require multiple data points, reflection, and even
identifying additional data that must be collected. School leaders and support staff should be
14
involved to aid with data storage and analysis programs, grouping and processing the data in a
way that is accessible for teachers (Marsh, 2012). However, Datnow and Park (2015) emphasize
that amidst the tools and support personnel, it is essential for school leaders to value the
professional judgement of their teachers; data use should build on teacher professionalism, not
threaten it. The focus of all structures and protocols should be on melding the professional
judgment of teachers with the measurements of student understanding that we get from data to
identify opportunities for student growth, not limitations (Datnow & Park, 2015).
Role of Stakeholder Group of Focus
While the combined efforts of all stakeholders seek to achieve the organizational goal of
a 6% increase in student achievement, it was important to evaluate how school leaders support
the efforts to use data-based decision making towards the performance goal. Therefore, the
stakeholders of focus for this study were the school administrators that oversee instruction and
learning. The stakeholders’ goal was to increase student proficiency by 2% each year using data
from multiple measures to inform the allocation resources for first time instruction and
intervention based on student need and growth. Failure to accomplish this goal would impact
student achievement, which would impact graduation rates, post-secondary careers and
education, and district enrollment. Inter-district transfer students could choose to return to their
home district, and the SUSD community could begin to transfer to another district. Both
outcomes would lead to a loss of funding for the district.
Clark and Estes’ (2008) Knowledge, Motivation and Organizational Influences Framework
Clark and Estes (2008) provide a framework for evaluating the performance gaps
between institutional goals and actual performance. The framework examines stakeholder
performance as being caused by three different influences: knowledge, motivation, and
15
organizational. When a performance gap has been identified, the analysis of the knowledge and
skill influences determines whether stakeholders require information, job aids, training, or
education to close the gap (Clark & Estes, 2008). For institutions where motivation is
influencing the performance gap, the framework examines whether stakeholders are choosing to
pursue institutional goals, persisting at the work, and whether the work requires an adequate
amount of mental effort. Stakeholders need to understand and value the goals, and too much or
too little mental effort tied to goals can lead to underperformance due to under- or
overconfidence respectively (Clark & Estes, 2008; Rueda, 2011). The third cause, according to
the framework, occurs when the performance gap is influenced by organizational factors of
culture, processes, and resources. Institutions may lack cultural support for the established goals,
need additional resources to do the work, or have inefficient processes in place, generating the
perceived gap in performance (Clark & Estes, 2008).
The three causes of the institutional performance, according to Clark and Estes (2008),
were utilized to examine the knowledge, motivation, and organizational influences on the ability
of school leaders to encourage their school sites to use data-based decisions to improve student
achievement. The first section discusses the assumed knowledge influences, followed by a
discussion of the motivational influences. Then the discussion finishes with an explanation of
the assumed organizational influences. Collectively, the three areas of influence, knowledge,
motivation, and organizational, are examined in the development of the methodology of this
study in Chapter 3.
Stakeholder Knowledge, Motivation and Organizational Influences
Knowledge Influences
SUSD seeks to improve student proficiency by 6% over the next three years. For the
16
2019-2020 school year, eighty percent of all students tested scored proficient in ELA and fifty
percent were proficient in Mathematics. Both areas of proficiency increased from the prior year.
Schools must work harder to reach the goal of increasing by 2% each year, identifying student
instructional needs quickly and delivering intervention and instructional support promptly. The
district has begun the process of addressing this challenge by providing additional support for
schools to develop the capacity for making data-based decisions using formative assessments.
Support includes training for school leaders and teachers on the use of online formative
assessments and interpreting data analysis reports. Each workshop focuses on increasing the
capacity for staff to gather, access, and interpret data from formative assessments, as well as
generate action steps to address student needs (Elmore, 2002; Marsh & Farrell, 2015; Rueda,
2011).
Developing a successful culture of data use relies on each school leader’s ability to model
the importance and purpose of using data to support instruction, and the ability of teachers to use
data to adjust instruction. While the district has provided data tools and formative assessments
for several years and encouraged the use of data, intentional data inquiry has not yet developed
into a universal practice. Addressing the challenge to improve the culture of making data-based
decisions within schools involves knowledge and skill influences related to building the capacity
of staff for using data. For example, school leaders need specific knowledge and skills to
promote and support effective data use at their schools, and teachers need a different set of
knowledge and skills to interpret formative data quickly and determine appropriate instructional
actions (Black & Wiliam, 2010; Marsh & Farrell, 2015). Informed professional judgement
provides new learning opportunities for students when data-based insights are combined with
school leader and teacher experience in pedagogy and learning practices (Katz & Dack, 2014).
17
Gaps in factual, conceptual, or procedural knowledge act as barriers to access, and may need to
be addressed by providing facts or teaching step-by-step procedures (Mayer, 2011). For
example, workshops and discussions could share about the benefits of effective data use or
model and provide practice opportunities for using data inquiry methods to inform instruction.
Metacognitive knowledge gaps involve reflecting on the thinking processes that need to be
developed (Baker, 2006). Coaching individuals in metacognitive processes involves exercises
such as critically analyzing and engaging in the data to identify student needs. For example,
developing metacognitive capacity requires staff to demonstrate and reflect on their analysis
using self-talk before, during, and after examining formative data.
The literature reviewed in this study examines knowledge and skill influences tied to
building capacity for data-based decisions to support instruction. Through the review,
examination includes influences sought to determine which stakeholder groups are affected and
the types of knowledge involved.
Effective collaboration models for data analysis. Developing a culture of data use
involves utilizing strategies that provide structure for the process as well as scaffolding to
compensate for the different levels of capacity among staff. Research suggests collaborative
inquiry as an effective strategy for facilitating change in instructional practices (Katz & Dack,
2014). In their review of current research, Katz and Dack (2014) summarize that improving
teaching and learning involves teachers working collaboratively to analyze current student
progress and asking questions about classroom and learning practices. The study also suggests
that collaborative groups need time to examine new ideas and concepts for teaching and learning,
as well as to examine methods for identifying and responding to needs for differentiation.
Furthermore, Katz and Dack (2014) clarify that the collaborative inquiry potential for school
18
improvement results from shared learning activities that intentionally interrupt the status quo by
supporting individual knowledge and professional development. Establishing the use of an
inquiry cycle within the culture requires that school leaders model the use of inquiry-based data
use with staff, building their ability to understand and apply the process. Gradually, groups of
staff can then be encouraged to utilize the process with support until their efficacy extends to
small group use, such as professional learning community (PLC) groups, and individuals (Katz
& Dack, 2014).
Van Gasse, Vanlommel, Vanhoof, and Van Petegem (2017) also support the use of
collaborative inquiry through their study. However, they conclude that the level of school-wide
teacher interaction declines as they work through the phases of data use. The study breaks
DDDM into a structure of four phases: discuss, interpret, diagnose, and action. Transitioning
from one data use phase to another, Van Gasse et al. (2017) found that teachers interacted with
fewer colleagues, yet the intensity of the interactions increased. The researchers also clarified
that it is important to distinguish between cooperation and collaboration. Cooperation activities
involve interactions such as storytelling, providing help, and sharing resources and strategies,
while collaboration involves working together to achieve shared goals. Van Gasse et al. (2017)
demonstrated that school social networks among teachers support low teacher interdependent co-
operation activities in the early data phases, and shift to smaller, highly interdependent,
collaborative groups as they approach the action phase. School leaders need to be intentional
about how they engage staff in learning and building capacity with the data phases. Staff
meetings and large workshops can support data discussion, interpretation, and some diagnosis
through cooperative learning activities (Van Gasse et al., 2017). However, according to Van
Gasse et al. (2017), content or grade level diagnosis and the action phase are best supported by
19
small interdependent groups that are truly collaborative.
School leaders need to learn about the concepts of collaborative inquiry and the types of
collaborative groups that work best during different phases of data analysis. Examining school-
wide data problems gives school leaders opportunities to challenge the status quo as well as
develop teacher capacity.
Support and accountability of data analysis in PLC groups. Huguet, Farrell, and
Marsh (2017) recognize that PLC groups are used by many schools to support instructional
change and school improvement, yet they also highlight that school leaders have an important
influence on how well PLC groups incorporate data-based practices. PLC groups are groups of
teachers organized around grade level or subject with the focus on collaborating with similar
content to share strategies and improve instruction for students. Van Gasse, Vanlommel,
Vanhoof, and Van Petegem (2017) agree with Huguet et al., identifying teacher collaboration as
an essential part of developing a strong culture of data-based decisions and that the collaboration
is most productive in small groups. School leaders can influence the routines and expectations
for PLCs and other staff meetings, through tools that guide the process and create artifacts for
accountability (Huguet et al., 2017). It is important for school leaders to know how the data
analysis process and concept works most effectively in a PLC group. The PLC process must
value the professional skill and experience of the participating teachers, while a data analysis
process provides a framework for informing their understanding of students, empowering
professional judgment (Katz & Dack, 2014; Marsh, 2012; Marsh & Farrell, 2015). Experts agree
that building teacher capacity for data-based decisions requires effective, dedicated time,
scaffolding of the process and supports, and purposeful outcomes where teachers identify action
steps to address student needs (Black & Wiliam, 2010; Dowd, 2005; Huguet et al., 2017).
20
Through their study, Huguet et al. (2017) demonstrate that school leaders should be careful
when attempting to structure data analysis in PLC. Add too much paperwork or requirements,
and the data analysis will overwhelm the collaborative process, removing the space for
professional discussion and inquiry (Huguet et al., 2017; Van Gasse et al., 2017). Huguet et al.
(2017) describe an effective model as a framework that supports the process, prompting inquiry
and leading to action step formation. Hence, school leaders must be aware of how much
accountability and guidance for data use they can establish as part of PLC groups without
disrupting the time required for the group to reflect and discuss possible actions.
School leaders supporting PLC groups and data use must ensure that groups have
informative data, a framework to guide their inquiry and create action steps, a method for sharing
the action steps that are developed, and resource needs met with support. Hence, addressing the
knowledge gap allows school leaders to provide guidance and accountability for data analysis
without hindering the effectiveness of PLC groups.
Table 2 below provides the organizational mission, organizational and stakeholders goals,
as well as the knowledge influences, knowledge types, and knowledge influence assessments.
The table contents indicate that knowledge of the conceptual, procedural, and metacognitive
influences tied to the effectiveness of collaboration models during data analysis, as well as
understanding how to provide support and accountability within the procedures and concepts of
PLC groups will impact the development of staff capacity for making data-based decisions.
Table 2.
Knowledge Influences, Types, and Assessments for Knowledge Gap Analysis
Organizational Mission
21
The mission of SUSD is that students will thrive in a caring, respectful, and encouraging
environment where they demonstrate academic excellence and develop unique talents in
preparation for their future goals.
Organizational Performance Goal
By June 2022, SUSD will use data-based decision making to improve learning
opportunities, closing the proficiency gap by 6%, or 2% per year.
School Leader Stakeholder Goal
By June of 2020, School leaders will adapt site messaging and support for data-based
decision making using student data from multiple measures to increase student proficiency
towards the 2% annual growth goal.
Assumed Knowledge
Influence
Knowledge Type Knowledge Influence
Assessment
School leaders must know
the phases of data analysis.
Factual Survey and Interviews
School leaders must know
which collaboration models
are most effective for each
phase of data analysis.
Conceptual Survey and Interviews
School leaders must know
how to effectively provide
accountability for PLC data
use.
Procedural Survey and Interviews
School leaders must know
how to effectively provide
guidance for PLC data use.
Metacognitive Survey and Interviews
Motivation Influences
Supporting a culture of data use to increase student achievement is impacted by the
motivation of school leaders and teachers to use formative data regularly, informing instruction
to meet student needs. Motivation influences staff by affecting whether they choose to use
formative data, how likely they are to persist in analysis and identifying courses of action, and
will fluctuate based on the level of mental effort required (Mayer, 2011). Identifying the
22
motivational influences impacting the culture of making data-based decisions within the district
will guide the district in how to remove motivational barriers to engaging in data use.
The literature review identified motivation influences tied to building capacity for data
use to support instruction. Motivational challenges are important to examine in addition to
knowledge and skill influences. An organization can address all the knowledge and skill needs
of their staff, but if the staff is not motivated to perform the task, they will not be productive
(Clark & Estes, 2008). Building capacity for data use must include addressing the motivational
needs of both school leaders and teachers. The literature identified two motivation influences:
factors attributed to the cause of student achievement, and the goal orientation of data use as it
relates to student achievement.
Attribution theory and student achievement. When reviewing student achievement,
beliefs play a strong role in attributing the cause of a student’s performance. E. Anderman and
Anderman (2006) explain that attribution, or belief of why an outcome occurred, impacts
motivation based on three factors: locus, stability, and controllability. Locus refers to whether
the cause is internal or external; stability describes whether the outcome is consistent through
time and different events; and controllability is whether they believe the cause of the outcome is
under their control (Anderman & Anderman, 2006). E. Anderman and Anderman (2006)
suggest that when school staff, consisting of school leaders and teachers, believe the reason for
low student achievement is due to deficits of student ability or the student’s parental support,
they are less motivated to improve instruction through data use. Student ability and parental
support are factors that school leaders and teachers perceive as out of their control and have little
power to change. Other studies suggest data talk sessions that do not provide enough context,
move too fast to go beyond cursory analysis, or make staff feel their reputation is threatened,
23
reinforce beliefs in student deficits (Datnow & Park, 2015; Miranda & Jaffe-Walter, 2018).
Studies identify that it is important for school leaders to believe achievement gaps connect to a
need for learning opportunities, and that they can close the gap by supporting teachers and
providing additional resources when necessary (Anderman & Anderman, 2006; Datnow & Park,
2015). School leaders can in turn coach teachers through shifting their beliefs and attribution
towards identifying additional classroom learning opportunities that students need to succeed.
Datnow and Park (2015) support the conclusions of Anderman and Anderman (2006),
sharing through their study the benefits of moving beyond achievement data, to examine multiple
measures tied to opportunities to learn. Staff can make stronger informed professional decisions
when they can analyze student needs based on multiple metrics, including attendance,
benchmarks, screening tools, and classroom-based formative assessments. Data can enhance the
big picture of progress for school leaders, motivating their ability to allocate school resources.
School leaders, in turn, can motivate teachers to respond to student needs by highlighting
examples in progress towards closing achievement gaps and providing supportive learning
opportunity options such as intervention specialists. Additionally, focus on supporting students
with learning opportunities will further adjust the attribution perspective away from an external
locus of the student, engaging an internal drive to develop new strategies and protocols for
responding to student needs (Anderman & Anderman, 2006; Datnow & Park, 2015).
Attribution and belief in the ability to bring about change are important motivators for school
leaders that can be passed on to teachers as resources are provided to support their data-informed
action steps. The school-wide focus, in turn, can shift the culture towards a goal orientation of
improving learning for all students.
Goal orientation of data use. Researchers have established that the goal orientation of
24
data-focused schools is an important factor, emphasizing that the purpose of data analysis is
improving student learning opportunities, not for evaluating the school’s performance (Black &
Wiliam, 2010; Dowd, 2005; Marsh, 2012; Marsh & Farrell, 2015). Schools that believe the goal
of data use is to evaluate their performance, fearing punitive measures, will lose motivation
(Marsh, 2012; Marsh & Farrell, 2015; Rueda, 2011). Performance goal orientation creates a
negative competitive culture that inhibits collaboration between schools, their leaders, and their
teachers when the focus should be on promoting collaborative inquiry and sharing impactful
practices. Thus, researchers recommend that district and school leaders communicate a clear
vision for supporting students aligned with data-based decisions (Marsh, 2012; Marsh & Farrell,
2015; Rueda, 2011). Sharing the vision is not enough; it must frequently be revisited through the
focus of leaders, work towards short-term goals, and demonstration of behaviors that utilize data
in a mastery goal orientation for improving learning opportunities (Marsh & Farrell, 2015;
Rueda, 2011). Mastery goal orientation encourages collaborative inquiry that analyzes the needs
of students and harnesses the experience and skill of teachers to find new solutions (Rueda,
2011; Yough & Anderman, 2006).
With an emphasis on improving how the school can support student learning, moving
beyond the fear of judgment, school leaders and teachers gain motivation to take professionally
informed risks and challenge the status quo in pursuit of better solutions for students (Dowd,
2005; Katz & Dack, 2014). While learning opportunities within the classroom can be adapted,
options may also include solutions such as addressing attendance issues through parent
conversations with counselors and principals or utilizing before or after school intervention
specialists (Datnow & Park, 2015). With increased motivation towards providing new solutions,
school leaders can align their support with teacher data-based action plans. Across the school, as
25
action plans are revisited and evaluated through additional data, resources can be reallocated to
successful strategies or new areas of inquiry (Katz & Dack, 2014; Marsh & Farrell, 2015).
Successful strategies can be shared between PLC groups and with other schools, developing a
culture of professional learning and improvement.
Table 3 below provides the organizational mission, organizational and stakeholders goals,
as well as the motivation influences, and motivation influence assessments. The table contents
indicate that it is assumed that beliefs about the factors attributed to student achievement, and the
goal orientation of staff, data use may impact a school’s ability to build capacity for making
data-based decisions.
Table 3.
Motivation Influences, and Assessments for Motivation Gap Analysis
Organizational Mission
The mission of SUSD is that students will thrive in a caring, respectful, and encouraging
environment where they demonstrate academic excellence and develop unique talents in
preparation for their future goals.
Organizational Performance Goal
By June 2022, SUSD will use data-based decision making to improve learning
opportunities, closing the proficiency gap by 6%, or 2% per year.
School Leader Stakeholder Goal
By June of 2020, School leaders will adapt site messaging and support for data-based
decision making using student data from multiple measures to increase student proficiency
towards the 2% annual growth goal.
Assumed Motivation Influences Motivational Influence Assessment
School leaders must attribute the cause of
student achievement gaps to be due to effort
Survey and Interviews
26
or learning opportunities, not a lack of
student ability.
School leaders must perceive the goal-
orientation of data use to be improving
learning opportunities for students, not the
judgment of school performance.
Survey and Interviews
Organization Influences
General theory. Clark and Estes (2008) describe organizational factors that contribute
to performance gaps as inefficient or ineffective processes or material resources, and the cultural
setting and models that influence work processes. The process component consists of the
integral links and interactions that must occur between people and resources in the organization
to drive productive work, while material resources are the equipment and supplies that must be
used to complete the work (Clark & Estes, 2008). According to Clark and Estes (2008),
performance gaps develop when processes are either inadequate or misaligned to complete
organizational goals, or when material resources are unavailable or ineffective for production
towards the goals. Organization performance barriers may be a direct result of issues with
processes or materials, or as Clark and Estes (2008) further explain, many of the gaps occur due
to cultural influences on the interactions that are part of work processes. The cultural influences
are defined by Rueda (2011) as the cultural models and settings. According to Rueda (2011), the
cultural model is the organization’s core understanding of how business works, such as values,
beliefs, policies, norms, and reward structures. The cultural model makes up the characteristics
of the organization, and changes over time as the organization adapts behaviors to different
contexts, or cultural settings (Rueda, 2011). The pressure to change the cultural model, based on
Rueda’s (2011) explanation, is driven by the interactions of the components of each cultural
27
setting in the organization: who, what, where, when, why, and how. Ultimately, while
performance gaps may involve addressing knowledge and motivational influences, dealing with
organizational barriers is essential to ensuring the process and materials interactions necessary
for productive organization work.
Stakeholder specific factors. Studying the organizational performance gaps of school
sites involves examining the cultural setting and models that are developed by the school leaders
and their staff. The importance of the components of the cultural setting is confirmed by the
research of Van Gasse et al. (2017), observing that collaboration in the formation of action goals
is most effective when the group consists of individuals with similar content or grade level of
students. Additionally, it is important for school leaders to model and practice a data inquiry
process with staff and collaborative teams consistently to build efficacy and a cultural belief the
data supports improvement goals (Huguet, Farrell, & Marsh, 2017; Marsh & Farrell, 2015;
Miranda & Jaffe-Walter, 2018). Based on these findings, the two cultural influences that may
impact school improvement of student achievement are the data analysis process utilized by
school leaders with their staff, and the school leader methods and communication that impact
staff assumptions and beliefs about the purpose of student data use.
Table 4 below provides the organizational mission, organizational and stakeholders goals,
as well as the assumed organizational influences, and organizational influence assessments. The
table contents indicate that it is assumed that beliefs about the purpose of data use, and the
context in which school leaders support data use may impact a school’s ability to improve
student achievement through data-driven decision making.
Table 4.
Organizational Influences, and Assessments for Organizational Gap Analysis
28
Organizational Mission
The mission of SUSD is that students will thrive in a caring, respectful, and encouraging
environment where they demonstrate academic excellence and develop unique talents in
preparation for their future goals.
Organizational Performance Goal
By June 2022, SUSD will use data-based decision making to improve learning
opportunities, closing the proficiency gap by 6%, or 2% per year.
School Leader Stakeholder Goal
By June of 2020, School leaders will adapt site messaging and support for data-based
decision making using student data from multiple measures to increase student
proficiency towards the 2% annual growth goal.
Assumed Organizational Influences Organization Influence Assessment
Cultural Model Influence 1: School
leaders must promote the value of data
use throughout the year for improvement
and intervention.
Survey and Interviews.
Cultural Model Influence 2: School
leaders must promote that belief that
assessment data is used for student
improvement not to judge the
performance of teachers or schools.
Survey and Interviews.
Cultural Setting Influence 1: School
leaders must incorporate data-informed
goal setting and reflection when
discussing plans and progress.
Survey and Interviews.
Cultural Setting Influence 2: School
leaders must work with staff groups, such
as PLC grade-level or content-team
meetings, supporting data use to build
capacity through application and action.
Survey and Interviews.
29
Conceptual Framework: The Interaction of Stakeholders’ Knowledge, Motivation, and
Organizational Context
The purpose of the conceptual framework is to represent the relationships between the
concepts and variables identified in literature specific to the problem of practice using data to
inform the improvement of student achievement. Maxwell (2013) describes the conceptual
framework as a construct built, utilizing what is learned through a literature review specific to
the problem, to explain how the body of knowledge defines the problem and how the study
intends to explore the phenomena further. The conceptual model includes all the beliefs,
assumptions, concepts, expectations, and theories the are necessary for understanding the
problem, and informing the research questions and design of the study (Maxwell, 2013).
Similarly, Merriam and Tisdell (2016) explain the need for a framework to provide an
explanation of how the concepts of the study are grounded in knowledge of literature and
provide a specific definition of the inherent factors and relationships the study addresses. The
conceptual framework provides the boundaries of this study, identifying the specific factors
connected to SUSD’s goal, and the influences that occur due to related factors.
Through the literature review, the organizational factors linked to SUSD’s goals involve
the culture, resources, and organizational processes. Huguet, Farrell, and Marsh (2017) identify
that the role and interactions that principals have with school professional learning communities
has a direct impact on whether data is accessible and utilized during collaboration time.
Furthermore, the level of data accessibility is tied to whether districts have resources and systems
available for organizing, sharing, and visualizing data (Marsh, 2012; Marsh & Farrell, 2015).
Once data is available and utilized, the interpretation and value of the data analysis in decision
30
making is influenced by whether schools have developed a process for data inquiry (Dowd,
2005; Katz & Dack, 2014). While these organizational factors play a role in defining the data
capacity of a school, the knowledge and motivation of school leaders in the use of data to inform
decision making are also connected. School leaders need knowledge of the concepts and
procedures involved in effective models for using data in collaborative groups, as well as for
establishing guidance and accountability for utilizing data to support decisions (Dowd, 2005;
Huguet, Farrell, and Marsh, 2017; Katz & Dack, 2014). The knowledge of the concepts and
procedures links directly to the effectiveness of the culture of professional learning communities
as well as establishing an organizational process for data inquiry. Additionally, school leaders
need to establish the motivational values and persistence, believing that all students can achieve,
and aligning the organization goal orientation of data use for improvement not judgement
(Datnow & Park, 2015; Huguet, Farrell, and Marsh, 2017; Miranda & Jaffe-Walter, 2018; Van
Gasse, Vanlommel, Vanhoof, & Van Petegem, 2017). These motivational influences tie directly
to the organizational influences impacting whether school staff are motivated to use data and
persist in following the data through inquiry and decision making. Ultimately, the connection of
organizational, knowledge, and motivational influences are essential to understand and address
in order to maximize progress to SUSD’s goal of improving student achievement. The
conceptual framework depicted below, as figure 1, represents the interaction of these knowledge,
motivational, and cultural factors in achieving the goal.
Figure 1. Building Data Use Capacity Conceptual Framework
31
Figure 1. The inner circle represents school leaders that work within the school organization,
interacting, guiding, and establishing accountability goals for their staff. The influence of their
own knowledge of data use concepts and procedures directly impact their ability to support data
use in collaboration as well as to establish accountability. Additionally, a school leader’s values
and persistence in data use defines how they engage in data analysis with their staff and
communicate beliefs of student achievement and the goal for improvement. The outer circle
represents the organization that is led by a school leader’s knowledge and motivation for data use
and will in turn impact the culture of data use throughout the school. School leaders must model
effective methods for data use in collaboration and the process of data inquiry. The arrow
32
demonstrates that addressing the three areas of influence will lead to progress in achieving
SUSD’s goal for improving student achievement.
Conclusion
The purpose of this study focused on evaluating the knowledge, motivation, and
organizational barriers that influence the capacity of school leaders for leading staff in the use of
data to improve student achievement. To summarize what has been reviewed from literature,
school leaders must have knowledge of effective methods for facilitating data-based decisions
and how best to incorporate collaboration into the process. School leaders must also be
motivated by the belief that all students can achieve and that the goal of data use is for
improvement, not judgment of staff. Finally, for real change and improvement to occur, school
leaders must share the knowledge and motivation with staff, establishing an inquiry process for
data analysis that staff can use and providing opportunities where they can model data use for
staff and support staff data-based instructional insights and goals. The study examined the
influences in the context of the Clark and Estes (2008) conceptual framework, evaluating the
influence of the knowledge, motivation, and organization barriers. Chapter three explains how
the study evaluated influences by detailing the study’s methodological approach.
33
CHAPTER THREE: METHODS
Chapter three presents the research design for this study, including the methods identified
for data collection and analysis. As the study evaluated the capacity of school leaders to lead the
use of data by staff to improve student achievement, the goal was to answer the following
questions:
1. To what extent did school leaders improve student achievement through the
promotion of data-based decision making in their schools?
2. What knowledge and motivation did school leaders have related to increasing
student achievement through data-based decision making?
3. What is the interaction between Sunshine Unified School District culture and
context and school leader knowledge and motivation for increasing student
achievement through data-based decision making?
4. What are the recommendations for organizational practice in the areas of
knowledge, motivation, and organizational resources for achieving increases in
student proficiency through data-based decision making?
In order to best explain the methodology and approach, the chapter begins by describing
the criteria that was used to select survey and interview participants, as well as the associated
recruitment strategies and rationale. The details regarding the composition and utilization of the
survey and interviews are also be provided, explaining how the tools worked together to provide
a mixed methods evaluation study. Next, the chapter explains how analysis of the collected data
was completed. Finally, the chapter concludes by evaluating the study’s credibility,
trustworthiness, validity, reliability, ethics, and limitations to provide insight into the measures
34
that were taken to portray an accurate depiction of school leader support for data use in their
schools.
Participating Stakeholders
The stakeholder population examined in this study consisted of school leaders in SUSD.
The effectiveness of school leaders in leading data-based decision making with their staff is
influenced by their knowledge of and motivation to use resources to access and organize data, as
well as their use of a developed inquiry process for data analysis (Huguet, Farrell, and Marsh,
2017; Katz & Dack, 2014; Marsh & Farrell, 2015). The sampling group only included school
principals that oversaw instruction and learning. Developing the knowledge, constructing
informative data systems and reports, and the modeling and practice of a data inquiry process
involves both school leaders, as well as the administrators that manage support systems and
provide services such as professional development opportunities. However, given that not all
administrators in the district were directly tied to supporting the instruction within schools, the
administrators that participated in the study were limited to leaders assigned to a specific school,
working directly with teachers throughout the year to improve student achievement.
Survey Sampling Criteria and Rationale
Criterion 1. The participants in the survey phase were either a school principal or
assistant principal at the school-site level. For teachers to embrace regular use of data to inform
decisions, school leaders must model effective data-based decision making and promote positive
values to motivate data use for improvement goals (Huguet, Farrell, and Marsh, 2017; Katz &
Dack, 2014; Marsh & Farrell, 2015). Additionally, data-based decision making effectiveness is
influenced by a district’s ability to organize and provide access to the data that administrators
and teachers need (Katz & Dack, 2014; Marsh, 2012; Marsh & Farrell, 2015). Thus, it is
35
important to study the how principals lead school data use as well as how they promote and
model growth through professional development, utilize data management systems, and oversee
curriculum and instruction.
Criterion 2. School leaders that participated in the survey were involved in supporting
the improvement of instruction and student achievement. Districts include many administrator
roles beyond principals and superintendents, including those that oversee areas outside of the
context of instruction, such as facilities management and food services. To build the data-based
decision making capacity of school staff, administrators must model and support teacher data use
regularly and consistently, both with the entire staff and with small collaborative groups (Marsh
& Farrell, 2015; Miranda & Jaffe-Walter, 2018; Van Gasse, Vanlommel, Vanhoof, & Van
Petegem, 2017). While administrators that work outside of instructional support may provide
resources necessary for safe and healthy schools, they do not have a direct influence on data-
based decision making of instruction and improving student achievement. Thus, the sample of
administrators taking the survey focused on the population whose role regularly supports or
develops teacher capacity to use data to improve student achievement.
Survey Sampling (Recruitment) Strategy and Rationale. Before the survey portion of the
study began, the study was approved by the SUSD school board through a presentation, sharing
the research questions the study sought to answer and the alignment with SUSD organizational
goals. The presentation provided a positive perspective of the study’s goal to improve SUSD’s
understanding of how data-based decision making supports student achievement. Given the
small population in the sample, less than 20 administrators, the survey strategy sought to take a
census of the beliefs and trends of the entire sample population (Johnson & Christensen, 2014).
The initial quantitative survey occurred early in the study, enabling an explanatory mixed-
36
methods approach, as the survey data identified key factors and trends that were followed-up
through qualitative interviews to develop greater understanding (Creswell, 2014). Prior to the
interviews, survey data was also used to further align the interview questions with the research
questions and revealed trends, as well as to inform the selection of which school leaders to
interview for a diverse perspective. Recruitment of participants for the initial quantitative survey
phase involved utilizing professional relationships and the motivation to improve organizational
practices, encouraging administrators to share their experiences and insights. All participants
were assured that they would receive access to the study’s recommendations, supporting the
improvement of their own leadership and improvement practices.
Interview Sampling Criteria and Rationale
Criterion 1. Like the quantitative phase, the individuals that participated in the interview
phase of the study served the district as school-site administrators. The survey phase identified
administrator trends and values around the process of data use for informing decisions, and the
interview phase sought to probe further into the meaning of data-based decision making and how
administrators viewed its role in the context of improving student achievement.
Criterion 2. Similarly, the administrators that participated in the interview phase
regularly supported instruction and learning at a school. Just as in the survey phase, the focus
was on understanding the influences on administrator ability to develop and support data use
capacity for improving student achievement, so only those administrators were studied. The goal
of the qualitative interviews was to explore the findings of the survey data to provide a deeper
understanding (Creswell, 2014). Thus, the survey data informed the recruitment of
administrators that represented important perspectives and trends for supporting data-based
decision making. According to Creswell (2014), the phase two qualitative interview strategy
37
should focus on individuals that were part of phase one, and the authors also note that the goal is
to explain how the variables interact. Hence, the mixed method approach enabled the study to
identify key administrators during the first phase of the study that would add additional
information about the relationships between the variables.
Interview Sampling (Recruitment) Strategy and Rationale. Transitioning into the second
phase, the recruitment of a purposeful sample involved reaching out to identified administrators,
sharing the results of the survey and requesting an interview for feedback and additional insights
revealed through the trends. Motivation to participate in the interview phase focused on the
value that could be gained through reviewing the findings and discussing the connections to the
administrator’s setting within SUSD. Merriam and Tisdell (2016) emphasize that a focus of
sample selection in qualitative studies should be to ensure information rich data, providing a
depth of knowledge around the problem of practice. Thus, to represent the challenges and
factors tied to the needs identified through the research questions, the sample population
included a range of administrator levels of experience, grade levels supported, backgrounds, and
proficiency with data use to inform decisions. The number of interviews was prioritized to
ensure that range of leadership experience, grade levels supported, and educational and
professional backgrounds present among SUSD administrators would be reflected in the sample
studied. Working within the cultural context and setting of the organization, each leader had
their own understanding and methods for supporting the organizational goals, as well as
alignment of motivational factors influencing their commitment to the goals (Clark & Estes,
2008; Schein, 2017). Additionally, the explanatory mixed-method nature of this study sought to
deepen the understanding of how data-based decision making is used in SUSD to improve
student achievement, so the intentional diversity of the sample provided more administrator
38
variables to consider. Whether an administrator is new to leadership, has many years of
experience to draw upon, or comes from a different professional background, it is important to
understand the range of factors involved district-wide and their impact on the effectiveness of
data-based decision making.
Data Collection and Instrumentation
The study utilized a mixed-method explanatory design. Through an explanatory design,
data collection began with a quantitative survey that provided an overview of the influences
followed by a qualitative interview to generate a deeper understanding (Merriam & Tisdell,
2016). Data collected from the survey informed the selection of interview participants.
The survey component of the data collection sought to gather a general understanding of
the knowledge, motivation, and cultural influences tied to school leader support of data-based
decision making to improve student achievement. The data was used to group the participants in
terms of their level of experience and the type of school they supported, as well as to examine
the reported level of effect each influence had on their ability to support data-based decision
making at their school.
Interviews were used qualitatively to collect data around the research questions,
providing rich data about the influences on school leader groups identified through the survey.
Interviews followed the survey so that the quantitative survey data could guide the purposeful
selection of interview participants. The goal was to select participants, maximizing variation by
identifying participants of different backgrounds and different levels of reported knowledge,
motivation, and cultural influences (Merriam & Tisdell, 2016). Interview data provided a rich
depth of understanding for the knowledge and motivation of school leaders for using data-based
decision making to improve student achievement. Additionally, interviews sought to describe
39
the cultural organization structures that were in place to support data-based decision making and
differentiate the structures in terms of those that are organization-wide and those unique to a
school site.
Finally, to measure the progress towards the stakeholder goal of supporting data-based
decision making to improve student achievement, the assessment data results of the district’s
participation in the state standards-based assessment, CAASPP, were to be compared with data
from the prior year. However, in the spring of 2020, California schools were closed due to
concerns with safety associated with the pandemic outbreak of COVID-19. As a result, the
CAASPP assessments were cancelled and data was not available for comparison to answer
research question 1.
Surveys
Survey instrument. The survey sought to gain a broad understanding of the knowledge,
motivation, and cultural influences that affect school leaders’ ability to support data-based
decision making at their school sites. For example, the orientation of organizational goals for
data use influences the motivation of teachers. If teachers perceive that the goal of data use is for
accountability and evaluation, they may be less motivated to use data to inform instruction on a
regular basis, limiting the use of data to times when it is mandated or required. However, if the
goal of data use is focused on supporting teachers to identify how to improve learning
opportunities for students and in turn student proficiency, teachers may be positively motivated
as their professional goals align with those of the organization. It is also important to determine
the types of structures or resources that are in place to support teachers with data analysis and
pursuing action steps, bolstering the persistence of data use when it becomes challenging. Thus,
40
the survey also sought to gain an initial description of the state of data use support by school
leaders across SUSD, so that qualitative interviews could dig deeper into specific school settings.
The survey included twenty-four questions. The questions were generated using a table
of specifications aligning each item with a research question and the corresponding influences
from the conceptual framework. When using a table of specifications, topics and associated
subtopics were grouped, ensuring full coverage and providing an organization that reduced strain
on the participant (Irwin & Stafford, 2016). The survey allowed for several nominal grouping
multiple choice questions such as identifying the type of school they served and how long they
had worked as a school leader. Following the nominal grouping questions, each participant was
presented with five questions per research question. The ease at which the data was manipulated
was dependent on the types of data gathered, so it was important to be intentional about the
nominal and rating items included in the survey. Each item covered only one topic or thought
and kept the language simple for comfortable reading (Fink, 2013; Irwin & Stafford, 2016).
Additionally, nominal and ordinal items focused on grouping or categorizing participants, while
rating-based items had points associated for ranking purposes (Salkind, 2017). Five multiple
choice or rating scale questions allowed for the measurement of knowledge, motivation, and
cultural influences tied to each research question. Functionally, the survey facilitated both the
groupings of participants, as well as ranking each school leader according to their knowledge,
motivation, and cultural influences through the point-based items aligned by the table of
specifications. The survey concluded with an open-ended question to allow for additional
insights to be shared (Fink, 2013).
Survey procedures. The online survey was created using Qualtrics and sent to
participants via email, so that the spreadsheet data output could be readily manipulated for
41
analysis (Irwin & Stafford, 2016; Salkind, 2017). Given the audience consisted of school
leaders, they were comfortable corresponding via email and familiar with online survey
mechanics.
Interviews
Interview protocol. To develop a depth of understanding that maximized each
participant’s input a semi-structured protocol was utilized for interviews. The interview
sequence adjusted, inserting probing questions and altering the order of questions in response to
participant answers. An overall interview guide was used, ensuring that all interview items were
covered, but it was flexible enough to allow for the researcher to mix up the order when
participant responses lead to deeper connections along particular paths of questioning (Patton,
2002). The sequence of items also included variation in question types to collect a range of data,
including participant beliefs, feelings, motivations, knowledge, and metacognitive decision
processes (Merriam & Tisdell, 2016). It was important to determine how school leaders interact
within the culture of teacher collaboration and utilize the data use resources, as well as how their
knowledge of data-inquiry and beliefs influence their ability to support the improvement of
student achievement. For example, providing a participant with a scenario, roleplay, or devil’s
advocate question provided a glimpse into their beliefs, priorities, and the metacognitive process
they go through when faced with a challenge (Merriam & Tisdell, 2016; Patton, 2002).
Interview procedures. Due to the nature of the mixed-method explanatory design, the
interviews did not occur until after the survey data analysis has concluded. The goal was to
complete eight interviews, selecting four school leaders supporting a secondary school, middle
school or high school, and four supporting an elementary school, while also representing
different levels of experience, knowledge, and motivation. School leaders were interviewed
42
individually, and qualitative data was reviewed throughout the process to ensure that a saturated
level of data was collected, describing the influences (Merriam & Tisdell, 2016). Interviews
lasted approximately one hour and took place in a comfortable environment for the participant,
typically in their school’s office. The interview environment and duration were established to
provide comfort for participants, lowering the anxiety they may experience (Bogdan & Biklen,
2007; Merriam & Tisdell, 2016). While handwritten notes were taken, a digital recorder was
utilized to record interviews after the researcher received permission from each participant
(Bogdan & Biklen, 2007; Patton, 2002). Recordings were then sent to a transcription service to
generate robust data documents for analysis.
Data Analysis
Survey data analysis included the calculation of frequencies for each response item.
Additionally, the percentage of stakeholders who strongly agreed or agreed were presented in
relation to those who strongly disagreed or disagreed. Descriptive statistical analysis was
conducted once all survey results were submitted.
For interviews, data analysis began during data collection. Analytic memos were written
after each interview. All thoughts, concerns, and initial conclusions about the data in relation to
the conceptual framework and research questions were documented. Upon leaving the field,
interviews were transcribed and coded. In the first phase of analysis, the use of open coding
looked for empirical codes and applied a priori codes from the conceptual framework. A second
phase of analysis was conducted to aggregate empirical and a priori codes into analytic/axial
codes. In the third phase of data analysis, pattern codes and themes that emerge in relation to the
conceptual framework and study questions were identified.
43
Credibility and Trustworthiness
Credibility and trustworthiness were established in three ways, dedicating enough time to
collect rich data describing the problem, revisiting interview participants for respondent
validation, and triangulating data using survey, interview, and current literature. If rich data is
collected and triangulated from multiple sources, then credibility is improved, minimizing the
perception of bias and reflexivity (Maxwell, 2013). An important aspect of rich data depended
on the connection between the survey and how it informed interview sampling. Survey data
provided a maximally diverse sample that described the rich data around the influences, such that
the interview data represented the full depth within the organization (Maxwell, 2013; Merriam &
Tisdell, 2016). The effect of researcher bias was also reduced by providing respondents the
opportunity to review the data they provided (Merriam & Tisdell, 2016). Following each
interview, participants had the opportunity to clarify their responses or correct a
misunderstanding, further nullifying any bias the researcher may have brought into the
interpretations.
Validity and Reliability
The initial survey items were piloted with peers that did not belong to the study’s
participant sample. Feedback from peers was gathered through a follow-up interview to review
the wording of the items, the sequence of the items, and confirm construct validity (Irwin &
Stafford, 2016; Salkind, 2017). Construct validity examines whether items that are difficult to
measure, such as those tied to motivation, were interpreted by the pilot participants in the way
that the researcher intended (Salkind, 2017). Feedback was used to update the survey items,
including reordering several items to improve the flow and reduce strain on the participants
(Irwin & Stafford, 2016). Before sending the survey, score-based items were reviewed to ensure
44
that data could be efficiently analyzed to identify a sample for interviews. Effort was made to
ensure the scoring was thoughtful with alignment to research questions and influence coverage
so the data would represent all aspects of the problem (Fink, 2013). When ready, the survey was
announced in a meeting of all school leaders, and then shared via email. Participant names were
collected to facilitate follow-up, and to ensure as close to census-level participation as was
possible. Reminders included information about how the results of the study would be shared
out to provide insight for all schools, and effort was made to ensure that all levels of school
leadership were represented in the data collected to minimize bias on non-responses.
Ethics
This research study followed all of the expectations and obligations for ethical research
practices (Glesne, 2011; Rubin & Rubin, 2012). The primary researcher was a district leader that
actively participated in supporting school leaders with data use to inform instructional practices.
As such, there was a direct link between the researcher’s organizational role and the study,
however the researcher’s relationship with the participants was peer-to-peer. In terms of a
conflict of interest, the peer-to-peer relationships limited the potential for the abuse of power
(Glesne, 2011; Rubin & Rubin, 2012). The primary researcher was, however, cognizant of their
biases and assumptions that they would have as part of the organizational culture (Glesne, 2011).
As part of the data use support at the district level, there were assumptions and procedures that
are recommended for sites to follow. However, the evaluation study sought to understand the
support that was provided to teachers at their school site, so the researcher approached the survey
and interviews with an open mind about the practices and methods that school leaders had
established for their school. Due to the peer-to-peer positionality of the survey and interviews, it
was necessary to remind participants that their names and schools would be anonymous and
45
protected from inspection through pseudonyms. Anonymity was emphasized in order to
encourage participant data to be as genuine as possible. It was also made clear that the findings
of the study would be shared with all participants, providing insights to guide school leaders in
improving how they support data use. Ethical considerations for preparing participants included:
demonstrating respect for all participants, following-through with sharing research findings,
ensuring informed consent for all participants, and confirming that all unnecessary risks to
participants are removed.
Given that this study focused on how school leaders, such as principals and assistant
principals, support teachers in their use of data to inform instruction, it was important to
approach all ethical considerations in a way that was respectful and sensitive to their position and
the reputation of the district. As recommended by Rubin and Rubin (2012), pseudonyms were
used for the district and, when necessary, to reference all schools and individuals that were
quoted. This included referring to school leaders generically as stakeholders so as not to reveal
their specific position within the district, protecting their identity. School leaders were reminded
of these protective steps in both stages of the study, when taking the initial survey and when they
were interviewed. The interviewer requested participant consent to record the interview to allow
for further analysis later and was willing to turn off the recorder if requested by the participant
(Rubin & Rubin, 2012). Participants were also assured that the files and data collected in the
study would be stored in a drive protected from access by individuals other than the researcher.
Additionally, the files were destroyed upon completion and acceptance of the dissertation study.
The main goal was to ensure participant cooperation and genuine communication by following
through with respecting school leader privacy.
The study also worked diligently to follow through on all promises and commitments to
46
participants. While the study did not provide any form of monetary or gift reciprocity, as is
common practice for research, the findings were shared with the participants (Glesen, 2011;
Rubin & Rubin, 2012). School leaders should find benefit in access to the findings to inform
their practices for supporting teacher data use. The key point communicated with participants
was that the research and findings would depend on their ability to provide candid and detailed
responses so that the study could accurately describe the conditions within the district.
Both explaining the steps that were to be taken to maintain participant privacy and
sharing the value of participating in the study were part of establishing informed consent for all
participants. It was important for participants to have a complete and truthful understanding of
the expectations, methods, and risks involved (Glesne, 2011; Merriam & Tisdell, 2016; Rubin &
Rubin, 2012). Informed consent is a core component of IRB-approved ethical research (Rubin &
Rubin, 2012). As such, informed consent also includes making sure that all participants
understand they are not obligated to participate and that they have the freedom to remove
themselves from the study at any time (Rubin & Rubin, 2012). Informed consent was tracked for
each participant through a signed document, and participants were reminded of their ability to
leave the study prior to all interviews. This included detailing the types of data that were
collected, the research questions the study seeks to answer, and detailed participant rights for
consent and withdrawal to their participation. Participants joined the study feeling protected and
understanding that if their feelings changed, they could withdraw from the study.
Ethically protecting participants at its core focuses on ensuring that the study does no
harm, which includes minimizing all risks to the participants that can possibly be eliminated
(Rubin & Rubin, 2012). For example, it is possible that a school leader may not want to
participate because they are not confident in their abilities to lead data use for their staff. They
47
may be concerned that the information about their personal abilities or challenges will be shared
with other school leaders or district administration. Thus, it was of utmost importance to provide
participants with details regarding the steps that would be taken to protect their confidentiality,
hiding their names from everyone but the primary researcher and storing all data in a digital form
that was only accessible to the primary researcher (Merriam & Tisdell, Rubin & Rubin, 2012).
Additionally, it was important that if any participant of a study was hesitant about participating,
they would not be further pressured into participation (Rubin & Rubin, 2012).
Preparing stakeholders to participate in the study included demonstrating respect for all
participants, following-through with all promises to participants, ensuring informed consent for
all participants, and confirming that all unnecessary risks to participants were removed. These
steps were essential to meeting the ethical obligations of formal research.
Limitations and Delimitations
The anticipated limitations of the study included the small number of participants, the
truthfulness of the respondents, and the role of the researcher in the organization. The mixed-
method approach facilitated a purposeful selection of participants within the small sample and
included open-ended and exploratory questions to strengthen the accuracy of responses. The
order of the items within the survey and interview intentionally sought to develop a less
threatening engagement and to allow time for engagement to build before asking more
challenging or personal questions. During the interviews, the researcher sought to listen
carefully, providing reflection and summaries to clarify responses and build empathy while
inserting probing questions where necessary to build on the accuracy of the response.
Additionally, all participants were informed before the survey and interview that they would
have the opportunity to validate their responses by reviewing the study results and providing
48
feedback, including the ability to request the removal of statements. Respondent validation
served as an opportunity to build trust with participants allowing them to check responses for
misinterpretation (Merriam & Tisdell, 2016). In terms of the researcher’s role within the
organization, the researcher served as a district administrator supporting data gathering and use.
While the researcher was a peer to school leaders and did not serve in any evaluative nature, it is
possible that participants may have responded in ways that they believed would positively
influence their superiors. Some of these concerns were addressed in the previous discussion of
the study’s ethics, and when combined with survey and interview procedures the authenticity of
responses should have been increased. Every effort was made to communicate and remind
participants that their participation would be anonymous and that at no point would their
responses be shared with their superiors.
The delimitations of the study include that both elementary and secondary school leaders
participated, providing data from the full range of grade levels where data was used within the
school district. Interview participants in each group were also selected based on experience,
knowledge, and motivation responses to the survey in order to capture data representative of both
new and well-experienced school leaders. The use of a survey and interview data analysis
sought to increase the amount and quality of the data gathered, providing the opportunity for
triangulation with current literature, as well as the identification of conflicting data points to
enrich understanding and generate valuable recommendations.
Conclusion
In order to evaluate the capacity of school leaders for leading staff in the use of data to
improve student achievement, a methodology was developed that considered the challenges of
understanding associated knowledge-based, motivational, and cultural influences. The
49
researcher approached data collection and analysis with careful and constant focus on providing
a credible, trustworthy, and ethical outcome to provide insight for the organization and
stakeholder group in their efforts to use data to inform the improvement of student achievement.
50
CHAPTER FOUR: RESULTS AND FINDINGS
Over the course of the study, two rounds of data collection occurred. Initially,
quantitative data was collected through a survey and then followed by the collection of
qualitative data through interviews. Data was then analyzed to describe the principal knowledge
and skill, motivation, and organizational influences involved with school leaders supporting
data-based decision making to drive the improvement of student outcomes. The results were
compared with the assumed influences of the knowledge and skills, motivation, and
organizational influences discussed in Chapter Three, to identify whether each served as an asset
or need as school leaders sought to lead staff data use. Recommendations are discussed for areas
of need, and assets rejected as a gap in knowledge and skills, motivation, or organizational
influences are identified.
This evaluation study sought evaluate the capacity of school leaders to lead their staff in
the use of data, driving the improvement of student outcomes. The research questions guiding
this evaluation study are the following:
1. To what extent did school leaders improve student achievement through the
promotion of data-based decision making in their schools?
2. What knowledge and motivation did school leaders have related to increasing
student achievement through data-based decision making?
3. What was the interaction between Sunshine Unified School District culture and
context and school leader knowledge and motivation for increasing student
achievement through data-based decision making?
51
4. What are the recommendations for organizational practice in the areas of
knowledge, motivation, and organizational resources for achieving increases in
student proficiency through data-based decision making?
In this chapter, the Clark and Estes (2008) KMO framework organizes the results of the
data into the following sections:
● Results and findings for knowledge influences;
● Results and findings for motivation influences; and
● Results and findings for organizational influences.
The results and findings for each section of influences emphasizes why each influence
was identified as an asset or need. Chapter Four concludes with a synthesis of how each
influence identified as a need need supported each research question for the study.
Participating Stakeholders
The participants in the quantitative survey portion of the study included sixteen school
leaders. Survey participation was 88%, as sixteen of the total eighteen school leaders employed
by SUSD chose to participate in this portion of the study. Each of the sixteen school leaders was
an employee of SUSD during the 2019-20 school year and served as a credentialed administrator
at a school site, supervising teachers and instruction. The participant group included
representation from each of the district’s nine schools. Additionally, fifty percent of the
participants represented administrators currently serving in an elementary school and the other
fifty percent represented administrators currently serving in a secondary school. Participation
from school leaders representing each of the schools and with equal representation of elementary
and secondary school environments provided broad understanding of the experience and
competencies across the district. Table 5 represents the school leaders that participated in this
52
study as related to the established criteria from Chapter Three, including the type of school they
supported and the total number years they have served in education.
Table 5
Demographic Information of Survey Participants’ Service Years in Education
Years of
Service
Service of Current Elementary
Administrators in Education
(Any Role)
Service of Current Secondary
Administrators in Education
(Any Role)
0-10 0 0
11-20 5 4
21-30 2 2
31-40 1 2
Table 6 represents the school leaders that participated in this study as related to the established
criteria from Chapter Three, describing the type of school they supported and the number years
they have served as administrators in education.
Table 6
Demographic Information of Survey Participants’ Service Years as an Administrator
Years of
Service
Service of Current Elementary
Administrators in Administration
(Any Role)
Service of Current Secondary
Administrators in Administration
(Any Role)
0-10 6 3
11-20 1 3
21-30 1 2
For the second half of the study, the participants consisted of eight school leaders
employed by SUSD, four from elementary schools and four from secondary schools. Each
school leader that was interviewed, previously participated in the quantitative portion of the
study, and had a primary role of overseeing and supporting classroom instruction. Table 7
represents the school leaders that participated in this study as related to the established criteria
53
from Chapter Three and includes the type of school they supported at the time of the study, as
well as the total number of years they have served education.
Table 7
Demographic Information of Interview Participants’ Service Years in Education
Years of
Service
Service of Current Elementary
Administrators in Education
(Any Role)
Service of Current Secondary
Administrators in Education
(Any Role)
0-10 0 0
11-20 2 2
21-30 1 1
31-40 1 1
Table 8 represents the school leaders that participated in this study as related to the established
criteria from Chapter Three, including the type of school they supported and the number of years
they have served as an administrator.
Table 8
Demographic Information of Interview Participants’ Service Years as an Administrator
Years of
Service
Service of Current Elementary
Administrators in Administration
(Any Role)
Service of Current Secondary
Administrators in Administration
(Any Role)
0-10 3 2
11-20 0 1
21-30 1 1
Determination of Assets and Needs
Sources of data for this research study were administrators leading schools within SUSD.
Data collected was analyzed to identify and develop a comprehensive understanding of the
knowledge, motivation, and organizational influences.
Survey data from school leaders provided a broad understanding of the data-based
practices and structures in place at each school in the district, while interview data provided a
deeper description of the similarities and differences between elementary and secondary schools,
54
as well as the challenges for supporting effective data use among staff. A baseline for the
presence of influenced-based practices was established using the survey, identifying practices
that may be lacking or in disagreement to follow-up on during the interview phase. The survey
questions focused on factors connected to the knowledge, motivation, and organizational
influences. However, for some knowledge and motivation influences the survey questions were
not specific enough to provide direct data supporting an influence as an asset or need. However,
the survey data was used to identify the possibility of a need that could be explored further
during an interview. The following sections for results and findings for knowledge and skill,
motivation, and organizational influences begin with school leader survey results to identify the
factors for each influence that the leaders agreed on as important to leading staff in the use of
data for improving student outcomes.
Further analysis of the results through interview data was used to support the
identification of each influence as an asset or need. Influences identified as a need are addressed
in the Chapter 5 recommendations for developing school leader capacity to lead effective data
use with staff. While influences identified as assets, are highlighted in the summary for how
they support data use for the improvement of student outcomes.
Results and Findings for Knowledge Causes
The study included four assumed knowledge influences. Table 9 shows the assumed
factual, procedural, and metacognitive assumed influences in terms their identification as assets
and needs.
Table 9
Knowledge and Skill Influences Assets and Needs
Category Assumed Influences Asset Need
55
Factual School leaders must know the phases
of data analysis.
√
Conceptual School leaders must know which
collaboration models are most
effective for each phase of data
analysis.
√
Procedural School leaders must know how to
effectively provide accountability for
PLC data use.
√ √
Metacognitive School leaders must know how to
effectively provide guidance for PLC
data use.
√
Findings for knowledge influences were organized using Krathwohls’ (2002) four
knowledge dimensions: factual, conceptual, procedural, and metacognitive.
Factual Knowledge
Influence 1. The factual assumed knowledge influence, stated as “School leaders must know the
phases of data analysis,” was identified as need by school leaders through interview questions
related to whether school leaders knew enough to model the data use process, as well as
implement the phases of data analysis leading to establishing goals and measuring progress.
School Leader Survey results. The survey results did not directly identify factual
knowledge influences for the study. Individual interviews were used to identify specific assets
and needs associated with factual knowledge.
School Leader Interview findings. The interview data provided insights that confirmed
factual knowledge-based gaps in terms of each school leader’s ability to support data-driven
improvement of instruction. All four elementary school leaders communicated confidence in
their knowledge of instructional practices and assessments for supporting literacy. School
Leader Participant #4, explained that, “we are strategic about data use to improve reading,
56
calendaring student assessments for the year in advance, and meeting proactively before data
deadlines for teams to look at the needs of students.” However, all four elementary school
leaders also expressed a need to improve their knowledge for identifying and supporting student
mathematics skill needs. School Leader Participant #3 summed the need up as, “we have a
wealth of qualitative understanding of each student's reading behaviors, but we have found it
difficult to find the same level of understanding for math skills.” In contrast, all four secondary
school leaders expressed that their data use was outcome focused, examining student grades and
work habits. School Leader #2 described data-focused work in secondary schools:
One of our primary goals is for zero D’s and F’s. Teachers review their D’s and F’s
throughout the year, identifying needs and gaps for students to pass. The grade students
have at the 12-week mark is generally indicative of their semester mark, so we need to
intervene early. The big three subjects we focus on are Biology, Algebra, and English,
which are key for freshmen.
Much of the data-driven work in secondary schools seeks to intervene when students are unable
to keep up with instructional expectations. All the secondary school leaders interviewed also
expressed a gap in factual knowledge for how to identify and support specific skill-based student
needs. School Leader Participant #7 shared a story supporting the value of knowing how to
effectively support student skill needs:
Knowing how to help students grow is difficult and takes time to identify. We often try
to meet the need by providing additional time through intervention services. There are
many ways to do this, and sometimes it works and sometimes it doesn’t. Through our
attempts we discovered the best method for math was adding more math time to the day
with their regular math teacher. Previous intervention models had students attending
57
extra time with an intervention teacher, but there was a disconnect with the regular math
teacher. Extra math time covering the same content with the same teacher provided a
bigger bang. The extra time allowed the struggling students to front load and make more
of the regular class time. Students with D’s and F’s changed to B’s and A’s. Whatever
you do with the extra time, it is critical that what you are doing is targeting the specific
skill gap. The challenge is knowing how to identify and provide the right type of
intervention for different skills.
Summary. The data collected through the interviews revealed most school leaders have
factual knowledge of data-driven practices in their schools. While the survey data did highlight
the possibility of gaps in knowledge by identifying school leaders that disagreed about using a
consistent data analysis process, the disagreement could have indicated a lack in knowledge of
data analysis processes, motivation-based influences for engaging in consistent data use, or
organizational structures to support a process. Thus, the interviews were necessary to confirm
that factual knowledge gaps, revealing the gaps as a lack of understanding for how to identify
and support different academic skills. It is important for school leaders to develop knowledge of
how to identify and select indicators of skill that can be measured in the search of solutions
(Marsh and Farrell, 2014). Furthermore, school leaders and teachers need knowledge of how to
formatively assess for needs, develop action steps, and adjust instruction (Datnow, Park &
Kennedy-Lewis, 2013; Elmore, 2002; Marsh & Farrell, 2015; Rueda, 2011). In the elementary
schools, leaders were confident in their tools for measuring student needs for ELA, but they
indicated a need for additional understanding of how to measure Mathematics. In contrast,
leaders of secondary schools described a focus on grades, not on measuring specific skills
students need. The secondary schools invest the time they have for data work on making sure
58
students graduate, and spend little time looking for specific student proficiency gaps that need
support. This is due in part to the revealed knowledge gaps and complexities involved in
creating standards or skills aligned assessments for measuring student needs in a secondary
school setting. However, the focus on grades is also influenced by the high priority secondary
schools put on student graduation rates, as well as staff that attribute a lack of achievement to
student-centered causes such as poor work habits. As school leaders increase their factual
knowledge of how data use can be leveraged to improve student achievement in different
academic subjects, they could approach the challenges of student proficiency and graduation in a
balanced approach where one outcome improves the other. Ultimately, moving closer to their
goal for increasing student achievement in English Language Arts and Mathematics.
Conceptual Knowledge
Influence 1. The assumed conceptual knowledge influence, stated as “School leaders
must know which collaboration models are most effective for each phase of data analysis,” was
identified as a need by school leaders through interview questions related to establishing a
process for data analysis and utilizing multiple sources of data.
School Leader Survey results. The survey results did not directly identify conceptual
knowledge influences for the study. Individual interviews were used to identify specific assets
and needs associated with conceptual knowledge.
School Leader Interview findings. Interview data provided insights that confirmed
conceptual knowledge-based gaps in terms of each school leaders’ conceptual knowledge
influences of data use. While every elementary school leader described that their staff and team
meetings regularly review data to inform planning, they also shared that teams were still in the
59
process of learning to build common formative assessments. School leader participant #1
shared:
As a school, we have a structure for assessing student learning and providing reteaching
or intervention, as needed. Teams have begun to develop common assessments aligned
with the district-wide benchmark assessment scope and pacing to further measure student
growth. Our shift grew out of each team recognizing the ability for formative
assessments to provide specific measures around a standard or skill, informing the
improvement of lesson alignment with standards and learning-goals. We are moving
towards a process for assessing skills, identifying needs, providing supports, and then re-
assessing for growth.
In contrast, all of the secondary school leaders shared that while teacher teams meet regularly, it
is challenging to provide enough time for teachers to regularly incorporate a data analysis
process, informing their plans and goals. Secondary school leader #6 described the challenge:
It is challenging to create time to analyze the data. It would help to build beyond the
collaboration time that currently exists. Many of our teachers support over a hundred
students and have multiple subject teams, forcing them to alternate their meetings
between subjects. They really need a half- day substitute to dive into the data with their
team, extract what they need to support, and plan for what to change. They may need an
admin present to help structure the work and understand the process, as well as offering
structural shifts, such as class sizes, support staff, etc. Finding ways that bring our
teachers together consistently and in the right context to establish a data-informed process
is a real challenge.
In addition to the challenge of time for establishing a process, three of the four secondary school
60
leaders described challenges related to utilizing multiple measures, Secondary school leader #7
shared:
Data is cumbersome and the content areas change. If you course doesn't match the grade
level standards that adds more complexity. It's not easy to track. Multiple measures at
secondary are subjective because most of the assessments are created by the teachers. If
the benchmark does not match the scope and sequence of the course, then we lose that
measure. You need the benchmarks to be spot on or we have no real objective measure
of proficiency that can inform adjustments.
Summary. To improve teaching and learning opportunities, teachers must work
collaboratively to analyze student progress and ask questions about classroom practices (Katz
and Dack, 2014). Furthermore, teams need to meet regularly to discuss data, developing a
common process and be able to monitor practice to adjust (Van Gasse et al., 2017). Elementary
school leaders have found it easier than leaders of secondary schools to establish collaborative
teams that meet regularly to examine data, but they are still growing in their ability to construct
tools for measuring growth effectively. While there are structural and cultural elements of the
challenge, the conceptual knowledge of how to embed skill or standard-based formative
assessments within learning to inform improvement is essential. Secondary school leaders
expressed that the complexity of different subjects and multiple course assignments, and the
larger number of students assigned to each teacher were barriers to establishing a consistent data
process. However, while the complexity of secondary school courses and population sizes may
play a role, the elementary school leaders indicated that a key element to success was the use of
common formative assessments aligned to a content standard or skill. The factual knowledge
gap for knowing how to identify skills needs may be combined with a conceptual knowledge gap
61
for how to establish assessments that provide informative data for improvement. School leaders
that support teams with developing formative assessments aligned to learning goals, that provide
quick and actionable direction for meeting learning needs, will see improvement towards the
organizational goal of improving student achievement in English Language Arts and
Mathematics.
Procedural Knowledge
Influence 1. The assumed procedural knowledge influence, stated as “School leaders
must know how to effectively provide accountability for PLC data use,” was identified as both
an asset and need by school leaders through survey and interview questions related to the
implementation of accountability procedures through an established structure for data-based
decision making and the creation of improvement goals.
School Leader Survey results. The survey results confirmed school leaders found
accountability tools, such as creating a structure through the provision of specific materials and
establishing specific and measurable goals through data use, to be priorities for improving
student achievement (Table 10). All the elementary school leaders indicated that they provide
teachers with specific materials and resources to structure data analysis, and that improvement
goals are clear, specific, measurable, and based on student data. However, among secondary
school leaders, 20% communicated a disagreement that the data-based improvement goals they
hold teams accountable for are clear, specific, and measurable. Additionally, 10% of the
secondary school leader responses indicated that teachers are not provided with materials and
resources to structure data use. The data informed interview participant selection and follow-up
questions during the interview to learn about the disagreements and identifying possible gaps.
Table 10
62
School Leader’s Procedural Knowledge of Practices that Support Data-Driven Schools
School Leader Interview findings. Interview data from the school leader participants
provided additional insight into the procedural knowledge influences impacting the
implementation of data use accountability procedures and confirmed the presence of procedural
knowledge gaps. Every elementary school leader interviewed confirmed that they provide
materials to support regular accountability for PLC teams, including the adjustment of school
resources, such as intervention teachers, and professional development plans. However, this
practice was not universal in implementation, with differences in timing and available resources.
Elementary school leader participant #5 shared an example of a structured implementation:
We meet as a staff after every data collection event, our benchmarks, data is printed out
for teams, we look at the gaps and areas of need, we model how to get to each question
item and find the trends, and then teams discuss how to adjust instruction. It’s a powerful
time when teachers are sharing instruction practices, learning from each other, and really
teaming up to help all students.
63
Three of the four elementary school leaders also indicated the support and accountability for
PLC data use in elementary schools extends beyond staff meetings, as elementary school leader
participant #4 shared:
I meet personally with grade-level teams, going into team meetings to answer questions
and follow-up, personally looking at their data and being a part of their planning. As I
manage the intervention lab support based on the data, I share the intervention schedule
with teams, making sure that students with a need are still getting support. Really,
anytime teachers say they need something, a day to plan, etc. I never say no. It's a
purposeful time and the day produces a whole plan, a 6-week plan for their intervention
or what they will do to help the kids that need it.
All the school leaders from secondary schools also confirmed that they hold PLC teams
accountable for data use to improve student achievement, but they all added that data work with
teachers was challenging due to the complexity of secondary school teaching assignments.
Secondary school leader participant #6 explained:
[Secondary school] teachers may teach multiple subjects and have one-hundred and
seventy-five or more students. The data should always be presented, especially data
associated with accountability, such as attendance, semester grades, or state test scores,
with the narrative behind it. Otherwise teachers can misinterpret the data as they draw
their own conclusions. We participate in the data conversations with teachers and focus
on conclusions and goals that are productive and within our ability to control.
Accountability is a strong word because it implies consequence or discipline, but in
reality, it's accountability to your practice. What are the reflections or things I can take
away from this to adapt my pedagogy and grow? How can I change my approach to meet
64
the needs of specific students or populations? Collaborative conversations help us
strengthen each other and could lead to building in professional development options or
instructional networking to facilitate learning, conversation, and observation of new
options for supporting students.
Further description of the challenge in secondary schools was provided by secondary school
leader participant #2:
PLCs are one place for accountability toward improving learning opportunities to be most
effective. Ideally, they need to meet once a week to develop common assessments and
evaluate progress. Accountability is a follow-up behavior, a check-in, and works best as
an immediate reflection. It's about the next step if follow-through towards growth and
progress. In [secondary schools] teachers have multiple preps, preventing them from
meeting with every content team every week. It’s difficult to create an ongoing
conversation for the team to work together, developing trust and accountability, as well
as build the habits of mind and the formalized process that we need.
Summary. Collaborative PLC work is essential for establishing a culture of data-based
decision making for the improvement of student achievement (Huguet et al., 2017; Van Gasse et
al., 2017). The capacity for using data to inform decisions and instruction requires time and
structure to develop, establishing professional efficacy for identifying needs, setting goals, and
perseverance to completion (Black & Wiliam, 2010; Dowd, 2005; Huguet et al., 2017). For
teams to be held accountable for data use, they must have the tools and resources necessary to
accomplish the task. The survey and interview data confirmed that there are procedural
knowledge needs that can be addressed to improve progress towards the organizational goal.
School leaders can directly impact the compliance and success of PLC data use by providing
65
resources and professional development, supporting the data inquiry process, as well as the
action steps in pursuit of the identified goals (Elmore, 2002; Marsh & Farrell, 2015; Rueda,
2011). Elementary school leaders shared that many of these structures are in place, serving as an
asset, and their teachers regularly assess to review progress towards data-based goals. In
contrast, secondary school leaders describe the multiple subjects and complex teacher schedules
in secondary school as a barrier to establishing strong PLC groups that follow a structured data
use process. It takes time to build a productive PLC team, and when the teachers collaborate it is
important for the data and the process to help focus the work (Datnow et al., 2013). Focused
data is essential to reveal patterns of student achievement, but the PLC must be productively
collaborative to be willing to share practices, ask questions, seek help, and determine what to do
differently in the classroom. School leaders that enable PLC data use through structures for
accountability, resources, and training will see improvement towards the organizational goal of
improving student achievement in English Language Arts and Mathematics.
Metacognitive Knowledge
Influence 1. The assumed metacognitive knowledge influence, stated as “School leaders
must know how to effectively provide guidance for PLC data use,” was identified as a need by
school leaders through survey and interview questions related to accountability for modeling
data-driven educational practices, supporting proposed changes with data during meetings, and
talking through the thought process of data-based decisions.
School Leader Survey results. The survey results confirmed that school leaders had
metacognitive knowledge of how to provide guidance for PLC data use (Table 11). All school
leaders agreed that they model data-driven educational practices and talk through the thought
66
process of data use. However, despite the full agreement, follow-up questions during interviews
identifying metacognitive knowledge needs.
Table 11
School Leader’s Metacognitive Knowledge of Practices that Support Data-Driven Schools
School Leader Interview findings. School leader participant interview data examining
metacognitive knowledge influences impacting the use of data analysis during collaboration
confirmed the presence of metacognitive knowledge needs. Three of the four elementary school
leaders identified the value of using standard-aligned assessment data to model data use with
staff, exploring student progress and needs. Elementary school leader participant #4 shared:
By using assessment tools that are aligned to the standards of each grade level, we can
take the time as staff to analyze data that is meaningful to us all. I can share my insights
regarding grade-level or school-wide data, and we bring the TOSAs [instructional
coaches] for reading and math to help with the data interpretation and development of an
intervention plan. Everyone is involved and knows what to expect; we even set the dates
67
before the school years begins, so every teacher knows when to give each assessment and
the data is available to inform intervention planning.
However, two of the four elementary school leaders described challenges with teachers
becoming defensive or guarded when school leaders model data use and facilitate discussions for
improvement. Elementary school leader participant #4 explained:
Understand that it is challenging for teachers, so I do my best to be non-threatening and
welcoming to any questions. Teachers don't want to seem ignorant of the technology or
the data, so it is intimidating, sometimes they may hold back their questions and choose
not to engage. I try my best to bring defenses down to open people up.
In comparison, while all four secondary school leaders did confirm that they model data-based
decision making for staff, they also identified challenges associated with staff that teach different
subjects, as well as examining large student populations. Secondary school leader #6 explained:
Modeling how to use data to inform decisions for staff can be a challenge, since
[secondary school] teachers don’t all teach the same subjects. It’s important to work with
data that is relevant to everyone, like disciplinary data. You want to be careful of
misconceptions too. A lot of the data we look at describes percentages of our student
population. When percentages change by large amounts it can perceived as a significant
disparities or disproportionality, especially with subgroups of our population that are
small compared to the overall population size. A change in just a few students may make
the change in percent look more extreme than when we look at raw numbers.
Understanding the sensitivity of a measure is important so we don't jump to conclusions
that are misguided or create false perceptions or narratives from that point.
Summary. Providing PLC groups with guidance through the mental thought processes of
68
making data-based decisions helps teachers to develop their own framework for using data,
especially if the shared learning model provides an intentional disruption to common
understanding (Katz & Dack, 2014). Through continued practice, a culture of data inquiry can
be developed, establishing a pattern of thought for facilitating change in instructional practices
that extends into teacher led meetings (Katz & Dack, 2014; Van Gasse et al., 2017). The survey
and interview data confirmed that there are metacognitive knowledge gaps that can be addressed
to improve progress towards the organizational goal. The key difference observed between
elementary and secondary modeling is the type of data used. In elementary schools, every
teacher is responsible for all the core subjects. This allows school leaders to model the analysis
of instructional data during staff meetings to develop common language and practice. In
contrast, secondary school teachers are responsible for a specific subject. During staff meetings,
secondary school leaders are challenged to find measures that all teachers can relate to and
discuss. In turn, while the modeling of the data analysis can build common language, it does not
provide the necessary modeling of how to examine subject-specific data from assessments to
identify instruction needs. To provide the subject-specific data analysis modeling, Datnow et al.
(2013) describe the importance of teachers working on a common problem of practice organized
around departmental or subject teams. School leaders that provide effective and relatable models
of data inquiry will see improvement towards the organizational goal of increasing student
achievement in ELA and Mathematics.
Four assumed knowledge influences were presented in Chapter Two. Table 12 shows a
summary of the assumed knowledge and skill influences identified as assets or needs through the
site administrator survey and interview analysis.
Table 12
69
Summary of Assumed Knowledge and Skill Influences Identified as Assets or Needs
Category Assumed Influences Asset Need
Factual School leaders must know the phases
of data analysis.
√
Conceptual School leaders must know which
collaboration models are most
effective for each phase of data
analysis.
√
Procedural School leaders must know how to
effectively provide accountability for
PLC data use.
√ √
Metacognitive School leaders must know how to
effectively provide guidance for PLC
data use.
√
Results and Findings for Motivation Causes
There were two assumed motivation influences presented in Chapter Two. Table 13
shows that how the assumed motivation influences were identified as assets and needs.
Table 13
Motivational Influences Identified as Assets or Needs
Category Assumed Influences Asset Need
Attribution
Theory
School leaders must attribute the cause of
student achievement gaps to be due to
effort or opportunities, not a lack of
student ability.
√
70
Goal
Orientation
School leaders must perceive the goal-
orientation of data use to be improving
learning opportunities for students, not the
judgement of school performance.
√ √
Attribution Theory
Influence 1. The assumed attribution theory motivation influence, stated as “School
leaders must attribute the cause of student achievement gaps to be due to effort or opportunities,
not a lack of student ability,” was identified by school leaders as a need through interview
questions related to whether teachers adjusted instruction based on assessment data and whether
student assessment data was used to set action goals.
School Leader Survey results. The survey results did not directly identify motivation-
based attribution theory influences for the study. Individual interviews were used to identify
specific assets and needs associated with attribution theory.
School Leader Interview findings. School leader interview data exploring attribution
theory-related motivation influences confirmed the presence of motivation needs. Three of the
four elementary school leaders described teacher collaboration processes that monitor student
proficiency around essential skills and provide additional learning opportunities to strengthen
weakness that are discovered. Elementary school leader participant #8 shared:
I think any child can achieve proficiency. They may just need additional time and
supports. Some kids learn at a different pace, we need to recognize what we are looking
for in mastery, the fenceposts, zeroing in on the standards and determining what the big
goals are, the key skills. You may not pass go until you have some level of proficiency
71
with the key standards. Any kid can be successful and achieve if provided the supports
and learning opportunities that they need.
Among the elementary school leaders, participant #3 did highlight that, “it’s important to
understand that some student achievement can be unpredictable due to controllable and
uncontrollable factors. You need to look at each individual kid's growth and set your goals
around the factors that are within your control.” The interview data indicated some
disagreement about secondary school teachers using assessments to adjust instruction, and school
leader participant #2 expanded, “we need to find a way to assess student needs weekly in
[secondary school], making room to adjust, addressing gaps identified through assessment”.
Secondary school leader Participant #7 provided additional insight into the need to adjust
instruction with assessment data:
Let’s say sixty percent of my class fails the test... what do I do? Do I keep teaching and
go on to the next unit, giving everyone Ds? Or do I throw the test out and spend another
four days reteaching and retest? Not all teachers in secondary will do the second. You
can attribute the failure to lack of effort or ability of the student, or you can think the
content might not have been clear enough. Let's go back and look at it again, practice
more. Other teachers might address it through warmups or other methods, backfilling as
they move forward. The key is to identify the students in need, keep them on your radar,
and figure out how to fill the gaps. Every teacher needs to make it part of their practice,
but it takes time to backfill or reteach. It's tough to find the time.
Summary. Anderman and Anderman (2006) emphasize that when examining an
outcome, it is important to be aware of whether the outcome is consistent through time and
whether the cause of the outcome is under their control. The researchers suggest that when
72
school leaders and teachers believe the reason for low student achievement is due to deficits of
student ability or the student’s parental support, they are less motivated to improve instruction
through data use. Additionally, Datnow and Park (2018) share that teachers may unintentionally
use data to confirm what they believe about a student and avoid data that challenges their beliefs.
The researchers caution that this can lead to a culture in which stereotypes and low expectations
diminish the opportunities available to students, as teachers attribute student data to the student’s
ability, not the result of the instruction the student received. The interview data confirmed that
there are attribution theory motivation gaps that can be addressed to improve progress towards
the organizational goal. Interview participants described elementary school practices where
teachers are using assessments to adjust instruction along the way, controlling learning
opportunities to meet the needs of students. However, there is room for growth in terms of
evaluating challenges in terms of factors that are controllable. In the secondary school setting,
while it may be difficult for all teachers to adjust instruction based on assessments, the system
focuses on multiple measures, including progress report and semester grades, attendance, and
discipline records. The team of professionals, including school leaders, guidance counselors, and
teachers, monitor student growth for indicators of need. Datnow and Park (2015) support this
structure, stating that staff can make stronger informed professional decisions when they can
analyze student needs based on multiple metrics, including attendance, benchmarks, screening
tools, and classroom-based formative assessments. The secondary schools demonstrate a need to
improve their use of common formative assessments and benchmarks yet can also leverage the
strength of their professionals to monitor other important metrics. As teachers and other
professionals raise concerns about measures such as progress report grades, the response and
support of school leaders and guidance counselors is essential. As the system addresses the
73
concerns through the provision of additional learning opportunities or other resources, such as
intervention or counseling, teachers are engaged in the process as the barriers to learning are
reduced and positive classroom learning increases (Anderman & Anderman, 2006; Datnow &
Park, 2015). Interventions and other non-classroom resources do not remove the need for
teachers to use assessments to adjust instruction, but the external supports can help demonstrate
to teachers the impact additional learning opportunities can have on student achievement.
School leaders that recognize the importance of attributing student achievement gaps to a lack of
learning opportunities or supports will see improvement towards the organizational goal of
increasing student achievement in ELA and Mathematics.
Goal-Orientation
Influence 2. The assumed goal-orientation motivation influence, stated as “School
leaders must perceive the goal-orientation of data use to be improving learning opportunities for
students, not the judgement of school performance,” was identified as both an asset and a need
by school leaders through interview questions related to whether the schools have open and
honest discussions about data, and whether teachers use data to identify students that need
additional academic support, such as intervention.
School Leader Survey results. The survey results did not directly identify motivation-
based goal-orientation influences for the study. Individual interviews were used to identify
specific assets and needs associated with goal-orientation.
School Leader Interview findings. School leader participant interview data confirmed
the presence of motivation assets and needs in terms of goal orientation. Interview data from all
eight participants described that school leaders support a student improvement-focused goals
74
orientation, and they have established some trust among staff that data is not used for judgement
or staff evaluation. School leader participant #6 shared:
I believe that all students can achieve proficiency, it just may be on a different time
schedule or trajectory, and they may need different levels of support during the process.
Students struggle for lots of reasons: learning disabilities, self-efficacy, social concerns,
or a myriad of things that work together to hold the student back. We need to take the
whole student into account. It may be a bad year because there is a set of bad
circumstances in their life that are impacting them. A teacher’s lens may not see the
whole student, so [school leaders] and counselors can be a big support.
The school leader interviews with seven of the eight participants also described systemic
supports that exist within each school to respond to other measures. School leader participant #2
shared:
[School leaders] and guidance counselors team up with teachers to support students by
seeing the whole student. Their academic profile, social emotional profile, behavior and
discipline records, homelife, and other roadblocks. We can see the roadblocks and
identify the proper supports, such as intervention lab or counseling, that will help a
student move beyond the barriers and get back to focusing on learning.
However, all eight school leaders also cautioned that they must always approach data use with
caution because there are still moments when data use is perceived as competition or judgement.
School leader participant #2 explained that, “school-wide data can be interpreted negatively at
times, especially if it was a tough year, so it is important to highlight areas of growth for
students. It’s not about ranking or competition, it’s about students, and sometimes we need that
reminder.”
75
Summary. Establishing a positive goal-orientation focused on student improvement, not
judgement, requires open and honest communication about data, as well as promoting the
support of student academic needs. Teachers that believe they are on their own to get all
students to proficiency may lose motivation, struggling under the consistent effort and limited
resources. School leaders that provide additional learning opportunities, such as intervention
support, can reinforce data use by teachers that focuses on identifying individual student needs
(Anderman & Anderman, 2006; Datnow & Park, 2015). This in turn reinforces the belief that
the purpose of the data use is to improve student learning, not to judge instruction or schools.
The survey and interview data confirmed that there are goal orientation motivation gaps that can
be addressed to improve progress towards the organizational goal. Researchers recommend
school leaders communicate focus to use data to support students clearly and consistently
(Marsh, 2012; Marsh & Farrell, 2015; Rueda, 2011). Sharing the goal is not enough; it must be
revisited consistently through goal-setting and modeled behaviors for staff (Marsh & Farrell,
2015; Rueda, 2011). School leaders that recognize the importance of establishing a positive
goal-orientation focused on student improvement will see improvement towards the
organizational goal of increasing student achievement in ELA and Mathematics.
Two assumed motivation influences were presented in Chapter Two. Table 14 shows a
summary of the assumed motivation influence assets and needs identified through the site
administrator survey and interview analysis.
Table 14
Summary of Assumed Motivational Influences Identified as Assets or Needs
Category Assumed Influences Asset Need
76
Attribution
Theory
School leaders must attribute the cause
of student achievement gaps to be due
to effort or opportunities, not a lack of
student ability.
√
Goal
Orientation
School leaders must perceive the goal-
orientation of data use to be improving
learning opportunities for students, not
the judgement of school performance.
√ √
Results and Findings for Organization Causes
The study included four assumed organizational influences presented in Chapter Two.
Rueda (2011) categorizes organizational culture into two themes, culture models and cultural
settings. Findings are presented using these two themes. Table 15 shows that both assumed
organizational influences identified as assets or needs.
Table 15
Organizational Influences Identified as Assets or Needs
Category Assumed Influences Asset
Need
Cultural
Model
School leaders must promote the value of
data use throughout the year for
improvement and intervention.
√ √
Cultural
Model
School leaders must promote the belief that
assessment data is used for student
improvement not to judge the performance
of teachers or schools.
√ √
Cultural
Setting
School leaders must incorporate data-
informed goal setting and reflection when
discussing plans and progress.
√
Cultural
Setting
School leaders must work with staff
groups, such as PLC grade-level or content
team meetings, supporting data use to build
capacity through application and action.
√
77
Cultural Models
Influence 1. The assumed organizational cultural model influence, stated as “School
leaders must promote the value of data use throughout the year for improvement and
intervention,” was identified as both an asset and need by school leaders through survey and
interview questions related to whether student data informs the allocation of school resources,
schools support teacher use of data to improve instruction, and the consistency of collaboration
focused on improving student learning outcomes.
School Leader Survey results. The survey results identified that school leaders promote
the value of data use throughout the year for improvement and intervention by using data to
inform the allocation of school resources, supporting the teacher use of data to improve
instruction, and ensuring that teachers can collaborate consistently around improving student
learning outcomes (Table 16). All elementary school leaders agreed with the practices. Ten
percent of the secondary school leaders surveyed, indicated student data does not consistently
inform the allocation of school resources, and twenty-five percent of secondary school leaders
did not agree that teacher data use is adequately supported, nor do teachers collaborate regularly
with the focus to improve student learning outcomes. The data informed interview participant
selection and follow-up questions during the interview to learn about the disagreements and
identifying possible needs.
Table 16
School Leader’s Cultural Model of Practices Associated with Influence 1
78
School Leader Interview findings. School leader interview data exploring the cultural
model influence confirmed the presence of cultural model organization assets and needs. Three
of the four elementary school leaders described teacher collaboration process assets that monitor
student proficiency around essential skills and provides additional learning opportunities to
strengthen weakness that are discovered. However, while some elementary school leaders
shared that data use is scheduled throughout the year on the staff calendar, and that data use is
revisited in staff-wide emails and during staff meetings, full-implementation of this practice was
not present at all schools. School Leader participant #4 shared an example that was further along
implementation, “we start using the data before the school year ends to begin preparing for the
fall. Teachers use data to create balanced groups of students for the new classes, and we use
reading data to pre-plan possible interventions.” Three of the four elementary school leaders
described how they have found value in embedding data use into their schedule, communication,
and processes such as shared by participant #5, “release days are available for teams to dive
deeper into data, planning intervention and updating formative assessments.” Three of the four
secondary school leaders explained that data use is promoted consistently in content areas that
79
are annually assessed by the state: ELA, Mathematics, and Science. However, for other
departments, such as physical education, visual arts, and world languages, data work is not a
consistent priority. For the departments tied to state assessments, school leader participant #7
described the promotion of data use:
Data is important to all of us. However, because it is important to me and I make it a
priority all the time, it's a priority to you. It's really about why it is important to us...
Data is about identifying the students that need help to succeed by the end of the year.
What do we need to do to help them? We look at [state assessment] data from the
beginning of the year, compare each teacher to their team to identify trends and plan
instructional updates. Then throughout the year, after each benchmark assessment, we
revisit the state assessment data for comparison. It’s about having hard conversations as
professionals, but it’s worth it for students.
Two of the four secondary school leaders also shared that promotion of data use includes
growing the capacity for formative assessments throughout the year. School leader participant
#1 explained:
We are encouraging our PLC groups to develop common formative assessments aligned
with the scope and sequence of our district benchmark assessments to further provide
data of student growth and progress. Our team recognized the connection between the
benchmarks and [state assessments] and realized the high value of the data’s ability to
provide specific measures, guiding our instruction and interventions. We provide release
time for teams to realigned curriculum and assessments. It’s an investment that will
provide data throughout the year aligned with standards and learning-goals.
80
Summary. The survey and interview data confirmed that there are cultural model assets,
as well as needs that can be addressed to improve progress towards the organizational goal.
Cultural strengths were described in the practices of school leaders, promoting the value of data
use throughout the year, involves making data use and the support of data-based goals a priority
(Schildkamp et al., 2019). The school leaders have prioritized data use by ensuring that time will
be available for key data collection events, like benchmarks, and that everyone is aware of the
events far in advance. The school leaders also promote data use throughout the year by investing
in teachers through release time. Consistently throughout the interviews, school leaders referred
to time as the most precious and valuable commodity, so providing teachers with extra time to
develop common formative assessments for data collection and to analyze the data is an essential
method for promoting data use. According to Schildkamp et al. (2019) it is essential that
teacher data teams feel supported and that the decisions and goals they establish will be
implemented. Based on the percent of school leaders that indicated a lack of support for teacher
data use, supporting for teacher data use is not fully implement for all schools, as there are teams
in need. Overall, the trend for data use across the schools, whether implemented or in progress,
is moving towards the use of common formative assessments aligned to district standards-based
benchmarks. This is common among school districts, as state and federal accountability
pressures have driven a shift to be more data focused (Datnow & Park, 2018). Datnow and Park
(2018) caution about sticking too rigidly to this model, as data use organized solely around
narrow data measures, such as benchmarks, coupled to strong accountability practices can
undermine the ability for teachers to make meaningful improvements to instruction. The
researchers recommend the use of a wide range of data for understanding student growth and
needs. School leaders that recognize the importance of promoting teacher data use throughout
81
the year will see improvement towards the organizational goal of increasing student achievement
in ELA and Mathematics.
Influence 2. The assumed organizational cultural model influence, stated as “School
leaders must promote the belief that assessment data is used for student improvement not to
judge the performance of teachers or schools” was identified as an asset and need by school
leaders through survey and interview questions related to whether they believe data has
improved the quality of decision making, as well as have established meetings with each teacher
to set and revisit goals in the fall and spring.
School Leader Survey results. The survey results confirmed that school leaders promote
the belief that assessment data is used for student improvement through their belief that data has
improved the quality of decision making, and establishment of meetings with each teacher to set
and revisit goals in the fall and spring (Table 17). All school leaders agreed that using data has
improved the quality of decision making in their school. While all elementary school leaders
also shared that they establish fall and spring meetings with each teacher to set goals and then
revisit progress towards the goals, ten percent of the secondary school leaders surveyed said they
do not meet with teachers to establish and revisit annual goals. The data informed interview
participant selection and follow-up questions during the interview to learn about the
disagreements and identifying possible needs.
82
Table 17
School Leader’s Cultural Model of Practices Associated with Influence 2
School Leader Interview findings. School leader interview data exploring the cultural
model influence confirmed the presence of cultural model assets and needs. All four of the
elementary school leaders described their work leading data use with teachers a relationship of
trust, identifying the influence as an asset. School leader participant #8 shared their strategy for
working with teachers that may be defensive about data:
It’s important to build positive relationships with staff to make sure they understand we
are not using data to attack them... if we see a gap, we need to figure out how we're best
going to be able to address it. Build the connections and make them feel supported.
When you find a gap, provide ideas, strategies, or make recommendations for the teacher
go see another teacher in action or share a strategy.
Additionally, three of the four elementary school leaders emphasized the importance they place
on fall planning meetings with each teacher and the subsequent spring reflection meeting.
School leader participant #5 explained:
83
I meet with every teacher, one-on-one, at the start of the school year in the fall. Fall
planning is all about data, looking at the teacher’s previous year student scores, as well as
their new students. We break it all down. We look at their previous class and their
present class so we can identify trends or gaps and set goals for improvement. The
conversation and trust is so important. Teachers need to trust me enough to share their
ideas, challenges, and questions. They need to feel safe enough to be vulnerable. Once
we have a plan in place, then my job is to support them throughout the year. We follow-
up on the goals, review updated data, and adjust. In the spring, we revisit it again,
reflecting on how it went by examining benchmarks and other assessments, as well as the
types of support and interventions we tried.
All four secondary school leader interviews supported the elementary perspective that building
relationships and trust are essential, confirming further identification of an asset. The
disagreement shown in the survey items about reviewing student data in the fall and spring was
clarified as a difference between departments and subjects. Participant #2 clarified that, “we do
meet to set goals with each teacher in the fall and reflect on growth in the spring. In [secondary
school] teachers support a specific subject and not all subjects have assessment data to review, so
our annual meetings focus on instructional growth.” The challenge of developing trust among
teams in a secondary school setting that represent different content-specialization and levels of
data use, identified a need among school leaders as they communicate about data to staff. As an
area of agreement, all eight school leaders agreed that communicating about data should be
approached with caution and was best facilitated in person so dialogue could clarify
misconceptions and context. All four secondary school leaders further explained that staff
representing core academics perceive that they are responsible for a majority of the
84
implementation of student achievement improvement, while other staff may not identify how
their work with students can support improvement efforts. In the interview with participant #6,
the school leader expanded on how they promote an improvement focus for data use:
We always communicate data as a conversation, emailing data leaves the opportunity for
it to be misread. The data conversation must be prefaced with trust and support, and we
must be willing to answer questions, and listen. The conversations lead to collaboration,
revealing needs, and discussing what could be done different to make a change.
Sometimes the conversation upfront is also necessary to build motivation for a teacher to
want to use data. We address what the data is saying, and influence or inspire teachers to
look at what is best for students, and in turn look into the data to see what it highlights as
needs. Collectively thinking of ways to improve the outcomes for students. Teachers
need a vested interest in what they can change. They use their knowledge of the
curriculum and content to identify the gaps but might not have the full answer for how to
bridge the gaps, so we are there to provide support.
Summary. Examining the cultural model influence revealed a unanimous agreement
among the school leaders that data use has improved the quality of decision making for their
schools. This is an indicator that the schools have established a baseline culture for improving
student outcomes with data, rather than judging or shaming teachers with their data. Research
has shown that school leaders can hinder or enable a data use culture (Park & Datnow, 2009;
Schildkamp & Poortman, 2015; Wayman et al., 2012). Specifically, school leaders that create
feelings of blame or shame around data with their staff can cause an aversion to data use
(Datnow & Hubbard, 2015). If data use at the schools studied was interpreted as judgement,
then the conflict could become apparent as impacting the quality of decisions. Interview follow-
85
up confirmed the presence of cultural model needs, exploring whether school leaders
encountered teachers that felt like data was being used to judge or shame. In all schools, the
leaders shared that they proactively work to build a positive improvement culture by investing in
relationships with teachers to build trust. In each case, the school leaders described the
importance of building trust and offering support to enable teachers to feel like they have
approval and the ability to facilitate change. Of the two groups, secondary school leaders were
the most concerned with making sure an administrator was present to provide context and
support for the data work. This was supported by the data that secondary school teachers are
responsible for a single subject, teacher teams are larger, and some departments have less
experience with data use. It is important for school leaders to stay connected to the
collaboration, and that can include serving as a role model for effective data use, providing
access to relevant data, and nurturing capacity for teacher data use leadership (Schildkamp et al.,
2019). School leaders that prioritize reinforcing the belief that assessment data is used for
student improvement will make progress towards the organizational goal of increasing student
achievement in ELA and Mathematics.
Cultural Settings
Influence 1. The assumed organizational cultural setting influence, stated as “School
leaders must incorporate data-informed goal setting and reflection when discussing plans and
progress,” was identified as a need by school leaders through survey and interview questions
related to whether school leaders use data to support proposals for change and focus staff
meetings on progress towards data-based goals.
School Leader Survey results. The survey results confirmed that school leaders
incorporate data-informed goal setting and reflection when discussing plans and progress by
86
using data to support proposals for change and focusing staff meetings on progress towards data-
based goals (Table 18). All school leaders agreed that they use data to support proposals for
changing school programs or process. However, while elementary school leaders indicated that
staff meetings focus on progress towards data-based improvement goals, fifty percent of
secondary school leaders shared that staff meetings do not have a data focus on progress. The
data informed interview participant selection and follow-up questions during the interview to
learn about the disagreements and identifying possible needs.
Table 18
School Leader’s Cultural Setting of Practices Associated with Influence 1
School Leader Interview findings. School leader interview data exploring the cultural
setting influence confirmed the presence of cultural setting needs. While all school leaders
shared that they provide supporting data when proposing change for school programs, the
frequency of and the types of data conversations they felt comfortable discussing with staff
differed. Three of the four elementary school leaders expressed that they regularly engage in
87
data analysis and review progress towards data-based goals during staff meetings. Elementary
school leader participant #4 shared that, “staff meetings have been an effective way of modeling
data use, for us. I share updates, we can explore [the online data management system], and break
up into grade-level teams to discuss.” Two of the four elementary principals added that they have
found embedding data use into staff meetings an effective method for providing short
professional development for teachers. Participant # 8 described staff meeting training on data
use:
Every staff meeting offers a forum to provide short profession development
opportunities, and I have found it works best if I utilize our teachers that know how to do
it as a model. Teachers share during every staff meeting. I make it intentional, and
relatable. It can help to make things tactile and nonthreatening like M&Ms. Build a
sense of the data while keeping the technical aspect low and reducing the dryness and
boredom. The meetings also allow us to bring together the whole staff and all their
perspectives. At times we can be too driven by the academic data, so as a group we are
working to grow better at looking at the whole child. We need to support students with
anxiety, stress, and getting better at handling failure. As a staff, we can discuss solutions
beyond academic interventions, such as behaviors or social emotional options.
For secondary school leaders, the survey indicated a large disagreement with focusing on
progress towards data-based goals in staff meetings. Follow-up during the interviews with all
four school leaders revealed that secondary schools have less whole-staff meetings than
elementary schools, and due to the larger number of staff it is difficult to adequately present the
data and answer all the questions. Two of the four secondary school leaders indicated that
sharing data in this way typically happens in school leadership meetings with department chairs,
88
teacher leaders, and they share the data in turn with their department teams. Although, all four
secondary school leaders agreed with elementary school leaders that staff meetings are an
excellent opportunity to provide training and modeling of data use. School leader participant #6
shared:
We know that despite our work and the trust we have built there are some teachers that
are resistant to data use, and are worried about judgement and evaluations. There are
some that feel like it's just a phase of education and they can keep teaching their content
regardless of the trends. People can get cynical. You also have new teachers/hires who
may be at the forefront, wanting to use data and new tools to lead the conversations and
grow in supporting students. Then you have the mid-ground, people that can be
influenced either way. They need to be led. They need a reason. They need to respect
the person who is leading them, and our folks, respond well to somebody who has
knowledge of what they're talking about and not just feeling like they're getting
something just thrown at them. We use staff meetings or department meetings to make
sure everyone is part of the conversation. That's big, and it takes time in a secondary
school to get conversation started and move it forward.
Summary. Establishing a collaborative setting for data use involves establishing norms
around data use and an environment where data use is common (Schildkamp et al., 2019;
Wayman & Jimerson, 2013). The school leader interviews revealed that all nine schools
leverage their staff meetings in some way to create conversations around data use. The survey
and interview data confirmed that the differences among the schools represent cultural setting
needs that can be addressed to improve progress towards the organizational goal. Based on the
challenges that secondary schools encounter, there is an optimal number of participants for these
89
meetings. If the meeting is too large then some voices do not have a chance to participate, and if
the meeting is too small it reduces the variety of perspectives for problem-solving. The schools
also use this time to establish expectations for data use, modeling how to approach data, set
goals, and reflect on progress. The school leaders know that their staff vary in skill levels for
using data, so they also distribute leadership to teachers that have demonstrated skill. Teacher
leaders can be a valuable resource for networking and sharing practices and support among the
staff (Datnow & Park, 2018; Schildkamp et al., 2019; Van Gasse et al., 2017). Finally, all of the
school leaders provided specific structure for the meetings, guiding and modeling, and making
sure that the data they present to staff in a large meeting is purposeful and clear. The caution is
warranted, as Datnow and Park (2015) warn that quick and superficial data analysis meetings,
seeking to address all concerns at once, can lead to oversimplified solutions and often miss the
nuances of data. Large meetings serve as an opportunity for intentionally modeling data use that
starts the conversation, reminds the staff of a problem, or provides specific training or updates.
School leaders were clear that the core data use worked best within the structure of grade-level or
subject-level collaboration teams. School leaders that incorporate data-informed goal setting and
reflection when discussing plans and progress will make progress towards the organizational
goal of increasing student achievement in ELA and Mathematics.
Influence 2. The assumed organizational cultural setting influence, stated as “School
leaders must work with staff groups, such as PLC grade-level or content team meetings,
supporting data use to build capacity through application and action,” was identified as a need by
school leaders through survey and interview questions related to whether school leaders hold
teams accountable for data-based goals, student data informs teacher professional development
and resources, and teachers collaborate regularly to look at student data and plan instruction.
90
School Leader Survey results. The survey results confirmed that school leaders work
with staff groups, such as PLC grade-level or content team meetings, supporting data use to build
capacity through application and action by holding teams accountable for data-based goals, using
data to inform teacher professional development and resources, and ensuring teachers collaborate
regularly to look at student data and plan instruction (Table 19). All school leaders agreed that
they hold teams accountable for data-based goals and use student data to inform decisions for
teacher professional development and resources. All Elementary school leaders also agreed that
their teachers meet regularly to look at student data and make instructional plans, while twenty-
five percent of secondary school leaders disagreed. The data informed interview participant
selection and follow-up questions during the interview to learn about the disagreements and
identifying possible needs.
Table 19
School Leader’s Cultural Setting of Practices Associated with Influence 2
School Leader Interview findings. School leader interview data exploring the cultural
setting influence confirmed the presence of cultural setting needs. Three of the four elementary
school leaders confirmed that they work with grade-level teams regularly, discussing and
91
questioning data, and hold the teams accountable for, as well as support their goals. The
collaboration helps school leaders to determine how to allocate support staff, such as intervention
teachers and instructional coaches, as well as provides insights weekly or bi-weekly into how
students are growing. Elementary school leader participant #8 explained:
Working with the grade-level PLC groups I help teachers find the difference and leverage
both qualitative and quantitative data. Elementary Teachers are fantastic with qualitative.
They can tell you a narrative and story of student learning like nobody's business. They
are not always as strong at backing up the observations, triangulating, with other
quantitative measures. My goal during collaboration is to help them consider multiple
measures, utilizing data to support what their intuition is telling them about a student's
struggles.
All four elementary school leaders also indicated that the trust and conversations they develop
during the year payoff during PLC collaboration time. Elementary school leader participant #5
shared:
PLC meetings are we get vulnerable with the data. You take your ego out of the data
analysis and focus on how to help a student grow. Step outside the personal aspect of
your craft and really look at what specific students need. It’s not natural and as an admin
you have to provide that support. Teachers grow most when their students are not all
meeting proficiency. We need to support teachers in that moment to figure out how to
break it down. Not all teachers will know how or where to start or how to move on. It
can be a very uncomfortable and scary moment, and I help them navigate a different
approach, strategy, or way of looking at it.
92
However, two of the four elementary school leaders expressed challenges with accountability
when the school year gets busy. As one school leader explained, “teachers are instructional
leaders, but their time is limited. It’s hard to be data driven during IEP meetings, Open House,
and other responsibilities.” Through follow-up questions, balancing accountability and data use
with other events and responsibilities is further along at some schools than others. All the
elementary schools are making progress, but the initial transition to embedding measurement
tools into instruction requires ongoing adjustments.
All four secondary school leaders shared that they recognize the value of working with PLCs, but
limited collaboration time makes it a challenge to do consistently. School leader participant #2
described the challenge:
PLCs are one place where improving learning opportunities is most effective. Ideally,
our PLCs would meet once per week, developing common assessments, evaluating
progress, adjusting. It’s about providing immediate feedback of student performance.
The challenge with PLCs in [secondary school] is that teachers may teach more than one
course and belong to more than one team. The teams have two PLC meetings a month,
so it is difficult to switch between teams and create an ongoing conversation for the team
to work together and respond in a timely fashion.
Two of the four secondary school leaders shared that they provide supports and release days for
teams when needed, lowering the barrier of time and scheduling to work collaboratively on a
specific problem. School leader participant #7 also shared how administrators can help to
streamline data meetings to give teachers more time for analysis:
Technology has hurt as much as it has helped. Navigating the online data system can be
as much of a barrier as working with the data itself. Teachers are not comfortable and
93
navigating the system can take up valuable time during collaboration. I prefer to give
everyone a printout copy of their data. Move beyond the challenges of navigating their
data, and just focus on reading the data. It would be best to give them list as a forced
ranking, sorted so we can look at the range of proficiency. I structure the data to provide
fast visual access, reducing the need to organize data, and getting them to thoughtful
discussion as fast as possible. If we anticipate a lot of discussion I arrange for substitutes,
so that a team can really dig deep into the data and we can discuss options for adjusting
instruction.
Summary. The data that was gathered from school leaders established PLC meetings as
the core data analysis time and where instructional change takes place. It is essential that school
leaders provide conditions for teachers to collaborate and examine the data deeply as a team
(Datnow & Park, 2015). The survey and interview data identified challenges that confirmed
cultural setting needs that can be addressed to improve progress towards the organizational goal.
Every school provided some form of structured time, making it a priority by building it into the
schedule. However, secondary schools provided less collaboration time than elementary, and
faced the challenge of teachers belonging to more than one PLC team if they taught more than
one course. As a result, secondary school PLC data work is spread throughout the year and takes
longer to build consensus since they meet less frequently. When teams identify significant work
is needed, they have the option of using release days. By providing teams with resources when
they have identified a need for collaboration, school leaders enable teachers to meld their
professionalism with data-based decision making (Datnow & Park, 2015). The experiences offer
specific times of growth when teachers can safely challenge their understanding of students and
instruction, finding new considerations and options (Katz & Dack, 2014). Secondary school
94
leaders are aware of the challenges and have adapted their support, enabling teachers to use the
time they have as effectively as possible. For example, school leaders provide teachers with pre-
formatted data reports, so they do not need to spend time navigating the data management
system. Research suggests that providing structured and organized data reports helps teachers by
getting to data analysis faster, looking at the data through a professional lens (Marsh & Farrell,
2014). However, the key difference between elementary and secondary PLC data work is the
pace and scope of work they can complete. Elementary PLC data use is further along in the
process of developing effective data practices because they can meet more frequently, act more
quickly, and learn from their actions. Thus, they have created common formative assessments
that are standards-aligned and can quickly inform instruction. Secondary school PLC groups are
working on these tools as well, but it takes longer to create, and follow-up meetings after an
assessment must wait for the next scheduled meeting. School leaders that support staff groups,
such as PLC grade-level or content team meetings, to build capacity for data use through
application and action will make progress towards the organizational goal of increasing student
achievement in ELA and Mathematics.
There were four assumed organizational influences presented in Chapter Two. Table 20
shows the four assumed organizational influences identified as assets or needs.
Table 20
Summary of Assumed Organizational Influences Identified as Assets or Needs
Category Assumed Influences Asset
Need
Cultural
Model
School leaders must promote the value of
data use throughout the year for
improvement and intervention.
√ √
Cultural
Model
School leaders must promote the belief that
assessment data is used for student
improvement not to judge the performance
of teachers or schools.
√ √
95
Cultural
Setting
School leaders must incorporate data-
informed goal setting and reflection when
discussing plans and progress.
√
Cultural
Setting
School leaders must work with staff
groups, such as PLC grade-level or content
team meetings, supporting data use to build
capacity through application and action.
√
Summary of Influence Assets and Needs
Table 21, 22, and 23 show the knowledge, motivation and organization influences for this
study and their determination as an asset or a need.
Knowledge
Table 21 Knowledge Assets or Needs as Determined by the Data
Summary of Assumed Knowledge and Skill Influence Assets and Needs
Category Assumed Influences Asset Need
Factual School leaders must know the phases of
data analysis.
√
Conceptual School leaders must know which
collaboration models are most effective for
each phase of data analysis.
√
Procedural School leaders must know how to
effectively provide accountability for PLC
data use.
√ √
Metacognitive School leaders must know how to
effectively provide guidance for PLC data
use.
√
Motivation
Table 22 Motivation Assets or Needs as Determined by the Data
Summary of Assumed Motivation Influence Assets and Needs
96
Category Assumed Influences Asset Need
Attribution
Theory
School leaders must attribute the cause of
student achievement gaps to be due to
effort or opportunities, not a lack of student
ability.
√
Goal
Orientation
School leaders must perceive the goal-
orientation of data use to be improving
learning opportunities for students, not the
judgement of school performance.
√ √
Organization
Table 23 Organization Assets or Needs as Determined by the Data
Summary of Assumed Organization Influence Assets and Needs
Category Assumed Influences Asset
Need
Cultural Model School leaders must promote the value of
data use throughout the year for
improvement and intervention.
√ √
Cultural Model School leaders must promote the belief that
assessment data is used for student
improvement not to judge the performance
of teachers or schools.
√ √
Cultural Setting School leaders must incorporate data-
informed goal setting and reflection when
discussing plans and progress.
√
Cultural Setting School leaders must work with staff
groups, such as PLC grade-level or content
team meetings, supporting data use to
build capacity through application and
action.
√
Chapter Four presented the results and findings data of the collected and summarized
influence assets and needs. The chapter concluded with alignment of the research questions and
knowledge and skills, motivation, and organizational influences. Chapter Five provides
97
solutions, based on data and literature, for addressing the perceived needs as well as
recommendations for an implementation and evaluation plan for the solutions, as well as answers
the fourth and final research question, What are the recommendations for organizational
practice in the areas of knowledge, motivation, and organizational resources?
98
CHAPTER FIVE: RECOMMENDATIONS, IMPELMENTAITON, AND EVALUTION
The purpose of this study was to evaluate the knowledge, motivation and organizational
influences on the capacity for school leaders to lead data use by staff, driving an increase in
student achievement. An analysis of how school leaders promote and support data use by their
teachers focused on gaps in knowledge and skill, motivation, and organizational structures. The
results revealed a need for a systematic program for school leaders to develop essential
knowledge and leadership skills if schools, and districts, would like to increase the effectiveness
of data use for raising student achievement.
Modeled after Clark and Estes’ (2008) gap analysis, the study sought to identify the
presence of any knowledge, motivation, and organizational gaps between how school leaders
support data use and what was recommended by current literature. School leaders were surveyed
to establish a base understanding of their practices for promoting data use at their school.
Interview participants were then selected from the survey participants, to provide additional
information to the researcher related to how they approach data use at their school. Data was
analyzed and categorized into knowledge, motivation, and organizational assumed influences.
The researcher evaluated each influence and determined whether the assumed influence was
interfering with the stakeholder group of focus, school leaders, in meeting their stakeholder
performance goal to lead staff in the use of data from multiple measures to improve student
achievement. Chapter Five explores recommendations for the influence-associated needs
identified in Chapter 4.
Recommendations for influence-associated needs use the Clark and Estes (2008)
categorical framework for knowledge, motivation, and organizational influences. Kirkpatrick
and Kirkpatrick’s (2016) New World Kirkpatrick Model’s framework was used to design an
99
integrated implementation and evaluation plan to address the identified needs. The New World
Kirkpatrick Model uses four levels that address reaction, learning, behavior, and results, using a
backwards planning model to ensure the desired outcomes are the focus.
Organizational Context and Mission
The organization, Sunshine Unified School District (SUSD), is a PreK-12 school district
of approximately 9,000 students in Southern California. The mission of the district focuses on
providing a caring, respectful, and encouraging environment where students thrive and
demonstrate academic excellence and develop unique talents to prepare them for their future
goals. Within the student population, forty-eight percent self-identify as White, twenty-five
percent as Hispanic or Latino, thirteen percent as Asian, seven percent as two or more races,
three percent as African American, and three percent as Filipino. The district’s schools consist
of six elementary schools, grades preschool to five, and three secondary schools, grades six to
twelve.
Organizational Performance Goal
The district has demonstrated that eighty percent of students assessed through the
CAASPP state assessments are at or above proficiency in English Language Arts (ELA) and fifty
percent are proficient in Mathematics. The skills associated with proficiency are broken down
through the California Common Core standards and are further assessed through screening and
formative assessments during the year. The proficiency levels suggest that thirty percent of
graduating students would not have the ELA proficiency and fifty percent would not have the
Mathematics proficiency necessary to prepare them for success in college and future careers.
The organizational goal was to use data-based decision making to improve student achievement,
closing the proficiency gap by 2% per year. Progress towards the goal was necessary to provide
100
an education of academic excellence and preparation for a future which includes college and
career readiness for as many students as possible.
The organizational performance goal at the time of the study was to improve student
achievement in ELA and Mathematics as measured annually by the state summative assessment,
CAASPP. Thus, by June of 2020, the district sought to increase student achievement on the
CAASPP assessments by two percent per year. Student proficiency is essential for college and
career readiness. Failure to achieve the district goal would result in diminished reported
performance values for the district which could lead to students transferring to another district.
More importantly, a lack of proficiency leads to less students graduating with the skills and
proficiency to succeed in college and future careers.
Description of Stakeholder Groups
The stakeholder groups that are directly invested and benefit from the organizational goal
are the students, parents, teachers, and school leaders. All four groups participate on the
advisory committee and provide input into the goals that are set and how organizational
resources are allocated for support. Students and parents are invested in the education of the
student, and directly benefit by improvements in how the district supports their growth and
success. Teachers are the primary facilitators of student learning, and directly responsible for
proficiency improvement. School leaders, such as principals and assistant principals, are
responsible for overseeing improvement efforts, providing support and intervention resources for
students that teachers identify as in need, and leading data-based decision making to support
student learning at their school. Overall, it was the goal of school leaders to facilitate and
support the efforts of other stakeholder groups in order to develop incremental progress to the
organizational goal for improving student achievement (Table 24).
101
Table 24.
Organizational Mission, Organizational Performance Goal, Stakeholders’ Goals
Organizational Mission
The mission of SUSD is that students will thrive in a caring, respectful, and encouraging
environment where they demonstrate academic excellence and develop unique talents in
preparation for their future goals.
Organizational Performance Goal
By June 2022, SUSD will use data-based decision making to improve learning
opportunities, closing the proficiency gap by 6%, or 2% per year.
School Leader Stakeholder Goal
By June of 2020, School leaders will adapt site messaging and support for data-based decision
making using student data from multiple measures to increase student proficiency towards the
2% annual growth goal.
Goal of the Stakeholder Group for the Study
While the combined efforts of all stakeholders join to achieve the organizational goal of
an annual two percent increase in ELA and Mathematics proficiency, school leaders are
responsible for the overall leadership of improvement efforts towards the performance goal.
Therefore, the stakeholders of focus for this study were administrators serving in school
leadership roles. The specific stakeholder goal for school leaders was to incrementally increase
ELA and Mathematics proficiency by two percent each year using data from multiple measures,
informing the allocation resources for first time instruction and intervention based on student
need and growth. Failure to accomplish this goal would impact student achievement, which
would impact graduation rates, post-secondary careers and education, and district enrollment.
Inter-district transfer students could choose to return to their home district, and students living in
the SUSD community could begin to transfer to other high performing districts. Both outcomes
would lead to a loss of funding for the district.
102
Purpose of the Project and Questions
The purpose of this study was to conduct an evaluation, examining the knowledge,
motivation, and organizational influences on the capacity for school leaders to lead staff in the
use of data, driving an increase in student achievement by two percent annually. The analysis
began by generating a list of possible or assumed interfering influences that were examined
systematically to focus on actual or identified interfering influences. While a complete analysis
would focus on all stakeholders, for practical purposes school leaders were focused on as the
stakeholder, exploring their use of data to inform improvement.
As such, the questions that guided the evaluation study are the following:
1. To what extent did school leaders improve student achievement through the promotion of
data-based decision making in their schools?
2. What knowledge and motivation did school leaders have related to increasing student
achievement through data-based decision making?
3. What is the interaction between Sunshine Unified School District culture and context and
school leader knowledge and motivation for increasing student achievement through
data-based decision making?
4. What are the recommendations for organizational practice in the areas of knowledge,
motivation, and organizational resources for achieving increases in student proficiency
through data-based decision making?
Introduction and Overview
Clark and Estes’ (2008) gap analysis, a systematic, analytical method was used to clarify
organizational goals and identify the presence of any knowledge, motivation, and organizational
gaps between how school leaders lead and support the use of data for improving student
103
achievement and what was recommended by current literature. In responding to Research
Question 4, “What are the recommendations for organizational practice in the areas of
knowledge, motivation, and organizational resources?” the researcher used the findings and
results from examining the SUSD to develop recommendations. An analysis of the school
leaders’ knowledge, motivation, and organizational influences on how they lead staff in the use
of data for improving student achievement revealed validation of four knowledge influences, two
motivation influences, and four organizational influences.
Recommendations for Practice to Address KMO Influences
Knowledge Recommendations
Introduction. School leader approach and support for data use to improve student
achievement is based on their knowledge of the best practices and strategies. The support for
data use at a school involves careful navigation of supporting teams with resources, times, and
space to collaborate, while holding teams accountable for using data to improve student
outcomes (Huguet, Farrell, & Marsh, 2017; Miranda & Jaffe-Walter, 2018). Thus, knowledge
influences impact whether data use successfully occurs, whether data use is structured in an
efficient and productive manner, and whether teacher teams are adequately supported and held
accountable for utilizing data effectively and consistently. Through the quantitative and
qualitative methods used in this study, Table 25 displays the identified knowledge influences that
were examined to determine whether each knowledge influence impacts school leaders, the
relative priority of each knowledge influence present, the core knowledge principle behind the
influence, and a specific recommendation to address each knowledge influence within the
context of the organization.
Table 25
104
Summary of Knowledge Influences and Recommendations
Assumed
Knowledge
Influence
Need
Yes,
No
(Y,
N)
Priority
Yes, No
(Y, N)
Principle and
Citation
Context-Specific
Recommendation
School leaders
must know the
phases of data
analysis. (Factual)
Y Y Information
learned
meaningfully
and connected
with prior
knowledge is
stored
more quickly
and
remembered
more accurately
because it is
elaborated with
prior learning
(Schraw &
McCrudden,
2006).
Provide a job aid diagram of the
phases of data analysis,
providing a crosswalk connecting
current organization practices
and resources with the
appropriate phase of data
analysis.
School leaders
must know which
collaboration
models are most
effective for each
phase of data
analysis.
(Conceptual)
Y Y How
individuals
organize
knowledge
influences how
they
learn and apply
what they know
(Schraw &
McCrudden,
2006).
The use of
metacognitive
strategies
facilitate
learning
(Baker, 2006).
Provide training for school
leaders to practice the data
analysis process, following a
structured procedure that
supports data use in collaborative
groups.
Utilize metacognitive talk
modeling as part of the provided
training.
School leaders
must know how to
effectively provide
Y Y Increasing
germane
Provide training, modeling
familiar data analysis activities,
resources, and the use of
105
accountability for
PLC data use.
(Procedural)
cognitive load
by engaging
the learner in
meaningful
learning and
schema
construction
facilitates
effective
learning
(Kirshner et al.,
2006).
Behavior that is
reinforced is
strengthened.
(Daly, 2009)
The use of
metacognitive
strategies
facilitate
learning
(Baker, 2006).
accountability tools that reinforce
behaviors.
Utilize metacognitive talk during
the training to explain how
different school leader behaviors,
tools, and feedback reinforce the
desired staff behaviors.
School leaders
must know how to
effectively provide
guidance for PLC
data use.
(Metacognitive)
Y Y Modeling to-
be-learned
strategies or
behaviors
improve self-
efficacy,
learning,
and
performance
(Denler,
Wolters, &
Benzon, 2009).
Modeled
behavior is
more likely to
be
adopted if the
model is
credible,
similar (e.g.,
Provide training for school
leaders modeling how an
instructional leader can provide
guidance to a team analyzing
data. Include time for each
school leader to practice
modeling effective behaviors for
supporting the team through the
data analysis process.
106
gender,
culturally
appropriate),
and the
behavior has
functional
value (Denler
et al.,
2009).
The use of
metacognitive
strategies
facilitate
learning
(Baker, 2006).
Utilize metacognitive talk during
training modeling to convey the
functional value of instructional
leader’s behaviors while guiding
the team.
Increasing school leader knowledge of the phases of data analysis. The results and
findings indicated that school leaders need additional factual knowledge about the phases of data
analysis supporting data use to drive improvement. A recommendation linked to information
processing system theory has been selected and shared in Table 25 to close the factual
knowledge gap. The research suggests that information can be learned faster and retrieved more
accurately if it is learned in a context that develops meaningful connections to prior knowledge
(Schraw & McCrudden, 2006). Thus, providing learners with a visual organizer supports the
learning process. The recommendation is that school leaders would benefit from access to a
visual organizer that connects appropriate phases of the data analysis process to current resources
and practices used by the organization to examine data. For example, when supporting teacher
grade-level or content team collaborative meetings, the visual organizer could recommend
accessing the district data system and provide a data overview report and teacher roster reports
for common assessments.
Marsh and Farrell (2015) explain that data use support includes access to practical
tools that organize strategies and actions, providing immediate utility during implementation.
107
The tool is also valuable for directing staff to methods that produce thoughtful use of data, not
misinformed data use (Datnow & Park, 2015). However, the tool must be flexible enough that
staff can adjust its use to support the needs and practices of each collaborative team, developing
their own meaning and value for the tool. The evidence supports providing a visual organizer to
support staff as they adjust to using data to inform the improvement of student achievement.
Improving school leader procedures accounting for and supporting data use for
instructional improvement. The results and findings indicated that school leaders lack
procedural knowledge for supporting and holding collaborative groups accountable for data use
to drive improvement. Of the two procedural knowledge influences identified, support and
accountability are the more challenging and important knowledge need for school leaders as they
work to manage and improve student achievement. A recommendation linked to cognitive load
theory has been selected and shared in Table 29 to close the procedural knowledge gap. The
research suggests that effective learning should involve a meaningful learning experience that
includes schema construction, connections to prior knowledge, and guidance and scaffolding
appropriate to support engagement and learning (Kirshner et al., 2006; Mayer, 2011). Thus,
providing training that involves familiar organizational procedures, linking them to the new
resources and procedural information in a way that facilitates the construction of new schema
will support the learning. The recommendation is that school leaders would benefit from
training that models the use of familiar data analysis activities and resources combined with
accountability tools and procedures that reinforce the desired behavior of utilizing data to drive
improvement.
Jimerson and McGhee (2013) suggest that an essential aspect to promoting data use
schools is setting collaboration as the priority and making data use a supportive structure.
108
Schools already value and utilize collaborative teams for improvement and connecting data use
as a tool for empowering the effectiveness of the teams makes sure that collaboration and
improvement remain the focus. Additionally, framing the data use processes within the
organization’s context and practices, encourages long-term integration and develops an
understanding for school leaders of the conditions that support productive data use, as well as
when data use will be counter-productive (Boudett & Steele, 2007; Jimerson & McGhee, 2013;
Wayman & Jimerson, 2013). The findings are supported by the research of Marsh, Pane, and
Hamilton (2006) which found that educators valued training that provided organizational context
for supporting instructional planning and improvement more than learning technical skills for
data processing and analysis. Thus, the evidence supports the recommendation for training that
connects familiar organizational procedures to new strategies and resources for data use.
Building school leader knowledge for effectively guiding staff in data use through
the modeling of metacognitive processes. The results and findings indicated that school
leaders lack metacognitive knowledge for effectively providing guidance for data use to drive
improvement. A recommendation linked to social cognitive and information processing system
theories has been selected and shared in Table 25 to close the metacognitive knowledge gap.
The research suggests that modeling the strategy, behaviors, and metacognitive process will
improve the self-efficacy of the learners, their success with adoption of the behaviors, and
improve their performance (Baker, 2006; Denler, Wolters, & Benzon, 2009). Thus, providing
training that incorporates the modeling of desired behaviors and the sharing of the metacognitive
process will support learning. The recommendation is that school leaders would benefit from
training modeling behaviors to use when providing guidance to staff analyzing data, including
109
time for school leaders to practice the behaviors and talk through the metacognitive process
highlighting the functional value of the desired behaviors.
Marsh and Farrell (2015) explain that modeling should be an essential element of training
school staff to use data, providing an observable demonstration of the activity paired with
detailed explanations of the thinking along the way. The research describes the training as a
shared-experience, providing horizontal-learning among the participants that promotes the
development of an integrated set of cognitive and metacognitive skills. School leaders need to
be able to lead and support the data analysis process while purposefully examining how their
behaviors will impact staff along the way. The success collaborative teams have with using data
is influenced by whether school leaders are able to model and promote a culture of continual
improvement that encourages teams to find meaning in the data and make adjustments (Huguet,
Farrell, & Marsh, 2017; Wayman & Jimerson, 2013). The evidence supports training that
provides observable demonstrations of the activity to be learned paired with metacognitive talk
detailing the importance of the decisions made throughout the process.
Motivation Recommendations
Introduction. The beliefs and perceptions of the goals for data use determines whether
school leaders are motivated to engage in, persist at, or self-regulate the encouragement of data
use for improving student achievement. It is essential that school leaders avoid deficit talk about
students, and instead focus on specific achievement targets and learning opportunities that can
overcome achievement gaps (Miranda & Jaffe-Walter, 2018). Educators must also move beyond
attributing the lack of student growth to uncontrollable factors such as parents and student
homelife and begin to focus on controllable factors such as providing learning opportunities that
target the specific needs of students (Diamond, 2008). It is also important to establish
110
measurable goals for data use that emphasize improvement and the empowerment of learning
opportunities to meet student needs (Jimerson, 2014). Through the quantitative and qualitative
methods used in this study, Table 26 explores the identified motivational influences to determine
whether each motivational influence impacts school leaders, the relative priority of each
motivational influence present, the core motivational principle behind the influence, and a
specific recommendation to address each motivational influence within the context of the
organization.
Table 26
Summary of Motivation Influences and Recommendations
Assumed
Motivation
Influence
Need
Yes,
No
(Y,
N)
Priority
Yes, No
(Y, N)
Principle and
Citation
Context-Specific
Recommendation
School leaders
must attribute
the cause of
student
achievement
gaps to be due to
effort or
opportunities,
not a lack of
student ability.
Y Y Learning and
motivation are
enhanced when
individuals
attribute success or
failure to effort
rather than ability.
(Anderman &
Anderman, 2009).
Attribute success or
failure to effort
(Anderman &
Anderman, 2009).
Incorporate real-
life, original source
materials that are
vivid, varied or
novel, and create
surprise or
Provide multiple
collaborative meetings that
focus on the analysis of real
organization achievement
gap examples. Identify each
school leader’s individual
attributions for the
achievement gap and
challenge school leaders to
view the root cause as a need
for student learning
opportunities.
Utilize examples that are real
and relevant to school
examples and provide a value
to learning opportunities as
compared to student ability.
111
disequilibrium
(Schraw &
Lehman, 2009).
School leaders
must perceive
the goal-
orientation of
data use to be
improving
learning
opportunities for
students, not the
judgment of
school
performance.
Y Y Focusing on
mastery, individual
improvement,
learning, and
progress
promotes positive
motivation (Yough
& Anderman,
2006).
Use task, reward,
and evaluation
structures that
promote mastery,
learning, effort,
progress, and self-
improvement
standards and less
reliance on social
comparison or
norm-referenced
standards (Pintrich,
2003).
Create a
community of
learners where
everyone supports
everyone else’s
attempts to learn
(Yough &
Anderman, 2006).
Establish data analysis
meetings that emphasize
improvement and goal
setting, not judgement or
comparison of individuals or
schools.
Utilize evaluation and
feedback during collaborative
meetings to examine progress
and growth, encouraging
schools, teams, and
individuals to reflect and set
goals that are actionable and
measurable.
Challenging attributed causes for low student achievement. As observed in the study,
school leaders shared that while progress has been made, some belief that student achievement
gaps are due to their abilities and family persists among staff. The belief limits the impact
schools can have on improving the students’ proficiency. A recommendation based on
attribution theory has been selected and shared in Table 26 to close the motivation gap.
112
Motivation increases when individuals believe that the factors leading to success are within their
control and are not tied to uncontrollable external influences (Anderman & Anderman, 2009).
Thus, motivating school leaders to support data use involves challenging beliefs that attribute
student achievement to their innate abilities or circumstances. The recommendation is to utilize
regularly occurring collaborative meetings to analyze achievement gap examples within the
organization, fostering dialogue that identifies school leader attributions for the gap and
challenges school leaders to discuss the root cause as an area of opportunity for student learning.
Clark and Estes (2008) state that the key motivator in whether an individual chooses to
act, persist, and endeavor to completion is believing that they can be effective and have an
impact. When seeking to change beliefs and attribution, research suggests using real-life,
genuine examples that reveal unexpected findings, thus challenging assumptions (Schraw &
Lehman, 2009). Katz and Dack (2014) support the recommendation, stating that collaborative
inquiry of existing practices and beliefs creates shared learning experiences essential for driving
change in beliefs and behavior. Thus, research provides theoretical support for improving
motivation by utilizing recurring collaborative data analysis experiences to examine real-life
examples, challenging the attributed causes for achievement gaps, and revealing the potential for
learning opportunities.
Shifting to an improvement-focused goal-orientation. Analysis of the study data
suggests that school leader leaders still encounter the perception that data use is used to judge
school performance rather than to improve learning opportunities for students. The
recommendations in Table 26 for addressing the motivation gap are supported by goal-
orientation theory. Motivation improves when learning and progress are the focus (Yough &
Anderman, 2006). School leaders and teachers are demotivated when they believe data use will
113
lead to negative judgements about the school’s performance, so it is important to frame data use
with the goal to study the school’s students and to improve learning opportunities to meet student
needs. The recommendation is to establish data analysis as part of regular meetings,
emphasizing improvement and goal-setting that leads to learning about students and school
practices and generates incremental progress adjusting learning opportunities to better serve
student needs.
Performance goals and feedback that are vague or inconsistent demotivates individuals,
especially if the feedback is negative or overly critical (Clark & Estes, 2008). Marsh and Farrell
(2015) recommend supporting data use by providing specific guidance and encouraging targeted
goal setting. According to research, processes that involve tasks, rewards, and feedback are
more effective if they emphasize the learning and effort that lead to progressive improvement
and eventual success (Pintrich, 2003). Miranda and Jaffe-Walter (2018) support the
recommendation, stating that it is important for data analysis to focus on specific goals or topics,
learning to identify patterns to generate specific testable adjustments for learning opportunities.
Therefore, the research provides theoretical support for the recommendation that developing
improvement-focused goal orientation will improve motivation for data use to improve student
achievement.
Organization Recommendations
Introduction. The ability for school leaders to support data use for student improvement
is influenced by the processes, models, and experiences they provide staff to develop the
organizational value and accountability for data use. Clark and Estes (2008) identified that
successful change requires that organizational structures and processes must be in alignment with
the organization’s goals. Hence, school leaders must purposefully construct staff messaging and
114
meetings to align their expectations and school processes with improving student achievement
through data use. Table 27 describes the identified cultural influences that were examined for an
impact on school leaders, the relative priority of each cultural influence, a core cultural principle
behind the influence, and a specific recommendation to address each influence within the context
of the organization as supported by analysis of the data collected in the study.
Table 27
Summary of Organization Influences and Recommendations
Assumed
Organization
Influence
Need
Yes,
No
(Y,
N)
Priority
Yes, No
(Y, N)
Principle and
Citation
Context-Specific
Recommendation
Cultural Model
Influence 1: School
leaders must
promote the value of
data use throughout
the year for
improvement and
intervention.
Y Y Accountability is
increased when
individual roles and
expectations are
aligned with
organizational goals
and mission.
Incentives and
rewards systems need
to reflect this
relationship.
Design of incentive
structure and use of
incentives are more
important than the
types of incentives
used (Elmore, 2002)
School leaders need to
regularly check in with
teachers, reviewing data-
supported needs and
shifting resources to
support data use for
improvement.
115
Cultural Model
Influence 2: School
leaders must
promote the belief
that assessment data
is used for student
improvement not to
judge the
performance of
teachers, or schools.
Y Y Leaders can create an
effective
accountability system
when they engage in
the challenging but
necessary process of
analyzing the
complex social and
political elements
within an
organization.
School leadership is
an important factor in
building capacity and
student achievement
(Waters, Marzano, &
McNulty, 2003)
School District offices
play an important role
in improving
accountability in
school systems
(Childress, Elmore, &
Grossman, 2006)
School leaders and district
leaders need to frame data
in ways that focus the
message on improving
student achievement, not
on comparing school
performance.
Cultural Setting
Influence 1: School
leaders must
incorporate data-
informed goal
setting and
reflection when
discussing plans and
progress.
Y Y People are more
productive when goal
setting and
benchmarking are
essential to evaluating
progress and driving
organizational
performance in
accountability.
Data-driven
benchmarking as a
common approach to
benchmarking (Dowd,
2005; Levy & Ronco,
2012)
School leaders need to
provide structured data-
focused meetings
throughout the year that
use benchmarks to
establish school goals and
measure progress.
Cultural Setting
Influence 2: School
leaders must work
with staff groups,
Y Y Measurement of
learning and
performance are
essential components
School leaders should
regularly join teacher
collaborative teams to
support the use of data to
116
such as PLC grade-
level or content-
team meetings,
supporting data use
to build capacity
through application
and action.
of an effective
accountability system
capable of improving
organizational
performance.
Measuring learning is
crucial in the
decision-making
process (Marsh &
Farrell, 2015)
analyze progress towards
identified learning goals
specific to their grade-
level or content.
School leader response to teacher data-based goals. The results and findings of the
study indicated that school leaders need to promote the value of data use for student
improvement throughout the year. A recommendation linked to principles of accountability has
been selected and shared in Table 27. The effectiveness of accountability relationships stem
from the alignment of organization and employee goals and expectations, which can be further
reinforced with incentives (Elmore, 2002). To sustain motivation for data use, school leaders
must carefully employ incentives that reinforce the alignment of teacher goals to school and
district goals. The recommendation is for school leaders to regularly check in with teachers,
reviewing the identified data-supported needs and responding by shifting resources to support the
needs that align with organizational goals.
Establishing specific performance goals that are measurable and linked to organization
goals is an effective way for individuals to develop a connection to the improvement process
(Clark & Estes, 2008; Dembo & Eaton, 2000). Furthermore, Schein and Schein (2017)
emphasized that the beliefs and values individuals develop about their organization are
influenced by what their leaders focus on and invest in regularly. School leaders that regularly
check in with teachers to discuss data-supported needs and invest in data-supported goals aligned
with the organizational goals encourage further data use by teachers. Research supports the
117
recommendation, emphasizing the importance of supporting teachers that have effectively used
data to identify actions for improving student growth by allocating resources (Datnow & Park,
2015; Datnow & Hubbard, 2016). Thus, there is theoretical support for the recommendation that
school leaders can incentivize data use by providing timely access to resources, supporting the
data-informed goals of teachers, the school, and the district for improving student achievement.
Improvement-focused messaging. The results and findings of the study indicate that
school leaders need to reinforce the belief that data use is for student improvement, not to judge
the performance of individual teachers, grade-level teams, content teams, or schools. A
recommendation has been selected and shared in Table 27 based on the principles of
accountability. To establish effective accountability, it is essential for school leaders to address
conflicting social and political perspectives of the expectations, maintaining the focus on
improvement and student achievement (Childress, Elmore, & Grossman, 2006; Waters,
Marzano, & McNulty, 2003). School leaders need to keep the focus of messaging and actions on
the mission to use data for improving student achievement. The recommendation is for school
leaders and district leaders to consistently frame the use of data in ways that focus on improving
student achievement and avoid messaging and actions that would compare or judge performance
of teachers or schools.
The roles of school and district leaders are essential to establishing a positive culture for
data use, influencing the data use values and practices for teachers (Datnow & Hubbard, 2016;
Hubbard, Datnow, & Pruyn, 2014; Long et al., 2008; Park, Daly, & Guerra, 2012). Schildkamp
and Poortman (2015) caution against leaders taking any actions that blame teachers or elicit
shame for instructional practices tied to student achievement, citing strongly negative
implications for motivation. Clark and Estes (2008) also caution leaders to be careful of the
118
destructive and demotivating impact of data when it is used in ways that create competition
between teacher teams or produce strongly negative feedback. School leaders should
communicate with staff consistently, providing relevant data use to inform decisions to develop
plans, monitor school progress, and identify learning opportunity needs (Clark & Estes, 2008;
Datnow & Park, 2015). Thus, there is theoretical support for the recommendation that school
and district leaders should consistently frame the use of data in ways that focus on improving
student achievement, and avoid messaging and actions that would compare or judge performance
of teachers or schools.
Measuring progress through data use. The results and findings of the study indicated
that school leaders need to provide settings for staff throughout the year in which data-informed
goals are established, revisited, and progress is measured. A recommendation has been selected
and shared in Table 27 based on the principles of accountability. Celebrating short-term
progress is important to maintaining staff motivation, measuring growth through goal-aligned
benchmarks to encourage increased value for the processes tied to the accountability (Dowd,
2005; Kotter, 2007; Levy & Ronco, 2012). Teachers are less likely to become discouraged if
school leaders can provide regular updates on the progress teams are making, taking the time to
find and highlight examples of success. The recommendation is for school leaders to provide
structured data-focused meetings throughout the year that utilize goal-aligned benchmarks to
measure progress in improving student achievement.
According to Clark and Estes (2008), establishing organizational change involves
regularly communicating a clear vision that includes organizational goals as well as reflecting on
progress. Organizations that lack a clear vision for data use risk misusing data in ways that only
target specific student groups for support, such as lowest achieving group, and fail to identify
119
root causes (Datnow & Park, 2015). Research supports the recommendation that school leaders
should frame the vision for data use as continuous improvement, recognizing short-term
examples of success as well as promoting inquiry through examples that require further data
collection (Datnow & Park, 2015; Firestone & Gonzalez, 2007). Thus, there is theoretical
support for school leaders to provide structured data-focused meetings throughout the year that
utilize goal-aligned benchmarks to measure progress in improving student achievement.
Supporting collaborative team specific learning goals and measurements. The
results and findings of the study indicated that school leaders need to work with teacher
professional learning communities (PLCs), such as grade-level or content-specific teams,
providing support to remove barriers and build capacity for data use through application and
action. A recommendation linked to principles of accountability has been selected and shared in
Table 27. In order to make effective decisions to improve student achievement, the progress
towards learning goals must be measurable (Marsh & Farrell, 2015). School leaders can support
PLCs as they establish relevant learning goals, measure progress towards the goals, reveal
barriers to achievement, and identify needed resources or interventions. The recommendation is
for school leaders to regularly join teacher collaborative teams to support the use of data,
analyzing progress towards identified learning goals specific to their grade-level or content, and
assisting in addressing the removal of barriers and the provision of additional resources.
The capacity for data use and productivity of teacher collaboration groups is strongly
supported by school leaders that participate in ways that promote a culture of data-informed
decision making (Datnow & Hubbard, 2015; Horn & Little, 2010; Hubbard, Datnow, & Pruyn,
2014). Effective management drives change by consistently participating in the processes tied to
organizational improvement, establishing visibility and credibility (Clark & Estes, 2008).
120
Datnow and Park (2015) support the recommendation, suggesting that teacher collaboration
groups are most effective at data use when school leaders participate, encouraging teachers to
look beyond professional intuition and standardized assessments. School leaders need to
encourage teams to create measurable learning goals relevant to their curriculum and students,
incorporating multiple measures of data for analysis. Thus, there is theoretical support for school
leaders to regularly join teacher collaborative teams to support the use of data, analyzing
progress towards identified learning goals specific to their grade-level or content, and assisting in
addressing the removal of barriers and the provision of additional resources.
Integrated Implementation and Evaluation Plan
Implementation and Evaluation Framework
When organizations set out to address performance problems, it is important to have a
clear concept of the desired outcomes. Kirkpatrick and Kirkpatrick (2016) provide a framework
for effective change that begins with identifying the desired outcomes or leading indicators that
support the organizational and stakeholder goals. With the outcomes in mind, a plan can be
backwards mapped, detailing the critical stakeholder behaviors and required drivers, as well as
the necessary organizational support. Establishing stakeholder behaviors and effective drivers
involves closing knowledge gaps through training, so the next step is to identify the learning
goals that must be met to prepare stakeholders. Finally, providing effective training takes
motivation into account by using methods to measure engagement, relevance, and customer
satisfaction. When properly implemented, engaging training opportunities should produce staff
that can implement what they learned through their behaviors and utilization of drivers, and
eventually result in the improvement of leading indicators towards stakeholder and
organizational goals.
121
Organizational Purpose, Need, and Expectations
Sunshine Unified School District seeks to provide a caring, respectful, and encouraging
environment where students thrive and demonstrate academic excellence and develop unique
talents to prepare them for their future goals. Currently, based on state standardized assessments,
only 80% of students graduate proficient in English Language Arts (ELA) and 50% are
proficient in Mathematics. This demonstrates that 30% of students lack full preparation for
reading and writing, and 50% lack full preparation for Mathematics in the area of college and
career readiness. Thus, the district has established the goal of leveraging data-informed decision
making to improve student learning opportunities with the target of increasing ELA and
Mathematics proficiencies by two percent per year. School leaders, as instructional leaders for
schools within the district, oversee improvement efforts and are directly responsible for building
the capacity of teachers to use data effectively. In alignment with the overall district
organizational goal, school leaders work to facilitate and support the use of data by their grade-
level or content teams of teachers to make incremental progress towards improving student ELA
and Mathematics proficiency. Progress toward the goals of school leaders and the organization
will result in consistent use of data to support decisions, allocation of district resources such as
intervention time based on data recommendations, and overall increases in student achievement
on district-developed benchmark assessments as well as state standardized assessments.
Level 4: Results and Leading Indicators
Kirkpatrick and Kirkpatrick (2016) state that any training initiative is not fully complete
without measuring its impact on the desired organization outcomes towards achieving overall
results. Every initiative begins with a desire to achieve some goal and progress towards the goal
can be measured in terms of an outcome. The short-term measurable changes that occur due to
122
implementation of training within the workplace are called leading indicators and, according to
Kirkpatrick and Kirkpatrick (2016), are used to account for progress towards the desired overall
results. Table 28 shows the proposed external and internal outcomes or leading indicators for
using data to improve student learning opportunities. If the internal leading indicators, such as
benchmark scores, intervention recommendations, and student grades can be achieved, then the
resulting improved student proficiency and achievement should also result in progress
completing the external leading indicators of improved state test scores. The external leading
indicators include the primary organization focus of improving ELA and Mathematics
proficiency as measured by the California Assessment of Student Performance and Progress
(CAASPP), as well as progress in improving English Language proficiency for English Learners
as measured by the English Language Proficiency Assessments for California (ELPAC).
Table 28
Outcomes, Metrics, and Methods for External and Internal Outcomes
Outcome Metric(s) Method(s)
External Outcomes
1. Increased student
proficiency on the
CAASPP English
Language Arts
assessment.
The reported percent of students
at the individual SUSD schools
as well as in the district overall
that score a three or four on the
state assessment each year.
Review final reported state
scoring data or public records
for district and school scores.
2. Increased student
proficiency on the
CAASPP Mathematics
assessment.
The reported percent of students
at the individual SUSD schools
as well as in the district overall
that score a three or four on the
state assessment each year.
Review final reported state
scoring data or public records
for district and school scores.
3. Increased student
scores on the ELPAC
assessment.
The reported percent of tested
students that score a three or four
on the ELPAC each year.
Review final reported state
scoring data or public records
for district and school scores.
Internal Outcomes
123
4. Increased student
proficiency on district
benchmark assessments.
The reported percent of students
at the individual SUSD schools
as well as in the district overall
that score a three or four on the
district benchmarks each term.
Review district benchmark
reports following the
completion of each benchmark
assessment window.
5. Increased number of
data-supported
recommendations for
support or intervention.
The percent of student study
team and teacher
recommendations for
intervention services that cite
student data as justification.
Review site records to identify
which intervention
assignments utilized
benchmarks or other
instructional data to inform the
recommendation.
6. Improved grades
related to ELA
proficiency.
The percent of students that
receive passing or satisfactory
marks for ELA each term.
Review site reports for ELA
grades by grade-level, school
site, and district wide.
7. Improved grades
related to Mathematics
proficiency.
The percent of students that
receive passing or satisfactory
marks for Mathematics each
term.
Review site reports for
Mathematics grades by grade-
level, school site, and district
wide.
8. Increased
reclassification of
English Learner students
to fluent English
proficient.
The reported percent of students
that are reclassified to fluent
English proficient each year.
Review district-tracked
reclassification date or public
records detailing annual EL
progress.
Level 3: Behavior
Critical behaviors. Table 29 details the critical behaviors school leaders must
implement in order to build the capacity for teachers to utilize data, improving student learning
opportunities. The behaviors of school leaders should model effective data use for analysis of
problems, setting goals, and monitoring progress, as well as provide supported opportunities for
staff to apply the data use methods. Additionally, school leaders must follow-up with teachers,
reviewing recommendations for intervention and providing resources and support in response to
teacher requests. The maintenance of ongoing motivation also requires regular progress updates,
including specific success examples where data use resulted in improved student learning.
Table 29
Critical Behaviors, Metrics, Methods, and Timing for Evaluation
124
Critical Behavior Metric(s)
Method(s)
Timing
1. School leaders use
data in staff meetings to
support the analysis of
challenges, goal setting,
progress monitoring, and
resource allocation.
The number of times
school leaders hold
data analysis meetings
or include a data-
supported topic in a
staff meeting.
School leaders will
log the data analysis
meetings as well as
data-supported topics
that they address in
staff meetings.
Ongoing -
following staff
and school site
leadership
meetings.
2. School leaders support
and participate in teacher
team meetings, grade-
level or content-specific,
encouraging the use of
data from benchmarks, as
well as other assessments
and measures of student
progress to inform the
planning of learning
opportunities.
The number of teacher
team meetings school
leaders participate in or
provide support for that
use data to adjust
instruction or provide
recommendations for
intervention.
School leaders will
log the data-focused
teacher team meetings
that they participate in
or when they provide
direct support for a
group.
Ongoing -
following
teacher
collaboration
days.
3. School leaders follow-
up with teachers on
which students have been
identified via data-
analysis as in need of
additional academic
support through
intervention and other
supplemental learning
opportunities.
The number of requests
by teachers for
classroom learning
opportunity needs or
recommendations for
students to receive
intervention as a result
of data analysis.
School leaders
collect teacher
collaboration data
logs completed by
each team. Log
includes teacher
requests and
recommendations
aligned with data-
based justification.
Weekly or bi-
weekly, based
on the teacher
collaboration
calendar for
the school.
4. School leaders are
responsive to teacher
data-based
recommendations,
providing intervention
and other resources.
The number of
intervention
assignments or
instructional supports
added by the school
leader in response to
data-based teacher
requests.
School leaders track
the intervention
assignments,
instructional
purchases, or
additional resources
they implement along
with the data-based
support for each.
Every six
weeks,
following the
review of
intervention
lab progress
and
adjustments.
5. School leaders share
out short-term examples
of positive progress,
emphasizing the
importance of improving
student learning and
impact it will have on
their future.
The number of
meetings or
communications in
which school leaders
are able to provide a
positive example of
progress or revisit the
goal of improving
student learning.
School leaders log the
meetings or
communications, and
the topics they
covered, which
involved sharing
progress or revisiting
the goal for improving
student learning.
Monthly,
following staff
and school site
leadership
meetings.
125
Required drivers. As part of the implementation, the district should establish specific
drivers (Table 30) that address aspects of motivation, as well as the organization’s resources and
settings to support school leader behaviors. Required drivers reinforce behaviors through job
aids that reference new processes, planning out intentional times for behavior to occur, and
communicating about the change (Kirkpatrick & Kirkpatrick, 2016). It is also important to
include drivers that encourage and reward the behaviors, such as providing culturally relevant
settings for implementing change, communicating out progress through examples of short-term
success, and providing timely resources for support. Finally, required drivers also provide a
means for monitoring behaviors, such as providing documentation or forms for tracking the
productivity and communication that comes from the work towards change.
Table 30
Required Drivers to Support Critical Behaviors
Method(s) Timing
Critical
Behaviors
Supported
1, 2, 3 Etc.
Reinforcing
District provided job aid for school
leaders to review data analysis process
and effective settings.
Ongoing - when preparing for staff,
data analysis, or teacher
collaboration meetings
1, 2, 5
School leadership team establishes
calendar of meetings, identifying each
as a staff data review, data analysis, or
teacher data collaboration.
Ongoing - before the start of each
month
1, 2, 5
District provided job aid calendar of
intervention and common assessment
windows to provide data-availability
context for creating regular
communication to staff
Weekly 1, 2, 5
Encouraging
Job aid to guide data analysis during
teacher collaboration team meetings.
Weekly or Bi-weekly, based on
school collaboration calendar
2, 3
126
Feedback and follow-up between school
leaders and teacher collaboration teams
Ongoing - during or following
teacher collaboration team meetings
5
Rewarding
Acknowledgment at admin meeting,
such as following a benchmark
assessment window when a school
leader has led the data analysis and
responded to data-informed needs
Every 12 weeks 4
Monitoring
Logs for school leaders to track teacher
data-supported resource requests for
improving learning opportunities in
their classrooms.
Ongoing, following data meetings 4
Logs for school resource specialists to
track the assignment and progress
of students identified through data as
needing intervention services.
Ongoing, following data meetings
and during intervention windows
4
Protocols for teachers to request
additional resources and supports based
on data-supported justifications.
Ongoing - part of follow-up with
teacher teams regarding data-
informed recommendations for
intervention and additional learning
opportunity needs
4
Drivers are only effective at supporting critical behaviors if they are implemented. Thus,
it is important to measure their use regularly as a part of tracking the occurrence of critical
behaviors. The accountability for implementation of the drivers includes regular monitoring of
teacher documentation completion by school leaders, as well as district level coaching and
support for school leaders and teachers to regularly schedule and participate in data-focused
collaboration.
Organizational support. Building the capacity for school data use to improve student
learning opportunities requires more than effective school leaders; district leaders and staff must
provide support. District leaders help school leaders by modeling district-level data use,
establishing the goals and timelines for implementation, and providing on-going coaching and
follow-up with data use progress. Similar to the reward mechanism at school sites, district
leaders can motivate school leaders by providing additional resources and supports in response to
127
their data-supported requests. District staff are also essential for providing timely access to
district assessments, relevant data reports, technical support, and on-site workshops and training
for data use. Ensuring access to data and lowering the cognitive load of reviewing the data will
minimize barriers to motivation, while workshops and coaching provide necessary supports for
bridging knowledge gaps.
Level 2: Learning
Learning goals. After completing the recommended program of workshops school
leaders should be able to:
1. Recall the phases of data analysis with 100% accuracy, (Factual).
2. Map the phases of data analysis to current organizational practices and resources with
100% accuracy, (Factual, Conceptual).
3. Carry out the phases of data analysis as part of a collaborative work group, (Conceptual,
Procedural).
4. Recognize data insights within the context of data analysis that suggest students need
additional learning opportunities, (Conceptual, Procedural).
5. Model reflection on the process of the data analysis, describing the data that was
significant, the questions that arose, and the train-of-thought that lead to action steps or
goals, (Conceptual, Procedural, Metacognitive).
6. Evaluate a group’s progress through the data analysis process, providing feedback on the
depth of the analysis and possible additional considerations, (Conceptual, Procedural).
7. Demonstrate the effective behaviors of an instructional leader that is guiding a team
through the data analysis process, (Conceptual, Procedural).
128
8. Analyze data-supported recommendations for learning opportunities and interventions,
determining appropriate support and feedback for the teacher collaboration team,
(Conceptual, Procedural).
9. Discuss post-data analysis goals with teacher teams, establishing a timeline, providing
appropriate supports, and follow-up, (Conceptual, Procedural).
10. Use communication channels, such as email and meetings, to convey the value of data
use for improving learning opportunities for students, (Factual, Conceptual).
Program. The learning goals listed in the previous section will be achieved through a
series of three two-hour training workshops followed by additional review and practice in
district-wide administrator meetings. School leaders will explore relevant data examples,
practice analysis in cooperative groups, and discuss the findings and goals to provide feedback
and elicit reflection on the effectiveness of the analysis.
In the first training workshop, school leaders will receive a job aid mapping the data
analysis process to organizational processes and resources. The workshop leader will introduce a
sample data set to the group of school leaders, modeling the process of data analysis
collaboratively. The initial workshop will focus on the first five learning goals, providing an
overview of the process and the organizational context for how the process connects to the
processes and procedures used in their schools.
The second training workshop will follow a district benchmark testing window, providing
access to their school and district-wide progress-measuring data sets for analysis. The workshop
leader will facilitate and coach groups of school leaders as they work collaboratively to apply the
data analysis process to their data. School leaders will be organized in job-alike groups of
elementary, middle, and high school leaders. Over the course of the workshop, school leaders
129
will demonstrate their ability to apply the first five learning goals with their collaborative group.
The workshop leader will model learning goals six and seven, providing metacognitive dialogue
at key points to exemplify important behaviors, checkpoints, and patterns-of-thought. School
leaders will leave the meeting with data-supported insights, goals, and action steps to
communicate to their site as they begin a site-based data analysis meeting.
The third training workshop will begin with a mock data analysis collaborative group that
consists of four to five school leaders serving as teachers collaborating with a sixth school leader
serving as the participating administrator. As the collaboration group examines district data, the
other school leaders will participate as part of an observation group, recording notes and
suggestions on the process in a fishbowl format. The fishbowl will progress through the phases
of data analysis, facilitated by the workshop leader and pausing between phases to dialogue with
the observation group, analyzing and reflecting on how to better coach the group and provide
support. The goal of the initial hour of the workshop is to utilize the fishbowl exercise to
practice and reflect upon learning goals one to seven, culminating with the mock producing data-
supported recommendations for additional learning opportunities and intervention support.
During the second hour, the workshop leader will guide the entire group of school leaders
through the application of learning goals eight and nine to analyze and develop a plan for
supporting the group. School leaders will also brainstorm examples of communication that the
school leader could provide to staff, highlighting data-based examples of growth and sharing
goals and support that have resulted from the process.
In subsequent monthly district-wide administrator meetings, time will be devoted for
school leaders to share about communication methods they have used for developing the value of
data use, data meetings they have facilitated, challenges they have encountered, and successes
130
they have made towards improving learning opportunities for students through data use. The
goal of the exercise is to serve as a forum for school leaders to learn from each other, building on
each other’s successes and problem-solving the roadblocks they encounter in the process.
Initially, this segment of the administrator meeting should allow up to 30 minutes, as well as
follow-up coaching time with the workshop leader. By mid-year or after three to four monthly
meetings, the portion of the administrator meeting devoted to data use should also shift to serve
as a reflective time for analyzing how the process is supporting data use at school sites and
whether further training workshops for school leaders are needed.
Evaluation of the components of learning. It is essential to measure each school
leader’s progress toward each learning goal, providing the workshop leader data to inform future
workshops and respond to participant needs during the workshop. In addition to checking
declarative, conceptual, procedural, and metacognitive understanding, it is also important to
gauge each school leader’s value for the process both at the start of the workshop series and
throughout. Table 31 provides a list of evaluation methods and timings to be used during the
training series, measuring and gathering evidence of school leader growth.
Table 31
Evaluation of the Components of Learning for the Program
Method(s) or Activity(ies) Timing
Declarative Knowledge “I know it.”
Knowledge checks through quick assessments during the
workshops, such as google forms, hand raising, or other
methods.
During each workshop.
Knowledge checks during group discussion and dialogue, such
as pair-sharing, brainstorming, and other activities.
During each workshop and
documented in participant
notes.
Procedural Skills “I can do it right now.”
Groups working through sample data analysis steps and sharing
their evaluation steps and insights.
During each workshop.
131
School leaders providing feedback during observation of a data
analysis fishbowl activity.
During the third workshop.
Effective use of the data analysis job aid throughout the process. During each workshop.
Follow-up application of the learned skills in workshops two
and three.
During workshops two and
three.
Attitude “I believe this is worthwhile.”
Workshop leader’s observations of individual school leader’s
words and actions throughout the three workshops,
demonstrating whether they value the data analysis process.
During each workshop.
Discussions about the value of the data insights throughout
workshops, as well as when sharing challenges and successes.
During each workshop.
Reflective discussions during workshop three and in the follow-
up district-wide administrator meetings.
During workshop three and
the administrator meetings.
Responses to follow-up surveys following workshops. After each workshop.
Confidence “I think I can do it on the job.”
Responses to follow-up surveys following workshops. After each workshop.
Discussions following the application practice segments and
feedback of the workshops.
During each workshop.
Discussions and reflection during district-wide administrator
meetings.
During each administrator
meeting.
Commitment “I will do it on the job.”
Goals and action plans created to communicate back to their
school sites.
During the first workshop.
Creation of timelines and plans for evaluating data, supporting
teacher teams, and communicating with their school site
throughout the process.
During workshops two and
three.
Responses to follow-up surveys following workshops. After each workshop.
Level 1: Reaction
In order to ensure that each workshop is engaging, relevant, and that school leaders have
a positive experience it is important to measure their reactions. The results will provide insights
into each school leader’s motivation as well as any barriers to motivation that may exist during
the workshop series. Table 32 provides a list of methods to be used, measuring the reactions of
school leaders, as well as the timing for each measurement.
Table 32
Components to Measure Reactions to the Program
Method(s) or Tool(s) Timing
Engagement
132
Attendance to workshops and district-wide
administrator meetings.
During each workshop and the follow-up
district-wide administrator meetings.
Participation in the workshop activities and
discussions.
During each workshop.
Completion of individual notes and activities
during the workshops.
During each workshop.
Observation by the workshop leader. During each workshop.
Completion of a workshop evaluation. At the end of each workshop.
Relevance
Relevance-related survey questions and
discussions in administrator meetings.
After each workshop.
Workshop evaluation. At the end of each workshop.
Customer Satisfaction
Satisfaction-related survey questions and
discussions in administrator meetings.
After each workshop.
Workshop evaluation. At the end of each workshop.
Evaluation Tools
Immediately following the program implementation. The instrument included as
Appendix C is a survey that will be sent out to school leaders directly following the first
workshop on the same day as the training. Section one of the survey aligns with Kirkpatrick
framework level one reaction evaluation measures. The questions seek to measure how engaging
the workshop was for individuals and collaborative groups, how relevant school leaders felt the
training was for supporting their school site improvements, and whether school leaders were
satisfied with the experience. Section two of the survey measures the success of achieving the
learning objectives for the training in alignment with level two of the Kirkpatrick evaluation
framework. The first few questions seek to determine whether school leaders learned the core
declarative knowledge and skills tied to the training. Finally, the remaining questions measure
school leader attitude, confidence, and commitment following the workshop. The data will be
used to direct support for school sites in the weeks that follow the training to ensure successful
application of the learning at school sites. Additionally, the survey results will inform the
133
structure of workshop two, incorporating elements to support school leader needs identified
through the data.
Delayed for a period after the program implementation. The instrument included as
Appendix D is a survey that will be sent out to school leaders three months after the first data
inquiry workshop. Section one of the survey seeks to measure the Kirkpatrick level one factors
of relevance and customer satisfaction, as well as whether school leaders remember key level
two knowledge essential to applying data inquiry. Section two of the survey measures the
behaviors of school leaders since the workshop in alignment with level three of the Kirkpatrick
framework. The focus of the questions is to determine how much school leaders are applying
data inquiry with their staff, as well as to identify barriers and effective supports to application.
Finally, the remaining questions seek to determine whether school leaders are observing positive
results towards the Leading Indicators and Desired Results of the training program in alignment
with Kirkpatrick level four.
Data Analysis and Reporting
Progress towards the desired outcomes can be measured through responses to the
immediate and delayed instruments or surveys found in Appendix C and Appendix D. In order to
communicate the progress, the data from the surveys will be analyzed and visualized through
charts aligned to the Kirkpatrick evaluation levels. Following the immediate survey, the primary
focus will be on visualizing data tied to Kirkpatrick Level two, providing insight into whether
participants left the training with improved knowledge and motivation. Later, the evaluation of
the delayed survey data will focus on showing how participants have demonstrated Kirkpatrick
Level three through critical behaviors and Kirkpatrick Level four through improved outcomes.
The delayed survey can be further modified to gather additional data at checkpoints throughout the
134
year and expand upon the initial visualizations to illustrate progress as depicted in Appendix E.
Reflection during administrator data-focused meetings will allow for time to revisit the
visualizations and explore progress, providing a stimulus to begin the discussion of challenges and
successes behind the data.
Summary
The implementation is structured to align with the New World Kirkpatrick Model and
organization goals in order to provide feedback on progress throughout the improvement
process. Focus on organization goals provides measurable outcomes that can be tracked to
determine if the recommendations are having a positive impact on results. Additionally, since
the Kirkpatrick model identifies the critical behaviors and required drivers, the progress
evaluation data will assist the organization with identifying where reinforcement is needed.
School leaders that excel in the identified practices can also be leveraged to support workshops
and modeling of best practices. Thus, the addition of the evaluations will expand traditional
progress monitoring beyond evaluating workshops to include long-term descriptive data on
whether the desired behaviors are applied, and outcomes are achieved. The organization can
adjust workshops and support resources in response to the evaluation data. Successful
implementation will promote effective school leader behaviors that promote the use of data to
inform learning opportunities and intervention for students in need.
Strengths and Weaknesses of the Approach
The combination of Clark and Estes (2008) gap analysis framework and Kirkpatrick and
Kirkpatrick (2016) New World Kirkpatrick Model provide a lens for evaluating the organization
for gaps in knowledge, motivation, and organizational structures, as well as creating an
evaluation-based plan with recommendations to address the identified influence-related needs.
135
However, a weakness of this approach for this study can be found with the level of disclosure
that each school leader was willing to provide during the survey and interview. School leaders
are invested in the reputation of their school and may not want to disclose anything they perceive
as a weakness in their programs. Additionally, the researcher was a member of the organization,
and despite all of the procedural measures taken to ensure anonymity, it is possible school
leaders may have withheld some concerns or details, leading to limitations on the data collected.
Limitations and Delimitations
Several limitations exist within the study that should be considered. The first limitation
was that the research was conducted by a member of the organization, working as a peer to the
school leaders that served as the focus stakeholders. As peers, the researcher did not serve in any
sort of supervisory role to the participants. However, there was concern about the level of
disclosure participants were willing to have with a fellow administrator in their organization. In
all cases, the participants did appear to be open and willing to share their insights freely, but the
limitation is still a possibility. Additionally, it should be noted that only eight interviews were
completed, representing the eighteen school leaders in the district. While the survey responses
were used to ensure a diverse range of interview participants and the qualitative data appeared to
reach a saturation of details, it is possible that additional insights could have been revealed if all
school leaders participated. However, it is also important to note that none of the school leaders
refused to participate, so the researcher did not believe that any information or insights were
withheld intentionally. It should also be noted that the results and findings were specific to the
context of the school district, limiting the ability for generalization to other organizations.
Finally, due to school closures as a result of COVID-19, the study was unable to compare annual
136
assessment data as means of addressing research question 1 and providing a measure of how
successful the school data practices were at improving student achievement.
The study entailed few delimitations. The primary delimitation of the study was that it
only focused on stakeholder group, school leaders. Other stakeholder groups, such as teachers,
district leaders, and curriculum coaches, may have provided additional insights into evaluating
the organization’s pursuit for increasing student achievement through date use. However, the
additional stakeholders were outside the scope of this study.
Future Research
Due to the limitations and delimitations of this study, it is recommended to conduct
more in-depth research, expanding the scope of the study. The study focused on a single school
district and a single stakeholder type within the school district. Replication of the study in other
school districts could provide a deeper understanding of how school leaders support effective
data use to improve student achievement and increase the ability to generalize the results and
findings to other organizations. Additionally, school leaders provide one lens of understanding
for establishing effective data use within schools. Future studies could expand the stakeholder
focus to include other professionals that contribute to improving student achievement, such as
district leaders, teachers, and curriculum coaches, providing an expanded understanding of how
each stakeholder group values and supports effective data use. Finally, a long-term replication of
the study is recommended to provide additional depth of understanding around the impact of data
use over multiple years, as well as elements that impact the consistency of data use. Effective
data use to improve student achievement requires consistent use of a process over multiple years,
and a long-term study of an effective program could reveal how organizations establish a
program capable of sustained growth.
137
Based on the findings of the study, additional research is also recommended to evaluate
the data-based decision-making training programs, and their effectiveness in providing
organizations with the skills and tools to drive change. Education in the United States has
established data-based accountability throughout the system, and schools are driven to improve
their student outcomes. An in-depth study exploring the data-based decision-making programs
available to schools, and the value they have provided districts could lead to improvements
across the country, as well as help districts to make effective funding investments.
Conclusion
The purpose of this study was to evaluate the knowledge, motivation, and organizational
influences on the capacity of school leaders to lead the use of data by staff, driving an increase in
student achievement within schools. Schools have access to multiple measures for informing the
support of student achievement yet may struggle to leverage the data for a variety of reasons. An
abundance of data is generated through learning opportunities, as well as to meet the
accountability requirements of State and Federal governments (Jimerson, 2014). However,
connecting the multiple measures for analysis in ways that increasingly provide insights for
improving student achievement requires additional intentional action (Portz, 2017). Schools
collect data to meet reporting requirements, but the true value for improving each student’s
achievement in the moment may be overlooked if they do not have a process in place to analyze
the data and inform ongoing instruction. Redundant data collection steps can also be revealed
providing increased levels of efficiency (Bin Mat, Buniyamin, Arsad, & Kassim, 2013). Schools
have limited funding and resources with which to provide support for students, so finding ways
to more effectively use funds provide additional value. Overall, opportunities to improve the
utilization and efficiency of data that is consistently collected will enable schools to make more
138
informed decisions on the allocation of resources yielding greater achievement for all students.
The study analyzed survey and interview data collected from SUSD school leaders. The Clarke
and Estes (2008) gap analysis framework provided a methodology to generate findings, revealing
that all the school leaders supported data use to improve student achievement at their schools.
However, it was also revealed that the influence assets and needs differ between the type of
school. School leaders supporting elementary schools encountered different challenges than
those leading secondary schools. All the assumed influences were identified as needs and the
Kirkpatrick and Kirkpatrick (2016) New World Model was used to develop recommendations
and an implementation plan for addressing the needs.
Student achievement is a primary goal for schools, and school leaders invest time and
energy into finding and implementing a range of solutions to drive improvement. As this study
revealed, schools have a lot of other goals as well, so the time for data use is limited and must be
used efficiently. Data use can often become siloed around a requirement, event, or problem,
creating inefficiency. School leaders equipped with a process and the skills of effective data use
have the access and ability to identify the connections and opportunities for effective data use
within their schools. Through education, influence, and the provision of supports, school leaders
can empower their teachers to blend the science of data use with the craft of their pedagogy. The
COVID-19 pandemic of 2020 impacted this study as it prevented schools from gathering annual
student proficiency data. The loss of the proficiency data coupled loss of instruction due to
school closure has created new problems for schools to solve. Addressing the problems will
involve staff developing assessments that can find the gaps in student learning, as well as
monitoring the progress of recovery toward learning goals. Effective data use within schools will
be essential for identifying gaps in learning due to the closures and learning loss. With limited
139
funding and confronted with a variety of students that represent different educational stories and
needs, school leaders are face a new challenge to build the data use capacity of their staff, so they
recover from the learning deficits due to COVID-19 and continue their goal to improve the
educational outcome for every student.
140
REFERENCES
Anderman, E., & Anderman, L. (2006). Attributions. Retrieved from
http://www.education.com/ reference/article/attribution-theory/
Baker, L. (2006). Metacognition. Retrieved from http://www.education.com/reference/article/
metacognition/
Bin Mat, U., Buniyamin, N., Arsad, P.M., & Kassim, R. (2013). An overview of using
academic analytics to predict and improve students’ achievement: A proposed proactive
intelligent intervention. IEEE 5th Conference on Engineering Education (ICEED), 126–
130. https://doi.org/10.1109/ICEED.2013.6908316
Black, P., & Wiliam, D. (2010, September). Inside the black box: Raising standards through
classroom assessment. Phi Delta Kappan, 92(1), 139–148.
http://pdk.sagepub.com/content/92/1/81.abstract
Bogdan, R.C., & Biklen, S.K. (2007). Qualitative research for education: An introduction to
theories and methods (5th ed.). Boston, MA: Allyn and Bacon.
Boudett, K.P., & Steele, J.L. (2007). Data wise in action: Stories of schools using data to
improve teaching and learning. Cambridge, MA: Harvard Education Press.
Bowen, G.A. (2009). Document analysis as a research method. Qualitative Research Journal,
9(2), 27-40.
Childress, S., Elmore, R., & Grossman, A. (2006). How to manage urban school districts.
Harvard Business Review, 84(11), 55–68.
Clark, R.E., & Estes, F. (2008). Turning research into results: A guide to selecting the right
performance solutions. Charlotte, NC: Information Age Publishing, Inc.
141
Creswell, J.W. (2014). Research design: Qualitative, quantitative, and mixed methods
approaches. Thousand Oaks, CA: SAGE.
Daly, E. (2006) Behaviorism. Retrieved from http://www.education.com/reference/article/
behaviorism/
Datnow, A., & Hubbard, L. (2016). Teacher capacity for and beliefs about data-driven decision
making: A literature review of international research. Journal of Educational Change,
17(1), 7–28. doi: 10.1007/s10833-015-9264-2
Datnow, A., & Park, V. (2015). Data use--For equity. Educational Leadership: Journal of the
Department of Supervision and Curriculum Development, N.E.A, 72(5), 48–54.
Retrieved from http://www.ascd.org/publications/educational-leadership/
feb15/vol72/num05/Data use%E2%80%94For-Equity.aspx
Datnow, A., & Park, V. (2018). Opening or closing doors for students? equity and data use in
schools. Journal of Educational Change, 19(2), 131-152.
doi:http://dx.doi.org.libproxy2.usc.edu/10.1007/s10833-018-9323-6
Datnow, A., Park, V., & Kennedy‐Lewis, B. (2013) Affordances and constraints in the context
of teacher collaboration for the purpose of data use, Journal of Educational
Administration, Vol. 51 Issue: 3, pp.341-362,
https://doi.org/10.1108/09578231311311500
Dembo, M.H., & Eaton, M.J. (2000). Self-regulation of academic learning in middle-level
schools. The Elementary School Journal, 100(5), 473–490. doi: 10.1086/499651
Denler, H., Wolters, C., & Benzon, M. (2006). Social cognitive theory. Retrieved from
http://www.education.com/reference/article/social-cognitive-theory/
142
Diamond, J. (2008). Focusing on student learning. In M. Pollack (Ed.), Everyday anti-
racism: Getting real about race in school (pp. 254-256). New York, NY: The New
Press.
Dowd, A.C. (2005). Data Don’t Drive: Building a Practitioner-Driven Culture of Inquiry to
Assess Community College Performance. Lumina Foundation for Education Research
Report. Lumina Foundation for Education. Retrieved from https://eric-ed-gov.
libproxy1.usc.edu/?id=ED499777
Elmore, R.F. (2002). Bridging the gap between standards and achievement. Washington, DC:
Albert Shanker Institute. Retrieved from http://www.shankerinstitute.org/
resource/bridging-gap-between-standards-and-achievement
Fink, A. (2013). How to conduct surveys: A step-by-step guide (5th ed.). Thousand Oaks, CA:
SAGE Publications.
Firestone, W.A., & Gonza ́lez, R.A. (2007). Culture and processes affecting data use in school
districts. In P. A. Moss (Ed.), Evidence and decision making. Yearbook of the National
Society for the Study of Education (pp. 132–154). Malden, MA: Blackwell.
Glesne, C. (2011). Chapter 6: But is it ethical? Considering what is “right.” In Becoming
qualitative researchers: An introduction (4th ed.). 162-183. Boston, MA: Pearson.
Harwati, Alfiani, A.P., & Wulandari, F.A. (2015). Mapping Student’s Performance Based on
Data Mining Approach (A Case Study). Agriculture and Agricultural Science Procedia,
3, 173–177. https://doi.org/10.1016/j.aaspro.2015.01.034
Horn, I.S., & Little, J.W. (2010). Attending to problems of practice: Routines and resources for
professional learning in teachers’ workplace interactions. American Educational
Research Journal, 47(1), 181–217.
143
Hubbard, L., Datnow, A., & Pruyn, L. (2014). Multiple initiatives, multiple challenges: The
promise and pitfalls of implementing data. Studies in Educational Evaluation, 42, 54–62.
doi: 10.1016/j.stueduc.2013.10.003
Huguet, A., Farrell, C.C., & Marsh, J.A. (2017). Light touch, heavy hand: principals and data
use PLCs. Journal of Educational Administration and History, 55(4), 376–389. doi:
10.1108/JEA-09-2016-0101
Irwin, C.W., & Stafford E.T. (2016). Survey methods for educators: Collaborative survey
development (part 1 of 3). Washington, DC: U.S. Department of Education, Institute of
Education Sciences, National Center for education Evaluation and Regional Assistance,
Regional Educational Laboratory Northeast & Islands. Retrieved from
http://ies.ed.gov/ncee/edlabs.
Jimerson, J.B. (2014). Thinking about data: Exploring the development of mental models for
“data use” among teachers and school leaders. Studies in Educational Evaluation, 42, 5–
14. https://doi.org/10.1016/j.stueduc.2013.10.010
Jimerson, J.B., & McGhee, M.W. (2013). Leading inquiry in schools: Examining mental
models of data-informed practice. Current Issues in Education, 16(1). Retrieved from
http://cie.asu.edu/ojs/index.php/cieatasu/article/view/1163
Johnson, R.B., & Christensen, L.B. (2015). Educational research: Quantitative, qualitative,
and mixed approaches (5th ed.). Thousand Oaks: SAGE.
Katz, S., & Dack, L.A. (2014). Towards a culture of inquiry for data use in schools: Breaking
down professional learning barriers through intentional interruption. Studies in
Educational Evaluation, 42, 35–40. doi: 10.1016/j.stueduc.2013.10.006
144
Kirkpatrick, J.D., & Kirkpatrick, W.K. (2016). Kirkpatrick’s four levels of training evaluation.
Alexandria, VA: ATD Press.
Kirschner, P., Kirschner, F., & Paas, F. (2006). Cognitive load theory. Retrieved from
http://www.education.com/reference/article/cognitive-load-theory/
Kotter, J.P. (2007). Leading change: Why transformation efforts fail. Harvard Business
Review, 85(1), 96-103.
Levy, G.D., & Ronco, S.L. (2012). How benchmarking and higher education came together.
New Directions for Institutional Research, 2012(156), 5–13. doi: 10.1002/ir.20026
Local Control and Accountability Plan (LCAP) - Resources (CA Dept of Education). (n.d.).
Retrieved November 28, 2018, from https://www.cde.ca.gov/re/lc/
Long, L., Rivas, L.M., Light, D., & Mandinach, E.B. (2008). The evolution of the homegrown
data warehouse: TUSDSstats. In E. B. Mandinach & M. Honey (Eds.), Data-driven
school improvement: Linking data and learning. New York: Teachers College Press.
Marsh, J. A. (2012). Interventions promoting educators' use of data: Research insights and
gaps. Teachers College Record, 114(11), 1–48.
Marsh, J.A., & Farrell, C.C. (2015). How leaders can support teachers with data-driven decision
making: A framework for understanding capacity building. Educational Management
Administration Leadership, 43(2), 269–289. doi: 10.1177/1741143214537229
Marsh, J.A., Pane, J.F., & Hamilton, L. S. (2006). Making sense of data-driven decision
making in education. Retrieved from https://www.rand.org/pubs/occasional_papers/
OP170.html?doc=OP-170-EDU
Maxwell, J.A. (2013). Qualitative research design: An interactive approach (3rd ed.).
Thousand Oaks, CA: SAGE.
145
Mayer, R.E. (2011). Applying the science of learning. Boston, MA: Pearson Education.
Merriam, S.B., & Tisdell, E. J. (2016). Qualitative research: A guide to design and
implementation (4th ed.). San Francisco, CA: Jossey-Bass.
Miranda, C.P., & Jaffe-Walter, R. (2018). When data use devolves into deficit talk: Creating the
conditions for productive teacher collaboration using data. Journal of Cases in
Educational Leadership. doi: 10.1177/15554589177515
NAEP Data Explorer. (n.d.). 2017 Reading Scores. Retrieved May 31, 2018, from
https://www.nationsreportcard.gov/ndecore/xplore/nde
Park, V., Daly, A.J., & Guerra, A.W. (2012). Strategic framing: How leaders craft the meaning
of data use for equity and learning. Educational Policy, 27(4), 645–675.
doi:10.1177/0895904811429295
Patton, M.Q. (2002). Qualitative research & evaluation methods (3rd ed.). Thousand Oaks,
CA: SAGE Publications.
Pintrich, P.R. (2003). A motivational science perspective on the role of student motivation in
learning and teaching contexts. Journal of Educational Psychology, 95, 667–686.
doi:10.1037/0022-0663.95.4.667
Portz, J. (2017). “Next-Generation” Accountability? Evidence From Three School Districts.
Urban Education. https://doi.org/10.1177/0042085917741727
Romero, C., & Ventura, S. (2013). Data mining in education. Wiley Interdisciplinary Reviews:
Data Mining and Knowledge Discovery, 3(1), 12–27. https://doi.org/10.1002/widm.1075
Rubin, H.J., & Rubin, I.S. (2012). Chapter 6: Conversational partnerships. In Qualitative
interviewing: The art of hearing data (3rd ed.) (pp. 85-92). Thousand Oaks, CA: SAGE
Publications.
146
Rueda, R. (2011). The 3 dimensions of improving student performance. New York, NY:
Teachers College Press.
Salkind, N.J. (2017). Statistics for people who (think they) hate statistics: Using Microsoft
Excel 2016 (4th ed.). Thousand Oaks, CA: SAGE.
Schein, E.H. & Schein, P. (2017). Organizational culture and leadership. Hoboken,
NJ: John Wiley and Sons, Inc.
Schildkamp, K., & Poortman, C. (2015). Factors influencing the functioning of data teams.
Teachers College Record, 117(4), 1–42.
Schildkamp, K., Poortman, C.L., Ebbeler, J., & Pieters, J.M. (2019). How school leaders can
build effective data teams: Five building blocks for a new wave of data-informed
decision making. Journal of Educational Change, 20(3), 283-325.
doi:http://dx.doi.org.libproxy2.usc.edu/10.1007/s10833-019-09345-3
Shraw, G., & Lehman, S. (2009). Interest. Retrieved from http://www.education.com/
reference/article/interest/
Schraw, G., & McCrudden, M. (2006). Information processing theory. Retrieved from http://
www.education.com/reference/article/information-processing-theory/
Van Gasse, R., Vanlommel, K., Vanhoof, J., & Van Petegem, P. (2017). Unravelling data use in
teacher teams: How network patterns and interactive learning activities change across
different data use phases. Teaching and Teacher Education, 67, 550–560.
https://doi.org/10.1016/j.tate.2017.08.002
Waters, T., Marzano, R.J., & McNulty, B. (2003). Balanced leadership: What 30 years of
research tells us about the effect of leadership on student achievement. Aurora, CO:
Midcontinent Research for Education and Learning.
147
Wayman, J.C., & Jimerson, J.B. (2014). Teacher needs for data-related professional learning.
Studies in Educational Evaluation, 42, 25–34. doi: 10.1016/j.stueduc.2013.11.001
Yough, M., & Anderman, E. (2006). Goal orientation theory. Retrieved from http://www.
education.com/reference/article/goal-orientation-theory/
148
APPENDIX A
Survey Items
1. I am: (Male, Female)
2. What type of school do you serve? (Elementary, Secondary)
3. How many years have you served as an administrator? ____
4. How many years have you worked in education as whole? ____
5. When proposing a change, data is provided to support the proposal. (strongly disagree,
disagree, agree, strongly agree)
6. Grade level or subject teams are provided with materials and resources that provide
structure for their data-based decision making. (strongly disagree, disagree, agree,
strongly agree)
7. As a school leader, I model data-driven educational practices. (strongly disagree,
disagree, agree, strongly agree)
8. Our school has a process for analyzing data to inform improvements that is used in staff
meetings as well as team planning. (strongly disagree, disagree, agree, strongly agree)
9. When working with teachers, I demonstrate data-based decision making by talking
through my thought processes as I decipher and interpret the data. (strongly disagree,
disagree, agree, strongly agree)
10. Staff meetings focus on progress toward data-based improvement goals. (strongly
disagree, disagree, agree, strongly agree)
11. As a school leader, I hold teams accountable to develop data-based goals for improving
student achievement. (strongly disagree, disagree, agree, strongly agree)
12. Student data is used to determine resource allocation. (strongly disagree, disagree, agree,
149
strongly agree)
13. The school’s improvement goals are clear, specific, measurable, and based on student
data. (strongly disagree, disagree, agree, strongly agree)
14. As a school we have open and honest discussions about data. (strongly disagree,
disagree, agree, strongly agree)
15. My school uses multiple data sources to improve how we support students and learning.
(strongly disagree, disagree, agree, strongly agree)
16. Student data informs decisions on teacher professional development needs and resources.
(strongly disagree, disagree, agree, strongly agree)
17. Teachers in my school use assessment data to identify students that need academic
support. (strongly disagree, disagree, agree, strongly agree)
18. Teachers in my school make changes in their instruction based on assessment data.
(strongly disagree, disagree, agree, strongly agree)
19. Teachers in my school use data from student assessments to set instructional action goals.
(strongly disagree, disagree, agree, strongly agree)
20. My school adequately supports teachers’ use of data to improve classroom instruction.
(strongly disagree, disagree, agree, strongly agree)
21. Using data has improved the quality of decision making in my school. (strongly disagree,
disagree, agree, strongly agree)
22. When meeting with teachers individually for fall planning, we review student data to
support goals for the year. (strongly disagree, disagree, agree, strongly agree)
23. When meeting with teachers individually for their spring review, we review student data
from the year as part of revisiting their annual goals. (strongly disagree, disagree, agree,
150
strongly agree)
24. Teacher teams in my school meet regularly to look at student data and make instructional
plans. (strongly disagree, disagree, agree, strongly agree)
25. When teachers in my school meet for collaboration, they usually focus on improving
student learning outcomes. (strongly disagree, disagree, agree, strongly agree)
151
APPENDIX B
Interview Protocol
Thank you for joining me to participate in this interview. The research is part of my
dissertation for the USC EdD program, and focuses on evaluating the key influences on how
school leaders support their staff in developing effective data practices. First, please trust me
that as a participant in this study your identity will not be revealed, and the results will be
presented in general themes of school leaders at either the elementary (pk-5) or secondary (6-12)
grade levels. If this interview leads to something that I would wish to quote or cite, I will use a
pseudonym such as “Secondary School Leader A”. Prior to publishing, you will have the option
to review the document and ensure that your identity is not inadvertently revealed. I would like,
with your permission, to record our conversation and transcribe it for use later as part of my data
review. Following this interview, if you agree to the recording, you will have the opportunity to
review the transcript to ensure the accuracy of your responses. The recording and transcript will
be stored in a locked safe in my home when not in use, and after my research is completed all
recordings will be destroyed and all identifying information will be purged from the archived
transcript records. Do I have your permission to record the interview?
(Pause for response and/or discussion)
The interview questions were created in line with a review of current literature on the
topic of supporting effective data practices. If you do not wish to answer any question let me
know, and if at any time you decide you wish to end the interview please let me know. Also,
please remember this is a voluntary study so you can withdraw from the study at any time prior
to publication.
Do you have any questions before we begin?
152
(Pause for response and/or discussion)
(Turn recorder on if permitted)
I would like to begin by learning about your background prior to serving in your current school
leadership role.
1. Tell me about you background before serving as a school leader. What did you do before
becoming a principal?
2. In your prior role, can you please describe a typical experience with data driven
practices?
3. Through your experiences with data use, what types of training have you received to
build your capacity for data use?
4. If you were to assign an overall goal for the data driven practices at your school, what
would it be?
Now if we can shift our focus a bit, I would like to discuss a related topic of student
achievement.
5. What do you feel are the most important factors for the use of data for improving student
achievement?
a. What tools do you use to evaluate these factors?
6. What are your beliefs about the ability for students to achieve proficiency?
7. What are your thoughts about the use of data for accountability?
a. In terms of data use, when do you feel like it most useful, and when is it least
useful?
b. How do you think data should be used within schools?
Thank you, so if we look at data-driven practices through the lens of instruction…
8. What do you think are the challenges to using data effectively?
9. What factors do you think lead to more effective data use??
10. In what ways do you communicate relevant data to your staff?
11. What do think are the best methods for training teachers to use data?
12. How do you provide support for teachers to use data?
13. What types of interactions do you have during teacher collaboration that are tied to data
use?
14. How would you describe the effectiveness of the current data use at your school to
improve student achievement?
15. Tell me about a time when a team used data to improve student ELA achievement?
16. Explain how support staff (IT, coaches, techs, etc.) contribute to data use for improving
student achievement?
17. What additional supports do you wish you could have for data use?
18. What role do you see accountability playing in supporting the use of data driven practices
for improving student achievement?
Okay, we are just about done, and I know I have asked a lot of questions.
153
19. Based on our interview, is there anything else that you feel would be helpful for me to
know or understand about data driven practices in your school?
Thank you, this has been a helpful interview. I am deeply grateful for your willingness to share
your opinions and thoughts on the topic. When I have a draft ready for your review I will let you
know.
154
APPENDIX C
Initial Workshop Evaluation Tool
Figure C1. Section one of the survey that will immediately follow the first workshop
implementing a program for addressing the knowledge, motivational, and organizational
influences identified by the study.
155
Figure C2. Section two of the survey that will immediately follow the first workshop
implementing a program for addressing the knowledge, motivational, and organizational
influences identified by the study.
156
APPENDIX D
Delayed Workshop Evaluation Tool
Figure D1. Section one of the survey that will follow the first workshop after six to eight weeks
of implementation of critical behaviors and drivers.
157
Figure D2. Section two of the survey that will follow the first workshop after six to eight weeks
of implementation of critical behaviors and drivers.
158
APPENDIX E
Data Visualization Concept for Displaying Progress Towards Desired Outcomes
Figure E1. Mock dashboard charts for visualizing the status of implementation progress as
determined through surveys of school leaders throughout the process.
Abstract (if available)
Abstract
Schools regularly collect data regarding student progress and learning gaps. However. the support, structures, and value school leaders place on the use of data for informing instruction and intervention influences whether data informs student achievement improvement. This evaluation study examines how Sunshine Unified School District (SUSD) school leaders support data use by teachers to improve student achievement in English Language Arts and Mathematics. A gap analysis framework was utilized, focusing on the knowledge and skill, motivation, and organizational barriers influencing school leaders at the SUSD schools. Quantitative and qualitative data was collected through a survey and interviews. The findings suggest that school leaders must have knowledge and organization structures that establish dedicated time for teachers to collaborate around data, as well as consistently reinforce the belief that the purpose of data use is for student achievement improvement. Successful implementations of the recommendations is anticipated to develop skills and structures for school leaders to effectively build the capacity for teachers to use data to inform their instructional practices and interventions, and ultimately lead to the accomplishment of the organization’s goal to increase student achievement.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
School leaders' use of data-driven decision-making for school improvement: a study of promising practices in two California charter schools
PDF
Perception of alternative education teachers readiness to instruct English language learners: an evaluation study
PDF
Building teacher competency to work with middle school long-term English language learners: an improvement model
PDF
Understanding the factors that contribute to successful school reconstitution: A promising practice
PDF
The interaction of teacher knowledge and motivation with organizational influences on the implementation of a hybrid reading intervention model taught in elementary grades
PDF
Implementation of the International Baccalaureate Middle Years Program in an urban secondary school: an improvement practice to address closing the achievement gap
PDF
Sustaining faith-based organizations through leadership pipelines and programs: an evaluation study
PDF
Trending upward: an evaluation study of teacher practices in serving special needs students in a public high school
PDF
Influencing teacher retention: an evaluation study
PDF
An evaluation study of... What do teachers know about gifted students?
PDF
Collaborative instructional practice for student achievement: an evaluation study
PDF
Allocation of educational resources to improve student learning: case studies of California schools
PDF
Examining the faculty implementation of intermediate algebra for statistics: An evaluation study
PDF
Effective implementation of technology in elementary schools: an evaluation study
PDF
Explicit instruction’s impact on the student achievement gap in K-12 English language learners
PDF
Minding the gap: an evaluation of faculty, staff and administrator readiness to close equity gaps at the California State University
PDF
Supporting teachers' mental health: an evaluation study
PDF
Development of intraorganizational post-merger collaboration plan: an evaluation study
PDF
Building leaders: the role of core faculty in student leadership development in an undergraduate business school
PDF
The role of international school teacher leaders in building leadership capacity within their teams
Asset Metadata
Creator
Bennett, Daniel Adam
(author)
Core Title
Building data use capacity through school leaders: an evaluation study
School
Rossier School of Education
Degree
Doctor of Education
Degree Program
Organizational Change and Leadership (On Line)
Publication Date
09/25/2020
Defense Date
07/27/2020
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
achievement improvement,data use,data-based decision making,OAI-PMH Harvest,school leader
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Hasan, Angela (
committee chair
), Malloy, Courtney (
committee member
), Ott, Maria (
committee member
)
Creator Email
dabennett7@gmail.com
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c89-381796
Unique identifier
UC11665981
Identifier
etd-BennettDan-9019.pdf (filename),usctheses-c89-381796 (legacy record id)
Legacy Identifier
etd-BennettDan-9019.pdf
Dmrecord
381796
Document Type
Dissertation
Rights
Bennett, Daniel Adam
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
achievement improvement
data use
data-based decision making
school leader