Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Data use in middle schools: a multiple case study of three middle schools’ experiences with data-driven decision making
(USC Thesis Other)
Data use in middle schools: a multiple case study of three middle schools’ experiences with data-driven decision making
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
DATA USE IN MIDDLE SCHOOLS:
A MULTIPLE CASE STUDY OF THREE MIDDLE SCHOOLS’ EXPERIENCES
WITH DATA-DRIVEN DECISION MAKING
by
Lars A. Nygren
A Dissertation Presented to the
FACULTY OF THE ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION
August 2009
Copyright 2009 Lars A. Nygren
ii
TABLE OF CONTENTS
LIST OF TABLES v
LIST OF FIGURES vi
ABSTRACT vii
CHAPTER ONE 1
Overview of the study 1
Introduction 1
Background of the Problem 1
Educational Reform and Data-Driven Decision Making 4
Research Questions 6
Significance of the Study 7
CHAPTER TWO 8
Literature Review 8
Introduction 8
Impact of Federal and State Accountability Policies K-12 Public Education 9
Middle School Reform 12
Merits of Data Use 16
School Site Leadership and Data-Driven Decision Making 18
Challenges of Creating a Data-Use School Culture 22
Building Capacity for Data-Driven Decision Making 24
Types of Data Used in Schools 27
Processes that Support Data Use 30
Summary of the Literature Review 33
CHAPTER THREE 36
Methodology 36
Introduction 36
Research Design 36
Sample and Population 38
Overview of District and Schools 39
Data Collection Procedures 42
Data Analysis Procedures 43
Limitations of the Study 44
Researcher’s Subjectivity 45
Ethical Considerations 45
CHAPTER FOUR 47
Data Analysis and Interpretation of Findings 47
iii
Introduction 47
Research Questions and Thematic Underpinnings 48
School Settings 49
How Middle School Principals Support Teachers with Data-Driven 51
Decision Making
Making Data Available 52
Expectations of Data Use 55
Supports in Use of Data 58
How School Structures and Cultures Facilitate or Inhibit Data-Driven 62
Decision Making
Time for Data Discussion 62
Culture 65
What Types of Data are Collected by Educators 70
Summative Assessment 71
Benchmark Assessments 73
Formative Assessment 76
How are Data Used by Leaders and Teachers and for What Purposes 78
Student Placement and Support 79
Teacher Use of Data 82
Conclusion 85
CHAPTER FIVE 88
Summary and Implications of Findings 88
Introduction 88
Connections to Prior Research 91
Impact of Federal and State Accountability Policies on Education 91
Middle School Reform 92
Merits of Data Use 93
School Site Leaders and Data-Driven Decision Making 94
Challenges of Creating a Data-Use School Culture 96
Building Capacity for Data-Driven Decision Making 97
Types of Data Used in School 98
Processes that Support Data Use 99
Implications for Future Research 100
Implications for Policy and Practice 101
Provide Time for Collaboration 101
Professional Development 101
Benchmark Assessment 102
Leadership Support 103
Conclusion 103
REFERENCES 105
iv
APPENDICES
APPENDIX A: Principal Interview Protocol 109
APPENDIX B: Teacher Interview Protocol 113
APPENDIX C: Harrison Data Forms 117
APPENDIX D: Code List 125
v
LIST OF TABLES
Table 1: Student Demographics – Burbank Middle School 39
Table 2: Accountability Profile – Burbank Middle School 40
Table 3: Student Demographics – Harrison Middle School 40
Table 4: Accountability Profile – Harrison Middle School 40
Table 5: Student Demographic – Lincoln Middle School 41
Table 6: Accountability Profile – Lincoln Middle School 41
Table 7: Subjects and Grades Assessed in the CST 71
vi
LIST OF FIGURES
Figure 1: Diagram of BASRC Cycle of Inquiry 32
vii
ABSTRACT
Over the last decade, schools in California have been experiencing the dramatic
effects of NCLB and the Public School Accountability Act of 1999. Both the state and
federal legislation have called for increased student achievement partnered with
accountability. As a result of such policies, data and accountability have grown to hold a
central place in large-scale school reform. With little guidance, many middle schools are
now being held accountable for having to put into practice data-driven decision making
in order to help improve teaching and learning.
The purpose of this qualitative multiple-case study was to examine the processes
and practices of middle schools that used data to drive their decisions in order to improve
student achievement, The following research questions were addressed: (1) What is the
role of the middle school leader in supporting teachers to use data to drive their
instruction? (2) How do school structures and cultures facilitate or inhibit data driven
decision making? (3) What types of data are collected by educators? (4) How are data
used by leaders and teachers and for what purposes?
I chose three high performing middle schools in Southern California for my study.
The schools chosen for the study were selected based upon meeting a specific criterion.
Selected schools had met their API and AYP targets in each of the three preceding school
years (2005-2006 through 2007-2008). Additionally, the schools selected attributed data
to having a role in helping them reach their API and AYP over the preceding three years.
Qualitative case study research methods were used in this study
1
CHAPTER ONE
Overview of the Study
Introduction
Background of the Problem
Over the years, there has been an increasing level of accountability placed on
schools. Nationally, the Elementary and Secondary Education Act (ESEA) was first
enacted into law in 1965 under President Lyndon B. Johnson. Its purpose was to help
support the efforts of states to provide a high-level of education for all students. Over the
years, the ESEA increased the accountability of states who received federal money. For
instance, with the 1984 reauthorization of the ESEA, the federal government began to ask
states to collect data on the population that the educational program served and use the
data gathered to determine the effectiveness of the program.
In the meantime, in 1983, a congressional report was released entitled “A Nation
at Risk” (National Commission on Excellence in Education, 1983). The landmark study
examined the achievement of students in schools in the United States. This report had the
effect of raising the sense of urgency to increase academic achievement of children in the
country. The report made statements such as:
We report to the American people that while we can take justifiable pride in what
our schools and colleges have historically accomplished and contributed to the
United States and the well-being of its people, the educational foundations of our
society are presently being eroded by a rising tide of mediocrity that threatens our
very future as a Nation and a people. (p. 8)
2
The “Nation at Risk” report caused a shift in the thinking about America’s educational
system and resulted in changes in federal, state, and local policies toward education
(Guthrie & Springer, 2004). It helped to launch a movement whereby schools would
increasingly be judged on the outcomes of their students’ achievements rather than on the
resources the schools received (Guthrie & Springer, 2004). Accountability for schools
was on the rise.
States across the nation felt the pressure to improve students’ academic
achievement. In California, a move toward a standards-based system began in the mid-
1980s. By the mid-1990s, California instituted reform in order to create a system where
financial, curriculum, professional development, assessments, and accountability systems
were aligned to the state’s academic standards (EdSource, 2005). Some believed that
through the alignment of systems, there would be an increase in students’ academic
achievement (EdSource, 2005).
California’s State Board of Education adopted state wide academic standards in
1999. State wide standards were created for all four core subject areas – math, science,
social studies, and English language arts. Schools were accountable for assessing their
students’ knowledge of the standards through the use of statewide tests.
The Public Schools Accountability Act (PSAA) of 1999 was passed in an effort to help
monitor the progress of students in the achievement of state standards. Schools were
assigned an Academic Performance Index (API) score based upon the performance of
their students on the statewide tests. In a move that brought more attention to schools in
California, the PSAA made the API scores of all schools available to the public. The API
3
score is seen by many to reflect the success or failure of schools to teach their students
the state content standards. California schools would be rewarded and sanctioned based
upon the performance of their students (Woody, Buttles, Kafka, Park, & Russell, 2004).
California made additional changes to their accountability system when the
federal legislation entitled No Child Left Behind (NCLB) was passed into law in 2002.
NCLB was the reauthorization of the earlier federal ESEA. Under NCLB, states and
their schools were held accountable for the academic progress of their students.
California schools and districts had to show their students were making gains toward
“proficient” or “advanced” status as defined by California. Furthermore, NCLB placed
an emphasis on the achievement of the different student subgroups. The different student
subgroups, which included special education, English Language Learners, African-
Americans, Hispanic, and Asian student groups were to make gradual increases in
academic achievement toward “proficiency.” Additionally, schools needed to have at
least 95% of their students tested on the state content standards. If schools and districts
failed to accomplish the requirements of NCLB, they would face sanctions.
Under NCLB, schools and districts that receive Title I funding face a number of
different sanctions if their students fail to make continual improvement. The AYP
criteria include four areas they are: participation rate, percent proficient (also known as
Annual Measurable Objectives), API, and graduation rate if applicable. Schools which
fail to make their AYP for two consecutive years enter into the Program Improvement
(PI) process. PI schools face increasingly harsher penalties under NCLB that can range
4
from having students be eligible to transfer to another school to shutting down the school
completely.
Educational Reform and Data-Driven Decision Making
As a result of the state and federal accountability measures, many schools have
undergone a process of reform to meet educational achievement goals. Schools have had
to analyze their practices, curriculum, and how they spend their resources in order to
make adjustments so they can meet state and federal goals. By going through this
process, the academic programs within some schools have improved and helped to raise
the level of student academic achievement (Gross & Supovitz, 2005). Some believe for
schools to succeed, they need to have a process in place to analyze state and local
assessment data in order for them to improve instruction and make program changes
(Gross & Supovitz, 2005). As a result of its documented effectiveness, analyzing student
assessment data has become a central practice for schools which work to raise the
academic achievement of their students (Black & William, 1998). Principals and
teachers are expected to make data-driven decisions that will change the educational
programs in schools and the instruction within the classrooms. However, steps needs to
be taken in order for a school to undergo successful reform regarding the use of data.
First, establishing a culture of data use is significant for schools that wish to
become data-driven (Datnow et al., 2007). Changing a school’s culture is a large and
demanding task. Copland (2003) cites Sarason (1971) who explains: “changing a school
culture is difficult work and must be done in a comprehensive way if it is to be effective
5
and or have lasting significance” (p. 377). Second, principals need help create structures
for teachers whereby time is allotted for collaborative data analysis, training is provided
in order to make data usable, and resources are given to support the process (Datnow,
Park, & Wohlstetter, 2007; Jandris, 2001). In other words, the structure and culture both
need to be changed to support data-driven decision making.
Many schools, however, face leadership and staff-capacity barriers which can
prevent them from becoming data-driven. In many cases, teachers and administrators
have an overwhelming amount of data about their students which they can access, but
they do not have the training to use it (Earl & Fullan, 2003; Lashway, 2002). Often, the
building of capacity for principals and teachers in data-driven decision making has not
occurred, leaving teachers and principals without direction (Lashway, 2002). In their
study, Earl and Fullan (2003) noted that principals had anxiety and felt ill-prepared to
make school reform decisions when they were based on using data. The lack of training
and confidence in making decisions using data in a time of accountability puts schools at
a distinct disadvantage. In sum, schools need to focus on building the capacity of
teachers and principals in using data before they can focus on building a school culture
and structure that supports data-driven decision making.
There have been studies at the elementary and high school level documenting
strategies and supports for data driven decision making (e.g., Datnow et al., 2007;
Datnow, Park, & Kennedy, 2008; Marsh et al., 2006). However, there has been a dearth
of research on data-driven decision making in middle schools. Meanwhile, middle school
education remains an area of significant concern, Twenty years ago, landmark studies
6
such as the California Department of Education’s Caught in the Middle (1987) and the
Carnegie Council on Adolescent Development’s Turning Points (1989) established the
importance of adolescent education. Both reports document that the middle school years
are the last opportunity for students to prepare to enter high school with the skills
necessary to succeed. Although these reports led to some changes in middle school
organization, the need to improve middle school student achievement is still a pressing
issue. Reform at the middle school level is critically important, as research has indicated
that success in eighth grade is a crucial predictor of graduation from high school
(Rumberger, 1995). By conducting research into data-driven decision making at the
middle school level, it will help us better understand how to reach the goals of improving
adolescent education.
Research Questions
This qualitative, descriptive-analytic case study’s purpose is to document and
examine data-driven decision making in high achieving middle schools. Three schools
will be chosen in order to compare and contrast data-driven decision making at both sites.
Through the analysis of the schools’ practices, it is hoped that knowledge gained can be
presented for other middle school sites to use. This study will specifically answer the
following overarching research question:
How does a high performing middle school use data to inform decision
making?
In conducting this multiple-case study, the following sub-questions will be addressed:
7
1. What is the role of the middle school leader in supporting teachers to use data
to drive their instruction?
2. How do school structures and cultures facilitate or inhibit data-driven decision
making?
3. What types of data are collected by educators?
4. How are data used by leaders and teachers and for what purposes?
Significance of the study
This study will add to the limited but growing body of data-driven decision
making research on middle schools. Presently, most research has been centered on
elementary and high school use of data-driven decision making. This study, however,
will focus entirely on the effective use of data-driven decision making in middle schools.
When this study is done, it will provide educators with three case studies of
middle schools that have implemented the use of data-driven decision making. This
study will document the process used for data-driven decision making within the three
schools. Conducting this study will add to the available body of research and provide
practical information for middle schools who wish to implement data-driven decision
making.
8
CHAPTER TWO
Literature Review
Introduction
In recent years, there have been growing accountability demands placed on
schools to collect and analyze data in an effort to improve instruction and raise student
achievement (Kerr, Marsh, Ikemoto, Darlek, Barney, 2006). As a result of such policies,
data and accountability have grown to hold a central place in large-scale school reform
(Earl & Fullan, 2003). Schools today operate in an environment where they need
accurate and actionable information about what their students know in order for them to
plan student learning (Heritage & Yeagley, 2005). Schools now are beginning to
understand that data can be a powerful ally for educators who are looking to create
positive change in order to help increase student academic achievement (Lachat & Smith,
2005).
As stated before, schools now are operating in an environment where data are
playing a large part in the decision making process in relation to student achievement. To
build a foundation on how a high performing middle school uses data to inform decision
making, the following literature review will examine issues regarding the context,
building, and implementation of using data to drive decisions. The following areas will
be discussed:
1. Recent federal and state policies which have had an impact K-12 public
education.
9
2. The middle school reform movement and effective middle school data-driven
making practices.
3. An examination of leadership issues related to data, data culture, building
capacity of staff, and successful school practices using data.
To begin, it is important to understand the impact of federal and state policy on
public education. The following section will discuss how data-driven decision making
has risen from the implementation of government policy.
Impact of Federal and State Accountability Policies K-12 Public Education
Across the entire United States there has been a movement for states to adopt
rigorous academic standards and content (Goertz & Duffy, 2001). During the 1980s on
and 1990s, schools began to use data gathered from assessing their students’ knowledge
against federal, state, and local standards, to drive their decisions in school improvement
(Coburn, Honig, & Stein, 2005). According to Goertz and Duffy (2001), after states
adopted standards-based reforms, assessments were to be used to measure the
achievement of all students. By establishing state content standards, accountability
measure could now be focused on student outcomes.
According to Goertz and Duffy (2001), it was in the environment of the
Improving America’s Schools Act (IASA) of 1994 when many states began adopting
standards-based reforms. The purpose of the IASA was to empower states to create high-
quality content and performance standards, and to build a system of accountability for
schools that would measure their students’ performance against the content and
10
performance standards in an effort to raise achievement (Goertz & Duffy, 2001).
Disadvantaged Title I students were not to be treated differently and they were to be
measured against the same standards as the other students. The components of standards-
based reform include:
• Standards that spell out what students should know and be able to do;
• Assessments to measure progress toward those standards;
• Strategies for building the capacity of educators to help their students meet higher
expectations;
• Rewards when students and schools meet or exceed standards; and
• Clear consequences when they fail. (EdSource, 2008, p. 1)
In partial response to IASA, California passed the Public Schools Accountability Act
(PSAA) in 1999 (Bitter & O’Day, 2006). The PSAA created a system for measuring how
well a school performed academically and ranked that school against others. The PSAA
system had three main components:
1. Academic Performance Index (API) – used to quantify the academic
performance of students or groups of students in a school;
2. Immediate Intervention/Underperforming Schools Program (II/USP), - a
program that gives assistance to underperforming schools, with the threat of
state intervention if schools fail to improve;
3. Governor’s Performance Award Program – a program that rewards schools
which make significant gains in their API scores (EdSource, 2008, p. 1)
11
According to Bitter & O’Day (2006), the PSAA had the effect of focusing the public’s
and educators’ attention on the academic performance of their schools. Making public
schools’ academic performance public intended to be a motivating force for educators
and schools to work toward continued student academic improvement (Bitter & O’Day,
2006).
California created its current accountability system several years before the
federal No Child Left Behind (NCLB) act was passed into law in 2002. In order to
inform the public, NCLB legislation requires public schools who receive Title I funds to
make data available in three areas: assessment, accountability, and teacher quality
(Knapp, Swinnerton, Copland, Monpas-Huber, 2006). Along with making data available,
NCLB required schools to make “adequate yearly progress” (AYP) in meeting the goal of
proficiency on state standardized tests in reading and mathematic by the year 2014 (U.S.
Department of Education, 2002). More specifically, in order for schools to make their
AYP, they had to:
• Have a certain percentage of students meet “annual measurable
objectives” (AMOs) to be proficient on state standardized assessments;
• Attain specific graduation rates or make improvement in the graduation
rate;
• Test 95% of their students (EdSource, 2007)
If a Title I school fails to make its AYP for two consecutive years in English, math, API,
or graduation rate, the school enters into Program Improvement. However, if a school is
in Program Improvement and then meets its AYP in all areas for two consecutive years, it
12
will exit the program (EdSource, 2007). If schools fall into Program Improvement, they
face progressively worse consequences such as having to bus their students to attend
another public school, provide tutoring, replace school staff, and restructure the school
until they exit the program.
NCLB and the PSAA have created an environment that can require educational
leaders to utilize data in order to improve teaching and learning (Coburn & Talbert, 2006;
Knapp et al., 2006). Knapp et al. (2006) and his colleagues state that “these events make
it hard to ignore the need for data; at best, they represent an opportunity to use data to
strengthen the planning and execution of educational programs, as well as public support
for them” (p. 9). However, neither law tells educators how they should use data, nor do
they provide capacity building for it to occur (Knapp et al., 2006). Nevertheless, NCLB
and California’s PSAA have changed how public schools are held accountable and
pushed many schools to consider their data.
In the following section, the literature on middle school reform movement is
presented. The section focuses on the how landmark reports in the 1980s had the effect
changing early adolescent education. Later in the section, the standards reform
movement is discussed and how it has created a focus on student academic achievement.
Middle School Reform
The middle school reform movement began to receive national attention as a
result of the dissatisfaction of many with adolescent education in the 1980’s
(Springboard, 2007). A number of reports came out during the 1980’s that highlighted
13
the faults of education for adolescents in junior high schools. One such report was the
Carnegie Council on Adolescent Development’s (CCAD) landmark work entitled
Turning Points which was published in 1989. The CCAD (1989) report highlighted why
many were dissatisfied with education during the middle years, it stated “Most young
adolescents attend massive, impersonal schools, learn from unconnected and seemingly
irrelevant curricula” (CCAD, 1989, p.13). In order to remedy these problems, the CCAD
(1989) report made recommendations that called upon middles schools to address
students’ physical, social, and emotional needs. Additionally, it was recommended that
schools “transmit a core of common, substantial knowledge to all students in ways that
foster curiosity, problem solving, and critical thinking” (CCAD, 1989, p.13). A second
significant report to call for change in junior high school education was the California
Department of Education’s (CDE) Caught in the Middle (1987). This influential CDE
report echoed the CCAD report when it stated that junior high schools which children
were attending were not meeting the needs of adolescents (Springboard, 2007). The CDE
and CCAD reports along with others helped to define the purpose of middle schools
(Clark & Clark, 1993). As a result of this, many schools underwent reform efforts to
transition from the junior high model to the middle school model in order to better help
educate students and provide for their social and emotional needs (Springboard, 2007).
Some of the structural changes from the junior high model to the middle school model
which took place were:
• A shift from grades 7-8 or 7-9 junior high school to a grades 6-8 middle
school structure;
14
• The introduction of the “advisory” period as a way of giving students an
adult mentor in school;
• An emphasis on “coring,” or creating interdisciplinary classrooms and
reducing the number of different teachers students experience in a day;
and
• The introduction of a broad spectrum of “exploratory” classes and
experiences. (Springboard, 2007, p.3)
In order to determine the effectiveness of middle school reform, a meta-analysis
of research was conducted by Flowers, Mertens, and Mulhall (2003). They examined
over 10 years of data on middle schools. The research they examined was based on: how
middle schools improve, what impact the improvements have on teachers and students,
and how outcomes can be replicated in other schools. Flowers et al. (2003) found that
teams, which consisted of a small core group of teachers who shared students, and who
had a common planning time, produced students who demonstrated high levels of
achievement. Additionally, their research showed if teachers had a middle grade
certification and were part of an interdisciplinary team with common planning time, it
would result in the highest levels of best team and classroom practices (Flowers et al.,
2003). Middle school reform has also benefited students in poverty, Flower et al. (2003)
found:
The combined effect of teaming with common planning time, length of time
teaming, and high levels of classroom practices had a collective impact on student
achievement in high poverty schools (p. 58).
15
The research from Flower et al. (2003) over the last 10 years documents that students
have benefited from districts implementing middle school reforms. According to the
research by Flower et al. (2003) the change from the junior high model to the middle
school model has provided young adolescent students with an environment to better suit
their needs.
The middle school reform movement was to create schools that were meeting the
physical, social, and emotional needs of adolescents; however, they still were not
providing a rigorous academic education (Springboard, 2007). The standards-based and
accountability reform movement came about, in part, to help address these weaknesses in
public education (Taking Center Stage, 2001). The reform movement’s intent was to
hold students to high common standards for academic performance along with the
schools and the people who work in them (Elmore, 2002). By doing this, it would ensure
that all students would be able to achieve those standards. The standards-based and
accountability movement was successful in creating more academic rigor for middle
schools and in the establishment of accountability measures for all middle schools in
California and virtually all over the United States (Taking Center Stage, 2001).
However, studies have also been done that question whether middle schools are
succeeding in their mission to address the social-emotional needs of students and whether
academic achievement has increased.
Juvonen, Le, Kaganoff, Augustine, and Constant (2004) from RAND Education
conducted their own meta-analysis study that was focused on middle schools. The
research team carried out a comprehensive study which examined research relevant to
16
middle schools over 20 years. The study focused on eight areas such as: instructional and
organizational practices, academic achievement, conditions affecting students, teacher
qualification, among others. There were many conclusions drawn from Juvonen’s et al.,
(2004) study regarding the effectiveness of middle schools. It was determined that school
leadership, school culture, and the degree to which interventions are carried out impacted
how effective middle school practices were (Juvonen et al., 2004). While Flowers et al.
(2003) highlights the benefits of practices such as advisory period, interdisciplinary team
teaching, and flexible scheduling for middle schools, their effectiveness ultimately
depends on the factors that exist at the middle school where they are being implemented.
To further support their conclusions, Juvonen et al., (2004) examined students’ social-
motivation, by conducting an analysis of U.S. middle school students and comparing
them to students from 11 other countries. It was found that middle school students from
the U.S. have higher levels of emotional and physical problems then their counterparts
from the other 11 countries. In addition to this, U.S. middle school students viewed their
school climate as being more negative than the students from the other countries. If
middle schools are to be successful in raising student achievement and addressing the
social-emotional health of their students, the factors of leadership, school culture, and the
level to which interventions are carried out must be addressed.
Merits of Data Use
Research studies have cited how effective middle schools use data to improve
student achievement. One such study was conducted by Symonds (2004). Her research
17
focused on the school-level policies and strategies of a middle school. Her team
conducted a case study at Roosevelt Middle School in Oakland, California. She found
that Roosevelt: (1) used frequent and reliable data; (2) had an infrastructure to support
teachers in data use; and (3) used data to examine the effectiveness of school policies. To
gather achievement data, Roosevelt administers two assessments biannually and
additionally uses STAR testing data. Teachers analyze the data from the assessments to
determine areas of weakness that need to be addressed accordingly (Symonds, 2004).
Additionally, Roosevelt provided collaboration time for teachers to meet to discuss data
every Wednesday in either staff or department level meetings. One teacher commented,
Everybody is given the data, and we take our time and look at it. We do a lot of
that. We throw ideas up on the table….Before, there’s been criticism that “Well,
so we’ve got all this data, what are we going to do differently?” And I think that’s
what’s happening now; we’re able to do more planning. (Symonds, 2004, p. 37)
Symonds (2004) also found that Roosevelt used data to improve student
achievement by analyzing student discipline data. Roosevelt determined that African-
Americans were being suspended at higher levels than other student groups. Professional
development was given to teachers to address discipline. This resulted in a drop in
African-American students being suspended and an increase in their attendance in class
and a rise API scores.
In another study conducted by Abbeit et al. (2007), they examined the best
practices of three middle schools which were considered to be high-achieving. For a
school to be identified as a “high-performer”, it had to meet its AYP requirements for
three consecutive years, which were between 2003-2005, and meet certain AYP
18
proficiency requirements in English Language Arts (ELA) and Mathematics (Abbeit et
al., 2007). In her study, Abbeit et al, (2007) found that high-performing schools would:
• Analyze data from CST, EduSoft, Benchmark and Teacher Assessments,
• Plan differentiated assignments based on assessments,
• Analyze embedded assessment results and adjust instructional planning as needed,
• Have a inquiry process in place,
• Use data in goal setting, and
• Conduct professional development
Through analysis of this study, it can be concluded that having a system in place that
supports educators in analyzing student data can have positive effects on academic
performance. Both studies cite the importance of having data available to analyze and
also a support structure in place to help facilitate the use of data.
In the following section school site leadership is discussed. In particular, the
decision making environment for leaders is considered, and later the qualities and skills a
school site leader must possess are examined.
School Site Leadership and Data-Driven Decision Making
There was a time in education when decisions were based on the best judgments
of the people in authority. It was assumed that school and district leaders, as
professionals in the field, had both the responsibility and the right to make
decisions about students, schools and even about education more broadly. They
did so using a combination of intimate knowledge of the context, political savvy,
experience and logical analysis. Data played almost no part in decisions. Instead,
leaders relied on their tacit knowledge to formulate and execute plans. In the past
several decades, a great deal has changed. Accountability has become the
watchword of education and data hold a central place in the current wave of large-
scale reform. (Earl & Fullan, 2003, p.383)
19
As noted by Earl and Fullan (2003) above, the environment in education has been
rapidly changing from one where leaders use their instincts to make decisions to one
where accountability now requires leadership to use data to drive decisions. Educational
leaders find themselves in situations where their judgments must be made in
consideration to the evidence that they have to support them (Earl & Katz, 2002).
According to research literature, educational leaders need to have the necessary skills and
qualities to lead a data driven school. Schmoker (2005) argues that educational leaders
are not people who simply manipulate data in prescribed ways; instead, they are people
who need to gather and analyze evidence in ways to deepen their understanding of it. He
argues that an effective data-literate leader:
• Thinks about purpose. Often, decisions are made without having data
available. The data-literate leader gathers data and understands what the
data is being used for and for what purpose.
• Recognizes sound and unsound data. A leader is able to interpret data and
determine its quality to make the proper decision.
• Is knowledgeable about statistical and measurement concepts. Leaders
need to understand measurement concepts to be able to read into student
achievement data to make decisions.
• Makes interpretation paramount. This gives the leader the opportunity to
provide different possibilities, pose questions, and find flaws of what the
data could represent.
20
• Pays attention to reporting and to audiences. School leaders need to be
aware of who their audience is and how to report data to them. Without
this ability, the message the leader is trying to get across to the audience
will be overlooked.
The research conducted by Earl and Katz (2002) supports much of what
Schmoker (2005) has found to be true. Earl and Katz (2002) note that school leaders
must have the skills to be able to interpret and use data effectively. They feel that to be
an effective school leader in a data-driven environment there are three requirements of
them. First, a school leader needs to develop an inquiry habit of mind. This occurs when
a school leader develops a mindset that they view data and evidence as a means to help
further understand a problem. An inquiry habit of mind leader is reflective and always
seeks out information to base their decisions on. Second, a school leader needs to
become data literate in order to interpret and use data. Third, a school leader must be
able to create a culture of inquiry within the school community. Without buy-in and
collaboration from the staff, the leader will have little chance of succeeding in building a
data-driven school.
Research literature indicates that strong leadership is needed in order to create
data-driven schools (O’Day, Bitter, Kirst, Carnoy, Woody, & Buttles, 2004; Choppin,
2002, Supovitz & Klein, 2003). In a research study conducted jointly by Policy Analysis
for California Education (PACE), Consortium for Policy Research in Education (CPRE),
and American Institutes for Research (AIR), it was concluded that “Schools that studied
data seriously, on the other hand, typically had a strong principal who helped the faculty
21
draw valuable information from it” (O’Day et al., 2004, p.9). This finding was supported
in a research study conducted by Choppin (2002). In his study on six Milwaukee public
schools, he examined the extent of which the schools implemented the use of data in their
decisions. He found the two most successful schools to use data had strong leadership
from the site administration which was able to encourage the staff to use data on a
consistent basis. Finally, Lachat and Smith (2005) came to a similar conclusion in
regards to the importance of leadership and data use, they note that if school leaders are
to have continuous improvement through the use of data, then they must view and
champion the use of data.
Studies have continually highlighted the crucial role the principal plays in leading
the school in becoming data-driven to increase student academic achievement. In
Symonds’ (2004) study of schools which have closed the achievement gap, she found that
effective leaders of schools created a systematic inquiry process to analyze data to better
understand why achievement gaps existed. Furthermore, Symonds (2004) found that
principals would facilitate the creation of measurable goals for schools in order for the
schools to gauge how well they were closing the achievement gap. Additionally,
Togneri’s and Anderson’s (2003) research study of five school districts across the
country found that principals were instrumental in leading instructional improvement.
Their research showed that principals in the districts being studied practiced regular
analysis of student performance data in order to improve instruction.
As cited in the above research literature, having strong leadership that is a driving
force behind data-driven decision making at a school is essential if the practice is to be
22
implemented (Supovitz & Klein, 2003). In many of the studies where leadership was
weak, or support for use of data was not encouraged, the implementation of data to
inform decisions fell by the wayside (Supovitz & Klein, 2003).
In the following section, the research literature focuses on the importance of
building a school culture around data use. Also, discussed are the different kinds of
barriers that may prevent a data use culture from developing within a school.
Challenges of Creating a Data-Use School Culture
Research literature on data emphasizes that creating a culture around the use of
data is necessary if it is to become a central component in helping to improve student
achievement (Heritage & Yeagley, 2005; Earl & Katz, 2005; Datnow et al., 2007).
Ingram, Louis, & Schroeder (2004) note “culture exerts a powerful influence on the way
decisions are made” (p.1280). Depending on the type of culture established within a
school it can be beneficial or detrimental to data use. For example, in a qualitative study
conducted by Datnow’s et al. (2007) , she found that of four school systems that made
positive gains in student academic achievement all had a school culture which supported
data use. However, there are many challenges that can prevent a data culture from
developing within schools.
According to Ingram et al. (2004), barriers must be overcome if a supportive
culture of data-driven decision making is to be established. They identify three areas
which need to be addressed in order to create a data culture: cultural challenges, political
23
challenges and technical challenges. They specifically identify cultural barriers to data
use:
• Teachers have their own personal metric for judging their own effectiveness;
• Personal experience, intuition, and anecdotal information is relied upon for
decision making;
• Lack on consensus on the importance of student outcomes and what is
meaningful data;
• Teachers disassociate their performance from that of their students’.
They identify technical barriers as:
• Lack of relevant data for teacher use;
• Lack of time for data collection and analysis.
Finally, the political barrier as:
• Data being used political, creating mistrust among teachers and data
avoidance.
Through examination of the research literature, other additional barriers have been
identified which prevents data use in schools. Kerr’s et al. (2006) study of three school
districts found there was a large decline in teachers’ interest to using data in the
classroom when teachers did not have access to it in a timely manner. Additionally, they
found that data were more likely to be used by teachers if it were deemed to come from
valid assessments. For example, as a result of teachers and principals believing state
assessment data were less useful for analyzing student performance and driving
instruction in the classroom, they chose not to make use of them. Kerr et al. (2006)
24
notes, that teachers were also reluctant to use data that came from state assessments
because they felt the tests were not aligned with daily instruction and limited in content
coverage.
In a case study conducted by Choppin (2002) on six Milwaukee Public Schools,
he found data use among in four schools dropped for various reasons. Choppin (2002)
found that regular usage of data to inform teachers’ decisions declined as a result of
technical capacity, district data management policies, and access to reliable data. He did
find, however, the two schools that had teachers who used data effectively had: 1) strong
leadership from the principals; 2) invested time and effort into data analysis; and, 3) were
in their second year of the project (Choppin, 2002). Without the capacity, reliable data
and sound data management polices, data use had not been institutionalized and become
part of the culture of the studied schools. In the next section, research studies on the
building of capacity for data-driven decision making will be reviewed.
Building Capacity for Data-Driven Decision Making
One of my weaknesses has always been documenting a student’s progress,
because I always found it such an overwhelming task. I would assess students,
hand in the scores to an administrator, and then file them away. I literally would
assess here and there, never use the results, and concentrate on whole-group
instruction. Individual needs based on assessment were never taken into
consideration. (Calderon [a kindergarten teacher]. cited in Moktari et al., 2007, p.
254)
As the above quote implies, there is a lack of capacity among some teachers and
principals that prevents them from effectively using data to improve instruction (Sharkey
& Murname, 2006; Kerr et al. 2006; Earl & Fullan, 2003; Earl & Katz 2002; Symonds,
25
2004; Wayman & Stringfield,, 2006; Supovitz & Klein, 2003). Research literature states
that professional development, which builds the capacity of teachers, has become
increasingly important in the current environment of school accountability (Elmore,
2002). Wayman and Stringfield (2006) state: Faced with increasing accountability
requirements, schools and school systems are implementing a variety of methods for
gathering, analyzing, and reporting all of the data, but they are moving forward with
strikingly little guidance from any quarter (p. 464).
A great need for professional development for administrators and teachers, along
with other forms of support to assist them in moving forward is apparent (Slavin, 2002).
However, professional staff development in education has tended not to focus on
worthwhile productive strategies to improve instructional strategies with follow-up
training (Schmoker, 2004). The level of human capacity and support with data-driven
decision making at a school impacts how effective and how often it is used (Marsh, et al.,
2004).
A school system needs to implement focused professional development and offer
support to be effective in using data to drive decisions. In a research study by Datnow et
al., (2007), she and her colleagues found that school systems which had success in raising
student achievement focused on four key strategies in regards to building teacher
capacity. The strategies they observed were:
1. Investing in professional development
2. Providing support for staff in how to use data and modeling use and data
discussions
26
3. Providing time for teacher collaboration
4. Connecting educators across schools to share data and improvement strategies
In another research study conducted by Symonds (2004), she analyzed 32 K-8
schools in the San Francisco Bay Area to determine how schools closed the achievement
gap. She found teachers at schools where the achievement gap closed were more likely
to have: an understanding of how to use data to close the skill gaps of low achieving
students; receive professional development on analyzing low-performing student data;
receive professional development on linking low-performing student data to instructional
strategies and; have leaders that encourage or lead systematic inquiry into gaps.
Assessments and the data gathered from them have the potential for improving the
academic achievement of students as long as the efforts of teachers to use data to guide
instruction are supported (Jandris, 2001, Supovitz & Klein 2003). According to Jandris
(2001), he believes principals need to provide the following for schools and teachers to
support the use of data:
• the time to practice their craft;
• the training to make it usable;
• the resources to support it; and
• a school culture that fosters quality assessment practices.
In explaining the above points, Jandris (2001) argues that teachers need time to
meet and collaborate in order to discuss data and develop “solutions”. Along with this,
he states teachers need an easy to use system that allows them to use assessment data to
create and access easily readable reports. Furthermore, he believes teachers need to be
27
trained in assessment literacy. By having training for teachers in assessment literacy,
Jandris (2001) argues that it will help teachers understand how to read and interpret test
results and make informed decisions. Finally, he believes a school culture needs to be
developed around a continuous and reflective use of data which produces school
improvement.
The task of providing professional development in order to build capacity so that
educators can fully implement data-driven decisions take a large investment in human
and economic capital (Heritage & Yeagley, 2005). Unfortunately, many districts and
schools have inadequate human and financial resources to provide training and support,
and this is particularly true in urban areas (Kerr et al. 2006).
To summarize, teachers and principals have a difficult task in taking large
amounts of data and turning this information into educational practice that improves
student achievement, especially when few educators have the background or experience
in doing so (Wayman, Midgley, Stringfield, 2006); therefore, the building of capacity on
data-driven decision making is very important.
In the following section, the types of data used to guide decisions will be
discussed.
Types of Data Used In Schools
In many instances, schools are overwhelmed by the amount data they have, so
much so that choosing the right kind of data can be a problem (Datnow et al., 2007;
March, Pane, Hamilton, 2006; Ingram et al., 2004) When data are collected, they are
28
generally organized into three categories which are student demographics, school and
educational program data, and performance data (Lachat & Smith, 2005). A problem
with data, however, is that after they have been gathered, the interpretation of them may
be subject to dispute by the different individuals who analyze it (Knapp et al., 2006). An
additional problem highlighted by Ingram et al. (2004), was that local achievement
measures, such as course grades and classroom assessments are seen by teachers as better
indicators rather than standardized achievement tests. Earl and Katz (2002) define data
as “summaries that result from collection of information through systematic measurement
or observation or analysis, about some phenomenon of interest, using quantitative and/or
qualitative methods” (p. 5). By collecting and analyzing data, they can help define a
problem if it exists and also determine what possible solutions could exist (Spillane &
Miele, 2007).
According to Heritage & Yeagley (2005), there are four main types of data that
can be used for school improvement. They are large scale achievement tests, benchmark
assessments, formative assessments and grading. Heritage and Yeagley (2005) describe
them as:
• Large-Scale Achievement Tests
They note that large-scale achievement tests demonstrate how well students perform
with regard to the state standards. Data from these tests provide information to the
public which show the effectiveness of the school’s academic program. The
weakness of this data is that it is usually collected once a year, and it covers a large
amount of material, which can impact its effectiveness.
29
• Benchmark Assessments
Provide data multiple times throughout the year. The tests are mostly aligned with
state standards and provide predictive feedback on how well students may do on
larger state standards tests. Teachers and administrators can use data to help guide
instruction.
• Formative assessment
Formative assessments are used by teachers to gather evidence about what students
have learned. Using this data gathered from formative assessments, teachers can
adjust their instruction based upon their students’ needs to improve achievement
(Black, 1998, Heritage and Yeagley, 2005; Schmoker, 2003). In Black’s (1998)
extensive research review of 160 journals, he found that formative assessment
produced significant and at times, substantial learning gains.
• Grading
Grading is the most common form of measurement used in classrooms to gauge
students’ understanding of the subject. However, researchers have found that grades
are subjective forms of measurement and thus are potentially invalid forms of
measurement for student achievement (Heritage & Yeagley, 2005; Marzano, 2000).
Heritage and Yeagley (2005) point to factors such as behavior and attendance,
weighted scores, and a single grade to represent a student’s performance for multiple
skills.
• Going Beyond Assessment Data
30
Other forms of valuable data can be used to help understand what may impact student
performance. Other forms of data that can be used are demographic, perception,
behavior, and attendance. These data can help further the understanding of what is
having an effect on students’ academic achievement.
In a study conducted by Supovitz and Klein (2003), they classify three main types
of student performance data used in schools, and they are: external data, individual
teacher assessment data, and school-wide data. More specifically, they define external
data as state and district test results; individual teacher assessment is assessment data that
are used within the classroom of individual teachers; school-wide assessments are
defined as assessments that are not only administered systematically across groups of
students within a school, but whose results are aggregated and systematically analyzed
for patterns that are then used to guide school and individual teacher decision-making.
To summarize, multiple types of data must be gathered in order to make decisions
that drive school improvement. When data is gathered from several different areas, such
as Supovitz’s and Klein’s (2003) external data, school-wide assessments, and individual
teacher assessments, cross-referencing of the data can occur in order to support decisions
which are made. In the following section, research literature on the processes that
support data use is discussed.
Processes that Support Data Use
Studies have been conducted in order to identify and analyze the processes for
using student performance data to inform decision making. In a CPRE study, conducted
31
by Supovitz and Klein (2003), they analyzed schools which had implemented America’s
Choice comprehensive school reform model. More specifically, they analyzed schools
that used a variety of student data to improve teacher instruction and how the school as an
organization supported instructional improvement. In their study, they found the most
effective schools: used data systematically in their decision making process and devoted
a large amount of time to use data. According to Supovitz and Klein (2003), a school
that practiced gathering data equally from external assessments, school-wide
assessments, and individual teacher assessments, would be able to use it to improve
academic achievement and support continual school improvement.
In order for data to be of practical use for teachers and principals, data feedback
systems, which rely on data from multiple sources, must be in place (Supovitz & Klein,
2003). By having data feedback systems in place at a school site, it helps teachers to be
able to gauge their students’ learning and to make instructional adjustments if necessary
(Schmoker, 2003).
One system that is used by schools to implement data-driven decision making is
called a cycle of inquiry. In Copland’s (2003) longitudinal study of 16 schools that took
part in the Bay Area School Reform Collaborative (BASRC), he examined distributed
leadership, how schools used continual inquiry in changing their practice, and collective
decision-making at the schools. Copeland (2003) defined the cycle of inquiry as a
process to “help schools pose, investigate, and respond to questions about policies and
practices” (p.380). According to Copeland (2003), there are six steps that need to be
followed in the cycle of inquiry (See Figure 1). The six steps are:
32
1. Identify problems and area of academic focus based on data;
2. Refine the focused effort;
3. Identify measurable goals for school, grade levels, and/or department;
4. Build a concrete work plan both schoolwide, and at grade levels and/or
departments;
5. Take action; and
6. Reflect on and analyze results from the data.
By going through the six steps in the BASRC’s cycle of inquiry, schools are informed of
whether their efforts to use data to resolve problems have been effective or not
(Copeland, 2003).
Figure 1: Diagram of BASRC Cycle of Inquiry
33
Copland (2003) found that schools which became proficient at using the inquiry
process understood that it was fundamental in helping students achieve. More over,
teachers turned to their own professional community as a resource in order to change
their teaching practices. For instance, Copeland (2003) found at the classroom level,
teachers who were using the process of inquiry, were able to identify effective teaching
practices used with students who performed well on the school’s literacy performance
standards. These teachers then shared their strategies with the entire school and helped to
improve teaching practices and student achievement in literacy.
As the research indicated, having established processes in using data are
beneficial to the school and students. As Copland’s (2003) research had shown,
developing a cycle of inquiry allows for continuous improvement to occur within a
school. In the following section, a summary of the literature will be presented.
Summary of the Literature Review
The importance of the use of data has increased dramatically with the advent of
NCLB. Public schools who receive federal funding are now required to use data in order
to create school improvement (Coburn & Talbert, 2006). As more research is conducted
on data-driven schools, it is becoming more apparent that data can be a powerful tool in
guiding instruction in order to raise student academic achievement (Lachat & Smith,
2005; Copland 2003; Datnow et al., 2007; Jandris, 2001). However, while schools are
faced with increasingly higher levels of accountability, and are required to use evidence
34
to help in school improvement, they are surprisingly given little direction on how to
accomplish this (Wayman & Stringfield, 2006).
In conjunction with the lack of direction in data-driven decision making, schools
face barriers in creating a data-use culture which need to be overcome if they are to
become “data-driven.” Ikemoto and Marsh (2007) note, that school culture is a
determining factor that distinguishes schools which implement data-driven decision
making at a high level and those who do not. As the research indicated, it is important to
have strong leadership to help create a school culture that uses data for continuous
improvement. It is important for leadership to build the capacity of their staff and to
establish processes for data use. Without this occurring, a data culture can not
materialize within the school.
The purpose of this study is to determine how middle schools effectively implement data-
driven decision making. By conducting this study, it will add to the literature about data
use in middle schools with the hope of benefiting educational leaders pursuing a data-
driven school to benefit students. Schools are now beginning to realize that data can be
the bedrock for which to create a course of action for school improvement (Supovtiz &
Klein, 2003). When conducting this study, the following factors that delineate middle
schools from elementary and high schools in regards to data-driven decision making will
be considered:
• structure and organization
• culture
• personnel
35
• teacher interaction
• leadership
• student social-emotional support
• student academic support
36
CHAPTER THREE
Methodology
Introduction
This chapter describes the design, sample, instrumentation, data collection, and
analysis of the proposed study. The purpose of this study is to examine the processes and
practices of middle schools that use data to drive their decisions in order to improve
student achievement. Particular attention is focused on the role the school administration
plays in support of data use. Additionally, teachers’ responses to questions were
examined in order to understand how they use data to inform their decisions about
instruction. Three high achieving middle schools in southern California were studied in
an effort to answer the following four research sub-questions:
1. What is the role of the middle school leader in supporting teachers to use data
to drive their instruction?
2. How do school structures and cultures facilitate or inhibit data driven decision
making?
3. What types of data are collected by educators?
4. How are data used by leaders and teachers and for what purposes?
Research Design
In order to answer the research questions of this study, a qualitative, descriptive-
analytic case study method was used. Case study research method was appropriate
37
because it provided an in-depth description about the process of how three middle
schools implemented data-driven decision making. Merriam (1998) argues that case
study design allows the focus to be on the “process” rather than the outcomes. In
addition, Patton (2002) comments on case studies stating, “The purpose is to gather
comprehensive, systematic, and in-depth information about each case of interest” (p.
447). By studying high achieving middle schools, I wanted to document the practices
that make them data-driven and help them to successfully make their AYP year after
year. Merriam (1998) notes that case studies can be used to investigate and gain
knowledge about a phenomenon. In my study, I examined three schools which can be
considered unique when compared to schools which are similar in socioeconomics and
demographics.
Qualitative inquiry methods were used in order to explore data use in middle
schools to inform their decision making. Inductive analysis was conducted using data
from observations, semi-structured interviews, and documents gathered at sites. From the
data gathered, analysis was done in order to identify any general patterns. This approach
then led to theory generation about data-driven decision making related to student
academic achievement.
The organizational unit of this study was the middle school. During the course of
the study, the focus of data gathering was on the personnel of each school. More
specifically, within the sites themselves, the subjects of analysis were primarily principals
and teachers.
38
Sample and Population
This study focused on middle schools from the Orange Valley Unified School
District (OVUSD) in Southern California. A colleague made the recommendation to
conduct the study within OVUSD. Upon further investigation, I did select OVUSD as
the district to conduct the study in as a result of it being a unified district of medium size
and for meeting its district AYP in each of the preceding school years (2005-2006
through 2007-2008). The school district has eight middle schools within its boundaries.
Of the eight schools, three were purposefully selected for this study based upon a specific
criterion. First, schools were chosen on the basis of making their API and AYP in each
of the three preceding school years (2005-2006 through 2007-2008). Six of the eight
schools met this requirement as determined by a search of STAR testing data on the
California Department of Education’s website. Second, the principal at the school site
had to attribute data use as playing a role in making their API and AYP. Finally, it was
determined through interviews of the principals that data use was indeed being
implemented at the school site. Three principals out of the six whose schools met the
criterion of the study volunteered to have their sites participate. By using purposeful
sampling, it allowed me to gather pertinent information related to how middle schools use
data to inform their decisions.
Interviews of teachers and principals were conducted at each school site. The
principals at each of the three school sites agreed to be interviewed for this study.
Additionally, the principals solicited teachers to volunteer to participate in this study.
This resulted in a range of teachers who taught different subjects and grades ranging from
39
sixth to eighth grade. By interviewing principals and teachers at three school sites from
different grade levels and subjects, it helped to give a broad perspective on data use and
support at the middle school level as opposed to a study conducted at one school. At
each of the three school sites four teachers and one principal volunteered for a total of 15
interviews.
Overview of District and Schools
All three of the schools selected for the study are located in Southern California.
Burbank, Harrison, and Lincoln Middle School are all a part of the Orange Valley
Unified School District.
1
The Orange Valley Unified School District has 30 schools
which serve approximately 25,000 students in kindergarten through twelfth grade. The
tables below contain information about Burbank, Harrison, and Lincoln’s students, staff,
and academic performance.
Table 1: Student Demographics – Burbank Middle School
Total Enrollment 754
Hispanic 10.21%
Asian 42.18%
White 37.93%
Filipino 2.52%
African American 2.52%
Pacific Islander 0.27%
American Indian or Alaskan Native 0.80%
Multiple or No Response 3.58%
English Learners 11%
Socioeconomically Disadvantaged 8%
Students with Disabilities 8%
Source: California Department of Education – http://www.cde.ca.gov
1
Pseudonyms are used for the purpose of confidentiality.
40
Table 2: Accountability Profile – Burbank Middle School
2005-2006 2006-2007 2007-2008
Academic Performance Index (API) 858 865 890
Statewide API Rank 10 10 10
Similar Schools API Rank 4 2 3
Made Adequate Yearly Progress (AYP) Yes Yes Yes
Program Improvement School No No No
Source: 2007-2008 School Accountability Report Card – Burbank Middle School/
California Department of Education – http://www.cde.gov
Table 3: Student Demographics – Harrison Middle School
Total Enrollment 661
Hispanic 9.98%
Asian 29.80%
White 51.59%
Filipino 3.03%
African American 1.36%
Pacific Islander .45%
American Indian or Alaskan Native 1.21%
Multiple or No Response 2.57%
English Learners 7%
Socioeconomically Disadvantaged 9%
Students with Disabilities 7%
Source: California Department of Education – http://cde.ca.gov
Table 4: Accountability Profile – Harrison Middle School
2005-2006 2006-2007 2007-2008
Academic Performance Index (API) 872 857 877
Statewide API Rank 10 10 9
Similar Schools API Rank 4 3 3
Made Adequate Yearly Progress (AYP) Yes Yes Yes
Program Improvement School No No No
Source: 2007-2008 School Accountability Report Card – Harrison Middle School/
California Department of Education – http://www.cde.gov
41
Table 5: Student Demographics – Lincoln Middle School
Total Enrollment 698
Hispanic 23.64%
Asian 40.97%
White 16.62%
Filipino 4.15%
African American 2.15%
Pacific Islander .72%
American Indian or Alaskan Native 0%
Multiple or No Response 11.75%
English Learners 9%
Socioeconomically Disadvantaged 19%
Students with Disabilities 6%
Source: California Department of Education – http://cde.ca.gov
Table 6: Accountability Profile – Lincoln Middle School
2005-2006 2006-2007 2007-2008
Academic Performance Index (API) 820 840 856
Statewide API Rank 9 9 9
Similar Schools API Rank 6 1 2
Made Adequate Yearly Progress (AYP) Yes Yes Yes
Program Improvement School No No No
Source: 2007-2008 School Accountability Report Card – Lincoln Middle School/
California Department of Education – http://www.cde.gov
The principals of the schools and the majority of teachers interviewed attributed data use
in part to helping them achieve the AYP and API listed in the tables. The data above also
indicates that each middle school has achieved at high levels for three consecutive years.
The information also reveals that the schools’ student demographics vary from school to
school. In comparing each of the schools’ demographics, Lincoln has the largest
population of Hispanic students at 23% and the lowest population of white students at
16%. Burbank Middle School, on the other hand, has the largest population of Asian
42
students at 42%, while Harrison has the largest population of white students at 51%.
However, each of the three schools have student populations that are similar in size.
Data Collection Procedures
As noted above, interviews were the primary method for collecting data. Merriam
(1998) notes “Interviewing is probably the most common form of data collection in
qualitative studies in education” (p.70). The gathering of qualitative data began with the
collection of information from semi-structured interviews of three site administrators.
The interviews focused on how the administrator first began creating a data culture; how
the administrator supports data use; and what are the types of data he/she uses in decision
making. Data was also gathered by conducting teacher interviews. The focus of the
interview of teachers was on how data use was implemented at the school site; how
teachers use data in their classrooms; and how administration supports teachers in the use
of data. Patton (2002) states, “We interview people to find out from them those things
we cannot directly observe” (p. 340). The protocols for all interviews are included in
Appendices A and B. Interviews with teachers were approximately 35 minutes in length
and took place in a classroom or conference room. Interviews with the principals were on
the average longer and lasted for approximately 45 minutes. The principal interviews
took place in their offices and a conference room. After interviews were completed, they
were transcribed verbatim by a professional service. In all, the transcriptions of the
interviews from twelve teachers and three principals amounted to over 200 pages of
interview data from a total of 15 interviews.
43
Finally, documents and artifacts were collected as an additional source of data for
this study. Patton (2002) refers to documents and artifacts as being a “particular rich
source of information” (p. 293). Documents and artifacts that were collected were forms
used to gather data, data reports, standardized testing data, and anything else deemed
pertinent to the study. The documents and artifacts that were gathered were from
administrators and teachers. Standardized testing data were also gathered from state
reporting websites. Selected documents and artifacts are provided in Appendix C.
Data Analysis Procedures
Interviews
Data gathered from the person-to-person interviews of teachers and principals were
transcribed verbatim for data analysis. Patton (2002, p. 380) refers to the analysis of
interview data as “making sense out of what people have said, looking for patterns,
putting together what is said in one place with what is said in another place.” Codes were
created to help classify and organize the data for analysis. They were developed
according to the research question and sub-questions of this study. In all, there were a
total of 13 codes used in the classification of research data. A list of all codes can be
found in Appendix D. By using codes to classify and organize information, different
themes in the research emerged. In order to code the research data, I used
HyperResearch coding software. Once all the data were coded, it allowed for me to
determine if the data gathered from individuals was reliable and valid. This was
accomplished by analyzing interview data for redundancy between multiple subjects. To
44
further help in organizing data, I also developed tables to compare findings across the
three school sites.
Documents and Artifacts
Patton (2002) comments that documents and artifacts can be a rich source of
information for the researcher. Documents and artifacts which related to data-driven
decision making were collected and examined to determine if there was consistency with
the gathered interview data or not. In addition to this, they were also indexed according
to the study’s research questions.
Limitations of the Study
As a result of limited financial funds and time, this study was small in scope. One
person conducted this study over a four month period of time. A more in-depth and
thorough study would have taken place if there were more time. In addition to a lack of
time to conduct this study, the number of schools involved in the study was small. With
only three middle school sites participating, fewer schools may find this study relevant.
The number of interviews conducted was small. At each of the three schools, only one
administrator and four teachers were interviewed. In addition to a small number of
teachers interviewed at the school sites, the sample of schools used in this study were all
located within one school district. By having all the schools in one district, it meant they
received the same type of support, so comparisons could not be made with schools in
other districts. Thus, the study was not broad in scope and comprehensive in nature. If
additional funds were made available, this study could have been carried out at a greater
45
number of schools across many districts for a longer period of time. As a result of the
short time frame and the small number of participating schools, this study may not be
generalized to all schools. However, despite this, it is hoped that the findings will be
relevant and transferable to other middle schools.
Researcher’s Subjectivity
As an administrator at a middle school, I may have brought certain biases when
conducting interviews. These biases may have been derived from my experiences with
data-driven decision making at my own school site. Having had both positive and
negative experiences in regards to working with teachers to implement data-driven
decision making, I was aware that they could impact the study. However, being
cognizant of this, I did make efforts to not allow any biases that I may have to influence
this study. The purpose for conducting this study was to produce a piece of research that
was not influenced by my subjectivity in an effort to aid middle school leaders in
bringing data-driven decision making to their school sites.
Ethical Considerations
This study followed the procedures set forth by the University of Southern California
and those written by the schools districts which the participating schools are a part of.
Additionally, this study underwent the IRB approval process before it took place to
ensure that proper protocol was followed in carrying out this research.
46
In order to protect the confidentiality and anonymity of participants and their schools,
pseudo names were used throughout the entire study. All participants were informed that
their anonymity would be protected before observations and interviews took place.
Additionally, participants were told of the purpose of the study before observations or
interviews were conducted so that they could determine if they wanted to participate or
not.
47
CHAPTER FOUR
Data Analysis and Interpretation of the Findings
Introduction
With the reauthorization of the Elementary and Secondary Education Act in 2002,
or commonly known as NCLB, schools which receive federal funds are expected to use
data to drive their decision making to raise student achievement. State and federal
policies require that schools be measured using the Academic Performance Index as well
as determining whether they have met their Adequate Yearly Progress. As a result of
schools trying to increase their API and make their AYP, a tremendous shift has occurred
within public schools to focus on using data-driven decision making to raise student
achievement. Thus, some middle schools and individual teachers have developed many
different data practices which are implemented school wide and within the classrooms.
Some of these middle schools have experienced success in raising their API scores and
making their AYP, and they attribute partial success to implementing data-driven
decision making at the school and classroom levels.
The findings in this chapter are based on analysis and interpretation of data
gathered from interviews and document analysis. The purpose of this study is to
understand how high performing middle schools use data to inform their decision
making. For this study, a total of fifteen interviews were conducted with principals and
teachers at three high performing middle schools. Various documents from the school
sites and teachers were reviewed and analyzed to inform this study.
48
Research Questions and Thematic Underpinnings
This chapter is organized to connect the thematic underpinnings revealed in the
data with the corresponding research questions. This chapter will present a data analysis
of each of the sub questions below and the major themes within each of these categories.
Research Question:
How does a high performing middle school use data to inform decision making?
Sub Questions and Themes
a) What is the role of the middle school leader in supporting teachers to use data to
drive their instruction?
1. Access to data
2. Expectations of data use
3. Support in use of data
b) How do school structures and cultures facilitate or inhibit data-driven decision
making?
1. Time for data discussions
2. School culture
c) What types of data are collected by educators?
1. Summative assessments
2. Benchmark assessments
3. Formative assessments
49
d) How are data used by leaders and teachers and for what purposes?
1. Student support and placement
2. Instructional changes
Before addressing these questions, I will first give an overview of the school contexts.
School Settings
Burbank, Harrison, and Lincoln Middle School are all at different stages in
becoming data-driven. Each school is unique in their features and approach to using
data, but they also shared commonalities. All three schools experiences with data-driven
decision making have provided valuable insights to inform this study.
The principal at Burbank Middle School is a strong proponent of data use in
schools. At the time of this study, the principal had been at the school for four years. He
is a proponent of data-driven decision making and believes strongly in its effectiveness.
In order to move his school forward in DDDM, he has been working closely with his
departments and having data discussions with them. One way the principal has helped to
facilitate discussions was by creating Professional Learning Communities or PLCs.
PLCs are groups of teachers who meet regularly to discuss their students and teaching
practices. He commented, “We’ve been working with PLCs and we’re in the third year
of PLCs, and the discussion of data is at various stages with each of the departments.”
Based on data from 2007-2008 school year, Burbank Middle School had 29 teachers on
staff. The average years of educational service for the staff was 11.2 years. The school’s
teaching staff is organized into departments based on subject areas. It was found that the
50
departments’ use of data varied. To highlight this, the principal identified the math
department as his most successful in utilizing data and his social studies department as
being the most challenged.
Harrison Middle School’s principal has been in place for a little over two years.
He was a former assistant principal at a school which had grades 6-12. According to the
principal, Harrison Middle School is located in a fairly high socioeconomic area. The
principal described the staff as being “pretty tight knit for the most part.” The school has
active parental involvement and the principal characterized his students as being
“successful.” The staff at Harrison clearly understood the importance the principal
placed on data use. Harrison’s data practices were at a more advanced stage than
Burbank’s and Lincoln’s. In the two years that the principal had been in place, he had all
of his departments create quarterly benchmark assessments to give to students.
Furthermore, it was reported by Harrison’s principal that each grade level team had to
create their own annual goals which were based on data. For instance, one goal for
seventh grade math teachers was to increase the percentage of seventh grade students
correctly answering multi-step problems, graphing, and functions from 58% to 70%. The
structure that was put into place by the principal was more fully developed than the other
two schools. This is evidenced by the department wide common assessments that were
established, and the PLC meetings organized around data discussions. The teaching staff
of 28 had the fewest years of educational experience across the three sites with an
average of 9.8 years. The principal at Harrison was optimistic about the academic growth
of his school, he stated, “We have the conditions to be a really, really, good school.”
51
Lincoln Middle School stands apart from the other two middle schools in this
study. The principal has been at Lincoln for four years which is similar to Burbank’s
principal. The school’s staff, however, has more years of experience in comparison to
the other two schools. According to the California Department of Education, the
teaching staff had on average 14.2 years of educational service. This stands in contrast to
Burbank’s 11.2 and Harrison’s 9.8 years of educational service. Interview data from the
principal suggests that the school culture was not as data-friendly as the other schools in
this study. The principal commented “when you’ve got that group of teachers that have
been teaching the same subject the same way for 30 years, they really don’t care to look
at data.” The data practices and structures on a school wide level were the least defined
of the three middle schools. The PLCs at Lincoln seemed to be less focused on the use of
data in their discussion than the other schools. However, individual teachers employed
their own use of data to drive instruction within their classrooms and the principal was
working with the district to implement benchmarks in her departments.
In the section that follows, I will describe in more detail how the principals at the
three sites supported teachers in the use of data.
How middle school principals support teachers with data-driven decision making.
The three themes in this section demonstrate the importance of the school site
leader in promoting data use within the school. The findings in this study reveal that an
important function of the principal is to facilitate easy access to state student assessment
data for teachers. In order to access the state student testing data, principals in the
52
Orange Valley School District use a data report service called Edusoft
2
. By utilizing the
resources available to them, principals have the ability to generate different school
reports on student achievement data. As a result of the principals’ familiarity with data,
they have become the main source of information for teachers who need state testing data
or customized reports.
In addition to providing their staff with data, principals also had the expectation
that the teachers use data to inform their decisions. At all three schools sites each
principal had reported that their staff knew of this expectation. Principals promoted the
use of data through staff meetings, agenda items, and use of Professional Learning
Communities. As a result of principals talking about the value of data and promoting its
use, it has helped in creating a data culture at the schools sites.
Principals’ support for benchmarks at the schools sites helped to promote data use
among the staff. All three school sites reported that benchmark assessments have become
valuable in helping to shape their instruction. The data from this study suggested that
teachers felt their principals’ expectations for the use of benchmarks and their support of
data had helped promote its use at the schools sites. The following section will elaborate
on these themes, detailing how the principals help to support data use at their schools.
1. Making Data Available:
Teachers regard the principals as the main source of information on student
achievement data. The majority of teachers indicated that they would go to their
2
Edusoft is a web service which gives schools the ability to conduct analysis of student performance data
from state assessments.
53
principals if they needed student data or had questions about it. At each of the three
schools, the administrators interpreted data for teachers and presented it in an
understandable way to them. One teacher noted about her administration,
They provide it and they explain it, because some people aren’t up on their
statistics and they give us all those graphs from the CST and it breaks them down
so nicely into populations and stuff. Sometimes people don’t really get that. So,
they really take the time to go through it and explain it.
Another teacher reported that his principal was very supportive of providing
access to data for his teachers. The teacher commented, “We get a lot of support because
there are times when we need certain data that is available out there and we don’t know
how to pull it.” One veteran teacher felt her principal also promoted data use by
providing them with data that reflected increased student achievement. Overall, nine of
the twelve teachers in this study reported they were provided CST data (also known as
STAR data) by their principals. Nine of the twelve also responded that they would go to
their principal if they needed to retrieve a report on student CST data. Of the three
teachers who would not first go to the principal for assistance, two said they would go to
their assistant principal and the other would use the school psychologist.
In addition to providing teachers with specific reports, principals also provided
data on the previous year’s CST at the beginning of the new school year to their staff.
The presentation of the previous year’s CST data served different purposes for principals.
They found it helped to determine the gains and losses in the different targeted subject
areas. The other purpose was to set student achievement goals for the different
departments for the upcoming school year. Harrison’s principal stated, “We look at
certain grade levels and the things that teachers have implemented to see how they did for
54
that set of students from the previous year.” He went on to say that “Teachers will use
specific data sets from their classes through their departments or their content areas and
they will create SMART (Specific, Measurable, Attainable, Relevant and Timeline)
goals.” See Appendix C for Harrison’s SMART goal sheet. Burbank’s principal noted,
“The beginning of the year, we start out with an overview of our STAR performance,”
He then explained, “and we talk about areas of significant deficit.” One teacher noted
that the beginning of the year presentation of the data let the staff know where the school
stood in comparison to the other middle schools within the Orange Valley School
District. Another teacher explained that the presentation of the data at the beginning of
the year helped teachers to understand the CST data because they are not used to seeing
it. She went on to explain that the presentation helped to drive discussions among the
staff about how to help the students.
In support of the findings above from the teacher interviews, the three principals
felt it was their responsibility to provide their teachers with student testing data and teach
them how to access it. One principal said, “my job is to make sure that they’re following
through with using data; that they get it and we help them learn how to access it and then
when they have accessed it, then what to do with it.” Another principal stated that she
pulled data to distill and refine it for teachers. This was done for the purpose developing
strategies for students and intervention classes. In order to improve teachers’ access to
data and be able to more easily understand it, the principal helped teachers use Edusoft
reports. Edusoft generates user friendly reports based on student achievement data. The
principal commented:
55
I’m constantly using tools in a different way and then giving that to them and
saying, “Look, here’s what we can generate. We can generate this report and this
report.” And that’s getting – especially those people that have been around for a
long time. It’s getting them to, “Look, I can do that myself.” You know? So
that’s great.
One principal noted that the student information system that is used in Orange Valley
called Zangle produces reports in a raw format that is hard to read. Therefore, he
commented, he frequently had to use different data manipulation tools, such as Microsoft
Excel or Access, to format the data to make it more usable for his staff.
Additionally, principals reported that they wanted teachers to analyze their grades
and student performance on CSTs. To accomplish this, two principals provided academic
grade data and CST scores to their teachers. The principals believed teachers would be
able to determine if there was a discrepancy between the students’ academic grades
which they received in the teachers’ classes and the CST scores, and thus, reflect on their
practices.
2. Expectations of Data Use
All three principals had expectations that the teaching staff would use student data
to improve student achievement. At the schools sites principals relied on several methods
to inform the staff about what their expectations were in regards to using data. One
common method that was corroborated through teacher and principal interviews was
setting the agenda for the teachers’ PLC meeting times. PLCs are regularly held
meetings that teachers have with each other in order to build their capacity and to
increase student learning. Once a week the teachers will meet by grade level,
department, and course alike. One principal explained this when he said,
56
It’s about creating those opportunities to ensure they’re doing it, and following
through with the activities that we do at each of our professional learning
community meetings that says, “I’ve got to be doing this. I’ve got be looking at it
and here’s how and what we’re going to be doing.”
Principals are able to focus the teachers’ group discussion on what they feel needs to be
accomplished by setting the agenda or placing items on the agenda. Lincoln’s principal
used this method less frequently, but she did place specific data analysis items on the
agenda if she felt they were needed. She noted the reason for this was that she wanted
the teachers to be able to create the agenda themselves. The principal from Burbank set
the expectation that data were to be used in discussions during teachers’ PLC time every
third week. Also, it was found that Harrison’s principal would provide set exercises
related to student achievement data for the teachers to complete during their PLC meeting
time. By all principals setting these expectations, teachers were able to come to the
meetings with material prepared in advance to make their data discussions more valuable.
The majority of the teacher interviews corroborated that the principals at the three
school sites set agendas or placed items on an agenda to inform teachers of their
expectations to use data. One teacher noted, “They give us time during PLC meetings to
analyze the data, decide where we’re going to go with this, how to plan lessons to really
pick help kids understand the information.” At one school site, teachers felt their
principal had a high expectation to use data to drive their discussions during their PLC
meetings. This was reflected in the teacher’s comment when it was reported that the last
four or five meetings focused on analysis of state testing data. This teacher stated,
“We’ve been meeting for the last several weeks going over data to start tweaking our
lesson plans according to the data. So the expectations are pretty high.”
57
After interviewing participants in this study, it was found that principals had the
expectation that teachers would use data provided from benchmark assessments to help
drive PLC discussions. This was found to be especially true in the subject of science.
Orange Valley has required its middle schools to administer science benchmark
assessments. Teachers reported that their principals expected them to use the data from
the benchmarks in their discussions to help student achievement. The principal of
Lincoln explained why she placed so much importance on using common assessments,
stating:
I really believe that they’re going to need to have those school wide assessments
and everybody gives the same thing, because at this point in time the real
dilemma for my school site, and I would imagine it’s the same for …a lot of
schools is that you’ve got teachers where a child is really just barely over the line
in the basic, and we really want to bring them up proficient and they’re getting
A’s in class.
Harrison’s principal expected his teachers to administer common benchmark assessments
across all of the departments at the school and to discuss their results in their PLC
meetings. See Appendix C for Harrison’s documents used in data discussions. The
principal noted that common benchmark assessments allowed for productive discussions
to be held about teacher instruction, whether the assessments were seen by the teachers as
being valid, and if improvements were needed. This comment was further supported by a
teacher who stated:
If we’ve given a benchmark test, we would come back with information as to how
our kids did, maybe some common problems that we experienced, and the hope
always is that if one teacher is able to teach that lesson very well, and someone
else didn’t feel that great about it, that there would be these conversations where
they would share.
58
Using protocols or placing items on the agenda set the expectation that teachers
were to have data discussions during their PLC time. The clear expectations from the
principals helped to focus the meetings on the topics deemed important by the principal.
The following section further explains how the principals supported data use at their
school sites.
3. Support in Use of Data
At each of the three schools, principals had different ways in which they
supported their staff in using data. As previously noted, all three principals provided help
to facilitate access to data for teachers. Principals would help take raw data and create
more readable reports for staff, pull data, provide academic grade data, and give access to
Edusoft, among others. In addition to providing and facilitating access to data, the
research in this study found that principals supported teachers through site based trainings
and facilitating discussions on data. Principals reported that they or other staff members
would provide training for teachers on using Edusoft so they could pull student CST data
reports. Lincoln’s principal exclaimed, “I really try to make sure that they have the
training so that they can use the tools that we have.” It was also noted that some teachers
had received training from the district to use Edusoft.
However, after interviewing the teacher participants, it became abundantly clear
that although there was some support with training, there was a need for organized
ongoing site based training to use Edusoft. One veteran teacher reported, “I’m not that
technically advanced. I’m old school. And that’s a challenge for me, Edusoft. We’ve
59
had training in it, but for me it hasn’t sunk in that well.” When teachers received training
on Edusoft that was either provided by the school site or the District, they found it to be
of some value. One teacher noted that, “The training was valuable, but I do think that,
for instance, the Edusoft training could have been a little bit more extensive.” In support
of this, another teacher stated that they received training during their PLC time and it was
“absolutely valuable to me.” In sum, the majority of teachers felt that training in Edusoft
was effective; however, trainings were not extensive and ongoing.
In addition to training with Edusoft, principals also supported data use for their
teachers by facilitating discussions with their staffs. One principal commented that
discussions are naturally held in August when STAR data is released, but also the
discussions are carried on throughout the year. To help facilitate discussions among the
staff and to increase use of data, the principal provided data packets to the teachers. He
stated, “You know I give people packets and we’ll continuously at meetings say ‘Okay,
bring your STAR data, you know bring some of that data we have been working on,’ to
kind of get them, again, looking at it and not just opening it and then it’s done.” The
same principal also touted the value of having discussions that are based on data rather
than ones that are not. He found that the focus of the entire meeting changes. He
exclaimed,
It’s been beneficial because they’re eye openers. You know we really focus – the
conversation is in how we can improve instruction rather than “Why are kids
screwing this up?” And it becomes a more positive thing; rather than creating
reasons of why kids can’t do it as opposed to “What can we do to help ensure that
they do stuff?”
60
He went on to say about the meetings, “So it’s very helpful. Again, it’s a challenge; it’s
something that you continuously follow through as a principal.”
Harrison’s principal used data tools to support his teachers in having focused
discussions on data. In PLCs, the principal had teachers reach consensus about a
common benchmark assessment and document the planned assessment on a form (see
Appendix C). After the assessment was given, he again had teachers meet in their PLCs
and they brought with them their student performance data from the common benchmark
assessment. At the meeting, teachers had to discuss whether students demonstrated
mastery of the standards assessed or not. In addition to this, teachers had to discuss what
they would do to remediate those students who did not demonstrate mastery of the
standards. Furthermore, the common benchmark assessments were analyzed for
strengths and weaknesses and changes were planned based on teacher input. See
Appendix C for the common benchmark analysis worksheet. By having a clear process
for creating and analyzing a common benchmark assessment, it supported teachers in
using data.
This study also found principals supported data discussions by attending PLC
meetings. Lincoln’s principal stated, “You know, my assistant principal and I go
between all those meetings whether it’s curricular or grade level, we’re always there kind
of helping facilitate.” At Burbank, the principal reported that he used guided discussions
with his staff in helping them analyze testing data. He also supported the different
departments depending on their level of need. For example, he commented that he has a
“heavy hand” with his social studies department which has not progressed significantly
61
on STAR testing, and as a result, the principal has been working with them on backwards
mapping. Backwards mapping targets the outcomes of what students should be able to
demonstrate at the end of a lesson or unit. Teachers plan their lessons with those
outcomes in mind, and only move forward once students are able to demonstrate that they
can achieve them. However, with the math department, which has been performing at
high levels, the principal had significantly less involvement. Data from teachers
corroborated that the principal at Burbank helped to lead data discussions. As one
teacher stated, “Sometimes the principal will come in with something he has analyzed
and then share it with us and have us respond to the data, ‘how did this go?’ And we can
give him a clear view of whatever happened.”
In sum, it was clear that all three the middle school principals supported their
teachers in data driven decision making on some level. The most common form of
support came in the form of providing CST testing data to the teachers and helping them
to interpret it. However, the strongest support for data use from a principal was seen at
Harrison. He was able promote a higher level of data use by providing data tools for
teachers to analyze their common benchmark assessments. The data tools helped to
facilitate a process of continual improvement at the school. Principals also provided
training and facilitated discussions around data to help promote it use within the school.
Finally, the principals’ expectations for data use helped inform teachers that it was a
priority. Related to this, the following section will discuss how school structures and
cultures further promote the use of data – or not.
62
How school structures and cultures facilitate or inhibit data-driven decision making
Two themes arose that either facilitated or inhibited data-driven decision making
at all three middle school sites. One finding was the need for time for teachers to have
data-discussions. At all three school sites additional time had been devoted for teachers
to meet in their PLCs so they could meet to discuss data. However, this study found that
there were differences in how the PLC time was used among the schools. Teacher
meeting time and how it impacts data-driven decision making will be explored in this
section.
In addition to examining time allotted for discussions, this section will also
examine the data cultures of each school. Research has previously been cited about the
importance of school culture in helping to enhance teacher instruction and student
learning (Bryk & Schneider, 2002). While the schools in this study were selected
because they were ostensibly high performers when it came to data use, in fact, at one
school that culture may be inhibiting the use of data to its fullest. This section will also
examine how the principals’ beliefs in data-driven decision making can impact the data
culture of the schools.
1. Time for Data Discussion
An overwhelming number of respondents, both principal and teachers, identified
PLC meetings as the time when data was discussed at length. At the time this study had
been conducted, the PLCs had been in place for three years. Each of the three middle
schools had set aside time in their schedules each week for teachers to meet and discuss a
range of topics. On Tuesdays, all middle schools have a late start day for students. On
63
late start days, students begin class at 9:30 a.m. instead of the typical time of 8:15 a.m.
During the late start days, teachers are able to meet with their PLCs to have discussions.
They are obligated to meet for one hour from 8 a.m. to 9 a.m., but they have the option to
continue meeting up to 9:30 am if they chose to. The interviews from teachers and
principals indicated that meeting time was very important. As one principal exclaimed,
The PLC time that we kind of carved out has just been fabulous. And I was kind
of in on the ground level. We sort of were covert about doing it. We met – we
did! We met and kind of got this put together and I don’t think that we really
realized at the time how powerful it was going to be.
All three schools had their PLCs structured so they would meet with grade level teachers
and content-alike (departments). It was also found at one school they held one PLC
meeting a month that was focused on at-risk students. During the at-risk meetings,
teachers would meet with the team they taught with to discuss possible interventions for
students that would help them be successful.
Teachers did use the PLCs to discuss student achievement data. One teacher
reported, “We have what are called PLCs. Every Tuesday school starts late and teachers
gather for an hour and a half and we meet in content-alike areas usually or sometimes the
meetings are not content alike. But in our content-alike social studies for the last four or
five meetings we’ve been going over data from the state test.” At another middle school,
a teacher also reported about using data in PLC meetings, “If we are having content-alike
– meaning English – it is used, I would say probably half the time. So 50 percent of the
time, depending on what we’re doing, we might use some sort of data, either what they
have provided for us, or the common assessment tools we’ve done.” Another teacher
64
further supported this statement by noting that at least one and sometimes two of the four
PLCs held each month are used to analyze common benchmark or CST data.
Each middle school’s PLC time was used somewhat differently. Decisions about
how time was used were typically made by the principals. At Harrison, the principal
used different tools to focus teachers on their tasks. For example, the principal would
have his departments create SMART goals which were based on student achievement
data. These goals were then reflected on and used throughout the year in the PLCs. He
stated, “Through our SMART goals, teachers will use specific data sets from their
classes, their departments, or their content areas and they will create SMART goals.” A
teacher confirmed the principal’s statement about using data for SMART goals. She
stated that CST data is “Used with the SMART goals.”
At Burbank Middle School, the principal had PLCs analyze CST testing data and
compare it to student academic grades each quarter. The principal stated, “The data they
are going to be looking at, you know, in conjunction with STAR data sheet that they have
will be grade data, and every quarter I generate reports on student achievement with
regards to grades.” The principal also reported that PLCs were to take minutes during
meetings and they were to have a goal to achieve at the beginning when they met. He
admittedly commented, “That’s not well established yet within our culture, but we’re
working towards it,” The principal also worked with a leadership team to help create the
agenda for the PLCs. By working with his leadership team in building the PLC agendas,
it also created buy-in from his staff. Having buy-in from his teachers made the PLC
meetings more productive.
65
At Lincoln Middle School, the principal’s approach to PLCs was different than
the others. The study revealed that the principal took a more hands off approach with the
PLCs. The principal involved in this study commented,
You know what, I’m pretty flexible with our – we have weekly PLC, so I have,
you know, almost an hour and a half every Tuesday if I want it. I really don’t try
to be strict with that time that it’s set up really far in advance because it’s
important that they actually can – if we need something that we can use it.
During the PLC meeting times at the schools, teachers were expected to bring their
STAR testing and assessment data on their students to discuss students’ needs. The
teachers would then meet in either their grade level or the same subject area prepared for
discussions.
The time and the structures built into the PLCs made for more focused and
productive discussions. This was indicated by many of the teachers who commented that
PLC meeting time was when the majority of data discussions were held. The principals
at the school helped to promote discussions by setting time aside for data discussions in
PLCs. They also helped by preparing either protocols, agendas, or having data available
for teachers when PLC time was dedicated for data discussions.
V. Culture
At all three schools each had developed their own unique data culture. This
became evident from the interviews of the principals and the teachers. At Burbank and
Harrison Middle School, it was found that their data cultures had developed at a higher
level than at Lincoln Middle School. At Lincoln Middle School, the culture of the staff
was not as favorable toward data, and as a result of this, it may have been inhibiting the
66
use of data. Not all teachers believed in using data and there was resistance in using it in
their PLC meetings to make decisions.
Meetings at Lincoln were not strictly focused on using data. One Lincoln teacher
stated, “I think we need to have more focused meeting time to discuss and analyze data. I
think, from that, we need to be able to come up with ideas to improve upon the results we
found.” This statement was supported by another teacher who commented on whether the
time to analyze data was effective or not. She stated,
Yes and no. I think if the time on – I think on Tuesday, for a hour, hour and a half
we get to work as a school or in our grade levels or content areas, I think it’s an
ample amount of time. I don’t think within each group – you know, I can’t speak
for them, but I don’t think people are using it as efficiently as they could have.
The principal noted that the culture of the school was not favorable toward using data at
one time. The principal commented,
When I got here, there was definitely an entrenched population here of teachers
that had been here for like 30 plus years. So getting them to actually use any data
that’s from benchmarks, getting them to use the benchmark exams was really hard
and getting them to look at data from test was real difficult.
Lincoln’s teachers also revealed that there are still challenges for a school wide
data culture to develop. As one teacher stated, “As a school, I think it has – I don’t know
how to say this. I want all the departments to be going in the same direction.” The
teacher went on to explain, “Our school district started with math, where the math folks
looked at the standards and tried to improve instruction, even make a map of the entire
school year. That hasn’t happened in other curricular areas.” It’s important to note, that
Lincoln only administered the district benchmarks school wide in math and science. The
principal was working with teachers from the other departments so they could create their
67
own; however, she was experiencing some resistance. This teacher also went on to
explain that when mapping was suggested for other departments that it “made some
waves” with staff members. In addition to this, two teachers at this school site who were
asked whether teachers share their CST results with one another, and both responded
“No”. One teacher went on to explain, “We can’t even raise our hand and vote in front of
each other. It upsets people.”
The above evidence supports the finding that the data culture of the school has not
fully developed and may be inhibiting the use of data to its fullest potential. Lincoln’s
principal commented that “I think the challenge really is getting everybody to buy into
it’s okay to look at it [data].” However, interview data suggests that lack of direction
from the principal may be inhibiting the growth of a data culture. Lincoln staff lack
training in using data and there is an absence of protocols to help teachers guide their data
discussions. When a Lincoln teacher was asked what type of training would help
teachers use data for instructional improvement, she exclaimed, “I think actually having a
training.” The same teacher also reported that protocols were not used in their data
discussions with their PLCs. The lack of training and use of protocols may contribute to
the lack of a fully developed data culture.
At Burbank Middle School, a more highly developed data culture existed. The
culture was supported in part by the principal’s beliefs in being data-driven. The
principal commented that data has been very useful for the school. At the school, the
staff has used data to specifically focus on students who are struggling academically.
After interviewing the four teacher participants, it became abundantly clear that they all
68
felt their principal was data driven. One teacher commented about the principal, that the
principal “loves data”, another teacher reported, “I think he really has a passion for the
data and he loves to crunch the numbers that are out there.” One teacher characterized
both the principal and the assistant principal as very “data driven.”
While the teachers reported that their principal was data-driven, the study also
examined whether the culture among the teaching staff was data friendly. All of the
teachers reported that data has contributed in someway to the success of the school and
students. In support of this, one teacher commented about data “I believe it very much
helps with student achievement.” Another veteran teacher commented that data was used
to help identify students who were in need of help. One teacher felt very strongly about
the effectiveness of using data, she commented, “I think the more that we’re using it, the
more that we’re able to see how it works and what we can gain from it.” However, some
reported that all the success could not be attributed to using data to drive instruction. A
veteran teacher stated, “From time to time definitely data has been very important part of
our success, but I think more of it can be attributed to the fact that we just have
conscientious teachers that really love kids.” Nonetheless, all teachers were proponents
of using data because they felt that it helped student achievement to a certain degree.
The principal of Harrison Middle School was also a proponent of using data in the
decision making process. The principal had been in place at the school for just over two
years and was working to build a strong data culture at his school site. He believed data
allowed him to have a much better understanding of his school so he could affect change
for the school’s students. The principal remarked, “By using data we have found that we
69
can really hone in on what’s going on with our school.” He also advocated the use of
data because of it was a tool to measure whether changes were effective or not. To
highlight his point, the principal likened not using data to throwing darts blind. Teacher
interview data supported that they felt the principal believed in using data to drive
instruction. One teacher noted that the principal expected teachers to analyze STAR
testing and common assessment data during their PLC and staff meeting times. She
stated about data meetings:
We get together and we work as a group, as a whole group, whole staff and we
talk about it. And it’s put on the board and maybe explained or just discussed,
why has this dropped here or if this has risen here, what did we do differently and
we talk about it as a group. And then often we get into our groups, just content-
alike and we talk about what we could do to improve in certain areas.
The data showed that the culture at Harrison was favorable to using data. One
teacher stated about data, “I think if you have different types, I think it’s effective.”
When asked if she would attribute data as an important reason for Harrison’s success in
reaching its AYP, she stated “I do.”
The different uses of data by teachers also indicated that the school has a data-
friendly culture. One teacher noted that by using data they could identify areas of
weakness for their students and increase instruction to address it. For instance, one
teacher stated that data verified that she was spending less time on vocabulary with her
students, thus she increased the number of exercises to improve student performance.
Teachers also indicated that they used benchmark assessments and common assessments
throughout the year. Teachers reported that the data gathered from the assessments were
valuable in guiding their instruction. However, one teacher at Harrison felt that data was
70
not widely used across the school and that it played a small role in the school’s meeting
its AYP and API. The teacher stated, “I think it has very little impact on how our scores
have changed.” The teacher, however, was a proponent of using data in the classroom.
She commented about data saying, “I want to be the best teacher I possibly can be and
that means I can’t ignore the data that’s out there.” The comments from the teachers
demonstrate that there is a friendly data culture present at Harrison Middle School.
At each of the three schools, it was found that culture can impact data use either
positively or negatively. At Harrison and Burbank Middle School, each had a staff that
felt data had a positive role in their school’s success; however, both schools had
principals that clearly supported data use. On the other hand, Lincoln’s staff was not as
favorable to using data and the principal’s vision and support for data use was not as clear
as the other two principals. As a result of not having clear direction from the principal
regarding data use and the support of it, Lincoln’s data culture had not developed to a
high level like Burbank or Harrison Middle School.
What types of data are collected by educators?
The three schools all collected different forms of student data to drive their
decision making process. However, the type of data and the amount varied at each site.
Three major forms of data – summative assessments, formative assessments, and
benchmark assessments – were found to be used. However, the degree to which each
form of data were used relied on how heavily the principal emphasized its importance.
71
This section will examine the different types of data used and importance that each
school site placed on them.
1. Summative Assessment
The most common summative assessment that was used by all three schools was
the California Standards Test. This assessment is required by the state of California to be
given to students starting in second grade and up to the eleventh grade. The assessment
is given statewide for middle schools during the month of May. In addition to this,
seventh grade students are given a writing test in March. The test results are then
released in August. At the middle school level, students are assessed on the subjects
noted in table 9.
Table 7: Subjects and Grades Assessed in the CST
Grade Social Science Math Language Arts Science
6 X X
7 X X
8 X X X X
Source: California Department of Education – http://cde.ca.gov
In this study, it was found that all three schools referred often to prior year’s CST
data during the beginning of the year. At Lincoln Middle School, when teachers were
asked what student data are used most frequently by the school, three out of four teachers
stated CST data was used. The principal confirmed the teachers’ responses by stating
that CST data was the data used most frequently at her school site. However, some
teachers expressed some dissatisfaction with using CST data. One teacher noted,
“Unfortunately, we focus a lot on the CST. We use those scores as a basis to decide what
students have what needs.” Another teacher commented that CST data was useful at the
beginning of the year but it became less important as the year went on.
72
At Burbank Middle School, evidence indicated that CST data was the most
heavily relied upon student data. This conclusion was reached after speaking with the
principal and teachers at the school site. When asked what types of student achievement
data does the school collect and examine the most frequently the principal stated, “I
would say we look at the CST assessments the most.” The study confirmed that the data
gathered from the CST was used frequently throughout the year by the different
departments and the principal. Teachers confirmed the principal’s statement that they use
CST data. Of the four teachers interviewed, three of them indicated that CST data was
used in their PLC discussions. In support of the principal’s statement, one teacher stated,
“We have weekly meetings and often times we’re looking over the state assessments.
And we’re taking a closer look at benchmarks in different areas now, but most of the data
is related to the state achievement test that we take a look at.”
At Harrison Middle School, the study found that CST data was frequently used
during the school year. Additionally, it was also found that Harrison often referred to
other assessments over the course of the year. The principal at Harrison indicated that
STAR data was most frequently used data source. He stated, “Yeah, obviously it’s the
STAR data, that’s there from the CSTs you have all of that data each year.” He then
when on to say, “In addition to STAR data, we also use- we do common assessments.”
Teachers at the school supported the principal’s statement. One teacher noted that CST
scores are used along with common benchmark assessments, which are the same
assessments used throughout the department.
73
All three schools relied significantly on the use of summative data gathered from
the CSTs. All principals indicated that they used the data to analyze the performance of
the prior year’s students. The data from the CST, along with benchmark data, was also
used to drive discussions with teachers throughout the year. This study found that
schools did not have another summative assessment which they felt played as large of a
role as the CST.
2. Benchmark Assessments
All three schools showed evidence of using benchmark assessments in order to
collect student performance data. The Orange Valley School District had required the
middle schools to administer a district benchmark assessment within the math and
science classes. According to interview data, the math benchmark assessments had been
given for the past three years, while science benchmarks had just begun for the 2008-
2009 school year. This was verified by a teacher from Lincoln stated, “We’re starting the
benchmarks this year in science. Math has already started them, and so, we’re doing
benchmark exams.” This statement was also supported by a teacher from Burbank who
stated, “This year we have instituted a benchmark test system for science.” She went on
to say, “And this year we’re meeting together after each quarter, analyzing the results,
making any adjustments to them.” The importance of the district’s support in requiring
the school’s to administer benchmarks can be seen in one teacher’s comment. When
asked if the science department has common benchmarks before the district’s
assessments she replied, “No. No, it was just basically the only thing we’ve had
department wide is basically the CST.”
74
In addition to district benchmarks, some school sites had created their own
common benchmark assessments to be used within their departments. The principal from
Harrison Middle School stated, “Every course that we have has a benchmark assessment
and we have two of those per year. So we do them in second quarter and some third, and
some fourth at this point in time.” The principal’s statement was supported by a teacher
who was asked how often she gave a common assessment, she commented, “Last year we
did it twice.”
At Burbank Middle School, the principal reported that benchmarks had been
established in math and science. In addition to these benchmarks, the principal indicated
that a writing benchmark would be given at his school site which would be an equivalent
to the state’s writing exam. The principal reported that his departments were at different
phases and he was working with his social studies and Language Art departments to
increase their use of data.
Lincoln Middle School administered the district’s math and science benchmark
assessments. The principal at Lincoln had also been working with her departments to
develop benchmarks to be administered at her school. However, she commented that she
had run into opposition to benchmarks from some of her staff. She stated, “Getting them
to use the benchmark exams was really hard and getting them to look at data from test
was real difficult.” She noted that she works with her staff and is able to move them
forward a little at a time. She reported,
I’ve been kind of trying to get those people to move and see that in fact there is
value. If you can see that the kid-it’s not that just that the child, you know, is
doing poorly in Language Arts but it’s something that’s real specific and we can
use that data to improve that kid.
75
With the district’s support, schools were able to administer school wide
benchmarks in math and science; however, while it was evident that Harrison Middle
School had established school site benchmarks in social studies and Language Arts, it
was not evident that this had occurred with Burbank and Lincoln Middle School. In lieu
of the district benchmarks or a common benchmark used by teachers who taught the same
subject, interview data from Burbank and Lincoln teachers indicated that some
administered their own individual benchmarks. One teacher stated, “There are some pre-
made benchmarks tests in the other subjects from the books that some teachers choose to
use and some do not.” A language arts teacher from Burbank Middle School was asked
if she gives a common benchmark the department uses or her own benchmark
assessment. She replied, “I have my individual ones in Language Arts.” Some teachers
who did not give department wide benchmarks used assessments from the textbooks used
in the class. When another teacher was asked if she used a department benchmark
assessment, she responded, “Here, no, no. Within my classes, yes I do. So, we do, and
I’m doing it every quarter. We do quarter benchmarks.” The response from all teachers
was that they gave a benchmark that came from the district, department, textbook, or one
they created themselves.
In sum, benchmarks were used at all three school sites, but to varying degrees.
The principal of Harrison had instituted the use of common benchmarks in all
departments; whereas Lincoln and Burbank Middle School just administered the district
benchmarks in math and science. At all three schools, teachers commented that they
created their own individual benchmarks to assess their students’ learning.
76
3. Formative Assessment
In addition to using summative and benchmark assessments, all three schools used
formative assessment to drive their decision making. Formative assessment uses data
gathered weekly or monthly from quizzes, projects, and written assignments to improve
teacher instruction with the end purpose of increasing student achievement. Formative
assessment allows teachers to identify areas of weakness in students and as a result,
teachers can target additional instruction as needed to improve learning. The following
section will describe how these assessments were used at the three school sites.
The majority of the teachers in this study reported that they believed the results
from their class assignments had the largest impact on how they instructed their students.
For instance, one teacher from Lincoln Middle School stated,
Day-to-day class assignments and the discussions that we have in class to check
for understanding things I do. I don’t find the CST scores as a big thing, a big
data component that I’m going to change and teach, because I realize that’s one
day and it’s a snapshot of what they did on that one day.
In support of the finding stated above, all four teachers at Lincoln Middle School
reported that daily assessments had the most significant impact on their instruction.
Another Lincoln teacher stated, “For me, it’s my own tests. It’s my own daily
observations of the kids. That gives me more information than this one test, I think, but I
mean, it’s all part of the package.” In examining the other two middle schools, similar
results were found. One veteran teacher from Burbank Middle School stated, “For my
particular classes I need to know what my students are being successful in, so I try to get
feedback through doing projects with them.” He went on to explain, “I get immediate
feedback on all – like these three and four day things. Every four days I can kind of tell
77
who’s getting it, who’s not.” Moreover, at Harrison Middle School, all the teachers
reported that their own daily assessments influenced their classroom instruction. In
support of this, when a teacher was asked “What types of data do you find most useful in
making decisions about your instruction and curriculum?” She explained, “It has to be
what we presently do in the class.” She went further commenting, “I mean, I look at the
test scores but the test scores are a one-day assignment and it’s really hard, I think, for the
kids that are like below basic. It’s hard for me to determine are they really below basic or
did they not choose to take this test?” It was found that while teachers believed in the use
of data derived from summative assessments and benchmark exams, they felt their daily
assignments gave them the most meaningful information about student performance.
Formative assessment was found to play a pivotal role in the math department at
Burbank Middle School. Burbank’s highest performing academic department, when
measured by STAR test scores, is math. The math STAR scores are the highest out of the
eight middle schools located in OVSD. The principal from Burbank noted,
The math department embraces data, of course. It’s easy for them to do, and they
do weekly assessments, and they actually talk about the scores in a global way,
and they identify maybe sections that were strong with some instructor and not as
strong as others, and they discuss whether or not we should move forward or
focus a little bit more on that one section.
The principal’s statement about his math department was supported by his math
department chair. She stated,
We use data at every juncture, for instance in mathematic department, we start
every Friday is an assessment day and then on Tuesdays when we meet at our
PLC time, Profession Learning Community time, the entire department, all of us
bring our data, talk about what was taught and what did we test them on and then
if we had any problems, what do we need to re-teach.
78
The teacher went on to explain the importance of daily class assignments for the math
department. When she was asked what data she relied upon most in her teaching, she
commented,
It is daily. Our tests are driven by daily homework. That is the beauty of what we
do. We give homework but out test data comes out of our homework. We don’t
even change the question. So if you did all your homework assignments, you’re
basically ready for the test. So that’s the data we rely on most.
As these comments reveal, teachers place a high level of importance on using
formative assessment in their classrooms. In their day-to-day teaching, formative
assessment determined whether teachers re-taught a concept or moved forward in the
curriculum. Data from formative assessments were also used by the math department at
Burbank in their PLC meetings. The math teachers used it to drive productive
discussions around student achievement, teacher instruction, and curriculum.
How are data used by leaders and teachers and for what purposes?
This section examines how teachers and principals use the student data at the
school sites. Student data were used most commonly in student placement, student
support, and instructional change. Principals based decisions on placing students in
support classes using student data derived from CST results in order to further their
academic achievement. Teachers also used data to understand the strengths and
weaknesses of their students. In addition to this, teachers also used data in the PLC
discussions to help identify the needs of students and recommend supports to address
them.
79
1. Student Placement and Support
Student data was utilized in the placement of students in academic support
classes. All three principals made use of student achievement data in helping to place
students in classes which would support the students’ academic growth. The most
commonly used data to inform a decision was STAR testing results. However, there
were other types of data such as academic grades and benchmark assessments that were
also found to play a role in determining whether a student needed to be placed in a
specific class.
At Burbank Middle School, the STAR test scores were a primary indicator of
whether the student needed to be placed in a support class. During the study the principal
commented on students who had low grades and STAR scores. He stated, ”Depending
on the degree of underachievement, they may end up with an intervention class that
substitutes for their elective.” The principal reported that students who tested as “far
below basic” or “below basic” on STAR testing would “fall on our radar” and become
candidates for targeted intervention either during or after school. To address the needs of
these “at-risk” students, the principal created a reading intervention and a writing
intervention. These support classes were designed to strengthen the areas where
struggling students were weak in order to raise their academic achievement.
At Harrison Middle School, it was found that students were also placed in support
classes based upon data. The principal reported that STAR data was used to identify
students who need additional support in areas where they had deficits. To address the
needs of their students, Harrison would double-block a student with an additional math or
80
Language Arts class. The school was able to double-block a student by eliminating their
elective class. The principal stated,
Right now I’ll pull a lot of data sets for teachers; things that can really hit home
with them and where it’s really concrete evidence that says “This kid struggles in
this particular area or multiple areas,” and we use that to get students extra help
you know and we’ll double-block a class, maybe an English or a math class if
we’re seeing that they’re struggling.
In addition to double-blocking, Harrison also implemented a Learning Center
model to support struggling general education and special education students. The
learning center uses research-based curricula to teach math or Language Arts. In the
Learning Center there is one primary teacher and an aide who supports students. To be
eligible to receive support from the Learning Center the students had to demonstrate a
need which was partially based on student achievement data. The principal felt his
school could improve in using data to place students in the Learning Center. He
exclaimed, “We’ve used the data to place some kids in classes, but we haven’t used it
well enough to maximize those classes.”
The principal at Lincoln Middle School used data to help support students. Using
data derived from CST scores, the principal found that Hispanic and low-socioeconomic
students would benefit from an after school support class. As a result, she had teachers
volunteer to run the class throughout the year. She felt by focusing on these student
subgroups that their academic achievement would rise, and as a result, it would be
reflected on STAR data.
Lincoln used STAR testing data to place students in math. By utilizing the STAR
testing data and determining the needs of the students, the number of math sections and
81
types of classes could be determined. The principal stated, “In our math department, she
[the department chair] analyzes the data from the minute we get it. She comes in during
the summer, in August, and looks at all the math scores and we really kind of try to put
classes together using that data.”
While academic grades and STAR testing data played a large role in the
principals’ decision making process, there were other types of data used. At Burbank
Middle School student discipline data was collected and utilized. The principal used the
discipline data to collect information on the types of discipline occurring within his
school. If he determined that there was a rise in a particular type of discipline he would
focus on it and try to resolve it. The principal reported,
One piece of data we didn’t talk about that’s important that doesn’t show up
anywhere that I can find is our student discipline data. That’s something we also
do visit yearly, and it’s another analysis that we do quarterly, numbers of U’s,
suspensions, and we use that data, which is hard to get, to target areas of need. So
if we have a sudden rise in cyber-bullying, parent instruction towards that would
occur.
By using an inquiry based decision model, the principal is able to identify what the
problem is by using data and to formulate a targeted response.
Data plays a large role in determining the kinds of supports students received at
the three middle schools. Using data, the principals are able to identify students who
need targeted intervention or students who would benefit from being placed in a
particular support program. By placing them in the classes and programs, the principals
are able to support the students’ academic achievement.
82
2. Teacher Use of Data
In a previous section the types of data used were discussed, the focus on this
section will be on how teachers use data. One of the findings of this study which helped
to support why all three schools achieved at high levels was the teachers’ support of the
use of data. All teachers were asked if they used student data in the classroom and if so,
how they made use of it. At the three school sites, it was found that all twelve teachers
responded that they used student data to drive their decisions about their students. One
teacher from Lincoln Middle School noted,
I am pretty data driven, I think. For example, I have here just a list of all the
seventh-grade kids, and I have the majority of seventh-grade kids, and it has their
individual reports. I can see what their strengths are, what their weakness are. It
gives me an idea of, okay, what are these kids coming to me are like.
The teacher explained that during PLC meeting time with her department they would
discuss strategies to help lower performing students. She stated, “We like to trade stories
and lessons, so that, you know, they can do better, and that’s based on the data from the
benchmark tests.” By working with her colleagues and using student performance data,
she was able to learn and share instructional strategies to support student academic
growth.
Teachers reported different ways of how they utilized student data to support their
students. One teacher exclaimed, “I do use data in my daily use of my class. For math,
especially, with the CST scores, it did help me understand right from the get-go what
students were going to be the most in need, and it affects how I see the students. It also
does affect how I teach.” The teacher went on to explain, “With my English language
learners, that’s the data I have on them, knowing what levels they are and make sure
83
they’re seated near certain students. I also provide different kinds of material for them.”
Teachers felt the CST data gave them a general idea of what their classes’ strengths and
weaknesses were so they could adjust their instruction accordingly.
Many of the teachers noted that student assessment data from daily assignments,
unit tests, and benchmarks were used frequently. For example, one teacher noted that she
felt observational data from daily assessments provided her the most valuable
information because she was able to determine whether students understood the lesson
being taught. A math teacher from Burbank Middle School commented about her
department, “Our practice has been for the longest time, we teach three lessons in a week
and then one day for review and a practice test and Friday is generally assessment.” The
math department would then meet during their PLC time to discuss the results of the
assessments and change instruction accordingly. The math teacher went on to explain
that “when some concept is not clear, we not only re-teach, we make sure that in
subsequent two, three exams we continue to place certain questions that assess the
standard to make sure there is mastery.” In support of this math teacher, another teacher
from Burbank Middle School commented that in their PLC data discussion meetings,
“We use basically student work and their grades and achievement in our classes and
discuss how they’re doing at this point. And then we create plans to try and help them to
achieve better.” One teacher commented about the power of analyzing his student data
and meeting with others. He stated, “It’s enlightening to see how they may be more
successful somewhere else and what sort of strategies are leading to that.”
84
At Harrison Middle School teachers also used data to drive their instruction. As
with the other two schools, teachers responded that data has an impact on their classroom
instruction. In support of this, one teacher noted, “What instruction I use is based on
what the students give me, but it’s – would have to be the data within my class, like the
results of weekly writings, reading logs, and obviously tests.” It was generally observed
that teachers felt the data derived from their classroom assessments was more valuable to
them in changing their instruction then STAR results.
Student data was used to plan for non-academic support for students at Burbank
Middle School. Teachers used data to select struggling students who they could mentor.
One teacher noted,
We’ve got students that we look to try and act as guardian angels for or to look
out for and those are usually from a list of students that data said that they’re –
either not achieving where they should be or they’re you know, they’re not where
we would like them to be in one way or another and a lot of that’s from the CST
data.
The teachers felt by acting as mentors and caring for struggling students, they could help
students emotionally as well as academically.
Data was used by middle school teachers to help students in a variety of ways.
First, teachers worked within their PLCs to discuss with their colleagues what strategies
were effective for helping lower performing students. Second, teachers used data to
determine what the strengths and weaknesses of the class. Finally, it was found that
middle school teachers at Harrison and Burbank were able to use data in PLCs to identify
at-risk students who needed social and emotional support. As one teacher said, “we look
to act as guardian angels” for struggling students.
85
Conclusion
This chapter represents the research and analysis on data-driven decision making
as reported by teachers and administrators at Burbank, Lincoln, and Harrison Middle
Schools. To focus the study, a research question was developed that addressed how high
performing middle schools use data to inform their decisions. From the analysis of the
data, several prominent themes emerged. These themes were explored and then linked to
the appropriate four sub questions posed in this study. The themes were critically
examined and the data relevant to the themes were analyzed to determine their impact on
the research questions. These sub questions address leadership support for data use, the
schools’ data cultures, types of data used by teachers and administrators, and finally how
data are used by the schools.
An analysis of the data showed that administrator support for data use affects
the degree to which it is implemented at the school site. As a result of the principals’
support of data use, the data practices of some schools were more highly developed than
others. At Harrison Middle School, the principal promoted the practice of using data to
create SMART goals and common benchmark assessments for all of the departments at
his school to help improve student achievement. There was no evidence that
demonstrated that these practices were being implemented to the same degree at the other
two schools. In the area of administration support, it was also found that teachers used
data more often at sites which have administrators who have high expectations of data
use by their staffs. When administrators communicated a clear message to teachers to
use data in their decision making process, and provided them with data tools such as
86
protocols, the staffs were more inclined use data. Finally, all three schools use STAR
data, math, and science benchmark data; however, the principals who were
knowledgeable about disaggregating data, were able to support their teachers by
providing more relevant and detailed data.
In addition to administrator support of data use at schools, school structures and
culture impacted data-driven decision making at sites. At all three school sites, teachers
were provided time to meet in their PLCs to discuss data. These “data discussions”
resulted in teachers speaking about their teaching practices, student performance, and
student behavior. The PLCs ultimately led to a higher level of data use at the schools.
Culture also clearly plays a role in the use of data. At Lincoln Middle School, whether it
was due to a more veteran teaching staff or a principal who was less supportive of data
use, it was clear that the school culture was not completely receptive to fully using data.
However, at the other two middle schools, the school cultures were more data-friendly.
The principals at the schools were proponents of extensive data use and their staffs were
more receptive to using data.
While there are many forms of data which can be used by educators, a common
theme found at all three schools was the teachers’ value they placed on formative
assessment data. Teachers expressed dissatisfaction with the use of summative STAR
data because of questions surrounding its validity. Teachers reported that STAR testing
was a snap-shot of how a student performed on one day. However, all teachers felt the
data gathered from their own class assignments, quizzes, and tests helped them to gauge
students’ learning, which ultimately decided whether re-teaching took place or not.
87
Lastly, it was found that benchmark testing was beginning to play a larger role in all three
middle schools. With support from the district, the math and science departments at the
middle schools had implemented the use of common benchmarks. Harrison, on the other
hand, had instituted common benchmarks across all departments.
Administrators and teachers used data in a multitude of ways. A common theme
that became apparent when analyzing the data from all three schools was the use the data
by administrators to provide support for students. All three administrators used data to
determine if students would benefit from being placed in support classes or after school
programs. Teachers used data during PLC collaboration time, guide their instruction, and
to identify struggling learners who may need a mentor to provide social and emotional
support.
In sum, the findings of this study indicate that all three school sites had
implemented data-driven decision making. How the school used data and the extent of
use was found to be impacted by a number of factors. Some of these factors were
determined by the school site’s leadership, while others were more dependent on the
culture and structures in place at the school. Nevertheless, the majority of teachers
indicated that data use at their school site played some role in helping their students
achieve academically.
88
CHAPTER FIVE
Summary and Implications of Findings
Introduction
The increasing focus on accountability of the country’s public schools has
resulted in many turning toward data-driven decision making to increase student
achievement. This heightened accountability has had the effect of shifting a school’s
decision making processes from one that relied on a teacher’s anecdotal evidence to a
system where decisions are guided by actual data. As stated previously, there is a dearth
of data with regards to how middle schools use data-driven decision making. The
purpose of this study is to provide the educational community – both researchers and
practitioners – with timely and informative data on how three high performing middle
schools implemented data-driven decision making.
In conducting this study, an analysis was done on the responses of principals and
teachers to questions related to the support and implementation of data-driven decision
making at the middle school level. The majority of data was in the form of principal and
teacher interviews at the different school sites. In addition to interviews, data was also
collected from the gathering of documents and artifacts from each school. The summary
of the findings below are derived from the analysis of the data.
There were a number of factors that influenced data-driven decision making at
each school site. This study focused on factors such as leadership, culture, school
89
structures, types of assessments, and lastly, how data is used among teachers and
principals.
This study examined leadership and the role it has in supporting and promoting
effective data use at the school site. The principal’s role at the school was critical in
providing access to data, training, and clear directions in the use of data during PLC
collaboration time. Teachers relied upon their administrators, for the most part, in
providing them with relevant and understandable data reports on their students’
achievement. Other than the training that teachers received when developing district
benchmarks, principals were primarily responsible for the professional development of
teachers using data practices and the data warehouse program called Edusoft. Lastly,
administration support was critical in providing clear direction for PLCs so they could
effectively engage in data analysis. If clear direction from administration was lacking,
then the effectiveness of teacher collaboration time became questionable.
In addition to the leadership being a factor in effective data use, school culture
and structures were considered as well. Teachers’ willingness to be receptive to using
data-driven decision making determined how widespread the practice was within a
school. In addition to culture being a factor, school structures such as time for
collaboration in PLC meetings, were critical for teachers to conduct data discussions
about instructional practices and discuss supports for students. The effect of providing
time for teachers to engage in meaningful discussions was significant in promoting data
use at the middle schools.
90
The different types of assessments used by teachers and principals were explored
in this study. Summative assessment data was relied upon by teachers and the principals
heavily at the beginning of each year; however, teachers felt their own formative
assessments had more validity in judging a student’s performance. In addition to these
assessments, benchmark assessments were examined. Both teacher created and district
created benchmarks were implemented at the school sites and were used in data-driven
decision making.
Beyond providing data on the type of assessments used in middle schools, this
study analyzed how principals and teachers utilize data. Principals were primarily found
to rely upon summative CST data when determining student placement for support
classes. In addition to this, administrators used data to help promote data discussions
among their staff members to improve teaching practices. It was found that teachers used
assessment data to determine the strengths and weaknesses of their students so they could
focus their instruction. Additionally, through using data to identify students, teachers
used their PLC time to discuss struggling learners and create plans to support their
academic or social-emotional needs if necessary.
In sum, there were some unique features about data-driven decision making in the
middle schools. As stated in Flowers et al. (2003) research, middle schools focus more
on the students’ social and emotional health than a junior high school. It was found at
two middle schools in this study, that they used PLC time to discuss at-risk students that
were identified using data. As one teacher noted, “We’ve got students that we look to try
and kind of act as guardian angels for.” In addition to trying to meet the social and
91
emotional needs of students, the middle school model was developed to facilitate more
collaboration time with teachers so that they could discuss their students amongst each
other. Once again, the PLC meeting time allowed teachers to meet and discuss student
data with each other. These data discussions have in turn impacted instructional
techniques, facilitated the creation of benchmarks, and overall, increased the level of
interaction of teachers with each other building upon the “team” concept.
Connections to Prior Research
The findings in this study can be linked to the existing literature on data-driven
decision making. Chapter 2 examined the research on data-driven decision making in
different domains. The following areas were examined: state and federal accountability
policies, middle school reform, merits of data use, leadership, school culture, capacity
building, types of data utilized, as well as supports for data use.
Impact of Federal and State Accountability Policies on Education
The impact of federal and state accountability policies on K-12 education
subsection of the literature review discussed the development of a system of
accountability for California public schools. In this subsection, the research discussed
how the state of California created the Public School Accountability Act (PSAA) in 1999
to hold K-12 schools accountable for their academic performance by making their API
scores public (Bitter & O’Day, 2006). In addition to the measures placed on schools by
California’s PSAA, the passage of the federal NCLB act in 2002 further heightened the
92
level of accountability. NCLB requires schools to make their “adequate yearly progress”
(AYP) to meet the goal of proficiency on state standardized tests in reading and
mathematics by 2014 (U.S. Department of Education, 2002). Coburn and Talbert (2006)
have found that, together, NCLB and the PSAA have placed California public school
administrators in a position that can require them to use student achievement data to
improve education. The findings in this study are consistent with Coburn and Talbert’s
(2006) conclusions. In an effort to improve academic achievement of their students,
principals reported at each of the three schools to have implemented systems to use data.
This study also found that the principals were aware of their schools’ API scores, which
were created under California’s PSAA. As such, each principal was making efforts to
raise their schools’ API scores and the proficiency levels of their students.
Middle School Reform
The middle school reform section of the literature review analyzed the schools
transitioning from a junior high model to a middle school model. Researchers who are
proponents of the middle school model believe that it can better educate students and help
provide for their social and emotional needs more than a junior high school (Springboard,
2007). Research conducted by Flowers et al. (2003) echoed the findings of the
Springboard (2007) study. One of the finding of Flowers et al. (2003) was that middle
schools which emphasized the importance of teachers having common planning time are
able to produce students with higher achievement levels. The findings in this study
support the research findings of Springboard (2007) and Flowers et al. (2003). Common
93
collaboration time for teachers provided them an opportunity to meet and discuss their
student achievement data with each other. However, a new finding in this study was the
use of PLC time by teachers to identify at-risk learners using student data and then
devising support plans to help them. Teachers reported in this study that by identifying
who the at-risk students were via a careful analysis of the data, they could together devise
a support plan to address their academic, social, and emotional needs.
It is also important to note the high level use of data in PLCs by the math
department at Burbank Middle School. Unlike all other departments at the three middle
schools, Burbank’s math department carried out the continuous improvement model at a
high level. Using collaboration time, the math department created and carried out a plan
where in the classrooms, students were assessed weekly and re-teaching occurred if
students did not reach mastery of the standards. Outside of the classrooms, the math
department teachers would use collaboration time to meet and discuss their instructional
practices with each other. Teachers would help strengthen areas of weakness of their
colleagues if needed. These discussions that stemmed from student data resulted in
improved teaching practices and thus higher student achievement.
Merits of Data Use
Prior research studies have shown a link between schools which make use of data
in their decision making processes and increases in student academic achievement
(Symonds, 2004, Abbeit et al. 2007). Research conducted by Abbeit et al. (2007) at three
high achieving middle schools that practiced using data found that they had processes in
94
place where they analyzed assessment data, used an inquiry process, and adjusted
instructional planning. This research was consistent with the findings of this study. At
each of the three schools, it was found that both teachers and principals contributed at
least partial success to their schools reaching its high API due to the use of data.
Principals at two schools reported that they were working with their teachers to further
expand the use of data-driven decision making across their schools. Specifically, the
principals wanted to expand benchmark testing. The principals recognized that
benchmarks were not in place in all of their departments and were making efforts with
teachers to move toward developing and implementing them.
School Site Leadership and Data-Driven Decision Making
One finding that played a significant role in data-driven decision making at the
school site was the support and leadership the principal provided. The research cited in
the literature review on school site leadership highlights the importance of having a
strong leader in order to create a data-driven school (O’Day, Bitter, Kirst, Carnoy,
Woody, Buttles, 2004; Choppin, 2002, Supovitz & Klein, 2003). The findings in this
study were consistent with those of the literature view. At the three school sites, data-
driven decision making was supported by the principals. They support their staff by
providing teachers with access to data, giving direction through agendas, and providing
professional development related to data use. The research of Lachat and Smith (2005)
concluded that leaders must be champions of data and use it if their schools are going to
experience continuous improvement. The principals at each of the school sites were all
95
strong advocates for data and they touted its benefits for students to their teachers.
Despite Lincoln’s principal facing some opposition from teachers, she continued to work
to persuade teachers of the benefits of using assessment data. To win over those teachers
who were resistant to using data, the principal presented data on students to show their
areas of weakness. The principal believed that once teachers understood that students
had specific deficits, the resistant faculty members would be more receptive to using data
to help students.
The findings of this study in regards to leadership at the middle schools should be
noted. Each of the three principals were advocates of data-driven decision making,
however, the level of implementation differed at each school site. This study found that
data use is promoted at school sites if there is clear direction given by the principal. An
example of this would be at Harrison Middle School. The principal had a clear plan in
place for his teachers to use data. He had developed data tools for each department to use
so they could create measurable goals based on data and then create plans to achieve
them. At the other sites, this was not fully apparent. However, teacher interviews did
reveal that some departments had goals to move a certain percentage of their students to
proficiency. This study did not find any written documentation of the goals. This study
found that leadership needs to be clear and consistent in regards to data use. One way to
accomplish this, as determined by this study, is for principals to provide data tools that
help teachers focus on the task at hand.
96
Challenges of Creating a Data-Use School Culture
Research indicates that creating a culture around the use of data is necessary if it
is going to contribute significantly to improving student achievement at the school site
(Heritage & Yeagley, 2005; Earl & Katz, 2005; Datnow et al., 2007). The findings in
this study indicate that data cultures existed at the three schools. One indicator that a data
culture existed was that all teachers indicated that data played a role in their school’s high
academic achievement. In addition to this, all teachers at the three school sites reported
that they used data to drive their instruction for students. However, the degree to which
data use had been become a part of the school culture can be argued. According to the
research of Ingram et al. (2004), barriers must be overcome if a culture that is supportive
of data-driven decision making is to be established. This is consistent with the findings
of this study. Lincoln Middle School’s principal was facing challenges from some of her
teachers when benchmarks were being established. Ingram et al. (2004) research shows
this could be a result of cultural barriers such as a lack of consensus of what is
meaningful data, teacher reliance on their own anecdotal information, or teachers
disassociating their performance to that of their students. However, at the other two
middle schools, interview data from principals and teachers did not indicate any signs of
resistance to using data. However, resistance could exist at the schools and it was not
revealed in this study due to only four teachers being interviewed.
97
Building Capacity for Data-Driven Decision Making
The research literature states that professional development, which builds the
capacity of teachers, has become increasingly important in the current environment of
school accountability (Elmore, 2002). Many of the teachers commented that they did not
receive training; however, the majority of teachers reported that they did go to the
principals when they needed support with data. This study found that the principals at the
school sites worked to build capacity among their staff with data use. This was consistent
with the findings of Jandris (2001) who found that principals needed to support their
teachers with training to give them the skills necessary to use data effectively. Some
teachers also reported that they received training from the district to use the data program
called Edusoft. However, the training was not ongoing and teachers felt more was
necessary. In addition to this, interview data indicated that teachers had questions about
how comprehensive the training was. Teachers indicated that the training was focused on
the functional aspects of the program such as how to log into Edusoft and how to print
out data reports; however, the training did not teach them how to conduct data analysis
with the reports they produced. This finding supports the research by Kerr et al. (2006)
who found that many districts and schools have inadequate human and financial
resources to provide comprehensive training. Findings in this study suggest that
comprehensive and ongoing training must be carried out at either the site or district level
to support teachers in data analysis. Although principals at the school sites supported
their individual staff members when they sought support from them, teachers as a whole,
felt they had not received training in regards to data analysis.
98
Types of Data Used in Schools
Benchmark assessment data in the subjects of math and science were available to
teachers and principals at each of the three school sites. In particular, at Harrison Middle
School, benchmark assessment data existed for all subjects. However, a finding in this
study showed that teachers believed their own formative assessments to be a more
reliable measure of students’ achievement than other assessments. This finding was
consistent with the research presented in the literature review. Ingram et al. (2004) also
found that teachers gave more merit to classroom assessments as a better indicator of
achievement than standardized tests. Another finding of this study was that teachers used
formative assessment data to determine if re-teaching was necessary. This was consistent
with Heritage & Yeagley’s (2005) study which found that teachers use data from
formative assessment to adjust their instruction.
Teachers were not the only users of data in this study. Interview data from the
three school sites indicated that principals used summative data to help inform their
decisions about the placement of students in support classes if needed. This finding was
supported by Supovitz and Klein (2003) who found that data from different assessments
can be cross-referenced in order to support decisions.
A new finding in this study was the use of discipline data to improve school
climate by the principal of Burbank. He used discipline data to target interventions for
students and parents. One example he gave was using data to determine that cyber-
bullying was a problem, and to help resolve the problem, he would provide parental
education in hopes that it would help resolve the matter.
99
Processes that Support Data Use
In the literature review research cited the importance of having processes in place
to analyze student data to inform decisions. Supovitz and Klein’s (2003) research in this
area determined that the most successful schools provided a sufficient amount of time
and had processes in place to collect and analyze data. Supovitz and Klein (2003) found
that schools were able to support continual school improvement and raise the academic
achievement of their students if time and data procedures were in place. This research
supports the findings in this study. The principal at Harrison Middle School had a
process in place that closely followed Copeland’s (2003) cycle of inquiry. The principal
had each of his departments identify problems based on data, create goals based on those
problems, and implement a plan to carry them out. To accomplish this, the principal had
his departments meet in their PLCs. This supports Supovitz and Klein’s (2003) finding
about the importance of time being provided to facilitate continual improvement in a
school.
One finding in this study which was very significant for all schools was the
importance of PLC meeting time. Every Tuesday teachers were given time to meet with
their colleagues. It was during this meeting time that teachers and principals all reported
that the majority of data use and analysis took place. When meeting together, teachers
were able to share the results from their benchmarks freely with one another. This study
showed that PLC meeting time helped to promote data-driven decision making among the
teachers.
100
Implication for Future Research
For data-driven decision making to effectively take place at the middle school
level, a number of variables need to be considered. Research has shown that there are
benefits to implementing data-driven decision making within our schools, one being
higher academic achievement of students. While this study provided valuable insight
into high achieving middle schools’ experiences with data use, future research is
necessary in order to further deepen our understanding of the factors that influence it.
Listed below are suggested future areas of research on data-driven decision making at the
middle school level.
• Evaluate the structure of the middle school and its impacts on data-driven
decision making.
• Examine the impact of coring academic subjects versus non-cored subjects at the
sixth grade level to determine the effect of having fewer teachers involved in data
discussions about students and the impact on achievement.
• Examine a broad array of teachers in order to analyze the impact of department
culture to determine if it impacts the willingness to engage in data-driven decision
making using a survey.
• Time provided for data discussions among teachers. How much time is necessary
to fully carry out data-driven decision making?
• A qualitative and quantitative study of high achieving middle schools and low
achieving middle schools to determine the differences in data use, if any.
101
• Evaluate the role technology has with data use and its impact on teachers and
administrators
Implications for Policy and Practice
Taking into account the findings of this study, in conjunction with prior research
on data-driven decision making, schools that are using data in their decision processes or
are contemplating how to put it into practice in the future, should consider the following:
Provide Time for Collaboration
The most significant finding in this study was the importance of collaboration
time provided for the teachers. Administrators must schedule enough time for teachers in
order for them to identify problems, create goals, develop assessments, analyze data, and
implement next steps throughout the year. At the middle school level, time for these data
discussions must be set aside outside a teacher’s conference period. Scheduling common
time outside of teachers’ conference periods allows everyone the opportunity to
collaborate together. If time is not devoted solely for the purpose of data-discussions,
then the school runs the risk of data-driven decision making not being carried out with
fidelity.
Professional Development
For a school to effectively implement data-driven decision making, administrators
and teachers need to be thoroughly trained in the practice. It is important for school
districts to invest in their human capital and train administrators with the skills necessary
for data-driven decision making. The principal must be a person who is able to help
102
educate his/her teachers in using data to drive decisions. If the principal does not have
the knowledge to carry this out, then the practice of using data school wide will falter.
When administrators have the skills needed, they in turn must build the capacity of their
own teachers. Administrators need to provide meaningful and relevant professional
development for their teachers related to data use. Too often teachers undergo “drive-by”
trainings that are “one and done” without follow up. Administrators need to plan
comprehensive training for their staff that follows a clear and logical path to reach the
goal of data literacy. If teachers are trained properly by school administration or district
administration, it decreases the responsibilities placed on the principal and builds
capacity among the staff. By doing this, it empowers the teachers and further builds the
data culture of the school.
Benchmark Assessments
The use of school wide benchmarks should be considered in order to provide
teachers with relevant data. Common benchmark assessments provide comprehensive
data on students’ knowledge of the curriculum up to the point they are administered.
When teachers have current student performance data that is derive from common
benchmarks assessments, it allows them to reflect together on their teaching practices
during collaboration time. This in turn promotes teachers to change their instruction to
benefit student achievement. In addition to promoting better teaching practices by
providing data to teachers, common benchmark data helps to identify struggling learners.
Once at-risk students have been identified, teachers can recommend different supports to
help their students’ academic achievement.
103
School districts should also consider implementing district wide benchmark
assessments. Schools may have cultures that are resistant to school wide assessments, as
such, principals may not be as willing to institute benchmark testing. As a result of this,
schools are not able to benefit from student achievement data, thus student learning may
suffer. However, if districts have the political will to institute benchmark assessments
district wide, teachers will have valuable student performance data to drive their
discussions.
Leadership Support
The importance of the administrator can not be emphasized enough. Without a
committed principal who has a clear and comprehensive plan for instituting data-driven
decision making, a lasting school data culture will not develop. To foster a data culture
the principal needs to show consistent support for teachers in different ways. First, the
principal needs to provide teachers with access to data. One example is providing
teachers with access to data technology. Second, the principal needs to provide sufficient
time for teachers to collaborate and discuss data. Lastly, principals need to provide clear
direction for teachers by utilizing agendas and data tools. By removing barriers for
teachers and being champions of data, principals can more easily create a data friendly
culture at their school and begin to go down the path of continuous school improvement.
Conclusion
With federal and state policies placing higher levels of accountability on schools,
the implementation of data-driven decision making to raise student achievement has
104
become even more important. While this study provides pertinent and timely research
about data-driven decision making within middle schools, there is a pressing need to
conduct even more. Further research will bring to light the strengths of using student
assessment data, and therefore, it will encourage even more schools to begin to
implement it. The success of schools using well thought out data practices will rely on
the leadership and of the teachers’ willingness to improve their practice.
105
REFERENCES
Black P., & William, D. (1998) Inside the black box: Raising standards through
classroom assessment. Phi Delta Kappan, 80, 139-148.
http://www.pdkintl.org/kappan/kbla9810.htm
California Department of Education (2001). Taking center stage. Retrieved March 17,
2008, from http://pubs/cde.ca.gov/tcii/documents/tcii.pdf
Carnegie Council on Adolescent Development. (1989). Turning point: Preparing youth
for the 21
st
century. New York: Author.
Choppin, J. (2002). Data use in practice: Examples from the school level. Paper presented
at the American Educational Research Association Annual Conference, New
Orleans, LA. Retrieved March 25, 2008 from
http://www.wcer.wisc.edu/mps/AERA2002/data_use_in_practice.htm
Clark, S. N., & Clark, D. C. (1993). Middle level school reform: The rhetoric and the
reality. The Elementary School Journal, 93, 447-460.
Coburn, C. E., Honig, M. I., & Stein, M. K. (2005). What is the evidence on districts’use
of evidence? In J. Bransford, L. Gomez, D. Lam, & N. Vye (Eds.) Research and
practice: Towards a reconciliation. Cambridge: Harvard Educational Press.
Coburn, C. E., & Talbert, J. E. (2006). Conceptions of evidence use in school districts:
Mapping the terrain. American Journal of Education 112(4), 469-495.
Datnow, A., Park, V., & Wohlstetter, P. (2007). Achieving with data: How high-
performing school systems use data to improve instruction for elementary
students. A report commissioned be the NewSchools Venture Fund. Los Angeles,
CAL Center on Educational Governance, University of Southern California.
Earl, L., & Fullan, M. (2003). Using data in leadership for learning. Cambridge Journal
of Education, 33(3), 383-394. Retrieved November 4, 2007 from
http://education.vermont.gov/new/pdfdoc/pgm_assessment/summit/research_reso
urces/using_data_leadership_learning.pdf
Earl, L., & Katz, S. (2002). Leading schools in a data rich world. In K. Leithwood, Pl.
Hallinger, G. Furman, P. Gronn, J. MacBeath, B. Mulford & K. Riley (Eds.) The
second international handbook of educational leadership and administration.
Dordrecht, Netherlands: Kluwer.
EdSource (PSAA) http://www.edsource.org/pub_edfct_ayp.cfm March 29, 2008.
106
EdSource (Standards Reform) http://www.edsource.org/edu_acc.cfm March 29, 2008
EdSource (NCLB) http://edsource.org/pub_edfct_ayp.cfm
Eilers, E., M., Camacho, A., (2007). School culture change in the making. Leadership
factors that matter. Urban Education, 42(6), 616-637.
Elmore, R. F. (2002). Bridging the gap between standards and achievement, Washington,
DC, Albert Shanker Institute. Retrieved on July 12, 2003 from
http://www.nsdc.org/library/results/res11-02.elmore.html
Felner, Robert D, Jackson, Anthony W, Kasak, Deborah, Mulhall, Peter, et al. (1997).
The impact of school reform for the middle years. Phi Delta Kappan, 78(7), 528-
532+. Retrieved May 18, 2008, from Research Library Core database.
(Document ID: 11188207).
Flowers, N., Mertens, S.B., & Mulhall, P.F. (2003) Lessons Learned from More Than a
Decade of Middle Grades Research. Middle School Journal 35(2), 55-59.
Heritage, M., & Yeagley. R. (2005). Data use and school improvement: Challenges and
prospects. In J.L. Herman & E. Haertel (Eds.), Uses and misuses of data for
educational accountability and improvement (National Society for the Study of
Education Yearbook, Vol. 104, Issue 2, pp. 320-339). Chicago: National Society
for the Study of Education. Distributed by Blackwell Publishing. Retrieved April
26, 2006 from http://blackwell-synergy.com/doi/pdf/10.1111/j.1744-
7984.2005.00035.x
Ikemoto, G. S., & Marsh, J. A. (2007). Cutting through the “data driven” mantra:
Different conceptions of data-driven decision making. In P.A. Moss (Ed.),
Evidence and decision making (National Society for the Study of Education
Yearbook, Col 106, Issue 1, pp.105-131). Chicago: National Society for the Study
of Education. Distributed by Blackwell Publishing.
Ingram, D., Louis K., S., & Schroeder, R.G. (2004) Accountability policies and teacher
decision making: Barriers to the use of data to improve practice. Teachers College
Record 106(6), 1258-1287.
Jackson, A. W., & Davis, G. A., (2000) Turning points 2000: Educating adolescents in
the 21
st
century. New York: Carnegie Corporation of New York.
Juvonen, J., Le, V., Kaganoff, T., Augustine, C., & Constant, L., (2004) Focus on the
wonder years. Santa Monica, CA: RAND Corporation.
107
Kerr, K. A., Marsh, J. A., Ikemoto, G. S., Darilek, H., & Barney, H. (2006). Strategies to
promote data use for instructional improvement: Actions, outcomes, and lessons
from three urban districts. American Journal of Education 112(4), 496-520.
Knapp, M.S., Copland, M. A., Swinnerton, J.A. et al. (2006). Data-informed leadership
in education. Seattle: Center for the Study of Teaching and Policy, University of
Washington.
Lachat, M., & Smith, S. (2005). Practices that support data use in urban high schools.
Journal of Education for Students Placed at Risk, 10(3), 333-349.
Marzano, R.J. (2000). Transforming classroom grading. Alexandria, VA: Association for
Supervision and Curriculum Development (ASCD).
Merriam, S. B. (1998). Qualitative research and case study applications in education.
San Francisco, CA: Jossey-Bass Publishers.
Middle Grade Task Force. (1987). Caught in the middle: The task of educational reform
for young adolescents in California schools. Sacremento, CA: California State
Department of Public Instruction.
O’Day, J., Bitter, C., Kirst, M., Carnoy, M., Woody, E., & Buttles, M., et al. (2004).
Assessing California’s accountability system: Successes, challenges, and
opportunities for improvement (Policy Brief 04-2). Berkeley, CA: Policy Analysis
for California Education. Retrieved March 17, 2008, from
http://pace.berkeley.edu/policy_brief.04-2.pdf
Patton, M. Q. (2002). Qualitative research & evaluation methods (3
rd
ed.). Thousand
Oaks, CA: Sage Publications, Inc.
Rumberger, R. W., (1995). Dropping out of middle school. American Educational
Research Journal 32(3), 583-625.
Schmoker, M., (2003). First things first: Demystifying data analysis. Educational
Leadership, Vol. 60, No. 5, 2003.
Schmoker, M., (2004). The tipping point: From feckless reform to substantive
instructional improvement. Phi Delta Kappan 85(6), 424-432.
Sharkey, N., & Murname, R.J., (2006) Tough choices in designing a formative
assessment system. American Journal of Education 112(4), 572-588.
Slavin, R. E., (2002) Evidence-based educational policies: Transforming educational
practice and research. Educational Researcher, 31(7), 15-21.
108
Spillane, J. P., & Miele, D. B., (2007) Evidence in practice: A framing of the terrain.
National Society for the Study of Education Yearbook.
Springboard Schools (2007) Balancing Act: Best practices in the middle grades. San
Francisco, CA. Springboard.
Supovitz, J., & Klein, V. (2003). Mapping a course for improved student learning: How
innovative schools systematically use student performance data to guide
improvement. University of Pennsylvania Graduate School of Education:
Consortium for Policy Research in Education.
Symonds, K. W. (2003). After the test: How schools are using data to close the
achievement gap. San Francisco: Bay Area School Reform Collaborative.
Togneri, W. & Anderson, S. E., (2003). Beyond islands of excellence: What districts can
do to improve instruction and achievement in all schools. Washington, DC:
Learning First Alliance.
The National Commission on Excellence in Education. (1983). A nation at risk: The
imperative for educational reform. Retrieved March 20, 2008, from
http://www.ed.gov/pubs/NatAtRisk/index.html
Wayman, J.C., Midgley, S., & Stringfield, S. (2006). Leadership for data-based decision
making: Collaborative data teams. In Danzig, A., Borman, K., Jones, B., &
Wright, B. (Eds), New models of professional development for learner centered
leadership. Erlbaum. Retrieved May 24, 2007 from
http://edadmin.edb.utexas.edu/datause/papers/Wayman,%20Midgley%20&%20St
ringfield.PDF
Wayman, J. C., & Stringfield, S. (2006). Data use for school improvement: School
practices and research perspectives. American Journal of Education 112(4), 463-
468.
Woody, E., Buttles, M., Kafka, J., Park, S., Russell, J. (2004) Educators respond to
accountability (pp.11-16 and 43-51). Berkley, CA:Policy Analysis for California
Education. Retrieved December 5, 2006 from
http://www.gse.berkley.edu/research/PACE/ERAP Report-WEB.pdf
109
APPENDIX A
Principal Interview Protocol
Participant’s Name: __________________________ Date: __________________
Position: _______________________________________________________________
[Introduction: Begin with a few minutes of explaining the study, who you are, and the
purpose of the study. Explain that while the interview will be taped, their responses are
strictly confidential. Let them know if there is something they would like to say off tape,
they can inform you and the recorder will be shut off for their comment. Also, let them
know the approximate length of the interview and ask if they have any specific questions
before beginning.]
VI. Background/Context
1. Could you tell us a little about the students and community that you serve?
2. How long have you been at this school/district? What is your prior experience
and training?
3. What are your duties and responsibilities, especially around data? (research
question 1)
4. When/why did the use of data become an important part of you school
improvement process? (research question 1)
II. Data Sources
1. What types of student achievement data does the school collect and examine most
frequently? (benchmarks, state assessments) How often? How do they use them?
(Research question 3 & 4)
2. What types of data do you find most useful in making decisions about your school
and curriculum? Why? (benchmarks, standardized tests) (research question 3 & 4)
3. Do you receive data from the district? What kinds of data? (research question 3)
110
III. Leadership
1. How does site administration/district administration support teachers in using
data? (research question 1)
2. How do you promote data use in the school? (research question 1)
3. Does your school provide time to analyze data individually or with other
teachers? Is this effective? (research question 1)
IV. Structural Support
1. What kinds of support are important in helping teachers look at and use data for
instructional improvement? (research question 1 & 2)
2. Do you have a person on staff to support teachers in their use of data? What are
their role/responsibilities? (research question 2)
3. Does your school have informal and/or formal grade-level teams, small learning
communities, or professional development opportunities that offer teachers
forums for talking about instruction and student achievement? How are these
meetings organized? (focus on uses of time) (research question 1 & 2)
4. Are any tools provided to teachers to help analyze their student data? (request
copy) (research question 1)
5. Has your district/school sponsored professional development for principals and/or
teachers that focuses on using data to make decisions? In what specific areas? Is
the professional development offered mandatory or voluntary? (research question
2)
6. What further types of training/support do you think will be needed to help
teachers use data for instructional improvement? (research question 2)
VI. Uses of Data
111
1. How do you use student performance data in making your decisions about your
school? How frequently do you do this? Example? (goal setting?) (research
question 4)
2. How do teachers at your school use data? Can you provide a specific example?
(research question 4)
3. What do you feel are the most useful ways to use data? (research question 4)
4. How would you describe a teacher who effectively uses data compare to one that
does not? (research question 4)
5. How do you think your school can improve upon its data practices? (research
question 4)
V. Culture of Data Use
1. What are your beliefs about the importance of data use? Where did they come
from? (research question 2)
2. What are the expectations of the district/principals/teams for using data? How do
you know what they are? (research question 2)
3. Do teachers share their achievement data with one another? Why? Are they
comfortable doing this? (research question 2)
4. What are the challenges that your school faces in using data? How have these
challenges been dealt with? (research question 2)
5. What do you think teachers’ beliefs are about using data? (research question 2)
6. What do you feel about using data overall? Does it help or hinder students
achievement and why? (research question 2)
112
7. Is there anything else you think might be making a difference in raising test scores
in this school? (research question 2)
Concluding Remarks/Questions: Is there anything else we should know? Thank them for
their cooperation and time. Inform them we will share our report with them once it is
done and that we might need to contact them for follow-ups]
Document Request:
• Samples of work to demonstrate data-driven decision making?
• Sample reports from the district
• Sample reports given to teachers/leads/coaches
• Sample reports you use and find most helpful
• Sample action plans based on data – school and/or classroom level
• Anything else you think would be helpful for us to look at to get a sense of how
you use data?
113
APPENDIX B
Teacher Interview Protocol
Participant’s Name: __________________________ Date: __________________
Position: _______________________________________________________________
[Introduction: Begin with a few minutes of explaining the study, who you are, and the
purpose of the study. Explain that while the interview will be taped, their responses are
strictly confidential. Let them know if there is something they would like to say off tape,
they can inform you and the recorder will be shut off for their comment. Also, let them
know the approximate length of the interview and ask if they have any specific questions
before beginning.]
VII. Background/Context
1. What subject & grade do you teach? How long have you been at this school?
What is your prior experience and training? (background)
2. Could you tell us a little about the students and community that you serve?
3. Would you attribute using data as an important reason for your school’s success
in achieving it AYP and the growth of its API? (research question 4)
II. Data Sources
1. What types of student achievement data does the school collect and examine most
frequently? (benchmarks, state assessments) How often? How do they use them?
(Research question 3 & 4)
2. Do you use data in your classroom? How often? (research question 4)
3. What types of data do you find most useful in making decisions about your
instruction and curriculum? (benchmarks, assessments, and projects) (research
question 3)
4. Do you receive data from the district? What kinds of data? (research question 3)
114
III. Leadership
1. How does site administration/district administration support teachers in using
data? (research question 1)
2. How does your administration promote data use in the school? (research question
1)
3. Do they provide you time to analyze data individually or with other teachers? Is
this effective? (research question 1)
IV. Structural Support
1. What kind of support do teachers have in using data to change their instruction?
(research question 1 & 2)
2. What meetings take place over data? What happens at the meetings and what do
you have to do to prepare for them? (research question 2 & 4)
3. Do you use any type of worksheet to help you analyze your data? (request copy)
(research question 2)
4. Have you attended any training on using data? Who provided them to you? Were
you mandated to attend or volunteered? Was it valuable to you? (research
question 1 & 2)
5. If you need help with data, who do you go to and why? (research question 1 & 2)
6. Have you run into any problems with data? What happened and why? (research
question 1 & 2)
7. What further types of training/support do you think will be needed to help
teachers use data for instructional improvement? (research question 2)
VI. Uses of Data
115
1. Can you think of an instance when you used student performance data to change
your instruction? How frequently doe this occur? Example? (research question 4)
2. What do you use data for? (research question 4)
3. What does your school use the data for? Student assignments? Goal setting?
(research question 4)
4. What do you feel are the most useful ways to use data? (research question 4)
5. How would you describe a teacher who effectively uses data compare to one that
does not? (research question 4)
6. How do you think your school can improve upon their data practices? (research
question 4)
V. Culture of Data Use
1. What are your beliefs about the importance of data use? Where did they come
from? (research question 2)
2. What are the expectations of the district/principals/teams for using data? How do
you know what they are? (research question 2)
3. Do teachers share their achievement data with one another? Why? Are they
comfortable doing this? (research question 2)
4. What are the challenges that your school faces in using data? How have these
challenges been dealt with? (research question 2)
5. What do believe are teachers’ beliefs about using data? (research question 2)
6. What do you feel about using data overall? Does it help or hinder students
achievement and why? (research question 2)
116
7. Is there anything else you think might be making a difference in raising test scores
in this school? (research question 2)
Concluding Remarks/Questions: Is there anything else we should know? Thank them for
their cooperation and time. Inform them we will share our report with them once it is
done and that we might need to contact them for follow-ups]
Document Request:
• Samples of work to demonstrate data-driven decision making?
• Sample reports from the district
• Sample reports given to teachers/leads/coaches
• Sample reports you use and find most helpful
• Sample action plans based on data – school and/or classroom level
• Anything else you think would be helpful for us to look at to get a sense of how
you use data?
117
APPENDIX C
Harrison Data Forms
118
119
120
121
122
123
124
125
APPENDIX D
Code List
The following is a list of codes used to analyze the data gathered through the interview
process:
Beliefs in Data
Challenges to Data Use
Data Tools
District Support
Frequency of Assessments
How Data are Used
Leadership Data Preference
Leadership Responsibilities
Leadership Support
School Data Culture
School Structural Support
Type of Assessments
Types of Data Used
Abstract (if available)
Abstract
Over the last decade, schools in California have been experiencing the dramatic effects of NCLB and the Public School Accountability Act of 1999. Both the state and federal legislation have called for increased student achievement partnered with accountability. As a result of such policies, data and accountability have grown to hold a central place in large-scale school reform. With little guidance, many middle schools are now being held accountable for having to put into practice data-driven decision making in order to help improve teaching and learning.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Holding on and holding out: why some teachers resist the move toward data-driven decision making
PDF
District level practices in data driven decision making
PDF
Does data-driven decision making matter for African American students?
PDF
A school's use of data-driven decision making to affect gifted students' learning
PDF
How urban school superintendents effectively use data-driven decision making to improve student achievement
PDF
Teacher education programs and data driven decision making: are we preparing our preservice teachers to be data and assessment literate?
PDF
The pursuit of equity: a comparative case study of nine schools and their use of data
PDF
How districts prepare site administrators for data-driven decision making
PDF
Examining the applications of data-driven decision making on classroom instruction
PDF
Multiple perceptions of teachers who use data
PDF
Beyond the numbers chase: how urban high school teachers make sense of data use
PDF
School leaders' use of data-driven decision-making for school improvement: a study of promising practices in two California charter schools
PDF
Data-driven decision-making practices that secondary principals use to improve student achievement
PDF
Do we really understand what we are talking about? A study examining the data literacy capacities and needs of school leaders
PDF
An examiniation of staff perceptions of a data driven decision making process used in a high performing title one urban elementary school
PDF
Charter schools, data use, and the 21st century: how charter schools use data to inform instruction that prepares students for the 21st century
PDF
Successful resource allocation in times of fiscal constraint: case studies of school-level resource use in southern California elementary schools
PDF
Closing the achievement gap in a high performing elementary school
PDF
The implementation of data driven decision making to improve low-performing schools: an evaluation study of superintendents in the western United States
PDF
Designing school systems to encourage data use and instructional improvement: a comparison of educational organizations
Asset Metadata
Creator
Nygren, Lars A.
(author)
Core Title
Data use in middle schools: a multiple case study of three middle schools’ experiences with data-driven decision making
School
Rossier School of Education
Degree
Doctor of Education
Degree Program
Education (Leadership)
Publication Date
08/07/2009
Defense Date
04/22/2009
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
data analysis,data driven decision making,evidence based,OAI-PMH Harvest
Place Name
California
(countries)
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Datnow, Amanda (
committee chair
), Brewer, Dominic J. (
committee member
), Love, Laurie (
committee member
)
Creator Email
lnygren5@hotmail.com,lnygren5@yahoo.com
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-m2534
Unique identifier
UC1154032
Identifier
etd-Nygren-2993 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-176521 (legacy record id),usctheses-m2534 (legacy record id)
Legacy Identifier
etd-Nygren-2993.pdf
Dmrecord
176521
Document Type
Dissertation
Rights
Nygren, Lars A.
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Repository Name
Libraries, University of Southern California
Repository Location
Los Angeles, California
Repository Email
cisadmin@lib.usc.edu
Tags
data analysis
data driven decision making
evidence based