University of Southern California Dissertations and Theses

Data -driven strategies to improve student achievement: A cross-case study of four California schools

(USC Thesis Other)

Data -driven strategies to improve student achievement: A cross-case study of four California schools

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content DATA-DRIVEN STRATEGIES TO IMPROVE STUDENT ACHIEVEMENT:
A CROSS-CASE STUDY OF FOUR CALIFORNIA SCHOOLS
by
Alan Stephen Lewis
A Dissertation Presented to the
FACULTY OF THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(EDUCATION)
May 2005
Copyright 2005 Alan Stephen Lewis
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UMI Number: 3180366
Copyright 2005 by
Lewis, Alan Stephen
All rights reserved.
INFORMATION TO USERS
The quality of this reproduction is dependent upon the quality of the copy
submitted. Broken or indistinct print, colored or poor quality illustrations and
photographs, print bleed-through, substandard margins, and improper
alignment can adversely affect reproduction.
In the unlikely event that the author did not send a complete manuscript
and there are missing pages, these will be noted. Also, if unauthorized
copyright material had to be removed, a note will indicate the deletion.
®
UMI
UMI Microform 3180366
Copyright 2005 by ProQuest Information and Learning Company.
All rights reserved. This microform edition is protected against
unauthorized copying under Title 17, United States Code.
ProQuest Information and Learning Company
300 North Zeeb Road
P.O. Box 1346
Ann Arbor, Ml 48106-1346
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
DEDICATION
To my mother, the first Ph.D. on both sides of the family, to my father whose
support helped me through the formative years, to Rachel Taylor, my mother’s
mother, whose memory always inspires me to do more than I might normally do, and
to my wife who reminded me how much I like learning and who painstakingly edited
this endeavor through its multiple revisions.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ACKNOWLEDGMENTS
A special thanks to Dr. David D. Marsh, my dissertation chair for selecting
me to be a part of the Ed.D. parallel dissertation process that resulted in the
foundation of my dissertation. Additional acknowledgments for Dr. Myron Dembo
whose humor and attention to detail helped maintain the quality of this final project,
and to Dr. Thomas Cummings whose guidance assisted me in investigating credible
business sources for this dissertation.
Many obstacles continued to cross my path as I kept my eye on the goal of
completing the requirements for a Ph.D. Computer problems were an incessant
thorn in my side. For this reason, it would be remiss of me to not express my
appreciation for the timely computer assistance that was given to me by both Shimon
Rothchild and Michael Knopf. Their knowledge made my travels much easier. I
only wish they had been available from the very beginning.
Finally, I would like to acknowledge all of the professors who played a role
in my graduate school education. Thank you for believing in me and encouraging
me to continue along the path that now culminates with a Ph.D. I couldn’t have done
it without you and I will make an honest effort to use the knowledge base I now have
to be a contributor to my chosen profession, education
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
TABLE OF CONTENTS
Page
DEDICATION ii
ACKNOWLEDGMENTS II
LIST OF TABLES v
LIST OF FIGURES b
ABSTRACT , x
CHAPTER 1 - OVERVIEW OF THE STUDY 1
Introduction 1
The State of Public Schools 1
Reform Efforts 5
Statement of the Problem 7
Purpose of the Study 9
The Research Questions 9
The Importance of the Study 10
Limitations, Delimitations, and Assumptions 11
Limitations 11
Assumptions 12
Definitions 12
Organization of the Study 15
CHAPTER 2 - INTRODUCTION 17
Selected Research on Systemic Organization Change 21
Representative Education Change Research 22
Reflection on First Seven Researchers 33
Achievement Gaps and Improved Student Achievement 34
A Business Lens on School Efforts to Improve Student 36
Key Themes from Education and Business Change Research 51
Representative Educational Research on Data-Drive Change 52
Key Themes from Education and Business Research on
Data-Drive Change 61
Exemplars of Successful Change Efforts in Education 63
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Page
Exemplars of Successful Change Efforts in Business 75
Overview of Critical Characteristics to Systemic Change 89
Research-based Analysis Variables 90
CHAPTER 3 - METHODOLOGY 91
Introduction 91
Sampling and Population Descriptions 95
■ Cross-case Study Instrumentation 97
Data Collection and Analysis 99
Obstacles to Cross-Case Study Data Analysis 100
CHAPTER 4 - THE FINDINGS 101
Introduction 101
Findings by Case Study and by Research Question 103
Comparative Review of Findings 161
Characteristics between the Case Studies in the Use of Data 162
Discussion of Findings 175
Links to Research 178
CHAPTER 5 - SUMMARY AND IMPLICATIONS 182
Introduction 182
Restatement of Study’s Purpose ’ 182
Review of Analysis Procedures 183
Restatement of Research Design Limitations 184
Summary of Findings 185
Implications 186
REFERENCES 191
APPENDICES 199
Appendix A- Major Research Themes by Researchers 200
Appendix B- Stages of Concern and Teacher Questionnaire 204
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
LIST OF TABLES
Table 1
Table 2
Table 3
Table 4
Table 5
Table 6
Table 7
Table 8
Table 9
Table 10
Table 11
Table 12
Table 13
Page
Student Demographics in Each Case-Study School and
In Each School District 96
Teacher Questionnaire Statements 17-22 104
Case Study #1 Data Analysis Summary (Southern
Elementary School)—Training Implementation 110
Case Study #1 Data Analysis Summary (Southern
Elementary School)—Data Collection 111
Case Study #1 Data Analysis Summary (Southern
Elementary School)—Data Analysis 112
Case Study #1 Data Analysis Summary (Southern
Elementary School)— Goals Implementation 113
Case Study #2 Data Analysis Summary (Divine
Elementary School)—Training Implementation 123
Case Study #2 Data Analysis Summary (Divine
Elementary School)—Data Collection 124
Case Study #2 Data Analysis Summary (Divine
Elementary School)—Data Analysis Implementation 126
Case Study #2 Data Analysis Summary (Divine
Elementary School)— Goals Implementation 127
Case Study #3 Data Analysis Summary (South
High School)—Training Implementation 138
Case Study #3 Analysis Summary (South High
School)—Data Collection Implementation 139
Case Study #3 Data Analysis Summary (South
High School)—Data Analysis Implementation 140
VI
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 14
Table 15
Table 16
Table 17
Table 18
Table 19
Table 20
Table 21
Table 22
Table 23
Table 24
Table 25
Table 26
Table 27
Table 28
Case Study #3 Data Analysis Summary (South
High School)—Goals Implementation
Responses to Selected Teacher Questionnaire
Statements
Case Study #4 Data Analysis Summary (Leno
High School)—Training Implementation
Case Study #4 Data Analysis Summary (Leno
High School)—Data Collection
Case Study #4 Data Analysis Summary (Leno
High School)—Data Analysis
Case Study #4 Data Analysis Summary (Leno
High School)— Goals
Data-Use Training Characteristics across
Case-Studies— at District Level
Data-Use Training Characteristics across
Case-Studies—at School Site
Data-Use Training Characteristics across
Case-Studies— Teacher Perceptions
Data Collection Characteristics across
Case-Studies—at District Level
Data Collection Characteristics across
Case Studies— at School Site
Data Collection Characteristics across
Case-Studies—Teacher Perceptions
Data Analysis Activities across Case
Studies—at District Level
Data Analysis Characteristics across
Case Studies—at School Site
Data Analysis Characteristics across
Case Studies—Teacher Perceptions
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 29 Student Achievement Characteristics
Across Case Studies— State and School Goals 170
Table 30 Leadership Characteristics Across Case-Studies 172
Table 31 Characteristics of High Performing Organization 174
Table 32 Local Control and Curriculum Alignment and
Data Analysis Characteristics Across Case Studies 176
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
LIST OF FIGURES
Figure 1
Figure 2
Conceptual framework of relationships between the
variables used to analyze the case studies.
Implementation Analysis Scales
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ABSTRACT
Public Schools in the United States are in the midst of a reform that is
attempting to significantly improve student achievement in all schools, for all
students. This cross-case study reviews research completed on four public schools
that operate within the context of California’s accountability system. These multi
ethnic urban schools, two elementary and two senior-high, share a reputation for
effectively utilizing student achievement data to continuously improve annual results
on state assessments.
This secondary study compares and contrasts the data-driven improvement
strategies central to efforts to improve student achievement. The analysis is guided
by four research questions which focus on the frequency and general characteristics
of data-driven improvement trainings, data collection and analysis o f student
performance data. The fourth research question investigates the degree to which
each of the four schools attain their student achievement goals.
This study revealed that all four schools had data related trainings ranging
from four times per year to one session each week. In addition to the state-mandated
assessments, all schools collected and analyzed student performance data in multiple
subject areas throughout the school year. All schools in this study showed
measurable improvement in state assessment results during the years their data-
driven improvement strategies were in place. In spite of this, most of the schools
x
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
shared the teacher concerns for more training and more immediate access to student
data.
This cross-case study also looks at the implementation of data-driven
improvement strategies from the perspective of successful improvement in high
performance organizations as well as the leadership qualities that facilitate
improvement. There were distinct differences among the schools in the areas of
developing a broad instructional leadership base and an active and visible
involvement of management in the improvement process. Site management at the
secondary schools appeared to be more involved in the improvement efforts than did
those at the elementary schools.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 1
OVERVIEW OF THE STUDY
Introduction
The State of Public Schools
Widespread low performance of students in urban public schools remains the
standard throughout the United States. International, national, and state student
performance data characterizes the relative academic significance of “low” student
performance in the United States. Studies of the results from several standardized
exams provide insight in to the performance of U.S. students in an international,
national, and state-specific context.
Internationally, the Third International Mathematics and Science Study
(TIMSS) provides a recent and systematic review of student achievement in science
and math among 41 participating countries, including the United States (Blank &
Wilson, 2001). Most analysts of the TIMSS results concur that the U.S. does not
have a “world-class” curriculum in math and science. Students from Singapore,
Korea, Japan, the Czech Republic and Hungary significantly outperform U.S.
students on the TIMSS exams. Schmidt, Houang, and Cogan (2002) further
underscore the relative weakness of student performance in the United States when
they report a previous study of TIMSS data by Martin, Mullis, Gregory, Hoyle, and
Shen (2000) that concludes that the achievement gap between U.S. students in high
performing schools and low performing schools is greater than in comparable
schools in most of the other TIMSS countries.
1
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Nationally, the National Assessment of Education Progress (NAEP) provides
data on 4lh , 8th , and 12tn grades in 44 states that describe student proficiency levels in
math and reading. The NAEP results portray a range of performance that runs from
67% of students testing below grade-level (Blank & Wilson, 2001) to as high as 80%
of students below grade-level proficiency (Bowslier, 2001).
California is an excellent example of a state whose public schools are
grappling with the multiple variables such as low income and large populations of
English Language Learners. The California Department of Education (2003) reports
2002 data that indicate, depending upon grade level, 47% to 66% of students are
below state goals in reading and 38% to 50% are below state goals in mathematics.
To place California student performance in perspective nationally, the average
NAEP test results rank California third from the bottom of the 44 states participating
in NAEP testing (Grissmer, Flanagan, Kawata, & Williamson, 2002). Within the
participating NAEP states Texas is often compared to California because of its close
match to key variables, such as student family income, language, and educational
characteristics. Grissmer et al. (2002) report Texas ranked 27th out of 44 NAEP
tested states. The end result is that even when California is compared to a state with
similar variables the performance of California students ranks near the bottom.
David Berliner (2001) counteracts some of the gloom and doom of the state
of public education by stating that the reported declines in The Scholastic Aptitude
Test (SAT) and other standardized test such as the California Achievement Test
(CAT) and the Iowa Test of Basic Skills (ITBS) are not as dramatic as a cursory
2
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
giance might communicate. With respect to SAT scores, Berliner explains that only
a decline of 3.3% has occurred over the course of 25 years. He contends that this
drop, which amounts to only about 55 fewer items answered correctly, is negligible
considering the large numbers of low income and minority students who now
participate in SAT testing. Berliner also reports that all of the minority subgroups
increased their SAT scores from 1976 to 1990. As far as norm-referenced tests are
concerned, Berliner states that because they are re-normed about every 7 years, it is
actually more difficult to even maintain scores because the bar is continually raised.
Berliner’s (2001) remarks are well taken; however, they do not adequately
compensate for the preponderance of evidence that significant numbers of students in
our public schools are not learning at the levels they should be. Nevertheless,
Berliner’s insights are important because they are part of a growing body of research
that represents progress in improving student achievement in all student populations.
As an example, multiple year results of California norm-referenced exams
(California Department of Education, 2003) confirm that significant gains are now
being made at the elementary levels. These student performance gains in California
are not yet apparent in the secondary grades. Furthermore, Delaine Eastin
(California Department of Education, 2001), the former California State
Superintendent, acknowledges that the traditional achievement gaps between
Caucasian and most minority students still continues. This trend, coupled with high
percentages of students who are not meeting minimal grade-level competencies,
continue to mirror the achievement concerns expressed in the Nation at Risk report.
3
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
These negative factors, even in the face of improvement, also act as a critical
evaluation of the effectiveness of educational reforms engendered 20 years ago by
the Nation at Risk report,
Bowsher (2001) presents additional statistics that disclose devastating
repercussions of an inadequate K-12 public school system. He cites 700,000
illiterate high school graduates per year, 80% of colleges with remedial reading and
math programs, and millions of dollars spent per year by businesses to re-teach entry
level applicants.
The impact of a poor education on the workforce and the employment future
of many minority populations further underscores the Nation at Risk’s premonition
that an eroding education system threatens the economic fabric of the U.S. society
(Bowsher, 2001). A weak system of public education affects even more acutely the
future of minority populations since they represent the largest workforce population
growth in the United States (Marshall, 1992). These are the very students that are
not well served by our current educational structures. Marshall underscores the
importance of this statistic when he reports that Black and Hispanic students have the
two highest dropout rates when American Indians are excluded from the equation.
Additionally, he reports that the academic achievement gap between Caucasians and
Blacks and Hispanics persists. This data significantly counteracts Berliner’s (2001)
views regarding the academic progress made by minority students in the United
States.
4
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
To add to the concerns regarding public education, Marshall and Tucker
(1992) state that non-college-educated students will only be able to obtain unskilled
and low paying positions. They also indicate that the level of expertise of U.S.
workers is far below entry level workers in other industrialized societies such as
Japan and Germany. Marshall and Tucker suggest that the United States can only
maintain its competitive edge in the global market place by investing in the
intellectual development of its workforce. They believe that quality education is the
linchpin of a prosperous economic future. Hence, they submit that the retooling of
the American education system is a necessity to belter meet the business challenges
of the 21st century.
Reform Efforts
The need for change in our K-12 education system continues to be
acknowledged by policymakers, educators, and researchers. President George Bush,
Sr., Bill Clinton, and George W. Bush are all known for publicly expressing their
concern regarding the state of public education. George Bush, Sr. and Bill Clinton
are both known for their governors’ summits of 1987 and 1996, respectively
(Marzano & Kendall, 1998). While George W. Bush’s No Child Left Behind
legislation (U.S. Department of Education, 2001) is impacting ail major public
school systems throughout the United States.
It is no coincidence that widespread political concern regarding the quality of
our public education is occurring while many change efforts are being attempted to
dramatically improve the quality of public education in the United States. Bowsher
5
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(2001) catalogues more than 50 programs to improve student achievement. He
separates these efforts into three categories. He labels one category as “low-cost,
quick-fix” programs, another category as “high-cost, long-term,” and the third
program as “alternatives and outside of the traditional” programs. Adopt-a-School,
Site-Based Management, and smaller schools are example of the first category.
Longer school days, more instructional days, smaller class sizes and school
desegregation represent the second category. Finally, vouchers, charter schools, and
magnet schools are examples of the third category. Bowsher (2001) reviews each
one of these reform efforts and provides explanations as to why they are ineffective
methods of transforming the quality of education received by public school students.
In general, he concludes that most of these efforts result of piecemeal attempts to
make change that do not cause significant structural and cultural change to the
education institution.
In response to the continued awareness of policymakers that a major
overhaul of the K-12 public education system is in order, the establishment of state
curriculum standards in 49 states now dominates educational change agendas
(Marzano & Kendall, 1998). Standards-based reform is heralded as the vehicle that
will affect major changes to the traditional school system (Marshall & Tucker, 1992;
Mid-Continental Regional Educational Laboratory, 1994; Tucker & Codding, 1998).
The salient areas to be impacted by standards-based education include curriculum,
learner-centered instruction, assessment, collaboration, governance, staff
development, parent and community involvement, and reallocation of resources.
6
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
California’s entrance in to standards-based education in 1996 (California
Department of Education, 1999) marks one of the earliest efforts by a state to
implement standards-based reform for every grade-level (Kendall & Marzano, 1997).
California’s implementation of standards-based instruction also represents a process
still in its infancy, a process in which there is a great deal of hope but as of yet with
few dependable implementation guidelines, despite its early beginnings.
Statement of the Problem
Systemic change that leads to high levels of student achievement with all
student populations continues to elude most schools in K-12 public education in the
United States. Echoes from the National Commission on Excellence in Education’s
1983 report, “A Nation At Risk: The Imperative for Education Reform” (as cited in
Education Manifesto, 1998), still reverberate loudly through the halls of policy
makers, educators, and the schools for which they are responsible. Evidence is
increasingly abundant that students from low-income families and all minority
students can succeed academically. Schmoker (2001) and Reeves (2000) provide a
list of schools that represent the success of historically poor performing students in a
public school setting. Unfortunately the evidence of change that positively impact
students in our K-12 public schools is for the most part school specific and does not
represent entire districts.
The dearth of examples of district-wide continuous high-level student
academic achievement exemplifies the lack of know-how with respect to realizing
systemic improvement in student achievement throughout entire school districts.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Much is known about characteristics of high performing schools that are beating the
achievement odds (Schmoker, 2001). Characteristics, however, are only the end
results of processes. It is these processes that remain unknown to most schools and
school districts. Hence, public school districts are still unable to demonstrate an
ability to transfer the success of a few individual schools throughout an entire
district. This inability to manifest widespread improvement in student achievement
is the primary problem facing education today.
Although current reform efforts purport to address systemic Improvement of
student performance in public schools, a recent synthesis of standards-based reform
and accountability by Chatteij i (2002) concludes that research on systemic school
reform generally utilizes non-systemic designs and therefore fails in its ability to
promote systemic reform. The lack of systemically designed research is an ancillary
problem. The need for education to develop the ability to foster a learning
environment that has the skills to investigate and implement the processes that lead
to continuously improving performance is one that educational research must
provide. By this means research can better assist schools and districts in the
systemic improvement they so passionately desire.
In summary, the problem addressed by this cross-case study is circumscribed
by three interconnected issues: First, public schools are still unable to produce high
levels of student achievement throughout schools and school systems; second, for the
most part, the research that studies efforts to improve student achievement system-
wide is non-systemic by design, and third, as a result, the research provides little
8
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
assistance for educators to understand the necessary processes that lead to successful
change implementation.
Purpose of the Study
This study provides a structure that identifies the critical variables to lead to
systemic change. Such an analytical framework enhances the ability to target critical
areas in the educational system so that innovation implementation is increased and
the momentum of improved student performance is maintained or accelerated.
Unlike a great deal of school reform research this study’s approach includes the
multidimensional and interconnected nature of organizations in its scope. This study
inspects successful change efforts in a variety of public school settings and looks at
the strengths and weaknesses of those schools as they relate to continuous
improvement in student performance.
The Research Questions
This research study is guided by the following four research questions:
1. How frequently are data-driven improvement trainings provided? What
are the characteristics of those trainings?
2. How often is student performance data collected for analysis? What are
the characteristics of the data collection process?
3. How often does the school instructional staff meet in teams to
collaboratively analyze student performance data? W hat are the
characteristics of the analysis?
9
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
4. To what degree were school student improvement goals attained since the
implementation of data-driven improvement strategies?
The Importance of the Study
This study contributes to the existing body of reform research by focusing on
basic implementation variables and processes that lead to systemic innovation and
improved student achievement. The framework developed and applied in this study
is a tool that can be employed to counter education’s resistance (Fullan, 1993) to
substantive change. This tool can be used by district support personnel and school
site educators to analyze the relative health of efforts to improve student
achievement, as well as provide a means of stimulating and maintaining continuous
improvement.
In the same vein, legislators and members of boards of education continue to
miss the mark of systemic implementation with their initiatives. The tradition of
policy makers and school boards to fashion generic policies for all situations
exacerbate the disconnect between generic designs and local needs. More
importantly, politically shaped innovations tend to ignore the interconnected and
multi-leveled nature of organizations. This includes the critical component of
motivating people to change. To underscore the inherent drawback of top-down
only, politically fashioned policies, McDonnell and Elmore (1991) present a
conceptual framework of policy instruments that provides a detailed description of
the assumptions and consequences of policy instruments. An inspection of their
conceptual framework clarifies several of the shortcomings of the policy instruments
10
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
employed by policymakers. The framework reveals a policy assumption base that
involves such restrictive consequences as coercion, uniformity, and adversarial
relations, all of which are antithetical to the promotion of systemic improvement.
The importance of this study for researchers is that it employs an analytic
foundation that looks systemicaily at change based on examples of successfully
initiated and sustained change in schools and in a variety of non-school and school
settings. Drawing upon multiple sources serves to strengthen the link between
research findings and practice. This study contributes to broadening the research
database and thereby improves the ability of practitioners to build their capacity for
maximizing the implementation of innovations.
Limitations, Delimitations, and Assumptions
Limitations
This cross-case study analyzes the data of four non-experimental case studies
of schools located in California. The non-experimental design of the case studies
reviewed in this study and the small number of cases, in addition to the limited
geographic location of the school sites, severely restrict the generalizability of the
findings and the ability to describe causal relationships. Another limitation is that
the design of the case studies does not ensure the disclosure of all pertinent data such
as multiple years of API scores and complete teacher responses to questionnaires.
Delimitations
The explicit purpose of the four dissertations was to examine the district’s
design for using student performance data and how that design was successfully
11
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
translated at a selected school site, The research questions in the four case studies are
different from the questions in this study. As a result, answers to the questions in
this cross-case study must be largely sought from the substantial amount of
qualitative data presented in each case study.
Assumptions
This study assumes that the data in the four case studies was collected and
documented accurately using standard procedures and instruments common to all
four studies. It also assumes that the schools studied are representative of urban
districts based on parameters of student population diversity, a significant number of
English Language Learners, and students from low socioeconomic neighborhoods.
The final two methodological assumptions of this study are that all four schools
demonstrate evidence of continuous student achievement and that the districts they
belong to or the schools themselves have formal designs for improving student
achievement by using student performance data.
One conceptual assumption of this cross-case study is that all of the schools
operate within the realm of larger external contexts that influence the ultimate
outcome of student performance. It is also assumed that the variables investigated in
this study represent key research-based variables over which schools and districts
have meaningful control.
Definitions
The following definitions are provided to clarify the meaning of vocabulary
used in this study:
12
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Accountability: The notion that people or an organization should be held
responsible for improving student achievement and should be rewarded or
sanctioned for their success or lack of success in doing so (EdSource Online, 2001).
Achievement: The level of student learning as evidenced in results from state
and district mandated assessments.
Alignment: The degree to which assessments, curriculum, instruction,
textbooks, and other instructional materials, teacher preparation and professional
development, and systems of accountability all reflect and reinforce the educational
program’s objectives and standards (EdSource Online, 2001).
Benchmark: Reachable (achievement) targets at various grade levels or ages
(Carr & Harris, 2001).
Content Standards: Broadly stated expectation of what students should know
and be able to do in particular subjects and grade-levels (Brooks-Dombrower, 2002).
Criterion-referenced Assessment: Assessment that compares a student’s
performance according to a description of the desired performance (Carr & Harris,
2001).
Multiple Measures: Relying on more than one indicator to measure a
student’s strengths and weaknesses (Kang-Smith, 2002).
Norm-referenced Assessment: An assessment in which an individual or
group’s performance is compared to a larger group. Usually the larger group is
representative of the cross-section of all United States students (EdSource Online,
2001).
13
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Performance Standards: Type of standards that describe how well or at what
level students should be expected to master the content standards (EdSource Online,
2001)
Professional Development: Programs that allow teachers and administrators
to acquire the knowledge and skills they need to successfully perform their jobs.
Program II/USP: A component of California’s Public Schools Accountability
Act (PSAA) designed to provide assistance and intervention for schools identified as
underperforming. Schools that meet improvement goals will be eligible for financial
and nonmonetary rewards; schools that fail to meet growth targets over time may be
subject to district or state interventions (EdSource Online, 2001).
Rubrics: An established set of parameters for scoring or rating students’
performance on tasks (Carr & Harris, 2001).
Stakeholders: Those individuals who are involved in the student learning
process; it includes the state department of education, the local board of education,
district office staff, site staff, parents, students, and community members.
Standardized Test: A test for which procedures have been developed to
ensure consistency in administration and scoring across all testing situations (Gall,
Borg, & Gall, 1996).
Standards: A degree or level of achievement. The “standards movement”
began as an informal effort grown out of a concern that American students were not
learning enough and that American schools did not have rigorous curriculum. The
14
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
U.S. Congress adopted this concept more formally with its 1994 reauthorization of
the federal Title I program (EdSource Online, 2001).
Standards-based: An interrelated system of instruction, curriculum., and
assessments that are based on a specific set of learning objectives (standards).
Student Performance Data: The measured results of student efforts on
mandated state and district assessments.
Systems Thinking: An analytic method of viewing the complex and
interdependent nature of organizations that is the cornerstone of organizational
learning (Senge, 1990).
Title I: A federal program that provides funds for educationally
disadvantaged students, including the children of migrant workers. The funds are
distributed to school districts, which make allocations to eligible schools according
to criteria in the federal law. It was formally called Chapter 1 (EdSource Online,
2001).
Variable: A characteristic that can assume several values or can represent
two or more classifications or categories, often quantitatively different (Isaac &
Michael, 1997).
Organization of the Study
Chapter 1 of this study describes the overarching problem to be addressed by
this study along with the research questions that direct this study and the importance
of this study to policy makers and educators. Chapter 2 discusses the literature and
research that focus on systemic implementation of innovations. Chapter 3 discusses
15
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the methodology of the case studies and the methodology of the cross-case study.
Chapter 4 presents the data from the case studies and the analysis of that data from
the perspective of the cross-case study. Chapter 5 summarizes the research findings
and suggests recommendations for districts, schools, policy makers and educators to
further the innovation implementation throughout individual schools and districts.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 2
INTRODUCTION
Underlying the need for systemic change in public education is the need for
improvement in student achievement. The fact that the public’s concern for the state
of public education continues to capture the attention of politicians explains the
persistence of policymakers to generate wave after wave of educational reforms. It
is believed by some that this political involvement in the improvement of our public
schools Is also a primary cause of our inability to significantly improve the quality of
learning throughout the entire system of public education (Jenkins, 1997).
Seymour Sarason, in his book The Predictable Failure of Educational Reform
(1990), suggests that as long as the existing power base that is inherent to public
school systems is not altered real change is impossible. He contends that the
complexities of the public school systems and the power elements that maintain their
status quo are inadequately addressed by reformers. Sarason argues that our
socialization to the accepted traditional power relationships of our school systems
prevents reformers from recognizing the strength of the school systems’ power
structure to resist change. Bascia and Hargreaves (2000) also attribute resentment
and resistance to change to this lack of understanding of the complexity of the
educational context. In short, Sarason states that lack of understanding results in
superficial actions that ultimately confuse policy for actual change.
McDonnell and Elmore’s (1991) conceptual framework of policy instruments
provides a description of policy actions the design of which contains serious
17
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
deficiencies. Their conceptual framework categorizes policy instruments into four
types: mandates, inducements, capacity building, and system changing. McDonnell
and Elmore attribute assumptions of policymakers to each type of policy instrument.
The consequences of these assumptions result in the perception that negative
reactions are required to affect positive change. As an example, mandates generally
result in coercion and uniformity, inducements assume the existence of change
capacity when it does not exist, and system changing encourages defensive responses
to the intended changes. When policy instruments are fraught with inappropriate
elements, the result is an exacerbation of problems and not improvement.
Other elements of reform design also plague the implementation process. It
is common for policymakers to expect too many reforms to be implemented too
quickly (Bascia & Hargreaves, 2000). In addition, with standards-based reform the
current premiere model of educational change in the United States (Marzano &
Kendall, 1998), it is important to reflect on particular difficulties that arise from the
standards environment. Three standards-based issues that cause implementation
problems involve which standards should be used, the appropriate instrument for
assessing student proficiency with regards to standards, and the best method of
grading student work in a standards-based curriculum.
Another significant fault in the current standards-based reform is that states
generally create more standards than can be reasonably taught and learned in a K-12
system. This makes it difficult to determine which standards should be taught and
assessed and what benchmarks should be employed (Kendall & Marzano, 1997;
18
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Schmoker & Marzano, 1999). Many states also use norm-referenced assessments
exams to track and compare student achievement. This is in contrast to the
prevailing understanding that criterion-referenced assessments provide a more
appropriate method of evaluating student achievement in a standards driven context
(Carr & Harris, 2001; Isaac & Michael, 1995). With the use of norm-referenced
assessment tools student achievement is measured in comparison to other students
instead of with respect to the benchmark curriculum prescribed by state standards.
An additional issue with current assessment practices is that the high-stakes
environment reinforced by the No Child Left Behind legislation brings into question
the accuracy of one assessment to evaluate a school’s effectiveness and the depth of
instruction which is increasingly structured by success on one test (Herman, 2003;
Schmoker & Marzano, 1999). A fourth problem that has not been identified by
many researchers is that the use of rubrics to grade student work often leads to
inconsistent grading. Furthermore, the standards-based rubric is largely utilized at
the elementary school level. As such, it is incompatible with the traditional grading
system that remains in middle and senior high school. The practical issues that stem
from these problems have yet to be satisfactorily resolved.
Two additional reform characteristics add to the difficulty of reform
Implementation success. First, reforms are seldom pilot tested. Instead, reforms are
usually mandated to be implemented on a broad scale, without the benefit of learning
from incremental implementation efforts in local environments (Jenkins, 1997 &
Warwick, 1995). Second, reform agendas are generally so focused on immediate
19
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
quantifiable results such that the efforts of educators are overly focused on achieving
acceptable numbers, to the detriment of understanding the processes that lead to
continued improvement in student achievement (Warwick, 1995). The resiliency of
the educational environment to change (Fullan, 1993) requires that all of the above-
mentioned problems be acknowledged and substantively resolved if real change is to
become manifest throughout the public school educational system.
The general purpose of the literature review that follows is to review
principal findings in educational and business research as they relate to securing
successful systemic change within an organization, and more specifically to review
key factors in the successful use of data-driven strategies to manage continuous
improvement in schools.
Affecting change and improvement by means of data is a relatively new
strategy for public schools. It is, however, a strategy that is seriously undertaken and
studied, for many decades, by the business world. This literature review presents
snapshots from both business and education or proven systems and strategies. Some
overlapping of findings between business and current education practices is an
understandable outcome of this review. Other findings that are either peculiar to
education or only implemented in the area of business are also disclosed in this
review.
Ultimately, the reason for this literature review is to identify variables that
can be applied to the four case studies under scrutiny in this dissertation. The
examples explored in this literature review are to be narrowed to specific systems
20
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and strategies that can be utilized for analyzing the case studies. The rationale for
this linkage between specific variables to be employed as analysis tools rests on what
can best be analyzed, based on the content and specific purposes of the case studies.
Using variables to analyze content that is non-existent in the case studies offers little
value for this cross-case study. Nevertheless, the variables disclosed in this literature
that are not pertinent to this cross-case study due serve a purpose in that they
represent a wealth of resources for future studies regarding data-driven strategies to
improve student achievement.
Educational research has a symbiotic relationship with change efforts in
education. Research is used to authenticate change efforts at the policy making level
as well as at the district and school site levels. As previously referenced in Chapter
1, Chatterji’s (2002) extensive review of standards-based reform research resulted in
the conclusion that most research involving educational reforms is non-systemic in
design. As a result, Chatterji concludes that most educational research fails to
provide adequate assistance to enact system wide change.
Selected Research on Systemic Organization Change
The references that follow in the remainder of this chapter are representative
of current research and practices regarding system wide change in the fields of
education and business. The perspectives detailed below are varied, which
exemplifies the complex nature of organizational change. The research literature is
organized into two categories, that of the general topic of implementing change to
affect continuous improvement and more specifically data-driven change. Examples
21
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
of successful change efforts follow the representative literature on change and data-
driven change. These examples are from both the educational and business realms
and serve as “real-life” exemplars of the research literature on change.
Representative Education Change Research
In his review of reform research in education Chatterji (2002) defines
“systemic evaluation” as a method of examining the inter-relatedness of two or more
components of an organization that impact efforts to significantly change aspects of
the organization. Peter Senge (1990) cogently describes “systems thinking” as a
discipline for seeing wholes. In other words, it is a way to view the inter-relatedness
of processes that clarifies patterns and insightfully assists in effectively
accomplishing change. Education’s objective to improve the outcomes of education
on a system wide basis necessitates reviewing reform research from a systems
perspective (Bowsher, 2001).
Chatteiji (2002) suggests that most educational reform research can be
categorized as (a) Context based, (b) input based, (c) process based, or (d) outcome
based. The focus of context designs is the needs of school stakeholders or political
and legislative actions. Input designs investigate actions taken to cause change in
education systems such as resource allocation, and teacher training and qualification
levels. Process designs focus on how reforms are actually implemented and outcome
designs investigate the intended and actual results of reforms.
Context, input, process, and outcome studies provide valuable information
regarding the characteristics of successful elements of reforms and successful
22
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
schools. Without a systemic perspective to the issues surrounding change the
problem solving processes leading to significant improvement continue to remain
unknown to the majority of schools and school systems.
The Effective Schools’ research (Taylor, 2002) is an excellent example of
research that provides a description of common elements found in schools that
successfully improve student achievement. Taylor lists seven correlates to the
Effective Schools’ program. They are:(a) Clearly stated and focused school mission,
(b) safe and orderly climate for learning, (c) high expectations for students, teachers,
and administration, (d) opportunity to learn and student time-on-task, (e)
instructional leadership by all administrators and staff members, (f) frequent
monitoring of student progress, and (g) positive home-school relations. Taylor’s
admission that the Effective Schools’ process is ineffective in promoting systemic
change in student achievement in large urban school districts such as New York, Los
Angeles, and Chicago is evidence that implementation guidelines must be a concern
of research. The critical question for all practitioners is how does one materialize the
characteristics such as the seven correlates of the Effective Schools program?
The previous research referenced several important variables that will impact
the analysis of the case studies central to this dissertation. Those variables include
the importance of providing a clear goal or focus and having high expectations,
having and opportunity to learn and the frequent monitoring of student progress.
With respect to this study, clear goals are found in the form of measurable
achievement goals, high expectations in the form of believing that students can
23
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
improve if instruction is improved and the opportunity to ieam represented in
improved instruction that may also include timely instructional interventions,
Fullan and Miles (1992) present seven reasons why educational change plans
fail. They also describe seven considerations they believe foster successful change.
The seven reasons for change failure are described under the following headings: (a)
Faulty maps of change, (b) complex problems, (c) symbols over substance, (d)
impatient and superficial solutions, (e) misunderstanding resistance, (f) attrition of
pockets of success, and (g) misuse of knowledge about the change process.
The seven causes of change failure describe characteristics that act as barriers
to change. Strategies that do not take into consideration the complexity of change
generally employ symbols instead of substance to guide change efforts. The result is
usually cosmetic at best. In addition, without understanding the change process
change managers often misunderstand the reasons for resistance to the imported
change and react to people instead of improving the processes of change. According
to Fullan and Miles (1992) these deficiencies place in jeopardy the sustainability of
any change that may come from the change efforts. Inappropriate efforts require an
inordinate amount of energy and time from a few' key people and burnout becomes a
real possibility. If change is unsustainable, then it is not institutionalized.
Fullan and Miles (1992) present solutions to the causes of failed change
efforts that only partially improve the odds of system wide change. Certainly having
everyone involved in educational change leam about the components of change is an
important foundation from which to build. Such learning would delve into how
24
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
linear change strategies clash with the nonlinear and uncertain qualities of the
journeys known as change. Knowledge of change can also clarify the inability of
plans to dictate every stage of a proposed change. Such knowledge further indicates
that the use of creativity to solve implementation problems is proven to be an
effective means of realizing continuous improvement.
Fullan and Miles’ solutions for change weaken with the statement of their last
three propositions. Proposition 4 discusses the importance of allocating assets to
activities that directly support the change. They indicate that this may mean
increasing existing funding and moving existing funding from where it has
historically been used to where it needs to be employed to better support the change
implementation process. The problem is that they do not address how to change
attitudes such that staff will actively support the realignment of resources based on
what is best for improving student achievement, as opposed to allocating resources
as they are habitually inclined. Furthermore, in times of shrinking state budgets,
acquiring additional funding is easier said than done, especially to increase funding
without the type of strings attached that reduce the amount of funding and limit the
use of supplemental resources.
Proposition 5 of Fullan and Miles’ presentation (1992) discusses the need for
collaborative decision making that is both legitimized by all site stakeholders,
empowered by the district and imbued with the ability to make decisions that are
removed from personal desires. The unanswered question is how are attitudes
altered in order to establish a site level decision making body that is legitimized by
25
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
both the site and the district? Without answering this question such an ad hoc
decision making body will never be able to make substantive decisions with respect
to improving student achievement.
Fullan and Miles’ exposition of proposition 6 is also void of how to achieve
non-superficial change. Simply listing actions that must occur such as cooperative
learning, peer coaching, focusing on interrelationships of key components, and the
culture of the existing system is, at best, a description of what schools which have
successfully managed change seem to manifest. How to engage in these activities is
the missing link that Fullan and Miles fail to provide in any meaningful way.
The final and seventh proposition emphasizes the importance of change being
generated locally. Fullan and Miles do not offer a method that will diminish the
impediments to change that the district and state legislators impose on the schools.
In fact, state-sanctioned training and district-sanctioned behaviors are often the root
of dysfunctional processes that lead to preserving the status quo (Jenkins, 1997).
Research must go beyond a mere cataloguing of desired outcomes if schools and
districts are to counter the forces that prevent real implementation of strategic
change.
Fullan’s and Miles’ findings support improvement in student achievement by
appropriate funding to support the desired change, by collaborative decision-making
and focusing on the importance of change being under the control of each school
site. Collaborative decision-making in this study materializes in the form of how
26
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
data Is analyzed and how instructional decisions are made based on the results of
data analysis.
Schaffer, Nesselrodt, and Stringrield’s (1997) study of 10 reform programs
establishes 10 threats to foil implementation at school sites. The fust live threats
Include: (a) Financial issues such as Insufficient or unstable funding, (b) leadership
issues that stem from a lack of understanding of the reform and an intolerance to
change, (c) commitment issues ranging from a lack of consensus to inadequate
support, (d) public, parent and student perceptions that are generally based on limited
knowledge of the program and that perceive schools as being unable to assist
students, and (e) staffing issues which include teacher recruitment difficulties and
inadequate skills to support the reform program.
The second five destabilizing factors identified by Schaffer et al. (1997)
include: (f) Curriculum issues based on programs that do not meet with student needs
at a particular school site, incongruent school and state goals, as well as multiple
program implementation that promotes incongruent foci, (g) political issues
exemplified by administrators’ alterations to and deletions from the program because
of political reasons, (h) racial conflicts based on racial beliefs among staff members,
(i) facilities issues such that the school provides an inadequate environment for the
reform program, and (j) management, communication and scheduling problems that
reduce program effectiveness due to dysfunctional communication that ultimately
allows for a range of problems including missed student assignments and
inappropriate course placement.
27
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Four of the 10 impediments discovered by Schaffer et al. (1997) were found
in up to eight of the 10 programs studied. The most frequently observed
impediments to reform were financial support, commitment, leadership, and
curricular issues. For reference purposes this study targeted ten programs instituted
in schools with a majority of economically disadvantaged students. The 10 reform
programs studied include the Comer Model, Success for All, the Paideia Program,
Coalition of Essential Schools, Schoolwide Projects, Extended Year Schoolwide
Projects, Reading Recovery, Computer Assisted Instruction, Extended Time
Projects, and locally adapted Tutoring Programs.
The impediments exposed in the Schaffer et al. study are real and frequent
interferences to successful program implementation. They represent areas that
should be monitored prior to and during all change efforts across schools or districts.
How to avoid these impediments is not provided in Schaffer et al.’s study.
Nevertheless, The alignment of instructional goals with state standards achievement
goals, the involvement of administration in change efforts and the broadening of the
leadership base with respect to change agents are all important aspects of the
methodology for analyzing the case studies.
Siskin (2003) proposes that the internal culture of a school, which she terms
the internal accountability of a school, is what ultimately determines whether a
school will successfully implement change or not. Siskin’s thesis is that a school’s
willingness to respond to the external demands depends on whether the internal
accountability of the school or district is congruent with the change mandate. She
21
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
further deconstructs the culture into what teachers are held accountable for formally
(formal mechanisms), what individual teachers perceive as their responsibilities as
educators (individual responsibility), and what teachers expect of each other
(collective expectations). The issue of changing culture is fundamental to
implementing change. Siskin’s insights shed additional light on basic areas that need
to be addressed in order for serious change to unfold.
Fullan (2001) and Sergiovanni (1992) bring to the fore leadership qualities
that their research demonstrates to be necessary for successfully managing change.
Fullan’s five leadership components include (a) Having a moral purpose, (b)
knowledge of the change process, (c) building relationships, (d) knowledge of how
to share, and (e) knowing how to create coherence. Sergionni’s five dimensions of
professional virtue complement Fullan’s leadership components. In Sergionni’s
framework the elements of professional virtue are: (a) A commitment to high quality
practice, (b) a commitment to basing practice on commonly shared values and goals,
(c) a commitment to a high quality of practice beyond one’s own practice, (d) a
commitment to collegiality, and (e) a commitment to caring. Research consistently
discovers all of these leadership qualities as important to reform initiatives.
Fullan (1993) provides eight additional lessons that can be used as a compass
for managing change. They are: (a) Change cannot be mandated, (b) change is
uncertain, non-linear, not static, and continual, (c) problems and obstacles should be
seen as learning opportunities, (d) creating a formal vision and change plan come
after reflection and after individual and group values coincide, (e) tension between
29
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
individualism and group perspectives must be tolerated and preserved to maximize
knowledge, (f) top-down and bottom-up strategies must be orchestrated to be
employed simultaneously, (g) connections must be nurtured with entities outside of
the immediate environment, and (h) everyone must be viewed as a change agent.
Unfortunately, Fullan and Sergiovanni only provide us with an extensive list
of characteristics observed in environments that successfully negotiate the
vicissitudes of change. How to accomplish those characteristics is left unspecified.
As with a great deal of educational change research the outcomes of successful
change efforts remain as guideposts for how to achieve improvement in student
achievement and learning but do not represent a road map to change.
Cawelti’s (1997) study of reform efforts at 10 high schools provides valuable
insights into change efforts in secondary schools. The 10 schools are from 10
different states and were primarily chosen because they have been in the midst of
restructuring activities for at least three years. Based on the research-based premise
that multiple restructuring efforts are required to improve student learning, the
coordination of those multiple efforts is extremely important.
Using findings from 1994 Educational Research Service survey based on
restructuring practices at over 3000 high school throughout the country, Cawelti
developed five general restructuring components. The five broad change areas are:
(a) Curriculum (teaching), (b) school organization, (c) community outreach, (d)
technology, and (e) monetary incentives. Cawelti defines the five broad areas of
restructuring with seven restructuring elements. The first three elements are
30
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
considered focal elements. These consist of curriculum standards, effective teaching,
and results orientation combined with performance assessment. The last four
elements are considered facilitating elements that consist of technology, human
resource development, parent-community involvement, and work redesign involving
schedules, roles, teams, and information systems.
A major purpose of Cawelti’s (1997) study of 10 high schools was to
determine how reform oriented schools dealt with the demands of giving attention to
the various elements of the five broad areas of change and the seven restructuring
elements. Cawelti’s study concludes that all 10 schools are at varying stages of
implementation with regards to the degrees of implementation in these multiple
areas. He does state that perhaps one of the ftiost important observations that
resulted from this study was that all ten high schools exhibited the active
understanding of change and improvement as a continual process that requires
constant reflection upon the efforts made.
The Malcolm Baldrige Education Criteria for Performance Excellence
Blazey, Davison & Evans, 2003) employs a battery of system assessments to provide
on-going feedback during a change process. The Baldrige surveys contain seven
categories. This organizational analysis provides a systemic analysis of the
processes of change, specifically targeting educational organizations. The core
categories inspect 16 areas, each of which is described below.
Organizational Leadership reviews the level and quality of behavior of senior
leadership during change efforts as it attempts to impact the entire organization.
31
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Social Responsibility assesses the degree to which the organization ensures ethical
behavior as it involves responsibilities to external stakeholders. The level of
Strategy Development is measured based on organizational strengths and
weaknesses, clarity of objectives and the degree to which processes in place are
systematic. The area of Strategic Deployment inspects the quality of the strategic
planning process and how well strategy is translated into action throughout the
organization.
The next group of Baldrige performance evaluators generates feedback on the
perspectives and expectations of students and the relevance of the programs and
services offered to students and other stakeholders based on their perspectives and
expectations. An assessment of the level of student and other stakeholder
satisfaction is an important aspect of this component. Measurement and Analysis of
Organizational Performance, Information and Knowledge Management and Work
Systems focus on determining the extent to which an organization is able to measure
the effectiveness of its decision-making, systemic performance, and its ability to
manage knowledge to support change. Evaluation of work systems establishes the
degree to which organizational systems support high levels of performance.
Staff levels of Learning and Motivation are also measured by the Baldrige
survey. Training Quality is the link in this area. In addition, the well-being and
satisfaction of staff are evaluated along with the Effectiveness of Institutionalized
learning-centered processes and the quantity and quality of internal processes that
support student growth.
32
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The Baldrige surveys constitute a system of self-assessment that thoroughly
investigates seven major categories. Those categories represent interconnecting
processes that are vital to any change efforts. The surveys provide insight into the
relative health of each of these areas and enable teachers and administrators to make
strategy adjustments during the continuous process of change and improvement.
Baldridge’s findings impact this study by emphasizing the measurement and
analysis of performance and providing high quality training that supports the change
efforts. These actions are vital to this study as they also represent concerns of the
four case studies.
Reflections on First Seven Researchers
The research described up to this point establishes a set of characteristics that
promote change in schools. The features that relate directly to the analysis of the
case studies are: (a) Active support of change by leadership, (b) some degree of local
control over change, (c) adequate allocation of resources to support change, (d)
collaborative decision making, (e) school-wide comment to high quality, (f) frequent
monitoring of student progress, (g) alignment of instruction with student needs and
State learning goals and (g) accountability systems to ensure change. These
characteristics are important to this cross-case study because they will be used as a
means of validating change and improvement characteristics observed in the four
case studies which are the focus of this dissertation.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Achievement Gaps and Improved Student Achievement
Closing the achievement gap between African American students, Hispanic
and middle-class Caucasian students is a common call to arms in public education
(Quindlen, 2002; Rothman, 2001/2002; WestEd, 2003). Even though there are
currently more examples of African American and Hispanic students significantly
improving their achievement then ever before, the gap between the achievement of
these two minority groups and middle-class Caucasian students continues. This
achievement gap is central to the overall achievement deficits in many public
schools.
There are several plausible reasons that can be that can be suggested for the
difficulties in reducing or eliminating the achievement gap between African
American, Hispanic and middle-class Caucasian students. The sheer number of
African American and Hispanic students in many of the large urban school districts
makes the efforts of positively impacting the achievement gap a daunting task.
The R & D Alert (WestEd, 2003) suggests four necessary conditions to
significantly narrow the achievement gap between African American students,
Hispanic students and Caucasian students. Those four conditions are: (a) Assign the
best teachers been assigned to the most difficult teaching situations, (b) increase
instructional and learning time, (c) increase time for teacher professional growth, and
(d) reduce class size to 20 or fewer students. The cost of achieving the last condition
is often prohibitive. How to achieve “(a)” given the union restrictions that are
ubiquitous in large urban school districts is a question yet to be resolved. Increasing
34
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
instructional time so that the quality and length of extra time is effective is also
difficult to achieve given the amount of additional time that may be required to bring
about the degree of improvement required to decrease the learning gap. Finally,
many schools and school districts are increasing the amount of teacher professional
development time. The quality of those professional developments, in general, have
not improved enough to create sufficient change in instructional practices such that
the learning gap is reduced (Guskey, 2000; Joyce & Showers, 2002).
Quindlen (2002) sets forth seven additional actions that need to be manifest
in order to decrease the achievement gap. These actions are self explanatory and
include: (a) Build a relationship with students and their families, (b) respect their
ethnic heritage, (c) increase their expectations of their capabilities, (d) expect high
achievement, (e) focus on individual student needs, (f) use professional development
and collaboration among teachers. How to accomplish any, some, or all of these
changes is not discussed by Quindlen.
The achievement gap between Caucasian and Hispanic students is a problem
that goes to the heart of student achievement improvement for many public schools.
Given the high Hispanic student population in three of the four case studies, it will
be an important aspect of this study to see how effectively these schools have
reduced or eliminated this traditional achievement deficit. The two high schools in
this study isolate the achievement gap between their Hispanic and Caucasian
students as a significant goal in improving student achievement.
35
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
A Business Less on School Efforts to Improve Student Performance
Research by Peter Senge (1990) bridges the realms of both business and
education. During an interview (O’Neil, 1992) Peter Senge contrasts learning
organizations with schools. Senge describes learning organizations as environments
where individuals, at all levels, constantly reflect on and leam from their practices.
This activity is a collective and collaborative activity that builds the organization’s
overall capacity for learning and solving problems. It also encourages reliance on
innovation as a corollary response to improving performance and practice. In
contrast, Senge describes schools as systems in which individuals, at all levels, feel
disempowered to leam and to impact the organization. Schools and school systems
reinforce conformance to rules and thereby stifle innovation. In addition, Senge
describes schools as tradition-ladened environments that quickly convert the sense of
personal purpose often found in new teachers into cynicism. This conversion
dramatically reduces any commitment and passion to improve practice.
Senge’s five disciplines for change (Senge, 1990; Senge, Cambron-McCabe,
Lucas, Dutton, & Kleiner, 2000) are the building blocks for transforming an
organization from a protector of the status quo to an organization that has learning at
its core. Although Senge et al. place no rigid ordinal requirements on the five
■ disciplines, personal mastery and mental models are discussed first because of their
link between individuals and the organization. Personal mastery is the development
of an individual’s ability to objectively reflect on his or her views of reality in the
context of the organization in which the individual works. Without an open mind at
36
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the individual level little new learning in the organization is possible. Mental models
is an extension of personal mastery in that they represent deeply ingrained patterns of
assumptions and generalizations that guide behaviors and thoughts. Obviating these
patterns, first inwardly on an individual level and then outwardly with respect to the
context of the organizational culture, allows for the scrutinizing of perceptions and
thoughts.
Building shared vision and team learning are developed based on personal
mastery and mental models. Shared vision is a distinctly different process than the
usual process of creating a vision statement. A shared vision manifests itself as a
result of objective and open dialogue in which the common features of individual
pictures of the organization’s future are captured. It represents the driving spirit of
the organization as opposed to a manufactured statement perceived as words to
comply with, which in many cases few people can recall or articulate.
Team learning is a logical outgrowth of shared vision in that it utilizes shared
vision and dialogue to foster a more focused learning environment in which team,
members work together by building on each others expertise. The benefits of
effective teamwork, in contrast to non-team efforts, are virtually unquestioned in
both research and practice. Senge (1990) goes so far as to state that the critical
learning unit of modem organizations is teams and not individuals.
The final and fifth discipline is the discipline that integrates the other four
disciplines. Systems thinking ensures that all aspects of an organization are viewed
wholistically as inter-linking systems that impact each other. This perspective
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
strengthens the ability of an organization to leam and support the need for
continuous, strategic improvement.
In response to the need for change and improvement, Senge (1990) promotes
the use of systems diagrams to see the dynamic complexity of cause-and-effect
beyond the traditional models. Instead of portraying causal relationships from a
linear perspective that depicts the processes of change in snapshots of significant
outcomes, Senge highlights the streams of interactive processes. Systems diagrams
assist in the problem solving that is an integral component of implementing change
and achieving a state of continuous improvement.
The Importance of Sense’s Work to this Study
The notion of learning organizations is a business concept that has recently
received frequent exposure in school change research (Fullan, 1993). Senge’s work
in this field, as primarily a business researcher, has recently garnered a great deal of
favor in educational research on change and improvements in schools. His “five
disciplines” describe and analyze actions that build an organization’s capacity to
sustain continuous improvement. Although there is research evidence that some
schools are beginning to actively employ Senge’s research in their improvement
efforts (Senge et al., 2002), it still represents uncharted waters for most schools for
most schools. In the context of this study, Senge’s emphasis on team and
collaborative learning has a direct bearing on the four case studies. Senge’s concept
of systems analysis also provides promising possibilities for future research for
studies that Investigate the use of data-driven strategies.
38
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Representative Business Change Research
Fullan (1992, 2001) references recent business research as he wrestles with
the problem of successful change in education. The paradigm shift that Fullan
describes so that educators have the capacity to be change agents is a topic that has
been the focal point of business for many decades. Fullan’s acknowledgment that
business research in change can provide important information regarding change in
education further supports the shortcomings of educational research to engender the
desired improvement in student achievement.
Businesses have a longer history of serious concern with change that
improves the end product much longer than have institutions of public education.
Dealing with imposed change, promoting change and using performance data to
manage continuous improvement are business concepts that are now making their
way in the lexicon of educators. It can be instructive to gain some awareness of how
successful businesses manage change and continuous improvement. Characteristics
of successful businesses can be used to gauge where schools are on a continuum of
change actions that support continuous improvement.
The research in this section is selected as representative of strategic change
implementation in the business arena. The research studies that follow have
publishing dates in the 1990’s or later and the researchers and practitioners presented
have extensive experience in the field of organizational change.
Mintzberg, Ahlstrand, and Lampel (1998) discuss and describe the ten
schools of thought that underlie strategies for change in the business arena.
39
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Mintzberg, et al. conclude their review with the statement that understanding
effective strategy formation requires a focus on both process and content. This
emphasis on a systemic, holistic perspective reaches beyond the superficial changes
most often enacted by organizations, of organizational behavior, people, and work
systems, change leadership, motivation, training, goal setting, decision-making,
organizational communication, high performance.
Change is at the heart of all improvement strategies and reforms. Business
research provides valuable insight into the nature of organizational change. In the
business world, understanding how to make successful changes is increasingly
critical to an organization’s survival. Schermerhom, Hunt, and Osborn (2000)
provide a comprehensive overview of the business research and theory that grapples
with the difficulty of responding to change and implementing changes in an
organization. The change topics covered by Schermerhom et al. include, but are not
limited to, paradigms of organizational behavior, people, and work systems, change
leadership, motivation, training, goal setting, decision making, organizational
communication, high performance teams, and managing organizational culture. This
array of topics typifies the complexity involved in initiating and sustaining
organizational change.
Nadler, Shaw, Walton and Associates (1995) indicate the importance of a
dual requirement of an organization entertaining change. The people that make-up
the organization must leam new ways of thinking and working, as well as unlearn
previously sanctioned and ingrained behaviors and thought patterns. It is clear that
40
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
al! forms of significant change require some degree of modification to the
organizational culture.
Changing an organization's culture is no easy matter (Fullan, 2001; Nadler et
al, 1994). Educational and business researchers discuss the need for top-down and
bottom-up and lateral interventions (Aguayo, 1990: Beckhard & Pritchard, 1992;
Fullan, 1993; Nadler et al., 1994). The top-down format is prevalent in education.
Bottom-up and lateral interventions are minimized in most education institutions, but
vibrant in those schools that exhibit significant improvement in student achievement
(Berliner & Biddles; Bracey & Darling-Hammond as cited in Reeves, 2000). The
absence of a robust lateral intervention in education is one of the obstacles to
systemic reform.
Fullan (1993) summarily dismisses Beckhard and Pritchard’s 1992 book on
managing change as paradigmatically incorrect. One can only surmise as to why
Fullan states that Beckhard and Pritchard’s model of change leadership is outmoded
since he does not describe the reasons for his convictions. It is probably because
Beckhard and Pritchard’s clear linear model of leadership actions counter Fullan’s
understanding of change as messy, unpredictable, complex, and definitely nonlinear.
Nevertheless, Beckhard and Pritchard’s work does detail important characteristics
that are confirmed by multiple streams of research that are the results of their
combined total of almost 60 years of research in the field.
Beckhard and Pritchard (1991) describe the importance of an organization
moving to a learning mode, with an emphasis on integrating the processes that lead
41
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
to implementation. Training must support the new learning mode and organizations
should be more process oriented than simply results oriented. They indicate that
learning organizations encourage innovation and take advantage of their repository
of knowledge already in the organizations’ memory, but untapped. Developing an
environment by means of training that supports continuous learning, innovation, and
a focus on processes that leads to continuous improvement are topics that are
reiterated by other researchers, yet to be discussed in this review of literature. These
topics are also central to research on how organizations manage knowledge
(Davenport & Pruseit, 2000; Hatten & Rosenthal, 2001; Housel & Bell, 2001).
Another topic which resonates through the change literature is that change is
always met with resistance. Beckard and Pritchard (1991) indicate that resistance to
change generally pervades an organization. Converting the negative energy of
resistance to positive energy, according to, according to Beckhard and Pritchard, is
important with any change effort. They suggest that communicating clear goals
throughout the organization while rewarding progress in achieving the change goals
is an effective strategy toward converting an environment filled with resistance to an
environment with a positive attitude toward change.
The importance of a learning mode for organization and training to engender
the learning of new behaviors is emphasized by Deckhard and Pritchard (1991).
This also underscores the importance of training in this study.
Larry Bossidy’s successful careers as CEO of General Electric and Allied
Signal combined with Ram Charan’s research and consultant work with senior
42
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
executives from numerous companies, ranging from start-ups to Fortune 500
companies, offers a perspective on achieving successful change that is rarely
understood and even less often pursued by architects of change (Bossidy & Charan,
2002). Bossidy and Charan state that most leaders, throughout all levels of an
organization, do not take personal responsibility for “getting the job done.”
Bossidy and Charan (2002) indicate that execution (“getting the job done”) is
dependent upon three processes: (a) People, (b) strategy and (c) operations. These
processes are integral aspects of all organizations and are usually separate entities
working simultaneously with each other. Inservices on change generally do not
result in commitment to action plans. Bossidy and Charan emphasize robust
dialogue throughout the organization as the key element in executing change.
Robust dialogue encourages and rewards honest and comprehensible talk. The top
leader must model this form of open and direct discussion in order to align the entire
organization with the change effort.
Bossidy and Charan (2002) detail seven essential leadership behaviors for
executing change in an organization. The behaviors include: (a) Know your people
and your business, (b) insist on realism, (c) set clear goals and priorities, (d) follow
through, (e) reward the doers, (f) expand people’s capabilities, and (g) know
yourself. In Bossidy’s and Charan’s framework these behaviors are best articulated
by means of robust dialogue and the active participation of the top leader in the
hiring, training, and promotion processes. These are qualities of leaders that are
43
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
rarely found in organizations, but are definitely found, in varying degrees, in
organizations that successfully accomplish continuous change and improvement.
Bossidy and Charan (2002) not only mention the importance of expanding
the capabilities of personnel through training, they also underscore the importance of
top leadership being personally involved in change efforts. Although the importance
of trainings has been previously mentioned, Bossidy’s and Charan’s clear description
of leadership’s involvement is an important emphasis that is an integral part of this
dissertation’s analysis.
Edwards Deming’s work on continuous improvement provides a seminal
foundation for systemic organization change. Although the statistical theory of
variation represents a cornerstone of Deming’s work, generally referred to as Total
Quality Management (TQM), his perceptions on the people-end of business have •
proven highly effective in organizations that thoroughly implement his quality
improvement framework (Aguayo, 1990; Deming, 1994).
Deming’s Plan-Do-Study-Act (PDSA) learning and improvement cycle
(Deming, 1994) is, on the surface, similar to other process or output evaluation
paradigms. However, underlying the four phases of the improvement cycle are a
knowledge of the system under scrutiny and the elimination of obstacles to services
by improving processes that impact and are integral to the system. Deming’s method
of improvement adheres to three central principles: (a) Cooperation between
management and workers, (b) appropriate training for all employees and (c) the
44
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
creation of a work environment free from fear of open communication and
experimentation (Aguayo, 1990).
Deming’s 14 points that lead to continual system improvement can be
subsumed under the three central principles mentioned in the previous paragraphs.
The following listing represents an amalgam of Warwick’s (1995) listings of
Deming’s 14 points: (a) Create constancy of purpose for improvement, (b) adopt a
new philosophy, (c) cease dependency on inspection to achieve quality, (d) cease
awarding contracts based on price alone, (d) constantly improve the process, (e)
institute training, (f) adopt and institute leadership that supports change, (g) drive out
fear, (h) break down barriers between staff areas, (i) eliminate slogans and targets
that pressure quality work without a means to achieve the goals, (j) eliminate
numerical quotas, management by objectives, and management by results, (k)
remove barriers to pride of workmanship, (1) institute a vigorous program of
education and self-improvement, and (m) make everyone’s aim quality
improvement.
Several of Deming’s 14 points require more explanation because they are not
readily understood by merely listing them. Adopting a new philosophy means
learning new behaviors that promote cooperation, open communication, and an
understanding that structured change is a ongoing process, not an achieved goal.
Non-reliance on inspection to achieve quality improvements requires the
understanding that quality is achieved by continual improvement of processes and
not micromanaging or mass testing. Micromanaging prevents the development of
45
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
system-wide leadership and testing only indicates where yon are. not how to get to
where you want to be. Not restricting purchases to initial cost only refocuses the
reason for the purchase on the most important consideration, which is the overall
value of the purchase. The actual value of a purchase to the improvement of the
system may far outweigh initial costs. The elimination of slogans and targets
underscores the preoccupation with setting goals that are vague or unrealistic and not
providing workers with the means to actually achieve the stated goals. Finally,
eliminating quotas and various forms of management by numbers is closely related
to the elimination of slogans and targets. Numbers reflect quantity not quality. In
Deming’s model the numbers will come with improvement in processes, which
should be the main goal, environment in which open communication throughout the
organization is the norm are the crux of Deming’s continuous change model.
It is important to understand that the principles of Total Quality Management
do not claim to enable the ultimate achievement of perfect improvement. Instead
they provide procedures for continuously working towards a perfectly functioning
system. It is this concept of continuous effort to improve that supports the need for a
systems analysis approach to improvement, as well as the importance of developing
a learning organization that can leam how to most effectively make continuous
improvement.
A concept that is newly introduced in this literature review by Deming is the
cooperation between management and workers. This speaks against the traditional
micromanaging inherent to hierarchical organizations, such as schools and school
46
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
districts. It is a form of self-management that is observed, to varying degrees, in
several of the case studies.
Gumming’s and Worley’s (2001) chapter on leading and managing change
describe the most recent focus on change to include creating visions, garnering
internal political support, and managing the organization’s transition toward
achieving and sustaining goals. Cummings and Worley describe five management
activities that support effective organizational change. The activities include: (a)
Motivating change, (b) creating a vision, (c) developing political support, (d)
managing transition, and (e) sustaining momentum.
The object of change is to change the status quo or culture of an organization
so that the organization’s performance is improved. Creating a readiness for change
in the organization and overcoming resistance to change are important aspects of
motivating change. Personnel throughout all levels of the organization must believe
in the beneficial need for change. Cummings and Worley mention how the presence
of “devils’ advocates” can foster an attitude that questions the status quo. They also
describe an organization’s ability to look at itself from the perspective of people and
organizations external to itself as a means of awakening its personnel to the need for
change.
Revealing discrepancies between current and desired performance, along
with targeting achievable and credible change goals also impact an organization’s
will to change. When members of an organization believe they can achieve
successful and meaningful change, they are more likely to pursue such change.
47
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Cummings and Worley (2001) describe how empathy and support,
communication, and participation and involvement can further attenuate resistance to
change. Learning how personnel are responding to the change by active listening
and nurturing a more open and more inclusive environment that encourages
collaborative problem solving can both lessen barriers to change can be lessened.
When communication is clear and responsive to concerns uncovered by a more
empathetic awareness of the responses to change, a great deal of the anxiety and
misinformation that arise in reaction to change can be effectively countered.
Broadening the participation base of personnel in the change efforts allows for
increased support of the change. Greater participation in the change process is a
two-way street in that the diversity of ideas and information is the foundation for
better innovations for the company and innovations that make sense to the
employees.
Creating a vision is understood by Cummings and Worley (2001) as both
describing the core values and purpose of the organization and communicating the
future direction of the company. The vision can provide a common goal and a
rationale for that goal. Leaders must make sure that future goals are enough in
alignment with basic organization values so that they do not become achievable
goals and not unrealistic dreams.
Developing political support for change requires determining the power of
the change agent, identifying the principal representatives of stakeholder groups, and
influencing those stakeholders (Cummings & Worley, 2001). It includes determining
48
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the actual power of the agent, identifying who are the key stakeholders, and
influencing those stakeholders. All organizations consist of coalitions of individuals
and groups with different interests and power-bases that are threatened by deliberate
change. These “political” alignments often respond to change in defensive reactions
that attempt to preserve the status quo.
In order to positively influence the political structure of an organization a
leader must assess his or her power-base in relation to the political status quo of the
organization. Change agents must use the,knowledge of their own power-base and
the knowledge of key stakeholder interests to promote successful and meaningful
change. Strategies for influencing key stakeholder individuals and groups include
informing the various stakeholders how the change will benefit them, using social
relationships to develop stakeholder support for the change, and working around the
existing organizational structures that usually act as the mechanism for acceptable
levels of change.
Managing an organization’s transition from status quo to change is an
important aspect of effective change management that entails planning activity,
commitment to activity planning, and developing structures to manage change.
Planning specific activities for carrying out change must be fully supported by top
management. Flexibility and openness to feedback should characterize the change
process plans. Setting measurable achievement goals throughout the change process
is also important, as well as identifying people and groups who are supportive of the
change.
49
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Sustaining change is the fifth activity of the effective management of change.
It consists of the four actions that ensure that there are appropriate and adequate'
resources to support the change, making sure that change agents have the
organizational support to carryout the change activities. This entails providing
personnel with the new skills to enable the requisite change in work procedures and
creating mechanisms to reinforce the behaviors that sustain the change.
Cummings and Worley (2001) conclude their chapter on leading and
managing change by describing two common barriers to the change process. First,
most organizations do not allow adequate time to accomplish change. Any
significant change requires time and is often accompanied by dips in performance
during the transition from familiar practices to becoming expert with new, behaviors.
The decline in productivity is often misinterpreted as a failure of the change process
instead of evidence that more time is needed to unlearn old routines and leam new
ones. Second, organizations often do not stay the course and move to the next
change fad before they have allowed the first change to be fully implemented. This
can be a result of impatience for more immediate results or a means of maintaining
the status quo while creating a veneer of change. In either case, the result is the
same. Significant change is abrogated. .
Inspecting discrepancies between current and desired performance and
setting performance goals are not new' concepts to this literature review. The main
difference is the insistence that performance goals be both achievable and credible.
50
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
These qualitative differences are glaring not addressed by either the state or the
public school system. This topic would be a subject for future research.
Key Themes from Education and Business Change Research
Both education and business research literature on change emphasize the
need for changing an organization’s culture as a part of implementing system-wide
change. In order to cause a change in culture clear change objectives and a
knowledge of the change processes and effective change processes are required.
Formal training and modeling are understood as fundamentals to building the
capacity of people in an organization to fully implement change.
Areas that are generally not valued in organizations are the very processes
that must be made an integral aspect of an organization’s day-to-day operations if
continuous improvement is to result. The research literature describes some of the
important areas to institute as: (a) Collaborative and creative problem solving, (b)
clear performance goals, (c) continuous learning of how to improve the end product,
(d) open dialogue between all organization levels, and (e) tenacity and patience to
materialize the desired changes over time. General change literature in education
and business are in agreement with each of these major thenjes.
Some apparent differences between educational and business research are not
differences, on closer inspection. For example, having smaller class sizes, increasing
opportunities for learning and having high expectations appear to be education
specific. However, having smaller sizes for teams or formal learning opportunities is
not antithetical to business. Neither is having high expectations for personnel output
51
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and product quality, which is comparable to seeking greater certainty that
instructional efforts will consistently produce high levels of student learning.
On the other hand, several findings from education research that appear equal
to the findings in business research do in fact represent important differences. Two
examples of this include the emphasis on management and workers to cooperate with
each other and the setting of achievable and credible goals. As previously stated,
education literature does not emphasize cooperation between administration and
teachers, nor does it emphasize credible and achievable goals.
Education is quickly learning to take advantage of the business research
regarding change. The once traditional boundaries between education and business
are being blurred. Major researchers from both education and business, such as Peter
Senge and Michael Fullan, are facilitating this exchange of information so that long
standing and successful change processes in business are now being introduced to
education’s efforts to initiate and sustain continuous and significant change.
Representative Educational Research on Data-Driven Change
Research by Schmoker (1999, 2001), Fox (2000, 2002), and Dataworks
(2000) provide more specific implementation guidelines for improving student
achievement with the use of student performance data. Schmoker’s (2001)
investigation of school districts that use student performance data to dramatically
improve student achievement involved five school districts. The school districts
reviewed by Schmoker are found in a variety of geographical locations, Chicago,
52
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Milwaukee, Detroit, Phoenix, and Houston. The district student populations range
from 3,300 to 100,000 and the districts serve a wide-range of student populations.
Schmoker (2001) synthesizes what he believes to be the common factors to
the success of schools in the five school districts. The strategies he distills from his
shady of five districts include (a) Simplifying all school improvement plans, (b)
involving all instructional staff in data analysis and goal setting, (c) beginning
change on a small scale with personnel who are open to implementing the change,
(d) targeting easily attainable goals, and (e) working collaboratively to attain the
desired goals. Schmoker (2001) acknowledges the importance of training in the
execution of these five strategies. Learning how to work collaboratively and how to
utilize information from standardized assessments to improve student learning and
achievement require learning new behaviors.
In addition to monitoring the success and quality of the change
implementation Schmoker states that educators must also continuously ask four
questions. Those questions are: (a) Do all teachers know their annual improvement
goals during the school year? (b) Can every teacher describe the areas of weakness
their team is concentrating on? (c) How often are improvement meetings scheduled?
(d) What evidence is there that team meetings are productive?
Although the results of Schmoker’s (2001) research findings offer evidence
of success in improving student achievement by using student performance data, the
success is nevertheless limited and there is definitely no clear evidence of district
wide success in large urban districts. A wholistic understanding of how the change of
53
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
using student performance data is achieved is in fact hinted at by the insistence that
all personnel must receive appropriate training and that leadership, to propel the
change, should exist throughout the school or district. Clearly, the schools in this
study created the processes that led to successful data implementation.
Dennis Fox (2002), in consonance with Schmoker’s study, delineates data
into three major categories. The first of those categories is outcome data. Outcome
data are the results of instruction. They distinguish between what is taught and what
is actually learned. Publishers’ tests, State tests, homework, department tests, and
report cards are examples of outcome data that provide important feedback on the
successfulness of instruction.
Demographic and Process data are the second and third categories described
by Fox (2002). Demographic data provides a source of information into the student
qualities that impact student learning. Some of those qualities include language
proficiency, mobility rate, socioeconomic status, family support, and attendance
records. This is the data that is often used by educators as an excuse for low student
performance, instead of as additional information about the learner that can assist in
improving instruction so that instruction better meets the individual needs of
students. Process data, on the other hand, describe attitudes, practices, and programs
used to instruct students and educators. Process data include such areas as textbooks,
instructional strategies, supplemental materials, interventions, parent involvement
levels, and assessment practices.
54
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Fox’s research (2002) indicates that the ability to decipher the types of data
available is important for producing high levels of student achievement. Each type
of data provides different tools with which to fine-tune instruction. Fox takes the
conceptualization of data an additional step by developing a data inquiry system that
uses all three data categories to improve instruction and learning.
Fox’s (2001) additional data usage steps answer the following four questions:
(a) What do I want students to know and be able to do? (b) How will I know students
know it and can do it? (c) How can I account for the students’ performance? (d)
What am I going to do for students who do not get it.. .and for those who do? The
first phase in Fox’s process is organizing the data. Fox allows, and almost insists, on
a creative rearranging of data by educators. This rearranging often takes graphic
forms that more dramatically highlight targeted concerns.
After reorganizing the data in a more meaningful format the data is then
analyzed for instructional decision-making by responding to the following two
questions: (a) What did you learn about the students’ performance.. .from organizing
these data in a user-friendly format? (b) How might these data be used for
instructional decision-making? Teachers are next guided to determine which
students require the lesson to be redone, reviewed or re-taught. Fox (2001) defines
“re-doing” a lesson as providing additional practice to ensure mastery, “review” is
for students who learned most of what was taught and “re-teach” requires starting a
lesson again, from the beginning. Fox cautions teachers to select the instructional
strategies that provide the largest effect size or impact on student learning.
55
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Fox further guides teachers in using data by having them pay attention to how
the data describes the distribution of scores to indicate how many and which students
require assistance in what areas. Finally, Fox suggests four prompts to enlist a more
in-depth and insightful analysis of the outcome data. Those prompts include the
questions: (a) What is known as a result of examining the data? (b) What are your
“hunches” concerning why the data resulted as they did? (c) How can the data be
used to inform instruction? The power of Fox’s method is its simplicity and its
procedural clarity.
Research by DataWorks (2000) advances a means of improving student
achievement based on the ability of schools to improve classroom productivity. The
DataWorks Productivity Index encompasses four instructional-learning components:
(a) Time on task, (b) degree instructional tasks are aligned to standards, (c) the
breadth of the curriculum in teaching all of the standards-based content strands, and
(d) the degree of instructional effectiveness based on the use of the most effective
teaching methods. According to DataWorks, if each of these components is raised to
a high level of implementation student achievement dramatically improves.
One technique used by DataWorks to assess the degree that curriculum is
aligned to the standards is by collecting actual work given to students over a period
of time and then evaluating the extent to which the work is grade-level appropriate
per state standard guidelines. This process is termed curriculum calibration by
DataWorks and has been performed, by DataWorks, on over 500 California schools
per year. In general, DataWorks research reveals that low performing schools,
56
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
beyond kindergarten, teach increasingly fewer grade-level standards-based material
in each successive grade-level. As an example, by 4th grade, 20% of the tasks in
mathematics that are given to students are at the 4th grade level, 40% are at the 3 rd
grade level, and 40% are at the second grade level. In language arts 33% of the
student work is at 4th grade level, 35% at 3rd grade level, 30% at 2n d grade level, and
2% at 1st grade level. This “curriculum slippage” accounts for low student
performance with grade-level curriculum. In general, research by DataWorks
discloses that high performing schools teach more at grade-level than do lower
performing schools.
Schmoker, Fox and DataWorks all emphasize collaboration, data analysis,
goal setting, and frequent data analysis. In addition, DataWorks emphasizes
standards-based, grade-level instruction as a means to improving student
achievement. These actions are mirrored in the case studies.
Representative Business Research on Data-Driven Change
The use of data to guide change is a common activity in business that is
nevertheless difficult to effectively execute. Nadler (1977) describes how data can
impact behavior in an organization. He presents two research-supported concepts
regarding data, behavior, and change. First, data used as information can energize
individuals and groups in an organization to bring about change. Second, data can
direct behavior by motivating action.
Energizing behavior is understood as a necessary precursor to directing
behavior. The type of data, how it is collected, and how the data is analyzed and
57
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
presented influence the outcome of the energizing and directive actions. Nadler
(1977) emphasizes the importance of perception as to whether individuals change
behavior in relation to data. He describes three ways in which data collection
energizes behavior. Implied sanctions, or rewards, evaluation and explicit rewards
can result in data collection energizing personnel.
When data is collected by individuals of importance in the organization the
common perception is that the area being measured is important and sanctions or
rewards could result. This perceived implication can generate focus on the measured
actions. Evaluation is often preceded by data collection and hence can also
galvanize employee energy and redirect behavior. Explicit rewards are often an
outgrowth of data collection that measures performance.
Nadler (1977) describes one other factor that generates energy around
measured activity, in addition to data collection or measurement. The perceived
degree of accuracy of the measurement is important with respect to whether people
in the organization will respond positively to the data collected. If the data collection
measurement is perceived to be inaccurate or threatening, employees may react
negatively instead of positively. In this case false information may be provided and
individuals may become so concerned about the data collection that work is
adversely affected by actions such as over dependence upon numbers and by
defensive behaviors that serve as a false protection from the data collection and data
analysis.
58
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Nadler (1977) also provides research-based insight into how data feedback
can impact employee behavior and support or frustrate change efforts. Feedback can
assist in the improvement of work processes by providing an organization with
comparative data and benchmark performance goals. Feedback can energize and
direct the behavior of employees by discontinuing preconceived beliefs regarding
the organization by providing facts that are contrary to currently held beliefs.
Feedback can also be linked to, or perceived to be linked to, extrinsic
sanctions or rewards. Internal rewards can stem from feedback, as well. Feedback
can be used to monitor progress toward achieving newly established performance
goals. Finally, Nadler (1977) indicates that feedback can impact behavior by
prompting learning through problem solving that finds ways to decrease the
performance gap disclosed by the feedback.
Additionally, Nadler (1977) describes three factors with which data
collection and feedback will increase, and not decrease, an organization’s
performance. The first factor is that feedback should have sufficient specificity to be
credible and measure progress toward goals. The second factor is that group
participation in using data for goal setting and goal attainment should be encouraged
in a non-punitive environment. The third factor is that the nature of the goals and the
difficulty of attaining the goals must be both meaningful and attainable from the
perspective of the employees.
The value of Nadlers’s research to this cross-case study is that it substantiates
the importance of important individuals collecting data with respect to elevating the
59
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
act of data collecting and analysis. This understanding is not directly addressed in
the education literature presented in this review. The 1977 date of Nadler’s book
also reinforces the reality that this is a topic studied by business researchers long
before education understood it to be a viable means of improvement.
Deming’s analysis of how to attain continuous organization improvement
, also involves the use of data. The statistical concept of stable systems cannot be
dismissed from its central role in Deming’s systemic and continuous improvement
model. For Deming, all organizational environments have variation, which is
described by purposefully collected data. In his framework if the variation is erratic
it is impossible to predict outcomes of that system. The first priority of change
agents should be using the PDSA cycle to bring control and predictability to the
system (Warwick, 1995). Defining acceptable performance ranges is an initial step
in bringing about control and improvement to any organizational system. The
statistical considerations combined with the principles of cooperation and
communication throughout the organization, effective training that supports
improvement, and creating an environment in which open communication
throughout the organization is the norm are the crux of Deming’s continuous change
model.
Another data-related improvement program is Six Sigma. Six Sigma is an
improvement program that enjoys a high degree of currency with many large
businesses such as IBM, General Electric, Ford, DuPont, and Microsoft (Bruce,
2002). It represents an extension of Deming’s quality improvement process in that it
60
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
provides a method that refines an organization’s ability to reduce system variation.
As with Deming’s improvement program, Six Sigma provides organizations with a
means of measuring variation within a system and reducing that variation, thus
achieving improvement in that system. Six Sigma purports to increase the ability to
decrease variation to a greater degree than Deming’s model. In fact, the name Six
Sigma is a statistical concept that indicates only 3.4 “defects” per million (Bmce,
2002). The use of data to make decisions is the foundation of both Deming’s and Six
Sigma’s improvement paradigms.
Deming and Six Sigma stress the use of data to manage continuous
improvement. Appropriate training to support managing improvement with data is
essential to Deming and Six Sigma. Deming and Six Sigma offer a new possibility
to managers of change and that is the possibility of being in control of improvement
without the active use of hierarchical micromanaging. This scenario is only partially
realized in several of the case studies in which schools are allowed to control aspects
of the data-driven improvement strategies.
Key Themes from Education and Business Research on Data-Driven Change -
The use of data to improve an organization’s performance is not new to
business. It is quickly becoming the fundamental operation in schools for promoting
continuous improvement in student learning. Education and business research, in the
area of data-driven change, are in agreement with the importance of using a
systematic data analysis procedure to determine areas of strength and weakness with
respect to goal achievement. Collaboration, analysis of data and the application of
61
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
innovation and creativity to achieve goals, while charting progress with data from
regular assessments, is also described as important aspects of data-driven change.
Business research on data-driven change provides an understanding of how
using data can energize organizations to change procedures for achieving goals. As
a subset of organization change, data-driven change is more clearly explained in
business research. Knowing how feedback and information from data can effect
change may facilitate the use of data as a vehicle for improvement in schools. As an
example, understanding how the perception of the accuracy of the data and linkage
to possible sanctions or rewards based on the progress documented by the data can
be powerful sociological forces to bring to bear on change efforts in schools.
Education research can be used to document the present levels of student
learning in terms of student achievement and the degree to which students are taught
grade-level standards. However, Deming’s (1977) admonition against focusing
solely on numerical goals, to the neglect of the processes and procedures that lead to
continuous improvement, provides educational change practitioners with an
important consideration to guide their efforts toward targeted improvement goals.
Giving a face to the research by describing real schools and businesses that
are successfully using data-driven change strategies is the purpose of the remainder
of Chapter 3. Such disclosures take research out of the realm of theory, making it
more believable as a potential reality for other organizations.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Exemplars of Successful Change Efforts in Education
Rothman (2002) lists several school systems for which data reflects success
in decreasing the achievement gap. The Houston school district, Department of
Defense schools, the 21 so-called “90/90/90 schools” in Milwaukee, and the
Charlotte Mecklenberg district in North Carolina are highlighted as schools and
school districts that managed to significantly reduce the achievement gap between
certain minority groups and non-low income Caucasian students. How did these
schools and districts accomplish this previously thought to be unachievable level of
success?
The Houston school district believes that it cut the gap in half within six
years for African American and Hispanic students by sharing information regarding
successful initiatives through a formal network and providing academic assistance to
students while making available opportunities for parents to improve their own
literacy, and participating in school decision making processes. The Milwaukee
study, that included schools where 90% of the students were from low-income
families and ethnic minorities, nevertheless met high academic standards. The
schools in this study exhibited common characteristics such as focusing on academic
achievement, curricular emphasis on reading, writing and math, frequent
assessments, multiple opportunities to succeed, written responses as a function of
performance assessments, and the external scoring of assessments (Reeves, 2000).
Rothman (2002) reports that these initial findings by Reeves, which encompassed
seven schools, has since tripled the number of schools duplicating this success.
63
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The success of the schools in Reeve’s study is predicated on schools
performing frequent assessments and providing students with multiple opportunities
to succeed. These actions also represent two of the basic improvement actions of
data-driven improvement.
Research by Smerkar, Guthrie, and Sims (as cited in Rotham, 2002) reports
that the Department of Defense schools have dramatically improved the academic
achievement of its African American and Hispanic students by addressing the
multiple causes of low performance. The study gives credit to the Department of
Defense schools for addressing the multiple causes of low performance, which
includes low self-esteem and low self-expectations. A concomitant result of
responding to the issues of low student self-expectation is that Department of
Defense teachers tend to become more resolute in assisting their students in
achieving academic success when they see students missing the mark.
The Charlotte-Mecklenberg district doubled the number of African American
students reading at grade-level to 70%. The district accomplished this in only four
years (Rothman, 2002). The dramatic change in student performance is attributed to
a concerted effort to have high-quality teachers in high-need schools. The district
offered financial incentives and improved work conditions to encourage highly
qualified teachers to work in low performing schools. It also developed specially
designed curricula and instructional assistance for less-qualified teachers.
One observes from the above four examples a variety of characteristics and
strategies that are present in high performing schools and school districts. This
64
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
reemphasizes the fact that a variety of strategies that are systemically adopted by a
school or district can be a potent formula for change. The good news is that there is
not just one correct strategy for manifesting improved student achievement. Four
additional studies will be reviewed in this section to underscore the multiplicity of
strategies that are credited with dramatically improving student achievement.
A multi-year study of eleven elementary public schools in Chicago supported
the theory that higher performing schools have greater program coherence than their
lower performing counterparts (Newman, Allenworth, & Bryk, 2001). Newman et
al. define program coherence as a common instructional framework that is active
throughout a school such that the school staff works collaboratively to support the
implementation of the change framework. This differs from the usual definition
which defines “coherence” as the alignment between a school’s instructional
program with achievement standards and assessments that are external to the school.
Newman et al. do not exclude the possibility of congruence with external standards
but their definition acknowledges that policies and mandates that are external to a
school can undermine or have nominal impact on the brand of coherence that they
believe maintains the focus of change efforts at a school.
The study of 11 Chicago elementary schools by Newman et al. (2001) also
confirmed that achieving high levels of program coherence, in terms of their
definition, is difficult to achieve. Only three of the schools out of the 11 schools
under study exhibited high levels of coherence. The consortium’s report attributes
higher levels of program coherence and student achievement to leadership that
65
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
begins with the principal assertively putting in place a focused instructional program,
while simultaneously spreading the leadership throughout the teacher base. This
broadening of the leadership base is accomplished by means of teacher collaboration
and professional development.
The focus on collaboration and on both definitions of “coherence,” broad
leadership base and professional development are readily found in the case studies.
The notion of coherence is found in the form of consistent practice throughout a
school and in terms of aligning instruction to the state standards.
McDougall, Sanders, and Goldenberg’s (2002) review of a school change
model called Getting Results describes a school improvement model that is currently
an active part of nine elementary schools in a large urban school district in
California. Comparisons of National Curve Equivalence (NCE) means of site testing
results between the nine Getting Results schools and the over 600 other elementary
schools indicate that the NCEs from 1997 to 2001 of the Getting Results schools
increased at a greater rate.
Fifteen elementary schools participated in the Getting Results study by
McDougall et al. Nine of the schools were using the Getting Results model and six
were demographically similar schools that agreed to be a part of the study as the
control group. All schools in the study were implementing 20:1 class size reduction
in grades K-3, state content standards in language arts and mathematics, annual state
standardized testing, and a common curriculum linked to required on-going training,
with assistance from site coaches in math and language arts.
66
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The Getting Results (GR) model was gradually piloted at schools over six
years. The GR model utilized five interdependent components as focal points for
improving student achievement. Those program components consisted of setting
achievement goals, establishing indicators of success, encouraging peer collaboration
in implementing actions to improve student achievement, the development of
leadership that sustains an environment which supports and pressures the change
necessary to achieve the targeted improvement.
McDougall et al. (2002) hypothesize that the GR model assists teachers in
focusing on improving teacher actions that impact academic achievement. They also
believe that the model helps develop a broader leadership base that simultaneously
places teachers in a more visible leadership role while developing more powerful
leadership linkages with administrators. This formally manifests itself in Academic
Leadership Teams. The GR model is also understood as a vehicle for producing
visible improvement in student performance. McDougall et al. believe that the
results of school wide success helps to generate increased collective support for the
Getting Results program.
Major fundamental supports of the Getting Results model are considered to
be continuous external expert support, and the readiness of the majority of the staff
to commit to the implementation of the program (McDougall et al, 2002). Data
from this study indicate that school staff responded well to guidance from external
experts and that the initial buy-in by the majority of the staff decreased resistance to
the proposed change program. Researchers also note that when the GR model was
67
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
implemented simultaneously with another major instructional change there was less
effective implementation in those grade levels. McDougall et al. also report that the
complexities of a multi-track school calendar versus a single-track, traditional school
calendar did adversely impact the important issues of planning and communication.
One of the strengths of the GR model is that it is both “curriculum-free” and
provides a step-by-step process that galvanizes site administrators and teachers to
prioritize their focus actions on areas over which they have control and that
positively impact student achievement. Two of the major drawbacks of the program
are its expense (about $100, 000 per school over three years) and the inability of the
Getting Results organization to sponsor more than a handful of schools at a time.
The sustainability of a program that relies heavily upon outside expertise could also
be a concern, especially for schools in which personnel change, both at the teacher
and administrative levels, are a common part of a school’s tapestry.
The final study in this section is by Cawelti and Protheroe (2001). It
reiterates the findings of a great deal of educational research on transforming
schools. Some of these findings include focused leadership, high student
performance expectations, clear achievement goals, and the analysis of student
performance data to modify instruction. Cawelti and Protheroe also indicate how
exceptionally difficult it is to achieve these successful manifestations of change.
The selection requirements for Cawelti and Protheroe’s (2001) study included
the need for the district to serve a significant number of students from low-income
families and to have demonstrated significant improvement in student achievement
68
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
for ail student groups during the past five years. It is significant that no large urban
districts met the selection criteria. Large urban school districts in Sacramento and
Houston came closest to meeting the selection requirements. Due to the progress
made by the districts in Sacramento and Houston Cawelti and Protheroe included
them in their study of four other districts that did meet the selection requirements.
Each of the six districts chronicled by Cawelti and Protheroe (2001) uses
different strategies along with common elements to improve the success rate of their
students. The first district is the Brazosport Independent School District (BISD)
which serves 13, 500 students in 11 elementary schools, 2 junior high schools, 2 high
schools and 1 alternative school. The ethnic composition of BISD is 55% Caucasian,
34% Hispanic, and 9% African American. From 1991 to 1997 BISD gradually
piloted Total Quality Management principles with an eight step data-driven process
to transform its student achievement success rate.
The basic principles of Total Quality Management are discussed in the
section on business research. The eight step instructional process employed by BISD
consists of an orderly cycle that begins with disaggregated student data, then moves
to developing an instructional timeline, with instructional focuses emphasized in the
instructional timeline. At this point in the process grade-level assessments, both
commercially and district developed, are administered to test for knowledge and skill
mastery. Once the level of mastery is determined students receive mastery or tutorial
instruction. To complete the cycle teachers periodically maintain the tutorial or
enrichment instruction process to develop a high level of mastery and a continuance
69
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
of monitoring the quality of instructional focus and the continued progress of
individual students.
The second school district in the study is the Twin Falls School District
(TFSD). It has 6, 850 students in 6 elementary schools, 2 junior high schools, 1 high
school, and 1 alternative high school. Ninety-three percent of the students are
Caucasian and only 6% of the students are Hispanic. Twenty-three different
languages reflect the primary language spectrum of this district.
The Twin Falls District used a seven-goal focus to significantly turnaround
student achievement. The seven goals of the Twin Falls School District are: (a)
Aligning articulating, and coordinating the curriculum, (b) using training and
effective classroom practices to implement instruction such that student learning is
improved, (c) designing and administering assessments that instill accountability in
students for their learning and that instruction is aligned to the district curriculum,
(d) redesigning the grade reporting process so that student progress is appropriately
communicated to all stakeholders, (e) improving the lines of communication, (f)
developing an environment that encourages teamwork, and (g) maintaining a safe
caring, and orderly learning environment.
It took approximately four years for the Twin Falls School District to make
dramatic changes to levels of student performance. Multiple strategies were at the
core of change with an emphasis on supporting change in the classroom that also
manifested in central office leaders changing roles from supervisors and inspectors to
facilitators and planners. Curriculum alignment, standards-based curriculum, a clear
70
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
line of accountability, extensive use of data, and collaborative teams were perceived
as the building blocks for successful change.
The third district in Cawelti and Protheroe’s study (2001) is the Ysleta
Independent School District (YISD). The Yselta Independent School District
instructs 47, 000 students in 35 elementary schools, 11 middle schools, and 7 high
schools in which 87% of the students are Hispanic, 72% are eligible for free or
reduced lunch, and 23% are Limited English Proficient.
A shared belief system is understood as being at the core of the YISD
turnaround. Its belief system entails expecting and demanding success from
students, no matter what their background, and a belief that all students can be
successful with a rigorous curriculum. The idea that improvement in student
achievement requires tenacity on the part of teachers and students strikes at the heart
of the traditionally held beliefs that background deficiencies are acceptable reasons
for poor student performance.
The insertion of a new belief system changed the Yselta Independent School
District. The change occurred from top to bottom. Change began with the
superintendent who first made sure that the central office management accepted the
new belief structure, and then the school site administrators. With management
behind the change, affecting change in teachers and teaching became a natural
consequence.
Curriculum alignment with state standards was only part of the solution in
YISD. Innovation became the clarion call for the district and a genuine interest in
71
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
improvement was cultivated in a variety of stakeholder groups. Stakeholders sensed
that their participation was valued by the formal district structure and as a result
action plans became an on-going process for collaborative efforts. In effect, YISD
understood that leadership is not relegated to certain echelons of an organizational
hierarchy. Problem solving ownership was broadly based. The YSD story is an
example of how important leadership from the top can be.
The fourth school investigated in this study is the Barbour County School
District (BCSD). The Barbour County School District educates 2, 700 students in 6
elementary schools, three middle schools, and one high school. Ninety-eight percent
of the students are Caucasian and 60% of students throughout the district qualify for
free or reduced lunch.
In 1997 schools in the Barbour County School District (BCSD) registered
some of the lowest scores in the state on the SAT 9 achievement exams. In one year
the BCSD schools dramatically improved their scores throughout all of the grade
levels, nearly reaching the state averages. The superintendent of the Barbour County
School District played a critical role in turning around the achievement level of
students in his district. Deliberately not overreacting to the dismal scores the
superintendent first clearly described the achievement problem to all school staff.
He then made sure that they had the tools they needed to improve the levels of
learning. He finally realigned central office support to address the achievement
issues.
72
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Five actions are attributed to the foundation of successful change efforts in
the Barbour Count School District. Those actions are: (a) Aligning the curriculum to
problem areas, (b) emphasizing instruction in reading involving a site Reading
Mentor teacher, (c) interventions from K-3m that include extended-dav programs, in-
class tutors providing one-to-one or small group assistance, (d) structured test
preparation for students and parents, and (e) an emphasis on continuous
improvement. In addition to these five actions the superintendent of BCSD also
connects successful change to the acceptance of the achievement problems and the
need to change by all teachers and administrators. This was something that he had
never experienced, to the same degree, In 30 years of experience in education.
The Houston Independent School District (HISD) and the Sacramento City
Unified School District (SUSD) complete Cawelti and Protheroe’s study (2001) of
districts that consistently improved their students’ achievement. The Houston
Independent School District serves 210, 000 students with 260 schools in which 53%
of the students are Hispanic and 34% are African American. Nearly three-quarters
of the students qualify for free or reduced lunches. The Sacramento City Unified
School District (SCUSD) enrolls 51, 000 students in over 70 schools. Twenty-five
percent of the students are Hispanic, 25% Asian, 25% African American, and 25%
are Caucasian.
The HISD attributes the success of improvement in student achievement
between 1994 and 2000 in mathematics’and reading to: (a) Setting high achievement
standards for all students, (b) restructuring accountability for improved student
73
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
achievement so that the school site personnel are allowed expanded site-level
budgeting powers and principals are provided with monetary incentives for improved
student performance as opposed to the possibility of stringent consequences for
inadequate progress in student achievement.
The SCUSD attempts to generate accountability by requiring that all district
employees sign a pledge agreeing to work together to realize improvement in student
achievement. The district establishes annual performance goals and awards of gold ,
silver, or bronze pennants are made to site personnel based on the degree of progress
made toward achieving district improvement targets.
The Sacramento City Unified School District also considers its nine key
factors as integral to its ability to improve student achievement. Those nine factors
are: (a) Core beliefs and trust, (b) high standards, (c) quality instruction, (d) staff
development enabling the development of new skills to support improvement, (e)
site-based decision-making and budgeting, (f) parent participation, (g) community
supports and services, (h) data-driven culture to make plans and decisions, and (I)
accountability for results fashioned by rewards, interventions, and sanctions.
Cawelti and Prothroe (2001) identify three key interrelated elements that
foster successful change. Those elements are establishing high standards, using a
knowledge base, and restructuring the accountability system. Cawelti and Protheroe
also list six characteristics common to all of the districts in their study. Those
characteristics are paraphrased as follows: (a) Leadership from the top develop and
promote a shared belief in high expectations and a focus on actual improvement, (b)
74
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
decentralized management and budgeting which increase accountability by better
focusing school personnel to student results, (c) align curriculum to state framework
and item-by-item, student-by-student test results, (d) assist teachers in administering
interim assessments, providing timely intervention, and providing adequate practice
to increase skill retention, (e) sustain change efforts over a period of years, and (f)
reflect on test items incorrectly answered by students to inform instruction.
This study by Cawelti and Protheroe (2001) is valuable because it does look
at districts that manifest significant change throughout the district. As with most
educational research, however, it provides common characteristics and no clear
answers as to how to achieve those characteristics or how to implement changes in
organizational structure so that structural changes actually occur system-wide.
Many of the strategies previously reviewed in both the educational and
business literature are apparent in the Cawelti and Protheroe (2001) study. The case
study shows a variety of school districts, not just schools, that are successfully
improving student achievement. These districts are valuable examples of successful
system-wide improvement.
Exemplars of Successful Change Efforts in Business
The following four examples of successful systemic change are presented
because of their similarities to a variety of urban public school settings. IBM
represents a large organization with decades of little or no motivation for change and
years of tradition that was built on a hierarchy that is known for its top-down
management. The New York Police Department is also a large service organization
75
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
with a tradition of top-down management. The Marine Corps mirrors all of the
previously described characteristics of the New York Police Department except that
it has an even more rigid reputation as a top-down hierarchy. Finally, It’s Your Ship:
Management Techniques From the Best Damn Ship in the Navy (Abrashoff, 2002),
describes how the successful systemic change of a unit is accomplished in the
context of a large and rigid, top-down organizational environment. Observing how
successful change is managed in these environments, through the eyes of the
principal change agents, can provide valuable understandings to successful
organizational change.
Who Says Elephants Can’t Dance? (Gerstner. 2002).
IBM was poised to be broken into many smaller companies when Louis
Gerstner, Jr. became CEO in 1993. Gerstner made several key observations about
IBM prior to and during implementing organization-wide change. He encountered a
corporate culture insulated from the market place and rigid in its adherence to its
traditional hierarchically top-down management. There was never a mention of
culture, teamwork or leadership in the IBM setting. When Gerstner arrived at IBM
he found fear, uncertainty, and a preoccupation with internal processes at every level
of the organization. Gerstner also found a bloated and inefficient redundancy laden
hierarchy that supported geographic fiefdoms, in which individuals promoted
individual agendas, and rubber-stamping was the rule. He also discovered that
financial responsibility was difficult to achieve because no consolidated budget
76
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
existed. The overarching positive observation was that IBM had some of the most
talented individuals in the industry.
Gerstner developed several strategies to accomplish a top to bottom change in
IBM’s culture. Some of the key strategies included creating an environment of open
exchange of ideas at and across all levels of the hierarchy, product and sendee value
by strategically employing information and knowledge internal and external to the
organization, promoting a problem-solving focus that relies on teamwork, and
sponsoring the ability of the organization to productively manage conflict and
opposing ideas, instead of suppressing them.
Modeling and training were the means by which Gerstner implemented the
desired changes, which were simultaneously accomplished at multiple levels of the
organization. Gerstner believed that the top leader should be highly visible
throughout the organization, articulating and demonstrating the changes. Mandatory
training for management to echo the CEO’s change actions included the four topics
of (a) Focus to win, (b) mobilize to execute, (c) sustain momentum, and (d) have
passion for the business. Gerstner learned after a few years that the training in these
areas needed to be simpler than initially structured and that the core topics needed to
be written in terms of every day actions and procedures.
The four training topics are important in that they represent individual actions
in critical areas pertinent to systemic change. Focusing to win is concerned with
developing insights into client needs, creative thinking to meet those needs, and
persistence in achieving them. Mobilizing to execute addresses the issues of
nn
i /
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
developing team leadership, teamwork, decisiveness, and straight talk. Sustaining
momentum develops skills in coaching, building organizational capability, and
cultivates personal dedication, while the core competency addresses the issue of
passion and work.
It took almost a decade for Gerstner to turn around IBM. Reflecting on that
period of time he acknowledged that two of the most difficult changes were (a)
Developing a leadership cadre that would actively participate in problem solving and
(b) reallocating resources to support strategic change and increase overall corporate
profitability. Other strategies that proved difficult in changing the culture of IBM
were retaining talent and knowledge while significantly downsizing personnel and
maintaining clear and continuous communication with all IBM employees so that
everyone understood the reasons for the radical changes from tradition, what those
changes were, and how the changes would be realized..
The results of Gerstner’s leadership were the creation of a “counterintuitive
corporation” that exhibits the contrasting characteristics he described in the 2001
Annual Report. The report describes IBM as big and fast, continually learns,
changes, disdains the constraining aspects of bureaucracy, rewards results, covets
talent, and manifests a passion for its activities. This represents both the
characteristics of a successful systemically implemented change and a blueprint for
such change.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Turnaround (Bratton,. W., 1998).
When Bill Bratton became police commissioner of the New York City Police
Department (NYPD) crime in the “Big Apple” was acknowledged by all to be out of
control. In only 27 months Bratton reduced crime by 33% and the murder rate by
half. This dramatic change was the result of an effective systemic change in the
NYPD organization.
Upon arriving, Bratton found the NYPD to be a dysfunctional organization in
which lack of trust was rampant and a continuous stream of scandals paralyzed the
department. All of these factors contributed to the city’s dismal crime figures. He
found that the centralized bureaucracy did not even extend power to precinct
commanders. Department leadership was inflexibly linked to preserving the status
quo. Morale was at an all-time low and the organization was fractured into
numerous fiefdoms.
In response to the NYPD’s condition, Bratton first assembled a hand-picked
team to thoroughly analyze the Department. The team consisted of carefully selected
insiders and outsiders. This group of individuals was also responsible for creating
and implementing a change plan. A more detailed analysis of the NYPD revealed an
organization resistant to creativity and an historical deployment of critical police
units that covered crime on a 9:00 to 5:00 time span.
Five other key findings included: (a) Prime objectives of the highest levels of
the Department were to avoid external criticism and protect the good name of the
Department, as well as, the careers of the most senior officers, (b) rank and file
79
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
officers did not believe that the Department supported them, (c) the greater the
distance from headquarters the less the trust, (d) officers felt that there was a major
difference between what their supervisors say and what the officers actually needed,
and (e) there was no evidence of a consistent focus on crime reduction. This last
observation was dramatically exemplified by priority lists of the seven most
important goals of rank and file officers versus departmental heads. The number one
priority for rank and file officers was reducing crime. Reducing crime was seven out
of seven for departmental heads.
Bratton’s leadership cadre assembled reengineering teams consisting of more
than 300 people, all of whom were selected from every rank in the NYPD. These
teams were responsible for focusing on making changes that would positively
support the significant reduction and prevention of crime. The focus areas included
productivity, discipline, training, community partnership, paperwork, equipment,
uniforms, technology, and integrity.
Bratton had several intermediate objectives in mind as a means to reducing
and preventing crime. He wanted to return a sense of pride, commitment, and
respect to the department. He wanted to empower and train individual officers to be
able to make appropriate decisions in the field. He also wanted to challenge all
department personnel to find ways of attaining seemingly unreachable goals. He
believed that the key to realizing goals was training and the taste of successfully
achieving goals at all levels of the organization.
80
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
To guide training, Bratton determined that answers to three questions would
achieve both intermediate and ultimate objectives. Those questions were: (a) Where
and when are the crimes occurring? (b) Are police efforts coordinated based on
question “(a)”? and (c) Are coordinated efforts working to reduce crime? These
questions generated a need for accurate and timely information, a rapid response to
crime, effective strategies applied to crime situations, and continuous follow-up and
assessment of results of the applied strategies. Bratton believed that training must
address both the understanding of the intricacies involved in the three questions and
the corresponding needs.
The catalyst for a cohesive systemic overhaul of the NYPD was Bratton’s
creative use of technology. His weekly computer statistic meetings (“Compstat”)
became the critical element that repeatedly answered the three core questions
discussed in the previous paragraph. They also assisted in determining policy needs
and the effectiveness of coordinated efforts. The Compstat meetings also provided a
forum for precinct commanders to initiate creative solutions to factual crime
problems and review the impact of their precincts’ efforts to respond to crime. In
fact, during this intense interactive process the worst criticism would come if no
innovative efforts had been attempted to reduce crime since the previous Compstat
meeting. Innovation and collaboration between precincts were actions most coveted
during the Compstat interaction. Dramatic reductions in crime were the resultant
outputs of the strategic changes Bratton and his teams implemented throughout the
New York Police Department.
81
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Gerstner and Bratton both rely on training, reallocating resources and their
personal participation in the change and improvement efforts. They both also
describe the importance of teamwork and the effective use of data to improve their
organization’s performance.
Corps Business (Freedman, P., 2Q00).
Excellence in leadership continues to be the cornerstone of the Marine Corps’
fabric. Recent changes in the demands placed upon the Marine Corps necessitate
radical changes in how marines must be trained to perform their missions. Military
operations are becoming increasingly complex as the distinction between military
and humanitarian aid is blurred and as technological advances allow the enemy to be
a faceless one that exploits conventional military superiority with non-conventional
tactics.
Rigid adherence to the traditional top-down hierarchy no longer allows the
Marine Corps to achieve its goals. To better achieve its objectives the Corps
refocuses its leadership throughout its organization, regardless of rank. In fact, the
Marine Corps established a “counterintuitive” blend between a rank-respected chain
of command and a high respect for an individual’s ability to make decisions, no
matter what his or her rank.
One of the primary responsibilities of all Marine Corps officers is to develop
leadership capabilities In the marines they command. Thirty management principles
are at the heart of the Corps’ efforts to develop effective leadership from top. to
bottom. These thirty principles can be catalogued into the following nine groupings:
82
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(a) Training, (b) teamwork, (c) communication, (d) technology, (e) innovation, (fj
questioning, (g) decision-making, (Ii) external resources, and (I) competitive edge.
The following paragraphs will briefly describe each category.
Training is paramount to the Marine Corps5 implementation of its 30
principles. Ten out of the 30 principles can be placed under the rubric of training.
Developing an organization’s capacity by developing talents is the foundation of the
Corps’ training program. Training every individual to be able to react effectively to
rapidly changing situations that require an immediate response is a priority that is
directly related to the types of missions that has become a staple for the Corps.
Based on research, the Corps has developed a “rule of three” to facilitate operating in
a chaotic environment. This rule states that most individuals can handle no more
than three responsibilities during stressful times.
Training in the Marine Corps commences with challenging each individual
from the very beginning of his or her Marine Coips career. The Corps calls it “trial
by fire,” which is believed to both assess the appropriate organizational fit, as well as
cultivate a bond between the organization and the individual. An extension of “trial
by fire” is the concept of providing training scenarios that are more extreme than
what would probably be encountered during a real mission (“extreme training”).
The emphasis on cross-training creates a depth of skills and knowledge that
provides versatility and insightful decision making capability throughout the
organization. Management decision-making skills are further enhanced by
instructing Individuals how to recognize situational patterns based upon previous
83
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
experience. This “decision by analogy” allows individuals to make effective
decisions without requiring a briefing on every new situation. As a by-product of
training the Corps firmly believes in developing the ability in all of its personnel to
work out the details for accomplishing the specified objective (“management by end-
state and intent”). By continually educating and training people at all levels
(“distributing competence”) the Marine Corps develops managers from within,
making sure that promising candidates gain experience in multiple areas such as
hiring, training, promotion, and personnel transfers (“personnel functions as a
stepping stone”).
Despite the importance of rank in the Marine Corps it is ingrained in its
highest ranking managers that they should continually communicate to the lower
ranking personnel that the rank and file are the heroes that ultimately accomplish the
tasks (“glorifying the lower levels”). This acknowledgment of and respect for the
lower ranks lessens the hierarchical chasm between the ranks and supports the ethos
that everyone in the organization is working on an aspect of the same mission
(“establishing a core identity’ and “instilling values that support the mission”).
The training and communication throughout the Marine Corps organization
create an environment that demands questions, even between subordinates and
supervisors (“demanding to be questioned”), and rewards failure as a means of
supporting innovation as a problem solving technique (“experimenting obsessively”
and “rewarding failure”). As a result, the Corps is able to utilize contradictory ideas
to discover the most effective solutions (“cultivating opposing traits”). Another
84
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
important offshoot of the Marine Corps training is that plans are kept simple and
flexible to adapt to changing situations (“keeping plans simple and flexible”) and any
organization edicts are considered to be living concepts, not cast in stone (“making
doctrines living things”).
As with all enterprises, technology maintains an important role in the Marine
Corps (“leveraging technology”). Contrary to popular belief, however, the Corps
trains for the use of technology by also ensuring that its people can function at high
levels without technology. The creative implementation of technology also requires
receiving feedback and direction from outside sources (“getting an outside
perspective”). It is interesting to note that reliance on outside perspectives extends
beyond technology and is actually an important characteristic of the Marine Corps’
maintenance of its on-going renewal of esprit de corps. The Corps hires outside
consultants to regularly review their training procedures. For such an apparently
insular community the Marine Corps thrives on the objectivity that can result from
the acceptance of perspectives external to the Corps.
Three of the Marine Corps management principles are directed specifically
toward competition and winning. They are: (a) Using one’s strength to combat
another’s weaknesses, (b) surprise and disorient the opposition, and (c) make tempo
a weapon. These three principles clearly stem from, an organization that is capable of
learning about itself and about other organizations.
After 226 years the Marine Corps continues to be chosen for special missions
tailored to their unique capabilities as a fighting organization to respond to rapidly
85
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
changing conditions. By the number of successful CEO’s who received their training
in the Marine Corps, it could also be said that the Corps is one of the best
management training programs in the United States (Freedman, 2000).
It is clear from the above description that the Marine Corps, also utilizes
training to develop broad leadership that generates teamwork and the ability of its
members to use data to make quick decisions in life threatening situations. Although
the staff in the four case study schools are not making life threatening decisions, they
do make decisions that impact life-long futures of their students.
It’s Your Ship (Abrashoff. P.. M.„ 2002),
The title of Captain Abrashoff s book, It’s Your Ship, succinctly references
the main thrust of strategies to change his ship from an unknown among numerous
naval ships to a ship with the highest level of battle preparedness rating in the entire
Navy. The ultimate goal of Captain Abrashoffs’ strategies is to empower all of the
USS Benfold’s crew members so that each one feels that the USS Benfold is his or
her ship, not just the captain’s.
At the conclusion of only 20 months, Captain Abrashoff transformed a ship
of 310 men and women from a dysfunctional ship with low morale to a ship that
displayed consistently high morale and high ratings. Eleven of the chapter headings
delineate the components of Abrashoffs systemic change strategy. Those heading
are: (a) Take Command, (b) Lead by Example, (c) Listen Aggressively, (d)
Communicate Purpose and Meaning, (e) Create a Climate of Trust, (f) Look for
Results, Not Salutes, (g) Take Calculated Risks, (h) Go Beyond Standard
86
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Procedures, (i) Build Up Your People, (j) Generate Unity, and (k) Improve Your
People’s Quality of Life.
From Captain Abrashoffs perspective taking command means knowing your
boss’s expectations and knowing the people you supervise as the most valuable
assets of vour organization. Taking command means breaking down the barriers
between people caused by organizational ranking. It means understanding that there
is always a better way to do things and that better ideas are not necessarily the
purview of an higher rank. Taking command is also understanding that change is
extremely difficult and requires that there is continuous dialogue that explicates the
reason for the change, how it will impact each individual and how it will benefit the
organization and the individuals within it.
In large command and control organization such as the U.S. Navy many
changes are top-down in nature and are not necessarily beneficial. Abrashoff
counters these detrimental change mandates by encouraging innovation and
creativity to work around the initial directives. Innovation and creativity are
generally the only way in which progress is made in an organization. Despite this
fact, many organizations promote people based on their ability to make no mistakes.
In Abershoff s mind, such people may actually be playing it safe by not taking risks
and thereby not contributing to organizational progress.
The Navy’s sclerotic bureaucracy traditionally prevents communication
across ranks, discourages innovation and creativity to maintain discipline, and
ultimately disempowers the majority of Navy personnel who are in lower ranks.
87
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Abershoff took the innovative risk of breaking down communication barriers
between ranks and encouraged collaborative problem-solving by making it the status
quo on his ship to be creative across ranks and to make review of positive and
negative outcomes a regular learning process.
Captain Abershoff had to personally model actions that empowered all
crew members to openly communicate ideas an to make decisions that normally
would be the responsibility of officers. This modeling was important for both his
officers and non-officers. Abershoff made clear that if a decision or idea might
injure an individual, damage the ship or other government property it had to be
cleared by him, personally. Beyond these provisions, his modeling consistently
showed every crew member that the USS Benfold was their ship too.
Abershoff found that by opening the lines of communication, emphasizing
creativity in the pursuit of continuous progress, discipline improved, the overall and
individual performances improved and everyone took responsibility for their actions
without the need to be micromanaged. He felt that training was the key, especially
training by his own example. Some of the positive results of this unconventional
management strategy included the USS Benfold operating under 75% of its budget,
setting new records for deployment training (19 days versus 52 days), increasing
retention rate from 28% to 100%, and posting one of the highest promotion rates
after being far below the Navy’s average prior to Abershoff s command. This level
of performance could only be achieved and propagated as long as the changes that
supported this degree of improvement were systemic throughout the ship. Such a
88
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
change blueprint offers possibilities for systemic change at a departmental or unit
level, as well as on a broader scale.
Overview of Critical Characteristics to Systemic Change
Not everyone is happy with the use of marketplace change models being
superimposed on educational environments. Alfie Kohn (1993) specifically targets
Ms disagreement with employing business models in education with respect to TQM.
His fundamental argument against using TQM is that it reduces learning to
quantifiable numbers. What he fails to realize is that TQM is about improving
processes that lead to improved outcomes. The only valid criticism that Kohn is
justified in making is that education reduced TQM to outcome numbers. This does
not invalidate the correct .application of the progressive improvement process
fostered by TQM and other business improvement models in education settings. In
fact, besides Warwick’s (1995) and Jenkins (1997), the American Society for Quality
records numerous successful change stories in school settings that are based on the
principles of TQM (ASQ Education Division, 2003).
Business research and examples of successful system wide changes in
business help in developing an understanding of how education can be more
successful in manifesting systemic change. Any substantive change in an
organization must be system-wide and necessarily requires changes in the
organization’s culture. The research and examples previously cited describe
elements of an organization that must be integral to any strategy that attempts to
manifest change that will improve the organization’s product or services.
89
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Research-based Analysis Variables
The selection of appropriate variables to analyze the four case studies must
consider the content and focus of those studies. The primary focus of the case
studies is to investigate the successful use of student performance data to improve
student achievement. The broader areas of analysis of the case studies include on
going training, periodic student assessments, data collection and collaborative data
analysis. Additional variables include leadership functions and tracking student
improvement.
Each category has sub-variables. Training includes curriculum alignment to
standards, data analysis, using data to inform instruction and understanding the
benefits of local change implementation. Leadership variables include active and
visible involvement of management, providing clear improvement goals and
developing a large instructional leadership base. Finally, the area of high
performance qualities includes using resources to support improvement actions,
emphasizing continuous improvement, having high expectations for improvement
and providing multiple opportunities to learn. These variables are the mainstay of the
case-study analysis.
Chapter 3 will introduce analytic methods based on several of the processes
listed above. These methods will be used as a means to analyze the quality of system
wide change implementation in each of the case studies under investigation as well
as a framework for future research to evaluate system wide change implementation
in schools and site-level improvement assessments.
90
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 3
METHODOLOGY
Introduction
The purpose of this cross-case study is to review four case studies from the
perspective of investigating the degree to which particular district data-driven change
designs are implemented throughout purposively selected schools. Such an analytic
study is ultimately meant to enhance the ability to target critical areas in the
education system so that change implementation can be increased and the
momentum of improved student performance can be maintained or accelerated.
This study attempts to use the quantitative and qualitative data presented in
the selected case studies to evaluate the level of data-driven strategy implementation
at each school site and to determine the effectiveness of those strategies in affecting
continuous improvement in student achievement.
This cross-case study is guided by the following four research questions:
1. How frequently are data-driven improvement trainings provided? What
are the characteristics of those trainings?
2. How often is student performance data collected for analysis? What are
the characteristics of the data collection process?
3. How often does the school instructional staff meet in teams to
collaboratively analyze student performance data? What are the
characteristics of the analysis?
91
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
4. To what degree were school student improvement goals attained since the
implementation of data-driven improvement strategies?
The over 40-change and data-driven variables described in Chapter 2 of this
dissertation can be organized into six genera! categories (refer to page 2 of Appendix
A). Those categories include: (a) Leadership, (b) high .performance qualities of
organizations, (c) on-going training, (d) knowledge of change process and (e) data
collection and analysis. Figure 1, on the following page, represents the more
detailed depiction of the hypothesized relationship of the variable to each other and
their relationship to promoting continuous improvement in student achievement
within the framework of data-driven improvement strategies. These variables are
from the literature review and are the most applicable for analyzing the four case
studies of this dissertation because they are closely aligned to the content and focus
of those studies.
The four case studies under scrutiny were completed by University of
California Ed.D. candidates during the 2001-2002 academic year. All of the case
studies employed the identical methodological procedures. The research instruments
used by each case study include interviews, artifact collection and analysis, informal
interviews, Teacher Questionnaire, Stages of Concern questionnaire, and a post data
collection researcher rating form. The case studies also had the same research focus,
which was to investigate the quality and level of implementation of district designs
to improve student achievement by means of using student performance data.
92
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Leadership
• Active and visible involvement of management
® Leadership provides clear improvement goals
• Development of broad instructional leadership base
High Performance Qualities of Organizations
• Resources support improvement efforts
• Emphasis on continuous improvement
® High expectations________________
On-going Training
Continuous Student Improvement
Alignment of
curriculum to
standards
Use of data
analysis
Data Collection & Analysis
Collaborative analysis
Benefits of local control over change
Knowledge of Change Process
Figure 1. Conceptual framework of relationships between the variables used
to analyze the case studies.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The case study format of the individual case studies substantiates the use of a
cross-case study approach. The strengths of the case study format include the ability
to provide in-depth information about the object of study in its natural context
(Cresweii, 1994; Gall, Borg, & Gall, 1996). The format also includes what Gall et
al. describe as “multivocality,” which is well suited for studying outliers. In
addition, case studies provide for multiple data sources, termed triangulation and the
inductive nature of the qualitative elements of case studies allows for the emergence
of theories that suggest more extensive research in future studies (Cresweii, 1994;
Gall, Borg, & Gall, 1996; Isaac & Michael, 1997).
The major weaknesses of the case study design are in the areas of external
and internal validity. Researchers are generally in agreement that the limited number
of units under study in a case study constitutes an inherent weakness to one’s ability
to generalize beyond the case study (Isaac & Michael, 1997). The concept of
triangulation, therefore obtaining data from multiple sources, is viewed by some
researchers as a means of strengthening the external validity of a case study (Anfora,
Brown & Mangione; Gall, Borg & Gall, 1996). Although the cross-case study
format is seen by some researchers to be a design that increases generalizability
(Miles & Huberman, 1994; Patton, 1990), well-designed experimental research is
universally considered to be the strongest format for external validity.
Experimental research is also considered to be the best approach for internal
validity. Even though Patton (1990) downplays infernal validity concerns with
94
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
respect to the case study design by stating that the level of skill of a qualitative
researcher greatly impacts the internal validity of a case shady, the fact that case
studies are susceptible to subjective biases does significantly compromise the ability
of case studies to demonstrate causal relationships (Gall et ah, 1996). The essential
nature of internal validity is considered to be an even greater issue than that of
external validity (Pedhazur & Schmelkin, 1991).
Sampling and Population Descriptions
The four individual case studies were selected because of their common
methodological procedures and research questions. They were also selected because
they are representative of urban elementary schools and high schools in the public
school systems throughout the country. The school districts were chosen because of
their accessibility to the researchers and their ability to meet the case study
population requirements. Those requirements included being an urban public school
district with both a diverse student population that includes some combination of
students from low-income families, multiple ethnic groups, English Language
Learners, and a data-use design to improve student achievement. The schools were
purposively selected by central office senior administrators based on the perceived
success of those schools in using student performance data to improve instruction
and student learning.
The districts involved in the four case studies are all urban school districts
with student populations that range from 15,000 students to 45,000 students. Three
95
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
of the districts are unified school districts that serve K-12 students, One of the
districts serves only high school students.
The school student populations in the case studies also vary. School sizes
range from 523 students to 18,900 students. Within that range, the elementary
schools serve 523 students and 1,006 students, while the secondary schools
servel,8Q0 and 2,300 students. One of the two elementary schools has an Hispanic
student population of more than 66%. The second elementary school has a
Caucasian population of more than 87%. The Hispanic student populations in the
two high schools are almost 37% and 50% of the general student population. Data
on student populations in the four case study schools also indicate that one
elementary school and one high school have minority student populations of 88%
and 70%, while the other elementary and high schools have predominantly
Caucasian student bodies. This data is summarized in Table 1 .
Table 1
Student Demographics in Each Case-Study School and in Each School District.
Student Populations Divine ES(K-12) Southern ES
(K-12)
South HS(9-12) Leno HS(K-12)
Total District Population 15000 45892 15400 18900
Total School Population 523 1006 1800 2300
Caucasian 12% 87.4% 30% 54.3%
Hispanic 66.2% 6.2% 50% 36.7%
Black
----
0.5% 8% 1.8%
Asian
----
3.8% 10% 4%
Other
----
2.1%
----
4.2%
Total Minority Population 88% 12.6% 70% 45.7%
Note. ES= Elementary School, HS= High School, (9-12)= High School District, (K-
12)= Unified School District.
96
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
individual Case Study instrumentation
All of the four individual case studies seek to answer the following three
research questions:
1. What is the District design for using data regarding student performance,
and how is that design linked to the current and the emerging state
context of assessing student performance?
2. To what extent has the district design been implemented at the district
level?
3. To what extent is the district design a good one?
The researchers of the four individual case studies utilize nine data collection
instruments to answer the three research questions. Those instruments
include: (a) Interviews with district and site administrators and selected
teachers, (b) data flow mapping throughout the district, (c) collection of
related quantitative data, (d) situated interviews, (f) teacher questionnaire, (g)
States of Concern questionnaire, (h) post data collection Research Rating
Form, and (i) post data collection innovation configuration summary (refer to
Appendix B).
Cross-case Study Instrumentation
An Implementation Analysis Scale was developed to measure key
frequencies and percentages of quantitative data found in the four case studies. This
data is connected to the four research questions as shown In Figure 2 and represents
a first tier analysis of the data in the four case studies.
97
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data-Driven improvement Training (Research Question #1)
• 3 points: Monthly or more frequent
• 2 points: Three to four times per year
• 1 point: Two times or less per year
Collection of Student Performance Data (Research Question #2)
• 3 points: Daily collection of student work and by teacher informal observation
@ 2 points: Monthly collection of student achievement data
• 1 point: Quarterly or less frequent collection of student data
Collaborative Analysis of Student Performance Data (Research Question #3)
• 3 points: Teacher analysis of student data on a weekly or more frequent basis
• 2 points: Monthly analysis of student data on
• 1 point: Quarterly or less frequent analysis of student data
Attainment of Student Achievement Goals (Research Question #4)
• 3 points: 68% or greater of student achievement goals attained
• 2 points: More than 34% and less than 68% of student achievement goals
attained
• 1 point: Less than 34% of the student achievement goals attained
Figure 2. Implementation Analysis Scales
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The implementation analysis scale ratings are based on the research literature
findings that the more frequent the training, collection of data and collaborative
analysis of data the greater the positive impact in the area of the desired change
(Brae, 2002; Joyce & Showers. 2002: Schmoker, 2001; Senge, 1990). In this case,
the desired improvement is in the area of student achievement. A second tier of
analysis is based on the remaining variables displayed in Figure 2. The combination
of first and second tier analyses variables provides a broad base from which to
understand the data in the case studies.
Case study displays are used to catalogue and analyze data from each case
study (Miles & Huberman, 1994). A meta-matrix is also used to assemble the data
from all of the individual case studies to facilitate analysis of data across case
studies (Miles & Huberman, 1994). As a replica of Miles’ and Huberman’s meta
matrix, a Case Study Summary analysis form was created. This summary contained
both the quantitative data from the Implementation Scale Ratings and the qualitative
data pertinent to each of the research questions. By using this display format,
answers to the four cross-case study research questions were sought and a ranking of
relative implementation levels of data-driven change strategies was established.
Data Collection and Analysis
Data for the cross-case study is collected directly from copies of the initial
drafts of the Ed.D. dissertations presented to the approval committee at the
dissertation defenses. Each case study is first reviewed to answer the four cross-case
study research questions. As data relating to each cross-case study research question
99
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
is found, post-its, labeled 1-4, identify the research question that is addressed on each
page of the reviewed dissertations. The quantitative and qualitative data is then
spread on the Case Study Summary display. After all of the case studies are analyzed
the data summaries from each case study is reviewed to provide a cross-case study
analysis of the data findings.
Obstacles to Cross-Case Study Data Analysis
The lack of complete data responses to the questionnaires, interviews and
API scores in each case study presents significant challenges to this comparative
study. A complete display of this data, by each case study, would provide evidence
as to the comparable nature of the data from all of the studies. It would also allow
for a more accurate assessment of questionnaire response, avoiding the reliance on
averaged questionnaire response scores.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 4
THE FINDINGS
Introduction
All public schools and public school districts are actively involved in
implementing school-wide and district-wide changes that continuously improve
student achievement as evidenced primarily by scores on state assessments. For the
most part, research provides description of characteristics common to “high
performing” schools or schools that are making significant and consistent gains in
student achievement. Despite the wealth of information that current research
provides, it falls short of providing educators and schools with a core set of actions
that provide a basis upon which systemic student improvement can be achieved
throughout the public school system.
Systemic research that investigates the interconnected nature of
organizational practices is infrequent in the realm of education (Chatterji, 2002).
Unfortunately, it is this very form of research that offers a promise of improving the
ability of schools and districts to affect system-wide improvement in student
achievement. The focus on a manageable set of processes that lead to systemic
change is a missing link to successful change strategies, extending change efforts
beyond mere structural changes that mimic the characteristics of “successful”
schools.
101
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The need for schools to monitor the success of their efforts to improve
student achievement is increasingly more important as public schools continue to be
in the public’s eye and under the constraints of state and federal guidelines that
require significant improvement in student achievement for all student populations.
Attaining mandated student achievement goals with student populations that
historically under-perform is integrally tied to quality training for data-driven
analysis, the regular collection of data for analysis, and the use of collaborative
teams to analyze the data and creatively plan improvement strategies based on the
data (Fox, 2002; Nadler, 1997; Schmoker, 2001).
This chapter will first analyze the data in four case studies based on the four
research questions. These questions are based on the first tier of analysis discussed
in Chapter 3. Then a second tier of analysis will be applied to the case study data
The four research questions guiding this analysis are:
1. How frequently are data-driven improvement trainings provided? What
are the characteristics of those trainings?
2. How often is student performance data collected for analysis? What are
the characteristics of the analysis?
3. How often does school instructional staff meet in teams to collaboratively
analyze student performance data? What are the characteristics of the
analysis?
4. To what degree were school student improvement goals attained since the
implementation of data-driven improvement strategies?
102
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The findings to each research question now follow.
Findings by Case Study and by Research Question
Case Study #1, Southern Elementary School (Dombrowen 2002)
Research question #1: How frequently are data-driven improvement trainings
provided? What are the characteristics of those trainings?
Interviews with key site personnel indicate that training to use student
performance data occurs in several formats at Southern Elementary School. Every
year, the week before school begins, teachers receive training from the district on
how to prepare charts and rubrics for results from state tests and the CORE battery of
tests. The school district also requires six trainings for teachers over the course of
the school year. This is reported by the Executive Director and the Vice Principal.
The Vice-Principal also trained teachers at Southern Elementary. The Vice-
Principal provided instruction to teachers on how to analyze data and how to prepare
flowcharts that describe student performance over several years. The frequency of
these trainings is not described by the case study researcher.
Teacher Questionnaire data confirmed that teachers attended staff
development trainings in the past six months. However, teacher responses to item 17
of the Teacher Questionnaire communicate that teachers are not totally in agreement
with the statements that the school offers regular professional developments to
increase the awareness of new data practices. Several of the interviewed teachers
remarked that only some of the district offered training sessions focused on data.
103
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Five more items on the Teacher Questionnaire evaluate teacher response to
data training in less formal settings. Although all of the Teacher Questionnaire
statements concerning professional developments, except item 18, describe the
frequency of training in terms of “frequent” or “frequently,” input from this data
source provides additional information on data related training. Table 2 below
provides teacher responses to items 17 through 22 from the Teacher Questionnaire.
The scores shown are averaged mean scores, as presented in the case study.
Table 2
Teacher Questionnaire Statements 17-22
Item # and Statement_________________________ 5 Point Scale
17. The school offers frequent professional develop
ment to raise awareness of new data practices. 2.92
18. I have attended professional development training
In the past six months related to new data practices. ?
19. I frequently discuss new data practices with teachers
who are about as experienced as I [am]. 3.08
20. I frequently discuss new data practices with teachers
who are more or less experienced than I [am], 2.86
21. I frequently discuss data practices with teachers in
different disciplines from mine. 2.05
22. School administrators have assisted me in implementing
new data practices. ?
Note. The points displayed are average mean scores, as presented in the case study.
“?” indicates that there was no data found for these statements. “2” = Disagree
Somewhat, “3” = Agree Somewhat and “4” = Agree Strongly.
104
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Out of the six Teacher Questionnaire statements involving data related staff
developments this case study provides responses to four statements. It is not
surprising that there is some disagreement with the statement that teachers frequently
discuss data practices with teachers in different disciplines. A higher agreement
rating would be expected in middle schools and high schools where
departmentalization is present. Responses to statements 19 and 20 indicate that
teachers at Southern Elementary School do discuss data practices with other
teachers.
The Stages of Concern Questionnaire provides information, indirectly,
regarding data related trainings. Out of 36 statements, 31 statements are related to
data training in that training or the need for more training is directly related to the
levels of teacher concerns in statements 1 through 9 and 23 through 36. This case
study provides the highest frequency of high levels of concern in the areas of the
need for more information and the practical consequences of adopting the new data
practices.
The case study researcher reports that only 11 of the 38 teachers completed
the Stages of Concern questionnaire. The majority of the 11 respondents expressed
only light to moderate concerns which counters the feedback described in the
previous paragraph that would indicate a general lack of teacher satisfaction with the
data-related training. Nevertheless, the school should investigate whether the
concerns expressed by the 11 respondents are more widespread.
105
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Interview and questionnaire data indicate relatively strong ratings on Data
related inservices. With more than four trainings a year in the use of student
performance data to Improve student achievement, the area of training at Southern
Elementary School receives the highest 3 point cross-case study Analysis Scale
rating.
Research question #2: How often is student performance data collected for analysis?
What are the characteristics of the analysis?
The Executive Director commented that the district stores, in a district data
base, the results of SAT-9, SAT-9+, CORE and CELDT assessments. The SAT-9,
SAT-9+ and CELDT assessments are annual state assessments, while the CORE
battery of tests is given by every school in the district throughout the school year. In
fact, a district directive indicates that these assessments are to be utilized at every
school as a means of increasing student performance and as vehicles for determining
the learning needs of individual students.
Teacher Questionnaire responses also provide insights in to how often
student performance data is collected. Respondents recorded average mean scores in
the Agree Somewhat range to statements 7 and 8. This supports the teacher
perception that data collection is a function of school activity at Southern Elementary
School.
The district scores the results of the school CORE tests and forwards that
scoring to each school. The researcher does not provide data collection frequency
for the CORE tests, however, it is known from the interviews that teachers map
106
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
student progress on a weekly basis. These progress flowcharts are presented to the
Vice Principal on a monthly basis. In accordance with the rating scale of this cross
case study, the weekly mapping of student progress qualifies the school for 3 points.
Research question #3: How often does school instructional staff meet in teams to
collaboratively analyze student performance data? What are the characteristics of
the analysis?
The district’s Executive Director stated in an interview that grade-level teams
at every school worked collaboratively to reflect on their efforts to improve learning
by using student performance data. The Vice Principal also disclosed that she
developed grade-level teams to reflect on student performance data. Contrary to the
comments of the Executive Director and the Vice Principal, interviews with teachers
at South Elementary School indicated a lack of familiarity regarding collaborative
teams working with student data. In contrast to this initial finding, teacher
interviews also disclosed that all teachers understood the process of analyzing
student data. Subsequent interviews also revealed that teachers actively support the
district’s plan to use data-driven strategies to improve student achievement.
The Executive Director describes data interpretation as a barrier to
improvement in student learning. Teachers, on the other hand, stated in interviews
that the primary challenge to data-driven improvement is the inadequate time
provided to accomplish the data strategies. This indicates awareness by both district
administration and teachers at the school site that some problems exist in the area of
data analysis. In addition, the perspective of teachers at the school site should be
107
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
weighted more heavily that that of the district Director. This is the case because the
teachers are the ones who have to implement the collaborative data strategies.
No specific frequency of analysis is described in the case study. Based on the
data presented in the case study, the area of collaborative data analysis is given a
rating scale score of one.
Research question #4: To what degree were school student improvement goals
attained since the implementation of data-driven improvement strategies?
According to the Executive Director of Elementary Operations, the district
focus is on improving student performance on the SAT-9 and SAT-9+ state exams.
The measure of improvement on the state assessments is determined by adequate
increases in scores and increases in each school’s Academic Performance Index
(API). The API ranks schools based on a school’s overall score. The API ratings
also take in to consideration school demographics, such as parent education, and
student primary language. Another feature of the API rating system is that the state
of California sets minimum score improvement goals for each school. These
minimum scores include school-wide improvement and improvement in all
statistically significant student sub groups (Los Angeles Unified School District,
2003).
The Vice Principal at Southern Elementary School stated in an interview that
she expected student performance to exceed the minimum state expectations. A
review of school documents reveal that in 2000 the school’s API score was 851. In
108
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2001 the school made a 21-point increase in its API score, with 86% of its students
scoring above the national average in reading and mathematics.
With a score of 851 in 2000, Southern Elementary received the state’s highest
ranking of 10. Even in 1999, the school had no official API growth target with a
score of 808. Despite the lack of formal external improvement goals the 21-point
increase is indicative of a school that is continuing its student performance
improvement efforts.
The Assistant Principal believed that analysis of the scaled scores of the
SAT-9 and CORE assessments in combination with the emphasis on standards-based
curriculum were the primary reasons for the improvement in student performance.
The implementation of the CORE battery of tests in 1998 and the continued increase
in state test scores lends support to the Vice Principal’s belief that the CORE exams
were integral to the improvement in student performance.
Although the case study presents no disaggregated data, Southern Elementary
is described, in the case study, as a school that continues to meet all of the state’s
student learning goals and all of its own student performance goals. Southern
Elementary School therefore qualifies for an Implementation Analysis Scale rating
of 3.
Table 3 below summarizes the implementation level of data-driven
improvement training at Southern Elementary School.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 3
Case Stadv #1 Data Analysis Summary (Southern Elementary SchoolV-Training
Qualitative Data Support
+ District requires training a week before school begins
+ District requires that teachers attend 6 trainings per year
+ Vice Principal also trains teachers
- Teachers are not in M l agreement that the school offers
regular
professional developments in data practices
Note. “+” = positive characteristic and = negative characteristic
Data presented in table 3 indicate that the area of training is a strength for
Southern Elementary School, with respect to its efforts to use student performance
data as a means of improving student achievement. The district and the school offer
more than six professional developments per year which train teachers how to use
data to improve instruction so that student learning needs are better met. The fact
that some teachers indicate their perception to be that the district does not offer
regular data practice training could be due to an ineffective monitoring system to
ensure that teachers actually attend the professional developments.
Implementation
Scale Rating
3 points
(6 times per year)
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 4
Case Study #1 Data Analysis Summary (Southern Elementary School)— Data
Collection
Implementation
Scale Rating Qualitative Data Support
3 points + District collects state test results
(weekly) + Weekly flow charts of student progress provided to Vice
Principal on a monthly basis
+ Teachers perceive data collection to be a part of the school
environment
Note. “+” = positive characteristic and = negative characteristic
The requirement that teachers collect weekly data on student progress and
provide that data to an administrator once a month creates a data-rich environment at
Southern Elementary School. The regularity of internally generated data is
undoubtedly responsible for the teacher perception that data is an integral aspect of
the school environment.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 5
Case Study #1 Data Analysis Summary (Southern Elementary School)— Data
Analysis
Implementation
Seale Rating Qualitative Data Support
1 point + Teachers seem to understand the process of
Case Study #1 Data Analysis Summary (Southern Elementary SchoolV-Data
Analysis
Implementation
Scale Rating Qualitative Data Support
(no frequency analyzing student data
provided)
+ Teachers seem to discuss data practices with other teachers
- Teachers indicate that there is inadequate time to
accomplish a variety of data strategies
- Teachers seem to be unfamiliar with regular collaborative
data analysis
Note. = positive characteristic and = negative characteristic
112
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Analysis of student performance data received the lowest implementation
rating out of the four dimensions. Although data shows that teachers generally
understand the steps in how to analyze data, teacher perceptions regarding
collaborative data analysis suggests that they are provided with few opportunities for
collaborative analysis. This possibility is further supported by the time requirements
for developing weekly flow charts described in Table 4. An additional cause for the
low implementation rating is that the case study did not specify any frequency of
collaborative data analysis.
Table 6
Case Study #1 Data Analysis Summary (Southern Elementary School)— Goals
Implementation
Qualitative Data Support
+ School goals to continuously increase API score
+ School expected student performance goals to surpass the
minimum state expectations
+ 21-point increase in API score
+ 86% of students scored above national averages in reading
and math
- No disaggregated data displayed in case study
Note. = positive characteristic and = negative characteristic
Scale Rating
3 points
(more than 68%
of students)
113
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 6 summarizes data on Southern Elementary School’s ability to achieve
goals in improved student achievement since the advent of data-driven instruction.
In 1999. the Southern Elementary School API score was already high at 808 out of a
possible 900 points. Southern Elementary School continued to surpass minimum
State improvement goals with a 43 point growth in State test results in the Spring of
2001 .
The fact that in the Spring of 2000 86% of students at Southern Elementary
School scored above the national averages in both Reading and Math emphasizes
that the majority of the student body is achieving at levels above the national
averages. The lack of disaggregated data in this case study makes it impossible to
determine if certain groups are falling behind, academically. This absence of data is
clearly a function of the case study and not the school.
Case Study #2. Divine Elementary School (Thompson. 2002)
Research question #1: How frequently are data-driven improvement trainings
provided? What are the characteristics of those trainings?
The second phase of the district’s policy on a Curriculum Review,
Improvement and Implementation policy states the need for teacher staff
developments to improve student learning by using multiple assessments to monitor
student learning and develop instructional interventions. In addition, as a II/USP
school, Divine Elementary School’s School Site Council developed five
improvement goals in 1999. One of the goals included providing training for
teachers so that they can effectively use interim assessment data to drive instruction.
114
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
This training goal was implemented by the school due to its II/USP status. At the
time of Thompson’s research, interviews with teachers and the site principal revealed
concerns regarding the continuation of funding to support teacher training after the
II/USP state designation is discontinued.
The district provides staff development to teachers that is monitored through
teacher evaluations. In turn, site administrators are expected to provide teachers with
staff development that develops their ability to use disaggregated student
performance data. The district and the site principal provide staff development to
teachers on an on-going basis. Although the district’s data-use policy allows the
school sites to formulate their own strategies and procedures for data use, teachers at
Divine Elementary School expressed the concern that district trainings are too
generic. The teachers did indicate tat they found the Literacy Coaches, Facilitators,
Curriculum Specialist and Mentor to be positive supports in their acquisition of data
based skills. These individuals play important professional development roles.
Twenty-four teacher responses to the Teacher Questionnaire statements
indicate that more data-use training is needed, despite the frequency of trainings
already offered. Thompson’s (2002) research reveals that the district annually
provides for more than four data related inservices and that the school site provides
additional trainings. Teachers identify another source of data training concerns.
They report that they have no on-demand access to disaggregated student
performance data. One of the recommendations made by the case study researcher is
115
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
that the district should fund improvement in site technology so that teachers can
readily access student performance data.
Teachers at Divine Elementary School do believe that their involvement in
the state mandated II/USP intervention program developed many of the data skills
they currently utilize. The II/USP intervention initiated data-driven change strategies
at Divine Elementary School prior to the district’s entry into the use of student
performance data to improve student achievement.
Research question #2: How often is student performance data collected for analysis?
What are the characteristics of the data collection process?
The district’s Strategic Master Plan mandates the use of multiple assessments
to improve student achievement. Action Plans are written to accomplish the goals of
the Strategic Master Plan. One of the Action Plans is to conduct norm-reference and
criterion-referenced assessments for K-12 students. The position of Coordinator of
Student and Program Evaluation is an offshoot of the district’s goals and action
plans. The position is responsible for creating, organizing and conducting the
structuring, dissemination and analyzing of student performance data district wide.
At the direction of the Coordinator of Student and Program Evaluation for the
district, all schools began using criterion-referenced tests in Mathematics and
Language Arts. Additionally, with passage of the Public Schools Accountability Act
(PSSA), under-performing schools were identified and supported by the II/SUP
program. This program supported the use of a uniform research-based interim
assessments from the Consortium on Reading Excellence (CORE). When CORE
116
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
proved instrumental in dramatically raising state test scores, the District mandated
their use at all elementary schools.
Several times each year the CORE battery of assessments are employed in
conjunction with other multiple measures of student achievement. The timeline for
administering the CORE assessments is initiated by a set of assessments. Based on
the results of these initial assessments some students are given additional
assessments and provided with differentiated instruction.
Required formative assessments are scheduled three times a year, in either
September or October, January or February and May. The Language Arts topics
assessed range from letter recognition to phonemic awareness, phonics decoding,
fluency, blending and comprehension. Kindergarten assessed 11 different topics,
first grade assessed 9 areas, and second through fifth assessed four distinct areas.
The majority of these assessments were based on CORE. Required assessments
were also found in mathematics, writing and English development, along with the
traditional progress reports.
The school board developed a three-phase implementation plan to foster a
program of instruction that prepares students for success at all levels of the district’s
curriculum. The second phase of this program is divided in to nine steps. The eighth
and ninth steps include the use of evaluation to determine the effectiveness of
instruction by using multiple assessments to monitor student progress toward
meeting standards. This process encouraged the alignment of curriculum, instruction
and assessment with state standards.
117
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The district also collects all of the Interim student performance data from
each school site In June. The data is disaggregated by using the DataWorks
computer program and the disaggregated data is provided to the principals in
September. Principals then share the data with their teachers. The Coordinator of
Student and Program Evaluation is available to all principals and teachers to answer
questions regarding the data. Teachers at Divine Elementary express concerns
regarding the usefulness of data from the previous school year.
Teacher responses to the Teacher Questionnaire statements concerning data
collection range from the “Agree Somewhat” to “Agree Strongly.” However,
teacher interviews expose a concern regarding data collection. Teachers believe that
there is a lack of understanding and appreciation on the part of district administration
regarding the value of the qualitative and quantitative student performance data
teachers collect on a daily basis. Teachers believe that the district’s preoccupation
with a limited data set is encouraged by the state’s high-stakes monitoring system.
It is clear from interview and questionnaire data that teachers at Divine
Elementary School collect measurable student performance results on a monthly
basis. This frequency of data collection at the site level warrants a 2 rating on the
Implementation Analysis Scale, for data collection.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Research Question #3: How often does school instructional staff meet in teams to
collaboratively analyze student performance data? What are the characteristics of
the analysis?
The district expects schools to use the student assessment data administered
throughout the school year to inform and shape instruction so that the academic
needs of all student populations and each student are met. The district’s data base
does some preliminary analysis of student performance results by indicating the
percentage of students by grade level who are scoring in the categories of
“Advanced,” “Proficient,” “Basic,” “Below Basic,” or “Far Below Basic.” These
groupings parallel the state’s groupings. The DataWorks program also disaggregates
data by subject, teacher, gender, grade level and student population. Not only can
the District Coordinator of Students and Program Evaluation monitor the
effectiveness of instruction at each school and throughout the District, teachers and
site administrators at each school can monitor the success of their instruction. The
district provides additional support to schools to assist their efforts to use data to
improve student performance.
The student performance data provided by the district is compiled in a
notebook that is also provided to each school’s Curriculum Specialist. The
Curriculum Specialist also reviews the school’s performance data and provides the
school with data analysis assistance. Another form of data analysis assistance to
schools is found in the district’s general analysis guidelines. Schools are to use a
six-variable analysis guideline in analyzing student data. These variables include
119
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
demographic characteristics, testing environment, student and staff attitudes toward
testing, levels of test taking skills, degree of curriculum alignment with state testing
and the relative effectiveness or quality of instructional programs. A district bulletin
explains the procedures for controlling these six variables as a function of data
analysis.
The II/USP plan for Divine Elementary School provided its teachers with
similar skills to analyze student performance data two years prior to the
implementation of the district’s plan. Searching for student achievement patterns
and developing instructional modifications to improve those achievement patterns
were already practiced skills at Divine Elementary School.
The multiple interim assessments necessitate the analysis of student
performance data throughout the school year. For students not satisfactorily meeting
grade-level standards, assessment results trigger target interventions that attempt to
assist students in making progress towards demonstrating acceptable facility with the
state content standards. The district continually communicates to the school sites
that analysis of the multiple data streams is solely for the purpose of reflecting on
instructional effectiveness so that instruction can be tailored to student learning
needs and, in turn, student achievement levels will improve.
Teachers at Divine Elementary meet monthly to review assessment data and
consider instructional implications of those test results. Interview and responses to
teacher questionnaire statements reveal concerns with sufficient time to
collaboratively analyze data and plan target interventions. These responses indicate
120
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
that teachers do not believe that the once a month data analysis is adequate for
providing the most beneficial instructional interventions for students. Teacher
interviews repeatedly reveal that Divine Elementary School’s teachers also link the
high teacher turnover with difficulties associated with the site’s data use plan.
Although the district sets aside 45 minutes every Tuesday for collaborative
planning, interview respondents unanimously report that this time is not honored for
collaborative planning and data analysis. A deeper probing of teachers during
interviews also revealed a dissatisfaction with the Pullian Group’s data-
disaggregating program. Teachers expressed frustration with the bugs in the
program because program problems prevented them from using the technology that
they believe would be an excellent tool for quickly organizing and analyzing student
achievement data. In the meantime, teachers collect their own data and do not
depend upon the flawed system provided by the district at the school site.
Divine Elementary School receives 2 points for monthly data analysis
meetings, since the weekly meetings are not consistently used for data analysis and
instructional modifications.
Research question #4: To what degree were school student improvement goals
attained since the implementation of data-driven improvement strategies?
In 1996 the district instituted its first Strategic Master Plan. This plan acted
as the foundation for the setting of district student performance goals and the
development of strategies to achieve those goals. In general, the district’ s Strategic
121
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Master Plan’s goal is to have all students exhibit continuous improvement in
mastering the state standards.
The case study indicates that all students at Divine Elementary School are
progressing towards high performance. Performance data indicates that all student
groups are also making progress towards high-performance goals. Based on this
data, Divine Elementary School is given 3 implementation Analysis Scale points.
Table 4 summarizes the quantitative and qualitative data provided by case study #2.
Table 7 summarizes characteristics that describe the implementation level of
data training. The school provides the majority of data training for its teachers. The
district does provide training in how to use disaggregated student data. The district
also provides support for the use of student performance data by making available to
schools Literacy Coaches, Special Facilitators and trained Mentors.
The II/USP state program provides data-driven training. In fact, one of the
concerns expressed by both teachers and site administration is what will replace the
support the program provides after it expires in a year. The case study provides no
evidence that the district is assisting the school in preparing for the future decrease in
data-based training soon to be caused by the expiration of the II/USP program.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 7
Case Study #2 Data Analysis Summary (Divine Elementary7 SehooD-Traininp
Implementation
Scale Rating Qualitative Data Support
1 point + As a II/USP school, training to enable teachers to effectively
use
(no frequency interim student data to drive instruction was required in 1999
stated) + District provides staff development to teachers that develops
teacher ability to use disaggregated student performance data
+ Site principal provides staff development to teachers on an
on-going basis
+ District allows each school to develop its own procedures
for analyzing student performance data
+ Teachers found Literacy Coaches, Facilitators, Curriculum
Specialist and mentors to be helpful in developing data-based
skills
- Site concerns regarding continuation of II/USP training after
the II/USP grant ends
- Teacher Questionnaire responses indicate a need for more
data use training
Note. “+” = positive characteristic and = negative characteristic
123
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 8
Case Study #2 Data Analysis Summary (Divine Elementary School)- Data
Collection
Implementation
Scale Rating
2 points
(monthly)
Qualitative Data Support
+ Formative assessments given 3 times per year
+ All schools in district are using criterion-referenced tests in
Mathematics and Language Arts
+ Use of CORE interim assessments several times per year
+ Results from initial testing trigger CORE testing
+ District requires the use of assessments to monitor the
effectiveness of instruction
+ District formally supports the use of multiple norm-
referenced and criterion-referenced assessments to improve
instruction and student achievement (Strategic Master Plan)
+ Surveyed teachers indicate teacher positive perceptions
regarding the level of data collection
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 8 (continued)
- Teachers report no on-demand access to disaggregated
student performance data
- Teachers indicate in interviews a belief that the district is
overly interested in the results of state testing and that district
personnel do not appreciate the importance of more regular
site level assessments
Note. “+” = positive characteristic and = negative characteristic
Table 8 describes 12 positive characteristics regarding data collection at
Divine Elementary School. The use of multiple criteria referenced assessments
throughout the school year provides teachers at Divine Elementary school with on
going student performance data to use to improve instruction.
Teachers expressed two concerns regarding data collection. First, teachers
had no direct access to disaggregated student performance data and second, teachers
believed that the district did not value the on-going assessments as much as the
annual State assessments.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 9
Case Study #2 Data Analysis Summary (Divine Elementary SchoolV-Data Analysis
Implementation
Scale Rating Qualitative Data Support
2 points + District expects schools to use and administer assessment
(monthly) data throughout the school year to inform instruction
+ District database gives preliminary analysis of test results based
on performance categories that parallel state categories
+ Curriculum Specialist reviews performance data from each
school and provides schools with data analysis assistance
+ District provides a 6-variable analysis method to be used at all
schools
+ II/USP plan previously provided school with data analysis
procedures similar to the district
+ District regularly communicates to teachers that the purpose of the
multiple assessments is to help data analysis
+ Teachers also meet monthly to review
assessment data to consider changes in instruction
- Weekly analysis time is not always honored for data analysis
- Teacher questionnaire responses indicate a need for more time to
analyze data and develop instructional modifications
- Teacher dissatisfaction with program that disaggregated data
Note. “+” = positive characteristic and = negative characteristic
126
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 9 describes five ways in which the district supports analysis at the
school. The district expected schools to analyze student data during the school year.
The district provided all of its schools with an analysis method to apply to student
performance data.
Although teachers meet monthly to analyze data and modify instruction
based on data analysis, teachers express the need for more time to analyze student
data and alter instruction accordingly. To underscore the need for more analysis
time is often used for topics other than data analysis.
Table 10
Case Study #2 Data Analysis Summary (Divine Elementary School)— Goals
Implementation
Scale Rating Qualitative Data Support
3 points + General goals: all students have continuous improvement in
(all students) their mastery of grade-level state standards
+ Researcher indicates that all student groups are making
progress towards high-performance goals
- No disaggregated data presented in study
Note. “+” = positive characteristic and = negative characteristic
Multi-year-disaggregated data would be important in establishing the actual
progress all students are making at Divine Elementary School. The absence of
disaggregated data is again the function of the case study and not the school. A
127
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
review of actual data might change the rating In the area of attainment of student
achievement goals, especially as they relate specifically to minimum State goals.
Case Study #3, South High School (Sevillano. 2002)
Research question #1: How frequently are data-driven improvement trainings
provided? What are the characteristics of those trainings?
The Assistant Superintendent of Educational Services in the district requires
all principals to be fully trained in the data analysis method mandated by the district
and developed in collaboration with the Orange County Office of Education. The
district also provides teachers with training in how to utilize standards-based
assessment strategies. Teachers at South High School receive training in the summer
and throughout the year. The majority of teachers interviewed expressed the belief
that the training they received in data analysis helped increase API scores.
The district assessment trainings impact the various departments at South
High School, as evidenced by interviews with teachers in the English, Math,
Reading, History and Spanish departments. Many of the departments at South High
School hold formal and informal meetings concerning the use of data to modify
instruction. Departmental participation in data analysis is extremely important for
data-driven analysis to be implemented on a school-wide basis.
South High School’s Single School Plan, which is reviewed yearly, formally
established support for professional development. Despite the formal support of the
Single School Plan, interview data describe the need for additional funding to
support teacher training on the use of data. Interviews also disclose that many
128
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
teachers only participated in data training once. Some teachers, at the time of
Sevillano’s (2002) research, still had not participated in even one professional
development on data use to improve student achievement.
Teachers interviewed who participated in the staff development on how to
analyze the data from the state exams stated that the training assisted them with
interpreting the student achievement results. Twenty-seven out of 32 teachers
corroborated that the school does offer professional development to increase teacher
facility with .the new data practices. Despite this response, responses to the Stages of
Concern Questionnaire statements describe teacher desires to leam a variety of data
analysis techniques.
Stages of Concern responses express a need to provide further explanation
regarding the implementation of the district’s data analysis process. The concerns
include the ability to manage the implementation of the district’s design to.use data
and whether there is adequate funding to support the district’s data-use design.
Teachers also expressed concerns over how the district’s data-use design may
adversely impact students. All of these concerns could be addressed by additional
district and site professional developments.
The district offers monthly inservices to district staff regarding emerging data
practices. Sevillano (2002) observed that teachers who have participated in data
inservices are the most comfortable in interpreting data and that those same teachers
are more inclined to use data to inform instruction. These teachers are also more
likely to call the district’s research department to obtain additional student data. The
129
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
comfoit level to call the research department is important because only site
administrators and counselors have internet access to the student data that is on the
student information system in the district’s data base.
Teacher responses to the Teacher Questionnaire support the effectiveness of
the district and school site data trainings. Sevillano (2002) reports that the most
effective aspect of the data procedures is the link made between standards,
curriculum and instruction. Teacher Questionnaire responses support regular
inservices on standards generated by both the district and the school site. Teacher
interviews corroborated that the expectation is high for department chairs to
participate in the data-standards-based workshop.
Despite the reports from a variety of sources, researcher observations
describe the need for additional teacher training to increase teacher knowledge and
skills on making data-driven, standards-based classroom decisions. The case study
confirms the teachers’ beliefs that instructional modifications, based on student
performance data, is the primary reason to use data so that student achievement can
improve.
Despite Sevillano’s findings that teachers believe that additional training is
needed, beyond the monthly trainings that the district and school site offer, South
High School receives 3 points on the Implementation Analysis Scale. It is possible
that the teachers’ need for more training is an indication that they are not yet
comfortable with using data, which may only be a function of inadequate time spent
130
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
in training or training that doe not .meet actual teacher needs, as opposed to needing
more frequent training.
Research question #2: How often is student performance data collected for analysis?
What are the characteristics of the analysis?
The district’s Department of Educational Services is responsible for
providing data to the schools. The Department of Educational Services uses the
DataWorks software to prepare disaggregated data reports of student performance.
This data is provided to principals. Although principals receive the student
performance data teachers have no direct access to the data. They must go to the
school office to obtain data. Teachers are able to request more student performance
data from the district’s Technology Department, but with the assistance of site
administrators who have direct internet access to student data.
The East Orange County School District collects criterion based and
authentic assessments. An example of the authentic assessments is the district’s
annual writing exams, which are administered to every student. Other authentic
assessments include informal observations of students at work, projects and
portfolios. The authentic assessments, in tandem with a variety of state assessments,
provide teachers with a base of multiple assessments with which to chart student
performance and determine appropriate instructional interventions.
Interviews with the teacher school leadership team members provide
feedback as to what elements of student performance data collection they believe to
be most important in producing student achievement gains. The English teacher
131
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
states that the regularly administered school-wide authentic assessments such as
portfolios, projects and collaborative assessments are effective improvement
instruments, The Spanish teacher described the district’s alignment of instruction
and assessed standards. The Social Studies teacher described the school and district
examination of national assessments and growth targets with respect to minority
students as an effective improvement tool. The reading teacher referred to the
regular assessment data forwarded from the district, while the math teacher cited the
importance of end of unit exams. What is clear is that the district and the school
provided a broad range of assessment data so that the various departments could
select a form of student data that is considered appropriate by each department to
assist as a guide to inform instruction.
Student achievement assessments are administered two times a year and
forwarded to the district to be organized by the DataWorks program. Results from
the state tests, which includes a norm-reference test (SAT-9), the California
standards tests (CST), an English test for English Language Learners (CELDT), and
the High School Exit Exam (CAHSEE), are presented to the school board once a
year. This presentation of data provides the board with student performance data
that measures student performance in several areas that are pertinent to monitoring
the effectiveness of the district’s instruction. With this data, the school board can
make more informed decisions regarding where to improve instruction in it schools.
The district requires that the schools develop their own data collection
strategies. This site-driven model allows schools to tailor their data collection to
132
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
their individual needs. This laissez-faire approach may not be appropriate for all
schools. Some schools may require more guidelines than others. Districts must
monitor the level of guidance and top-down structure that should be provided to
schools based on the degree of improvement in student performance achieved by
each school. This form of site level self determination has been effective for South
High School.
Teachers at South High are using a variety of assessment tools to collect
student achievement data on a daily, weekly and end-of-unit basis, in addition to the
results from state assessments and the annual data base provided by the school
district. This would qualify for an Implementation Analysis Rating of 3.
Research question #3: How often does school instructional staff meet in teams to
collaboratively analyze student performance data? What are the characteristics of
the analysis?
The cycle of reviewing student performance data begins with the Assistant
Superintendent meeting with each principal and content experts to review and
analyze data. The base data reviewed consists of a four-year comparison of
disaggregated state test scores. The Assistant Superintendent makes sure that all
principals are trained in a four-step data analysis method developed in collaboration
with a consultant from the Orange County Office of Education. With all principals
learning the same data analysis technique the chances are greater that the technique
will be transferred to teachers throughout the district.
133
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The district’s writing exam is an example of how the analysis of student
performance leads to changes in instruction. The district-wide writing assessment is
administered at the end of each school year. If students do not meet the district
standards in writing, they are placed in a special class the following school year so
that they can receive extra support in achieving writing proficiency. Although end-
of-year assessment is too late to guide students and teachers throughout the school
year, the ultimate result of this activity is an intervention makes a dramatic change in
instruction for individual students.
South High School’s leadership team developed its own plan for improving
the performance of the students who score in the lowest quartile on state exams.
Data on these students are disaggregated across content areas and departments. The
school leadership team used this data to implement specific strategies to improve the
learning of the lowest performing students. The leadership committee analyzes the
student data on a quarterly basis and their findings are provided to the staff on data
worksheets.
The results from the California High School Exit Exam and the California
Standards Tests are also analyzed twice a year. Sevillano (2002) indicates that the
leadership team meets monthly to review school-wide data. He does not specify the
amount of time made available to teachers to analyze student performance at the
departmental level. Interview responses reveal that standards-based analysis of
lessons and student learning is viewed by the teachers and administrators at South
High School as the primary focus that provides the most effective forum for
134
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
improved student learning. Teachers expressed the belief that analysis of student
performance data was most beneficial when done with other teachers. This
collaborative format was the predominant activity during both staff developments
and department meetings. Post-lesson teacher evaluations also continued the focus
on standards-based lesson structure and the gauging of student learning based on
grade-level standards. This too is a collaborative data analysis model that reinforces
the more frequently utilized model of teachers working in teams.
Even though the case study researcher found evidence of widespread analysis
of student performance data, teachers, nevertheless, indicated a need for more data
use inservices and collaborative time to actually analyze data. Quarterly student
performance data analysis appears to be the predominant analysis frequency at South
High School, but with the leadership team meeting monthly to review school-wide
data, the area of collaborative analysis receives a 2-point rating on the
Implementation Analysis Scale.
Research question #4: To what degree were school student improvement goals
attained since the implementation of data-driven improvement strategies?
Since 1999 through 2001, South High School has experienced a 34-point
increase in its API score. During two school years, the school has met the state’s
API growth targets. In addition to the mandated state growth goals, the district has
set its own student performance goals for each school site. The Assistant
Superintendent developed student achievement reports in conjunction with the
Technology Department. These reports were analyzed in consultation with the
135
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
teachers and principals at each school. Annual instructional goals and learning
objectives were set based upon the learning trends presented in the reports.
Although South High School teachers and administrator focused on
improving the achievement of those students in the lowest quariile, the real target for
improvement mirrored an achievement gap that is all too familiar in public schools
across the nation. The 2000-2001 API scores described a continuing achievement
gap between the school’s white students and its Hispanic students. Caucasian
students registered a 48-point increase, surpassing their minimum growth target.
Hispanic students, on the other hand, did not attain their minimum growth target.
The Hispanic student population represents over 50% of the school’s student
population. It is the student subgroup that prevented the school from qualifying for
state award money. It is also the subgroup that is specifically targeted by the school
for improvement.
Despite the emphasis on student performance goals, Sevillano (2002) reports
that teacher interviews reveal that not all teachers at the school are cognizant of the
site achievement goals. On the surface, this may seem contradictory. However, it
may also reflect two weaknesses in the level of data-driven improvement practices at
the school. First, the site spends a significant amount of time focusing on state API
targets. This represents attention to a one-time-a-year high stakes exam. This focus
does not easily translate in to day-to-day or monthly student achievement goals.
Second, the researcher does not describe any use of the state tests to narrow the
number of grade-level standards to be targeted by instruction for instruction for
136
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
mastery learning. When the state tests axe analyzed by item and standards the
voluminous number of standards to be taught and learned can be greatly reduced.
For some, approaching instruction from the standpoint of what is tested is
pejoratively referred to as teaching to the test. For others, having a greater alignment
between instruction and assessment is considered providing clear expectations for
both students and teachers (Institute for Learning, 2004). Current research shows the
most successful trend to be reviewing the state tests for those standards that will be
tested and then making sure that these so called “power” or “enduring” standards are
taught and emphasized in the curriculum.
South High school receives 3 points on the Implementation Analysis scale
because 74% of its student body achieved the State’s annual achievement goals, and
26% of its student body did not achieve its minimum achievement goals. This 26%
represents the Hispanic student population that was specifically targeted by the
school.
Table 11 describes ways that data training is supported at South High School.
Teachers receive data-use training in the summer and during the year. The Orange
County Office of Education collaborated with the development of data analysis
procedures. School administration makes sure that teachers in all departments
receive training in the use of data to Improve student achievement.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 11
Case Study #3 Data Analysis Summary (South High SchoolV-Training
Implementation
Scale Rating Qualitative Data Support
3 points + All principals required to be folly trained in data analysis
(mpnthly) + Teachers also receive training in the summer and throughout the
school year
+ Majority of interviewed teachers indicated that data analysis
training was helpful in raising school API scores
+ All departments received training in the use of data to improve
student achievement
+ School’s Single School Plan officially supports professional
development for data analysis
+ Teachers participating in data analysis trainings are more
comfortable using data than those who did not attend trainings
- Need for more funding to support increased time for training
- The district requires participation in data analysis training
but some teachers only participated in one training while some
had not participated in even one training
- Teachers expressed concerns about their ability to fully implement
data analysis procedures
Note. “+” = positive characteristic and = negative characteristic
138
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Teachers at South High School believe that data training is an important
aspect o f why API scores continue to rise. Despite this belief, some teachers stated
that they had not participated in most of the trainings. This underscores the need for
an internal monitoring system that ensures teacher participation in professional
developments. This lack of training could also explain teacher concerns regarding
their ability to folly implement data analysis procedures.
Table 12
Case Study #3 Data Analysis Summary (South High School)— Data Collection
Implementation
Scale Rating Qualitative Data Support
3 points + Teachers collect student achievement data on a daily,
(daily) weekly, and end-of-unit basis
+ A variety of data is collected and provided to each department
+ District is responsible for providing disaggregated data to schools
+ Students assessed two times a year, in addition to State testing
+ District requires each school to develop its own its own data
collection strategies
+ District presents data to school board once a year
- Teachers have no direct access to student performance data
Note. “+” = positive characteristic and = negative characteristic
139
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 12 describes characteristics of data collection at South High School
Teachers gather student performance data in a variety of subject areas that provide
all 'departments with data so that instruction can be differentiated to better meet
student learning needs. This inability of teachers to easily access student
performance data counteracts the ability of teachers to efficiently use the data
collected at the school site.
Table 13 describes characteristics of data analysis at South High School.
South High School teachers are provided with four consecutive years o f comparative
student data. Data is analyzed by a four-step process used throughout the district.
Student placement is determined by assessment results and the student leadership
team monitors the instruction and assessment results of students scoring in the lowest
quartile on State exams.
Table 13
Case Study #3 Data Analysis Summary (South High School)-Data Analysis
Implementation
Scale Rating Qualitative Data Support
2 points + Data analysis methods mandated by the district
(monthly) collaboration with Orange County Office of Education
+ Teachers are required to turn in assessment data notebooks
to principal
+ Assistant Superintendent and content experts meet with each
principal to review student performance data
140
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 13 (continued)
+ 4 year comparative data presented
+ 4-step data analysis method developed in collaboration with the
district and Orange County Office of Education
+ Next year’s student course placement is determined by
assessment results
+ Leadership team monitors progress and instruction of
students scoring in the lowest quartile on State exams
+ Data is disaggregated for each student across subject areas
+ Results from the High School Exitjexam, the State’s English
Language Development Test and the California Standards Test
are analyzed twice a year
+ Teachers view standards-based analysis of lessons as a primary
cause of improved student learning
+ Teachers expressed positive benefits from participating in
collaborative data analysis
- Teachers express a need for more data-usage training
- End-of-Year assessment results are inadequate for determining
instructional needs during the year
Note. = positive characteristic and - negative characteristic
1 4 1
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 14
Case Study #3 Data Analysis Summary (South High School)— Goals
Implementation
Scale Rating Qualitative Data Support
3 points + 34 point increase in API scores from 1999-2001
(74% o f students) + District set student performance goals for each school
+ Improvement of performance of Hispanic student population
is a major school emphasis
- Not all teachers interviewed were aware of the school5 s
student achievement goals
- Achievement gap continues between Caucasian and Hispanic
students
Note. “+” = positive characteristic and = negative characteristic
Table 14 describes the results of the school’s efforts to meet improvement
goals for student achievement. South High School shows an increase in API of 34
points over a two-year period. The school continues to meet the State’s growth
targets for two consecutive years. However, the school has yet to achieve its goal to
decrease the performance gap between Caucasian and Hispanic students. In fact, the
142
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Hispanic student population was the reason why the school did not qualify for the
State’s monetary awards in 2001.
South High School continues to make steady improvement in student
achievement, as measured by State assessments. When South High School is able to
significantly reduce this learning gap, it will be well on its way to improving student
achievement.
Case Study #4. Leno High School (Miller, 2002)
Research question #1: How frequently are data-driven improvement trainings
provided? What are the characteristics of those trainings?
The Principal, Assistant Principal and department chairs held meetings with
teachers to instruct them in emerging student assessments. The California High
School Exit Exam (CAHSEE) was one of the emerging assessments that received
attention during the meetings. These training sessions linked the assessment to
instruction by means of the state content standards tested.
The school district conducts three staff developments prior to the beginning
of each school year. The focus of the staff developments include analysis of State
exam results, standards-based instruction and analysis of interim district assessments.
Interviews disclose that the English and Social Science teacher leaders believed these
trainings to be very useful for developing instructional modifications that target
student learning weaknesses and helping address the Hispanic male achievement gap
that exists at Leno High School.
143
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
An increase in computer technology by means of the Digital High School
Grant initiated a series of training sessions that included instruction in data analysis.
Additionally, due to an accreditation process, staff was meeting every Friday
morning to revise -writing rubrics and practice analyzing student performance data.
These weekly meetings were considered training sessions by the teachers, Although
these weekly training sessions were considered to be pilot trainings, site
administration and teachers acknowledged the potential usefulness of such data-
driven activities. The accreditation training format also included twice a month
meetings. In order to provide site administration with a means of directly
participating in the training sessions, the Principal and Assistant Principals at Leno
High School are assigned to departments based on their areas of expertise.
Funding to support training in data-driven change is currently inadequate for
providing the entire staff at Leno High School with the skills to effectively analyze
data so that instruction and learning are positively impacted on department-wide
basis. This financial backing is also considered important for the development of
collaborative team analysis and the shadowing of fellow teachers to observe best
practices in a real classroom setting. Teacher release time is essential for both of
these activities.
Responses to teacher questionnaire statement indicate that teachers perceive
the data training as an important tool in their efforts to improve student achievement.
Teacher leaders, in fact, expressed a need for more data analysis training. Responses
to the Stages of Concern questionnaire describe strong concerns with respect to
144
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
teacher ability to actually implement the district’s and school’s design to use student
performance data to improve student achievement. Further training is the key to
resolving these concerns.
Weekly training sessions to learn how to revise writing rubrics and analyze
student performance would qualify for 3 points on the Implementation Analysis
Scale. The fact that these meetings are part of a pilot program has both positive and
negative ramifications. On the positive side, piloting a program before it is officially
implemented school-wide allows the teachers to explore and familiarize themselves
with both training techniques and the subject matter of those trainings. On the
negative side, a pilot program indicates that the training and the focus of the training
are not institutionalized. Data from this particular case study does not provide a
clear indication as to whether the school will be able to sustain the training initiated
by the accreditation process after the accreditation process is completed.
Research question #2: How often is student performance data collected? What are
the characteristics of the data collection process?
The school district’s accountability design requires multiple measures that
include criterion-based interim assessments that are aligned to the California content
standards. End of course assessments is one method of data collection encouraged
by the school district that is a mainstay at Leno High School, especially in the Math
and Social Science departments. Authentic writing assessments is another example
of multiple assessments found at Leno High School that are tied to State standards.
145
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The most widespread example of multiple assessments in the school district
axe the CORE assessments that the district directs all schools to administer during the
school year. The district Coordinator of Student Programs and Evaluation describes
the CORE assessments as criterion-referenced measurements. Scores from the
CORE assessments are forwarded to the district office in June for use during the
following school year. Unfortunately, the procedure of forwarding student
performance data from the CORE assessments to the district counteracts the potential
power of having regular student performance feedback throughout the year. This is
an area that the district needs to improve. Investment in the appropriate technology
could quickly remedy the lack of regular student performance data in a format easily
used by schools.
The district also requires student performance data collection throughout the
winter months. This data collection is made to provide information that is used to
identify instructional intervention that is needed by individual students. Teachers
also collect daily class work to support their efforts to provide appropriate and timely
instructional interventions.
Leno High School follows district policy by generating student data in the
form of lists that describe students who are meeting grade-level standards and
students who are not achieving grade-level standards. In addition, semester grades in
Language Arts and Mathematics are reviewed after being disaggregated by the
DataWorks program. The grades are then compiled in individual school site binders.
Schools also receive a School Achievement Summary School Report once a year.
146
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
This report provides schools with a brief summary of multiple measures over
multiple years.
Responses to the Teacher Questionnaire describe a perception that the
collection of weekly is an important activity for teachers in the use o f data to
improve instruction and student learning. The teacher emphasis on weekly
collection of data earns a 3-point rating in the area of Data Collection.
Research question #3: How often does school instructional staff meet in teams to
collaboratively analyze student performance data? What are the characteristics of
the analysis?
Leno High School utilizes the results from the State tests as the primary
means of determining student placement. However, it uses results from other
assessments to determine what type of instructional intervention each student
requires and in what areas of the curriculum. All assessments, whether state or
district generated, are analyzed from the perspective of State content standards. This
increases the need for district assessments and instruction to be aligned with State
standards.
In order to make uniform the data analysis process, district-wide, the school
district published a five-step data analysis process for all schools to use. The five
steps included: (a) Review test reports by subject matter, (b) departments identify
performance patterns in terms of quartile distribution, (c) departments determine the
degree to which subject matter content is successfully taught, (d) departments meet
to identify instructional areas requiring re-teaching and to establish progressive
147
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
student performance goals and (e) the departments create action plans to meet the
student performance goals they have set,
The district created a special data collection process for identifying secondary
students in need of instructional intervention. Student grades, results from CORE
assessments and daily student work are considered. At the middle school level
students at risk of not meeting grade-level requirements are scheduled for a Student
Study Team review. During this meeting, student strengths and weaknesses are
documented and improvement strategies are developed, while responsibilities for
implementing improvement strategies are assigned. This process begins in
November and ends in April.
The priority of the district and the school appears to be on successful
instructional intervention rather than retention. One student performance
accountability model used by Leno High School lists all individual assessments used
*
to track student performance with set cut point that establish a “meets standards” or
“does not meet standards” parameter. This model is applied to all students, including
English Language Learners, identified gifted students and those students receiving
Special Education services.
The weekly data analysis program, generated by an accreditation process,
was previously mentioned in the section under training. What is characteristic of all
of the data analysis activities at Leno High School is the active involvement of site
administration in some phase of the analysis process. In the Spring of 2001 the
principal requested, in writing, that all Algebra teachers submit a list of standards
148
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
that are most difficult for students to master. This action led to placing responsibility
on Algebra teachers to develop instructional interventions based on analyzing
student performance data. This personal involvement by the principal is arguably
one reason that the Math department is identified as an example of effective use of
performance data to improve student achievement.
The case study researcher reports that the principal at Leno High School
encouraged all department chair persons to develop intra- and inter-discipline
strategies to use interim assessments to guide instruction. This encouragement was
not as formal as was the request made upon the Algebra teachers but is nevertheless
an example of the principal’s involvement in the data analysis process. According to
Miller (2002), many of the departments meet both formally and informally, once a
week or more often.
The principal also influences the use of data to improve student achievement
by collecting specific data from selected teachers, every five weeks. This data
describes the distribution of grades in each classroom. The principal meets with all
teachers who have 20% or more D ’s or F’s or a classroom G.P.A. of 2.5 or less. The
case study researcher had the opportunity to observe one of the meetings between the
principal and a teacher, During the meeting the principal asked the teacher questions
like: Have the parents been contacted to help their children? How many of the failing
students are Hispanic males? What standards are the students failing? How can I help
you help these students?
149
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The final strategy initiated by the school administration to support teacher
efforts to improve student achievement was by encouraging teachers to observe each
other while teaching. In this way, teachers were able to obtain observational data in
terms of best practices in instructions and the learning outcomes that resulted from
that instruction.
The process of collaborative data analysis Is refined at Leno High School in
that special treatment is afforded the ninth and tenth grade students. Students in
these two grade levels are treated as one group of students who are instructed by and
whose progress is monitored by a team of teachers. Formal data analysis department
meetings are held once a week, while it is common for informal data analysis
meetings occur on a daily basis. Interviews with teachers who participate in the
ninth and tenth grade clusters provide anecdotal evidence that this team approach to
clustering students is successful at improving student performance. The case study
a
researcher indicates that the 10 -week data report also supports significant
improvement in student performance since the inception of the team-cluster model.
If students are better prepared in the ninth and tenth grades, performance
improvements should ultimately materialize in the eleventh and twelfth grades.
Teachers and site administration indicated a concern that the one-hour, twice
a month allotted for collaborative data analysis is inadequate. All interviewed parties
also expressed a belief that more time for collaborative data analysis is needed to
reach an effective data-use implementation level throughout the school. The use of
data to drive improvement is still a new process to schools.
150
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Teachers also commented in interviews that they found analyzing their own
student's performance provided important insight in to the effectiveness of their
standards-based instruction. Additionally, interviews indicated that the emphasis on
analyzing student performance data encouraged veteran teachers to assist less
experienced teachers in reflecting on their instructional practices, based on the level
of student performance exhibited by the data.
Leno High School’s Math department was considered by all interviewed
parties to be the exemplar of data analysis that drove instructional delivery. This
department had a Final Exam Evaluation Committee, which analyzed end-of-course
standards-based examination. This committee’s analysis covered all subjects offered
by the math department and often resulted in major modifications to the instructional
sequence. For example, certain chapters in a given subject might be moved from the
first semester to the second semester.
The Teacher Questionnaire has six statements that address the use of data
analysis. Those statements are: (a) I use data to monitor student programs, (b) I use
data to guide my instruction, (c) I use data to improve student outcomes, (d) I use
data to compare past and present performance, (e) I use data to compare students
within my class and (f) I use data to compare students across the school. Table #15
compares mean score responses to these statements from teachers and from teacher
leaders.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 15
Responses to Selected Teacher Questionnaire Statements
Questionnaire Statements
#4... to monitor student programs
#5... to guide my instruction
#6... to improve student outcomes
#12...to compare past and present..
#13...to compare students in m y...
#14...to compare students across...
Teacher Teacher Leadership
Responses (N-37) Responses (N-61
3.27 3.33
2.89 3.33
2.81 3.00
3.22 2.67
3.11 3.50
2.14 2.83
Note: scores represent mean score responses to statements (“2” = Somewhat
Disagree, “3” = Agree Somewhat and “4” = Agree Strongly)
The above data presents similar scores for statements 4, 5, 6 and 13. This
demonstrates a consistency in teacher perception regarding their use of assessment
data to monitor instructional programs, guide instruction, improve student
performance and compare student performance within each teacher’s classroom.
Leno High School receives a 3-point rating on the Implementation Analysis
Scale because of the weekly data analysis exhibited by the Math department. This
“3” rating is qualified by qualitative data that will be summarized in Table 7.
152
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Research question #4: To what degree were school student improvement goals
attained since the implementation of data-driven improvement strategies?
The 2001 Academic Performance Index (API) reported a 19-point growth for
Leno High School. This far exceeded the 5-poiiii growth required by the State. The
school became eligible for the Governor’s monetary award program. The Leno High
School principal’s focus for the 2000-2001 school year API results was improvement
in the performance of the school’s Hispanic student population. Only about 33% of
the 599 Hispanic students at Leno High School met grade-level student expectations
during the 2000-2001 school year. Department chairpersons and other teachers also
had the improvement of the Hispanic students as a priority, especially Hispanic
males. The fact that the passing rate of the Hispanic males on the California High
School Exit Exam is 34-points below other student groups dramatically underscores
the challenge Leno High School faces in bringing parity to academic achievement
between Hispanic and other students.
Over the past three years, the API score at Leno High School has increased
159-points. This is a part of the evidence that supports Leno High School’s focus on
improving the academic of its students. Although the achievement gap between
Hispanic and other students persists, the Spring 2001 State assessment scores
registered an 18-point growth in the category of Economically Disadvantaged
Students. This category probably includes a significant number of Hispanic students.
Unfortunately, the case study researcher did not present disaggregated data that
would clearly describe the performance level of Hispanic students. What was
153
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
disclosed was the 18-point increase in scores for economically disadvantaged
students exceeded the point growth of the Caucasian students. The increase for this
student population indicates that the achievement gap was closed.
Although the case study researcher did not describe any specific school
student performance goals, it is clear from the data presented that continuous student
achievement progress is an integral aspect of Leno High School. The fact that Leno
High has a solid high performing student base, and still continues to produce
statistically significant improvement in student achievement is an important finding.
“High-performing” schools also have the challenge of continuous improvement. As
can be seen from the data presented on Leno High School, even high-performing
schools have student populations that lag behind the average school scores.
Leno High School’s state test results exceeded the minimum State
improvement requirements. It therefore qualifies the school for a “3” rating. Table 7
summarizes the key data findings for Leno High School.
Table 16
Case Study #4 Data Analysis Summary (Leno High SchoolV-Trammg
Implementation
Scale Rating Qualitative Data Support
3 points + Principal, Assistant Principals and department chairs meet
(weekly) with teachers to instruct them in emerging student
assessments
154
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 16 (continued)
+ District conducts three staff developments prior to each
school year
+ Increase in availability of school technology prompted a
series of training on data analysis
+ Weekly department meetings to leam how to revise writing
rubrics and analyze student performance data
+ Twice a month training sessions also generated by
accreditation process
+ Assistant Principals assigned to lead that data training of
departments in which they have expertise
+ Teachers perceive data training to be important to their
ability to improve student achievement
- Inadequate funding for data-driven change training,
especially for teacher release time
- Personal management for implementing data
analysis is a concern of teachers
Note. “+” = positive characteristic and = negative characteristic
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Weekly department meetings are the comer stone to data training at Leno
High School. Assistant Principals are responsible for guiding departmental trainings.
The expression of concerns regarding the ability of teachers to implement data
analysis techniques is can be interpreted as both a sign of buy-in and the desire for
more training so that individual teacher concerns for implementing data analysis
procedures can be addressed.
Table 17
Case Study #4 Data Analysis Summary (Leno High School)— Data Collection
Implementation
Scale Rating Qualitative Data Support
3 points + District requires multiple measures that are aligned to the
(daily) State content standards
+ CORE assessment used throughout the year is forwarded to
district once a year in June
+ Data is required to be collected throughout the winter
months to determine appropriate instruction instructional
intervention for individual students
+ Semester grades in Language Arts and Mathematics, are
disaggregated and reviewed in the form of a School
Achievement Summary Report
156
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 17 (continued)
- Once a year is not frequent enough to inform instruction
during the year
- Inadequate technology at site level for teachers to readily
access student performance data
Note. “+” - positive characteristic and = negative characteristic
Table 17 describes characteristics of student performance data collection at
Leno High School. The district requires that multiple measures are used in all of its
schools for the purpose of monitoring student progress in learning. State content
standards, semester grades in Language Arts and Mathematics are disaggregated and
reviewed with the purpose of providing instructional intervention to targeted
students. This tracking system is vital to the ability of Leno High School to
differentiate instruction to students and student populations whose successful
academic progress is in jeopardy.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 18
Case Study #4 Data Analysis Summary (Leno High School)— Data Analysis
Implementation
Scale Rating Qualitative Data Support
3 points + Assessment results used to determine student placement and
the (weekly) need for intervention
+ District published a 5-step data analysis process to make
uniform data analysis throughout the district
+ Departments meet to review data and create action plans for
student improvement
+ District created special data analysis process for identifying
and supporting secondary students in need of intervention
+ Principal focused on the math department by formally
making that department responsible for creating appropriate
interventions to affect student academic improvement
+ Principal assigned administrative team to be personally
responsible for departments to employ effective interventions
+ Principal collects and reviews, with individual teachers,
student performance data every 5 weeks
158
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 18 (continued)
+ 9th and 10th graders are given an added level of data analysis
and intervention
+ Data analysis also focuses on daily collection and analysis
of homework
+ Each department has a frnai exam evaluation committee that
can make major modifications to Instructional sequencing
- Teachers would like more data analysis training
opportunities
Note. “+” = positive characteristic and = negative characteristic
Table 18 details several characteristics that describe aspects of data analysis
at Leno High School. The district’s 5-step analysis process standardizes the analysis
procedures. This is especially valuable in a departmentalized environment where
procedural uniformity can be difficult to achieve. The principal’s visible role in the
data analysis process sends a strong message, throughout the school, that the use of
data to improve instruction and student achievement is important. The principal’s
strategy to initially direct the majority of his attention on the Math department,
which would probably be most amenable to using data because of its relationship to
numbers, narrows the scope of focus and increases the probability of establishing a
high level of implementation.
159
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 19
Case Study #4 Data Analysis Summary (Leno High School)— Goals
Implementation
Seale Rating Qualitative Data Support
3 points + 2001 API score reflect a 19-point growth
(68% or more + School qualified for State monetary award because all
of students met statistically significant student populations met minimum
goals) State improvement goals on 2001 exams
+ API score Increased 159 points in the past 3 years
+ State test scores for economically disadvantaged students
increased 18 points, exceeding the increase in Caucasian
students by 1 point
- 67% of Hispanic students did not meet school grade level
expectations
- Hispanic male scores on the CAHSEE 34 points below that
of Caucasian student population
- Achievement gap between Hispanic and Caucasian students
continued with Spring of 2001 State test
Note. “+” = positive characteristic, negative characteristic
160
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 19 reflects data that describes the main characteristics of student
performance goal achievement at Leno High School. As with South High School,
%
Leno High School has two main goals for continuous improvement o f student
achievement. Meeting the minimum requirement in student achievement on State
assessments and closing the achievement gap between Hispanic and Caucasian
students are the two primary objects.
Leno High School met the minimum improvement goals for increased
achievement on State exams for all statistically significant student populations.
Increases in scores for economically disadvantaged students also seems to suggest
that Hispanic student achievement is improving and that the achievement gap is
being reduced. However, without disaggregated data it is difficult to validate or
quantify the degree to which the achievement gap has been reduced.
Comparative Review of Findings
This section will describe and then discuss the similarities and differences
across case studies. This discussion will first occur by research question area. Each
research question will be divided in to conceptual categories to help clarify the data.
Training, data collection and data analysis will be divided in to the three categories
of District Actions, School Site Actions and Teacher Perceptions. The final research
area of Attainment of Student Performance Goals will be divided in to the two
categories of State Goals and School Goals. Significant differences between case
study characteristics will be discussed in the context of a second tier of analysis
variables.
161
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Characteristics between the Case Studies in the Use of Data
Table 20 depicts characteristics of data-use training, at the district level, that
are similar across case studies.
Table 20
Data-Use Training Characteristics across Case-Studies— at District Level
Southern Elementary Divine Elementary South High Leno High
District requires teachers to
attend 6 trainings per year,
including a training prior to
the beginning of the school
year
District provides at
least 4 trainings to
teachers
District provides
training in
Summer and
throughout the
school year
District conducts 3
trainings prior to the
beginning of the
school year
One immediately sees that all four case study schools have districts that
provide data training throughout the school year. These trainings include, but are not
limited to, how to understand and use disaggregated data from State exam results to
improve instruction and provide students with appropriate and timely interventions.
The value of the training throughout the year is that it also provides important
guidance to teachers in terms of using the student performance data from the on
going assessments during the school year. The case studies also suggest that the
trainings are used to clearly communicate that student performance data, no matter
the source, is to be used to improve instruction and student achievement. Multiple
opportunities to communicate this message and provide practical steps to realize the
message in the classroom are a common feature in all four schools.
162
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Moving the focus to the school level, table 21 indicates that all of the school
sites provide training to their teachers, in addition to the district training.
Table 21
Data-Use Training Characteristics across Case-Studies— at School Site
Southern Elementary Divine Elementary South High Leno High
Vice Principal trains
teachers at school
Principal trains teachers
at school
Principal provides
training to English,
Math and Social
Studies departments
Assistant
Principals lead
trainings for
Math, English,
Science and
Social Study
departments
In two of the schools the training responsibilities seem to rest largely with the
principal, while in the other two schools the training responsibilities appear to be
with the assistant principals. What is interesting to note is that the site management
training responsibilities are not determined by elementary or high school
configurations. One might assume that the high schools would be more prone to
having assistant principals lead trainings because of the departmentalized structure,
and principals be responsible for training in elementary schools where the
instructional breakdown is by grade levels, and not departments. However, this data
shows an elementary school and a high school with a principal directing on-site
training and the other high school and elementary school having assistant principals
in charge of data training at the school.
General teacher perceptions of the district and school data trainings are also
consistent across all four case study schools. Table 22 makes clear that teachers
163
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
desire more training. In no school do teachers seem to say that the training they
received was of poor quality. What teachers seem to be saying is that the trainings
do not meet all of their needs with respect to implementing data strategies to improve
instruction and levels of student learning.
Table 22
Data-Use Training Characteristics across Case-Studies— Teacher Perceptions
Southern Elementary Divine Elementary South High Leno H igh
All teachers
interviewed not in
agreement as to the
regularity of training
Responses to
teacher
questionnaire
indicate a need for
more training
Teachers indicate a
need for funding to
support more training
time
Teachers express
concerns regarding
sufficient training to
implement the
required data
analysis procedures
Table 23 describes the data collection activities common to all four schools.
Table 23
Data Collection Characteristics across Case-Studies— at District Level
Southern Elementary Divine Elementary South High Leno High
• District collects • District collects ® District collects • District collects
and provides and provides and provides and provides
schools with schools with schools with schools with
results from State results from results from results from
tests. Reading and State tests. State tests. State tests.
Math test results Reading and Reading and Reading and
are also Math test Math test results Math test
disaggregated. results are also are also results are also
disaggregated. disaggregated.
® District requires
school to
develop its own
collection
methods
disaggregated.
164
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Each school’s district gathers data from the State assessments, disaggregates
it and provides it to the schools. The district’s emphasis on State assessments is not
surprising in light of the potential sanctions or awards that accompany the results.
Despite this emphasis, all four districts support and encourage assessments
throughout the school year. Each of the school’s districts recognizes the value of on
going assessments for impacting instruction and student learning. Table 24 describes
the on-going assessment activity at the schools.
Table 24
Data Collection Characteristics across Case-Studies— at School Site
Southern Elementary Divine Elementary South High Leno High
Teachers gather
assessment data during
the year in the form of
weekly flow charts
Teachers gather
assessment data
during the year in
the form of
criterion-referenced
tests in Math and
Language
Teachers collect
student
achievement data
on a daily, weekly
and end-of-unit
basis
Teachers gather
assessment data during
the year in the form of
criterion-referenced tests
in Math and Language
The regularity which teachers, at all four schools, collect student performance
data creates an environment in which data collection becomes routine and an
important activity at each school. Teacher concerns at Divine Elementary, South
High School and Leno High regarding their ability to access disaggregated data is an
outgrowth of the importance of data collection. This is especially the case with the
non-State assessments administered during the school year. Table 25 summarizes
teacher perceptions of data collection across the four schools.
165
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 25
Data Collection Characteristics across Case-Studies-Teaefaer Perceptions
Southern Elementary Divine Elementary South High Leno High
• Teachers show • Teachers show ® Teachers show • Teachers show
signs that they signs that they signs that they signs that they
understand the understand the understand the understand the
importance of importance of importance of importance of
data in data in data in data in
improving improving improving improving
student learning. student student student
Teachers state learning. learning. learning.
that data ® Teachers have • Teachers have • Teachers have
collection is an no on-demand no on-demand no on-demand
integral part of access to access to access to
the school disaggregated disaggregated disaggregated
culture. student data student data student data
The ability of teachers to access disaggregated data, especially the data they
collect during the year, is an important concern. It probably means that both the
district and the school need to allocate funds to purchase programs and hardware to
provide teachers with disaggregated performance data at any time during the year.
This would enhance the ability of teachers to modify instruction in a timely manner.
The districts of all four schools communicate clearly that all of the
assessment data is to be used for improving instruction and developing instructional
interventions for the purpose of improving student achievement. Table 26 describes
the level of each district’s participation in analyzing data.
166
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 26
Data Analysis Activities across Case-Studies— at District Level
Southern Elementary Divine Elementary South High Leno High
District expects • District expects • District • District expects
schools to use schools to use expects schools to use
assessment data assessment data schools to use assessment data
during the year to during the year assessment during the year
inform instruction. to inform data during the to inform
instruction. year to inform instruction.
• District instruction. • District provides
provides 6- • District 5-step data
variable data
analysis method
provides 4-
step data
analysis
method
analysis method
Southern Elementary School is in the only district, out of the four districts,
that does not utilize a district-wide mandated data analysis process. The Southern
Elementary case study does not describe the API performance level of the schools in
its district. If the district, in general, is a high performing district this could explain
why it leaves the manner of data analysis up to each school. With an API score of
872, certainly Southern Elementary can be considered to be a high performing school
and this level of current and historical performance lends itself to more site-
determined decisions. Generally speaking, districts with schools that are not
performing at relatively high levels are not given procedural latitude in today’s high
visibility, high accountability world.
167
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 27 describes characteristics of Data Analysis at school sites.
Table 27
Data Analysis Characteristics across Case-Studies— at School Site
Southern Elementary Divine Elementary South High Leno High
Results from State
exams and non-State
exams are analyzed at
school site to
determine instructional
and learning
ramifications
Results from State
exams and non-State
exams are analyzed
at school site to
determine
instructional and
learning
ramifications
• Results from State
exams and non-
State exams are
analyzed at school
site to determine
instructional and
learning
ramifications
• Data
disaggregated for
each student
across subject
areas
• Next year’s
student course
placement
determined by
assessment results
• Results from
State exams
and non-State
exams are
analyzed at
school site to
determine
instructional
and learning
ramifications
® Assessment
results
determine
placement and
intervention
needs
Data Analysis at the schools is one area in which the greatest distinction
between elementary and high schools is apparent. In addition to the regular analysis
of State, district and school assessments the two high school case studies report a
higher incidence of site administration participation in the data analysis process.
This public involvement in the analysis activities affords management the
opportunity to be directly involved in the data analysis process. This allows site
management to be better able to support teachers and students during the process of
data-driven improvement of student achievement.
168
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
During the course of responding to a myriad of management issues, it is easy
for administrators to have only superficial levels of participation with improvement
efforts. Personal, active and visible participation of management in the improvement
efforts creates a level of broad collaboration that is more potent than would be the
case with minimal management participation in the processes involved in data-driven
improvement.
Table 28 documents the prevailing teacher perceptions regarding the regular
analysis of student performance with respect to its ability to assist in the
improvement of instruction and levels of student learning.
Table 28
Data Analysis Characteristics across Case-Studies— Teacher Perceptions
Southern Elementary Divine Elementary South High Leno High
® Teachers are ® Teachers are • Teachers * Teachers are
expressing a expressing a express expressing a growing
growing belief growing belief positive belief that regular
that regular that regular results from analysis of student
analysis of analysis of participating performance data can
student student in help improve student
performance data performance data collaborative learning
can help improve can help improve data analysis
student learning student learning activities
® Teachers want ® Teachers want
more time to more time to
collaboratively collaboratively
analyze data analyze data.
Weekly analysis
time is often
used for other
topics
• Teachers
want more
training in
data analysis
• Teachers want more
training in data
analysis
16 9
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The growing belief in the power of regularly analyzing data is expressed at
the four case study schools. Both positive comments by teachers regarding data
analysis and requests for more time and training for collaborative data analysis
underscore the growing buy-in at each of the four schools.
Finally, Table 29 describes the two-year growth in API scores that every one
of the case study schools has made.
Table 29
Student Achievement Characteristics across Case-Studies— State and School Goals
Southern Elementary Divine Elementary South High Leno High
• Growth in State • All student sub ® Growth in State • Growth in State
API scores: 1999- groups and API scores: API scores:
2000, 808-851 = overall student 1999-2000 = 21 1999-2000 = 10
43 point increase; population points; 2000- points to 648;
2000-2001,851- exceeded their 2001=13 2000-2001 = 19
872 = 21 point API growth
targets for two
points points to 667
« School qualified consecutive ® School qualified
for monetary years • School did not for monetary
award from State • School qualified qualify for award from State
• School for monetary monetary award • School felt that it
improvement goal award from State from State did not
met by having • School goal met • School did not adequately
86% of its by meeting API meet its goal of reduce the
students score growth targets significantly achievement gap
above national reducing between
averages in Math achievement Hispanic and
and Reading gap between
Hispanic and
other students
other students
even though
State growth
targets for
Hispanic
students was met
After the Spring of 2001 State assessments, only South High School did not
qualify for monetary awards from the state. This was due to the Hispanic student
population not achieving the State’s minimum growth in its API score, as a student
170
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
subgroup. With respect to the achievement of Hispanic students, it is noteworthy
that out of the four schools in this cross-case study, only the high schools expressed
achievement concerns. One would expect that Divine Elementary School would also
express some concerns with over 66% of its student population Hispanic. Possibly
with only 523 students, its small size compensates for the commonly found
achievement lags of Hispanic students, or perhaps there are other causes, not
disclosed in the case study, for the teachers and administrator at Divine Elementary
not to express concerns about this student sub-group. In any event, teachers, site
administrators and central office administrators, in each of the case studies, all
attributed the continued improvement in API scores to the regular implementation of
data-driven improvement strategies and a corresponding focus on State standards.
The first tier of analysis portrayed in Tables 20 through 28 generally describe
how similar the four case-study schools are with respect to data-use training, data
collection and data analysis. It is noteworthy that all four schools show evidence of
exploiting these basic data-driven and change strategies. The next set of tables
regarding leadership, high performance organization qualities, the benefits of
allowing local sites to direct the implementation of change, methods of aligning
curriculum to state standards and knowing the purpose of data analysis may assist in
distinguishing the four case study schools from each other.
Table 30 describes district and site level administrators involved in the
raining of teachers at Southern Elementary, Divine Elementary and South High.
Table 30 also depicts District and school administrative personnel actively
171
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 30
Leadership Characteristics Across Case-Studies
Subject Southern
Elementary
Divine Elementary South High Leno High
Broad-based
—
• Conscious effort ® School • Teachers
leadership: to train teachers formal acknowledge
and build their leadership District
capacity to be team based Coordinator
instructional on curricula of Student
leaders departments Programs and
Evaluation,
Principal,
Assistant
Principal and
department of
chairs
actively
involved in
data analysis
172
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
participating in the analysis of student performance data at Southern Elementary and
Leno High. TMs table also shows formal and clear goals stated for the use of data-
driven strategies at Southern Elementary, Divine Elementary and South High.
Despite these similarities, it is evident that Divine Elementary and South High are
distinguished from Southern Elementary in that they specify the use of student
performance data for improving the performance of all students, including the lowest
achieving students.
The final topic in Table 30 inspects the development of broad-based
leadership. South High has a formal leadership team constructed around
instructional departments. Leno High’s recognition of the role of both site
administrators and teachers as a bona fide leadership team is not only similar to
South High but also comes the closest to minimizing leadership boundaries between
site administration and teachers. The fact that Divine Elementary consciously
trained teachers to build their capacity is also an example of consciously building a
broad base of instructional leadership.
Table 31 shows a common need at all four case study schools to provide
more resources for additional data related training. Table 31 also describes specific
examples of high expectations expressed by Southern Elementary and South High.
Although all case study schools strive to meet state achievement standards, South
Elementary expects its students to exceed state achievement standards. Even more
interesting is the fact that South High cultivates an environment in which there are
173
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 31
Characteristics of High Performing Organizations
Subject Southern Divine South High Leno High
Elementary Elementary
Resources: » Need more • Flaws in • More • More
trainings in district resources resources
data analysis program that needed to needed to
and how to disaggregates support support
use the student data teachers’ teachers’
results of • School ability to ability to
data analysis concerns access access
regarding student student
impact of the performance performance
end of state data and to data and to
intervention increase data increase data
program training. training
(II/USP) with
respect to
High data-related
Expectations:
« School
expects
student
training
• High
expectations
performance
—
for
goals to department
—
surpass chairs to
minimum participate
. .
state in data
achievement oriented
targets trainings
174
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
high expectations for department chairs to participate in standaxds-based professional
development and the training of their departments. While it can be argued that all of
the case study schools probably have high expectations for students, this is the only
school that consciously expresses high expectations for staff.
Table 32 describes the conscious efforts made in permitting local site control
of change efforts, attempts to align curriculum and instruction to state content
standards and the formal description of the underlying purpose of data analysis.
Table 32 indicates that all four schools enjoy some freedom to control aspects of
their implementation of data-driven improvement strategies. However, Divine
Elementary and Leno High consciously employ instruction to assist teachers in
aligning instruction to state standards. Finally, Table 32 shows that the case study
schools use a variety of methods to use periodic student performance data to
ultimately improve student achievement.
Discussion of Findings
Data from the four case studies paint a picture of four different schools that
employ similar basic data-improvement strategies. The variations in structure and
data-improvement strategies act as a signature that distinguishes the schools from
each other. This reinforces the notion that although change and improvement
strategies have the same basic framework, their success is related to how those
strategies are tailored to each entity.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 32
Local Control and Curriculum Alignment and Data Analysis Characteristics
Across Case Studies
Subject Southern
Elementary
Divine
Elementary
South High Leno High
Local site • District • District data • Departments
control: expects each use policy ® District determine
school to allow schools requires that reteaching
determine its to formulate schools needs based
goals and their own develop on student
how to use data-use their own performance
student strategies data patterns
performance collection
data strategies
(Top-down
support
provided
when
student
achievement
Curriculum
results are
alignment:
® Teachers not positive) ® School
"" taught to
align
formally
analyzes the
curriculum
—
degree of
and alignment
instruction with
Purpose o f
with state instruction
data standards and state
analysis:
• District
directive to
standards
use ® Differentiate • 4 years of • List
assessments instruction to comparative published of
to determine accommodate student students not
student student performance mastering
learning learning data used to grade-level
needs needs track student
achievement
and monitor
student
learning
needs
standards.
This list
triggers
instructional
interventions
176
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The first tier of analysis involved the primary variables of training, data
collection, data analysis and continuous improvement in student achievement,
Tables 2 through 29 paint a picture of schools with minimal variation in these four
primary areas of analysis. The most salient differences in this first tier analysis
between schools appear in Table 27. Table 27 describes general data analysis
characteristics of the four case study schools. The two high schools, in contrast to
the elementary schools, are identified as using student data to generate intervention
and placement across subject areas.
The second tier of analysis, involving high performance organization
qualities and more specific aspects of training and data collection., exhibit more
significant distinctions between case-study schools. Based on the sheer number of
data-driven actions, Divine Elementary and South High record actions in seven of
the eight variables. Divine Elementary is further distinguished from South High in
the area of leadership. Divine Elementary is officially provided with more data
analysis support than all of the other schools. The one area in which South High
could be considered in a better position, with regards to implementation of data-
driven strategies, is in the area of high expectations for its teaching staff.
General concerns arise from the cross-case study analysis. The need for
resources to support more data related trainings is universal. Teachers at Divine
High, South High and Leno High also indicate problems teachers have in
immediately accessing periodic student performance data. A reallocation of funding
resources to employ improved technology can easily resolve this concern.
177
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Although contrasts between schools can be a beneficial result of a cross-case
study analysis, similarities between schools can also be informative. As mentioned
in Chapter 2, these four schools were selected by their districts for their success in
using data to continuously improve student achievement on state exams. What is
common to all schools in this study is the use of periodic district and school
generated assessments to monitor student progress and the need for instructional
modifications. In addition to aligning instruction with state content standards, the
use of periodic assessments is the most obvious mechanism for the four schools for
promoting continuous success in improving student results on state exams.
Links to Research
This research study begins with the hypothesis that training, data collection
and data analysis are the fundamental building blocks of data-driven improvement.
The data disclosed in this cross-case study lend support to this hypothesis. With
respect to training, all four case studies show districts and schools applying a
considerable amount of resources towards establishing regular data-driven
improvement trainings at the district and school levels. Teachers and site
administrators are the prime recipients of these trainings because they are the
personnel held directly responsible for affecting continuous improvement in student
achievement.
Training inherently acknowledges the learning of new ways of thinking and
working (Nadler et ah, 1995). Using data to guide changes in instruction that better
meet student learning needs can still be considered a relatively new process in public
178
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
school systems. The research literature emphasizes the importance o f training as a
vehicle for promoting systemic change (Blazey et al., 2003) and as a means of
expanding the capabilities of an organization to learn from its previous actions
(Bossidy & Charan, 2002; O’Neil, 1992: Senge et ah, 2000). Ail four schools also
display evidence of making efforts to insure that ail appropriate personnel receive
data-driven improvement training (Schmoker, 2001). This is especially critical with
high schools because of their departmentalized structure.
All four case studies provide evidence of collecting student performance data
from a variety of assessment sources throughout each school year. Districts and
school sites in this study are actively disaggregating data from both state and district
mandated assessments. Frequent data collection is one of the seven correlates of the
Effective Schools program’s efforts to promote systemic improvement in student
achievement (Taylor, 2002). Nadler (1977) emphasizes the importance of data
collection prior to making decisions. With specific reference to schools, Fox (2002)
and Schmoker (2001) both describe how outcome data can assist in determining the
difference between what is taught and what is actually learned. All four case studies
present data that substantiates the ability of student data to be used by teachers to
decrease the discrepancy between what teachers are attempting to get students to
learn and what students are in fact learning.
Data is only useful when it is analyzed with the purpose of improving the
processes that lead to the continuous improvement of the end product (Brace, 2002).
In the case of education, the end product is improved student learning that is
179
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
minimally exhibited in improved achievement on standardized assessments. Ai! four
districts in this dissertation send very clear messages to their schools that student
data is to be used to improve instruction so that student learning is measurably
improved (Fox, 2001).
Deming’s Plan-Do-Study-Act analysis cycle (Aguayo, 1990) is transparent in
the data collection and analysis procedures of all four schools. What is also evident
from teacher interview and questionnaire responses is that when teachers experience
noticeable improvement in student learning that is coupled with their use of data to
improve instruction, there is increased buy-in to the strategy of using data to improve
student outcomes. This outcome is central to Nadler’s (1977) affirmation that data
feedback can provide employees with reasons to support the use of data to promote
performance improvement throughout an organization.
The four case studies strongly support the importance and interconnected nature of
continuous data-usage training, data collection and data analysis. The number of
similar activities in each of the four schools presented, and the degree of student
improvement that occurred at all of the schools suggests that data practices can be
successfully employed at small and large schools located in urban settings. This
study additionally suggests that data-driven strategies can cause continuous
improvement in student performance in high and low performing schools, if student
data is consistently used to address learning needs by means of variations to
instructional delivery. Finally, the data in this study strongly suggests that when site
and district personnel work together with a common set of tools to achieve the
180
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
commonly held goal of significant improvement in student achievement, continuous
improvement in the “high stakes” world of State assessments can be managed, and
does not have to be left up to chance.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 5
SUMMARY AND IMPLICATIONS
Introduction
The predominant population in many urban public schools consists of low-
income and/or minority students who historically perform poorly in the school
environment. These student populations represent the largest workforce population
growth in the United States (Marshall, 1992). This situation dramatizes the need for
system-wide improvement in student achievement.
As states increase student performance accountability by means of annual
state exams, the use of student performance data to meet minimum state student
achievement goals is rapidly becoming a major preoccupation of districts and
schools in public school systems. A growing body of research presents examples of
schools, with historically low performing students, that have used student data to
significantly improve student achievement (Schmoker, 2001). An important aspect
of these data-driven improvement efforts is the use of regular assessments,
throughout the school year, to monitor student achievement and modify instruction
to better meet student learning needs.
Restatement of Study’s Purpose
The purpose of this study is to inspect successful change efforts in a variety
of public school settings to document the strengths and weaknesses of those schools,
as they relate to continuous improvement in student performance. An analytical
182
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
framework, based on four variables derived from the literature review, is used to
describe the strengths and weaknesses of the data-driven improvement activities of
each of the schools under investigation in this dissertation. Four research questions
that are based on the four research-based variables guide this investigation. Those
questions are:
1. How frequently are data-driven improvement trainings provided?
What are the characteristics of those trainings?
2. How often is student performance data collected for analysis? What
are the characteristics of the data collection process?
3. How often does the school instructional staff meet in teams to
collaboratively analyze student performance data? What are the
characteristics of the analysis?
4. To what degree were school student improvement goals attained
since the implementation of data-driven improvement strategies?
Review of Analysis Procedures
The procedures for analyzing the data in the four case studies involved
reading chapters 4 and 5 of each case study, marking each page with post-its to
denote information concerning each of the four research questions. This information
was reported in a narrative format for each case study and each research question.
The narrative information was then summarized in table format, by case study and
by research question. After all of the case studies were analyzed, a cross-case study
evaluation was completed. A comparative quantitative table and several qualitative
183
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
data tables were created to support the relative implementation rankings of the four
case studies. Finally, several tables were created to reflect characteristics of high
levels of implementation represented by the case study schools in the areas of
Training, Data Collection and Analysis. .
Restatement of Research Design Limitations
Several difficulties inherent to the case studies and the case study format
limit the scope of this dissertation. One of the initial difficulties encountered in
analyzing quantitative data is that complete multi-year API scores are not available
in every case study. The same applies to the teacher responses to the two teacher
surveys. It is difficult to compare quantitative data across case studies or to verify
and quantify general statements made in the case studies.
Two additional limitations of this study involve the case-study format, itself.
External validity, or the ability to generalize findings, is a well-documented concern
in the literature on research design, and case studies in specific. The small number
of cases and the non-experimental design format make generalizing any findings
impossible to accomplish. Although there is discussion among some researchers
regarding the ability of case studies to provide internal validity (Creswell, 1994; Gall
et ah, 1996; Miles & Huberman, 1994), all researchers are in agreement that internal
validity or the ability to describe causal relationships is the strength of well-designed
experimental research. As a result of the internal and external validity issues
involved in case studies, the findings in this cross-case study are to be understood as
suggested relationships that can be further investigated by additional research.
184
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Summary of Findings
Teachers in each of the four case studies receive data-driven improvement
instruction from district and school site personnel. Despite these trainings, teachers at
all of the schools report that more training is needed. In three of the four schools,
concerns were expressed regarding adequate funding to support training needs.
With respect to the collection of student data, all of the case study districts
provide disaggregated test results from the annual State exams. All of the schools in
the cross-case study collect student performance data throughout the year, from a
variety of assessments that are aligned to the State content standards. Teachers at
two of the schools report that they have no direct access to the student performance
data collected during the school year.
The analysis of student performance data has two aspects, as defined in this
study. The results from student assessments are regularly analyzed and then
instructional modification are developed and implemented based on the analyzed
data. These modifications include reteaching, preteaching, additional practice and
the creation of interventions that may include after-school instruction or placement in
a class that will better address individual student learning needs. Teachers in all of
the schools under study analyze more than the results from State exams. All of the
schools also use assessments to evaluate the effectiveness of instruction and to
improve instruction. This analysis is done in collaborative grade-level or
185
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
departmental groups. Teachers at two of the schools report that, despite the current
training, they need additional training in data analysis.
The fourth variable of academic goal achievement is generally addressed by API
scores. All four schools report increases in API scores and all statistically significant
student populations achieved the State’s minimum performance growth on the State
assessments. Two of the case studies indicate that the Hispanic student population is
targeted for improvement in academic achievement. Both of those schools report
increases in the achievement of the Hispanic students. They also report, however,
that the achievement gap between Hispanic and Caucasian students continues. The
reduction of this achievement gap are school goals that extend beyond the State
minimum improvement targets. Teachers and administrators at all of the case study
schools express the opinion that their regular analysis of student performance data is
an important reason why State test scores increase each year.
Implications
Policymakers can impact the successful use of data to improve student
achievement. They must first understand that an accountability system that over
emphasizes ranking schools based on average scores, that are in turn based on once a
year high-stakes exams is faulty in that it focuses on quantitative outcomes, instead
of the continuous improvement of quantitative and qualitative aspects of the
improvement strategies that lead to the actual change in performance. As a result,
under the banner of increasing accountability, a great deal of education legislation
has, in Deming’s terms, tampered with the education system. The result of
186
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
tampering is that often the intended solution exacerbates instead of resolves the
problem. This is caused by policymakers not understanding the real causes of
continuous improvement in student achievement. Policymakers, just as district
management, should establish feedback mechanisms to continuously improve
legislation based on the needs of those who are expected to implement policy. This
feedback should come from both low and high-performing schools. This would
better address the capacity building needs that could vary from school to school. The
findings from this study support existing research that makes it clear that schools can
be successful in using student performance data to improve student achievement.
If policymakers focus on the importance of the use of student performance
data throughout the year, this will accomplish two important needs that are evident in
this cross-case study of four schools. Number one, the district and school emphasis
would be shifted from the once-a-year snapshot that relegates a school’s performance
to the realm of high-stakes averages. Number two, schools and their districts will be
provided with an additional impetus for utilizing student performance data on a
monthly, if not a weekly basis. The schools in the cross-case study show the
importance of on-going assessments to student achievement.
Research on data-driven change emphasizes the importance of improving the
processes that lead to the goal or product. The attention of teachers must be
redirected away from the fear of the high-stakes state exams to the practicality and
power of regularly scheduled collaborative analysis of student performance data for
the purpose of “informing” instruction.
187
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Researchers also have an important role to play in the shift of attention from
the uncertainty of high-stakes testing to the certainty of the regular use of data to
improve instruction and learning. Researchers can provide policymakers, districts
and schools with feedback paradigms that can be used to refine improvement efforts.
Furthermore, educational researchers should not limit themselves to the field of
education to develop, test and observe models of feedback that promote positive
change. The literature review in Chapter 2 clearly established business as an arena
that can offer additional feedback models that are effective in promoting continuous
improvement throughout an organization.
Although the unmarked topics on page two of Appendix A represent a
number of potential research lenses that can be applied to future research on data-
driven improvement strategies in the public school system, more studies based on the
conceptual framework model in Chapter 2 could positively add to the current body of
data-driven change research. One such study could include a more in-depth
investigation of the actual data analysis strategies employed by schools and a
corresponding study of how the data analysis actually changes instruction and
student learning. Another study could research the measurable impact of the
leadership qualities and the high performance characteristics on continuous
improvement in student achievement.
Other promising research perspectives include more links between the fields
of psychology and sociology. Education is a people business. Just as teachers
188
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
attempt to motivate students to leam the education system, policymakers on down,
must become more effective motivators of teachers to change and improve their
instruction. Research from the perspective of psychology and sociology could offer
important “people” insights for motivating change.
It is clear from this cross-case study that teachers and site administrators bear
the majority responsibility for getting the job done of increasing student
achievement. What is also evident from this study is that a review of key variables
can provide important information regarding the relative health of a school’s
strategies to improve student achievement, using student performance data. The
effective and continuous training of teachers and site administration, coupled with
the collection of student performance data and collaborative analysis of that data for
the purpose of improving instruction to meet student learning needs, appears to be
the cornerstone of successful data-driven improvement.
The initial responsibility to provide effective professional development
regarding the use of student performance data to improve student achievement rests
primarily with the school district, if improvement is to occur throughout the district.
However, one other variable that is under the auspices of a district is the regular
collection and availability of student performance data throughout the school year. If
this is an area of weakness for schools, then it requires the district to reallocate
funding to support rapid and regular access, by teachers, to student performance data.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
In addition to high quality continuous professional development, ea sy access to
current student data is a must if student achievement is to continuously improve
throughout our public schools.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
REFERENCES
Abrashoff, D. M. (2002). I t ’ s your ship: Management techniques from the best damn
ship in the navy. New York: Warner Books, Inc.
Aguayo, R. (1990). Dr. Deming: The American v^ho taught the Japanese about
quality. New York: Fireside
Anfora, Jr., V., Brown, K. M., & Mangione, T. L. (2002, October). Qualitative
analysis on stage: Making the research process more public. Educational
researcher, 31(7), 28-38.
ASQ Education Division. (2003). Successful application o f quality systems in K-12
schools. Milwaukee, WI: ASQ Quality Press.
Bascia, N., & Hargreaves, A. (Eds.). (2002). The sharp edge o f education change:
Teaching, leading, and the realities o f reform. New York: Routledge Faimer.
Beckhard, R., & Pritchard, W. (1992). Changing the essence: The art o f creating and
leading fundamental change in organizations. San Francisco: Jossey-Bass,
Inc.
Berliner, D. C. (2001, July). Educational reform in an era of disinformation.
Education Policy Analysis Archives 1(2), 1-35. Retrieved July 10, 2001,
from http://epaa.asu.edu/epaa/vln2.htm/
Blank, R. & Wilson, J. (2001). Understanding naep and timss results: Three types of
analysis useful to education. ERSSpectrum, Winter, 23-33.
Blazey, M., Davison, K., & Evans, J. (2003). Insights to performance excellence in
education 2003: An inside look at the 2003 baldridge award criteria for
education. Milwaukee, WI: ASQ Quality Press.
Bossidy, L., & Charan, R. (2002). Execution: The discipline o f getting things done.
New York: Crown Business.
Bowsher, J. (2001). Fix schools first: Blueprint for achieving learning standards.
Gaithersburg, MD: Aspen Publishers.
Bratton, W. (1998). Turnaround: Hovs America’ s top cop reversed the crime
epidemic. New York: Random House.
191
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Brooks-Dombrower, D. M. (2002). Data; Policies, strategies, and utilization in
public schools. Unpublished doctoral dissertation, University of Southern
California, Los Angeles.
Brae, G. (2002). Six sigma for managers. New York: McGraw-Hill.
California State Department of Education. (2003). California high school exit
examination: Demographic summary for all students tested- State report of
english-language arts (combined 2002). Retrieved April 6, 2003, from
http://cahsee.cde.ca. gov/2002/Ex itProff. asp?cYear=2001 -
Q2&T est=E&cAdxnin~C&D...
California State Department of Education. (2003). Star 2000-2002 score level
comparisons o f total scores fo r all students. Retrieved April 6, 2003, from
http :H www.edataonline.com/starComp20Q2/creport.idc7co-
o&dist=o&sch=o&grpl=4&gr...
Cantor, J. (2002). Design, implementation and adequacy o f data utilization in
schools: A case study. Unpublished doctoral dissertation. University of
Southern California, Los Angeles.
Carr, J., & Harris, D. (2001). Succeeding with standards: Linking curriculum,
assessment, and action planning. Alexandria, VA: Association for
Supervision and Curriculum Development.
Cawelti, G. (1997). Effects o f high school restructuring: Ten schools at work.
Arlington, VA: Educational Research Service.
Caelti, G., & Protehroe, N. (2001). High student achievement: How six school
districts changed into high-performance systems. Arlington, VA: Educational
Research Service.
Chatterji, M. (2002). Models and methods for examining standards-based reforms
and accountability initiatives: Have the tools of inquiry answered pressing
questions on improving schools? Review o f Educational Research, 74(3),
345-386.
Cummings, T., & Worley, C. (1999). Organization development & change. Mason,
OH: South-Western College Publishing.
Creswell, I. (1994). Research design: Qualitative and quantitative approaches.
Thousand Oaks, CA: Sage Publications.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
DataWorks. (2000). Curriculum calibration, lessons learned from underperforming
schools. The Dataworks Assessment Newsletter, 1(1).
DataWorks. (2000). Raise the api by increasing classroom productivity. The
Dataworks Assessment Newsletter, .2(2).
Davenport, T., & Prusak, L. (2002). Working knowledge: How organizations
manage what they know. Boston, MA: Harvard Business School Press.
Doming, E. (1994) The new economics for industry, government, education (2n d ed.).
Cambridge, MA: MIT Press.
Dombrower, D. (2002). Data: Policies, strategies, and utilization in public schools.
Unpublished doctoral dissertation. University of Southern California, Los
Angeles.
Drucker, P. (1990). Managing the non-profit organization: Principles and practices.
New York: Harper Collins Publishers, Inc.
Drucker, P. (1999). Management challenges o f the 21st century. New York: Harper
Collins Publishers, Inc.
Duncan, T. (2002). How effective schools use data to improve student achievement.
Unpublished doctoral dissertation. University of Southern California, Los
Angeles.
EdSource Online. (2001). Glossary. Retrieved July 29, 2001, from
http://www.edsource.org/do-cfm
Education manifesto. (1998). A nation still at risk. Washington, DC: Center for
Education Reform.
Epstein, J., Sanders, M., Simon, B., Salina, K., Jansom, R., & Voorhis, F. (2002).
School, family, and community: Your handbook for action (2n d Ed.).
Thousand Oaks: Corwin Press, Inc.
Fox, D. (2002, November). Classroom assessments data: Asking the right questions.
Leadership, pp. 22-23.
Fox, D. (2002). Using classroom data to build better readers: User-friendly tools for
teachers and administrators. Presented at Reading Excellence Act seminar,
September 17, 2002, sponsored by the Southern California Comprehensive
Assistance Center Division for School Improvement, Los Angeles County of
Education.
193
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Freedman, D. (2002). Corps business: The 30 management principles o f the its.
marines. New York: Harper Collins.
Fullan, M. (1993). Change forces: Probing the depths o f educational reform.
Levittown, PA: Falmer Press.
Fullan, M. (2001). Leading in a culture o f change. San Francisco, CA: Jossey-Bass.
Fullan, M., & Miles, M. (1992, June). Getting reform right: What works and what
doesn’t. Phi Delta Kappan, pp. 745-752.
Gall, D. G., Borg, W. R., & Gall, J. P. (1996). Educational research: An
introduction. (6th ed.) New York: Longman.
Gerstner, Jr., L. (2002). Who says elephants can’ t dance?: Inside IB M ’ s historic
turnaround. New York: Harper Business.
Grissmer, D., Flanagan, A., Kawata, J., & Williamson, S. (2002). Improving student
achievement: What state naep test scores tell us. Santa Monica, CA: Rand.
Guskey, T. (2000). Evaluating professional development. Thousand Oaks, CA:
Corwin Press, Inc.
Hatten, K., & Rosenthal, S. (2001). Reaching for the knowledge edge: How the
knowing corporation seeks, shares and uses knowledge for strategic
advantages. New York: AMACOM.
Herman, J. (2003, Spring). State test lessons: The effects on school reform. Voices in
urban education, 1, 46-55.
Housel, T., & Bell, A. (2001). Measuring and managing knowledge. New York:
McGraw/Irwin.
Institute for Learning. (2004). [Data file]. Available from Learning Research &
Development Center-University of Pittsburg web site, www.institutefor
leaming.org/
Isaac, S., & Michael, W. (1995). Handbook in research and evaluation. (3rd ed.). San
Diego, CA: Educational and Industrial Testing Services.
Joyce, B., & Showers, B. (2002). Student achievement through sta ff development.
(3r Q ed.). Alexandria, VA: Association for Supervision and Curriculum
Development.
194
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Kendall. I,. & Marzano, R. (1995). The systemic identification and articulation o f
content standards and benchmarks: Update. Aurora, CO: MCREL.
Kendall, J., & Marzano, R. (1997). Content knowledge: A compendium o f standards
and benchmarks fo r K-12 education (2n d ed.). Aurora, CO: MCREL.
Kohn, A. (1993, September). Turning learning into a business: Concerns about total
quality. Educational leadership, pp.58-61.
Los Angeles Unified School District. (2003). 2002 api base report. [Memorandum
No. BT-99]. Division of Planning, Assessment and Research.
Marshall, R., & Tucker, M. (1992). Thinking fo r a living: Education and the wealth
o f nations. New York: Basic Books.
Martin, M., Mullis, 1, Gregory, K., Hoyle, C., & Shen, C.(2000). Effective schools in
science and mathematics, iea ‘ s third international mathematics and science
study. Chestnut Hill, MA: International Study Center, Lynch School of
Education, Boston College.
Marzano, R., & Kendall, J. (1998). Implementing standards-based education.
U.S.A.: NEA
Mattingly, D., Prislin, R., McKenzie, T., Rodriguez, J., & Kayzar, B. (2002).
Evaluating evaluations. The case of parent involvement programs. Review of
educational research, 72(4), 549-476.
McDonnell, L., & Elmore, R. (1991). Getting the job done: Alternative policy
instruments (A.R. Odden, Ed.). Education policy implementation. New York:
State University of New York Press.
McDougall, D., Saunders, W., & Goldenberg, C. (2002). Inside the black box and
other change elements at getting results schools. Manuscript submitted for
publication.
Mid-Continental Regional Educational Laboratory. (1994). What is noteworthy on
school reform. Aurora, CO: MCREL.
Miles, M., & Huberman, A. (1994). An expanded sourcebook: Qualitative data
analysis (2n d ed.). Thousand Oaks, CA: Sage Publications.
Miller, K. D. (2002). An analysis o f the use o f data to increase student achievement
in public schools. Unpublished doctoral dissertation. University of Southern
California, Los Angeles.
195
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Mintzberg, H., Ahlstrand, B., & Lamel, J. (1998). Strategy safari: A guided tour
through the wilds o f strategic management. New York: The Free Press.
Nadler, D., Shaw, R., Walton, A. & Associates. (1995). Discontinuous change:
Leading organizational transformation. San Francisco, CA: Jossey-Bass, Inc.
Newman, F., Smith, B., Allensworth, E., & Bryk, A. (2001). School instructional
program coherence: Benefits and challenges. Chicago, IL: Consortium on
Chicago School Research.
O’Neil, J. (1995, April). On schools as learning organizations: A conversation with
Peter Senge. Educational Leadership, pp. 20-23.
Patton, M. (1990). Qualitative evaluation and research methods. (2n d ed.). Newbury
Park, CA: Sage Publications.
Quindlen, T. (2002). Reaching minority students: Strategies for closing the
achievement gap. Education update, 44(5), 1, 6-8.
Reeves, D. (2002). Accountability in action: a blueprint fo r learning organizations.
Denver, CO: Advanced Learning Press.
Rothman, R. (2002). Closing the achievement gap: How schools are making it
happen. Challenge journal, 5(2), 1-11.
Sarason, S. (1990). The predictable failure o f educational reform: Can we change
course before i t ’ s too late? San Francisco, CA: Jossey-Bass, Inc.
Schaffer, E., Nesselrodt, P., & Stringfield, S. (1997). Impediments to reform: An
analysis o f destabilizing issues in ten promising programs. Arlington, VA:
Educational Research Service.
Schermerhom, J., Hunt, I., & Osbome, R. (2002). Organizational behavior (7th ed.).
New York: John Wiley & Sons.
Schmidt, W., Houang, R., & Cogan, L. (2002, Summer). A coherent curriculum: the
case of mathematics. American Educator, pp. 10-26, 47.
Schmoker, M. (1999). Results: The key to continuous school improvement (2nd. Ed.).
Alexandria, VA: ASCD.
Schmoker, M. (2001). The results fieldbook: Practical strategies from dramatically
improved schools. Alexandria, VA: ASCD.
196
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Schmoker, M., & Marzano, R. (1999, March). Realizing the promise of standards-
faased education. Educational Leadership, pp. 17-21.
Senge, P. (1990). The fifth discipline: The art and practice o f the learning
organization. New York: Currency Doubleday.
Senge, P., Cambion-McCabe, N., Lucas, T., Smith, B., Dutton. J., & Kleiner, A.
(2002). Schools that learn: A fifth discipline fieldbo ok for education, parents,
and everyone who cares about education. New York: Doubleday.
Sergiovarmi, T. (1992, February). Why we should seek substitutes for leadership.
Educational leadership, pp. 41-45.
Sevillano, P. J. (2002). An analysis in the use o f student performance data in
schools. Unpublished doctoral dissertation. University of Southern
California, Los Angeles.
Siskin, L. (2003, Spring). Colleagues and “yutzes: Accountability inside schools.
Voices in Urban Education, pp. 24-31.
Snyder, K., Acker-Hocevar, M., & Snyder, K. (2000). Living on the edge o f chaos:
Leading schools into the global age. Milwaukee, WI: ASQ Quality Press.
Taylor, B. (2002, January). The effective school process: Alive and well. Phi Delta
Kappan, 53(5), pp. 375-378.
Thompson, R. (2002). Design implementation and adequacy o f using student
performance data and the design’ s link to state context for assessing student
performance: A case study. Unpublished doctoral dissertation. University of
Southern California, Los Angeles.
Tissen, R., Andriessen, D., & Deprez, F. (2000). The knowledge dividend: Creating
high performance companies through value-based knowledge management.
London: Pearson Education Limited.
Tucker, M., & Codding, J. (1998). Standards for our schools: How to set them,
measure them, and reach them. San Francisco, CA: Jossey-Bass.
U.S. Department of Education. (2001). No child left behind: Executive summaiy.
Jessup, MD: ED Pubs, U.S. Department of Education.
Warwick, R. (1995). Beyond piecemeal improvements: How to transform your
school using deming’ s quality principles. Bloomington, IN: National
Educational Service.
197
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDICES
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
A ppendix A
Major Change Themes in Research
THEMES RESEARCHERS (Bold Print = Business)
1. Clear Mission/Goals Effective Schools, Fullan, Sergiovanni, Cawelti, Baldridge,
Bossidy
2. Safe Environment: Effective Schools, Cawelti, Baldridge
3. High Expectations: Effective Schools
4. Student Opportunity to Leam: Effective Schools, WestEd, Quindlen, Rothman,
Data Works, Bladridge
5. Broad Instructional Leadership: Efective Schools, Cawelti, Newman,
McDougall, Baldridge
6. Frequent Monitoring of Student Progress: Effective Schools, Cawelti, Rothman,
Schmoker, Baldridge, Fox
7. Positive Home-School Relations: Effective Schools, Schaffer, Quindlen, Cawelti,
Baldridge
8. Capacitiy for Organization to Leam: Fullan, Schaffer, Aguayo, Baldridge,
Nadler, Senge, Beckhard, Denning, Six Sima
9. Change is Unpredictable and Requires Flexibility: Fullan
10. Obstacles to be seen as Opportunities: Fullan
11. Resources Must Support Change: Fullan, Schaffer, Baldridge, Deming, Six
Sigma
12. Ability to Manage Change: Fullan, Baldridge, Deming, Cummings & Worley
13. Change Must be Systematic: Fullan, Baldridge, Nadler, Senge, Beckhard,
Demming, Six Sigma
14. Local Change Implementation Yields Large-scale Chamge: Fullan, Sergiovanni,
Cawelti, Baldridge, Deming
15. Commitment to Change: Schaffer, Baldridge, Bossidy, Deming, Six Sigma
16. Appropriate Staffing: Schaffer, WestEd, Rothman, Bossidy
17. Curriculum Alignment: Schaffer, Data Works, Fullan, Rothman, Newman
18. External Forces Can Impact Change: Schaffer
19. Organization-wide Communication: Schaffer, Baldridge, Bossidy, Deming, Six
Sigma, Senge
20. Moral Purpose: Fullan
21. Building Relationships: Fullan, Baldridge, Deming, Senge, Cummings &
Worley
22. Collaboration: Fullan, Fox, Data Works, Sergiovanni, Qunidlen, Newman,
Baldridge, Senge, Nadler
23. Commitment to High Quality Practice: Sergiovanni
199
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
24. Caring: Sergiovanni
25. Manage Tension Between Individual and Group. Perspectives: Fullan, Deming
26. Nurture Connections with External Support Entities: Fullan, McDougal,
Baldridge, Deming
27. Alter School Organization/Structure: WestEd, Cawelti
28. Use Technology to Support Change: Cawelti, Gerstner, Bratton
29. Accountability Aligned to Change: Cawelti, Baldridge
30. Professional Development: WestEd, Quindlen, Rothman, Baldridge, Joyce &
Showers, Beckhard, Bossidy, Deming, Six Sigma
31. High Expectations (teacher & student): Cawelti, Quindlen
32. Shared Belief Systems: Cawelti, Baldridge, Bossidy, Deming, Senge, Six Sigma
33. Timely Instruction Intervention: Cawelti, Fox
34. Formal Test Preparation: Cawelti, Fox
35. Emphasis on Continuous Improvement: Cawelti, Baldridge, Deming, Senge
36. Focus on Areas over which School has Control: McDougall, Fox
37. Change Organizational Culture: Baldridge, Nadler
38. Management of Use of Organization’s Knowledge: Beckhar, Senge
39. Conversion of Negative Energy of Resistance to Change: Fullan, Beckhard
40. Promote Innovation: Fox, Schmoker, Baldridge, Senge
41. Focus on Continuous Improvement: Baldridge, Schmoker, Fox, Deming, Six
Sigma
42. Top Leadership’s Personal Commitment to Executing Change: Baldridge,
Bossidy
43. Know Your People: Bossidy, Deming
44. Know Your Self: Bossidy, Senge
45. Brign about Predictable Control to Product Variation: Deming, Six Sigma
46. Take a Systems Analysis Perspective: Baldridge, Deming, Six Sigma, Senge
47. Use Data to Support Change: Schmoker, Fox, Data Works, Deming, Six Sigma,
Nadler
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
47 Change Themes Divided into 6 Key Topics
(bold print indicates that topics/variables used in this cross-case study)
Leadership
Clear goals (1)
Broad instructional leadership (5)
Ability to manage change (12)
Commitment to change (15)
Manage tension (25)
Management and use of organization’s knowledge (38)
Promote innovation (40)
Top leadership committed to change (42)
Know your people (43)
Know yourself (44)
Hfph Performing Organization Qualities
Safe environment (2)
Students have opportunity to leam (4)
Positive home-school relations (7)
Capacity of organization to leam (8)
Resources must support change (11)
High expectations (13)
Appropriate staffing (16)
Organization-wide communication (19)
Moral purpose (20)
Building Organizational relationships (21)
High Performing Organization Qualities
Commitment to high quality practice (23)
Caring (24)
Alteration of organization structure (27)
Nurture connections with external support entities (26)
Emphasis on continuous improvement (35)
Focus on areas over which schools have control (36)
Change culture (37)
Training
Curriculum alignment (17)
On-going professional development (30)
Formal test preparation (34)
201
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data Collection and Analysis
Frequent monitoring of student progress (6)
Collaboration (22)
Use of technology to support change (28)
Accountability aligned to change (29)
Timely instmction intervention (33)
.Bring about predictable control of product variation (45)
Use systems analysis (46)
Use data to support change.(47)
Knowledge of Change Process
Change requires flexibility (9)
Obstacles to be understood as opportunities (10)
Change must be systemic (13)
Local Change implementation yields large-scale change (14)
External forces impact change (18)
Conversion of negative energy of resistance to change (39)
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Appendix B
Use of Data in School Study
Stages of Concern (Teachers)
Name (optional)_____________________ ' ______ ____
In order to identify these data, please give us the last four digits of your Social
Security number:
This is a questionnaire about the district’s design to use student data to improve
student performance. The purpose of this questionnaire is to determine what
teachers who are using or thinking about using the district’s design to use data to
improve student learning are concerned about at various times during the innovation
adoption process. A good part of the items on this questionnaire may appear to be of
little relevance or irrelevant to you at this time. For the completely irrelevant items,
please circle “0” on the scale. Other items will represent those concerns you do
have, in varying degrees of intensity, and should be marked higher on the scale.
Please respond to the items in terms of your present concerns, or how you feel about
your involvement or potential involvement with the district’s design to use data to
improve student learning. We do not hold to any one definition of this innovation, so
please think of it in terms of your perception of what it involves. Remember to
respond to each item in terms of your present concerns about your involvement or
potential involvement with the district’s design to use data to improve student
learning.
Thank you for taking time to complete this questionnaire.
Please circle the number that best reflects your response to each statement based on
the following rating scale:
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
0 1
Irrelevant Not true for me
now
2 3 4 5
Some what true for me
6 7
Very true for me
1. I am concerned about student attitudes toward the district’s design to use
student data to improve student performance. 01234567
2. I now know of some other approaches that might work better than the district’s
design to use student data to improve student performance. 0 1 2 3 4 5 6 7
3. I don’t even know what the district’s design to use student data to Improve
student performance is. 0 1 2 3 4 5 6 7
4. I am concerned about not having enough time to organize myself each day
because of the district’s design to use student data to improve student
performance. 01234567
5. I would like to help other faculty I their use of the district’s design to use
student data to improve student performance. 01234567
6. I have a very limited knowledge about the district’s design to use student data
to improve student performance. 01234567
7. I would like to know how the implementation of the district’s design to use
student data to improve student performance would affect my classroom, my
position at my school and my future professional status. 01234567
8. I am concerned about conflict between my interests and responsibilities with
respect to implementation of the district’s design to use student data to
improve student performance. 01234567
9. I am concerned about revising my use of the district’s design to use student
data to improve student performance. 01234567
10.1 would like to develop working relationships with both our faculty and outside
faculty while implementing the district’s design to use student data to improve
student performance. 01234567
11.1 am concerned about how the district’s design to use student data to improve
student performance affects students. 0 1 2 3 4 5 6 7
12.1 am not concerned about the district’s design to use student data to improve
student performance. 01234567
13.1 would like to know who will make the decisions in the district’s design to use
student data to improve student performance. 0 1 2 3 4 5 6 7
14.1 would like to discuss the possibility of using the district’s design to use
student data to improve student performance. 0 1 2 3 4 5 6 7
15.1 would like to know what resources are available to assist us in implementing
the district’s design to use student data to improve student performance. 0 1
2 3 4 5 6 7
16.1 am concerned about my ability to manage all that is required by the district’s
design to use student data to improve student performance. 0 1 2 3 4 5 6 7
204
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
17.1 would like to know how my teaching or administration is supposed to change
with the implementation of the district’s design to use student data to improve
student performance. 01234567
18.1 would like to familiarize other departments or people with the progress of this
new approach to use district’s design to use student data to improve student
performance. 01234567
19.1 am concerned about evaluating my impact on students in relation to the
district’s design to use student data to improve student performance. 0 12 3
4 5 6 7
20.1 would like to revise the district’s design to use student data to improve
student performance. 01234567
2 1 .1 am completely occupied with other things besides the district’s design to use
student data to improve student performance. 0 1 2 3 4 5 6 7
2 2 .1 would like to modify our use of the district’s design to use student data to
improve student performance. 01234567
23. Although I don’t know about the district’s design to use student data to
Improve student performance, I am concerned about aspects of the district’s
design. 01234567
24 .1 would like to excite my students about their part in the district’s design to use
student data to improve student performance. 0 1 2 3 4 5 6 7
25.1 am concerned about time spent working with nonacademic problems related to
the district’s design to use student data to improve student performance. 0 1
234567
26 .1 would like to know what is the use of the district’s design to use student data
to improve student performance. 01234567
27 .1 would like to coordinate my effort with others to maximize the effects of the
district’s design to use student data to improve student performance. 0 12 3
4567
28.1 would like to have more information on time and energy commitments required
by the district’s design to use student data to improve student performance.
01234567
2 9 .1 would like to know what other faculty are doing in the area of implementing the
district’s design to use student data to improve student performance. 0 12 3
4567
30. At this time, I am not interested in learning about the district’s design to use
student data to improve student performance. 01234567
31.1 would like to determine how to supplement, enhance or replace the district’s
design to use student data to improve student performance. 01234567
32.1 would like to use feedback from students to change the district’s design to use
student data to improve student performance. 01234567
33.1 would like to know how my role will change when I am using the district’s
design to use student data to improve student performance. 0 1 2 3 4 5 6 7
205
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
34. Coordination of tasks and people in relation to the district’s design to use
student data to improve student performance is taking too much of my time. 0
12 3 4 5 6 7
35 .1 would like to know how the district’s design to use student data to improve
student performance is better than what we have now. 01234567
36 .1 am concerned about how the district’s design to use student data to improve
student performance affects students. 01234567
THANK YOU!
Please place this questionnaire, along with the Teacher Questionnaire, in the attached
envelope. To insure confidentiality while ascertaining that everyone has completed
the surveys, please seal the envelope, write your name on the outside, and return it to
the principal’s office.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

How effective schools use data to improve student achievement

PDF

How districts and schools utilize data to improve the delivery of instruction and student performance: A case study

PDF

"Self" in self -worth protection: The relationship of possible selves to achievement motives and self -worth protective strategies

PDF

A case study: An analysis of the adequacy of one school district's model of data use to raise student achievement

PDF

Districtwide instructional improvement: A case study of Algiers Elementary School in the Orchard Unified School District

PDF

Implementation of educational policy: Technology implementation in a California middle school

PDF

Connecting districts and schools to improve teaching and learning: A case study of district efforts in the Los Coyotes High School District

PDF

Connecting districts and schools to improve teaching and learning: A case study of district efforts in the Media Unified School District

PDF

Connecting districts and schools to improve teaching and learning: A case study of district efforts in Los Coyotes High School District

PDF

Districtwide instructional improvement: A case study of a high school in the Los Coyotes High School District

PDF

An empirical examination of customer perceptions of service quality utilizing the extended service quality model under the condition of multiple subunit service providers

PDF

Districtwide instructional improvement: A case study of an elementary school in the Beach Promenade Unified School District

PDF

A study of the relationship between student achievement and mathematics program congruence in select secondary schools of the Archdiocese of Los Angeles

PDF

Design, implementation and adequacy of data utilization in schools: A case study

PDF

Curriculum policy and educational practices: A study of primary classroom music education in Kern County, California

PDF

Coevolution of technology and organization: A case study of BrokerHouse

PDF

Inaccuracies in expert self -report: Errors in the description of strategies for designing psychology experiments

PDF

Does autonomy matter? Investigating research -based practices in charter and other public schools

PDF

College students' attributions about non -condom use

PDF

An analysis of the use of data to increase student achievement in public schools

Asset Metadata

Creator Lewis, Alan Stephen (author)

Core Title Data -driven strategies to improve student achievement: A cross-case study of four California schools

School Graduate School

Degree Doctor of Philosophy

Degree Program Education

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag education, administration,OAI-PMH Harvest

Language English

Contributor Digitized by ProQuest (provenance)

Advisor Marsh, David D. (committee chair), Cummings, Thomas (committee member), Dembo, Myron (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c16-343025

Unique identifier UC11335843

Identifier 3180366.pdf (filename),usctheses-c16-343025 (legacy record id)

Legacy Identifier 3180366.pdf

Dmrecord 343025

Document Type Dissertation

Rights Lewis, Alan Stephen

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA