University of Southern California Dissertations and Theses

A case study: An analysis of the adequacy of one school district's model of data use to raise student achievement

(USC Thesis Other)

A case study: An analysis of the adequacy of one school district's model of data use to raise student achievement

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content A CASE STUDY: AN ANALYSIS OF THE ADEQUACY OF ONE SCHOOL
DISTRICT’S MODEL OF DATA USE TO RAISE STUDENT ACHIEVEMENT
by
Ruben Zepeda II
A Dissertation Presented to the
FACULTY OF THE ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment o f the
Requirements for the degree
DOCTOR OF EDUCATION
August, 2005
Copyright 2005 Ruben Zepeda II
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UMI Number: 3196923
Copyright 2005 by
Zepeda, Ruben, II
All rights reserved.
INFORMATION TO USERS
The quality of this reproduction is dependent upon the quality of the copy
submitted. Broken or indistinct print, colored or poor quality illustrations and
photographs, print bleed-through, substandard margins, and improper
alignment can adversely affect reproduction.
In the unlikely event that the author did not send a complete manuscript
and there are missing pages, these will be noted. Also, if unauthorized
copyright material had to be removed, a note will indicate the deletion.
®
UMI
UMI Microform 3196923
Copyright 2006 by ProQuest Information and Learning Company.
All rights reserved. This microform edition is protected against
unauthorized copying under Title 17, United States Code.
ProQuest Information and Learning Company
300 North Zeeb Road
P.O. Box 1346
Ann Arbor, Ml 48106-1346
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
DEDICATION
This work is dedicated to my wife Sarah. You are the love of my life! The
realities of my successes large and small begin and end with the love, support and
encouragement I receive from you. My life and my world make sense to me because
o f you. I love you, always and forever!
I also dedicate this work to Ruben and Martha Zepeda, m y father and my
mother. You will forever be my greatest teachers and role models. Without your
love, guidance and support I would not have become the man that I am.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ACKNOWLEDGMENTS
I entered USC to further my education in education policy and administration
in an effort to improve the educational experiences o f youth who are confronted by
all o f the challenges of living and learning in an inner-city urban environment. The
faculty of the USC Rossier School o f Education exceeded m y expectations and
provided me with a richer, deeper and more personalized experience than I ever
imagined. I am deeply indebted to Dr. David D. Marsh, my mentor, my dissertation
committee chair and my friend, for his wisdom, his leadership and his ability to walk
through the dissertation process. My sincere appreciation goes to Dr. Stuart E.
Gothold for inviting me to study at USC, for allowing me to pursue my own path at
USC and for sitting on my dissertation committee. I want to express a special thanks
to Dr. Lawrence O. Picus for helping me expand my thoughts about access and
equity, for sitting on my dissertation committee and for always asking about my
father. Finally, I am especially grateful to Dr. Robert Rueda for maintaining the
highest o f standards in research and scholarship, for encouraging me to pursue
excellence, for focusing my efforts towards inner-city Latino and African-American
youth and for always finding the right words to ease my academic anxiety.
I also want to thank a few leaders and colleagues in Los Angeles Unified
School District who have supported me throughout this journey. First and foremost I
need to thank Dr. Toni Marsnik for encouraging me to pursue a doctorate at USC.
Toni, your leadership and legacy will continue to make a difference wherever I go. I
also want to thank J.L. “Bud” Jacobs, you have provided me with the best advice, a
iii
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
multitude of professional experiences and a friendship I am honored to possess. My
special thanks goes to my many other colleagues in LAUSD who showed their
concern for me daily, by reminding me, prodding me and urging me to finish.
This dissertation has special significance for my dearest friends and
colleagues from the Southern California Council Social Science Association and the
California Council for the Social Studies. They have collectively shared my
journeys, my challenges, my frustrations and now I share with each and every of
them this accomplishment and all that it represents! Your private words of
encouragement, your relentless commitment to seeing me through to the end
exemplify the notion o f “community.” I can only hope you understand the depth o f
respect and love I have for you.
I have reserved my closing words o f acknowledgment for my brother and his
family, my sister and her family and to the entire Magana family. Gilbert and Maria
you are doing a phenomenal job raising families that make me proud to be your
brother and your children’s uncle. However distant we may find ourselves we will
always share an unbreakable bond as family. Finally, I have to share some thoughts
that I have rarely expressed but I hope are clearly evident. I have been blessed to be a
member o f Matilde and Angelina Magana’s extended family. You have treated me
more like a son and brother rather than a son-in-law and brother-in-law. You have
always welcomed me to your homes, to every family event and you have forgiven
me when I have had to miss. Your presence and participation in my university
celebrations means the world to me!
iv
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
TABLE OF CONTENTS
DEDICATION ii
ACKNOWLEDGMENTS iii
LIST OF TABLES viii
LIST OF FIGURES ix
ABSTRACT x
CHAPTER 1 - OVERVIEW OF THE STUDY
Introduction 1
International Perspective on the Status o f Student Performance 3
National Perspective on the Status o f Student Performance 4
State Perspective on the Status of Student Performance 6
Education Reform Efforts 6
Role o f Student Performance Data 9
California Context 11
Current Status o f Data Utilization 13
Statement o f the Problem 14
Purpose o f the Study 15
Importance o f the Study 16
Limitations 16
Delimitations 17
Definitions 17
Organization o f the Dissertation 22
CHAPTER 2 - LITERATURE REVIEW 24
Current Status on Student Performance 25
Comprehensive School Reform 40
Role o f Data in Comprehensive School Reform 58
California State Expectations for Students 73
Research o f Data in Schools 83
CHAPTER 3 - RESEARCH METHODOLOGY 87
Research Questions 88
Research Design 88
Sample and Population 89
Setting 90
Instrumentation 96
Data Collection 102
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data Analysis 113
Summary 114
CHAPTER 4 - RESEARCH FINDINGS 115
Research Question One 116
A Results Driven Orientation
Student Performance Assessed in the Context o f Current and
Emerging Assessments 117
Emerging Context 124
Overview of the Elements of District Design o f Data Use
To Raise Student Achievement 126
District Decisions and Rulings that Support Use o f Design 132
Intended Results o f Design Plan 134
Data Use Policy and Strategy Funding 137
Summary of Research Findings for Question One 140
Research Question Two 141
Degree o f Design Implementation Current Data Practices 141
Implementation of Emerging State Data Practices 161
Accountability for Data Use at District, School and
Individual Levels 164
Improving Student Achievement Through Implementation
of Data Use 168
Barriers to Data Utilization 172
Summary o f Research Findings for Question Two 174
Research Question Three 178
District Support o f Standards-based Instruction and
Assessment 178
District and School Accountability to Standards Based
Instruction and Assessments 181
Determination o f “High” Student Performance 185
Summary o f Research Findings for Question Three 187
Discussion of Research Findings 188
CHAPTER 5 - SUMMARY, DISCUSSION OF THE FINDINGS, 199
CONCLUSION AND RECOMMENDATIONS
Statement of the Problem 199
Purpose of the Study 200
Methodology 201
Summary of the Findings 202
Conclusions Based on Research 208
Implications o f Research Findings 210
Recommendations 213
Suggestions for Further Research 215
vi
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
REFERENCES
APPENDIX
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
LIST OF TABLES
PAGE
Table 1 United States Averaged Scaled Scores on TIMSS
1995-2003 30
Table 2 New American Schools Design Teams and Founders 51
Table 3 STAR Testing Program 77
Table 4 Students by Ethnicity 2003-04 92
Table 5 List o f Participants Interviewed for Study 112
Table 6 Yearly Cycle o f Data Collection 131
Table 7 VUSD Student Achievement Goals and Objectives 136
Table 8 Disaggregated Subgroup API 146
Table 9 Rank o f Teacher Identified and Desired
Staff Development 15 6
Table 10 District Support of Standards-based Instruction
and Assessment 179
Table 11 District an School Accountability to Standards-
based Curriculum 181
Table 12 Determination of “High” Student Performances 185
viii
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
LIST OF FIGURES
Figure 1
Figure 2
Figure 3
The Richness and Complexity o f Student
Assessment Data
2003-04 API Cycle Elementary and Middle Schools
Teacher Ratings of Attitudes Toward Educational
Practices
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ABSTRACT
The purpose o f this study was to evaluate how one high school used data to
raise student achievement. It describes the model developed at the school district
level and compares how the school identifies, collects and evaluates data. It focuses
on the emerging state o f school reform and accountability in California and measures
the extent to which the district design addresses state goals and expectations for
student achievement. This study describes the extent to which the district design was
implemented and to what extent the model used data to raise student achievement.
The study focused on three research questions to design the data collection
instruments: 1) What is the district design for using data regarding student
performance and how is the design linked to current and emerging state context for
assessing student performance? 2) To what extent has the district design actually
been implemented at the district, school and teacher level? 3) To what extent is the
district design a good one?
This study utilized a moderately sized urban school district in Southern
California. The case study was conducted in a medium sized high school consisting
of a diverse student population. The superintendent o f the school district, one school
board member, two central office administrators, the Chief Education Officer of the
high school, two assistant principals and four classroom teachers were interviewed
using interview guides designed to gather data to answer the three research
x
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
questions. Informal observations o f the school and teachers’ classrooms were
conducted using a site observation guide as a framework for data collection.
The data collected provide evidence that the superintendent, district staff, site
administrators, as well as teachers are all keenly aware o f how the public and the
California State Board of Education measure student success. The data documents
that the school district maintains a fundamental belief in the value of data collection
and data utilization to raise student achievement while closing the achievement gap.
The major findings of this study will add to the body of research emerging around
the collection and analysis o f data to improve student achievement.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 1
OVERVIEW OF THE STUDY
Introduction
Public schools face an uncertain future. Among certain populations of
citizens there is a strong belief that public schools have failed and radical changes
need to be instituted to save America’s children from a life o f ignorance, poverty
and despair. In response, some critics o f public education push for private school
vouchers, legislation allowing parents to enroll their child in private schools if the
child’s current school is identified as a “failing school” (Kohn, 2004). Others argue
that public schools will improve if strong, measurable accountability systems are
instituted that include incentives and sanctions. The best articulation o f this
approach can be found in the speech delivered by President George W. Bush on
January 8, 2002, the day he signed the “No Child Left Behind” legislation.
No longer is it acceptable to hide poor performance. No longer is it
acceptable to keep results away from parents. One o f the
interesting things about this bill, it says that we're never going to
give up on a school that's performing poorly; that when we find
poor performance, a school will be given time and incentives and
resources to correct their problems. A school will be given time to
try other methodologies, perhaps other leadership, to make sure that
people can succeed. If, however, schools don't perform, if,
however, given the new resources, focused resources, they are
unable to solve the problem of not educating their children, there
must be real consequences. There must be a moment in which
parents can say, I've had enough o f this school. Parents must be
given real options in the face of failure in order to make sure reform
. is meaningful.
1
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
In contrast, findings from the 36th Annual Phi Delta Kappa/Gallup Poll, a
majority o f the responding American general public still supports public schools and
thinks they are doing a decent job, assigning them grades o f “A, B or C” at the
combined rate o f between 80% and 85%. However, the same poll indicated that
88% respondents believed closing the achievement gap was very important 64%, or
somewhat important 24% (Rose & Gallup, 2004). Additionally, the poll revealed
that 56% believed that public schools have the responsibility to close the
achievement gap and 66% believe reform should occur in the existing system. What
seems clear is that both critics and supporters of public education see a strong need
to improve public schools by raising student achievement.
Perhaps critics and supporters o f public education believe that the
consequences o f doing nothing or maintaining the status quo could be devastating to
the social and the political fabric o f the United States and could also place the
country’s economic health in peril. Politicians, business people and the general
public have made educational quality a public policy issue. Merely providing a free
and public education to every child has given way to policies that demand the
establishment of high quality, internationally competitive content and student
performance standards with the expectation that students in the United States be first
in the world in mathematics and science achievement (Goals, 1994, 2000). Maybe
the sentiment was best expressed by Tucker (2002b) that
it is still possible for young people to leave high school with an
8th grade level of literacy or less and get a low-skill job, but it
is not possible for that young person to support a family on that
wage or with that kind o f education.
2
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Campbell, Hombo and Mazzeo (2000) reported that over the last 30 years
this nation made slow progress in improving the quality o f education. In 1995,
Berliner and Biddle argued in their book The Manufactured Crisis: Myths, Fraud,
and the Attack on Am erica’ s Public Schools, that criticisms o f the public
educational system were unfounded and without merit. Stedman (1996) argued that
a careful analysis of the available data indicated the opposite and that the crisis in
education is real. The following discussion will explore the data currently used to
measure student achievement and consider whether these data are sufficient to make
meaningful judgments about educational quality and student achievement. The use
o f data to make improvements in the quality of education will be explored.
International Perspective on the Status o f Student Performance
Claims that public schools are not providing America’s students with a high
quality education appear to be supported by a variety o f evidence produced from
state, national and international assessments. Results from international assessments
detailed how American students are falling behind in comparison to students from
around the world (Peak, 1996, 1997; Takahira, Gonzales, Frase, & Salganik, 1998).
The Third International Math and Science Study (Peak, 1996, 1997; Takahira,
Gonzales, Frase, & Salganik, 1998 ) indicated a troubling trend in student
performance in the United States. It is important to note that studies comparing
educational outcomes among counties are not likely to report matched student
groups, due to differences in the makeup of school populations at various grades as
well as laws governing who is educated, for how long and under what circumstances
3
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(Bracey, 1998). Nevertheless, the trends reported by TIMSS must be taken seriously
when speculating on the future of the United States, economically, politically, and
socially.
Analyses o f TIMSS showed that students from the United States in grade 4
performed above average in mathematics and placed second only to students from
Korea in science, while performance of students in grades 8 and 12 fell significantly
lower (Peak, 1996, 1997; Takahira, Gonzales, Frase, & Salganik, 1998). By grade
8, students from the United States scored below the international average in
mathematics and slightly above the international average in science (Peak, 1996).
By grade 12, students from the United States were among the lowest scoring nations
in both mathematics and science education. When the scores o f American students
enrolled in Advanced Placement Calculus were compared to those of the highest
performing students from other nations, they only performed at the international
average. When the scores of American students enrolled in Advanced Placement
Physics were compared to those o f students from other nations, they performed
below the international average (Takahira, Gonzales, Frase, & Salganik, 1998). In
short, although our younger students appear to be doing well in mathematics and
science, by the time they are ready to leave high school and enter college or the
workforce, too many leave without the necessary knowledge and skills demanded
by today’s global economy (Greenspan, 2004).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
National Perspective on the Status o f Student Performance
In contrast to international assessments, results from our own national
assessments seem to indicate that public schools are doing, at the very least, an
adequate job. In fact, average student outcomes on national assessments support the
view that educational quality and student achievement have improved over time.
Longitudinal data on the National Assessment o f Education Progress (NAEP),
commonly known as the nation’s report card, showed improvement in the reading,
mathematics and science (Campbell, Hombo & Mazzeo, 2000). Data collected in all
three areas for over 30 years indicated improvement in all cohorts of students ages
9,13 and 17. Data also indicated a drop in scores from their highest points in
reading and science. Scores in mathematics are currently at their highest levels.
Disaggregated data on the NAEP assessments point out a less positive
assessment o f student performance. In mathematics, only 16% of twelfth graders,
24% o f eighth graders and 21% o f fourth graders scored at the “proficient level”
while 31%, 38% and 36%, respectively, were identified as being “below basic”
(Reese, Miller, Mazzeo, & Dossey, 1997). In reading, 25% o f twelfth graders, 30%
o f eighth graders, and 40% o f fourth graders were identified as being “below basic”
(White, 1996). In all three subject areas, and at each grade level, white students
significantly out-performed Black and Hispanic students (Campbell, Hombo, &
Mazzeo, 2000). NAEP data indicated that when student performance was
disaggregated by race, proficiency levels showed wide disparities of achievement
between various groups of students.
5
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The national data on student performance o f California’s students are even
more discouraging. California’s students in grades 4 and 8 scored below the national
average on NAEP assessments in mathematics, reading and science. For illustrative
purposes, California’s students at grade 4, approximately more than half, 54% of the
students scored at “below basic” while the national average was significantly less at
38%. In grade 8, approximately half, 49% o f the student scored at “below basic”
while the national average was approximately 39% (Reese, Jerry, & Ballator, 1997).
State Perspective on Student Performance
California uses both a nationally norm-referenced test, and a criterion
referenced test to measure student achievement. The nationally norm-referenced test
changed from the Stanford Achievement Test, version 9 (SAT-9) in 2003 to the
California Achievement Test, Sixth Edition (CAT-6). The CAT-6 is not standards-
based but it is California’s choice o f the norm-referenced tests available. California
also uses the California Standards Test (CST) as its standards-based criterion
referenced assessment o f student achievement. Student achievement results as
measured in 2003 by either the CAT-6 or the CST, for almost any grade and for
th
almost any subject area tested, show large percentages o f students below the 25
NPR and/or in the categories o f “below basic” and “far below basic.”
Recent data collected on student achievement at the international, national
and state levels reveal a significant need to make improvements in the quality of
education our students in public schools receive in an effort to raise student
achievement.
6
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Education Reform Efforts
National concerns about student achievement have been traced back to the
late 1950’s when the Soviet Union launched Sputnik. In 1983, The National
Commission on Excellence in Education published the landmark report A Nation at
Risk that once again focused attention on the devastating effects to the United States
if student achievement continued to decline or if it remained at dismally low levels.
The report warned that the United States was in danger o f losing its economic
competitiveness due to declining student achievement.
If only to keep and improve on the slim competitive edge we
still retain in world markets, we must dedicate ourselves to the
reform of our educational system for the benefit o f all— old and
young alike, affluent and poor, majority and minority. Learning
is the indispensable investment required for success in the
"information age" we are entering.
A Nation at Risk offered few new insights as to what needed to be done to
improve the quality o f education o f America’s public schools. The alarm raised by
the report about the United States’ ability to remain economically competitive in a
global economy is now viewed by many as mere propaganda and without merit. At
a national and international level, there appears to be a debatable relationship
between the economic vibrancy and competitiveness o f a nation and the quality of
education a nation provides its children and citizens (Bracey, 2003). Alan
Greenspan, (2004) Chairman o f the Federal Reserve, noted that:
Intensive research in recent years into the sources of economic
growth among developing and developed nations generally
point to a number of important factors: the state of knowledge
and skill o f a population, the degree o f control over indigenous
natural resources; the quality o f a country’s legal system,
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
particularly a strong commitment to a rule o f law and
protection o f property rights; and yes, the extent o f a country’s
openness to trade with the rest of the world.
Maybe the most important message in A Nation at Risk was the concern
voiced to reform the education system for the benefit for all. Implicitly, there was a
reiteration of an acknowledgment o f an educational opportunity gap that existed
between old and young, affluent and poor, majority and minority. Most educational
reforms, including the Effective Schools movement, the Standards movement, the
Comprehensive School Reform movement and the Accountability movement, and
most notably No Child Left Behind can all be viewed as an attempt in one way or
another to close the opportunity gap and/or the achievement gap.
Concern over an achievement gap, primarily between white students and
minority groups, especially African-Americans and Hispanics was not new. The
question o f an ability to resolve or eliminate the achievement gap was a different
matter. Coleman (1966) concluded that a student’s family background, not the
school was the primary determinate of student achievement. The Coleman report led
to a belief among some educators that schools did not make a difference. Harvard
Professor Ronald Edmonds challenged Coleman’s findings and identified five
common correlates found in successful schools in high minority, high poverty
urban, rural and suburban school districts (Edmunds, 1979). Edmund’s findings led
to the Effective Schools Movement which in turn would set a research-based
foundation upon which many future educational reform movements would build.
8
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Contemporary school reform began in response to the call for a return to
excellence, a renewed belief that all children can leam and a strong desire to
improve the achievement levels o f all children. In 1989, President George Herbert
Walker Bush convened the governors of all 50 states in an effort to address a
national response to pressing educational issues. The first National Education
Summit ended with an agreement to develop a set of national education goals, raise
student achievement levels of all children, raise academic standards and hold
schools accountable for results and carefully defined federal role in improving
education (Goals 2000: Building, 1996). By 1992, eight national goals were
established and a process began to develop a set of national voluntary standards in
the core curriculum areas. The resulting conceptual framework for the setting of
standards emerged from models adopted by industry in order to operate more
efficiently. In educational terms, standards would define what students should know
and be able to do within given periods of time.
By 1998, school reforms were making slow progress. The complexity o f
schooling and raising student achievement led to the notion that much more was
needed to be done to reform schools. In 1998, Comprehensive School Reform
required educators to incorporate 11 elements based in scientifically valid research
and effective practices (ECS, 1999). Concerns at the national, state and local levels
led to the development o f common curricular standards, the search for valid and
reliable school accountability measures, and a focus on using student achievement
data to improve teaching and learning.
9
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Role of Student Performance Data
Student performance data is essential to education reform efforts. Data use is
not a new concept in education reform, teachers, principals and other education
leaders are familiar with the use of data for understanding school and student
performance (Harvey, 2003). However, schools working toward change will need
sophisticated data on individual student achievement as measured against state
standards. Without data, principals and teachers fly blindly in any kind of school
improvement effort (Kress, 2003). According to Schmoker (1996), data illuminate
what was previously hidden and has the power to reveal strengths and weaknesses,
and enables data users the opportunity to judge the effectiveness or ineffectiveness
of innovation.
In the search for effective practices and evidence o f enhanced student
learning, student performance data provide certainty and alternatives to anecdotal
evidence, testimonials or subjective opinion (Scherer, 2003; Schmoker, 1996). In
previous practice, the examination of data has often been met with resistance and
fear. Data threatens the status quo because once confronted with reasonably valid
information, one is forced to act. Collection, analysis and dissemination of data,
make users accountable. In a goal-oriented organization, the use of data could be
threatening, but in effective organizations data informs practice and facilitates
greater efficiencies. Schools are no exceptions (Schmoker, 1996).
Policymakers and scholars envision the use o f student performance data as a
school’s vehicle for improving student achievement. State mandated assessments
10
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
are designed to provide common data through which schools, teachers and the
public can discuss areas for school and student improvement. Reformers and
scholars use data and performance indicators in at least five overlapping ways: 1) to
describe achievement, 2) to advance policy agendas, 3) to serve as the basis for
accountability, 4) to evaluate policies and programs and 5) to serve as information
management systems (Ogawa & Collom, 2000).
California’s Public Schools Accountability Act uses data and performance
indicators primarily for the dual purposes of describing progress in student
achievement and for holding schools accountability for results.
California Context
California has a history o f collecting student performance data. Until 2000,
individual school districts could select which assessments to use or could decide not
to administer assessments. Data, if it was used, was used to inform teachers of the
general skill strengths and weaknesses of students. There were few external
incentives for teachers to examine data and few consequences if they chose not to.
Unless teachers searched through a student’s cumulative record, most of the data
remained unexamined.
The Public Schools Accountability Act (PSAA) o f 1999 sought to make
schools accountable for student achievement results. Accountability forced schools
and educators to start thinking about data in new ways. For the first three years,
California schools and students had been evaluated using a norm-referenced test, the
Stanford Achievement Test, version 9 (SAT-9). Each Spring, California’s
11
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
elementary and middle schools were ranked on the basis o f student scores on the
SAT-9 in reading, language, spelling and mathematics. California’s high schools
were ranked on the basis of student scores on the SAT-9 in reading, language,
mathematics, science and social science, each with a 20% weight factor. The
ranking was accomplished using a formula that resulted in the Academic
Performance Index (API). Educators often viewed the rankings as invalid because
the SAT-9 was a non-standards based assessment and produced data inconsistent
with what was actually taught. Marzano (2003) called this “indirect learning data”
that could lead to false conclusions.
In 2003, the SAT-9 was replaced with a new norm-referenced test called the
California Achievement Test, 6th Edition, (CAT-6). The CAT-6 weight has been
reduced and given way to the California Standards Test in calculating the API. The
California Standards Test (CST) is a criterion referenced test based on the California
Academic Content Standards. Beginning in the Spring o f 2003, 80% an elementary
school’s API depended upon student scores on the California Standards Test and
only 20% on the new CAT- 6 (CDE, 2003). The API for high schools was
composed o f the SAT-9 English-language arts, mathematics and science, the CST in
English-language arts, mathematics and social science and the California High
School Exit Exam, a standards-based assessment in English-language arts and
mathematics. In addition scores for science and history were added to the API in
grades 9-11 (CDE, 2003).
12
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
According to the PSAA, API results were reported to school districts,
schools and to the general public. Schools that failed to meet state expectations were
warned that their position was unsatisfactory and were provided with resources and
assistance to improve student outcomes. Schools that were unable to raise their API
ranking faced a variety of sanctions including the removal o f administrators and
teachers, allowed students to attend any public school where there was space
available, allowed parents to open a charter school at the existing school site, and
allowed a college or university or other appropriate institution to assume the
management the school or the school may be closed (California Education Code
52055.5). Schools that met or exceeded state expectations had the potential to earn
fiscal rewards.
Students also faced serious consequences in this high-stakes educational
environment. Important decisions about promotion, retention and graduation made it
necessary for students to make adequate progress each year. In the near future,
students who failed to pass the high school exit exam would be in danger of not
receiving a high school diploma.
Current Status o f Data Utilization
Teachers, administrators and schools are under great pressure to utilize data
to raise student achievement but many have yet to figure out how to efficiently
access data, interpret data and use data to raise student achievement. Schools are
struggling to gain faster access to student achievement data that might assist them in
planning instruction and in providing additional assistance to struggling students.
13
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Schools are also struggling to understand the data they have available as most
teachers and administrators have limited training in data interpretation and
utilization. More importantly, schools, administrators and teachers struggled to
know how to modify instructions to improve student achievement. In many cases
this quest for using and understanding data is leading to organizational and
operational changes previously unexplored.
Recent thinking about data utilization for organizational improvement
indicates that a focus on data alone may not guarantee improvement without
common institutional goals, adequate fiscal and human resources and continuous
evaluation and adjustment. Models of data use guide the decisions that either build
or destroy business. Information is what makes an organization effective by
bringing focus to problems, coordination of solutions and allocations of fiscal and
human resources (Lawler, 1986).
Current models for using data must be identified that address the complexity
of schooling and provide a greater causal relationship between schooling and
student achievement. Some models already exist to move schools towards better and
more efficient use o f data to improve student achievement (Khanna, Trousdale,
Penuel & Kell, 1999).
Statement o f the Problem
California has implemented a high-stakes accountability system that
measures the performance of students and ranks schools on the basis of student
outcomes. The consequences are severe for students, administrators and teachers
14
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
when student achievement does meet state expectations. A proposed solution to low
student achievement is the collection and analysis o f student achievement data in an
effort to make informed decisions about curriculum, instruction and assessment.
Current data collection methods are inadequate to accurately capture the complexity
of teaching and learning. Data utilization practices are only now emerging that may
prove beneficial in helping schools raise student achievement. Lacking effective
models for data collection and utilization, it is conceivable that large numbers of
schools, teachers and students will suffer unnecessary consequences.
Purpose o f the Study
The purpose of this study is to describe and evaluate how one high school
used data to improve student achievement. The study will describe the model
developed at the school district level and will describe how the school identified,
collected and evaluated data during the year o f the study. The study will 1) assess
the extent to which the district design addressed state goals and expectations for
student achievement 2) describe the extent to which the district design was
implemented at the school site, and 3) attempt to assess the extent o f its usefulness
in achieving the goals of increased student achievement.
It is hoped that the results will shed light on the emerging state of school
reform and accountability in California.
Research Questions
1. What is the district design for using data and how was the design linked to
the state context for assessing student performance?
15
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2. To what extent has the district design been implemented at the district,
school and teacher level?
3. To what extent was the district design a good one?
Importance o f the Study
The recent explosion of data availability provides school districts, schools,
teachers and parents with valuable information about student achievement.
Unfortunately the process for evaluating data and making substantive changes in
curriculum, instruction and assessment has been problematic due to the lack of
effective models. Additionally, data use to raise student achievement is still
emerging as a practice and not one best model currently exists. Effective use o f data
may also provide beneficial to school intent on improving instruction to better meet
the academic needs o f students. In a political environment that emphasizes penalties
and sanctions against schools that fail to raise student performance, the use of data
may be the missing link that is needed to guide reform and improvement in
education quality.
Limitations
The focus o f this study is one secondary school during a specific given time
period of nine months. The resulting information may or may not be typical of other
secondary schools in the same district or the state. The objectivity o f the information
identified is only as accurate as the researcher’s ability to capture the responses of
the participating subjects in the study. Additionally, this qualitative analysis is
subject to the researcher’s selection and interpretation of the data collected. The data
16
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
collected and not collected may reflect unique characteristics and attitudes o f the
respondents that were caused by unknown forces or the dynamic nature of any
school setting.
Delimitations
This qualitative study examined one high school in California during
January 2004 - December 2004. The school under study had a diverse student
population o f medium size. The school is located in an urban setting and was
specifically selected rather than randomly selected. In addition to the characteristics
noted above the school was selected because 1) it was located in a school district
that had a strong reputation for academic success, 2) emphasized data use and 3)
school leaders had attempted to implement the design model to raise student
achievement.
Definitions
Academic Performance Index (API). One o f three components o f California’s
Public Schools Accountability Act (TSSAT State legislation was designed to
measure the performance of schools, especially the academic performance of pupils,
and to demonstrate comparable improvement in academic achievement by all
numerically significant ethnic and socio-economically disadvantaged subgroups
within schools. The API in 2001 was composed only o f the student test results on
the Stanford Achievement Test 9. By 2004, it has evolved to include a combination
o f norm-referenced and criterion referenced assessments. California’s public
schools are ranked according to the Academic Performance Index.
17
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Achievement gap. Persistent differences in achievement among different types of
students as indicated by scores on standardized tests, teacher grades, and other data.
The gaps most frequently referred to are those between whites and minority groups,
especially African-Americans and Hispanics.
Accountability, the notion that students, teachers, or an organization (a school,
school district, or department of education) should be held responsible for
improving student achievement and should be rewarded or sanctioned for their
success or lack o f success in doing so.
Adequacy. An approach to school funding that begins with the premise that the
amount o f funding schools receive should be based on some estimate of the cost of
achieving the state’s educational goals. This approach attempts to answer two
questions: How much money would be enough to accomplish the goals and where
would the money be best spent?
Alignment. The degree to which assessments, curriculum, instruction, textbooks and
other instructional materials, teachers’ preparation and professional development,
and system o f accountability all reflect and reinforce the educational program’s
objectives and standards.
Augmented Stanford 9 . Standards-based multiple choice items added to the
California Stanford 9 test. These items reflect the K-12 academic content standards
adopted by the California State Board of Education.
18
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
AVID. An educational program designed for students to help raise their
achievement levels and success rates in school. AVID, Advancement Via Individual
Determination.
CAT - 6 California Achievement Test. 6th Edition. A basic skills test, nationally
normed, and composed of multiple choice type questions. California adopted the
CAT-6 in 2002 to replace the SAT-9 as the primary measure o f student
achievement.
California Standards Test. Items added to the SAT-9 in 2001 to augment the test
specifically to test California standards.
Class Size Reduction (CSR). Reduction or maintenance o f now more than 20
students per teacher in a classroom, but the average must be lower than 20.4 over
the school year.
Comprehensive School Reform. A federally funded school reform initiative that
offers schools and districts the opportunity to implement school-wide research-
based reform strategies to increase student achievement. Formerly known as the
Comprehensive School Reform Demonstration Program (CSRD), the program was
renamed with the passage of the federal No Child Left Behind. Act of 2001. The
purpose o f the CSR program is to
improve student achievement by supporting the
implementation of comprehensive school reforms based on
scientifically based research and effective practices so that all
children, especially those in low-performing, high poverty
schools, can meet challenging State content and academic
achievement standards.
19
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
An outside provider is required to assist the school in achieving goals.
Content Standards. These academic standards describe what students should know
and be able to do in core academic subjects at each grade level and by extension
what teachers should teach.
Criterion-referenced test. A test that measures specific performance or content
standards, often along with a continuum from total lack o f skill to excellence (e.g.,
the California Standards Test). This type o f test can also have cut scores that
determine whether a test-taker has passed or failed the test or has basic, proficient or
advanced skills.
Curriculum. The courses o f study offered by a school or district.
Data. The resulting collection of information, evidence or facts gathered for some
purpose of analysis.
Equity. The belief that state governments have an obligation to equalize student’s
access to educational opportunities and thus life chances.
Formative assessment. Any form of assessment used by an educator to evaluate
students’ knowledge and undertaking o f particular content and then to adjust
instructional practices accordingly toward improving student achievement in that
area.
GPA. Commonly known as a student’s grade point average. The cumulative result
of student grades as defined by the end o f the semester marks which are given a
numerical value, totaled and then divided by the number o f credits attempted by the
student.
20
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
High-stakes test. A test that results in some kind of significant consequence for
those who score low, some kind o f reward for those who score high or both.
High School Exit Exam. In the near future, California public school students must
pass the state’s new exit exam before receiving their high school diploma. The exit
exam is not a college entrance exam or an honors exam. Instead, its purpose is to
test whether students have mastered the academic skills necessary to succeed in the
adult world. It is a pass-fail exam divided into two sections: language arts (reading
and writing) and mathematics. Students in grades 9 through 12 will have multiple
opportunities to take the test. Once students pass a section of the test they do not
have to take it again.
Multiple measures. Relying on one or more than one indicator to measure a
student’s academic strengths and weaknesses.
Norm-referenced assessment. An assessment in which an individual or group’s
performance is compared to a larger group. Usually the larger group is
representative o f the cross-section o f all students in the United States.
Performance assessment. Sometimes referred to as alternative or authentic
assessment. A performance assessment requires students to generate a response to a
question or prompt rather than chose from a set o f possible answers provided for
them.
Performance standards. Standards that describe how well or at what level students
should be expected to master the content standards.
21
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Professional development. Programs that provide opportunities for teachers or
administrators to acquire the knowledge and skills they need to perform their jobs
successfully.
Retention. The act or policy of holding students back from advancing to the next
grade level if they do not meet established performance standards.
Reliability. In testing, reliability is a measure o f consistency. For example if
someone took the same test on two different days, scores on both tests should be
similar.
SAT-9. The Stanford Achievement Test version 9. A basic skills test, nationally
normed, and composed of multiple choice type questions. California adopted the
SAT-9 in 1999 as the primary measure o f student achievement.
Social promotion. The policy of promoting students from one grade to the next with
their age group rather than based on student’s acceptable performance.
Standardized test. A test that is in the same format for all test takers. It often relies
heavily or exclusively on multiple choice questions. The testing conditions,
including instructions, time limits and scoring rubrics are the same for all students.
Sometimes accommodations on time limits and instructions are made for students
with special needs as identified with an official IEP.
Standards. Clearly identified, recognized and accepted definitions o f what students
should know and be able to do. Standards clearly define expected levels of
performance (performance standards) allocations o f resources (delivery standards),
and/or knowledge and skills (academic content standards).
22
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Standards-based reform. A recent shift in education policy and school reform that
focuses resources and student learning on clearly identified standards that identify
what students should know and be able to do and sometimes to level of
performance.
Summative assessment. A form o f assessment that gauges the final outcome o f any
innovation.
Organization o f the Dissertation
This dissertation is a qualitative analysis o f one high school’s use of data
utilization. Chapter 1 summarizes the current conditions leading to this study
including a statement o f the problem and research questions. Chapter 2 reviews the
literature about the collection of the data, how data is used in various settings and
how it is being used to raise student achievement. Chapter 3 provides an overview
of the study design and methodology used in collecting data. Chapter 4 presents the
findings of the study. Chapter 5 presents conclusions, implications and
recommendations for future research.
23
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 2
REVIEW OF THE LITERATURE
Introduction
The United States as a nation faces great challenges as it moves into the
twenty-first century. Among these challenges is the question of how to keep the
United States first in the world in power and economy. Politicians, businessmen and
the general public have focused their attention on education as one way o f
accomplishing this task. Yet as early as 1983 with the publication o f A Nation at
Risk, leaders and citizens alike have had to face the reality that too many schools
were inadequately preparing children for their future roles and responsibilities.
Reports over the last 30 years have documented the slow progress of
improving student achievement (Campbell, Hombo & Mazzeo, 2000). As student
achievement in the United States plateaued during the 1980’s and 1990’s, other
nations quickly moved ahead in key subjects such as mathematics and science
(TIMSS, 1996). Although there is not a strong researched based link between
student performance and economic competitiveness, governments continue to
perceive that national competitiveness depends on high levels of skills and
knowledge (Bracey, 2003; MCEETYA, 2002). Greenspan (2004) notes “that the
second most critical element of wealth creation in the United States, and doubtless
globally, is the level of knowledge and skill o f the population.” As a result, political
pressure and new accountability measures have forced school districts around the
24
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
nation to move away from educational “fads” and “hunches” to data analysis as a
means to improve student achievement.
Current Status on Student Performance
The rapid changes occurring in society and the workplace require citizens
that are life-long learners capable of solving complex problems by analyzing
information and data. For the most part, the skills and foundation knowledge needed
to accomplish these tasks are formally taught in schools in the academic subjects of
mathematics, science, and English language arts (Larson, Guidera & Smith, 1998).
Greenspan (2004) notes that generic capabilities in mathematics, writing, and verbal
skills are key to the ability to leam and to apply new skills and thus to earn higher
real wages over time. Results from assessments given to student over the last 30
years in math, science and reading paint a troubling picture for the United States.
While the world has become increasingly complex and in effect smaller as a result
of new technologies and innovations, student achievement in the United States in
math, science and reading has stagnated.
As a nation we evaluate student performance and school effectiveness using
a variety o f measures, most notably student assessments. On an international level
the United States participates in a number o f international assessments that measure
primarily mathematics and science content, concepts and application (Gonzales et
al. 2004; Lemke et al. 2004) and reading literacy (Mullis, Martin, Gonzales, &
Kennedy, 2003). On a national level, the United States administers assessments in
key academic areas through the National Assessment o f Education Progress (NAEP)
25
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(Johnson, 2004). According to state and national legislations, every state now
requires students to be assessed in English language arts, mathematics and science,
with some states having additional requirements in history, civics, social sciences or
social studies (U.S. DOE, 2002). At a local level, some school districts have
instituted their own annual assessments or they have developed and introduced
periodic assessments in English language arts and mathematics. Each of these
various assessments provides valuable data on student learning and other school
related functions.
International Assessments
The United States and other nations around the world participate in a number
of international assessments to gauge how their students’ understanding o f key
academic content knowledge and skills measure up to the best in the world (Blank
& Wilson, 2001; MCEETYA, 2002). In the last 10 years, these international
assessments have shown that the United States has been primarily in the lowest
ranks or near the international average (Gonzales et al., 2000; Lemke et al., 2004;
Peak, L. 1996; Takahira, Gonzales, Frase, & Salganik, 1998). Some of the more
recent studies showed that students from the United States outperformed the
international average scaled score but again, the United States has not been the
world leader (Mullis, Gonzales, & Chrostowski, 2004; Mullis, Martin, Gonzales,
and Chrostowski, 2004; Mullis, Martin, Gonzales, & Kennedy, 2003; Peak, 1997).
The largest of these international studies was developed and administered by
the International Association for the Evaluation o f Educational Achievement (IEA),
26
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
an independent, international cooperative o f national research institutions and
governmental agencies. IEA began in 1958 and has conducted more than 23
research studies o f cross-national achievement. The Third International
Mathematics and Science Study, now renamed the Trends in International
Mathematics and Science Study (TIMSS), is the best known and most commonly
cited international study. TIMSS conducted three similar cross-national studies that
were administered in 1995, 1999 and 2003. The main purpose o f TIMSS was to
provide achievement data on mathematics and science in order to better inform
policymakers about educational policies, practices and outcomes that hold strong
potential to enhance math and science learning within and across systems of
education (Mullis et al.,1997). IEA also administers the newly developed Progress
in International Reading Study (PIRLS). The main purpose o f PIRLS was to provide
achievement data on reading comprehension in the areas o f literary and
informational text. PIRLS collected additional data about the context of literacy
development in the home and school settings in order to better inform policy makers
about educational policies, practices and outcomes that hold strong potential to
enhance reading comprehension (Mullis, Martin, Gonzales, & Kennedy, 2003).
A third international study, the Program for International Student
Assessment (PISA) measures 15-year-olds, capabilities in reading literacy,
mathematics literacy and science literacy. PISA was first established in 2000 by the
Organisation for Economic Co-Operation and Development (OECD) established in
1960 for the purposes o f promoting economic growth, employment and a rising
27
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
standard o f living in and among member nations. OCED established the Center for
Educational Research and Innovation in 1968 in an effort to analyze and develop
research, innovation and key indicators in current and emerging education and
learning issues and their links to other sectors o f policy. Additionally, the Center
explores forward-looking coherent approaches to education and learning in the
context o f national and international cultural, social and economic change. It has
conducted two study cycles. PISA 2000 primarily focused on reading literacy. PISA
2003 primarily focused on mathematics literacy. PISA 2006 will primarily focus on
science literacy. In addition to reading, math and science literacy, PISA measures
general or cross-curricular competencies such as learning strategies (Kirsch, 2002;
Lemke et al., 2004; OCED, 2004).
TIMSS: The Third International Math and Science Study. TIMSS-R. and Trends
In 1995, the Third International Math and Science Study (TIMSS) provided
the United States with valuable, if startling, information about student achievement
in math and science (Peak, 1996; Peak, et al 1997; Takahira, Gonzales, Frase &
Salganik, 1998). Unfortunately the news was not good for the United States. The
TIMSS study placed the achievement level o f students from the United States in
comparison to the achievement o f student in 41 other countries. The results o f the
TIMSS study raised the level of concern among politicians, scholars, and
businessmen in the United States. The results revealed that the student achievement
in math and science in the United States had fallen to distressingly low levels for
students just about to leave high school.
28
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The TIMSS 1995 study showed a troubling trend in student achievement in
the United States. It showed that students from the United States in grade 4
performed above average in mathematics and placed only second to Korea in
science (Peak, 1997; Stevenson, 1998). B y grade 8, students from the United States
scored below the international average in mathematics and only slightly above the
international average in science (Peak, 1996). By grade 12, the TIMSS data revealed
that students from the United States were among the lowest performing students in
both mathematics and science of all the nations participating in the study (Takahira,
Gonzales, Frase, & Salganik, 1998). Even when American students in Advanced
Placement Calculus classes were compared to other nations’ highest performing
students, American students only performed at the international average. In science,
students from the United States in Advanced Placement Physics classes, when
compared to students from other nations, performed below the international average
(Mullis et al., 1998).
Performance of students from the United States in TIMSS-R and Trends-
TIMSS show signs of reserved promise. On the 2003 TIMSS, U.S. grade 8 students
reduced the gap between the leading scoring nation o f Singapore but the gap
widened between the U.S. grade 4 students and the leading scoring nation of
Singapore. In Math, U.S. grade 8 students’ average scaled scores increased and
again reduced the gap between the leading nation o f Singapore. There was no
change in the average scaled scores for grade 4 students o f the U.S.. Interestingly,
the number o f countries outranking the United States at grade 4 increased from four
29
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
countries in 1995 to seven countries in 2003 as a result o f a few countries that
significantly improved their scaled scores while the U.S. scores stagnated. It is also
important to note that students in grade 8, for both mathematics and science subject
areas, improved their average scaled scores for each administration o f TIMSS.
Table 1 provides a good visual of student performance for all three administrations
of TIMSS for grades 4, 8, and 12.
Table 1
United States Averaged Scaled Scores on TIMSS 1995-2003
Subject/Grade TIMSS
1995
TIMSS-R
1999
Trends
2003
Difference
(2003-1995)
Science Grade 4 542 536 -6
Science Grade 8 513 515 527 +15
Science Grade 12 480
. > , - ' .
Math Grade 4 518 518 0
Math Grade 8 492 502 504 +12
Math Grade 12 TIMSS
1995
International
Average
Differences between U.S.
and International Average
Math and Science
Literacy Grade 12
471 500 -29
Math Literacy Final
Year-Grade 12
461 500 -39
Science Literacy
Final Year - Grade 12
480 501 -21
On PIRLS another international assessment, students from the United States
performed with better success. PIRLS, an international assessment on the reading
literacy achievement o f 4t h graders studied two principal purposes for reading,
literary experience and to gather and apply information. Students from the U.S.
30
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
averaged scaled score (542) was significantly above the international average scaled
score of (500). Their average scale score performance was statistically outranked
only by the top scoring nation o f Sweden (561), the Netherlands (554) and England
(553). U.S. students were statistically comparable to eight other nations and
significantly outranked 23 other nations (Mullis, Martin, Gonzales, & Kennedy,
2003).
PISA, the Programme for International Student Assessment, measures
mathematical literacy, problem-solving, reading literacy and scientific literacy
among 15-year-old students. In 2000, PISA primarily measured reading literacy and
in 2003 primarily measured mathematical literacy. PISA is built on a different
framework that attempts to capture an understanding o f how well equipped
adolescent teens are for their future by emphasizing items that have a real-world
context. The framework was developed not necessarily from school curricula but
from an internationally agreed upon understanding o f reading, math and science
literacy (Lemke et al., 2001). PISA tasks are more likely to be multi-step and
require interpretation in an effort to reflect the dynamics of everyday situations.
PISA also differs from TIMSS and PIRLS in that PISA is designed to generate
useful indicators to benchmark performance and inform policy rather than to
directly improve classroom instruction.
Under the above noted circumstances students from the U.S. performed in
2000 at the approximate international OECD average in each o f the three areas
assessed. The 2000 PISA results showed that the performance o f America’s 15-
31
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
year-old students were amazingly undistinguished, except to show that they were
not among the lowest international performers. In the 2003 PISA, America’s student
performance remained basically “flat” with the exception o f the math assessment. In
the 2003 math portion of the PISA, the primary focus o f the 2003 PISA, American
students performed below the international average scale score in almost all areas.
In essence, American students performed distinguishably less well than they did in
2003 (Lemke et al., 2004).
All o f the above mentioned international assessments provide valuable
information to participating nations about education systems around the world.
While much o f the public is obsessed with the national rankings and standings
highlighted in follow up reports, much o f the education establishment is interested
in the other findings produced that can help inform new policy. International
assessments have provided comparative data about teacher training, educational
preparation, credentialing and instructional practices. They have provided
information about the curricula and the opportunities to leam that school systems
provide students especially information about content depth and breadth, and
concept topics covered. In most cases they confirm national trends in both
achievement gaps among gender, class and race.
International assessments often have the power to alarm publicly
conscientious businessmen, researchers at public policy institutes, newspaper
reporters but often fail to alarm state and local officials charged with the
responsibility for educating American youth. Critics o f international assessments
32
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
often state that international assessments document what local schools teach and
don’t teach. Unlike many counties in the world where decision-making rests at a
national ministry or at a nationally centralized institution, most o f the functions of
education in the United States remain at state and local levels (Mullis, 1998).
International assessments mean 50 different things to 50 different state educational
agencies and innumerable things to thousands o f local school districts officials.
National Assessments: National Assessment o f Education Progress: NAEP
In an effort to monitor the progress o f student achievement in the United
States, Congress established the National Assessment o f Education Progress
(NAEP) in 1969. For over 30 years, the United States has been monitoring student
performance in many subject areas through the National Assessment of Education
Progress. NAEP is actually a collection o f several types o f assessments. “Long-term
trend” NAEP tests a nationwide sampling o f students ages 9, 13, and 17 in an effort
to estimate student achievement over long periods o f time in reading and
mathematics. “National” NAEP assesses student performance at grades 4, 8, and 12
by periodically administering tests in core subject areas such as mathematics,
science, reading, United States history and civics. NAEP serves as the nation’s
report card and is the only ongoing measure o f the student achievement across time.
Consequently, NAEP serves as one of the most important measures by which the
public can judge the effectiveness and quality o f the education its children receive
(Johnson, Princiotta, & Vanneman, 2003).
33
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The Long-term trend NAEP’s longitudinal data on math, science and reading
achievement over time since 1971 and 1973 show positive improvement over the
years (Campbell, Hombo & Mazzeo, 2000). Data collected for all three subject
areas indicate higher achievement for student cohorts ages 9, 13, and 17. In reading,
average performance for 17 year olds steadily improved from 1971 through the
early 1990’s but has fallen slightly since. For 13 year olds, reading scores have risen
since 1971, reached a peak in 1992 and have remained basically unchanged. Scores
for students 9 years old have risen since 1971 and are currently at their peak level.
In science, average performance for 17 year olds is about the same as it was in 1973
but has improved significantly after a decline in the late 1970’s and 1980’s. Scores
for students 13 years old are significantly higher than they were in 1973 but have
slightly fallen off since their 1992 high mark. The same pattern shown by 13 year
olds holds true for 9-year-old students as well.
The greatest improvement in average student achievement for America’s
youth is found in the area of mathematics. In all three cohorts o f students, student
achievement is at an all time high. Seventeen year old students showed sustained
improvement through the 1980’s but has slowed since. For 13-year-old students
trends indicate two growth periods o f student achievement from 1973-1990 and a
second growth period from 1990 to 1999. Scaled scores for 13 year olds are 10
points higher in 1999 than they were in 1973. Students age 9 showed significant
growth from 1973-1990 and slower growth since. The overall scores are 13 points
higher than they were in 1973.
34
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
In the most recent National NAEP report cards in reading (Donahue, Daane,
Grigg, and Educational Testing Service, 2004; Donahue, Voelkl, Campbell &
Mazzeo, 1999), science (O’Sullivan, Reese & Mazzeo, 1997) and math (Braswell,
Daane, Grigg, Education Testing Service, 2003; Braswell, et al, 2001), nationwide
student achievement increased in most grade levels. National N A EP’s main
assessments are used to issue report cards in particular subject areas. These new
assessments were developed beginning in 1992 in response to demands for
additional and specific state-level data on student performance in key subject areas.
These assessments evaluate and report on the educational progress o f students at
grades 4, 8, and 12 instead of reporting on students at ages 9, 13 and 17, as in the
longitudinal trend assessments.
The improving level of student achievement in reading, science and math as
reported by NAEP does not reveal the entire story. NAEP also disaggregates its data
on a variety o f levels. NAEP’s disaggregated data point to a less positive assessment
of student performance. The NAEP report cards distinguish between levels of
achievement using labels of “advanced,” “proficient,” “basic,” and “below basic.”
The “advanced” level signifies superior performance at a given grade. A
“proficient” level is the desired level o f performance. A “basic” level designation
represents partial mastery and is a step towards proficiency. The “below basic”
designation represents significant underachievement. Nationwide in 2003, in
reading at grade 4, only 8% of students were considered at the advanced level o f
achievement while 31% were considered proficient, 63% were predicted to be at or
35
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
above “basic” and 37% were considered at “below basic” reading (Donahue, Daane,
Grigg, and Educational Testing Service, 2004). In 2003 at grade 8 only 3% o f the
students were considered “advanced,” 32% were identified as “proficient,” 74% are
at “basic” and 26% of 8th graders were considered “below basic” (Donahue, Daane,
Grigg, and Educational Testing Service, 2004). By grade 8 in math, 5% were
considered “advanced,” 23% were considered “proficient,” 39% were at “basic” and
32% were identified as being “below basic” (Braswell, Daane, Grigg, and Education
Testing Service, 2003). The disaggregated data for science was similar to student
achievement in math. The point to take notice here is that at nearly every grade level
and in all three subject areas, between 25% and 37% o f students were identified as
“below basic” levels of achievement.
National NAEP Achievement Gap
When the NAEP report card is disaggregated by race for the nation, an even
greater concern has to be raised (Braswell, Lutkus, Grigg, Santapau, Tay-Lim &
Johnson, 2001). The disaggregated math achievement levels for white students
grade 4 were in 1999-2000, 3% “advanced,” 30% “proficient,” 46% were at “basic”
and 20% were considered “below basic.” The math achievement data for Black
children grade 4 reveals that less than 1% were considered “advanced,” only 5%
were considered “proficient,” 33% were considered at the “basic” level and 61% of
Black children were identified as being “below basic.” The achievement level for
Hispanic children in grade 4 was not much better with 1% identified as “advanced,”
10% as “proficient,” 38% as basic and 52% o f Hispanic children were considered at
36
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
“below basic” levels o f achievement. The gaps at grade 8 widen with only 23% of
White students at “below basic” in comparison to 68% for Black and 59% for
Hispanic. B y the 12t h grade, 26% of White students, 69% o f Black students and 56%
o f Hispanic students were identified at “below basic.”
When the 1998 NAEP report cards are disaggregated by race for the nation
for reading achievement similar trends occur (Donahue, Voelkl, Campbell &
Mazzeo, 1999). The 4th grade “below basic” level o f achievement for White students
was about 27%, 64% for Black students and 60% for Hispanic students. By the 8t h
grade the gap was narrowed with only 18% White students at “below basic,” 47% of
Black students and 46% of Hispanic students at “below basic.” By the 12th grade the
gap narrows further but was still significant.
The achievement gaps between White, Black and Hispanic students is wide
in math, science and reading. These gaps remain true for the most recent
assessments administered by NAEP and are consistent with the longitudinal data
studies o f NAEP as well. The NAEP longitudinal report shows increased
achievement for all races during the last 30 years but the gaps in achievement
between races appear to be about the same (Campbell, Hombo, & Mazzeo, 2000).
Student Achievement in California
Education in the United States is a decentralized endeavor and is largely a
responsibility o f the individual states. Be that as it may, politicians, educators,
businessmen and parents across the nation are concerned with how well their
37
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
students compare with students in other states. The NAEP assessments provide
information comparing student achievement across the states.
Unfortunately the news is not good. NAEP assessments show California
students near the bottom o f the student performance when compared to students in
other states in math and reading. In reading fourth grade students in California
perform below the national average (Donahue, Voelkl, Campbell & Mazzeo, 1999).
In fact it has one o f the highest percentages o f students identified as “below basic”
at 52% and a total o f 80% reading below proficient. California’s closet comparison
states are Mississippi, Louisiana and Hawaii. At the eighth grade reading level,
California improved slightly passing Alabama and tying South Carolina as having
the highest percentage o f students “below basic.” Only Mississippi and Hawaii have
a greater percentage o f eighth graders identified as “below basic.” In mathematics,
eighth grade students in California also performed below the national average and
outperformed only Mississippi (Braswell, Lutkus, Grigg, Santapau, Tay-Lim &
Johnson, 2001). At the eighth grade level California remained below the national
average scaled score sharing the bottom with Alabama, Arkansas, Louisiana,
Mississippi and Tennessee. Finally, California has also done little to close the
existing gaps in achievement between White, Black or Hispanic students (Barton,
2001).
The results from NEAP and TIMSS can add significantly to the information
base available for policymakers in individual states so that they may make informed
decisions on how to improve student achievement. Although NAEP and TIMSS are
38
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
different in many respects, they share enough of a conceptual framework that allow
comparisons to be made between states and nations, if states would have
participated in TIMSS (Blank & Wilson, 2001; Johnson, 1998). It is estimated that
California students in eighth grade mathematics would score similarly to students in
the United States but would only outperform nations such as Iran, Kuwait and South
Africa. In eighth grade science it is estimated that students from California would
score below the United States, but not significantly different from Denmark, France,
Spain, Switzerland, Hong Kong and several other nations. It would also only
outperform Belgium, Colombia, Cyprus, Iran, Kuwait and South Africa.
Long-Term Consequences of Low Student Achievement
The consequences for long-term, low student achievement might have
devastating effects on the lives of poor performing students, and society in general.
The world has significantly changed during the last 30 years and will continue to
change rapidly in the future. Some politicians, businessmen and thoughtful citizens
question whether the United States will be able to maintain its comparatively high
standard o f living. Others question how the United States will maintain it economic
competitiveness in the world market place. And yet others question whether or not
the United States will be able to maintain its fragile institutions of democracy amid
the political, social and economic pressures building from within resulting from the
disparities in real income and accumulation of wealth (Greenspan, 2004). These
questions become increasingly important as America moves from a nation of
industry and manufacturing to one economically grounded upon information and
39
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
knowledge. This paradigm shift has caused many leaders to focus on comprehensive
school reform in education as a means to improve student achievement and to
academically prepare students for the 21st century (Goals, 2000).
Comprehensive School Reform
National concerns about the quality o f schooling and about student
achievement have been prominent throughout the years of educational reform.
Reforms in education during the 20th century have addressed issues o f compulsory
attendance, the establishment o f comprehensive high schools, desegregation,
education for students living poverty, bilingual education and a host o f other social,
political and educational issues. Basom (1999) believes that “Whole-School
Reform” or “Comprehensive School Reform” (CSR) emerged over the last three
decades from four evolutionary movements namely: the effective schools
movement; school improvement movement; systems change movement; and the
standards movement. CSR addresses school reform from the perspective that in
order to improve student achievement, reform must be systemic, encompassing all
areas o f schooling from policy development, use o f resources, to professional
development (Slavin, 2001). Borman, Hewes, Overman, and Brown (2002) added
clarity by stating that, “CSR focuses on reorganizing and revitalizing entire schools,
rather than on implementing a number o f specialized and potentially uncoordinated,
school improvement initiatives.”
The roots o f Comprehensive School Reform can be found in the effective
schools’ research o f Ron Edmonds, 1979. Edmonds’effective schools researched
40
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
linked school properties and characteristics to school improvement by comparing
“effective and “ineffective” schools. The research was primarily conducted to
disprove the findings o f the 1966 Coleman Report and popular beliefs about student
achievement o f mainly poor, urban and minority children as measured by
achievement tests. Edmonds identified a few large urban schools that should have
performed poorly on standardized achievement tests, as previously predicted by
family background and the low socio-economic status of the students, but instead
scored higher than expected (Edmonds, 1979). Edmonds and other researchers’
(Astuto & Clark, 1985; Brookover & Lezotte, 1979; Firestone, 1991) findings
indicated that these schools had a set o f common components that led to increased
student achievement in urban and low socio-economic schools. These factors
included established high expectations by teachers an clearly stated academic goals
for students, strong principal leadership, an emphasis or concentration on basic
skills, expectations of learning occurring in an orderly environment, and frequent,
systemic evaluation of students and connecting what was taught to what was tested
(Cuban, 1998; Goddard, Sweetland, & Hoy, 2000).
During the eighties, schools embarked on numerous reforms emerging from
the research on the effective schools components and were generically categorized
as “school improvement.” Most o f these reforms were school-based aimed at
making schooling a better experience for students and teachers. While many of
these schools formed a more collaborative and reflective environment, Marsh
(1996) reported that many of these reforms failed because they lacked explicit focus
41
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
on improving student performance. While much energy was spent on reforms such
as school-based management, teacher empowerment and school reorganization,
these efforts did little to improve student achievement. In other words change and
structure became the goal rather than the means to the goal of increased student
achievement.
The “systems change movement” o f the early nineties expanded the way
reformers conceptualized change. In his book The Fifth Discipline, Peter Senge
(1990), brought forth the notion that change was more about the interrelationships
of system elements rather than linear cause-effect chains. In contrast to school
improvement reforms that isolated rather connected system elements, systemic
reform sought to capitalize on the interrelationships among system elements. Smith
and O’Day (1991) proposed a strategy that sought to bring coherence to educational
systems by coordinating policy that focused on developing a unified vision and
goals, a well-structured instructional guidance system, and a restructured
governance system. Odden (1995) summarized the elements considered important to
successful reform as:
understanding o f the nature o f problems with the current
education system; a vision o f what a high performance
school could look like; a series o f state policies supporting
that vision including student academic goals, curriculum
frameworks, instructional materials, performance-based
assessment, revised teacher training and licensure, and new
forms o f professional development; and restructured
governance and finance systems, including new forms of
accountability.
42
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The systems change movement resulted in many challenges for states
attempting to integrate complex reform elements. Fuhrman (1994) found that some
states had done a better job than others in building an integrated educational system,
some had encountered obstacles and setbacks and some had overcome obstacles and
began to demonstrate success in the form o f increased student achievement.
Standards-Reform Movement
The desire for increased student achievement led to a re-emergence o f the
importance o f maintaining high expectations and clearly stated student academic
goals for all students. High expectations and clear academic goals were key
components of both the “Effective School Movement” and the “Systems Change
Movement.” In 1989 President George H.W. Bush called together the governors of
all 50 states to discuss the current status o f education and the need to make
significant improvements in the education provided to the nation’s children
(Building, 1996; Goals, 2000). The National Education Summit o f 1989 ended with
a commitment to:
• The recognition that states must focus on raising the achievement levels of
all students rather than on simply creating models of success.
• A broad consensus among state and business leaders, parents and the
education community that education reform must raise academic standards;
measure student and school performance against those standards; provide
schools and educators with the tools, skills, and resources needed to prepare
students to reach the standards; and hold schools accountable for the results.
43
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• A clear statement of an important and carefully defined federal role in
improving education, including financial, research, and dissemination
support and greater flexibility in administering programs.
In 1990, President George Herbert Walker Bush established the National
Education Goals Panel (NEGP) as an independent agency in the Executive Branch.
The purpose of the NEGP was to develop, monitor and report on the national goals
that emerged from the summit. By 1992, NEGP established eight national goals:
1. All children in America will start school ready to learn.
2. The high school graduation rate will increase to at least 90%.
3. American students will leave grades 4, 8, and 12 having demonstrated
competency in challenging subject matter, including English, mathematics,
science, foreign languages, civics and government, economics, arts, history,
and geography; and every school in America will ensure that all students
learn to use their minds well, so they may be prepared for responsible
citizenship, further learning, and productive employment in our Nation's
modem economy.
4. The Nation's teaching force will have access to programs for the continued
improvement of their professional skills and the opportunity to acquire the
knowledge and skills needed to instruct and prepare all American students
for the next century.
5. U.S. students will be first in the world in science and mathematics
achievement.
44
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
6. Every adult American will be literate and will possess the knowledge and
skills necessary to compete in a global economy and exercise the rights and
responsibilities o f citizenship.
7. Every school in America will be free o f drugs, violence and the unauthorized
presence o f firearms and alcohol and will offer a disciplined environment
conducive to learning.
8. Every school will promote partnerships that will increase parental
involvement and participation in promoting the social, emotional, and
academic growth of children.
Goal 3 o f the national goals led to the early efforts at developing government
sponsored academic content standards. In the early 1990’s the federal government
promoted the development o f voluntary national curriculum standards in the core
curriculum areas o f mathematics, English language arts, science, history, civics and
geography. Other organizations also responded and developed voluntary national
standards in an effort not to be squeezed out o f the curriculum. By 1994, some of
the standards projects had garnered approval from a broad consensus o f policy
makers. Under the Improving American School Act o f 1994, states were required to
submit a plan demonstrating that it had “developed or adopted challenging content
standards and challenging student performance standards that will be used by the
State, its local educational agencies, and its schools” (H.R. 6, 1994). At the same
time the movement for national voluntary standards was collapsing. The standards
developed in history were causing an uproar and led in 1995 to a Senate vote of 99-
45
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1 to censure the standards (Ravitch, 1996). The English standards had their funding
stopped when it was perceived that the effort was not producing anything usable.
Since 1995, the states have assumed the primary responsibility for setting academic
content standards.
Role o f Standards in Comprehensive School Reform
Standards are a key component o f education reform. Thompson (2001)
argued that at the heart o f standards-based reform is a deeply rooted and
fundamental concern with equity. Academic content standards identify what
students should know and be able to do in key content areas such as mathematics,
English language arts and reading, science and history-social science. Since the goal
o f comprehensive school reform is to raise student achievement, it only makes sense
that everyone, students, teachers, parents and the general public, know what is to be
specifically learned (Carr & Harris, 2001; Merrow, 2001). In the past too many
children were victims of low expectations and a “watered-down curriculum.
Common “high expectation” academic content standards help teachers and students
focus their efforts on essential subject matter. Depending on what has been adopted
by state and local school boards o f education, academic content standards could
have been developed at the national, state or local levels in one or each of the core
academic content areas. In California, students are at a minimum responsible for
learning the state identified academic content standards in mathematics, science,
English language arts and history-social science. Local school districts could also
46
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
impose additional and different academic standards on their students in an effort to
meet localized expectations and needs.
Standards done well in a CSR model would address a variety o f areas of
teaching, learning and running a school (Carr & Harris, 2001). Standards reform
should inform decisions made about curriculum and instruction, especially what
teachers choose to teach and what students will be required to learn. Standards
should influence instructional guidelines and provide access to instruction to a
diversity o f students. They should also guide, inform and influence assessments of
student work. Standards-based reforms will also impact the professional
development opportunities that teachers have available to them. If students are
going to be held accountable for learning standards-based lessons, then teachers
should have the knowledge, skills and abilities to help students reach the identified
standards and they should be accountable for using standards to drive instruction. In
other words, administrators need to incorporate standards into their supervisory and
evaluation responsibilities o f teachers and ensure that teachers have the tools to get
the job done. These areas address key elements in school life that when done well
can systemically change schools but they do not list all that has to be done.
Standards also serve other purposes as well, especially in comprehensive
school reform. In comprehensive school reform all efforts focus on improving
student achievement. In the past, school resources were squandered in a variety of
ways that may or may not have addressed student achievement. A perfect example
is in the area of professional development. Before CSR, professional development
47
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
was often a one-shot experience that rarely had anything to do with instruction.
Under CSR, professional development is designed to improve teachers’ skills and
abilities in an effort to help their students meet standards. Thompson (2001) stated
that a standards-based approach breaks down teacher isolation and calls for
collaboration so that there is a wide understanding o f educational goals. The mere
establishment of academic content standards forced teachers to look at what they
were teaching and to modify their instructional practices (Carr & Harris, 2001).
Teachers now have to focus on helping all students meet standards, even those who
have never excelled in school. Professional development should be based on proven
models that have clear and measurable outcomes for student achievement.
Another area where the establishment o f standards has had an effect in CSR
is in the alignment o f instructional resources across grade levels and subject areas.
School districts and teachers are now examining the textbooks they use and the
instructional materials they buy to supplement instruction. This careful examination
of materials is forcing publishers to change materials they offer to students, teachers
and to schools so that the materials reflect the standards student need to know and
meet.
Standards help schools communicate to parents and the community the
learning expectations for students. When parents and community members clearly
know what students should be learning they can mobilize the fiscal and moral
support that is essential if schools are to raise student achievement. Public support
for learning could result in a community passing financial bond measures to assist
48
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the school in raising badly needed funds in order to provide adequate instructional
supplies and school facilities. It could also take the form o f maintaining the
continuity of elected school board members so that the comprehensive reform
measures could continue. Clear standards for learning help unite students, parents,
teachers, administrators and the larger community in the mutual goal o f raising
student achievement (Greene, 1998).
A significant contribution o f standards to the CSR movement is the creation
o f standards-based assessments. Standards-based assessments allow teachers and
students to focus on results. Standards-based assessments in broad terms link
student learning with opportunities for students to demonstrate their understanding
as part o f the daily experience o f school (Rothman, 1999). Assessments aligned with
standards, when done well, will provide students with criteria to determine whether
the student work produced meets standards. These assessments should provide
important data to evaluate students but should also be used to identify areas of need
or remediation in an effort to raise the student achievement o f students. Standards-
based assessments also facilitate a move to criterion referenced assessments rather
than norm referenced assessments. In other words, the goal o f creating standards is
to provide students with opportunities to meet standards by comparing student
achievement to an absolute level o f performance, rather than to compare students to
their peers on a “bell-shaped” curve. The end result of standards should be a
comprehensive assessment system that is standards-based and provides teachers
49
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
with valuable information that will help individual students, groups within schools,
schools and school districts raise student achievement.
New American Schools
Whole-school reform gained significant support from the establishment of
the New American Schools Development Corporation (NAS) in 1991. Under the
leadership o f David Kerns, NAS raised over $130 million from private business
leaders and foundations. The idea behind NAS was that private-sector venture could
develop “break the mold” whole-school restructuring models. The guiding premise
o f NAS was that characteristics of high-quality schools could be distilled and re
packaged into newly designed coherent reform programs. These newly designed
whole-school reform programs could then be introduced lower-performing schools.
Low performing schools would adopt a reform program that met their individual
needs in an effort to improve student learning and performance (Berends, Bodily &
Kirby, 2002).
The NAS experiment was a unique opportunity to study school reform from
design conceptualization through efforts at large scale-up implementation. The
entire process was to take place over the course of seven years and four phases.
During the first phase, design teams submitted proposals to NAS. The second phase
started with 11 teams that developed reform designs and ended with nine. The
demonstration phase marked the implementation phase o f nine designs into 147
schools. The demonstration phase lasted two years from 1993-95 with seven designs
making into the final phase. Table 2 identifies the seven design teams and the
50
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
founding organizations. The final phase marked the scale-up component o f the NAS
theory o f action. Seven teams entered the scale-up period.
Table 2 New American Schools Design Teams and Founders
Team Founders
Authentic Teaching, Learning and
Assessment for All Students
(ATLAS)
Coalition o f Essential Schools,
Education Development Center,
Project Zero, School Development
Program
Purpose-Centered Education Audrey Cohen College
Co-nect Schools BBN Corporation
Expeditionary Learning Outward
Bound
Outward Bound, USA
Modem Red School House Hudson Institute
National Alliance for Restructuring
Education (America’s Choice)
National Center on Education and
the Economy (NCEE)
Roots and Wings Robert Slavin, Nancy Madden, and
a team of developers from Johns
Hopkins University
The underlying conceptual framework for all o f the above designs consisted
o f putting into practice elements or sets o f activities defined by each of the design
teams as core components (Berends, Bodily & Kirby, 2002). Core components
generally included professional development, curriculum and instructional
materials, content and performance standards, assessments, organization and
governance requirements, and parent and community involvement. The idea was
that when these core components were consolidated into a coherent design and
implemented into a school setting, the learning environment would change for the
better thus leading to enhanced student learning outcomes.
51
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Comprehensive School Reform Demonstration
In 1998 the United States Congress established the Comprehensive School
Reform Demonstration (CSRD) program to improve student achievement. The
Federal government made $120 million available to schools eligible for Title I funds
to participate in the CSRD program. The legislation identified 17 researched-based
programs or models that faculties could use to implement reform in their schools.
Comprehensive school reform models had to include certain characteristics if they
expected to participate in the federal CSRD program. Originally the U.S.
Department of Education required schools to adopt CSRD programs that addressed
nine elements:
1. Innovative strategies and proven methods for student learning, teaching,
and school management that are based on reliable research and effective
practices, and have been replicated successfully in schools with diverse
characteristics;
2. A comprehensive design for effective school functioning that (a) aligns
the school's curriculum, technology, and professional development into a
school-wide reform plan designed to enable all students to meet
challenging state content and performance standards and (b) addresses
needs identified through a school needs assessment;
3. High-quality and continuous teacher and staff professional development
and training;
52
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
4. Measurable goals for student performance and benchmarks for meeting
those goals;
5. Support by school faculty, administrators, and staff;
6. The meaningful involvement o f parents and the local community in
planning and implementing school improvement activities;
7. High-quality external technical support and assistance from a
comprehensive school reform entity (e.g., a university or independent
research organization) with experience or expertise in school-wide
reform and improvement;
8. A plan for the evaluation o f the implementation of school reforms and
the student results achieved; and
9. Identification o f how other resources (Federal, State, local, and private)
available to the school will be utilized and coordinated to support and
sustain the school reform effort.
The programs that schools may have selected for CSRD funding were varied
in their approach to reform. Programs put forth by Comer -Comer School
Development Program, Sizer -Coalition o f Effective Schools (Sizer, 1992) and
Levin-Accelerated Schools Project (Hopfenberg & Levin, 1993) were process
models of school reform. Comer, Sizer and Levin’s models applied general
principals of reform but allowed educators flexibility to implement the principles
into practice in an effort to meet individual school needs. Additionally, these
programs provided consultants who work with school staffs as “critical friends.”
53
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Slavin’s Success fo r All program introduced change through a scripted program with
specific materials, teacher’s manuals, and training procedures (Slavin, 2001).
Research conducted by 52 individual researchers, in dozens o f studies, in districts
across the United States and in five foreign countries, showed Success fo r All was
effective in raising student achievement (Slavin, 2001). Although the evidence has
grown that comprehensive school reform improves student achievement, there is
much evidence that the effects of these designs are uneven and yield a variety of
results (Odden, 2000).
One issue o f CSRD that has not received much attention is the costs
associated with reform (Odden, 2000). The cost associated with CSRD models
might appear to be prohibitive for many schools. In order to implement a CSRD
design in a school the costs appeared to run between $114,000 to over a million
dollars depending on the model and the number o f years o f components a school
might engage in the program. It also appears that schools that receive federal
funding for Title I, bilingual education programs and programs for students with
learning could sufficiently fund a CSRD model. Evidence suggests that although
more complicated, schools without federal funding could still engage in
comprehensive school reform. The issue here is, would it be more cost efficient to
spend valuable fiscal resources on other items that might produce the same or
greater effect sizes. To accomplish this cost/benefit analysis o f comprehensive
school reform, solid data must be collected and analyzed (Odden, 2001).
54
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Kress (2003) stated the nature of CSR rested on the notion that
implementing certain practices in failing school can lead to a dramatic improvement
in the achievement o f students enrolled in them. Recent analyses o f comprehensive
school reform programs have begun to show that such a simplistic notion of school
improvement ignores what is known about program implementation in an
educational setting. Lytle (2002) noted that implementation indicated that success
or lack thereof emanate from a variety o f sources. Berends, Bodilly and Kirby
(2002) identified several factors that enhanced success or inhibited success were
found to be constructs inherent within the program designs themselves. Other
factors identified were outside of the control o f CSR program designs and providers
and were due primarily to the complex nature o f school district organizational
structures, goals and leadership priorities. Additionally, school site factors played a
significant role as well.
Lytle (2002) found that some CSR program models failed to build-in quality
system checks that led to questionable gains resulting from program adoption. In
many CSR programs, consultants played almost as large o f a role in the success or
failure in the change process as the design o f the program itself. Consultant quality
was identified as one o f the key factors in program success as defined by increased
student achievement. Some CSR providers delivered to their customers, consultants
of inconsistent quality and capacity to move school reform forward. High employee
turnover suffered by some CSR providers led to the hiring o f inexperienced
55
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
consultants who failed to gain credibility with teachers consequently lead to further
difficulties.
Researchers from the RAND Corporation have conducted a several
evaluations o f various CSR program models and have also identified a number of
barriers to large school reform conducted under NAS leadership. The barriers that
RAND identified resulted from the following factors: 1) CSR designs themselves
and designed-based assistance; 2) school selection process o f CSR; 3) school
capacity including principal leadership and reform overload; 4) school context
including student demographics and school size; and 5) district context including
level of district support, lack of crisis in leadership or budget, coherent program of
reform, resource support and the relationship between the district and school,
management and union leadership (Berends, Bodilly & Kirby, 2002). RAND
concluded that variation o f implementation o f CSR designs greatly influenced
student achievement results in math and reading. In short, if RAND’s analysis
holds, weak implementation will lead to weak impacts on student performance as
measured by tests.
Mike Schmoker (2004) argued that record regarding whole-school reform
has been disappointing. After a decade of whole-school reform Berends, Bodilly
and Kirby, (2002) concluded that “by and large, schools are not fertile ground” for
externally developed comprehensive school reform. Fullan (2003) argued that
“whole-school reform models make the mistake o f thinking that a comprehensive
external reform model will solve the coherence problem within schools. It doesn’t
56
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
work because it feeds into the dependency o f teachers and principals.” Fullan
concluded that comprehensive school reform models fail because they don’t change
the school culture and therefore they never become embedded in the school culture.
Borman et al. (2002) conducted a meta-analysis o f CSR program studies to
see if there were any designs that demonstrable evidence o f effectiveness as
measured by test scores. Borman used three criteria to determine effectiveness
including the quality of evidence from highest quality studies, the quantity of the
evidence and statistically significant and positive results from control-group studies.
Borman categorized his analysis o f 29 CSR models into four categories:
• Strongest Evidence of Effectiveness;
• Highly Promising Evidence of Effectiveness;
• Promising Evidence o f Effectiveness; and
• Greatest Need for Additional Research
Based on the criteria established, Borman found only three programs that
met all of the criteria to be included in the category o f strongest evidence of
effectiveness. The three design models were: D irect Instruction. School
Development Program and Success fo r All. Only three programs met the criteria for
highly promising evidence of effectiveness and these included: Expeditionary
Learning Outward Bound, Modern Red Schoolhouse and Roots and Wings. The
designs categorized under promising evidence o f effectiveness were: Accelerated
Schools, A m erica’ s Choice; Atlas Communities; Montessori; Paideia; and The
Leaning Network. The remaining seventeen program designs did not meet the
57
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
minimum criteria to measure program effectiveness. Borman findings suggest that
student attending schools implementing CSR programs can expected to score higher
than control groups on achievement test.
The evidence presented by Borman et al. (2002) holds promise for CSR
designs but there remain concerns about test-based accountability measures
promoted by No Child Left Behind. Berends et al. (2002) noted that the typical
outcome measures used in public school accountability systems provide a very
limited measure o f student and school performance. In fact, the use of tests as the
sole measure o f student performance will prove to be a significant hindrance to
understanding the power o f innovative reform approaches unless alternative
indicators and assessments are developed. High-stakes testing, the overwhelming
dependence on test scores and an immediate demand for increased test scores place
in jeopardy CSR designs that might have been useful in transforming schools,
teaching and raising student achievement.
Role of Data in Comprehensive School Reform
Schools are inundated with data. Schools collect tremendous amounts of
data in a variety o f areas relating directly to students including student family
background, student health and immunization, student language preferences and
proficiencies, student demographics including race and gender, student
achievement, student participating in co-curricular activities, student discipline, and
student attendance. Schools also collect significant data about teachers and staff
including teacher certification, family life, job history, demographic data, staff
58
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
attendance, professional development experiences, preferences for subjects and
grade levels to be taught and of course performance evaluations. School districts
collect significant information about schools including building history, repair and
maintenance, budgets, staffing, student achievement, and demographic information.
By themselves these data do little to shape school practices and o f course some data
collected about students, staff and schools are more valuable than others when it
comes to raising student achievement.
Data play a significant role in comprehensive school reform. Popham (2003)
asserted that policymakers at the national and state levels maintain a strong belief
that data play a central role in improving student achievement. The increased calls
for improved student achievement have pressured educators to align local district
standards with state standards, to align local curriculum with state frameworks, to
identify effective instructional practices and to examine in-class teacher developed
assessments with local district and state assessments. Schools and districts are
challenged by intensified demands for accountability in meeting the instructional
needs o f all students. Cromey (2000) believed that in order for educators to meet the
expectations o f increased student achievement, teachers and administrators must be
given the time, the tools, the training and the skills to properly use data.
Data Defined
Data can be defined in a variety of ways, but in its most basic sense data is
information. These days when people think about data in relationship to schools
they think o f student test scores on standardized tests. For the purposes o f this paper,
59
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
limiting data to student test scores on standardized test is too narrow to adequately
convey the complexity of what occurs at schools and in relationship to teaching and
learning (Bernhardt, 2000; Conzemius, 2000; Olsen, 2001). Additionally, limiting a
definition o f data to standardized test scores would remove valuable opportunities to
embark on meaningful whole-school change. Schmoker (1999) stated the value of
data in simple but clear terms, “data make the invisible visible, revealing strengths
and weaknesses that are easily concealed.” Johnson (2002) noted that low
expectations, low-level curricula and low level instructional strategies for low-
income students easily evolve into institutionalized practices. She argued for the use
o f data from high achieving students from low-income and minority backgrounds to
“kill the myth and build dissatisfaction” in an effort to inspire a commitment to
change.
In a comprehensive school reform context, understanding data is vital to
improving student achievement, closing the achievement gap and in making sure
that students receive equal access to a high quality educational experience.
Understanding a broad range of data will help educators make wise decisions,
allocate scarce resources in the most beneficial manner and provide students the
assistance they need. Kozal (1991) documented the “savage inequalities” of some
school districts where poor and minority students are trapped in deplorable schools.
Many times these schools are situated in poverty stricken communities, populated
by a diversity of students that have special needs. Tucker (2002) lamented that
as long as we are a world leader among the developed
nations in the field of income inequity and fail to take
60
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
adequate measures to protect our children from that
situation by providing them with adequate nutrition, health
care, housing and pre-school education
we should never expect to beat the educational record of other industrialized
nations. Without various types of data, without looking after the needs o f the whole
student, their heart, their mind and their body, schools and educators will continue
to fight a loosing battle. Nevertheless, without a laser-sharp focus on data,
especially data on student achievement, schools cannot identify problems, solve
problems or measure student performance against the achievement goals they
establish. Data keep students, teachers and administrators focused on results
(Schmoker, 1999).
Uses o f Data Analysis
Data utilization in CSR provide the only means and the only real evidence
for success or failure o f educational goals. Bernhardt (2000) found that using data
helped to replace hunches, guesswork, and hypotheses with facts. Using data helped
to identify root causes of problems and not just symptoms. It helped assess needs
and led to targeted resources to address those needs. Additionally, data use helped
shape and focus professional development and track the impact o f the professional
development. Analyses o f data aided school districts, schools and teachers in setting
goals and keeping track o f accomplishments. Noyce, Perda and Traver (2000)
contended that data can be used to monitor, evaluate and to revise programs and
policies. They added that data provided opportunities to reveal important patterns
and to identify gaps between policy and practice. Wade (2001) stated that data could
61
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
reveal strengths and weaknesses in students’ knowledge and skills and they could
provide meaningful guidance on how teaching practices can and should be altered.
Ogawa and Collom (2000) found policymakers and scholars envision the use
o f student performance data as a school’s vehicle for improving student
achievement. State mandated assessments are designed to provide common data
through which schools, teachers and the public can discuss areas for school and
student improvement. They found reformers and scholars use data and performance
indicators in at least five overlapping ways: 1) to describe achievement, 2) to
advance policy agendas, 3) to serve as the basis for accountability, 4) to evaluate
policies and programs and 5) to serve as information management systems.
Categories o f Data
Bernhardt (2000) grouped various data under four primary categories. The
most important category of data in the United States are test data because they are
most often used to evaluate educators’ effectiveness (Popham, 2003). The second
category broadens the discussion o f data by including student demographic data. A
third way we could think about data is as process data or data that describes what is
occurring on school campuses to raise student achievement.. A fourth type of data
Bernhardt identifies is perceptual data on how people, namely teachers, students,
administrators, parents and the community feel or think about schools. Odden
identifies a fifth type of data that focuses on inputs and fiscal resources that schools
receive to accomplish the task of education but CSR focuses on outcomes rather
62
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
than inputs. Each of these aspects o f data is important in improving schools and
must be considered in a CSR model.
Demographic data are important in CSR because it provides data about our
student’s lives. It is also the most troubling aspects o f the data that is collected
because school have such little control over who attends what school, what needs
they have or the backgrounds from which the students come. Public schools are
required to educate the students that arrive on their doorsteps. Schools can record
and monitor demographic information such as gender, age, race, home background
and family circumstance but they cannot do much to these factors. If this is the case,
what good is demographic data in comprehensive school reform and what role can it
play in improving student achievement? The answer lies in what the schools can
actually do with demographic data if they are committed to meeting the academic
needs of all students.
One o f the most important benefits o f demographic data is in the
disaggregation o f demographic data so that we “leave no student” or no student
groups behind. Demographic data allow schools to see the gaps in student
achievement and to monitor those gaps to see if they widen or narrow over time.
Demographic data also allow school personnel to see trends overtime. Typical
trends that school personnel could monitor include changes in enrollment, changes
in the population o f students speaking diverse languages or changes in student
achievement for any particular group within a school (Bernhardt, 2000; DuFour,
2000; Olsen, 2001)
63
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Process data are important because information is revealed about how things
get done on a school campus (Bernhardt, 2000; Killion & Bellamy, 2000). Process
data are closely connected to what schools do and the manner in which things get
done. Process data include information about scheduling, grouping, and school
leadership decision making processes. Process data extend to the classroom via the
ways in which teachers decide to teach, the programs and materials they decide to
use, and the decisions they make to determine what information is relevant or
inelevant to teach. Process data look at traditions and how things get done currently
in contrast to how things were done in the past and how things might get done in the
future. In a CSR model, processes that increase student achievement should be
encouraged while processes that inhibit student achievement should be either
changed or discontinued. The important question to ask about process data is to
what extent can changing the way things get done raise student achievement?
Perceptual data help schools understand what students, teachers, parents an
community members think about school and the learning environment (Bernhardt,
2000). Teachers for example might think that their classes are decorated in a
stimulating manner while students might find the very same room distracting and an
administrator might find the rooms inappropriate. Perceptions of schools are
important because the perceptions influence behavior and behavior impacts student
learning. In a CSR model, if one group perceives a school to be disruptive and
disorderly it will most certainly affect the learning process (Conzemius, 2000;
DuFour, 2000). Examining perceptual data will allow students and other school
64
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
personnel to seek ways to improve the learning environment so that more learning
can take place.
Traditional Forms o f Student Data Available at Schools
Student learning goes to the heart o f comprehensive school reform. While
student achievement data include information about standardized tests, student
achievement data can include much more. Student achievement data include grade
point averages, daily quizzes, dropout rates, proficiency rates in language and
standards-based assessments, grade distributions, percentages of students going to
college, percentages o f student taking classes that allow them to get into universities
and the list can go on to include many more examples o f student achievement data.
Student learning data provide a wealth o f information about a school’s ability to
educate all children to their utmost potential. Student achievement data provides the
foundation upon which schools can make meaningful change in instruction and in
instructional services such as providing tutoring or special intervention strategies.
Student achievement data not only provide invaluable information about how
individual students are performing, but how groups o f students are performing.
Analyzing student performance data can lead to a revelation that certain groups o f
students are not having their instructional needs met (Olsen, 2001).
There is also an expectation by policymakers that schools will begin a
process of organizing change to improve student achievement. Schools will invest in
their teachers by providing meaningful professional development based on
identified needs as revealed by student achievement data. And, because change is
65
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
difficult and student achievement is the central mission o f schools, administrators
will begin to take their responsibility to supervise instruction seriously. In other
words, the days o f laisse-faire administration are quickly fading away.
Policymakers are interested in student achievement data for several reasons
included those listed above. They also intend to use data to hold students, teachers
and administrators accountable. Unfortunately schools have done such a poor job in
monitoring and utilizing student achievement data. Consequently there is a
perception, real or erroneous that only a small percentage o f students actually
receive a quality educational experience and as a result schools, teachers and
administrators are under pressure to change. Accountability for students, teachers,
and administrators is taking the form o f rewards and sanctions. For students,
policymakers have instituted high-stakes assessments in numerous states around the
nation. In order to move to the next grade level students are expected to learn and
achieve at a publicly identified level of performance. Teachers are expected to
teach the standards and monitor student achievement or face increased public
ridicule, poor performance evaluations, or even relocation or the possibility of
termination. Building administrators have been placed in the most difficult
circumstances o f any group mentioned thus far, as they are the ones most easily
identified with low student achievement. In many places the demands on site
administrators have grown significantly. In addition, administrators have become
the focal point for poor student achievement. If schools do not improve, site
administrators can be removed, demoted, or terminated. However, Rallis and
c
66
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
MacMullen (2000) remarked that the assumptions held by policymakers that high
standards + assessment + incentives (or consequences) = higher student
achievement rest on the faulty belief that schools have the capacity to improve if
only they had all three components. Such a simplistic view o f organizational
change, data use and the real complexities associated with teaching and learning
demonstrate how distant policymakers are from the classroom experience.
Barriers to Using Data
For much o f the history o f public education in the United States, teaching
has been an isolating endeavor for teachers, with teachers having few opportunities
to interact with each other. Rarely have teachers engaged in meaningful
conversations about what works and what doesn’t work based on common data.
Teachers have not had the time nor the incentive to establish common expectations
for student performance and this has very specifically led to problems such as grade
inflation, varied quality of instruction and in one sense discriminatory practices
based on low expectations for certain student populations.
Khanna, Trousdale, Penuel, and Kell (1999) found significant barriers to
using data exist at all levels within the school system. Teachers and administrator
often cited finding time to collect and analyze data as a major barrier. They also
reported a lack o f training and useful professional development that would enable
them to better read the data. Closely related to the same issue, school staff often
complained that the reports generated by central office research staff or data experts
were constructed in a technical manner and contained assessment specific language
67
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and terminology with which they were not familiar and thus did not understand the
relevance o f the terms used. School administrators and teachers also reported that
large scale assessment data returned to them far too long after the assessments were
administered. In fact, assessment data often arrived after students had left for
summer vacation and teachers were soon to be faced with a new group of incoming
students. Choppin (2002) reported that the lack o f school-level technical capacity
remained an enduring problem. Complicated software packages, antiquated
computer technology, inadequately designed data bases and a lack of integrated data
systems also complicated the desire to use data.
Rallis and MacMullen argued that policymakers share a responsibility in the
accountability movement. In various states across the country they found legislators
intent on using student performance data to improve schools within their jurisdiction
permits. Based on the best data available to them, state policymakers initiated
legislation to remedy problems exposed through data analysis. As a result in some
cases they have approved new instructional materials, created new instructional
frameworks or provided more money for intervention programs for
underperforming students. If schools were performing well they might reduce state
regulations in an effort to encourage innovation and further achievement growth. If
certain schools were consistently performing poorly, they might decide to take over
the school or close it down.
State legislators remain responsible for passing state budgets which directly
impact schools. Odden (2003) and Picus (2004) found that the accountability
68
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
movement has shifted its attention from finance equity focusing on fiscal “inputs” to
funding adequacy based on educational “outputs” or simply stated as student
achievement. Picus states with absolute resolvement “if states are holding districts
and schools accountable for what students should know and be able to do;” then he
states “must provide the resources to enable schools and districts to meet the state-
set-standards.” While there has been a long standing debate whether money matters
in improving school outcomes, the request or demand for new funding levels must
ensure that new and often complex funding models become comprehensible to
policymakers and the tax-paying general public. A failure to do so would jeopardize
the resources identified that lead to improved student performance.
Promising Model for Enhancing the Use o f Data
Student performance data can be used as a powerful tool to drive change in
schools. Beyond monitoring student performance, scholars, politicians, businessmen
and citizens want schools to use data to make deep and profound changes to
improve instruction and student achievement. At the closest point o f contact to the
student, student performance data can promote teachers sharing “best” instructional
practices. In some schools, teachers are supported and encouraged to share ideas and
work together, collaboratively with their fellow teachers. They are engaged in
developing new lessons and curricular change based on standards (Reeves, 2001;
Sizer, 1992).
69
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Rallis and MacMullen (2002) advocated using six activities that lead to an
inquiry model schools could use. They argued that successful schools are inquiry-
minded schools. In inquiry minded schools data play a central role.
• Establish outcomes for which we accept responsibility - define
satisfactory outcomes and establish standards forjudging the quality
outcomes;
• Identify important questions concerning student learning - articulate key
questions about learning in the school, based on experience, concerns, or
data;
• Collect and manage data derived from the assessment o f performance
(e.g., test scores and writing samples) - seek multiple measures and
obtain data that are generated at the school;
• Conduct mindful analyses o f the data in light o f the desired outcomes,
and interpret information in light o f the school's purposes - transform
raw data into information that can be used to improve instructional
practice
• Take action based on knowledge - face the results of the inquiry process
and make changes where needed;
• Assess the effects of actions - begin a new cycle o f inquiry.
Khanna, Trousdale, Penuel, and Kell (1999) advocated a five-step data
planning model to help administrators use data better. Their plan included the
following steps:
70
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• Analyze - examine trends in test data and other sources of data
• Plan - articulate the particular reform strategies to be used in the school
• Implement - identify what is to be done, when and by whom
• Monitor - identify ways to document whether what was planned was
actually implemented
• Evaluate - test assumptions and theories o f reform against the evidence.
After two years o f implementation nine o f 17 schools demonstrated “exemplary
evidence” in meeting the objective to demonstrate the thoughtful use of data in
developing district and site plans.
Cromey (2000) summarized how different types o f assessments can be tied
to the two variables that influence the utility o f data use. The two variables
identified by Cromey were “rate of feedback” referring to time frequency
(infrequent vs. frequent) and “type of feedback” referring to the specificity of
information. Cromey suggests that the range o f various assessments and the type of
feedback returned to schools and educators provide multiple opportunities to learn
from the data as well as creates numerous challenges that could eventually
overwhelm a school. Figure 1 illustrates typical range o f assessments, the feedback
rate and evaluative questions various assessments might address.
Advocates for using data suggest that school faculties interested in using
data to raise student achievement start with the data for which they already have
access. In addition to the demographic data schools collect, they should also utilize
the data provided from the various assessment data identified by Cromey. Other
71
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
recommendations for using data include understanding and capitalizing on where
the data converge, diverge and intersect. Bernhardt (2000) suggests that school first
try to utilize data by crossing two categories o f data (e.g. demographic and student
test data) to get a more representative understanding o f the school and student
Figure 1: The Richness and Complexity o f Student Assessment Data
u
< 4
£ >
4 >
0 >
0 4
Annually to
students
in selected grades
Annually to students
in selected grades
As needed - usually
once a year
National/
International Assessments
Are students performing
nnt.im.nl.lv?
Large-Scale Assessments
Are students meeting the
state standards?
Diagnostic Assessments
What are students ‘ cognitive
strengths and needs?
Once/
Approx. 6
weeks
Once/
Curriculum unit
Weekly
Daily
Student Report Cards
How are students performing in general
and in specific classes?
Performance Assessments
Can students apply what they
have learned?
Classroom Curricular Unit Tests and Quizzes
D id students learn what was taught?
Formative Assessments
Are students learning? ,
O
< T >
3
c t >
i - t
3
( Z J
" 3
n
r >
3
n
©
3
3
a
©
3
00
©
o
3
o
Figure from Cromey (2000)
72
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
performance. As school faculties or data coordinators developed a deeper
understanding o f how data can be utilize, she suggested that they move from two
intersecting categories, to three intersecting categories o f data (e.g., demographic
data, student test data and process data), to eventually intersecting four categories.
An analysis o f Cromey’s model suggests that schools could focus primarily on
various student assessment data.
Whatever the case may be, the potential for utilizing data to raise student
achievement is ripe for experimentation. N ew research is needed to identify multiple
models o f data use that schools and school districts may adopt based on evidence of
effectiveness and efficiency and the extent that school have the capacity to utilize
data.
California State Expectations for Students
California has a history of collecting student performance data (Carroll et al.,
2005). Until 2000, individual school districts could select which assessments to use
or could decide not to administer assessments. Data, if it was used, was used to
inform teachers o f the general skill strengths and weaknesses o f students. There
were few external incentives for teachers to examine data and few consequences if
they chose not to. Unless teachers searched through a student’s cumulative record,
most o f the data remained unexamined.
California schools are now becoming data driven schools as a result of new
accountability measures. Efforts to improve education have been largely assumed
by the state legislature, the governor or the State Board o f Education. Each policy-
73
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
making body at the state level expects schools to improve student achievement and
to put into place structures to facilitate improvement. While California has a long
history o f providing curriculum materials, assessments and measures of school
success, nothing in recent history has had the impact o f the current accountability
system for California’s schools. Accountability came to California, like it did to
most states as a result o f federal requirements for Title I o f the Improving America ’
School Act, 1994.
Standardized Testing and Reporting system fSTARl
At the center o f California’s accountability plan is the Standardized Testing
and Reporting system (STAR). The STAR program was authorized by Senate Bill
(SB 376) in October 1997. The STAR program is composed of four tests
administered to students in grades 2 through 11 (CDE 2004). Each test measures
student achievement in one o f two ways either through a norm-referenced test or a
criterion referenced test. The norm-referenced tests are the California Achievement
Tests, Sixth Edition Survey (CAT-6) that replaced the Stanford Achievement Test,
Ninth Edition, (SAT-9) and the Spanish Assessment o f Basic Education, Second
Edition (SABE-2). The criteria referenced tests are known as the California
Standards Tests (CSTs) and the California Alternative Performance Assessment
(CAPA).
The original single performance assessment indicators o f the STAR were the
test results o f the Stanford Achievement Test-9, (SAT-9). As required by statute, the
California State Board of Education designated a norm-referenced assessment test,
74
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the Stanford 9, in November 1997. This test was first administered statewide in the
of Spring 1998. The Stanford 9 is a multiple-choice test that allows comparisons to
be made to a national sample of students across the nation during the norm year.
The SAT-9 was an off-the-shelf, pre-constructed, commercial test. The questions
asked o f students on SAT-9 were not standards-based and may or may not have
reflected what students in California were required to learn. The SAT-9 tested
students in grades 2 through 11 at the elementary and middle school levels in
reading, language, spelling and mathematics. At the high school level, history-social
science and science were added to the tests.
In 2000, California provided a series o f exams to augment the SAT-9 in
English and mathematics. The augmented assessment items were added as a result
o f the criticism raised by advocates o f standards-based instruction. Critics argued
that students were being tested on content not explicitly identified by the academic
content standards therefore resulting in invalid measurements o f student learning.
The newly added augmented test items were developed to reflect the content
identified in the California academic content standards. In 2001, California added
similar augmented exams in science and history at the high school level. These
augmented tests would become the foundation for the California Standards Tests
(CSTs)
In 2002, the State Board of Education decided to replace the SAT-9 with
CAT-6 beginning in 2003. The CAT-6 is similar to the SAT-9 but it is owned and
published by CTB/McGraw-Hill. During the transition period the San Diego City
75
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Schools warned parents that “the CAT-6 is a different test than the SAT-9. Results
for the CAT-6 should not be compared with previous years' results on the SAT-9.”
Since 2003, the state has lowered the weight o f the CAT-6 to 20% where the norm-
reference test was once 100% of how the state measured school performance.
Student test scores for the CAT-6 are reported to schools according to the percent of
students performing at or above national percentile rankings. Schools and parents
are also provided information about each individual student scores on each tested
area on the CAT-6.
The SABE-2 is taken by Spanish speaking English-language learning
students who have been enrolled in California schools for less than 12 months.
Students who take the SABE-2 also take the CAT-6 and the CSTs.
The CSTs and CAPA are standards-based assessments constructed to
measure student progress towards meeting proficiency on the state adopted content
area standards developed in English language arts, mathematics, science and social
science. The standards describe what students in California should know and be able
to do in each grade level. The CSTs for English-language arts and mathematics
became part o f the STAR program in 1999. The CSTs for science and social science
became part o f the STAR program in 2001. Additionally, the writing tests for grades
4 and 7 were added to the STAR in 2001. The last CST test added to the STAR
program was the science assessment added to the grade 5 battery in 2004. The
CAPA is administered to students with significant cognitive disabilities who are
unable to take the CSTs and CAT-6 even with accommodations. The CAPA was
76
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
added to the STAR in 2003. The CAPA only assesses English-language arts and
mathematics. Test scores on the CSTs and CAPA are reported under five levels of
proficiency: Advanced, Proficient, Basic, Below Basic, or Far Below Basic. Each
level of proficiency is represented by a scaled score between a high and low
number. Table 3 shows the components o f the STAR testing program in detail.
Table 3: STAR Testing Program
Grade Number of CSTs
at each grade level
Subjects tested by CSTs
2 Two English-language arts
3
4
6
7
Mathematics
5 Three English-language arts
Mathematics
Science
8 Three English-language arts
Mathematics (end-of-course tests)
History-Social Science
9 Three English-language arts
Mathematics (end-of-course tests)
Science (end-of-course tests)
10 Four English-language arts
11 Mathematics (end-of-course tests)
Science (end-of-course tests)
History-Social Science (end-of-course tests)
The Academic Performance Index (APB
The Academic Performance Index (API) was established in 1999 with the
California Public Schools Accountability Act (PSAA). The PSAA also established
the Intermediate Intervention/Underperforming Schools Program (II/USP) as an
77
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
assistance program of schools that do not meet expectations. The third element o f
the PSAA was the High Achieving/Improving Schools Program (HA/ISP) that
provided rewards for schools that exceed expectations. The purpose of the API is to
measure the academic performance and growth o f schools.
The API is a rating scale that ranges from a score of 200-1000, with 1000
being the highest score possible. Schools in California are placed on the scale
according to their school’s performance level. The target goal for all schools is a
score of 800. A school’s API is a single-number index score based on a compilation
o f CST and CAT/6 scores for elementary schools and the CAHESS scores have
been added for high schools. The reporting criteria for the API consists o f two
components, the score and information reported in the base year and growth
information reported each year. Each year in January, beginning in 1999, schools
were given an API base score. It is against the base score that a school’s growth is
measured and reported. Schools are required to meet their yearly growth targets on
the API or they begin the processes that lead to consequences for schools,
administrators and teachers if their test score continue to fall below expected targets.
The scores o f all of the students at a given school are combined into a single
weighted number to represent the average performance o f the school. Schools were
then ranked statewide by the type of school, elementary, middle or high school.
Schools were then separated and then banded together by criteria which included
school and student demographics and background characteristics. Schools are then
compared to schools o f similar student composition and circumstance and were
78
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
again rated within each band or cohort. The results o f the CAT-6 are posted on the
Internet for public review in an effort to communicate with schools, parents, and
citizens about the current average performance of student achievement at individual
schools. Parents are provided information by their school district about how well
their individual child scored on the CAT-6, the CSTs, CAPA, SABE-2 and for
parents of high school students the CAHSEE.
Figure 2: 2003-04 API Cycle for Elementary, Middle and High Schools
2003-04 API cycle
Elernentary and Middle Schools
79
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2003-04 API cycle
High Schools
C A T O !
L a n g u a g e
C A H S 6E
W iSittiiftf'fistJca; &%
C A N S iiS a s lis h '
L a n g u a g e A m :
10%
- S f e s T f in p M i '• /
Language Art*: 32%
CATdS-Scieinc«i3%
CAT-6 Mathematics-
3%
O S T ~ S c * 6 n « $ .S %
'O'-M C-^y5^i‘ jiJSt ZXihC'fri£i
California High School Exit Exam (CAHSEE)
Also in 2001, the first cohort of ninth graders took the newly developed
High School Exit exam in English and mathematics. California High School Exit
Exam (CAHSEE) was also a criterion referenced test is directly related to the
California Academic Content Standards. The CAHSEE was originally to hold
students beginning with the class or 2004 accountable for passing the CAHSEE as a
condition for earning a high school diploma. In July 2003 the State Board of
Education decided to postpone the passing the CAHSEE for the classes of 2004 and
2005 and is now requiring the class o f 2006 to pass the tests in English-language
arts and mathematics as a condition to receive a diploma (EdSource, 2003).
CAHASEE scores are reported as pass/fail.
80
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Information Provided to Schools
The California Department o f Education encourages parents, student and
concerned citizens to consider a comprehensive approach to assessing the academic
progress of students. Tests, teacher observations, and other measures usually
assigned by teachers as in-class assignments as well as school, district and state
assessments should all be carefully considered when analyzing STAR results.
Although this is the official stance o f the Department o f Education and the
California School Board of Education, the fact remains that only tests administered
under the STAR program get factored into a school’s API report and the same goes
for the federal No Child Left Behind accountability goal known as adequate yearly
progress (AYP).
Each year schools are provided a School Report that includes information
about school demographics and data on student subgroups. The ethnic/racial
subgroups are categorized as: African-American not Hispanic; American Indian or
Alaska Native; Asian, Hispanic or Latino; Pacific Islander; and White not Hispanic.
Students are also identified by a subgroup known as "socio-economically
disadvantaged" and are defined as a student whose parents both have not received a
high school diploma or a student who participates in the free or reduced price lunch
program (also known as the National School Lunch Program). Schools are also
provided student information about sex (male or female). The last major category of
information pertains to students’ English language proficiency. These several
81
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
categories o f disaggregated student data allow schools to look for trends and gap in
student achievement by sub-category.
The information provided to schools resulting from the CAT-6 assessments
encouraged schools to examine the results o f their student’s performance. Schools
are provided the raw test results that will provide a breakdown o f the content
clusters in each o f the tested subject areas. It is assumed that schools will review
their data and begin to make adjustments in instruction, the purchasing of
instructional materials and in designing professional development in an effort to
improve. Schools were also encouraged to purchase new materials and align their
curriculum with state standards.
Holding Schools Accountable
According to the PSAA, API results were reported to school districts,
schools and to the general public. Schools that failed to meet state expectations were
warned that their position was unsatisfactory and were provided with resources and
assistance to improve student outcomes. Schools that were unable to raise their API
ranking faced a variety of sanctions including the removal of administrators,
teachers, allow students to attend any public school where there is space available,
allow parents to open a charter school at the existing school site, allow a college or
university or other appropriate institution to assume the management the school or
the school may be closed (California Education Code 52055.5).
82
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Weaknesses in California Accountability System
Stecher (EdSource, 2003a) argued that California’s accountability system
needs to be more robust. One of the primary missing links is a unique student
identifier that would allow educators to monitor individual student progress over
time. The current API system according to Stecher does not accurately portray
student achievement at schools because the students from one year maybe very
different from the student the following year. Phil Daro, director o f the Public
Forum on School Accountability, argued that the current API system provides
adequate information to answer global type questions but he continued and stated
that the data would not answer the question, “How is Johnny doing?” What
California lacks is a means to measure value-added growth for individual students.
Florida on the other hand has that capability. Daro also added that California’s once-
a-year assessment system was more like an annual checkup that really requires local
schools and teachers to figure out what is working and not working (EdSource,
2003a).
Research o f Data in Schools
The literature on data use in the schools is varied and covers a wide range of
topics. Within the last several years there has been a growing trend of using data to
improve student achievement (Carr & Harris, 2001; Schmoker, 2000). There are two
fundamental beliefs inherent in the accountability movement to improve student
achievement, a belief that educators can change and improve teaching and learning
and second, a belief that effective use o f data can support that effort (Keeney, 1998).
83
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data can be used at a variety o f levels, national, state, local and the classroom in
order to improve learning. Yet, districts, and schools underutilize the power o f data
and miss valuable opportunities to make change (Noyce, Perda, & Traver, 2000).
Englert, Fries, Goodwin Martin-Glenn (2003) found that superintendents
used data in their own practices and are encouraging staff to use data as well. The
study indicated that superintendents were more frequently using data to inform
parents and the community about testing data. The second highest response
indicated that superintendents used data to monitor schools and apply rewards and
sanctions. The most frequently cited sources were dropout rates, parent-feedback,
district created assessments and other standardized assessments. Although nearly
three-quarters o f the superintendents stated that there was a district policy or
expectation that teachers would use data, most also believed that schools were slow
to develop a culture o f using data to improve classroom practice.
It would be both unfair and inaccurate to say that teachers do not use data
related to student achievement. In fact, teachers use data on a regular basis in
evaluating student work and in reflecting on what needs to be done for the next
lesson. Teachers use data to change their own curriculum and instructional
practices. They share exemplary work o f students with other students to provide
models o f assignments. Teachers use student performance data to provide
remediation for students not meeting standards. Student data can also be very
powerful when trends start to emerge in instructional settings (Schmoker, 2000).
84
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Mason (2002) reported that in order to be effective, data must become an
active part o f school planning and improvement processes. Data use must become
infused and accepted into the school culture and organization and staff members
must develop the analytical capacity to understand and to apply data strategically. In
a study of six Milwaukee public schools, Mason identified six challenges school
need to address as they build their capacity to use data for decision making: 1)
cultivate the desire to transform data into knowledge; 2) focus on a process for
planned data use; 3) commit to the acquisition and creation o f data; 4) organize data
management; 5) develop analytical capacity; and, 6) strategically apply information
and results.
On a school-wide level data utilization is used to make informed decisions to
optimize curriculum and instruction and to focus resources. Some schools use data
to create action teams to work on and address major issues (Conzemius, 2000).
Data driven school improvement utilizes both qualitative and quantitative data to
make substantive changes in the classroom and in the school. Greater school change
occurs when multiple data categories “cross.” An example o f crossing occurs when
variables intersect during questioning and analysis (Bernhardt, 2000) Crossing more
categories o f data can provide great insight in order to gain a fuller picture of the
school setting and its challenges.
Data action planning teams examine a wide range o f data in order to provide
useful information to the entire school staff and community (Carr & Harris, 2001).
They focus on student performance in relation to standards; they examine testing
85
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
results from national, state and local assessments; they place student performance
data in the context o f other pertinent information; they set priorities for improving
student performance and develop a process to evaluate data on a continuous basis
(Carr & Harris, 2001). In school settings where there is not focused collaboration,
data are counter productive (Conzemius, 2000).
Reeves (2000) in his study o f schools that are characterized by over 90
percent poverty, over 90 percent minorities, and yet over 90 percent achieving at
high proficiency levels found that the faculty and administration used data in
significantly different ways. Among the most notable data used, Reeves noted that
these schools devoted time for teacher collaboration that focuses on student work
and proficiency. He also noted that parents were provided significantly more
frequent feedback than a typical report card by teachers. He also noted that teachers
created data through action research and made mid-course corrections based on the
created data. Another common characteristic o f the 90/90/90 schools was that the
entire system included an intensive focus on student data from multiple sources, and
— teachers compared students to themselves rather than to other student groups.
Finally, Reeves noted that these schools consistently used common assessments (as
opposed to tests) and quickly received feedback to help improve performance.
86
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 3
RESEARCH METHODOLOGY
Introduction
The purpose o f this case study was to investigate the design,
implementation, and adequacy o f a district’s model o f data use to raise student
achievement. Given the rise o f high-stakes accountability measures in California, it
was critical to examine how school districts, especially diverse, urban school district
in southern California, had begun to conceptualize and utilize various forms of data
to raise student achievement. As o f 2004, very few school districts in California
have met the state’s expectations for excellence as described by the Academic
Performance Index. Some districts were close but most would have to raise student
achievement significantly. Consequently, school districts across the state would
have to develop a strategy to raise student achievement because the consequences
for doing nothing were too great to leave to chance or circumstance. It seemed
logical that if a school district and its schools were to develop local school
improvement plans, they would need to identify, collect and analyze various data.
This individual case study and analysis were designed to describe one school
district’s effort under the conditions and expectations generated by an accountability
system imposed on schools by California’s policymakers and legislators. It was
hoped that this study would add to the growing body o f evidence o f how schools use
data to raise student achievement and to provide some insight about practices
educators use in their effort to close the achievement gap.
87
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Research Questions
Three research questions guided this study and framed the resulting analysis:
• What is the district design for using data regarding student performance,
and how is that design linked to the current and the emerging state
context for assessing student performance?
• To what extent has the district design actually been implemented at the
district, school and individual teacher level?
• To what extent is the district design a good one?
Research Design
The study was conducted using a qualitative case-study format. Qualitative
research was appropriate for this study because qualitative research focuses on the
collection and the subjective interpretation o f data collected rather than on the
testing of theory (Harris, 1998). Schools offer complex settings where human
interaction and social structures, mores and values play a significant role and
capturing such a phenomena was best accomplished using a qualitative research
approach that provided the opportunity for a rich description o f the case studied.
This case study was conducted for the reason it offered a critical analysis of
events in a natural contextual setting (Gall, Borg, & Gall, 1996). The case study
consisted of staff interviews, teacher questionnaires, an innovation configuration
and researcher ratings. The purpose of this case study was to investigate the design,
the implementation, and the adequacy of a district’s model of data use to raise
student achievement. The findings o f this study are not generalizable and may
88
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
reflect unique characteristics and attitudes of the respondents that were caused by
unknown forces or the dynamic nature o f any school setting.
Sample and Population
The sample size was small and purposive. This case study examined one
high school from a mid-sized southern California school district. The school was
selected in an effort to meet four criteria: 1) the school had a diverse student
population; 2) the school had an emerging state context for assessing student
performance; 3) the school utilized multiple measures for assessment; and 4) the
school was believed to have a design for using data regarding student performance.
The school district chosen was agreed upon by the researcher and his chair. The
researcher initiated contact with the Assistant Superintendent and secured
administrative level permission to observe and conduct research in the selected
school district. To maintain the school district’s and school’s anonymity, the school
district will be referred to as Valley Unified School District and the school as River
Valley High School.
The Valley Unified School District was chosen because it had a strong
reputation for academic success and for graduating students who would attend some
o f the nation’s most prestigious universities. River Valley High School was selected
for this study because o f its outstanding reputation among urban high schools and
because the school had a history o f self-study and data use. The River Valley High
School’s administration welcomed the opportunity to participate in the study and
provided significant support to ensure its successful completion.
89
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Setting
The Valley Unified School District provides its students with rich
educational opportunities. Situated in the urban sprawl o f Los Angeles County,
Valley Unified School District (VUSD) educates students from diverse ethnic,
religious and socioeconomic backgrounds representative o f the larger Los Angeles
community. Most schools within VUSD had a reputation as being some o f the best
schools in the county and never had to worry about the community’s perception of
the quality o f education provided to neighborhood youth.
According to data collected through the California Basic Educational Data
System (CBEDS), Valley Unified School District is composed o f 16 schools. Ten of
the schools are elementary with students in grades K-5. Two schools are middle
schools with students in grades 6-8. Two schools are comprehensive high schools
with students in grades 9-12. There is one alternative school and one continuation
school. The total student population served by VUSD during the 2003-2004 school
year was approximately 12,842. These schools were staffed by approximately 637
full-time equivalent teachers, 71 administrators and 62 other certificated personnel.
The district is fortunate to have 87.3% if its teachers fully credentialed by the state
o f California and only 10.1% on emergence permit, 1.4% university intern, district
intern or pre-intem status and 1.1% on waiver.
The Valley Unified School District enjoys great community support
resulting from alumni that continues to monitor the school system from which they
graduated and o f which some will enroll their own children. The community has
90
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
established an educational foundation in an effort to provide supplemental support
and enrichment opportunities for the school district’s youth. The community
considers the school district a vital institution in the city and shares in its success
and becomes concerned with its challenges. Community members are engaged in
the educational and governance issues o f the school district and they play an active
roll in the public policy issues confronting the school district and its schools.
The superintendent was hired three years ago to address a number of issues
confronting the school district. He was hired as a result o f the school board’s
decision to conduct a national search for its next superintendent. The
superintendent’s strong leadership and vision reflects the VUSD Board of
Education’s desire to provide each child with a quality o f education that will enable
its graduates to pursue their life goals. The vision statement o f the Valley Unified
School District states:
As a community o f learners the Valley Unified School District
works together in a nurturing environment to help students be
visionary, versatile thinkers; resourceful, life-long learners;
effective, multi-lingual communicators and global citizens. We
are a richly varied community that values the contributions of all
its members. We exist to prepare all students in their pursuit of
academic achievement and personal health and to support and
encourage them in their development o f intellectual, artistic,
technological, physical and social expression.
For the last four years the school district and the high school have been in
transition. With the hiring of the current superintendent, VUSD experienced a
complete change in administrative leadership including all three assistant
superintendents and a number of high level central office staff positions. The central
91
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
office has been restructured to better meet the instructional support needs o f the
schools within the district and to better address the philosophy o f the superintendent
and his goals. Several o f the new administrative leaders were hired from outside the
school district and state.
Valley Unified School District has always been concerned with educational
excellence and has worked hard toward that goal. As with all school districts in
California, VUSD has had to become increasingly cognizant of how well all o f its
students perform on state mandated assessments. The superintendent frequently
speaks out against the overwhelming emphasis the press and other media place on
the STAR system noting that the assessments of student learning measured by the
state are limited and serve minimal value. Since the arrival o f the current
superintendent the school district has increasingly used data to make decisions and
to measure student achievement. There is an emerging culture o f data use within the
district offices that has purposely been exported to all local school sites.
River Valley High School by most standards is considered a medium sized
high school with an enrollment o f around 3,450 students. The student body is
composed o f approximately 50% White (not Hispanic), 31% Hispanic or Latino,
11.1% African-American, 7.0% Asian, 0.5% Filipino, 0.3% American Indian or
Alaska Native, and 0.1% Pacific Islander students. According to the California
Department o f Education’s language census, River Valley High had approximately
8.6% or 298 students who were English learners and approximately 700 or 20.4% of
students received free or reduced priced meals. In comparison to the student
92
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
population attending River Valley High, the state had a significantly larger
proportion o f English learners at 25.4% and a significantly higher proportion of
students who received free or reduced priced meals. River Valley High has no
students participating in Compensatory education programs whereas nearly 50% of
the students in state o f California participated in such programs.
Table 4:
S tu d en ts by Ethnicity 2003-04
Los
River Valley High
School
Valley Unified
School District n
Angeles
County Total California Schools
Ethnicity Enroll
ment
Percent of
Total
Enroll
ment
Percent
of Total s
Percent
of Total Enrollment
Percent of
Total
American
Indian
7 0.2%
34 0.3%
0.3%
52,706 0.8%
Asian 247 7.2% 745 5.8% 7.6% 504,534 8.0%
Pacific
Islander
4 0.1%
20 0.2%
0.5%
39,744 0.6%
Filipino 10 0 3% 44 0.3% 2.1% 160,400 2.5%
Hispanic 1,053 30.5% 3,505 27.3% 61.3% 2,897,806 46.0%
African
American
398 11.5%
1,068 8.3%
10.5%
510,691 8.1%
White 1,730 50.2 7,423 57.8% 17.0% 2,046,285 32.5%
Multiple/No
Response
0 0 0%
3 0.0%
0.7%
86,247 1.4%
Total 3,449 100% 12,842 100% 100% 100% 100%
River Valley High (RVH) has also undergone a radical transformation. The
principal and all o f the assistant principals who were there three years ago are gone
and a whole new leadership structure has emerged. The principal now approaching
the beginning o f her third year at RVH is a longtime and well established, respected
instructional leader within the district. She has received her doctorate from the
93
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
University of Southern California and has been a guest lecture at Harvard
University’s Principal Leadership Academy. The faculty has had significant
turnover during this period but enough veteran teachers remain to provide an
institutional memory. A number o f faculty members are well known in local, state
and national education settings and frequently write editorials for local newspapers
and education journals. The school itself has been broken up into 6 smaller schools
each with their own house administrator, faculty and students. The enormous
changes that have transpired are clearly evident.
The deleterious effects of the somewhat chaotic state o f affairs in the school
district and school became public during the 2000-2001 academic year. In 2000 the
API base score was 717, a score considered very good and not too far off from the
state’s API goal of 800 or above. In 2001 the API base score dropped to 712, a
negative growth of 5 base points. In 2002 the declined continued and the school’s
initial “state” generated report on the API base score fell to 705. The two-year
decline in the base API began to signal a trend at RVH and a new principal was
hired at the end o f that year. While the API base had fallen nearly 12 points in three
years it was still considered respectable by most accounts and in comparison to most
neighboring high schools in neighboring school districts. A revised API report for
the 2002 year actually indicated an additional decline to a score o f 694, an aggregate
decline of 23 points. In 2002-2003, the school experienced a dismal one point
growth. The declining scores concerned nearly everyone at RVH, except those that
94
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
viewed the state STAR system as an invalid measure o f a school’s performance and
student achievement.
Traditionally, student success at RVH is a mixed story. The long history of
self-study led to the revelation that RVH was providing two types o f education to its
students, one primarily for White and Asian students whose socioeconomic status
was high and one for African American and Latino students whose socioeconomic
status was significantly less high or even low. One was designed for students on a
rigorous college preparatory track experience and the other one designed for
students going somewhere else. The achievement gap condition between the two
groups o f White and Asian students and African American and Latino students had
become known among the schools’ faculty as the “two schools phenomena” long
before the state required disaggregated data on student achievement. The
disaggregated data from the 2000 API report showed that the education system
allowed for a colossal range o f student performance between Asian and White
students with API scores o f 812 and 833, respectively, and African American,
Latino and Socially Disadvantaged students with API scores o f 546, 560 and 545,
respectively. The achievement gap between the highest performing students and
lowest performing students as measured by the 2000 API was 287 points.
The 2003-2004 API report, the most recent report available, indicated a
significant academic turn-around. RVH’s base API growth climbed 20 points from
695 to 715, a level similar to the 2000 API report. Moreover, significant gains were
realized in the three lowest performing subgroups. African American student
95
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
achievement grew 22 base points from 570 in 2003 to 592 in 2004. Latino student
achievement for the same grew 44 points from 565 to 609. Socio-economically
Disadvantaged student achievement grew from 556 in 2003 to 603 in 2004, a total
o f 47 points. California’s STAR accountability measures showed a large reduction
in the achievement gap at River Valley High School between Asian students, the
highest performing subgroup with an API o f 824 and African American students,
the lowest performing subgroup with an API o f 592. The total gap between the high
and the low subgroup scores resulted in a 5 5-point reduction over the API results of
2000. The gap between Asian students and Socio-economically Disadvantaged
students decreased by 58 points using the 2000 API results and 2004 API results.
Although there remains over a 230-point gap between the two most distant student
subgroups, RVH has made significant progress in reducing the achievement gap.
Instrumentation
The study was conducted using a qualitative case-study format. The research
design was collaboratively developed by 15 doctoral students participating in a
parallel dissertation study group facilitated by Dr. David D. Marsh, Professor and
Associate Dean for Academic Affairs, Rossier School o f Education, University of
Southern California. This researcher fully participated in the following described
activities as one member of the parallel dissertation team of doctoral students and as
a member of a small group “high performance” team charged with specific duties
and tasks. As a large group, these students discussed in detail the emerging
accountability movement sweeping across the United States but focused more
96
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
specifically on the emerging context of accountability in California. The dissertation
group mapped a series o f issues related to California’s accountability movement and
decided to conduct research about how school districts were utilizing data to raise
student achievement.
During the summer o f 2001, the parallel dissertation study group met in
teams and developed a pool o f potential research questions. Each team presented
their questions to the larger study group explaining what the questions were
attempting to find. After all o f the groups had presented, the students sorted and
categorized the questions and framed three new questions around large issues that
were judged worthy to research and could hold promise for future research and local
district policy development. With the case study questions identified, each of the
three small group teams met to craft a conceptual framework for one of the
questions under investigation. After each team completed its task o f developing a
conceptual framework they reported back to the larger group and obtained
constructive feedback, analysis, and specific suggestions for improving the final
conceptual framework model. A similar process was followed to develop the data
collection tools linked to each conceptual framework.
The dissertation team worked collaboratively, developed three conceptual
frameworks, and designed a set o f instruments and interview questions to
investigate each o f the three research questions:
97
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1. What is the district design for using data regarding student performance,
and how is that design linked to the current and the emerging state
context for assessing student performance?
2. To what extent has the district design actually been implemented at the
district, school and individual teacher level?
3. To what extent is the district design a good one?
A case study guide provided structure and organization to the data collection
process. The first research question asked, “What is the district design for using data
regarding student performance, and how is that design linked to the current and the
emerging state context for assessing student performance?” Conceptual Framework
A was constructed to gain to a deep understanding o f Valley Unified School
District’s design for using data. The conceptual framework provided this researcher
several methodologies to understand the district’s design in light o f the current and
emerging context for assessing student performance as implemented by the state of
California. Conceptual Framework A (CF-A) incorporated staff interviews, informal
observation, artifact analysis, and various forms o f quantitative data as a means for
answering research question one.
The tools California used to measure student achievement provided the
context by which Valley Unified School District’s design would be compared.
California had a test driven accountability system and relied solely on data resulting
from student tests to determine student achievement. California students at the high
school level were required to take a battery o f assessments including the California
98
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Achievement Test, Version 6 (CAT-6), the California Standards Tests (CSTs) in
English-language arts, and end o f course CSTs in mathematics, science and history
social science, the California High School Exit Exam (CAHSEE). Only students
with special characteristics had to take the California English Language
Development Test (CELDT), the California Alternative Performance Assessment
(CAPA), or the Spanish Assessment of Basic Education, Second Edition (SABE-2).
The information provided to schools as a result o f these test scores marginally
benefited schools because o f the nature and frequency o f the exams and the system
itself only provided snap-shots in time and were deemed by some to be inadequate.
Due to huge budget shortfalls at the state level, no foreseeable changes in the state
accountability system appeared forthcoming.
Conceptual Framework A framed five areas o f inquiry that would help shape
the answer to research question one. The five areas of inquiry that would lead to a
full understanding o f Valley Unified School District’s design were:
• student performance assessed in the context o f current and emerging
instruments;
• overview o f the elements of district design o f data use to improve student
performance;
• district decisions and rulings that support use o f district design;
• intended results o f design plans to improve student performance (district,
school, and classroom); and
• data use policy and strategy funding.
99
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Conceptual Framework B (CF-B) guided the data collection process for
answering research question two. Research question two asked, “To what extent has
the district design actually been implemented at the district, school and individual
teacher level?” The original conceptual framework constructed during the summer
of 2001 to answer research question two consisted o f several methodologies
including interviews with one district administrator, a site administrator and six
teachers from varied backgrounds and with diverse school responsibilities. The case
study guide also called for collecting data through situated interviews with six
teachers, informal observation, artifact analysis, and various forms of quantitative
data in the form o f API reports, CAT-6 scores, CST scores, CAHSEE results,
performance assessments and other available quantitative data. Originally a teacher
questionnaire and Stages of Concern questionnaire were included in CF-B. Another
tool included in CF-B was an innovation configuration to be utilized during the
post-data collection phase to help the researcher determine the extent of
implementation o f the district’s data design.
Several factors led to a revision o f CF-B. Details will be fully explained in
the Data Collection segment that follows this section. Certain difficulties led to the
elimination o f using both the Teacher Questionnaire and the Stages of Concern
Questionnaire. Both questionnaires were to be distributed faculty wide and used to
gain a deep understanding of the level of implementation o f district’s design model
at the teachers level. The school principal requested that the surveys not be
distributed to teachers as a pre-requisite for conducting the research study on her
100
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
campus. The request was honored by this researcher and additional elements were
added to augment CF-B. A modified CF-B resulted in widening the range of
interviews conducted and the inclusion of various quantitative data from a large
scale whole-school survey administered in the spring o f 2004.
This researcher’s quest for data that could provide alternative evidence to
corroborate or contradict the level of use and implementation o f the district’s data
design by teachers led to district support staff and on-site administrators. Research
identified in the literature review highlighted several barriers that inhibited the use
o f data by teachers and schools. Several o f the most prevalent excuses identified
were the schools’ inability to provide adequate support in the forms of human
resources, professional development or comprehensible data reports (Khanna,
Trousdale, Penuel, & Kell (1999). Cromey (2000) also found that some schools
lacked skilled analytically trained administrators to work with staff or they lacked
the resources to build such capacity. CF-B was altered by eliminating the two
teacher questionnaires. It was enhanced by adding interviews with two house
principals, traditionally identified as assistant principals who might have supported
teachers’ use o f data. Three central office administrators who were responsible for
producing and disseminating data reports, for providing professional development
about how to understand and use student achievement data and for setting up
technological computer-based technologies were also added to the original list of
people to be interviewed.
101
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Conceptual Framework B Degree o f Design Implementation sought data to
describe:
• implementation o f current data practices (at the district, school site, and
classroom levels)
• implementation o f emerging state data practices (at the district, school
site, and classroom levels)
• accountability for data use at district, school, and individual levels
• improving student achievement through implementation of data use
The third research question asked, “To what extent was the district design a
good one?” The previously mentioned case study guide, interviews, observations,
artifact analysis and quantitative data were also used to gather evidence. A
researcher rating form was used to help determine the adequacy o f the data design
using descriptive levels o f high, medium and low to assess data use and practice.
Data Collection
Originally data collection for this research project took place between
January and June 2002 during an extremely chaotic period in the organizational life
of the Valley Unified School District and River Valley High School. Significant
change engulfed VUSD and RVH. Administrative personnel changes, declining
achievement scores and low teacher morale at RVH characterized the period.
Securing teacher interviews was difficult as was getting a significant number of
teacher questionnaires returned. In fact, the two teacher questionnaires were
102
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
distributed to the faculty a total of four times over a period o f five months with only
38 pairs of surveys out o f 600 returned and some o f those were returned incomplete.
One of the two co-principals explained in the interview process with great detail
some of the potential reasons for the difficulty experienced with this researcher’s
attempts to data collection. Two o f the most frequently reasons cited were:
• extensive attempts by the school and district administrations to gather
teacher input through surveys o f the faculty that both preceded and
competed simultaneously with the teacher questionnaires distributed by
this researcher
• teachers expressed to the administrator a feeling o f being abused and
overloaded with meaningless surveys
This researcher was able to briefly interview the newly hired superintendent
in an effort to understand the data design the school district was using to raise
student achievement. He expressed a strong commitment to the value of using data
but felt that he had not had adequate time to develop a structure or a set of policies
within the school district to encourage the use o f data. He outlined a series of
organizational, structural and personnel changes he wanted to make in the coming
months that would lay the groundwork for a future study o f the same sort. He
remarked that this research request appeared to be “just a little premature.”
Shortly after the interview with the superintendent, the researcher decided to
postpone the study and declared the initial attempt a false start.
103
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
A fresh attempt at starting this research project began in March 2004.
Contact was made with the relatively new principal. She had been promoted from
the middle school principal to high school principal around the same time the
original study was postponed. The principal agreed to meet with the researcher to
discuss the research study and to decide whether it was a feasible project under new
circumstances. After a brief meeting it was mutually agreed that the study could
take place if both the Teacher Questionnaire and the Stages o f Concern
Questionnaire were eliminated from the process. She explained that the faculty
would not tolerate the intrusion.
The researcher met with Dr. Marsh to discuss an alternative plan to capture
the evidence that was potentially lost when the two surveys were eliminated. The
researcher suggested that several additional strategically planned interviews could
accomplish the desired result without sacrificing the quality o f the study. Gall
(1996) noted two benefits o f questionnaires, one the cost over a wide geographic
area is lower and the time required to collect the data typically much less. He also
noted however, that questionnaires cannot probe deeply into the respondent’s
opinions and feelings and that they cannot be altered once the questionnaire has
been distributed. Interviews in contrast provide the researcher an opportunity to
obtain more information, to clarify vague statements and to adapt the interview as
necessary. Interviews also make it possible to obtain information that would not
normally be revealed in other data-collection methods. Gall also noted that case
study researchers might start with one or more methods o f data collection and
104
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
gradually shift to, or add other collection methods. The goal in this particular case
was to ensure adequate triangulation o f sources and data to increase the validity of
the study’s findings.
Data collection began in mid March 2004 and continued into February 2005.
Gall notes that ending data collection in a case study results from both practical and
methodological considerations. The decision to end the data collection phase in this
study resulted from two primary considerations. The first consideration was the
necessity to terminate the study according to a planned schedule. The second
consideration resulted from the researcher’s impression that the participants had
grown tired o f the researcher’s presence or requests for additional information.
The data collection process concentrated on identifying individuals who
would be willing to share their insights about the use o f data to raise student
achievement. The superintendent was the first person interviewed in this study. He
was very willing to share 30 minutes o f his time to continue the conversation that
was started nearly two years earlier. The only record o f the interview was the notes
taken by the researcher in a pen and paper format. The interview was conducted in
the superintendent’s office in the informal area away from his business desk. The
conversation was relaxed and unhurried and the superintendent elaborated on a
number o f answers providing depth and breadth to the questions posed by the
researcher. When appropriate or to make a point crystal clear the superintendent
accessed data reports and took the time to explain what the student achievement
data had indicated and how the district staff was responding. The superintendent
105
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
spoke unguardedly about his impressions and the extent to which data was
collected, used or not used. The superintendent’s interview was extremely valuable
in setting the stage for all remaining interviews by providing the “big picture” o f
data use in Valley Unified School District. Upon completing the interview he
suggested that I speak with three other individuals who might provide additional
insight into the evolution and expanded use o f data as a tool to raise student
achievement. The superintendent suggested that I contact one o f the veteran school
board members who might provide an historical context to the study and two
relatively new central office administrators who he had hired and whose job
responsibilities revolved around data in various ways.
The second and third interviews were with the Chief Academic Officer and
the Director o f Educational Services. Both administrators had been hired within the
last six months, had excellent credentials and a wide range o f leadership
experiences. As a result o f the limited time in the school district both felt that they
could only make limited contributions to the research. Each interview took place in
a small meeting room not too far from each administrator’s desk and one followed
the other. What became readily apparent as each interview progressed was that both
women were well versed in the uses and benefits o f data to raise student
achievement. They spoke candidly about the district’s lack o f desire to use data and
described their jobs in terms of changing the institutional culture and of
demystifying the utility o f data. Both interviews lasted approximate 45 minutes.
106
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The fourth interview was conducted at a local Starbucks coffee shop. The
interviewee was one o f the two school board members recommended by the
superintendent. The school board member had spent nearly 10 years on the school
board, had children in district schools and felt very comfortable describing the
history o f the VUSD. She was also very explicit about what she liked about the
district and where she wanted the district to go. She was clearly an admirer o f the
superintendent and expressed satisfaction on the school board’s ability to hire a
superintendent with vision and a commitment to raising student achievement while
simultaneously closing the achievement gap. Much of the time was spent discussing
the history o f the achievement gap and what steps had been taken or not taken to
narrow the gap. She expressed frustration of previous, well-intentioned school board
members who gave lip service to supporting policy aimed at closing the
achievement gap but in practice did little to support the efforts. She spoke
extensively about the district’s previous hesitation to share student achievement data
with board o f education members and noted that the situation had made a 180-
degree turn. She appreciated the professional manner that the new administration
had been sharing information with the board members and attributed much of
progress to the superintendent. The interview lasted nearly two hours in length.
The fifth interview was with a house principal of one o f the small schools.
He had been part o f the school for two years, hired from another school district with
six years of administrative experience and fourteen years in education. He was a
doctoral student at USC but had not yet entered the dissertation stage of his
107
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
schooling. He was the only Hispanic staff member interviewed and some o f his
opinions appeared to reflect much more concern for African American and Latino
students. We met in his office on campus and w e were frequently interrupted. By his
secretary who passed on messages from mostly teachers but a few parents who had
called. He was extreme knowledgeable and literate in the schools computer and data
systems. As we spoke was able to generate several reports on student achievement
and manipulate the data contained on each distinct report to focus on several
different criteria. His technological skills made him the logical designee to assist
other administrators who had requests for data reports. The interview lasted
approximately one hour.
The sixth interview was a lead teacher in one o f the houses at RVH. She was
a house lead which meant she assisted other teachers and the house administrator
with various tasks. She was not considered an administrator. She was very
experienced having over 20 years of classroom experience with almost all of her
career being at RVH. We met and had the interview at a local Starbucks coffee shop
at her request. At first she was reserved in her responses but as the interviewed
progressed she became much more relaxed and her answers reflected the change.
The seventh and eighth interviews were conducted on campus in two
different teacher’s classrooms. Both were English teachers, White, female, very
articulate and very knowledgeable about the school. Both women had additional
leadership responsibilities on campus and both had students coming in and out of
their classrooms during the interview. Their interactions with students were very
108
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
friendly and they appeared to be very well liked by the students. Neither teacher had
student work on their boards but had plenty o f books displayed as one might expect
the case to be in an English class. The surroundings, the interviews, the context and
the interactions between interviewees and the researcher were friendly and
respectful. Both women had strong opinions about almost every issue discussed
about the school and the direction they felt it was headed. There was a significant
difference in attitude expressed when the interview shifted to student achievement,
while both articulated a need to maintain high standards, both required students to
have a strong work ethic and both cared deeply about their students, they held very
different beliefs about high and low performing students, minorities students and
who they felt they should be teaching. Both interviews lasted about one hour 15
minutes and both were held after school.
The ninth interview was with an early-mid career teacher social studies
teacher. He coached and had been at the school most o f his teaching career. He was
very articulate, cared deeply about his students. He answered the interview
questions concisely without much elaboration. His class lacked student work but it
was easily identifiable as a social studies classroom. The interview was the shortest
of all and occurred during lunch which meant about 25 minutes. He did hold
instructional leadership positions at the school.
The tenth interview was with a younger science teacher. He was very
enthusiastic and you could tell that he enjoyed teaching. This teacher was
specifically identified by the principal as one to interview. He was not the science
109
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
department chair but he took initiative in leading a rewrite of the biology course of
study at RVH. The interview lasted about 40 minutes and it occurred on his
planning period.
The eleventh interview was a second house principal. She was very
accommodating in granting time for an interview. The interview actually occurred
during summer school during the period when she was the only administrator on
campus. It was clearly not a good time to take her away from her job but she was
very willing to help. As with almost every interview this administrator was very
knowledgeable and had a good sense o f the school and its needs. This was her first
administrative position and she was responsible for accreditation and school
redesign. Much o f time and efforts were directed at transforming the school from
one large comprehensive high school to six small schools. She was knowledgeable
about many issues, especially about standards, assessment and the interventions
provided to students who needed special assistance. She appeared to be in her mid
thirties and took her administrative responsibilities seriously. She was very
thoughtful about her responses to the question asking for clarification if she was
unsure o f what was being asked, but her answers were very deliberate. The
interview lasted about 45 minutes.
The final interview took place at the district central offices. Besides the
superintendent, this interviewee was the only one interviewed in both the false start
study and the current study. In both cases he was not a planned interview but he
seemed to have special knowledge and a special role in the area of informational
110
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
technology. He was partially responsible for maintaining the District’s technological
needs and services in working condition, at one point he was responsible for the
district’s website and at the same time he was responsible for assisting the
superintendent in gathering data and creating reports from the data for the board of
education. He was very knowledgeable about the district’s use o f special software
used at the schools for recoding grades and student tracking and achievement
information. The following table summarizes the school personnel interviewed.
Each of the interviewees were asked specific questions relating to their job
responsibilities in an effort to gain general and specific understanding o f how the
district and school used data to raise student achievement. The main interview
questions were consistent but were adjusted as each situation warranted. As the
interview progressed some questions were asked by the researcher to the
interviewees that were designed to get a more personal level response. Typical
questions that were added asked the interviewee about the opinions about certain
matters that they had discussed that the research needed additional elaboration. All
interviews were conducted as much as possible in private settings where the
interviewees could express themselves without the concern o f being overheard. The
researcher never used a recorder and only took paper and pen notes that needed to
be reworked following the interview to maintain fidelity to the interview.
Interviewees were guaranteed anonymity as much as possible. References made to
interviewees will only be made to their generic job title such as “teacher” or “school
administrator” in an effort to comply with the promise.
I l l
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 5: List o f Participants Interviewed for Study
District Employee Perspective
Superintendent Central Office and District
Chief Instructional Officer Central Office and District
Director o f Ed Services Central Office and District
Director o f Information Services Central Office and District
Chief Instructional Officer/Principal School Administration
House Principal #1 School Administration
House Principal #2 School Administration
Teacher #1 Math Teacher
Teacher #2 English Teacher
Teacher #3 Teacher
Teacher #4 Teacher
Teacher #5 Teacher
School Board Member Public servant/Elected official
The researcher also gathered a number o f artifacts for analysis from a variety
o f sources. All artifacts were in the public domain unless they were generated with
the understanding that they were only representative documents. Real student
achievement records and specialized reports that originally had personal and
confidential information were cleansed o f any identifiers that would violate
confidentiality requirements. Some artifacts such as student-parent newsletters,
faculty bulletins and website information were collected as they were publicly
available in various offices, and were provided to students, parents and staff
members. They were collected without the explicit knowledge o f any staff member.
School generated reports and surveys that were also collected in the same manner
but they were collected because they conveyed school information that would help
triangulate specific information conveyed during the interview.
112
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data Analysis
The data generated as a result o f this study were context specific within the
school district, school and with the participants engaged in the study. All data
gathered were analyzed, sorted and used or discarded as deemed relevant to the
purposes o f the study. The data were analyzed according to the conceptual
frameworks in an effort to look for distinct patterns and themes.
The data were also sorted for the purposes o f triangulation. Artifact analysis
pertaining to student achievement, school district and school policy, direct
observations and diverse and multi-organizational interviews were the primary
elements for cross-verification and corroboration. The data and findings resulting
from the research were analyzed within the context o f the policies, goals and
intentions o f California’s public schools accountability system.
Summary
Chapter 3 provides an overview o f the study design and methodology used
in collecting data. It includes a discussion o f the research design and the
modifications made to the design. It includes a description o f the sample and
population, the instrumentation, the data collection tools and the conceptual
frameworks. It includes a thick description o f the participants and the interview
process. It describes how additional artifacts were collected. It concludes with a
113
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
discussion on how the resulting data are sorted and analyzed in an effort to
establish a triangulation o f evidence. Chapter 4 presents the findings o f the study.
1
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 4
RESEARCH FINDINGS
Introduction
The purpose o f this study was to describe and evaluate how one high school
used data to improve student achievement. The study described the model
developed at the school district level and described how the school identified,
collected and evaluated data during the year o f the study. The study assessed the
extent to which the district design addressed state goals and expectations for student
achievement, described the extent to which the district design was implemented at
the school site, and attempted to assess the extent o f its usefulness in achieving the
goals o f increased student achievement.
The study was conducted using a qualitative case-study format. The data
was collected during 14 situated interviews involving the superintendent, district
support staff, school site administration and teachers. Additional data was collected
from district and school documents, publications, test results, school and district
conducted surveys and researcher observations. This chapter presents the data
collected and the resulting analysis. The findings o f the study will be discussed
within the context o f study and linked to the emerging state of school reform and
accountability in California. The three research questions provide the framework
around which the discussion of the findings o f the study and the resulting analysis
will be built and addressed in this chapter.
115
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Research Question One
What is the district design for using data and how was the design linked to
the current and the emerging state context for assessing student performance?
A Results Driven Orientation
In a recent local newspaper interview the superintendent of the Valley
Unified School District raised the following question, “What explains success in
school?” According to several VUSD employees interviewed for this study, the
most common answer previously cited was “student ability.” Upon his arrival, the
superintendent has worked diligently at changing the notion that school success is a
result o f “ability” rather than “effort.” The superintendent strongly supported the
work of Lauren Resnick, University o f Pittsburgh, who argued that “effort leads to
ability.” The superintendent believed that in a standards-based instructional program
“effort matters” but effort itself is not rewarded directly “only results count.” He
argued that if school success was not a function o f innate ability and student effort
mattered, then a keen focus on student achievement would lead to improved results.
The superintendent’s focus and thus the school district’s focus were “to work
towards raising academic achievement for all students while closing the existing
achievement gap.”
According to the superintendent, both the current Federal and the state
policies on education accountability fail students. In the Federal case, he believed
the “No Child Left Behind” policy will collapse under it own weight and “in reality
all children will be left behind” as long as full funding o f public education remained
116
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
unrealized. In the California case, he voiced similar concerns but added that the
state’s measures were not rigorous enough and that the academic standards were too
low. While he acknowledged the effort being made at the state level to improve
student achievement, he criticized the state for its ever-changing criteria and
measures that describe how well or how poorly school districts are doing in
improving student achievement. He provided particularly clear examples of the
dynamics when he spoke about the state’s initial use of the norm-referenced SAT-9
as the sole measure o f Academic Performance Index (API), then the inclusion o f the
criterion-referenced California Standards Tests, and then it changed from the SAT-9
to the CAT-6 and then added the California High School Exit Exam into the API
calculations. He additionally warned that “we must be cognizant of the fact that
testing is but one form o f assessment used when evaluating a child’s progress.”
Student Performance Assessed in the Context of Current and Emerging Assessments
In VUSD data is a driving force in the district and it is used incessantly. The
superintendent reported that the district collected huge stores of data from a variety
o f sources. It collected and disseminated to all stakeholders data collected from the
California Standardized Testing and Reporting program (STAR). According to the
superintendent and other central office administrators the district viewed the most
useful data gleaned from STAR was that obtained from the California Standards
Tests (CSTs). The district preferred the data from these criterion-referenced
assessments because they directly related to content standards being taught in
schools. While the district used data from the CAT-6, it relied less on the norm-
117
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
referenced assessments because they do not necessarily reflect what students were
required to learn in California.
Student test results from all o f the STAR assessments were forwarded to the
district’s schools soon after the data arrived at the central office. In previous years,
the principal and teachers at River Valley High School noted that test data came
back in a very delayed fashion. The slow manner in which test data arrived at
schools usually resulted in the data not being used. This year the principal, the
assistant principals and the teachers interviewed all agreed that the central office
provided STAR test results with record speed. The school received whole school
(aggregate) test data as well as disaggregated data for all subgroups tested and
identified at the school site level. Individual school reports were generated by the
central office staff and school site administrators met with the superintendent and
discussed and analyzed the results. As the district’s schools prepared to open,
central office staff met with school site staff to share the test data, they provided
professional development to help teachers and staff “read and interpret” the data and
helped school staffs to identify gaps in student achievement.
The district has focused on raising the API as defined by the state and has
developed a plan to achieve this goal. The superintendent set student achievement
targets at an 8% increase for all schools. Additionally the superintendent set a target
goal of improving API subgroup scores by 16% in the subgroups o f race, language
and poverty/socio-economically disadvantaged. After setting the student
achievement targets, the superintendent met individually with all school principals
118
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
to communicate his expectations and to discuss with principals how they could
collaborate in this effort.
During the last few years the district has moved towards implementing a
standards-based instructional program for all students based on the California
content standards. The district believed that high standards promoted rigorous
learning. They developed a theory o f action for excellence and equity, based on
building capacity through coaching to integrate standards, rigorous instruction and
culturally responsive strategies. The California standards served as the foundation
and the centerpiece o f improved instruction. From the content standards, curriculum
was to be aligned, textbooks adopted and assessments developed. To help
communicate the district efforts the Chief Academic Officer and the Director of
Educational Services visited schools on a regular schedule to explain the role of
standards and why they were important. More specifically the message around
standards was translated into the need “to provide high expectations for all
students.”
Valley Unified School District has a history o f an achievement gap between
White and Asian students and African American and Latino students. The
superintendent’s vision for closing the gap relied on several factors. First the
superintendent made it clear in a number o f venues that the gap must be closed.
Next he communicated a belief that teachers were responsible for developing
intelligence in students and that the central office must support the work o f teachers
and site leaders. Third, the district’s emphasis on holding all students to high
119
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
expectations and standards led to changing the status quo. He denounced the idea
that business as usual would close the gap and bold new goals were needed to
transform an educational system that allowed the gap to exist. Closing the
achievement gap meant providing students with “best first teaching.” It meant a
focused effort at implementing a standards-based literacy program that was
balanced and comprehensive. In grades 1-8 students received extended learning
opportunities in English-language arts by increasing daily instructional time to 2.5
hours. Time allocated to math instruction increased to a minimum o f 1 hour per day.
At River Valley High School the superintendent supported a complete
redesign o f the high school’s organizational structure to provide students with a
more personalized educational experience by transforming the high school into six
small high schools. River Valley received a small learning communities grant. As a
requirement o f the grant, the school had to identify a set of broad overarching goals
and a set o f measurable goals. The school leadership stated that the move to a small
school redesign was created to achieve three broad goals:
1. to increase personalization
2. to increase achievement for all students
3. to eliminate the achievement gap.
Measurable goals for the small schools redesign at RVHS were set by the
superintendent in collaboration with the school’s Chief Instructional Officer, some
goals dealt with issues of implementation and some goals focused on student,
teacher, and parent perceptions. In respect to raising student achievement in the
120
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
context of current and emerging assessments in California, measurable goal two
stated:
Academic achievement is improved across all subgroups (whether or not
they are deemed significant by the State o f California). Scores will
demonstrate improved achievement on standardized test scores such as
STAR and CAHSEE as well as improvements to the school’s overall grade
point average (GPA). Specifically, in each of the following three years,
student measures of achievement will show a:
1. Three percentage point increase on STAR testing;
2. One percentage point increase in the student pass rate on the
CAHSEE English-language arts and CAHSEE Math tests; and
3. Five percentage point decrease in students with a GPA below 2.0.
The heavy focus on standards-based instruction in the school district and at
River Valley High School ensured that administrators, teachers, students and parents
were aware o f STAR, the CAT-6, the CSTs, the California English Language
Development Test (CELDT), the CAHSEE, and the Academic Performance Index
(API) score. Administrators were provided district briefings at the central
administrative offices throughout the last three years to build their depth and breadth
o f knowledge o f these state assessments and programs. Several times during the
year the superintendent and staff discussed with school administrators the
importance o f raising the student achievement levels in each o f these measures of
student achievement for all students and in closing the achievement gap. At the
121
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
beginning o f the school year faculty members were provided a day of professional
development explaining how most o f these assessments and measures were based
on the California academic content standards. The Director o f Educational Services
also informed the faculty of their students’ most recent performance and pointed out
improvements of student achievement as well as pernicious gaps. Faculty members
were provided an opportunity to meet in departments to discuss what needed to be
done to move to a standards-based instructional program and how they could work
together to raise student achievement.
Students whose achievement levels on state identified assessments were low
were identified and provided special assistance. Most o f the special assistance took
the form o f after school tutoring. Students were also recommended for tutoring if
their in-class performance indicated a need for additional instructional support.
Teachers primarily used grades on teacher made tests and quizzes as the criteria to
make their recommendation for tutoring.
The superintendent, the school administration and the faculty at River Valley
High School made specific attempts to inform students and teachers of the STAR,
the CSTs, the CAHSEE and the API. The superintendent informed the public mostly
through reports to the Board of Education o f Valley Unified School District. He also
raised the level of awareness through newspaper interviews and through his monthly
open letters to the public and to district personnel. The administration used parent
newsletters and its web-site to inform parents o f current and emerging state
122
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
assessments. Teachers informed students and parents of these measures primarily
through their course syllabi.
Recent technological improvements funded by the school district helped
district employees to better utilize student achievement data. The programs
purchased by the school district allowed school administrators greater access to
specific student achievement data. The programs facilitated data utilization by
making the tracking and sorting o f student data easier to access and easier to
manipulate. The superintendent, his administrative staff and school site
administrators produced reports that looked at specific student assessment results for
unique student populations. The superintendent shared a recent report that reported
individual student test results by school, grade and by teacher. The Chief Academic
Officer independently produced a report showing growth or lack o f grow at a
specific school, by grade level and disaggregated by race, language and socio
economically disadvantaged status. At the school site a house principal
demonstrated how the computer program could produce various reports using
specific data fields. He produced in “real-time” a report identifying students who
scored at the “below basic” and the “far below basic” in Algebra 1 CST for the 2003
test administration. All three administrators specifically mentioned the importance
o f technological advances, particularly advanced computer software and the Internet
as reasons for being able to utilize data more frequently and more efficiently.
123
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Emerging Context
The superintendent of Valley Unified School District (VUSD) voiced strong
concern over the state’s accountability and assessment practices. He specifically
noted the frequently changing targets and tools used to measure student
performance as a point o f contention. At the same time he was keenly aware o f the
need and importance for remaining up-to-date on emerging developments at the
state level. He was remarkably fluent in his understanding o f emerging state
assessments and identified specific measures the district has implemented to raise
student achievement and provide access and opportunity for students in the district.
He explained that the district instituted two new programs at the high school level to
address CAHSEE and to address the state’s newest emerging assessment, the Early
Assessment Program (EAP).
The district required all 10th grade students district wide to take a CAHSEE
math pretest. The purpose o f the CAHSEE pretest was to gauge how well VUSD
students were prepared for the actual CAHSEE and to identify students who might
need additional assistance. Students needing additional assistance were
recommended for math tutoring programs and for other types of assistance to
increase their math proficiencies.
The teachers at River Valley High School on their own accord took special
measures to prepare for the CAHSEE English examination. The California
Department o f Education published a Student Study Guide to inform students about
the assessment. The RVHS English department chair edited the document for the
124
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
state and made a strong case to distribute it to RVHS students. In addition to a
strong emphasis on standards and several other important factors, one teacher
credited the study guide with helping RVHS students obtain a first-time pass rate of
89% on the 2004 CAHSEE English assessment.
The second element was an awareness program that was created to address
the new EAP. High school English teachers were informed o f the new opportunity
provided to students in grade 11 who did well on the EAP. Teachers were informed
that students who took the additional multiple choice questions on the 11th grade
CST for English and who wrote the additional EAP essay became eligible if they
scored high enough to become exempt from California State University placement
exams. The program had two effects, it increased awareness o f the new opportunity
and it assisted teachers to better focused lessons that specifically addressed the skills
and design o f the EAP assessment.
The district began a process two years ago to develop or purchase
assessments that were administered to students at various grade levels in English-
language arts and in mathematics. The district-wide periodic criterion referenced
assessments based on California academic standards were first implemented in
elementary schools during the 2002-03 school year and in middle schools during
2003-04. Criterion-referenced periodic assessments at the high school level are yet
to be developed or purchased. The current district design called for three periodic
assessments in English-language arts and mathematics to be administered in grade
kindergarten through grade 8. The high school design called for two periodic
125
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
assessments in English in grades 9-12 and three assessments by course in math in
grades 9-12. The purpose of the periodic assessments was to identify periodic
benchmarks for student achievement and to help teachers plan future lessons and to
provide insight into what remediation might be needed for certain students or
student groups.
When this researcher asked the superintendent about the importance of
international assessments, the superintendent responded with absolute resolve. He
stated that, “Our standards should be internationally benchmarked — that is, we
should be asking our students to perform at the same levels as their counterparts in
the rest o f the developed world.” He continued his thoughts on the matter and said:
If American workers are to receive high wages and obtain jobs
in the high-value-added industries, they must be able to meet
the same educational standards required o f students in our
competitor nations. Our standards in the Valley Unified School
District meet this expectation.
Overview o f the Elements o f District Design o f Data Use to Raise Student
Achievement
The superintendent’s often stated primary educational goal for Valley
Unified School District centered upon raising the student achievement levels of all
students while simultaneously closing the achievement gap. To measure progress
towards meeting this explicit goal the district collected a wide variety of student
achievement data. According to the superintendent, “data is a driving force in the
district. We use it incessantly and have collected huge stores of data.” Although the
district collected and used data extensively, the superintendent admitted that the
126
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
district did not have an official explicit Board adopted model design for using data
to raise student achievement.
The superintendent, the Chief Academic Officer and the Director of
Educational Services explained during independent interviews the various types of
data collected and their intended use. Amazingly, all three central office
administrators listed the data collected at the district level. Foremost among the
categories o f data identified and collected were those traditionally mandated by state
and federal agencies, especially student demographic data, school climate data,
district financial status data, teacher demographic, employment and credential data,
as well as student achievement data on state mandated assessments and STAR. The
district and school were mandated according to the California Education Code to
provide a School Accountability Report Card (SARC) and "shall provide data by
which parents can make meaningful comparisons between public schools enabling
them to make informed decisions on which school to enroll their children."
Each o f the central office administrators acknowledged the importance of the
above identified data categories but insisted that the main purpose o f schools was to
educate children. As such, the superintendent wanted to focus the discussion around
student achievement data. The superintendent emphasized that STAR, the state data
model for measuring student performance was inadequate and needed to be
augmented by local district and school efforts. The district efforts he noted
concentrated on the established periodic assessments in English-language arts and
mathematics at the K-8 grade levels. He maintained a steadfast conviction that
127
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
frequently collected student achievement data held the greatest potential o f
improving student achievement. He also believed that teacher developed
assessments were very valuable, especially if they were o f high quality, rigorous,
and were used to inform teacher practice.
The superintendent also instituted a practice of collecting data through a
survey o f important school-based processes modeled after Rhode Island’s, School
Accountability fo r Learning and Teaching (SALT). The National Center on Public
Education and Social Policy at the University o f Rhode Island assisted VUSD in
administering and reporting out the findings o f the survey. The SALT-like survey
has been administered to school administrators, school faculty and support staff,
parents and students. River Valley High School was the only school community in
the district to have administered the survey three consecutive years. The survey
examined four research-based specific learning support indicators used to examine
and capture the various and complex elements existing in schools. The four learning
support indicators measured in the survey were:
1. Instruction - specifically the levels of standards and
researched-based instruction taking place in the school and
district, teacher preparation levels, barriers and support for
implementation o f standards-based instruction as reported by
teachers;
128
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2. Parental Engagement and Involvement - identified levels and
patterns of and needs for, parental engagement and
communication;
3. Climate - a set o f indicators that measured school climate as
reported and experienced by teachers and students in the
areas o f personalization, learning expectations, safety and
support for student achievement; and
4. Health - looked at levels o f student nutrition and rest that
have been linked to high levels o f student performance.
The superintendent also brought with him a desire to work with Dr. Lauren
Resnick o f the University o f Pittsburgh. Dr. Resnick leads the Institute for Learning
(IFL) a research-based institution focused on improving teaching and learning. The
IFL used a construct called the Principles o f Learning (POL). The nine principles
provided various lenses through which administrators and teachers could discuss and
make judgments about instructional practices. The POL also provided data
collection tools, analysis and facilitated data use to occur using observations of
teachers’ instruction, student engagement and student work. The superintendent held
an expectation that the POL would be deeply embedded over an unspecified amount
o f time into the school culture and in practices o f teachers. The POL, as designed,
was to be introduced over several years rather than a quick rollout program. The first
principles introduced into the VUSD system were, “Clear Expectation” (clearly
articulated content standards), “Academic Rigor” (academically rigorous
129
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
instructional practices and student assignments with rubrics and criteria charts) and
“Accountable Talk” (holding students and teachers to a protocol used during
discussions of academic content). Additionally, school administrators and teachers
were expected to collect data about instructional practices during “Learning Walks,”
structured classroom observations o f teaching and learning.
Unlike some school districts that rely solely on annual measures o f student
achievement, Valley Unified School District engaged in year-around data collection,
analysis and utilization. Table 6 illustrates the yearly cycle o f data collection
experienced in Valley Unified School District.
The results o f these assessments were reported out in a variety o f venues.
The superintendent made periodic reports to the Board of Education about the
progress o f improving student achievement and in closing the achievement gap. He
noted with particular pride that this board of education was deeply concerned about
what the data indicated. One long-time school board member stated,
This superintendent has made it regular practice to inform
board members about student achievement. In the past we had
to make formal requests to the previous superintendents to gain
access to student achievement data. The practice o f data
sharing was usually guarded. This superintendent has made
data public. We as board members used the data reports to help
set policy and for strategic planning purposes.
130
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 6 Yearly Cycle o f Data Collection
Assessment Tool Administration Results
Available
Grade
Levels
CST’s Spring August 2-11
CAT-6 Spring August 2-11
CELDT Spring August 2-11
CAHSEE September,
November,
February,
March and
May
Within 60
days of
administration
10-12
Advanced
Placement
May Summer 10-12
EAP Spring August 11
SABE Spring August 2-11
CAPA Spring August 2-11
Writing Test Spring August 4 and 7
Periodic Bench
mark exams in ELA
and Math
Fall, Winter
Spring
Within two
weeks
k-8
CAHSEE Pre-test Fall Within four
weeks
10
SALT-like Survey May September 2-12
The superintendent has also made the use of data a component of
administrative accountability. He commented that in his weekly meetings with
school site administrators he noticed a geometric increase, a “J curve” increase in the
knowledge and skill of data analysis among principals. The use o f data was an
important indicator for him about the quality o f leadership at the school site. When
this researcher asked the superintendent about why he thought there was such an
131
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
increase in data use by school administrators, he paused and said, “They have no
choice, either they do or they are out.” He accepted the fact that this sort o f demand
led to an exodus o f administrators.
District Decisions and Rulings that Support Use of District Design
The Valley Unified School Board o f Education use o f student achievement
data has never been greater than current times. The current superintendent regularly
reported to the Board of Education about student achievement and how the district
staff intended to raise student achievement and to close the achievement gap. The
current superintendent and his staff made numerous reports to the board o f education
during the year o f this research.
District decisions to utilize and support data utilization emerged three years
ago when the previous superintendent’s contract expired. The one school board
member interviewed stated that the simple fact that they hired “this current
superintendent was a perfect example” o f their determination to change the direction
o f the school district.” The board member inferred that previous superintendents
had not been as forth coming with them about the extent o f student success or lack
o f student success until the passage o f the Public Schools Accountability Act. She
said that when the state started to compare schools statewide, some members
realized that all schools were going to have to report student achievement data and
that data was going to be made very public. When the opportunity to hire a new
superintendent arrived, there was a strong motivation to hire a superintendent who
would facilitate a more open discussion o f the status of student achievement.
132
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Shortly after becoming superintendent, a new reorganization plan for the
central office was presented to the Board of Education. The plan radically changed
the structure and function o f the central office to vastly improve the focus and
support to schools in their effort to raise student achievement. Over the last three
years, the superintendent’s original staff has been replaced with administers who
were known to have a reputation of focused attention on improving student
achievement and in actuality a strong working knowledge o f research and data
analysis. The decision to reorganize the district’s central staff was extremely
important and controversial. Many o f the previous assistant superintendents and high
level administrators voiced concern directly to board members. Board members
were confronted with a political dilemma to support the newly hired superintendent
or to support an administrative staff that had been around for a number o f years, who
had long established relationships with local community leaders and with local
school administrators. The board almost universally supported the newly hired
superintendent and that decision led to an exodus of high level central office
administrators. The effect rapidly increased plans for reorganization and facilitated
the hiring of a new central office staff supportive of district goals and o f the
superintendent.
In a more local context for River Valley High School, student achievement
data and huge stores o f data collected over the last 10-14 years led to a request by
RVHS to reorganize. The superintendent and the school administrative staff met to
discuss school reorganization as a result o f data. The superintendent moved a local
133
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
middle school principal to the high school and promoted her to Chief Instructional
Officer. Her experience with middle school reform, especially her experience with
middle school “houses,” made her an excellent candidate. The decision to break up
the high school into six small schools was supported by the superintendent, the
Board o f Education, and the community. Teacher and staff buy into the
reorganization was also an important factor. All o f the teachers interviewed by this
researcher specifically mentioned that “a strong case for school reorganization had
been made based upon the information known over the years and the faculty’s strong
desire to improve student achievement.”
The district did not have an official model for using data although various
types o f data were widely used throughout the school system. The Board of
Education did not mandate any particular training to support data use, however
numerous optional professional development opportunities were provided by the
Director of Education Services to teachers and administrators. The workshops were
primarily held after school or during vacation time.
Intended Results o f Design Plan to Improve Student Performance (District. School
and Classroom)
The Valley Unified School District monitors and tracks data to improve
student achievement o f all students while simultaneously closing the achievement
gap. In March, 2004, the VUSD adopted a detailing specific bench marks for student
achievement. The student achievement goals were identified by the superintendent
and his staff. School administrators were informed o f the proposal and it was unclear
134
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
if the any revision or modification to the proposal was made as a result of the
superintendent and principals.
The benchmark goals and objectives noted in Table 7 were designed to make
district student achievement targets concrete. The performance measures and target
growth rates as well the identified student population identified was useful for
school administrators to explain the district’s expectations for student achievement
gains. The greatest implication for the targeted growth for schools and classrooms
occurred as a result o f periodic assessments where and when they were available.
The district maintained a strong expectation that administrators and teachers would
use the results from period assessments to make mid-course corrections in
instruction and to provide additional assistance to students who needed additional
time and coaching.
The district also intended that information gained about teaching and
learning while using the Principles of Learning would help shape measures designed
to raise student achievement. Specifically, professional development opportunities
would be developed to provide support for the POL. Expectations were that schools
would use professional development time to discuss issues o f literacy across the
content areas, the use o f rubrics, the use of criteria charts, the development of
multiple disciplinary based assessments that supported academically rigorous
instruction. The expectation was also articulated that teachers would meet together
to discuss student work, develop anchor papers for writing exercises and some form
o f common subject area department-based assignments and assessments.
135
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 7 VUSD Student Achievement Goals and Objectives
Performance Measures Target Student Population Percent
State and district assessments All students and disaggregated
subgroups including English-
language learners (ELLs) and
students with disabilities
95%
participation
rate
Language Arts
CSTs English-language arts
(ELA)
All students scoring at “proficient” level 8% growth
annually
CSTs ELA Students in disaggregate subgroups at or
above “proficient”
16% growth
CAHSEE ELA All grade 10 students and disaggregated
subgroups pass
85% pass rate
District administered ELA
standards-based program
assessments
Students meet district benchmarks on
standards-based instructional program
85% pass rate
District administered ELA
standards-based program
assessments
ELLs meet district benchmarks on
standards-based instructional program
between winter and spring administrations
85%
demonstrate
progress
District standardized
Spanish-language arts
assessments
English-only and ELL’s in Two-Way
Immersion
85%
proficiency
CELDT English-language learners:
Beginning- two levels
Early and Early Advance- one level
80% demonstrate
annual increase
District administered ELD
standards-based program
assessments
ELLs 85% pass rates
Mathematics
CSTs Mathematics All students scoring at “proficient” level 8% growth
annually
CSTs Mathematics Students in disaggregate subgroups at or
above “proficient”
16% growth
CAHSEE Math All grade 10 students and disaggregated
subgroups pass
85% pass rate
District administered math
standards-based program
assessments
Students meet district benchmarks on
standards-based instructional program
85% pass rate
District standardized Spanish-
language math assessments
English-only and ELL’s in Two-
Way Immersion
85% pass rates
College Readiness
Grade point average All high school students with
GPA below 2.0
10% decrease
Advanced Placement Courses 11th and 12th grade students
completing AP courses
10% annual
increase
UC/CSU a-g requirements 1 2 * grade students completing a-g
requirements
10% annual
increase
136
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The district, in an effort to keep parents up-to-date with their youth’s
academic progress, used Internet based technology. The District purchased a grading
program that allowed teachers to post student grades on a secure computer server
that allowed parents to access student grades “on-demand.” The grading program
also allowed teachers to post assignments for student who were absent and for
parents who wanted to keep a close eye on their son’s or daughter’s classroom
assignments.
Data Use Policy and Strategy Funding
Valley Unified School District funded its data collection, analysis and
utilization primarily through the use o f general fond monies. District personnel did
most o f the data collection and analysis. The superintendent reorganized the central
office to better assist schools with understanding the data collected and to provide
preliminary analysis. The job responsibility for data dissemination previously fell to
the district’s Director of Information and Technology. More recently the tasks
associated with data collection, analysis and distribution were divided and
distributed to the Chief Academic Officer, the Director o f Education Services and a
new Assessment and Data division within Education Services.
In 2003, city residents who live within the school district’s boundaries passed
a “save our schools” parcel tax to provide additional fonds for Valley Unified School
District. The school district received $225 per parcel o f land or about $6.25 million
per year. The monies raised saved 90 teacher jobs, and eliminated the need to
increase class size during the 2003-04 school year. The parcel tax will remain in
137
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
effect for 6 years. The district benefited from the parcel tax by being able to avoid
cutting the periodic assessment programs which were a major component o f the
superintendent’s plan for raising student achievement at the elementary and middle
school levels. The SALT-like school survey was only administered three consecutive
years at River Valley High School.
The VUSD also established an “equity fund” that was created as part o f the
new district-wide gift policy approved at the last Board meeting. The cornerstone of
the policy will be administered by the Valley Unified Education Foundation, which,
with the Board o f Education, “will, annually, distribute block grants to all schools to
address district and local school goals and mitigate the effects of the unequalized
enrichment o f district schools.” The equity fund was comprised o f “fifteen percent of
all cash gifts to either the district or any department or individual schools, donated
during a school year,” or from grants and gifts donated to the district. Some of the
money funded reading interventions (Grades K-8), mathematics interventions
(Grades 4-8), programs held before/after school or on Saturdays, summer school and
additional tutors for students needing additional assistance. The effect of this policy
was designed to reduce the funding gap between schools in the more economically
exclusive areas of the district that received enormous amounts o f money to augment
school budgets and those schools in economically distressed areas that received
virtually no monetary gifts. Prior to school board adoption, some parents and
citizens voiced opposition to the proposed policy but in the end the percentage of
138
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
money diverted away from the originally targeted schools was considered marginal
by most accounts.
Individual schools in Valley Unified School District were fiscally responsible
for all other aspects o f data collection, analysis and utilization. River Valley High
School has secured a number of grants to fund its data collection, analysis and use
over the years. In the early 1990’s RVHS secured a large study grant from the
Annenberg Foundation. The Annenberg grant was the first significant amount of
money and first significant collection o f data aimed at school reform. The
Annenberg grant provided data but the data did not facilitate any real change for
almost a decade. In 2001-02 RVHS, wrote and received $60,000 Small Learning
Community Planning grant. The grant paid for the release time for one teacher to
collect various types o f data, to interview teachers, students and parents. The
findings o f that grant added to the data and the findings already collected from the
Annenberg grant. In 2002-03 RVHS received a large grant from the Stuptki
Foundation, this grant led the school redesign effort moving RVHS from one large
comprehensive high school to six small high schools on one campus. The decision to
redesign was based on the huge amount o f data collected over the last decade,
especially the persistent and pernicious gaps in student achievement between various
ethnic subgroups. In the 2003-04 year, RVHS received a Small Learning
Community Implementation grant o f $500,000. The above mentioned grants all
required evaluation reports describing how the school used the money and what the
school discovered as a result of the data collection and analysis. The practice of data
139
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
utilization for substantive school change and focused attempts at raising student
achievement have only recently been implemented and realized.
Summary o f Research Findings for Question One
Research question one asked, “What is the district design for using data and
how was the design linked to the current and the emerging state context for
assessing student performance?”
Valley Unified School District did not have an explicit Board adopted model
design for using data. The superintendent maintained an unshakeable conviction in
the power o f data and was deeply committed towards establishing a culture o f data
analysis and use throughout the district at all levels. The central office has been
reorganized to support data use in an effort to raise student achievement.
Valley Unified School District collected, analyzed and disseminated data
collected by the state to district stakeholders. The district’s practice o f data
collection, analysis and utilization vastly exceeded the de-facto model provided by
the state. More specifically, the district began to institute multiple measures o f
student achievement, periodic assessments, instructional practices in teaching and
learning and school performance indicators specifically aimed at raising the
achievement levels o f all students while simultaneously closing the achievement
gap.
In addition to collecting student achievement data the district and the high
school utilized a self-study tool used to provide schools with reliable and systematic
information for use in planning and monitoring school improvement efforts. It
140
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
provided a mechanism for making important decisions by raising key questions that
lead to deep engagement of all staff in the school's transformation.
Research Question Two
The second research questions asked, “To what extent has the district design
been implemented at the district, school and teacher level?” Conceptual Framework
B Degree o f Design Implementation sought data to describe:
• implementation o f current data practices (at the district, school site, and
classroom levels);
• implementation o f emerging state data practices (at the district, school
site, and classroom levels);
• accountability for data use at district, school, and individual levels; and
• improving student achievement through implementation of data use.
Degree o f Design Implementation Current Data Practices
Valley Unified School District did not have an explicit Board adopted model
design for using data. The superintendent maintained an unshakeable conviction in
the power o f data and was deeply committed towards establishing a culture o f data
analysis and use throughout the district at all levels. The district began to institute
multiple measures of student achievement, periodic assessments, instructional
practices in teaching and learning and school performance indicators specifically
aimed at raising the achievement levels o f all students while simultaneously closing
the achievement gap.
141
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The superintendent o f Valley Unified School District has worked hard at
establishing a culture utilizing data to raise student achievement. He expects
administrators and teachers to utilize various forms o f data to raise the student
achievement levels o f all students while simultaneously closing the achievement
gap. He acknowledged that he thought the collection, analysis and utilization o f data
was probably better embedded at the elementary school level where most o f the
periodic assessments have been instituted. He spoke eloquently about the “shifts of
instructional practices” realized at the elementary level resulting from the frequency
of data collected and analyzed. District-wide periodic assessments provided a
valuable vehicle to address student achievement. He believed that data use was less
prevalent at the secondary level at the present time, noting that periodic assessments
at the middle school and high school levels were not fully instituted.
The Chief Academic Officer provided a multi-month calendar identifying
the various professional development meetings in which she had participated. The
calendar o f trainings verified the superintendent’s statements about the use of data at
the elementary level. The CAO provided a number o f data worksheets, protocols
and reflection questions that have been provided to elementary principals but had
not yet been provided to high school principals mainly because the were tied to the
periodic assessments by Houghton-Mifflin and Holt. Here are just a few example
prompts provided to elementary principal:
• Quickwrite: What did you first notice when you looked at you data? What
jumped out at you?
142
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• What are our observations and reflections when we look at our grade-level
data?
• When we examine test questions for the subtest, what is the specific content
being tested? What California State Standard is being addressed through the
content?
• How will we share our “data-driven instruction protocol with your faculty?
(grade-level team meetings, department meetings, whole school banked
time, etc.)
In his observations of data use at various secondary schools, the
superintendent believed that there was a genuine awareness o f the benefits o f using
data. He also noted that realistically, practice varies among school administrators
and teachers. He has noticed very little willful objection to data practices but he said
there were challenges. He proceeded to make several generalizations about data use
among teachers. For the novice teacher, the chances were great that data use was not
a priority and that they were probably overburdened not only with daily rigors of
being a teacher, but when confronted with large amounts o f data they appeared
overwhelmed. For the older, veteran teacher, he imagined that they just did not care
much about analyzing data. He thought that many had become set in their ways and
probably thought that they would use what they could and not much more. He was
most hopeful when speaking about the middle career teacher. He stated that mid
career teachers were making the most gains in data use. Specifically he noted that in
looking at school committees, he noticed that most participants still had many good
143
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
years ahead o f them in education and thus probably had a vested interest in using
improving student achievement. At the time o f the interview, the superintendent
believed that “we are probably about half o f the way there” to effectively use data to
raise student achievement.
The Chief Educational Officer (CEO) at River Valley High School described
data use in similar terms as the superintendent. The CEO thought that the faculty’s
level o f understanding o f the various types o f data being collected by the district
remained for most teachers at an awareness level but with a growing understanding.
She also noted that certain faculty on campus used data more than others as a result
of their job responsibilities and assignments. In the organizational structure o f River
Valley High School, one house administrator had the primary responsibility of
analyzing data and generating reports as requested by other house principals or as
requested by the faculty. Data users among the faculty included those who were
helping to implement school redesign, writing reports as a condition of grant funded
programs, planning for the WASC accreditation visit and monitoring intervention or
special assistance programs.
Each year since the establishment o f State mandated testing under the 1999,
Public Schools Accountability Act, the faculty at RVHS has had at least one
professional development explaining the school’s test scores. Nearly all of the
teachers interviewed thought that the SAT-9 scores, CAT-6 scores and the CST
scores were presented in an interesting manner but few teachers really thought that
they needed to do anything specific once the scores were presented. Until recently,
144
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
teachers reported that there were few scheduled opportunities for teachers to meet
and discuss the student achievement results and few professional development
trainings on how to better understand and utilize data. When RVHS started school
redesign in 2002-03, the sharing o f student performance data became more frequent.
Non-mandatory professional development opportunities were developed and
available for all teachers interested in using data. The professional developments
were usually off-site on weekends, after school and during vacation periods.
The superintendent’s restructuring o f the central office also led to another
significant change identified by the school site administrators and teachers who
were interviewed. The hiring o f a new Chief Academic Officer and a new Director
of Educational Services increased the availability of central office staff to RVHS for
the purposes o f providing technical assistance in data use and analysis. Prior to the
arrival of these two administrators support for data use had been weak. The
restructuring o f the central office also facilitated the dissemination of the results
generated from the STAR system. Teachers reported that prior to this last year state
testing data arrived at the school very late and had almost no impact on their
instructional practices. This year, teachers reported receiving student test results in
time for the opening of the school year.
This year teachers reported a heightened interest in what the STAR results
indicated. They were concerned about what the data showed especially because the
schools’ overall API score had dropped four years in a row. The entire faculty was
pleased to see an increase in student performance, as noted by the API, for all
145
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
students and for student subgroups. Table 8 details RVHS’s disaggregated API
growth for the 2004 administration. The administration at RVHS reported that one
of the main reasons why they thought that they did a much better job in the
administration o f STAR tests in comparison to previous administrations. In a review
of last year’s procedures they noticed that too many students had missed one or
more sections o f the exams. To remedy this situation the administration and the
testing coordinator changed the testing bell schedule to allow all student to take the
exams in their registered English class. This procedure provided better oversight and
less confusion for students and faculty.
Table 8 Disaggregated Subgroup API____________ __________________________
Subgroup # 2004 Score 2003 Score 2004
Target
2004
Growth
African
American
256 592 570 4 22
Asian 179 824 820 A* 4
Hispanic 722 609 565 4 44
White 1209 788 786 4 2
SES Dis 605 603 556 4 47
*A=>800
The superintendent and RVHS’s CEO noted a greater reliance on the
California Standards Tests CSTs because they reflected content that teachers should
teach and students should leam. During the last two years there has been a
movement to align instruction to the California academic content standards in each
of the core subject areas o f English-language arts, mathematics, science and history-
social science. All three administrators and all five teachers interviewed stated that
time had been allowed for departments to meet, discuss and plan a transition to a
146
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
standards-based instructional program during professional development days.
Teachers reported that most of the professional development focused on curricular
issues occurred in departmentalized subject matter meetings. They also reported that
most o f the instructional strategies type professional development occurred in house
meetings. The administration provided release time o f content specific groups to
meet to align instruction with the content standards. It was thought important by the
administration that consistent academic rigor across classes might also be achieved
if the attempt to align instruction to content standards was achieved.
The teachers interviewed stated that “Making Standards Work” a
professional development opportunity provided by the Center for Performance
Assessment moved the work on standards forward. They stated that departments
met and identified “power standards,” those standards that were perceived as most
important and fundamental to disciplinary knowledge. The started the process to
align performance tasks and scoring rubrics for each o f departments and some o f the
specific classes. They also took time to discuss the quality o f tasks teachers assigned
their students in an effort to eliminate interesting but not relevant assignments for
meeting standards. Teachers expressed a concern about not having enough time to
cover the required standards. They stated that they felt pressure to teach what they
knew students were required to know but felt that some of the assignments they
knew were really beneficial to students were on the verge o f being eliminated from
their instructional battery. Teachers were divided between those who valued their
professional judgment and those who felt compelled to follow state and district
147
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
guidelines. Both sets of teachers strongly believed that their instructional practices
were in the “best interests o f kids.”
The Chief Educational Officer and the two house principals stated that
implementation o f a standards-based instructional program was uneven across the
curriculum at RVHS. Based-upon their observations during “learning walks” the
administrators noted that some academic disciplines were “deeper” into the work
than others. English-language arts teachers for example were fairly advanced,
followed by math, science with the least progress made by the social studies
department. These observations by the administrative staff were confirmed during
the teacher interviews as well.
The two teachers interviewed in the English department stated that they had
made much progress in moving towards a standards-based instructional program.
They had met together as a department to make some important decisions about
instruction, instructional materials and assessments. They both spoke about using
data from the CSTs and the CAHSEE as a force in changing instruction. One o f the
teachers said,
Before the standardized testing there were few incentives for
English teachers to meet and discuss how we might work
together. We were all professionals and we knew what we
needed to do. We all realized that there was an achievement
gap and many of us thought it unconscionable. Some in our
department worked as individuals with their student and others
blamed the gap on students and parents. After several years of
declining test scores we decided that we needed to do
something as a department. We decided to move from a mainly
literature-based approach where we used specific works of
literature, each of using literature we wanted to teach, to one
where we adopted textbooks aligned to standards. It was a
148
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
difficult decision because no one wanted to give up what they
were doing and change. I will say that even before state
mandated testing we had occasionally met to grade papers
together, mostly on the weekends. We have developed
department rubrics for our assignments and for various types of
essays or writing prompts. Now we meet a little more often and
we also give a department, end o f semester common final
exam.
The other English teacher spoke about using data in a different manner but
still related to standards.
The district supports the use o f data tremendously. Data
matters, standardized test scores matter but test scores don’t tell
you everything, but you can’t ignore them either. We use data
to place students in classes, we use the STAR test and CSTs to
decide if students need additional help such as getting students
into AVID or placing them in Highpoint. W e also use the
STAR 7th grade writing assessment to see who gets into
“honors.” About half o f the entering 9th grade class enters
honors classes. Now that I think about it w e use all o f the
STAR assessments as a means to improve student
achievement. We try to find the right placement for kids, we
are upfront with them about the skills they need to work on but
if they are up for a challenge they can take A.P. The
department uses the content standards to guide instruction
specifically in grades 9 and 10 but less so in grades 11 and 12.
The math teacher interviewed also noted that her department was deeply
involved in using data that the state and district provided. She specifically noted
looking at the CSTs for math classes and the CAT-6. “W e looked at them by
ethnicity, free and reduced lunch and by permit. This year was the first year we had
ever seen this data.” She noted that one of the main reasons for seeing the data was
that the district had hired a new person to do the “data stuff.” She described the use
o f standards and instructional practice as follows:
Sure we use data, we look at the API, STAR and CAT-6. In
some cases in the past, teachers encouraged students to have
149
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
their parents take the waiver but that was less so this last year.
We have always used various forms o f data to guide our
instruction. Before the current superintendent RVHS had a
practice o f “paper norming” to ensure a consistency among
classes. We found that grading might vary by teacher but not
extremely dissimilar. This year we looked took a professional
development called “Making Standards Work.” It was amazing
to me how many teachers had never seen the test blueprints
that the state provided. The school district is supposed to be
focus in standards but there are some teachers who still think
they have academic freedom. Other teachers here RVH think
that they do not need to look at data, why? We had 45 students
go to Berkley. We have lots o f student take Advanced
Placement and the score were high. There was an emphasis on
helping students get higher grades, all it did I think was
promote grade inflation. Some teachers gave legitimate “A ’s”
because some kids are at really high levels. W e’re doing a
better job in using standards, things have been changing around
here because we have a lot o f new staff, they know about
standards but some don’t know much content or they have poor
classroom management. We have adopted books with
standards, but like I said, if the teachers don’t know their
subject... RVHS also increased math requirements from 4
semesters to 6 semesters to align to UC/CSU requirements.
Overall, I think we are moving in the right direction, as the
“old guard” leaves thing will continue to change. W e’re still
not there yet.
The science teacher interviewed confirmed much of what the three previous
teachers noted in their interview. He confirmed that the school was using STAR
tests, especially the CSTs to guide instruction. He said that department discussions
focused on how to better align instruction to the standards and that much o f this
discussion occurred after the looking at the testing blueprints. He was stunned at
what the blueprints revealed and took it upon himself to work at rewriting the
biology course of study. He stated:
When we looked at the end o f course exams for science and the
testing blueprints we (members o f the science department and
myself) did a close analysis of what was tested in comparison
150
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
to what we taught. We looked at the textbooks we were using
and decided that several things needed to occur. First, we
needed to rewrite the biology curriculum to better reflect the
state standards and the testing blueprints. Second, we needed to
buy the new science books that included the standards. Third,
we needed to look at the assignments we required then to do
and find a way to make sure they focused much more
specifically on what the state tested. We finished the new
curriculum at the end o f last year, so this year is the first year
we are using it. We will have to wait and see if what we did
makes a difference in student achievement based on the
biology CST. I also want to comment on that it was decided
that all student would take biology and chemistry. That
decision means a lot more students who did not have access in
the past, would, and that they were probably going to need
additional help. The school decided to implement a three
period biology-algebra block course where students have
additional concentrated time to learn both subjects. It provided
more time for struggling students and smaller classes.
The CEO reported that after one year, CST test data showed that biology-
algebra block did little to raise student achievement scores on the biology and
algebra exams. Most o f the students remained below proficient on those tests.
Transcript records showed that grades improved for student in the program. The
program was going to continue with an emphasis on preparing students for the CSTs
and not just improving their grades.
The teacher from the social studies department described a similar situation
with very different outcomes. He explained that the school and the district focused
on providing students a standards-based education, it had a focus on closing the
achievement gap and that everyone looked at the state test results both as a whole
school, in departments and in house meetings. He also spoke negatively about the
SALT-like survey the faculty has had to complete the last three years as “a complete
151
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
waste o f time and money." He also thought that the small schools redesign was
“forced down our throats.” Here is how he described RVHS.
We spend a lot o f time, especially in departments talking about
and being told we need to be standards-based. We have spent
several specific department meetings talking about some sort of
pacing so that we are all around the same topic at the same
time. The fact o f the matter is that what sounds good doesn’t
always actually workout. Our department is really independent.
Teachers teach what they want to teach. We meet mostly to
share lessons but I don’t think many of those lessons are
implemented by other teachers. The new state testing system
does not work. How can you judge schools when there is no
support or back-up. Students do not do assigned homework.
They are apathetic in their studies and there is too much value
on being in the socially in crowd. There is a student mentality
at RVHS that it is not cool to be too successful. As a school,
we are trying to bend them into having high expectations
because right now their expectations are too low. Also my
department still does not have standards-based text books. The
state and the textbook publishers have not produced them yet.
We have to wait until at least next year.
Interestingly, administrators and all o f the teachers admitted that there was
very little connection between student grades and academic content standards. One
teacher commented that “in theory we are standards driven but in reality we are
really not.” Another teacher stated,
Grades and report cards do not reflect attainment o f standards,
but the fact that students show up to class and do the work
assigned to them. I would think academic standards are part of
student’s grades but we really have not spent much time talking
about performance standards to measure how well students
understand or can demonstrate mastery particular standards.
One house principal concurred and stated, “student grades are a reflection
of teacher demands. The most often represent the subjects and topics
152
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
covered by the teacher.” She continued and stated that most of the grades
were representative o f what was tested. She thought it was desirable to
establish some common district-wide assessments much like they had done
at the elementary level.
River Valley High School’s faculty, students, administrators and parents
participated in a SALT-like survey used to capture important school-based
processes modeled after Rhode Island’s, School Accountability fo r Learning and
Teaching (SALT). The survey measured specific learning support indicators used to
examine and capture the various and complex elements existing in schools. In later
part of the 2003-2004 school year RVHS took the survey. The total number of
teachers responding to the SALT-like survey was 144 out o f a teacher faculty of
162. Results were returned to RVHS in September 2004. Figure 3 describes
teacher’s self-reporting rating o f attitudes toward educational practices when asked,
“How much do you agree or disagree that each o f the following practices is essential
to effective education in the grade level(s) that you teach?” The scaled score was
obtained by the average responses to each item on the scale. The scores on the chart
were based only on the responses o f classroom teachers. Teachers at RVHS
“agreed” that standards-based instruction was important arid rated it slightly behind
reading skill development and integration across the curriculum. It is interesting to
note that the teachers’ attitude towards standards-based instruction does not
necessarily teacher match practice as closely as one might have expected.
153
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Figure 3
T e a c h e r R a tin g s o f
ATTITUDES TOWARD EDUCATIONAL PRACTICES
Question: How much do you agree or disagree that each of the
following practices is essential to effective education in the grade
level(s) that you teach?
Strongly
Agree
Neither
Agree nor
Disagree
Disagree
Strongly
Disagree
4 .0
3.0
2.0
3 .8
4 .3
3 .7
4 .1
1.0 Community-
based
learning
opportunities
Reading skill
developm ent
and
integration
across the
curriculum
Mathematical Standards-
skill
developm ent
and
integration
across the
curriculum
based
instruction.
Teachers also reported on the SALT-like survey that they used standards-
based practices “monthly” up to about “several times a month.” Teachers reported
incorporating standards-based practices for “literacy instruction” and for
“numeracy” at the frequency rate o f several times a month. The reported about
“monthly” for “cross-content area standards-based practices.” The reported their
154
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
lowest rating on classroom practices in “standards-based practices for applied
literacy: analysis and interpretation” at a frequency o f between several times a year
and monthly. For each of the classroom practices identified above, teachers could
have rated their use at higher rates of, “weekly,” “several times a week,” and
“daily.”
The SALT-like survey revealed other interesting data regarding standards
use and implementation. Teachers reported that in their house, on an average
“curriculum coordination practices,” and “coordination o f student assignments,
assessments and feedback,” take place “once a year.” Teachers also reported that,
on average, the activity o f “development o f performance standards” takes place
“several times a year.” These data provide a triangulation o f consistent information
reported at the district, school and classroom levels.
The superintendent and the CEO reported that for the last three years
protocols from the Institute for Learning (IFL), University o f Pittsburgh have been
used to improve teaching practice. One o f the procedures used by the IFL and
RVHS faculty and administration was the “learning walk.” Participation in the
Teaming walks” provided time and opportunity to see teaching in an authentic
setting as it occurred in a daily fashion. Administrators reported that the “learning
walks” were non-evaluative. Teachers stated that the “learning walks” gave them a
chance to “see their colleagues in action.” Some teachers used what they had
learned during the “learning walks” to design, plan and implement professional
development. The professional development was usually provided in the house staff
155
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
meeting and they usually revolved around instructional strategies that could be
useful across the curriculum. Teachers reported that the use of Cornell Notes were a
perfect example o f a strategy tool resulting from a discussion from the learning walk
activity. Participating teachers perceived a need to improve students’ note taking
skills in all classes as this would be an important skill for all students.
The SALT-like survey also provided information about the type of
additional staff development wanted or needed by teachers. Teachers were asked to
rate on a 5-point scale where a score of; 1 indicated “none;” 2 indicated “little”
(e.g., single workshop/in-service); a 3 represented “moderate amount;” 4 indicated
“much” and 5 indicated teachers’ strong desire for staff development, “very much.”
Table 9 shows teacher identified professional development o f the five highest
ranking and the five lowest ranking responses. An interesting finding of the survey
was that teachers desired “little” future staff development in the area o f data-based
decision making. In fact the survey indicated that data-based decision making was
the least desired staff development of the options available for their selection.
Table 9 Rank o f Teacher Identified and Desired Staff Development
Rank Professional development topic desired Mean N
1 Strategies for teaching broad range ability levels in the
same classroom
3.40 103
2 Active “hands-on” learning 3.31 100
3 Using computers as part of instruction 3.29 102
4 Reading skill development 3.14 107
5 Developmentally appropriate instructional methods 3.12 106
39 Service-learning 2.42 106
40 Integration of mathematical reasoning and concepts
throughout the curriculum
2.39 100
41 Data-based decision-making 2.28 103
156
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
At RVHS, administrative responsibilities were divided among the various
administrators. Each administrator had some academic department responsibilities
and some school-wide administrative responsibilities. One o f the house principals
interviewed was charged with “test score data analysis” and with “grade analysis.”
This administrator exhibited great proficiency at manipulating the school’s student
information system. During the interview he created a variety o f reports on demand.
He identified, sorted, disaggregated and manipulated student achievement data in a
variety o f formats. He could create reports by class, grade, teacher, student
characteristics, achievement levels, tests taken and a host of other criteria. He had
been trained by personnel in the Information and Technology department on several
occasions. He admitted that he was extremely computer literate before coming to
the school as a newly hired administrator two years ago. When this research first
requested an opportunity to speak with the Chief Educational Officer o f RVHS, she
recommended that I work directly with this other administrator who had “data
analysis” responsibility. He clearly had her confidence and she relied on him almost
exclusively to produce reports for her consumption and for any data other
administrators or faculty needed. During the interview he stated that all
administrators had been trained on the student information system but that he had
was probably the most proficient and most other administrators relied on his skills.
Teachers also admitted that this particular administrator fulfilled requests for data
and that he occasionally reported out at whole school faculty meetings data that
either been requested or identified as showing interesting information.
157
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The “data” administrator described an incident resulting from a question the
superintendent posed to the faculty. The superintendent asked, “Why were so many
students o f color and low socio-economic status (SES) failing classes?” The faculty
responded that the answer was in the student attendance records. Teachers stated
that students who failed classes or received low grades were the students who were
most often truant or absent from class. To test the hypothesis, this administrator
created a report composed o f student absentee rates, grades and ethnic background
and SES status. The sorted data revealed that the majority of low grades and failing
marks were given to students of color and low SES but who were actually in class.
Some teachers he stated were clearly irritated by the report. The report also caused
both teachers and administrators to question the phenomenon o f some primarily
White, well to do, students missing classes - high absentee rates - and yet still
earning high grades. The superintendent posed a similar question later in the year
when he raised the question, “how come, so many student pass the CSTs, the
CAHSEE and the CAT-6 with at high levels o f achievement but fail classes?” These
questions led to house discussions about the level o f academic rigor of teacher
assignments and the need to better support the instructional needs o f Black and
Hispanic students.
RVHS used a variety of methods to keep parents informed o f their youth’s
academic progress. The most readily available means o f communication was via the
internet. RVHS used a computer grading and reporting program called Pinnacle.
Teachers used Pinnacle as their electronic roll book and grade book. The grade data
158
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
were housed on a district server and available to both students and parents. All
parents needed to do access their son’s or daughter’s grades was log on to the
server, enter their youth’s name and personal identification number. This tool
allowed parents an access 24 hours a day. Teacher reported that computer program
was fairly easy to use and it eliminated the hassle with bubble sheets for both
progress reports and report cards. The school website also had a link to
communicate important news to parents about various issues but testing information
and college application information was often posted. One teacher commented that
the web site and computer program was great for parents and families that had
access to a computer. She said,
Sometimes people still don’t get it. We assume that every
family has access to a computer and the Internet. I would say
that most families do in this area but there are still families,
even in this area that do not have a computer. They are poor
and barely make ends meet. If we are going to close the
achievement gap, we have to be cautious about our
assumptions.
RVHS used traditional methods o f informing parents o f individual student
progress and o f whole school academic achievement progress. Parent newsletters
went home in English and Spanish. The CEO and the house principals informed
parents o f the school’s progress and expectation in meetings with several parent
groups. Back to school night was another opportunity used by the school
administration to inform large numbers o f parents. The CEO and the two house
principals thought that they had utilized multiple lines o f communication with
parents and that they had reached as best as they could a majority o f parents. The
159
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
school has a strong relationship with the local community newspaper and student
achievement results are often reported out to the community through the newspaper
as well.
The most important communications sent to parents address interventions
the school recommended as a result o f student achievement scores on STAR testing
instruments. Parents were frequently informed through letters sent home with
students about the opportunities available for additional academic help. The STAR
testing instruments provided academic advisers enough data to appropriately
program students into a class schedule. More importantly advisers identified
students who could benefit from the many intervention and special assistance
programs the school had in place to raise student achievement. Students could
receive additional help from a satellite program run out o f UCLA, called the
“Student Outreach Program.” The program provided free tutoring for RVHS
students who needed help primarily in math but tutors were available for other
subjects as well. Students could also have access to after school tutoring created
from district funds and the district’s foundation. The PTSA also funded Xinachtli an
after school tutoring program for advanced math and science students.
District officials and RVHS faculty reported that AVID (Advancement Via
Individual Determination) has become a very important support program to raise
student achievement and to close the achievement gap. Students who were just
below average or average based on grades were encouraged to participate in AVID
and parents were notified about the availability o f the program in the student
160
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
newspaper, special mailings and through the Internet. While in the past AVID
classes had a small enrollment at RVHS, it has become very well known and many,
many students now enroll. House principals reported that they see improvements in
student attitudes and grades for AVID students. They thought that it might help
students “over the hump” for who were mostly receiving “C” grades.
Implementation of Emerging State Data Practices
California legislators and the Department of Education expect school
districts, schools and teachers to use data. Since the passage o f the Public Schools
Accountability Act the depth and variety o f assessments administered to
California’s students have increased and become more useful in measuring student
achievement. The transition from a single nationally norm-referenced test to a whole
battery o f standards-based criterion-referenced tests provided significant insight into
the status o f student achievement in California grades 2-11. The state plan for
measuring student achievement continued to emerge during the last two years and
there remained a lack of clarity regarding certain state administered assessments.
The rapidly approaching high-stakes consequences for students not passing the
California High School Exit Examination (CAHSEE) continued to be a point of
controversy. As late as March 2005, some state legislators had proposed legislation
to delay holding students accountable for the CAHSEE until adequate and equitable
resources and opportunities to learn were in place. Many school policy analysts
predicted that a previously enacted delay would encourage a second delay as the
revised deadline approached. Some school districts, schools, teachers and students
161
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
have believed that the state will never hold high school students accountable and the
proposed legislation reinforced that perception and belief.
The state also introduced for the first time an English-language arts
assessment to measure 11th grade students’ readiness for university level work. The
test was administered and provided students who took the additional multiple choice
questions on the 11t h grade CST for English and who wrote the additional EAP
essay an opportunity to become eligible if they scored high enough to become
exempt from California State University placement exams. While the assessment
was purely voluntary and there were no consequences attached to the newest
assessment administered by the state, the potential existed for it to become a future
component o f the API. Universities could continue to press for students who should
enter the system academically prepared for university level work. In a time of
shrinking fiscal resources, it seems logical that universities would want to work
more efficiently and not duplicate effort, responsibility and resources on matters
originating at the K-12 level of instruction.
The superintendent of Valley Unified School District was not willing to risk
the district’s reputation or place in jeopardy its students’ future opportunities by
ignoring these emerging trends and assessments. The effort to prepare students for
university work as the superintendent noted begins at home and must be fostered as
students move through the school system. He thought it improbable that schools that
waited until their students entered high school would ensure all students would be
well prepared for high stakes assessments required for a diploma or for university
162
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
admission. Consequently, he believed that the entire school system had to
concentrate its efforts early and continuously to raise the levels o f student
achievement while simultaneously closing the achievement gap. To accomplish this
effort he and his staff put in place at the elementary level and at the middle school
levels, periodic assessments in English-language arts and in mathematics. He
believed that starting early was essential. At the high school level he mandated that
all students take a CAHSEE pre-test at the 10th grade.
River Valley High School’s pass rate for the CASHEE for the 2003-04
iL
school year was excellent. The aggregate pass rate for 10 grade students in
mathematics was 88% and 90% for English-language arts. Both pass rates exceeded
count and state averages. The student newspaper even reported on the successes of
RVHS students on the CAHSEE. After the initial excitement waned, a closer look at
the disaggregated scores by school administrators and the math and English
departments revealed that an achievement gap continued to exist. Although RVHS’s
English-language learner (ELL) population and socio-economically disadvantaged
students outperformed the county and state students o f similar characteristics, the
scores were still significantly lower than those o f White and Asian students. The test
results were analyzed and students who did poorly were identified and received
mandatory additional assistance and services.
In preparation for the coming CAHSEE, RVHS administered a CAHSEE to
all grade 10 students. The CEO stated that,
The test results indicated those who would likely pass, those
who were close and with some tutorial assistance had a chance
163
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
to pass on the first time, and those who were struggling and
would need more intensive long term intervention. They were
scored, so the District determined break points for these
categories. The decision was to provide intervention and
support to those who, with support, we thought would be able
to pass on the first time...and we contacted students and parents
to get them to attend the support classes. We did not provide
support for the students we predicted would pass, and we did
not provide additional support for those far from passing, likely
special education students who are already supporting
intervention with their program.
The EAP assessment opportunity prompted a different response for the
English department staff at RVHS. In anticipation of receiving the scores from the
EAP assessment, the department decided to create a new class for seniors who
wanted to improve their preparedness for university work. Student demand for the
class was so high two full sections were opened in September.
The district has not yet identified or developed periodic assessments at the
high school level although the superintendent indicated a strong desire to do so. He
believed that the periodic assessment work at the elementary level improved
teachers’ effectiveness and provided opportunities for mid-course corrections. In the
absence o f any district periodic assessments, only the English department had
developed some common assignments and assessments. Implementation of these
assignments and assessments appeared to be uneven and inconsistent across the
school.
Accountability for data use at district, school and individual levels
The superintendent, RVHS’s CEO, the school’s administrative staff and the
teachers interviewed all thought that the State held most districts accountable. They
164
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
questioned the sincerity of a system that held school districts accountable without
properly and fully funding schools. They raised concerns about the unpredictable
nature o f school funding in California. RVHS for example has had to wait until the
state passed its budget before the CEO was permitted to hire new teachers to fill
vacancies that had occurred the previous year or during summer. The CEO
expressed great concern because by the time she was able to hire, all o f the best
teachers had signed contracts with neighboring school districts.
Nearly everyone recognized the benefit of the state’s testing and reporting
system. Almost all interviewees stated that at the very minimum, the STAR system
forced schools to pay attention to the academic needs o f Black, Latino and students
who lived in poverty. They thought the data indicating an actual achievement gap
confirmed their experiences and helped them to recommit towards its reduction.
The superintendent stated that principals were required to understand and
use data. He noted a geometric increase in discussions and in the ability levels o f his
administrators in the analysis and use o f data. He specifically stated that principals
who did not use data to raise student achievement would not have their contracts
renewed. Since meeting with principals to discuss their job performance goals, one
o f which was to better use the data available to them, he has found no real
objections.
The CEO and the several house principals said they hold teachers
responsible for raising student achievement levels for all students. The teachers
interviewed expressed the same sentiment. While teachers did not explicitly feel
165
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
compelled to use API, CST, CAT-6, CAHSEE, CELDT or any o f the other data
available from state tests, they did feel administrators held them accountable for
using state content standards. They said they would occasionally have their lesson
plans reviewed to see if they were teaching to the standards. Teachers said that there
were no consequences for not using the standards to guide instruction if they had
valid reasons. One teacher said, “Administrators do their job, and they do their best
to keep teachers on track.” Some administrators are better than others in this regard.
One teacher best described instructional practice at the school when he said,
We use standards to guide our instruction, we do not march
lock-step and if you were to look at all o f the teachers teaching
the same grade and class you would find that teachers were
teaching at different places in the curriculum. That is not
necessarily bad because not every teacher chooses to focus the
same amount of time on similar topics but I would say that I
have confidence that my fellow teachers were making sure
their students studied what they were supposed to study. Our
test score are pretty good compared to surrounding school
districts. I guess we are lucky in that respect, maybe if we were
identified as underperforming the story might be different.
The CEO stated that newly hired teachers were really accountable for good
standards-based instruction. She stated that she has not hired back numerous newly
hired teachers who were either poor teachers in general or were bad for kids. The
faculty has had significant number of teachers leave the school for a variety of
reasons including retirement, pregnancy, other job opportunities or because RVHS
was just not for them. The instability caused by such large teacher turnover forced
administrators to concentrate their efforts on newly hired teachers rather than
veteran teachers.
166
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The one house principal who was responsible for data analysis said that he
actually looked at the grades teachers assigned students. He shared a teacher
evaluation that had been cleansed o f any sort o f information that was confidential
and pointed out how one teacher under his supervision had been counseled about the
large number o f fails issued in her class. He said he used the opportunity to bring
the issue to the attention of the teacher and to help make her think about how she
might better reach the students she was failing.
Interestingly enough the school board member this researcher interviewed
thought that the issue of accountability as framed by the state was not the driving
force for improved student achievement. She noted that the community she
represented expected the superintendent to lead the school district to higher student
achievement and to close the achievement gap not because o f any state mandate but
because it was the morally right thing to do. She thought the community more than
anyone or anything else held the superintendent, his staff, the schools, the teachers
and the students accountable. “There is no way” she said, “that the community
would have supported the school district like it has if they felt the schools were
doing a poor job.”
Several teachers interviewed were concerned that students were left out of
the accountability discussion. There appeared to be some frustration about students,
primarily o f color who just didn’t give a care to learning anything in school. They
were there to socialize and to be seen. One teacher didn’t blame the students so
much as he blamed the parents for not setting some boundaries for their son or
167
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
daughter. He spoke about teachers trying to maintain “high expectations for all
students” but felt unsupported by parents who felt that their kid was being singled
out. In essence there was some resentment among a couple of the teacher who felt
students should work hard and care more about their studies than they do about
distractions outside of school. Confidentially, one administrator confessed that more
than a few teachers felt that way. The administrator also noted that it was really
difficult to make such a broad generalization. The administrator pointed out that
many minority students were enrolled in intervention classes, participating in after
school tutoring programs and were working hard and their hard work was leading at
least to better grades.
Improving Student Achievement through Implementation o f Data Use
The superintendent articulated a fundamental belief that the use of data
would raise student achievement. He believed if schools and teachers had to have an
accurate picture o f the current state of achievement if progress was going to be
made. He acknowledged that all of the support systems had not been put into place
across the district but the district was moving forward. He held his Chief Academic
Officer and his Director of Educational Services in the highest regard and praised
their efforts to build a deeper awareness o f how students at each school were
performing on the multiple state assessments. He reiterated that the district was
probably realizing a great return on data utilization at the elementary level where the
district has provided significant support in resources and technical assistance. The
same would be true for one middle school that faced greater challenges that the rest
168
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
of the middle school within the district. As for the high school the superintendent
thought that the school redesign would lead to increased student achievement and
that the shift to six small schools facilitated the use o f data to raise student
achievement. He was very specific about setting benchmark targets and goals as one
way to focus school efforts. He stated,
Not only do you have to have an accurate assessment o f where
student achievement is at any one point, you need to know your
targets and be very explicit about how you are going to
measure progress. I think we have provided the public,
administrators, teachers and student with sufficient information
to move the agenda forward.
The Chief Educational Officer at RVHS agreed with the superintendent’s
plan of action. She thought that since her arrival the school has started to see a turn
around in student achievement. She pointed to the API and CST data that showed
improvement, especially improvement for African American, Latinos and Socio
economically disadvantaged students. Some o f the increases she thought could have
been attributed to procedural data they gleaned from previous test administrations
and the adjustments the school made as a consequence. She thought some of the
data the school had collected about the previous large comprehensive school had
been used to provide a more personalized learning environment for students. She
pointed to a number of targeted interventions for struggling students that data
showed made a difference:
• AVID - students enrolled in AVID classes earned higher grades and
reported greater self-efficacy by enrolling in more difficult classes.
169
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• Algebra-Biology block - students enrolled earned higher grades but only-
small test score gains on CST tests, students also had a higher attendance
rate.
• Literacy support classes - students showed improvement in grades but small
gains in test scores.
• After school tutoring - many students took advantage o f special tutoring
programs after school, little direct evidence was available to prove a direct
connection to increased test scores but students much have felt it made a
difference or else they would have stopped coming. Tutoring has increased
significantly over the last two years.
• Academic advisers in the small school house - prior to school redesign
student rarely met with their counselors, since school redesign, students
report they have greater contact with their academic adviser. Students feel
better connected to the school and feel that they are receiving better
academic advice.
One house principal was not so sure that the data collected and analyzed was
actually having that much o f an impact to raise student achievement. In fact, when
different groups use the data they sometimes come up with different conclusions or
recommendations. She used the example o f what student should take in summer
school.
We had an interesting situation this last summer. The central
office looked at the data for in-coming 10th grade students and
11th grade student who previously had taken the CAHSEE. The
170
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Chief Academic Officer sent out a letter to specific students
telling them to take the CAHSEE classes this last summer
school. The RVHS advisers looked at students’ grades in their
math classes and told them to take the classes they failed such
as Algebra I. Here is a perfect case where two groups are
looking at student math data and making contradictory
recommendations. I am not sure what most students did, I do
know that it caused some contusion.
Teachers interviewed spoke about using data differently to raise student
achievement. The teachers interviewed spoke about the use of the Principles of
Learning (POL). The POL provided a common language to speak about teaching,
learning and student achievement. Teachers reported that during the last year they
had learned the value of using rubrics and had started to incorporate them on a
regular basis into classroom instruction. One o f the house principals interviewed
reinforced the teacher’s sentiments, “ The POL are very nice, they have been the
vehicle through which we speak about instruction.” The POL “Clear Expectations”
component helped teachers to make sure students understood what was expected in
terms o f content to be learned, tasks to be completed, and work quality. Teachers
were divided as to value of the POL to instruction but they did believe that rubrics
helped to clarify for students what they needed to do to earn higher grades.
All five teachers reported that the most important and useful data were the
data provided about the specific students in their house or their classrooms. In fact
teachers were initially interested in the opening year professional development that
reported out test score gains on the CSTs and CAHSEE and the CAT-6, but they
reported getting ‘blinded by the amount” o f data presented. One teacher stated that
the numbers stopped being meaningful because they became abstractions. She said,
171
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
“I would have been more interested if I knew how my students from last year did on
these tests. At the very least I would have wanted to know how my newly enrolled
students did not last years tests.” Another case and point centered upon the biology
department’s desire to access student achievement data by student and class. At the
time of the request, the central office would have had that information did not have
the ability to provide what was being asked. The teachers became disenchanted that
they were being encouraged to look at data, they wanted to utilize data but they
could not get the data they wanted.
Barriers to Data Utilization
The superintendent described the district as “data driven.” School site
personnel agreed to some extent with the superintendent’s account. All o f the
administrators and all of the teachers interviewed were able to identify a number of
barriers or challenges that impeded data analysis and utilization. Three common
themes and barriers emerged as well as several unique barriers identified by one or
two teachers.
School site administration and faculty identified “lack o f time,” “insufficient
professional development,” and “competing reform agendas” as challenges that had
to be addressed if they were going to become better producers and consumers of
data. The lack o f time was usually the first barrier interviewees mentioned. They
thought that finding “adequate time” for teachers to meet and to work outside their
classrooms was the biggest factor that affected any attempt at improving student
achievement.
172
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Teachers remarked that their jobs “were tough enough” given all o f the other
things they were expected to do. Teachers preferred to concentrate their time on
planning for instruction and grading papers. Only one teacher made the connection
between planning for instruction, assessment and data analysis and use as non
exclusive, mutually beneficial processes. Another criticism voiced by teachers was
that their class sizes were too large. The move to small schools did not necessarily
mean a move to smaller classes. The push to “unwrap” or “unpack” the standards,
also consumed much of the department time allocated on banked professional
development time.
A second common barrier identified was the lack of sufficient professional
development on data analysis, and how to utilize the power o f the districts student
information and data system. Teachers and administrators reported that training on
using Pinnacle, the district’s electronic roll book was minimal, just enough to get
teachers to enter grades and get them working with the program. This research
asked four o f the five teachers to demonstrate the program and what it could do
besides record grades and not one could move beyond data entry and grade
calculations. The house principal who was responsible for data demonstrated the
power o f the program by disaggregating data by a variety o f categories such as
assignment types, students above or below a certain percentage or score. He ran a
few charts graphically showing the distribution o f grades in a certain class, and he
was able to compare two classes’ grades on a particular assignment and also their
total aggregate performance for a period o f time.
173
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The CEO and the other house principal reported that much of the data
analysis training came from the Chief Academic Officer at the district office. They
reported that most of the training involved hands on review o f reports and a
significant number o f PowerPoint presentations. Both felt that their training on
using the power to improve student achievement could have been more explicit.
Interestingly, the house principal noted that the best training she received was from
the superintendent. She believed in this district, “you leam by the superintendent
way. He models and explains data very well.”
The third impediment identified by teachers and the administrators
interviewed was the complexity o f school redesign. School redesign as most
described it, “was hard work.” One administrator confessed that, “It takes a lot of
time and energy to change a school and faculty that was set in its ways.” Another
administrator said that, “The house system is ending the reign o f the untouchables.”
The most striking consequences o f redesign as reported by a significant number of
teachers who took part in the Second Year School Redesign Survey were, “low
teacher morale” and “a feeling o f isolation from fellow department members.” The
CEO noted that some o f issues that arise from the difficulties o f implementing such
a huge reform will just take some time to work out the glitches. You can’t do
everything at once.
Summary o f Research Findings for Question Two
Research question two asked, “To what extent has the district design been
implemented at the district, school and teacher level?”
174
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The district’s practice of data collection, analysis and utilization varies in
levels of implementation as delineated by organizational structure, employee job
responsibility and by academic subject.
Implementation o f the district’s design for using data was best established at
the central offices o f the district and at the elementary school site levels where
resources and data analysis tools were made widely available. The establishment of
periodic assessments at the elementary level required technical support to enable
site administrators and school lead teachers opportunities to understand what the
data exhibited. Elementary school site administrators received multiple
opportunities for professional development on the topic o f data collection, analysis
and utilization. Although the superintendent and central office staff were available
to high school administrators, the level of support, based on the fact that the district
had not yet implemented periodic assessments was significantly limited.
Implementation o f the district’s data model at RVHS reflected the evolving and yet
complete status of the district design.
Utilization o f the district’s data model or the available features of the model
remained situated in the context of job related responsibilities. Certainly the
superintendent, and the central office accessed and utilized student achievement
data frequently and for a variety of purposes including, monitoring student
achievement at the district and at individual school site levels. They accessed
student achievement data on a classroom level to identify students who need special
assistance. They analyzed disaggregated data to monitor student achievement in an
175
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
effort to close the achievement gap and they used student achievement data to
develop policy and to engage in strategic planning.
School site administrators at RVHS also used the district’s data model
features that were available for the high school level. They utilized student
achievement data from the CSTs, the CAT-6, the CELDT, the CAHSEE and EAP.
Additionally the school site administrators utilized student achievement data from
Advanced Placement assessments, teacher grades, school survey’s o f teacher and
student attitudes and classroom practices. The administration primarily utilized the
data in several ways, specifically to identify and enroll students into instructional
support programs such as tutoring, special summer school classes, AVID, or special
programs such as the algebra I/biology block class. Certain house administrators
used data as a result o f specifically assigned responsibilities including preparing for
a future WASC accreditation visit, to complete grant reporting responsibilities and
as the designated data administrator.
Teachers used some o f the same data identified in the district’s data model.
In most cases the data was mostly reported out to teachers although they were ask to
examine the data to see how they might plan instruction differently based on the
data. Data sources most commonly identified were the CSTs and CAHSEE.
Teachers felt informed about what the state assessment data indicated about student
achievement at RVHS but most teachers felt it remained at an abstract level unless
they were given specific information about their previous students or the
achievement scores o f their current students. Teachers did acknowledge that they
176
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
looked more specifically at test scores when they met to discuss individual students
in house meetings or when they participated in student success teams. Teachers
identified student grades as being the most relevant data they regularly collected and
analyzed.
A second finding indicated that the explicit collection o f data resulting from
various surveys and other instruments at the school site level leads to an expectation
by teachers and administrators that the data will lead to utilization for the purposes
of improving student achievement and improved educational experiences for all
students.
VUSD and RVHS collected data from a variety o f surveys and
questionnaires. Teachers reported that some surveys such as the SALT-like survey
were very time intensive and questioned their utility. Other surveys such as the
SCORE survey on school redesign asked for teacher’s perspectives. Several
teachers expressed concerns that the surveys did little to change the practices on
campus even though the data clearly indicated there was a problem. The
administration felt that these surveys yielded useful data to monitor the attitudes and
expectations of teachers, students and parents especially about school redesign and
perceptions o f teaching and learning. The District’s model, through the use of
surveys took into account contextual data about the learning environment and the
processes of teaching and learning that they believe might also affect student
achievement.
177
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Research Question Three
The third research question asked, “To what extent was the district design a
good one? A Researcher’s Observation and Rating Form and a Rating Matrix was
developed by the parallel dissertation team and two Ph.D. students to determine the
degree to which the District’s design is a good one. The three Rating Matrixes
addressed:
• District Support of Standards-based Instruction and Assessment
• District and School Accountability to Standards-based Curriculum
• Determination of “High” Student Performance
Each matrix had its own criterion for determining the extent to which the
District addressed the topic identified above. The matrix used a 1-5 rating scale for
each o f the criterion. A researcher rating o f 1 described Districts effectiveness as
“not effective.” A rating of 2 indicated the District was “somewhat effective,” a
rating o f 3 indicated that the District’s effectiveness was “unclear” as a result of
insufficient data to rate and a 5 rating indicated a design that was “very effective.”
District Support o f Standards-based Instruction and Assessment
Table 10 addressed elements of Conceptual Framework A (CF-A), the
District’s support for standards-based instruction and assessment. The rating score
takes into account the District’s intended impact o f the design and the observable
evidence at River Valley High School. A score was awarded based on the
researcher’s judgment and analysis of the design and observed evidence.
178
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 10 District Support o f Standards-based Instruction and
Assessment
Criterion 4* Rating Score ^ 1 2 3 4 5
Clear performance goals
(aligned to or higher than state goals)
X
Disaggregated standards-based assessment data X
Inservice on how to use data X
District is preparing schools for emerging
State assessments
X
1. Clear performance goals: This researcher assigned the District a score of 4.
The Board of Education at Valley Unified School District and the
Superintendent adopted student performance goals for students at all grade
levels based upon State and District administered assessments, performance
growth rates and targets in mathematics and English-language arts
assessments and college readiness. Administrators and teachers were well
informed o f the performance goals and received at faculty meetings and
through district communications. It was unclear to this researcher whether or
not students were well informed o f the Districts performance goals.
2. Disaggregated standards-based data: This researcher assigned the District a
score o f 5. The District’s mantra o f raising achievement levels o f all students
while simultaneously closing the achievement gap permeates every
discussion and presentation about assessment and student achievement.
River Valley High School has been monitoring a “Two Schools”
phenomenon since the mid 1990’s. The faculty was well informed about the
179
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
gap that existed between ethnic subgroups, student language groups and
student identified socio-economic status. The District provided to RVHS
student achievement data in a variety o f formats, such as District reports,
PowerPoint presentations and through a variety inservices. The school
district and RVHS complied with state education code by providing a School
Accountability Report Card (SARC) available on the Internet. The SARC
provided the public with disaggregated data on gender, ethnicity and SES.
3. Inservices on how to use data: The District received a score o f 3. The
District provided numerous professional development opportunities on how
to use data. At the beginning o f the year the District provided each school a
professional development workshop to report the school’s most recent data
and how to read and read the data. Throughout the year the District provided
optional professional developments on how to use data at after school
workshops and weekend trainings. Very little evidence was collected as to
the extent of which RVHS faculty or administrators participated in the
professional development opportunities provided by the central office.
Teachers reported on the SALT-like survey that workshops on data use were
not highly demanded.
4. District is preparing schools for emerging state assessments: The District
received a score o f a 5. The District implemented a plan to prepare for
emerging State assessments. Specifically the District has instituted a
mandatory 10th grade Pre-CAHSEE examination to assess student
180
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
preparedness for the CAHSEE. Students who were identified as not having
performed well enough to pass were required to take support classes to
increase the likelihood that they would pass the first time they take that
actual CAHSEE. The District and RVHS encouraged students to take the
Early Assessment Program (EAP) and RVHS established two sections of
English for 12th grade students who wanted to improve their readiness for
university work after finding out that they did not do as well as they had
hoped. RVHS also established an algebra-biology block to provide students
with a combined time o f three periods o f instruction rather than the
traditional two periods used to teach the classes.
District and School Accountability to Standards Based Instruction and Assessments
Table 11 addressed elements o f Conceptual Framework B (CF-B), the
District’s and schools accountability to standards-based curriculum. The rating score
takes into account the District’s intended impact o f the design and the observable
evidence at River Valley High School. A score was awarded based on the
researcher’s judgment and analysis o f the design and observed evidence.
Table 11 District an School Accountability to Standards-based
Curriculum
Criterion ^ Rating Score 4 1 2 3 4 5
Motivation to use standards-based assessments and
instruction
X
Regular assessment of student performance X
Collaborative teams review data X
Instructional strategies individualized to promoted
learning
X
181
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1. Motivation to use standards-based assessment and instruction: VUSD and
RVHS received a score o f 3 indicating insufficient information to determine
rating. District policy and directives indicated that the District was highly
supportive of standards-based instruction. Teachers at RVHS “agreed” on
the SALT-like survey that standards-based instruction was essential for an
effective education. However during the situated interviews with
administrative staff and with teachers there appeared to be an inconsistency
between what was supposed to occur and what was actually practiced. The
variation o f practice and utilization o f standards versus teacher attitudes
towards the importance o f standards appeared to be related to academic
content taught. English-language arts, mathematics and science teachers
reported a greater tendency to be standards-based whereas social science
teachers were somewhat apathetic to the power of standards. Additionally,
there did not appear to be a strong accountability mechanism in place at
RVHS to encourage teachers to provide standards-based instmction. Data
collected from the SALT-like survey also reflect an inconsistency between
teachers’ attitude towards standards-based instmction and actual teacher
practice. Teachers reported using standards-based practices at the rate of
only “several times a month” rather than “daily” or at least “several times a
week.”
2. Regular assessment o f student performance: Under this criterion VUSD
received a score of 2. While the VUSD provides system and periodic
182
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
assessment at the elementary school level it has not provided similar support
at the high school level, thus leaving high school teachers to their own
ingenuity. Practice at RVHS varies between academic departments.
Uniformed end of semester standards-based assessments exist for English-
language arts and mathematics classes and appear to be used as designed.
End of semester standards-based assessments also exist for some science
classes and for history classes; however they did not appear to be used
universally throughout the department. With the teachers reporting limited
standard-based practices, it would appear unlikely that they would engage in
regularly monthly assessments and tend to provide occasional standards-
based assessments.
3. Collaborative teams review data: This researcher assigned the District a
score o f 2 for this criterion. First and foremost this district and school does
not appear to have a collaborative teams approach to using data. Certainly
the District administrative staff and Superintendent met with school site
administrators at regularly scheduled meetings to discuss student
achievement data, but RVHS did not appear to have a data team in place.
Rather, individuals were assigned job responsibilities that directly related to
data utilization. One house principal was identified as the lead
“administrator for data.” Another house principal was identified as the house
principal for “grades and assessments.” A lead teacher and another house
principal were responsible for WASC accreditation and grant reporting
183
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
responsibilities. Teachers in contrast used house meetings to discuss the
achievement levels o f individual students but it was unclear at to how often
and under what circumstances. Data analysis and utilization appeared to be
fragmented and job responsibility dependent.
4. Instructional strategies individualized to promote learning: The District and
RVHS received a 2 for their effort in providing individualized instruction to
promote learning. The District has in place the potential to promote such
practice at the elementary level but does not have any similar strategies at
the high school level. The teachers at RVHS commented that class sizes
were too large to provide individualized instruction. Students who needed
extra help or specialized instruction were provided additional assistance
through special tutoring programs available after school or on weekends.
Teachers reported also reported that some students did receive
individualized instruction again through special programs and that some
programs utilized special instructional strategies to promote learning. The
program most often cited by teachers and administrators was the AVID
(Advancement Via Individual Determination) program. Teachers reported on
the SALT-like survey that they provided students occasional opportunities
each reporting period to pursue topics of individual interests. This was done
by assigning research projects or reports but these assignments were not
specifically aimed at providing students formal instruction in skill areas that
were identified as needing improvement.
184
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Determination of “High” Student Performance
Table 12 addressed elements o f Conceptual Framework C (CF-C), the
District’s determination o f “high” student performance. The rating score takes into
account the District’s intended impact o f the design and the observable evidence at
River Valley High School. A score was awarded based on the researcher’s judgment
and analysis o f the design and observed evidence.
Table 12 Determination o f “High” Student Performance
Criterion ^ Rating Score ^ 1 2 3 4 5
High Performance Rubrics X
Teachers, students & parents understand standards-
based curriculum
X
Report cards aligned to state standards X
All student progressing towards high performance X
1. High Performance Rubrics: The District received a score o f 4 for this
criterion. Teachers at RVHS developed and used rubrics as part of regular
practice. Each academic department either developed their own rubrics or
modified other rubrics from other disciplines. Teachers reported and
produced rubrics on demand and the English teachers interviewed provided
sample student work graded by rubric. Teachers reported in the SALT-like
survey that they used rubrics on a quarterly basis. The administration
reported that in the previous school year the use o f rubrics was encouraged
through house meetings and professional development. No information
about parents’ understanding o f rubric use was available and no parents were
interviewed during this study.
185
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2. Teachers, students and parents understanding standards-based curriculum:
The District received a score o f 4 for this criterion. Every teacher
interviewed indicated that they understood the purpose and goals of a
standards-based curriculum. Additionally, they indicated that they had gone
through several processes in department meetings, professional development
and house meetings to better understand standards. Administrators promoted
the use o f standards as a means to increase student performance on state
assessments and as a way to close the achievement gap. Teachers were told
to teach using the standards but none o f the teachers interviewed said that
they used only standards to plan instruction. There was no evidence that
parents understood standards-based curriculum even though teachers said
that they wrote about standards in the course syllabus they sent home to
parents. The use o f standards in planning lessons and teaching remained
somewhat stratified by academic course and subject.
3. Report cards aligned to state standards: The District received a score of 2 on
this criterion. The District implemented a standards-based report card at the
elementary school level as a pilot field test. There was no such movement at
the middle or high school level. The teachers and administrators interviewed
reported that grades were supposed to reflect standards-based curriculum but
they all admitted that at this time report card grades reflect what teachers
taught and what teachers taught was only partially aligned to standards in the
best of circumstances.
186
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
4. All students progressing toward high performance: The District and RVHS
in particular received a score o f 5. The student achievement data reported on
STAR indicated that all student groups improved their performance across
the spectrum or battery o f assessments. Socio-economically Disadvantaged
students and Hispanic made the greatest gains followed by African-
American students. Asian and White students made small gains but their
sub-group scores were already above 800. The District’s stated goal of
raising student achievement for all students while simultaneously closing the
achievement gap made realized significantly progress but the Superintendent
admitted that the District had much hard work and progress to go before the
goal is to be realized.
Summary o f Research Findings for Question Three
The extent o f the District’s practice of data collection, analysis and
utilization was evolving at the time o f this study. The District design supported
standards-based instruction adequately by setting clear performance goals aligned to
and higher than state goals. The District design was uneven in its use of assessments
to raise student achievement as a result of providing periodic assessments at the
elementary level but not at the secondary level. The District did exceedingly well in
providing schools disaggregated data, RVHS appeared to report disaggregated data
for multiple measures of student achievement, school climate and student perception
survey data. At the high school level the District instituted various measures to
prepare for emerging State assessments most notably the California High School
187
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Exit Exam and the Early Assessment Program o f college readiness in English-
language arts.
The District’s methods o f holding schools accountable to a standards-based
curriculum were somewhat effective but could be significantly improved. Teacher
motivation and utilization o f standards-based assessments were unclear. It appeared
that some academic departments were accepting standards-based instmction and
assessment while others were unenthusiastic. The uneven development of the
District’s data model provided for a range o f use o f regular assessments of student
performance that promoted effective use at the elementary level but resulted in less
effective use at the high school level. The District’s uneven development and
minimal oversight also allowed, at least at the high school, the faculty to remain
somewhat uncommitted to developing a cohesive plan for utilizing data teams and
for providing individualized instruction to students.
The quality o f the district’s collection, analysis and utilization of data at the
high school level appears to hold much promise and should be more than adequate
to raise student achievement. The use o f high performance rubrics at RVHS, the
realization by teachers that standards-based instruction was essential for an effective
education were all good signs that the District’s model was moving schools in the
right direction. Reporting student performance related to standards to students and
parents appeared underdeveloped and only partially aligned to the implicit
intentions o f using standards to raise student achievement. Nonetheless, a majority
o f students are progressing towards high performance goals with Socio-
188
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
economically Disadvantaged, Hispanic and African American students making the
largest gains as evidenced in River Valley High School’s disaggregated API scores.
Discussion of Research Findings
This study attempted to research the extent to which one school district’s
model o f data use to raise student achievement was adequate when compared to the
current and emerging context o f California’s education system. The three research
questions, and the research finding based on the evidence collected and presented in
this chapter frame the ensuing discussion.
Question One
What is the district design for using data regarding student performance and
how is the design linked to the current and emerging state context for assessing
student performance?
1. A results oriented district design to raise student achievement and to close
the achievement gap has been developed and articulated.
Valley Unified School District did not have an explicit Board adopted model
design for using data. However, the superintendent maintained an unshakeable
conviction in the power o f data and was deeply committed towards establishing a
culture o f data analysis and use throughout the district at all levels.
The unofficial data model design the District utilized included a rich
collection o f multiple student achievement measures that included the all of the
State Accountability and Reporting (STAR) assessments, as well as a host of
District identified student achievement indicators and performance goals. The
189
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
intended results o f the monitoring student achievement and using the data from
multiple measures was explicitly stated to “raise the student achievement levels o f
all students while simultaneously closing the achievement gap.” Rallis and
MacMullen (2000) argued that much of the data schools needed to improve already
existed because o f external mandates or because o f routine recording. Much of the
data Valley Unified School District and River Valley High School used was easily
available and often provided as a result o f State and District policy.
The Superintendent and the Board o f Education adopted specific student
performance goals for all students k-12. Valley Unified School District
implemented school reform and avoided a fate common to previous reform efforts
by maintaining an explicit focus on improving student performance (Marsh, 1996).
The goals were discussed with each school principal who then shared the goals with
the faculty through meetings and other communication venues. In addition to
collecting student achievement data the district and the high school utilized a self-
study tool that provided schools with reliable and systematic information used in
planning and monitoring school improvement efforts. It provided a mechanism for
making important decisions by raising key questions that led to deep engagement of
all staff in the school's transformation. Goals, Schmoker (2000) suggested “lead not
only to success but also the effectiveness and cohesion o f a team.”
The focus on student achievement led to an effort by some academic
departments at River Valley High School to align instructional materials,
instructional practices and in a few cases in-class teacher developed assessments
190
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
with State identified academic content standards. Carr and Harris (2001) noted that
standards done well would address a variety o f areas of teaching, learning and
running a school. The belief among teachers at RVHS that standards matter forced
some teachers to look at what they were teaching and to modify their instructional
practices. Although the practice o f aligning planning lessons, teaching, learning and
assessing student work were not common for all subject areas or teachers, the
expectation permeated most school discussions and decisions.
Scott Thompson (2001) believed that “authentic, standards-based reform is
fundamentally concerned with equity.” The Superintendent, the CEO and the faculty
at RVHS have long acknowledged an achievement gap, and the RVHS in particular
has had an unenviable distinction o f maintaining a “two schools phenomenon” o f
inequity and unequal access. Rather than just being aware o f the inequities, RVHS
redesigned, instituted specific actions to increase the number o f Hispanic, African
American and Socio-economically Disadvantaged students in AVID classes, in
tutoring programs, in special assistance programs and into Advanced Placement
classes. Additionally, the school district funded and administered a pre-CAHSEE
assessment to identify students who were unlikely to pass the CAHSEE on the first
administration and provided tutoring and special assistance. The planning for
emerging state assessments was in a sharp contrast to previous practice of providing
“after-the-fact” assistance, as one administrator stated.
After three years of declining student performance indicators. RVHS
students in 2003-2004 performed exceedingly well on state assessments. Socio-
191
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
economically Disadvantaged, Hispanic, and African American students showed
strong gains in student performance, some o f which must be attributed to the
policies and practices in VUSD and at RVHS.
Question Two
To what extent has the district design been implemented at the district,
school and teacher level?
2. The District’s practice of data collection, analysis and utilization varies in
levels of implementation as delineated by organizational structure, employee
job responsibility and by academic subject.
The District’s design was not fully developed at all level o f the organization.
Consequently, implementation was uneven and coherence throughout the system
was weak. Tucker (2002) argued that coherence o f the instructional program was
essential to achieve high student performance across the board. The District design
was best implemented at the central offices where the Superintendent and his staff
collected, reviewed and analyzed student performance data frequently. The
Superintendent was well aware of the inadequacies o f the data model for secondary
school leaders and teachers but he did not excuse them from using the data they had
available to them, including teacher assigned grades. The Superintendent on several
occasions raised questions about the relationship between teacher assigned grades
and the STAR battery o f assessments implying that more could be done to raise
student performance levels.
192
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Abundant evidence existed to suggest that the data model at the elementary
level was strongly supported by the District and that it was having a beneficial effect
on student learning. The incorporation of district wide periodic assessments
provided student achievement data to the Superintendent, to elementary school
principals and to specific classroom teachers about the progress their students were
making in the content areas o f literacy and mathematics. The Superintendent
reported that at the elementary level if teachers and administrators were using the
District’s data model, they would be meeting on a regular schedule to plan
instruction, make mid-course modifications to remedy identified patterns of student
underachievement and to identify individual students who needed special assistance.
The Superintendent’s frequent visits to elementary school sites confirmed his belief
that the data model o f using state test data, district mandated periodic assessments
and teacher judgment were all being used to raise student achievement.
Evidence collected at the high school level suggested that the District’s data
model was less well developed and only sporadically utilized. Teachers and school
site administrators interviewed indicated that some school personnel used data
relatively frequently and some school personnel rarely used data. An accurate
description o f the District’s data model would be very difficult to construct with the
exception o f stating that data use was somewhat dependent on explicitly identified
job responsibilities and subject areas taught. Teachers in general appeared more
interested in the student performance data on students previously enrolled in their
classes or currently enrolled students. General student performance data after being
193
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
initially reported to teachers was rarely examined in an effort to identify unique
distinguishable pattern in student performance.
Fullan (2000) warned that school reforms fail because we fail to understand
that both local school development and the quality o f surrounding infrastructure
were critical for lasting success. Hargreaves and Fullan (1998) also suggested that
teachers needed to become more assessment literate in an effort to build their
capacity to develop action plans aimed at altering instruction or other factors to
improve student learning. The minimal and situational use of available data at River
Valley High School suggests that the district’s model for using data remained fragile
and only marginally supported by teachers. Rallis and MacMullen (2000) suggested
statewide assessments do not always accurately measure local or state standards and
consequently schools remain unclear about how to use data. A change in the
utilization o f the District’s data model by teachers at RVHS would appear unlikely
given the current minimal support provided to RVHS by the District. Additionally,
only those staff members with a specific need or responsibility for seeking data and
utilizing data would continue to rely on the District model.
3. The explicit collection of data resulting from student achievement measures,
various surveys and other instruments at the school site level leads to an
expectation by teachers and administrators that the data will lead to utilization
for the purposes of improving student achievement and improved educational
experiences for all students.
194
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Rallis (2000) reported that schools are deluged by data but by themselves
these data do little in shaping school practices. Some teachers at River Valley High
School reported a frustration with the amount o f data collect and the time it took
away from more meaningful endeavors. Several teachers expressed concerns that
the various surveys and the data they yielded did little to change the practices on
campus even though the data clearly indicated that there were problems. Schmoker
(2000) argued that the survey data collected provided the school opportunities to
focus on short-term, local goals and “quick wins” to in the effort to achieve long
term goals. Nichols and Singer (2000) pointed out that merely gathering data was
not enough. Administrators and teachers must leam to analyze the data and apply
this information to the classroom.
School site administration and faculty identified “lack o f time,” “insufficient
professional development,” and “competing reform agendas” as barriers to the
consistent use o f data utilization. Tucker (2002) stated that the question of capacity
was the central question provoked by the standards and accountability movement.
Bernhardt (2000) noted that there were multiple reasons for not using data including
the lack o f training, equipment and time. Teachers and administrators expected data
to be utilized if it was collected. However, teachers appeared apprehensive to invest
too much effort in data use because they felt they were being asked to do too much
with what little time they had. Teachers cited systemic obstacles such as the lack of
professional development opportunities about how to use data, the lack of desire by
teachers for such professional development as reported in the SALT-like survey,
195
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and the amount o f work and effort at other reforms such as school redesign as
additional barriers. Fullan (2000) concurs by stating, “the main enemies of large
scale reform are overload and extreme fragmentation.” In reflecting upon the
evidence discovered in this study, this researcher believes that expectations for data
use collide with the realities and complexities of the school day and the work
demands of teachers to the point o f frustration and a feeling o f overload.
Question Three
To what extent was the district design a good one?
4. The extent of the District’s practice of data collection, analysis and
utilization was evolving but it had not yet created the conditions to transform
previous school culture.
The District’s data design supported-standards based instruction. Teachers
attended “Making Standards Work” a five day inservice designed to assist teachers
in understanding the standards and providing them with practical strategies to use
the standards. Teachers were well informed o f the academic content standards they
were to use to guide instruction. In some disciplines teachers used standards to
guide instruction, to select and to purchase instructional materials and textbooks and
to develop assessments that measured student achievement o f standards.
The Superintendent reported probably the best examples o f standards-based
instruction would be found at the elementary level. He admitted that much more
work and support had been directed at improving instruction and at improving
student achievement at the elementary level. The District’s progress on providing
196
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
standards-based instruction on literacy and numeracy was regularly monitored
through a variety o f periodic assessments. Teachers at the k-5 level administered
between 3 and 4 District purchased periodic assessments for English-language arts
and mathematics. The assessments were scored at the District office with individual
student scores reported by ethnicity, SES, gender, school and teacher. The student
performance results were analyzed and utilized to make adjustments in instruction
and to identify students who needed additional assistance.
District support for standards was much more limited at the high school level
and did not necessarily translate into teachers regularly using standards. The use of
standards was found to be uneven and entirely dependent upon the individual
teacher. Even when departments at the high school had aligned the curriculum to the
standards, had purchased instructional materials based on the standards and had
developed end o f semester assessments based on the standards, teachers admitted
that standards at most marginally influenced instruction rather than having standards
“drive” instruction. Actual utilization o f standards by RVHS teachers can best be
described as “weak.”
The use o f high performance rubrics at RVHS and the realization by teachers
that standards-based instruction was essential for an effective education were all
good signs that the District’s data model was m oving schools in the right direction.
Additionally, the District’s setting of clear and unambiguous setting of student
performance goals provided targets against which schools could measure their
students’ growth and performance.
197
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
A majority o f students at RVHS were progressing towards high performance
goals with Socio-economically Disadvantaged, Hispanic and African American
students making the largest gains as evidenced in River Valley High School’s
disaggregated API scores at the time o f this study. Causal links to this increased
student performance remained unclear, especially since teacher practice remained
virtually unchanged in compared to previous traditional practices. Teachers were
still found to be working in relative isolation from others. They did not work in
teams and they were not held accountable to any larger goal or purpose. They rarely
met to collectively discuss student work and almost never met to discuss student
performance standards. These practices run counter to a culture o f data utilization
(Schomker, 2000).
Chapter 5 summarizes this study, the methodology, data sources and
instrumentation and the findings. The chapter also presents the researcher’s
conclusions, implications and recommendations for future research.
198
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 5
DISCUSSION OF FINDINGS: CONCLUSIONS, IMPLICATIONS AND
RECOMMENDATIONS
Introduction
A quality education has long been thought to provide significant benefits and
new opportunities for youth who excelled or those who came from privileged
families. Efforts to democratize education for the masses differentiated the United
States from other nations and led to an idealism o f social, political and economic
justice for all. Initial education gains emerging from the 1960’s and early 1970’s for
youth from poverty stricken families and for youth from African American and
Latino heritages eventually plateaued leaving a persistent and insidious achievement
gap. Numerous education reform movements previously tried have failed to close
the achievement gap and have failed to significantly improve student performance
for all students. Additionally, American students have failed to remain competitive
with the academic performance levels of students from nations in Asia and Europe.
These failures have resulted in demands for greater accountability.
Statement o f the Problem
California’s emerging accountability system relied on student test scores
obtained from a battery o f assessments administered under the Standardized Testing
and Reporting system. The system used test data and other performance indicators
to describe progress in student achievement and for holding schools accountability
for results. Results from the 1999-2003 statewide assessments indicated low student
199
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
performance for large numbers o f African American, Latino and Socio
economically Disadvantaged students. A potential solution to low student
achievement was the collection and analysis o f various student achievement data in
an effort to make informed decisions about curriculum, instruction and assessment.
The limited range o f California’s test-based accountability system and the data
collection methods resulting from the assessments were inadequate to accurately
capture the complexity o f teaching and learning. Consequently, local school districts
were left on their own to design data utilization models to raise student
achievement. Data utilization practices were emerging and were not yet proven to
help schools raise student achievement. Lacking effective models for data collection
and utilization, it was conceivable that large numbers o f schools, teachers and
students will suffer unnecessary consequences.
Purpose of the Study
The purpose o f this study was to describe and evaluate how one high school
used data to improve student achievement. The study described the model
developed at the school district level and described how the school identified,
collected and evaluated data during the year o f the study. The study assessed the
extent to which the district design addressed state goals and expectations for student
achievement. It described the extent to which the district design was implemented at
the school site and it attempted to assess the extent o f its usefulness in achieving the
goals of increased student achievement.
200
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Three research questions guided this study:
4. What is the district design for using data and how was the design linked to
the current and the emerging state context for assessing student
performance?
5. To what extent has the district design been implemented at the district,
school and teacher level?
6. To what extent was the district design a good one?
Methodology
A qualitative case-study approach was selected because it offered a critical
analysis o f events in a natural contextual setting (Gall, Borg, & Gall, 1996). The
qualitative research design also provided the opportunity for rich description o f an
organizational setting where human interactions, social structures, mores and values
created a uniquely interesting and dynamic environment for school reform to take
place. This study utilized a moderately sized urban school district in Southern
California and was conducted in a medium size urban high school consisting o f a
diverse student population.
Data was collected according to conceptual frameworks that were developed
by a team of doctoral students conducting parallel dissertations. Thirteen interviews
were conducted within the school district to obtain a full range o f perspectives
including: the Superintendent and three additional central office administrators, the
Chief Educational Officer and two House Principals, five classroom teachers, and a
school board member. The interviews allowed the researcher to probe deeply into
201
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the district’s data design model and to ask additional questions for clarification or
for elaboration. Information gained from the interviews were heavily augmented by
readily available extant data, mainly in the forms o f various teachers surveys,
district reports, faculty bulletins, administrative PowerPoint presentations, teacher
lessons, rubrics and demonstrations, program and grant evaluation reports, parent
and student communications and District and school website information.
The data were analyzed using a researcher’s observation and rating form and
a rating matrix. Self reported data was corroborated by numerous artifacts, surveys
completed by the entire school faculty and researcher observations. Most of the
interviews were conducted without the general knowledge o f the various
participants. Confidentiality o f responses was guaranteed by identifying participants
only by their job title when quoted. Triangulation o f data resulted from a thorough
synthesis o f all o f the data collected and by using the available tools and the
conceptual frameworks.
Summary o f the Findings
California established an accountability system in an effort to provide all
students with a quality education. The system used student test scores from a variety
o f criterion and norm referenced assessments to gage student performance and
school effectiveness. Rather than allow schools to be measured by their aggregate
scores, California and the Federal government required the scores to be
disaggregated by several student subgroups based on student ethnicity, English-
language proficiency, special needs and socio-economic status. California schools
202
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
were assigned an Academic Performance Index number as a school and for each
student subgroup a means to monitor student performance.
The case study conducted in Valley Unified School District and at River
Valley High School suggests that California’s school accountability system can
work when augmented by local district initiatives focusing on student achievement
results. Specifically, this case study found that after several years of declining
student performance and a wide achievement gap between various student sub
groups, River Valley High School students significantly improved their performance
while closing the achievement gap. Causation for one year’s improvement in student
performance was indeterminate but promising. The findings presented below were
identified as a result o f the research.
1. A results oriented district design to raise student achievement and to close the
achievement gap has been developed and articulated.
2. The District’s practice of data collection, analysis and utilization varies in levels
o f implementation as delineated by organizational structure, employee job
responsibility and by academic subject.
3. The explicit collection of data resulting from student achievement measures,
various surveys and other instruments at the school site level leads to an expectation
by teachers and administrators that the data will lead to utilization for the purposes
of improving student achievement and improved educational experiences for all
students.
203
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
4. The extent o f the District’s practice o f data collection, analysis and utilization
was evolving but it had not yet created the conditions to transform previous school
culture.
The following section reviews the research questions, the findings, and
summarizes the context and the evidence that led to the findings.
Question One
What is the district design for using data regarding student performance and
how is the design linked to the current and emerging state context for assessing
student performance?
Research Finding One: A results oriented district design to raise student
achievement and to close the achievement gap has been developed and
articulated.
Valley Unified School District did not have an explicit Board adopted model
design for using data. The Superintendent set the standard for data use and became
the role model and leading advocate within the District. His commitment to data use
created an environment where leaders increased their ability to analyze data to make
data-driven decisions. The primary purpose o f analyzing data as articulated by the
Superintendent was to “improve the academic achievement o f students while
simultaneously closing the achievement gap.”
The unofficial data model design the District created included various
student achievement measures collected through STAR, emerging state assessments
beyond STAR, as well as District and school-based student achievement results.
204
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The Superintendent set student performance goals for all students k-12 and the goals
were shared with all stake-holders. The District also collected data from a school
climate survey that addressed educational processes that could impact student
learning. The District set in place periodic assessments at the elementary level and
planned on developing similar assessments at the high school level in the future.
Student achievement results for River Valley High School in the 2003-04
year performed exceedingly well in comparison to prior yearly results. African
American, Hispanic and Socio-economically Disadvantaged students significantly
improved their scores on STAR assessments across all assessments.
Question Two
To what extent has the district design been implemented at the district,
school and teacher level?
Research Finding Two: The District’s practice of data collection, analysis and
utilization varies in levels of implementation as delineated by organizational
structure, employee job responsibility and by academic subject.
The District’s design was not fully developed at all levels o f the
organization. Implementation was uneven and coherence throughout the system was
weak. The central offices exhibited greatest level o f implementation followed by
school site administrators and then teachers. Elementary schools exhibited greater
use of the district data model in comparison to middle schools and high schools
mostly as a result o f having a continuous flow o f student achievement data. At the
high school English-language arts and mathematics teachers used data better than
205
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
science teachers and social studies teachers. The situational use o f available data
suggests that the district’s model for using data remained fragile but with a growing
base of users and consumers.
Research Finding Three: The explicit collection of data resulting from student
achievement measures, various surveys and other instruments at the school site
level leads to an expectation by teachers and administrators that the data will
lead to utilization for the purposes of improving student achievement and
improved educational experiences for all students.
VUSD and RVHS collected data from a variety o f teacher and student
surveys and questionnaires. Teachers reported that some surveys such as the SALT-
like survey were very time intensive and questioned their utility. Other surveys such
as the SCORE survey on school redesign asked for teacher’s perspectives but
several felt that they were not used to make changes at the school but were used to
let teachers vent their frustrations. Several teachers expressed concerns that the
surveys did little to change the practices on campus even though the data clearly
indicated there was a problem. The administration felt that these surveys yielded
useful data to monitor the attitudes and expectations o f teachers, students and
parents especially about school redesign and perceptions o f teaching and learning.
The District’s model, through the use of surveys took into account contextual data
about the learning environment and the processes o f teaching and learning that they
believe might also affect student achievement. The Superintendent believed that
206
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
school climate data was important enough to use general funds to pay for the SALT-
like survey at the expense o f other competing fiscal demands.
Question Three
To what extent was the district design a good one?
Research Finding Four: The extent of the District’s practice of data collection,
analysis and utilization was evolving but it had not yet created the conditions to
transform previous school culture.
The District’s data design supported-standards based instruction. Teachers
were well informed of the academic content standards they were to use to guide
instruction. In some disciplines teachers used standards to guide instruction, to
select and to purchase instructional materials and textbooks and to develop
assessments that measured student achievement o f standards. District support for
standards was limited at the high school level and did not necessarily translate into
teachers regularly using standards. The use o f standards was found to be uneven and
entirely dependent upon the individual teacher. Actual utilization o f standards by
RVHS teachers can best be described as “weak.”
The District set o f clear and unambiguous student performance goals
provided targets against which schools could measure their students’ growth and
performance. A majority o f students at RVHS were progressing towards high
performance goals with Socio-economically Disadvantaged, Hispanic and African
American students making the largest gains as evidenced in River Valley High
School’s disaggregated API scores. Causal links to this increased student
207
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
performance remained unclear, especially since teacher practice remained virtually
unchanged when compared to previous traditional practices. Teachers were still
found to be working in relative isolation from others. They did not work in teams
and they were not held accountable to any larger goal or purpose. They rarely met to
collectively discuss student work and almost never met to discuss student
performance standards. At the time o f this study, there was a big variance between
stated beliefs and practices in the value o f standards, assessments and the use of
data.
Conclusions Based on Research
Improved student achievement results and a closing o f the achievement gap
at River Valley High School suggest that the District design for using data might be
an effective and an influential factor. Although one cannot ascertain the extent of
the relationship between improved student achievement scores and the data model
and data usage, it is clear that the District needs additional support for the model at
the secondary level and continued reinforcement at the elementary level. The
Superintendent has made essential changes in the District’s organizational structure
based on a fundamental commitment to educational equity and educational
excellence for all students. He used indisputable student achievement data as lever
to move a district self-satisfied with educating the top twenty percent o f the student
population well, while disregarding the educational needs of the masses.
The Chief Educational Officer faced a similar malaise at River Valley High
School. Her primary challenge was to break free from the historical context in
208
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
which previous faculties allowed the school to develop a “two schools
phenomenon” reputation. She capitalized on huge stores o f data both recent and
historical to redesign the school into several small schools. The additional impact of
a three year trend o f declining API scores convinced the faculty to take drastic steps.
She has faced some unforeseeable challenges that have slowed progress, namely a
thirty percent turnover in faculty during the last two years and a seventy five percent
turnover in her administration. Nevertheless, her experience with middle school
redesign and her commitment towards improved student achievement results have
stopped the decline and facilitated a modest revival o f student success.
Pockets o f teachers at River Valley High School have embraced standards
and have begun to examine at student work. School redesign challenged the existing
power structure and moved many teachers beyond their zone o f comfort. Many
faculty members were unwilling to change and left the school. Others stayed with a
“wait and see” attitude. Initially the professional development work around
standards moved the faculty to realize the importance o f standards-based instruction
but not to the point o f changing their instructional practices. The vast changes
resulting from school reorganization, the pressure to meet accountability
expectations, the continual barrage of data may have had the unfortunate outcome of
frustrating many o f the faculty to the point o f saturation. What teachers really need
at this point is time to reflect and a renewed focus on improving instruction.
Improving teachers’ understanding of student achievement data might facilitate the
later but carving out time in an already hectic schedule will remain a challenge.
209
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Implications o f Research Findings
The public demands that schools ensure every student has access to a high
quality education that prepares them for the demands o f the world o f work, provides
for a quality o f life that matches or exceeds their parents and ensures the
continuance o f the American experiment in participatory democracy. Public
policymakers impatient with the slow progress o f improving student achievement to
meet these goals have imposed accountability systems that have serious
consequences for school districts, school faculties including administrators and for
students failing to meet performance expectations. Some school districts have begun
to look at evidence using student performance data to raise student achievement and
to close the achievement gap. Although schools are awash in data, data utilization to
improve student achievement remains problematic.
The evidence collected and analyzed during this study indicates that data
utilization to raise student achievement was a relatively new phenomenon emerging
from decades o f other school reform efforts. Strong accountability measures
coupled with the relative newness of data analysis as an innovation in schools have
encouraged some school districts and some school leaders to develop data models as
a logical tool for improving student achievement. Innovative leaders challenge the
status quo and work toward change. They face numerous obstacles that must be
overcome and they must demonstrate in very concrete ways the benefits of their
innovation. Data utilization as an innovation is no different.
210
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Policymakers and school leaders committed to the use of data can apply
lessons previously learned from other school reform efforts. First, a focus on results
is essential for success. Early successes at using data will gain some converts but
many teachers and administrators will remain unconvinced or ill-prepared to
adequately tap the power o f analyzing student achievement data. Data users will
need concentrated and continued support in collecting, understanding, analyzing and
utilizing data. Specific and relevant professional development will help staff use
data but it will not be enough. Additional structures and supports will need to be put
in place. Whatever the data utilization model the District adopts will need to be
sensitive and responsive to the needs o f teachers. Since teachers work with students
daily, they have the greatest potential to influence student achievement.
Policy makers and teachers themselves will need to find opportunities to
collect student achievement data that can be used to improve teaching and learning.
Daily lessons, teacher generated assessments and department developed assessments
might provide information teachers would accept as valid in contrast to how they
generally discount the data when looking at annual state mandated assessment
results. If teachers can’t see the relevance to particular innovations they rarely work
to make them part o f their daily or weekly practice. Data utilization must be made as
much as possible teacher friendly, efficient and productive.
Technology has played in important role in making data accessible in a
variety of formats. Unfortunately there is a generation gap between those that lived
before the personal computer existed and those who have only known a world filled
211
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
with computers. Since most student achievement data is stored in electronic files,
teachers with limited technological skills will certainly be less likely to effectively
access data. Additionally, much o f the technology used in data analysis was created
by software programmers who have little understanding o f teacher’s technological
needs, skills and proficiencies.
The conditions and expectations of most teachers and school systems have to
change. Specifically the isolating nature of teaching will have change for effective
data use to become systemic. As long as teachers are allowed to close their door to
their peers and the administration any innovation is bound to fail. Accountability
comes with a need for transparency. Data sheds light on the hidden. The quest for
data will do the same. What most educators fail to understand is that data is neutral.
Data’s power comes in the form of providing evidence rather than best hunches as
to the current condition o f student achievement. Too many educators still fear data.
The challenge for many educational leaders will be to convince their colleagues that
student achievement data can be used as a vehicle for change rather than as a
weapon available to others to attack them.
Finally, the conditions of schooling in California must also change. It seems
unreasonable to place the entire burden on teachers and administrators to raise
student achievement if data indicate that too many schools are large and impersonal
institutions. It seems unreasonable that little data has been collected to determine
adequate funding for the desired student outcomes the public has demanded of
schools. Data collection, analysis and utilization practices hold much promise in
212
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
providing targeted assistance to students with the greatest academic needs, but the
scope and range o f physical, emotional and psychological needs o f students must
also be addressed and not necessarily by schools.
Recommendations
1. The District should develop and implement periodic assessments at the
secondary level in core curricular areas of English-language arts, mathematics,
science and social studies.
The District has not fully implemented the stated data model and must fulfill
the expectation that regularly collected data provides the best vehicle to make mid
course corrections in teaching and learning. The current STAR system fails to
provide frequent and adequate measures of student achievement. Completing the
District’s system-wide assessment system should enhance the likelihood that timely
instructional modifications could be made to benefit student learning.
2. The school should sharpen its focus on student achievement by ensuring that
regularly scheduled professional development addresses state standards and
effective models of assessing student work.
Records o f professional development time indicated a mish-mash of topics
that lacked coherence. The school had not provided teachers with adequate
professional development opportunities in the areas o f standards, assessment or data
analysis. Professional development opportunities should provide adequate time for
observing other teacher’s classroom instructional practices and student work.
Teachers reported they rarely meet collectively to set academic performance
213
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
standards for student work. It seems unreasonable to expect teachers to improve
instructional practices without adequate time, support and accountability.
3. The school leadership must develop a plan to stabilize the faculty and
administration.
River Valley High School has been plagued by high teacher and
administrator turnover. Investments in professional development, human
relationships and school reform efforts all become casualties and more importantly
become less effective with high staff turnover. The newness o f data use to raise
student achievement remains relatively foreign to educators. Initial reluctance to
innovation will have to continually be addressed if the faculty continues to change.
Additionally the unevenness in data experience and use will form a barrier between
those that invested time and effort versus those who have not.
4. The District and administration should carefully evaluate the consequences
of introducing any new innovations to the organizational system.
Carelessly introducing new innovations to a system that has not adjusted nor
internalized a series o f previous innovations overloads the system. Doing too much,
results in accomplishing too little. Faculties become frustrated and begin to think
that any innovation is the flavor o f the month and it too shall pass. Making a
commitment to an innovation means being patient with the development,
implementation, evaluation and the revision processes. Completely eliminating new
innovations maybe difficult given the complex nature o f schools and the demands
214
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
for rapid results, coherence and compatibility with priority innovations should be
paramount.
Suggestions for Further Research
1. This case study focused on one school in a single school district. Future
studies should examine the effectiveness of the District data model at the
elementary, the middle and the high school levels.
2. Further research is needed to evaluate specific student achievement data
driven decisions and their effect on raising student achievement.
3. Additional research is needed to shed light on the effects of data analysis
on systems change and the organizational structure o f schools.
215
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
REFERENCES
Astuto, T.A., & Clark, D.L. (1985). Strength o f organizational coupling in the
instructionally effective school. Urban Education. 19.
Bassom, R. (1999). The meaning o f knowledge-based, comprehensive school
reform. Challenges of Comprehensive School Reform: A working
Conference. October 29-30 1999, Portland, Oregon. Northwest Regional
Educational Laboratory.
Bemhart, V.L. (2000). Intersections: New routes open when one type o f data crosses
another. Journal o f Staff Development. 21(1).
Berends, M., Bodilly, S., & Kirby, S.N. (2002). Looking back over a decade of
whole-school reform: The experience o f the New American Schools. Phi
Delta Kappan. 8412). 168-175.
Berliner, D.C. & Biddle, B.J. (1995). The Manufactured Crisis: Myths, Fraud, and
the Attack on America's Public Schools. Reading, MA: Addison-Wesley.
Blank, R.K., & Wilson, L.D. (2001). Understanding NAEP and TIMSS results:
Three types o f analyses useful to educators. ERS Spectrum. 19(1), 23-32.
Borman, G.D., Hewes, G.M., Overman, L.T., & Brown, S. (2002). Comprehensive
school reform and student achievement: A meta-analysis. John Hopkins
University, Baltimore, MD. [Online]. Available:
http://www.csos.ihu.edU/CRESPAR/techReports./Rei3oit59.pdf
Bracey, G. (1998). TIMSS, rhymes with 'dims,' as in 'witted. Phi Delta Kappan.
79(9), 686-687.
Bracey, G. (2003). April foolishness: The 20th anniversary of A Nation at Risk. Phi
Delta Kappan. 84(81. 616-621.
Braswell, J., Daane, M., Grigg, W., and Education Testing Service. (2003). The
nation’s report card: Mathematics highlights 2003. (NCES 2004-451) U.S.
Department o f Education, National Center for Education Statistics.
Washington, D.C.: U.S. [Online]. Available:
http://nces.ed.gov/nationsreportcard/pdf/main2003/20Q4451 .pdf
Brookover, W., & Lezotte, L. (1979). Changes in school characteristics coincident
with changes in student achievement. East Lansing: Michigan State
University, College o f Urban Development.
216
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
California Department o f Education (CDE). (2003). Academic performance index
base report: 2002 Information guide. California Department of Education,
Policy and Evaluation Division. [Online]. Available:
http://www.cde.ca.gov/ta/ac/ap/documents/infoguide02b.pdf
California Department o f Education (CDE). (2004). STAR 2004: Reporting results
to parents and guardians, assistance packet for school districts and schools.
California Department o f Education. [Online]. Available:
http://www.cde.ca.gov/ta/tg/sr/docii.ments/star04respkt.pdf
Campbell, J.R., Hombo, C.M. & Mazzeo, J. (2000). NAEP 1999 Trends in
academic progress: Three decades o f student performance, (NCES 2000-
469) Office of Educational Research and Improvement, National Center for
Education Statistics. Washington, D.C.. [Online], Available:
http://nces.ed.gov/nubsearch/nubsinfo.asp7pubid~2000469
Carr, J.F, & Harris, D.E. (2001). Succeeding with standards: Linking curriculum.
assessment and action planning. Association for Supervision and Curriculum
Development, Alexandria, VA.
Carroll, S.J., Krop, C., Arkes, J., Morrison, P.A. & Flanagan, A. (2005). California’s
K-12 public schools: How are they doing? RAND Corporation, Santa
Monica, CA. [Online]. Available:
http://www.rand.Org/pubs/m.onograph.s/2004/RAND M G186.pdf
Celio, M.B., & Harvey, J. (2005). Buried treasure: Developing a management guide
from mountains o f school data. Center on Reinventing Public Education.
University o f Washington. [Online]. Available:
http://www.crpe.org/pubs/txlf7BuriedTreasure cello.pdf
Choppin, J. (2002). Data use in practice: Examples from the school level. Paper
presented at the annual meeting o f the American Education Research
Association, New Orleans, Louisiana, April 2, 2002. [Online]. Available:
http://www.wcer.wisc.edu/mps/AERA2002/data use in practice.htm.
Christie, K. (2004). Virginia’s excellent adventure. Phi Delta Kappan. 85(8). 565-
567.
Coleman, J., et al. (1966). Equality o f Educational Opportunity. Washington D.C.:
U.S. Government Printing Office.
Conzemius, A. (2000). Framework: Systems builds change effort beyond hopes,
hunches, guesses. Journal o f Staff Development. 21(1).
217
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Cuban, L. (1998). How schools change reforms: redefining reform success and
failure. Teachers College Record. 99(31. 453-477.
Donahue, P.L. Daane, M., Grigg, W., and Educational Testing Service. (2004). The
Nation’s Report Card: Reading Highlights 2003. (NCES 2004-452) U.S.
Department of Education, National Center for. Education Statistics.
Washington, D.C.: U.S. [Online]. Available:
http://nces.ed.gov/nationsreportcard/pdiymain2003/2004452.pdf
Donahue, P.L., Voelkl, K.E., Campbell, J.R. & Mazzeo, J. (1999). NAEP 1998
Reading Report Card for the Nations and States. (NCES 1999500), U.S.
Department o f Education, National Center for Education Statistics.
Washington, D.C.: U.S. [Online]. Available:
h ttn ://nces.ed. go v/nati on sreportcard/Zpd f/main 1998/1999500. pdf
DuFour, R. (2000). Data put a face on shared vision. Journal of Staff Development.
21(1).
Edmunds, R. (1979). Effective schools for the urban poor. Educational Leadership
37, 15-25.
EdSource. (2003). The California High School Exit Exam. Palo Alto, CA. (April,
2003).[Online]. Available: http://www.edsource.org/pdf/QA hseefinal.pdf
EdSource (2003a). Show me the data: EdSource forum focusing on using and
improving data and on NCLB. Palo Alto, CA. (May 2003) [Online].
Available: http://www.edsource.org/pdfTomm03report finaltxlf
Education Commission o f the States (ECS). (1999). The progress o f education
reform 1999-2001. 1(3) [Online]. Available;
http://www.ecs. or g/clearin ghou.se/16/42/1642. him
Englert, K., Fries, D., Goodwin, B., and Martin-Glenn, M. (2003). Understanding
how superintendents use data in a new environment of accountability.
Regional Educational Laboratory, #ED-01-CO-0006, United Stated
Department o f Education, Washington, DC.
Firestone, W. (1991). Rethinking effective schools. Englewood Cliffs, NJ: Prentice-
Hall.
Fuhrman, S.H. (1994). Challenges in systemic education reform. CPRE Policy
Briefs. Consortium for Policy Research in Education. University of
218
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Pennsylvania. [Online]. Available:
http:// www.CDre.org/Publi cat! ons/rb 14 .pdf
Fullan, M.G. (2004). Change agent. In Dennis Sparks interview with Michael
Fullen. Journal of Staff Development. Winter 2003, 55-58. [Online]>
Available: h ttp://www.nsdc.org/1 ibrarv/publi call ons/i sd/fis 1 1 an241.pdf
Gall, M.D., Borg, W.R., & Gall, J.P. (1996). Educational research: an introduction.
Sixth edition. White Plains, NY. Longman Publishers.
Glennan, T.K., Bodilly, S.J., Galegher, J.R., & Kerr, K.A. (2004). Expanding the
reach o f education reforms: Perspectives from leaders in the scale-up o f
educational interventions. Rand Corporation, Santa Monica, California.
[Online]. Available:
http://www.rand.org/pubs/monographs/2004/RAND M G248.pdf
Goals 2000: Educate America Act. (1994). Public Law No. 103-227. [Online].
Available: http://wv vw.&d.gov/legislation/GOALS2000/TheAct/index.htnil
Goals 2000: Building on a decade of reform. (1996). [Online]. Available:
http://www.ed-.gov/G2K/ProgRpt96/build.html
Goddard, R.D., Sweetland, S.R. & Hoy, W.K. (2000). Academic emphasis on of
urban elementary schools and student achievement in reading and
mathematics: A multilevel analysis. Educational Administration Quarterly.
36(5).
Gonzales, P., Guzman, J.C., Patelow, L., Pahlke, E., Jocelyn, L., Kastberg, D. &
Williams, T. (2004). Highlights from the Trends in International
Mathematics and Science Study (TIMSS) 2003 (NCES 2005-005). U.S.
Department o f Education, National Center for Education Statistics.
Washington, D.C.: U.S. Government Printing Office. [Online]. Available:
http://nces.ed.gov/pubs2005/timss03/
Green, L. (1998). A parents guide to understanding academic standards. Council for
Basic Education, Washington, D.C.
Greenspan, A. (2004, February). The critical role o f education in the nation’s
economy. Remarks by Chairman Alan Greenspan at the Greater Omaha
Chamber o f Commerce 2004 Annual Meeting, Omaha, Nebraska. [Online],
Available: http://www.bis.org/review/r040301 a.pdf
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Greenspan, A. (2004, March). Testimony o f Chairman Alan Greenspan before the
Committee on Education and the Workforce, U.S. House o f Representatives,
Washington, DC. [Online] Available:
http://www.federaireserve.gov/boarddocs/testimonv/2004/2004Q311/default.
htm
Greenspan, A. (2000, April). Remarks by Chairman Alan Greenspan at the U.S.
Department o f Labor National Skills Summit, Washington, D.C. [Online],
Available:
http://www.federalreserve.gov/boarddocs/speeches/2000/2000Q41 i.htm
Hamilton, L., & Stecher, B. (2004) Responding effectively to test-based
accountability. Phi Delta Kappan. 85(8L 578-583.
Harvey, J. (2003). The urban superintendent: Creating great schools while surviving
on the job. Council o f the Great City Schools. [Online]. Available:
http://www.wallacefound.ation.org/NIUrdonlvres/0B91F0E.l-l.7A7-4584-
AE40-B5582A8A3833/0/UrbanSuptRenort.pdf
Hiebert, J. et al. (2004). Teaching mathematics in seven countries: Results from the
TIMSS 1999 video study. (NCES 2003-013 revised), U.S. Department of
Education, National Center for Education Statistics. Washington, D.C.:
Government Printing Office. [Online]. Available:
http://nces.ed.gov/pubs2003/2003Q 13 .pdf
Hopfenberg, W. S., Levin, H. M., et al. (1993). The Accelerated Schools Resource
Guide. San Francisco, CA: Jossey-Bass
H.R. 6., Improving America's Schools Act o f 1994. (1994). 103r d Congress of
United States o f America. [Online]. Available:
http://www.ed.gov/legislation/ESEA/toc.html
Johnson, C. (2004). Nation’s report card: An overview o f NAEP. (NCES 2004552)
[Online]. Available: http://nces.ed.gov/pubs2004/2004552.pdf
Johnson, C., Princiotta, D., & Vanneman, A. (2003). The nation’s report card:
Parent’s guide to NAEP. (NCES 2003-480), U.S. Department o f Education,
National Center for Education Statistics. Washington, D.C.: Government
Printing Office. [Online]. Available:
http://nces.ed.gov/pubs2003/2003480.pdf
Johnson, E.G. & Siegendorf, A.(1998). Linking the National Assessment of
Education Progress and the Third International Mathematics and Science
Study for eighth grade: A research report. (NCES 98-500) U.S. Department
220
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
o f Education, National Center for Education Statistics. Washington, D.C.:
Government Printing Office. [Online]. Available:
http://:nces.ed.gov/pubs98/98500.pdf
Johnson, J.H. (1997). Data-driven school improvement. ERIC Digest 109, (ED 401
595), Clearinghouse on Education Management. Eugene, OR. [Online],
Available: http://eric.uoregon.edu/publications/digests/digestl09.html
Johnson, R.S. (2002). Using data to close the achievement gap. Thousand Oaks,
CA, Corwin Press.
Keeney, L. (1998). Using data for school improvement: Report on the second
practitioners’ conference for Annenberg Challenge sites. Brown University,
Providence, RI.
Khanna, R., Trousdale, D., Penuel, W. & Kell, J. (1999, April). Supporting data use
among administrator: Results from a data planning model. Paper presented
at the annual meeting of the American Education Research Association,
Montreal, Quebec, Canada, April 19-23.
Kirsch, I., de Jong, J., La Fountaine, D., McQueen, J., Mendelovits, J., & Monseur,
C. (2002). Reading for change: Performance and engagement across
countries, results from PISA 2000. Organisation for Economic Co-Operation
and Development. Paris. [Online]. Available:
http://www 1 .oecd.org/pu.blicaf ions/e-book/9602071E. PDF
Kohn, A. (2004). Test today, privatize tomorrow: Using accountability to ‘reform’
public schools to death. Phi Delta Kappan, 85(81. 569-577.
Kozal, J. (1991). Savage inequalities: Children in America’s schools. New York:
Harper Perennial.
Kress, S. (2003, April). Memorandum on struggling schools. Paper presented at a
policy forum meeting at the National Center on Education and the Economy
on Implementing the No Child Left Behind Act, Washington, DC. [Online].
Available: http://colosus.ncee.org/pdf/policv/fi.xing essav.pdf
Larson, K., Guidera, A. & Smith, N. (1998). The formula for success: A business
leader's guide to supporting math and science achievement (Publication No.
SAI-98-3028). Washington, D.C. (ERIC Document Reproduction Service
No. ED 419 914).
Lawler, E.E. (1986). High involvement management. San Francisco: Jossey-Bass.
221
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Lemke, M. et al. (2001). Outcomes of learning results from the 2000 Program for
International Student Assessment o f 15-year-olds in reading, mathematics
and science literacy. (NCES 2002-115), Washington, D.C.: U.S. Department
o f Education, National Center for Education Statistics. [Online]. Available:
http://nc-es.ed. gov/pubs2002/2002115.pdf
Lemke, M., Sen, A., Pahlke, E., Partelow, L., Miller, D., Williams, T., Kastberg, D.,
& Jocelyn, L. (2004). International outcomes o f learning in mathematics
literacy and problem solving: PISA 2003 results from the U.S. perspective.
(NCES 2005-003). Washington, D.C.: U.S. Department o f Education,
National Center for Education Statistics. [Online]. Available:
http://nces.ed.gov/imbs2005/20Q5003.pdf
Lewis, A. (2004). High schools and reform. Phi Delta Kappan. 85(8). 563-564.
Livingston, A. & Wirt, J. (2004). The condition o f education 2004 in Brief (NCES
2004-076). U.S. Department o f Education, National Center for Education
Statistics. Washington, D.C.: U.S. Government Printing Office. [Online].
Available: http://nees.ed.gov/pubs2004/2004Q76.pdf
Lytle, J.H. (2002). Whole-school reform from the inside. Phi Delta Kappan. 84(2L
164-167.
Marsh, D. (1996, Feb/Mar). Making school reform work: lessons from successful
schools. Thrust for Education Leadership. 25. 10-14.
Martin, M.O., Mullis, I.V.S, Gonzales, E.J., & Chrostowski, S.J. (2004). TIMSS
2003 International Science Report: Findings from IEA’s Trends in
International Mathematics and Science Study at the eighth and fourth grades.
Chestnut Hill, MA: Boston College. [Online]. Available:
http://timss.bc.edu/PDF/t03 download/T03INTLSCIRPT.pdf
Martin, M.O., Mullis, IV.S., Gonzalez, E.J., Gregory, K.D., Smith T.A.,
Chrostowski, S.J., Garden, R.A., & O’Connor, K.M., (2000). TIMSS 1999
International Science Report: Findings from IEA's repeat o f the Third
International Mathematics and Science Study at the eighth grade. Chestnut
Hill, MA: Boston College. [Online]. Available:
http://isc.bc.edu/timssl 999i/pdPT99i Sci All.pdf
Marzano, R. J. (2003). Using data: Two wrongs and a right. Educational Leadership.
60(5), 56-60.
222
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Mason, S. (2002). Turning data into knowledge: Lessons from six Milwaukee public
schools. Paper presented at the annual conference of the American Education
Research Association, New Orleans, April 2002.
Merrow, J. (2001). Undermining standards. Phi Delta Kappan, 82(9). 653-659.
Ministerial Council on Education, Employment, Training and Youth Affairs
(MCEETYA). (2002). Benefits o f participation in international studies of
student achievement. Ministerial Council on Education, Employment,
Training and Youth Affairs, Performance Measurement and Reporting
Taskforce, Melbourne, Australia. [Online]. Available:
www.mceetva.edu.au/pdfybenefits o f internat studies.pdf
Mullis, I.V.S, Martin, M.O., Beaton, A.E., Gonzales, E.J., Kelly, D.L., & Smith,
T.A. (1998). Achievement in the final year o f secondary school: IEA”s Third
International Mathematics and Science Study (TIMSS). Chestnut Hill, MA:
Boston College. [Online]. Available:
http://isc.bc.edu/timss 1995i/TIMSSPDF/C full.pdf
Mullis, I.V.S, Martin, M.O., Gonzales, E.J., & Chrostowski, S.J. (2004). TIMSS
2003 International Mathematics Report: Findings from IEA’s Trends in
International Mathematics and Science Study at the eighth and fourth grades.
Chestnut Hill, MA: Boston College. [Online]. Available:
http://tim.ss.bc.edu/PDF/t03 download/T03INTLMATRPT.pdf
Mullis, IV.S., Martin, M.O., Gonzalez, E.J., Gregory, K.D., Garden, R.A.,
O'Connor, K.M., Chrostowski, S.J., & Smith T.A. (2000). TIMSS 1999
international mathematics report: Findings from IEA's repeat o f the Third
International Mathematics and Science Study at the eighth grade. Chestnut
Hill, MA: Boston College. [Online]. Available:
http://isc.bc.edu/timssl999i/pdfiT99i Math A ll.pdf
Mullis, I.V.S., Martin, M.O., Gonzales, E.J., & Kennedy, A.M. (2003). PIRLS 2001
international report: IEAs study o f reading literacy achievement in primary
schools in 35 countries. International Study Center, Boston College, Boston.
[Online]. Available: http://timss.bc.edu/pirls2001 i/pdf/p 1 IR book.pdf
National Center for Education Statistics (NCES). (2003). NAEP Mathematics 2003
state snapshot reports: California grade 4. U.S. Department o f Education.
National Center for Education Statistics, (NCES2004457), Washington
D.C.: [Online]. Available:
http://nces.ed. gov/nationsreportcard/t)d.fystt2003/2004457CA4 .pdf
223
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
National Center for Education Statistics (NCES). (2003). NAEP Mathematics 2003
state snapshot reports: California grade 8. U.S. Department o f Education.
National Center for Education Statistics, (NCES2004457), Washington
D.C.: [Online]. Available:
http://nces.ed,gov/nationsreportcard/pd£/stt2003/2004457CA8.pdf
National Center for Education Statistics (NCES). (2003). NAEP Reading 2003 state
snapshot reports: California grade 4. U.S. Department o f Education.
National Center for Education Statistics, (NCES2004456), Washington
D.C.: [Online]. Available:
http://nces.ed.gov/nationsreportcard/pdf/stt2003/2004456CA4.pdf
National Center for Education Statistics (NCES). (2003). NAEP Reading 2003 state
snapshot reports: California grade 8. U.S. Department o f Education.
National Center for Education Statistics, (NCES2004456), Washington
D.C.: [Online]. Available:
http://nces.ed.gov/natiopsreportcard/pdf/stt2003/20Q4456CA8.pdf
National Commission on Excellence in Education. (1983). A nation at risk: The
imperative for education reform. Stock No. 065-000-00177-2, Washington,
D.C., U.S Government Printing Office. [Online]. Available:
h ttp : //ww w . ed. go \7p nb s/N at A tRisk/i nd.ex.htm l
Nelson, D. (2003, April). Resuscitating failing schools in Georgia. Paper presented
at a policy forum meeting at the National Center on Education and the
Economy on Implementing the No Child Left Behind Act, Washington, DC.
[Online]. Available: http://colosus.ncee.org/pdf/policv/fixing essay.pdf
Noyce, P., Perda, D., & Traver, R. (2000). Creating data-driven schools.
Educational Leadership. 57(51.
O’Sullivan, C.Y.O., Lauko, M.A., Grigg, W.S., Qian, J., & Zhang, J. (2003). The
nation’s report card: Science 2000. (NCES 2003-453), U.S. Department of
Education. National Center for Education Statistics, Washington D.C.: U.S.
Government Printing Office. [Online]. Available:
http://nces.ed.gov/pationsreportcard/pdf/main2000/2003453.pdf
O’Sullivan, C.Y., Reese, C.M., & Mazzeo, J. (1997). NAEP 1996 science: Report
card for the nation and the states. (NCES 97497), U.S. Department of
Education. National Center for Education Statistics, Washington D.C.: U.S.
Government Printing Office. [Online]. Available:
http://nces.ed.gOv/nationsreportcard//pdf/main 1996Z97497.pdf
224
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Odden, A.R., with Odden, E.R. (1995). Educational leadership for America’s
schools. New York. McGraw-Hill.
Odden, A. (2000) The costs o f sustaining educational change through
comprehensive school reform. Phi Delta Kappan. 81161.
Odden, A. (2003). Equity and adequacy in school finance today. Phi Delta Kappan.
85(2), 120-125.
Ogawa, R.T., & Collom, E. (2000). Using performance indicators to hold schools
accountable: Implicit assumptions and inherent tensions. Peabody Journal of
Education. 75(41 200-215.
Olsen, L. (2001). Holding schools accountable for equity. Leadership. 30(41.
Organisation for Economic Co-Operation and Development (OECD). (2004).
Learning for tomorrow’s world: First results from PISA 2003. Organisation
for Economic Co-Operation and Development. Paris. [Online]. Available:
http ://www.pisa.oecd.org/dataoecd/1/60/340022.16.pdf
Peak, L. (1996). Pursuing excellence: A study of U.S. eighth-grade mathematics and
science teaching, learning, curriculum, and achievement in international
context: Initial findings from the Third International Mathematics and
Science Study. U.S. Department o f Education. National Center for
Education Statistics, NCES 97-198, Washington D.C.: U.S. Government
Printing Office. [Online]. Available: http://nces.ed.gov/piibs97/97198.pdf
Peak, L. et al. (1997). Pursuing Excellence: A Study o f U.S. Fourth-Grade
Mathematics and Science Achievement in International Context. U.S.
Department of Education. National Center for Education Statistics, NCES
97-225, Washington D.C.: U.S. Government Printing Office. [Online].
Available: http://nces.ed.gov/pubsearch/pubsinfo,asp?pubid=97255
Picus, L.O., & Blair, L. (2004). School finance adequacy: The state role. SEDL
Letter X V im . 3-8.
Pogrow, S. (2002). Success for All is a failure. Phi Delta Kappan. 83/61. 463-468.
Popham, W J. (2003). The seductive allure of data. Educational Leadership. 60151.
48-51.
Protheroe, N. (2001). Improving teaching and learning with data-based decisions:
Asking the right questions and acting on the answers. ERS Spectrum.
225
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Summer 2001. [Online]. Available:
http ://w w w .e rs.org/spectrum/sumO 1 a.htro
Rallis, S.F., & MacMullen, M.M. (2000). Inquiry-minded schools: Opening doors
for accountability. Phi Delta Kappan. 81(101 766-773.
Ravitch, D. (1996). 50 states, 50 standards: The continuing need for national
voluntary standards in education. The Brookings Review. 14(31. 6-9.
[Online]. Available:
ht(p://www:brookings.edu/press/review/sumrner96/ravitch.htm
Reese, C.M., Jerry, L., & Ballator, N. (1997). NAEP 1996 Mathematics State
Report for California, Washington, D.C.: National Center for Education
Statistics, 1997. [Online]. Available:
http://nces.ed.gov/nationsrenortcard/pdPsttl996/97974ca.pdf
Reese, C. M., Miller, K.E., Mazzeo, J., & Dossey, J. A. (1997). NAEP 1996
Mathematics Report Card for the Nation and the States: Findings from the
National Assessment o f Educational Progress, Washington, D.C.: National
Center for Education Statistics, 1997. [Online]. Available:
http://nces.ed.gOv/nationsreportcard//pdf/mainl996/97488.pdf
Reeves, D. B. (2000) Caught in the middle. American School Board Journal.
187(121.
Reeves, D. B. (2000). Accountability in Action: A Blueprint for Learning
Organizations. Advanced Learning Press, Englewood, CO., 185-208.
Rose, L. C., & Gallup, A. M. (2004). The 36th Annual Phi Delta Kappa/Gallup Poll
o f the public’s attitudes toward the public schools. Phi Delta Kappan 86(11
41-56.
Rothman, R. (1999) Standards and assessments: The foundations o f high student
achievement. In D.D. Marsh and J.B. Coding and Associates (Eds.) The new
American high school (pp. 37-60). Corwin Press, Thousand Oaks, CA.
Scherer, M. (2003). Perspectives/blind data. Education Leadership. 60(51. 5.
Schmoker, M. (1996). Results, the key to continuous school improvement.
Alexandria, VA: Association for Supervision and Curriculum Development.
226
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Schmoker, M. (1999). Results, the key to continuous school improvement.
Alexandria, VA: Association for Supervision and Curriculum Development.
Schmoker, M. (2004) Tipping point: From feckless reform to substantive
instructional improvement. Phi Delta Kappan. 85(61. 424-432.
Senge, P.M. (1990). The fifthdDiscipline: The art and practice o f the learning
organization. New York. Doubleday & Company
Sizer, T. R. (1992). Horace’s school: Redesigning the American high school.
Houghton Mifflin Co. NewYork.
Slavin, R.E. (2001). Putting the school back in school reform. Educational
Leadership. 58(4).
Smith, M.S., & O’Day, J. (1991). Putting the pieces together: systemic school
reform. CPRE Policy Briefs, Consortium for Policy Research in Education,
University o f Pennsylvania. [Online]. Available:
http://www.cp.re.org/Publications/rbQ6.Ddf
Stedman, L. (1996). The achievement crisis is real: A review o f The Manufactured
Crisis. Arizona State University, Education Policy Analysis Archives 4(1)
[Online] Available: http://epaa.asu,edu/epaa/v4n. 1 .html
Stevenson, H.W. (1998). A TIMSS primer: Lessons and implications for U.S.
education. Thomas B. Fordham Foundation.
Supovitz, A.A., & Klein, V. (2003). Mapping a course for improved student
learning: How innovative schools systemically use student performance data
to guide improvement. Consortium for Policy Research in Education,
University o f Pennsylvania. [Online]. Available:
http://www.cpre.org/Publications/AC-08.pdf
Takahira, S., Gonzales, P., Frase, M., & Salganik, L. H. (1998). Pursuing
Excellence: A Study of U.S. Twelfth-Grade Mathematics and Science
Achievement in International Context. U.S. Department o f Education.
National Center for Education Statistics, NCES 98-049, Washington D.C.:
U.S. Government Printing Office. [Online]. Available:
littp://nces.ed.gov/pubsearch/piibsinfo.asp?pubid=98049
Thompson, S. (2001). The authentic movement and its evil twin. Phi Delta Kappan.
82(5), 358-362.
227
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Tucker, M. (2002). Building the capacity of schools, districts and states to educate
all students to high standards: The case o f the America’s Choice school
design. National Center on Education and the Economy, Washington, DC.
[Online]. Available:
http://colosus.neee.org/pdf/acsd/global/promo/rand paper.pdf
Tucker, M. (2002b). High school and beyond: The system is the problem- and the
solution. National Center on Education and the Economy, Washington, DC.
[Online]. Available:
http://colosus.ncee.org/pdf/acsd/g1obal/promo/gates paper.pdf
U.S. Department o f Education. (2002). Testing for results: Helping families, schools
and communities understand and improve student achievement. [Online].
Available: http://www.ed.gov/nclb/accoimtability/avp/testingforresults.htnil
Wade, H.H. (2001). Data inquiry for analysis and educational reform. ERIC Digest
153, (ED 461 911), Clearinghouse on Education Management. Eugene, OR.
[Online], Available:
White, S. (1996). NAEP 1994 reading report card for the nation and the states.
National Center for Education Statistics, NCES 96-045, Washington D.C.:
U.S. Government Printing Office. [Online]. Available:
http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=96045
Whitehouse. (2002). President signs landmark education bill. Office of the Press
Secretary, January 8, 2002. [Online]. Available:
http://www.whitehouse.gOv/news/re1eases/2002/01 /20020108-1 .html
228
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data Study
Case Study Guide
The purpose of the Case Study Guide is to provide a framework for data
collection and data reporting in Chapter 4 o f your dissertation. Note that the
conceptual frameworks for the case study guide are Conceptual Frameworks A and
B. During data collection, the conceptual frameworks and the guiding questions
should therefore serve as the focus when conducting formal and informal interviews
and while informally observing at the district and site levels. You do NOT need to
ask each o f the guiding questions during the interview, but instead use the guiding
questions and the conceptual frameworks to make sure you are getting the
information that you will need in the data collection.
Your case study should be written up as a major part o f Chapter 4 o f your
dissertation. The headings and questions from Conceptual Frameworks A and B will
serve as the guide in the write-up of your district and school as a case study. You
do not need to submit a case study write-up except as it appears in your Chapter 4.
Staff Interviews
You are to formally interview at least one district leader, preferably the
superintendent or an assistant superintendent, one site principal and six teachers.
The teachers should come from both positions o f leadership in the school (i.e.,
chairs, coordinators, etc.) as well as typical classroom teachers. For each interview
provide a description o f the interview with relevant background information about
the person. Please include, if applicable, the title, years o f experience, years of
230
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
experience in current position, and position prior to the current one. The individual
researcher should create data collection instruments for the formal interviews.
Remember to focus on the questions from Conceptual Frameworks A and B.
Observation
All o f the observations in this case study will be informal. Observing for
environmental information should be a part o f all on site data collection days.
Valuable data can be gathered while informally observing and interviewing district
and school staff. The researcher should be collecting evidence from district and
school records, district and school policies, district and school publications, and by
other means. This analysis of this evidence is very important. In addition, informal
observations will aid in the formation o f vignettes.
Artifact Analysis
The artifact analysis is an informal environmental school and district data
collection. While informally observing and interviewing district and school staff,
collect evidence o f the following:
• State Reports (State of California Department o f Education webpage, STAR
reports, API, etc.)
• District Records (Check with research department, Superintendent’s office, etc.)
• School Records (check files, with Office Manager, testing coordinators, etc.)
• District Correspondence (Bulletins, Memos, Addendum’s, Handbooks, School
Data Systems (i.e., SIS)
• School Correspondence (Bulletins, Memos, etc.).
231
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• District Publications (District Newsletters, Newspaper Articles, Training
Manuals, etc.)
• School Publications (Monthly Newsletters, PTA Bulletins, School Handbook,
etc.)
• Classroom Publications (Parent Newsletters, Back to School Night Agendas,
Classroom Newspapers, etc.)
• Teacher Management Systems (goal setting charts, portfolios, rubrics, lesson
plans, etc.)
Reporting Out
The Case Study Guide requires that all o f the questions under Conceptual
Frameworks A and B be answered. A short explanation should be provided if no
data was collected to address a question. This will serve as a guide for the case
study portion o f your Chapter 4.
Provide several examples o f data use on the district, school site, and
classroom level.
232
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Suggested Quantitative Data
Data (Current) Collected Data (Emerging) Collected
API Reports
(1999-2001)
Performance
Assessments
(Multiple Years)
SAT-9 scores
(1999-2001)
ELA scores (math,
science, S.S.)
STAR
CST (STAR
augmentation)
HSEE
CELDT
(California English
Language
Development)
Other Other
Indicate for every data set that the data was collected or if necessary indicate
that the data was unavailable. If possible, accept or make estimates, but clearly
indicate that the data was estimated and what information was used to make the
estimation.
Mapping of Data Flow
Create a schematic map or drawing (flowchart) o f data analysis from the district
to the school site and ultimately to the classroom.
• How does the State o f California disseminate/ record data from school districts
and individual schools (specifically look for mode, pathway, and recording
mechanisms)? You may want to look at API scores, SAT-9, HSEE, Performance
Assessments, etc.
• How does the school district analyze the data and disseminate it to the local
school sites? Does the district do research based on this data?
233
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• What system o f checks and balances are in place to see if this is carried out
effectively (if any)?
• How does the district level implement new state data mandates (i.e., High
School Exit Exam)? What is the procedure and who is involved?
• Is any training conducted throughout the district that involves data on the
district/ local school site level?
• Can you identify any innovative practices with data (i.e., emerging practices)?
• What occurs with data use at the local school site level? Who is in charge o f
disseminating the data? How is it used at the local school site?
• What training about data use is conducted at the local school site?
• What emerging data practices do you see present at the local school site?
• What part do department/grade level chairs play with use o f data?
• How is the average classroom teacher using data?
• Are data used in the regular classroom to drive instruction?
• Are students involved in the data analysis process?
These questions should help define your informal observations, informal
interviews, and situated interviews. After you have gone through these questions,
create a flow chart to illustrate the data trail in the district you are conducting your
data collection. This may be as simple or complex as needed. In addition, this may
be used as a reference table in your Chapter 4.
234
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
At some point you will need to reconcile the proposed district plan for using
data and what you actual observe at the school site. You may want to create two
data maps if you find a large discrepancy between the intended and actual flow of
data.
235
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data Collection Schedule
The data collection should occur in two rounds. The days for each round
would ideally be scheduled close together, or if possible, consecutively. The
schedule should serve as a guide; the researcher may need to alter the schedule
based on circumstances.
Round 1
Day 1 District level interviews; collect artifacts and quantitative data.
Day 2 Site level interviews with teachers (case study oriented and
situated interviews) and administrator; collect artifacts including
quantitative data.
Day 3 Continue interviews and pass out Teacher Questionnaire and
Stage o f Concern (8 mins. each) to all teachers on staff.
Break for preliminary data analysis.
Begin mapping the data flow in the district, school site, and
classroom. Create a schematic.
Round 2
Day 4 reentry with principal; revisit any issues with the artifact
analysis.
Day 5 Finish any unfinished situated interviews to form vignettes.
Day 6 Collect teacher questionnaires/ stages o f concern and exit with
principal.
236
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Debriefing
Day 7 Post Data Collection Debriefing with dissertation study group
at USC to fill out Researcher Rating Form in mid-January. Address
Innovation Configuration.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The Relationship of Data Collection Instrumentation to Research Questions
Data Collection
Instruments
RQ1: Design RQ 2:
Implementation
RQ 3: Adequacy
of Design
Case Study Guide
• Interviews: District
Administrator, Site
Administrator, 6
Teachers (made up
of grade level/
department leaders
and average
teachers)
• Mapping o f Data
Flow at District and
School Site
• Artifact Analysis/
Collection
• Quantitative Data
X X X
Situated Interviews
• Forming vignettes
with 6 teachers
X
Teacher Questionnaire X
Stages of Concern
Questionnaire
X
Researcher Rating
Form (Post Data
Collection)
X
Innovation
Configuration
(Post Data Collection)
X
238
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
D ata Use Study
Conceptual Frameworks A and B
Conceptual Framework A
Description o f Data Use Policies and Strategies: The Design
• Student Performance Assessed in the Context of Current and Emerging
Instruments
• Overview o f the Elements o f District Design o f Data Use to Improve Student
Performance
• District Decisions and Rulings that Support Use o f District Design
• Intended Results o f Design Plans to Improve Student Performance (District,
School, and Classroom)
• Data Use Policy and Strategy Funding
Conceptual Framework B
Implementation o f Data Use Policy and Strategy in Practice
• Degree o f Design Implementation
• Implementation o f Current Data Practices (District, School Site, and Classroom)
• Implementation o f Emerging State Data Practices (District, School Site, and
Classroom)
• Accountability for data use at district, school, and individual level
• Improving Student Achievement through Implementation of Data Use
239
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data Use Study
Conceptual Framework A
Description of Data Use Policies and Strategies: The Design
This conceptual fram ework addresses research question 1: What is the district
design fo r using data regarding student performance, and how is that design linked
to the current and the emerging state context fo r assessing student performance?
1. Student Performance Assessed in the Context o f Current and Emerging
Instruments
Current
• Are the district and school focusing on improving Stanford 9 scores and API
ratings as defined by the State?
• Are the district and school using the California Content Standards to improve
student performance?
• Are authentic assessments linked to state standards throughout the district,
school, and classrooms?
• Are norm-referenced assessments used?
• Are interventions with students linked to performance assessments?
• Is there awareness on the district and school levels o f the Stanford 9, API scores,
etc.?
• Is state-of-the-art technology used to address current and emerging student
assessments on the state, district, and school levels?
240
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Emerging
• Is there awareness on the district and school levels o f emerging state
assessments?
• Is there preparation for the High School Exit Exam (Sr. High only)?
• Is there preparation for the California English Language Test (CELDT)?
• Is there evidence o f district planning to prepare students for emerging
assessments?
• Are there examples of the use of criteria-referenced tests on the district and
school levels?
• Is there measurement o f student performance against international performance
standards?
2. Overview o f the Elements o f District Design o f Data Use to Improve Student
Performance
• What types o f data are collected in the district design o f data use?
• What is the timeline for the receipt and use o f data in the district?
• What assessment instruments is the district using to collect data?
• What is the rationale for collecting these instruments?
• What methods are used to analyze these data?
• What is the rationale for selecting these methods o f analysis?
• Did reputable research guide the district’s design o f data use?
• What methods does the district have for disseminating data?
241
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• What training does the district provide for improving/modifying data?
• Are multiple stakeholders involved in the decision-making process within the
district?
• What steps do district and schools take to use data to improve student
performance?
• What outside influences affect policy and strategy design?
• What inside influences affect policy and strategy design?
• What role does fiscal and funding issues play in the design of the policies and
strategies?
• What state legislation was considered in the design?
• What current data practices are included in the policy?
• What emerging data practices are included in the policy?
3. District Decisions and Rulings that Support Use o f District Design
• What board rulings directly support the district design?
• To what degree are multiple stakeholders involved in establishing/ influencing
boards?
• Are research studies guiding the board support o f the district design?
• Does the board consider State legislation?
• What is the timeline to implement the district design?
• Is there board mandated training for the district design o f the use o f data?
• What is the process for establishing district-wide high-performance goals?
242
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• Is there money allotted to develop and implement the district design?
• Is technology designated to be used in the implementation o f the design?
• Are outside sources used to implement the design?
4. Intended Results o f Design Plans to Improve Student Performance (District,
School, and Classroom)
District
• What are the intended results of district design?
• Does the district want to use data to increase student performance, and is there
an increased focus on improving student performance?
• Was there a process created to establish district-wide high-performance goals?
• Does the district support schools’ efforts to improve student performance by
providing student data?
• Does the district use multiple measures to increase performance and guide
instruction?
• Does the district provide schools with the results o f these multiple measures?
School
• What is the school doing to use data (directly or indirectly) to promote student
learning?
• Are there any school-wide plans/ efforts to improve how data are gathered,
analyzed, and used?
243
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• Does the school try to improve administrator’s and teachers’ ability to use data
to increase student performance through training, etc.?
• Does the school try to improve administrator’s and teachers’ ability to analyze
data effectively to guide instruction?
• Is the school trying to raise student performance expectations with teachers and
parents?
• Does the school administrator(s) support school-wide implementation of
standards-based curriculum to improve student performance?
• Does the school regularly inform the students, parents, and community of
student performance?
• Was the effectiveness o f instructional programs evaluated?
• Were there any roadblocks or challenges to making these improvements?
Classroom
• Are teachers able to effectively analyze performance data and other forms o f
data?
• Are schools training teachers to improve their ability to use data?
• Are administrators and teachers using standards and rubrics to improve student
performance? Are teachers executing high-level standards-based curriculum?
• Are classroom teachers using data to evaluate the effectiveness of instructional
programs?
• Are teachers placing a high priority on improving student performance?
244
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• Are students expected to raise their ability and achieve high performance goals?
• Are classroom teachers regularly informing students, parents and community of
student learning performance and how they can assist student improvement?
5. Data Use Policy and Strategy Funding (District, and School Site)
• How is data use implementation funded?
• Does the district control the budgeting and funding o f the programs and
strategies, or do schools have a role?
• Do all grade levels and subject areas receive equal shares o f funding?
• How is the funding affecting curricular and instructional decisions?
• Where is additional funding needed?
245
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data Use Study
Conceptual Framework B
Implementation of Data Use Policy and Strategy in Practice
This conceptual fram ework addresses research question 2: To what extent has the
district design actually been implemented at the district, school, and individual
teacher level?
1. Degree o f Design Implementation ('in the current and emerging contexts)
• To what degree does the school have knowledge o f the district design?
• Were the district policies and strategies fully implemented at the school?
• How long did this implementation take? Was it phased in? What was the
timeline?
• Who was given the job o f implementation at the site level?
• What factors were considered important in the implementation?
• How comfortable is the school with the district design?
• What outside influences affected implementation?
• What roadblocks were identified in implementation?
• Was technology used to implement the design?
• Were outside sources used to implement the design?
• Was there a timeline for the receipt and use o f data from the district or state?
246
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• What data assessment instruments are used at the school site?
• How supportive is the principal (and other site administrators) of the district
design?
• Are school in-services being offered on how to use data effectively?
• Can the school site link implementation o f the district design with improved
student performance?
• Does the school have a method for communicating standards-based curriculum
and assessments to students, parents, and community?
• What data was collected on student performance?
• Are current data practices implemented?
• Are emerging data practices being implemented at the school?
• Is the district plan implementing current data practices occurring: Sat-9, API,
norm-referenced tests etc.?
• Is the district plan implementing emerging data practices: criteria-referenced
tests, performance assessments, international comparisons, High School Exit,
etc.?
2. Accountability for data use at district, school, and individual level
• How are the district policies and strategies monitored for effectiveness?
• Where is the data stored?
• How is the data disaggregated?
• What is done with the data?
247
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• Who is accountable for the implementation o f the district’s plan to improve
student achievement through the use o f data at the school site?
• What responsibility do classroom teachers have for linking instructional practice
to data use?
• What responsibility do students have to be accountable for their achievement?
3. Improving Student Achievement through Implementation o f Data Use
• Does the implementation of data use show any student achievement gains at the
district level and school site level?
• What assessments were used?
• What elements o f data use are the most effective in bringing about student
achievement gains?
• How is data used in the classroom to improve student performance?
• Do classroom teachers keep track of students’ growth? How?
248
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
School Interview Guide - Leadership Team, Question #3
CFa
• Describe the inservices that were provided to allow you to effectively use
student data as a team.
• What did the data tell you about how well all student groups are performing?
• Describe the relative effectiveness o f individually reviewing student data as
compared with evaluating it in a collaborative team environment?
• How often do you work in collaborative units to evaluate student
performance?
CFb
• How do you know how to alter instruction to improve student performance?
Give an example (vignette).
• How do you determine that your instruction is promoting strong student
performance? Give examples (vignette).
CFc
• What standardized assessments do you use to evaluate student performance?
• How do you ensure that parents understand standards-based instruction and
assessments?
249
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• What are the barriers from a teaching perspective, to make sure that all
students perform at a high level?
• What changes would have to made to reduce these barriers?
• How are your report cards based on state standards?
• Describe how you know when a student is performing at a “high” level.
250
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
School Interview Guide - Principal, Question #3
CFa
• What standardized assessments were forwarded to the school site?
• Which o f the assessments are aligned to state standards?
• What assessments disaggregated student performance data?
• What strategies do you use to assist your teachers in understanding the
information in the assessment data?
• What reflective structures do you promote among your teachers to enhance
necessary changes to instruction based on student performance data? Give
examples (vignettes).
CFb
• How do you motivate teachers to maintain high energy and enthusiasm
towards the improvement o f student performance? Give examples
(vignettes).
• What methods and instruments do you use to evaluate student performance?
• How frequently does the school evaluate student performance?
251
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CFc
• How are teachers, students and parents made aware o f the learning goals of
standards-based assessments?
• Describe high performance by students as it relates to state standards.
• What strategies are used at your school to ensure that all students achieve
levels o f high performance. Give examples (vignettes).
• What are some of the barriers you perceive to developing a high level of
student performance in a standards-based environment?
• Describe ways that these barriers can be reduced or eliminated.
252
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
District Interview Guide - Superintendent, Question #3
CFa
• Describe in-services developed by the district to promote the successful
implementation o f standards-based instruction.
• What in-services were developed by the district to assist school staff in the
use o f student data?
• What standards-based assessment data was forwarded to the school?
• How was student performance data disaggregated for the schools?
• How many o f the assessments were based on state standards?
• In what ways is the district helping schools prepare for future changes in
state assessments?
CFb
• Describe state and district incentive plans that are in place to support high-
quality standards-based student performance?
• How often do district personnel review student performance data?
• What corrective actions are taken by the district when student data indicates
that student achievement is not at an acceptable level? Give examples
(vignettes).
• What district strategies are used to determine possible strategies for
improving student performance on assessment instruments?
253
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CFc
• What are the district’s “high” performance goals for students?
• How do district performance standards correspond to state standards?
• Describe district strategies for ensuring that all students master the state
standards for their grade-level?
• In what ways does the district communicate to parents about standards-based
curriculum and student assessment results?
• Describe the district process for determining what student data is to be
disseminated to schools?
• What is your overall assessment o f student achievement as it relates to state-
based standards?
254
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

How districts and schools utilize data to improve the delivery of instruction and student performance: A case study

PDF

Connecting districts and schools to improve teaching and learning: A case study of district efforts in the Los Coyotes High School District

PDF

An analysis of the impact a district design on the use of data has on student performance

PDF

Design, implementation and adequacy of data utilization in schools: A case study

PDF

An analysis in the use of student performance data in schools

PDF

An analysis of student -level resources at a California comprehensive high school

PDF

An analysis of the use of data to increase student achievement in public schools

PDF

Design, implementation and adequacy of using student performance data and the design's link to state context for assessing student performance: A case study

PDF

Addressing social promotion and retention issues: A look at one middle school in the process of reform

PDF

An analysis of the elementary principal's role in implementing school accountability within California's High Priority School: A case study

PDF

How effective schools use data to improve student achievement

PDF

A case study of social promotion and retention policies, strategies, and programs

PDF

A case study of teacher evaluation and supervision at a high -achieving urban elementary school

PDF

An analysis of the implementation of content standards in selected unified school districts of California

PDF

Implementation challenges of a school district's technology plan at the middle school level

PDF

Cognitive and motivational factors that affect the achievement of elementary students on standardized tests

PDF

An analysis of the elementary principal's role in implementing school accountability within California's high -priority school: A case study

PDF

Implementation and levels of use of technology plans in a Southern California K--12 district and its middle school

PDF

A longitudinal look at what's important in comprehensive reform: A case study

PDF

A longitudinal comparative study of the effects of charter schools on minority and low-SES students in California

Asset Metadata

Creator Zepeda, Ruben, II (author)

Core Title A case study: An analysis of the adequacy of one school district's model of data use to raise student achievement

School Rossier School of Education

Degree Doctor of Education

Degree Program Education

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag education, administration,Education, Secondary,education, tests and measurements,OAI-PMH Harvest

Language English

Contributor Digitized by ProQuest (provenance)

Advisor Marsh, David D. (committee chair), Gothold, Stuart (committee member), Picus, Lawrence O. (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c16-462680

Unique identifier UC11336630

Identifier 3196923.pdf (filename),usctheses-c16-462680 (legacy record id)

Legacy Identifier 3196923.pdf

Dmrecord 462680

Document Type Dissertation

Rights Zepeda, Ruben, II

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA