University of Southern California Dissertations and Theses

An analysis in the use of student performance data in schools

(USC Thesis Other)

An analysis in the use of student performance data in schools

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content AN ANALYSIS IN THE USE OF STUDENT
PERFORMANCE DATA IN SCHOOLS
by
Paul John Sevillano
A Dissertation Proposal Presented to the
FACULTY OF THE ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
Doctor O f Education
December 2002
Copyright 2002 Paul John Sevillano
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UMI Number: 3093916
Copyright 2002 by
Sevillano, Paul John
All rights reserved.
®
UMI
UMI Microform 3093916
Copyright 2003 by ProQuest Information and Learning Company.
All rights reserved. This microform edition is protected against
unauthorized copying under Title 17, United States Code.
ProQuest Information and Learning Company
300 North Zeeb Road
P.O. Box 1346
Ann Arbor, Ml 48106-1346
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UNIVERSITY OF SOUTHERN CALIFORNIA
School of Education
Los Angeles, California 90089-0031
This dissertation, written by
, •''T h k .o
under the direction of h___ Dissertation Committee, and
approved by all members of the Committee, has been
presented to and accepted by the Faculty of the School
of Education in partialfulfillment of the requirementsfor
the degree of
D o c t o r o f E d u c a t io n
D-ene.n.hfi.r._J.a.T ._-2DQ2
Bean
Dissertation Committee
Chairperson ,
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ACKNOWLEDGMENTS
I wish to acknowledge a few special individuals whose contributions left an
indelible mark on me throughout the dissertation process. First, I wish to thank my
wife Karen for her unlimited support, love, and sacrifice. Second, thank you Lauren,
Michael, and Rachel for your love and understanding. Third, I want to thank father
Mando for his wisdom, unconditional love and support throughout the dissertation
process. Finally, I wish to thank Dr. David Marsh and my fellow colleagues for
challenging me to achieve at the highest level. Thank you all.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
TABLE OF CONTENTS
PAGE
ACKNOWLEDGMENTS ii
LIST OF TABLES v
ABSTRACT vi
CHAPTER
1 BACKGROUND OF THE PROBLEM 1
Introduction 1
Statement of the Problem 11
Purpose of the Study 12
Importance of the Study 13
Definitions 15
Outline of the Remainder of the Study 21
2 REVIEW OF RELATED LITERATURE 23
Student Performance 23
Standard-Based Reform 32
Assessment and Accountability 43
The Role of Data Use in the U.S. 44
Expectations of Student Performance Data
In California 52
Public Schools Accountability Act 53
Academic Performance Index 55
Immediate Intervention/Under-Performing Schools 56
Governor’s Performance Awards 57
The Current Use of Data in Schools 59
Emerging State Context 66
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER PAGE
3 METHODOLOGY 69
Methods Description 70
Instrumentation 74
Data Collection Schedule 81
4 ANALYSIS AND INTERPRETATION OF FINDINGS 87
Introduction 87
Findings by Research Questions 90
Discussion of Research Findings 135
Conclusion 138
5 SUMMARY, CONCLUSIONS, AND IMPLICATIONS 140
Conclusions 143
Implications 146
Purpose of the Study 150
Overview 151
Research Design 151
Research Questions 152
Sample and Populations 152
Instrumentation 154
Conceptual Frameworks 154
Data Collection and Instruments 156
Main F indings 157
Implications 158
Recommendations for Additional Future Research 158
BIBLIOGRAPHY 160
APPENDICES 169
iv
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
LIST OF TABLES
TABLE
1
2
PAGE
Teacher Questionnaire Summary Analysis 114
Stages of Concern Summary Teacher Responses 116
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ABSTRACT
This case study was designed to investigate the methods in which school
districts use student performance data to improve student achievement. School
accountability has increased the emphasis on student achievement testing and data.
Literature findings on the use of student performance data in schools suggest that the
use of such data can assist both teachers and school leaders in the identification of
student achievement gaps both at the classroom level and program level. This study
investigates the district design for data use in schools and how the design is linked to
the current and emerging state context for assessing student achievement. The extent
of implementation of the district design was investigated and the adequacy of the
district design in regards to data use for improving student performance.
The qualitative study was part of a cross-site study with 13 different districts
and 15 schools in Southern California. Thirteen Ed.D. Doctoral candidates at a large
research institution worked as a cohort to investigate how schools and districts
throughout the state of California use data. This case study observed and investigated
data use practices and design systems from one school district, and one school site,
over a six-day period. Each researcher selected ethnically diverse urban schools that
have been recommended for having employed advanced practices in data use.
A researcher developed case study guide was used to organize, collect, and
document the multiple types of data. The case study guide provided the framework
vi
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
for data collection and data reporting. Conceptual frameworks were developed to
examine each of the three research questions. Data collection instrumentation
included staff interviews, situated interviews, observations, vignettes, questionnaires,
and document analysis. The combined findings of these dissertations formed this
larger comparative case study.
These case study findings substantiated current research on data use by
teachers, principals, and district leaders. These findings revealed that school
practitioners who use data effectively not only identified the standards and
assessments needed to measure student achievement, but established clear
benchmarks to assess student mastery and progress towards established standards.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 1
BACKGROUND OF THE PROBLEM
Introduction
The findings of many research and studies on the distribution of data
pertinent to student performance on standardized tests in the U.S indicate a long-term
trend of low student achievement in math and science during the past quarter
century. (Larson, Guidera, Nelson, Rodstag & Smith 1998). Math and science
achievement scores in the U.S. have changed little in comparison to other countries.
In 1997, the U.S. Department of Education released the results of the NAEP
Mathematics and Science Assessments study that was conducted from 1996-1998.
Those findings indicate that, “ ... over two-thirds of the U.S. students tested in grades
four, eight and 12 in 1996 were performing below the proficient level" (Blank &
Wilson, 2001, p. 23).
In 1997, the TIMSS study confirmed that 41 other countries outperformed the
U.S. Students in the U.S. were below average in eighth grade and almost last in 12th
grade (Blank & Wilson, 2001). Further analysis by Barton (2001) of NAEP data
between 1990 and 1996, indicated that the achievement gaps between whites and
minorities and the top and bottom quartile "generally" did not decline. However,
gaps in performance in eighth grade mathematics between the white and minority
students increased in two states with no narrowing (Johnson & Siegendorf, 1998).
Recently, politicians in Texas proclaim that minority students are performing
well on the Texas Assessment of Academic Skills (TASS). These data suggest that
1
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Texas accountability test results are mixed. Texas has built an assessment system
that promotes equity, thereby, narrowing the gap between minority and white
achievement, while increasing achievement for all students over time. The TASS
data indicate score discrepancies between white and minority students have
narrowed. However, the percentage of Black and Hispanic 6th graders who make it
to the 12th grade in Texas declined from 80% in the 80's to 70% in 1999 (Keller,
2001). As the student drop-out rate of minority students in Texas continues to
increase, the consequences for low student performance of minority students in
Texas is likely to have a long-term economic impact on the state in the years to
come.
Consequences of long-term, low student achievement in U.S. schools will
have serious impact on our ability to compete in the global economy (Education
Commission of the States, 1996). American students stack up poorly against students
from other industrialized nations in reading and math. While the existence of a well-
educated workforce does not guarantee economic vitality, it is clear that the nation
and its communities cannot sustain prosperity in the knowledge age without a highly
skilled workforce (Education Commission of the States, 1996). A 1998 study
conducted by the University of Pennsylvania showed that a 10% (or about one year)
increase in the education level of a company’s workforce increased productivity by
8.6%, while a comparable increase in capital equipment increased productivity by
3.4%. For non-manufacturing companies, the result was even higher at 11%
(Larson, Guidera, Nelson, Rodstag & Smith, 1998).
2
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The increased role of school accountability policies in comprehensive school
reform efforts in the U.S. has led to increased emphasis on student performance data
and raised equity issues for minority students (Conzemius, 2000). School
accountability has assumed many forms in education. Most recently, the external
model of accountability has policymakers holding schools accountable for what
ought to be assessed based on student outcomes (Pearson, Vyas, Sensale, & Kim,
2001). There are two distinct objections to increased school accountability policies.
These objections include the increasing use of standardized test dependence of states,
who uses standardized testing to drive the curriculum, and the effects of standardized
tests on minority students.
A strict dichotomy by which state accountability systems are viewed as either
all good or all bad also currently exists. Of particular concern are questions of
equity, notably in what ways accountability systems affect the education of low
income children of all races, especially children of color (Scheurich, Skrla &
Johnson, 2000). The use of high-stakes standardized assessments by state education
agencies has raised concerns of equity and learning opportunity issues for all
students.
Comprehensive school reform efforts in the U.S. have focused on a standards-
based reform approach in order to improve student performance and reshape the key
elements of school change and accountability nationwide (Odden, 1999). Standards-
based reform began in 1983 as a response to the national concerns that students in
America were not learning enough to compete in a global economy. Currently, 49
3
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
states (excluding Iowa) have adopted state standards. As a result of this adoption,
standards are changing and student achievements are increasing (Laboratory
Network Program, 1998).
Standards are called doctrines, and they call for a more intellectually
demanding content of studies for students. Standards specify what students should
know and learn at each grade level and what should be taught (Tucker & Codding,
1998). Unfortunately, the number of standards also has created some confusion
among educators. There are so many standards to assess that measurement people
are unable to do a reliable job of assessing students' mastery of them (Popham,
2001).
Assessments are an integral part of standards-based reform. Well-designed
tests can provide objective measures of how well students are progressing towards
meeting standards (Achieving the Goals of Standards-based Reform, 2001).
However, testing has gotten ahead of developing and implementing standards-based
reform efforts (Merrow, 2001). Teachers need time to align curriculum, instruction
and, assessments to the standards before testing students to see whether they have
attained them (Laboratory Network Program, 1998).
School funding is a key element of the standards-based reform efforts. A shift
in school finance in support of standards reform, challenges policymakers to identify
new school finance structures that are more directly linked to strategies that raise
student achievement (Odden, 2000).). For the standards-based reform movement to
work in schools, district finance systems need to be aligned more closely to
4
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
standards-based reform efforts and the national goal of teaching all students to
rigorous performance standards (Odden, 1998). Standards-based reform efforts in
the U.S. may be our best alternative yet in reforming schools, once the key elements
of standardized assessment, national standards agreement, and school finance are
decided. Standards-based performance data trends over time will assist
policymakers in the analysis and decision-making process of school reform.
The role that data/information about student performance and related matters
is intended to play in comprehensive reform is not always carefully explained by
scholars and policymakers and, therefore, can easily be misunderstood. Each state's
accountability system is a complicated arrangement of multiple and interacting
components, including the tests used, the grades ands subjects tested, and the levels
of performance mandated (Scheurich, Skrla & Johnson, 2000). Many states are
implementing assessment systems that do not contain a balance between external and
internal assessments in which to assess student performance. External assessments
or norm referenced standardized tests can provide data for making global decisions
on state, district, or school instructional program progress (Pearson, Vyas, Sensale
& Kim, 2001). Internal assessments provide direct information links between
teachers, parents, and students. Externally-driven state tests can monitor district and
state progress in effective and meaningful ways, but only if viewed as one piece in a
broader system of indicators.
Critics of standardized tests argue that today's evaluation of student
performance, based on students' scores on standardized achievement tests, do not tell
5
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
us exactly what and how well those students have been taught. Some studies suggest
that more than 50% of the content in many standardized achievement tests is not
taught in the curriculum of most schools (Popham, 2000). Critics argue that
standardized tests contribute to superficial coverage of content and less depth in the
curriculum (Blank & Wilson, 2001). In addition, pressure to raise standardized test
scores in schools results in teachers teaching to the test without changing the
students' knowledge or achievement (Hoffman, Czop, Assaf & Paris, 2001).
However, the current rhetoric of educational reform, sounds as if improving
the educational system is synonymous with improving test scores (Haetel, 1999).
While all the pieces of the puzzle are not in place, early returns from new state
testing systems are spurring changes that lead to improved student learning (Hoff,
2000). Advocates of standardized testing in schools are convinced that national
standards or state curriculum and testing programs are essential elements for
improving schools (Gillbom & Youdell, 2000). There seems to be little argument
that standardized performance assessments can affect the curriculum if the data is
used correctly to improve student performance, and guide instruction (Madaeus &
O'Dweyer, 1999).
The state of California has given large sums of money to schools that meet
their Academic Performance Index (API) growth targets on the Stanford Ninth
Edition Achievement Test (SAT 9) and other measures since the passage of Senate
Bill 376 in 1997 authorized the Standardized Testing and Reporting Program
(STAR) and (PSAA) Public Schools Accountability Act program (Alpert, 1999).
6
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Currently, state policymakers are investing millions of dollars into improving
statewide school performance and assessment practices as measured by the SAT 9.
In three years, the emerging California Standards Exams and the High School Exit
Exam for secondary schools will replace the SAT 9 Achievement Test for assessing
statewide student performance.
Policymakers in California, like those in other states, are detemiined to shift
the focus of educational reform from rules and process to student achievement and
accountability (Hart & Brownell, 2001). The Public Schools Accountability Act
(PS A A) of 1999 led the way for the development of the Academic Performance
Index (API) and growth targets for accountability. The statewide measure was
designed to affect student achievement through the use of sanctions and incentives
(Duffy, Goertz & Le Floch, 2001). PSSA is California's statewide accountability
system, its' comprised of three components that include the Academic Performance
Index used to measure and rank school performance. The Immediate
Intervention/Underperforming Schools Program is designed to provide support for
schools not meeting growth targets, and the Governor's Performance Award Program
designed to reward schools and individual teachers whose students show
improvement or high achievement.
The implementation of the Public Schools Accountability Act in California
has resulted in increased accountability pressure on schools and districts (Alpert,
1999). Schools are expected to examine SAT 9 standardized performance data
results, identify low student achievement gaps, determine appropriate intervention
7
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
strategies to improve school performance, and meet state established Academic
Performance Index (API) growth targets (Burr, Haywood, Fuller & Kirst, 2000).
Problematic with this current approach is the emerging state context for measuring
student performance in three years. It does not include SAT 9 measures, but instead
relies upon standards- based assessments for students.
The emerging goal of PSSA is to provide an accountability system in
California that will become standards-based. When fully developed, the API will
use various performance indicators to create a composite score for a school.
Currently, only the results of the SAT 9 were used to calculate the numeric scale
(Hart & Brownell, 2001). Over time, scores from the augmented Standardized
Reporting Tests will be phased in to assess student progress toward content
standards, and the High School Exit Exam for secondary schools. State content
standards will eventually align with new assessment practices and, in turn, create an
accountability system that monitors the progress of schools, holds them accountable
for helping students achieve high standards, and allocates resources where they are
needed (Baker, Brown & Herman, 2000).
Current research on the use of data in schools has indicated that the use of
student performance data can assist both teachers and school leaders in the
identification of student achievement gaps both at the classroom level and program
level (Holcomb, 1999). Several common themes were shared by high-performing,
high-poverty urban elementary schools in Texas. Educators aligned instruction with
the standards and assessments used to determine growth. They identified what
8
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
students need to know and be able to do at each grade level. Next, they planned
instruction to meet the standards and assessments (Charles A. Dana Center, 1999).
Many state education departments put little emphasis on schools gathering
data and, thus, provide little incentive for districts and schools to devote time,
money, and staff resources to using data in new ways (Bernhardt, 2000). Early
patterns on the use of data in schools reflected the hunches or guesses of student
hoped-for outcomes without reflection on data. The current use of data reflects
data-driven school improvement which encompasses a myriad of quantitative and
qualitative measures (Conzemius, 2000).
Schools who use data effectively not only identified the standards and
assessments needed to measure student achievement but established clear
benchmarks to assess student mastery and progress towards established standards
(Marx, Hunter & Johnson, 1997). Additionally, data were disaggregated and
resources were distributed based upon student needs. The disaggregation of data
involves the use of multiple measures of student performance to assess the strengths
and weaknesses of instructional programs (Khanna, Trousdale, Peneul & Kell,
1999).
School improvement requires that decision-makers make comparisons
between and among data, including school input and process variables (Johnson,
1997). Data-based decision-making is a process of inquiry, analysis, and decision
making inferences from the interplay of these input, process and outcome measures
(National Education Association, 2000). New technologies will significantly affect
9
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
educational decision-making. First, they will reduce the time necessary to gather and
analyze data; second, they will enable school leaders to test hypotheses and quickly
solve problems through the analysis phase of data-based decision-making process
(Streifer, 2000).
Technology and private industry can provide support for schools in data
analysis, staff development, and instructional program evaluation to improve student
performance. The advent of new computer technology and software systems will
allow states to retrieve and disperse accountability data quickly (Christie, 2000). In
Ohio, the state legislature created the Educational Information Management System,
an infrastructure to house data necessary to determine how well schools are doing.
The Texas legislature established a similar system in 1995 (Keller, 2001). On school
report cards published in Texas, data are disaggregated by student groups: African
American, Hispanic, White, and economically disadvantaged. In 1999, the
Massachusetts legislature required each school district to adopt and maintain a
reliable data collection system (Christie, 2000). Part of the system requires each
district and charter school to have a unique and permanent and unduplicated ID
number for each student. In Minnesota and California, each state possesses a
website that gives the education system and common data elements for its public
schools.
As the amount of data used in schools has increased, so has the use of data on
student achievement to inform decision-making. The most striking trend in school
districts in their use of data is for schools to be data-driven (Goertz, 2000). Data can
10
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
be complied within different ways to inform instruction. School leaders can no
longer ignore the value of data to assist them in making data-based decisions. Using
data to judge performance extends to the classroom level teachers as well.
Ask any classroom teacher whether a standardized test data tells
them what to do tomorrow with a student, and they will tell you
no. However, give them some important data such as student
work, combined with standardized test scores and they can tell
you what is going on in terms of student outputs (Goertz, 2000).
Much remains to be learned by educators in terms of the use of data to judge
student achievement. Until current research in this field improves, school districts
must continue to discover best practices in the use of data by trial and error. Further
investigation is needed to investigate district design systems in the use of data and
adequacy of those district design systems. Only then, can we truly measure the true
impact of using data to increase student achievement in schools.
Statement of the Problem
In an era of accountability, assessment of student achievement has created a
focus on testing and data. As increased emphasis on student achievement and
accountability policies unfold across the nation, federal, state, and local communities
have focused their concerns on the measurement of student performance. Each state
has developed accountability systems that rely heavily on testing and the use of data
as the primary measure of student success.
In California, there is a growing belief that the use of student performance data
can bring about school change and comprehensive school reform (Burr, Hayward,
11
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Fuller & Kirst, 2000). The use of data to increase student achievement has provided
the state with a key measurement tool to evaluate progress towards increased student
achievement. Although many schools, districts, and local communities have adopted
policies and legislation for assessing student achievement, more information on the
use of data needs to be known.
There is evidence that school leaders, teachers, and administrators are not
using data effectively in order to bring about increased student achievement
(Khanna, Trousdale, Peneul & Kim, 1999). Furthermore, questions remain in
regards to the quality of district data use design systems. Currently, school districts
are struggling with implementation and adequacy concerns of design systems that are
linked to the current and the emerging state context for assessing student
performance. Throughout the state, key elements of data use are in place (Alpert,
1999). However, increased knowledge regarding the extent in which districts use
and implement design systems to assist school leaders, teachers, and administrators
in the use of data to improve student achievement remains unknown. Additionally,
much remains to be learned about the adequacy of such design systems in school
districts throughout the state.
Purpose of the Study
The purpose of this study is to investigate the district design for data use in
schools and to determine how the design is linked to the current and emerging state
context for assessing student achievement. The extent of implementation of the
12
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
district design also will be investigated. In addition, this study will analyze the
adequacy of the district design in regards to data use for improving student
performance.
Consistent with the major purpose of this investigation, the following specific
research questions were posed:
1. What is the district design for using data regarding student
performance, and how is that design linked to the current and the
emerging state context for assessing student performance?
2. To what extent has the district design actually been implemented at
the district, school, and individual level?
3. To what extent is the district design a good one (use of multiple
criteria)?
Importance of the Study
The importance of this study is to provide a paradigm shift in education
among practitioners to place an emphasis on schools to implement data use designs
and provide support for schools in their use of data to improve student performance.
This study is designed to assist state policymakers in the use of data to assess schools
and link statewide assessment practices to standards-based reform efforts. Currently,
state assessment policies are not directly tied to data use and standards-based reform
efforts. Findings from this study will assist policymakers in linking standards-based
reform policies to assessment practices in schools.
13
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
District administrators will benefit from a study such as this, for it will assist
them in the analysis of data design systems to improve student achievement. The
analysis of effective data design systems will provide knowledge to district level
practitioners the quality indicators for design and design implementation systems
that are tied to emerging state assessment practices.
This study will provide knowledge that will assist school leaders and
principals in analysis and use of data to increase student performance in schools.
Evaluative understanding will assist practitioners in the effective use of data to
evaluate and improve instructional practices in schools. Additionally, this study will
provide assistance in staff development to school leaders and principals, particularly
in the area of effective data use practices.
Teachers also will benefit from these findings on the effective use of data by
improving student assessment and student performance in the classroom. Teachers
can utilize data more effectively to align instruction and provide intervention
strategies to meet student needs and improve student achievement.
The researchers will benefit from this study both as practitioners in the field
of education and discovery of new knowledge in their use of data to improve student
performance. Additionally, the researchers will learn current techniques and
approaches of data use design linked to both the current and emerging context for
assessing student performance.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Limitations
The following limitations are noted:
1. The district design may reveal divergent outcomes regarding the use
of data in the examination of student achievement.
2. The district design may reveal unaccounted for support mechanisms
in the use of data that are linked specifically to increasing student
achievement.
3. Teacher knowledge and practices of data use may skew data results.
4. School leader perceptions of data use and best practices may be
different from teachers.
Delimitations
1. School case study sample was confined to one school district only.
2. Data gathering was designed to measure each subgroup use of data.
3. Small sampling size may limit the findings and generalizability of this
study.
Definitions
The purpose of the definitions listed below is to provide clarification for the
reader. These definitions and acronyms are used extensively throughout the study
and are taken directly from the literature.
15
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
For the purpose of this study, the following terminology were defined as:
accountability: A set of clearly defined standards and goals for student performance.
Accountability models developed by state policymakers or educators tells schools
what they should accountable for and hold themselves accountable for in public
education (Pearson, Yyas & Sensale, 2001).
adequacy: School-level data that can be used to establish measures for adequate
resource levels needed to educate students and inform districts to meet specific
achievement measures (Odden, 1998).
alignment: Curriculum activities and assessments that reflect alignment of
curriculum with content and performance standards that states have established for
students. They can be content specific that possesses themes, priority principles,
concepts, and topics of the discipline (Me Neil, 2000).
assessment: An instrument, tool, process, or exhibition composed of a systematic
sampling of behavior for measuring student's competence, knowledge, skills or
behavior towards leaning outcomes identified in state curriculum frameworks
(Stiggens, 2001).
authentic assessment: Performance assessments that demonstrate student mastery of
a particular content through student work samples of portfolios, speeches, skits,
projects and extended papers (Haertel, 1999).
best practice: The collection of knowledge on best educational practices or lessons
learned, the sharing and understanding of those practices and lessons so they can be
16
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
used, and the adaptation and application of those practices and lessons for the
purpose of intervention or innovation (Cistone & Stevenson, 2000).
comprehensive school reform: Reform that seeks to have schools create adopt, or
adapt an educational approach that integrates all students and programs into a
cohesive, schoolwide educational strategy (Odden, 2000). _
content standards: Sometimes called curriculum frameworks, content standards
define the specific subject matter that students are expected to master (Nave, Miech
& Mosteller, 2000).
data: A collection of things known or assumed: facts, figures, records, statistics-
information that describes or from which conclusions can be inferred. A piece of
data can be a number (statistical/ quantitative data) or a paragraph of text (textual
data) (Patton, 1999).
data-based decision-making: Data serves as a basis for answering questions and
making decisions (Tucker & Codding, 1998).
demographic data: Demographic data about schools that include student population,
enrollment, LEP students, percentage of students eligible for free or reduced lunch,
gender, ethnicity (Reeves, 2000).
disaggregation: Student achievement and behavioral data that can be analyzed by
breaking down scores into various categories or variables (Khanna, Trousdale &
Penuel, 1999).
district design: The district design is the design a school district uses to collect or
train administrators on data use or analysis of school data. This can include the
17
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
methodologies used or practices on data use, or resources expended to use data
effectively at the district or school site level (Marsh, 2001).
emerging state assessments: When fully developed the California Academic
Performance Index will use various performance indicators to create a composite
score for a school. Over time scores from the augmented Standardized Testing and
Reporting program with test questions selected to assess student progress towards the
content standards (Hart & Brownell, 2001).
high-stakes tests: A high-stakes test means that one test is used to make important
decisions about students, teachers, and schools (Ransom et al., 1999).
incentives: The rewards a system offers to those who are instrumental in raising
student performance (including the students) and the consequences it provides for
those who are not (Tucker & Codding, 1998).
multiple measures: The effective use of a variety of data both standardized tests and
classroom performance assessments to inform and make critically important
decisions about school (Patton, 1999).
outcome data: Describes and educational system in terms of standardized test
results, grade point averages, graduation rates, and other formal assessments
(Bernhardt, 2000).
performance indicators: Performance indicators are statistics that indicated
something about the performance or health of a district, school, or program
( Educational Resources Information Center, 1996).
18
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
performance standards: Written standards consisting of performance descriptions,
samples of student work, and commentaries on that work. The performance
description is a succinct narrative statement of what students are expected to know
and be able to do that describes what is most essential to learn in each discipline and
is confined to things that can actually be assessed. Performance descriptions are
often referred to as content standards (Tucker & Codding, 1998).
process data: School process data define programs, instructional strategies, survey
trends, and classroom practices (Bernhardt, 2000).
professional development: The search for information and professional competence
that a faculty engages in to improve their capacity to help students in their school
meet the standards (Tucker & Codding, 1998).
qualitative data: Data presented not in numerical form at the onset. These data can
include open-ended surveys or questions (Patton, 1990).
quantitative data: Data that can either start out in numeric form or may be placed
easily in categories that can be assigned numbers (Patton, 1990).
raw data: The measurements of objects of your research. The raw data may come in
one of two basic forms: qualitative or quantitative.
site-based management: The practice, implemented in a number of districts, of
shifting authority over school policies from the central office to local schools
(Tucker & Codding, 1998).
standards: Standards are doctrines that call for more intellectually demanding content
of studies for students. These standards specify what students should know and learn
19
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
at each grade level, and, by extension, what should be taught (Tucker & Codding,
1998).
standards-based education: A way of operating schools and education systems so that
standards for student performance are at the center, and the sole objective for
everyone in the system is ensuring that students meet the standards (Tucker &
Codding, 1998).
standardized achievement test: A test for which procedures have been developed to
ensure consistency in administration and scoring across all testing situations. Any
examination that is administered and scored in a standard, predetermined manner
(Popham, 2000).
student achievement data: Student achievement data are student scores on
standardized achievement tests that include but not limited to the following
standardized assessments: SAT 9, CELDT, Advanced Placement Exams, Academic
Performance Index, Student Aptitude Test, ACT, Golden State Exams, California
Standards Examinations (Marsh, 2001).
student performance data: Student performance data are student scores on
classrooms based assessments that include but not limited to the following : student
portfolios, classroom tests, student projects, classroom grades, authentic assessments
(Marsh, 2001).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Acronyms
For the purpose of this study the following acronyms are defined as:
API: Academic Performance Index
CAHSEE: California High School Exit Examination
GPA: Governor’s Performance Awards
II/USP: Immediate Intervention Underperforming Schools
LE£: Limited English Proficient
NAEP: National Assessment of Educational Progress
NASB: National Association of School Boards
PSAA: Public Schools Accountability Act
SAM: Student Achievement Model
SES: Socioeconomic Status
STAR: Standardized Testing and Reporting
SAT-9: Stanford Achievement Test Ninth Edition
TASS: Texas Assessment of Academic Skills
TIMSS: Third International Mathematics and Science Study
Outline of the Remainder of the Study
The remainder of the study is presented in four chapters. Chapter 2 presents a
review of relevant research literature and current research conclusions. Chapter 3
describes the methods and procedures of the inquiry with particular emphasis upon
the measures employed and the methods of data analysis. Chapter 4 sets forth the
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
findings of the study within the framework of each of the three research questions
posed in Chapter 1. Chapter 5 provides a summary of this investigation along with
the author’s conclusions and recommendations.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 2
REVIEW OF RELATED LITERATURE
Student Performance
Measurement of data pertinent to consistent student performance in the U.S.
provides the base for this study. Student performance data in the U.S and in
particularly California is limited primarily to nationally norm and criteria referenced
National Assessment of Educational Progress (NAEP) results in reading and math
and the Third International Mathematics and Science Study (TIMSS) studies in math
and science results. My research of student performance data will focus primarily on
the norm and criteria referenced assessment findings and the long-term effects on
low student performance upon society.
NAEP and TIMSS student performance data suggests that student
achievement is not improving fast enough. Across our nation, far too many students
are still not meeting the standards that will prepare them for the challenges of today
and tomorrow (U.S. Department of Education, 1997). While America’s fourth
graders read as well as ever on average, today the data indicate that over 40% cannot
read as well as they should in order to hold a solid job in tomorrow’s economy. A
1998 study conducted by the Bureau of Labor Statistics show that the average 28-
year-old who tested in the top quartile of math skills on the National Assessment of
Educational Progress earns 37% more than those in lower quartiles (Larson, Guidera,
Rogstad, Nelson, & Smith 1998). Additionally, the same study showed that 40% of
23
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
all 17 year-olds do not have the necessary math skills to hold down a production job
in manufacturing.
The view of student achievement was set by President Bush and the
nations’governors for National Goals in the Education Summit of 1989. The
objective for student achievement for the nation according to that document is that
the academic performance of all students at the elementary and secondary will
increase significantly in each quartile, and the distribution of minority students in
each quartile will more closely reflect the student population as a whole (Barton,
2001). A closer examination of student performance data indicates that the nation is
not improving fast enough (Barton, 2001).
NAEP
In 1988, the National Assessment of Educational Progress, a tool that could
monitor continuously the knowledge, skills, and performance of the nation’s
children, was mandated by Congress (National Center for Educational Statistics,
2000). The National Assessment of Educational Progress provides regular, periodic
reports on the knowledge and skills of students in America’s schools and tracks the
progress of learning. NAEP provides the Nation with an ongoing survey of student
performance indicating what students know and can do. Prior to 1990, NAEP was
required to assess reading, mathematics, and writing at least once every five years.
Currently, NAEP provides assessment in reading and mathematics every two years,
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
in science and writing every four years, and history and geography, and other
subjects every six years (Johnson & Siegendorf, 1998).
Since 1969, NAEP assessments have been conducted with a representative
sample of the nation’s students at grades four, eight, and 12. The NAEP state-level
assessments began in 1990, using a representative sample of 2,000 students in grades
four and eight. Administered by the U.S. Department of Education, NAEP regularly
reports on student achievement in core academic subjects (Blank & Wilson, 2001).
NAEP test items are written by subject specialists and teachers and are reviewed by
policymakers, parents, assessment specialists, and others.
The 1994 NAEP reading assessment provides data on student reading at
grades four, eight, and 12. Students’ reading performance is summarized on the
NEAP reading proficiency scale that ranges from 0 to 500. In addition, results from
each grade are reported according to three achievement levels: Basic, Proficient, and
Advanced. The findings indicated that although student achievement in reading and
writing improved dramatically during the 1980’s and early 1990’s, progress has
slowed (National Center for Educational Statistics, 2000).
The 1994 NAEP reading assessment found that the average reading
proficiency of 12th grade students had declined significantly from 1992 to 1994,
especially among lower performing students, those scoring at the 10th, 25th, and 50th
percentiles. In 1994, 30% of the 4th graders, 30% of 8th graders and 36% of 12th
graders attained a “proficient” level in reading. Across the three grades, only 3-7%
reached the “advanced” level. Furthermore, between 1992 and 1994, the average
25
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
reading proficiency of fourth graders declined in California, Delaware, New
Hampshire, New Mexico, Pennsylvania, South Carolina, and Virginia (National
Center for Educational Statistics, 2000).
In California between 1992 and 1998, 4th grade NAEP average reading scores
were unchanged and, in fact, actually declined in some areas. The bottom quartile
(Q l) and top quartile scores (Q4) of California grade four students in reading showed
no improvement, and the White / Minority Gap Closing also showed a lack of
change in reading. In 4th grade mathematics, California students in the bottom
quartile (Q l) did improve while closing the quartile gap between Ql and Q4.
However, the White/Minority Gap Closing indicated a lack o f change in
mathematics (Barton, 2001). In 1996, out of 38 states California fourth grade
students in reading were dead last (Herman, Brown & Baker, 2000).
The declines in California’s NAEP reading scores can be attributed to student
demographic changes in the 1990s. Since the population of second-language students
increased throughout the state (Maxon & Schwartz, 2001). During this time, student
scores on standardized tests, especially in reading, declined. In 1992 and again in
1994, California fourth graders ranked next to last among students from the 34 states
participating in the NAEP reading assessments. California students also performed
poorly or below proficient levels in math.
Between 1990 and 1996, the average scores of California students on fourth
grade NAEP math assessment showed a large percentage of students performing
below the proficient level (Barton, 2001). The gap between whites and minority
26
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(Black and Hispanic) student achievement levels also showed no improvement in
closing the distance between Ql and Q4 student performance levels in math.
Although, eighth grade scores in math increased between 1990 and 1996 across all
quartiles, in California the gap between white and minority students’ achievement
levels remained unchanged. In the 1996 assessment of eighth grade mathematics,
California ranked 31st out of 41 states (Herman, Brown & Baker, 2000).
Traditionally, California students have performed poorly on NAEP state-by
state comparisons. Theoretically, national norms on standardized tests are
representative of the student population taking the test. In California, the
characteristics of students do not match the national group of test takers. California
has a higher percentage of test takers with limited proficiency in English (LEP), over
21% in 1999 versus less than 2% nationwide (Frey, 2000). Furthermore, over 35% of
California students are considered to be low-income, or economically disadvantaged,
compared to 29% in national samples. Testing experts note that economic
background is an important predictor of student performance on standardized tests
(Frey, 2000).
A further analysis of NAEP state-level student achievement trends in
mathematics shows significant math learning gains in some states in the 1990’s, and
almost no change in other states. The National Assessment for Educational Progress
assessments were conducted in 1996. In 1997, the U.S. Department of Education
released national and state-level results from NAEP Mathematics and Science
Assessments in mathematics and science across 41 countries (Johnson & Siegendorf,
27
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1998). The data indicated that over two-thirds of the U.S. students tested in grades
four, eight, and 12 in 1996 were performing below the “proficient” level, and long
term trend data showed only small improvement in student math performance over
the past quarter century (Johnson & Siegendorf, 1998).
Despite the negative reports, between 1990 to 1996 positive gains were made
by some states in mathematics, according to NAEP state-level findings. In grade four
improvement, seven states made substantial gains in the percentage of students
achieving at or above the Proficient level. Gains were the largest in fourth grade
mathematics in Texas, Indiana, and North Carolina, each improving from 8 to 10
percentage points. In grade eight improvements, eight states made the highest gains
in mathematics, with Michigan, Minnesota, and North Carolina student scores
improving 11 to 12 percentage points. In total, 27 states made significant
improvement in NAEP mathematics scores at grade eight over six years (National
Center for Educational Statistics, 2000).
TIMSS
In 1995, the Education Commission of the States released its annual report
examining progress towards educational reform, which confirmed that American
schools and students were losing ground (Education Commission of the States,
1996). The report confirmed that American students stack up poorly against students
from other industrialized nations in mathematics and reading achievement. The
report further examines the release of the Third International Mathematics and
28
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Science Study (TIMSS). The report describes the largest and most comprehensive
comparative international study of education ever undertaken by the US Department
of Education, National Center for Educational Statistics in 1996.
The purpose of the Third International Mathematics and Science Study was to
measure student learning on a specific set o f objectives in math and science in the 41
participating counties, and to determine the key factors in explaining differences in
student performance (Blank & Wilson, 2001). Student testing and data collection in
schools were completed during the 1994-95 school year. A representative national
sample of schools and classrooms was selected at each of the grade levels selected.
The reports of TIMSS provided comparisons of student scores by countries showed
mixed results for the U.S. in mathematics. U.S. students in mathematics ranked
high among TIMSS countries in the fourth grade, but were below average in the
eighth grade, and almost last in grade 12. U.S. 12th graders scored below the
international level average and among the lowest of the 21 participating nations in
both mathematics and science general knowledge (Larson, Guidera, Nelson, Rogstad
& Smith 1998).
The international comparative results offered by TIMSS results show fourth
graders’ scores to be well above the international average on TIMSS, and only seven
countries outperformed U.S. students. But the problem is that U.S. eighth graders
scored below the international average of the 41 TIMSS countries (National Center
for Educational Statistics, 2000). In mathematics, our eighth grade students’
international standing is stronger in algebra and fractions than geometry and
29
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
measurement. One explanation of the math gap between fourth and eighth grades is
the opportunity to learn that is provided for students, in terms of topics covered. A
curriculum survey of math teachers in the U.S. and Japan conclude that US teachers
spread time very thinly among a wide variety of topics, in comparison to Japanese
teachers who focus on covering greater depth and less breadth in their curriculum
(Blank & Wilson, 2001).
In 2000 a repeat study of TIMSS-R for eighth grade students in math and
science revealed that U.S. students scored at the international average with 14
countries scoring significantly higher and 14 countries scoring below U.S. students
(National Center for Educational Statistics, 2000).
Consequences of Low Student Performance
The long-term consequences of low student performance in America’s
schools can have a devastating impact on society. Today, schools are educating more
children who face significant barriers to academic success. As the demographic
profile of the student population becomes more diverse, more students have needs
that require special programs and services. The most telling indicator of these
changes is the child poverty rate. Today, one-quarter of American children live in
poverty (National Center of Educational Statistics, 2000). Since 1989, a period of
overall strength in the national economy, the number of children in working-poor
families jumped 30%, and 40% of parents in working-poor families are high school
drop-outs. Another 35% have no education or specialized training beyond high
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
school (National Center of Educational Statistics, 2000). The Urban Institute
estimates that federal welfare reform legislation will throw an additional 1.1 million
children into poverty (Education Commission of the States, 1996).
The NAEP and TIMMS student performance data mirror our current societal
trend. Too many students are leaving school underachieving intellectually, are
deficient in basic academic and critical reasoning skills, and are lacking the strong
sense o f social and ethical obligation essential to their own growth and growth of our
society overall (Charles A. Dana Center, 2000). Schooling must break the powerful
grip that poverty and race exerts on academic performance. Schools must work
intensively to educate all students. They must work to overcome systematic variation
in outcomes due to race and class (Charles A. Dana Center, 2000).
California’s low student performance in public schools will have a long-term
negative economic impact on the state if current reform efforts are not successful. In
this high-tech age, high information century, educators face the daunting challenge
of ensuring that all students demonstrate higher levels of learning than ever before
(Hart & Brownell, 2001). Given the size and scope of education in California, with
8,000 public schools of vast geographic and demographic differences, creating an
accountability system that positively affects the quality of education is a formidable
(Hart & Brownell, 2001).
Schools are struggling to respond to the social and demographic contexts in
which schools must operate. Schools are struggling to respond to these changes, even
as their mandate to educate all children to high levels of achievement is expanding
31
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(Levine, Rathburn, Seldon & Davis, 1998). The Improving America’s Schools Act of
1994 fundamentally reformed the federal Title I program. This was a seven billion-
dollar funding program designed to teach basic and advanced skills to students in
high poverty schools. The intent of this legislation is to get rid of lower educational
expectations for poor children and ensure that disadvantaged students are held to the
same standards as other children (U.S. Department of Education, 1997). Every state
has been given a new priority to improve schools that serve low-income and at-risk
students to achieve high academic standards. In order to meet the diverse needs of
today’s students, educators and state policymakers are now examining alternative
methodologies in which to measure and assess student performance data and are
aimed at achieving high standards.
In summary, NAEP and TIMSS student performance data suggests that
student achievement is not improving fast enough. Across our nation, far too many
students are still not meeting the standards that will prepare them for the challenges
of today and tomorrow (U.S. Department of Education, 1997).
Standards-Based Reform
The question of what kind of comprehensive school reform is to be
undertaken ultimately has received much attention and varied response. A great deal
of opinion and other input has been given, and only now are final decisions being
formulated. Such has been a long tunnel, and are just beginning to see the light.
Comprehensive educational reform efforts in the U.S. have focused on a standards-
32
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
based reform approach to improve student performance and reshape the key elements
of school change and accountability (Odden, 1999). Most experts agree that the
standards-based reform efforts in the U.S. may be our best alternative yet in
reforming schools, once the key elements of national standards agreement,
standardized assessment, and school finance are decided.
Standards-based performance data over time will assist policymakers in the
analysis and decision-making process of school reform. Standards-based reform
provides a common language and conceptual framework for educators and
researchers to examine student performance data. A closer examination of each of
the key elements of standards-based reform is important for the purposes of this
study if we are going to investigate new ways to examine student performance data
in schools.
Historical Context
Standards-based reform began in 1983 as a response to the national concerns
that students in America were not learning enough to compete in a global economy.
The changes in the knowledge and skill requirements of work and citizenship in 21st
century America helped fuel the standards-based movement (Odden, 2000). In 1983,
the United States government published A Nation At Risk, a study asserting that
American industry and our leadership in the world were endangered because of our
poor public school system (National Commission on Excellence in Education, 1983).
33
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
In the 1980’s and early 90’s, a review of a broad range of performance data
during this time led to vigorous debates and policy implications. Worrisome
performance data about NAEP student achievement results and changing social and
demographic issues facing schools divided the nation. One view declared failure of
U.S. schools, while the other view is satisfied with progress in the face of social and
demographic issues facing schools (McNeil, 1996). The U.S. Department of
Education released student performance data showing poor performance of U.S.
students in comparison with other countries (Sirtnik & Kimball, 1999). This finding
set off a stream of negative public perceptions of schools, and efforts to restructure
our nation’s schools produced experimental reform models.
During the late 1980s and early 90s, an agenda for restructuring schools
began to move ahead throughout the nation influencing national policy and school
organization and governance (McNeil, 1996). One example of this was the New
American Schools Development Corporation experimental project. This corporation
invited nationwide teams to foster a new generation of “break the mold schools.”
Teams included school districts, business associates, colleges and think tanks to
come up with a design to reform schools. Nearly 700 designs were reviewed.
Researches concluded that the designs were laudable but viewed them largely devoid
of new ideas (McNeil, 1996). While the nation begin to examine the experimental
restructuring efforts of the Collation of Essential Schools, Accelerated Schools, and
Alternative Schools, the standards-based reform movement began to gather steam.
34
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The push for national standards went into force back in 1989 when President
George Bush called the nation’s governors together for the first ever National
Education Summit as part of the Goals 2000 efforts (Merrow, 2001). The 1989
education summit produced vague goals for education; however, the follow-up 1996
summit reached agreement on the following: states should develop their own
standards, states should develop tests that can hold students accountable, and states
should establish independent entities to assist in the implementation of standards and
accountability (Education Commission of the States, 1996).
In 1997, President Clinton unveiled his education plan “National Standards of
Academic Excellence: A Call to Action for American Education in the 21st Century”
(U.S. Department of Education, 1997). That every child can learn and meet
challenging standards are the cornerstone themes of the plan. The President’s
Education Plan advocates that every state and every school must establish
meaningful standards for what students should master in the core subjects. The plan
puts the responsibility on states to ensure that students master the basics and test
every student to make sure the standards are met. Further, the U.S. Department of
Education will develop a new national test to replace NAEP and TIMSS tests. The
new national test will eventually be administered to all students in every state to
ensure world-class standards of excellence are met in America’s schools. Standards-
based reform efforts in the U.S. has made considerable progress (The National
Education Goals Panel 2001); however, the emerging shift to world-class standards
35
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
may not be possible until all the key elements in which to measure progress of the
current standards-based reform movement are in place.
Standards
Standards are doctrines that call for a more intellectually demanding content
for students. Standards specify what students should know and leam at each grade
level, and by extension, what should be taught (Tucker & Codding, 1998). In theory,
students balance educators’ and scholars’ professional judgement about what
constitutes challenging and important academic content with the views of parents,
the business community, and the public concerning what students need to know.
Standards create a common language for educators to measure and examine, and to
chart data on student progress towards meeting the standards (The National
Education Goals Panel, 2001).
In 1998, according to the National Association State Board of Education
(NASB), states were granting school districts increased autonomy to make critical
decisions about the appropriate techniques to be used to implement standards. High
performing schools benefit from increased autonomy that allows them to make more
decisions on how best to organize their resources (National Association State Board
of Education, 1998).
Currently, 49 states (excluding Iowa) have adopted state standards and most
school districts are providing assistance to schools in bringing standards-based
reform efforts to the classroom level. State standards are statements of skills that
36
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
students need to meet or exceed at grade level. The majority of states are committed
to standards-based reform efforts. In 1999, 47 states of the U.S. had committed to
measuring student achievement towards standards (Yinger & Hendricks, 2000). As a
result, practices are changing, and student achievement is increasing (Levine,
Rathburn, Seldon & Davis, 1998).
A recent study on the progress of standards in the U.S. titled “Making
Standards Matter” was conducted by American Federation of Teachers in 1999. It
examined the progress of standards- based reform nationwide (Achieving Goals
of Standards Based Reform, 2001). The findings of the study conclude that every
state except Iowa has a common set of academic standards for students. In addition,
the quality of standards continues to improve; every state is committed to measuring
student achievement towards the standards, and 29 states, as of 1998, have academic
Intervention Programs to assist students in achieving the standards. A recent second
multi-year study involving 16 school districts in 13 states, conducted by Schools
Engaged in Educational Reform Program from 1998 to 2000, provided evidence that
standards-based reform efforts are improving student achievement (Making
Standards Matter, 2000). This study investigated educational reform at three levels,
the state, district, and school level. The study cited each level as having a distinct
role in supporting the implementation and communication of standards that lead to
increased student achievement.
Critics of standards-based reform argue that disadvantaged students from
low-income homes and children of color will be left behind in the nation’s standards
37
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
movement. Recently, a few examples of sustained, districtwide academic success for
children of color and children from low-income homes have appeared in states such
as New York, North Carolina, and Texas (Elmore, 2000; Ragland, Asera & Johnson,
1999). These states have highly developed and stable accountability systems.
Recently, the Charles A. Dana Center and the Educational Productivity Council
conducted a study on state accountability and the Texas Assessment of Academic
Skills (TASS) state accountability system, concluded that equity-driven student
achievement was achieved in the four school districts studied (Charles A. Dana
Center, 2000). Standards-based reform addresses equity and is attainable for both
poor and wealthy students (AFT Executive Summary, 2001).
Human Resources
Standards-based reform efforts have considerable implications for the
allocation and use of human resources for school systems (Roelike, 1997). Roelike
reported his research findings on a study conducted in New York State involving 10
schools and 4 school districts. The study focused on the curriculum and related
standards-based reform movement undertaken by New York State Board of Regents
and the State Board of Education. Roelike concluded that policy changes regarding
allocation of human resources have occurred at the local level in response to state-
initiated reforms. These policy changes have positively impacted managerial and
classroom practices. The implications of these findings need future research due to
their ongoing nature (Roelike, 1997).
38
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Schools need to make academic standards everybody’s business; everybody
needs to know what the standards are and why they are important (Reeves, 2000).
School boards, superintendents, and principals are the key figures in forging the
standards-based reform movement, but ultimately teachers have the responsibility for
successful delivery of the standards. The National Association School Board in
Education cite the following factors in building a standards-based curriculum:
number and type of students involved; teacher knowledge and skills; student
motivation; time; improved curricula and materials; and necessary technology and
supplies (NASB, 1999). Additionally, all stakeholders must have easy access to the
standards and the full compliment of clarifying documents and supplemental
materials that states develop to illustrate the standards (AFT Executive Summary,
1999). Human resources development in the standards-based reform movement is
vital, and support must be given to all stakeholder involved in the process.
Time for Professional Development
Time for professional development is a critical factor in the implementation
of standards-based reform. Each state’s standards are different; none is perfect. Some
state that standards are vague and teachers are not sure what they mean, others state
that some standards are so specific, and numerous, that it is impossible to cover
everything in a K-12 curriculum (Jones, 2000). Teachers, administrators, and school
leaders need time to increase their knowledge about the standards. Teachers need
time to learn how to best deliver the standards through focused staff development
39
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and time to create lessons to attain the standards (Scherer, 2000). School districts
must narrow the state standards down to “power standards” at each grade level, ones
that have stood the test of time that apply in other disciplines, and prepare students
for the next level of instruction (Reeves, 2000). Teachers and administrators only
rarely experienced the practice in their own education and staff development. A
sustained accountability is needed to achieve the standards. Time is needed in
standards-based reform to chart curriculum, assessment of standards, and delivery of
instruction, in order to map a clear course and how we want to get there (Noyce,
Perda & Traver, 2000).
More work needs to be done in aligning assessments to the standards in terms
of what is adequate performance to ensure students participate in a rigorous
curriculum (Laboratory Network Program, 1998). Additionally, more time is needed
to provide objective measures of how well students are doing to determine if all
students have the knowledge and skills to achieve at the next level. Investment in
time for adequate professional development is necessary and will require states to
invest substantial resources as part of the standards-based movement.
School Finance
As the standards-based reform movement develops, finance structures need
to be viewed differently. Researchers say school districts should put their money
where their standards are when making standards-based decisions (Jones, 2000).
One of the most problematic issues raised in comprehensive school reform has to do
40
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
with cost. The attempt to standardize school costs by identifying the necessary
elements of a successful reform model is necessary (Odden, 2000). Odden suggests
that it may be time to redesign state and district school finance systems to align them
more closely with the standards-based reform movement (Odden, 1997).
Those who study education policy have struggled to understand how money
matters in education and why educational research finds no strong or consistent
relationship between expenditures and student performance (Fowler, 1997). Since
educational policy is designed at the state level, the state should establish new school
finance structures (Odden, 1997). Others argue that districts play crucial roles in
redesigning school finance policy, since they will help guide implementation of state
policies at the school level. Problematic with this approach is that district resource
allocation is inappropriately aligned with areas in which decisions should be made to
improve student perfomiance (Guthrie, 1988). To increase productivity in schools,
schools-sites must have the flexibility to use school-site resources differently to
improve results. One way schools can use resources differently is to invest in teacher
quality. Numerous studies have identified the importance of teacher quality as
measured by teacher’s education, experience, and test scores on teacher exams have
shown that theses factors have more impact on student achievement than any other
single factor (Jones, 2000).
A school-site-based finance system is a crucial element of standards-based
reform (Odden & Picus, 2000). This strategy allows schools to use education dollars
better. When absent, it constitutes a major obstacle to any meaningful form of school
41
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
restructuring (Bodily, 1998). Additionally, school districts must provide
organizational conditions at the school level including control over the allocation and
use of its resources to increase student achievement (Odden & Picus, 2000).
States are beginning to explore alternatives to school-based funding (National
Center for Educational Statistics, 1996). For example, Texas used two different
approaches to restructure Texas school finance in order to implement school-based
funding. The first approach is a campus foundation program allotment that allows the
flow of revenue from the district to each school based on calculations of campus
allotments. The second approach is to implement a block grant system that depends
on major tax restructuring to generate additional state revenues for education
(National Center for Educational Statistics, 1996).
Some states are beginning to make the shift to a more performance-oriented
education system for school funding. Currently, a few states are providing
performance enhancement elements to support student achievement. Performance
enhancement elements can provide schools with a greater control over school
resources (Odden, 1997). In Texas, Kentucky, and California, performance
enhancements are linked to accountability systems in order to implement standards-
based reform efforts. Performance enhancements in these states are used as rewards
and sanctions, and then offer potential for motivating students, improving teaching,
learning, and student performance as the common sustaining goal for the entire
school and community. In California, the Governor’s Performance Awards (GPA)
program uses award money to increase student achievement, provide regulatory
42
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
relief, and /or other incentives to schools, teachers, students, and all public schools in
the state eligible to receive GPA recognition (Hart & Brownell, 2001).
Assessment and Accountability
As part of a national accountability movement for schools, output systems are
being developed in some states. Some states are shifting the emphasis from inputs of
credentialed teachers, books in the library, and providing special needs for pupil
services to a more “outputs” system of accountability. These outputs are based upon
standards that set expectations for what students should know and be able to do
(Goertz, 2000). The new output system relies on measures of student performance,
combined with incentives such as rewards or sanctions. Furthermore, the new
accountability system includes sanctions applied to low performing schools and
school districts.
Traditional accountability systems have focused primarily on school districts.
States relied on the accreditation processes and program monitoring to ensure
compliance with the rules. But with standards-based reform, the focus is on
assessments of individual students and their schools (Goertz, 2000). The standards
based reform system holds states responsible for assessing student progress through
rewards and sanctions. Sanctions and rewards are aimed at schools and students,
often with high-stakes penalties such as state takeover of low performing schools or
no diploma for a high school student who fails an exit exam.
43
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Assessments are an integral part of standards-based reform. Well-designed
tests can provide objective measures of how well students are progressing towards
meeting the standards (Achieving the Goals of Standards-based Reform, 2001).
Some states hold schools accountable by using both tests and knowledge (cognitive)
and non-cognitive measures of performance (Goertz, 1997). Cognitive measures of
student learning such as assessments systems, are typically based on state developed
standards. States like Kentucky, Maryland and Vermont rely on writing samples and
tests of critical thinking as well as multiple-choice tests. California has developed its
assessment system primarily based upon multiple-choice questions. However, the
state is moving towards broader assessments as evidenced by written responses on
the (CAHSEE) California High School Exit Examination (Goertz, 2000).
In summary, the key elements of standards-based reform equity, human
resources, time for professional development, school finance, assessment and
accountability provide a conceptual framework and a common language for
researchers and educators to assess student performance and chart school progress
towards attainment of state standards. The information derived from standards-based
assessments will allow this researcher to measure, use, and make judgements about
these types o f student performance data for the purposes of this study.
The Role of Data Use in the U.S.
The role of data use as it relates to student performance in U.S. education,
currently is changing and influencing policy along the way. Educators no longer
44
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
view data as information needed only to justify program expenditures. Data as it is
used today, can tell us a great deal about student performance. Student performance
data, can assist educators and policymakers in decision-making about schooling, and
provide valuable information on how well students are actually performing in
schools (Goertz, 1997).
This study intends to show student performance data as it is currently used -
to shape educational reform, requires further investigation and clarification. Further
analysis of the different kinds of student performance data used to hold schools
accountable for comprehensive school reform is needed. This analysis will provide
insight into how data has been used and currently is used to influence comprehensive
school reform.
The debate over norm-referenced or criterion-standardized assessments versus
classroom performance assessments continues to be controversial, particularly
regarding the question of which assessment more accurately measures student
achievement, however, both types of data give us information on how well students
are doing in U.S. schools. However, the amount and kind of influence they have on
comprehensive school reform policies requires further investigation. Today, the role
of data use in the U.S., provides educators with a comprehensive structure by which
they can assess student performance in schools. The intended use of data is linked to
national accountability movement goals established for educational reform (Goertz,
2000). A closer examination of how the intended use of data and its influence on
45
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
education came about is critical for the full understanding, as well as clarification of
the intended use of data that this study
addresses.
Student Outcome Data Movement
Today, comprehensive school reform currently is going through a paradigm
shift. States have spent years focusing on internal methods for comprehensive school
reform. States have long tinkered with school reform through re-organization of
schools, curricula changes, and school choice. These internal changes offered little in
the way of reforming schools on the national level (McNeil, 1996). Comprehensive
school reform today, is focused on student outcomes. The widespread demand to
improve student performance is at a fever pitch. Forty-nine states now have
mandated curriculum standards (Pierce, 2000). In this era of accountability and high-
stakes testing, raising student achievement scores has changed our public schools
dramatically.
The role of student outcome data is the centerpiece of the national standards-
based accountability movement in schools. States like Kentucky and Maryland have
been using standards for over 10 years. Theses states have set high standards, but the
stakes were at the school level, not the student level (Frey, 2000). Now that states are
more strongly compelled to take responsibility for what and how well students learn,
policymakers are struggling with ways to develop appropriate accountability systems
to measure student outcomes.
46
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Some states hold schools accountable by using both measures of state
developed standards knowledge (cognitive) and noncognitive measures such as
attendance rates and dropouts or high school completion rates. States vary, in how
they weigh these different measures (Goertz, 2000). Kentucky uses noncogitive
measures that comprise 10% of its accountability index. On the other hand, Texas
gives equal weight to student performance, attendance, and dropout rates. All
accountability systems use student data, however, successful accountability systems
use outcome data as the cornerstone for improvement.
State policymakers are struggling with ways to develop accountability
systems that measure absolute progress (the score on a specific test) or relative
progress (improvement over time). Florida requires students to meet an absolute
standard of performance. In Florida, at least 60% of students must score at a level 2
in the state’s reading and math tests. Texas has an absolute system, it also measures
and rewards growth over time and annually increases its requirements (Reeves,
2000). California has set 800 as an ultimate target API score on a scale of 200 to
1,000. Each year schools with API scores below 800 must close the gap between
their scores and the target 800 by 5%.
As states develop accountability systems based upon student outcomes, states
must decide which students will be assessed. Some states test certain grades, while
others like California test all grades each year. Several states include all students,
such as Special Education students and English Learners. Some states, such as
Florida, exempt some students based upon their length of time in the state as an
47
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
English as-a-Second Language Learner. Issues of race and equity on standardized
measures for school accountability remains controversial. Texas addresses the issue
of equity by giving its reading test and math assessment in Spanish in grades third
through sixth and its writing assessment in Spanish in grade four. But all tenth
graders must take the high school graduation test in English.
Standardized Assessments
There is no argument that norm-referenced standardized tests can affect the
curriculum and provide critical student performance data on a state’s progress
towards achieving educational outcomes. Currently, 28 states have or will have a
high school exit exam for students to pass in order to graduate from high school
compared to 24 in 1998 (AFT Executive Summary, 1999). Linking standardized
testing to state accountability systems has raised the issue of equity, notably in what
ways do these tests affect the education of low income children of all races, but
especially the children of color who typically perform poorly on standardized tests.
Recently, there is a range of data that suggests that low-income children and
children of color have successfully improved student achievement scores in Texas
and North Carolina (Scheurich, Skrla & Johnson, 2000). States should stay the
course in the standards and accountability movement asserts the Committee on
Economic Development (Hoff, 2000). While all the pieces of the puzzle are not in
place, early returns from new testing systems are spurring changes that lead to
improved student learning says the committee. Test scores in many of America's
48
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
urban school districts are up due to a common set of traits, such as setting fewer and
clearer goals, improving teacher training, establishing high academic standards, and
using data to set incentives and consequences for performance (Johnston, 2001).
Social policy has continued to dominate the educational specter, advocates of
rigorous academic standards and high stakes testing argue that the use of
standardized test scores to index educational success or failure is almost never
questioned (Haertel, 1999). In the rhetoric of educational reform, it often sounds as
if improving the educational system is synonymous with improving test scores. In
such a climate, the logic of high stakes testing seems compelling. It is imperative that
schools are held accountable if they fail to make the grade (Robinson & Timperley,
2000).
In the U.S. in recent years, there has been an increase in policymaker and
educators’ reliance on high stakes standardized testing in which single scores are
used to make important educational decisions (Ransom, Santa, Williams & Farstrup,
1999). High-stakes standardized testing means that one test is used to make
important decisions about students, teachers, and schools. The student performance
data generated from high-stakes standardized tests remains controversial. Educators
argue that high-stakes standardized tests do not accurately tell us how well students
have been taught. First, because the makers of standardized tests must design tests to
one-size-fits-all curriculum. What is taught in specific localities often does not match
what is tested. Second, according to some studies, 50% of the content on
standardized achievement tests is not taught in the classroom (Popham, 2000).
49
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
States are beginning to use alternative types of assessments to assess student
outcomes. Criterion-referenced standardized assessments are tests that are
comparable across schools, over time, and can be linked to state accountability
systems. Criterion standards-based testing can be linked to state tests on writing,
literacy, and math. They act as a dipstick that pierces several important layers to
check on the level of student achievement at a particular point in time (Sirotnik &
Kimball, 1999). Criterion standards-based tests can be linked to state performance
standards testing as part of state accountability systems.
The data derived from criterion-referenced standards tests can be used to chart
student progress and achievement of standards over time. In California, the
emerging California Standards Tests will drive assessment over time. California
Standards Tests are based on state content standards, and over the next three years
these will be infused into the state’s Academic Performance Index (Duffy, Goertz &
Le Floch, 2001). Criterion- referenced tests serve to gauge state standards
benchmarks, assess student achievement over time, drive the curriculum, and can be
linked to state accountability systems to assess standards-based student achievement.
Classroom Assessments
Assessment of student performance data should consider the use of multiple
measures for evaluating student achievement, especially classroom-based
performance assessments (Fierros et al., 1997). Classroom performance assessments
or authentic assessments are portfolios, student projects, and hands-on tasks that
50
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
require higher-order thinking skills that engage students in their own learning
(Haertel, 1999). The advantages of classroom performance assessments are that they
can build on proceeding instruction, draw on component skills, develop special
vocabulary, scoring rubrics, and content knowledge that teachers and students build
up over time in order to assess higher order thinking skills. In completing a
classroom performance assessment, students often present their work to an audience
of their peers, their teachers, and possibly their parents. Students are motivated to
show what they can do, and they can expect to receive individual scores and
immediate feedback.
Classroom performance assessments do have limitations in data use. The
major disadvantage of classroom performance assessments is their lack of
generalizabilty. The selection of assessment tasks on the basis of their inherent value
complicates inferences of scoring rubrics, especially if teachers and students
participate in that selection, as with the use of portfolios (Haertel, 1999). Scoring
rubrics are dependent upon teacher experience and knowledge and are not
generalizeable. However, what they lack in generalizability, they gain in support of
high quality teaching. Authentic assessments are more closely aligned to
instructional outcomes. Furthermore, authentic assessments can involve teachers,
students, parents, and other active participants in developing and refining common
learning standards (Thompson, 2001). Additionally, authentic assessments can
provide equity for all students and provide a high level of support for all students,
teachers, and educational leaders.
51
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The collection of classroom-based assessment data can prove to be a difficult
task to accomplish, especially if standards-based rubrics to assess student progress
towards a given content standard have not been developed beforehand. For the
purposes of this study, all classroom-based assessments will require standards-based
rubrics the development of which are based upon achievement of a particular content
standard.
To summarize, the intended use of student outcome data is to provide schools
with useful information on student performance. Student performance data can
provide indicators of relative performance of a school in much the same way that
national economic indicators track the performance of the U.S. economy (Goertz,
2000). Student outcome data are derived from various types of assessments that
provide educators and policymakers with information on how students are actually
performing in schools. Comprehensive school reform currently is being driven by
student outcome data. The success o f this reform will derive from the way the data is
used to drive future decision-making.
Expectations of Student Performance Data in California
The California school accountability system requires school improvement and
high student expectations for all schools, teachers, and students. Schools are
expected to use student performance data derived from the state Academic
Performance Index (API), Stanford 9 (SAT 9) test results, and the California
Standards Test results to make decisions regarding student achievement of API
52
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
growth targets, and the teaching and learning of the California content standards. All
schools are given API growth targets, SAT 9 test results, and California Standards
Test results, and each is expected to achieve growth targets for all subgroups in their
schools (Guth et ah, 1999).
For the purposes of this study, our focus will be on California schools that
utilize this information effectively to improve student achievement and achieve API
growth targets for all subgroups. A review of California’s school accountability
system is needed for increased understanding of the state’s expectations for the use
of data in schools.
Public Schools Accountability Act
In 1991, California began to develop an accountability model based upon the
report card model. The California law established an advisory committee to
recommend performance indicators to the legislature. The committee recommended,
and the legislature approved annual indicators in five areas: the population context,
fiscal context, student preparation, student access, and student outcomes. No
measures of student learning achievement or even goals for learning were mentioned
in 1991 (Wellman, 2001). Since that time, the state’s role in the accountability
system has seen a dramatic shift towards increased accountability.
Policymakers in California, like those in other states, were determined to shift
the focus of educational reform from rules and processes to student achievement and
accountability (Hart & Brownell, 2001). The Public Schools Accountability Act
53
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(PSAA) of 1999 defined the laws of school accountability in California. With a
single act of legislation, the most comprehensive new law launched an era of higher
expectations for California schools. The PSAA legislation outlined educational
reform and accountability for improved student performance for students, teachers,
schools, and school districts (Alpert, 1999). Students are accountable for
demonstrating achievement on standardized tests each year in grades 2-11, and
beginning with the high school freshman class of 2000, every student must pass the
California High School Exit Exam as a new graduation requirement. Accountability
for teachers comes through the Peer Assistance and Review Program, which seeks to
enhance the daily practice of teaching by using expert teachers. Accountability for
schools and districts is defined in the Public Schools Accountability Act of 1999.
The PSAA is comprised of three sections that include the Academic
Performance Index (API), the Immediate Intervention/Underperforming Schools
Program (II/USP), and the Governor’s Awards Program (GPA). The goal of the
PSAA is to provide an accountability system that will become standards-based
educational reform effort (Calvo et al., 1999). The PSAA led the way for the
development of the Academic Performance Index (API) and growth targets for
accountability. It is a statewide measure designed to effect student achievement
through the use of sanctions and incentives (Duffy, Goertz & Le Floch, 2001).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Academic Performance Index
The purpose of the Academic Performance Index (API) is to measure
academic performance and progress in schools. It is a numeric scale that ranges from
a low of 200 to a high of 1000, and a school’s placement on the API is an indicator
of its performance level. A school is ranked by type (elementary, middle, or high
schools) then placed according to its API scores (Hart & Brownell, 2001).
Additionally, each school’s API scores are compared to those of similar
characteristics, including student mobility rate, SES status, percentage of English
Language Learner students, percentage of fully credentialed teachers, and other
factors. Satisfactory growth is considered relative to an interim statewide API
performance target of 800. The standard growth target for a school is 5% of the
distance between a school’s API and the interim target of 800 (Duffy, Goertz & La
Floch, 2001). Additionally, the state system compares similar schools across the
state. Within a district, it is equally important to analyze data comparing schools
across all schools and neighborhoods in the district (Olsen, 2001).
The API ranks the academic performance of all schools, and identifies
disaggregated growth targets for schools. The state accountability system measures
not only improvement in achievement overall, but also the improvement of
numerically significant subgroups of students (Olsen, 2001). An analysis of current
test data was considered in the design of the API to ensure each type of school in the
state could be measured fairly against such a standard. Preliminary data indicate that
schools in each decile have an equal chance to improve student performance and to
55
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
work towards incentives (Hart & Brownell, 2001). When fully developed, the API
will use various performance indicators to create a composite score for a school. In
1999, only results of the Stanford 9 (SAT 9) tests were used to calculate the numeric
scale. Over time, criterion referenced California Standards Tests will be added to the
API score in 2001, as well as the California High School Exit Exam. The
assessments should become aligned to the standards with the California Standards
Tests over the next two years (Burr, Hayward, Fuller & Kirst, 2000).
Senate Bill 376 in 1997, authorized the Standardized Testing and Reporting
program (STAR). The SAT 9 Test, currently is the most powerful indicator of
student achievement in California and plays a major role in the STAR program
(Calvo et al., 1999). The STAR Test results are posted on the Internet. It displays the
SAT 9 Test and California Standards Test results by grade level and by content areas
for the state, counties, districts, schools, and for all students. The STAR student
achievement data on the Internet is easily accessible for educators, parents, and the
public.
Immediate Intervention/Under-Performing Schools
In California, the Immediate Intervention/Under-Performing Schools program
(II/USP) is designed to provide resources and assistance to low performing schools
that do not meet their growth targets. The II/USP program grants resources in the
form of external evaluators who work with low performing schools to analyze
student performance data to determine causal effects for low student achievement
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and develops school improvement action plans (Boese, Burkhardt, Carstens, Devine,
Gaffney & Just, 2001). In 1999-2000, over 1,100 low performing schools
volunteered to participate in the II/USP program, and only 430 were funded (Hart &
Brownell, 2001).
Low performing schools that both qualify and participate in the II/USP
program must meet their 2000-01 growth targets, or be subject to local interventions.
These interventions include action plans that require schools to show significant
growth to meet 2001-02 targets, or they may be subject to state reorganization
sanctions, removal of staff, or state takeover of management plans (Boese,
Burkhardt, Carstens, Devine, Gaffney & Just, 2001).
Governor’s Performance Awards
All schools in California that meet and exceed their identified growth targets
from year to year are eligible to receive monetary and nonmonetary awards under the
Governor’s Performance Awards (GPA) program. The GPA is a significant step in
California’s school accountability system and provides rewards for motivating
students, improving teaching and learning, and expanding school improvement
efforts (Wellman, 2001).
The GPA program awards money, regulatory relief, and/or other incentives to
schools, teachers, students, and all public schools in the state eligible to receive GPA
recognition. Eligible schools must meet or exceed growth targets for all populations
of students and all socioeconomic backgrounds (California Department of Education,
57
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1999). School monetary awards are based on per-pupil expenditures of $150 per
student. Nonmonetary awards may include honor role distinction or distinguished
school recognition. Beginning in the fall of 2000, teaching staff in underachieving
schools (defined as the bottom 50% on the API) that significantly improve beyond
their API growth targets will receive performance bonuses ranging from $5,000 to
$25,000.
The GPA provides students merit-based scholarships as a reward for
demonstrating high academic achievement scores on the Stanford 9 and Advanced
Placement Exams. The Governor’s Scholar Awards program offers students in 9th,
10th, and 11th grades who score in the top 10% of their high school class can are
eligible for a $1000 to be used for higher education. Additionally, students who score
in the top 5% statewide on the Stanford 9 test and score a 5 on Advanced Placement
j
Exams in calculus and science are eligible for GPA awards for a $2,500 scholarship
to be placed in a trust for higher education (Frey, 2000). The GPA program will
expand from its current level of funding from 400 million to in excess of 1 billion in
the future years.
The results of the state accountability system are difficult to assess at this time
due to newly established performance outcomes recently established for schools.
However, preliminary data suggests that California students lagged behind national
norms on the Stanford 9 reading and math tests by substantial margins (Betts,
Rueben & Danenberg, 2001). Only 40-45% of the state’s students scored at or above
the national median. The unusually high proportion of Limited English Proficient
58
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(LEP) students in California accounts for at least two-thirds of the gaps in math and
reading performance (Betts, Rueben & Danenberg, 2001). If LEP students are
excluded from the sample results, 44 to 53% of California’s students perform at or
above the national norms in math.
To summarize, California’s role in the school accountability system is to
monitor the progress of schools and districts, hold them accountable for helping
students achieve high standards, allocate resources where they are most needed, and
improve educational equity and access for every student (Hart & Brownell, 2001).
The role expectations of school is to monitor student achievement through the use of
student performance data provided by the state, set clear goals for standards-based
student outcomes, and to provide assistance to school staff in the use of this
information to improve student achievement.
The Current Use of Data in Schools
The current research on the uses of data in schools both at the individual
teacher level and schoolwide level, indicates that there is a lack of alignment
between student data and student achievement (Khanna, Trousdale, Penuel & Kell,
1999). The research on data use will show that currently, educators lack training in
the effective use and interpretation of data to meet student needs and enhance student
achievement in schools. For the purposes of this study, the research on data use will
investigate: data use patterns, the types of data used currently in schools, how data is
59
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
used by teachers and schoolwide levels, and the emerging state context for future
data use in California schools.
Data Use Patterns
State legislatures across the U.S. are struggling how best to "monitor what
matters" (Christie, 2000). Although, federal guidelines to help standardize the
collection of data on school dropouts came out several years ago with the goal of
improving the consistency in how school districts and schools collect data, the
guidelines did nothing to help establish the information systems necessary. To
simplify the process and ensure more accurate numbers, every state requires districts
to produce report cards. The purpose of these report cards is to measure academic
performance and progress of schools (Goertz, 2000).
In California, when fully developed, the API will use various performance
indicators to create a composite score for a school. In Ohio, the state legislature
created the Educational Information Management System, an infrastructure to house
data necessary to determine how well schools are doing. The Texas legislature
established a similar system in 1995 (Keller, 2001). Report cards published in Texas
are disaggregated by student groups: African American, Hispanic, White, and
economically disadvantaged. In 1999, the Massachusetts legislature required each
school district to adopt and maintain a reliable data collection system (Christie,
2000). Part of the system requires each district and charter school to have a unique
60
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and permanent and unduplicated ID number for each student. In Minnesota and
California, each state possesses a website that gives the education system and
common data elements for its public schools.
The policy context in which policymakers in the U.S. make decisions
regarding the use of information in schools has hastened a paradigm shift from
qualitative assessments to quantitative assessments and has changed public schools
assessment systems (Madeaus & O’Dweyer, 1999). Changes in assessment
technology have led to streamlined, machine-scorable, standardized multiple-choice
test as the technology of choice by policymakers. Standardization of assessments
assumes that states provide a substantially equal standards-based high quality
education for all students (Nave, 2000). Many educators believe a link exists
between standards set by reformers and academic achievement. The ability of states
to monitor progress towards national benchmarks or standards will assist states in
making standards work to improve schools (Jones, 2000).
Types of Assessment Data Collected by Schools
The primary aim of assessment is to foster learning of worthwhile academic
content for all students. Assessment is a tool for evaluating and monitoring a
school’s academic progress and for addressing its shortcomings (Herman & Winters,
1992). Many state departments put little emphasis on schools gathering data, and,
thus, provide little incentives for districts and schools to devote time, money, and
staff resources to using data in different ways (Bernhardt, 2000). School leaders must
61
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
advocate for a balanced development and use of assessments for their schools
(Stiggins, 2001). Assessments that enable the gathering of dependable and quality
information about student achievement are the responsibility of school districts.
Providing the resources to do so, is the states responsibility (O’Donovan, 2001).
School districts across the state are now developing more structured and
complicated accountability systems with high-stakes assessment measures leading
the way of each district’s system (Doherty, 2001). The implementation of a complete
program of data collection and use can lead to the improvement of education as no
educational innovation of the last century has (McLean, 1995). Both standardized
tests and classroom performance assessments must be of high quality for teachers to
utilize effectively (Patton, 1990). Both must be used for schools to improve, because
both inform critically important decisions.
Gathering data assessments at the schoolwide level means looking at students,
teachers, and the school community in many different ways. An effective data
analysis of a school or program can include four different types of data: student
learning data, demographic data, perceptions data, and school process data
(Bernhardt, 2000). Student learning data describe an educational system in terms of
standardized test results, grade point averages, standards assessments, and other
formalized assessments. Demographic data provides descriptive information on
items such as enrollment, attendance, grade levels, ethnicity, gender, socioeconomic
status, and language proficiency. Perception data help us understand what students,
parents, and teachers think about the learning environment. School process data
62
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
define programs, instructional strategies, and classroom practices to systematically
examine their practices and student achievement (Bernhardt, 2000, p. 35).
Use of Data
There are many reasons why schools don’t use data well. In many schools and
districts data analysis has never been viewed as a high priority (Bernhardt, 2000).
The empirical research in this area found that administrators and teachers lack the
necessary tools and training in the use of data in schools. Many principals and
teachers have not been given the opportunity to develop the assessment literacy
needed to fulfill their assessment responsibilities in standards-driven schools
(Stiggins, 2001).
Researchers have learned much about how teachers view and use results from
standardized tests of student achievement (Khanna, Trousdale, Penuel, William &
Kell, 1999). For example, we know that teachers use these data very rarely to inform
educational practices (Penuel, Kell, Frost & Khanna, 1998). While teachers feel
pressure to improve test scores, they believe that such scores are not particularly
useful in helping drive instruction in a positive way. In addition, teachers may not
understand that improved test scores may not be attributable to improved classroom
teaching, but to improved test preparation practices (Popham, 2000).
Individual school districts have taken the initiative to involve teachers in
professional development to improve the use of data to improve student achievement
(Levesque, 1996). The Student Achievement Model (SAM) was an action research
63
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
project designed to give teachers a system for collecting data and analyzing data in
order to increase student achievement (Frye, Fugerer, Harvey, McKay & Robinson,
1999). The findings concluded that through the implementation of this model,
teachers used the information in powerful ways to increase student achievement.
Additionally, as teachers became more comfortable with the SAM model, teacher
attitudes and actions about the use of data changed in profound ways to increase
student achievement. The model is currently used in 63 elementary schools and 8
secondary schools in Florida.
School principals also lack the necessary skills in the use of data to make data-
based decisions for their schools (Striefer, 2000). Data-based decision-making is a
process of inquiry, analysis, and decision-making inferences from the interplay of
these input, process and outcome measures. As school districts across the country
develop more structured accountability systems, there is increasing pressure on
principals to understand and use data to improve instruction (Khanna, Trousdale,
Penuel, William & Kell, 1999). In 1999, a study was conducted by San Francisco
Unified School District that focused on professional development for middle school
principals for the implementation of a data training model program to read, interpret,
and use multiple sources of data to improve student achievement. The study findings
concluded o f the 17 middle school principals that participated in the study, all but
one middle school principal benefited from the data training model implementation
at their school sites. The 16 middle school principals planned with data to utilize
resources more effectively, increased their ability to identify school priorities, and
64
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
improved student academic achievement (Khanna, Trousdale, Penuel, William &
Kell, 1999).
School districts and teachers can use student data to help make decisions about
school policy, curriculum, and instruction (Noyce, Perda & Traver, 2000). School
improvement requires that decision-makers make comparisons between and among
data, including school input and process variables (Streifer, 2000). Creating data-
driven schools cultures require school districts to make data use a priority. In schools
where data use was given priority, several themes emerged. In Texas, nine high-
performing, high-poverty urban elementary schools were studied on their exemplary
use of data to improve student achievement (Charles A. Dana Center, 1999). These
highly successful schools not only identified the standards and assessment used to
measure student achievement, but established benchmarks to periodically assess
learning and progress towards established benchmarks. Data was disaggregated and
resources were reallocated, based on student needs. Additional time for instruction
was added both during the school day and after school to meet the greatest areas of
need identified by the analysis of data (Charles A. Dana Center, 1999).
The research done in Texas is supported by the Education Trust that examined
top performing, high poverty schools across the United States. The 1200 schools
were selected because they were identified by states as their top scoring and/or most
improving schools with poverty levels over 50% (Education Trust, 1999). The study
findings concluded that these schools monitored student achievement data, assessed
student work in relation to the standards, analyzed pre- and post-data by core
65
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
curricular areas, and used data to inform instruction. A case study on the 90/90/90
Schools also yielded similar findings. 90/90/90 Schools are identified as having the
following characteristics: 90% of students are eligible for free and reduced lunch,
90% are ethnic minorities, and 90% of the students met or achieved high academic
standards. The findings concluded that 90/90/90 Schools had a laser like focus on
student achievement (Reeves, 2000). These schools posted charts, graphs, and tables
about student achievement information throughout the school, and conducted weekly
assessments of student progress. Common performance assessment practices were
used and students were provided with multiple opportunities to improve (Reeves,
2000).
In summary, the current research on the uses of data in schools both at the
individual teacher level and schoolwide level, indicates that there is a lack of
alignment between student data and student achievement (Khanna, Trousdale, Penuel
& Kell, 1999). Despite the gains made by a few selected schools, the research on
data use shows that currently, educators lack training in the effective use and
interpretation of data to meet student needs and enhance student achievement in
schools.
Emerging State Context
Many principals and teachers have not been given the opportunity to develop
the assessment literacy needed to fulfill their assessment responsibilities in
standards-driven school (Stiggins, 2001). School organizations must develop
66
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
assessment literate staffs that train teachers and principals to make standards-driven
decisions using dependable information on student achievement. In a data-intensive
society, schools are being challenged to find a simple system where teachers,
administrators, students, and parents can access and share information, then put it to
work to improve learning (Heck, 2000). Financial incentives are being offered to
schools and district that show initiative in this area. In California, the state's
economy boom provided substantial new resources for education (Brownell & Hart,
2001). New thinking policies to improve schools include Governor's Performance
Awards, training administrators on the use of data, and additional financial
incentives for improving low-performing schools.
In California, educational reform is currently focused on improving student
performance and increasing school accountability through the use of the (API)
Academic Performance Index growth targets for SAT 9 (Duffy, Goertz & Le Floch,
2001). Problematic with this current approach of educational accountability, is that
the Academic Performance Index is primarily based upon standardized student
performance assessments on the norm referenced Stanford 9 Achievement Test (SAT
9). Schools are investing a tremendous amount of resources in the analysis of
student performance data results as measured on the Stanford 9 Achievement Test
(Baker, Brown & Herman, 2000). The inherent problem with this current data use
strategy is the SAT 9 will disappear in three years and will be replaced by the
California State Standards Tests, and current California High School Exit
Examination in the secondary schools. These data assessment sources will prevail as
67
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
primary measures in which to assess student performance in schools statewide. The
emerging state context of standards-based educational reform will force schools and
districts to analyze the design, adequacy, and extent of implementation of their data
systems. School districts will be required to effectively assess student performance,
meet state API growth targets, and standards-based criteria for improving student
performance in schools (Calvo et ah, 1999).
In conclusion, school leaders, teachers, and district leaders have learned to
analyze student performance data as measured on the SAT 9, and must now shift
their emphasis in the use of standardized assessment data to assess curriculum,
instruction, and assessment practices in relation to the state content standards.
Schools and districts that fail to make this shift in the use of student performance
data will lessen their ability to assess instruction, curriculum, and assessment
programs to assist students in mastery of the state standards and meet graduation
requirements.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 3
METHODOLOGY
Chapter 3 has a four-fold purpose. First, the chapter will delineate the design
of the research study and the methodology used for instrumentation development,
data collection, and data analysis, and the purpose of the research study, which is to
investigate the district design for data use in schools. Next, it will show how the
design is linked to the current and emerging state context for assessing student
achievement. Then, it will inform the extent to which the design has been
implemented at the district, school, and individual teacher level. And finally, it will
measure and rate the adequacy of the district design through use of multiple criteria
Thirteen Ed.D. candidates at a large research institution worked as a cohort to
investigate and describe how schools and school districts throughout the state of
California use data. The uses of data employed by the schools and districts under
scrutiny were linked to the current and emerging state context for assessing student
performance. The combined findings of these dissertations formed this larger
comparative case study. The focus of the cross-case analysis was to identify how
effectively individual teachers and school leaders use data in the context of school
and district design. Each researcher selected an ethnically diverse urban school
within the state of California that had been recommended for having employed
advanced practices in data use.
Qualitative analysis has been applied in this study in order to provide a
description of the program. The cohort selected the case study method for it seemed
69
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
to be the most appropriate design for in-depth study of instances of phenomena in
their natural context, as well as from the perspective of the participants involved
(Gall, Borg & Gall, 1996). The intent is to identify promising perspectives of data
use for future research in this area. The qualitative study will include a researcher-
developed case study guide, designed to provide a conceptual framework for
collection and reporting of data organized by the three research questions.
Therefore, three research questions will guide and organize this study:
1. What is the district design for using data regarding student performance, and how
is that design linked to the current and emerging state context for assessing
student performance?
2. To what extent has the district design actually been implemented at the district,
school, and individual teacher level?
3. To what extent is the district design a good one?
Methods Description
The case study will observe and investigate data use practices and design
systems from one school district, and one school site, over a six-day period. The
researcher will self-collect and self-analyze the data from the case study. A case
study guide was developed to enable the researcher to anticipate and formulate the
types of data that should be collected in a finish-to-start approach (Gall, Borg &
Gall, 1996). The purpose o f the case study guide is to provide a framework for data
collection and data reporting which will appear in Chapter 4. Additionally,
70
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
conceptual frameworks for the case study guide are Conceptual Frameworks A and
B. The conceptual frameworks and the guiding questions serve as a focus for the
researcher while conducting both formal and informal interviews and while
informally observing at the district and site level. Results of this study will be
discussed in Chapter 4.
Sample and Population
This research study is limited to one school district and one comprehensive
high school within the district. It is part of a cross-site study among 13 different
districts and 15 schools. In order for a school or district to be selected, it must meet
the following specific sampling criteria: the district design for collecting and using
data must be is in place, and the district must use both SAT 9 data (as measured on
the California Academic Performance Index), and use multiple measures to gauge
student achievement. Additionally, the district in the sample must have an ethnically
diverse student population, and not have a majority student population with either a
too high or too low socioeconomic status (SES) base.
The process used for the selection of the district for this case study was to
select a secondary school district that met the specified sampling criteria. Research
on Orange County schools was conducted using the California Department of
Education Academic Performance Index website and informal interviews with
school leaders in Orange County. A district list was developed and narrowed to a few
schools that met the specified criteria. The district was selected after an in-depth
71
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
interview with the Assistant Superintendent regarding criteria for the study. The
selection of East Orange County High School District for this case study was made
because of their effective use of data and their relating it to the current and emerging
state context for assessing student performance. The East Orange County High
School District met the student SES population, Academic Performance Index, and
additional sampling criteria for the case study.
District
The purposive sampling and rationale for the selection of this district school
was its reputation for producing high student achievement. Additionally, the district
supports its school sites by producing quality student achievement data in an
efficient and useful manner. The East Orange County School District student
population consists of 15,400 total students in grades 9-12. There are six
comprehensive high schools and one alternative school. During the past three years
(1989-2001), the student population has increased by 1,000 new students. The
student population is ethnically diverse with 26% Hispanics, 20% Asian, 36% white,
11% Pacific Islanders, and 7%African American. Only 6% of the student population
qualify for free or reduced lunch program. Students identified as Limited English
Proficient (LEP) comprise 12% of the total student population. Both blue and white
collar jobs are represented by the parents in the community. The surrounding
community is primarily a residential area, with light industry, and commercial
enterprises.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
School
South High School was selected after an in-depth interview with the Principal
regarding the criteria. South High offers a wide range of educational programs for
students. School Magnet Programs include Academy of the Arts, International
Baccalaureate Program, Agricultural-Science Academy, Advanced Placement,
AVID, CISCO Networking Academy, and Culinary Academy. Students graduating
from the school attend California State Universities, community colleges, and trade
schools.
South High School has a diverse student population that reflects its
surrounding community. O f the 1,800 students (9th -1 2 th graders), over 50% are
Hispanic, 30% White, 10% Asian, and 8% are African American. According to the
2001 Academic Performance Index School Report, approximately 300 are
socioeconomically disadvantaged.
The school’s Academic Performance Index School Report shows steady
student achievement growth gains since 1999. Additionally, for the past two years,
South High School has qualified for state award moneys. South High has met its API
growth target for two consecutive years. South High API growth was a 21-point API
increase in 1999-2000, and a 13-point API increase in 2000-2001. Overall the
school has achieved a 34-point API growth during the past two years. The 2001
Similar Schools Report ranks South High a 2 out of 10 for schools with similar
school characteristics and student population.
73
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Instrumentation
Case Study Guide
The purpose of the case study guide is to provide a framework for data
collection and data reporting which will appear in Chapter 4. Additionally,
conceptual frameworks for the case study guide are Conceptual Frameworks A and
B. The conceptual frameworks and the guiding questions serve as a focus for the
researcher while conducting both formal and informal interviews and while
informally observing at the district and site level. The qualitative study will include a
researcher-developed case study guide, designed to provide a conceptual framework
for collection and reporting of data organized by the three research questions. The
case study guide addresses the three research questions on district’s design, the
implementation of the district design, and the adequacy of the district design (see
chart below and Appendix A).
Data Collection
Instruments
RQ1: Design RQ 2:
Implementation
RQ 3: Adequacy of
Design
Case Study Guide
• Interviews: District
Administrator, Site
Administrator, 6
Teachers (made up of
grade level / department
leaders and average
teachers)
• Mapping of Data
Flow at District and
School Site
• Artifact Analysis/
X X X
74
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Collection
• Quantitative data
Situated Interviews
• Forming vignettes
with 6 teachers
X
Teacher Questionnaires X
Stages of Concern
Questionnaires
X
Researcher Rating Form
(Post Data Collection)
X
Innovation
Configuration (Post
Data Collection)
X
Conceptual Frameworks
The Case Study Guide requires that all questions under Conceptual
Frameworks A and B are answered. Conceptual Frameworks A and B address
research questions #1-2 both the district design and implementation of the district
design (see Appendices B and C).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Conceptual Framework A
Description of Data Use Policies and Strategies
• Student Performance Assessed in the Context of Current and Emerging
Instruments
• Overview of the Elements of District Design of Data Use to Improve Student
Performance
® District Decisions and Rulings that Support Use of District Design
® Intended Results of Design Plans to Improve Student Performance (District,
School, and Classroom)
® Data Use Policy and Strategy Funding
Conceptual Framework B
Implementation of Data Use Policy and Strategy in Practice
« Degree of Design Implementation
® Implementation of Current Data practices (District, School, and Classroom)
• Implementation of Emerging State Data Practices (District, School Site, and
Classroom)
• Accountability for data use at district, school, and individual level
• Improving Student Achievement through Implementation of Data Use
Staff Interviews
The study requires formal interviews of one district leader, one site principal
and six teachers. The teacher interviews will represent both positions of leadership in
the school (i.e., chairs, coordinators, etc.) as well as typical classroom teachers. The
76
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
researcher created data collection instruments for the formal interview process using
the Conceptual Frameworks A and B. Staff interviews address research questions
#1-3. Formal interviews with the district and principal address specifically question
#3 (see Appendix D).
Observations
All observations for this case study are informal. Observing for environmental
information is part of the on site data collection process. Valuable data is gathered
while informally observing and interviewing district staff and school staff. The
analysis of both the informal observations data and school evidence will assist in the
formation of vignettes. Observations of the environment will address research
questions #1-3.
Mapping of Data Flow
A data map or flowchart will be included as part of the reporting out process
in Chapter 4. The mapping out of the collected data provides a schematic analysis
from the district to the school site and ultimately to the classroom. Questions help
define informal observations, informal interviews, and situated interviews. These
questions illustrate a data trail in the district and in the school site. A second
flowchart will be included to reconcile the proposed district plan for data use and
actual observations at the site. The mapping of data flow at the district and school
site will address research questions #1-3 (see Appendix E).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Artifact Analysis
The artifact analysis provides an informal environmental school and district
data collection. The researcher will include collected evidence from district and
school records, district and school policies, district and school publications, and
other means. Artifact analysis are used to address research questions # 1-3 (see
Appendix F).
Qualitative Data
The researcher is required to collect quantitative data during the data
collection process. The quantitative data collected is recorded in chart form and
addresses research questions # 1-3. A short explanation will be recorded if no data
was collected to address a question. Quantitative data includes examples of data use
on the district, school site, and classroom level (see Appendix G).
Situated Interviews
The researcher will conduct six situated interviews with the same six teachers
with whom the researcher conducted formal interviews. The purpose of these
interviews is to generate stories and examples about the way data are used (and not
used) in the school of study. These stories are used to form vignettes that will help
illustrate the information gathered through some of the other instruments. Therefore,
the intent of these interviews should aim to be factual yet personal. The bases for
these questions are from research question #2, implementation of the district design.
78
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Each section that follows comes directly from the conceptual framework headings
for this research question (see Appendices A and B).
Teacher Questionnaire
The researcher developed teacher questionnaire is used to address research
question #2. For each interview the researcher will provide a description of the
interview with relevant background information about the person. This includes title,
years of experience in current position, and position prior to the current one.
Questions from the teacher questionnaire are used to address both Conceptual
Frameworks A and B (see Appendix H).
Stages of Concern
The teacher questionnaire Stages of Concern which addresses research
question #1. The purpose of this questionnaire is to determine what teachers who are
using or thinking about using the district's design to use data to improve student
learning are concerned about at various times during the adoption process. Teachers
respond to each item on the survey in terms of their present concerns about their
involvement with the district's design to use data to improve student learning (see
Appendix I).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Researcher Rating Form
The researcher observation and rating form is used to address research
question #3, which is the adequacy of the district’s design. The researcher rating
form rates the district design using a 1-5 rating for the three areas of adequacy (see
Appendix J).
Rating Matrix
The researcher rating matrix rates the district design by the three areas on
adequacy. The researcher rating matrix rates the district’s design on adequacy and
addresses research question #3 (see Appendix K).
Instrumentation Development
This researcher met with 15 other doctoral students throughout the summer to
develop the instruments used for this study. The Summer Institute held at the
University of Southern California required the group to meet weekly and complete
necessary work in order to develop the instrumentation tools. Additional data
collection training was also required in the Fall. The Fall training included
refinement of the instruments used for this study, interviewing training, and data
collection protocol. Interview training consisted of review of summary guidelines for
interviewing, protocol review, and practice interviewing and questioning sessions.
Between training sessions, data collection deadlines were established. Each
80
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
individual researcher prior to the data collection at the school sites was required to
gain approval for Human Subjects Research.
Data Collection
A data collection schedule was established as part of the data collection
training session. The pre-data collection training session consisted of an extensive
review of instrument protocol prior to the data collection phase by the research team.
The three research questions and conceptual frameworks focused data collection
training. The researcher had to submit required credentials to the district office and
school site. The data collection schedule at the district was arranged by mutual
agreement with the Assistant Superintendent and the site Principal. Teacher selection
for the interview process was arranged with the Principal and the Assistant Principal
working with researcher criteria. Data collection schedule and timelines were closely
adhered throughout the data collection process.
Data Collection Schedule
The data collection schedule occurred in two rounds. The days of each round
were scheduled over six consecutive days. The researcher did not alter the schedule
throughout the data collection process.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Round 1
Day 1 District level interviews; collect artifacts and quantitative data.
Day 2 site level interviews with teacher (case study oriented and
situated Interviews) and administrator; collect artifacts including
quantitative data.
Day 3 Continue interviews and pass out Teacher Questionnaire and
Stage O f Concern (8 min. each) to all teachers and staff.
Break for preliminary data analysis
Begin mapping the data flow in the district, school site, and classroom.
Create a schematic.
Round 2
Day 4 Reentry with principal; revisit any issues with the artifact
analysis. Day 5 Finish any unfinished situated interviews to form
vignettes.
Day 6 Collect teacher questionnaires/ stages of concern and exit with
Principal
Debriefing
Day 7 Post Data Collection Debriefing with the group of researchers
at the University of Southern California to complete the Researcher
Rating Form.
The data collection process began with the formal interview of the district
administrator. The Assistant Superintendent of Educational Services provided me
with a one-hour interview on the district design for data use in her office. Color-
coding of red was used to organize notetaking and separating district documents
from other documents. State documents were color-coded as yellow. She provided
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
me with supporting documents that support the district and state design for using
data and how this design is implemented at the school site. Her background included
28 years of professional experiences in education as a teacher, site and district
administrator. Conceptual Frameworks A and B were used to guide the interview
process and questions. Extensive notetaking by the researcher was utilized
throughout the interview and organized by conceptual framework questions.
Document classification cover sheets were attached to supporting documents and
rated by research question #3 adequacy of the district design. Artifact analysis and
quantitative data were collected as part of the interview process.
The site principal formal interview consisted of a one-hour interview in her
office. The site principal of South High provided me with an overview of the district
design on data use and how this design was implemented at the school site and the
adequacy of the design itself. The principal provided me with both site documents
and state documents that addressed research questions #1-3. Conceptual framework
questions both A and B were used to focus the interview of the site principal. Color-
coding of green was used to differentiate site documents from district documents and
yellow was used to differentiate and classify state data documents (CBEDS, API,
etc.). Document classification sheets were attached as cover sheets for organization
and classification of each document. Extensive notetaking was used throughout the
interview process.
Interviews of the six teachers consisted of two separate interviews on two
separate occasions by the researcher in each teacher’s classrooms. The six teachers
83
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
provided the researcher with personal information during the interview that consisted
of their title, years of experience, and position prior to this one. Of the six teachers
interviewed, four were department chairs in math, language arts, social studies,
foreign language, and Reading Coordinator, and all had five years or more of
classroom teaching experience. The two remaining teachers, one was the technology
coordinator at the site and part-time math teacher, and the other teacher was an
AVID (Advancement Via Individual Determination) Coordinator and part-time
social studies teacher, and both had five years or less of teaching experience. Five of
the six teachers interviewed began their teaching careers at South High and have
remained there since, while the sixth teacher taught for five years prior to teaching at
South High.
The first set of teacher interviews consisted of formal questions focused on
the Conceptual Frameworks A and B questions. Informal observations were used to
gather environmental data during the classroom interviews. Observations of
classroom standards based documents were observable in the classrooms. The formal
interviews of the six classroom teachers provided the researcher with artifact analysis
of standards-based documents, samples of student work, lesson-plan design, rubrics,
and standards-based grading systems. Document classifications were color-coded
green to differentiate between site and district document classifications. Extensive
notetaking was used throughout the interview process. The length of the teacher
interviews consisted of one-hour in length for each interview.
84
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The second set of teacher interviews consisted of situated interviews that
specifically focused on research question #2, the implementation of the district
design. The purpose of the situated interviews is to provide the researcher with
stories and examples of the way data is used to (and not used) in the school. These
stories were used to form vignettes that allowed the researcher to illustrate the
information gathered through some of the other instruments used. Teachers appeared
to be more relaxed throughout the second set of interviews. Storytelling on data use
was recorded through extensive note taking and focused on Conceptual Framework
B. The length of each interview consisted of 45 minutes each in length.
The researcher distributed the teacher questionnaires and Stages of Concern
questionnaire to the staff during a staff development time and of the 65 full and part-
time teachers, 25 teachers completed and returned the questionnaires back to the
researcher. The questionnaires provided the researcher with data for research
question #2, the implementation of the district design. During the instrument
administration, teachers asked clarifying questions and appeared to be resistant to the
types of questions asked on the teacher questionnaires. Teacher questionnaires were
collected and organized by years of teaching experience. Responses from each of the
teacher questionnaires were analyzed based upon teacher responses on the district’s
implementation of the design for data use and the results reported in Chapter 4.
Additionally, the Stages of Concern questionnaire was recorded into an excel
spreadsheet and results reported in Chapter 4.
85
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data Analysis
Quantitative data was collected during the formal interview process and
classified to address research questions #1-3. Data was color-coded, organized, and
analyzed based upon current constructs such as SAT 9, STAR, API scores, and the
emerging constructs such as HSEE, CELDT, and CST scores. The principal and
district leader provided the majority of quantitative data along with teacher provided
student work samples. Quantitative data is reported in Chapter 4. Artifact analysis
provided a holistic approach to the data collection process. Quantitative data and
artifact analysis processes and the documentation classification process provided a
rich description o f the environment both at the site and district levels.
At the conclusion o f round 1 of the data collection process the researcher
created a preliminary schematic map of data analysis from the district to the site to
the school site and ultimately the classroom. Additionally, at the conclusion of round
2 of the data collection process, the researcher adjusted the schematic data map. The
mapping of data flow provides both the reader and researcher with a schematic
interpretation of data analysis in comparison to the intended impact of the district
data use design. The data map analysis is reported in Chapter 4.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 4
ANALYSIS AND INTERPRETATION OF FINDINGS
The purpose of Chapter 4 is to present the findings of the study. The research
findings are organized by each of the three research questions set forth in Chapter 1.
The results are presented in both a quantitative and qualitative manner. An
introduction is presented prior to the research findings.
Introduction
The purpose of this study is to document the experiences of the East Orange
County School District's utilization of accountability and student performance data to
foster increased student achievement through the district data design. The adequacy
of the district's data design will be examined as well as the extent to which the design
has been implemented and is aligned with the current and emerging state content for
student achievement through the results of several data instruments.
The Researcher Rating Form executed by the researcher will examine the
adequacy of the district's design. A formal interview with the Assistant
Superintendent of Educational Services will investigate the alignment of the district
design to current and emergent practices and the implementation of the design at the
district, school, and individual level and the improvement of student achievement
through the implementation of data use. The Stages of Concern questionnaire was
administered to the staff at South High School to examine the staff-wide
87
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
interpretations and concerns regarding the implementation of the district design at
the school site level and the individual classroom level. One Teacher Questionnaire
was administered to the staff at South High School to create awareness and quantify
the implementation of the design utilizing current data practices, accountability for
data use at the district, school, and individual levels, and improvement of student
achievement through the implementation of data use.
Therefore, three research questions will guide and organize this study:
1. What is the district design for using data regarding student performance, and how
is that design linked to the current and emerging state context for assessing
student performance?
2. To what extent has the district design actually been implemented at the district,
school, and individual teacher level?
3. To what extent is the district design a good one?
The following sections of this chapter will define each critical component of the
research questions utilizing Conceptual Framework A, Conceptual Framework B,
and the Researcher Rating Forms.
Conceptual Framework A addresses Research Question 1, investigating the
district's assessment of student performance in terms of current and emerging
performance practices, an overview of the district design to improve student
performance, district's decisions and rulings that support the use of the district
design, intended results of the design plans to improve student performance at the
88
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
district, school, and classroom level, and the funding sources for data use policy and
supporting strategies.
Conceptual Framework B addresses Research Question 2, the implementation
of the district design. Specific components utilized in Conceptual Framework B to
answer Research question 2 are the degree of design implementation (in the current
and emerging contexts), the accountability for data use at the district, school, and
individual levels, and the improving of student achievement through implementation
of data use.
The Researcher's Observation and Rating Form addresses Research Question
3, and investigates the following components of whether or not the district design is
adequate in three areas:
1. Researcher's Rating on how effectively the district design improves student
performance as demonstrated in standardized assessment results in terms of
impact on the school site. A rating matrix with rating of 1-5 (1 being not
effective, 5 being very effective) serves as the scale of effectiveness based on the
following components of Conceptual Framework A: Clear Performance Goals
aligned to or higher than state goals; (B) Dissaggregated standards-based
assessment data; (C) In-services on how to use data; (D) District is preparing
schools for emerging assessments.
2. Researcher Rating in the degree that the district provided student data used by
the school in terms of the actual student performance data forwarded to the
school is utilizing the student data to impact student performance. The
89
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
measurement of the adequacy of the district providing useful student data to the
schools is scored on the rating matrix of 1-5 scores and is based upon the
following critical components of Conceptual Framework B, District and School
Accountability to Standards-based Curriculum; (A) Motivation to use standards-
based assessment and instruction; (B) Regular assessment of student
performance; (C) Collaborative team's review data; and (D) Instructional
strategies individualized to promote learning.
3. The Researcher Rating on how effectively high student performance is
developed throughout the learning community in terms of the degree to which
student performance is aligned to the standards and communicated to teachers,
parent, and students. The critical component of data utilization is measured with
1-5 rating based on the following components of the Conceptual Frameworks A
and B; (A) The presence of high performance rubrics; (B) The understanding by
teachers, students, and parents of the standards-based curriculum; (C) Report
cards aligned to the state standards; and (D) All students progressing toward
high performance achievement.
Findings by Research Questions
Research Question 1: What is the district design for using data regarding
student performance, and how is that design linked to the current and the emerging
state context for assessing student performance?
90
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
To address this question the conceptual framework “Description of Data Use
Policies and Strategies: the Design” Cfa was used. Interviews along with observation
provided the primary data for evaluation. The case study guide was used in the data
collection. Interview questions included the district leader interview guide, site
leader interview guide, and teacher interview guide.
Student Performance Assessed in the Context of the Current and Emerging
Instruments
Current Practices. East Orange County School District Assistant
Superintendent of Educational Services developed the district’s data use design with
the intent of creating a standards-based system to improve student achievement. Her
staff consists of two directors, consultants, several teacher facilitators, technology
division experts, and three clerical support staff. This department provides data to the
school and provides “standards-based support to the schools” (Assistant
Superintendent, 2001).
District. Due to state and federal demands on accountability through
assessment, the East Orange County School District (EOCSD) has given
considerable attention to improving Stanford 9 and API ratings. The Educational
Services Department first receives the data from district or state assessments and test
publishers. The Assistant Superintendent uses the “Data Works” software program to
develop and examine the data to statewide comparison bands. The Assistant
Superintendent creates and prepares the reports with the support o f the Technology
91
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Department for the district and site principals. The Assistant Superintendent analyzes
the site and student data, then consults with teachers and principals to set annual
instructional goals and objectives for each site.
SAT 9 scores across the district showed steady positive gains across most
grade levels. The 2000-01 district API Growth Report showed mixed results. Not all
schools met or exceeded their Academic Performance Growth Targets. In the 2000-
01 school year, only two out of six schools met their API Growth Target and were
eligible for the Governor’s Performance Award Program. Two schools actually
decreased or showed no gain towards its intended API Growth Target/Three schools
overall showed positive gains in their API Growth.
The Assistant Superintendent meets with each principal and district content
experts to review the data, monitor, and analyze reports on the school’s data.
Principals are in-serviced using the “The Four Steps When Using Assessment
Information” model when analyzing data. SAT 9 scores are dissaggregated using
four-year comparisons by grade levels and in each content area. The Assistant
Superintendent requires that each principal be fully trained using this model and can
train his or her staff of teachers. The preparation program for site principals was
developed using a researched-based model for data use in collaboration with a
consultant employed by the Orange County Office of Education.
The research-based model uses a four-step approach in the use of assessment
information. The first step is to structure the analysis creating an environment to
ensure that staff knows and believes that this is important work. Second, is how to
92
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
summarize and analyze the data and to establish the need for staff to do this, this part
includes an exhaustive overview of student scores. Third, is to establish cause which
examines the physical environment of testing, attitudes of the testing program, test
taking skills, alignment of instruction, and quality of the instructional program. This
model is used extensively throughout the district “to create a standards-based
educational system” states the Assistant Superintendent.
The EOCSD assessment data is criterion based rather than norm referenced
based assessment. Authentic assessments in the form of district writing exams are
administered district-wide. Each student is assessed and administered a district
writing proficiency standard at the end of each year to ensure each student is meeting
the district writing standards. Students who are not meeting district performance
assessments are placed into classes that will assist them. Additionally, the district
supports technology and its use in the classroom for instruction purposes and to link
authentic assessments (student projects) to standards-based instruction. Students are
required to pass a district technology proficiency in order to graduate. Students are
enrolled in computer literacy courses if they do not pass the district technology
assessment.
The Assistant Superintendent uses the “Data Works Program” to
dissaggregate the data to generate reports that track every student’s progress towards
the standards yearly. “The district can track and document every students’ progress
towards each content standard.” These reports are published for the school board
twice a year. These reports link student achievement over a four-year continuum.
93
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The EOCSD data primarily includes only high school data, and the data only
includes limited middle school data; however, no elementary school is currently
available.
School. South High School is focused on improving SAT 9 scores and API
ratings. Student performance showed positive gains in the 2000-01 API School
Report. The school overall exceeded its API Growth Target Score by one point;
however, the school did not qualify for the Governor’s Performance Awards because
the school did not achieve the necessary gains in its API Subgroup scores. White
students showed a 48-point increase in their API Subgroup Growth Target, while
socioeconomically disadvantaged students achieved a 31-point increase in their API
Growth Subgroup Target score. Hispanic/Latino was the only subgroup that did not
attain their API Growth Subgroup Target score, thus preventing the school from
receiving state award moneys.
According to the South High School first-year Principal, SAT 9 scores are
one of the primary focuses of the school staff. One of the central challenges is to
examine SAT 9 scores and create a plan for change while improving student
achievement. The school plan developed by the leadership team includes how to
move students in the lower quartile Q1 which include student scores 17-24 (National
Percentile Ranks) NPR up to Q2 (23-30). Copies of the data were provided to the
staff in each content area. Each content area or department, in turn, provided the
leadership team with specific strategies on how they were going to address the goals
of the school plan. Specific strategies included Cornell notetaking, the use of graphic
94
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
organizers, and the use of reading strategies. Progress towards the school plan was
reviewed by the leadership committee each school quarter and published to the staff.
Data worksheets were distributed that included Q1 and Q2 student names, were sent
to each department, and teachers were held accountable to keep track of student
progress.
Improving student performance in SAT 9 scores and API rating is a focus to
the administrators and to the teachers. South High has used data consistently due to
the Western Association for Schools and Colleges (WASC). Last year, South High
School received a six-year clear term of accreditation. Teachers interviewed
indicated that the former principal “provided the leadership and spent a great amount
of time on WASC accreditation.” According to the WASC report, teachers were
using data effectively to increase student achievement before the SAT 9 existed. One
veteran teacher interviewed indicated that “the culture of using data at South High
School has been around for a while.” Teachers at South High were using authentic
assessments on observing student work, projects, and portfolios during the WASC
review. In the classroom, teachers stressed oral presentations, student projects,
artistic and creative products, and application of hands-on student projects.
Norm referenced assessments, such as the SAT 9, are used to assess student
progress towards the standards. SAT 9 scores and the SABE test for the Spanish
speaking population also serve as the primary tool for student placement. Students
who are assessed in the Q1 NPR level are placed into remedial math and language
classes at South High School. Even if a particular student’s grades are in the average
95
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
range, the chances of this student possessing below the Q2 range can exist. The
discrepancy between the average grade and lower SAT 9 scores are investigated and
the student may be placed in remediation to bring the student to grade level standards
and improved SAT 9 performance.
Technology is used in the classroom for instruction to link authentic
assessments (student projects) to standards-based instruction. The district
technology infrastructure does not provide a link for direct student assessment data
access for classroom application. The majority of classrooms observed has at least
one computer and television monitor most connected to the Internet. ROP classes
provide additional computer literacy classes for students. Teachers state “that they
must go to the front-office to access school assessment information.” Interviewed
teachers indicated that access to student assessment information was not available.
Emerging Practices. Interviewed teachers indicate that the district is moving
towards a standards-based system for improving student achievement. The Assistant
Superintendent has designed the district’s use data design to include a laser-like
focus on the standards in order to achieve this reform.
District Practices. The Assistant Superintendent has focused her efforts to
include the use of data to include the emerging assessments for aligning curriculum
to the California Standards Tests, CASHSEE, and CELDT measures. Student
performance assessment data are disaggregated for the sites biannually and the
district using these emerging measures that meet district and states criterion and are
aligned to the state standards. Examination of quantitative data and artifact analysis
96
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
collected both at the site, classroom, and district level reveal that student
performance is assessed by using these emerging data. Student intervention programs
are linked to the standards. Students who have not passed the California High School
Exit Exam (CAHSEE) and who are deficient in making progress towards the subject
area content standards are enrolled into remedial courses.
The district provides support for teachers align curriculum to the content
standards. This support can be seen in in-services for teachers and administrators on
standards-based instaiction and data use. California Content Standards Tests provide
the framework for aligning curriculum to the standards. The East Orange County
School District’s goals and objectives were aligned to the standards, and end-of-the-
semester examinations in the core content areas were also aligned to the standards.
The district provides teacher training that utilized standards-based instruction and
assessment strategies. The district developed standards-based assessments and
scoring rubrics for the school sites. The district scored districtwide-writing
examinations that are aligned to the state standards and sent these assessments to the
individual schools. Authentic assessments in the form of district writing exams are
administered districtwide. Technology is used in the classroom for instruction to link
authentic assessments (student projects) to standards-based instruction.
School Practices. South High teachers provide site administrators with
assessment data notebooks that include evidence of standard-based lessons to
improve student achievement. Teacher evaluations and instruction are assessed based
on the standards. Teachers are provided time to analyze student data and target
97
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
student deficiencies on standardized assessments. Intervention programs for students
are linked these emerging assessments. Remedial skills were emphasized for students
who fell below the 30th percentile on the SAT 9 and district assessments. Second
Language Learners presented the most difficult challenge in meeting the
instructional goals and objectives outlined by the district.
All of the South High School teachers interviewed (from the major content
areas of Mathematics, Language Arts, Social Science and Science) stated that they
actively engaged in standards-based instruction. The researcher’s informal and
formal observations would indicate active standards-based instruction as the teachers
had a standard listed on the board and objective to be learned for a particular lesson
accompanied by instructional strategies formulated to reinforce the day’s standard.
The South High School principal indicated that by requiring teachers to list the
standard on the board provided focused instruction on that particular standard for
both students and teachers (see chart below).
School Leadership Teacher Interview Summary Analysis
Teacher Years CFa
English 7 Authentic assessments
Spanish 9 Data use workshops (multiple measures)
AVID/ Social Studies 4 Alignment of curriculum to standards
Reading 12 Use of multiple measures
Math 14 Set attainable & measurable outcomes
Social Studies 18 Quintile targets for low performers
98
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
School leadership teacher interviews provided the most in-depth research
findings for this study. Teacher responses to Conceptual Framework A (Cfa) on the
district data design provided the most insight into the district data design when
teachers were asked what types of student performance data they use in their
classrooms to analyze how students are performing. The English teacher's response
cited that authentic assessments provide the most information on student
performance. The Spanish teacher cited his site training on data use workshops
helped him assess students using multiple measures as most important. The AVID
teacher (Advancement Via Individual Determination) cited the examination of
student performance data in comparison to the state content standards as most
important. Examination of student performance data that includes both standardized
assessments and classroom data (multiple measures) was cited as most important for
the Reading teacher. The math teacher cited having students set measurable and
attainable goals as most important, while the reading teacher cited targeting students
in the lower quintiles as most important. While the Social Studies teacher indicated
that a focus on students who fall in the lowest quintiles on standardized assessments
provided the most information.
Teachers at South High School have been trained during the summer and
throughout the school year to align curriculum, textbooks, criterion and standardized
assessments including SAT 9, CAHSEE, and the State Content Standards test. One
Language Arts teacher leader who was interviewed stated that she felt keenly aware
99
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
of the CAHSEE and felt immense pressure from administration to have high student
passing scores on the CAHSEE. The teacher also remarked very positively on the
district in-services on the CAHSEE. Many teachers at South High School have
participated in these standards-based trainings. Teachers were trained to serve as lead
teachers to help other teachers develop lessons emphasizing standards-based
instruction. The staff utilizes staff development time for careful analysis of student
performance data that may yield gaps in the curriculum, stimulate discussion on
instructional strategies that can target weaknesses or improve support, and further
development of assessments aligned to the standards (i.e., CST, CAHSEE) to
improve student achievement. Students at South High School take grade level
writing proficiency test, Golden State Exams, and Advanced Placement tests as well.
Additionally, students take ACT and SAT exams as part of college admission
requirements.
Overview of the Elements of District Design of Data Use to Improve
Performance. The types of data collected at the district for data use consists of all the
state mandated tests such as SAT 9, API, HSEE, and CELDT. The district writing
authentic assessments are collected and scored at the district. Academic test results
that consist of Advanced Placement, Golden State Exams, SAT, and ACT scores are
tracked at the district office. Student specific data and state required data that
includes student attendance rates, grade level, gender, ethnicity, and socioeconomic
levels for free levels and reduced federal programs are collected. Process data such
100
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
as student, parent, and staff surveys are also collected and stored in the district
design for data use.
The Assistant Superintendent collaborates with the Technology Department to
generate student performance reports for the school sites and the school board. Each
principal is provided with student performance reports that include student data
disaggregated into various report formats. These student performance report formats
consist of bar graphs, block graphs, line graphs, and pie charts that are disaggregated
by ethnicity, school, grade level, schools, and by content clusters. Each principal is
expected to review his or her data reports with their teaching staff and to develop a
plan to increase student performance. The Technology Department receives requests
for additional student performance report formats from the sites and Assistant
Superintendent. The Technology Department has recently expanded their staff to
increase their ability to create various report formats of student performance data.
The Assistant Superintendent provides the structure and the training in the
analysis of student performance data. The Assistant Superintendent and hired
consultants work with various stakeholder groups in the training, analyzing, and
interpreting student performance data. The “Four Steps When Using Assessment
Data Information” approach is used in the use of data at the district level and school
site levels.
District Decisions and Rulings that Support Use of District Design. The
EOCSD school board and district management support the district design for data
use. Biennial data reports of district goals and objectives are presented to the board.
101
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The board has not established data use policies for student achievement. District
parent and student surveys, and standardized assessments are used to make
districtwide policy changes to the instructional program. Student achievement yearly
district goals and site goals are developed using survey and standardized assessment
data in consultation with the school board.
The EOCSD board supports the instructional goals and objectives as outlined
by the Assistant Superintendent and the survey data. However, when the board
decides to make changes in the instructional program, a working group is established
under the direction of the Superintendent’s cabinet who then consults with teachers
and principals to develop policies and to make changes in the instructional program.
Data drives decision-making and supports improved standards-based student
performance at the district level. Site principals are responsible for the
communication of established board policies regarding and standards-based student
achievement goals and objectives at the school site level. However, teacher
interviews indicated that not all teachers at the school site were aware of site goals to
improve student achievement or how instructional policies were established.
Intended Results of Design Plan to Improve Student Performance (District- School.
and Classroom)
District Intended Results. The EOCSD intended results of the district design
consists of bringing about standard-based reform to improve student achievement
through the use of data. It is the Superintendent who provides the district leadership
102
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
with the board’s support and is ultimately held responsible to ensure the district
instructional goals are met and drives this reform effort. The Assistant
Superintendent and her staff are involved in every process to establish a standards-
based system. Her daily decisions must center on increasing student performance and
attainment of the standards. She states that school districts are using data more and
must do things differently in order to prepare students for these emerging
assessments. Increased accountability requires that assessment, curriculum and
instruction provide a seamless education.
Artifact and quantitative data analysis indicated that the district effectively
uses multiple measures to assess students and increase student performance.
Districtwide authentic assessments such as district writing samples and end-of-
semester student mastery projects were evident at the school site. Student
performance did improve as a result of the district design. Standardized assessment
measures consist of SAT 9, API, CELDT, CAHSEE, and CST data. The district
provides site staff these multiple measures collected by the district and forwarded to
the school site. Site principals use this information to guide instructional practice
while increasing student performance.
The district hired teachers and placed them on special assignment to provide
support services districtwide. These consulting teachers provide a variety of support
services in many areas. New programs include project PRIDE that supports teachers
in the district. Project PRIDE is the district’s Peer Assistance and Review (PAR)
program that provides a framework to assist teachers in subject matter knowledge
103
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and/or teaching strategies. Project START is the district’s Beginning Teacher
Support and Assessment (BTSA) program that provides professional development
for teachers in their quest to teach to the standards. Additionally, the district has
hired technology trainers to ensure technology is infused into teaching the content
standards. The T3 project supports staff development activities that provide for
teacher training in technology.
The district provides funding to support standards-based instruction. The
EOCSD sets a high priority and funding mechanisms to support student
achievement. The Assistant Superintendent provides a plethora of staff training
opportunities for teachers to learn the standards. Additional resources are provided
for standards-based programs if they address standards-based reform efforts. No
evidence of long-range planning such as a strategic plan or district master plan was
available to the researcher.
School Intended Results. South High School’s leadership team formed
comprehensive plan to improve student performance in 1999. The plan included an
intense literacy program to increase reading scores as measured on the SAT 9.
Sustained Silent Reading was implemented and a Literacy Task force was formed to
oversee the reading goals of the school. Hispanic student scores and second
language learners were performing at the low levels on the SAT 9 in reading. If the
school was going to meet its API Growth Target improving scores of Hispanic
students were vital. The 2000 Academic Performance Base Report showed a
104
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
remarkable 11-point increases for Hispanic students. All subgroups in 2000 met
their growth target and qualified for the Governor’s Award Program.
Student performance showed positive gains in the 2000-01 API School
Report. The school overall exceeded its API Growth Target Score by one point;
however, the school did not qualify for the Governor’s Performance Awards because
the school did not achieve the necessary gains in its API Subgroup scores. White
students showed a 48-point increase in their API Subgroup Growth Target, while
socioeconomically disadvantaged students achieved a 31-point increase in their API
Growth Subgroup Target score. Hispanic/Latino was the only subgroup that did not
attain their API Growth Subgroup Target score, thus preventing the school from
receiving state award moneys.
The teachers at South High held department meetings on data use. Department
meetings, both formal and informal, are primary sources o f data training to improve
instruction. The Social Science chair stated that attendance was high at department
lunch meetings that had data analysis as the central topic of discussion. The
mathematics teacher stated that colleagues were always eager to take
recommendations and strategies on the use of data back to their classrooms.
The staff at South High School systematically, through the researcher’s
attendance in staff meetings, through interviews, and actual class observation
sessions, the staff seems to utilize some forms of student data as a basis for
instructional modifications and accompanying, standards-based assessments to
increase student achievement.
105
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
South High administrators provide teachers with assessment notebooks that include
evidence of standard-based lessons to improve student achievement. Teachers are
provided with department time to analyze student data and target student deficiencies
on standardized assessments. Intervention programs for students are linked to
standards-based assessments. Remedial skills were emphasized for students who fell
below the 30th percentile on the SAT 9 and district assessments. Second Language
Learners number over 30% at the school site and present the most difficult challenge
in making large increases in standards-based reform efforts. The transient rate is
very high for these students and parent communication is difficult. Parents are
provided results of student performance on standardized assessments; however,
many parents are limited English proficient and have difficulty understanding school
information sent home. Few ESL parents can converse with teachers regarding
standard-based student progress.
The principal at South High School utilizes norm referenced and criterion
referenced assessment data to with his department chairs to diagnose gaps in the
curriculum. Teachers address those concerns with improved instruction based on the
standards. The principal requires that teachers document standards-based instruction
and assessment. Teacher evaluations require extensive analysis of standard-based
lesson design scrutiny. Post-observation teacher conferences consists of the objective
for the day; description of the content standard; description of lesson and activities;
closure of lesson; and commendations and recommendations. The principal states
106
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
that administrators and teachers are held accountable for teaching to the standards at
South High School.
Classroom Intended Results. As a result of having a standards-based
curriculum, teachers were familiar with state content standards and, in general, made
standards-based decisions in the classroom. In most classrooms, standards were
evident in the classroom and posted on the board along with the objectives for the
day. Teachers at South High placed a high priority upon improving student
achievement. Teachers interviewed were confident in the use of student performance
data. Data-driven decision-making has become part of the school’s culture. One
teacher interviewed stated that teachers did not receive timely student data from the
district and were unable to access student data information from their classrooms.
However, most teachers interviewed indicated that data use was more effective to
analyze in-groups rather than individually and that having the right data to examine
was always prepared in advance for the groups. Teachers are given access to Data
Works materials and other information but are expected to realize where the
organization’s needs to be based on the data. The district provides consultants and
department experts in instruction to assist teachers in lesson design and teaching
standards-based curriculum.
The majority of teachers interviewed at South High School feel that training,
their adherence to content standards instructional strategies, and their analysis of
department trends has helped their school increase their API score yearly. South
High for the third year in a row met their 2001 API target with a 14-point increase in
107
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
their score. In spite of the school’s API success, its Similar Schools Rank remains a
low rank of 2 out of a possible 10. The school’s Expected Schoolwide Learning
(ESLR). Results are aligned to the standards and measurable indicators of student
learning. ESLRs were posted throughout the school and the goal of high student
performance was evident throughout the school.
South High School informs parents with eight reports on student academic
progress in a school year. At the beginning of the school year, teachers send home
classroom expectations for the school year. One interviewed teacher states that he
sends home classroom expectations that include every content standard that will be
taught in his classroom for the school year and indicated that this was a department
wide commitment.
Language Arts and Social Science departments were using rubrics and
benchmarks to assess student performance. These departments were developing
writing rubrics for standardizing essay formats and quality indicators for student
mastery. All levels of student proficiency were aligned to the appropriate grade level
and content standard.
Researcher observations of classrooms indicated a high priority was placed on
improving student performance. Various projects displayed in the classroom to
standards-based assignments posted on the board as well as comments from teachers,
all indicate diligent work towards teaching the standards. One particular classroom
observation in Social Studies had students assessing their lesson on a rubric (1-4) for
each standard addressed during that day.
108
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Parents and students are informed of student progress towards the standards
through teacher notices mailed home, Parent Nights, and parent notification letters
for students who are not meeting the standards. Teachers stated that the
communication to students, parents, and community was regularly done. Artifact
analysis did indicate that classroom teachers regularly inform students, parents, and
the community of student performance goals and solicit suggestions for ways to
improve student performance.
Data Use Policy and Strategy Funding
District Policies and Funding. Many of the design elements for funding data
use implementation are grounded in the district’s mission statement, which states
“the EOCSD has a tradition of educational excellence for over a century.” This
“educational excellence” is translated into action that drives the district plan for
expecting high student performance. The major design elements consist of specific
training on standards-based curriculum design, expert teachers on special
assignment, increasing technology and increased access to student data, remediation
programs that are linked to standards-based student achievement, and research-based
programs are all district funded programs.
Funding for the district design originates from several places at the district
level and formulates a comprehensive district plan for improving student
achievement. The district and schools have benefited from Governor’s Performance
Award money, SB 1969 (CLAD), SB1882 (Staff Development), and grant monies to
109
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
fund the logistics and expenses of the district’ data use design. Title I, GATE testing
apportionment, and other categorical programs also fund the district’s vision of
student achievement. Remedial student programs and additional staffing to fund
additional math and literacy classes for students who are not meeting grade level
standards was funded by the district.
Funding does play an important role in affecting curriculum and instruction
decisions. The Assistant Superintendent works directly with the business and
operations division and the Superintendent’ cabinet to ensure that instructional
priorities are funded. Instructional decisions and policies adopted must be linked
directly to student performance data.
District interviews indicate that policies are established at the district level
and are implemented at the site level. Data collection strategies and policies for East
Orange County School District was district driven. Policies and strategies of the
district design are assessed from a district-centralized approach, and then
communicated, supported, and funded from the district level. No evidence was
available to confirm at the district or school site that the funding of the district data
design is linked to a long-range planning mechanism or strategic planning.
School Policies and Funding. Schools do have a role in funding sources.
Once the funds are earmarked for the school sites, the school assumes the
responsibility for the appropriation of monies. The school administration and School
Site Council have monetary discretion over funding resources at South High School.
The principal states that “the decisions made on funding must be aligned to
110
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
improving student performance.” Funding curricular programs is dependent on the
needs or focus for the school year.
A single school plan for improving student performance is developed yearly
by the school. This plan consists of connecting the objective(s) of the instructional
activity to the content standard. This plan includes the purchase of instructional
materials, attending professional development conferences, or hiring of consultant to
support a particular instructional program. The school plan drives the instructional
program by allocating resources are linked directly to instructional objectives, links
the resources of the school to the standards, and uses data to inform funding
decisions.
As the school1 s instructional needs have changed yearly, so have the funding
decisions of the school. The Advancement Via Individual Determination (AVID)
program was added to the instructional program to address WASC
recommendations. This program addressed the WASC recommendation to meet the
needs of the school’s increasing minority student population that want to go to
college. AVID students are challenged and provided additional support to take
college preparation courses. The AVID program has significantly raised student
performance at South High School while increasing the school’s budget with
additional grant funds.
Additional funding is needed in the area of teacher training on the use of data.
Since professional staff development days have been removed from the state budget,
schools have had to bank instructional minutes for staff development time. More
111
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
time is needed for data use training. Many teachers have only participated in data use
training once or not at all. Data training is needed to increase the school’s capacity to
use data effectively.
Research Question 2: To what extent has the district design actually been
implmented at the district, school, and individual teacher level?
To address question 2 “To what extent has the district design actually been
implemented at the school, district, school and individual level?” To answer this
question, the analysis of East Orange County School District’s extent of
implementation of the data design, elements of conceptual framework (CFb) guided
the data collection and analysis. Interviews and observations including the district
leader interview guide, the principal interview guide, and the teacher interview guide
were conducted based on the Case Study Guide. Additionally, the situated interview,
teacher questionnaire, and Stages of Concern questionnaire were utilized as well.
Degree of Design Implementation tin the current and emerging contexts)
Current. Administrators were clearly knowledgeable about the district’s
design for data design of using data to improve student achievement. The principal
was aware of the key elements of the district design. The principal has used staff
development time and department meetings to analyze student performance data to
improve student performance. Her administrative staff that consist of three Assistant
Principals and one Dean of Attendance were clearly knowledgeable about the
district’s design and also shared elements of the design through informal discussions.
112
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
South High School teacher leaders had a very mixed review of whether the
school had knowledge of the district design. Six different teacher leaders on campus
were interviewed stated that they have general knowledge of the district design, they
could not describe the key elements of the design. They also stated that the core four
departments (Language Arts, Mathematics, Social Studies, and Science) at the school
were more aware of the district design than other departments.
Teachers at South High School have knowledge of the district design and are
comfortable using and analyzing data during staff development time. Teachers
interviewed stated that the training on the use of data, particularly the analysis of
SAT 9 scores, has assisted them with the interpretation of the scores. One teacher
stated that he would like to have more time to learn different approaches in the use of
data. Teachers work with site administrators and counselors in obtaining reports
through the Technology Department. Teachers also obtain reports through the Title 1/
School Improvement Coordinator.
One teacher who teaches Limited English Proficient (LEP) students states that
the Title I Coordinator reviews student progress data with teachers regularly.
Interviewed teachers stated that the Title I Coordinator use of student data is
beneficial for both students and staff. An informal interview with the Title I
Coordinator confirmed the teacher comments on her use and knowledge of data use.
She states that both state and federal programs require that schools document the use
of student performance goals and objectives to funding. The Title I Coordinator uses
multiple measures when assessing students for placement in the ELL program.
113
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Along with SAT 9 scores, other assessments consist of CELDT, SABE, CAHSEE,
and other district criterion based student assessments. Teachers utilize her knowledge
of using data inform instruction regularly.
The implementation of data already existed at the school due to previous
WASC recommendations. The Technology Department generates reports and
consults with district level administration on the types of reports needed for the
district. Currently, there is no direct classroom technology link for teachers to access
student assessment information in the classroom.
The data based on the teacher questionnaire, Table 1, seems to support the
mixed reviews of the teachers’ awareness of the district design for using data.
Teacher questionnaires provided valuable information on research question two
regarding the implementation of the district design. Thirty-two teacher
questionnaires were collected at the school site; teacher responses were then
analyzed, summarized, and compared for pattern similarities. Listed below are the
thirty-two teacher questionnaire responses summarized in a table format. The
complete Teacher Questionnaire instrument is located in Appendix H.
Table 1: Teacher Questionnaire Summary Analysis
Question #
Number o f
Responses
Don’t
Know
0
Disagree
Strongly
1
Disagree
Somewhat
2
Agree
Somewhat
3
Agree
Strongly
4
*1 32 10 15 4 3
2 32 7 8 1 15 1
*3 32 2 27 3
*4
32 5 6 12 9
*5 32 1 3 22 6
6 31 6 7 16 2
7 32 3 4 2 10 10
8 30 2 17 11
9 32 1 23 7
114
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
10 32 1 1 6 14 9
11 32 2 4 3 18 5
*12 31 1 13 8 9
*13 30 7 21 2 1 1
14 32 3 11 2 14 2
*15 32 3 16 8 2 3
16 31 7 3 16 5
*17 32 3 2 18 9
*18 32 1 2 1 24 4
19 31 6 8 5 11 1
20 32 12 4 16
21 32 5 17 7 2
*22 32 9 3 15 5
*23 31 1 17 2 11
* Indicate significant findings (items further addressed in the narrative)
The teacher questionnaire provides significant research findings for further
discussion in regards to the implementation of the district data use design. Teachers
at South High were not aware of the district design for using data as indicated in
question 1. Twenty-five teachers responded that they did not know or disagreed
strongly that they were not aware of the district design for using data. Overall, the
teachers’ awareness of the district design was 1.45/N=25 out of 4.0. Questions 3, 4,
and 5 showed that teachers collected, monitored, and used data on a weekly basis to
monitor and guide instruction as indicated by responding that they strongly agreed
with these questions. The majority of teacher responses to questions 12 and 13
resulted in strong teacher disagreement with their use of data to compare past and
present performance on individual students and within classes. Twenty-four out of
32 teachers indicated that they disagree that reports are sent home to parents
regularly. Twenty-seven out of 32 teachers responded that the school offers
professional development to raise awareness o f new data practices, while 28 out of
115
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
32 teachers indicated that have participated in these training's during the past six
months. Twenty teachers responded strongly to somewhat agree that school
administrators have assisted them in implementing new data practices, while 17
teachers strongly disagreed that school administrators have monitored their
utilization of new data practices.
The Teacher Questionnaire used on the Conceptual Framework B on
implementation of data use policy and strategy in practice was confusing to many
teachers. The result of questions under the degree of design implementation of
current practices was at an average of 1.45 out of a scale of 4.0 (see Appendix H).
Teacher comments written on completed and returned surveys substantiated this
teacher confusion. Only 32 out of 85 staff members filled out the Teacher
Questionnaire, which lacks the representative number of teacher responses.
Stages of Concern Questionnaire was administered to 32 teachers who are
thinking about using the district design to improve student learning and are
concerned at various times during the innovation adoption process. Table 2 Stages of
Concern Summary Teacher Responses addresses the degree of implementation of the
district data use design (see Appendix K).
Table 2: Stages of Concern Summary Teacher Responses
Question
#
Responses Irrelevant Not
True
Somewhat
True
Very True
2 32 7 18 6 1
3 30 2 1 18 10
7 32 1 3 17 11
13 31 2 6 19 4
15 32 6 10 16
16 31 1 3 22 5
116
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
29 32 2 8 20 2
33 31 4 2 9 16
36 32 2 1 5 24
The Stages of Concern teacher questionnaire indicates that teachers at South
High were concerned about the use of data in the areas of teacher expectations,
professional growth, decision-making, resources, student achievement. Questions 2
and 29 indicate that teacher training and sharing of data use is evident at South High.
Question 2 showed 25 teachers did not know of other approaches for data use and
question 29 indicated that 25 teachers are interested in learning new techniques from
their peers. Question 3 had 28 teachers in agreement that the district implementation
process needs further explanation and communication. Teacher responses were
mixed when asked if they knew about the district data use design to improve student
achievement. Question 7 indicates that 27 teachers were concerned about the district
data use design and the effects on the classroom. Question 16 indicates that 27
teachers are concerned about their ability to manage all that is required to implement
the district data use design. Question 33 directly addresses teacher concerns
regarding their changing role in the use of data to improve student achievement.
Twenty-five teacher responses indicate their concern with their changing role in this
area. Question 13 indicates that 23 teachers were concerned with who makes the
decision for data use implementation, while question 15 indicates that teachers are
concerned with resources available for data use implementation. Question 36
indicates that teachers overall are concerned with the effect of the district's data use
design on students.
117
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Teacher stages of concern responses at South High School indicate that the
district data use design implementation needs further communication and refinement.
Teacher concerns are evident in many areas at the school site. Teacher concerns were
evident in the area of resource allocation and decision-making. Teacher concerns
were evident in the area of changing role expectations particularly in the classroom.
As the demand for increased student accountability has increased for schools, so has
the need for quality teacher training. Teachers at South High and districtwide share
information, but teachers overall are lacking in classroom assessment literacy
strategies.
Teachers who have attended district in-services on data use are the most
comfortable in interpreting data. Less experienced teachers or teachers with two or
less years of teaching experience are the least comfortable using data. One
interviewed teacher stated that she would request reports from counselors.
Experienced teachers call the research department directly to access information.
Technology is available in most classrooms, however, only administrators and
counselors have direct access to online student information. The student information
system contains student score reports that consist of SAT 9, API, AP, GSE, ACT,
HSEE, and CELDT disaggregated testing information. Teachers interviewed
primarily use SAT 9 and API student assessment scores provided for them to inform
instruction.
Emerging. The Assistant Superintendent and her staff are putting increased
emphasis on implementing emerging data practices. She stated that educational
118
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
reform in California is moving away from norm referenced testing and is shifting its
attention to standards-based reform efforts and criterion-based testing such as the
California Standards Tests and the CAHSEE.
The East Orange County School District implementation of emerging data
practices involves a collaborative effort that includes participation by district
personnel, site administrators, department chairpersons, and teachers in standards-
based professional development. The district staff provides for monthly meetings
and inservices to discuss emerging data practices and methods needed to implement
standards-based reform. The Assistant Superintendent stated that a collaborative
effort by district staff, site administrators, and teachers was needed to implement
emerging data practices districtwide.
The district relies heavily on end-of-course exams aligned to the standards.
The four core departments (Language Arts, Mathematics, Science, and Social
Studies) conduct analysis meeting with fall exam diagnosis. Department diagnosis of
end-of-course exams was vital to achieve standards-based reform. The data can serve
as benchmarks for longitudinal data sources, standards-based modification including
criterion lesson design for each standard and student knowledge. District curriculum
experts have aligned authentic assessments in writing to the CAHSEE exams and
have established district wide writing rubrics linked to the standards.
The most effective element o f emerging data practices at the school site was
that the standards were linked to instruction and curriculum. Teacher Questionnaire
Degree of Design Implementation of emerging data practices contradicted teacher
119
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
interviews and researcher observations. The mean teacher respondent was 2.21 N=32
for emerging data practices indicating somewhat disagree response (see Appendix
H). Teachers received regular in-services on the standards at the district and site
levels. Teacher interviews and questionnaires indicating high levels of standards-
based training opportunities were provided for teachers.
Interviewed teachers used standards-based instruction and assessments to
guide curriculum decisions in the classroom. Data use was evident in most
classrooms to improve student achievement in the forms of charting student
achievement on posters, lesson plan design, grading policies, and technology
applications. Teachers posted reading charts on the walls that showed student-
reading increases linked to standards-based assessments (CELDT, CST, and HSEE).
Classroom observations showed teachers listing the standards and objectives for the
day on the board. Teacher interviews indicated that this was valuable for both
students and teachers in order to focus during instruction on the particular standard
for the day. Technology use during instruction was evident in most classrooms,
teachers used technology during instruction that linked content standards materials
and software to instruction.
Conceptual Framework B (CFb) provides design implementation information
when teachers were asked what elements were most effective in bringing about
student achievement gains. The English teacher cited the use and regular
examination of schoolwide authentic assessments (portfolios, cumulating senior
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
projects, collaborative assessments) as most effective. The Spanish teacher cited
district alignment o f instruction to the standards activities as most important. While
the AVID teacher cited the use district and school efforts to examine national
assessments and growth targets particularly for minority students as most effective.
Regular and usable assessment data forwarded by the district to the school site was
cited as most important to the reading teacher, although, no evidence of this strategy
was evident at the school or district levels. The math teacher cited alignment of end
of the unit exams and the creation of question stems aligned to the standards
activities as the most effective element of the district design. The Social Studies
teacher cited the district strategy of alignment of all classroom assessments to state
content standards as most important (see chart below).
School Leadership Teacher Interview Summary Analysis
Teacher Years CFb
English 7 Assessment feedback
Spanish 9 Alignment of instruction to the standards
AVID/ Social
Studies
4 National assessments and growth targets
Reading 12 Need for immediate data resources at school
site
Math 14 End of unit exams aligned to the standards
Social Studies 18 Alignment of all classroom assessments
Interviewed teachers indicated that standards-based professional
development was just beginning and that much more training was needed to fully
121
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
implement the district design. Teacher questionnaire data revealed that teachers
strongly agree range that staff development opportunities were available to teachers.
Accountability for Data Use at District. School, and Individual Level
District. The district policies and strategies for effectiveness are monitored
through data use. The Assistant Superintendent is held accountable for monitoring
through data use. The Superintendent places complete trust in her ability to produce
the desired results of standards-based reform efforts. Accountability for data use
occurs at all levels-district, school, and teacher.
Student performance assessments were completed biannually for the district
and the school sites by the Assistant Superintendent’s “Data Works” program and
the Technology Department. East Orange County School District provided data for
the district and school in the forms of SAT 9, CELDT, CST, district standards
testing, and the CAHSEE. SAT 9 results were disaggregated yearly and goals and
objectives were established for each school site and grade level cohorts. Biannual
databases and references which include district goals and objectives are presented to
the board yearly and reviewed twice yearly. As a result of accountability for data
use, student achievement is increasing on standardized measures across the district.
The East Orange County School District forwarded disaggregated assessment
data to the school sites. District administrators developed student performance
reports for all schools at the beginning of the year and reviewed with site principals.
The district infrastructure for communicating assessment data is still under
122
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
development. Online student data is not available for classroom teachers and must be
requested by individual teachers from the school site administration, and then from
the district.
School Accountability for Data Use. Principal interviews indicated that site
principals are held accountable to be instructional leaders at their school sites. The
principal stated that she is expected and held accountable to improve both SAT 9
student scores and meet API growth targets.
The API for South High School increased over the past three years. Last
year, the South High School missed their API growth target by 1 point as a result of
one sub-group failure to meet its expected growth target. The district holds principals
accountable for their school site standardized assessment scores. SAT 9 State
Academic Performance Growth Targets are examined yearly and are included as part
of the principal’s yearly evaluation. Department chairpersons are expected to attend
in-services and workshops designed to increase student performance on standardized
assessments.
The California Standards Tests and SAT 9 scores are examined in the core
content areas, alignment of curriculum and content score increase is expected to
occur in these areas. Teacher interviews indicated that key teachers in the core
content areas along with department chairs are expected to participate in district
developed standards-based workshops or in-services.
Accountability for increased student performance effects the enrollment of the
school site. South High School’s student population has shifted dramatically during
123
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the past five years. Student longitudinal data reveal an increasing Latino population
shift of ESL students. White fight has occurred from South High School to district
magnet schools and programs. Reduced enrollment affects school funding and all
programs. Teacher interviews indicated that high student achievement will assist
with community perception issues and increase enrollment.
Individual. Teachers are held accountable in delivering standards-based
curriculum. Teachers are held accountable for test results and grades. Teachers must
submit assessment notebooks to site administrators. The teacher questionnaires for
data use somewhat disagree that schools hold teachers accountable. Teacher results
on question 26 were rated a 2.2 out of 4.0.
The PRIDE and START programs were added to support teachers and
instruction in the district. Both programs use PAR and BTSA money to provide
interventions and support for teachers to be successful. Teachers were hired as
teachers on special assignment to support new and veteran teacher learn new
strategies to support standards-based instruction.
Teacher interviews indicated that parents support was lacking in helping
students learn the standards. Parents received student data regarding school progress
towards standards-based curriculum in the form of school information mailed to their
homes. Teachers claim that parents often did not fully comprehend standards. Report
cards in the district are not standards-based. According to teachers, parents did not
view school progress towards a standards-based instruction as essential. Students and
parents were not interviewed for this case study; therefore, there is no data available
124
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
to rate student and parental support for standards-based instruction and assessment.
However, student scores received on standardized assessments affected their student
placement into specific courses. Accountability for the use of data to support
standards-based instruction was high for all students and low for parents.
Improving Student Achievement through Implementation of Data Use
District. The use of data has shown significant student achievement gains.
During the past two years, the district’s API scores have increased across the six
comprehensive schools a total of 47 points. Two schools in the district are ranked a 9
and 10, respectively, in the 2001 Similar Schools Report, while a third school was
scored an 8 on this measure.
Research Question 3: To what extent is the design a good one?
Research question 3 asked, “To what extent is the design a good one? To
answer this question, the Case Study Guide was used. To address this question, the
Conceptual Framework C as well as the Researcher Rating and Observation Form
provided the tool to answer the adequacy of the design.
District Support for Standards-based Instruction
District. The East Orange County School District intends to increase student
performance with the continued use of the State adopted standards-based curricula.
The Assistant Superintendent supports principals in making standards-based
decisions to improve student performance. The district is aware of the state and
125
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
national efforts to be standards-based and applies concepts to facilitate changing
curricula. Teachers in the East Orange County School District believe that standards
help create a coherent instructional program that results in increased student
achievement levels. Principals and teachers are aware of district performance goals
and work collaboratively to standards-based goals.
The district disaggregates and forwards received SAT 9 and CST, CELDT,
and CAHSEE assessments from the state to individual schools. Quantitative student
performance data was developed, scored and charted by the district for the school
sites by the Assistant Superintendent and her staff using the “Data Works” program
(see Quantitative Data chart located in Appendix G). The district assists in the
scoring of the authentic assessments and sends these assessments to the individual
schools. The district requires each school to develop its own data collection policies
and strategies at the school site. Data collection policies and strategies are site
driven. Student data is collected by district staff and the site to monitor student
progress towards standards-based reform efforts. Student data consists of end-of-
course exams, writing samples, and teacher tests aligned to the standards. Principals
also collect teacher data, that includes assessment data documenting standards,
lesson design linked to the standards, and teacher evaluation data, which is then
reported to the district.
The district expects yearly grade level increases on student performance
measures and tests and keeps track of them yearly. The district offered several
standards-based staff development workshops for grade-level teams in the core
126
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
content areas. Data was used to examine student performance and most teachers
were in-serviced on data use strategies and standards-based instruction. Expert
teachers aligned curriculum to the State content standards and the HSEE.
The district has a plan for data use and standards-based reform efforts but it
has not been fully implemented at the school sites. Researcher-teacher observations
indicate that standards-based efforts are under development at the school site.
Teachers are phasing in the use of standards in the classroom, aligning assessment to
the standards, and aligning instructional methodologies to the standards.
Rating of District Support for Standards. The rating of the district support for
standards as based on a scale of 1 to 5, indicates how effectively the district support
standards-based performance instruction. Rating rubric terms for each of the
numbers are as follows: l=not effective; 2=somewhat effective; 3=unclear;
4=effective; and 5=very effective. On this scale, district support for standards-based
instruction was 4= effective.
School. South High School utilizes standards-based instruction and assessment
practices. The principal, teachers and students are aware of clear performance goals
expected by the school. Students are required to perform at or above the state
standards or they are placed into remedial classes of English, reading or
Mathematics. South High School’s expected schoolwide learning results are aligned
to the standards and are measurable. As a result o f standards-based curriculum and
instruction, teachers were familiar with standards and collected student data as part
of the teacher evaluation process. Additionally, teachers charted and logged student
127
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
assessment notebooks and described standards-based instruction as part of the
teacher evaluation process.
Teachers were using available disaggregated data to make changes in
curriculum, assessment, and instruction. The teachers cited difficulty in obtaining
student performance data for analysis as a concern. South High School teachers do
not have direct access to student performance data from their classrooms, although
many teachers do possess computers. Teachers preferred prepared disaggregated
data for analysis to occur in group or department settings rather than individual
analysis.
South High School teachers participated in district data use in-services both
on site and at the district. Teachers indicated a need for increased data use in-services
at the school site. Staff development time to explore the student performance results
is being addressed through special bell schedules and staff in-services. Teachers at
South High School work collaboratively in the use of data to align curriculum to the
standards, share teaching strategies, and assess student progress.
Rating of School Support for Standards. The rating of the school support for
standards-based instruction was based on a scale of 1 to 5, indicating how effectively
the school supports standards-based instruction and assessment. Rating rubric terms
for each of the numbers are as follows: l=not effective; 2=somewhat effective;
3=unclear; 4=effective; and 5=very effective. On this scale the district design was 4-
effective.
128
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
District and School Accountability to Standards-based Curriculum
Forwarding Performance Data to School. The district forwarded all assessment
to the school sites following disaggregation of the data by the Assistant
Superintendent who uses a “Data Works” software program. The district Technology
Department worked with the Assistant Superintendent to develop a computer
program format for disaggregated student performance data into a usable form.
Disaggregated data generated for the site consist of the following data:
• API Results (API 1999-2001) - by school; API subgroups; API growth targets;
and Similar School Reports
• Stanford 9 Results - 4 year school summaries; percentile ranks; subgroups; grade
level; language fluency; SES; English, Math, Science, and Social Science; and
Quintile ranks.
• California Content Standards - Percentage of students at or above or below
English/Language Art performance levels
• CAHSEE School and Individual Results- for the class of 2004
« District Assessments - writing; technology; grade distribution; math and
language arts by language, subgroups, grade levels and four-year trends
• ACT/SAT I and II; GSE; AP Report results- for college bound students
CELDT; SABE, SAT 9 and CAHSEE Report results- for ELL students
Researcher observations and teacher interviews indicate that teachers did not receive
individual student score reports and class listing of student score reports on a
consistent basis.
129
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Rating of Provided Data. The rating was on a 1 to 5 scale on the degree to
which the district provided student data that was to be used by the individual schools.
Expressed results for each numbers 1 to 5 are as follows: l=not effective;
2=somewhat effective, 3= unclear; 4=effective; 5=very effective. On this scale the
rating of provided data to schools was 2- somewhat effective.
Utilization of Student Data at the School Site. Teachers at South High
School utilize data use practices. Teachers worked on collaborative teams to analyze
student performance. Department teams analyzed department-related data and
instructional approaches, while the leadership team met monthly to review
schoolwide data. One example of this is the school’s Literacy Task Force Committee
was formed to improve literacy schoolwide. The committee examined student
reading scores and reading interventions were formulated to improve reading all
based on the data analyzed. These interventions included implementing schoolwide
reading strategies, motivating students to read with a “Read A Book” contests, and
SSR.
Researcher observations revealed that many teachers require additional
training or increased knowledge on the use data in regards to making standards-
based decisions in the classroom. Since student data was not readily available for
classroom teachers, many teachers relied on classroom-based assessments to make
decisions regarding student achievement and standards. Teachers were most
comfortable making standards-based decisions and using data in the classroom in the
core content areas (Math, Science, Language Arts, and Social Studies). The 2000
130
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
WASC Visiting Team Report commended the staff for their use of data as a strength
area in the school’s Self-Study Report.
The API at South High increased yearly each of the last three years. This
increase may be attributed to grade-level teaming and staff collaboration on
curriculum and the standards during staff development activities. Regular assessment
of student performance was completed by end-of-course district exams. Teacher
questionnaires did not provide sufficient information as to how often teachers met to
review district and site assessment data and was not available for the researcher.
Instructional strategies were individualized to promote student learning and were
evident through teacher observation. Instructional strategies were individualized.
Teachers charted student progress in some classrooms on SAT 9, CELDT, and
CAHSEE. These assessments were helpful in providing intervention programs for
certain students.
Rating of School Utilization of Data. The rating was on a 1 to 5 scale on the
degree to which the school utilized student data. Expressed results for each numbers
1 to 5 are as follows: l=not effective; 2=somewhat effective, 3= unclear;
4=effective; 5=very effective. On this scale the rating was 4= effective.
Determination of “High” Student Performance
Researcher observations indicate that the focus of South High School is to
achieve high student achievement. The school expects that all students meet high
standards. South High School meets the needs of all students and expects students to
131
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
perform at high levels. Many students go to college and college preparation courses
are offered for these students. ROP classes are offered to students who want to take
career-oriented classes. A variety of ELL classes are offered to meet the needs of
these students. These classes offer intensive literacy programs that are aligned to the
ELD state content standards and subject area content standards. Students are placed
into remedial classes if they do not perform up to grade level standards.
Instruction in the core content areas is currently being aligned with the
California State Standards. The majority of teachers understood the standards and
the advantages of standards-based instruction and assessments. Teachers promote
individualized instructional strategies to promote student learning. Teachers meet
during monthly department and WASC cluster groups to align instruction and
materials to the standards.
Leadership committees meet regularly to monitor department progress, which
increases school accountability to meet the standards. Researcher observations did
not indicate the evidence of high performance rubrics use at the site level. District
materials did contain rubrics in the core content areas and teachers did indicate use at
the classroom level (see chart below).
School Leadership Teacher Interview Summary Analysis
Teacher Years CFc
English 7 Lack of depth in curriculum
Spanish 9 Assessment and placement
132
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
AVID/ Social
Studies
4 Narrowing of curriculum
Reading 12 In-service parents on standards
Math 14 Increase student accountability
Social Studies 18 Too many standards
Leadership team interview responses provided information and substantiated
researcher observations and ratings Conceptual Framework C (CFc) the degree to
which high student performance is aligned to standards and communicated to
teachers, students, and parents. The English teacher cited that the school learning
community was misinformed regarding standards and high student performance. The
English teacher cited the lack of depth in the school curriculum and superficial
learning occurred when too much emphasis was placed directly on the standards. The
Spanish teacher cited that students were misplaced when too much emphasis was
placed on standardized assessments. The teacher cited student tracking resulted in
students being placed into remedial classes was a form of tracking. The AVID
teacher cited the narrowing of the school curriculum as a problem when asked to
describe the degree high student performance is aligned to the standards. The reading
teacher cited in-services for parents were needed to help parents understand the
standards. No comprehensive school program was cited to help parents to understand
the standards, although a few ESL parent support programs in the school did exist.
The math teacher cited increased student accountability as most important. The
teacher cited the future of the CAHSEE exit exam would raise future student
133
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
accountability in terms of student attainment of the standards. Too many standards
for teachers and students were a major education problem cited by the Social Studies
teacher. The teacher indicated that too many standards to choose from and learn
prevented high student performance.
Rating on “High” Student Performance. The rating on a scale of 1 to 5 was
on how effectively high student performance was developed throughout the school
learning community. Expressed results for each of the numbers 1 to 5 are as follows:
l=not effective; 2=somewhat effective; 3=unclear; 4=effective; 5=very effective. On
this scale the rating was 4=effective.
Communication of “High” Student Performance to Student and Parents.
Parents and students did receive regular communication meeting “high” student
performance goals of the district and home communication of student progress
towards the standards. Teachers cited language issues as the most compelling reason
for parents not understanding the standards. South High has a 60% English Second
Language Learner student population. Parents did receive eight reports and other
communications from the district and from the school throughout the year regarding
student progress. Student report cards are not aligned to the standards; however
grade reports and student standards are communicated to parents and students in
regular mailings, parent nights, and parent-student conferences.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Rating on Communication of “High” Student Performance to Parents and
Students. The rating on a scale of 1 to 5 was on communication of “high” student
performance to is communicated to parents and students. Expressed results for each
of the numbers 1 to 5 are as follows: l=not effective; 2=somewhat effective;
3=unclear; 4-effective; 5=very effective. On this scale the rating was 4-effective.
Discussion of Research Findings
Based on an analysis of the research data, the following findings were
discovered:
1. Teachers are currently using a variety of student assessments both classroom
based assessments and standardized assessments to assess student performance
and progress towards state and district content standards.
2. Professional development in the area of assessment literacy training and data
based decision-making for teachers and administrators is needed at the school
site level.
3. Decision-making regarding standards-based student achievement goals and
objectives, as well as student intervention strategies and programs, and resource
allocation require decision-making and communication at the school site level
with district support for implementation.
4. Development of district technology support systems that are flexible and that
provide direct teacher and classroom access to on-line resources for multiple
135
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
student and teacher assessments that are linked directly to the emerging standards
should be the primary focus of the district data use design system.
Finding #1: Teachers are currently using a variety of student assessments both
classroom based assessments and standardized assessments to assess student
performance and progress towards state and district content standards.
Case study research findings reveal that teachers on one hand are beginning to
increase their use of data to make decisions about student performance. Teachers are
making substantial progress in the area of student assessment analysis and standards-
based decision-making (Khanna, Trousdale, Penuel, Williams & Kell, 1998).
Teachers are learning to use both standardized assessments and classroom
assessments in which to gauge student performance towards attainment of the
standards (Popham, 2000). Additionally, teacher training in the area of the emerging
state standards was a positive finding in this study. Teachers are participating in
professional development activities to align curriculum to State content standards.
Finding #2: Professional development in the area of assessment literacy training and
data based decision-making for teachers and administrators are needed at the school
site level.
Case study findings suggest that schools and districts are not doing enough to
close the glaring gap in assessment literacy for teachers and site leaders (Goertz,
2000). Teachers and site leaders are not currently trained to use student achievement
136
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
data, demographic data, and process data to make decisions about student
perfonnance (Bernhardt, 2000).
Finding #3: Decision-making regarding standards-based student achievement goals
and objectives, as well as student intervention strategies and programs, and resource
allocation require decision-making and communication at the school site level with
district support for implementation.
Case study findings revealed that the decision-making regarding the
development and implementation of the district design in matters regarding student
achievement goals and objectives, student intervention programs, and resource
allocation should be made at the school site level and coordinated with district
support (Levesque, 1996). This finding was clearly evident with teachers at the
school sites unfamiliarity with the district design. Teachers could not describe the
key elements of the district's data use design. Elements of the district design were
evident in many areas of this study, particularly in the area of standards-based reform
efforts. Teachers were aware of the standards, API index and participated in the
alignment of curriculum to the emerging state standards (Duffy, Goertz & Le Floch,
2001). However, this study revealed that without site input into the decision-making
process, gaps in communication and implementations were problematic in a district
driven data use design system. Researcher findings suggests that the school site must
be actively engaged in the development process of establishing student achievement
objectives, intervention programs, and resource allocation in the data use design.
137
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Finding #4:Development of district technology support systems that are flexible and
that provide direct teacher and classroom access to on-line resources for multiple
student and teacher assessments that are linked directly to the emerging standards
should be the primary focus of the district data use design system.
Case study research findings revealed that the development of district
technology systems that are flexible and include direct teacher and classroom access
to a variety of student assessment data and that are linked to the emerging state
standards should be the primary focus of the district design (Christie, 2000). School
site databases should have on-line software and resources that allow both student and
classroom assessment tutorials that provide progress towards the attainment of the
standards (Keller, 2001). Flexible technology and infrastructure systems that provide
school sites with standards-based data, standards-based lessons, and site data
management systems have emerged as critical elements for districts to achieve
adequacy in the district data use design.
Conclusion
A case study was completed at South High School and the East Orange
County School District to determine their design strategies in using data to improve
student achievement. The extent of the implementation of the data use policy and
strategy in practice was reviewed, and an analysis was completed to determine the
adequacy of the design.
138
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
In summary, case study findings require further discussion and investigation
in Chapter 5. School districts throughout the state are implementing data use design
systems without empirical research literature to support these systems. Case study
analysis of district data use design systems and emergent findings provided by
university cohort case studies groups such as this one, address critical concerns
among educators. Chapter 5 presents a summary, research conclusions, and
implications perpetuated as a result of this study.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER 5
SUMMARY, CONCLUSIONS, AND IMPLICATIONS
A case study was completed at South High School and the East Orange
County School District to determine their design strategies in using data to improve
student achievement. The extent of implementation of data use policy and strategy in
practice was reviewed, and an analysis was completed to determine the adequacy of
the design.
In summary, the East Orange County School District has taken the initiative
to involve teachers in professional development to improve their use of data. Case
study findings reveal that the district design for data use is linked to both the current
and emerging context for assessing student performance. These findings were
revealed through exhaustive data analyses of South High and district interviews,
questionnaires, artifact analysis, quantitative data analyses, data mapping, and
researcher ratings. Furthermore, case study findings suggest the implementation and
extent of the district design is effective primarily at the district level. Site level data
indicate a need to improve communication and site articulation of the district design
for it to be effective. At the classroom level elements of the design are in place;
however, due to inadequate technology infrastructure issues, the design cannot be
fully implemented.
Finally, the adequacy of the district design was researcher rated as to how
competently the district design improves student performance. The district design
was rated “somewhat effective” on the degree to which the district provided student
140
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
data to the school site. This less than optimum rating was primarily due to
technology infrastructure coordination issues at the district and school sites. The
district design was rated “effective” on how adequately high student performance is
developed throughout the school learning community.
Teachers at South High cite positive participation in many staff professional
development activities on data use and the alignment of standards. The research cited
by Noyce, Perda and Traver in 2000, cites how teacher involvement in data use
activities will assist teachers in making decisions about school policy, curriculum,
and instruction.
Teacher leadership team responses indicate that teachers use a variety of
strategies to assess how students are performing at the school and classroom level.
The intended result of the district design provides teachers with multiple strategies
for the analysis and the use of data to increase student performance. This finding
confirms the school research trend on the use of data on student achievement to
inform decision-making (Goertz, 2000). Data can be compiled in different ways to
inform instruction. School districts strive for teachers in their use of data to be data-
driven. Classroom teachers’ use of standardized assessments combined with student
work samples provide valuable information regarding student outputs and decision
making (Goertz, 2000).
The findings of leadership team interviews regarding the district data
design implementation and strategies provide key information for this study. These
data findings suggest that the district design implementation elements and strategies
141
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
provide effective methods for bringing about student achievement gains. However,
current research on school district data use design is limited and cannot be
substantiated by irrefutable research. Therefore, this researcher concludes that this
area requires further investigation for this case study to be conclusive.
Teacher questionnaire responses provide descriptive information cited in the
literature regarding the district data design implementation. Teachers at South High
could not describe the district design, although key elements of the design were
evident. Communication of a comprehensive data use design and integration of this
design appears to be the breakdown in the implementation of the district design at
the teacher-administrator level. Bernhardt cites that "the implementation of a
complete program of data collection at the school level and district level will
improve student performance" (Bernhardt, 1999). Teachers collected, used, and
monitored student performance data at South High. The East Orange County School
District is in the process of developing a more structured approach to assist teachers
in the gathering of quality information about student achievement. This emphasis
requires school districts to devote time, resources, and staff resources to be used in
different ways (Bernhardt, 2000).
Leadership team interviews responses regarding the communication of
standards to the school community provided key information on the adequacy of the
district design. Teacher responses substantiated standards-based research conducted
by Laboratories Network Program in 1998.. The researchers conclude, “teachers
need additional time in aligning assessments to the standards to ensure students
142
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
participate in a rigorous curriculum” (Laboratory Network Program, 1998).
Leadership team teacher responses recognized the need for additional teacher time is
needed to achieve standards-based reform and accountability in the district design.
Teacher interviews indicate that South High does not have the technology to
access student performance at the classroom level. State legislatures are currently
struggling with how best to monitor what matters. Current federal and state programs
do nothing to help assist school districts in the development of consistent data
collection (Christie, 2000).
In summary, researcher findings suggest that the district data use design
implementation process is effective at the site and district levels. Technology
infrastructure issues and communication issues were significant findings and both
require further exploration.
Conclusions
The case study research conducted at South High School and the East Orange
County School District on the use of data to improve student achievement have
paved the way for several key themes to emerge which require further investigation.
These findings verify that schools are under a tremendous amount of pressure
by the public and state legislatures to improve student performance (Pearson, Vyas,
Sensale, & Kim, 2001). As a result, increased school accountability has emerged as
the central reform effort to achieve standards-based goals, while establishing
standards for states, districts, individual schools, and students (Hart & Brownell,
143
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2001). School reform as we know it today, includes a multitude of standardized
assessments which, in turn, produce a great deal of student performance data for
teachers to examine (Bernhardt, 2000).
Case study research findings reveal that teachers on one hand are beginning
to increase their use of data to make decisions about student performance. Teachers
are making substantial progress in the area of student assessment analysis and
standards-based decision-making (Khanna, Trousdale, Penuel, Williams & Kell,
1998). Teachers are learning to use both standardized assessments and classroom
assessments in which to gauge student performance towards attainment of the
standards (Popham, 2000). Additionally, teacher training in the area of the emerging
state standards was a positive finding in this study. Teachers are participating in
professional development activities to align curriculum to State content standards.
On the other hand, case study findings suggest that schools and districts are not
doing enough to close the glaring gap in assessment literacy for teachers and site
leaders (Goertz, 2000). Teachers and site leaders are not currently trained to use
student achievement data, demographic data, and process data to make decisions
i •
about student performance (Bernhardt, 2000).
Case study findings revealed that the decision-making regarding the
development and implementation of the district design in matters regarding student
achievement goals and objectives, student intervention programs, and resource
allocation should be made at the school site level and coordinated with district
support (Levesque, 1996). This finding was clearly evident with teachers at the
144
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
school sites unfamiliarity with the district design. Neither teachers nor site
administrators could describe the district's data use design. Elements of the district
design were evident in many areas of this study, particularly in the area of standards-
based reform efforts. Teachers were aware of the standards, API index and
participated in the alignment of curriculum to the emerging state standards (Duffy,
Goertz & Le Floch, 2001). However, this study revealed that without site input into
the decision-making process, gaps in communication and implementation were
problematic in a district driven data use design system. Researcher findings suggest
that the school site must be actively engaged in the development process of
establishing student achievement objectives, intervention programs, and resource
allocation in the data use design.
Case study research findings revealed that the development of district
technology systems that are flexible and include direct teacher and classroom access
to a variety of student assessment data and that are linked to the emerging state
standards should be the primary focus of the district design (Christie, 2000). School
site databases should have on-line software and resources that allow both student and
classroom assessment tutorials that provide progress towards the attainment of the
standards (Keller, 2001). Flexible technology and infrastructure systems that provide
school sites with standards-based data, standards-based lessons, and site data
management systems have emerged as critical elements for districts to achieve
adequacy in the district data use design.
145
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
In conclusion, key research findings have emerged from this case study
analysis. Further research in the area of how schools are using data to increase
student achievement can provide empirical evidence and substantiate the above
findings. Conclusions derived from this study are primarily qualitative in nature and
require further investigation.
Implications
School districts across the state are struggling with increased student and
teacher accountability by the public and the state legislature. As a result, school
districts are seeking ways to make better use of student achievement data (Goertz,
2000). Case study findings on data use provide significant implications for teachers,
district administrators, principals, school leaders, and policymakers.
Teachers
The research implications from this study and their impact on classroom
teachers are significant. Teachers can benefit from this case study research findings
by effectively using data to improve student assessment and performance in the
classroom. These findings suggest that teachers can utilize data more effectively to
align instruction and provide intervention strategies to meet student needs and
improve student achievement (Frye, Fugerer, Harvey, McKay & Robinson, 1999).
Ongoing teacher training in the area of assessment literacy is needed for teachers to
make better use of the classroom and standardized assessments (Khanna, Trousdale,
146
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Penuel, William & Kell, 1999). Technology that provides teachers with direct access
to standards-based student assessments and on-line resources that support teachers in
the teaching of standards at the classroom level require further investigation.
Furthermore, teachers need to be actively engaged in the development and
implementation of the district design for data use. Teacher participation is necessary
for the success of the district design. Teacher input can provide significant data and
support for the implementation and development of the district design. Further
research is required in the area of teacher participation in the development and
implementation of the district's data use design.
District Administrators
The research implications for district level administrators are significant.
District administrators can benefit from case study research findings in the analysis
o f the district data use design systems to improve student achievement. The analysis
of effective data design systems can provide district level practitioners with quality
indicators for implementing data use designs that are linked to the emerging state
assessment practices (Levesque, 1996). Research findings from this study suggest
that data use design systems developed from a district-driven approach and without
site collaboration cannot reach their full capacity in their use of data. Data use design
development requires coordination of site specific needs and district coordination
and allocation of district resources that includes data management technology,
147
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
assessment literacy training, and curriculum alignment activities aligned to the
emerging state content standards.
Principals
The research implications regarding these research findings for principals in
this study are significant. Principals can benefit from these research findings that will
assist them with the development and implementation of the data use design to
increase student performance in schools. Principals can use these research findings to
provide leadership, resource allocation, staff development training, and district
coordination for schools to increase their use of data more effectively (Khanna,
Trousdale, Penuel, William & Kell, 1999). Case study findings suggest that
principals need to take an active role in leading school sites towards database
decision-making. Additionally, principals are expected to provide site leadership and
communication of data use efforts to improve student performance for teachers,
parents, students, and the community. These findings also suggest that principals will
need to acquire the assessment literacy skills needed for them to facilitate change.
Currently, principals are lacking in assessment knowledge and assessment literacy
skills (Striefer, 2000). Research findings indicate that principals need assessment
literacy skills in the analysis and use of multiple data assessments that include
student achievement data, demographic data, and process data (Bernhardt, 2000).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
School Leaders
The research implications on school leaders at the county and state level can
shift the educational paradigm for school support. School leaders can use these
findings to provide training, resource allocation, and support for data use design
development and implementation directly to the schools. School leaders at the
county and state levels can advocate to state policymakers the resources required to
assist schools in making data-based decisions to improve student achievement.
School leaders can reorganize county and state structures to provide support and
resources to school sites with the highest needs (Odden & Picus, 2000). Currently,
California's Public School Accountability Act (PSSA) is a program that includes the
Immediate Intervention/Under Performing Schools Act (II/USP) that is designed to
provide resources and assistance to low performing schools (Hart & Brownell,
2001). However, this program only addresses the needs for low performing schools
and does not include all schools. School leaders at the county and state levels would
require a paradigm shift of extraordinary efforts to shift available resources to ensure
that all schools are using data to improve student achievement.
Policymakers
The implications of these case study research findings for state policymakers
are significant. Case study research findings can provide assistance to state policy
makers in the use of data to assess schools and link statewide assessment practices to
the emerging state standards. Emerging state assessments that include broader
149
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
assessments such as written or performance based assessments that are linked to
school accountability systems that measure both cognitive and noncognitive
measures are needed (Frey, 2000). Policymakers can use these case study findings to
develop policy and legislation to align funding and resource allocation to ensure that
schools are using data to make informed decisions regarding student achievement.
To conclude, the implications derived from case study findings such as this
one, can have a tremendous effect on public education. These research findings can
have a far-reaching impact on education in five areas: first, the use of data in schools
for improving student and teacher classroom performance, second, the development
and collaboration of district data use designs structures by districts and school sites,
third, future principal leadership skills and knowledge required to implement data
use design systems, fourth, State and county school leader support, resource
allocation and assistance for all schools in the use of data, and finally, the paradigm
shifts for state policymakers to link state assessments to both cognitive and
noncognitive school accountability systems of reform.
Purpose of the Study
The purpose of this study is to investigate the district design for data use in
schools and to determine how the design is linked to the current and emerging state
context for assessing student achievement. The extent of implementation of the
district design also will be investigated. In addition, this study will analyze the
150
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
adequacy of the district design in regards to data use for improving student
performance.
Overview
Three researchers developed research questions guided the investigation of
this study. The three research questions were: 1) What is the district design for data
use regarding student performance, and how is that design linked to the current and
emerging state context? 2) To what extent has the district data use design actually
been implemented at the district, school and individual level? 3) To what extent is
the district data use design a good one? The importance of this study was to provide
new knowledge for teachers, school leaders, district leaders, and policymakers in the
analysis and use of data to improve student performance in schools.
Research Design
Qualitative analysis has been applied in this study in order to provide a
description of the program. The cohort selected the case study method for it seemed
to be the most appropriate design for in-depth study of instances of phenomena in
their natural context, as well as from the perspective of the participants involved
(Gall, Borg & Gall, 1996). The intent is to identify promising perspectives of data
use for future research in this area. The qualitative study included a researcher-
developed case study guide, designed to provide a conceptual framework for
collection and reporting of data organized by the three research questions.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Research Questions
Consistent with the major purpose of this investigation, the following specific
research questions were posed:
1. What is the district design for using data regarding student performance, and how
is that design linked to the current and the emerging state context for assessing
student performance.
2. To what extent has the district design actually been implemented at the district,
school, and individual level?
3. To what extent is the district design a good one (use of multiple criteria)?
Sample and Populations
District. This research study is limited to one school district and one
comprehensive high school within the district. It is part of a cross-site study among
13 different districts and 15 schools. In order for a school or district to be selected, it
must meet the following specific sampling criteria: the district design for collecting
and using data must be is in place, and the district must use both SAT 9 data (as
measured on the California Academic Performance Index), and use multiple
measures to gauge student achievement. Additionally, the district in the sample must
have an ethnically diverse student population, and not have a majority student
population with either a too high or too low socioeconomic status (SES) base.
The process used for the selection of the district for this case study was to
select a secondary school district that met the specified sampling criteria. Research
152
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
on Orange County schools was conducted using the California Department of
Education Academic Performance Index website and informal interviews with
school leaders in Orange County. A district list was developed and narrowed to a few
schools that met the specified criteria. The district was selected after an in-depth
interview with the Assistant Superintendent regarding criteria for the study. The
selection of East Orange County High School District for this case study was made
because of their effective use of data and their relating it to the current and emerging
state context for assessing student performance. The East Orange County High
School District met the student SES population, Academic Performance Index, and
additional sampling criteria for the case study.
School. South High School was selected after an in-depth interview with
the Principal regarding the criteria. South High offers a wide range of educational
programs for students. School Magnet Programs include Academy of the Arts,
International Baccalaureate Program, Agricultural-Science Academy, Advanced
Placement, AVID, CISCO Networking Academy, and Culinary Academy. Students
graduating from the school attend California State Universities, community colleges,
and trade schools.
South High School has a diverse student population that reflects its
surrounding community. O f the 1,800 students (9th -1 2 th graders), over 50% are
Hispanic, 30% White, 10% Asian, and 8% are African American. According to the
2001 Academic Performance Index School Report, approximately 300 are
socioeconomically disadvantaged.
153
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The school’s Academic Performance Index School Report, shows steady
student achievement growth gains since 1999. Additionally, for the past two years
South High School has qualified for state award moneys. South High has met its API
growth target for two consecutive years. South High API growth was a 21-point API
increase in 1999-2000, and a 13-point API increase in 2000-2001, overall the school
has achieved a 34-point API growth during the past two years. The 2001 Similar
Schools Report ranks South High a 2 out of 10 for schools with similar school
characteristics and student population.
Instrumentation
Thirteen Ed.D. candidates at the University of Southern California
developed the research methodology, instrumentation development and training, and
sampling criteria for this case study investigation. A researcher case study guide was
developed to provide a framework for data collection and reporting. Conceptual
Frameworks A /B and C guiding questions served to focus this case study while
conducting both formal and informal interviews and observations at the district and
site. Over a six-day period, this researcher investigated and observed data use
practices and design systems from one school district and one school site.
Conceptual Frameworks
The Case Study Guide requires that all questions under Conceptual
Frameworks A and B are answered. Conceptual Frameworks A and B address
154
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
research questions #1-2 both the district design and implementation of the district
design. The Researcher Rating and Observation Form provides Conceptual
Framework C for the adequacy of the district design.
Conceptual Framework A
Description of Data Use Policies and Strategies
• Student Performance Assessed in the Context of Current and Emerging
Instruments
® Overview of the Elements of District Design of Data Use to Improve Student
Performance
® District Decisions and Rulings that Support Use of District Design
• Intended Results of Design Plans to Improve Student Performance (District,
School, and Classroom)
• Data Use Policy and Strategy Funding
Conceptual Framework B
Implementation of Data Use Policy and Strategy in Practice
® Degree of Design Implementation
• Implementation of Current Data practices (District, School, and Classroom)
® Implementation of Emerging State Data Practices (District, School Site, and
Classroom)
• Accountability for data use at district, school, and individual level
• Improving Student Achievement through Implementation of Data Use
155
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Data Collection and Instruments
Data collection consisted o f staff and situated interviews with one district
leader, one site principal, six teachers, teacher questionnaires, observations, artifact
analysis, quantitative data analysis, data mapping, and researcher rating. Research
findings were reported using Conceptual Frameworks A/B/C in which to organize
and collect the data (see chart below).
Data Collection
Instruments
RQ1: Design RQ 2:
Implementation
RQ 3:
Adequacy of
Design
Case Study Guide
• Interviews:
District
Administrator, Site
Administrator, 6
Teachers (made up of
grade level /
department leaders
and average teachers)
• Mapping of Data
Flow at District
and School Site
• Artifact Analysis/
Collection
• Quantitative data
X X X
Situated Interviews
• Forming vignettes
with 6 teachers
X
Teacher
Questionnaires
X
Stages of Concern
Questionnaires
X
Researcher Rating
Form (Post Data
X
156
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Collection)
Innovation
Configuration (Post
Data Collection)
X
Main Findings
Based on an analysis of the research data, the following findings were
discovered:
1. Teachers are currently using a variety of student assessments both classroom
based assessments and standardized assessments to assess student performance
and progress towards state and district content standards.
2. Professional development in the area of assessment literacy training and data
based decision-making for teachers and administrators is needed at the school
site level.
3. Decision-making regarding standards-based student achievement goals and
objectives, as well as student intervention strategies and programs, and resource
allocation require decision-making and communication at the school site level
with district support for implementation.
4. Development of district technology support systems that are flexible and that
provide direct teacher and classroom access to on-line resources for multiple
student and teacher assessments that are linked directly to the emerging standards
should be the primary focus of the district data use design system.
157
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Implications
The implications derived from case study findings such as this one, can have
a tremendous effect on public education. These research findings can have a far-
reaching impact on education in five areas: first, the use of data in schools for
improving student and teacher classroom performance, second, the development and
collaboration of district data use designs structures by districts and school sites, third,
future teacher and principal leadership skills and knowledge required to implement
data use design systems, fourth, State and county school leader support, resource
allocation and assistance for all schools in the use of data, and finally, the paradigm
shifts for state policymakers to link state assessments to both cognitive and
noncognitive school accountability systems of reform.
Recommendations for Additional Future Research
Investigate the Types of Assessment Data Used to Improve Student Performance
Case study research findings reveal that teachers are beginning to increase
their use of data to make decisions about student performance. Teachers are making
substantial progress in the area of student assessment analysis and standards-based
decision-making (Khanna, Trousdale, Penuel, Williams & Kell, 1998).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Investigate Methods to Increase Assessment Literacy in Schools
Case study findings suggest that schools and districts are not doing enough to
close the glaring gap in assessment literacy for teachers and site leaders (Goertz,
2000). Teachers and site leaders are not currently trained to use student achievement
data, demographic data, and process data to make decisions about student
performance (Bernhardt, 2000).
Investigate Best Practices for Site Implementation of District Design
Case study findings revealed that the decision-making regarding the
development and implementation of the district design in matters regarding student
achievement goals and objectives, student intervention programs, and resource
allocation should be made at the school site level and coordinated with district
support (Levesque, 1996).
Investigate Standards-based Technology Systems
Case study research findings revealed that the development of district
technology systems that are flexible and include direct teacher and classroom access
to a variety of student assessment data and that are linked to the emerging state
standards should be the primary focus of the district design (Christie, 2000). School
site databases should have on-line software and resources that allow both student and
classroom assessment tutorials that provide progress towards the attainment of the
standards (Keller, 2001).
159
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
BIBLIOGRAPHY
Achieving the Goals of Standards Based Reform. (2001). Available:
www .AFT.org/edissues/standards/standardsreformers. htm
Alpert, S. (1999). Senate Bill IX: Public School Accountability Act 1999.
Sacramento, CA: Author.
Baker, E., Brown R. & Herman, J. (2000). Student assessment and student
achievement in the California Public School System, CSE Technical Report
519, Center for the Study of Evaluation National Center for Research on
Evaluation Standards and Student Testing.
Barton, P. (2001). Executive Summary, Raising Achievement and Reducing Gaps:
Reporting Progress Toward Goal Achievement, NAEP Executive Summary
1990-1998.
Bernhardt, V. (2000, Winter). Intersections: How crossing data can help you piece
together a clear picture of your school. Journal of Staff Development. 21(1):
32-37.
Blank, R. & Wilson, L. (2001). Understanding NAEP and TIMMS results: Three
types of analyses useful to educators. ERS Spectrum. 1911): 23-33.
Bodily, B. (1998). Lessons from New American School’s Scale-Up Phase:
Prospects for Bringing Designs to Multiple Sights. Santa Monica , CA: The
RAND Corporation.
Boese, L., Burkhardt, R., Carstens, L., Devine, M., Gaffney, T. & Just, A. (2001).
Public School Accountability (1999-2000) Immediate
Intervention/Underperforming Schools Program (II/USPJ: How Low
Performing Schools in California Are Facing the Challenge of Improving
Student Achievement. Sacramento. CA. California Department of Education,
Division of Policy and Evaluation.
Brownell, S. & Hart, G. (2001). An era of accountability in California. The Clearing
House. 74141: 183-186.
Burr, E., Hayward, G., Fuller, B. & Kirst, M. (2000). Crucial issues in California
education 2000: Are the reform pieces fitting together?
160
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
California State Department of Education Publication. (2000). Appreciation and
Update on Standards. Assessments, the API, the California Reading List, and
the State Budget. Sacramento, CA. California Department of Education,
Division of Policy and Evaluation.
Calvo, N., Carlos, L., Guth, G., Hayward, G., Holtzman, D. & Schneider, S. (1999).
Evaluation of California's Standards Based Accountability System: Final
Report November 1999. WestEd, California Department of Education.
Charles A. Dana Center, University of Texas at Austin. (1999). Hope for Urban
Education: A Study ofNine High Performing, High-Poverty, Urban
Elementary Schools. Washington, D.C.: U.S. Department of Education,
Planning and Evaluation Services.
Charles A. Dana Center. (2000, September). Equity-driven achievement-focused
school districts: a report on systemic school success in four Texas school
districts serving diverse student populations. University of Texas at Austin:
Author.
Christie, K. (2000, September). Monitoring what matters. Phi Delta Kappan. 82(11:
5-6.
Clune, W. (1998). The shift from equity to adequacy in school finance. Educational
Policy. 8141: 376-394.
Consortium for Policy Research in Education (CPRE). (2000). Assessment and
accountability in the fifty states: 1999-2000.
Conzemius, A. (2000, Winter). Framework: System Builds Change Efforts Beyond
Hopes, Hunches, Guesses. Journal of Staff Development. 21(11: 38-41.
Davis, D., Hagans, R. & Sagmiller, K. (1999). Implementing School Reform
Models: The Clover Park Experience. Northwest Regional Educational
Laboratory: United States Department of Education, Office of Elementary
and Secondary Education.
Doherty, K. Assessment (2001). Hot Topics: Assessment. Education Week.
Duffy, M., Goertz, M. & Le Floch, K. (2001). Assessment and Accountability
Systems in the 50 States: 1999-2000. Consortium for Policy Research in
Education.
161
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Education Commission of the States. (1996, November). The progress of educational
reform: 1996 Denver, CO. (Eric Document Reproduction Service No. ED
402 653).
Executive Summary. (2001). Seeking stability for standards-based reform. Education
Week on the Web (On Line serial), www.edweek.org/sreportsqc01
Exploring alternatives for school-based funding. (1996). Available:
www.nces.ed.gov/pubs98/finance9817-6.html#6-l
Fierros, E. et al. (1997). Using multiple methods of assessment to promote district
level reflection about instructional improvement. Paper presented at the
annual meeting of the American Educational Research Association, Chicago,
IL.
Fowler, W. (1997). Selected Papers in School Finance. National Center for
Education Statistics, Washington, D.C.
Frey, S. (2000). National accountability movement offers lessons for California.
Palo Alto, CA: EdSource.
Frye, B., Fugerer, B., Harvey, S., McKay, L. & Robinson, O. (1999). SAM: A
student achievement model designed to empower teachers and increase
student achievement through action research. ERIC Document Reproduction
Service No. ED 430 015.
Gall, M., Borg, W. & Gall, J. (1996). Educational Research : An Intoduction (6th
edition). New York, N.Y: Longman Publishers.
Gillbom, D., & Yudell D. (2000). Rationing Education. Philadelphia: Open
University Press.
Goertz, M. (1997). The challenges of collecting school-based data. Journal of
Education Finance. 22: 291-302.
Goertz, M. (2001). California's new academic standards take hold. EdSource:
December, 2001.
Goertz, M. (2000). Local accountability: the role of the district and school in
monitoring policy, practice and achievement. EdSource: November, 2000.
Grissmer, D. & Ross, M. (2000). Analytic issues in the assessment of student
achievement tNCES Publication No. 2000-050). Jessup, MD. (ERIC
Document Reproduction Service No. ED 443 890).
162
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Guth, C., Holtzman, D., Schneider, S., Carols L., Smith, J., Hayward, G. & Calvo, N.
(1999). Evaluation of California's Standards-Based Accountability System.
Final Report. West Ed, San Francisco, CA.
Guthrie, J. (1988). Understanding School Budgets. Washington D.C.: U.S.
Department of Education.
Haertel, E. (1999). Performance assessment and education reform. Phi Delta Kappan.
80(9): 662-666.
Hart, G. & Brownell, N. (2001). An era of educational accountability in California.
The Clearing House. 74(4): 183-186.
Heck, R. (2000). Examining the impact of school quality on school outcomes and
improvement: A value-added approach. Educational Administration
Quarterly. 3614): 512-552.
Herman, J. & Winter, L. (1992). Tracking your school’s success: A guide to sensible
evaluation. Los Angeles Center for Research on Evaluation, Standards and
Student Testing, University of California.
Hoff, D. (2000). Business group urges sticking with standards. Education Week.
20(24): 10.
Hoffman, J., Czop Assaf, L. & Paris, S. (2001). High-stakes testing in reading: today
in Texas, tomorrow? The Reading Teacher. 54: 482-492.
Holcomb, E. (1999). Getting excited about data: How to combine people, passion,
and proof. Thousand Oaks, CA: Corwin Press.
Johnson, E. & Siegendorf, A. (1998). U.S. Department of Education. National Center
for Educational Statistics. Linking the National Assessment of Educational
Progress (NAEP) and the Third International Mathematics and Science Study
(TIMSS): Eighth-Grade Results, NCES 98-500, Washington,
Johnson, J. (1997). Data driven school improvement. ERIC Clearinghouse on
Educational Management Digest #109. University of Oregon, Eugene,
Oregon.
Johnston, R. (2001). Test scores up in urban districts, report says. Education Week.
38(20): 3.
163
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Jones, R. (2000, September). What researchers say will make standards work. The
American School Board Journal. 187: 13-19.
Keller, B. (2001). Test-score gains in Texas traced to policies on minority progress.
Education Week. 20138): 12.
Khanna, R., Trousdale, D., Penuel, W. & Kell, J. (1999). Supporting data use among
administrators: Results from a data planning model. Paper presented at the
annual meeting of the American Educational Research Association,
Montreal, Quebec, Canada.
Killion, J. & Bellamy, G. (2000, Winter). On the job: data analysts focus school
improvement efforts. Journal of Staff Development. 21(1).
Laboratory Network Program.(1998). Taking stock of states’ curriculum-based
reform efforts. An interim report of the Laboratory Network Program’s
Curriculum. Learning, and Instruction Project. Aurora. CO.: Midcontinent
Regional Educational Laboratory, 8-19.
Larson, K., Guidera, A, Nelson, S., Rodstag, M. & Smith, N. (1998). The formula
for success: A business leader's guide to supporting math and science
achievement (Publication No. SAI-98-3028). Washington, D.C. (ERIC
Document Reproduction Service No. ED 419 914).
Levesque, K. (1996). Using data for program improvement: how do we encourage
schools to do it? National Center for Research in Vocational Education. 12:
1 - 6 .
Levine, R., Rathbun, A., Seldon, R. & Davis, A. (1998). NAEP's constituents: What
do they want? Report of the NAEP Constituents' Survey and Focus Groups.
Washington, DC: National Center for Education Statistics.
Madeaus, G. & O ’Dweyer, L. (1999). A short history of performance assessment:
Lessons learned. Phi Delata Kappan. 80191: 688-695.
Making standards matter 1999: Executive summary. (1999). Available:
www.AFT.org.edissues/standards99/Index.htm
Marx, G., Hunter, D. & Johnson, C. (1997). Increasing student achievement: an
urban district's search for success. Urban Education.31f33: 529-544.
Maxson, S., & Schwartz, D. (2001). School-university collaboration for reform in
California: the DELTA project: The Clearing House. 74: 251-256.
164
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Mayer, D., Mullens, J., & Moore, M. (2001). Monitoring School Quality: An
Indicators Report. Education Statistics Quarterly. 3: 38-44.
Merrow, J. (2001, May) Undermining standards. Phi Delta Kappan. 653-59.
National Association State Board of Education (1998). Standards based efforts and
progress in schools. NASB. 4 : 36-42.
National Center for Educational Statistics, U.S. Department of Education. (2000).
Highlights from Third International Mathematics and Science Study-Repeat
(TIMSS-R). Washington D.C.: U.S. Government Printing Office.
National Center for Educational Statistics, U.S. Department of Education. (2000).
Pursuing excellence : Comparisons of International Eighth-Grade
Mathematics and Science Achievement from a U.S. Perspective, 1995 and
1999. Washington D.C.: Government Printing Office. NCES 2001-028.
National Education Association. (2000). Data-driven decision-making and student
achievement. Available: www.nea.org/issues/high-stakes/dddm.html [July,
2000],
National Educational Tests, Congressional Digest (1997, November). Setting
standards and measuring achievement. 257.
Nave, B. (2000). Linking standards-based reform with student achievement. Phi
Delta Kappan. 82(2): 128-132.
Noyce, P., Perda, D. & Traver, R. (2000) Creating data-driven schools. Educational
Leadership 57(51: 52-6. The Association for Supervision and Curriculum
Development, VA.
O' Donovan, E. (2001). Dealing with information overload. Phi Delta Kappan
21(6): 9-11.
Odden, A. (1998, September). Creating school finance policies that facilitate new
goals. Consortium for Policy Research in Education. 26: 1-6.
Odden, A. (1999). Making better use of resources for educational reform. Marsh
(Ed.), 1999 ASCD Yearbook: Preparing our schools for the 21st century.
Alexandria, VA: Association for Supervision and Curriculum Development.
Odden, A. (2000, February). The costs of sustaining educational change through
comprehensive School reform. Phi Delta Kappan 81(6): 433-438.
165
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Odden, A. & Picus L. (2000). School Finance A Policy Perspective (second edition').
Dubuque,IA: McGraw-Hill Companies.
Olsen, L. (2001, March). Holding schools accountable for equity. Leadership. 28-
31.
Patton, M. (1990). Qualitative evaluation and research methods. N ewberry Park, CA:
Sage Publications.
Pearson, D., Vyas, S., Sensale, L., & Kim, Y. (2001). Making our way through the
assessment and accountability maze: where do we go now? The Clearing
House. 74: 175-182.
Penuel, W.R., Kell, J.G., Frost, J., & Khanna, R. (1998). School administrators
reasoning about data: An action research perspective. Paper presented at the
Annual Meeting of the American Educational Research Association,
Montreal, Quebec, Canada, April 19-23,1999. 20 pages.
Popham, W. J. (2000). The Mismeasurement of Educational Quality. The School
Administrator. 12-15.
Popham, W.J. (2001). Whittling Wish-List Standards Down to Reality. National
Association of Secondary School Principals, 84. 30-36.
Public Policy Institute of California Publication (2001, March). School Resources
student outcomes. Leadership. 20-22.
Ransom, K., Minnick Santa, C., Williams, C. & Farstrup A. (1999). High-stakes
assessments in reading: A position statement o f the international reading
association. Journal of Adolescent and Adult Literacy, 43: 305-312.
Reeves, D. (2000). The 90/90/90 schools: a case study. Accountability in action: a
blueprint for learning organizations. Citibank Resource Bank: Center for
Performance Assessment, Chapter 19,185-196.
Reeves, D. (2001). 101 questions and answers about standards assessment and
accountability. Denver, CO.: Advanced Learning Press.
Robinson, V. & Timperley, H. (2000). The link between accountability and
improvement: The case of reporting to parents. Peabody Journal of
Education. 75(4): 66-89.
166
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Roelike, C. (1997, March 24-28). The process of allocating Human Resources in the
era of educational reform: The practitioner’s perspective. Paper presented at
the Annual Meeting of the American Educational Research Association,
Chicago, Illinois, 150 pages.
Rumberger, W. & Gandara, P. (2000). The schooling of English Learners. Crucial
Issues in California. PACE 23-41.
Sandham, J. (2001). Driven by Data [4 pages]. Education Week on the Web [On-
Line serial]. Available www.edweek.org/sreports/qc01.
Scherer, M. (2000). Standardized instruction: Effects may vary. Journal of the
Association for Supervision and Curriculum. 58111: 5.
Scheurich, J., Skrla, L. & Johnson, J. (2000). Thinking carefully about equity and
accountability. Phi Delta Kappan. 82141: 293-299.
Schmoker, M. (1999). Results: The key to continuous school improvement 2nd ed.).
Alexandria, VA: Association for Supervision and Curriculum Development.
Seeking stability for standards-based reform (2001). Executive Summary [5 pages].
Education Week on the Web [On-Line serial]. Available
www.edweek. org/sreports/q cO 1
Sirtnik, K. & Kimball, K. (1999). Standards for standards-based accountability
systems. Phi Delta Kappan. 8113V . 209-214.
Stiggens, R. (2001). The principal’s role in assessment. National Association of
Secondary School Principals. 851621: 13-26.
Striefer, P. (2000). School improvement: Finding the time. National Association of
Secondary School Principals. 612(84): 66-71.
The National Education Goals Panel. (2001). Promising practices: Progress toward
the Goals. 2000. Available www.negp.gov: Author.
The National Education Goals Panel. (2001). Raising achievement and reducing
gaps: Reporting progress toward goals for academic achievement. Available
www.negp.gov: Author.
Thompson, S. (2001). As the standards turn: The evil twin o f authentic standards. Phi
Delta Kappan. 82: 358-362.
167
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Tucker, M. & Codding, J. (1998). Standards for our schools: How to set, them.
measure them, and reach them. San Francisco, CA: Jossey Bass Publishers.
U.S. Department of Education. (1997). National Center on Education and the
Economy. Standards Driven Curriculum: Washington D.C.
Yinger, R. & Hendricks-Lee, M. (2000). The language of standards and teacher
education reform. Education Policy. 14(1): 94-106.
Zach, T., Zon, C. & Ackley, D. (1998). Standards for improvement. Thrust for
Educational Leadership. 27: 14-18.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDICES
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX A
Case Study Guide
Research Question #1: What is the district design for using data regarding student performance,
and how is that design linked to the current and emerging state context for assessing student
performance?
A. District Interview Guide- interview of district leader description of intended results of data
use policies.
® Researcher Rating Form- researcher rates the importance of each intended result.
B. District Interview Guide - district leader description of data use
• Researcher Rating Form- researcher rates the extent of use of each policy.
C. District Interview Guide- description of key elements of the district design.
• Researcher Rating Form- researcher rates the importance of each element.
D. District Interview Guide- description of the district data use design tied to the emerging state
context for assessing student performance.
Research Question #2: To what extent has the district design actually been implemented at the
district, school, and individual teacher level? __________________________________________
A. How does it work? Description of practice/vignettes of practice (current vs. emerging)
• What does data use to improve student performance look like at the district, school, and
classroom level?
• Interview Guides— district, school leadership, and teacher
® Observation Guide- selected teachers
• Researcher Rating Guide- document review
• Self-Report— of district leaders, school leaders, and teachers
B. What does the best practice look like?
• To what extent does actual use fit with a model fully implemented?
Mapping out extent of implementation
Teacher Questionnaire
• What is the nature of concerns and extent regarding the implementation of the district design
for data use?
Stages o f Concern (How am I Feeling?)
Question #3: To what extent is the district design a good one?
A. Use of Multiple Criteria (to examine design adequacy)
• Description of good design
• Description of the type o f district
• Describe the current practices based on norm-referenced assessment
• Describe the emerging practices moving towards criterion-referenced assessment
® Describe what are the criterion-referenced assessment tools being utilized currently/planned
• Describe collaboration
• Rate the district design
® Rate the site's implementation of the district design
170
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX B
Conceptual Framework A
Description of Data Use Policies and Strategies: The Design
This conceptual framework addresses research question 1: What is the district design
for Using data regarding student performance, and how that design linked to the
current and the emerging state context for assessing student performance?
1. Student Performance Assessed in the Context of Current and Emerging
Instruments
Current
• Are the district and school focusing on improving Stanford 9 scores and API
ratings as defined by the State?
® Are the district and school using the California Content Standards to improve
student performance?
• Are authentic assessments linked to state standards throughout the district,
school, and classrooms?
® Are norm-referenced assessments used?
a Are interventions with students linked to performance assessments?
® Is there awareness on the district and school levels of the Stanford 9, API scores,
etc.?
• Is state-of-the-art technology used to address current and emerging student
assessments on the state, district, and school levels?
Emerging
® Is there awareness on the district and school levels of emerging state
assessments?
• Is there preparation for the High School Exit Exam (Sr. High only)?
• Is there preparation for the California English Language Test (CELDT)?
• Is there evidence of district planning to prepare students for emerging
assessments?
• Are there examples of the use of criteria-referenced tests on the district and
school levels?
® Is there measurement of student performance against international performance
standards?
171
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2. Overview of the Elements of District Design of Data Use to Improve Student
Performance
• What types of data are collected in the district design of data use?
® What is the timeline for the receipt and use of data in the district?
• What assessment instruments is the district using the collect data?
• What is the rationale for collecting these instruments?
• What methods are used to analyze these data?
• What is the rationale for selecting these methods of analysis?
® Did reputable research guide the district's design of data use?
• What methods does the district have for disseminating data?
® What training does the district provide for improving/modifying data?
• Are multiple stakeholders involved in the decision-making process in the
district?
® What steps do district and schools take to use data to improve student
performance?
• What outside influences affect policy and strategy design?
® What inside influences affect policy and strategy design?
• What role does fiscal and funding issues play in the design of the policies and
strategies?
• What state legislation was considered in the design?
® What current data practices are included in the policy?
® What emerging data practices are included in the policy?
3. District Decisions and Rulings that Support Use of District Design
• What board rulings directly support the district design?
• To what degree are multiple stakeholders involved in establishing/influencing
Boards?
• Are research studies guiding the board support of the district design?
• Does the board consider State legislation?
• What is the timeline to implement the district design?
• Is there board mandated training for the district design of the use of data?
• What is the process for establishing district-wide high-performance
goals?
• Is there money allotted to develop and implement the district design?
• Is technology designated be used in the implementation of the design?
• Are outside sources used to implement the design?
1. Intended Results of Design Plans to Improve Student Performance (District,
and Classroom)
172
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
District
• What are the intended results of district design?
® Does the district want to use data to increase student performance, and is there an
Increased focus on improving student performance?
• Was there a process created to establish district-wide high-performance goals?
• Does the district support schools' efforts to improve student performance by
Providing student data?
• Does the district use multiple measures to increase performance and guide
instruction?
• Does the district provide schools with the results of these multiple measures?
School
® What is the school doing to use data (directly or indirectly) to promote student
learning?
• Are there any schoolwide plans/ efforts to improve how data are gathered,
analyzed, and used?
® Does the school try to improve administrator's and teachers' ability to analyze
data effectively to guide instruction?
• Is the school trying to raise student performance expectations with teachers and
parents?
® Does the school administrator(s) support school-wide implementation of
standards-based curriculum to improve student performance?
« Does the school regularly inform the students, parents, and community of student
performance?
• Was the effectiveness of instructional programs evaluated?
• Were there any roadblocks or challenges to making these improvements?
Classroom
® Are teachers able to effectively analyze performance data and other forms of
data?
® Are schools training teachers to improve their ability to use data?
• Are administrators and teacher using standards and rubrics to improve student
performance? Are teachers executing high-level standards-based curriculum?
• Are classroom teachers using data to evaluate the effectiveness of instructional
programs?
• Are teachers placing a high priority on improving student performance?
® Are students expected to raise their ability and achieve high performance goals?
® Are classroom teachers regularly informing students, parents and community of
student learning performance and how they can assist student improvement?
173
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
5. Data Use Policy and Strategy Funding (District, and School Site)
• How is data use implementation funded?
• Does the district control the budgeting and funding of the programs and
strategies, or do schools have a role?
• Do all grade levels and subject areas receive equal shares of funding?
® How is the funding affecting curricular and instructional decisions?
• Where is additional funding needed?
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
A PPEN D IX C
Conceptual Framework B
Implementation of Data Use Policy and Strategy in Practice
This conceptual framework addresses research question 2: To what extent has the
district design actually been implemented at the district, school, and individual
teacher level?
1. Degree of Design Implementation fin the current and emerging contexts)
• To what degree does the school have knowledge of the district design?
• Were the district policies and strategies fully implemented at the school?
• How long did this implementation take? Was it phased in? What was the
timeline?
® Who was given the job of implementation at the site level?
• What factors were considered important in the implementation?
® How comfortable is the school with the district design?
• What outside influences affected implementation?
® What roadblocks were identified in implementation?
® Was technology used to implement the design?
• Were outside sources used to implement the design?
® Was there a timeline for the receipt and use of data from the district or state?
• What data assessment instruments are used at the school site?
• How supportive is the principal (and other site administrators) of the district
design?
® Are school in-services being offered on how to use data effectively?
® Can the school site link implementation of the district design with improved
student performance?
® Does the school have a method for communicating standards-based curriculum
and assessments to students, parents, and community?
® What data was collected on student performance?
• Are current data practices implemented?
• Are emerging data practices being implemented at the school?
• Is the district plan implementing emerging data practices: criteria-referenced
tests, performance assessments, international comparisons, High School Exit,
etc.?
2. Accountability for data use a district, school, and individual level
• How are the district policies and strategies monitored for effectiveness?
• Where is the data stored?
175
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• How is the data disaggregated?
• What is done with the data?
• Who is accountable for the implementation of the district's plan to improve
student achievement through the use of data at the school site?
• What responsibility do classroom teachers have for linking instructional practice
to data use?
• What responsibility do students have to be accountable for their achievement?
3. Improving Student Achievement through Implementation of Data Use
• Does the implementation of data use show any student achievement gains at the
district level and school site level?
« What assessments were used?
• What elements of data use are the most effective in bringing about student
achievement gains?
• How is data used in the classroom to improve student performance?
• Do classroom teachers keep track of students' growth? How?
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX D
Situated Interview Questions
Degree of design implementation of current data practices
1. Tell me your impression of how aware staff members are of the district design for
using data.
• Talk about a teacher who exhibits a high level of awareness.
® Talk about a teacher who has a low level of awareness
2. Can you give me an example of how data has been used in your classroom?
® What kinds of data have been used?
• Where does this data fdter down to you from (i.e., Who sees it before you? After
you?)
3.Tell me how data has been used to improve your instructional practices.
® What was the problem?
® What was the relevant data?
® What was the eventual solution/outcome of the problem?
Degree of design implementation of emerging state data practices
1. Tell me about any professional development training you’ve been involved in to
learn about emerging state practices.
® Who organized the training?
• Who attended the training?
• What policies were discussed?
• Was design or implementation the focus of the training?
2. Tell me about any professional development training you feel you would like to
have received but haven’t.
• Is there a cohort of teachers interested in this same topic?
3. What other existing avenues outside of formal training have you been involved
with?
• Teacher-teacher informal discussions
• Teacher-teacher formal meetings
177
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• Beginning teacher to experienced teacher informal discussions/formal meetings
4. In what ways has the school provided time for you to analyze data?
• Time release
• Professional development/non-teaching days
Accountability for data use at district, school, and individual level
1. How has your accountability been affected by the use of data?
• Promotion within the school
® Salary-dependence
® Required reporting to subject coordinators
Improving student achievement through implementation of date use
1. Tell me about a student with low performance whose performance improved
through the use of data.
• Who formulated an intervention plan based on the data?
• How long did it take from the time the data was analyzed to the time
improvement was seen?
2. In what ways is data effective in motivating student achievement?
• “fear factor:” (of retention, of parental dissatisfaction, etc.)
• in-class competition
® internal motivation to improve
178
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX E
Data Mapping Questions
® How does the State of California disseminate/ record data from school districts
and individual schools (specifically look for mode, pathway, and recording
mechanisms)? You may want to look at API scores, SAT 9, HESEE,
Performance Assessments, etc.
• How does the school district analyze the data and disseminate it to the local
school sites? Does the district do research based on this data?
• What system of checks and balances are in place to see if this is carried out
effectively (if any)?
• How does the district level implement new state data mandates (i.e., High School
Exit Exam)? What is the procedure and who is involved?
• Is any training conducted throughout the district that involves data on the
district/local school site level?
• Can you identify any innovative practices with data (ie: emerging practices)?
• What occurs with data use at the local school site level? Who is in charge of
disseminating the data? How is it used at the local school site?
• What training about data use is conducted at the local school site?
• What emerging data practices do you see present at the local school site?
• What part do department/grade level chairs play with use of data?
• How is the average classroom teacher using data?
• Are data used in the regular classroom to drive instruction?
• Are students involved in the data analysis process?
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX F
Artifact Analysis
While informally observing and interviewing district and school staff, the
researcher collects the following evidence:
• State Reports (State of California Department of Education webpage, STAR
reports, API, etc.)
• District Records (Research Department, Superintendent's office, etc.)
• School Records (review files with Office Manager, testing coordinators, etc.)
® District Correspondence (Bulletins, Memos, Addendum's, Handbooks, School
Data Systems (i.e., SIS)
• School Correspondence (Bulletins, Memos, etc.)
• District Publications (District Newsletters, Newspaper Articles, Training
Manuals, etc.)
• School Publications (Monthly Newsletter, PTA Bulletins, School Handbook,
etc.)
• Classroom Publications (Parent Newsletters, Back to School Night Agendas,
Classroom Newspapers, etc.)
• Teacher Management Systems (goal setting charts, portfolios, rubrics, lesson
plans, etc.)
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX G
Quantitative Data Collection Chart
Data (Current) Collected Data (Emerging) Collected
API Reports
(1999-2001)
X Performance
Assessments
(Multiple Years)
X
SAT-9
(199-2001)
X ELA scores (math,
science, S.S.)
X
STAR/
CST(STAR
Augmentation)
X HSEE
CELDT
(California English
Language
Development)
X
Other Other
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX H
Teacher Questionnaire
We would appreciate it if you would provide the following demographic
data for purposes of the study only. Again, complete confidentiality will
be maintained.
Credential(s)
(Indicate if it is an emergency
credential)
Years of Experience
Years in current position
Grade level(s) currently
teaching
Courses Currently Teaching
(By department only)
Gender
Ethnicity
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Don’t
Know
0
Disagree
Strongly
1
Disagree
Somewhat
2
Agree
Somewhat
3
Agree
Strongly
4
Degree of design implementation of current data practices
1 .1 am aware of the design
for using data.
0 1 2 3 4
2 .1 use data in my classes
on a weekly basis.
0 1 2 3 4
3 .1 collect data on a
weekly basis.
0 1 2 3 4
4 .1 use data to monitor
student progress.
0 1 2 3 4
5 . 1 use data to guide my
instruction.
0 1 2 3 4
6 .1 use data to improve
student outcomes.
0 1 2 3 4
7 .1 collect data on test
scores.
0 1 2 3 4
8 .1 collect data on class
participation.
0 1 2 3 4
9 .1 collect data on student
attitudes.
0 1 2 3 4
10. I collect data. 0 1 2 3 4
11. My department head
collects data.
0 1 2 3 4
12.1 use data to compare
the past and present
performance of an
individual student.
0 1 2 3 4
13.1 use data to compare
students within my class.
0 1 2 3 4
14.1 use data to compare
students across the school
in the same grade.
0 1 2 3 4
15 . Reports are sent to
parents on a regular basis
(about once a month).
0 1 2 3 4
16. The school completes
reports of data
implementation for district
databases.
0 1 2 3 4
183
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Don’t
Know
Disagree
Strongly
Disagree
Somewhat
Agree
Somewhat
Agree
Strongly
Degree of design im plem entation of emerging state data practices
17. The school offers
frequent professional
development to raise
awareness of new data
practices.
0 1 2 3 4
18.1 have attended
professional
development training in
the past six months
related to new data
practices.
0 1 2 3 4
1 9 .1 frequently discuss
new data practices with
teachers who are about
as experienced as I.
0 1 2 3 4
2 0 .1 frequently discuss
new data practices with
teachers who are more or
less experienced than I
(mentor/ mentee format).
0 1 2 3 4
2 1 .1 frequently discuss
data practices with
teachers in different
disciplines from mine
0 1 2 3 4
22. School
administrators have
assisted me in
implementing new data
practices.
0 1 2 3 4
23. School
administrators have
monitored my utilization
of new data practices.
0 1 2 3 4
184
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX I
Use of Data in School Study
Stages of Concern, Question #1
(Teachers)
Name (optional)______________________ _______ ______
In order to identify these data, please give us the last four digits of your Social Security
number:______________________
This is a questionnaire about the district’s design to use student data to improve student
performance. The purpose of this questionnaire is to determine what teachers who are using or
thinking about using the district’s design to use data to improve student learning are concerned about
at various times during the innovation adoption process. A good part o f the items on this
questionnaire may appear to be of little relevance or irrelevant to von at this time. For the completely
irrelevant items, please circle “O” on the scale. Other items will represent those concerns you do
have, in varying degrees of intensity, and should be marked higher on the scale.
Please respond to the items in terms o f vour present concerns, or how you feel about your
involvement or potential involvement with the district’s design to use data to improve student
learning. We do not hold to any one definition of this innovation, so please think of it in terms of
vour perception of what it involves. Remember to respond to each item in terms of vour present
concerns about your involvement or potential involvement with the district’s design to use data to
improve student learning.
Thank you for taking time to complete this questionnaire.
Please circle the number that best reflects your response to each statement based on the following
rating scale:
0 1 2 3 4 5 6 7
Irrelevant Not true for me Somewhat true for me Very true for me now
1 .1 am concerned about student attitudes toward the district’s design to use student data
to improve student performance.
0 1 2 3 4 5 6 7
2 .1 now know of some other approaches that might work better than the district’s
design to use student data to improve student performance.
0 1 2 3 4 5 6 7
3. I don’t even know what the district’s design to use student data to improve student
performance is.
0 1 2 3 4 5 6 7
185
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
4 .1 am concerned about not having enough time to organize myself each day
because of the district’s design to use student data to improve student performance.
0 1 2 3 4 5 6 7
5.1 would like to help other faculty in their use of the district’s design to use student data
to improve student performance.
0 1 2 3 4 5 6 7
6 .1 have a very limited knowledge about the district’s design to use student data to
improve student performance.
0 1 2 3 4 5 6 7
7 .1 would like to know how the implementation of the district’s design to use student
data to improve student performance would affect my classroom, my position at my
school and my future professional status.
0 1 2 3 4 5 6 7
8 .1 am concerned about conflict between my interests and responsibilities with
respect to implementation of the district’s design to use student data to improve student
performance.
0 1 2 3 4 5 6 7
9 .1 am concerned about revising my use of the district’s design to use student data to
improve student performance.
0 1 2 3 4 5 6 7
10.1 would like to develop working relationships with both our faculty and outside
faculty while implementing the district’s design to use student data to improve student
performance.
0 1 2 3 4 5 6 7
11. I am concerned about how the district’s design to use student data to improve student
performance affects students. ;
0 1 2 3 4 5 6 7
186
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
12. I am not concerned about the district’s design to use student data to improve student
performance.
0 1 2 3 4 5 6 7
13.1 would like to know who will make the decisions in the district’s new design to use
student data to improve student performance.
0 1 2 3 4 5 6 7
14.1 would like to discuss the possibility of using the district’s design to use student data
to improve student performance.
0 1 2 3 4 5 6 7
15.1 would like to know what resources are available to assist us in implementing
the district’s design to use student data to improve student performance.
0 1 2 3 4 5 6 7
1 6.1 am concerned about my inability to manage all that is required by the district’s
design to use student data to improve student performance.
0 1 2 3 4 5 6 7
1 7.1 would like to know how my teaching or administration is supposed to change
with the implementation of the district’s design to use student data to improve student
performance.
0 1 2 3 4 5 6 7
18.1 would like to familiarize other departments or people with the progress of this
new approach to use district’s design to use student data to improve student performance.
0 1 2 3 4 5 6 7
19.1 am concerned about evaluating my impact on students in relation to the district’s
design to use student data to improve student performance.
0 1 2 3 4 5 6 7
20. I would like to revise the district’s design to use student data to improve student
performance.
0 1 2 3 4 5 6 7
187
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2 1 .1 am completely occupied with other things besides the district’s design to use
student data to improve student perform ance.
0 1 2 3 4 5 6 7
2 2 . 1 would like to modify our use of the district’s design to use student data to improve
student performance.
0 1 2 3 4 5 6 7
23. Although I don’t know about the district’s design to use student data to improve
student performance, I am concerned about aspects of the district’s design.
0 1 2 3 4 5 6 7
2 4 . 1 would like to excite m y students about their part in the district’s use of student
data to improve student performance.
0 1 2 3 4 5 6 7
2 5 .1 am concerned about time spent working with nonacademic problems related to
the district’s design to use student data to improve student performance
0 1 2 3 4 5 6 7
26. I would like to know what the use of the district’s design to use student data to
improve student performance.
0 12 3 4 5 6 7
2 7 .1 would like to coordinate my effort with others to maximized the effects of the
district’s design to use student data to improve student performance.
0 1 2 3 4 5 6 7
28.1 would like to have more information on time and energy commitments required
by the district’s design to use student data to improve student performance.
0 1 2 3 4 5 6 7
2 9 .1 would like to know what other faculty are doing in the area of implementing the
district’s design to use student data to improve student performance.
0 1 2 3 4 5 6 7
188
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
30. At this time, I am not interested in learning about the district’s design to use student
data to improve student performance.
0 1 2 3 4 5 6 7
31.1 would like to determine how to supplement, enhance, or replace the district’s
design to use student data to improve student performance.
0 1 2 3 4 5 6 7
3 2 .1 would like to use feedback from students to change the district’s design to use
student data to improve student performance.
0 1 2 3 4 5 6 7
3 3 .1 would like to know how my role will change when I am using the district’s
design to use student data to improve student performance.
0 1 2 3 4 5 6 7
34. Coordination of tasks and people in relation to the district’s design to use student
data to improve student performance is taking too much of my time.
0 1 2 3 4 5 6 7
35. I would like to know how the district’s design to use student data to improve student
performance is better than what we have now.
0 1 2 3 4 5 6 7
3 6 .1 am concerned about how the district’s design to use student data to improve student
performance affects students.
0 1 2 3 4 5 6 7
Stages of Concerns Results
Insert stages of concern Results Table
Page ?
189
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX J
Use of Data in Schools Study
Researcher’s Observation and Rating Form, Question #3
District support fo r standards-based instruction and assessment
Intended District Impact:
Observed Impact on School Site:
Researcher’ s rating on how effectively district design improves student performance
as demonstrated in standardized assessment results.
1= not effective 2= somewhat effective 3= unclear 4= effective 5= very effective
Researcher’s Rating: 1 2 3 4 5
190
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Use of Data in Schools Study
Researcher’s Observations and Rating Form, Question #3
District and school accountability to standards-based curriculum
Student Performance Data forwarded to School:
How School is Utilizing Student Data to Impact Student Performance:
Researcher’ s rating on degree that district provided student data is used by school.
1= not effective 2= somewhat effective 3= unclear 4= effective 5= very effective
Researcher’s Rating:________ 1 2 3_____ 4_____ 5___________________
191
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Use of Data in Schools Study
Researcher’s Observation and Rating Form, Question #3
Degree to which High Student Performance is aligned to Standards and
Communicated to Teachers, Students and Parents:
Researcher’ s rating on how effectively high student performance is developed
throughout the school learning community.
1 - not effective 2 - somewhat effective 3= unclear 4= effective 5= very effective
Researcher’s Rating: 1 2 3 4 5
192
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX K
D istrict Interview Guide
CFa
• Describe in-services developed by the district to promote the successful
implementation of standards-based instruction..
• What in-services were developed by the district to assist school staff in the use of
student data?
• What standards-based assessment data was forwarded to the school?
9 How was student performance data disaggregated for the schools?
• How many of the assessments were based on state standards?
• In what ways is the district helping schools prepare for future changes in state
assessments?
CFb
• Describe state and district incentive plans that are in place to support high-quality
standards-based student performance?
9 How often do district personnel review student performance data?
• What corrective actions are taken by the district when student data indicates that
student achievement is not at an acceptable level? Give examples (vignettes)
® What district strategies are used to determine possible strategies for improving
student performance on assessment instruments?
CFc
9 What are the district’s “high” performance goals for students?
9 How do district performance standards correspond to state standards?
9 Describe district strategies for ensuring that all students master the state
standards for their grade-level.
9 In what way does the district communicate to parents about standards-based
curriculum and student assessment results?
9 Describe the district process for determining what student data is to be
disseminated to schools.
9 What is your overall assessment of student achievement as it relates to state-
based standards.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
School Interview Guide-Principal
CFa
® What standardized assessments were forwarded to the school site?
• Which of the assessments are aligned to state standards?
® What assessments disaggregated student performance data?
• What strategies do you use to assist your teachers in understanding the
information in the assessment data?
• What reflective structures do you promote among your teachers to enhance
necessary changes to instruction based on student performance data? Give
examples (vignettes)
CFb
• How do you motivate teachers to maintain high energy and enthusiasm towards
the improvement of student performance? Give examples (vignettes)
• What methods and instruments do you use to evaluate student performance?
• How frequently does the school evaluate student performance?
CFc
• How are teachers, student and parents made aware of the learning goals of
standards-based assessments?
® Describe high performance by students as it relates to state standards.
• What strategies are used at your school to ensure that all students achieve levels
of high performance. Give examples, (vignettes)
• What are some of the barriers you perceive to developing a high level of student
performance in a standards-based environment?
• Describe ways that these barriers can be reduced or eliminated.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
School Interview Guide-Leadership Team
CFa
• Describe the in-services that were provided to allow you to effectively use
student data as a team.
• What did the data tell you about how well all student groups are performing?
® Describe the relative effectiveness of individually reviewing student data as
compared with evaluating it in a collaborative team environment.
• How often do you work in collaborative units to evaluate student performance?
CFb
• How do you know how to alter instruction to improve student performance?
Give an example, (vignette)
• How do you determine that your instruction is promoting strong student
performance? Give examples, (vignettes)
CFc
® What standardized assessments do you use to evaluate student performance?
• How do you ensure that parents understand standards-based instruction and
assessments?
• What are the barriers, from a teaching perspective, to making sure that all
students perform at a high level?
• What changes would have to be made to reduce these barriers?
• How are your report cards based on state standards?
• Describe how you know when a student is performing at a “high” level.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX L
Document Classification
(CFa) Support of Standards-Based Curriculum and Assessment:
District School
(CFb) Student Performance Data Documents:
Standards-based
*
Disaggregated
*
Non/partiallv Standards-based
(CFc) Incentive Programs:
District School
*
(CFd) Reflective Practices:
District School
*
(CFe) Support for Strong Instructional Improvement:
District
*
School
*
(CFf) Standards-based instruction of Parents, Community Members and Students:
Parents Community Students
District-
School-
196
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

An analysis of the impact a district design on the use of data has on student performance

PDF

An analysis of the use of data to increase student achievement in public schools

PDF

How effective schools use data to improve student achievement

PDF

A case study: An analysis of the adequacy of one school district's model of data use to raise student achievement

PDF

How districts and schools utilize data to improve the delivery of instruction and student performance: A case study

PDF

Design, implementation and adequacy of using student performance data and the design's link to state context for assessing student performance: A case study

PDF

Design, implementation and adequacy of data utilization in schools: A case study

PDF

An analysis of student -level resources at a California comprehensive high school

PDF

An analysis of the elementary principal's role in implementing school accountability within California's high -priority school: A case study

PDF

Connecting districts and schools to improve teaching and learning: A case study of district efforts in the Los Coyotes High School District

PDF

Implementation and levels of use of technology plans in a Southern California K--12 district and its middle school

PDF

Implementation challenges of a school district's technology plan at the middle school level

PDF

An analysis of the elementary principal's role in implementing school accountability within California's High Priority School: A case study

PDF

Data: Policies, strategies, and utilizations in public schools

PDF

"Show me the money": Analyzing student-level resource allocation and use in a large urban elementary school

PDF

An analysis of per-pupil resource allocation at an urban elementary school

PDF

Essential elements of superintendent training and preparation programs as indicated by practicing superintendents in the state of California

PDF

How classroom teachers react to and implement California's Beginning Teacher Support and Assessment educational reform policy

PDF

An analysis of the implementation of content standards in selected unified school districts of California

PDF

Implementation of educational policy: Technology implementation in a California middle school

Asset Metadata

Creator Sevillano, Paul John (author)

Core Title An analysis in the use of student performance data in schools

School Rossier School of Education

Degree Doctor of Education

Degree Program Education

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag Education, administration,education, educational psychology,OAI-PMH Harvest

Language English

Contributor Digitized by ProQuest (provenance)

Advisor Marsh, David D. (committee chair), [illegible] (committee member), Picus, Lawrence O. (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c16-260978

Unique identifier UC11339314

Identifier 3093916.pdf (filename),usctheses-c16-260978 (legacy record id)

Legacy Identifier 3093916.pdf

Dmrecord 260978

Document Type Dissertation

Rights Sevillano, Paul John

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA