University of Southern California Dissertations and Theses

Evaluation of the effects of longitudinal tracking of student achievement to assess school quality

(USC Thesis Other)

Evaluation of the effects of longitudinal tracking of student achievement to assess school quality

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content EVALUATION OF THE EFFECTS OF LONGITUDINAL
TRACKING OF STUDENT ACHIEVEMENT TO
ASSESS SCHOOL QUALITY
by
Cynthia M. Anderson
A Dissertation Presented to the
FACULTY OF THE ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirement of the Degree
DOCTOR OF EDUCATION
May 2005
Copyright 2005 Cynthia M. Anderson
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UMI Number: 3180487
Copyright 2005 by
Anderson, Cynthia M.
All rights reserved.
INFORMATION TO USERS
The quality of this reproduction is dependent upon the quality of the copy
submitted. Broken or indistinct print, colored or poor quality illustrations and
photographs, print bleed-through, substandard margins, and improper
alignment can adversely affect reproduction.
In the unlikely event that the author did not send a complete manuscript
and there are missing pages, these will be noted. Also, if unauthorized
copyright material had to be removed, a note will indicate the deletion.
®
UMI
UMI Microform 3180487
Copyright 2005 by ProQuest Information and Learning Company.
All rights reserved. This microform edition is protected against
unauthorized copying under Title 17, United States Code.
ProQuest Information and Learning Company
300 North Zeeb Road
P.O. Box 1346
Ann Arbor, Ml 48106-1346
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ACKNOWLEDGEMENTS
To Andy, my husband, without whose support and faith this endeavor would
not have been completed; to my cohort, whose mantra “We all walk in 2005” kept
the spark simmering; to Dr. Mike Me Laughlin; for giving me the chance to succeed;
to Dr. Carl Cohn, committee member and respected professor; and to Dr. Dennis
Hocevar, mentor and inspiration.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
TABLE OF CONTENTS
ACKNOWLEDGMENTS.................................................................................... ii
LIST OF TABLES................................................................................................. v
LIST OF FIGURES............................................................................................... vi
ABSTRACT............................................................................................................ vii
Chapter Page
1. INTRODUCTION..................................................................................... 1
2. REVIEW OF LITERATURE................................................................... 20
Adequate Yearly Progress (AYP)...................................................... 20
Value-Added Analysis and Longitudinal Tracking........................... 32
Conclusion........................................................................................... 38
3. RESEARCH METHODOLOGY.............................................................. 40
Introduction.......................................................................................... 40
Research Question............................................................................... 40
Research Design.................................................................................. 44
Research Population............................................................................ 45
Case Study 1 (School 1)............................................................... 46
Case Study 2 (School 2)................................................................ 47
Case Study 3 (School 3)............................................................... 48
Case Study 4 (School 4)............................................................... 49
Case Study 5 (School 5)............................................................... 51
Case Study 6 (School 6)............................................................... 52
Sample Size.......................................................................................... 53
Data Collection.................................................................................... 53
Instrumentation................................................................................... 54
Data Analysis....................................................................................... 56
Implications.......................................................................................... 57
4. RESULTS................................................................................................... 58
Introduction.......................................................................................... 58
Case Study 1 (School 1)................................................................... 59
Case Study 2....(School 2)................................................................... 60
Case Study 3 (School 3)................................................................... 61
Case Study 4....(School 4)............................................... 62
Case Study 5 (School 5)................................................................... 63
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
iv
Chapter Page
Case Study 6 (School 6)...................................................................... 65
5. DISCUSSION............................................................................................ 67
Implications.......................................................................................... 73
Limitations........................................................................................... 76
Recommendations............................................................................... 77
REFERENCES....................................................................................................... 79
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
V
LIST OF TABLES
Tables Page
1. English/Language Arts and Math.............................................................. 5
2. School District A Demographics............................................................... 44
3. API/AYP School 1...................................................................................... 46
4. API/A YP School 2 ...... 48
5. APEAYP School 3 ..................................................................................... 49
6. API/A YP School 4 ..................................................................................... 50
7. API/A YP School 5 ...................................................................................... 51
8. API/A YP School 6 ..................................................................................... 52
9. District and School Site Population Data................................................... 54
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
vi
LIST OF FIGURES
Figures Page
1. California Elementary and Middle Schools and Elementary Districts
AMOs: English/Language Arts......................................................... 22
2. California Elementary and Middle Schools and Elementary Districts
AMOs: Math....................................................................................... 23
3. California High Schools and High School Districts AMOs:
English/Language Arts........................................................................ 23
4. California High Schools and High School Districts AMOs:
Math...................................................................................................... 24
5. API Gains: School District A ..................................................................... 55
6. Cross Sectional and Cohort Matched Data: School 1............................. 60
7. Cross-sectional and Cohort Matched Data: School 2 ............................ 61
8. Cross-sectional and Cohort Matched Data: School 3 ............................ 62
9. Cross-sectional and Cohort Matched Data: School 4 ............................ 63
10. Cross-sectional and Cohort Matched Data: School 5 ............................ 64
11. Cross-sectional and Cohort Matched Data: School 6 ............................ 66
12. Cross-sectional and Cohort Matched Data: School 6 ............................ 71
13. Cross-sectional and Cohort Matched Data:....School 2............................ 72
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ABSTRACT
Accountability is the dominant theme of the federal No Child Left Behind Act
of 2001, and at the core of this Act is Adequate Yearly Progress, or AYP. All
schools, including charter schools and schools not receiving federal funding under
Title 1, must demonstrate adequate yearly progress by showing that a percentage of
students scoring at proficient or above increases by a determined amount each year,
with all students proficient by 2013-2014.
In California, test score data are reported as “snapshots” of student
performance, or current status indicators, which represent the average score of
students enrolled in a district or school, and are assessed using percentile rankings.
Accountability plans should require fair analytic methods that support school-wide
improvement and planning; this study demonstrates that the current status indicators
may actually provide less information about school quality, the very crux of No
Child Left Behind.
Using standardized test data from 807 students tested over a 3-year period in
six schools in California, three of which are or have been labeled as Program
Improvement Schools, value-added analysis, particularly longitudinal tracking of
student data, reveals clearly a different picture of each school’s academic success
than the State’s status indicators. The indication of greater school quality and
improvement under a value-added system was particularly true for those schools
with high student mobility rates, low socioeconomic status, and/or low Academic
Performance Indicators (API). Evidence supports the punitive labels and sanctions
permission of the copyright owner. Further reproduction prohibited without permission.
attached to low-performing schools may be unjustified when data are examined
longitudinally as opposed to the current successive cohort approach.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1
CHAPTER 1
INTRODUCTION
The purpose of this evaluation research project, is to determine whether
schools in California are fairly and accurately measured as demonstrated under the
federal guidelines of the No Child Left Behind Act of 2001 through the Academic
Performance Indicators (API) and Adequate Yearly Progress (AYP) accountability
plans, in comparison to the impact that value-added indicators might have on school
achievement, specifically, longitudinal tracking of student test scores. Stated in
another way, this study evaluates the impact of value-added indicators that focus on
the growth in student achievement from one grade level to the next for given cohorts
of the same students, rather than on the trend over time in average test scores for
students at a given grade level. In other words, longitudinal value-added data as
opposed to cross-sectional student data will be compared.
Can data gathered by the State for accountability purposes provide helpful
information to both schools and the public? As states and districts across the United
States struggle to implement high-stakes accountability plans, a major component of
the federal legislation No Child Left Behind, questions and uncertainty of how to
analyze and use this data are surfacing, and application of this data will most
certainly have dramatic impacts on school districts nationwide.
Accountability is the dominant theme of No Child Left Behind, and at the
core of this Act is Adequate Yearly Progress, or AYP. All schools, including charter
schools and schools not receiving federal funding under Title 1, must demonstrate
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2
adequate yearly progress by showing that a percentage of students scoring at
proficient or above on standardized testing increases by a determined amount each
year, or face corrective actions, which can eventually include restructuring. These
incremental amounts per year are determined by individual states, but the ultimate
goal of No Child Left Behind is to have every child in the nation at grade proficiency
or above by the year 2013-2014.
In California, AYP is based on English-Language Arts and Mathematics
Annual Measurable Objectives (AMOs). Data from 2001-2002 are used to
determine a baseline or starting point for measuring the percentage of students
proficient in math and English/language arts. Baselines for science are to be set by
the 2007-2008 school year, when mandatory testing in this area begins.
All students are held to the same high academic standards, and as schools and
districts begin to disaggregate data from these standardized tests, achievement gaps
between ethnic groups, students of low socioeconomic status, and special education
students are predictably surfacing. These gaps can be attributed to the fact that the
test score data are reported as “snapshots” of student performance, or as current
status indicators. These indicators represent the average score of students enrolled in
a district, school, grade level, or classroom, and are assessed using percentile
rankings. These indicators may be useful in describing performance for a given
student population in a particular year; however, these same indicators actually
provide less information about school quality, the very core of the No Child Left
Behind legislation.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
3
How then, can a school or district or state accurately and reliably demonstrate
that schools are successful in achieving state and district standards, AND are also
able to report the extent of their rates of improvement? Currently, under No Child
Left Behind, the Adequate Yearly Progress index provides an erroneous indication of
school quality, and the performance indicators used to determine AYP tend to be
biased against schools and programs that disproportionately serve academically
disadvantaged students; this student variability may have implications for the
validity of accountability decisions. Differences in groups of students from one class
to the next, from one year to the next, may result in a school meeting its AYP one
year, not the next year, then meeting AYP the third year even though no changes
have occurred in the school’s instructional program. Therefore, accountability plans
need analytic methods that more fairly depict the impact of a school on student
learning and can provide results that support school-wide improvement and planning.
A growing number of states and districts are opting to use value-added
analysis as a new methodology for analyzing student achievement data, thereby
enhancing the usability and acceptability of achievement tests. Value-added analysis
seeks to determine how much value a school added to a student’s learning during the
academic school year. It measures an individual’s growth in learning from one
standardized measured test to the next standardized measured test, tracking the
progress of individual students over time. Longitudinal tracking of student
achievement, then, is one way this measurement can be implemented.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
4
School District A in California is a K-8 school district consisting of four K-5
elementary schools, two K-8 elementary schools, one 6-8th grade middle school, four
K-8 charter schools, and one charter high school. For purposes of this evaluative
study, district-wide data of the six elementary schools’ second through 5th grade
student performances in English/Language Arts and Math on the California
Standards Test (CST) for years 2002 through 2004 are indicated in table 1.
Preliminary analyses of these data indicate a high percentage of students
district-wide scoring at the basic level and below. But do these scores accurately
reflect the individual school’s effectiveness and quality? A longitudinal analysis of
each school’s students will give a clear picture of the school’s effectiveness, and will
show exactly how much growth each student has made from year to year.
An evaluation of the effectiveness of longitudinal tracking of student
achievement as a means to assess school quality is necessary to accurately depict
how a school is demonstrating success. Evaluation of the impact of this program is
critical to understanding that value-added indicators focus on the growth in student
achievement from one grade level to the next for given cohorts of students, rather
than on the change or trend over time in average test scores for students at a given
grade level. Results from this evaluation will be useful to school leaders to
determine the effectiveness of the school’s policies and programs, instruction and
teacher effectiveness, and will provide a sound foundation for professional
development. Value-added indicators are thus based on longitudinal as opposed to
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
5
Table 1. English/Language Arts and Math
Grade 2 Grade 3 Grade 4 Grade 5
ELA MATH ELA MATH ELA MATH ELA MATH
2002
% tested 92.0 99.0 94.0 96.0 89.0 93.0 95.0 96.0
Mean scaled score 315.2 339.2 324.3 324.7 335.5 336.4 332.3 321.3
% advanced 5.0 12.0 8.0 8.0 13.0 17.0 9.0 6.0
% proficient 22.0 280.0 22.0 25.0 25.0 25.0 22.0 21.0
% basic 33.0 290.0 34.0 34.0 34.0 34.0 44.0 37.0
% below basic 23.0 260.0 22.0 27.0 20.0 22.0 20.0 31.0
% far below basic 17.0 50.0 14.0 8.0 7.0 7.0 50.0 6.0
2003
% tested 98.0 98.0 98.0 98.0 99.0 99.0 100.0 100.0
Mean scaled score 335.9 368.5 325.9 346.7 345.4 346.4 337.4 331.7
% advanced 10.0 28.0 9.0 18.0 13.0 16.0 8.0 6.0
% proficient 29.0 34.0 20.0 27.0 27.0 30.0 32.0 29.0
% basic 33.0 20.0 39.0 29.0 42.0 30.0 36.0 29.0
% below basic 20.0 19.0 19.0 19.0 11.0 18.0 12.0 25.0
% far below basic 7.0 2.0 12.0 7.0 76.0 6.0 11.0 11.0
2004
% tested 98.9 98.7 98.1 97.9 98.0 97.8 99.3 99.3
Mean scaled score 332.3 357.6 338.7 375.2 343.6 347.5 345.4 334.1
% advanced 9.0 17.0 11.0 32.0 17.0 19.0 17.0 8.0
% proficient 26.0 35.0 27.0 28.0 23.0 37.0 28.0 30.0
% basic 36.0 30.0 34.0 21.0 40.0 34.0 35.0 28.0
% below basic 20.0 16.0 17.0 15.0 14.0 11.0 11.0 23.0
% far below basic 9.0 2.0 10.0 3.0 6.0 9.0 9.0 11.0
cross-sectional student data, thereby more reliably assessing school quality; this is
consistent with the No Child Left Behind stipulation that AYP should not only define
progress, but should also result in “continuous and substantial academic
improvement for all students.”
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
6
The No Child Left Behind Act of 2001(NCLB), an amendment to the
Elementary and Secondary Education Act of 1965 (ESEA), represents a much
expanded federal role in public education by outlining provisions for strengthening
accountability in academic achievement in all public schools. Besides requiring
annual testing, the law seeks various methods to measure school effectiveness,
creates a timetable to track progress, and establishes a sequence of specific
consequences if schools, districts, and states fail. With this legislation, the federal
government has essentially assumed the role of judge of the performance goals and
progress made, leaving each state to establish the base for the judgments, including
curriculum and standards development and assessment tools. NCLB’s immediate
objective then, appears to create a set of procedures to eventually link assessments
over time, across systems, and with external assessment validation of the system
from the National Assessment o f Educational Progress (NAEP) as stated in section
1112(b)(F) of the Act, thus creating an instrument of measurement for monitoring
the educational progress of the nation’s children.
Adequate Yearly Progress (AYP) is at the very core of NCLB. The essential
components of the AYP requirements as presented in a paper by the (2002) Council
of State School Officers entitled Implementing the State Accountability System
Requirements Under the No Child Left Behind Act o f2001 include:
1. An aligned system of academic content standards, academic student
achievement standards, and assessments of student performance;
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
7
2. Annual assessments of student progress in attaining the student
academic achievement standards;
3. School, district, and state accountability decisions based on the
performance of specific subgroups of students designed to ensure that all students
are proficient in reading or language arts and mathematics by 2013-2014; and
4. A system of rewards and required, progressive sanctions to encourage
and support high and low performing schools.
Within these boundaries, the Act provides states with some discretion. State
education agencies may establish a uniform procedure for averaging data over
multiple years and across grades, or make their own determination of AYP as long as
it is:
1. Based primarily on academic indicators (e.g., student performance on
tests);
2. Technically rigorous; and
3. Applied to school, district, and state levels of progress.
Section 111 l(b)(J) of the Act provides that, “For the purpose of determining
whether schools are making [AYP], the state may establish a uniform procedure for
averaging data” that generates average proficiency levels over multiple years and
across grades in a school. Such an approach for measuring involves establishing a
baseline of “proficiency” across all schools and subgroups in 1 year, and then
calculating increases in the percentage of proficient students in various increments of
time.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
8
Schools must be able to demonstrate students are making academic progress
as evidenced by annual gains in test scores. AYP is the minimum level of progress
that schools, districts, and states must achieve each year. In addition to the required
reporting of test results, states may select one or two other measures of academic
progress. In California, components of AYP include:
1. Achievement of English/Language Arts and math Annual Measurable
Objectives (AMOs): percent proficient or above;
2. Ninety-five percent student participation rate;
3. Academic Performance Indicator (API) scores for all schools; and
4. Graduation rate for high schools.
Central to this accountability system, many variables have profound impact
on the reliability of the components of determining AYP and the validity of the
decisions that emerge from such a system. Answers to the following questions must
be considered in evaluating this accountability system:
1. Is the system focusing on the “right goals?
2. Does the accountability system identify the schools that truly need to
improve?
3. Is the accountability system related to improved student learning?
Clearly, one is hard-pressed to dispute the lofty goals of this federal
education law in its pursuit of high standards and accountability, yet as each state
releases more and more information and data regarding AYP as mandated by NCLB,
it has become clear to many teachers, parents, administrators, and other educational
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
9
leaders that the one-size-fits-all AYP system has many serious flaws that prevent a
fair and accurate assessment of student progress. For example, the amount of growth
stipulated for discrete populations, i.e., those identified by race/ethnicity, English
Language Learners, and students with disabilities, as well as the general population,
is theoretically possible, but practically impossible. Financial resources and human
resources are insufficient in many communities to overcome the deficits of every
child in elementary and secondary education. The law requires schools to test every
student, and that every student will make progress toward a targeted “proficiency”
level, regardless of intellectual ability, language barrier, or learning disability. This
could potentially keep low-performing schools in that status perpetually, or until the
school’s doors are closed through the NCLB process.
Next, schools must ensure at least 95% student participation in state tests,
including 95% o f the children in each subgroup. This participation requirement
ensures a clear picture of student achievement. In California, parents can request
that their child not be tested, but under NCLB legislation, that child is counted in the
participation rate o f the school. In small schools with few children in each grade, the
absence of even one student for medical, family or other reasons can mean the
difference between a school meeting its AYP goals or being classified as “needing
improvement.”
Additionally, high schools must have a graduation rate of 95%, or if below
95%, show improvement from the previous year toward the goal of 95%. These data
compare 9th grade enrollments with 12th grade graduates. Because schools do not
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
10
track individual students from 9th grade through 12th grade, and because students
often move from one school district to another during the high school years, there is
potentially room for inaccuracy in these numbers.
In California, schools are recognized and rewarded as high performing or fast
improving if they meet or exceed state standards. A gap in the “fairness” of meeting
AYP goals is thus illustrated by an example of a school that reaches or surpasses the
English/language arts and math standards, but fails to meet AYP requirements
because it falls short on just one of the other above-listed criteria. That school,
should it fail to meet any one component of AYP the following year is then labeled
as “in need of improvement” and faces mandated sanctions.
Finally, although purported to be for diagnostic purposes only, the extensive
assessments of students is being used for “high-stakes” consequences, and the data
surfacing are certainly misinterpreted by the public at large. The punitive approach
of the legislation should be replaced with incentives to improve and exemptions.
Data should not be used to punish low-performing schools if student achievement
shows improved student growth.
These “gaps” in the AYP index clearly require calibration. The major goal of
AYP is to reform schools, not reform kids, and as long as schools are providing 1
year of learning or more every year to every child, that school is being successful.
But under the current AYP system, these very schools are being mislabeled as “low-
performing.”
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
11
The most fundamental question for any educational system, then, is: How
much are students learning? In California, specifically, this question cannot be
answered fully because the current data system is cross-sectional in nature. The
Standard Assessment and Reporting (STAR) system is reports annually on each class
of students within a given school, for example, this year’s 3rd grade class at school
“A.” These types of scores are generally referred to as “level” or “status” indicators
(Carlson ,2002; Meyer, 1995). In the next year, STAR reports how the next “cohort”
of school “A” 3rd graders performed. The Academic Performance Indicator (API), a
component of California’s AYP criteria, is based on the difference between the
performance of last year’s 3rd grade students of school “A,” and this year’s 3rd
grade students of school “A.” This “successive cohort” approach (Carlson, 2002;
Linn, 2002) has serious weaknesses, the most obvious of which is that differences
between the composition of 1 year’s class of 3rd graders and the next year’s class of
3rd graders can cause changes in aggregate test results. The current API provides a
misleading picture of academic progress within schools and districts. It does not
measure growth in individuals over time; it merely compares overall achievement of
successive classes over time.
Robert Linn and Carolyn Haug (2002), in a paper entitled “ Stability o f School
Building Accountability Scores and Gains, ” report that:
A number of states have school-building accountability systems that rely on
comparisons of achievement from one year to the next. Improvement of the
performance of schools is judged by changes in the achievement of
successive groups of students. Year-to-year changes in scores for successive
groups of students have a great deal of volatility. The uncertainty in the
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
12
scores is the result of measurement and sampling error and nonpersistent
factors that affect scores in one year but not the next. The level of
uncertainty was investigated using fourth grade reading results for school in
Colorado for four years of administration of the Colorado Student
Assessment Program. It was found that the year-to-year changes are quite
unstable, resulting in near zero correlation of the school gains from years one
to two with those from years three to four. (p. 29)
In California, STAR and the API could be modified to follow the same class
from year to year, that is, to compare the scores of the 3rd grade class of school “A”
in 1 year to the scores of the 4th grade class of school “A” in the following year. But
this approach, sometimes referred to as “quasi-longitudinal,” also has serious
limitations. For example, given high levels of student mobility, many members of
last year’s 3rd grade class may have transferred out of school “A.” Many other
students may have transferred in during the year, further affecting the mix of
students from year-to-year.
A true longitudinal data system statewide would be able to track students
from year-to-year regardless of whether or not they re-enrolled in school “A.”
Unless they transferred out-of-state or dropped out of school, such a system would
be able to measure their progress from third grade to fourth grade regardless of
where they went, thus providing a much clearer and more accurate portrait of student
learning.
The need for accurate data to measure school effectiveness is paramount
under the NCLB requirements to meet AYP goals. How this is accomplished relies
on the accountability systems at the state and local levels. It must be noted that
research and evaluation efforts also depend on such a system to answer questions on
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
13
student achievement. Therefore, the following components should be evaluated as
critical to an accurate and reliable accountability system:
1. Focus schools on improving each student’s achievement as well as
increasing the number who score above the standard passing score.
2. Provide more accurate and stable data from year-to-year to sustain
public and parental support.
3. Longitudinal tracking enables scientific control for prior student
achievement, thus holding schools accountable for the value they add to student
achievement.
4. Volatility in year-to-year scores based on comparing different
students leads to erroneous conclusions. More accuracy can be drawn from
combining same grade comparisons with longitudinal results of the same students
over time.
California has invested billions of dollars in improving schools over the past
decade. Class size reduction, expanded teacher professional development, increased
emphasis on achievement testing at all grade levels, high school exit exams, and new
college outreach programs have all promised improved academic outcomes for
California’s schools.
According to the California Legislature’s Joint Committee’s Master Plan for
Education in California (2000) the opportunity to implement a statewide data
tracking system has arisen which would make it possible to track the state’s progress
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
14
toward the larger goal of educational excellence for all students. Specifically, the
Plan proposes:
The State should develop and report yearly on a comprehensive set of
educational indicators, constructed from the data provided by an integrated,
longitudinal, learner-focused data system and from other school-level data
about educational resources, conditions and learning opportunities, (p. 46)
Longitudinal tracking and quasi-longitudinal tracking of student assessment
scores are mere components of a larger, more comprehensive form of accountability
referred to as “value-added assessment.” From a historical perspective, before
standards-based reform swept across the educational landscape, the quality of
schools was measured almost exclusively in terms of what Harold Doran (Horan &
Izumi, 2004) calls “inputs”; for example, the number of books in school libraries or
the qualifications of teachers. However, the subsequent demands of greater
accountability in public education led states and districts to supplement these
indicators with ones defined in terms of student performance, especially as measured
by standardized test scores.
These “snapshots” of student performance are commonly referred to as
current-status indicators, and represent the average scores for students enrolled in a
district, school, grade level, or classroom, and are assessed using percentile ranks,
the proportion of students meeting a state or district designated performance
standard. Although useful in describing performance for a given population in a
particular year, these indicators may actually provide less information about school
quality than the traditional input measures they have replaced. In the absence of
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
15
other measures, current-status indicators are invalid and potentially misleading.
According to Robert Meyer, in an article entitled Value-Added Indicators o f School
Performance: A Primer (1997), current-status indicators:
1. Reflect the combined influence of family, background, community,
and years of prior schooling on student achievement, and unfairly judge schools that
serve disproportionately high numbers of disadvantaged students.
2. Reflect the cumulative impact of school and nonschool related
influences on achievement over multiple years, and therefore can be skewed in any
given year by prior influences on student learning.
3. Fail to localize performance to a specific classroom or grade level,
making them less useful as a means of evaluating reform initiatives, teacher
effectiveness, and program innovation.
4. Tend to be highly “contaminated” by the influence of multiple
educational settings due to widespread student mobility in and out of different
districts.
Value-added assessment, with longitudinal tracking of individual student data
as the prime component of this system, is an approach to analyzing and reporting
test-score data that addresses many of these pitfalls. Whereas current-status
indicators measure the performance of a group of students at a single point in time,
value-added assessment focuses on the achievement gains of individual students over
time, i.e., from spring to spring. Regardless of where on the achievement scale a
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
16
child begins, this model measures the value added to his or her performance by the
educational system each year.
The value-added assessment model of accountability was developed in the
early 1980s at the University of Tennessee by statistician Dr. William Sanders.
Several other states have begun to follow Tennessee’s lead and develop account
ability systems based on Sanders’ (1998) model. For example, using the Measure of
Academic Progress, Arizona measures student growth with the expectation that each
student achieve 1 year’s growth from one grade to the next. While no state other
than Tennessee has yet implemented Sanders’ system on a statewide basis, several
have initiated trials, from state-funded pilot programs to privately funded initiatives,
to introduce his methods of holding schools and districts accountable for student
growth. Like Arizona, other areas, including Florida, North Carolina, and the
Denver Public School District, have developed their own particular accountability
programs based on value-added concepts (Bianchi, 2003, p. 3).
To implement a value-added accountability system, three key data
requirements must be met. First, students must be tested at least annually. Second,
test scores must be reported on a common scale so students’ test scores can be
compared from one year to the next. Finally, to track student achievement gains
over time, students must be assigned individual identifications codes that remain
consistent over all school years, regardless of the school attended. School District A
in California has met these requirements, a necessary component of this evaluation.
Moreover, it is only a matter of time before the State of California will also have
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
17
these requirements in place, which would allow statewide longitudinal tracking of all
students enrolled in California schools. Furthermore, there are two United States
Department of Education provisions for developing longitudinal databases: under
Title VI, Part A, Section 6111 and under Title I, Part E, Section 1501 (b). In
addition, Educational Technology funds under Title II, part D (1) can be used to
support the development of school information management systems.
However, critics argue that for an accountability system to be effective, it
must be easily understood, and that the advanced statistical methods underlying
value-added are well beyond most educator’s grasp. Trading rigor and accuracy for
simplicity is indefensible in these days of accountability and high stakes. Others
claim that value-added assessment inflates the degree of random error involved in
measuring student performance, thereby increasing the risk that teachers or schools,
may be rewarded or punished for inaccurate outcomes of their students’ learning.
But confidence in value-added assessment results can be increased by a variety of
methods, including averaging data over several school years or incorporating
multiple achievement measures in the analysis (Rogosa, Brandt, & Zimowski, 1982).
Finally, because value-added indicators adjust for student background
characteristics associated with academic growth, some critics argue that they create
lower performance expectations for disadvantaged students and the schools that
serve them. This concern is misguided for two reasons: first, the value-added
approach does not preclude a state or district from setting high achievement
standards for all its students, a mandated requirement under No Child Left Behind.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
18
Secondly, schools serving more disadvantaged students will typically have higher,
not lower, value-added performance goals. Because these schools generally start at a
lower performance level, they must cover more ground than other schools in order to
achieve the same end. As Robert Meyer (1997) observes, “This is a strength, not a
weakness, of the value-added approach.”
With the passage of No Child Left Behind, the external accountability role of
value-added assessment will be determined in large part, by the degree to which
states embrace it as a means of defining Adequate Yearly Progress (AYP).
Currently, most states (California included) define AYP solely in terms of
performance categories. However, the United States Department of Education has
issued clarifications that seem to signal approval of measurement models
recognizing both growth and proficiency. According to one statement from the
United States Department of Education concerning definitions of AYP, “states and
LEAs are strongly encouraged to develop systems to recognize . . . low-performing
schools that are making . . . improvement (U. S. Dept, of Education, 2002).
It makes little sense to continue to define AYP solely in terms of the
percentage of students crossing an arbitrary bar of proficiency, while ignoring the
growth that occurs within broad performance categories. Relying on average test
scores and proficiency cut-offs to measure AYP exclusively can lead to unintended
consequences, such as leading teachers and administrators to target instruction to
those students most likely to cross the proficiency cut point, leaving others without
focused instruction. The present approach to measuring AYP is also unfair to
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
schools serving large numbers of disadvantaged students since they may make great
strides and still be labeled deficient. Therefore, rates of improvement should factor
into the accountability system of measurement.
NCLB’s requirement for annual testing enables districts and states to track
performance over time, but it does not require them to do so. Even so, NCLB
recognizes the importance of value added assessment and specifically encourages its
use. Title 1, Part A, Section 1111(b), subsection 3(b) states that: “Each state may
incorporate the data from the assessments under this paragraph into a State-
developed longitudinal data system that links student test scores, length of
enrollment, and graduation records over time.”
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
20
CHAPTER 2
REVIEW OF THE LITERATURE
Learning can be defined in terms of change: (a) understanding, (b) gains in
knowledge, (c) skills, and (d) the ability to do things. Therefore, fundamentally,
education is about change. Robert Linn (1999) asserts that program evaluations, as
well as student and school accountability systems, feature the assessment of
progress. No one will argue against students and schools achieving adequate yearly
progress, but how much progress should be required to be considered adequate or
exemplary? A simple question and the crux of this evaluation research study, but a
good deal more difficult to answer than might be expected. Thus, a broad yet
thorough review of the No Child Left Behind Act of 2001 and a clear explanation of
Adequate Yearly Progress (AYP) is a necessary starting point of this literature
review, with a narrowing focus on value-added analysis and the effects it can have in
determining student and school achievement. A specific examination of longitudinal
tracking of student achievement will be compared to the current status indicators of
cross-sectional data used in California, and implications gleaned from the research
will be presented.
Adequate Yearly Progress (AYP)
The federal No Child Left Behind Act (NCLB) of 2001 builds on the
reauthorization of the Elementary and Secondary Education Act (ESEA) of 1965,
restoring the annual testing obligations of 1988 and retaining the standards-based
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
21
emphasis of 1994. It also provides more specificity in terms of defining how student
progress should be measured and reported in an effort to reduce the variation from
state-to-state.
Title 1, subpart A of the legislation requires states to develop accountability
systems that include all public schools, including charter schools, administer annual
testing in Grades 3-8, define “Adequate Yearly Progress (AYP) toward a goal of
100% proficiency in key subject areas within 12 years (the end of the 2013-2014
school year), and report assessment data annually at the state, local education agency
(LEA), and school level, disaggregated by student demographic subgroups including:
1. Economically disadvantaged students;
2. Students with disabilities;
3. Students with limited English proficiency;
4. Major racial and ethnic groups; and
5. Gender.
Before the enactment of No Child Left Behind in 2001, AYP applied to Title I
schools only and was determined through statewide accountability systems. AYP
was based on whether or not the school met the Academic Performance Indicator
growth targets, and did not apply to districts. The federal model of NCLB modified
AYP by specifying annual status targets in English/Language Arts and mathematics,
and applying these targets to all public school districts, as well as schools and student
subgroups (ethnic, socioeconomically disadvantaged, English learners, students with
disabilities).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
22
In California, AYP is measured through the following components:
1. Achievement of English/Language Arts and Math Annual Measurable
Objectives (AMOs) by percent proficient or above ;
2. Ninety-five percent participation rate;
3. API for all schools; and
4. Graduation rate for high schools.
Annual Measurable Objectives in elementary and middle schools are based
on results of the California Standards Tests (CST) in English/Language Arts and
Math, or on the California Alternative Performance Assessment (CAPA) given to
students with disabilities. High School AMOs are based on results of the Grade 10
California High School Exit Exam (CAHSEE) and CAPA for students with
disabilities (Figure 1-4).
100.0 %
100%
9.2%
9 0 %
8 0 %
7 0 %
i7U%
6 0 %
5 0 %
4 0 %
3 0 %
20%
1 0 %
24.4%
13.6%
o %
20 0 1 - 2 0 0 2 - 2 0 0 3 - 2 0 0 4 - 200 5 - 200 6 - 200 7 - 20 0 8 - 2 0 0 9 - 2 0 1 0 - 2 0 1 1 - 201 2 - 20 1 3 -
20 0 2 2 0 0 3 2 0 0 4 2 0 0 5 2006 2007 2 0 0 8 20 0 9 2010 2011 2 0 1 2 20 1 3 2014
Figure 1. California Elementary and Middle Schools and Elementary Districts
AMOs: English/Language Arts
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
23
AMOs: Math
100.0%
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
3.5%
8.5%
7.5%
770%
^ 2 6 .5 %
16.0%
2001- 2002- 2003- 2004- 2005- 2006- 2007- 2008- 2009- 2010- 2011- 2012- 2013-
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Figure 2. California Elementary and Middle Schools and Elementary Districts
AMOs: Math
AMOs: English/Language Arts
100%
80%
60%
40%
20%
0%
3.4%
& C \^ C? 5 CV*
VJ _ cV _CV* -CV^ -CV^ -CV^ _C\N
Figure 3. California High Schools and High School Districts. AMOs:
English/Language Arts
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
24
AMOs: Math
100%
50%
0%
100.0%
rjfa * *2u.y%
Figure 4. California High Schools and High School Districts AMOs: Math
Ninety five percent participation rates are required on any assessment used
for AYP calculations under the No Child Left Behind Act. The remaining 5% is the
maximum allowable percentage of non-participants, including students who are
exempted from testing at parental request.
The Academic Performance Indicator (API) is used as another indicator for
all grades in determining AYP. API scores must be above the “status bar” OR show
growth of at least 1 point. In high schools, the graduation rate is an additional
indicator of AYP, and schools must demonstrate an increase of 1% per year until the
school reaches 100%.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
25
This legislation calls for consequences if states, LEAs, or schools do not
make their prescribed annual gains. If students do not improve over time, schools
and districts will need to provide public school options to parents and can ultimately
be subject to reconstitution. Likewise, failing states run the risk of losing some of
their federal funds if they fail to meet AYP over consecutive years.
According to the California Department of Education (CDE), under NCLB,
the state educational agency (SEA) and local educational agency (LEA) must
identify for corrective action and restructuring any Program Improvement (PI)
school that for 2 years or more has not made AYP. According to the federal
guidance, corrective action identification means that the LEA needs to take greater
control of the school’s management and that more radical action is needed to
improve learning conditions and to increase the likelihood that all students enrolled
in the school will become proficient in math and reading. Years 1 and 2 of Program
Improvement specifically call for the LEA to provide technical assistance to the
school, notify parents of the PI status of the school, give parents the option transfer
their children to another non-PI school in the LEA with paid transportation, and
provide supplemental educational services to all eligible students.
Should the PI school fail to meet AYP in Year 2, Year 3 of Program
Improvement calls for specific corrective action by the LEA, including at least one
of the following:
1. Replacing school staff;
2. Implementing new curriculum;
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
26
3. Decreasing management authority at the school level;
4. Appointing outside experts;
5. Extending the school year or day; and
6. Restructuring the internal organization al structure of the school.
A school is identified for Year 4 restructuring if, after one full year of
corrective action, the school still fails to make AYP. Restructuring call for the LEA
and the school to prepare a plan for alternative governance of the school through one
of the following choices:
1. Reopen the school as a charter school;
2. Replace all or most staff, including the principal;
3. Contract with an outside entity to manage the school;
4. State takeover; and
5. Any other maj or restructuring.
If a school completes Year 4 and does not make AYP, it must be identified as
a Year 5 school. Year 5 is the second year of restructuring, and the LEA must
implement at the start of the school year the restructuring plan it developed for the
school during Year 4.
The sanctions schools must face for not making AYP are indeed financially
punitive; the costs for these actions must be borne by already financially strapped
school districts. Resources, both financial and human, are insufficient to overcome
the deficits many students bring to 13 years of elementary and secondary education.
However, the doom and gloom of schools being identified as in need of Program
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
27
Improvement is growing with each passing year as the accountability performance
bar of No Child Left Behind is raised.
The gap of meeting AYP goals reflects simply whether a school and all of
its significant subgroups of students met a single benchmark for achievement in a
single year. A school not meeting AYP may have fallen short in every category, or
missed the mark narrowly by failing one of many criteria measured. In both cases,
however, the designation of not meeting AYP is the same. Jack O’Connell, State
Superintendent of Public Schools in California, writes about California’s 2003-2004
STAR test results and AYP in a News Release (August, 2004), “The fact that 317 of
our schools grew 30 points or more [API], yet failed to make the federal benchmark
[AYP], illustrates why I believe a growth model of accountability such as we have in
California more accurately reflects actual student learning.”
This phenomenon of schools achieving growth in math and reading scores on
standardized tests year-to-year, but failing to make AYP is not restricted to
California. Most states in the nation have schools and districts facing a similar
dilemma. A case in point is illustrated in Pennsylvania. In September 2003, the
Pennsylvania Department of Education released a final list of schools, identifying
those who met, and did not meet their goals for AYP (NEA, 2004). The publication
followed a period of appeals by the state’s 501 school districts, objecting to some
schools being labeled as “needing improvement” in the department’s initial report in
August. The appeals resulted in hundreds of schools being removed from the lists of
schools needing improvement and being recognized as having met their AYP goals.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
28
The numbers in August 2003 showed that 48.7% met AYP goals, while
51.3% were labeled as in need of improvement. After the appeals 1 month later, the
September 2003 figures showed that actually 61.3% of the schools in the state were
making progress and had met AYP, with 38.7% failing AYP goals. However, public
perception was that more than half of the public schools in Pennsylvania were
substandard (NEA, 2004). It is little wonder the general public exhibits ambivalence
for public schools.
But are the schools that comprise the 38.7% of schools needing improvement
truly failing to teach children proficiency in reading and math at the levels required
by NCLB? Not according to the data released by the Pennsylvania Department of
Education. The total number of schools meeting all AYP goals was 61.3%. The
number of schools meeting reading and math requirements, but not other AYP goals,
was 19.6%. The total number of schools achieving reading and math proficiency
goals was 80.1%. Of these, 66% of the schools failed to meet AYP goals in only one
of the other AYP requirements (disadvantaged student populations, student
participation, and graduation rate). This example illustrates the necessity to promote
a greater understanding of AYP’s negative consequences for school districts,
schools, and students, and should serve as a roadmap for correcting its flaws (NEA,
2004).
AYP, then, appears to be the most problematic provision of the No Child Left
Behind Act. The law requires that every child make progress toward a targeted
proficiency level, regardless of intellectual capability or disability. Realistically, not
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
29
all children are going to attain proficiency. Would it not be better, then, to continue
the National Assessments of Educational Progress (NAEP)/Educational Reform
philosophy of continuous growth as opposed to targeted goals (Learning First
Alliance, August, 2003)? The categorical reporting component (disaggregated
subgroups) of the law has the very real potential to keep so-called low-performing
schools in that status perpetually, or until the school doors are closed through the
Program Improvement process.
In many states, including California, ambitious proficiency levels were
established as a result of NAEP, before No Child Left Behind established severe
consequences for failure to meet designated target percentages. Schools were not
informed that the state’s goals would become the state’s standards to determine
student and school or district proficiency (Learning First Alliance, 2003). Allowing
states to set their own standards of proficiency was certainly appealing, with the
appearance of being state-friendly. However, major flaws in this provision have
surfaced by the fact that some states have set their standard as low as 8% or 9% to
avoid having schools in the “need of improvement” category. Other states,
California included, established high state goals; the target of proficiency is much
more elevated in these states. As Robert Linn (Linn & Haug, 2002) points out,
“States are not starting on a level playing field because of differences in
accountability systems. Content standards, performance standards and test rigor vary
from state to state. If current standards and tests are used to set AYP objectives,
some states will have much farther to go to reach AYP than others. This will not be
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
30
because students are achieving less but because of the differences in state
accountability systems, including varying definitions of proficient performance.”
One solution is to establish positive incentives for states to raise the
achievement levels of specific subgroups of students, or perhaps high-achieving
states that demonstrate continuing progress in reducing and closing achievement
gaps in identified populations should be exempted from many of the provisions of
the law, leaving the No Child Left Behind Act applicable to states where progress is
slow, backsliding, or nonexistent. Providing evidence that growth is taking place
by comparing what states say they are doing with the data to NAEP scores, could
reduce the need for extensive reporting and paperwork requirements of already
overburdened schools and school districts.
However, Robert Linn and Carolyn Haug (2002) point out that “ . . . the
NAEP achievement levels have been sharply criticized by several national panels of
both the National Academy of Education and the National Research Council that
have been asked to evaluate NAEP.” In addition to finding fault with the process
used to set the NAEP performance achievement levels, the National Academy of
Education Panel concluded “that the achievement levels were set unreasonably high”
(Shepard, Glaser, Linn, & Bohmstedt, 1993). However, using the NAEP basic level
as an independent monitor of state achievement, would help level the variations in
states’ assessment programs and performance standards.
NAEP relies on a small sampling of students from public schools in each
state and a few major cities. In California, persistent achievement gaps exist. In
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
31
2003, NAEP results indicate white students outscore Hispanic and African American
students significantly, and higher-income students outscore their lower-income
counterparts. Performance gaps were not significantly different from similar gaps in
1992 with the notable exception that African American fourth grade students
narrowed the gap in math by nine points (EdSource, 2003).
How can states determine if their tests adequately reflect their academic-
content standards, and what is the best way to determine the value that schools add to
student learning? One of the central tenets of standards-based education is that states
should align standards, tests and instruction, so that they all reflect the same learning
goals. Daniel Koretz (2004), professor at Harvard University’s graduate school of
education states, “We need a systematic research agenda addressing both assessment
design and accountability design.”
In particular, the need to focus more on classroom assessments that can
actually promote student learning is paramount; such assessments should not only
provide teachers with information about what to do next, but also help students
reflect on their own thinking.
Value-Added Analysis and Longitudinal Tracking
How, then, can schools and school districts overcome the challenges of AYP
objectives, and what strategies can be taken to implement more effective methods of
data collection? According to Robert Linn and Carolyn Haug (2002), a number of
states, including California, use the successive groups approach to compare
achievement of students. In this approach, the performance of students in one grade
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
32
is compared to the performance of students in the same grade from the year before.
Year-to-year changes in scores for successive groups of students, however, have a
great deal of volatility. The uncertainty in the scores is the result of measurement
and sampling error and nonpersistent factors that affect scores in one year but not the
next. Linn and Haug (2002) point out,
These comparisons of student performance at a grade level in different years,
rest on the implicit assumption that student characteristics that affect
achievement levels are relatively stable from year-to-year for students
attending a given school. This assumption is questionable for schools serving
neighborhoods whose demographic characteristics are changing rapidly . . .
(p. 31)
Change scores for students tested at a given grade from one year to the next,
can prove to be quite unreliable. Using data from the state of North Carolina, Kane
and Staiger (2001) estimated that a substantial part of the variability found in change
scores for schools was due to non-persistent factors that influence scores in one year
but not the other. Therefore, single point-in-time analyses may reflect demographics
rather than effectiveness, and cannot distinguish between schools that accelerate
skills and those that allow students to languish.
Cross-sectional measures (current status indicators) do not show whether
students entered with high or low skills or whether they have lost ground as a result
of instruction. Flicek and Wong (2003) characterize the cross-sectional percent-
proficient model as “one of the least valid evaluation methods.” Schools with high
percentages of historically low-performing groups (students in poverty, ethnic
minorities, Limited English Proficient students, special education students) tend to
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
33
be identified as failing to meet AYP more often than those with lower percentages of
these groups. The performance of these students is reflected in the overall group
performance and in each selected subgroup. Schools that serve primarily White,
English-speaking students who are not in poverty have higher results overall, and
frequently have subgroup numbers too low to report. The data do not show which
schools have been effective with the population that they serve, and often end up
measuring social differences in two successive groups of students rather than the
effect of the school (Baker & Linn, 2002; Buchanan, 2004; Kim & Sunderman,
2004).
In the absence of other measures, current status indicators are invalid and
potentially misleading, especially in schools and districts with widespread student
mobility. These indicators tend to be highly contaminated by the influence of
multiple educational settings.
Concerns such as these have led researchers to focus increasingly on value-
added analysis, an approach to analyzing and reporting test score data that addresses
many of the pitfalls of current status indicators. Value-added analysis uses statistical
methods that, in effect, separate out or adjust for the influence of non-school related
variables, i.e., students’ socioeconomic background, on academic growth.
Value-added assessment is not a new or different type of test. Rather, it is a
model used to statistically analyze test data to determine the influence of teachers,
schools, and school districts on student learning. Instead of comparing students to
each other or to an established level of proficiency, value-added assessment
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
34
compares students to themselves to determine if they are advancing academically,
and if so, at what pace. For example, a school that lifts its students from the 35th
percentile on national tests to the 50th percentile and using an accountability system
that uses value-added assessment, would prove to be more effective than a school
whose students consistently score at the 60th percentile.
The value-added statistical model uses test scores accumulated year-to-year
from each student to track change in achievement. This allows creation of academic
growth charts for each student’s progress, measuring the value added to the
knowledge the student already had. The curve is rarely even; the record will show
flat spots or spurts of accelerated learning. By calculating statistically significant
variances in a group of students’ test scores, determination can be made as to how
well a particular teacher, school, or district is educating a particular student (Meyer,
1997).
The most prominent value-added assessment model was developed by Dr.
William Sanders, a former statistics professor at the University of Tennessee. His
method, called the Tennessee Value-Added Assessment System (TVAAS), uses
mixed-model methodology, a type of statistical analysis developed originally for use
in agriculture (Sanders, 1998).
Essentially, value-added assessment is like a pre-and-posttest given to
students to determine what they have learned during a particular course of study.
Unlike those tests, however, value-added assessment seeks results that can be
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
35
compared across classrooms and years. Thus, the value-added model must be
overlaid on tests that have the following characteristics (Sanders, 1998):
1. The tests must be highly correlated with curricular objectives.
2. The tests must have sufficient stretch to measure the progress of both
previously low and high scoring students.
3. The tests must demonstrate appropriate reliability.
Most schools use value-added assessment with readily available standardized
achievement test results. These are norm-referenced tests, comparing one student
with the average performance of all students. But a sufficiently reliable and
consistent criterion-referenced test could be used instead, comparing students with an
established standard of achievement, and the closer it correlates to established
learning goals, the more valuable the results of the analysis will be.
Because students’ scores are compared with their own prior test scores,
external factors such as socioeconomic status are blocked out. Gathering the data
over several years accommodates for statistical variations, while the TVAAS model
is constructed in a way to allow for variables such as missed tests, transferred
students, skipped grades, and other complications (Sanders, 1998).
Critics of value-added analysis argue that, to be effective, an accountability
system must be easily understood, pointing to the fact that the advanced statistical
methods underlying value-added analysis are well beyond most educator’s grasp.
Drury and Doran (2003) argue, however, that the accountability stakes are too high
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
36
“to trade rigor and accuracy for simplicity.” Sophisticated statistical controls must
be used if the assessments of teachers, schools, or programs are to be accurate.
Others claim that value-added analysis inflates the degree of random error
involved in measuring student performance, thereby increasing the risk that teachers
or schools may be rewarded or punished for over or under estimates of their
students’ learning. But Rogosa et al. (1982), stresses that confidence in value-added
assessment results can be increased by a variety of methods, including averaging
data over several school years or incorporating multiple achievement measures in the
analysis. Texas, for example, rewards schools based on value-added achievement
gains calculated using two-stage regression analysis. The state does not hand out
awards based simply on decimal point differences among schools, but reward a
previously set percentage of top-ranking schools (Summers, 2002).
Because value-added indicators adjust for student background characteristics
associated with academic growth, some critics argue that they create lower
performance expectations for disadvantaged students and the schools that serve
them. Robert Meyer (1997), however, points out that because these schools start out
at a lower level of performance, they must cover more ground than other schools in
order to achieve the same end. Meyer (1997) observes, “This is a strength, not a
weakness, of the value-added approach” (p. 3).
Anita Summers (2002) furthers this argument by stating, “There is already
considerable evidence from several places—such as Tennessee and Florida, where
value-added analysis has been used for accountability purposes—that low-achieving
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
37
students are the main beneficiaries of the changes that occur when these techniques
are implemented. When low-achieving students are taught the same body of
knowledge over and over again, and when they are taught how to work under a time
constraint, they benefit. Value-added assessment techniques reveal that
information.”
Value-added assessment is based on student progress, not on the level of
achievement. Schools and teachers are accountable for how much students gain in
achievement. They are not given credit for students entering at a high level, or
penalized when students start far behind. In effect, value-added assessment takes
into account the influence that family income, ethnicity, and other circumstances
may have on students’ initial level of achievement (Ballou, 2002).
Linn et al. (2002), offer four approaches to reduce the magnitude of year-to-
year fluctuations or results due to differences in cohorts of students attending a
school. These approaches are specific to value-added analysis, and include:
1. Longitudinal tracking of students from year-to-year.
2. The use of “rolling averages” of two or more years of achievement
results.
3. The use of composite scores across subject areas and grades.
4. The use of separate grade by subject area results while not requiring
that all combinations show improvement (e.g., 5 out of 8 or 7 out of 10 possible
grade by subject combinations).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
38
Further, the benefits of value-added analysis, and a longitudinal data system for
tracking student achievement, are many. Some of these include:
1. Cohorts of students could be followed throughout their educational
careers to show the impact of current reform efforts.
2. True gains in learning could be used to access impacts rather than just
status indicators.
3. States could follow the progress of the numbers of highly mobile
students (Linn et al., 2002).
Conclusion
The problem for California is its focus on school-level growth. Under the
current accountability system, schools rather than students are given growth targets
and those targets require minimal annual growth. This essentially assures that even
if schools hit their yearly growth targets, many students may not be proficient by the
2013-2014 deadline of the No Child Left Behind Act. Also, the state’s growth targets
for the percentage of students hitting the proficient mark encourage schools to focus
year-by-year on those students closest to the proficiency benchmark, potentially
ignoring the lowest achieving students (Doran & Izumi, 2004).
As Drury and Doran (2003) point out:
It makes little sense to continue to define AYP solely in terms of the
percentage of students crossing an arbitrary bar of “proficiency,” while
ignoring the growth that occurs within broad performance categories. This is
tantamount to measuring a child’s height with a yardstick but acknowledging
growth only when his or her height exceeds 36 inches. Relying exclusively
on average test scores and proficiency cut-offs to measure AYP can lead to
unintended consequences, such as leading teachers and administrators to
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
39
target those students most likely to cross the proficiency cut point, while
leaving others without focused instruction, (p. 3)
Jason Millman (1997) concludes in his book on school accountability,
Grading Teachers, Grading Schools: Is Student Achievement a Valid Evaluation
Measure? that,
Any method of evaluating teachers and schools with an eye toward making
them accountable should be fair to the teachers and schools, should be
comprehensive in terms of the types of learning objectives measured, should
be competitive in relation to other methods of evaluating teachers and schools
for an accountability purpose, and should not cause undesirable effects when
used properly, (p. 243)
Hence, a value-added approach to analyzing student data is more valid, more precise,
and more equitable than the current “successive groups” approach currently used in
California.
Providing necessary longitudinal annual growth information for individual
students to meet the eventual goal of subject matter proficiency, schools will have to
ensure that all students are hitting their growth targets rather than concentrating on
groups of students just under the proficiency bar, thus changing the focus of schools
to achievement progress among all students, and the ultimate goal of leaving no child
behind.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
40
CHAPTER 3
RESEARCH METHODOLOGY
Introduction
The purpose of this study, was to determine whether schools in California are
fairly and accurately measured as demonstrated under the federal guidelines of the
No Child Left Behind Act of 2001 through the Academic Performance Indicators
(API) and Adequate Yearly Progress (AYP) accountability plans, as compared to the
impact that value-added indicators might have on school achievement, specifically,
longitudinal tracking of student test scores. The study evaluated the impact that
value-added indicators focus on the growth in student achievement from one grade
level to the next for given cohorts of students, rather than on the trend over time in
average test scores for students at a given grade level; in other words, on longitudinal
as opposed to cross-sectional student data.
Research Question
“Teaching and learning are reciprocal processes that depend on and affect
one another. Thus, the assessment component deals with how well the students are
learning and how well the teacher is teaching,” (Kellough & Kellough, 1999, p. 417).
Under No Child Left Behind, accountability through standardized testing is measured
by summative assessment only. These high stakes standardized tests given to all
students in Grades 2 through 11, has intensified the domination of summative tests
over curriculum and instruction. Summative assessment is an attempt to summarize
student learning at some point in time; in California, this translates into the year-end
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
41
STAR testing. High quality summative data can shape what schools offer their
students; however, these tests are not designed to provide the immediate,
contextualized feedback useful for helping the teacher and student during the
learning process. Research conducted by Black and William (1998) supports the
conclusion that summative assessments tend to have a negative effect on student
learning, and that comparing students competitively causes low achieving students to
believe that they cannot learn.
How, then, can assessments be effectively utilized? Kellough et al. (pp. 418-
419) outline seven purposes of assessment:
1. To assist student learning.
2. To identify students’ strengths and weaknesses.
3. To assess the effectiveness of a particular instructional strategy.
4. To assess and improve the effectiveness of curriculum programs.
5. To assess and improve teacher effectiveness.
6. To provide data that assist decision-making.
7. To communicate with and involve parents.
Kellough et al. (1999) further state that,
Because the welfare and, indeed, the future of so many people depend on the
outcomes of assessment, it is impossible to overemphasize its importance.
For a learning endeavor to be successful, the learner must have answers to
basic questions: Where am I going? Where am I now? How do I get where I
am going? How will I know when I get there? These questions are integral
to a good program of assessment, (p. 421)
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
42
The American Association for Higher Education (AAHE) has established the
following nine principles to guide assessment’s implementation:
1. The assessment of student learning begins with educational values.
2. Assessment is most effective when it reflects an understanding of
learning as multidimensional, integrated, and revealed in performance over time.
3. Assessment works best when the programs it seeks to improve have
clear, explicitly stated purposes.
4. Assessment requires attention to outcomes but also and equally to the
experiences that lead to these outcomes.
5. Assessment works best when it is ongoing, not episodic.
6. Assessment fosters wider improvement when representatives from
across the educational community are involved.
7. Assessment makes a difference when it begins with issues of use and
illustrates questions that people really care about [sic].
8. Assessment is most likely to lead to improvement when it is part of a
larger set of conditions that promote change.
9. Through assessment educators meet responsibilities to students.
Given the current, artificially stratified nature of the K-12 system, summative
assessment is unavoidable. However, it is incumbent upon not only educators, but
those forces behind the regulations of No Child Left Behind, to minimize adverse
effects that such assessment might have on students, schools, and school districts
throughout California. Indeed, to maximize the efficacy of summative assessment,
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
43
several factors need to be considered, including authenticity, variety, volume,
validity and reliability.
STAR testing in California consists of both criterion-referenced and norm-
referenced tests, with each type weighted for API indicators. The CAT/6 Survey is a
norm-referenced test, and assesses the achievement of basic academic skills in key
subjects that are commonly taught in public schools throughout the United States. It
is used to compare the performance of California students to the performance of
students throughout the nation. The CAT/6 tests comprise 20% of the weight of the
API in the elementary grades. The California Standards Tests, on the other hand, are
criterion-referenced tests and are designed to tell how well students are doing with
respect to the California academic standards. The academic standards describe what
students should know and be able to do at each grade level. Scores are designated
through five performance levels or tiers, including advanced, proficient, basic, below
basic, and far below basic. Proficiency is the goal for all students under No Child
Left Behind. The CSTs comprise 80% of the API weights. For purposes of this
study, only the California Standards Tests were used for the comparative study.
Thus, the following research question is generated:
Question. What patterns or trends for schools can be discovered by tracking
individual student gains over long periods of time?
Hypothesis. Longitudinal tracking of student test scores will demonstrate
more valid estimates of the school, system, and teacher effects on the academic gains
of students.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
44
Research Design
Currently, California employs the successive groups approach to compare
achievement of students. Scores of this approach tend to be highly volatile, and
assume student characteristics that affect achievement levels are relatively stable
(Linn, 2002). Furthermore, single point-in-time analyses reflect a school’s
demographics rather than its effectiveness (Kane & Steiger, 2001). On the other
hand, value-added analysis uses statistical methods (including longitudinal tracking)
to adjust for the influence of non-school variables on academic growth.
School District A is a school district in northern California. It consists of
four K-5 elementary schools, two K-8 elementary schools, one 6-8 grade middle
school, four K-8 charter schools, and one charter high school. Table 2 below
indicates the percentages of student demographics for the entire district:
Table 2. School District A Demographics
Student factor District State
Subgroups:
English language learners 2.3% 25.6%
Special education students 8.5% 10.2%
Low income students 43.6% 48.7%
Compensatory education (Title 1) 64.8% 47.9%
Student ethnicity:
Asian 3.2% 8.1%
Filipino 0.4% 2.5%
Hispanic 5.5% 45.2%
African American 2.5% 8.3%
White 83.8% 33.7%
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
45
Table 2 shows that compared to districts statewide, School District A has far
fewer English Language Learners, about as many low income students, and more
students requiring compensatory educational service. The table also shows that the
district student ethnicity is primarily White.
This study focused on the evaluation of the individual California Standards
Test scores in language arts and math of those students in School District A in
Grades 2 through 5 during the 2001-2004 school years. Only those students who
participated in the testing all 3 years were measured. Following a quantitative
research design in six case studies of School District A’s six elementary school’s
Grade 2 through 5 California Standards Test (CST) scores, a comparison of school
level measurements in comparable year API scores with longitudinal data of students
tested in same 3-year period was evaluated.
Research Population
Students were tracked on the California Standards Test (CST) in
English/Language Arts and math and growth was measured on the mean scaled
scores of these tests for all Grades 2 through 5 students tested in the 2001-2002,
2002-2003, and 2003-2004 school years. CST norms were used to compare year to
year results. These scores were equated to enable comparison over time.
Six schools in School District A were part of this study; three of these
schools have been deemed “underperforming” through Adequate Yearly Progress
indicators, and two were in “Program Improvement” status although API scores
grew each school year. A longitudinal study comparison was proposed as an
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
46
alternative to the API scores for all six school sites, citing criteria of a longitudinal
distinction to the cross-sectional data of the current API/AYP indicators.
Case Study 1 (School 1)
School 1 is a K-5 school with 352 students enrolled during the 2003-04
school year. Two percent of this student population is comprised of English
Language Learners, and forty seven percent are low income students. The students
whose native language is not English speak Punjabi or Spanish at home. The ethnic
background of the school is as follows:
African American 3%
Asian American 1%
Hispanic 4%
White 92%
Table 3 below shows the school’s API/AYP reports for 2002-2004.
Table 3. API/AYP School 1
% Cal-
Works
% Free
and
Reduced
Lunch
AMOs and
Participation
(Phase I)
API
(Phase II)
Graduation
Rate
(Phase II)
All Phase
I and
Phase II
2002 13% 42% Base No
(664)
N/A Base
2003 13% 45% Yes Yes
(732)
N/A Yes
2004 14% 50% Yes Yes
(756)
N/A Yes
Cal-Works and Free and Reduced Lunch data are used as the poverty
indicators for Title I purposes. The 2002 AYP Base Report does not identify a
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
47
school as not making AYP. The results in the report relate only to participation rates
and the percentages of students proficient or above in English/Language Arts and
math. The report includes data on the API, graduation rates, and whether the 2002
starting points are above/at or below appropriate Annual Measurable Objectives
(AMOs), 13.6% in E/LA and 16% in math. In the case of School 1, it did not meet
its API in 2001, and was below in growth targets of the base values of the AMOs in
2002. Therefore, it entered year 1 of Program Improvement (PI) in 2001-2002, and
exited from PI status in 2003-2004.
Case Study 2 (School 2)
School 2 is a K-5 school with 603 students enrolled during the 2003-2004
school year. Two percent of this student population is comprised of English/
Language Learners, and 34% are low income students. The students whose native
language is not English speak Spanish or Mien at home. The ethnic background of
the school is as follows:
African American 2%
Asian 4%
Hispanic 5%
White 89%
Table 4 shows the school’s API/AYP reports for 2002-2004.
Cal-Works and Free and Reduced Lunch data are used as the poverty
indicators for Title I purposes. The 2002 AYP Base Report does not identify a
school as not making AYP. The results in the report relate only to participation rates
as the percentages of students proficient or above in English/Language Arts and
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
48
Table 4. API/AYP School 2
% Cal-
Works
% Free and
Reduced
Lunch
AMOs and
Participation
(Phase I)
API
(Phase II)
Graduation
Rate
(Phase II)
All
Phase I
and
Phase II
2002 22% 40% Base No
(768)
N/A Base
2003 24% 36% Yes Yes
(815)
N/A Yes
2004 14% 40% Yes Yes
(822)
N/A Yes
math. The report includes data on the API, graduation rates, and whether the 2002
starting points are above/at or below appropriate Annual Measurable Objectives
(AMOs), 13.6% in E/LA and 16% in math. In the case of School 2, it met its API in
all three years, as well as the base values of the AMOs.
Case Study 3 (School 3)
School 3 is a K-5 school with 292 students enrolled during the 2003-2004
school year. Two percent of this student population is comprised of English
Language Learners, and 68% are low income students. The students whose native
language is not English speak Spanish or Lao at home. The ethnic background of the
school is as follows:
African American 3%
Asian 6%
Hispanic 7%
White 84%
Table 5 shows the school’s API/AYP reports for 2002-2004.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
49
Table 5. API/AYP School 3
% Cal-
Works
% Free
and
Reduced
Lunch
AMOs and
Participation
(Phase I)
API
(Phase II)
Graduation
Rate
(Phase II)
All
Phase I
and
Phase II
2002 22% 66% Base Yes
(687)
N/A Base
2003 20% 66% Yes Yes
(735)
N/A Yes
2004 27% 71% Yes Yes
(754)
N/A Yes
Cal-Works and Free and Reduced Lunch data are used as the poverty
indicators for Title I purposes. The 2002 AYP Base Report does not identify a
school as not making AYP. The results in the report relate only to participation rates
and the percentages of students proficient or above in English/Language Arts and
math. The report includes data on the API, graduation rates, and whether the 2002
starting points are above/at or below appropriate Annual Measurable Objectives
(AMOs), 13.6 % in E/LA and 16% in math. In the case of School 3, it met its API in
all 3 years, as well as the base values of the AMOs.
Case Study 4 (School 4)
School 4 is K-8 school with 683 students enrolled during the 2004-2004
school year. Three percent of this student population is comprised of English
Language Learners, and 40% are low income students. The students whose native
language is not English speak Spanish, Lao, or Punjabi at home. The ethnic
background of the school is as follows:
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
50
African American 3%
Asian 6%
Hispanic 7%
White 85%
Table 6 below shows the school’s API/AYP report for 2002-2004.
Table 6. API/AYP School 4
%Cal-
Works
% Free and
Reduced
Lunch
AMOs and
Participation
(Phase I)
API
(Phase II)
Graduation
Rate
(Phase II)
All
Phase I
and
Phase II
2002 14% 42% Base No
(701)
N/A Base
2003 19% 38% Yes Yes
(727)
N/A Yes
2004 17% 41% Yes Yes
(744)
N/A Yes
Cal-Works and Free and Reduced Lunch data are used as the poverty
indicators for Title I purposes. The 2002 AYP Base Report does not identify a
school as not making AYP. The results in the report relate only to participation rates
and the percentages of students proficient or above in English/Language Arts and
math. The report includes data on the API, graduation rates, and whether the 2002
starting points are above/at or below appropriate Annual Measurable Objectives
(AMOs), 13.6 % in E/LA and 16 % in math. In the case of school 4, it met its API in
all 3 years, as well as the base values of the AMO’s.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
51
Case Study 5 (School 5)
School 5 is a K-5 school with 334 students enrolled during the 2003-2004
school year. Seven percent of this population is comprised of English Language
Learners, and 83% are low income students. The students whose native language is
not English speak Spanish at home. The ethnic background of the school is as
follows:
African American 4%
Asian 4%
Hispanic 8%
White 84%
Table 5 below shows the school’s API/AYP reports for 2002-2004.
Table 7. API/AYP School 5
% Cal-
Works
% Free
and
Reduced
Lunch
AMOs and
Participation
(Phase I)
API
(Phase II)
Graduation
Rate
(Phase II)
All Phase
I and
Phase II
2002 51% 100% Base No
(625)
N/A No
2003 44% 77% Yes Yes
(673)
N/A Yes
2004 35% 85% Yes Yes
(692)
N/A Yes
Cal-Works and Free and Reduced Lunch data are used as the poverty
indicators for Title I purposes. The 2002 AYP Base Report does not identify a
school as not making AYP. The results in the report relate only to participation rates
and the percentages of students proficient or above in English/Language Arts and
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
52
math. The report includes data on the API, graduation rates, and whether the 2002
starting points are above/at or below appropriate Annual Measurable Objectives
(AMOs), 13.6% in E/LA and 16% in math. In the case of school 5, it did not meet
its API in 2001 and 2002, and entered year 1 of Program Improvement (PI) in 2002-
2003, and exited from PI status in 2003.
Case Study 6 (School 6)
School 6 is a K-8 school with 342 students enrolled during the 2003-2004
school year. Three percent of this student population is comprised of English
Language Learners, and 56% are low income students. The students whose native
language is not English speak Lao, Spanish, or Thai at home. The ethnic background
of the school is as follows:
African American 2%
Asian 6%
Hispanic 6%
White 86%
Table 8 shows the school’s API/AYP reports for 2002-2004.
Table 8. API/AYP School 6
% Cal-
Works
% Free and
Reduced
Lunch
AMOs and
Participation
(Phase I)
API
(Phase II)
Graduation
Rate
(Phase II)
All Phase
I and
Phase II
2002 38% 91% Base No
(625)
N/A Yes
2003 24% 71% Yes Yes
(646)
N/A Yes
2004 24% 76% Yes No
(651)
N/A No
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
53
Cal-Works and Free and Reduced Lunch data are used as the poverty
indicators for Title I purposes. The 2002 AYP Base Report does not identify a
school as not making AYP. The results in the report relate only to participation rates
and the percentages of students proficient or above in English/Language Arts and
math. The report includes data on the API, graduation rates, and whether the 2002
starting points are above/at or below appropriate Annual Measurable Objectives
(AMOs), 13.6% in E/LA and 16 % in math. In the case of school 6, it did not meet
its API in 2001 or in 2004.
Sample Size
This study employed non-probability purposive sampling of each Grade 2
through Grade 5 student who participated in the STAR testing (specifically
California Standards Tests in English/Language Arts and math) for 3 consecutive
years at the six school sites. The population breakdown for both the cross sectional
data and the longitudinal data collected by district and school site is illustrated in
Table 9.
Data Collection
The study was of a quantitative, non-experimental, causal-comparative
approach to research through the collection of longitudinal post-test data of student
test scores from the California Standards Tests (CST) for 3 consecutive years in
English/Language Arts and 3 consecutive years in math. Quantifying the overall
progress of individual student scores in Language Arts and math was through the
comparison of the mean scaled scores one year minus the mean scaled scores in the
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
54
Table 9. D istrict and School Site Population Data
Cross School School School School School School
sectional District 1 2 3 4 5 6
2002 1164 206 306 133 255 146 118
2003 1195 198 293 150 264 161 129
2004 1106 170 277 159 227 153 120
Cohort
matched
2002 807 152 220 96 180 91 68
2003 807 152 220 96 180 91 68
2004 807 152 220 96 180 91 68
next year (simple gain score), and equating these levels to allow for comparisons
over time. These scores were then compared to each site’s grade level measurements
in the comparable year Academic Performance Indicator (API) scores using the
mean scaled scores of the CST measurements.
Instrumentation
The researcher used archival data, specifically student scores in the California
Standards Test (CST). The STAR Program consists of four components:
1. California Standards Tests (CSTs)
2. California Alternate Performance Assessments (CAPA)
3. California Achievement Test, Sixth Edition Survey (CAT/6 Survey)
4. Spanish Assessment of Basic Education, Second Edition (SAB E/2)
The most important components of the STAR Program are the CSTs and the
CAPA. The CAPA assesses the performance of students with significant cognitive
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
55
disabilities on subsets of California’s Academic Content Standards for English/
Language Arts and math. However, for purposes of this evaluative study, only CSTs
were used. The CSTs measure student performance on California’s Academic
Content Standards, and identify students who achieve at each performance level:
advanced, proficient, basic, below basic, or far below basic. The state’s target is for
all students to score proficient or advanced.
The CSTs carry the most weight (80%) for calculating school and district
Academic Performance Indexes (APIs). Figure 5 below shows the API gains for
■ 2001-2002
□ 2002-2003
□ 2003-2004
Figure 5. API Gains: School District A
The CSTs are also used in determining Adequate Yearly Progress (AYP)
toward meeting the federal No Child Left Behind requirement to have all students
District A and its six school sites for 2002, 2003, and 2004.
ill
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
56
score proficient or above by 2014. Because the CSTs are the most important
component of the STAR Program, teachers and administrators are encouraged by
California Department of Education to use the CST results to determine if
modifications may be needed in instructional programs to better help students
become proficient on the California Academic Content Standards.
CST records are contained in the databases of School District A, Measures
and Edusoft. Measures is based on a relational database, where information is
organized into several key tables. Fundamental tables include students, tests, scores,
and teachers. Edusoft is similar, but allowed for more extensive reporting, including
longitudinal data reports for 3 years of test scores for district and school level
inquiries.
Powerful assessment tracking is possible because of ID numbers assigned to
each student, each test (for each grade and date given), each score, and each teacher.
The Power Filter allowed the researcher to select a group of students by indicating
criteria, i.e., grade level, school, type of report (cross sectional, cohort matched, etc.).
The Power Filter and Select Tests features allowed the researcher to select the tests
to process for the selected group of students.
Data Analysis
This study used descriptive statistics to summarize and describe a group of
scores, specifically in the areas of Language Arts and math for 2nd through 5th grade
students during 3 consecutive years of testing. Non-standardized regression
correlation analysis was used to determine if a significant relationship exists between
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
57
longitudinal tracking of student achievement as a more accurate assessment of school
quality than the current cross-sectional approach used in California.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
58
CHAPTER 4
RESULTS
Introduction
The purpose of this study was to determine whether schools in California are
fairly and accurately measured as demonstrated un the federal guidelines of the No
Child Left Behind Act of 2001 through the Academic Performance Indicators (API)
and Adequate Yearly Progress (AYP) accountability plans, in comparison to the
impact that value-added indicators might have on school achievement, specifically
longitudinal tracking of student test scores. The impact of value-added indicators
that focus on the growth in student achievement from one grade level to the next for
given cohorts of the same students rather than on the trend over time of average test
scores for students at a given grade level was evaluated.
This chapter describes the research findings of the study, focusing on the data
that was yielded from the California Standards Tests (CSTs) in English/Language
Arts and math over a 3-year period, comparing cross-sectional student data to
longitudinal value-added data. Student scores from six California schools in the
same school district were tracked on the CSTs mean scaled scores for all Grades 2
through 5 students tested. CST norms were used to compare year-to-year results,
and analyzed in six case studies to determine the assessment of progress each school
achieved year-to-year for 3 consecutive years. Mean scaled scores were used
because of their reliability and correlation to this study. The California Department
of Education’s STAR Post-Test Guide (2004, p. 6) explains that:
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
59
Scaled scores provide a more precise measure of performance than raw
scores and are used to equate the tests at each grade level from year-to-year.
The equating is used to ensure that differences in the difficulty levels of the
CSTs from year-to-year do not affect scaled score ranges and performance
levels. The scaled score performance level cut-points are held constant from
year-to-year for each grade level and content area.
Case Study 1 (School 1)
School 1 is a K-5 school whose population of students tested in Grades 2
through 5 was 206 in 2001-2002,198 in 2002-2003, and 170 in 2003-2004. These
numbers indicate a declining enrollment in these grade levels at this site. The
longitudinal (cohort matched) population, that is, the number of students who tested
in all 3 test years was 152. This indicates a fairly low mobility rate, or a high
percentage of a static core grouping of 73.8% in 2001-2002, 76.8% in 2002-2003,
and 89.4% in 2003-2004. Figure 6 below shows the combined mean scaled scores of
the 2002-2004 CSTs in English/Language Arts and math, comparing the cross-
sectional data (current status indicators) to the cohort matched data. The school’s
API scores for the same 3 years are shown in figure 6 next to the scaled scores.
While the data clearly indicate higher scores each year in both comparisons,
it is evident that the cohort matched (longitudinal) data surpass the cross sectional
data slightly in 2002 and 2003, but more significantly in 2004. The API scores
indicate growth each year at this school. This school entered year one of Program
Improvement status in 2001-2002, and exited PI status in 2003-2004.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
60
1 Cross
Sectional
i Cohort
Matched
■ API
2002 2003 2004
2002 2003 2004
Figure 6. Cross-sectional and Cohort Matched Data: School 1.
Case Study 2 (School 2)
School 2 is a K-5 school whose population of students tested in Grades 2
through 5 was 306 in 2001-2002,293 in 2002-2003, and 277 in 2003-2004. The
numbers indicate a slight decline in enrollment in these grade levels at this site. The
longitudinal (cohort matched) population, that is, the number of students who tested
in all 3-test years was 220. This indicates a fairly low mobility rate, or a high
percentage of a static core grouping of 71.9% in 2001-2002,75% in 2002-2003, and
79.4% in 2003-2004. Figure 7 shows the combined mean scaled scores of the 2002-
2004 CSTs in English/Language Arts and math, comparing the cross sectional data
(current status indicators) to the cohort matched data. The school’s API scores for
the same 3 years are shown in the chart next to the scaled scores.
The data show significant growth in the cross sectional comparison, but the
cohort matched data take a significant dip in 2004. The API scores for this school
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
61
2002 2003 2004
■ C ross
Sectional
■ Cohort
M atched
2002 2003 2004
®API
Figure 7 . Cross-sectional and Cohort Matched Data: School 2
indicate very high achievement levels with significant improvement in 2003 (815)
and a slight increase in 2004 (822).
Case Study 3 (School 3)
School 3 is a K-5 school whose population of students tested in Grades 2
through 5 was 133 in 2001-2002, 150 in 2002-2003, and 159 in 2003-2004. These
numbers indicate a slight increase in enrollment in these grade levels each year. The
longitudinal (cohort matched) population, that is, the number of students who tested
in all 3 years, was 96. This indicates a fairly high mobility rate, or a low percentage
of a static core grouping of 69.9% in 2002, 64% in 2003, and 60.4% in 2004.
Figure 8 shows the combined mean scaled scores of the 2002-2004 CSTs in
English/Language Arts and math, comparing the cross-sectional data (current status
indicators) to the cohort matched data. The school’s API scores for the same three
years are shown in the chart next to the scaled scores chart.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
62
■ C ross
Sectional
■ C ohort
M atched
2002 2003 2004
Figure 8. Cross-sectional and Cohort Matched Data: School 3
While the data clearly indicate higher scores each year in both comparisons,
the cohort matched (longitudinal) data surpass the cross sectional data in 2003, but
there is a significant increase in 2004. These data correspond to the school’s API
trend of significant improvement in test scores in 2003.
Case Study 4 (School 4)
School 4 is a K-8 school whose population of students tested in Grades 2
through 5 was 255 in 2001-2002, 264 in 2002-2003, and 227 in 2003-2004. These
numbers indicate a significant decline in enrollment in these grade levels in 2004.
The longitudinal (cohort matched) population, that is, the number of students who
tested in all 3-test years, was 180. This indicates a fairly low mobility rate, or a high
percentage of a static core grouping of 70.6% in 2001-2002, 68.1% in 2002-2003,
and 79.3% in 2003-2004. Figure 9 shows the combined mean scaled scores of the
2002-2004 CSTs in English/Language Arts and math, comparing the cross-sectional
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
63
■ C ro ss
Sectional
■ Cohort
M atched
700
750
730
690
720
710
740
■ API
2002 2003 2004
680
670
2002 2003 2004
Figure 9. Cross Sectional and Cohort Matched Data: School 4
data (current status indicators) to the cohort matched data. The school’s API scores
for the same 3 years are shown in the figure next to the scaled scores figure.
While the data clearly indicate higher scores each year in both comparisons,
the cohort matched (longitudinal) data show a significant gain in 2004. The API
scores indicate growth each year at this school.
School 5 is a K-5 school whose population of students tested in Grades 2
through 5 was 146 in 2001-2002,161 on 2002-2003, and 153 in 2003-2004. These
numbers indicate an increase in enrollment in these grade levels in 2003, and a slight
decline in 2004. The longitudinal (cohort matched) population, that is, the number
of students who tested in all 3 years, was 91. This indicates a significantly high
mobility rate, or a low percentage of a static core grouping of 62.3% in 2001-2002,
Case Study 5 (School 5)
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
64
56.5% in 2002-2003, and 59.5% in 2003-2004. Figure 10 shows the combined mean
scaled scores of the 2002-2004 CSTs in English/Language Arts and math, comparing
the cross-sectional data (current status indicators) to the cohort matched data. The
school’s API scores for the same 3 years are shown in the chart next to the scaled
scores chart.
I C ro ss
Sectional
l C ohort
M atched
2002 2003 2004
2002 2003 2004
Figure 10. Cross-sectional and Cohort Matched Data: School 5
While the data clearly indicate higher scores each year in both comparisons, it is
evident that the cohort matched (longitudinal) data surpass the cross-sectional data
significantly in 2002, and slightly in 2003 and 2004. This school did not meet its
API targets in 200 land 2002, and entered year one of Program Improvement (PI) in
2002-2003. It was exited from PI status in 2003. The API scores indicate gains all 3
years.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
65
Case Study 6 (School 6)
School 6 is a K-8 school whose population of students tested in Grades 2
through 5 was 118 in 2001-2002, 129 in 2002-2003, and 120 in 2003-2004. These
numbers indicate an increase in enrollment in 2003, with a slight decline in 2004.
The longitudinal (cohort matched) population, that is, the number of students who
tested in all 3-test years was 68. This indicates a very high mobility rate, or a low
percentage of a static core grouping of 57.6% in 2001-2002, 52.7% in 2002-2003,
and 56.7% in 2003-2004. Figure 11 shows the combined mean scaled scores of the
2002-2004 CSTs in English/Language Arts and math, comparing the cross-sectional
data (current status indicators) to the cohort matched data. The school’s API scores
for the same 3 years are shown in the figure next to the scaled scores chart.
The data clearly indicate high scores each year in the cohort matched
(longitudinal) fields, while the cross sectional data show an increase from 2002 to
2003, but a significant decrease in 2004. The API scores indicate a large jump from
2002 to 2003 and a slight increase in 2004. This school did not meet its API targets
in 2004.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
66
2002 2003 2004
B Cross Sectional
I Cohort
Matched
655
650
645
640
635
630
625
620
615
610
r r ~
■ < F
i
. / r
■ 'T
2002 2003 2004
Figure 11. Cross-sectional and Cohort Matched Data: School 6
3 API
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
67
CHAPTER 5
DISCUSSION
The purpose of this study was to determine whether schools in California are
fairly and accurately measured under federal guidelines of the No Child Left Behind
Act of 2001through the Academic Performance Indicators (API) and Adequate
Yearly Progress (AYP) accountability plans, in comparison to the impact that value-
added indicators might have on school achievement, specifically, longitudinal
tracking of student test scores. This study evaluated the impact of value-added
indicators that focus on the growth in student achievement from one grade level to
the next for given cohorts of the same students rather than on the trend over time in
average test scores for students at a given grade level. Longitudinal value-added
data as opposed to cross-sectional student data was compared.
Currently, California employs the successive groups approach, or cross-
sectional measures, to compare achievement of students. The score data are reported
as “snapshots” of student performance, or current status indicators. These indicators
represent the average score of students enrolled in a district, school, grade level, or
classroom, and are assessed using percentile rankings. While these indicators may
be useful in describing performance for a given student population in a given year,
research demonstrates that these indicators actually provide less information about
I
school quality.
Researchers (Carlson, 2002; Linn, 2002) report that scores of this approach
are highly volatile, and that differences between the composition of one year’s class
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
68
of 3rd graders and the next year’s class of 3rd graders can cause changes in
aggregate test results. Thus, the current API accountability plan provides a
misleading picture of academic progress with schools and districts. It does not
measure growth in individuals over time; it merely compares overall achievement of
successive classes over time.
Other researchers in this area (Kane & Staiger, 2001) show that single point-
in-time analyses reflect demographics rather than effectiveness; that is, that a
substantial part of the variability found in change scores for schools, was due to non-
persistent factors that influence scores in 1 year but not the other. Cross-sectional
measures do not show whether students entered with high or low skills or whether
they have lost ground as a result of instruction. Researchers Flicek and Wong (2003)
characterize the cross-sectional percent proficient model as one of the least valid in
terms of evaluation methods.
Value-added analysis uses statistical methods to adjust for the influence of
non-school variables on academic growth. Longitudinal tracking is part of value-
added analysis, and compares students to themselves to determine academic growth.
Following a quantitative research design, this study examined six schools Grades 2
through 5 California Standards Tests (CSTs) scores, and compared the school level
measurements in comparable year API scores for 3 consecutive years with
longitudinal data of students tested in the same 3-year period.
Student scores were measured on different tests of the CSTs, namely
English/Language Arts and math. Using combined mean scaled scores of the CSTs
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
69
to compare year-to-year results, the combined scaled scores were equated through
the Measures and Edusoft databases to enable comparison over time.
School District A is a school district in Northern California comprised of four
K-5 elementary schools, two K-8 elementary schools, one 6-8th grade middle school,
four K-8 charter schools, and one charter high school. For purposes of this study,
only the four K-5 elementary schools and 2 K-8 elementary schools Grades 2
through 5 CST scores were evaluated, and each was organized as a case study.
Overall, school district A has a good API and AYP ranking in the state, and
the district demographics show this district to be comprised primarily of White
students (83.8% compared to 33.7% statewide) and has about as many low-income
students when compared to state demographics, far fewer English Language
Learners, and more students requiring compensatory educational service (Title 1).
However, taken individually, each school site’s demographics varied moderately.
Of the six schools, one has reached and surpassed the 800 bar of the
California API measurement scale, the state goal of “proficiency.” Three of the
schools have been deemed “underperforming” and were in or face Program
Improvement status resulting from not meeting AYP for 2 years of more.
Interestingly, two of these “underperforming” schools are school-wide Title I
schools.
School 1 did not meet its API targets in 2001 or 2002 and was labeled a
Program Improvement school in 2002, enduring the punitive sanctions required
under the federal guidelines of No Child Left Behind for failure to meet AYP for 2
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
70
consecutive years. Because of substantial growth in API scores in 2003 (from 664 to
732), it exited PI status in 2004. But was this school truly “failing” its students?
According to the longitudinal analysis of this school’s students who tested all 3
years, it did not fail. Its longitudinal data showed significant growth all three years
for these cohort matched students. This school had a fairly low mobility rate, or a
high percentage of a static core grouping (73.8% in 2002 to 89.4% in 2004) of
students who tested in the 2002, 2003, and 2004 years. Therefore, it would appear
that this school used its student test data to adjust its curriculum to align with the
state standards, and succeeded in improving its test scores significantly in the cohort
matched students during these test years.
Another case in point is perhaps the most important to this study. School 6 is
a K-8 school comprised of 56% low-income students. It is a designated school-wide
Title I school. According to the cross-sectional data, this school did not make its
API targets in 2002 (625) or in 2004 (651). Even though its API score improved
from 2003 (646) to 2004 (651) by 5 points, according to the API regulations
regarding “target growth,” it needed to score 8 points higher than the 2003 score, or
654 in 2004. Should this school fail to meet its targets again in the 2005 test year, it
will be placed in Program Improvement status. Again, has this school failed?
A careful analysis of the data show that this school has made the most
significant gains of any of the other school within the six schools evaluated in this
study when looked at its longitudinal data comparisons with the cohort matched data
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
71
of the combined mean scaled scores of the CSTs. The results of this comparison are
best indicated in the Figure 12.
340i
B Cross Sectional
■ Cohort Matched
2002 2003 2004
Figure 12. Cross-sectional and Cohort Matched Data: School 6
The longitudinal (cohort matched) population, that is, the number of students
who participated in testing in all 3 years was 68. This indicates a very high mobility
rate at this school site, or a very low static core grouping of 57.6% in 2002, 52.7% in
2003, and 56.7% in 2004. The cross-sectional, or current status indicators, show an
increase from 2002 to 2003, but a large decrease in 2004. However, for that static
core grouping of students, significant growth has occurred from year-to-year. It
would appear that this school is, in essence, being “punished” for the scores of
students whose mobility in and out of the school during the test years affected the
school’s API scores. Is this failure of the academic program and student
achievement at this school site? Has this school been fairly and accurately measured
as demonstrated under the federal guidelines No Child Left Behind? Using a value-
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
72
added approach to measure the school’s effectiveness proves it is successful with
those students with year-to-year attendance.
School 2, on the other hand, is a high-achieving school, with different set of
demographics than School 6. School 2 is a larger school, with 34% low-income
students. Its longitudinal (cohort matched) number of students who participated in
testing all 3 years was 220. This indicated a fairly low mobility rate, or a high
percentage of a static core grouping of 71.9% in 2002, 75% in 2003, and 79.4% in
2004. Its API scores ranged from 768 in 2002 to 822 in 2004. It scored above the
800 “proficiency” bar in both 2003 and 2004. In order for a school to meet its
growth targets each year once it has hit the 800 mark, it must show improvement of
at least 1 point the following year.
However, a severe discrepancy exists between School 2’s cross-sectional data
and cohort matched data when compared to School 6. Figure 13 show these data of
School 2.
■ Cross Sectional
■ Cohort Matched
2002 2003 2004
Figure 13. Cross-sectional and Cohort Matched Data: School 2
360 f
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
73
The data indicate comparable growth in both the cross-sectional and cohort
matched figures in 2002 and 2003, but the cohort-matched data take a significant dip
in 2004. Cross-sectional data indicate high achievement levels on the CSTs for all
students in Grades 2 through 5 who participated in the testing. In fact, the
achievement level exceeds the API growth differences significantly. These data
speculatively indicate that this school’s measure of success benefits from its mobile
students, that is, those students who likely come from a higher income base, with
probable more parental support than those students who are in the highly mobile
population of School 6.
Implications
The implications of this study provide additional data for a comprehensive
(state wide) evaluation of longitudinal tracking versus the current status indicators of
the successive cohort approach currently used in California. The California
Legislature’s Joint Committee’s Master Plan for Education in California (2000),
specifically proposes that, “The State should develop and report yearly on a
comprehensive set of educational indicators, constructed from the data provided by
an integrated, longitudinal, learner-focused data system and from other school-level
data about educational resources, conditions and learning opportunities” (p. 46).
With the implementation of the California School Information Service (CSIS)
database in 2006, it will be able to track all California public school students no
matter where they are enrolled in California.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
74
Under the federal legislation No Child Left Behind Act of 2001, every state
system of public education must implement a statewide accountability program that
measures the progress of its students and schools over time through the collection
and analysis of disaggregated data. In response, the California Legislature enacted
SB 145 3, which established two key components necessary for a long-term
assessment and accountability system;
1. Assignment of a unique student identifier to each K-12 pupil enrolled in
a public school program or in a charter school that will remain with the student
throughout his or her academic “career” in the California public school system.
2. Establishment of a longitudinal database of disaggregated student
information that will enable state policy-makers to determine the success of its
program of educational reform.
This study provides evidence that the achievement gap is narrowed through
a value-added approach to measurement. Ethnic groups, students of low
socioeconomic status, and special education students benefit from value-added
analysis because it measures an individual’s growth over time. True gains in
learning can be used to access impacts rather than just status indicators. As pointed
out by Meyer (1997), current status indicators:
1. Reflect the combined influence of family, background, community,
and years of prior schooling on student achievement, and unfairly judge schools that
serve disproportionately high numbers of disadvantaged students.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
75
2. Reflect the cumulative impact of school and non-school related
influences on achievement over multiple years, and therefore, can be skewed in any
given year by prior influences on student learning.
3. Fail to localize performance to a specific classroom or grade level,
making them less useful as a means of evaluating reform initiatives, teacher
effectiveness, and program innovation.
4. Tend to be highly “contaminated” by the influence of multiple
educational settings due to widespread student mobility in and out of different
districts.
A more accurate picture of school achievement for schools with high
mobility rates, low API scores, or schools with high numbers of at-risk students can
be gleaned through the implementation of a value-added approach. Indeed, as
Rogosa et al. (1982) point out, confidence in value-added assessment results can be
increased through a variety of methods, including averaging data over several school
years or incorporating multiple achievement measures in the analysis.
Schools serving more disadvantaged students will typically have higher, not
lower, value-added performance goals. Because these schools generally start at a
lower performance level, they must cover more ground than other schools in order to
achieve the same end. As Meyer (1997) observes, “This is a strength, not a
weakness of the value-added approach.” Hence, the current punitive approach to
low-performing schools may be unjustified.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Lastly, the “good news/bad news” implications for low-performing and high-
performing schools can be further broken down through data by grade levels or even
classrooms. These data can be used to guide school policies and programs,
professional development, and even teacher evaluation.
Limitations
While the data collected and studied from the combined scaled scores of the
California Standards Tests over 3 years from six schools in Northern California
provide additional information about the merits of value-added analysis, there are
limitations to the study. Findings from this particular study cannot be generalized to
other schools, districts or states, because the databases needed to perform
longitudinal or value-added analyses may not necessarily be in place.
Other limitations to this study include the fact that the schools did not have a
high level, as compared to state averages, of student diversity. The factions that
nearly equaled that of the state were the percentages of low-income students and
special education students. Title I percentages were much higher than the state
average, but the ethnic breakdown was significantly lower in student diversity.
There are other factors besides longitudinal tracking that can influence the areas of
interest within this study. Alignment of curriculum to assessment measures, texts
and materials utilized in the classrooms, teacher methodology and practices, and
levels of home support also affect student performance, participation, and
development.
permission of the copyright owner. Further reproduction prohibited without permission.
77
Recommendations
More longitudinal studies are needed at school, district, and state levels.
Studies need to be conducted in districts and schools with high demographics of
English Language Learners, high percentages of diverse ethnicities, low API scores,
high numbers of at-risk students, and schools and districts in Program Improvement
status. Additionally, more years of longitudinal data need to be collected to assure
statistical validity.
Perhaps a two-tiered model of accountability needs to be in place in
California, one of status and one of value-added. Low-performing schools need to
be able to show that they are having an impact on student learning and school-wide
achievement. Certainly, rates of improvement should factor into the accountability
system of measurement.
One new piece of API legislation slated to be implemented in this year’s API
base scores will be the mobility exclusion rule. Under previous requirements, this
rule applied only at the district level. Under the new ruling, it will apply at the
school level, and states specifically that test scores included in calculating a school’s
API will be only those of students who were counted as part of the school enrollment
in the annual California Basic Educational Data System (CBEDS) data collection for
the current fiscal year and who were continuously enrolled in the school during that
year.
Several issues need to be considered to implement a fair and reliable
accountability system, most notably that the system is focusing on the “right” goals,
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
that the system identifies schools that truly need to improve, and that the system is
geared to improved student learning. California has invested billions of dollars in
improving schools over the past decade. To be able to report yearly on a
comprehensive set of educational indicators with a fair and reliable learner focused
data system, is paramount to the goals of the No Child Left Behind Act.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
REFERENCES
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
80
REFERENCES
Baker, E. L., & Linn, R. L. (2002). Validity Issues for Accountability Systems, CSE
Technical Report 585. National Center for Research on Evaluation,
Standards, and Student Testing. Los Angeles, CA: UCLA.
Ballou, D. (2002, Summer). Sizing up Test Scores. Education Next, 2.
Bianchi, Alison (2003). A New Look at Accountability: ‘Value-Added ’ Assessment.
Forecast, New York State School Boards Association: Latham, NY (2003).
Black, P., & William, D. (1998, October). Inside the Black Box: Raising Standards
Through Classroom Assessment. Phi Delta Kappan. Retrieved March 26,
2004, from www.pdkintl.org/kappan/kbla9810.htm
Brookhart, S .M. (1999). Educative assessment: Designing assessments to inform
and improve student performance. San Francisco, CA: Jossey-Bass.
Buchanan, B. (2004, February). Defining ‘Adequate Yearly Progress’. American
School Board Journal.
California Department of Education (CDE). (2004). California standardized testing
and reporting: STAR test guide, Sacramento, CA: Educational Testing
Services.
California School Information Service. (2004, February 11). CSIS Frequently Asked
Questions About the Statewide Student Identifiers (SSID) Program (p. 3).
Retrieved January 18, 2005, from http://www.csis.kl2.ca.us/faq
California Department of Education (CDE). (2003, November) A YP Guidelines.
Retrieved January 12, 2004, from www.cde.gov/ta/ac/ay/index.asp
California Department of Education (CDE). (2004, August 26). Office of Jack
O’Connell, State Superintendent of Public Instruction, Letter to
Superintendents. LEA and School Improvement Non-Regulatory Guidance,
Sacramento, CA.
California Department of Education, Office of Jack O’Connell, State Superintiendent
of Public Instruction, August 31, 2004 News Release. API Scores Increase,
Meet Federal AYP Criteria. Retrieved September 1, 2004, from
www. cde. ca.gov/nr/ne/yr04
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
81
California Legislature’s Joint Committee, Master Plan for California Education,
draft 1, p.46, Sacramento, CA: California Department of Education.
Carlson, D. (2002). Four Methods o f Judging School Quality and Improvement
Relationships and Implications. Center for the Study of Evaluation, National
Center for Research on Evaluation, Standards, and Student Testing, UCLA,
Los Angeles, CA.
Council of State School Officers. (2002, December). Implementing the State
Accountability System Requirements Under the No Child Left Behind Act o f
2001. Washington, DC: Council of State School Officers.
Doran, H., & Izumi, L. T. (2004). Putting education to the test: A value-added
model for California. San Francisco, CA: Pacific Research Institute.
Drury, D., & Doran, H. (2003). The Value o f Value-Added Analysis. Policy
Research Brief, Vol. 3, No.l, 1-4. Alexandria, VA: National School Boards
Association.
EdSource. (2003, November). California Assessment News. Retrieved September
1, 2004, from www.edsource.org/edu_ass_cal.cfm
Feuer, M. J., Holland, P. W., Green, B. F., Bertenthall, M. W., & Gemphill, C. (Eds).
(1999). Uncommon measures: Equivalence and linkage o f educational tests.
Washington, DC: National Academy Press.
Flicek, M., & Wong, K. (2003). The Challenge o f Using Large-Scale Assessment to
Hold Schools Accountable. Lake Oswego, OR: Northwest Evaluation
Association.
Kane, T. J., & Steiger, D. O. (2001). Volatility in School Test Scores: Implications
for Test Based Accountability Systems. Brookings Papers on Educational
Policy, 2002.
Kellough, R. D., & Kellough, N. G. (1999). Secondary school teaching: A guide to
methods and resources; Planning for competence. Upper Saddle River, NJ:
Prentice-Hall.
Kim, J., & Sunderman, G. L. (2004). Large mandates and limited resources: State
response to the no child left behind act and implications for accountability.
Cambridge, MA: The Civil Rights Project at Harvard University.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
82
Koretz, D. (2004). Toward a Frameworkfor Validating Gains Under High-Stakes
Conditions. CSE Technical Report 551, December, 2001. Center for the
Study of Evaluation, National Center for Research and Evaluation. Standards
and Student Testing (CRESST), Los Angeles, CA: UCLA.
Learning First Alliance. (2003, August). A Practical Guide to Talking With Your
Community About No Child Left Behind and Schools in Need o f
Improvement. Des Moines, I A: Iowa Department of Education.
Linn, R. (1999, Spring). Standards Based Accountability Systems, Adequate Yearly
Progress: Absolutes, Wishful Thinking, and Norms. Newsletter of the
National Center for Research on Evaluation, Standards, and Student Testing:
The CRESST Line, Los Angeles, CA: UCLA
Linn, R., Baker, E., & Betebenner, D. (2002). Issue Paper: Accountability.
Denver, CO: Education Commission of the State.
Linn, R., & Haug, C. (2002, Spring). Educational Evaluation and Policy Analysis,
24(1), 29-36.
Linn, R., & Haug, C. (2002). Stability o f School Building Accountability Scores and
Gains, CSE Technical Report 561. Center for the Study of Evaluation,
Standards, and Student Testing, Graduate School of Education and
Information Studies, Los Angeles, CA: UCLA.
McCall, M., Kingbury, G., & Olson, A. (2004). Individual Growth and School
Success. Lake Oswego, OR: Northwest Evaluation Association.
McMillan, J. H. (2000). Classroom Assessment: Principles and Practice for
Effective Instruction, Pearson Technology Group. Retrieved March 26, 2004,
from www.fairtest. org/examarts/winter99/k-formsa3. html.
Meyer, R. H. (1995). Educational Performance Indicators: A Critique. Institute for
Research on Poverty. Madison, WI: University of Wisconsin.
Meyer, R. (1997). Value added indicators of school performance: A primer.
Economics o f Education Review, 16, 3.
Millman, J. (1997). Grading Teachers, Grading Schools: Is Student Achievement a
Valid Evaluation Measure. Thousand Oaks, CA: Corwin Press Inc.
National Education Association. (2004, January). AYP Absurdities. Retrieved
September 1, 2004, from www.lae.org/nea/ayp/absurdities.html
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
83
Pausch, L. M., & Popp, M. P. (1997). Assessment o f Information Literacy: Lessons
from the Higher Education Assessment Movement. Retrieved March 26,
2004, from www.ala.org/acrl/paperhtm/d30.html
Pellegrino, J. W., Jones, L. R., & Mitchell, K. J. (Eds). (1999). Grading the
Nation’ s Report Card: Evaluating NAEP and Transforming the Assessment
o f Educational Progress, Washington, DC: National Academy Press.
Rogosa, D., Brandt, D., & Zimowski, M. (1982). A growth curve approach to the
measurement of change. Psychological Bulletin, 92(3), 726-748.
Sanders, W., & Horn, S. (1994). The Tennessee Value-Added Assessment System
(TVAAS): Mixed Methodology in Educational Assessment. Journal o f
Personnel Evaluation, 9 ,299-311.
Shepard, L. A., Glaser, R., Linn, R. L., & Bohrnstedt, G. (1993). Setting
Performance Standards for Student Achievement: A Report o f the National
Academy o f Education Panel o f the Evaluation o f the NAEP Trial State
Assessment, An Evaluation o f the 1992 Achievement Levels, Stanford, CA:
Stanford University, National Academy of Education.
Summers, A. (2002). Expert measures. Stanford, CA: Hoover Institution, Stanford
University.
The White House. (2001). No child left behind. Washington, DC: The White
House.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Evaluation of the effects of a continuous improvement program on special education student achievement

PDF

An evaluation of the REMEDY project with an emphasis on sustainability: A Safe Schools /Healthy Students federal grant

PDF

An evaluation of probation supervision and its role in performance of students identified as living in poverty in Shasta County Court and Community Schools

PDF

A quantitative and qualitative study of computer technology and student achievement in mathematics and reading at the second- and third-grade levels: A comparison of high versus limited technolo...

PDF

Adult school student achievement on the California High School Exit Examination: Are adult schools ready for the challenge?

PDF

Into, Through and Beyond: Evaluation of migrant education reading tutor program innovations

PDF

Family group conference: An innovative approach to truancy in schools. A case study

PDF

Effects of the STAR testing program on teachers and the curriculum

PDF

Evaluation of a two -period double math program on the academic achievement of underperforming seventh grade math students

PDF

Cognitive coaching training for master teachers and its effects on student teachers' ability to reflect on practice

PDF

Implementation of performance indicators for the district office of the Dos Palos -Oro Loma Joint Unified School District

PDF

Corrective action plan for program improvement of a middle school

PDF

Institutional factors affecting academic persistence of underprepared community college freshmen

PDF

A comparative study of the role of the principal in conventional public schools and in charter schools

PDF

A comparative analysis of academic achievement for CalWORKs students in a K--12 public school system

PDF

Effects of an integrated content and methods course on preservice teachers' beliefs and efficacy toward mathematics

PDF

Increasing student retention through benchmarking and organizational improvement

PDF

Evaluation of CLAD training in northern California

PDF

Implementing computer assisted instruction in a multilevel -multigraded classroom evaluation /action plan

PDF

A study of student retentiveness, achievement, and program completion in a school -to -career program

Asset Metadata

Creator Anderson, Cynthia M. (author)

Core Title Evaluation of the effects of longitudinal tracking of student achievement to assess school quality

School Rossier School of Education

Degree Doctor of Education

Degree Program Education

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag Education, administration,education, tests and measurements,OAI-PMH Harvest

Language English

Contributor Digitized by ProQuest (provenance)

Advisor McLaughlin, Michael (committee chair), Cohn, Carl (committee member), Hocevar, Dennis (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c16-387275

Unique identifier UC11340979

Identifier 3180487.pdf (filename),usctheses-c16-387275 (legacy record id)

Legacy Identifier 3180487.pdf

Dmrecord 387275

Document Type Dissertation

Rights Anderson, Cynthia M.

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA