University of Southern California Dissertations and Theses

Use of accountability indicators to evaluate elementary school principal performance

(USC Thesis Other)

Use of accountability indicators to evaluate elementary school principal performance

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content

USE OF ACCOUNTABILITY INDICATORS TO EVALUATE ELEMENTARY
SCHOOL PRINCIPAL PERFORMANCE

by

Chiae Byun-Kitayama

__________________________________________________________________

A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION

May 2012

Copyright 2012 Chiae Byun-Kitayama
ii
DEDICATION
For my husband, Glen Kitayama who has been loving, encouraging, and
incessantly compassionate; my two beautiful daughters, Sejin and Yejin, who
have been my inspiration and source of energy; my mom and my sister who have
always been unconditional supporters; and amazing God.
I am eternally grateful for all of you who made it possible for me to dream
and to achieve.
iii
ACKNOWLEDGMENTS
To my chair, Dr. Dennis Hocevar; to my committee members, Dr. Pedro
Garcia and Dr. Michael Escalante. To the USC Doctoral Support Center,
especially to Dr. Linda Fischer; to my professors, especially to Dr. Rudy Crew; to
my mentors, Dr. Lori Kim, Chin Kim, Grace Yoon, Mary Kurzeka, and Robert
Bilovsky. To my faithful friends, Dr. Hyerim Choi, Joy Park, Anne Lee, Allison
Choi, and Grace Cho. To my fellow 2009 Cohort, especially to the thematic
group, and to my LAUSD colleagues, especially Rey Munda who took time to
edit my paper. I am forever indebted for your kindness, help, support, direction,
leadership, guidance, motivation, encouragement, and inspiration throughout the
process.
iv
TABLE OF CONTENTS
DEDICATION........................................................................................................ii
ACKNOWLEDGMENTS .....................................................................................iii
LIST OF TABLES.................................................................................................vi
LIST OF FIGURES ..............................................................................................vii
ABSTRACT.........................................................................................................viii
CHAPTER 1 THE PROBLEM.............................................................................. 1
Background of the Problem ................................................................................ 2
Statement of the Problem.................................................................................... 7
Purpose of the Study ........................................................................................... 9
Importance of the Study.................................................................................... 11
Definitions of Terms ......................................................................................... 12
Organization of the Study ................................................................................. 16
CHAPTER 2 LITERATURE REVIEW .............................................................. 17
Overview of Accountability.............................................................................. 17
Impacts of Accountability on Student Achievement ........................................ 24
Roles of Principal on School Accountability.................................................... 26
Instructional Leadership and Student Achievement ..................................... 30
Principal Evaluation to Support Student Outcomes.......................................... 36
Accountability School Status Indicators........................................................... 43
Academic Performance Index........................................................................... 44
AYP................................................................................................................... 45
Value-Added Methods ...................................................................................... 47
Discussion ......................................................................................................... 49
CHAPTER 3 METHODOLOGY ........................................................................ 51
Research Design................................................................................................ 52
Instrumentation and Procedures........................................................................ 57
Achievement. ................................................................................................ 57
California Standards Test.............................................................................. 58
Academic Performance Index....................................................................... 59
Value Added Method........................................................................................ 62
Academic Growth Over Time........................................................................... 63
Instrumentation: Surveys ................................................................................. 64
School Experience Surveys........................................................................... 64
Procedure: Ratings and Surveys ....................................................................... 65
v
Analysis............................................................................................................. 67
Limitations (Threats to Validity) ...................................................................... 67
CHAPTER 4 RESULTS ...................................................................................... 69
Descriptive Results ........................................................................................... 70
Stability of API and AGT Scores Over Multiple Years.................................... 72
Inter-correlation of the Three Dimensions of School Experience Surveys ...... 76
Correlation of Independent Variables and Dependent Variables...................... 80
CHAPTER 5 DISCUSSION................................................................................ 88
Summary of Findings........................................................................................ 89
API Scores..................................................................................................... 91
School Experience Survey ............................................................................ 92
Biased and Unbiased School Accountability Indicators ............................... 93
Limitations ........................................................................................................ 94
Implications for Practice ................................................................................... 95
Conclusions....................................................................................................... 96
REFERENCES...................................................................................................... 98
APPENDIX SCHOOL EXPERIENCE SURVEY .......................................... 107

vi
LIST OF TABLES

Table 1. Frequency and Distribution of Literature on Principal
Evaluation by Publication Type From 1980 to 2010 ............................ 6
Table 2. Student Enrollment History of LAUSD.............................................. 56
Table 3. Test Weights ....................................................................................... 61
Table 4. School Experience Survey Elements .................................................. 66
Table 5. Descriptive Table ................................................................................ 71
Table 6. Stability Coefficients of the Academic Performance Indices
for LAUSD Elementary Schools......................................................... 73
Table 7. Stability of Coefficients of the AGT Scores in LAUSD..................... 74
Table 8. Descriptive Table of the Academic Performance Indices
for LAUSD Elementary Schools......................................................... 75
Table 9. Descriptive Table of the AGT Scores in LAUSD............................... 75
Table 10. Correlations Between School Experience Dimensions....................... 77
Table 11. Correlation Between School Experience Surveys and Accountability
Indicators............................................................................................. 78
Table 12. Correlation Among Independent Variables and Dependent Variables82
Table 13. Correlation Among Dependent Variables........................................... 86
Table 14. LAUSD API 2006 to 2010.................................................................. 91

vii
LIST OF FIGURES

Figure 1. Conceptual Model I....................................................................... 53

Figure 2. Conceptual Model II ..................................................................... 54
Figure 3. Student Ethnic Percentage (2009)................................................. 57

viii
ABSTRACT
Recently, the federal government has pressured states to add student
growth data as a part of the evaluation system. State legislatures in New York
and Colorado have passed legislation to revamp teacher and principal evaluation
to include student growth data. Numerous researchers acknowledged the critical
impact of school principals on student achievement, and asserted the need to
develop a coherent evaluation that would assess and expand the capacity of
principals.
This study examined accountability indicators designed to evaluate
elementary school principals. A quantitative approach was used to study
correlation between six accountability indicators (Academic Performance Index
(API) scores, similar-school ranks, residualized difference (RD) scores, Value
Added Method (VA) scores, Academic Growth over Time (AGT) scores, and
school experience surveys) and three external factors (socio-economic status,
mean parent education level, and percentage of English Learners).
The participants included elementary schools in the Los Angeles Unified
School District from 2005 to 2010. The data files were collected from the public
websites, and were downloaded and analyzed.
The study’s conclusions are: API scores were extremely stable, but AGT
scores were not nearly as stable over the years. There was a steady growth in API
over the years. Employee and student survey results were strongly correlated, but
ix
neither correlated to parent ratings. None of the school experience surveys were
correlated with test-based accountability indices. API scores were strongly
related to all three external factors, but VAM scores, AGT scores, RD scores, and
school experience survey results were not related to them.
API was found to be biased against principals who were working at a
school with one or more disadvantageous external factors. Thus, using API as a
part of a principal performance evaluation would be unfair.
Results of this study highlights similar-school ranks, RD scores, VAM
scores, AGT scores, and school experience surveys as potential accountability
indicators that were unbiased and fair to be included in principal performance
evaluation system. RD scores were found to be the most reliable accountability
indicator. Due to their low cost and simplicity, RD scores should be considered
as possible alternative to complex and expensive value-added systems.
1
CHAPTER 1
THE PROBLEM

Education then, beyond all other devices of human origin, is the great
equalizer of the conditions of men, the balance-wheel of the social machinery.
Horace Mann

The founding fathers believed that education was a key to build productive
citizens in a democratic society. Thomas Jefferson believed that a man’s destiny
should not be determined by the family he was born to, rather a man could
determine his own destiny through a guaranteed universal education (Honeywell,
1969). Since the inception of the first public school in Boston in the 1600’s,
public education has been an essential part of America. It all started with a one
room schoolhouse with one teacher who was responsible for educating all of the
children in the village. Four centuries later, however, school has become a
complicated and complex bureaucracy. The nation’s second largest school
district, the Los Angeles Unified School District serves 671,648 students in 1,092
schools with $ 5.1 billion budget in the 2010-2011 school year (Los Angeles
Unified School District, 2011a). The responsibility of the adults in the school
buildings has also evolved as the system became increasingly convoluted.
2
Today, the primary responsibility of creating an effective school is placed
on school principals. They are facing ever-increasing high stakes accountability
for student achievement because school principals are critical to school success
(Kearney, 2010). It is vital to support and nurture the particular principal
behaviors and leadership skills that foster teacher quality and student achievement
through precise, useful, reasonable, coherent, appropriate, and dependable
performance evaluation (Davis, 2010). Historically principal evaluations have
been used as routine procedures to hold principals accountable to renew their
contract, or to substantiate personnel and/or bargaining units requirements. As
more researchers conclude that school leadership is a key component of school
improvement, it is imperative to develop a coherent evaluation that assesses and
expands the capacity of school leaders (Wallace Foundation, 2009). The purpose
of this study was to evaluate the reliability and validity of using accountability
indicators and school experience survey results as a part of the principal
performance evaluation. In particular, potential methods to evaluate principals in
a way to improve principal effectiveness were examined.

Background of the Problem
School principals wear multiple hats. They are expected to perform
sufficiently both instructionally and managerially. According to Portin, Shen, and
Williams (1998), traditional principals’ roles were ensuring building maintenance
3
and repair, maintaining a safe and orderly campus, fulfilling requests from
teachers and staff, conducting teacher evaluations, balancing budgets, managing
student and staff disciplinary issues, complying with district/state/ federal
mandates, and more. As years passed, untraditional duties started to be added to
the principals’ list of responsibilities. In the 1990s, with the school reform
movement, principals were expected to make collaborative school site decisions
involving all stakeholders, to become a change agent, a personnel specialist, and a
problem solver while trying to grasp the ongoing school climate changes such as
increased diversity in student population, socioeconomic, stratification, and
family complexity.
Before No Child Left Behind (NCLB) (United States Department of
Education, 2002), principals focused their attention on managerial aspects of the
job because it was more explicit and visible to their superiors (Portin, Shen, &
Williams, 1998). Principals often were not responsible for the student outcomes,
especially when schools were only accountable for 10% of achievement variation,
and the rest was determined by students’ background based on the Coleman
Report (Marzano, 2003). However, the Nation at Risk Report in 1982 (National
Commission of Excellence in Education, 1983) shocked America, and American
people began to doubt the educational system and its ability to educate students
compared to its international counterparts. This outcry for a better and more
effective public education leads to bipartisan efforts to enact an ultimate
4
educational act--NCLB Act of 2002 (United States Department of Education,
2002). The pendulum started to swing towards principal responsibility for
instruction as more research pointed towards their impact on school achievement.
The NCLB Act has changed the realm of a multifaceted educational field.
The job descriptions for school principals changed forever. No longer can a
school principal sustain his/her job by complying with the managerial
responsibilities, but his/her survival depends on whether they make the Annual
Yearly Progress goals or not. Currently, principals have to answer to the district
superiors as well as to the community, realtors, parents, and newspaper media
regarding their ability to improve publicly available student outcomes.
Furthermore, there is an increase in interest in improving principals’ effectiveness
through the principal evaluation because of principals’ direct and indirect impact
on student achievement (Sun & Youngs, 2009).
Under the NCLB Act of 2002, any principal can be under scrutiny of
sanction or reconstitution depending on meeting the accountability goals that are
set by the federal and state governments. The pressure of accountability
continued in 2008 when the new Democratic administration enacted an education
initiative, Race to the Top (RTT). The Education Recovery Act funds RTT as
part of American Recovery and Reinvestment Act of 2009 (United States
Department of Education, 2009). The purpose of the RTT is to encourage states
5
to adopt rigorous educational reforms by funding the top scoring states. The RTT
defines an effective principal as
A principal whose students, overall and for each subgroup, achieve
acceptable rates (e.g., at least one grade level in an academic year) of
student growth. States, Local Education Agency (LEA), or schools must
include multiple measures, provided that principal effectiveness is
evaluated, in significant part, by student growth. (United States
Department of Education, 2009, p.12)
Subsequently, highly effective principals are defined as ones who can
produce one and a half grade-level growth within an academic year from all
students (United States Department of Education, 2009). States are required to
propose a principal evaluation plan that includes student growth data that
measures the effectiveness of a principal.
Compared to teacher evaluation, there has been little research done on
principal evaluations, and there is a lack of in-depth study, literature, and research
to provide comprehensive theoretical perspectives on principal evaluation (Davis,
2010). There are 1,612 articles and 1,347 doctoral dissertations on the topic of
teacher evaluation where as there are only 68 articles and 107 doctoral
dissertations on the topic of principal evaluation (Davis, 2010). Table 1 shows
the frequency distribution of studies done on principal evaluation. This asserts
the dire need for a broader range of studies and research on principal evaluation to
improve the policies, implications, implementation, procedure, performance, and
outcomes to foster effective leadership practices. Furthermore, the existing
6
studies do not provide ultimate solutions for effective evaluation protocols (Davis,
2010).

Table 1

Frequency and Distribution of Literature on Principal Evaluation by Publication
Type From 1980 to 2010

Publication Type

Total

1980 to 1989

1990 to 1999

2000 to 2010

Original research
studies

30(44%)

20%

23%

57%

Narrative based on
research by others

38(56%)

21%

42%

37%

All publication

68(100%)

21%

34%

45%

Source: Davis (2010)

Davis (2010) summarizes the common findings and themes from
implementation studies
(a) evaluation systems were typically constructed around various
performance criteria rather than outcomes, (b) evaluations were loosely
linked to professional standards, (c) most evaluation systems used a
variety of methods to gather and analyze performance, and (d) most
evaluation systems lacked reliability or validity, and were unevenly
applied (p. 10).
The missing link of effective leadership was revamped in the RTT as it
tried to rectify the current principal systems through reinforcement as an
educational act (United States Department of Education, 2009). States continue
7
to look for the optimal evaluation system that fosters successful school principals
not only to obtain the funds from the federal government, but also to improve the
failing educational system.

Statement of the Problem
Much research has been done to assert the critical impact of school
principals, particularly in their instructional leadership role in student
achievement (Darling-Hammond & Rothman, 2011; Gentilucci & Muto, 2007;
Kearney, 2010; Marks & Printy; 2003, Marzano, 2003; Sun & Young, 2009). For
many years, instructional leadership roles were not viewed as the most important
job of a school principal. However, the imposed school-accountability pressures
from the state and federal governments revamped the roles of school principals
forever. For professional survival, school principals depend on making the
Adequate Yearly Progress (API) objectives, and for California principals, in
addition to the federal accountability, they must meet the state accountability
standards, the Academic Performance Index (API) goals. The RTT redefined a
highly effective principal as a person who can improve one and a half grade level
of student academic growth based on state assessment, and other supplemental
measures such as “high school graduation rates; college enrollment rates;
evidence of providing supportive teaching and learning conditions, strong
instructional leadership, and positive family and community engagement; or
8
evidence of attracting, developing, and retaining high numbers of effective
teachers” (United States Department of Education, 2009, p.12). Moreover, RTT
requires the states to improve principal evaluation systems by using constructive
feedback, using student growth data as a part of the evaluation, and using the
evaluation for personnel decisions (United States Department of Education,
2009).
Nevertheless, there is lack of empirical, comprehensive, and relevant
studies done on principal evaluation (Davis, 2010). The limited number of studies
and journal articles currently available point to a common predicament that most
principal evaluation systems developed by school districts lack reliability and
validity, consistent methods and effective tools to evaluate the principals.
Furthermore, most Local Education Agencies (LEA) provide inadequate training
for the principal evaluators, and have a lack of communication between the
superintendents and principals regarding the purpose of evaluation and
performance expectations (Davis, 2010). In other study, researchers found that
there was disconnect from a conceptual framework based on literature on
effective principal leadership and instruction (Porter et al., 2010). Effective
principal evaluations are critical to enhance principals’ performances at the school
site that, in turn, promote school improvement for all stakeholders. Principal
evaluations should be congruent to their job demands as an instructional leader
because evaluation is a powerful tool that guides its constituents to desirable
9
behaviors and outcomes. Moreover, more empirical research is needed to identify
how states and districts can adequately support principals through evaluations
designed to maximize the potential of their impact on student outcomes.

Purpose of the Study
In this study, public assessment data on elementary schools in Los
Angeles, California were used to examine the correlation between six
accountability indicators: API scores, Similar-school ranks, residualized
difference scores, Value Added Method scores, Academic Growth over Time
scores, and school experience survey scores, and three external factors were used:
socio-economic status, mean parent education level, and percentage of English
Learners. API scores and Similar-school ranks were used as a part of the state
and federal accountability systems. Value Added Method scores for each school
were published in newspapers as indicators of school accountability. Academic
Growth over Time scores and school experience survey scores were a part of the
accountability measure within the Los Angeles Unified School District (Los
Angeles Unified School District, 2011b).
More specifically, this study examined the extent of the six following
accountability indicators: (a) API scores, (b) Similar-school ranks, (c)
Residualized Difference Scores (actual scores minus expected achievement), (d)
Value Added Method (VAM)—school effectiveness, (e) Academic Growth over
10
Time (AGT)—school effectiveness, and (f) School Experience survey scores to
stability over the years, changes, and inter-correlation among the variables. The
correlation of the six accountability indicators to percentage of Socio-economic
status (SES), mean parent education level and percentage of English Learners
within LAUSD was examined. Furthermore, the reliability and validity of using
student test scores to evaluate principals were analyzed. The following research
questions were posed:
1. To what extent are API scores and AGT scores stable over
multiple years, and has a change in the mean API and AGT scores
been observed in LAUSD?
2. To what extent are the three dimensions of school experience
surveys (employees, parents, and students) inter-correlated, and do
these dimensions correlate with test-based accountability indices?
3. To what extent are the six accountability school indicators (API
scores, similar-school ranks, residualized difference scores, VAM
scores, AGT scores, and the school experience survey scores)
correlated to SES, parent education level, and percent of English
Learners?
4. To what extent are accountability school indicators (API scores,
similar-school ranks, residualized difference scores, VAM scores,
11
AGT scores, and the school experience survey scores) inter-
correlated?
API measures are known to be biased against the schools with low socio-
economic status, low parent education level, and high percentage of English
learners. Thus API should be used with caution as a school accountability
indicator (Hocevar, 2011; Powers, 2003; Schochet & Chiang, 2010). In this
study, quantitative data were used to comprehend how performance of principals
were evaluated to improve their leadership skills, to increase student achievement,
and to examine unbiased accountability indicators.

Importance of the Study
School Accountability has changed many facets of the educational realm,
especially for the school principals. Many experienced principals profess that the
principal job used to be enjoyable, but not anymore. The principals are
scrutinized by the multi-layered accountability from all stakeholders regardless of
their actual performances. Moreover, the state assessment results have become
the primary concern for most principals more than any other principal
responsibility because the principals are judged by publicly reported API scores
and AYP indicators by the media, parents, and districts.
Albeit all principals do not have equal opportunities to improve their
schools regardless of external factors, RTT recognizes highly effective principals
12
as those who can raise measurable student outcomes such as assessment scores,
graduation rates, college acceptance rates, and teacher retention rate without
consideration of the school context. The federal government is also urging states
to adopt a strenuous principal evaluation system that includes outcomes and
scores to measure the effectiveness of principals. Hence, the unintended
consequences will be enormous if evaluations are unreliable or invalid.
At the district and state levels, the information from a principal evaluation
can be critical in terms of improving desirable behaviors and outcomes by the site
principals. The information can help districts make critical decisions such as
retaining, hiring, demoting, and promoting principals. Also, the information can
help the principals reflect on their practices to make necessary modifications and
to increase their school’s performance. An improved and relevant performance
evaluation can reduce the pressure off the principals so that they can find
satisfaction in their jobs. Moreover, the information can be used to guide
legislators to implement and to create policy that fosters school improvement.
Finally, this study can be added to the number of scarce studies done on principal
evaluation.

Definitions of Terms
The following are operational definitions of the key terms that are used
throughout the study.
13
Academic Growth over Time(AGT). The Los Angeles Unified School
District chose AGT as a system to measure the effectiveness of schools. The
AGT is a value-added model that controls for external factors such as student’s
prior CST performance, grade level, gender, race/ethnicity, low-income status,
English Language Learner (ELL) status, Individualized Education Plan (IEP)
status, homelessness, and mobility (Los Angeles Unified School District, 2011b).
LAUSD has chosen to use a five-color system to interpret the AGT results. Based
in statistics, these colors indicate whether or not student growth was far above,
above, at, below or far below the prediction or the district average. The results
are color-coded based on the location of the result and the associated confidence
interval. AGT rates schools as follows: (a) Blue - far above predicted; AGT
estimate significantly above 4, (b) Green - above predicted; AGT estimate
significantly above 3, (c) Gray - within the range of predicted; AGT estimate
around 3, (d) Yellow - below predicted; AGT estimate below 3, and (d) Red - far
below predicted; AGT estimate less than 2.
Academic Performance Index (API). “The API is the cornerstone of
California’s Public School Accountability Act of 1999 that measures the
academic performance and growth of schools on a of variety academic measures”
(California Department of Education, 2011). The score ranges from 200 being the
lowest to 1,000 being the highest. The API is a weighted average of students’
California Standards Test (CST)/California Modified Assessment
14
(CMA)/California Alternate Performance Assessment scores across assessed
content areas and targeted grade levels. The statewide target is 800, and each
school has a base score and a target score. Each school is accountable each year
for a targeted growth of 5% of the base score or 800.
Los Angeles Times Effective Schools Rating: “A teacher whose students
achieve acceptable rates (e.g., at least one grade level in an academic year) of
student growth” (United States Department of Education, 2009, p.12). For the
study purposes, the effective schools are identified by the Los Angeles Times’
reports (Los Angles Times, 2010). The Los Angeles Times reported the Los
Angles school ratings based on the value-added study done by a Rand statistician.
Each elementary school in the LAUSD was rated as one of the five ratings; least
effective value-added school, less effective value-added school, average value-
added school, more effective value-added school, and most effective value-added
school (Los Angeles Times, 2010).
Reliability. Reliability is a trustworthy and accurate measurement that
shows consistency across different times, items, judges, or samples.
Residualized Difference Score. “Residualized difference scores, although
not widely known in California, is the difference between a school’s API and the
school’s predicted API” (Park, 2009, p.17). The school’s predicted API was
calculated with the weighted sum of index of the percentage of SES, parent
education level, and percentage of English Learners.
15
School Experience Survey. The School Experience Survey is a survey
developed by LAUSD to be included in the annual School Report Card to inform
all stakeholders about the progress of each school. Since 2008, LAUSD surveyed
students, staff, and parents in grades 3 to 12 about “the opportunities they have to
learn and be leaders at their schools, how welcoming and collaborative the
environment is, the cleanliness and safety of the campus, and parent engagement”
(Los Angeles Unified School District, 2011c, p.1). The details of the survey are
in Appendix A.
Similar-school ranks. Similar school rank is 10 tier based. Each
school is placed in a rank between 1 to 10 based on a school’s API score in
comparison to 100 schools with similar characteristics including : (a) student
mobility, (b) student ethnicity, (c) school socioeconomic data, (d) teachers with
full credential, (e) teachers with emergency credentials, (f) percentage of ELs, (g)
average class-size, (h) school calendar, (i) percentage of grade-span enrollment,
(j) percentage of students with disability, (k) percentage of Reclassified students,
and (l) percentage of migrant students (California Department of Education,
2010a).
Validity. Validity is how well a test measures what it was designed to
measure and the extent to which it provides accuracy in proposed interpretation
of results.
16
Value Added Model (VAM). VAM is one type of growth model. “The
main purpose of VAM is to separate effects of non-school related factors such as
family, peer, and individual influence from a school’s performance”
(Goldschmidt, Roschewski, Choi et al., 2005, p.5). It measures the difference
between the expected growth and the actual growth.

Organization of the Study
This study is organized into five chapters. Chapter 1 is a background of
the principal evaluation, the statement of the problem, the purpose of the study,
importance of study, and definitions of terms. Chapter 2 is a review of literatures
related to this study of the principal evaluation including overview of the
accountability, impact of accountability on student achievement, role of principal
on school accountability, principal evaluation to support student outcomes, and
accountability indicators. Chapter 3 describes the methodology of the study that
includes, sample and population, instrumentation, limitations, delimitations,
assumptions, data collection, and data analysis. Chapter 4 provides reporting of
findings of this study and discussion. Finally, Chapter 5 provides a summary of
findings, implications, limitations, areas for future research, and conclusions.
17
CHAPTER 2
LITERATURE REVIEW

This chapter contains a review of literature in relation to accountability, its
impact on student achievement, the roles of principal in school accountability,
principal evaluation to support student outcomes and accountability indicators.
For the purpose of this study, a variety of studies and literature were analyzed for
background information and research.

Overview of Accountability
Thomas Jefferson declared education was a right for citizens in a
democratic society. In 1779, he proposed guaranteed universal education for
three years that gave birth to the public education system in America (Honeywell,
1969). Since then, American public education has had multiple face-lifts and
modifications to meet the growing nation’s need to educate its perpetual shifting
citizens. Moreover, criticism and reform for better public education have been in
public and political debate.
Demands for a better education increased sharply since the launching of
Sputnik in 1957 (Marzano, 2003). The Coleman Report that was initiated by
President Johnson in 1964 confirmed the myriad of arguments against the
adequacy of public education (Coleman, 1966). The Coleman Report
18
corroborated the report by Jencks, “Inequality: A Reassessment of the Effects of
Family and Schooling in America.” Jencks (1969) and his colleagues asserted
that schools had no or little effect on student achievement. In their study, they
concluded that “schools account for only about 10% of the variance in student
achievement—the other 90% is accounted for by student background
characteristics” (Marzano, 2003, p. 2). The Coleman (1966) and Jencks (1969)
reports left the nation wondering about why any effort would be put into school
reform, and contradicted the common belief that American public education was a
pathway to equal opportunity and access for a better future for underprivileged
students (Marzano, 2003).
In 1983, America was startled again by A Nation at Risk: The imperative
for Educational Reform report by the National Commission of Excellence in
Education (1983). In the report, it was revealed that America was no longer the
leading nation in the world. America was losing its competitive edge in
commerce and industry to other countries. The nation was at risk because the
mediocre education system was no longer producing citizens with necessary
skills, literacy, and training (National Commission of Excellence in Education,
1983). Six years later, President George H. Bush, along with the 50 state
governors, announced the ambitious goal of being number one in the world in
mathematics by year 2000 by launching Goals 2000: Educate America Act
(Darling-Hammond, 2007). Following this act, the funding for education has
19
increased tremendously. For the following years, the per-pupil spending
increased 40%, and the states adopted some type of assessments to measure the
academic progress. Teachers’ salaries were raised, and class sizes were reduced
in an attempt to create a world-class education system (Kirkpatrick, 2003).
However, another avalanche of disappointment soon shook the nation.
In 1995, the United States’ eight grade students ranked 28
th
out of 41
countries in mathematics, and fourth grade students ranked 23
rd
in the Trends in
International Mathematics and Science Study (TIMSS) (Trends in International
Mathematics and Science Study, 1997). America was still at risk according to the
TIMSS’ report as American students were far from academic proficiency
compared to their counterparts in the international realm. American 12
th
grade
students placed 19
th
out of 21 developed nations in mathematics, 16
th
out of 21 in
science, and last in physics (A Nation Still at Risk, 1998).
The report purported that America was the only country where years of
schooling provided worsened achievement for students. Thirty-seven prominent
educational leaders asserted the urgency of an education reform to address
eminent needs of higher standards, standardized assessment, accountability,
public school choice, charter schools, voucher programs, less credential
requirements for highly qualified people, and merit pay (A Nation Still at Risk,
1998). Policy makers and politicians believed that traditional school reform
approaches such as professional development, class-size reduction, and school
20
improvement were costly and ineffective. Moreover, a lack of school
accountability, sanctions, and rewards were a major cause for the school failures
(Linn, 2006).
Repeated failures to achieve desirable educational outcomes led to
President George W. Bush to pass the bipartisan educational act, No Child Left
Behind (NCLB) in 2001. Under NCLB (United States Department of Education,
2002), schools were accountable to bring all students to academic proficiency by
2014 or would face a heavy sanction. By this time, multiple studies purported
that highly effective schools made an impact on student achievement (Barth,
Haycock, Jackson, Mora et al., 1999, Darling-Hammond & Rothman, 2011,
Kearney, 2010; Marzano, 2003). Barth et al., (1999) studied 365 schools with
high poverty in 21 states, and found that certain schools with high numbers of
socio-economically disadvantaged and minority students produced soaring
student outcomes despite a mainstream expectation of low performances.
Furthermore, Marzano (2003) purported that schools had remarkable impact on
students by implementing research-based strategies. Marzano (2003) used
Binominal Effect Size Display (BESD) to prove that 84.7% of the students in
effective schools reached proficiency. That clearly beat the odds of their expected
outcome, 50% proficiency given independent variables such as poverty,
background knowledge, student motivation, and parents. According to Marzano
(2003), highly effective schools, regardless of their background possess five
21
common factors; “(a) guaranteed and viable curriculum, (b) challenging goals
and effective feedback, (c) parent and community involvement, (d) safe and
orderly environment, and (e) collegiality and professionalism” ( p. 15).
Therefore, effective schools can make a difference in the lives of students
regardless, and this was another catalyst for the school accountability.
Scheurich and Skrla (2001) asserted that accountability was instrumental
to bringing equity to students, especially to children of color. The educators had
been color-blind and neglected to provide an equitable education to students from
low-income families, African American, Hispanic, and English Learners. It had
been prevalent among education community to blame the failures on external
factors such as parents, student attitudes, neighborhoods, culture, and language
(Scheurich and Skrla, 2001). NCLB was designed to stipulate accountability in
schools, districts, and states to meet annual measurable objective (AMO) goals for
all subgroups of students including socioeconomic background, ethnicity, English
language proficiency, and disability in attempt to provide equity in education for
all (Linn, Baker, & Betebenner, 2002).
Under NCLB, states must develop standards and assessments to measure
adequate yearly progress (AYP) of all students including the subgroups. The goal
of NCLB is to bring 100% of students to be proficient and above in their grade
level standards by the end of the 2013-2014 school year (Linn, Baker, &
Betebenner, 2002). There are other requirements of meeting the AYP goal: 95%
22
participation in state assessment, segregated subgroup reports, and one additional
academic indicator (i.e., graduation rates). For schools that do not meet the AYP
goals for two consecutive years, they would be sanctioned with the program
improvement (PI) status (Linn, Baker, & Betebenner, 2002).
According to Linn, Baker, & Betebenner (2002), there are unintended
negative consequences of NCLB as it is implemented as an accountability
mechanism to close the achievement gap. First of all, it was left for each state to
develop its own standards, assessments, and proficiency levels, which created a
great discrepancy among the states. That further resulted in watered-down
standards and assessments in order to meet the AYP goals in some states (Linn et
al., 2002). Second, for some states, the performance standards were already in
place before NCLB. The pre-NCLB performance standards might have been too
rigorous and geared towards the students from affluent communities that would
not fit for all students. Third, AYP goals left schools with considerable
challenges to meet them. The goal only focused on meeting the proficiency-and-
above category while neglecting students in the lowest-achievement category.
Fourth, comparing test scores from one year to another year was subject to errors
of measurement due to both internal and external factors.
The students who attended one year were different from another year.
Also, there were many uncontainable changes such as teacher-turn-over, a teacher
strike, and disruptive students within a school that would affect the results.
23
Finally, schools were focusing on teaching to the test to avoid stiff sanctions
instead of fostering deeper understanding of content (Linn et al., 2002).
Linn (2005) ascertained suggestions to modify implausible NCLB goals in
order to sustain its primary goals of bringing all students to academic proficiency
by closing the persistent academic gap. He suggested successive or longitudinal
approaches should be used to measure the growth instead of fixed targets. The
definition of proficiency should be revamped to be more meaningful and
comparable in all states. Moreover, there was a great need to praise the efforts of
success by implementing realistic goals to both the safe harbor provision of the
law and the AMO.
Under NCLB, all public schools in California are mandated to meet the
projected goals in Academic Performance Index (API) and AYP set by the state
and the federal governments each year (California Department of Education,
2010b). All Local Education Agencies (LEA) are accountable to meet the API
and AYP goals. Furthermore, administrators and teachers are accountable to
implement the standards-based education to meet the proficiency goals at each
site. When Title I funded schools and when LEA fail to meet the API and AYP
goals for two consecutive years, the schools will enter Program Improvement (PI)
status. There are timeline and requirements for each year of PI that schools must
meet. The LEA must notify schools’ PI status to the parents, and allow parents to
have a Public School Choice with paid transportation. The schools must improve
24
the academic achievement of all students within the timeline by establishing a
strategic plan to allocate appropriate funds to improve parent involvement and
professional development. If a school does not improve in 5 years, it can be taken
over by the state. The goal of AYP is to have 100% of American students to be
proficient and above by the year 2014. The school accountability continues to be
an intricate part of American schools that makes a significant impact on the realm
of education.

Impacts of Accountability on Student Achievement
Impacts of accountability on students achievement is inconclusive. Even
before NCLB, two-thirds of the top-performing schools were under some type of
accountability system where the adults were accountable for the learning, despite
the socio-economic status of the students or the ethnic groups (Barth et al., 1999).
There was no proven effect on academic achievement, but when there were
sanctions or rewards for performance of the schools or teachers, the academic
performance rose. Barth et al. (1999) ascertained that placing a state or district
accountability system on schools with viable consequences had a major impact on
student achievement. Therefore, the federal government imposed the school
accountability from all states under the umbrella of the NCLB Act to ensure the
quality of education throughout the nation.
According to Bracey (2008), there was no Robin Hood effect from NCLB.
Although the law was enacted to close the achievement gap of minority students,
25
there was no or little evidence of narrowing the gap. Moreover, by fifth grade,
white students who entered the kindergarten with average reading level surpassed
the black students who entered the kindergarten with similar reading level.
Overall the average student assessment scores have been on the rise in some states
when the scores were compared from pre-NCLB and post-NCLB. The
researchers were skeptical about the post-NCLB assessment increase because the
scores were on the rise regardless. Furthermore, the trend on National
Assessment of Education Progress showed no significant difference (Manzo,
2006). In addition, Bracey (2008) asserted that the gap between above-average
black students and above-average white students grew even larger since the
inception of NCLB. The researchers warned policy makers of jumping into early
conclusion of success of NCLB based on a false impression (Manzo, 2006).
According to a 3-year study conducted by the Santa Monica, California-
based RAND Corp., superintendents, principals, and teachers in California,
Pennsylvania, and Georgia expressed positive development in their classrooms
after NCLB (Viadero, 2007). Across the three states, 75% of administrators and
40%-60% of the teachers expressed improvement of instruction in the classrooms,
especially in schools with more struggling students (Viadrero, 2007). Manzo
(2006) ascertained growing public interest as well as public scrutiny on schools,
administrators, and teachers due to publishing the AYP results in the media. The
teachers expressed that the staff morale had declined as the accountability
26
pressures increased due to more focus on test taking skills, narrow focus on
standards heavily tested, and modified instruction focusing on students who were
closest to being proficient on state assessments (Viadero, 2006).
Although data-driven instruction, periodic assessments for monitoring
progress, and identifying the needs based on data, were prevalent in schools in the
three states (Viadero, 2006), Darling-Hammond (2007) asserted that with
implementation of NCLB, use of performance assessment was discouraged. The
focus had been shifted from challenging activities that promote high level
thinking skills to rote memorization of test-taking strategies to avoid any type of
sanctions due to NCLB. The consistent focus on high-stake testing had not
helped to overcome the national crisis of needing a policy to ensure the equity in
education to meet the intellectual demands of the new century.

Roles of Principal on School Accountability
The role of a principal has changed rapidly as the demand on
accountability has increased. Before, principals were expected to maintain and
manage the building with limited requirements for student achievement (Catano
& Stronge, 2006). Presently, principals are expected to produce optimal student
academic performance as prescribed in AYP. In addition, they are required to
meet their managerial and operational duties as site administrators with limited
resources, state and district mandates, and other policies and regulations (Lyons &
Algozzine, 2006). Tucker & Codding (2002) describe the cruel reality of a
27
principal in this era of accountability; “Why would anyone want the job of
principal? Many schools we know have the look these days of the proverbial deer
caught in the headlights. Almost overnight, it seems, they have been caught in the
high beams of the burgeoning accountability movement” (p. 1). It is not enough
to implement the school reforms, but school administrators are held accountable
to provide the evidences of performance outcomes of all students to the public or
else their school risks being taken over by the state or reconstitution (Marks &
Nance, 2007). In order to survive in this era of accountability, principals are
called foremost to be an instructional leader (Gentilucci & Muto, 2007).
Barth et al. (1999) surveyed principals in 366 schools in 21 states, and
found that schools that were producing high-academic results despite the students’
background had common thresholds of effective leadership. The key findings
from their study of the high-performing schools in high-poverty areas were:
(a) implementation of standards-based curriculum, assessment, and teacher
evaluation; (b) increased instructional time in core curricular areas such as
reading and math; (c) systematic professional development focused on
instructional strategies; (d) parent involvement around instructional practices; and
(e) implementation of state or district accountability systems to hold principals
accountable for student learning. They concluded that highly effective schools
made a difference in students’ lives regardless of their backgrounds, poverty
level, or ethnicity (Barth et al., 1999).
28
Similarly, Lyons and Algozzine (2006) surveyed principals in North
Carolina to gather principals’ perception on main elements of the North Carolina
state accountability program known as ABC’s of Public Education. Lyons and
Algozzine ascertained that responses were similar across the school level, gender,
years’ of experience, school size, or school demographical differences. Lyons
and Alozzine (2006) found the following:
Principals uniformly voiced their concerns over the high stake side of
programs and their implications for school leaders: the expectation for
schools to meet NCLB AYP, the testing requirements for limited English
proficiency students, the testing requirements for exceptional students, the
sanctions for schools that do not meet expected growth, and the school
status designation labels assigned to schools based upon student academic
achievement as measured by test scores. (p 11)
Consequently, a pool of prospective school leaders is shrinking due to
enormous accountability with less support. Principal retention is also low; many
principals report to quit their jobs before their eligibility for retirement.
Moreover, only 22% of current secondary principals in the United States (US)
wish to return to their position (Kearney, 2010). Kearney (2010) asserts that it is
critical for state policy makers to shed a light on leadership development and
support the schools’ leaders who are critical components of student achievement.
Some qualitative researchers pointed out that school leaders had little or
no impact on student achievement. They determined that direct instructional
influence from teachers, not school leadership, is a strong indicator for student
achievement (Robinson, Lloyd, & Rowe, 2008). This paradoxical difference
29
prompted Robinson et al. (2008) to examine the available studies on relationship
between school leadership and student outcome. They were baffled by the fact
that so little research had been done on this topic.
At the time, there were less than 30 published studies in English studying
school leadership and student outcome. They conducted meta-analysis on 27
studies completed between 1978 and 2006. Eighteen out of 27 studies were
conducted in U.S. schools, two studies were done in Canada, and one study in
each of the following countries: Australia, England, Hong Kong, Israel, the
Netherlands, New Zealand, and Singapore. The 27 studies took place in various
settings such as elementary, middle, and high schools. Fifteen of the 27 studies
sorely focused on the principal, and the others focused on the broader concept of
school leadership in general. Diverse measures of student outcomes were used in
the 27 studies such as various assessment scores, student surveys, pre- and post-
tests, SAT scores, standardized tests, and national tests. Furthermore, Robinson et
al., (2008) substantiated a strong correlation between school leadership and
student outcomes through their meta-analysis research on 27 published studies.
Robinson et al., (2008) purported a strong correlation between the impact of
school leadership and student outcomes; transformational leadership (ES = 0.11),
instructional leadership (ES = 0.42), and other types of leadership (ES = 0.30).
Thus, instructional leadership has a major impact on student achievement.
30
According to Marzano (2003), school leaders can impact student
achievement indirectly by supporting and sustaining quality instruction in the
classrooms through offering staff development focused on student achievement
results and pedagogical skills. In addition, principals can promote positive
learning environments by augmenting teacher retention and satisfaction that
connects to greater student outcomes (Sun & Youngs, 2009). Marzano (2003)
asserted that school leaders make a tremendous impact on student achievement by
ensuring that the indispensable curriculum is being implemented in appropriate
sequence and time to support the academic goals for all students and to provide
effective monitoring and feedback. School principals were second only to
teachers in terms of student achievement. Countries with high-performing
educational systems such as Finland, Singapore, and Canada recognized the
importance of instructional leadership as a key for student outcomes (Darling-
Hammond & Rothman, 2011).

Instructional Leadership and Student Achievement
Instructional leadership has also found to be a key factor in creating
positive student outcomes. The mean effect size estimates for instructional
leadership was four times more influential than transformational leadership.
Instructional leadership provided direct and indirect influences on student
outcomes whereas transformational leadership could affect staff attitudes and
31
culture, but not necessary provide positive influences on student outcomes
(Robinson et al., 2008). Based on the meta-analysis, Robinson et al., ascertained
five dimension of strong leadership qualities that brought positive results: (a)
establishing goals and expectations (ES = 0.42), (b) strategic resourcing (ES =
0.31), (c) planning, coordinating, and evaluating teaching and the curriculum (ES
= 0.42), (d) promoting, and participating in teacher learning and development (ES
= 0.84), and (e)ensuring an orderly and supportive environment (ES = 0.27). In
short, school leaders who participated as learners in the professional development
and other curriculum development with their constituents could make the most
impact in improving student achievement (Robinson et al., 2008).
Instructional leadership is considered a primary responsibility for a
principal in order to foster effective instructional skills for teachers, to hold high
expectation from teachers and students, to monitor classroom performances and
students’ progress, and to ensure the quality of curriculum provided (Darling-
Hammond & Rothman, 2011; Marks & Printy, 2003). Successful instructional
leaders promote and demand a high-level professional learning community from
all stakeholders (Sun & Youngs, 2009).
The modern view of instructional leadership has shifted to keeping the
same components while collaborating with teachers (Catano & Stronge, 2006).
Shared instructional leadership involves more than being a conventional
instructional leader. Shared instructional leaders cultivate competent and
32
empowered teacher leaders by providing instructional resources and support to
their teachers and constituents (Marks & Printy, 2003). The reciprocal
relationship between the principal and teachers has existed in order to bring
change in the instruction at the school site. Although Robinson et al. (2008)
purport transformational leadership as less effective leadership than instructional
leadership, shared instructional leadership alone cannot get the job done.
According to Marks and Printy (2003), the transformation leadership has to
accompany the shared instructional leadership in order to bring a substantial
change at a school.
Marks and Printy (2003) conducted a mixed-method research on 24
nationally selected elementary, middle, and high schools in 16 states and 22
school districts to find the relationship between transformational and shared
instructional leadership and their effects on student performance. The research
results were inconclusive to make generalized findings. However, according to
the positions of transformational and shared instructional leadership coordinates
on the scatterplot, instructional leadership without transformational leadership
could only exists in a case of instructional managerial leadership. Marks and
Printy (2003) suggested nine distinctive functions of transformational leadership
in three areas:
(a) mission centered (developing a widely shared vision for the school and
building consensus about school goals and priorities); (b) performance
centered (holding high performance expectations, providing individualized
support, supplying intellectual stimulation); and (c) culture centered
33
(modeling organizational values, strengthening productive school culture,
building collaborative cultures, and creating structures for participating in
school decision). (p. 375)
The schools with high transformational and shared instructional leadership
shared a common threshold of strong pedagogical skills among teachers and
higher National Assessment of Educational Progress (NAEP) scores. The schools
with high transformational leadership but low shared instructional leadership
tended to focus on other innovation such as community building, coordinated
social services, and structural changes rather than focusing on instructional
innovation. The schools low in both transformational and shared instructional
leadership had no principal, a new principal, or a very weak principal.
Gentilucci and Muto (2007) asserted a strong correlation between school
leadership and student achievement. Based on qualitative research on 39 middle
school students in the central coast, principals can provide direct and indirect
instructional leadership to increase student outcomes at schools. Principals
imparted indirect instructional leadership by changing the school climate to make
it conducive for learning, providing adequate resources, effective communication,
and implementing a principal-teacher relationship that would support the teachers
to provide positive learning in the classrooms. However, they also engaged in
direct instruction through visiting classrooms, interacting with students,
monitoring students’ academic progress, public and private recognition of
individual academic successes, and providing tutorial services (Gentilucci &
34
Muto, 2007). Students felt that their principal was more instructionally sound
when he/she exhibited more of direct instructional leadership behaviors such as
being visible, engaging in and out of the classrooms, and checking on their
academic progress which motivated them to do better at school. However,
Gentilucci and Muto (2007) did not make a direct correlation between the
students’ responses on their survey and actual achievement of the school.
Although no one disputes the importance of school leadership as an
intricate part of closing the student achievement gap, teacher retention, and school
improvement, there is a lack of support for high-quality leadership building,
training, and commitment for developing principals (Kearney, 2010). Under
NCLB and initiatives like Race to the Top, the federal government mandates that
state government improves the schools by strengthening the quality of teachers
and principals. However, there is no state-level commitment to develop school
leadership (Kearney, 2010). Often, principals are not prepared to perform the
most important task or given time to provide meaningful reflection to examine
what is going on in the classrooms (Schmocker, 1999). Rather, they are expected
to be instructional leaders to comply with stringent accountability demands.
School principals are not spending enough time in the classrooms to address the
critical instructional needs, nor are school principals trained to identify the
instructional needs and to create a strategic plan to target the problem. Less than
10% of the school principals are capable of such a task (Schmocker, 1999).
35
Marks and Nance (2007) purported that school principals were constrained
by contexts from both external and internal factors. In other words, even when
the principals possessed all quality leadership skills, they might not be able to
exercise their potential due to unique circumstances and challenges facing them at
the particular site or the district. Internal challenges could be available resources,
infrastructure within the organization, and/or human resources, including teaching
staff. External challenges include school districts, school boards, school councils,
and/or parent associations.
Marks and Nance (2007) surveyed 8,524 public school principals from 50
states and the District of Columbia to identify principals’ perceptions on their
actual influence on curriculum and instruction and school supervisory decisions
against other influential actors such as state, school districts, school boards,
school site councils, teachers, and parent associations. According to Marks and
Nance’s study, principals viewed school districts, school boards, school site
councils, and teachers as having a positive influence on their own curriculum and
instructional influence on school improvement, but they perceived the state as
negative in regards to state accountability and policy mandates. Moreover, for
supervisory influence, principals perceived parent associations as having an
adversarial influence on their ability to make decisions on personnel and budget
(Marks & Nance, 2007).
36
More principals relied on the school district, school board, and teachers
for support behind their impact on student achievement. Furthermore, principals
were constrained by the multiple contextual influences played by other influential
actors. More empirical and theoretical research needs to be conducted to identify
key components of school leadership to foster student achievements (Robinson et
al., 2008).

Principal Evaluation to Support Student Outcomes
It is apparent that school leadership matters. Behind every high-
performing, successful school, and 90/90/90 schools, there is an effective
principal (Marzano, Waters, & McNulty., 2005). Catano and Stronge (2006)
purports that an effective principal evaluation can ensure quality leadership in
schools. Furthermore, policy makers and researchers develop “five key leverage
points” to expand the quality of school leadership. Five key leverage points are
“standards, licensure, program accreditation, professional development, and
leadership evaluation and consequences” (Porter et al., 2010, p. 136). The first
four points (standards, licensure, program accreditation, and professional
development) are ascertained attention by creating and implementing the
Interstate School Leaders Licensure Consortium (ISLLC), the National Council
for Accreditation of Teacher Education, the Interstate School Leaders Licensure
37
Assessment by Educational Testing Service (ETS), standards based professional
development, and a basis for principal evaluation.
Davis (2010) ascertains that the principal evaluation is most effective
when it is connected to clear goals and expectations based on the standards
established by the principals, schools, and districts. Multiple studies point out
that there is strong a correlation between an effective principal evaluation and
student outcomes (Catano & Stronge, 2006; Davis, 2010, Goldring, Cravens,
Murphy, Elliott, Carson, & Porter, 2008; Porter et al., 2010). However, there has
been little or no light shed on principal evaluation and consequences. Davis
(2010) asserts that there have been a few empirical, theoretical, or psychometric
studies on the subject of principal evaluation.
In most schools and districts, principals are required to be evaluated for
their performance. Albeit effective principal evaluations are critical to enhancing
principals’ performances at the school site and improving schools for all
stakeholders. The principals’ evaluation should be congruent to their job
demands as an instructional leader because evaluation is a powerful tool to guide
its constituents to desirable behaviors and outcomes. Clear expectation for a
school leader’s job can be communicated effectively through evaluation (Catano
& Stronge, 2006). Catano and Stronge (2006) conducted a study on principal
evaluations of 132 school districts in the state of Virginia using both quantitative
and qualitative methods of content analysis. They segregated the words and
38
phrases into the five categories defined in the Interstate School Leaders Licensure
Consortium (ISLLC) standards, and nine categories defined in the Virginia
Standard of Accreditation (SOA).
They found that most of the ISLLC standards were incorporated in the
principal evaluations. The heaviest emphasis was placed on organizational
management and instructional program, and least emphasis on responsibility to
the larger society. On the analysis of SOA categories, instructional quality and
staff/parent communication were heavily emphasized, but there was little or no
emphasis on student drop-out statistics, keeper of student records, keeper of
teacher licensure, and teacher training. Catano and Stronge (2006) ascertain that
school districts in Virginia align with ISLLC standards and the Virginia SOA with
their evaluation instrument with emphasis on managerial and instructional
leadership. The evaluation provides clear expectation to principals regarding their
job responsibilities. However, there was no clear indication about principals’ job
satisfaction related to the evaluation or if the evaluation promotes principal
effectiveness or positive student outcomes. Porter et al. (2010) also examined 65
principal evaluation instruments from different school districts, and found that all
but two evaluations were “homemade,” and were disconnected from the
conceptual framework based on literature of effective principal leadership and
instruction.
39
Goldring et al. (2008) conducted a study on 65 instruments being used for
principal evaluations from 35 schools districts in 9 states. One third of 74 school
districts they contacted to be a part of the study did not have a point person for
principal evaluation procedures. With the collected samples, they used an
iterative and deductive process to code for content analysis. They categorized the
contents of the collected principal evaluations into four broad categories:
management, external environment, school and instruction, and personal
characteristics. Furthermore, in-depth content analysis was used to examine
congruence to “the core components of learning-centered (LC) leadership: high
standards of student learning, rigorous curriculum, quality instruction, a culture of
learning and professional behavior, connections to external communities, and
performance accountability” (Goldring et al., 2008, p. 9).
Goldring et al. (2008) asserted that no two evaluations were alike. The
evaluations were diverse in terms of number of items (10 to 199 items), scoring
methods (2 to 5 overall rating scales), and inconsistency of summative to
formative evaluation. About a half of the evaluations were not based on any
professional standards, whereas 41% of the evaluations were based on state
standards and the rest used the ISLLC standards. Although most districts focused
the principal evaluation in the category of school and instruction, there was no in-
depth evaluation of principals’ abilities to provide rigorous curriculum,
instructional quality, and accountability measures. Goldring et al. (2008)
40
conclude that there was a great need to create relevant assessment tools that
would enhance complicated demands of today’s principals and improve learning.
The authors alluded a necessity of future research for effective usages of principal
evaluation, but the direct link between the district’s success in student outcome
and principal evaluation was missing.
Sun and Youngs (2009) researched the study on relationship between
principals’ instructional leadership behaviors and district evaluations using a
Hierachical Multivariate Linear model in 13 school districts in Michigan. The
authors based their theoretical framework on the Learning-Centered (LC)
leadership essential behaviors and the correlation to the district evaluation
purposes, focus, and leadership activities. Based on the survey, principals felt that
the evaluation was irrelevant to augment their leadership skills to bring positive
student outcomes. However, the study indicated that when the district focused the
principal evaluation on a principal’s instructional knowledge and skills, their
leadership activities at the school, and accountability on student learning
outcomes, principals were more likely to display LC leadership core component
behaviors. Moreover, the district’s efforts to provide school leaders’ professional
development focusing on instructional knowledge and skills and accountability
had shown strong correlation to increased LC behaviors (Sun & Youngs, 2009).
Therefore, in order to raise student achievement, it is crucial for districts to
connect the principal evaluation to “school goal setting, curriculum design,
41
teacher program development and evaluation, and monitoring student
performance” (Sun & Youngs, 2009, p.438).
Sun and Youngs (2009) also found that years of experience as a principal
was not a strong indicator for LC leadership, but the years of experience as a
teacher correlated with likelihood of increased LC behaviors. They purported that
the more experiences a principal had as a teacher provided them with the
confidence to push for rigorous academic goals at their school. However, more
experienced principals were also less likely to be accustomed to take on an
instructional leadership role. The findings confirmed with other research on the
correlation between principal evaluation and instructional leadership practice, but
the authors failed to make direct connection between principal evaluation and
student outcomes.
Porter et al. (2010) conducted an extensive study on validity of Vanderbilt
Assessment of Leadership in Education (VAL-ED) that was developed by
researchers. The instrument is developed based on the ISLLC standards, 360°
feedback, and evidence-based model. The VAL-ED focuses on six core
components of LC leadership and six key processes that reflect the effective
behaviors of school leaders to foster learning. Six key processes refer to
“planning, implementing, supporting, advocating, communicating, and
monitoring” (Porter et al., 2010, p. 141). The VAL-ED instrument contains 72
items framed around two major concepts, core components and key processes that
42
are scored through a 5-point scale. The VAL-ED was field tested and piloted for
its validity and reliability as an evaluation instrument through multiple measures
and methods. Porter et al. (2010) purported that the VAL-ED could be used for
both summative and formative evaluation to support the instructional leadership
that would yield affirmative student outcomes. However, the long-term effects of
the VAL-ED are unknown, so it remains a research instrument. A conflict of
interest concern arose, as the developers of the instrument were the ones who
conducted the research study.
As mentioned in Chapter 1, the Race to the Top (RTT) redefined a highly
effective principal as a person who could improve academic growth by an average
of one and a half grade levels based on state. School principals make tremendous
impact on school improvement and quality of instruction at schools (Catano &
Stronge, 2006; Davis, 2010; Goldring et al., 2008; Porter et al., 2010). However,
not every principal has the skills, knowledge, and/or experiences to be an
effective instructional leader with less than 10% falling into that category of
effective instructional leaders (Schmoker, 1999). The evaluation process could be
used as a tool to shape and to promote an effective instructional leader. However,
the current principal evaluation implementation systems in the country need to
improve to maximize student outcomes (Davis, 2010). Moreover, RTT requires
the states to use a principal evaluation system that will improve principal
effectiveness based on performance through designing and implementing efficient
43
evaluation systems with timely and constructive feedback, using student growth
data as a part of the evaluation, and using the evaluation for personnel decisions
(United States Department of Education, 2009). It is imperative to identify the
eminent principal evaluation that will foster effective leadership to improve
student achievement by carefully selecting the instruments that measure the
leadership qualities.

Accountability School Status Indicators
Using tests to measure the effectiveness of programs or schools is nothing
new to the educational realm. According to Linn (2006), a variety of assessments
have been used to monitor progress of students, schools, and teachers, to evaluate
programs and schools, to identify educational needs for students and schools, and
to inform all stakeholders. Moreover, achievement test results are easy for the
public to understand, and became a policy tool for politicians to use to influence
what was happening in schools and classrooms (Linn, 2006). Under NCLB,
results from standards-based assessment became a core of the school performance
indicators (Linn, Baker, & Betebenner, 2002). Next the performance indicators
used by the schools such as Academic Performance Index (API), Adequate Yearly
Progress (AYP), and Value Added Method (VAM) is discussed.

44
Academic Performance Index
As a result of passing the California’s Public Schools Accountability Act
of 1999, the state created the Academic Performance Index (API) to suffice the
law mandating all schools in California to be ranked based on their performance
based on a standardized assessment (Powers, 2003). Since the inception, the API
has been widely used as an accountability measure and success indicator in
California due to immense publicity. The API is a type of growth model that
compares the performance of successive cohorts of students. This approach has a
threat to external validity due to student sampling errors that necessitates an
assumption that fifth grade students from the previous year and fifth grade
students this year are comparable (Linn, 2006). Regardless of its validity or
reliability, it was easy for the public to understand and to compare how different
schools were doing.
Powers (2003) conducted a correlation study using independent variables
measuring student socioeconomic status, mobility, English language learners,
teacher training and experience, teacher’s education, student/teacher ratio, school
size, school calendar, and API as a dependent variable for two largest school
districts in California: LAUSD and San Diego Unified School District. Powers
(2003) purported that “more than 75% of the variability in the API can be
explained by three factors: percentage of students eligible for reduced-price or
free lunch, mobility, and percentage English learners” (p. 569). Also significant
45
on the API growth were sizes of percentage of fully credentialed and years of
teaching experience, school size, and school calendar (Powers, 2003). Marzano
(2003) purported that 80% of the results of school academic test scores are based
on school demographic characteristics beyond a school’s control. Hocevar (2011)
asserted unadjusted school status scores such as API or AYP are based in favor of
schools with less SES, and they are useless indicators to measure performance or
accountability purpose. Multiple researchers concluded that the API was a biased
measure of school performance, and should be used with caution as a school
accountability indicator (Hocevar 2011; Powers, 2003; Schochet & Chiang,
2010).

AYP
The AYP is an accountability measure that is required by NCLB. The
AYP is a current status model that requires all schools and students to meet mean
performance proficiency level, Annual Measurable Objectives (AMO), each year
including all subgroups (Linn, 2006). The purpose of the AYP is to ensure the
success of all students by closing the achievement gap between minority and non-
minority students. However, the AYP “poses the greatest challenges to high-
poverty schools, which enroll a large percentage of students who have
traditionally scored poorly on standardized achievement test” (Kim & Sunderman,
2005, p.4).
46
Researchers asserted the strong correlation between high-poverty schools
and schools identified as Program Improvement (PI) schools based on not
meeting AMO for one or more subgroup after comparing the student achievement
data and the poverty level and ethnic groups in six states including Arizona,
California, Georgia, Illinois, New York, and Virginia (Kim & Sunderman, 2005).
PI schools have predominantly Black and Hispanic students compared to schools
who meet the AYP goals that are predominantly Asians and Whites. Many high-
poverty schools made comparable percentage improvement compared to low-
poverty schools; however, the substantial student achievement increase in high-
poverty is not recognized but often labeled as PI schools due to one size fits all
AYP requirements. Kim and Sunderman (2005) also found a strong correlation
between the number of subgroups and PI school status. A substantial number of
PI schools have multiple sub-groups because in many instances, Black and
Hispanic students belong to more than one subgroup. Researchers pointed two
design flaws for the AYP as an unsuitable accountability indicator: (a) use of
AMO rather than using a growth model and (b) the subgroup accountability
policy rather than multiple indicators (Kim & Sunderman, 2005, Powers, 2003).
Therefore, the AYP is not a suitable accountability that should be used as a
measure to make high stake decisions (Hocevar, 2011).

47
Value-Added Methods
The Los Angeles Times published a series of articles on their analysis
about effective and ineffective teachers based on the Value Added Methods
(VAM) and shed some lights on VAM as school performance indicators (Felch,
Song, & Smith, 2010). The Times published a list of highly effective, more
effective, average value added, less effective, and least effective teachers of all
elementary schools in Los Angeles Unified School District based on VAM. The
articles have triggered many controversial discussions on validity of VAM to
measure effectiveness of a teacher. Nonetheless, with a growing demand from the
federal government for data-based evaluation, the VAM is considered to be a part
of teacher and principal evaluations in Los Angeles (Felch, Song, & Smith, 2010).
VAM is a type of a growth model, which compares the difference between
prior and current achievement data to assess an academic progress of a particular
school, program, or teacher (Goldschmidt et. al, 2005). “The main purpose of
VAMs is to separate the effects of non-school-related factors (such as family,
peer, and individual influence) from a school’s performance at any point in time
so that student performance can be attributed appropriately” (Goldschmidt et. al,
2005, pg. 5). VAM infers a causal interpretation that the student achievement is
caused by teachers or the school rather than other variables such as student
background characteristics or school characteristics (Linn, 2006). However, the
VAM excluding the student background variables is biased. Moreover, Buddin
48
(2011) purports “a concern for VMA is the teacher or school effects might vary
substantially for alternative specification of the statistical model “(p. 14).
Students are not randomly placed in a classroom or school to control for variables
such as student motivation and parental support. Nevertheless, multiple studies
conclude that VAM is a better solution than AYP or API, but there is no strong
scientific evidence for correlation between student achievement and school
effectiveness. In other words, the VAM can be an effective tool to use as a part of
teacher evaluation along with other evaluation methods such as principal
observation (Buddin, 2011; Baker et at, 2010; Linn, 2006; Schochet & Chiang
2010,.
Schochet and Chiang (2010) asserted that VAM data, based on a few
years, would be inaccurate to assess teacher or school performance due to its
design flaws. Errors occurred during making a growth estimation stemming from
two factors: “(a) random differences across classrooms in unmeasured factors
related to test scores, and (b) idiosyncratic unmeasured factors that affect all
students in specific classrooms, such as a barking dog on test day” (Schochet &
Chiang, 2010, p.1). The VAM could be used to evaluate school programs such as
class-size, curriculums, and teacher prep programs because the VAM fell
somewhere between high and low stake-based on its reliability and validity
(Braun, Chudowsky, & Koenig, 2010). Schochet and Chiang (2010) purported
from their research that teacher level error rate using VAM was 26%, but the error
49
rate decreased by adding multiple years of data. However, the school level error
rate was 5% to 10% less than the teacher level due to having larger student
sample size. Thus, VAM could be a potential replacement for the current school
accountability model, AYP with a stipulation of bias against schools compared to
other schools with diverse external factors (Schochet & Chiang, 2010).

Discussion
A desire to sustain the American dream and to regain American
superiority through a compensatory education led to the scrutiny of public
schools. Increased public school accountability in an effort to close student
achievement gaps between minority and non-minority students and American
students versus the world seemed like a logical step for policymakers. Over the
century, substantial studies have been done on school effectiveness and attempts
have been made to find silver bullets for an optimal educational system or school.
Unlike the Coleman Report (1966) findings, further research has shown that
schools and teachers do make a difference in student achievement. That led to the
NCLB Act of 2002, a law that every school in America must follow a school
reform. The school reform is to promote effectiveness in principals who make
enormous impact on school improvement.
A principal evaluation can be a useful tool to guide and to shape school
principals into changing agents in their schools. However, the lack of research on
50
principal evaluation is disheartening compared to its potential impact on schools.
Continuous efforts to find the effective methods and predictable accountability
indicators in principal evaluation are critical to making school reform.
Nevertheless, pointing out a perfect accountability indicator based on testing is
not a simple matter. There are many variables to be considered before making the
causal interpretation based on the test results. In Chapter 3, a methodology to test
for the predictable accountability indicators to be included in the future principal
evaluation will be explored.
51
CHAPTER 3
METHODOLOGY

The purpose of this study was to examine the correlation between
accountability indicators/school accountability measures and selected
demographics and their conceptual relation to principal performance in the Los
Angeles Unified School District (LAUSD). Although a school principal is a
critical part of increasing student achievement, relatively few studies have been
done on principal evaluations. In this study, public assessment data on
elementary schools in Los Angeles, California were used to examine the
correlation between six school performance indicators and selected demographics
in relation to principal performance. Two of the six indicators were being used as
parts of the state and federal accountability systems. Two of the six indicators
were published in the national newspaper as school accountability indices. One
indicator was a part of the accountability measure within the LAUSD. More
specifically, the correlation between the three independent variables and six
dependent variables were examined. The three independent variables were: (a)
socio-economic status, (b) parent education level, and (c) percentage of English
learners. The six dependent variables were: (a) API scores over 5 years, (b)
similar-school ranks, (c) residualized difference scores, (d) Value Added Method
(VAM)–school effectiveness scores, (e) Academic Growth over Time (AGT)
52
Method–school effectiveness scores, and (f) School Experience Survey scores.
Furthermore, multiple bivariate correlation analyses were conducted to identify
the potential predictors that should be used for principal evaluation.

Research Design
A quantitative approach was used for this study. More specifically, a
correlational design was used to examine covariance of independent variables and
dependent variables to use potential school accountability indicators for principal
evaluation. In Figure 1, the correlation between independent variables and
independent variables are described and analyzed to conceptualize potential
predictors for principal evaluation.
In Figure 2, the correlation among dependent variables: API scores over 5
years, similar-school ranks, residualized difference scores, VAM—school
effectiveness rating, AGT—school effectiveness rating, and School Experience
Survey results are described and analyzed to conceptualize potential predictors for
principal evaluation.
53
Figure 1.
Conceptual Model I

SES
Parent
Educati
on
Level
Percentag
e of ELL
API
AGT
VAM
Residualized
Difference
Score
Simila
r
School
Rank
School
Experien
ce Survey
54
Figure 2
Conceptual Model II

School
Experienc
e Survey
Residualize
d
Difference
Score
Similar
School
Rank
API
AGT
VAM
55
Participants and Setting
The participants of this study included K-5 and K-6 elementary schools in
the Los Angeles Unified School District over 5 years, 2005-2006, 2006-2007,
2007-2008, 2008-2009, and 2009-2010. A stratification process was involved in
selecting elementary schools with valid Academic Performance Index (API)
Growth scores for at least 3 of the 5 designated academic years, and schools that
had a population of at least 130 students with valid test scores (N = 442 for 2005-
2006, 2006-2007, 2007-2008, and 2008–2009; and N=441 for 2009-2010).
During the selection process of the samples, the elementary schools with less than
20 participants in the School Experience Survey in any of the stakeholders
(parents, staff, or students) were excluded due to small sample size. Also, charter
schools under LAUSD were excluded in the samples because those schools were
not under the same principal performance evaluation. The API, AGT, and School
Experience Survey results data were downloaded from the LAUSD website. The
VAM data were downloaded from the Los Angeles Times website.
Samples were derived from 495 elementary schools in the LAUSD that
were located in Los Angeles County, California. The LAUSD covers a total of
710 square miles within the Greater Los Angeles Metropolitan area including
cities such as Cudahy, Gardena, Huntington Park, Lomita, Maywood, San
Fernando, Vernon, West Hollywood, and cities partially within the LAUSD such
as Bell, Bell Gardens, Beverly Hills, Carson, City of Commerce, Culver City,
56
Downey, El Segundo, Inglewood, Long Beach, Los Angeles, Lynwood,
Montebello, Monterey Park, Rancho Palos Verdes, Rolling Hills Estates, Santa
Clarita, Santa Monica, and Torrance (Los Angeles Unified School District, 2010).
The total number of students enrolled in the district from 2005 to 2010 is
shown in Table 2. The district serves diverse group of students, but the majority
of the student population is Hispanic, and detailed ethnic break-down for the
district is shown in Figure 2.

Table 2

Student Enrollment History of LAUSD

Year Total Enrollment
2009-2010 677,538
2008-2009 676,420
2007-2008 680,167
2006-2007 704,427
2005-2006

723,964
Source: LAUSD Profile (2010a)

57

Figure 3. Student Ethnic Percentage (2009)

Source: LAUSD Profile (2010).

Instrumentation and Procedures
Achievement.
Under the Public School Accountability Act (PSAA) of 1999, all public
schools in California are subject to be measured and monitored for their academic
growth through annual Standardized Testing and Reporting (STAR) program,
various standards-based assessments; California Standards Test (CST) for
students in grades two through eleven; California Modified Assessments (CMA)
for students with specific learning disabilities in grades three through eleven;
58
California Alternate Performance Assessment (CAPA) for students with
significant cognitive disabilities in grades two through eleven; California High
School Exit Exam (CAHSEE) in grade ten. The Academic Performance Index
(API) based on the standardized assessment results is the corner stone of
California Accountability Progress Report (APR). Moreover, the STAR program
is to suffice the federal mandated accountability, Adequate Yearly Progress
(AYP), and Program Improvement (PI). The achievement instrument was limited
to CST, CMA, and CAPA administered to students in grades second to six in
elementary schools in the LAUSD for the purpose of the study.

California Standards Test
The CST is one of the STAR programs that assess students’ academic
achievement. It is a criterion-referenced exam based on California content
standards that measure how well the California school system and students are
doing in the areas of English Language Arts and Mathematics for grades two to
six, and in addition, writing for grade four and general science for grade five. The
purpose of the CST is for teachers, parents, and schools to use the results to
improve student learning (California Department of Education, 2010a).
The CST scores range from a scale of low 150 to high 600. The
performance bands range from a continuum of far below basic, below basic,
basic, proficient, and advanced (FBB, BB, B, P, and A). Each CST score and
59
performance band represent the demonstration of students’ achievement based on
their grade-level standards. The proficient level scores range from 350 and 400
and “represents a solid performance with competent and adequate understanding
of the knowledge and skills measured by this assessment, at this grade, in this
content area (California Department of Education. 2011).” The goal is for all
students in California to reach the proficient or advanced level in each content
area.

Academic Performance Index
Each year, all schools in California receive the API Base and API Growth
report in an effort to measure the school effectiveness and as an accountability
measure. The API growth is an improvement model that compares a change
between 2 years. The API is calculated based on a weighted average of CST
scores in English Language Arts and math for grades 2-11, science for grade 5, 8,
9, 10, and 11 and history for grades 8-11. The test weight of each content area is
listed in Table 3. Students with IEP can take California Modified Assessment and
California Alternate Performance Assessment in lieu of the CST, and their
weighted scores are counted towards the school API. A Scale Calibration Factor
(SCF) is added to make negative or positive adjustment for year-to-year
consistency and preservation of the API. The API score ranges from 200 to
1,000. The goal of the API is for all schools to reach 800. Similar-school ranks.
60
Schools are divided into three groups: elementary, middle, and high school, and
in each group, all schools in California are placed in 10 tiers based on their API
scores to determine the relative API placement in the state.
To control for SES, a similar school rank has been created. The similar
school rank is determined based on comparison to 100 schools with equivalent
demographic characteristics including: (a) student mobility, (b) student ethnicity,
(c) school socioeconomic data, (d) teachers with full credentials, (e) teachers with
emergency credentials, (f) percentage of ELs, (g) average class-size, (h) school
calendar, (i) percentage of grade-span enrollment, (j) percentage of students with
disability, (k) percentage of Reclassified students, and (l) percentage of migrant
students (California Department of Education, 2011). Similar school rank ranges
from 1 being the lowest to 10 being highest.

Residualized Difference Score. A difference between actual API and
predicted API was calculated in response to rule out any biased external influence
such as SES, parent education level, and percentage of ELs. The expected API
for each school was calculated by regressing actual API with independent
variables (SES, parent education level, and percentage of ELs).

61
Table 3

Test Weights

Content Area 2010-2011 Api
Test Weights
CST/CMA, Grade 9/CAPA in ELA, Grades 2-8 0.48
CST/CMA Algebra I/CAPA in Mathematics, Grades 2-8 0.32
Cst/CMA/CPA in Science, Grades 5 And 8 0.20
CST In History-Social Science, Grade 8 0.20
Assignment Of 200, ST in Mathematics, Grade 8 0.10
CST/CMA, Grade 9/CAPA in ELA, Grades 9-11 0.30
CST/CMA Algebra I/CAPA in Mathematics, Grades 9-11 0.20
CST/CMA/CPA in Science, Grades 9-11 0.22
CST in Life Science, Grade 10 0.10
CST in History-Social Science, Grades 9-11 0.23
CAHSEE ELA, Grades 10-12 0.30
CAHSEE Mathematics, Grades 10-12 0.30
Assignment of 200, CST in Mathematics, Grades 9-11 0.10
Assignment of 200, CST in Science, Grades 9-11 0.05

Source: California Department of Education, 2011.

Park (2009) purported that the residualized difference scores were not
biased against external factors. According to MacKinnon (2008), the residualized
difference score is used as an alternative to avoid problems with the reliability of
difference scores.
62

Value Added Method
The Los Angeles Times in conjunction with a Rand Corporation consultant
measured the effectiveness of schools using a value added method (VAM). Based
on their study, each school was rated from least effective value added school to
most effective value added school, and these ratings were published in the Los
Angeles Times in August 2010. For the purpose of this study, the Los Angeles
Times rating is converted to numerical rating as follows: least effective value
added = 1, less effective valued added = 2, average effective value added = 3,
more effective value added = 4, and most effective value added = 5.
The VAM is a method to measure the performance of a teacher by
isolating “teacher and school contributions to student learning, conditional on
individual student background and preparation (Buddin, 2011).” Teachers and
schools were rated as “highly effective” when students make above average
increase in the standardized assessments compare to their counterparts of similar
background and preparation. Buddin (2011) asserts that VAM provides
summative measures of teacher and school performance on student academic
improvement by using a “Lagged student achievement model.” The Lagged
model is
T
ijt
=T
ijt-1
λ+x
ijt
β
1
+φ
j
+ε
ijt
, where T is the test scores for the i
th
student
assigned to the j
th
teacher in year t; x is a set of time-varying
characteristics of students, teachers, and schools; β is an estimated
parameter, φ is a teacher fixed effect, ε contains student and teacher time
63
variant unobserved characteristics, and observed time-invariant student
characteristics are incorporated in x. The values of λ reflect the
persistence of student learning inputs from one year to the next. The
contemporaneous value-added model implicitly assumes that λ is zero, so
knowledge acquired in one year does not persist into the next. The value-
added gains model implicitly assumes that λ is -1, so knowledge from one
year fully persists into the next. (Buddin, 2011, pp. 12-13)

Academic Growth Over Time
In 2011, the LAUSD announced that they would use the Academic
Growth over Time (AGT) to measure the effectiveness of schools and teachers.
The AGT is a value-added model using the statewide assessments for 3 years that
controls for external factors such as student’s prior CST performance, grade level,
gender, race/ethnicity, low-income status, English Language Learner (ELL)
status, Individualized Education Plan (IEP) status, homelessness, and mobility
(Los Angeles Unified School District AGT Portal, 2011a). The AGT scores are
measuring students’ academic growth based on students’ prior CST scores in
grades 2-8. “Average growth for similar students across LAUSD is set to three”
(Los Angeles Unified School District, 2011b, p.18). AGT estimate scores ranged
from 1 to 5 scale; score 1 represents schools meeting the far below predicted
growth and score 5 represents schools meeting the above predicted growth. There
were two AGT estimate scores—one in mathematics and one in English language
arts. For the purpose of the study, mean of two scores was used as a single AGT
estimate score for each school.
64
The VAM and AGT are growth models. However, results on schools are
different for two reasons. First, the Los Angeles Times did not have the same
access on certain demographic characteristics of each student such as low-income
status, English Language Learner (ELL) status, Individualized Education Plan
(IEP) status, and homelessness and mobility as only the LAUSD was able to
incorporate into the AGT analysis. Second, the Los Angeles Times VAM
published a school-wide result, but the AGT was separated into results of English
Language Arts and Mathematics. The Los Angeles Times combined results of
multiple years, but the AGT analyzed on over 1-year and over 3-years (Los
Angeles Unified School District, 2011b).
For the purpose of this study, AGT scores from 2007-2008 school year
were indicated as AGT 2008, 2008-2009 school year as AGT 2009, and 2009-
2010 school year as AGT 2010.

Instrumentation: Surveys
School Experience Surveys
LAUSD has developed a School Report Card (SRC) to inform all
stakeholders about the progress of each school annually. As a part of the SRC,
the district conducts annual survey to staff members, students in grades 3-12, and
parents about their experience at the school. The survey was sent home for
parents, and given to students and staff at the school using the questionnaire.
65
However, there is no validity or reliability of information available on this survey
(Los Angeles Unified School District 2011c). The survey was a 4-point Likert-
type response scale: strongly agree, agree, disagree, and strongly disagree. The
survey questions can be found in Appendix A. The survey was collected and
analyzed by the district, and the percentage of responses of strongly agree and
agree was added to the School Report Card. For the purpose of the study, mean
percentage of agree and strongly agree from each stakeholder was calculated as
shown on Table 4.

Procedure: Ratings and Surveys
The district data specialist provided the API scores from 2006 to 2010 in
Excel. The school experience survey results for 2010 for schools in the district
were downloaded from the district website (Table 4).
The value-added school ratings were downloaded from the Los Angeles
Times’ website (Los Angeles Times, 2010). All school data were entered into the
Excel document by hand. The similar-school ranks, parent education level, SES
information, and EL percentage were download from the CDE website, and were
entered into the excel document by hand. All mentioned data are public
knowledge, no permission was necessary to obtain the data. The AGT scores
were downloaded from the district website. Finally, residualized difference
scores for each school were calculated using the SPSS software.
66
Table 4
School Experience Survey Elements

Stakeholders Percentage of Agree and Strongly Agreed
Staff Overall opportunities for professional development
Overall school safety
Overall cleanliness
Overall support, commitment, and collaboration
Overall support, commitment, and collaboration
Overall teacher collaboration
Parents Overall opportunities for involvement
Overall school safety
Overall feeling of welcome
Overall home involvement
Overall parent center
Overall school involvement
Student Overall opportunities for learning
Overall school safety
Overall cleanliness
Overall school involvement
Overall school support
Overall school order

Source: Los Angeles Unified School District School Survey Reports (2011d).
67
Analysis
The collected data were coded, computed and analyzed through the SPSS
software. Means, modes, median, and standard deviation of the variables were
also computed using the SPSS software. Pearson correlation coefficient was
calculated to measure a degree of relationship between three independent
variables and the six dependent variables.

Limitations (Threats to Validity)
Statistical conclusion validity, internal validity, and external validity were
investigated to assess the validity of various school accountability indicators used
in this study.

Statistical Conclusion Validity
Statistical conclusion validity refers to whether relevant statistical analyses
were used to find the reliable covariance among the independent and dependent
variables to make an inference based on the results. Because the sample size in
this study was large, statistical power in the study was not an issue.
Internal validity. There is a threat to internal validity because the study is
not experimental. In this study, only the correlation between the independent and
dependent variables was measured; therefore, causal relationship cannot be
determined because correlation does not prove causation.
68
External validity. External validity refers to generalizability of the study
to broader students, context, interventions, and measurements. Threats to external
validity in this study were students, context, and measurements. Students behave
differently depending on geographical locations as well as their surroundings.
Therefore, what is found in Los Angeles, California may not be found in suburban
Park City, Utah. Furthermore, there is a range of school accountability indicators,
standards, and assessments depending on states. What is measured in California
might not be feasible in other states.
69
CHAPTER 4
RESULTS
This chapter presents the statistical results of the correlation between
independent variables (SES, parent education level, and percentage of EL
students) and dependent variables (API scores, similar-school ranks, residualized
difference scores, Value Added Method (VAM)—school effectiveness scores,
Academic Growth over Time (AGT) Method—school effectiveness scores, and
School Experience Survey scores). The statistical results of the inter-correlation
among independent variables, as well as among dependent variables, are also
presented. The results are analyzed to answer the following research questions:
1. To what extent are API scores and AGT scores stable over
multiple years, and have changes in the mean API and AGT scores
been observed?
2. To what extent are the three dimensions of school experience
surveys (employees, parents, and students) inter-correlated, and do
these dimensions correlate with test-based accountability indices?
3. To what extent are accountability school indicators (weighted API,
similar-school ranks, residualized difference scores, VAM, AGT,
and the school experience survey scores) correlated to SES, parent
education level, and percent of EL?
70
4. And to what extent are accountability school indicators (weighted
API, similar-school ranks, residualized difference scores, VAM,
AGT, and the school experience survey scores) inter-correlated?
This chapter contains the descriptive statistics and results, stability of API
and AGT scores over multiple years, inter-correlation of the school experience
surveys, correlation of independent variables and dependent variables, and inter-
correlation of accountability school indicators.

Descriptive Results
For the purpose of this study, data files were collected from various
sources as indicated in chapter three, and recorded by hand on a single Excel
document according to each school. Data files included SES, parent education
level, percentage of EL, API scores, similar-school ranks, residualized difference
scores, VAM, AGT scores, and school experience survey dimensions. The
completed collection of data was downloaded and analyzed using IBM SPSS
Statistics Data Editor, which follows. The descriptive statistic results of all
variables are listed in Table 5.

71
Table 5
Descriptive Table

Descriptive Statistics
N Minimum Maximum Mean
Std.
Deviation Kurtosis

Statistic Statistic Statistic Statistic Statistic Statistic
Std.
Error
# SES

442 2% 100% 79.59% 25.89% 1.01 .232
Parent Ed

442 1.19 4.51 2.39 .72 .059 .232
# EL

442 .00% 80% 32.9% 17% -.913 .232
2010 API

441 610 978 780.11 69.25 .067 .232
Similar-school
ranks
441 1 10 5.33 2.73 -1.16 .232
Standardized
Residual
441 -3.44 2.59 .0000 .996 .972 .232
VAM

442 0 10 3.00 1.48 .296 .232
AGT2010

441 1.35 4.99 3.00 .51 .423 .232
Employee
Survey
440 42.22% 99.05% 77.29% 9.22% .426 .232
Parent Survey

435 66.79% 92.88% 76.93% 3.39% 1.168 .234
Student
Survey
425 64.78% 91.42% 78.37% 4.41% -.234 .236
Valid N
(listwise)
421

72
Stability of API and AGT Scores Over Multiple Years
The results of statistics of API and AGT scores from elementary schools
over the years were observed to answer the first research question: what extent
are API scores and AGT scores stable over multiple years, and have changes in
the mean API and AGT scores been observed.
Correlation among Academic Performance Indices for 442 elementary
schools for five school years, 2006 to 2010, were observed (Table 6). A higher
API score in one year is associated with a higher API score in another year
consistently, e.g., correlation between 2006 API and its subsequent years, 2007, to
2010; r
1
(440) = .972, p
1
= .001; r
2
(440) = .948, p
2
= .001; r
3
(440) = .922, p
3
=
.001; r
4
(439) = .894, p
4
= .001 (Table 6). The correlations among Academic
Performance Indices over 5 years are statistically significant, and API scores are
extremely stable over multiple years.
Correlations among Academic Growth over the Years (AGT) scores for
442 elementary schools for three school years, 2008 to 2010, were observed. A
higher AGT score in one year was associated with a higher AGT score in another
year consistently, e.g., correlation between 2008 AGT and its subsequent years,
2009 and 2010; r
1
(434) = .316, p
1
= .001; r
2
(434) = .199, p
2
= .001 (Table 7).
73

Table 6

Stability Coefficients of the Academic Performance Indices for LAUSD
Elementary Schools
Correlations

2006
API
2007
API
2008
API
2009
API
2010
API
Pearson
Correlation
1 .972
**
.948
**
.922
**
.894
**

Sig. (2-tailed)

.000 .000 .000 .000
2006 API
N 442 442 442 442 441
Pearson
Correlation
.972
**
1 .968
**
.945
**
.914
**

Sig. (2-tailed) .000

.000 .000 .000
2007 API
N 442 442 442 442 441
Pearson
Correlation
.948
**
.968
**
1 .964
**
.935
**

Sig. (2-tailed) .000 .000

.000 .000
2008 API
N 442 442 442 442 441
Pearson
Correlation
.922
**
.945
**
.964
**
1 .937
**

Sig. (2-tailed) .000 .000 .000

.000
2009 API
N 442 442 442 442 441
Pearson
Correlation
.894
**
.914
**
.935
**
.937
**
1
Sig. (2-tailed) .000 .000 .000 .000

2010 API
N 441 441 441 441 441
**. Correlation is significant at the 0.01 level (2-tailed).

74
However, a correlation between 2008 AGT and 2010 AGT was not as
high as a correlation between 2008 AGT and 2009 AGT. Moreover, correlations
between 2010 AGT and its previous years are consistent, but low. The
correlations among AGT scores over three years are somewhat statistically
significant, and AGT scores are not nearly stable enough over the multiple years
to use as an accountability indicator.

Table 7
Stability of Coefficients of the AGT Scores in LAUSD

Correlations
2008 AGT 2009 AGT 2010 AGT
Pearson Correlation 1 .316
**
.199
**

Sig. (2-tailed) .000 .000
2008 AGT
N 436 436 436
Pearson Correlation .316
**
1 .148
**

Sig. (2-tailed) .000 .000
2009 AGT
N 436 438 437
Pearson Correlation .199
**
.148
**
1
Sig. (2-tailed) .000 .000
2010 AGT
N 436 437 441
**. Correlation is significant at the 0.01 level (2-tailed).

API scores have increased consistently every year from 2006 to 2010; the
mean API score in 2006 was 731.74 among 442 elementary schools in LAUSD
(Table 8). The mean API score in 2010 had increased to 780.11 among 441
elementary schools in LAUSD. The difference was 48.37 points from 2006 and
2010. The trend showed overall improvement of API scores in elementary
75
schools. Elementary schools are moving towards the meeting the API 800 goals
that California set for each school if they already have not met the goal. In
contrast to the AGT, scores did not increase over the years (Table 9).

Table 8

Descriptive Table of the Academic Performance Indices for LAUSD Elementary
Schools

Descriptive Statistics
Mean Std. Deviation N
2006 API 731.74 83.569 442
2007 API 745.09 78.356 442
2008 API 759.83 73.048 442
2009 API 770.43 72.322 442
2010 API 780.11 69.254 441

Table 9

Descriptive Table of the AGT Scores in LAUSD

Descriptive Statistics
Mean Std. Deviation N
AGT2008 3.0303 .41745 436
AGT2009 3.0215 .45409 438
AGT2010 3.0073 .51317 441

The mean AGT decreased slightly from 3.03 in 2008 to 3.00 in 2010
(Table 9). The mean AGT score in 2008 was 3.0303 among 436 elementary
76
schools in LAUSD. The mean AGT score in 2010 had decreased to 3.0073
among 441 elementary schools in LAUSD (Table 8). The difference was 0.023
points from 2008 and 2010. This is a contrast to a significant increase in the API
scores over the 5 years (Table 8). However, the trend showed that schools were
meeting the overall expected district growth goal of schools getting close to an
average growth of score 3.

Inter-correlation of the Three Dimensions of
School Experience Surveys
Correlations among the three dimensions (staff, parents, and students) of
school experience surveys in 2010 were observed to answer the second research
question: to what extent are the three dimensions of school experience surveys
(employees, parents, and students) inter-correlated, and do these dimensions
correlate with test- based accountability indices.
A higher percentage of satisfaction in one dimension was associated with
a higher percentage of satisfaction in another dimension consistently, e.g.,
correlation between employee survey and parent survey in 2010; r
1
(434) = .224,
p
1
= .001; correlation between employee survey and student survey, r
2
(424) =
.624, p
2
= .001; and correlation between parent survey and student survey, r
3
(423)
= .170, p
3
= .001 (Table 10).

77
Table 10

Correlations Between School Experience Dimensions

Correlations

Employees
Survey
Parents
Survey
Students
Survey
Pearson Correlation 1 .224
**
.624
**

Sig. (2-tailed)

.000 .000
Employees
Survey
N 440 435 425
Pearson Correlation .224
**
1 .172
**

Sig. (2-tailed) .000

.000
Parents
Survey
N 435 435 424
Pearson Correlation .624
**
.172
**
1
Sig. (2-tailed) .000 .000

Students
Survey
N 425 424 425
**Correlation is significant at the 0.01 level (2-tailed).

The correlations among the three dimensions of school experience surveys
in 2010 were statistically significant. Interestingly, the strongest correlation is
between the employee survey and student survey (r=.624).
There is an extremely low correlation between the school experience
surveys and test- based accountability indices (Table 11). A higher API score in
2010 was associated with lower employee and parent surveys; correlation
between API scores in 2010 and employee survey, r
1
(438) = -.086, p
1
= .071;
correlation between API scores in 2010 and parent survey, r
2
(433) = -.051, p
2
=
.669.
78
Table 11

Correlation Between School Experience Surveys and Accountability Indicators

Correlations

2010
API
Similar-
school
ranks
Standardized
Residual VA
2010
AGT
Pearson
Correlation
-.086 -.046 -.031 -.009 .095
*

Sig. (2-tailed) .071 .340 .523 .853 .047
Employe
e Survey
N 439 439 439 440 439
Pearson
Correlation
-.051 .005 .002 .074 .044
Sig. (2-tailed) .288 .921 .975 .125 .365
Parent
Survey
N 434 434 434 435 434
Pearson
Correlation
.021 .044 .018 -.073 .071
Sig. (2-tailed) .669 .364 .711 .131 .145
Student
Survey
N 424 424 424 425 424

**. Correlation is significant at the 0.01 level (2-tailed).
*. Correlation is significant at the 0.05 level (2-tailed).

The probability of correlation between API scores in 2010 and employee
survey and parent survey having occurred by chance were not less than .05;
therefore, there was no significant correlation between API scores in 2010 and the
employee and parent survey results.
A higher similar-school ranks was associated with lower employee
surveys; correlation between similar-school ranks and employee survey was
79
r(438) = -.046, p = .340. The probability of correlation between similar-school
ranks and employee survey having occurred by chance was .340; therefore, there
was no significant correlation between similar-school ranks and employee survey.
A higher residualized difference score was associated with lower
employee surveys; correlation between residualized difference score and
employee survey, r(438) = -.031, p = .523. The probability of correlation between
residualized difference scores and employee survey having occurred by chance
was .340; therefore, there was no significant correlation between residualized
difference scores and employee survey.
A higher Value Added Method (VAM) score was associated with a lower
employee survey and student survey; correlation between VAM and employee
survey; r
1
(439) = -.009, p
1
= .853; and correlation between VAM and student
survey, r
2
(424) = -.073, p
2
= .131. The probability of correlation between VAM
and employee survey and student survey having occurred by chance were .853
and .131 respectively; therefore, there were no significant correlations between
VAM and employee survey or student survey.
A higher AGT score in 2010 was associated with a higher employee
survey, parent survey, and student survey; correlation between AGT scores in
2010 and employee survey; r
1
(438) = .095, p
1
= .047; correlation between AGT
scores in 2010 and parent survey, r
2
(433) = .044, p
2
= .365; and correlation
between AGT scores in 2010 and student survey, r
3
(423) = .071, p
3
= .145. The
80
probability of correlation between AGT scores in 2010 and employee survey
having occurred by chance was .047. Thus, the correlation between AGT scores
in 2010 and employee survey was statically significant, but not nearly enough.
Probability of correlations between AGT scores in 2010 and parent survey and
student survey and having occurred by chance were .365 and .145, respectively;
therefore, there were no significant correlations between AGT scores in 2010 and
parent survey or student survey.

Correlation of Independent Variables and Dependent Variables
Correlations between three independent variables (SES, percentage of
parent education level, and percentage of EL) and six dependent variables (API
scores, similar-school ranks, residualized difference scores, VAM, AGT scores,
and school experience survey dimensions) were observed to answer the third
research question: to what extent are accountability school indicators (weighted
API, similar-school ranks, residualized difference scores, VAM, AGT, and the
school experience survey scores) correlated to SES, parent education level, and
percentage of EL. Results are shown in Table 11.
A higher percentage of low SES and EL students were associated with
lower API scores consistently (Table 12): correlation between SES and API
scores in 2010; r
1
(440) = -.768, p
1
= .001; and correlation between percentage EL
81
and API scores in 2010, r
2
(440) = -.670, p
2
= .001. Similarly, a higher parent
education level was associated with higher API scores in 2010, r
3
(440) = .785, p
3

= .001. To summarize the correlation between SES and API scores in 2010,
percentage EL and API scores in 2010, and parent education level and API scores
in 2010 are large and statistically significant.
SES and parent education were not associated with higher similar-school
ranks; r
1
(440) = .001, p
1
= .981; r
2
(440) = .033, p
2
= .489. Thus, the correlations
between SES and similar-school ranks and percentage of parent education and
similar-school ranks are statically insignificant. However, a lower percentage of
EL was associated with higher similar-school ranks; r
3
(440) = -.148, p
3
= .002.
The correlation between EL and similar-school ranks is small, but statically
insignificant.
There is no correlation between three independent variables (SES,
percentage of parent education level, and percentage of EL) and residualized
difference scores: correlation between SES and residualized difference score,
r
1
(440) = 0, p
1
= 1; correlation between percentage of parent education level and
residualized difference score, r
2
(440) = 0, p
2
= 1; and correlation between
percentage of EL and residualized difference score, r
3
(440) = 0, p
3
= 1.
82
Table 12

Correlation Among Independent Variables and Dependent Variables

# SES Parent Ed # EL
Pearson Correlation -.785
**
.785
**
-.670
**

Sig. (2-tailed) .000 .000 .000
2010 API
N 441 441 441
Pearson Correlation .001 .033 -.148
**

Sig. (2-tailed) .981 .489 .002
Similar-school ranks
N 441 441 441
Pearson Correlation .000 .000 .000
Sig. (2-tailed) 1.000 1.000 1.000
Standardized Residual
N 441 441 441
Pearson Correlation .061 -.051 .053
Sig. (2-tailed) .203 .285 .268
VAM
N 442 442 442
Pearson Correlation .027 -.020 .025
Sig. (2-tailed) .578 .671 .598
AGT2010
N 441 441 441
Pearson Correlation .077 -.094
*
.052
Sig. (2-tailed) .109 .048 .280
Employee Survey
N 440 440 440
Pearson Correlation .058 -.074 .031
Sig. (2-tailed) .225 .123 .519
Parent Survey
N 435 435 435
Pearson Correlation -.010 .014 -.013 Student Survey
Sig. (2-tailed) .839 .768 .784

Because the correlations among the three independent variables and
residualized difference scores were statistically insignificant, residualized
difference scores are a good way to control for SES bias.
A higher percentage of low SES and EL students were associated with a
higher VAM, and a lower parent education level was associated with a higher
VAM. However, the correlations among the three independent variables and
83
VAM were not significant. Because the correlations among the three independent
variables and VAM were statistically insignificant, VAM is a good way to control
for the bias of low SES.
A higher percentage of SES and EL were associated with a higher AGT in
2010: correlation between SES and AGT 2010; r
1
(440) = .027, p
1
= .578; and
correlation between percentage EL and AGT 2010, r
2
(440) = .025, p
2
= .598.
Similarly, a higher parent education level was associated with a lower AGT in
2010: r
3
(440) = -.020, p
3
= .671. Because the correlations among the three
independent variables and AGT in 2010 were statistically insignificant, AGT is a
good way to control for the bias of low SES and EL.
A higher percentage of low SES and EL were associated with higher
employee survey results, but the relationship was not significant: r
1
(439) = .077,
p
1
= .109; and correlation between EL and employee survey, r
2
(439) = .052, p
2
=
.280. However, parent education was associated with higher employee survey
results: r
3
(439) = -.094, p
3
= .048. Thus, the correlation percentage of parent
education level and employee survey is statistically significant, but the size of the
correlation is very small.
A higher percentage of low SES and EL students was associated with
higher parent survey results, but the relationships are insignificant: r
1
(434) =
.058, p
1
= .225; r
2
(434) = .031, p
2
= .519. However, a higher parent education
level was associated with lower parent ratings: r
3
(434) = -.074, p
3
= .123. To
84
calculate, the correlations among the three independent variables and parent
surveys are statistically insignificant, and the size of the correlation was very
small.
A higher percentage of low SES and EL students were associated with a
lower student survey. Results parallel to the results on the parent survey.
However, a higher parent education level was associated with a higher student
survey. Nevertheless, the size of the correlation was very small; therefore, the
correlations among the three independent variables and student surveys were
statistically insignificant.
It is observed that API is biased by SES, but VAM, AGT, and residualized
difference scores are not biased by SES. Maybe, VAM, AGT, and residualized
scores are useful for accountability purpose. The school experience surveys are
also unbiased.

Inter-correlations of Accountability School Indicators
Correlations among six dependent variables (API scores, similar-school
ranks, residualized difference scores, VAM, AGT scores, and school experience
survey dimensions) were observed to answer the fourth research question:
To what extent are accountability school indicators (weighted API, similar-school
ranks, residualized difference scores, VAM, AGT, and the school experience
survey scores) inter-correlated? Results are shown in Table 13
85
A higher API score in 2010 was associated with a higher similar-school
ranks, residualized difference score, VAM, and AGT in 2010; correlation between
API score in 2010 and similar-school ranks; r
1
(439) = .488, p
1
= .001; correlation
between API scores in 2010 and residualized difference scores, r
2
(440) = .601, p
2

= .001; correlation between API scores in 2010 and VAM, r
3
(440) = .182, p
3
=
.001, correlation between API scores in 2010 and AGT scores in 2010, r
4
(339) =
.190, p
4
= .001. All correlations between API scores in 2010 and similar-school
ranks, residualized difference scores, VAM, and AGT scores in 2010 are statically
significant.
A similar-school ranks was associated with a higher residualized
difference score, VAM, and AGT in 2010; correlation between similar-school
ranks and residualized difference scores: r
1
(439) = .785, p
1
= .001; correlation
between similar-school ranks and VAM, r
2
(440) = .381, p
2
= .001; and correlation
between similar-school ranks and AGT scores in 2010, r
3
(439) = .406, p
3
= .001.
All correlations between similar-school ranks and residualized difference
scores, VAM, and AGT scores in 2010 were statically significant.
A higher residualized difference score was associated with a higher VAM,
and AGT in 2010: correlation between residualized difference scores and VAM;
r
1
(440) = .383, p
1
= .001; and correlation between residualized difference scores
and AGT scores in 2010, r
2
(439) = .353, p
2
= .001.
86
Table 13

Correlation Among Dependent Variables

Correlations

2010
API
Similar-
School
ranks
Std
Residual VAM
AGT
2010

Emp.
Survey
Parent
Survey
Student
Survey
Pearson
Correlation
1 .488
**
.601
**
.182
**
.190
**
-.086 -.051 .021
Sig. (2-
tailed)

.000 .000 .000 .000 .071 .288 .669
2010 API
N 441 440 441 441 440 439 434 424
Pearson
Correlation
.488
**
1 .785
**
.381
**
.406
**
-.046 .005 .044
Sig. (2-
tailed)
.000

.000 .000 .000 .340 .921 .364
Similar-
school
ranks
N 440 441 440 441 440 439 434 424
Pearson
Correlation
.601
**
.785
**
1 .383
**
.353
**
-.031 .002 .018
Sig. (2-
tailed)
.000 .000

.000 .000 .523 .975 .711
Standardiz
ed
Residual
N 441 440 441 441 440 439 434 424
Pearson
Correlation
.182
**
.381
**
.383
**
1 .373
**
-.009 .074 -.073
Sig. (2-
tailed)
.000 .000 .000

.000 .853 .125 .131
VAM
N 441 441 441 442 441 440 435 425
Pearson
Correlation
.190
**
.406
**
.353
**
.373
**
1 .095
*
.044 .071
Sig. (2-
tailed)
.000 .000 .000 .000

.047 .365 .145
AGT2010
N 440 440 440 441 441 439 434 424
Pearson
Correlation
-.086 -.046 -.031 -.009 .095
*
1 .224
**
.624
**

Sig. (2-
tailed)
.071 .340 .523 .853 .047

.000 .000
Employee
Survey
N 439 439 439 440 439 440 435 425
Pearson
Correlation
-.051 .005 .002 .074 .044 .224
**
1 .170
**

Sig. (2-
tailed)
.288 .921 .975 .125 .365 .000

.000
Parent
Survey
N 434 434 434 435 434 435 435 424
Pearson
Correlation
.021 .044 .018 -.073 .071 .624
**
.170
**
1
Sig. (2-
tailed)
.669 .364 .711 .131 .145 .000 .000

Student
Survey
N 424 424 424 425 424 425 424 425

**. Correlation is significant at the 0.01 level (2-tailed).
*. Correlation is significant at the 0.05 level (2-tailed).

87
All correlations between residualized difference scores and VAM, and
AGT scores in 2010 were statically significant. A higher VAM was associated
with a higher AGT in 2010: correlation between; r
1
(440) = .373, p
1
= .001 The
correlation between VAM and AGT scores in 2010 was statistically significant.
A significant finding is that there are inter-correlations among API scores,
similar-`school ranks, VAM, and AGT scores. However, there is no notable
correlation between standardized test- based accountability school indicators (API
scores, similar-school ranks, residualized difference scores, VAM, and AGT
scores) and school experience surveys. Correlations among three dimensions of
surveys (employee, parent, and student) were discussed in the previous section.
The interpretation of results and findings from the study are discussed in
Chapter 5.
88
CHAPTER 5
DISCUSSION

This study examined the correlations between selected demographics and
accountability indicators and school accountability measures and their conceptual
relation to principal performance. More specifically, the inter-correlations
between three demographics (the percentage of SES, parent education level, and
the percentage of ELs) and six school performance indicators (API score, similar-
school ranks, residualized difference score, VAMM score, AGT score, and school
experience surveys) were used. Although a school principal is a critical part of
increasing student achievement, relatively few studies have been done on
principal evaluations to improve their practices.
Multiple bivariate correlation analyses were conducted to identify the
unbiased accountability performance indicators used for principal evaluation by
answering the research questions:
1. To what extent are API scores and AGT scores stable over
multiple years, and has a change in the mean API and AGT
scores been observed in LAUSD?
2. To what extent are the three dimensions of school experience
surveys (employees, parents, and students) inter-correlated, and
89
do these dimensions correlate with test-based accountability
indices?
3. To what extent are the six accountability school indicators (API
scores, similar-school ranks, residualized difference scores,
VAM scores, AGT scores, and the school experience survey
scores) correlated to SES, parent education level, and percent of
English Learners?
4. To what extent are accountability school indicators (API scores,
similar-school ranks, residualized difference scores, VAM
scores, AGT scores, and the school experience survey scores)
inter-correlated?

Summary of Findings
There were significant findings from this study for each research question.
Research Question 1. To what extent are API scores and AGT scores stable over
multiple years, and have changes in the mean API and AGT scores been
observed:
• The API scores were extremely stable.
• AGT was not nearly stable enough over 2 years to use.
• There was a steady growth in the API scores over the 5 years.
90
Research Question 2. To what extent are the three dimensions of school
experience surveys (employees, parents, and students) inter-correlated, and do
these dimensions correlate with test- based accountability indices:
• Employee and student surveys were strongly correlated, but neither
correlated to parent ratings.
• Employee, parent, and student surveys were uncorrelated with test- based
accountability indices.
Research Question 3. To what extent are accountability school indicators
(API scores, similar-school ranks, residualized difference scores, VAM, AGT, and
the school experience survey scores) correlated to SES, parent education level,
and percent of EL:
• The API scores were strongly biased by SES, parent education level, and
EL. Schools in wealthy neighborhood had higher API scores.
• VAM, AGT, and residualized difference scores were unbiased, and may
be useful for accountability purposes.
• The school experience surveys were unbiased.
Research Question 4. To what extent are accountability school indicators
(API, similar-school ranks, residualized difference scores, VAM, and AGT) inter-
correlated:
• Residualized difference scores were the most correlated with API and
VAM, but AGT scores were the least correlated with API.
91
API Scores
Based on the analysis of statistics of API scores from elementary schools
over multiple years, API scores were stable over the years, and the correlations
among the scores were statically significant. This indicated validity of API scores
for the study.
There is prevalent improvement of API scores over the years. The mean
API score consistently increased every year from 2006 to 2010. There was a
difference of 48.37 points augment in 5 years (Table 14). These findings were
congruent to previous research findings. After implementation of NCLB and state
accountability measures, administrators and teachers supported improvement of
instruction in classrooms as supported by the improvement of API (Viadero,
2007). Increased accountability was instrumental in bringing a major impact on
student achievement equity to students, especially to children of color (Barth et
al., 1999, Scheurich & Skrla, 2001).

Table 14

LAUSD API 2006 to 2010

2006 2007 2008 2009 2010
API 655 664 684 694 709

Source: California Department of Education, 2011.
92
Many researchers have ascertained the strong correlation between school
improvement and school leadership (Barth et al., 1999; Lyons & Alozzine, 2006;
Marzano, 2003; Robinson et al., 2008; Scheurich and Skrla, 2001; Sun & Youngs,
2009; Wallace, 2009). The results point to an improvement in the instructional
leadership of principals in Los Angeles as instructional leadership is a key factor
in increased academic achievement (Catano & Stronge, 2006; Robinson et al.,
2008; Sun & Youngs, 2009).

School Experience Survey
School experience surveys were inter-correlated significantly, and there
was a large strong correlation between employee and student surveys. Thus,
satisfied employees yielded satisfied students or vice versa. Moreover, there was
a cohesive survey result among employees, parents, and students as indicated in
the mean overall satisfaction, 77.29%, 76.93%, and 78.37%, respectively (Table
5). In contrast to findings by Manzo (2006), overall staff morale and satisfaction
was sustained despite the public scrutiny on schools as a result of increased
accountability pressures, at least in the elementary schools in Los Angeles.
Similar results from parents and students ascertained positive outlook on their
schools despite of prevalent public sentiment of dissatisfaction of the nation’s
public schools.
93
Marzano (2003) and Barth et al. (1999) purported that a common factor of
highly effective schools, regardless of their background, are parent and
community involvement around instructional practices, and a safe and orderly
environment. Collegiality and professionalism involve systematic professional
development under effective principals. This was congruent to the hypothesis that
might have satisfied stakeholders and led to overall school improvement shown in
API growth in Los Angeles.
In contrast, the school experience surveys had no or very little correlation
with school accountability indices. Similarly, the school experience surveys had
no or very little correlation with SES, parent education, and EL. Thus, the school
experience surveys were unbiased by the school characteristics. Peterson,
Wahlquist, Brown, and Mukhopadhyay (2003) purported that parent surveys
could be used as a part of teacher evaluation, but they also reported parents were
biased towards teachers and schools satisfying their children’s safety and well-
being over academic achievement. Nevertheless, the findings ascertained that the
school experience surveys should be used for accountability purposes.

Biased and Unbiased School Accountability Indicators
A significant correlation between API scores and selected school
demographics ascertained that 50%-75% of the API can be explained by SES,
percentage ELs, and mobility (Hocevar, 2011; Marzano, 2003; Powers, 2003).
94
Although, parent education level was used in this in lieu of mobility, the findings
suggest that the API is a biased measure of school performance, and should be
used with great caution as a school accountability indicator.
Insignificant correlations between selected school demographics and the
five school accountability indicators (similar-school ranks, residualized difference
score, VAM, AGT, and school experience survey) used in this study suggests
statistical means can be used to remove bias (i.e., level the playing field). The
finding was congruent with previous research findings that a growth model or
value added, including school backgrounds, could be an effective tool to be used
as a part of teacher evaluation along with other evaluation methods such as
principal observation (Baker et al, 2010; Buddin, 2011; Linn, 2006; Schochet &
Chiang 2010).
Lastly, this study suggests residualized difference scores are the most
reliable accountability indicator to be used because they were the most correlated
with other accountability indicators. Residualized difference scores are easy to
calculate and cost effective for the results can be analyized through the IBM SPSS
Statistics Data Editor. Most of all, residualized difference scores are relatively
easy for public to understand compared to VAM and AGT.

Limitations
There are several limitations to this study. First the statistical analysis was
based on school level results downloaded and hand scored from different public
95
websites that depend on the accuracy of responsible schools and agencies. The
inferences were made based on assumed reliable covariance among the
independent and dependent variables, and these different measures could have
lead to different results. Moreover, only the correlation between the independent
and dependent variables was measured; therefore, a causal relationship cannot be
determined because correlation does not prove causation. Lastly, the study was
based on data from elementary schools in Los Angeles Unified School District;
therefore, the generalizability of findings is very limited.
Notwithstanding these limitations, the study demonstrates imminent needs
of an effective performance evaluation system for principals to use to improve
schools. Also, the study tests impending school performance indicators to be used
as a part of principal evaluation that will benefit the school accountability system.

Implications for Practice
A number of implications were drawn from this study. First, the API was
found to be biased for a principal who was working at a school with a high
percentage of low SES students, low parent education levels, and ELs. Therefore,
considering API as a part of principal performance evaluation would be unfair.
Statistical analysis of this study highlights the potential school
performance indicators that are unbiased and fair to be included in impending
principal performance. The similar-school ranks, residualized difference score,
96
VAM, AGT, and school experience survey are not associated with SES, parent
education, and ELL factors. Thus, one, part, or all of the unbiased school
performance indicators (similar-school ranks, residualized difference score,
VAM, AGT, and school experience survey) can be considered as a part of a
principal performance evaluation.
The conclusive evidence on the reliability of unbiased school performance
indicators on student achievement or principal behavior was not a part of the
study. Therefore, any of the unbiased school performance indicators should be
used with caution. This fact highlights the needs for future study on unbiased
school performance indicators that are reliable connections to specific behaviors
of effective principals and student achievement. Moreover, research compared
complex VAM system to simpler residualized difference scores is needed.

Conclusions
With growing political and societal pressure to reform public education,
many politicians, educational leaders, and scholars have been searching for a
panacea to close the achievement gap in the nation. One of the potential panaceas
for school improvement is developing effective school leaders to bring the reform
to schools. Multiple studies assert the critical impact of school principals,
particularly in their instructional leadership role on student achievement (Darling-
Hammond & Rothman, 2011; Gentilucci & Muto, 2007; Kearney, 2010; Marks &
97
Printy, 2003; Marzano, 2004; Sun & Young, 2009). In order to improve principal
practices at school sites, there is a colossal need to develop relevant principal
evaluations in order to bring about school reform. Furthermore, federal and state
mandates have recently directed school districts to create principal evaluation
with student growth data in attempt to improve schools (United States Department
of Education, 2009).
There is a strong correlation between an effective principal evaluation and
student outcomes (Catano & Stronge, 2006; Davis, 2010; Goldring et al., 2006;
Porter et al, 2010). Carefully designed evaluations that included reliable,
equitable student achievement data is a key to school improvement. The
policymakers should walk in the principal’s shoes before making decisions on
principal evaluation procedures that would impact school leaders as well as
school reform. It is pivotal to select viable accountability indicators that connect
a conceptual framework based on literature on effective instructional leadership
and principal evaluation (Porter et al., 2010). Sufficient time should be given to
review current research, and active practitioners such as school principals and
teachers should be a part of the development. Ultimately, fair, reliable, and
equitable accountability school indicators (similar-school ranks, residualized
difference scores, VAM, AGT, and school experience survey) that are highlighted
in the study should be considered.
98
REFERENCES
A Nation Still at Risk. (1998). Policy Review, 90, 23-29. Retrieved March 6,
2011, from ABI/INFORM Global. (Document ID: 30766279).
Baker,E. L., Barton, P. E., Darling-Hammond, L., Haertel, E., Ladd, H. F., Linn,
R. L., Ravitch, D., Rothstein, R., Shavelson, R., & Shepard, L. A. (2010,
August). The Economic Policy Institute, 279. Retrieved from
http://epi.3cdn.net/b9667271ee6c154195_t9m6iij8k.pdf.
Barth, P., Haycock, D., Jackson, H., Mora, K., Ruiz, P., Robinson, S., & Wilkins,
A. (1999). Dispelling the myth: High poverty schools exceeding
expectations. Washington, DC: The Education Trust.
Bracey, G. W. (2006, March). The skinny on value-added. Principal Leadership,
6-7.
Bowe, B., Cronin, J., Kingsbury, G., & McCall, M. (2005). The impact of the
NCLB Act on student achievement and growth: 2005 ed. Northwest
Evaluation Association. Retrieved from
http://nepc.colorado.edu/files/EPRU-0504-117-OWI[1].pdf.
Bracey, G.. (2008, November). NCLB: Achievement Robin Hood? Principal
Leadership, 9(3), 47-49. Retrieved February 25, 2011, from Education
Module. (Document ID: 1596697711).
Braun, H., Chudowsky, N., & Koenig, J. (eds.). (2010). Getting value out of
value-added: Report of a workshop. Washington, DC: National
Academies Press.
Brennan, R. L. (1998). Misconceptions at the intersection of measurement theory
and practice. Educational Measurement: Issues and Practice, 17, 5-9.
Brennan, R. L. (2001). Some problems, pitfalls, and paradoxes in educational
measurement. Educational Measurement: Issues and Practice, 20(4), 6-
18.
Buddin, R. (2011 May 7). Measuring teacher and school effectiveness at
improving student achievement in Los Angeles elementary schools. Los
Angeles Times, Retrieved from http://documents.latimes.com/buddin-
white-paper-20110507/.
99
California Department of Education (2010a). Overview of California’s 2009-
2010 accountability progress reporting system. Retrieved from
http://www.cde.ca.gov/ta/ac/ay/documents/overview10.pdf
California Department of Education (2010b). 2010 Adequate Yearly Progress
information guide. Retrieved from
http://www.cde.ca.gov/ta/ac/ay/documents/infoguide10.pdf.
California Department of Education (2010c). Standardized testing and reporting.
Retrieved from http://starsamplequestions.org/about.html.
California Department of Education (2011). Academic Performance Index.
Retrieved from http://www.cde.ca.gov/ta/ac/ap/.
Catano, N. & Stronge, J. H. (2006). What are principals expected to do?
Congruence between principal evaluation and performance standards .
NASSP Bulletin, 90, 221. Retrieved from
http://bul.sagepub.com/content/90/3/221.
Chanter, C. L. (2002). The relationship of the accountability practices of
elementary principals to student achievement. (Ed.D. dissertation.)
University of Central Florida, Florida. Retrieved February 13, 2011, from
Dissertations and Theses: Full Text.(Publication No. AAT 3054588).
Coleman, J. S. (1966). Equality of educational opportunity. United States
Department of Heath, Education, and Welfare, Office of Education.
Superintendent of Documents, United States Government Printing Office.
Darling-Hammond, L. (2007, May 21). Evaluating No Child Left Behind. The
Nation, 11-18. Retrieved August 22, 2008, from
http://www.thenation.com/doc/20070521/darling-hammond.
Darling-Hammond, L. & Rothman, R. (2011). Teacher and leader effectiveness
in high performing education systems. Stanford, CA: Alliance for
Excellent Education and the Stanford Center for Opportunity Policy in
Education.
Darling-Hammond, L., Snyder, J., Ancess, J., Einbender, L., Goodwin, A. L., &
Macdonald, M. B. (1993, May). Creating learner-centered
accountability. New York: Columbia University, National Center for
Restructuring Education, Schools, and Teaching.
Davis, S. H. (2010). The policies and practices of principal evaluation. San
Francisco: WestEd.
100
DuFour, R. (2002). The learning centered principal. Educational Leadership,
59(8), 12-15.
DuFour, R. (2003). Building a professional learning community. School
Administrator, 60(5), 13-18.
DuFour, R. (2004). What is a "professional learning community"? Educational
Leadership, 61(8), 6-11.
DuFour, R., DuFour, R., Eaker, R., & Many, T. (2006). Learning by doing.
Bloomington, IN: Solution Tree.
Elmore, R.F. (2000). Building a new structure for school leadership. New York:
Albert Shanker Institute.
Felch, J., Song J., & Smith, D. (2010, August 15). Grading the teachers: Who's
teaching our kids? Los Angeles Times, p. A.1. Retrieved February 12,
2011, from Los Angeles Times. (Document ID: 2109894511).
Fullan, M. & Striegelbauer, S. (1992). The new meaning of educational
exchange, 2nd ed. New York: Teachers College Press, Columbia
University.
Fullan, M. (2001). How to make a turnaround succeed. Journal of Staff
Development 22(1), 80.
Gentilucci, J. L., & Muto, C. C. (2007). Principals’ influence on academic
achievement: The student perspective. NASSP Bulletin, 91,:219.
Retrieved from http://bul.sagepub.com/content/91/3/219.
Goldhaber, D., & Hansen, M. (2008, November). Assessing the potential of using
value-added estimates of teacher job performance for making tenure
decision. Brief 3. Seattle, WA: National Center for Analysis of
Longitudinal Data in Education Research.
Goldring, E., Cravens, X., Murphy, J., Elliott, S., Carson, B., & Porter, A. (2008,
March). The evaluation of principals: What and how do states and
districts assess leadership? Paper presented at the annual meeting of
American Educational Research Association, New York, NY.
101
Goldschmidt, P., Roschewski, P., Choi, K., Auty, W., Hebbler, S., Bank, R., &
Williams, A. (2005). Policymakers’ guide to growth models for school
accountability: How do accountability models differ? A paper
commissioned by the CCSSO Accountability Systems and Reporting State
collaborative on Assessment and Student Standards. Washington, DC:
Council of Chief State School Officers.
Grisham, D. L. (2001). High-stakes testing in our schools: A new report from
California. Reading Online, 5(1). Retrieved from
http://www.readingonline.org/editorial/july2001/index.html.
Hanushek, E. A., & Raymond, M. E. (2004). Does school accountability lead to
improved student performance? Journal of Policy Analysis and
Management, 24(2), 297-327.
Highlights of Results from TIMSS (1995). Third International Mathematics and
Science Study. Boston: Boston College. Retrieved from
http://timss.bc.edu/timss1995i/TIMSSPDF/P2HiLite.pdf.
Hocevar, D. (2011, Summer). Leveled accountability modeling: Can state test
data be used by school principals to make grade-level instructional
decision? Presented at EDUC 790. Research Class in Summer, 2011 at
University of Southern California, Los Angeles, CA.
Honeywell, R. J. (1969). Note on the educational work of Thomas Jefferson.
History of Education Quarterly, 1(9), 64-72.
Interstate School Leaders Licensure Consortium. (1996). Standards for school
leaders. Jossey Bass Reader on Educational Leadership. San Francisco:
Jossey-Bass.
Jencks, C. (1969, August 10). A reappraisal of the most controversial educational
document of our time. New York Times Magazine, 12-14.
Kearney, K. (2010). Effective principals for California schools: Building a
coherent leadership development system. San Francisco: WestEd.
Kim, J. S., & Sunderman, G. L. (2005). Measuring academic proficiency under
the No Child Left Behind Act: Implications for educational equity.
Educational Researchers, 34(8), 3-13.
102
Kirkpatrick, D. W. (2003, April). A nation still at risk: Little to show for 20
years of effort. School Reform News, 7(4), 1. Retrieved from
http://news.heartland.org/newspaper-article/2003/04/01/nation-still-risk.
Linn, R. L. (2005). Fixing the NCLB accountability system [CRESTT policy
brief]. Washington, DC: National Center for Research on Evaluation,
Standards, and Student Testing. Retrieved from,
nces.ed.gov/pubsearch/pubsinfo.asp?pubid2007480.
Linn, R. L. (2006). Educational accountability systems. (CSE Technical Report
No. 687). University of Colorado, Boulder. National Center for Research
on Evaluation, Standards, and Student Testing.
Linn, R. L., Baker, E. L., Betebenner, D. W. (2002). Accountability systems:
Implication of requirements of the No Child Left Behind Act of 2001.
Educational Researcher, 31(3). Retrieved from
Http://edr.sagepub.com/content/31/6/3.
Los Angeles Times. (2010). Los Angeles teacher rating. Retrieved from
http://projects.latimes.com/value-added/rank/school/4/.
Los Angeles Unified School District (2011a). Los Angeles Unified School
District fingertip facts 2010-2011. Retrieved from
http://notebook.lausd.net/pls/ptl/docs/page/ca_lausd/lausdnet/offices/com
munications/10-11fingertipfacts_revised228.pdf.
Los Angeles Unified School District (2011b). Los Angeles Unified School
District Growth over Time Portal. Retrieved from
http://portal.battelleforkids.org/BFK/LAUSD/Home.html?sflang=en.
Los Angeles Unified School District (2011c). REF-5422 School Experience
Survey. Retrieved from
http://notebook.lausd.net/pls/ptl/docs/page/ca_lausd/fldr_organizations/fld
r_plcy_res_dev/ses%20ref%20guide%205422.pdf.
Los Angeles Unified School District (2011d). School survey reports . Retrieved
from http://reportcardsurvey.lausd.net/surveys/reports.jsp.
Los Angeles Unified School District (2011e). Supporting all employee strategy
and Academic Growth over Time (AGT) frequently asked questions.
Retrieved from
http://sae.lausd.net/sites/default/files/LAUSD_AGT_FAQs.pdf
103
Lyons, J. E. & Algozzine, B. (2006). Perceptions of the impact of accountability
on the role of principals. Education Policy analysis Archives, 14(16).
Retrieved from htt://eppaa.asu.edu/epaa/v14n16.
MacKinnon, D.P. (2008). Introduction to statistical mediation analysis. New
York: Taylor & Francis Group, LLC.
Manzo, K. K. (2006). Study questions NCLB Law's links to achievement gains.
Education Week, 25(41), 11. Retrieved February 25, 2011, from
Research Library Core. (Document ID: 1090642421).
Marks, H. M., & Nance, J. P. (2007). Contexts of accountability under systemic
reform: Implications for principal influence on instruction and
supervision. Educational Administration Quarterly, 4 (1), 3-37.
Marks, H. M. & Printy, S. M. (2003). Principal leadership and school
performance: An integration of transformational and instructional
leadership. Educational Administration Quarterly, 39, 370. Retrieved
from http://eaq.sagepub.com/content/39/3/370.
Marzano, R. (2003). What works in schools. Alexandria, VA: Association for
Supervision and Curriculum Development.
Marzano, R., Waters, T., & McNulty, B. A. (2005). School leadership that
works: from research to results. Alexandria, VA: Association for
Supervision and Curriculum Development.
National Association of Elementary School Principals. (2004). Leading learning
communities: Standards for what principals should know and be able to
do. Alexandria, VA: National Association of Elementary School
Principals.
National Commission of Excellence in Education (1983). A nation at risk: The
imperative for educational reform. Washington, D.C. Retrieved August
6, 2008, from http://www.ed.ove/pubs/NatAtRisk/index.html.
National Educational Association. (2003). Balanced assessment: The key to
accountability and improved student learning. Student Assessment Series.
G. Washington DC: National Education Association. Retrieved from
http://www2.visalia.k12.ca.us/technology/balanced.pdf.
National Research Council. (2010). Front matter. Getting value out of value-
added: Report of a workshop. Washington, DC: The National Academies
Press, 2010
104
National Research Council. 2010. Getting value out of value-added. H. Braun,
N. Chudowsky, and J. Koenig (eds.). Washington, DC: National
Academy Press.
Park, G. (2009). California under the microscope: An evaluation of the practical
effect of different approaches to accountability. (Unpublished doctoral
dissertation). University of Southern California, Los Angeles.
Peterson, K., Wahlquist, C., Brown, J. E., & Mukhopadhyay, S.. (2003). Parent
surveys for teacher evaluation. Journal of Personnel Evaluation in
Education, 17(4), 317-330. Retrieved December 20, 2011, from
abi/inform global. (Document ID: 975794631).
Porter, A. C., Polikoff, M. S., Goldring, E., Murphy, J., Elliott, S. N., & May, H.
(2010). Developing a psychometrically sound assessment of school
leadership: The VAL-ED as a case study. Educational Administration
Quarterly, 46, 135. Retrieved from
http://eaq.sagepub.com/content/46/2/135.
Portin, B. S., Shen J., & Williams R. C. (1998). The changing principalship and
its impact: Voices from principals. NASSP Bulletin, 82, 1.
Powers, J. M. (2003). An analysis of performance-based accountability: Factors
shaping school performance in two urban school districts. Education
Policy, 17(5), 558-585.
Robinson, V. M., Lloyd, C. A., & Rowe, K. J. (2008). The impact of leadership
on student outcomes: An analysis of differential effects of leadership
types. Educational Administration Quarterly, 44, 645.
Scheurich, J. J. & Skrla, L. (2001). Continuing the conversation on equity and
accountability: Listening appreciatively, responding responsively. The
Phi Delta Kappan, 83(4), 322-326.
Schmoker, M. (1999). Results: The key to continuous school improvement, 2nd
Ed. Alexandria, VA: Association for Supervision and Curriculum
Development.
Schmoker, M. (2005). Tipping point: From feckless reform to substantive
instructional improvement. Phi Delta Kappan, 85(6), 424-432. Retrieved
March 10, 2007, from http://www.pdkintl.org/kappan/k0402sch.htm#1a.
105
Schochet, Peter Z., & Chiang, H. (2010). Error rates in measuring teacher and
school performance based on student test score gains (NCEE 2010-4004).
Washington, DC: National Center for Education Evaluation and Regional
Assistance, Institute of Education Sciences, U.S. Department of Education
Smith, W. & Andrews, R. (1989). Instructional leadership: How principals
make a difference. Alexandria, VA: Association for Supervision and
Curriculum. Retrieved March 7, 2007, from
http://eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/00000
19b/80/1f/cb/a9. pdf.
Sun, S. & Young, P. (2009). How does district principal evaluation affect
learning-centered principal leadership? Evidence from Michigan School
Districts. Leadership and Policy in Schools, 8(4), 411-445.
Tekwe, C.D., Carter, R.L., Ma, C., Algina, J., Lucas, M.E., Roth, J., Ariet, M.,
Fisher, T., & Resnick, M. B. (2004). An empirical comparison of
statistical models for value-added assessment of school performance.
Journal of Educational and Behavioral Statistics, 29(1), 11-36.
Terenzini, P. (1997). Student outcomes information for policy-making. Final
report of the National Postsecondary Education Cooperative Working
Group on Student Outcomes from a Policy Perspective. Retrieved
September 24, 2008, from
http://eric.ed.gov/ERICWebPortal/custom/portlets/recordDetails/detailmin
i.jsp?_nfpb=true &_&ERICExtSearch_SearchVA.
Trends in International Mathematics and Science Study (1997, March 29). The
Economist, 25.
Trends in International Mathematics and Science Study International Study
Center (1996, November). Highlights of results from TIMSS, Third
International Mathematics and Science Study. Boston: Boston College.
Retrieved from http://timss.bc.edu/timss1995i/TIMSSPDF/P2HiLite.pdf
Tucker, M. S., & Codding, J. B. (eds.). (2002). The principal challenge: Leading
and managing schools in an era of accountability. Hoboken, NJ: Jossey-
Bass/Wiley.

United States Department of Education. (2002). No Child Left Behind Act of
2001. Retrieved from http://www.nclb.gov/next/overview/index.html.
106
United States Department of Education. (2009). . Race to the top executive
summary. Washington, D.C. Retrieved from
http://www2.ed.gov/programs/racetothetop/executive-summary.pdf.
Vanderhaar, J. E., Muñoz, M. A., & Rodosky, R. J. (2006). Leadership as
accountability for learning: The effects of school poverty, teacher
experience, previous achievement, and principal preparation programs on
student achievement. Journal of Personnel Evaluation in
Education, 19(1-2), 17-33. Retrieved March 13, 2011, from
ABI/INFORM Global. (Document ID: 1269225831).
Viadero. D. (2007, June). Teachers say NCLB has changed classroom practice.
Education Week, 26(42),6-22. Retrieved February 25, 2011, from
Research Library Core. (Document ID: 1365006401).
Wallace Foundation. (2009, March). Assessing the effectiveness of school
leaders: New directions and new processes. Perspective, 1-13. Retrieved
from www.wallacefoundation.org.

107
APPENDIX
SCHOOL EXPERIENCE SURVEY
Students
What we are learning takes a lot of thinking.
Adults at my school know my name.
My school is clean.
I feel safe on the school ground.
Parents
I feel welcome to participate at the school
The school offers me opportunities to participate in councils, parent
organizations
I talk with the teacher about my child’s school work.
My child is safe on school grounds.
Staff
I get the help I need to communicate with parents.
I am proud of this school.
My school is clean.
I feel safe on school grounds.
Source: LAUSD School Report Card (2010c, p.16).

Abstract (if available)

Abstract Recently, the federal government has pressured states to add student growth data as a part of the evaluation system. State legislatures in New York and Colorado have passed legislation to revamp teacher and principal evaluation to include student growth data. Numerous researchers acknowledged the critical impact of school principals on student achievement, and asserted the need to develop a coherent evaluation that would assess and expand the capacity of principals. ❧ This study examined accountability indicators designed to evaluate elementary school principals. A quantitative approach was used to study correlation between six accountability indicators (Academic Performance Index (API) scores, similar-school ranks, residualized difference (RD) scores, Value Added Method (VA) scores, Academic Growth over Time (AGT) scores, and school experience surveys) and three external factors (socio-economic status, mean parent education level, and percentage of English Learners). ❧ The participants included elementary schools in the Los Angeles Unified School District from 2005 to 2010. The data files were collected from the public websites, and were downloaded and analyzed. ❧ The study’s conclusions are: API scores were extremely stable, but AGT scores were not nearly as stable over the years. There was a steady growth in API over the years. Employee and student survey results were strongly correlated, but neither correlated to parent ratings. None of the school experience surveys were correlated with test-based accountability indices. API scores were strongly related to all three external factors, but VAM scores, AGT scores, RD scores, and school experience survey results were not related to them. ❧ API was found to be biased against principals who were working at a school with one or more disadvantageous external factors. Thus, using API as a part of a principal performance evaluation would be unfair. ❧ Results of this study highlights similar-school ranks, RD scores, VAM scores, AGT scores, and school experience surveys as potential accountability indicators that were unbiased and fair to be included in principal performance evaluation system. RD scores were found to be the most reliable accountability indicator. Due to their low cost and simplicity, RD scores should be considered as possible alternative to complex and expensive value-added systems.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

A comparison of value-added, orginary least squares regression, and the California Star accountability indicators

PDF

Secondary school counselor-principal relationships: impact on counselor accountability

PDF

Use of Kotter’s change model by elementary school principals in the successful implementation of inclusive education programs for students with disabilities in K-6 elementary schools in Southern ...

PDF

Data-driven decision-making practices that secondary principals use to improve student achievement

PDF

Effective leadership practices used by middle school principals in the implementation of instructional change

PDF

Efective leadership practices used by elementary school principals in the implementation of instructional change

PDF

Let's hear it from the principals: a study of four Title One elementary school principals' leadership practices on student achievement

PDF

Superintendents increase student achievement by selecting effective principals

PDF

In the implementation of standards-based reform: what is the leadership role of the principal in building school capacity and accountability to sustain student academic growth?

PDF

Comparative study of the networked principal vs. the isolated principal

PDF

Model leadership: discovering successful principals' skills, strategies and approaches for student success

PDF

The teacher evaluation: key components that positively impact teacher effectiveness and student achievement

PDF

Elementary principal perceptions of teacher to student bullying within classroom management practices

PDF

The role of principal leadership in achievement beyond test scores: an examination of leadership, differentiated curriculum and high-achieving students

PDF

A case study of promising leadership practices employed by principals of Knowledge Is Power Program Los Angeles (KIPP LA) charter school to improve student achievement

PDF

The secondary school principal's role as instructional leader in teacher professional development

PDF

Digital teacher evaluations: principal expectations of usefulness and their impact on teaching

PDF

Factors contributing to the high performance of an urban high school

PDF

Quantifying student growth: analysis of the validity of applying growth modeling to the California Standards Test

PDF

Impact of Masters in governance training on school board member practice

Asset Metadata

Creator Byun-Kitayama, Chiae J. (author)

Core Title Use of accountability indicators to evaluate elementary school principal performance

School Rossier School of Education

Degree Doctor of Education

Degree Program Education (Leadership)

Publication Date 04/09/2012

Defense Date 02/27/2012

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag accountability,elementary school principal,evaluation,OAI-PMH Harvest,performance evaluation,principal performance,school principal performance evaluation,student growth data based evaluation

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Hocevar, Dennis (committee chair), Escalante, Michael F. (committee member), García, Pedro Enrique (committee member)

Creator Email byunkita@usc.edu,cbyun1@lausd.net

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-25196

Unique identifier UC11288034

Identifier usctheses-c3-25196 (legacy record id)

Legacy Identifier etd-ByunKitaya-585.pdf

Dmrecord 25196

Document Type Dissertation

Rights Byun-Kitayama, Chiae J.

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA